\n \n \n HTMLEOF\n\n - name: Publish to ludwig-ai/schema\n uses: cpina/github-action-push-to-another-repository@v1.7.2\n env:\n SSH_DEPLOY_KEY: ${{ secrets.SCHEMA_REPO_DEPLOY_KEY }}\n with:\n source-directory: schema-out\n destination-github-username: ludwig-ai\n destination-repository-name: schema\n target-branch: main\n commit-message: \"Update Ludwig JSON schema from ${{ github.sha }}\"\n"
},
{
"path": ".github/workflows/test-results.yml",
"content": "name: test results\n\non:\n workflow_run:\n workflows: [\"pytest\"]\n types:\n - completed\n\njobs:\n test-results:\n name: Test Results\n runs-on: ubuntu-latest\n if: github.event.workflow_run.conclusion != 'skipped'\n\n steps:\n - name: Download and Extract Artifacts\n env:\n GITHUB_TOKEN: ${{secrets.GITHUB_TOKEN}}\n run: |\n mkdir -p artifacts && cd artifacts\n\n artifacts_url=${{ github.event.workflow_run.artifacts_url }}\n\n gh api \"$artifacts_url\" -q '.artifacts[] | [.name, .archive_download_url] | @tsv' | while read artifact\n do\n IFS=$'\\t' read name url <<< \"$artifact\"\n gh api $url > \"$name.zip\"\n unzip -d \"$name\" \"$name.zip\"\n done\n\n - name: Publish Unit Test Results\n uses: EnricoMi/publish-unit-test-result-action@v2\n with:\n commit: ${{ github.event.workflow_run.head_sha }}\n event_file: artifacts/Event File/event.json\n event_name: ${{ github.event.workflow_run.event }}\n files: \"artifacts/**/*.xml\"\n"
},
{
"path": ".github/workflows/upload-pypi.yml",
"content": "name: Upload to PyPI\n\non:\n # Triggers the workflow when a release or draft of a release is published,\n # or a pre-release is changed to a release\n release:\n types: [released]\n # Allows you to run this workflow manually from the Actions tab\n workflow_dispatch:\n\njobs:\n pypi-publish:\n name: upload release to PyPI\n runs-on: ubuntu-latest\n # Specifying a GitHub environment is optional, but strongly encouraged\n environment: release\n permissions:\n # IMPORTANT: this permission is mandatory for trusted publishing\n id-token: write\n steps:\n - name: Checkout\n uses: actions/checkout@v4\n with:\n submodules: \"recursive\"\n\n - uses: actions/setup-python@v5\n with:\n python-version: \"3.12\"\n\n - name: Build source distribution\n run: |\n pip install setuptools\n python setup.py sdist\n\n - name: Publish package distributions to PyPI\n uses: pypa/gh-action-pypi-publish@release/v1\n"
},
{
"path": ".gitignore",
"content": "###################\n# ludwig specific #\n###################\n\n*.lock_preprocessing\nresults/\nludwig/results/\nresults_*/\nludwig_arm64/\n\n# ailabs-utils\nailabs_util\ndocker_assets\n\n# data\nmnist_data/\nprofile_images/\n./profile_images/\n\n###########\n# General #\n###########\n\n# Mac stuff\n.DS_Store\n\n# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\nenv/\nenv*\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\n./downloads/\n./dataset/\neggs/\n.eggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\nwheels/\n*.egg-info/\n.installed.cfg\n*.egg\n\n# PyInstaller\n# Usually these files are written by a python script from a template\n# before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*,cover\n.hypothesis/\n\n# Data\n*.csv\n*.hdf5\n*.meta.json\n*.parquet\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\nlocal_settings.py\n\n# Flask stuff:\ninstance/\n.webassets-cache\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\ntarget/\n\n# Jupyter Notebook\n.ipynb_checkpoints\n\n# pyenv\n.python-version\n\n# celery beat schedule file\ncelerybeat-schedule\n\n# SageMath parsed files\n*.sage.py\n\n# dotenv\n.env\n\n# virtualenv\n.venv\nvenv*\nENV/\n\n# Spyder project settings\n.spyderproject\n\n# Rope project settings\n.ropeproject\n\n# PyCharm\n.idea\n\n# ctags\ntags\n\n# examples\nexamples/*/data/\nexamples/*/visualizations/\n\n# benchmarking configs\nludwig/benchmarking/configs/\n\n# Aim tracking\n.aim/\n\n# Comet\n.comet.config\n\n# Test-generated artifacts (image/audio features)\n*.png\n*.wav\ngenerated_audio/\ngenerated_images/\n"
},
{
"path": ".nojekyll",
"content": ""
},
{
"path": ".pre-commit-config.yaml",
"content": "# Apply to all files without committing:\n# pre-commit run --all-files\n# Apply to changed files:\n# pre-commit run\n# Update this file:\n# pre-commit autoupdate\n# Run a specific hook:\n# pre-commit run \n\nci:\n autofix_prs: true\n autoupdate_commit_msg: \"[pre-commit.ci] pre-commit suggestions\"\n autoupdate_schedule: weekly\n\nrepos:\n - repo: https://github.com/pre-commit/pre-commit-hooks\n rev: v6.0.0\n hooks:\n - id: check-ast\n - id: fix-byte-order-marker\n - id: check-case-conflict\n - id: check-executables-have-shebangs\n - id: check-json\n - id: check-toml\n - id: check-yaml\n - id: debug-statements\n - id: detect-private-key\n - id: end-of-file-fixer\n - id: trailing-whitespace\n - id: mixed-line-ending\n - repo: https://github.com/asottile/pyupgrade\n rev: v3.21.2\n hooks:\n - id: pyupgrade\n args: [--py312-plus]\n - repo: https://github.com/PyCQA/docformatter\n rev: v1.7.7\n hooks:\n - id: docformatter\n args: [--in-place, --wrap-summaries=115, --wrap-descriptions=120]\n - repo: https://github.com/PyCQA/isort\n rev: 8.0.0\n hooks:\n - id: isort\n name: Format imports\n - repo: https://github.com/pycqa/flake8\n rev: 7.3.0\n hooks:\n - id: flake8\n - repo: https://github.com/psf/black\n rev: 26.1.0\n hooks:\n - id: black\n name: Format code\n - repo: https://github.com/asottile/blacken-docs\n rev: 1.20.0\n hooks:\n - id: blacken-docs\n args: [--line-length=120]\n - repo: https://github.com/hukkin/mdformat\n rev: 1.0.0\n hooks:\n - id: mdformat\n additional_dependencies:\n - mdformat-gfm==1.0.0\n - mdformat_frontmatter==2.0.10\n exclude: CHANGELOG.md\n - repo: https://github.com/yoheimuta/protolint\n rev: v0.56.4\n hooks:\n - id: protolint\n"
},
{
"path": ".protolint.yaml",
"content": "# Adapted from\n# https://github.com/yoheimuta/protolint/blob/master/_example/config/.protolint.yaml\n---\n# Lint directives.\nlint:\n # Linter files to walk.\n files:\n # The specific files to exclude.\n exclude:\n # NOTE: UNIX paths will be properly accepted by both UNIX and Windows.\n - ../proto/invalidFileName.proto\n\n # Linter rules.\n # Run `protolint list` to see all available rules.\n rules:\n # Set the default to all linters. This option works the other way around as no_default does.\n # If you want to enable this option, delete the comment out below and no_default.\n # all_default: true\n\n # The specific linters to add.\n add:\n - FIELD_NAMES_LOWER_SNAKE_CASE\n - MESSAGE_NAMES_UPPER_CAMEL_CASE\n - MAX_LINE_LENGTH\n - INDENT\n - FIELD_NAMES_EXCLUDE_PREPOSITIONS\n - FILE_NAMES_LOWER_SNAKE_CASE\n - IMPORTS_SORTED\n - PACKAGE_NAME_LOWER_CASE\n - ORDER\n - PROTO3_FIELDS_AVOID_REQUIRED\n - PROTO3_GROUPS_AVOID\n - REPEATED_FIELD_NAMES_PLURALIZED\n - QUOTE_CONSISTENT\n\n # Linter rules option.\n rules_option:\n # MAX_LINE_LENGTH rule option.\n max_line_length:\n # Enforces a maximum line length\n max_chars: 120\n # Specifies the character count for tab characters\n tab_chars: 2\n\n # FILE_NAMES_LOWER_SNAKE_CASE rule option.\n file_names_lower_snake_case:\n excludes:\n - ../proto/invalidFileName.proto\n\n # QUOTE_CONSISTENT rule option.\n quote_consistent:\n # Available quote are \"double\" or \"single\".\n quote: double\n"
},
{
"path": ".vscode/settings.json",
"content": "{\n \"editor.rulers\": [\n 120\n ],\n \"editor.formatOnSave\": true,\n \"[python]\": {\n \"editor.defaultFormatter\": \"ms-python.black-formatter\",\n \"editor.formatOnSave\": true\n },\n \"black-formatter.args\": [\n \"--line-length\",\n \"120\"\n ],\n \"flake8.args\": [\n \"--config=setup.cfg\"\n ],\n \"python.testing.unittestEnabled\": false,\n \"python.testing.pytestEnabled\": false,\n \"python.envFile\": \"${workspaceFolder}/.env\"\n}\n"
},
{
"path": "CODEOWNERS",
"content": "# Default code owners for the entire repository\n* @w4nderlust @tgaddair @justinxzhao @arnavgarg1 @geoffreyangus @jeffkinnison @Infernaught @alexsherstinsky\n"
},
{
"path": "CODE_OF_CONDUCT.md",
"content": "# Code of conduct\n\nLudwig adopts the [Linux Foundation code of conduct](https://lfprojects.org/policies/code-of-conduct/).\n"
},
{
"path": "CONTRIBUTING.md",
"content": "# Contributing\n\nEveryone is welcome to contribute, and we value everybody’s contribution. Code is thus not the only\nway to help the community. Answering questions, helping others, reaching out and improving the\ndocumentation are immensely valuable contributions as well.\n\nIt also helps us if you spread the word: reference the library from blog posts on the awesome\nprojects it made possible, shout out on X every time it has helped you, or simply star the\nrepo to say \"thank you\".\n\nCheck out the official [ludwig docs](https://ludwig-ai.github.io/ludwig-docs/) to get oriented\naround the codebase, and join the community!\n\n## Open Issues\n\nIssues are listed at: \n\nIf you would like to work on any of them, make sure it is not already assigned to someone else.\n\nYou can self-assign it by commenting on the Issue page with one of the keywords: `#take` or\n`#self-assign`.\n\nWork on your self-assigned issue and eventually create a Pull Request.\n\n## Creating Pull Requests\n\n1. Fork the [repository](https://github.com/ludwig-ai/ludwig) by clicking on the \"Fork\" button on\n the repository's page. This creates a copy of the code under your GitHub user account.\n\n1. Clone your fork to your local disk, and add the base repository as a remote:\n\n ```bash\n git clone git@github.com:/ludwig.git\n cd ludwig\n git remote add upstream https://github.com/ludwig-ai/ludwig.git\n ```\n\n1. Create a new branch to hold your development changes:\n\n ```bash\n git checkout -b a-descriptive-name-for-my-changes\n ```\n\n *Do not*\\* work on the `master` branch.\n\n1. Set up a development environment by running the following command in a virtual environment:\n\n ```bash\n pip install -e .\n ```\n\n The above command will install only the packages in \"requirements.txt\" in the developer mode. If you would like to\n be able to potentially make changes to the overall Ludwig codebase, then use the following command:\n\n ```bash\n pip install -e .[full]\n ```\n\n Please note that in certain Shell environments (e.g., the `Z shell`), the dependencies in brackets have to be quoted:\n\n ```bash\n pip install -e .\"[full]\"\n ```\n\n If you do not need access to the entire Ludwig codebase, but just want to be able to run `pytest` on the essential\n functionality, then you would replace the above command with:\n\n ```bash\n pip install -e .[test]\n ```\n\n (Please use `pip install -e .\"[test]\"` where your Shell environment requires quotes around the square brackets.)\n\n For the full list of the optional dependencies available in Ludwig, please see\n [Installation Guide](https://ludwig.ai/latest/getting_started/installation/) and \"setup.py\" in the root of the Ludwig\n repository.\n\n1. On MacOS with Apple Silicon, if this installation approach runs into errors, you may need to install the following\n prerequisites:\n\n ```bash\n brew install cmake libomp\n ```\n\n This step requires `homebrew` to be installed on your development machine.\n\n1. Install and run `pre-commit`:\n\n ```bash\n pip install pre-commit\n pre-commit install\n ```\n\n1. Develop features on your branch.\n\n1. Format your code by running pre-commits so that your newly added files look nice:\n\n ```bash\n pre-commit run\n ```\n\n Pre-commits also run automatically when committing.\n\n1. Once you're happy with your changes, make a commit to record your changes locally:\n\n ```bash\n git add .\n git commit\n ```\n\n It is a good idea to sync your copy of the code with the original repository regularly. This\n way you can quickly account for changes:\n\n ```bash\n git fetch upstream\n git rebase upstream/master\n ```\n\n Push the changes to your account using:\n\n ```bash\n git push -u origin a-descriptive-name-for-my-changes\n ```\n\n1. Once you are satisfied, go the webpage of your fork on GitHub. Click on \"Pull request\" to send\n your contribution to the project maintainers for review.\n\n## Other tips\n\n- Add unit tests for any new code you write.\n- Make sure tests pass. See the [Developer Guide](https://ludwig-ai.github.io/ludwig-docs/latest/developer_guide/style_guidelines_and_tests/) for more details.\n\n## Attribution\n\nThis contributing guideline is adapted from `huggingface`, available at .\n\n## Code of Conduct\n\nPlease be mindful of and adhere to the Linux Foundation's\n[Code of Conduct](https://lfprojects.org/policies/code-of-conduct) when contributing to Ludwig.\n"
},
{
"path": "LICENSE",
"content": " Apache License\n Version 2.0, January 2004\n http://www.apache.org/licenses/\n\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n 1. Definitions.\n\n \"License\" shall mean the terms and conditions for use, reproduction,\n and distribution as defined by Sections 1 through 9 of this document.\n\n \"Licensor\" shall mean the copyright owner or entity authorized by\n the copyright owner that is granting the License.\n\n \"Legal Entity\" shall mean the union of the acting entity and all\n other entities that control, are controlled by, or are under common\n control with that entity. For the purposes of this definition,\n \"control\" means (i) the power, direct or indirect, to cause the\n direction or management of such entity, whether by contract or\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\n outstanding shares, or (iii) beneficial ownership of such entity.\n\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\n exercising permissions granted by this License.\n\n \"Source\" form shall mean the preferred form for making modifications,\n including but not limited to software source code, documentation\n source, and configuration files.\n\n \"Object\" form shall mean any form resulting from mechanical\n transformation or translation of a Source form, including but\n not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n \"Work\" shall mean the work of authorship, whether in Source or\n Object form, made available under the License, as indicated by a\n copyright notice that is included in or attached to the work\n (an example is provided in the Appendix below).\n\n \"Derivative Works\" shall mean any work, whether in Source or Object\n form, that is based on (or derived from) the Work and for which the\n editorial revisions, annotations, elaborations, or other modifications\n represent, as a whole, an original work of authorship. For the purposes\n of this License, Derivative Works shall not include works that remain\n separable from, or merely link (or bind by name) to the interfaces of,\n the Work and Derivative Works thereof.\n\n \"Contribution\" shall mean any work of authorship, including\n the original version of the Work and any modifications or additions\n to that Work or Derivative Works thereof, that is intentionally\n submitted to Licensor for inclusion in the Work by the copyright owner\n or by an individual or Legal Entity authorized to submit on behalf of\n the copyright owner. For the purposes of this definition, \"submitted\"\n means any form of electronic, verbal, or written communication sent\n to the Licensor or its representatives, including but not limited to\n communication on electronic mailing lists, source code control systems,\n and issue tracking systems that are managed by, or on behalf of, the\n Licensor for the purpose of discussing and improving the Work, but\n excluding communication that is conspicuously marked or otherwise\n designated in writing by the copyright owner as \"Not a Contribution.\"\n\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\n on behalf of whom a Contribution has been received by Licensor and\n subsequently incorporated within the Work.\n\n 2. Grant of Copyright License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n copyright license to reproduce, prepare Derivative Works of,\n publicly display, publicly perform, sublicense, and distribute the\n Work and such Derivative Works in Source or Object form.\n\n 3. Grant of Patent License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n (except as stated in this section) patent license to make, have made,\n use, offer to sell, sell, import, and otherwise transfer the Work,\n where such license applies only to those patent claims licensable\n by such Contributor that are necessarily infringed by their\n Contribution(s) alone or by combination of their Contribution(s)\n with the Work to which such Contribution(s) was submitted. If You\n institute patent litigation against any entity (including a\n cross-claim or counterclaim in a lawsuit) alleging that the Work\n or a Contribution incorporated within the Work constitutes direct\n or contributory patent infringement, then any patent licenses\n granted to You under this License for that Work shall terminate\n as of the date such litigation is filed.\n\n 4. Redistribution. You may reproduce and distribute copies of the\n Work or Derivative Works thereof in any medium, with or without\n modifications, and in Source or Object form, provided that You\n meet the following conditions:\n\n (a) You must give any other recipients of the Work or\n Derivative Works a copy of this License; and\n\n (b) You must cause any modified files to carry prominent notices\n stating that You changed the files; and\n\n (c) You must retain, in the Source form of any Derivative Works\n that You distribute, all copyright, patent, trademark, and\n attribution notices from the Source form of the Work,\n excluding those notices that do not pertain to any part of\n the Derivative Works; and\n\n (d) If the Work includes a \"NOTICE\" text file as part of its\n distribution, then any Derivative Works that You distribute must\n include a readable copy of the attribution notices contained\n within such NOTICE file, excluding those notices that do not\n pertain to any part of the Derivative Works, in at least one\n of the following places: within a NOTICE text file distributed\n as part of the Derivative Works; within the Source form or\n documentation, if provided along with the Derivative Works; or,\n within a display generated by the Derivative Works, if and\n wherever such third-party notices normally appear. The contents\n of the NOTICE file are for informational purposes only and\n do not modify the License. You may add Your own attribution\n notices within Derivative Works that You distribute, alongside\n or as an addendum to the NOTICE text from the Work, provided\n that such additional attribution notices cannot be construed\n as modifying the License.\n\n You may add Your own copyright statement to Your modifications and\n may provide additional or different license terms and conditions\n for use, reproduction, or distribution of Your modifications, or\n for any such Derivative Works as a whole, provided Your use,\n reproduction, and distribution of the Work otherwise complies with\n the conditions stated in this License.\n\n 5. Submission of Contributions. Unless You explicitly state otherwise,\n any Contribution intentionally submitted for inclusion in the Work\n by You to the Licensor shall be under the terms and conditions of\n this License, without any additional terms or conditions.\n Notwithstanding the above, nothing herein shall supersede or modify\n the terms of any separate license agreement you may have executed\n with Licensor regarding such Contributions.\n\n 6. Trademarks. This License does not grant permission to use the trade\n names, trademarks, service marks, or product names of the Licensor,\n except as required for reasonable and customary use in describing the\n origin of the Work and reproducing the content of the NOTICE file.\n\n 7. Disclaimer of Warranty. Unless required by applicable law or\n agreed to in writing, Licensor provides the Work (and each\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n implied, including, without limitation, any warranties or conditions\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n PARTICULAR PURPOSE. You are solely responsible for determining the\n appropriateness of using or redistributing the Work and assume any\n risks associated with Your exercise of permissions under this License.\n\n 8. Limitation of Liability. In no event and under no legal theory,\n whether in tort (including negligence), contract, or otherwise,\n unless required by applicable law (such as deliberate and grossly\n negligent acts) or agreed to in writing, shall any Contributor be\n liable to You for damages, including any direct, indirect, special,\n incidental, or consequential damages of any character arising as a\n result of this License or out of the use or inability to use the\n Work (including but not limited to damages for loss of goodwill,\n work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses), even if such Contributor\n has been advised of the possibility of such damages.\n\n 9. Accepting Warranty or Additional Liability. While redistributing\n the Work or Derivative Works thereof, You may choose to offer,\n and charge a fee for, acceptance of support, warranty, indemnity,\n or other liability obligations and/or rights consistent with this\n License. However, in accepting such obligations, You may act only\n on Your own behalf and on Your sole responsibility, not on behalf\n of any other Contributor, and only if You agree to indemnify,\n defend, and hold each Contributor harmless for any liability\n incurred by, or claims asserted against, such Contributor by reason\n of your accepting any such warranty or additional liability.\n\n END OF TERMS AND CONDITIONS\n\n APPENDIX: How to apply the Apache License to your work.\n\n To apply the Apache License to your work, attach the following\n boilerplate notice, with the fields enclosed by brackets \"[]\"\n replaced with your own identifying information. (Don't include\n the brackets!) The text should be enclosed in the appropriate\n comment syntax for the file format. We also recommend that a\n file or class name and description of purpose be included on the\n same \"printed page\" as the copyright notice for easier\n identification within third-party archives.\n\n Copyright [yyyy] [name of copyright owner]\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n\n--------------------------------------------------------------------------\n\nCode in ludwig/api_annotations.py adapted from\nhttps://github.com/ray-project/ray (Apache-2.0 License)\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n https://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n\n--------------------------------------------------------------------------\n\nCode in ludwig/utils/structural_warnings.py adapted from\nhttps://github.com/ray-project/ray (Apache-2.0 License)\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n https://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n\n--------------------------------------------------------------------------\n\nCode in ludwig/utils/logging_utils.py adapted from\nhttps://github.com/ray-project/ray (Apache-2.0 License)\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n https://www.apache.org/licenses/LICENSE-2.0\n\nUnless required by applicable law or agreed to in writing, software\ndistributed under the License is distributed on an \"AS IS\" BASIS,\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\nSee the License for the specific language governing permissions and\nlimitations under the License.\n"
},
{
"path": "MANIFEST.in",
"content": "include *.txt\nrecursive-include ludwig/datasets *.yaml\nrecursive-include ludwig/automl/defaults *.yaml\nrecursive-include ludwig/schema/metadata/configs *.yaml\n"
},
{
"path": "NOTICE",
"content": "Ludwig includes derived work from TensorFlow(https://github.com/tensorflow/tensorflow) under the Apache License 2.0:\n\nCopyright 2016 The prometheus-operator Authors\n\n\nLicensed under the Apache License, Version 2.0 (the \"License\");\n\nyou may not use this file except in compliance with the License.\n\nYou may obtain a copy of the License at\n\n\nhttp://www.apache.org/licenses/LICENSE-2.0\n\n\nUnless required by applicable law or agreed to in writing, software\n\ndistributed under the License is distributed on an \"AS IS\" BASIS,\n\nWITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n\nSee the License for the specific language governing permissions and\n\nlimitations under the License.\n\n\nThe derived work can be found in the files: ludwig/models/modules/convolutional_modules.py\n\n\n------\n\nLudwig includes derived work from Keras(https://github.com/keras-team/keras) under the MIT License:\n\n\nCOPYRIGHT\n\nAll contributions by François Chollet:\n\nCopyright (c) 2015 - 2018, François Chollet.\n\nAll rights reserved.\n\n\nAll contributions by Google:\n\nCopyright (c) 2015 - 2018, Google, Inc.\n\nAll rights reserved.\n\n\nAll contributions by Microsoft:\n\nCopyright (c) 2017 - 2018, Microsoft, Inc.\n\nAll rights reserved.\n\n\nAll other contributions:\n\nCopyright (c) 2015 - 2018, the respective contributors.\n\nAll rights reserved.\n\n\nEach contributor holds copyright over their respective contributions.\n\nThe project versioning (Git) records all such contribution source information.\n\nLICENSE\n\nThe MIT License (MIT)\n\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\n\nof this software and associated documentation files (the \"Software\"), to deal\n\nin the Software without restriction, including without limitation the rights\n\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n\ncopies of the Software, and to permit persons to whom the Software is\n\nfurnished to do so, subject to the following conditions:\n\n\nThe above copyright notice and this permission notice shall be included in all\n\ncopies or substantial portions of the Software.\n\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n\nSOFTWARE.\n\n\nThe derived work can be found in the files: mkdocs/code_docs_autogen.py\n"
},
{
"path": "README.md",
"content": "
\n\n_Declarative deep learning framework built for scale and efficiency._\n\n[](https://badge.fury.io/py/ludwig)\n[](https://discord.gg/CBgdrGnZjy)\n[](https://hub.docker.com/r/ludwigai)\n[](https://pepy.tech/project/ludwig)\n[](https://github.com/ludwig-ai/ludwig/blob/main/LICENSE)\n[](https://twitter.com/ludwig_ai)\n\n
\n\n# 📖 What is Ludwig?\n\nLudwig is a **low-code** framework for building **custom** AI models like **LLMs** and other deep neural networks.\n\nKey features:\n\n- 🛠 **Build custom models with ease:** a declarative YAML configuration file is all you need to train a state-of-the-art LLM on your data. Support for multi-task and multi-modality learning. Comprehensive config validation detects invalid parameter combinations and prevents runtime failures.\n- ⚡ **Optimized for scale and efficiency:** automatic batch size selection, distributed training ([DDP](https://pytorch.org/tutorials/beginner/ddp_series_theory.html), [DeepSpeed](https://github.com/microsoft/DeepSpeed)), parameter efficient fine-tuning ([PEFT](https://github.com/huggingface/peft)), 4-bit quantization (QLoRA), paged and 8-bit optimizers, and larger-than-memory datasets.\n- 📐 **Expert level control:** retain full control of your models down to the activation functions. Support for hyperparameter optimization, explainability, and rich metric visualizations.\n- 🧱 **Modular and extensible:** experiment with different model architectures, tasks, features, and modalities with just a few parameter changes in the config. Think building blocks for deep learning.\n- 🚢 **Engineered for production:** prebuilt [Docker](https://hub.docker.com/u/ludwigai) containers, native support for running with [Ray](https://www.ray.io/) on [Kubernetes](https://github.com/ray-project/kuberay), export models to [Torchscript](https://pytorch.org/docs/stable/jit.html) and [Triton](https://developer.nvidia.com/triton-inference-server), upload to [HuggingFace](https://huggingface.co/models) with one command.\n\nLudwig is hosted by the\n[Linux Foundation AI & Data](https://lfaidata.foundation/).\n\n**Tech stack:** Python 3.12 | PyTorch 2.6 | Pydantic 2 | Transformers 5 | Ray 2.54\n\n\n\n# 💾 Installation\n\nInstall from PyPI. Be aware that Ludwig requires Python 3.12+.\n\n```shell\npip install ludwig\n```\n\nOr install with all optional dependencies:\n\n```shell\npip install ludwig[full]\n```\n\nPlease see [contributing](https://github.com/ludwig-ai/ludwig/blob/main/CONTRIBUTING.md) for more detailed installation instructions.\n\n# 🚂 Getting Started\n\nWant to take a quick peek at some of Ludwig's features? Check out this Colab Notebook 🚀 [](https://colab.research.google.com/drive/1lB4ALmEyvcMycE3Mlnsd7I3bc0zxvk39)\n\nLooking to fine-tune LLMs? Check out these notebooks:\n\n1. Fine-Tune Llama-2-7b: [](https://colab.research.google.com/drive/1r4oSEwRJpYKBPM0M0RSh0pBEYK_gBKbe)\n1. Fine-Tune Llama-2-13b: [](https://colab.research.google.com/drive/1zmSEzqZ7v4twBrXagj1TE_C--RNyVAyu)\n1. Fine-Tune Mistral-7b: [](https://colab.research.google.com/drive/1i_8A1n__b7ljRWHzIsAdhO7u7r49vUm4)\n\nFor a full tutorial, check out the official [getting started guide](https://ludwig.ai/latest/getting_started/), or take a look at end-to-end [Examples](https://ludwig.ai/latest/examples).\n\n## Large Language Model Fine-Tuning\n\n[](https://colab.research.google.com/drive/1c3AO8l_H6V_x37RwQ8V7M6A-RmcBf2tG?usp=sharing)\n\nLet's fine-tune a pretrained LLM to follow instructions like a chatbot (\"instruction tuning\").\n\n### Prerequisites\n\n- [HuggingFace API Token](https://huggingface.co/docs/hub/security-tokens)\n- Access approval to your chosen base model (e.g., [Llama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B))\n- GPU with at least 12 GiB of VRAM (in our tests, we used an Nvidia T4)\n\n### Running\n\nWe'll use the [Stanford Alpaca](https://crfm.stanford.edu/2023/03/13/alpaca.html) dataset, which will be formatted as a table-like file that looks like this:\n\n| instruction | input | output |\n| :-----------------------------------------------: | :--------------: | :-----------------------------------------------: |\n| Give three tips for staying healthy. | | 1.Eat a balanced diet and make sure to include... |\n| Arrange the items given below in the order to ... | cake, me, eating | I eating cake. |\n| Write an introductory paragraph about a famous... | Michelle Obama | Michelle Obama is an inspirational woman who r... |\n| ... | ... | ... |\n\nCreate a YAML config file named `model.yaml` with the following:\n\n```yaml\nmodel_type: llm\nbase_model: meta-llama/Llama-3.1-8B\n\nquantization:\n bits: 4\n\nadapter:\n type: lora\n\nprompt:\n template: |\n Below is an instruction that describes a task, paired with an input that may provide further context.\n Write a response that appropriately completes the request.\n\n ### Instruction:\n {instruction}\n\n ### Input:\n {input}\n\n ### Response:\n\ninput_features:\n - name: prompt\n type: text\n\noutput_features:\n - name: output\n type: text\n\ntrainer:\n type: finetune\n learning_rate: 0.0001\n batch_size: 1\n gradient_accumulation_steps: 16\n epochs: 3\n learning_rate_scheduler:\n decay: cosine\n warmup_fraction: 0.01\n\npreprocessing:\n sample_ratio: 0.1\n\nbackend:\n type: local\n```\n\nAnd now let's train the model:\n\n```bash\nexport HUGGING_FACE_HUB_TOKEN = \"\"\n\nludwig train --config model.yaml --dataset \"ludwig://alpaca\"\n```\n\n## Supervised ML\n\nLet's build a neural network that predicts whether a given movie critic's review on [Rotten Tomatoes](https://www.kaggle.com/stefanoleone992/rotten-tomatoes-movies-and-critic-reviews-dataset) was positive or negative.\n\nOur dataset will be a CSV file that looks like this:\n\n| movie_title | content_rating | genres | runtime | top_critic | review_content | recommended |\n| :------------------: | :------------: | :------------------------------: | :-----: | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |\n| Deliver Us from Evil | R | Action & Adventure, Horror | 117.0 | TRUE | Director Scott Derrickson and his co-writer, Paul Harris Boardman, deliver a routine procedural with unremarkable frights. | 0 |\n| Barbara | PG-13 | Art House & International, Drama | 105.0 | FALSE | Somehow, in this stirring narrative, Barbara manages to keep hold of her principles, and her humanity and courage, and battles to save a dissident teenage girl whose life the Communists are trying to destroy. | 1 |\n| Horrible Bosses | R | Comedy | 98.0 | FALSE | These bosses cannot justify either murder or lasting comic memories, fatally compromising a farce that could have been great but ends up merely mediocre. | 0 |\n| ... | ... | ... | ... | ... | ... | ... |\n\nDownload a sample of the dataset from [here](https://ludwig.ai/latest/data/rotten_tomatoes.csv).\n\n```bash\nwget https://ludwig.ai/latest/data/rotten_tomatoes.csv\n```\n\nNext create a YAML config file named `model.yaml` with the following:\n\n```yaml\ninput_features:\n - name: genres\n type: set\n preprocessing:\n tokenizer: comma\n - name: content_rating\n type: category\n - name: top_critic\n type: binary\n - name: runtime\n type: number\n - name: review_content\n type: text\n encoder:\n type: embed\noutput_features:\n - name: recommended\n type: binary\n```\n\nThat's it! Now let's train the model:\n\n```bash\nludwig train --config model.yaml --dataset rotten_tomatoes.csv\n```\n\n**Happy modeling**\n\nTry applying Ludwig to your data. [Reach out on Discord](https://discord.gg/CBgdrGnZjy)\nif you have any questions.\n\n# ❓ Why you should use Ludwig\n\n- **Minimal machine learning boilerplate**\n\n Ludwig takes care of the engineering complexity of machine learning out of\n the box, enabling research scientists to focus on building models at the\n highest level of abstraction. Data preprocessing, hyperparameter\n optimization, device management, and distributed training for\n `torch.nn.Module` models come completely free.\n\n- **Easily build your benchmarks**\n\n Creating a state-of-the-art baseline and comparing it with a new model is a\n simple config change.\n\n- **Easily apply new architectures to multiple problems and datasets**\n\n Apply new models across the extensive set of tasks and datasets that Ludwig\n supports. Ludwig includes a\n [full benchmarking toolkit](https://arxiv.org/abs/2111.04260) accessible to\n any user, for running experiments with multiple models across multiple\n datasets with just a simple configuration.\n\n- **Highly configurable data preprocessing, modeling, and metrics**\n\n Any and all aspects of the model architecture, training loop, hyperparameter\n search, and backend infrastructure can be modified as additional fields in\n the declarative configuration to customize the pipeline to meet your\n requirements. For details on what can be configured, check out\n [Ludwig Configuration](https://ludwig.ai/latest/configuration/)\n docs.\n\n- **Multi-modal, multi-task learning out-of-the-box**\n\n Mix and match tabular data, text, images, and even audio into complex model\n configurations without writing code.\n\n- **Rich model exporting and tracking**\n\n Automatically track all trials and metrics with tools like Tensorboard,\n Comet ML, Weights & Biases, MLFlow, and Aim Stack.\n\n- **Automatically scale training to multi-GPU, multi-node clusters**\n\n Go from training on your local machine to the cloud without code changes.\n\n- **Low-code interface for state-of-the-art models, including pre-trained Huggingface Transformers**\n\n Ludwig also natively integrates with pre-trained models, such as the ones\n available in [Huggingface Transformers](https://huggingface.co/docs/transformers/index).\n Users can choose from a vast collection of state-of-the-art pre-trained\n PyTorch models to use without needing to write any code at all. For example,\n training a BERT-based sentiment analysis model with Ludwig is as simple as:\n\n ```shell\n ludwig train --dataset sst5 --config_str \"{input_features: [{name: sentence, type: text, encoder: bert}], output_features: [{name: label, type: category}]}\"\n ```\n\n- **Low-code interface for AutoML**\n\n [Ludwig AutoML](https://ludwig.ai/latest/user_guide/automl/)\n allows users to obtain trained models by providing just a dataset, the\n target column, and a time budget.\n\n ```python\n auto_train_results = ludwig.automl.auto_train(dataset=my_dataset_df, target=target_column_name, time_limit_s=7200)\n ```\n\n- **Easy productionisation**\n\n Ludwig makes it easy to serve deep learning models, including on GPUs.\n Launch a REST API for your trained Ludwig model.\n\n ```shell\n ludwig serve --model_path=/path/to/model\n ```\n\n Ludwig supports exporting models to efficient Torchscript bundles.\n\n ```shell\n ludwig export_torchscript -–model_path=/path/to/model\n ```\n\n# 📚 Tutorials\n\n- [Text Classification](https://ludwig.ai/latest/examples/text_classification)\n- [Tabular Data Classification](https://ludwig.ai/latest/examples/adult_census_income)\n- [Image Classification](https://ludwig.ai/latest/examples/mnist)\n- [Multimodal Classification](https://ludwig.ai/latest/examples/multimodal_classification)\n\n# 🔬 Example Use Cases\n\n- [Named Entity Recognition Tagging](https://ludwig.ai/latest/examples/ner_tagging)\n- [Natural Language Understanding](https://ludwig.ai/latest/examples/nlu)\n- [Machine Translation](https://ludwig.ai/latest/examples/machine_translation)\n- [Chit-Chat Dialogue Modeling through seq2seq](https://ludwig.ai/latest/examples/seq2seq)\n- [Sentiment Analysis](https://ludwig.ai/latest/examples/sentiment_analysis)\n- [One-shot Learning with Siamese Networks](https://ludwig.ai/latest/examples/oneshot)\n- [Visual Question Answering](https://ludwig.ai/latest/examples/visual_qa)\n- [Spoken Digit Speech Recognition](https://ludwig.ai/latest/examples/speech_recognition)\n- [Speaker Verification](https://ludwig.ai/latest/examples/speaker_verification)\n- [Binary Classification (Titanic)](https://ludwig.ai/latest/examples/titanic)\n- [Timeseries forecasting](https://ludwig.ai/latest/examples/forecasting)\n- [Timeseries forecasting (Weather)](https://ludwig.ai/latest/examples/weather)\n- [Movie rating prediction](https://ludwig.ai/latest/examples/movie_ratings)\n- [Multi-label classification](https://ludwig.ai/latest/examples/multi_label)\n- [Multi-Task Learning](https://ludwig.ai/latest/examples/multi_task)\n- [Simple Regression: Fuel Efficiency Prediction](https://ludwig.ai/latest/examples/fuel_efficiency)\n- [Fraud Detection](https://ludwig.ai/latest/examples/fraud)\n\n# 💡 More Information\n\nRead our publications on [Ludwig](https://arxiv.org/pdf/1909.07930.pdf), [declarative ML](https://arxiv.org/pdf/2107.08148.pdf), and [Ludwig’s SoTA benchmarks](https://openreview.net/pdf?id=hwjnu6qW7E4).\n\nLearn more about [how Ludwig works](https://ludwig.ai/latest/user_guide/how_ludwig_works/), [how to get started](https://ludwig.ai/latest/getting_started/), and work through more [examples](https://ludwig.ai/latest/examples).\n\nIf you are interested in [contributing](https://github.com/ludwig-ai/ludwig/blob/main/CONTRIBUTING.md), have questions, comments, or thoughts to share, or if you just want to be in the\nknow, please consider [joining our Community Discord](https://discord.gg/CBgdrGnZjy) and follow us on [X](https://twitter.com/ludwig_ai)!\n\n# 🤝 Join the community to build Ludwig with us\n\nLudwig is an actively managed open-source project that relies on contributions from folks just like\nyou. Consider joining the active group of Ludwig contributors to make Ludwig an even\nmore accessible and feature rich framework for everyone to use!\n\n\n \n \n\n## Star History\n\n[](https://star-history.com/#ludwig-ai/ludwig&Date)\n\n# 👋 Getting Involved\n\n- [Discord](https://discord.gg/CBgdrGnZjy)\n- [X (Twitter)](https://twitter.com/ludwig_ai)\n- [Medium](https://medium.com/ludwig-ai)\n- [GitHub Issues](https://github.com/ludwig-ai/ludwig/issues)\n"
},
{
"path": "README_KR.md",
"content": "
\n\n_확장성과 효율성을 위해 설계된 선언적 딥러닝 프레임워크_\n\n[](https://badge.fury.io/py/ludwig)\n[](https://discord.gg/CBgdrGnZjy)\n[](https://hub.docker.com/r/ludwigai)\n[](https://pepy.tech/project/ludwig)\n[](https://github.com/ludwig-ai/ludwig/blob/main/LICENSE)\n[](https://twitter.com/ludwig_ai)\n\n
\n\n# 📖 Ludwig란?\n\nLudwig는 **LLM** 및 기타 심층 신경망과 같은 **맞춤형** AI 모델을 구축하기 위한 **로우코드** 프레임워크입니다.\n\n주요 기능:\n\n- 🛠 **손쉬운 맞춤형 모델 구축:** 선언적 YAML 설정 파일만으로 최신 LLM을 데이터에 맞춰 학습시킬 수 있습니다. 멀티태스크 및 멀티모달 학습을 지원합니다. 포괄적인 설정 검증으로 잘못된 매개변수 조합을 감지하고 런타임 오류를 방지합니다.\n- ⚡ **확장성과 효율성 최적화:** 자동 배치 크기 선택, 분산 학습([DDP](https://pytorch.org/tutorials/beginner/ddp_series_theory.html), [DeepSpeed](https://github.com/microsoft/DeepSpeed)), 매개변수 효율적 미세 조정([PEFT](https://github.com/huggingface/peft)), 4비트 양자화(QLoRA), 페이지 및 8비트 옵티마이저, 메모리 초과 데이터셋 지원.\n- 📐 **전문가 수준의 제어:** 활성화 함수까지 모델을 완전히 제어할 수 있습니다. 하이퍼파라미터 최적화, 설명 가능성, 풍부한 메트릭 시각화를 지원합니다.\n- 🧱 **모듈식 및 확장 가능:** 설정에서 몇 가지 매개변수만 변경하여 다양한 모델 아키텍처, 태스크, 피처, 모달리티를 실험할 수 있습니다. 딥러닝을 위한 빌딩 블록이라고 생각하세요.\n- 🚢 **프로덕션을 위한 설계:** 사전 빌드된 [Docker](https://hub.docker.com/u/ludwigai) 컨테이너, [Kubernetes](https://github.com/ray-project/kuberay)에서 [Ray](https://www.ray.io/) 실행 네이티브 지원, [Torchscript](https://pytorch.org/docs/stable/jit.html) 및 [Triton](https://developer.nvidia.com/triton-inference-server)으로 모델 내보내기, 한 번의 명령으로 [HuggingFace](https://huggingface.co/models)에 업로드.\n\nLudwig는 [Linux Foundation AI & Data](https://lfaidata.foundation/)에서 호스팅합니다.\n\n**기술 스택:** Python 3.12 | PyTorch 2.6 | Pydantic 2 | Transformers 5 | Ray 2.54\n\n\n\n# 💾 설치\n\nPyPI에서 설치합니다. Ludwig는 Python 3.12 이상을 요구합니다.\n\n```shell\npip install ludwig\n```\n\n모든 선택적 의존성을 포함하여 설치:\n\n```shell\npip install ludwig[full]\n```\n\n더 자세한 설치 방법은 [기여 가이드](https://github.com/ludwig-ai/ludwig/blob/main/CONTRIBUTING.md)를 참조하세요.\n\n# 🚂 시작하기\n\nLudwig의 기능을 빠르게 살펴보고 싶으시다면 이 Colab 노트북을 확인하세요 🚀 [](https://colab.research.google.com/drive/1lB4ALmEyvcMycE3Mlnsd7I3bc0zxvk39)\n\nLLM 미세 조정을 원하시나요? 다음 노트북을 확인하세요:\n\n1. Fine-Tune Llama-2-7b: [](https://colab.research.google.com/drive/1r4oSEwRJpYKBPM0M0RSh0pBEYK_gBKbe)\n1. Fine-Tune Llama-2-13b: [](https://colab.research.google.com/drive/1zmSEzqZ7v4twBrXagj1TE_C--RNyVAyu)\n1. Fine-Tune Mistral-7b: [](https://colab.research.google.com/drive/1i_8A1n__b7ljRWHzIsAdhO7u7r49vUm4)\n\n전체 튜토리얼은 공식 [시작 가이드](https://ludwig.ai/latest/getting_started/)를 확인하시거나, 엔드투엔드 [예제](https://ludwig.ai/latest/examples)를 살펴보세요.\n\n## 대규모 언어 모델 미세 조정\n\n[](https://colab.research.google.com/drive/1c3AO8l_H6V_x37RwQ8V7M6A-RmcBf2tG?usp=sharing)\n\n사전 학습된 LLM을 챗봇처럼 지시를 따르도록 미세 조정(\"인스트럭션 튜닝\")해 봅시다.\n\n### 사전 요구 사항\n\n- [HuggingFace API 토큰](https://huggingface.co/docs/hub/security-tokens)\n- 선택한 베이스 모델에 대한 접근 승인 (예: [Llama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B))\n- 최소 12 GiB VRAM의 GPU (테스트에서는 Nvidia T4를 사용했습니다)\n\n### 실행\n\n[Stanford Alpaca](https://crfm.stanford.edu/2023/03/13/alpaca.html) 데이터셋을 사용합니다. 다음과 같은 테이블 형식의 파일로 구성됩니다:\n\n| instruction | input | output |\n| :-----------------------------------------------: | :--------------: | :-----------------------------------------------: |\n| Give three tips for staying healthy. | | 1.Eat a balanced diet and make sure to include... |\n| Arrange the items given below in the order to ... | cake, me, eating | I eating cake. |\n| Write an introductory paragraph about a famous... | Michelle Obama | Michelle Obama is an inspirational woman who r... |\n| ... | ... | ... |\n\n`model.yaml`이라는 YAML 설정 파일을 다음 내용으로 생성하세요:\n\n```yaml\nmodel_type: llm\nbase_model: meta-llama/Llama-3.1-8B\n\nquantization:\n bits: 4\n\nadapter:\n type: lora\n\nprompt:\n template: |\n Below is an instruction that describes a task, paired with an input that may provide further context.\n Write a response that appropriately completes the request.\n\n ### Instruction:\n {instruction}\n\n ### Input:\n {input}\n\n ### Response:\n\ninput_features:\n - name: prompt\n type: text\n\noutput_features:\n - name: output\n type: text\n\ntrainer:\n type: finetune\n learning_rate: 0.0001\n batch_size: 1\n gradient_accumulation_steps: 16\n epochs: 3\n learning_rate_scheduler:\n decay: cosine\n warmup_fraction: 0.01\n\npreprocessing:\n sample_ratio: 0.1\n\nbackend:\n type: local\n```\n\n이제 모델을 학습시켜 봅시다:\n\n```bash\nexport HUGGING_FACE_HUB_TOKEN = \"\"\n\nludwig train --config model.yaml --dataset \"ludwig://alpaca\"\n```\n\n## 지도 학습 ML\n\n[Rotten Tomatoes](https://www.kaggle.com/stefanoleone992/rotten-tomatoes-movies-and-critic-reviews-dataset) 영화 평론가의 리뷰가 긍정적인지 부정적인지 예측하는 신경망을 만들어 봅시다.\n\n데이터셋은 다음과 같은 CSV 파일입니다:\n\n| movie_title | content_rating | genres | runtime | top_critic | review_content | recommended |\n| :------------------: | :------------: | :------------------------------: | :-----: | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |\n| Deliver Us from Evil | R | Action & Adventure, Horror | 117.0 | TRUE | Director Scott Derrickson and his co-writer, Paul Harris Boardman, deliver a routine procedural with unremarkable frights. | 0 |\n| Barbara | PG-13 | Art House & International, Drama | 105.0 | FALSE | Somehow, in this stirring narrative, Barbara manages to keep hold of her principles, and her humanity and courage, and battles to save a dissident teenage girl whose life the Communists are trying to destroy. | 1 |\n| Horrible Bosses | R | Comedy | 98.0 | FALSE | These bosses cannot justify either murder or lasting comic memories, fatally compromising a farce that could have been great but ends up merely mediocre. | 0 |\n| ... | ... | ... | ... | ... | ... | ... |\n\n[여기](https://ludwig.ai/latest/data/rotten_tomatoes.csv)에서 데이터셋 샘플을 다운로드하세요.\n\n```bash\nwget https://ludwig.ai/latest/data/rotten_tomatoes.csv\n```\n\n다음으로 `model.yaml`이라는 YAML 설정 파일을 생성하세요:\n\n```yaml\ninput_features:\n - name: genres\n type: set\n preprocessing:\n tokenizer: comma\n - name: content_rating\n type: category\n - name: top_critic\n type: binary\n - name: runtime\n type: number\n - name: review_content\n type: text\n encoder:\n type: embed\noutput_features:\n - name: recommended\n type: binary\n```\n\n이게 전부입니다! 이제 모델을 학습시켜 봅시다:\n\n```bash\nludwig train --config model.yaml --dataset rotten_tomatoes.csv\n```\n\n**즐거운 모델링 되세요**\n\nLudwig를 여러분의 데이터에 적용해 보세요. 질문이 있으시면 [Discord에서 문의](https://discord.gg/CBgdrGnZjy)해 주세요.\n\n# ❓ Ludwig를 사용해야 하는 이유\n\n- **최소한의 머신러닝 보일러플레이트**\n\n Ludwig는 머신러닝의 엔지니어링 복잡성을 기본으로 처리하여, 연구자들이 가장 높은 수준의 추상화에서 모델 구축에 집중할 수 있게 합니다. `torch.nn.Module` 모델에 대한 데이터 전처리, 하이퍼파라미터 최적화, 디바이스 관리, 분산 학습이 완전히 무료로 제공됩니다.\n\n- **손쉬운 벤치마크 구축**\n\n 최신 기준 모델을 만들고 새 모델과 비교하는 것이 간단한 설정 변경만으로 가능합니다.\n\n- **새로운 아키텍처를 여러 문제와 데이터셋에 쉽게 적용**\n\n Ludwig가 지원하는 광범위한 태스크 및 데이터셋 세트에 새 모델을 적용하세요. Ludwig에는 간단한 설정만으로 여러 데이터셋에서 여러 모델 실험을 실행할 수 있는 [전체 벤치마킹 도구](https://arxiv.org/abs/2111.04260)가 모든 사용자에게 제공됩니다.\n\n- **데이터 전처리, 모델링, 메트릭의 높은 설정 가능성**\n\n 모델 아키텍처, 학습 루프, 하이퍼파라미터 검색, 백엔드 인프라의 모든 측면을 선언적 설정에서 추가 필드로 수정하여 파이프라인을 요구 사항에 맞게 커스터마이즈할 수 있습니다. 설정 가능한 항목에 대한 자세한 내용은 [Ludwig 설정](https://ludwig.ai/latest/configuration/) 문서를 확인하세요.\n\n- **멀티모달, 멀티태스크 학습 기본 지원**\n\n 코드 작성 없이 테이블 데이터, 텍스트, 이미지, 오디오까지 복잡한 모델 설정으로 혼합하여 사용할 수 있습니다.\n\n- **풍부한 모델 내보내기 및 추적**\n\n Tensorboard, Comet ML, Weights & Biases, MLFlow, Aim Stack 등의 도구로 모든 시도와 메트릭을 자동으로 추적합니다.\n\n- **멀티 GPU, 멀티 노드 클러스터로 학습 자동 확장**\n\n 로컬 머신에서 클라우드로 코드 변경 없이 전환할 수 있습니다.\n\n- **사전 학습된 Huggingface Transformers를 포함한 최신 모델의 로우코드 인터페이스**\n\n Ludwig는 [Huggingface Transformers](https://huggingface.co/docs/transformers/index)에서 제공하는 사전 학습된 모델과 네이티브로 통합됩니다. 사용자는 코드를 전혀 작성하지 않고도 방대한 최신 사전 학습 PyTorch 모델을 사용할 수 있습니다. 예를 들어, Ludwig로 BERT 기반 감성 분석 모델을 학습시키는 것은 다음과 같이 간단합니다:\n\n ```shell\n ludwig train --dataset sst5 --config_str \"{input_features: [{name: sentence, type: text, encoder: bert}], output_features: [{name: label, type: category}]}\"\n ```\n\n- **AutoML을 위한 로우코드 인터페이스**\n\n [Ludwig AutoML](https://ludwig.ai/latest/user_guide/automl/)을 사용하면 데이터셋, 대상 컬럼, 시간 예산만 제공하여 학습된 모델을 얻을 수 있습니다.\n\n ```python\n auto_train_results = ludwig.automl.auto_train(dataset=my_dataset_df, target=target_column_name, time_limit_s=7200)\n ```\n\n- **손쉬운 프로덕션화**\n\n Ludwig는 GPU를 포함한 딥러닝 모델 서빙을 쉽게 만들어 줍니다. 학습된 Ludwig 모델에 대한 REST API를 실행하세요.\n\n ```shell\n ludwig serve --model_path=/path/to/model\n ```\n\n Ludwig는 효율적인 Torchscript 번들로 모델 내보내기를 지원합니다.\n\n ```shell\n ludwig export_torchscript --model_path=/path/to/model\n ```\n\n# 📚 튜토리얼\n\n- [텍스트 분류](https://ludwig.ai/latest/examples/text_classification)\n- [테이블 데이터 분류](https://ludwig.ai/latest/examples/adult_census_income)\n- [이미지 분류](https://ludwig.ai/latest/examples/mnist)\n- [멀티모달 분류](https://ludwig.ai/latest/examples/multimodal_classification)\n\n# 🔬 예제 사용 사례\n\n- [개체명 인식 태깅](https://ludwig.ai/latest/examples/ner_tagging)\n- [자연어 이해](https://ludwig.ai/latest/examples/nlu)\n- [기계 번역](https://ludwig.ai/latest/examples/machine_translation)\n- [seq2seq를 통한 대화 모델링](https://ludwig.ai/latest/examples/seq2seq)\n- [감성 분석](https://ludwig.ai/latest/examples/sentiment_analysis)\n- [시아미즈 네트워크를 이용한 원샷 학습](https://ludwig.ai/latest/examples/oneshot)\n- [시각적 질의응답](https://ludwig.ai/latest/examples/visual_qa)\n- [음성 숫자 인식](https://ludwig.ai/latest/examples/speech_recognition)\n- [화자 인증](https://ludwig.ai/latest/examples/speaker_verification)\n- [이진 분류 (타이타닉)](https://ludwig.ai/latest/examples/titanic)\n- [시계열 예측](https://ludwig.ai/latest/examples/forecasting)\n- [시계열 예측 (날씨)](https://ludwig.ai/latest/examples/weather)\n- [영화 평점 예측](https://ludwig.ai/latest/examples/movie_ratings)\n- [다중 레이블 분류](https://ludwig.ai/latest/examples/multi_label)\n- [멀티태스크 학습](https://ludwig.ai/latest/examples/multi_task)\n- [단순 회귀: 연비 예측](https://ludwig.ai/latest/examples/fuel_efficiency)\n- [사기 탐지](https://ludwig.ai/latest/examples/fraud)\n\n# 💡 추가 정보\n\n[Ludwig](https://arxiv.org/pdf/1909.07930.pdf), [선언적 ML](https://arxiv.org/pdf/2107.08148.pdf), [Ludwig의 SoTA 벤치마크](https://openreview.net/pdf?id=hwjnu6qW7E4)에 대한 논문을 읽어보세요.\n\n[Ludwig의 작동 방식](https://ludwig.ai/latest/user_guide/how_ludwig_works/), [시작 가이드](https://ludwig.ai/latest/getting_started/), 더 많은 [예제](https://ludwig.ai/latest/examples)를 확인하세요.\n\n[기여](https://github.com/ludwig-ai/ludwig/blob/main/CONTRIBUTING.md)에 관심이 있으시거나, 질문, 의견, 공유하고 싶은 생각이 있으시거나, 최신 정보를 받고 싶으시다면 [Discord 커뮤니티에 참여](https://discord.gg/CBgdrGnZjy)하시고 [X](https://twitter.com/ludwig_ai)에서 팔로우해 주세요!\n\n# 🤝 함께 Ludwig를 만들어 갈 커뮤니티에 참여하세요\n\nLudwig는 여러분과 같은 분들의 기여에 의존하는 활발하게 관리되는 오픈소스 프로젝트입니다. Ludwig를 모든 사람이 사용할 수 있는 더 접근 가능하고 기능이 풍부한 프레임워크로 만들기 위해 활발한 Ludwig 기여자 그룹에 참여하는 것을 고려해 주세요!\n\n\n \n \n\n## Star History\n\n[](https://star-history.com/#ludwig-ai/ludwig&Date)\n\n# 👋 참여하기\n\n- [Discord](https://discord.gg/CBgdrGnZjy)\n- [X (Twitter)](https://twitter.com/ludwig_ai)\n- [Medium](https://medium.com/ludwig-ai)\n- [GitHub Issues](https://github.com/ludwig-ai/ludwig/issues)\n"
},
{
"path": "RELEASES.md",
"content": "# Releasing\n\n## Release procedure\n\n1. Update version number in `ludwig/globals.py`\n1. Update version number in `setup.py`\n1. Commit\n1. Tag the commit with the version number `vX.Y.Z` with a meaningful message\n1. Push with `--tags`\n1. If a non-patch release, edit the release notes\n1. Create a release for Pypi: `python setup.py sdist`\n1. Release on Pypi: `twine upaload --repository pypi dist/ludwig-X.Y.Z.tar.gz`\n\n## Release policy\n\nLudwig follows [Semantic Versioning](https://semver.org).\nIn general, for major and minor releases, maintainers should all agree on the release.\nFor patches, in particular time sensitive ones, a single maintainer can release without a full consensus, but this practice should be reserved for critical situations.\n"
},
{
"path": "docker/README.md",
"content": "# Ludwig Docker Images\n\nThese images provide Ludwig, a toolbox to train and evaluate deep learning models\nwithout the need to write code. Ludwig Docker images contain the full set of pre-requisite\npackages to support these capabilities\n\n- text features\n- image features\n- audio features\n- visualizations\n- hyperparameter optimization\n- distributed training\n- model serving\n\n## Repositories\n\nThese four repositories contain a version of Ludwig with full features built\nfrom the project's `master` branch.\n\n- `ludwigai/ludwig` Ludwig packaged with PyTorch\n- `ludwigai/ludwig-gpu` Ludwig packaged with gpu-enabled version of PyTorch\n- `ludwigai/ludwig-ray` Ludwig packaged with PyTorch\n and Ray 2.3.1 (https://github.com/ray-project/ray)\n- `ludwigai/ludwig-ray-gpu` Ludwig packaged with gpu-enabled versions of PyTorch\n and Ray 2.3.1 (https://github.com/ray-project/ray)\n\n## Image Tags\n\n- `master` - built from Ludwig's `master` branch\n- `nightly` - nightly build of Ludwig's software.\n- `sha-` - version of Ludwig software at designated git sha1\n 7-character commit point.\n\n## Running Containers\n\nExamples of using the `ludwigai/ludwig:master` image to:\n\n- run the `ludwig cli` command or\n- run Python program containing Ludwig api or\n- view Ludwig results with Tensorboard\n\nFor purposes of the examples assume this host directory structure\n\n```\n/top/level/directory/path/\n data/\n train.csv\n src/\n config.yaml\n ludwig_api_program.py\n```\n\n### Run Ludwig CLI\n\n```\n# set shell variable to parent directory\nparent_path=/top/level/directory/path\n\n# invoke docker run command to execute the ludwig cli\n# map host directory ${parent_path}/data to container /data directory\n# map host directory ${parent_path}/src to container /src directory\ndocker run -v ${parent_path}/data:/data \\\n -v ${parent_path}/src:/src \\\n ludwigai/ludwig:master \\\n experiment --config /src/config.yaml \\\n --dataset /data/train.csv \\\n --output_directory /src/results\n```\n\nExperiment results can be found in host directory `/top/level/directory/path/src/results`\n\n### Run Python program using Ludwig APIs\n\n```\n# set shell variable to parent directory\nparent_path=/top/level/directory/path\n\n# invoke docker run command to execute Python interpreter\n# map host directory ${parent_path}/data to container /data directory\n# map host directory ${parent_path}/src to container /src directory\n# set current working directory to container /src directory\n# change default entrypoint from ludwig to python\ndocker run -v ${parent_path}/data:/data \\\n -v ${parent_path}/src:/src \\\n -w /src \\\n --entrypoint python \\\n ludwigai/ludwig:master /src/ludwig_api_program.py\n```\n\nLudwig results can be found in host\ndirectory `/top/level/directory/path/src/results`\n\n### View Ludwig Tensorboard results\n\n```\n# set shell variable to parent directory\nparent_path=/top/level/directory/path\n\n# invoke docker run command to execute Tensorboard\n# map host directory ${parent_path}/src to container /src directory\n# set up mapping from localhost port 6006 to container port 6006\n# change default entrypoint from ludwig to tensorboard\n# --logdir container location of tenorboard logs /src/results/_/model/logs\n# --bind_all Tensorboard serves on all public container interfaces\ndocker run -v ${parent_path}/src:/src \\\n -p 6006:6006 \\\n --entrypoint tensorboard \\\n ludwigai/ludwig:master \\\n --logdir /src/results/experiment_run/model/logs \\\n --bind_all\n```\n\nPoint browser to `http://localhost:6006` to see Tensorboard dashboard.\n\n### Devcontainer\n\nIf you want to contribute to Ludwig, you can setup a Docker container with all the dependencies\ninstalled as a full featured development environment. This can be done using devcontainers with VS Code:\nhttps://code.visualstudio.com/docs/devcontainers/containers\n\nYou can find the `devcontainer.json` file within the top level `.devcontainer` folder.\n"
},
{
"path": "docker/ludwig/Dockerfile",
"content": "#\n# Ludwig Docker image with full set of pre-requiste packages to support these capabilities\n# text features\n# image features\n# audio features\n# visualizations\n# hyperparameter optimization\n# distributed training\n# model serving\n#\n\nFROM python:3.12-slim\n\nRUN apt-get -y update && apt-get -y install \\\n git \\\n libsndfile1 \\\n build-essential \\\n g++ \\\n cmake \\\n ffmpeg \\\n sox \\\n libsox-dev\nRUN pip install -U pip\n\nWORKDIR /ludwig\n\nRUN pip install --no-cache-dir torch==2.6.0 torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cpu\n\nCOPY . .\nRUN pip install --no-cache-dir '.[full]'\n\nWORKDIR /data\n\nENTRYPOINT [\"ludwig\"]\n"
},
{
"path": "docker/ludwig-gpu/Dockerfile",
"content": "#\n# Ludwig Docker image with full set of pre-requiste packages to support these capabilities\n# text features\n# image features\n# audio features\n# visualizations\n# hyperparameter optimization\n# distributed training\n# model serving\n#\n\nFROM pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel\n\nRUN apt-get -y update && DEBIAN_FRONTEND=\"noninteractive\" apt-get -y install \\\n git \\\n libsndfile1 \\\n cmake \\\n ffmpeg \\\n sox \\\n libsox-dev\nRUN pip install -U pip\n\nWORKDIR /ludwig\n\nCOPY . .\nRUN pip install --no-cache-dir '.[full]'\n\nWORKDIR /data\n\nENTRYPOINT [\"ludwig\"]\n"
},
{
"path": "docker/ludwig-ray/Dockerfile",
"content": "#\n# Ludwig Docker image with Ray support and full dependencies including:\n# text features\n# image features\n# audio features\n# visualizations\n# hyperparameter optimization\n# distributed training\n# model serving\n#\n\nFROM rayproject/ray:2.44.1-py312\n\nRUN sudo apt-get update && DEBIAN_FRONTEND=\"noninteractive\" sudo apt-get install -y \\\n\tbuild-essential \\\n\twget \\\n\tgit \\\n\tcurl \\\n\tlibsndfile1 \\\n\tcmake \\\n\ttzdata \\\n\trsync \\\n\tvim \\\n\tffmpeg \\\n\tsox \\\n\tlibsox-dev\nRUN pip install -U pip\n\nWORKDIR /ludwig\n\nRUN pip install --no-cache-dir torch==2.6.0 torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cpu\n\nCOPY . .\nRUN pip install --no-cache-dir '.[full]' --extra-index-url https://download.pytorch.org/whl/cpu\n"
},
{
"path": "docker/ludwig-ray-gpu/Dockerfile",
"content": "#\n# Ludwig Docker image with Ray support and full dependencies including:\n# text features\n# image features\n# audio features\n# visualizations\n# hyperparameter optimization\n# distributed training\n# model serving\n#\n\nFROM rayproject/ray:2.44.1-py312-cu124\n\nRUN sudo apt-get update && \\\n DEBIAN_FRONTEND=\"noninteractive\" sudo apt-get install -y \\\n build-essential \\\n wget \\\n git \\\n curl \\\n libsndfile1 \\\n cmake \\\n tzdata \\\n rsync \\\n vim \\\n ffmpeg \\\n sox \\\n libsox-dev\nRUN pip install -U pip\n\nWORKDIR /ludwig\n\nRUN pip install --no-cache-dir torch==2.6.0 torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu124\n\nCOPY . .\nRUN pip install --no-cache-dir '.[full]'\n"
},
{
"path": "examples/README.md",
"content": "# Examples\n\nThis directory contains example programs demonstrating Ludwig's Python APIs.\n\n| Directory | Examples Provided |\n| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| hyperopt | Demonstrates Ludwig's to hyper-parameter optimization capability. |\n| kfold_cv | Provides two examples for performing a k-fold cross validation analysis. One example uses the `ludwig experiment` cli. The other example uses the `ludwig.experiment.kfold_cross_validate()` api function. |\n| mnist | Creates a model config data structure from a yaml file and trains a model. Programmatically modify the model config data structure to evaluate several different neural network architectures. Jupyter notebook demonstrates using a hold-out test data set to visualize model performance for alternative model architectures. |\n| titanic | Trains a simple model with model config contained in a yaml file. Trains multiple models from yaml files and generate visualizations to compare training results. Jupyter notebook demonstrating how to programmatically create visualizations. |\n| serve | Demonstrates running Ludwig http model server. A sample Python program illustrates how to invoke the REST API to get predictions from input features. |\n| class_imbalance | Demonstrates using our class balancing feature to over-sample an imbalanced dataset. |\n"
},
{
"path": "examples/calibration/README.md",
"content": "# Calibration Examples\n\nDrawing on the methods in\nOn Calibration of Modern Neural Networks (Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger), Ludwig supports\ntemperature scaling for binary and category output features. Temperature scaling brings a model’s output probabilities\ncloser to the true likelihood while preserving the same accuracy and top k predictions.\n\nTo enable calibration, add calibration: True to any binary or category feature:\n\n```\noutput_features:\n - name: Cover_Type\n type: category\n calibration: True\n```\n\nWith calibration enabled, Ludwig will attempt to find a scale factor (temperature) which will bring the probabilities\ncloser to the true likelihoods using the validation set. This calibration phase is run after training is complete. If\nno validation set is provided, the training set is used for calibration.\n\nTo visualize the effects of calibration in Ludwig, you can use Calibration Plots, which bin the data based on model\nprobability and plot the model probability (X) versus the observed rate (Y) for each bin.\n"
},
{
"path": "examples/calibration/train_forest_cover_calibrated.py",
"content": "#!/usr/bin/env python\n\nimport copy\nimport logging\nimport shutil\n\nimport numpy as np\nimport yaml\n\nimport ludwig.visualize\nfrom ludwig.api import LudwigModel\nfrom ludwig.datasets import forest_cover\n\n# clean out prior results\nshutil.rmtree(\"./results_forest_cover\", ignore_errors=True)\nshutil.rmtree(\"./visualizations_forest_cover\", ignore_errors=True)\n\n# Download and prepare the dataset\ndataset = forest_cover.load()\n\nconfig_yaml = \"\"\"\ninput_features:\n - name: Elevation\n type: number\n - name: Aspect\n type: number\n - name: Slope\n type: number\n - name: Horizontal_Distance_To_Hydrology\n type: number\n - name: Vertical_Distance_To_Hydrology\n type: number\n - name: Horizontal_Distance_To_Roadways\n type: number\n - name: Hillshade_9am\n type: number\n - name: Hillshade_Noon\n type: number\n - name: Hillshade_3pm\n type: number\n - name: Horizontal_Distance_To_Fire_Points\n type: number\n - name: Wilderness_Area\n type: category\n - name: Soil_Type\n type: category\noutput_features:\n - name: Cover_Type\n type: category\ncombiner:\n type: transformer\ntrainer:\n batch_size: 256\n learning_rate: .001\n epochs: 1\n\"\"\"\n\nuncalibrated_config = yaml.safe_load(config_yaml)\n\nscaled_config = copy.deepcopy(uncalibrated_config)\nscaled_config[\"output_features\"][0][\"calibration\"] = True\n\nuncalibrated_model = LudwigModel(config=uncalibrated_config, logging_level=logging.INFO)\nuncalibrated_model.train(\n dataset,\n model_name=\"uncalibrated\",\n experiment_name=\"forest_cover_calibration\",\n output_directory=\"results_forest_cover\",\n)\n\nscaled_model = LudwigModel(config=scaled_config, logging_level=logging.INFO)\nscaled_model.train(\n dataset, model_name=\"scaled\", experiment_name=\"forest_cover_calibration\", output_directory=\"results_forest_cover\"\n)\n\n# Generates predictions and performance statistics for the test set.\nuncalibrated_test_stats, uncalibrated_test_predictions, _ = uncalibrated_model.evaluate(\n dataset, collect_predictions=True, collect_overall_stats=True\n)\n\nscaled_test_stats, scaled_test_predictions, _ = scaled_model.evaluate(\n dataset, collect_predictions=True, collect_overall_stats=True\n)\n\nuncalibrated_probs = np.stack(uncalibrated_test_predictions[\"Cover_Type_probabilities\"], axis=0)\nscaled_probs = np.stack(scaled_test_predictions[\"Cover_Type_probabilities\"], axis=0)\n\nludwig.visualize.calibration_1_vs_all(\n probabilities_per_model=[uncalibrated_probs, scaled_probs],\n model_names=[\"Uncalibrated\", \"Calibrated\"],\n ground_truth=dataset[\"Cover_Type\"],\n metadata=uncalibrated_model.training_set_metadata,\n output_feature_name=\"Cover_Type\",\n top_n_classes=[7, 7],\n labels_limit=7,\n output_directory=\"visualizations_forest_cover\",\n file_format=\"png\",\n)\n"
},
{
"path": "examples/calibration/train_mushroom_edibility_calibrated.py",
"content": "#!/usr/bin/env python\n\nimport copy\nimport logging\nimport shutil\n\nimport numpy as np\nimport yaml\n\nimport ludwig.visualize\nfrom ludwig.api import LudwigModel\nfrom ludwig.datasets import mushroom_edibility\n\n# clean out prior results\nshutil.rmtree(\"./results_mushroom_edibility\", ignore_errors=True)\nshutil.rmtree(\"./visualizations_mushroom_edibility\", ignore_errors=True)\n\n# Download and prepare the dataset\ndataset = mushroom_edibility.load()\n\n# This dataset has no split, so add a split column\ndataset.split = np.random.choice(3, len(dataset), p=(0.7, 0.1, 0.2))\n\nconfig_yaml = \"\"\"\ninput_features:\n - name: cap-shape\n type: category\n - name: cap-surface\n type: category\n - name: cap-color\n type: category\n - name: bruises?\n type: category\n - name: odor\n type: category\n - name: gill-attachment\n type: category\n - name: gill-spacing\n type: category\n - name: gill-size\n type: category\n - name: gill-color\n type: category\n - name: stalk-shape\n type: category\n - name: stalk-root\n type: category\n - name: stalk-surface-above-ring\n type: category\n - name: stalk-surface-below-ring\n type: category\n - name: stalk-color-above-ring\n type: category\n - name: stalk-color-below-ring\n type: category\n - name: veil-type\n type: category\n - name: veil-color\n type: category\n - name: ring-number\n type: category\n - name: ring-type\n type: category\n - name: spore-print-color\n type: category\n - name: population\n type: category\n - name: habitat\n type: category\noutput_features:\n - name: class\n type: category\ncombiner:\n type: concat\ntrainer:\n batch_size: 256\n learning_rate: .0001\n epochs: 10\n\"\"\"\n\nuncalibrated_config = yaml.safe_load(config_yaml)\n\nscaled_config = copy.deepcopy(uncalibrated_config)\nscaled_config[\"output_features\"][0][\"calibration\"] = True\n\nuncalibrated_model = LudwigModel(config=uncalibrated_config, logging_level=logging.INFO)\nuncalibrated_model.train(\n dataset,\n model_name=\"uncalibrated\",\n experiment_name=\"mushroom_edibility_calibration\",\n output_directory=\"results_mushroom_edibility\",\n)\n\nscaled_model = LudwigModel(config=scaled_config, logging_level=logging.INFO)\nscaled_model.train(\n dataset,\n model_name=\"scaled\",\n experiment_name=\"mushroom_edibility_calibration\",\n output_directory=\"results_mushroom_edibility\",\n)\n\n# Generates predictions and performance statistics for the test set.\nuncalibrated_test_stats, uncalibrated_test_predictions, _ = uncalibrated_model.evaluate(\n dataset, collect_predictions=True, collect_overall_stats=True\n)\n\nscaled_test_stats, scaled_test_predictions, _ = scaled_model.evaluate(\n dataset, collect_predictions=True, collect_overall_stats=True\n)\n\nuncalibrated_probs = np.stack(uncalibrated_test_predictions[\"class_probabilities\"], axis=0)\nscaled_probs = np.stack(scaled_test_predictions[\"class_probabilities\"], axis=0)\n\nludwig.visualize.calibration_1_vs_all(\n probabilities_per_model=[uncalibrated_probs, scaled_probs],\n model_names=[\"Uncalibrated\", \"Calibrated\"],\n ground_truth=dataset[\"class\"],\n metadata=uncalibrated_model.training_set_metadata,\n output_feature_name=\"class\",\n top_n_classes=[3, 3],\n labels_limit=3,\n output_directory=\"visualizations_mushroom_edibility\",\n file_format=\"png\",\n)\n"
},
{
"path": "examples/class_imbalance/README.md",
"content": "# Credit Card Fraud Detection Example\n\nThis API example is based on Kaggle's [Imbalanced Insurance](https://www.kaggle.com/arashnic/imbalanced-data-practice) dataset for detecting whether customers will buy vehicle insurance.\n\n### Preparatory Steps\n\nCreate and download your [Kaggle API Credentials](https://github.com/Kaggle/kaggle-api#api-credentials).\n\nThe Imbalanced Insurance dataset is hosted by Kaggle, and as such Ludwig will need to authenticate you through the Kaggle API to download the dataset.\n\n### Examples\n\n| File | Description |\n| ---------------------------- | -------------------------------------------------------------------------------------------------------------- |\n| model_training.py | Demonstrates the use of oversampling by training two different models: one with no oversampling, and one with. |\n| model_training_results.ipynb | Example for extracting training statistics and generating visualizations. |\n\nEnter `python model_training.py` will train a standard model with no class balancing and a balanced model with class balancing applied. Results of model training will be stored in this location.\n\n```\n./results/\n balance_example_standard_model/\n balance_example_balanced_model/\n```\n\nThe only difference between these two models is that the balanced model uses a small amount of oversampling in addition to the other configuration parameters.\nThe way this is done is by specifying the ratio that you want the minority class to have in relation to the majority class.\nFor instance, if you specify 0.5 for the `oversample_minority` preprocessing parameter, the minority class will be oversampled until it makes up 50% of the majority class.\nIn this example, we had an imbalance where the minority class was 19% of the majority class in size. We decided that we wanted to increase that to 26%.\nThough this doesn't seem like much, it is enough to get some small performance improvements without experiencing performance degradation due to over-fitting.\n\nHere are the performance differences in the two models followed by some plots showing different metrics over training:\n\n| Metric | Standard Model | Balanced Model |\n| :------: | :------------: | :------------: |\n| Loss | 0.3649 | 0.2758 |\n| Accuracy | 0.7732 | 0.8237 |\n| ROC AUC | 0.8533 | 0.8660 |\n\nHere are the learning curve plots from both models:\n\n\n\n\n\nHere is the comparison of model performances on ROC_AUC and Accuracy:\n\n\n\nThe creation of the learning curves is demonstrated in the Jupyter notebook `model_training_results.ipynb`. The comparison plot was generated using the ludwig visualize [compare performance](https://ludwig-ai.github.io/ludwig-docs/0.4/user_guide/visualizations/#compare-performance) command.\n"
},
{
"path": "examples/class_imbalance/balanced_model_config.yaml",
"content": "input_features:\n - name: Gender\n type: category\n - name: Age\n type: number\n - name: Driving_License\n type: binary\n - name: Region_Code\n type: number\n - name: Previously_Insured\n type: binary\n - name: Vehicle_Age\n type: category\n - name: Vehicle_Damage\n type: category\n - name: Annual_Premium\n type: number\n - name: Policy_Sales_Channel\n type: number\n - name: Vintage\n type: number\noutput_features:\n - name: Response\n type: binary\npreprocessing:\n oversample_minority: 0.26\ntrainer:\n learning_rate: 0.0001\n learning_rate_scheduler:\n decay: exponential\n decay_rate: 0.9\n decay_steps: 30000\n staircase: True\n epochs: 50\n"
},
{
"path": "examples/class_imbalance/model_training.py",
"content": "#!/usr/bin/env python\n\n# # Class Imbalance Model Training Example\n#\n# This example trains a model utilizing a standard config, and then a config using oversampling\n\nimport logging\nimport shutil\n\n# Import required libraries\nfrom ludwig.api import LudwigModel\nfrom ludwig.datasets import imbalanced_insurance\nfrom ludwig.visualize import compare_performance\n\n# clean out old results\nshutil.rmtree(\"./results\", ignore_errors=True)\nshutil.rmtree(\"./visualizations\", ignore_errors=True)\n\n# list models to train\nlist_of_model_ids = [\"standard_model\", \"balanced_model\"]\nlist_of_eval_stats = []\n\ntraining_set, val_set, test_set = imbalanced_insurance.load()\n\n# Train models\nfor model_id in list_of_model_ids:\n print(\">>>> training: \", model_id)\n\n # Define Ludwig model object that drive model training\n model = LudwigModel(config=model_id + \"_config.yaml\", logging_level=logging.WARN)\n\n # initiate model training\n train_stats, _, _ = model.train(\n training_set=training_set,\n validation_set=val_set,\n test_set=test_set,\n experiment_name=\"balance_example\",\n model_name=model_id,\n skip_save_model=True,\n )\n\n # evaluate model on test_set\n eval_stats, _, _ = model.evaluate(test_set)\n\n # save eval stats for later use\n list_of_eval_stats.append(eval_stats)\n\n print(\">>>>>>> completed: \", model_id, \"\\n\")\n\n\ncompare_performance(\n list_of_eval_stats,\n \"Response\",\n model_names=list_of_model_ids,\n output_directory=\"./visualizations\",\n file_format=\"png\",\n)\n"
},
{
"path": "examples/class_imbalance/model_training_results.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"8c1e31e4-d8d4-4e83-8f4c-f868365d14d7\",\n \"metadata\": {},\n \"source\": [\n \"# Model Analysis\\n\",\n \"\\n\",\n \"This notebook will analyze the training results of the standard and balanced model on the [imbalanced insurance](https://www.kaggle.com/arashnic/imbalanced-data-practice) dataset. In order for the cells in this notebook to run, you must first run the following command to train the models:\\n\",\n \"```\\n\",\n \"python model_training.py\\n\",\n \"```\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"a3b3b6c1-5d11-4a03-9dfa-b070e45b2adb\",\n \"metadata\": {},\n \"source\": [\n \"## Import required libraries\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 226,\n \"id\": \"6a6bbd43-1333-4a0e-b895-c3e393d5ee07\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"from ludwig.utils.data_utils import load_json\\n\",\n \"from ludwig.visualize import learning_curves\\n\",\n \"import pandas as pd\\n\",\n \"import numpy as np\\n\",\n \"import os.path\\n\",\n \"import matplotlib.pyplot as plt\\n\",\n \"%matplotlib inline\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"b5e4d240-7607-4036-9573-6b452523c18f\",\n \"metadata\": {},\n \"source\": [\n \"## Learning Curves\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"725dde7d-f7a5-4836-8cf1-a3f50acafd30\",\n \"metadata\": {},\n \"source\": [\n \"### Create Plotting Data Function \"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 227,\n \"id\": \"d8c35325-cfab-4ead-8699-1e98e55a8c7b\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"def create_plot_ready_data(list_of_train_stats, model_names, metric, target):\\n\",\n \" # List of splits to evaluate statistics for\\n\",\n \" list_of_splits = ['training', 'validation', 'test'] \\n\",\n \" \\n\",\n \" # Empty list to fill with dfs for each models' stats\\n\",\n \" list_of_train_stats_df = []\\n\",\n \" \\n\",\n \" # For each models' stats, create a df with columns of stats for each split listed above\\n\",\n \" for name, stats in zip(model_names, list_of_train_stats):\\n\",\n \" list_of_dfs = []\\n\",\n \" for split in list_of_splits:\\n\",\n \" df = pd.DataFrame(stats[split][target])\\n\",\n \" df.columns = [split + '_' + c for c in df.columns]\\n\",\n \" list_of_dfs.append(df)\\n\",\n \" \\n\",\n \" combined_df = pd.concat(list_of_dfs, axis=1)\\n\",\n \" combined_df.name = name\\n\",\n \" combined_df['epoch'] = combined_df.index + 1\\n\",\n \" list_of_train_stats_df.append(combined_df)\\n\",\n \" \\n\",\n \" # holding ready for plot ready data\\n\",\n \" plot_ready_list = []\\n\",\n \" \\n\",\n \" # consolidate the multiple training statistics dataframes\\n\",\n \" for df in list_of_train_stats_df:\\n\",\n \" for col in ['training', 'validation']:\\n\",\n \" df2 = df[['epoch', col + '_{}'.format(metric)]].copy()\\n\",\n \" df2.columns = ['epoch', '{}'.format(metric)]\\n\",\n \" df2['split'] = col\\n\",\n \" df2['model'] = df.name\\n\",\n \" plot_ready_list.append(df2)\\n\",\n \"\\n\",\n \" return pd.concat(plot_ready_list, axis=0, ignore_index=True)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"id\": \"722f5f48-7026-427e-9bb7-388fec15a24f\",\n \"metadata\": {\n \"tags\": []\n },\n \"source\": [\n \"### Create Plotting Data\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 228,\n \"id\": \"6b48aef8-11a8-4bba-8d52-114adb9cb2f2\",\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"standard_stats = load_json(os.path.join('results/balance_example_standard_model','training_statistics.json'))\\n\",\n \"balanced_stats = load_json(os.path.join('results/balance_example_balanced_model','training_statistics.json'))\\n\",\n \"\\n\",\n \"accuracy_learning_curves = create_plot_ready_data([standard_stats, balanced_stats], ['standard_model', 'balanced_model'], 'accuracy', 'Response')\\n\",\n \"roc_auc_learning_curves = create_plot_ready_data([standard_stats, balanced_stats], ['standard_model', 'balanced_model'], 'roc_auc', 'Response')\\n\",\n \"loss_learning_curves = create_plot_ready_data([standard_stats, balanced_stats], ['standard_model', 'balanced_model'], 'loss', 'Response')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 229,\n \"id\": \"a06aaa41-e692-4348-aab5-6bd78711f6f1\",\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"image/png\": \"iVBORw0KGgoAAAANSUhEUgAAAmkAAAGICAYAAAAagXdoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAACwC0lEQVR4nOzddXhb5/nw8e85OkJLMtuJw3jCaaCctiljSiutTNu6des6eLeO6TeGjte1a7uuzExpm2KKYT5hTswk1oH3jyM7dmKQSbLj53NdvhLr0K1j2br1wP1IlmUhCIIgCIIg9C9ytgMQBEEQBEEQDiWSNEEQBEEQhH5IJGmCIAiCIAj9kEjSBEEQBEEQ+iGRpAmCIAiCIPRDIkkTBEEQBEHoh5RsByAIQs+pqroCmAkcpWnaZ1kOJ2tUVf0p8G1N0/zZjqU9qqq+A4Q0TTsvQ9c7Cfg6cAyQC2wHHgH+rGlaOBMxCILQPaIlTRAGOFVVpwMzgHXAzVkOR+jcV4BvZeJCqqp+F3gbkIDbgAXAA6nrL1RVNScTcQiC0D2iJU0QBr7rgJXA/4Cfqar6TdFC0n9pmrYuE9dJtaD9GviNpmnfb7Fpkaqq7wOLgW8Cv8hEPIIgdJ1I0gRhAFNV1QFcid068jjwB+Ay4P4W+4wCfg+cBljAO8A3NE3b2dl2VVWvT52rWNO0qtT+eUAtcIOmaf9NdTGeB7wP3ACs1TTteFVVhwK/BM4CioFK4Angu5qmxVPn8gI/Bz4P5AGrUtvfV1X1aUDVNG3aQc9ZA17UNO3bPbhvnwe+D0wA9mB3/f2txfYgdvJyITAUqAdeAb6uaVpdah8L+AFwFTAE+FLqPvhT9+KbQAnwMfAVTdPWp457h1R3p6qq87Fbuk4EfgPMAfYCv9I07T8t4pkJ3AkcDZQDPwZ+CjykadpP23ma38a+5z8/eIOmaR+pqvpjYGvq/E1xHKlp2pIW161L3Zufpl4LfwB+C3wXqMb+cDBT0zT1oPu7BFinadq1qe9vA74GjAQ2Az/XNO3xFvufk4pzChACXsLutq5p57kJwqAgujsFYWA7HTuJeFjTtL3AW7To8kwlGx9gd4d+BbgemAS8qqqqo7PtXYhjJnAk8DngV6qqysBrwGzgVuBM4EHssVFfbHHcY6nvf4edEJWnrj0eu2Vwaqo7t+n5HAlMTJ2rW1RVvQ57TNa7wPnYCe6dqqr+vxa7PQJcANwBnIGdnFwJ/Oig0/0U+DtwC3ZiBnaye13quV6NnQj+t5OwHgWeBs4BlgP3qKo6JRVvKXYC5QWuwE6S/gqM6OA5StivjUWapsXa2kfTtP/TNO2RTuI6WB52In4VdoL6H2CiqqozWlx7DHay+Wjq+58Af8T+WS8A3gAeVVX10tT2UcAz2C1752B3xS4A/tHF2AThsCNa0gRhYLsWWK5p2prU9/8DHlRVdXKq5eYG7FaeiZqmbQNQVXUX8Cx2MnZaJ9vTpQC3N7XCqKo6Aru17TZN01al9lmkqupZwEnA31KtQ+cD12qa9mDquPewk5TjsROlSuzk6Hupc1wFrNE0bWUXYmuWSh5/hZ3UfjX18MJUq9iPVFX9J2AALuAWTdNeS+3zjqqqx6Vib2mhpmn/anF+gABwrqZp+1KPDQP+oqpqoaZp1e2E9ldN0/6U2n8ZcBFwNvY4w9uwP1Cf3aIVrwp4qoOnWgS4gR0d3Y9ucAA/1jTt9VQcDuzE+lLsVlCAy4Eq4I1Uq+sdwG81TWtKcBeqqhrAbjl8EjgqFetvWtyzEDCql2MXhAFHJGmCMECl3uguAH6dejMEWAREsFvTvgUch939uK3pOE3TVgBjUuf4cSfbj+xCSOtbnGMXMF9VVVlV1QnYrV8zgVJgZ2q341L/vtjiuAQwtcVzfAy79eh7qYTgCuxWme6aCJQBL6uq2vLv36vY3W1HaZr2NnbrGaqqjk4dMw27K+7gVqm2xpftaEo2Unan/s3B7iJsy8dN/9E0rS6VpDQN6p8PvNOUoKU8B+jtnAvsRBP6prek+TlrmmaoqvoEdpLWlIRdBjypaZququoxgIe27/eNqVa3pUAc+DT1834ZeEHTNANBGOREd6cgDFyXAj7ssVO1qa89qceuVVXVBRQAFR2co7Pt6QofPFlBVdWbsMdXbQTuwe4OjWLPNGy6dvKg5ONgDwCjVVU9FrvVrxi7ha27ClP/PgIkW3w1lS0Zmor9fFVVtwDbgIexuw4jLWJv0ta9ixz0vZn6t6O/t20d07R/EXaLYrNUAlPV3slSY7lC2GPA2qSqaomqqs4OYmrPwc/5Eft06nRVVccBs0h1dXLgfn9I6/v9ZOrxoZqmbQVOxR7f9jXsrt3dqqpe1I3YBOGwIlrSBGHguhb4FHsQd0tTscdJXYA94H3cwQeqqno2sCyN7VbqoZYJRqc1yFIzC+/BTiD/rmlaZerxT1vsVg84VVXN1TStvsWxxwK1mqZt0DRtqaqqa4FLsLsRF2matqez63eg6Tq3Yt+7g21Ltfw9iZ0gnqRp2u5UXE9gt6Zl2l7s5LRZqtu2sO3dm70BnKyqqivVQnmw++1TqRNo4+ecGtfWaYkOTdM+VlV1K/Z4xDiwC3ucIxy43xdxoEWx1eGpcywGzlNV1YedsH0HeFJV1ZGpsZaCMCiJljRBGIBUVR2JPSPwQU3T3mn5BdwF7Mfu8vwQmJYanN107GTsmYoz09jekHq4rMXlT0gjxGOw3/j/r0WCVgZM50Br1Iepf5uLuqZa/57ATkCbPIidcJ5LDyYMpGzA7nIcrmnakqYv7ITnF9jFXmdjj0n7TYsELQeYx6EtaZnwHnbXcbDFY2cDnbWC/Rl7dunBkx2aZnOeCTyiaZpF2z/nY0j/g/xj2D+fi4HHUucE+AS75azkoPs9DXuGqqSq6s2qqm5VVdWpaVpE07QXgR9ij38blub1BeGwJFrSBGFguhY7CXr64A2pcUKPY3cdfRH4BvBSqlSGgZ2MfIo9fu3TTrb7scdh/UVV1f/D7j77EXaLSUc+w/4Q+GdVVZ9MHfcD7AHivlScy1RVfQl7EkEQuzTDLditN/9uca6HsAf7x7BnAXbGqarq7W08vkrTtEWp5/mn1CD/t7DH3/0a2ITdvamk7sNvVVX9F3Z347exJ1h09rz7wl+xf5Yvq6r6W+xWtV+ltpntHaRp2nuqqv4e+KGqqpOwuyVD2En2N4GPsEukgD3ofw/wC1VVk0AQe4xe/SEnbtvDHJjc0Tx7V9O0SlVV/wr8UVXVfOzX1RGp6z6vaVpDarLI37Bbzv6JnSD/EPtnsSLN6wvCYUm0pAnCwHQ1sPigAeotPYz9+30jdovbZuwyEP/BfuNboGmanhoP1tn2y7ATg5ewuwmvwX6zb5emaYuwE4HzsAeJ/wh7NuLPgVmqqrpTu16O3Tr2E+wZpQXAqZqm7Whxrj3YScSzmqZ1eN0UF3ZNsYO/Lkudr6lkxvnYLYY/x+7ePFfTNEvTtI3YSfCM1PbfAUuwS5SMTLUIZkxqRujp2D/Pp7Dv5TdSmzv7OXwHe7JFEXbi+xx21+P/AWc21atLjXG7jAOJ8E+wuxw3pxnjOmA1sFHTtOUHbf4OduL/BeyyLF/HbuW7PnXsRuySGyWp5/cI9ozR0zVNS6ZzfUE4XEmWZXW+lyAIQpaodlHcXcBZmqa9me14Mi01Rs+nadpbLR6biD2e6wJN017IWnCCIPQp0d0pCEK/lJopeDV2kdv12F2Tg9E44D5VVb+H3Y1cit11vBFYmM3ABEHoW6K7UxCE/koCbseutH91i8Hog4qmaQ9hdxneDLyOXYl/DXBye6sJCIJweBDdnYIgCIIgCP2QaEkTBEEQBEHohzI2Ji1VfPGf2LWX4sDNmqZtbrH9KuxlbAzgvqb18FRVXc6BaeDbNE27IVMxC4IgCIIgZEsmJw5cCHg0TTs2tZ7bH7ELVDb5A3al9BCwLrWGWxRA07T56V7ENE3LMDruwjVNu7SQLIuGxM6Ie5UecZ/SJ+5V+sS9So+4T+kT9yo9mbxPTqejioNWFWmSySRtHnaNnKZlROYetH0VdrVvHXvAsIXd6uZTVXVhKtbva5r2MR1IJg3276/pMJBo1F4mz+v1df1ZDDLiXqVH3Kf0iXuVPnGv0iPuU/rEvUpPJu/TqFGlO9rblslUOkjr6tWGqqotk8Q1wFJgLfBSqohmBLuF7Uzs4pMPH3SMIAiCIAjCYSmTCU8D9gLJTWRN03QAVVVnYK/7Nga7u/MhVVUvBV4ANqem3m9UVbUaaCps2SZZlvH50st8091PEPcqXeI+pU/cq/SJe5UecZ/SJ+5VerJ9nzLZkrYYOAcgNSZtdYtt9djjz6Kp5UkqgHzsJW3+mDqmDLs1rr1lcARBEARBEA4bmWxJexY4XVXVD7HHnN2gquqVgF/TtLtVVf038IGqqglgC/Y6ggD/VVX1A+wxajc2tb4JgiAIgiAczjKWpGmaZmKPK2tpQ4vtdwF3tXHolX0ZlyAIgiAIQn8k5uAKgiAIgiD0QyJJEwRBEARB6IdEkiYIgiAIgtAPiSRNEARBEAShHxJJmiAIgiAIQj8kkjRBEARBEIR+SCyx1A2RaJS2lnB3OJxIsoyh61iWcch2SXLgcTuRJanvgxQEQRD6DcO0CCd0IgmDIUEPAJsrwzgdEoU5LnJcDiTx3iAcRCRpXfTy2nKOWnQxU+RD10M9J/4r1lmj+bHyP25UXjtk+8+T17Bu+JX87ZLpmQhVOMzsb4jxwdYaNlWG2VEboaIxTm00idvhoDToJp402FEbPeQ4WYJcrxPDtGiI6ZiW/RHDskCSQJYkCnxOctwKsaRJ0jBxOiRcDhmXIuNRZEbl+yjL9RDXDWqiSXxOB4osEddNYrqJz+mgyO+iIaazdl8DScMiYdjnSpoWsiRRFvQQN0y2VoUxTAunIuOUJZwOGZdDYvaIXHK9LvbVx2iM6bidMl6nA2/q35lluYwu9FHRGGNjZRjdsNDNpi+TYr+b8UU5NMSSfLStFt2yMEz7SzdN3A6Z2SPyiOsmizZVEYrrdnyGRdK0/51SGsC0LLZUhamJJDEtC9MC07JwyjLTygLMHZGH1+kgrhtMLg0wssBLid+N09G7HRO6adEY1zFNC11KIskSMiDLEhJ2dW8rFZtlgYUdKxaYqf+bpoXR4j6Ylp0sND2mp553Qjeb/6+bFrkeJ7OG5+JSRGdLE8uyiCQNTAsaYzpaRYglO+uoiyZJGCYJ3SRhmBTkuDh6ZD4Jw+CBT3cT0w2iSZO4bgLgdcr86rzJyJLE797azJ76GAAuh0y+z0lhjosfnDGB8UU5LNlZx576GIU5LgpzXBTluCjwOXv9tSb0X5JltdUmNHAlk4ZVVxfpcJ9IxN7enTW5KhrjfPri3/Emqg/ZtnvkRVg5pfj2vE9hw9pDtr9nTOOT5Fheu+WYLl83W3pyrwaT3rhP++pjLN1dx8aKMNtrIuxviFMTSRD0OIkmDarCCQAkTKZL25ghb2WWYxsjlVpqPSPY5RjBvZH5GLJCy8/jbkXmiGG5yDIs31VP3DCxsJf9MFJJzqzhQSxLYn15I5WhRHNy0sQhgdHNPxWyBE6HzITiHFwOmc1VIZLGgYShKclwKzJx3WyzlbovSRxIVocE3PhcDhpiOjHdQJFlFFlCkSXCCbt1vDaabPM8XqfMkSPyuWx2GbkehRW76xkS9FAScGNZdsLldTqYURakLprkrsXbqYkkqY0kaYzrhOI6cd1k2tAg1eEEm6vC6Gb2/j4X+JycOrGI648eSYnf3afX2lDeyOPL93LdkcMZXZjDok1VvLK2nKBHIehxkutVyPUoTB4SYHJpgLhuUpv63fA6ZaJR+8NJy98/07KTz5huJ0j2l8GEYj8AK/fUUx1OENNNokmDxphOY9zgwhlDcEgSTyzfw9ubqgglDKJJg2TqF6ApQc4ESbIT8YONzPMwJOihPpZkd10MWZJwyKT+lRgadDNlSBDdMFm+ux6HLKE47A9FMhaXzizl9CllGXoW3ZSMoNRuQg6XI8XriY87F5yZex/K5HtfcXFgKTC3rW0iScugpx76M2Nr3mH2bc9l9Lo9IZK09HT3Phmmycvrynng093sqj20G93rMDk+WMXJ/l0UKAmWDbmcycUeLnxrHrIRw/QWYviH4ajfBkD1zetAkgi+eA1yMoyePwGjYAJ6wUSM/AmYOUPsv/xpanqjc8gSkiRRHY6zuy6WaoWy8Lkc+FwOCn1OSgMeHDIkDAunw05uunqvLMsiljQIJQwaYkkaYwahuE5jQkdGwrDsLqNo0kCRZJwOCcUhocgyPpeDgNtOUBOGidNhb7f/td+gmloGPU4HbsWOMZ0uJt+nf0QvmkZi7JlEkwabK8Os2lvPxsowO1IJdV1Ux+jk76nLYV+vqVWlre1jCnwU+d1EEjoyFg5ZQnY4aPpbrZb4CXqc7KyNsKc+1vwmbqZa0yYU5TC2KIeqUJwlu+pRZKn5TVyRJUoCbo4bXQASvLWxCkUGRZaRJcm+n7LE7BF5PLF8L5/sqAVg6pAAt88fwxHD8jq9V+nSDbtF85Gle1i7vxE5lZB4nHYsCcPEssBIJfEAE4pzmDsij8a4zktrywH7A4TX6cClyIzI83Ll3OEkdIMfvaK1ed0vHTcS3YSX1pZT3hg/ZLsMHPzT8SoyQY9Cns/JxBI/M8uC5HkcHLHhdyiBUqpn3orX6bA/+Fj269iwLEzTjt885P/2h5SE0TKBtJPIuG4SS7W8hVO/Bw0xnca4TjhhEEno+N0KiixTH01QfVCLr2XZP2eHLJE0TAxDp4Q6hkrVyJgssSaR61F489bjeu1nmTYr1dQryThqN6PsX4YjvB85XI4c3o8c3k9s8uXEpl2Lc9d75L1wYMGhxhN/SWz6dRkLtb8kaaK7M4MK9X2c6VjCjlgcn6dvP5kK/d/+hhg3PrKCynCCqUMCXDhjCH63wrR8i/n77yO/fg3O6rVI0RhEwQiMYO6x3wFJosH7AEZwFGZgWPPHbSlW25yAGQUTkcpX4N7yMvK6uuZrVl/1PmbeGNza08iRKpJDj0QfMrvdGGVJwuN0NH9fGvBQGvB0+Ly8PeiJkSQJr0vB61Io7uPWm3Q5qjeQ89md6HnjSIw9E6/TwfSyINPLgq32syyLylCCHbURtldH2FgZZktVmH0NMRSHTL7XSZHfxcg8H4U5Tgp8Tor8brsby+ci6FUOGa/a128U88cXtXwCuHYswrPmARpH/IPjRk/lg3de4IENsGY/fOGxVZT4Xdx0zEjOnz603SQ8He9vqeYnr2o0xg8sxTw818sJ4wqxsAgnDKIJg0jSIJKwE/VQwqA+FGbZmh1oCTvu2xzPMFfWGMN+vEacZZUT+OTlyTxjzAOCbV773x/uBECRwa1IKJKM1+VgWNBNWZ6HobleyoJuhgY9lOV6KA3YXdlyeD+uHYtQypcTmvY7kCTyVm7EufMxgtPOxcif0u370W2WiRypRA7tRQ7txcwZgj5kDo7azQQWfct+PFyOZNlpZ7xoGl9w/o6PdtQRSxqtfrf7lJEk7/nLUCrXUH/2PSRHzse1bSH+j34FgOnOw8wpxfQPwXLaLZ168XTqz74X0z+E3JeuRalclZlY+xmRpGWQmXrxVdXWMnLokCxHI2TTwg0V/PhVDcO0uHREmP9z/wvLzKPxxH+BkaTwk+cxCiYQnXo1evEM9NIjMHJHNydhyeHHtz6hJGF5C5q/DR//I/s/loUUrUap3YijZiNmcCQA7i2v4N72OgDR6dcTOu4HoHj7/HkPRJ71TwBQd/EzHe4nSXYrVUnAzZEj8zMRWu8wddybX8K37B8o1esx/ENx1G3DKp7OxXt/x6XspGbITB6KHM1/G2bz6zcT3P/JLi6ZOZQLZwwl1+tM6zIrdtfx4bZadtRG+WBrNQnDIt+rcN60IZw1qYQJxTl2q6ZlgiSDZeFddS+Oum046rfjqN+GnNgNDqj4+iZipkL+Gw/jaIAGzxyikpMT6lZyRngpCy6+CTMwnJKtT+BMNhAfegxGyXQUxYnikHFIpNWCqlSswrX0DVzb38KZShKMwHCkaDWWr4j68/5HwYPHkvPZnTScfU+Pfgxd4VvyFzzrHkMO70cyD3S/RydfQWjIHCzFh+XwkBw+D8NfhukfiukvwwiO5IvrlnP+7mdZsnMK88YVdXCV3qNUrcG57zNi4xfYrflAbPLlxMedi5lT0ubfHsuTT2LsmQDoxdNQKtdkJNb+RiRpmeS2k7S6epGkDVamZfGzVzVeWV+BLMEdp4zmps1fxlG5ndjUa+ydHE6qb1plv1H1lCRh+YpI+opIDjvQvdFwzr1I0Rp8S/+Gb+U9OPd8RMMZ/8AonNTzax5OjASejU8TH3s2lrcQ19bX8Kx/goaz7wZ54P/5dO54m8B7P8TRsAM9fzwNp/yJ+MQLweECoO7CJ3Fveo6g9gy3Je7ma16F7XnHcIfj2/z9g+3844PtHDUqj9tPGsf44pxDzh9N6Pzn4528sGY/dVG71Szfq3DRjKGcMamEGXlJnFWrce58CWXpKpSq9ZiuAHWXvwaShG/p38FIYOSNIVk6C2PiRRh5Y5Cxu9rj5/4bgESqxTHk8xEJ72dYKhEIlr/X/GHEdOagD51LouxYYpMvx/IVH3pDEmFc+z4hMfJkkCT879yBUrUGfcgcQsfcQWL0aRgFavOHJcuTR3TGjeQs+QuOqnUYRZlpTYuplyI37MTyFqaSMPvLCAy3n2ugjPoLH2/z2ImuRcxzvMc173/KvHHnZCRe5/6lgP3h0fTbY+EsbyGWtzCt4yMzv4CkHzopajAY+H9lBhCH225+DzXWZjkSIRuqwglufXIVW6sjBN0Kd18xg2l7HsNZsZKG0/9uvzk26Y0ErROWt4DwvJ+QGHEiwbe+Qc5Hv6LhvP/1+XUHEteOt5Cj1cQmXwGAlAzh3r6QnA9/SXjeT7IcXfdIiUakaDVm7mgslx/Tk0fo+B+SGHPmIa87MzCM6Oxbic6+FUfVOjwbn6Wsfjt/P3sujyzZRc7Hv+a9XZO5+n/VDMvzc8vxozllYhHLd9fz1/e2sqE8ZE9SkeCYUvjy+HqmD8nBGD0eObyfwvsPDMPRc8eQLJmBUXgg0am56j0sV6Br4yhzDnwAbjjnXuRwOc69n+Dc+zHOPR/j//g3xCecjwV41j6EHK3GdAVw71iEc89HSEacmivfxcgfR+Mpf8DMGdKqlfpg0Zk34111HzlL/kLDWf9OO85uMxKYgTJCp/yxW4c3DW/Ir12FbpyFkoGZosr+Zc3JZHckR87v3YAGEJGkZZDTZydp0VBddgMRMu7DbTX87DWNUFznmNH5/P78Kfii+8j5+HfER55MfMIFWYstOepkaq54o3nciqN6PaavtMM3psFCjlSh508gMfIkAOLqJUTKV+JbeQ96yQziEy/q8xiUhh3gHA3OnnVHS9FqvCvvxbvmAfTCSdRf9DT60COpu+SltJIgo2gK4RYtRVdPUshf/i43Sy9QTR7Ph47hsVeO5yevjSNpQDH1fMP7AacF9zDR3IJSvwuWQrJ4BnWjT8H0lRKa9zP0wknoxdOx3IeOIWvrsa4yc0qJTzif+ITzm++D5bFf287dH+LZ/AJgJ4nRadeRGH0qRmpYQDotY5Ynn8hR38JKddF2JaHsKueu9wks+hb15z2IUah26xzJ/InEJTez5E28tqGC86b2fa+Oc/9SkkPmdP8Epo5bexojbyz60CN7L7ABQMzuzKBVm7fy8IsvcOr8szljVvd+wTJNzO5MT3v3KWmY/GrhJl5aV86oAi+/O38KYwvtbiHvirvJ+eQP1Hx+EWZweMZjbpNlkv/oKUjxRhpP+zPJESf0+iUG3Gvq4DdeI0nu81fgrFxJ7cXPYxRP7bNLR8JhRjw8BzkZwvAPxcgdY3/ljSU6/TpQPAfGcLVDbtiNb8VdeNY/BnqcxLizicz6CnrpET0P0Ijbkw20Z3BufxPZTLIoeDF75v6Q4wvCjH3qBIzgKJIlM9CLp9vjK4unYXnyen7tFnrympJitUiJxubxmv2WZZH31HnIkSpqrnrX/tl3QyQSIfeFK9leWcNPi//KPVcc0btxtkEO7UXSYxh5Y7t3Asui8N5pxMcvIDT/N70bXHvWPElgw0OEzrkXy9e3Y/fE7M5+orB4KIvM2cxz5mY7FCEDdtZGue3p1eypj+FWZH50xsTmBA0gesQXiY87DzPQj+oVSTINZ/yT4MJbyXvh80Rm3UL46O80j1EaTBzV6zEDI7Bc/oM2OGk46y7ynzib4BtfpfaKN0HuxVlypk7O4p8Tm/J58Iyg+rhf4IvtxVG3FUfdNtxbX0FKhInOvBmA3Oc/j6NxVyp5G92cyCXLjsZSvOQ9exFypIqYejHRWV/GyB/fe7E63CTGnk1i7NlI8XrcW17hqGg10cklYFlU3bQay9O/J1FYnvxeiVGK1eJb8ldik6/oditXR1xbX7GHRpzyp24naE2kstlMrbqXreU1vRRdx7rbzdlMktCLpqFUru6dgNLgLl+Cq3pD1l+/IknLIK8V5WbHy4R2hmHa2dkOR+hDL63dzy8XbkI3LUbkefjXZTMpDdglJaRoDc69H5EYe07/StBSjKIp1F76Cv7FP8e3/C6cuz+k8Yy/d/9T8EBkWQRfuwUzMIz68x85dLOvmIaz/2N/05sJWjJC8PVbcO9YhBkYDhOvJjL2XDiohUiKNzRfNzH6NJSKFXYCpy1DTjQCUHPFmxiFk2g85U6MvLF9/lqz3Ll2YtkcpJT1N7hM86x7BDlcTuOZ/+zdE5s6OR//Dj1/InH1cz0+XWjC5/jznvFE9sLGihATS/ydH9RN3pX3Ijfu7vEYTr14Gt5V94ORBEd6M4p7wl21ikTRtN79/e4GkaRlkF+BHzof5r4KHyCStMNROKHzmzc28dqGSgBOV4v56Vlqq+V1/It/jnvTc9Rc9V7/7WJxegnN/zWJkScRWPRtHDXaoErSlP1LUeq20DD7K+3u09xdaOq4dr5LYvSpPbqmFK1O1YNaTeNJvyE27WqItD10o+VYregRX2ixwS654qjfhpE3BoDkiHk9iktIj+XJJzb9BrzL/kHkyNsxCib22rk9G55EqdtC/dn/6ZWkQc8dwzHzhnH/E6vYUN63SZp784u9ch69ZAaSmcBRu6nvZ9FaFsm8CSTye+9n2F1iAbAMcqUmDih6OMuRCH1hQ0WYqx9cxkKtkjnDc/l/p4zjl+dOapWgOXe+i0d7isisr/TfBK2FxNizqLnmQxJj7Q8VnjX/Q4rXZzmqvudZ/xiW4iM+7rxO9/Wu/i+5L1+Ha/NL3b6eXL+DvKcvQKneQMNZ99gJWnekSq7oQ48ER/8oBjyYRI74IihefEv+0qvnNQIjiE663J6B20uObXyVb/pe4/2thy5x2GuMOErFqp5NGkjRi+01r50VGShqK0lUn/AbGqfd2PfX6oRI0jJJVohYblxGKNuRCL3sox11fPmZ9TTEdP592Uzuunwml80a1rpgZjJC4N3voeeNJTL3tuwF20VNrTaOmk343/8x+Y+dgbL30yxH1YcSYdybXyQ2YQG4Dq39dbDotGtIls4m+NY3cVS3vRRRZ5zly5Dj9dRd8HhzAU9h4LG8BURnXI970ws4ajb12nmTI+YROvWPvTpz1LV7MVfzMu9urqI20vZ6tD2lVK5BMhMkO1jVJF1G7mgis29FL5rcC5F1TA7ttYcU9AMiScuwEF5cxuAsyne42lwV5kevb8G0YFS+lxnD2i4bkPPpH3E07CQ0/7c9HvibDUbBBOoufhZkhbznLiHwxtfwv/1dlNQnW+eu9wi88TUCC28l8NotBF/7IsFXbsKz9iEA5IZd5L5wJSWvXUdw9b3ZfCodcm95GTkZbq6N1imHm4az78Zy5hB89eYutTTK9dsBiE+8iJqr3kcf2uYEL2EAiRzxJSxnDs7dH/T4XFK8gcAbX8NRt7UXImstOWQOBWY1Q6nm8eV7ev38AM79ywDQe6ElDUkmfOz30Etm9vxcncj56NeUPdd5K3omiCQtwyJ48Vqiu/NwURtJ8LUnV5E0LKYN8XPXZTMPWX8RaC5VEJ1yJclhx2Y+0F6il86i9vLXiU26FNeu93FtfwM5bC90LUcqcO5fhlKxCqVmA47aLTgadiBHU8WbLRMpEcIRqSB/yW9x7l6cxWfSPiN/PJEZN6IPST9hMnOGUH/Wv3E07iLw5tftshidcG94ioJH5uPe+BxAr5elELLD8hZSc+1HxGbc0ONzeVf8G8/GZ5GSvf+e0VTUdpa8mbc2VvX6+cEe22kEhrcqMNwTcmgfbu0pMPXOd+4BpXw5iaLpfXqNdImJAxn2iutMKsxg2wVRhAEloZvc/swaqiJJinOc/Prs8a3Gn7XicFN72atIffzHJRMsl5/QKX/k4E77uHoJcfWSdo8zc0dRd8kLRBtqGPrsOfjf+wG1ly/sd+U99CGzO1x0vt3jyo4idPxPcG9biJSMHFq6o4ll4V32D/wf/4bEsONJjDqlhxEL/Y3lyQfLRClf0a3XEoAUqcS34h5i489vHo/Vm/TCKViKh+OlLbxaewyGaeKQe7fdJnz8j5HD+3rtfM49iwm+eTs1xTN6dWJGS1KsFqV+O43jez6LtjeIlrQMW152FYtcJ2c7DKGHLMvi129sZF15CLci84cFEwl62v7M49r8EnL9DlA87b9xDyKW4qHmmB+j1G7Gu/q/2Q6nFbf2NK5tC7t9fGz69dQveMj+ObfVmmYa+N//Ef6Pf0NswgXUL3iwV6rqC/2Pd/m/yHvmwm53VeYs+QsYcSJHf7uXI0txOEkWz+Q411ZMC97Qer81zQyU9U5XZ4qeat1SKvtu8oBSvgKAeHHfd6umQyRpGVas72NIZEO2wxB66OGle3hpXQWfmzGEP144ldH5bS/ZI9dtI/jm18n5+LcZjrB/i42YT8OpdxKdclW2QznA1Mn56Fd41j7c/XNIEsgOlMo15D962iFv0DmLf4539X+JHPElGk//W79rRRR6T2zSZeBwdWump1y/A8/ah4lN+Xyflr6JHPVNIsf/EICFGyp69dzOXR/gf/cHSLG6XjunkT8OS/GgVK7ptXMezFmxAguJROG0PrtGV4gkLcNOrbifP1h/ynYYQg+8v6Wav7y7lfnjC/nOaRM4elQ7BTsti8A7d2A5XISP/1FmgxwA4pMuBVcOUrTGXnopy1w738URLic2+fIen8t05yFHqwi+chNS4kDHcGz6dTSe+H/266GDpZyEgc/yFROdei3ujc92uTVNqVqD5QoQOfL2vgkuJTn8eIonn8SE4hzqY707FMO14y27lI2zF5d/kxX0oql9uvKA5XCTHHkSyA4Cax/o8/FvnRF/JTIsqeTgl6JE4gN/bNJgtLkqzB0vrgNgYrG/7UkCKe4NT+Das5jwsd/H9A/NVIgDiqNyLQUPHodry8vZDgXPhscxPQUkRp/W43OZweE0nPkvHHVb7Nmub30TklGMvLHEpl/f82D7MSlcgdy4Fzm0DzlcjhSuQIpUIUVr7HUyY3VI8QY7eU1GIBkFPQZGIq0JFwNJZNYtdmva0r916bjEuHOpvu7TdgfcO6rW4dz1Hug9rBRgmXiX/5ubizewam8D1aFEz87XgnP/EnsmZi+3FuvF01Aq1/bZayU6+yvUL3iInM3PU/DpL5HDvdvC2FVi4kCGGUoOfqJsbowxxi3GJw0ktZEEX3tqFQnDYkqpn+uOGtHuvlKkEv/in5McehSxqf2oS6+fMQpVjNxR+D/4CbUj52dtzJ4Urca17Q2i06/vtTeV5PDjCR/7A/wf/gLTFUCZcUPXBoBbFkr9duRkGMXjBqzmx9v8f/P32I/JTvt6GVrWRopW43/vh3h6UGHecrhJlh1NYuR8EiNOsgeH92JtsEMvaNllUBzuPlk2y8opITr1Gryr7iM85zbM1CoQHXFveoH46NOgnRYoz7pH8b9zB5Jl2Pdr2LEH7lf++K7dL0nGu/p+TvBPAa7nb+9v46dn98K6o3oMpXIN0Zk3NT+0dFcdk0r95Lh6lnYkRp2KJTvtxL43W+kAklEkS8dyBfDsXYyeMzTrH7BFkpZhliuASzKobggxpkgkaQNFQjf5xrNrqAonKfQ5ufPiae3P5CRVH8iyaDz5d6JbqyOyQuikX5P39AX4Pv0D4Xk/zUoYno3PIZnJXunqbCl6xBex3AGSpXPSW3Tb1HHu/QTX9jdwb1uIo2Fnj66fLJlJ6MRfHljCqo+4Nr9E4N3vIyUaicz+KkbuKLuloymJtEy7BItlpr4/8BjY+0mWhRStxLXrffyLfw6A4R9KYsSJJEaeTHL4vJ6XKTHiKBWrce5fgnPfZzj3L0WOVmHJLurPvY/kyPk9O38bIrO+jF44yV6LtRPKvs8ILvwKoeN/TPSIL7beaFn4Pv0DOUv+QmLkSUSn34Bz1/u4dr6D/4Of2k/PP4zEyJNIjDzJvl/u3E6vmRwyh9K9nyJL8MmO3llw3S5imySZKmOzdn8jtzyxis/PHsY3Tx7Xo3MnRp3SZzOi3dsXElj4VWovX4hn38dER53etx8S0iCStAyT3AEA6hvqgN6pHSP0Lcuy+OXCjazdH8LpkPjr56ZT4Ou4tSUx9kxqhn+C5QpkKMqBSx8ym9jUq/Guuo+4egl6ceYH7EanXoWROxqjcFLvnliSiE25suNdEiGcO9/BvW0hrh1vIcfrsRxuEsPnUTf1ZgxfMW63G5BavGG0eOOQJKyDt0kSjsY9+D75A3lPLSA25UrCx97R6wueS5FKAu/9APeWV0iWzKTxlD/2+B6GAblxD65d7+La+Q7uLa/iXf84liSjlxyRSkLmo5cc0WkroRSpshOy/Utw7luCUrEKybS79PTc0SRGzic5ZC6etQ+S+8pN1J/3P5LDj+9R/AezckqINyX/ltX+m75l4f/o1xi+UqJTr2m9zUgQeOe7eDY8SXTy5YRO+g04nCRGn2bfr4ZduHa+i2vXO7g3v4h33SNYkgN9yOzmVja9ZEabHxj10ll4Nj3P3LwIn9b6aIwlCXh6toC5c/9SgObloP69eDsAT6/cy1dPGNPhB9x0OOq22sMHiqf26DwHU8qXg8OFlAzhSNQTLTuuV8/frZiyHcBgI+UNZ8nuiSSTvdf3L/Sth5fu4ZX1FYwu8PKVeWM6XIxYSoTwrHmQ6MwbRYLWBeFjvot766v43/8RdRc9k/lPr4qnxwukd4UcLse17Q1c217HtXsxkpnAdOeRGH068TFnkBhxErhyiKQWWJd8Xe/WSQLx8efh+/ROvKvuxb31FcLHfs9eSaGnrbuWhXvT8/jf/xFSIkzomDuIzroF5N55SzEDw4hNudJOcE0dpXwFrp3v4Nr5Dr7P/kzOZ3diunNJDD+B5MiTiBcfjeErwVGt4dxvt5Ap+z5DSa3oYMku9JIZRGfcQHLokSSHzMHyFTdfLz7uHPKeu5Tcl6+nbsHD6GVH9crzaMn/3g/B1AnN/02b2107FuHc9ymNJ/0anAdmi0uJRoKvfhHX7vcJH/UtInNvP+T3wwyOIDbtanvNVyOJs3wZzp12kpvzye/J+eT3mJ58EiNORCo9lsiYswH7NdWUSF1Rup9Pa8fyxPK93HTsqB491/jECzByR2L5ivl0Ry0fba/F53QQSRo8snQ31x/ds3WLA298DcsVoP6Cx3p0noM5y1egl8zAtecjAGJlx5HttWEkqx/MqupNyaRh1dVFOtyn6Q+frxt/+HpqY0WIqx5cxm/Pn8IpE4oyfv2uyua96g/e31LNN59by6kTi/jluZPaLfbYdJ9KlvwKz+oHqLvkxT7vYhqo2ntNOXe+g+krwSiaktF4chb/AikZbvfNs1dYFo6ajXZr2bbXcVasAMAIjrKTsjFnkBx65CFJTm/9/jmqN+B/7we49n5CsnQWoZN+1e0CqVK4gsC738O97XWSpbPs1rM+Kiza5vVjtbh2vW8nIbvewZFa8cJUfMi6fb9MbyHJIXPtr6Fz7efayVJsUqSSvGcvQQ6XU3/+I90uQtuenPd/jHf1A9Rc9R5m7kFJkGWS//iZSMkINVe+Aw67JUsO7SX3petw1G6i8eTf2zOiu0iKVuPa9V4qyX0POVpJaNyFRM/6u72DkaDonsnUT76aI5acyrhCH49d33vl1i++91N21cW486KpfOPZtRT7XbzypWN6dE7/29/FveUlqm9a03sf6IwkRfdMIjrtOpTKVVixBvZf8FxG3vuKiwNLoe0a96IlLcP8bgWwaIyKlrT+zp7JuR6HBFfPGd5pNW5XxXI8qx8gOv16kaB1Q/N4IFNH0mOZmUSgR/Gse7TvWtH0GL6lf8Oz8TkcDTsAe5xY+OjvEB9zBkaBmpFWQ6NwEvUXPoV74zP4F/8feU+cQ2zatYSP/n/pj/OyLNwbn7Vbz/QYoeN+SHTmFzI2MaE5DE8+8QnnE59wfir53YC0+U0cod0wbC76kDkYuWO6fF8tXzH1FzxG3rOXkPvi1dRf+HivVvqPzv4K3rUP41v6N0Kn/KHVNufuD1Cq19Nw+t+bEzRH1TpyX7oWKRGyu2FHnNit61reQuITLyI+8SJ7Nucb38C37VWiyajdYudw0XjSr7EKVQrXRdhTH8OyLKRuvi7l0D58y/5BdMaNfFSfz666GOMKfcwbW8jE4hw2VobZXBlifHH3f7/14ul41z2M3LgbM9j+BK6uUGo2IBlx9MLJeFffT8OU63vlvD0lRjRnmLt6DRvd1xJa92q2QxE6UBtJcNtTq0gaJmOLchhfnNPxAUaCwsU/wPQPJXLMdzMT5OHINMh75mL8734/I5dzb30NOdGQ/mLqXSDXbyfv6QvJWfIXjLzRNJ70a6qvX0LdpS8TmXubPXYrk926kkRc/Rw1V71DdMYNeNY+SMEjJ+Fe/0Sn5Qzk8H6Cr9xI8M3bMPLHU3v566nuzcwmaIeQJIzCyTRMv4naY39CfNKldvHXbt5X0z+Uugsex3IFyH3hShzV63stVDNnCLEpn8ejPYXcsKvVtuSIE6m9+Fk78QScu94n71l7WaK6i5/pdoJ2CEkmPPZ8ZD2Ca8dbzQ/HJ1+GXjKTG44eSUw32VHT/dIezn2f4l39X6RkmF+/uQmAH5xht7TenOpG/fv723rwJGget9qbKw9I0RqMwHCwDHsS0bDsj0cDkaRlXF4gF5dkICUasx2K0I6kYfLNZ9dQGU6S61X480XT8DjbfzOSYrUMefVqXHWbCZ3068Gz9FNfDJWQHSRGnIBn4zMZWYDds/5xjMCIXl/03rX1NfKfOAdH4y7qz/0v9QseJjbtml5baLonLHcu4RN+Tu1lr2HkjiG46JvkPfs5HFXr2tjZwr3hSfIfPRXXrvcIHf9j6i56xi71cJgyg8Opu/BxLIebvOc/j6N2c6+dOzL7VkBuVTdNStXh0oceCZKMe8NT5L50Daa/jLpLXuj17v/YkKMwvEV4Nr9wIIZoDd4Vd3N6SQMAb26s7Pb5lf1LsRQvH4ZK2V0XY3yRj+ll9tJn88cX4nXKfLqzDt3ofp0zvXASlqz06soDyZEnUXPtxyg1G7AcbmIl/WOFbZGkZZjis6dEy8lwliMR2mJZFr98fSNr9odQZIk/XzSNkoC742NcQXRfCZUn/zWjg8+zRQ7tI/fFqyj612gK75tF/mOnkfv85wks/Co5H/wU79K/41n3GK7tb6KUL7dbDbpQdDMy56sYwVH43/0eGPG+ex4Nu3Dt/oDY5Mt6r0yKkSRn8S/IffVmjLwx1F72Wq8Ux+0LRtEU6i5+hoZT/oSjbiv5T5xFzvs/Rorbb9RyaB/Bl68j+NY3MApUaq94wy4Lke3Wswwwc0dTf8HjgETuc5cj1/Ws5af5vP6hRKdfB7LT/pCTCFPw+JnkfPhLu8TGkr8QfOt2kmXHUHfxs5j+3q/dhuwgPOpMXNvfal4NQzLi+Bf/nGHVi/G7Hfzvs93dPr1z/1KSJTP539JynLLEj848UHpGkiS+MX8cScPig609KPeheIhPvKh370/S/hvl2vU+ybKjQen4736miDFpGdbUyqLoIknrjx5euoeX11cgAT85S2Xq0HYWv7YsvMvvIll2FPqQOVSdYg/CPdynV7g3vYj/3TuQjATR6dch6THkaDVypBJnw06kaFW7H0BMZw6Wt4igO5/Y0KNJHHdH8/ibVhQvjSf+H3kvXYNv+V1E5n69T56LUrESS/ESU7s+GLstcmgvwYW34tz3GdFp1xGa92Nw9I8/9O2SZOKTLyMx5gxyPvk93lX349n0IrHJl+FZ8yCSmSA072dEZ9ww6Or9GfnjqLvgMfKeu5S856+g7qKnMYOd1zrrTPj4Hzd3x/pW3YscrSQ+6lT873wH77pHiamX2PUV+3Bd18iYcwhueBjX9jdSyc5QDP9QlPLlqMVHsnR3PVuqwowr6mSYx8H0KErVWnZPuIFPVtbypeNGMWVI61nuC6YN4Z6PdvDsqn3M78HkucZT7+z2sQeTEo0U3jud8FHfQqnRCKmX9Nq5eypjSZqqqjLwT2AmEAdu1jRtc4vtVwHfAgzgPk3T/tXZMQOS4sWwJNyGSNKyqS6aZFt1hK3VYbZWRdhaE2FrVZiaSJJTJhTx1RNGMyK/7ZRLSjQSeOubuLe+SmT6DeipKeyHMyneYFeT3/gMyZIjaDz9r+0v/JyM2olbtMr+ilTZyVvqMRr2kLvq3ySqV9Fw5r+xvAWHnmLUycTGnYdvyV+JTfxcr7w5Hiwx/jyqRp3SK1XLnbveI7jwq0h6jIYz/kF8wgW9EGHmWJ48Qif9ktjky/G/9wN8y/5BouxoGk/+Q1pV8g9XRqFK3fmPkvf8ZeQ9fzl1Fz3Z89YbSQIjiXflf8j55HfER51CzrK/49r5DuG5Xydy1Lf7fKxivHQORs4Q3JtetCcUAMnSOTj3L+XCI4ewdHc9jy7dww/P7NqsXaViNZKpc9e2YjyKzBWzhx26jywxrtDHh9tr2VEbYVQ7f2c7lVotwvSVgKuLyeQhca9CMvXmxeATvTUGsBdksiXtQsCjadqxqqoeA/wRaPmX7A/AVCAErFNV9THg5E6OGXgkiRA+3FbHZUKE3lEfTbI1lYxtqQqzpSrC9poINZFk8z6yBE6HjGlaHDUyj5+eNRFvO0uXOGo2Enz1Czjqt9tVwWd+IVNPJWucez8m8MbXkcP7CR/5DSJzbmu7Baz5AC+mc3i7iVUkEiFny/MULv4h+U+dR/0597VZADV8wk9JjpzfJ8v1SJEqLHeg5wmaaeBb8hd8n92JUTCRhrP+PaDHa+klM6j73PMoVWvRi6YOutazthjFU6lf8DC5z19B7vNXUHfhU1g5JT06pxypxP/RL+3/1+9Aqd9O48m/67Twca+RZOLjz8O7+n9I8Xosdy76kDl4trzEmcNNfiLBh9u73h1pFExgycxf8+InhYwq8aaqGRxqwfQhfLyjjr+9u40/XNi9grRK+TLyn76A+nPuIzHmjG6d48C5lgPgaNyN6S3EKJoM0ViPztlbMpmkzQNeA9A07WNVVQ8elbcKyAV07HLaVhrHHMI0zebaQu2JRrObIN1W/CDVUZjXSZz9QbbvVVcZpsV9n+3hk5317KqLE9NbD04t8bs4emQQCXhlQzV+l4PSgIsSv4tSv4thuW5isRiWfuibk3f76+S9/10sxUv5WQ8QH3IURO1xDAPtPqXFSJC37C8E1/wHPTCCinMeJVFyBMST2KVSuycajRAtO53k2aMpfutW8p66gKoTf0901EFjt6QgjD4fojGkZAjL2XsTMore/RGuqtXsvfj1brdayLEait79Ft69iwmNu4CaY3+G5fRBL/5eZ+11lTOu37xJpaPP71NgIvHT76Fk4U0En7uM8rMfwvQc2gKcNikH56QrydnyIo7QPipOu4vY8JN69bXTnqZ7VT/8DHwr/wMbXiQy4WL0vCn4AXZ/wvDcMnbWxaisbSCnnUSrbW6+tm4CDST400kj230vPn54Dh5F5sNtNTSGwjjkrv8OSt5R5Eky1t5lRErndfn4lnL2LiEZHI1z7ydEhx5LJBrL8O9e+4XPM5mkBYH6Ft8bqqoqmqbpqe/XAEuxVwV5RtO0OlVVOztmQPJ6fEQaRXdnX9hUFeGhZfvJ9ygkDJNcj0JRjpPhuR5G5ns4ZmQu04b4SRgmt80bibeDWZuHkBUSBZOomv9njH4wS68vOWs3UfTet3HVrKdx4mXUHvU9LGfPuhQOliieyf4FT1G86FZKFn2F2tnfoGHGLYckTcHV9xJY91/2XvRqr8ycleP1+Ha8TuOES7udoLnLl1L0zu044rVUH/cLQhMvy/oaf0LfipfOoeK0f1Pyxs2Uvn495Wf9D9Odl9axcrwed8Uy3BXLcVcsw1W5CtmIoXuLqTjtbhJFvbu8UToSxTPR/cPI2fYK4QkXkyicSt2s20nmTWD+OBf/W7qPxTvqOWNiYXontCwib/+SoaGxBIqOYEIH49kkSeLEsXks3FjDi+squXBa11smLaePZO5YXNVru3xs6xNZuCpXkSicgm/3O0TLepbw9bZMJmkNtE4X5aZkS1XVGcC5wBjs7s6HVFW9tKNj2iPLctoVgrNVRf/8qruoj8Tw+f6Tlet3x0BZcWD5fnvq+F2Xz2RMoa/dgozpPhspXIFHe4rorC/DpAU0qufi7qALaKDcp3ZZJt5V95HzkV1KpKkrwdv5kV3m8/nAN5aGzz1DYNH/I3/ZnXgbttB4yh9bLYvD6BNwLPkdRav/TviEn/f4up6tTyEZCYwZV3f952VZeFfeQ85Hv8L0D6PuvBcwi6f1+YSRAf+6ypA+v0/jTqbBeR+5L9/AkDe/QP35j2K5D5pclBor1byI+74lKLUb7U2SA714GrGpV9nLUw0/HsWTn5UZfD6fj8SEBfbYODmO5csnedy3cQE3FCR5ZPl+dtQn076ncsNORu34H1PkGzj3nM93etzt8yewcOMnPLmqgiuPGt2t52CWzMC9Z3GPfu5SrA7J4cDhtCf5SONObXW+bP/uZfK1sRhYADyRGl+2usW2eiAKRDVNM1RVrQDyOzlmwBqu72C01JDtMA5LizZWATAiz9PtitlNlH2fEXztFuREPYmxZ6WKZB6+Y3Tk0D4Ci76Fa9d7xEedSuMpf2i1vmGfUbw0nv439KLJ5Hz0Gxz122g4+97msWh66RHEpl2Ld/V/iU+6tMdV4D3rH0cvnIJe1LWF3KV4PYFF38K99TXiY86k8dQ/YblzexSLMPAkR55Ew9l3E3z1ZnJfuob6c/+Lo25rKiGz1w2Vo/bfIdOdS7J0NvGJF5IcOpdkyRG9MlGlt8THn49v+V24t75KbMqVyA07cW9bCNOuY86IXN7eWMltJ45J629pxYYPKATq8md2XvwbKA64GV+Uw5aqMNXhBIU5XZ/NqhdPx7PxGaRwRbfHCVqePGqu+4zcF65Ez5+A6R/arfP0lUwmac8Cp6uq+iH2mLMbVFW9EvBrmna3qqr/Bj5QVTUBbAH+iz0+rdUxGYy3zyQcOeRRTiRh4HMd/jWHMmlPfQyvU8ap9OC+WpbdmvThLzACw6ld8GD7MxkPE67NLxF457tIRpzGk35NbOrVGa+GH519K0aBSmDhV8l/8lzqz/lP88zZ8DHfwb3lFfzv3EHd517odq0uR9U6nBUrCc37WZeen1K52k7YQ3sOTBgR3ZuDVmL0aTSc8U+Cr3+ZonsPfGgwgqNIjJxvJ2RD5tprmvbjD3Z68XSM4Cjcm14kNuVKnOUr8X/wU5JDj6TE7+GTHXW8t6WGk8Z33uW5d/0HjLLcfPmCsw/daCQIvHEbyA704hnoxdPQi6fzq/Mmc9l/l/DKunKuObLryzslS2eRLJ2FHKvF6O5kDssEI4Fz36dEMzVxowsylqRpmmYCtxz08IYW2+8C7mrj0IOPGfB0xY9firIrFGd0Qf/5VDXQJQ2TcMJgXGEP7mkyQuDt7+DZ9Bzx0WfQeNqd/a+1xIjjqN+Bo24rjrotyJFKLFcAy52L6cnDcudhunMP/OvJbbdelxRvwP/+j/FoT5EsmUnj6X/LakKaGH0adZe8QO7LN5D37KU0zv8N8cmXYblzCc37CcE3vopz78ckhx8PRrLjWaZtkCyT+KhTiaXKDnTGUbcV32d/xr3pOUxfCXUXPmlXhhcGvcS4c6g/7wFcez4kWXIEySFzezzrM+MkidiE8/Et+wdSpIpkalF5Zf8yPjfzc7y4tpxnVu7tNEn7aFsNYxpWURGcSkneoa1oHu1pPFtewvQW49n0fPPjebmjeTAwnOWfjUYZci5G8TQsT37a4etD51J3yYtp79+W3BfsxEzSY723/FYvEsVsu0gpX4Fv2d+7vSROcthxmE4/OcSoaBRJWm9attueY6KW9mxwuaN2E6Fj7iA6+yvZ+xRsWcjhfTjqtuGo22InZLVbUOq2IjfuQmqx1qKl+JD0jmciWYo3lbgdSOQsdy7OPR8hh/YQnnu7XTS2i0lPXzAKJlJ76UsEX/8ywUXfJFK9gfBx3yc+4QJi298gOdSe5B189SaUag29aErqayp60RTM4Mh2f2568TQaznug0xgcdVvxLfkL7o3PgsNFdOYXiMy+tc2absLglRw5n+TI+dkOo0fi4xeQs/RvuLe+QmzqNRg5pTj3L2XqjBtwKzIr93Y+NOdPC1fzprSTmtFttKKZBt5l/yRZPJ26S19BilajVK7GWbkGpXIVk0PLOMH6AF54CAAjOBK9eBrJ4hnoxdPRi6d3/HtnWUjxui4ldy1jc+5fhp4/DktWSJYd0/Vz9DGRpHWRlGhEbtjd6k0yXXJor70grO9MAlKUqtDAmeI+ECxOLTNy9Khu/LI2cfrsT2Z9WO37EHoM97bXcdRsSrWObUWp29oq8bIUL3reWJIlMzEmXoiRNw4jbyxG3lh74LKp26/NWC1SvB4pXo8cq7P/jdfbg2Pj9cjxOqR4HY6GHan6SHnUXfxsvyvIa3nyqT/vQXIW/xzfyrtRajUazvgnjWf8o3mfxOgzsFwBlKr1uHa81fw7WXvJi+ils3Btfws5Um6PPytUUao3ICUj9jqd7SRxbSZns27JzNg8QcgCo3Ayev543JteIDbtWvQhc3CWLwNgcqmfFXsa2FkbYWQ7RWff2VzF/lCCPxfczo1Tz8I4aLt7yyso9duoP/MukCQsXxHJUSeTHHUyAPsbYpx2z1ucGtzLz+YkUCpW46xcjXvLKwdiDI6i4ax/Ny+s3pJ/0bdx7VlMzbUfd/m5O2o3IukRpFgdydI5/XLdZZGkdVFyxAnUXf5at47N+eCneNY9Rmz2pVy6ayRXuMV4tN5U3hhHlmDe2DSnjLdkWQQWfoX4+AUkxp3T+8G1QwpXkPvqTTjLl2NJMmZgBEbeGKJlR6eSsHEY+WPthbk7atWTFSxPPkZ3Pk32Vw4n4RN/gVE4Cf97PyTvqQU0nHM/Rv44AGLTriY27Wp7Xz2KUrPRLsJaYBfGdW94Es+WlwCwJBlL8WK5AtRc+4k9wrXlpeq24lvyV9wbnxHJmTC4SBLx8QvwffZn5PB+kqWzcW95BSlSyflTh7BiTwOPLN3DHadNaPPwPyzaQgw3J51/C8bBQ00sC++yv6PnjSMxto1WNmBI0EMgr5in6wLcOOEoSmZ77LBidShVa1EqV+Nb9g98n/6JhnPvO+R4I388jg2PI8Vqu9ya5ixfAYCjcRfxyZd36dhMEUlaBlmuAHIyRLBkNJ9Z9VxiioHHvak+lmRyaYCgp+sva6V8GZ7NL5IccUIfRNY2R9U6cl++HjlWS8MZ/yQ+5gxQPBm7/kARm3oVRv54gq9+gbynFhA+5jvopbPQCyaCkirVoXjRS2ail8xsPq7xzH8SbrgDpWqd/ce+egPxcee2mnQg120jZ+lfcWvPgMNJdMbNdnI20MYWCUIPxMefT85nd+Le/DKJkScT1mOAxNlTSvjVm5vYXNl2Xc+3N1VR3hjnWwUfMSFkkCw8udV25853cFatpeGUP3Y42efao0bwy4Wb+Pv72/n5OfaHLMuTR3L48SSHH4+UaMS35K/I9Tswc0e1OrZptrdSubrLY8qU8uVYihdJj5LI4N/+rhBJWgY1DUD3163hG8qT7N1zHUwUn9R7g2VZrNvfyJmTuvfm6ln3KJbiIz7+/F6OrG2ubQsJLvwqpifX7m5soxlfOCBZdjS1l75C8NWbCbz3Q8BuHTPyxtnj0QonY6TGppm+UnvmpSRj5o4mkTv6kNbRVsmZrBCdcZNIzoRByyiYgF44CffmF4nOvIlIoQrYCcLpajEfbqvBMK1DVgb449tbAItbjIcxNu9v7sJs4lv6dwz/0Ob1QdtzwbQh/P6tzby9qQrLsg4p+RGbdg2+Zf/Au/oBwvN+3GqbXmwXAu5OkuZo2IXpzkWSna0+4PUnIknLINNlFz0MNm7l68qz/KJyPtDpSldCGtaXh4gmTepjXV+uSEqE8Gx6gdiE8/t+TIJl4V3+L3I++jV6yQwazrkPM6e0b695mDCDw6m77BV7rcPqdakWsvU49y9tNWPM9BSkErcpBxK4ggngcCHXbydnyV9xa0+nkrMbicz6skjOhEEvPv58cj75HXLjXqREI47aTSTGn8cJ4wp5dX0F726u4pQWjQqbq8KUN8b5/Jgkzn01xEpbj2tV9n6Ka98ndrmbTsb4SpLE+dOG8NTKfazd18C0stYz6s2cIcTHnYtn/WOEj/pWqwXVLU8+RnAkSuWaLj/n+gUPU/DgMSSHHwdy/0yH+mdUhynLbS+eEEithWYlGrMZzmHlgy3VAMwZkdflY92bX0DSI8SmfL6XozqIESfwzvfwbHiC2PgFNJ76pwPddUJ6JBkzbwyJvDEkxp174OF4PUr1ehxVqeStej3eNQ8gGXEALFnByB2Lo25LKjm7gcisr4jkTBBSYuMXkPPJ73BvfhE5tAfvukeoGnsWs4fbCdN9n+xqlaTd8+F2clwOvj6hBvZBcmjrJM237O+YnoK0a4/desIYXlpbznNryg9J0gCiM27Es+l5PNpTxKZf12pbsnQ2ktH1iXiOhu04QnuJzPlql4/NFJGkZZCVaklzOuymXCkRymY4h5Xle+3yG/PHF3X5WOeu99ELVPTS2b0dVjMpWkPw1S/g2vcJ4SO/QeTIb4piqL3IcueSLDum9RR6U8dRty3V6rYeR80GEiPnE531JdF6KQgHMfPGkCyejnvzC0SP+CLSqvtQqtdTWDydPI/Clqpwc1fkG1olizZVs2BqKbk1r2A6/Rj5E5vP5ahci3vHIsJHf6f1Em8d8LsVjhqVx0tr9vOVeaMo8LWu7aiXziZZMhPv6vuJTbum1USqxtP/1uW/p96V/8G7/N8AJIb3z/FoAP23FPJhqHmNN9OepKzoIknrLduroyiyRGmg7aKtHWk845/Unf9onyVNjpqN5D91Hs6KFTSc8Q8iR31LJGiZICsYBROIT7iA8LF30HDufwnP+7FI0AShHfHxC3BWrMTw2b8jyn67FMfckXnopsWH2+0yR396ewsANx0zEmX/EvTSWa0mBviW/QPT6Sd6UItXZ44elY9hwT/e337oRkkiOuMmlNrNOHe9d8g2LAvMDpf2bkXZtwQ5XocRGIGZO7pLcWaSSNIyqGlMmmTZLySn0faMGaHraiMJCnK6XohVijfYtXv6qNvLueNt8p6+ACkZpe7CJ4lPuKBPriMIgtBT8fELAHDu+wzDV4Jz/1IALp89DICnV+zj9fUVVIUTzB6ey7A8L9EjvkR0xo3N53DUbcW95SVi06/t8motl8wcilOWeFOraie+8zB8JXhXtS7FISVCFN4/65DHO+KsWA5m0p7V2Y8/NIskLYOaWtIkPc59jsvY7p6U5YgODxWNMQwLJpUEunagHqPgoePxLflr7wdlWXhX3kvuy9dhBkZQe+nL6EP6rjtVEAShp8zgCJKls3BveQm9dBZKqqjtEcNycTkklu+p58537Va0n5xpzwCNqxeTGHN68zm8y/8FspPIjJu7fH1Zljl2TAGRpMHbmyoP3cHhIjb1atw7FuGo29r8sOXyYzmcKBWr0rqOFKnE0bgHydRJ9MOloFoSSVoGWS47iZD0MG+V3MgW1+QsR3R42FRpV+a/au6wLh3n3voqcqyWZG9X2zeS+N/9Pv4PfkJi9OnUXvwsZqCsd68hCILQB+Ljz8dZtZbE8OOJq5+zFyAHZg7LJRQ3qA4nmTMil7I8D85d7+Pa8nLzsXJoH54NTxGbfHm3eye+duIYAP7z0c42t0enXYMlO/Gsur/V43rRdJSq9GZ4NhWxtcBeB7gfE0laJsmKvc5ivJHxiXXkNW7MdkSHhZWpSQMTi7tWPsOz7lGM4Ch7maBeIsXqyH3pGrxrHyQy+1Yazr6n1XRxQRCE/iw+3p41LccbiBz5jeYB+jcdM9J+XDrQiuZd+R9yPvl987HeFXeDZRKZdUu3rz+6wMeQgJtNlWEiiUPHmFm+YuITzsez4QmkFhUS9JLpOGq3QKLzYURK9Tos7EK43VrzM4NEkpZhpjuIlKjnCzW/44rkM9kO57Dw2voKHJI9Oyhdct02XHs+JDb5il5bRF2p30be0+fj3PsJDafeSfjY72VvgXZBEIRuMP1lJIcehXvziyiVa1BS49JmlgUZGnRz87GjGJrrAcvCWb6MZKo+mhSrxbv2YeITLsAMjuxRDF8/aSwW8M7m6ja3R2fciJwM41n/ePNjevF0JCyU6nWdnj86/UaQHCRHnNSjODNBvINkmOUKIicaiUk+cohmO5zDQlU4Qa63a5MGvOsfx5IcxCZf2isxeHe+xZCXLkOO1VF3wePEJ/XOeQVBEDItNn4BSo1G4I3byPn4twAoDpnnbj6Km1Mtao76bcix2uaxtt5V9yHpESKzb+3x9U+ZWMTwPA/PrNzX5na9ZCbJIXPxrrq/uTtWL56GJSs4GtruJm3JufdjJMvot0tBtSSStAyz3EGkeANxRw4BKUokYWQ7pAGtJhwnaViMzO9aUVjTHSQ26VJ74fKeSEbwv3MHJW99GcNfRu0lL6KXHdWzcwqCIGRRfNy5WJKM5fTa47dSpS1kSWpesqmphS05ZC5SIoR31X3Ex5yJkVpSqidkSeKIYUFW7m1g8db2W9McDTtw7XgbANNXStUXNtjj6DrgqNuK/+1vYzlcJIf2/xV/RJKWYaYrgJRoJOHIwU+UilA82yENaO9tsev2zCgLdum46OyvEDrlDz26tlKxivwnzsKz9mHqp32Bfec9ecjiv4IgCAONlVNCsuwY5NBeJD2Co+bQ8dPO/UsxXUGMggl41j6MHK/vlVa0JlfNHQHAvz/c0eb2+NizMXKG4F11r/2AJIHi6fS8yv5lOKLV6MUzwNH1upqZJpK0DLNb0uoxnH5yiFHZKJK0nvh0Zx0AJ4wrTPsY19ZXkaI13b+oaeBd+nfynj4fSY9Sf8Fj1B35/zpdn04QBGGgiI8/H0fELoPRVC+tpcSoU4kc9U0wk3hX3E1i2PG9WmZofFEOxX4XG8pDhOJtFKl1OIlNuw7Xrveak0i39hQFDxwNevtDiVy7FwMQH3tWr8Xal0SSlmGWOxc50Uhj3iRWWuOIJkV3Z0/srosiSzB9aHotaXJoH8HXvoRvxd3dup7cuIfc5y/D//FviI85i9rL3+j3U7gFQRC6Kj7uHCxkLMWLM1UvraXEmNOJzrwZz4YncUTKicz5Wq/HcMnMMizg34u3t7k9OvVKLIcb7+r/AmA5c3CE9qBUb2j3nM69HwOQGNn/Jw2ASNIyznIFkOIN6LNu4evJr+JWHJ0fJLQrppscNzofh5xexWjPhieRLJPo5Mu7fC33pufJf+x0lMo1NJx6J41n/gvLk9fl8wiCIPR3lreA5IgT7LFbJTNbbXPUbrHroyVC+Jb9i2TJzD75sHrNkcNxSPDyuop2YiwkNuFC++96rA69aDoASmU79dL0GHLjbkzFh1EwMIrJiyQtw0x3EMlMEnAk8RCnLiK6O7srmtDZXh1hUmmaKw1YJp71j5EYdhxm3pi0ryPFGwi8cRvBhbdiFEyg9vLX7dmb/XgpEUEQhJ6Kj1+AHK9HLzmi1ePuzS+S+9qXcG96AUfDDiJzvtonfw+dDpkjR+bRGNfZWRtpc5/ojBuR9Cie9Y9jBoZhevJRKle3ua9StRYJC714+oD5+y2StAyzUut3Fqy/jw2eG1i7vfPpwkLbPtxWa1eMNsy09nfuXoyjYSexKVemfQ1l76fkP34G7k3PEz7ym9Rd9HS/XoxXEASht8THnoUlKXhW3Y/csKv5cWX/UvT8ifhW34eeP4HEmDP7LIYfnDGREr+Lhljbi6cbxVNJlB1td3laJnrx9HaTNEu2JwrEJl7cV+H2OpGkZVjT+p0+t13XS482drS70IGPttcCMHNYeuPRPOsexXTnpjdg1Eji++T35D13CUgO6i5+xh4kK6dfMFcQBGEgszx5JIbPw7PxaTzrHk09aOIsX4bpH4JSvcGe0dmHRbuHBD28+MWjmdbBuOPojBtxNO7Ctf0N9OJpKLWbwEgesp9r9/sAJMec2mfx9jaRpGVY0/qdfpedpJnxhmyGM6CtK7cT3CNHpresR3TmzYRO/GWn07Tlum3kPXMROUv+Qly9hNrLX0fv7fU9BUEQBoC4eiES4Nr5DmDXGZPj9cj1OzACw4lPuKDPY5A76ZpMjDkTwz8M76r7iMz6MlU3rgbHoQXOfSv+jZFT2vP6mBkkkrQMM925ACiO1IsuEcpiNAPb3voYXqeMx5ne5At9yGziEy/scB/3+scpePxMHPXbqD/zLhpP/ROWq2trggqCIBwuEmPOxJJke7kl08C5bwkASsMOe43ONpKhjJMVotOvw7XnQ+TQPnAeWtxcCu1DjlZhBHq2ZFWmiSQtw5pa0pqWsnAkRZLWHbphEk4YDA12XrwQyyLw1jdwpipTt8e56wOCi75FsvQIaq94g8T483opWkEQhIHJcgXQC6cimTqO6g3o+eMw/GWYnkJi3Zgl31diUz6PpXjwrr6fwJu34/ukdbFyj/YUYNd3G0hEkpZhTWPSJFMnYTlwmGJ2Z3es2mt3E08q6byVSylfZtfyCbe9DlwT78p7ML3F1C94ENNf1itxCoIgDHSxSfZAe8/Gp8HhxhHaS+SIL4DSteX4+pLlySc28XN4tGdw1G/HtevdVttd2xdhAbHJl2UnwG4SSVqGmanZnZbDzWVFz7Mp/+QsRzQw7a6PAXD1kcM73dez7hEsxUd8/Pnt7uOo24p7x1tEp109IJYKEQRByJTYpM/bXZ6Va/G/90NMp5/YtGuzHdYhojNuQDLiWJKEUrWuec1RAKV6PZbixcopyWKEXSeStExTvFiygpxoxO9WCMXFigPdsbEihNcpM64op8P9pEQjnk0vEJtwfodjyzyr/4slO4lOvaa3QxUEQRjY3H7iY89BKV+Gs3wZ8dGnNvcK9SdG4SQSw45HqdmEZMRx1G4CQIpUISdDGPnjsxxh14kkLdMkqXnVgTv2387xtc9kO6IB6Q2tErfi6HTWj3vT80h6tMPaaFKiEc/6J4iPXzDgPmUJgiBkQnzcOcipNTEjR/2/LEfTvujMm5DjdcCBlQdcuz8AsIvuDjAiScsCyxVESjQw2tzJMGt/tsMZcEzLojaSxO/qfFana9sb6AUqeumsdvfxrH8cORkiOvOm3gxTEAThsNGyJ8LMG529QDqRGHUqRmBEqnvWLmrr3P0+pjuXxJiBsah6SyJJywLTnYuUaCQq+fATxbKsbIc0oGwob8QCJpR03NUJ0HDOvdSf+9/2lwAxDbyr7ic5ZC76QevTCYIgCLbk0KMB0HPHZjmSTsgOe2yaZRIfdx5YFu4tr2J6C0EeeGtliyQtCyxXADnRQEz24peiRBJiXFpXvL+lBoC5I/I63lGPgqxgBke0u4tr59s4GnYQnXFjL0YoCIJwmHHlULfgYeoveiLbkXQqNvlyLMWHd/2jdvHdRAOY6S0f2N+IJC0LLHcQKd5AwpGDnyiV4US2QxpQVuypB+DEcYXt76RHKfzfsXhX3NPhubwr78XIGUJ87Nm9GaIgCMJhJznypAFRrd9y5xIffRruDU/hWfkfAJLDjslyVN0jkrQsMFNj0pJKDn4pSmVI1Errih01URyyxJAOCtm6t7yKHK1CL5rS7j6Oag3X7veJTr++f1TNFgRBEHpFfMIFSFh41z4EQHLEiVmOqHtEkpYFdktaIxunf49vJ2/BJXc8Q1E4wLIs4rrBqROKOtzPs/5RjOAoksOObXcf76r7sBzuDmd+CoIgCANPYvSpWJKMhD3mO9nB5LH+TCRpWWC5AsjJEL4hE9lqlRHVB2ZfeTaUN8ZpiBscMTy33X0cdVtx7fmI2OQrQGr7JS7FavFsfJrYxIuwvAV9Fa4gCIKQDbKCkTsaANMVwAx0Xvi8PxJJWhY0FQHM3/k6dyiPsLEinOWIBo5Fm6oAyOmg/IZn/WNYkoPY5Evb32fdo0h6TJTdEARBOEwlhx0PQMNZd7c/w7+fE0laFjQtDVVYu4wvOl5mZ41I0tL12c46ANQO1uy0ZCfxCRe0P8DV1PGu/i+JYcdhFE7ugygFQRCEbNNLpgNgdDDDv79TMnUhVVVl4J/ATCAO3Kxp2ubUtiHAYy12PwK4Q9O0u1RVXQ7Upx7fpmnaDZmKua80taR5XS5kySIWDWU5ooFjS1UYWYKxhb5294kc3XE1bNe213GE9hI64ee9HZ4gCILQTyRGnUrdhU8MiBmp7clYkgZcCHg0TTtWVdVjgD8CFwBomrYfmA+gquqxwC+Be1RV9aS2z0/3IqZpEolEOtwnGu14e18zLBe5gDM1YSAZqes05mzJ9r06WGUoQdCjEI1G29zuiJRjOdyY7rx2zxFYfg9J/3DqSo6HXrrv/e0+9WfiXqVP3Kv0iPuUvkF1r6QA5B8BCRMSXXvemb1PgXa3ZLK7cx7wGoCmaR8Dcw/eQVVVCfgb8GVN0wzsVjefqqoLVVVdlEruBrym7k5HKkmTEqIlLR3V4QS6aTEyr/3SG7nL/0rZ02dAO6s4OKvX4SlfQuPkqwdk9WlBEARh8MhkS1qQA92WAIaqqoqmaXqLxxYAazVN01LfR4A/AP8BJgCvqqqqHnRMK7Is4/O13xXWUrr79TY5WQyA02knCQ4jkrVY0tUf4lu42X75HDE8r914PDXrMYqn48tpe8mowEcPYyk+zJnX4HP3/nPqD/dpoBD3Kn3iXqVH3Kf0iXuVnmzfp0y2pDXQuk1PbiPZuhq4u8X3G4GHNE2zNE3bCFQDQ/s2zL7XNCbNcvq5U76OpLc0yxENDLXRJAAXTG9nfIERR6nRmgeLHkyKVOHe+DyxSZdiudsv4SEIgiAI/UEmk7TFwDkAqW7L1W3sMwf4sMX3N2KPXUNV1TLs1rh9fRtm37NcqVxVdvBe/qU0OjsuzCrYNlWGKQu6GZnf9icbpXoDkpkkWTyjze3etQ8hmQmxTqcgCIIwIGQySXsWiKmq+iFwJ/ANVVWvVFX1iwCqqhYDjZqmtRxMdC+Qp6rqB8DjwI0ddXUOGLKC6cxBitYwJ/4xcv2ObEc0IHyyvZYcd/s99EqlnffrxW20pBkJPGv+R2LkfIz8cX0VoiAIgiD0moyNSdM0zQRuOejhDS22V2KX3mh5TAI4LNfssdxBpFgtPwzdz8/064Hzsx1SvxaKJ2mI6wyT2580ABLJ4hmYwZGHbHFvfglHpILQjD/0XZCCIAiC0IsyOXFAaMFyBZF1e4qvzxpEU6K76ZPtdQBMHdL+VOXY1KuITb2qzW3eVfeh540lMXJ+H0QnCIIgCL1PrDiQJZY7iJSMkETBL0Wx2ikZIdg+3F4LwHFj8tvewdSR4g1tblL2L8VZscIei9bOWp6CIAiC0N+Id6wsMV0BpHgDUcmHnyjRpJHtkPq19eWNAMwdkdfmdqVqHUX/mYJr+5uHbPOuug/TFSCmtr+WpyAIgiD0NyJJyxLLFURKNBCTc/BLUSpD8WyH1K/trY/hdcp4XW330CuVqwDQ88e3elwO7cO95WVik68AV9u10wRBEAShPxJJWpZY7iByvIFteceyzhxFY0y0pLUnoZtEkwYnjS9sdx+lYjWmOxczOKrV4541D4JpEJ1+fR9HKQiCIAi9SyRpWWK3pDWy76ifcI9xHkjZjqj/2lIdxrRg/vj268kplavRi6aB1OJG6jG8ax8iMeYMzNxR7R4rCIIgCP2RSNKyxHQHkMwkAbOBfBqoiSSyHVK/9c6mKgCGBNxt72AkUKo3HLLSgGfjc8ixGlG8VhAEQRiQRJKWJZbLXpZowtIf8YzrJ6zc0/bMRAE+3VkHwLDctmukyeFyTP8Q9OKZBx60LLvsRoFKcthxGYhSEARBEHqXqJOWJZbbrveluNx4pSj1qXUphUPtqo3ickjk+VxtbjeDI6i55kNoUcbEufdjlOp1NJ78u9ZdoIIgCIIwQIiWtCwxXfYi626nCz8x6mMDf7WrvmCYFg0xnZL2ujoBzNS9a5GMeVfdi+nOIzbxoj6OUBAEQRD6hkjSssRyNyVpCl4pQSQWy3JE/ZNW0YgFTChqv3xG3jMXE3jz9ubv5YZduLYttFcfULx9H6QgCIIg9AGRpGWJlWpJczjsH4EVD2UznH7rvS3VAMwZmdf2DkYSpWotpvdAeQ7v6v8CEtFp1/V5fIIgCILQV0SSliVNLWlIDqqtIA5DtKS1pSaSRJElTpnQdvkNR81GJCOOXpya2ZmM4ln/GPFx52AGyjIYqSAIgiD0LjFxIEuaxqQZ+eO4IvggZcG2Zy4Odjtro6glfor9bY9JczatNFAyw/6+YgVyvJ64eknGYhQEQRCEviBa0rJF8WDJTuR4I363QighJg4czLIs1uxrpNjf9qxOsIvYmk4/Ru5o+/vy5QAkS2dlIkRBEARB6DMiScsWScJyBZAbdvCnqi8RrPg42xH1OztqI8R1k5hutruPHNqHXjwNJPul7KxYgREcheUtyFSYgiAIgtAnRHdnFpnuIFIizBh2EzDqsx1Ov/PBlhoAjhiW2+4+DefeD/qB8XxK+XKSQ4/q89gEQRAEoa+JlrQsslxBSE0Y8FnRLEfT/yzdXQfAieM6aRVT7PF8cng/jtA+dNHVKQiCIBwGRJKWRZY7iKzbyVkOEawWFfMF2FwVQZJgXDs10tzaU+Q/eipSxF7bUylfAYjxaIIgCMLhQSRpWWS5AkiJMAABKUo0aWQ5ov6lKpQgz+tEbmdZJ2f5cuTGPc3jz5zlK7BkBb1oSibDFARBEIQ+IZK0LDLdQaRkIzHJi58olaFEtkPqN6pCcXTT4pjR+e3uo1SsRi+e2jxpQKlYgV44RawyIAiCIBwWRJKWRZYriBRv4MU5/+Mf+gXopujubKJV2C2MF04f0vYOpo5SvQ692K6PhmWiVKxELz0iMwEKgiAIQh8TszuzyHIHkZNhXEXjqCVJJCG6O5t8tN2e2TmmwNfmdkftJiQ91rzSgKN2C3KiUYxHEwRBEA4boiUtiyxXAIBRG+/hC46X2Fwl1u9s8tH2WgD87rY/RyjVG4ADKw00FbHVS47o++AEQRAEIQNES1oWmW67/ldZ9WLOdMR4sS6e5Yj6j/LGOH63gtPR9ueI+MSLqB52HKavGLCL2JquAEb+uEyGKQiCIAh9RrSkZVFTS5pD8eAnSm00meWI+of6aIK4bjI8t+P1TM2c0gOTBsqXo5fMbP5eEARBEAY68Y6WRZbbXmRdcbrwS1EaYyJJA/hwm93VOWNYoO0dTJ28pxbg3vS8/b0eRaleL8ajCYIgCIcVkaRlkemyuzudTid+ojTGxSLrAO9vrQZg/viiNrc7ajfjLF8Opn2/lMq1SKYuxqMJgiAIhxWRpGWR5U51d8oyfqKEYiJJg6bxaA5mlrW9ZqdSuRqgeWans2KF/b0ovyEIgiAcRsTEgSyyXHZ3p5E3ll9JEyjKET8Oy7LYVRtj/vgiXErbnyGUilVYig8jz54koJQvx/CX2WPUBEEQBOEwIVrSsqhp4oDlCvBh4Gxkh0jSdtfFqI0mGV/Udn00AGdlaqUB2WF/X75CtKIJgiAIhx2RpGWT7MB0+pFD+zglvoj62spsR5R172y2F0uPJs22dzANlKq1JFNdnVK0GkfDDpIlYtKAIAiCcHgRTTdZZrkDyA27+F7iUT4X/VW2w8m6T3bYMztPmdD2pAEkmZor323+1lm+AhDj0QRBEITDT1otaaqqLlBV1dHXwQxGliuIZNoLq3vNSJajyb5NlWEcEowubKe7U5IwA2WYgTLAXlTdkmSSTWt4CoIgCMJhIt3uzkeBPaqq/klVVfFu2IssdxBJt1ca8DG4kzTdMKmNJCn2u5Elqc19vCv/Q84HP23+3lm+HKNgIrhyMhSlIAiCIGRGut2dpcAlwNXAMlVVVwMPAA9rmiYGUvWA6Qoix3cDkGNFsSwLqZ0E5XC3bn8IC5hU6m93H/eWV8BKjVezLJTyFcTHnZ2ZAAVBEAQhg9JqSdM0Laxp2gOapp0OjAIeBi4Fdqqq+pyqqheI7tDusVwB5EQYgBwpRiRhZDmi7Fmxpw7oYDyaaaBUrmmeNCDXb0eO14kitoIgCMJhqTuzOxuBaqAm9f1Y4F/AJlVVj+2twAYLy52LlAyxLHgqu6wS6gfx0lA7aqPkeZ2cNbmkze2Ouq1IegS9xO5xbypiK5aDEgRBEA5HaXV3qqqqAOdid3eei52oPQL8SNO0Fantd6UeG9POOWTgn8BMIA7crGna5tS2IcBjLXY/ArgDuLu9Yw4XliuAlAyx/vjf8e4rGl/R2yk9MQgs21WPWpzTbnevUrkKOLDSgFK+HEvx2mPSBEEQBOEwk25LWjnwBOAErgSGaZr2DU3TVgBomqYDrwGeDs5xIeDRNO1Y7ATsj00bNE3br2nafE3T5gPfA5YB93R0zOHCdAeRTJ28+D6KqKc6PDhb0sIJnd31MSrCiXb3USpXYykejPzxgF1+I1kyA2RRSUYQBEE4/KT77vYL4CFN06o62OcF4OkOts/DTuTQNO1jVVXnHryDqqoS8DfgKk3TDFVVOz3mYKZpEol0PEsyGu0/syhlyYMfOOHTG/mecyJr90xlapEr22E1y9S9em+r3Xs+pdjX7s8vrl6Lc8gJxGIJMEIUVa2hYfI1nf68M6E/vab6O3Gv0ifuVXrEfUqfuFfpyex9CrS7Jd2WtL8BX1dV9ctND6iqukRV1Z+kEis0TUtommZ1cI4gUN/ieyPVTdrSAmCtpmlaF44Z0Eyn/cMxHR4CRKmLDs5F1j/cbv+Y543Ja3cfI2cIsTJ72KOrVkMyEiSKREUYQRAE4fCUbsLzK+Aa4OYWj90N/ASQgJ+mcY4GWqeLcqqbtKWrgb908ZhWZFnG52t/3ceW0t2vLzmDxQDILg854RghvX/EdbC+jml9hf2p5bjxpfhch04Ulht24lt+F9GZN2PkjcVTv95+fOQx/ep+9adY+jtxr9In7lV6xH1Kn7hX6cn2fUq3Je0q4EpN015pekDTtLuB64Eb0jzHYuAcAFVVjwFWt7HPHODDLh4zoDUtsu5QXPilKA3RwTkmbV9DjIBbaTNBA3DuW4J3zf9Aj9nfl6/A9BZj+ssyGaYgCIIgZEy6LWl5wP42Ht8JFKd5jmeB01VV/RC79e0GVVWvBPyapt2tqmox0HhQl+khx6R5rQHDcucCqSSNahrjg6+7s6IxTsKwmD28/X55pXI1lsPdPJNTqVhhl94YpIV/BUEQhMNfuknap8Dtqqp++aAk6qvYMzE7pWmaCdxy0MMbWmyvxC690dkxh5WmljTJlUMtAYyORvUdptbubwTgi8eNancfpXIVetEUkBWkeD1K7WbiEy/OVIiCIAiCkHHpJml3AIuAU1VVXZp6bDYwBDirLwIbLEx3EIDkiHncVnkkMwPuLEeUeZ/trMMhwcSSdpaDskyUyrXE1c8BoFTY9dJEEVtBEAThcJbuslCfAtOBp4AcwAU8CUzSNO3Djo4VOuHwYMlO5HgDfpeD0CDs7nx3cxUWoMhtd1066rcjJ0PNRWyd5csBmlceEARBEITDUdrlLDRN24ZdaFboTZKE5Q6ilC/noYZH+FzjncC0bEeVMYZpURVOUOBz4WgnSTO9RTSc8S+SQ+cAoJSvQM8f3zyeTxAEQRAOR+kuC+UBvojdmtY0/U4C3MBcTdPEujw9YLoCSMkIJVItTiOU7XAyanNlCNOCCcU57e5juYPEJyxIfWPhLF9OYuRJGYpQEARBELIj3Za0fwCfx55AMA94DxgHDOcwXKop0yx3Lph26Q23ObiqQb+9uRqAY0fnt7uPd+V/MHLHkBh9KnJoL3K0UoxHEwRBEA576dZJWwBcl1pbcytwKzAWexmodkZ7C+myXAEkw16z0msNriRtyc46AE4cX9j2DpaJ79M/4trxFmAvqg6glx6RgegEQRAEIXvSTdJygU9S/18LzNE0zQB+TarYrNB9ljvYXKTVTxTLGjx1OPY1xMlxOSgLetrc7qjfjpxobDVpwHK40QsnZzJMQRAEQci4dJO0fcCw1P83Ak3T6upJv5it0A7TFUTS7RY0P1HCicExwzOWNKgOx7l8VhlSO0VplUp7kYlksf2SU8pXoBdNBUf/WYReEARBEPpCuknaM8B/VVU9FngTuE5V1QuAHwFb+iq4wcJyBZETEe6f/QJvmbMJxY1sh5QRa/c1YFgwdWiw3X2UilUHVhowdZyVq0iKrk5BEARhEEh34sD3ACcwRtO0R1RVfQF7PFojcFlfBTdYWO4gkhHFkzeUOCHCicGRpC3UqgAwO+jeVSpXoxdOAocTR9U6JD2KLiYNCIIgCINAukna9cAvNE2rANA07Quqqn4DiGmaNjj65vpQ09JQc1bewXnyFLZXT2ZcUfslKQ4Xq/Y2ADB7ePv1zqIzbgDTTlqbitgmS47o89gEQRAEIdvSTdJ+A7wNVDQ9oGna4Cro1YfMVFHWMbUfMEP2sD8Uz3JEBzywZC9jC7ycOc3X6+feXRfF53QQ9Djb3Scx9uzm/ysVKzDdeZi5o3s9FkEQBEHob9Idk7YcOL0vAxnMmlrSDIcXPxHqIoksR3TAfZ/t5Yevb+nWjFPX1tdQ9i1pc1t1OE5MNxmR3/asTgClcg1u7enmma/O8uV26Y12JhkIgiAIwuEk3Za0CuCvqqp+H7tOWrTlRk3TzujtwAYTK7XIuuVwE5Ci1EWTWY7IFmkxNm5XbZSRBem3pknRanJfvZnkkDnUfe75Q7Z/tL0WgFnD2u/qdG96Du/K+4iPXwCJMI6ajcRbtKwJgiAIwuEs3Za0KPA/YCGwGdhz0JfQA6bLTtIkxY2fKPXR/jHMb2t1uPn/z68p79Kxrh1vAxA+6tsAeNY9RuCNryHX7wBg2a56AOZPKGr3HEpF06QBF87KVUiWiS7GowmCIAiDRFotaZqm3dDXgQxmTS1pssNJjhSjMd4/krRt1XbtNpdD4rFlu7n+qOEEOhg/1pJr+5sYvhKSw48HQIrX4d7yCu7NLxKbehWJhjOYUOxvf9KAZaFUrSE+7jzAro8GiOWgBEEQhEEj3QXWr+xou6Zpj/ROOINT05g0ffhx/LtmJEU5/aNQa1O366xhAT7Z2cC/Fu/gO6eO7/xAI4lr17vEx50Dkt1YG511C/GJF+L77C941jzEneZjfFBwCZI5DRzuQ04hN+xAjtejl6RWGqhYjhEcheUt6L0nKAiCIAj9WLrdnQ+18/Uf4Kd9Etkg0pSkWb5iNvjm9JuB8X63ncPfPm8kDlnitfXpdXk693+GnGgkMfq0Vo+bOUMIzf81q855lTfN2UwIfQpyqmXObN166KywVxrQW6w0IIrYCoIgCINJWkmapmlyyy/swrZTgU+Bn/RlgIOC7MB0+nHUbuY8/XV21PSPRdYrUqVAivwu5ozIpTFu8ElqwH9HkkOOpO7CJ0gMP7HN7QvL/dyW/BrPTLsbJBlH1ToKHjwOz9qHm5M1I280kRk3oReqyOFyHKG9ooitIAiCMKik25LWiqZphqZp64FvAr/o3ZAGJ8sdRKnZyLf1/7C3Ptr5ARnwplaJLIHLIfO1E8YAcNfi7Z0f6HCSHHYcuNouyPvpzjoA5k0ssx+wLEx/GYF3vkv+Iyfj3vQietFUwif8DBzuA+PRxKQBQRAEYRDpVpLWgg6U9UYgg53lCoBl4JQMMPpHMduGmI4i212vk0oDFPqcrCtvJKG3v2yV3LCT4EvX4qha1+4+myvDOCQYXWiX9DCKp1J38bPUn3M/OFwEF36ZggeOQg7vB+z6aJasoBdP7cVnJwiCIAj9W08mDgSBLwKf9GpEg5TlzkWKVALgNvpHd2ckYeB1Hsjjv3riGH722kbe21LDaWpxm8e4tr+Je8ciQvN+1ub2hG5SG00yNOhGbjn2TpJIjDmdxKhTcG96Fu+q+3HUbMTMGYJSsQK9cDIo3l59foIgCILQn6VbzPahNh5LAh8BX+m9cAYv0x3EEdoLgM/qH0lawjAp9B2YeXn25FLu/nAHT63c226S5t7xFnreOMy8MW1u31DeCMDkUn/bF5UdxNVLiKuX2N9bJkrFSuITL+r+ExEEQRCEASjdOmk97RYVOmG5Aki63c2ZQwzTslq3NGVY0jAxLcjzHniJOGSJ6WVBFm6oZOWeemYevFpAIoxz90dEp1/f7nk3VNhLvn7txLaTuIM5arcgJxrFeDRBEARh0Ek7+VJV9SZVVa9o8f0zqqpe1zdhDT6WO4hkxHkveD71+FotyZQNu2vtyQvFB9Vsu2TmUAD+8cH2Q45x7f4AyUyQGH1qu+dds6+BohwXw3LT67pUKlYAiJmdgiAIwqCTVpKmquq3gT/TuuVtHfB3VVVv7YO4Bh3TFURKhtFm/JDdVgmhLK86YKTWU583Jq/V47OG55HrUVi5p/6QCQSuXe9iOv0khx7Z7nnf3VKDaVlIabYSOsuXY7oCGPnjuhS/IAiCIAx06bakfQW4WtO05rFpmqb9ELgeuL33wxp8LFcQyTIoDm8gnwZqsrzIelXY7not8R+6+sF5U0sxLXjg092tHg/N+yl1n3sOHG2vmNAQSxJJGOR701taCuwitnrJzOaVCwRBEARhsEj3na8UWNvG4yuA4b0WzSBmue1VBy5cfi3nOD5lZ5YL2r63pRoAt3Joi9cXjh2FBDy9am/rDQ4XRuGkds/5yQ67EO6MsmB6QehRlOp1YlF1QRAEYVBKN0lbDVzdxuNXABt6L5zBy3IdGITvJ0p1OLstaTtq7DFpZcFD19XMcStMHRqgOpxsTia9K+4m+MpNhyzv1NL7qcTvpPGFacWgVK1DMnWxHJQgCIIwKKVbguNnwIuqqp6IvRQUwFzgJODivghssDFTLWkmMn4pSlUku0ladSQBQNDTdtfkbxdM5vx7PuX5Nfv52oljcW9+CSwT5PZfUmv22eU3Zg3PSysGZ/lyQEwaEARBEAandNfufBU4AdgPnAucAZQDR2ma9mLfhTd4WC67C9B0ePATpT6W3SStPqrjdLQ/uL8k4OH4sQU8s3IfRqgSpXx5h7M6LcuivDFOwK3gcznSikEpX47hH4qZU9rl+AVBEARhoEu3JQ3sFrTbNU2rAFBV9ThgTZ9ENQhZbjtJsxQ3gUSU+lh2Z3dGEjoepeNkqjTgJpQwWP7+M5yNRWJU+0navoY4CcPiK/NGpB2Ds3yFaEUTBEEQBq10S3BMBDYB/6/Fw88Cq1VVTa8qqdAhs6klzVdKlRXEsqysxhPXTQLujpO0m48ZBYBz25sYvhL04mmttud89CtyPvgZzp3vsGGPveTV7BF5aV1fitbgaNghitgKgiAIg1a6LWl/BZYBv27x2ATgfuz6aRf0bliDT9PszqR6IXfVzeHUnLbLWGQkFsvCIUvMHpHb4X4FOS4mFHqYGtKoLj0FqUWZDEfdVnzL/gmAb+U9XISTMuckxm09B4dyGkbBJOigVpoYjyYIgiAMdukmaccBczRNq2l6QNO0BlVVf4C9fqfQUw4PluxCTjTgdztozOKYtHDCIGFYjC3M6XTf648ZzUkv38lpETctl1R3bX0dgJor30Fu3M3LLz3GXGkFZct+C8t+i+ErJTnyRBIj7C/L23rGp1KxAkuSSRbP6M2nJgiCIAgDRrpJWgQow+7ybKkIyO76RYcLScJyB3FtW8hd0Ve5cc/vsxbK+tQi6On0uJ42sYhfvO5m4S6LHxomTofdmube9jrJomkY+eOJ547jh1GZIv/1vHbVSJy73se1811c297As+FJAJLF00mOOJHEyJNIDpmLUr4Co2AiuDpPFAVBEAThcJRukvY08C9VVb8EfJZ6bC7wL+D5vghsMDJdATASFNNIXDezFsf6cnsRdKWD2Z1NCp5ewD/K5nLTzlN5Z3M1p6vFSJFKlP1LiRz5DQC08kYsYGKxH9NfRnzy5cQnXw6mgVK5Gteu93DufBfvin/jW/YPLMUHpk5MFdVdBEEQhMEr3STtu8CTwLtAU/uKBDwHfKP3wxqcLHcQKVqNX4qSMLKXpO2pswvZji7wdbifXL8DZ8VKjjz+IobWuXl21T5OV4txb38DCYv42LMAeGezXcT22DH5B53AgV56BHrpETD3NqREI849H+Ha+S7K/iXEJ5zf689NEARBEAaKtJI0TdNCwNmqqqrANCCJXTPtaOADoNOBQ6qqysA/gZlAHLhZ07TNLbYfCfwJO/nbj71WaExV1eVAfWq3bZqm3ZDmcxtwLFcQOVxBDjEMI3u9yPsb7XU7xxf5gPaTRdeOtwBIjj6VE6oNnlixl+3VEWZsW4gRGIFROBmAZbvrADhhbEGH17VcARJjziAx5oyePwlBEARBGOC6tGq1pmkasA97lYG3gb+Q/pi0CwGPpmnHAncAf2zaoKqqBNwD3KBp2jzgNWCUqqqe1HXnp74O2wQNUrXSTHvCgMeMZi2OpiWpiv2HLgnVknv7W+h54zDzxjC+2G51u//9dbh2vU98zBnNszdDcZ05w3MZGvT0beCCIAiCcBhJqyVNVdVc4Frgi8CU1MMLgd9pmvZ2mtdqSr7QNO1jVVXnttg2EagGbldVdTrwsqZpmqqqRwM+VVUXpmL9vqZpH3d0EdM0iUQ6Xpw8Gs3u4uXt8cheFMNuxfIRJRQOI3dQpqKv1EbiOGSIRqPt3ispGaZoz4c0Tr6aSCTCKWOC/E6WkLe/g+SM01A2n3gkQiRhsK06yklz84lGs5d49rX++prqj8S9Sp+4V+kR9yl94l6lJ7P3KdDulg5b0lRVPV5V1QeAvditZnHge9h9YN/qQoIGEORAtyWAoapqU5JYhF3m45/AacCpqqqeij2r9A/AmcAtwMMtjjnsmK4gkqHzx+mvsp8CosnsjEsLuBWGdNKK5qzdhCUrRIfPt793yBw7KpeT5SXElCDx0jkArNjbgEWHJdEEQRAEQWhDuwmPqqprgMnAcuCXwBNNY8hUVf1lN67VQOt0UdY0rWnto2pgs6Zp61Lnfw2Yg50YbtY0zQI2qqpaDQwFdrV3EVmW8fk6HvDeJN39MkXJKUA2ouTl5gG1GLITny/zXYSmJTGxNNDq/hxyr0YfR/VNq5FlBV9qUfVbjx/BpCeX87Z5JHP99goKH+/aDcCE0tx+d7/7wmB4jr1F3Kv0iXuVHnGf0ifuVXqyfZ86akmbhF0X7SXgvZaD/LtpMXAOgKqqxwCrW2zbCvhVVR2f+v4EYC1wI6mxa6qqlmG3xu3rYRz9lplav/PEFV9nhrSFvfXxjMdgWRb7GmK4HR28NCwLjAQoHpAP5PmTkuvIk8I8FzuCXbV21+bqfQ0AHDkyry/DFgRBEITDTkdJ2jDgP8DngHdVVd2rqupfVVU9kQNlOLriWSCmquqHwJ3AN1RVvVJV1S9qmpYAbgIeUVX1M2CXpmkvA/cCeaqqfgA8DtzYovXtsGOl1u+cEPqUYVIVFY2xjMdQFU4Q003qO1jxQKlaS+G9M3DuXtzqcde21zFkNx+YM3hutZ1L766L4nM6yPU6+zRuQRAEQTjctNvdqWlaOfZ4sD+kymNcD1wJ3Jra5RZVVX+vaVq7XY8Hnc/EHlfW0oYW2xcBRx10TCJ1zUHBSrWkAfilKNWRRMZj2FIVBqA00P6YNNeOt5CTIfSCiQcetCzc2xaijzyBI/VhvLB6PxfPGErCsBhX5O3rsAVBEAThsJNWCQ5N0z7TNO1W7PFgVwCvAl8Gtqqq+kwfxjeoWK4DQ/b8RKmNZL7RcHuNPaNlWF77Y+Fc298kWXIElq+4+TFH1TocjbtJjDmT0QU+6mI6v19k95DPGt7xQu2CIAiCIByqq3XSkpqmPalp2nnAcOD7wIQ+iWwQMt0Hkhk/UeqimV9kfU+d3cU6Kr/twZJStBqlfAWJ0ae2ety97XUsJOKjT+e4MXbR2sXbanFIcOWc4X0btCAIgiAchrpdziLVHfr71JfQC5pa0kzZSY4Uo6GDcWF9ZV9qtYFxhW0naa4dbyNhkRjVOklzbXsdfehcLF8RR3gt8rxO6qJJJpb4O+w6FQRBEAShbV1qSRP6VtOYtMiEz/G0eVKnFf/7gkOy1+Vqr7tTDu/HCI5CL5524LGG3Tir1hIfcyYAkiRxycyhAFldKF4QBEEQBjKRpPUjliuAhYQUHEqVezSG1Z1JtD2T41Io8rtwyG2/NKJzvkrNVe+BdGC7e9vrAK3W3Lx0VhkAhT5XH0YrCIIgCIevw7Z6/4AkyVguP0r5cs6wTDZWnZ7xEHbXRclvp1yGFG/AUrzgaL3dte119PyJGHljmx8r8Ll46OrZDAmKrk5BEARB6A7RktbPWK4gStVaPme+zp76zNdJW7O/kZpI22PhfEv+QuEDR4JxYLsUq8W595NWrWhN1FK/qI8mCIIgCN0kkrR+xnLbkwf8Uiwr47mShkXQ03YDq2vHW+hFU1q1pLl2vIVkGcTHnpmpEAVBEARhUBBJWj9junLBsghIURJGZpO0xrhdl63Qd2jrl1y/HaV2M4lRp7R63L1tIYavFL1kZkZiFARBEITBQiRp/YzlDoBl4idK0sjsxIEtlfZqAyVtlMxwb38LgHjL0ht6DNeOd+yuTkm8lARBEAShN4l31n7GcgWRTJ0cIhhmZpO0banVBspyDy2/4dqxCD1vLGbemAOP7V6MpEeItzEeTRAEQRCEnhFJWj9jt6QZvBG8BDAxM1iGoya1Vujo/IPW2rQsLMVDYuw5rR52bXsN0+knOfy4TIUoCIIgCIOGKMHRz5iuXCQ9xqZJX8eq2EYkYeB3Z+bHlJ+qaTZj2EFrbUoSDefce1CgBu5tb9hj1ByizIYgCIIg9DbRktbPWK4AkmUwvP4zvMSoiyYydu2qkL0kVGFO6wK0SsNOMFsv9q6UL0eOVrVZekMQBEEQhJ4TSVo/07Q01CXabYyR9rO/IZ6xay/UKnFIEk5Hi5eFZVH66pUEFn2r1b7uba9jyc5DZnsKgiAIgtA7RJLWz1iuYPP//USpCmdukfX6aBLFIbV6zFmzHiVSQWLY8QcetCxcW18jOezY5qRSEARBEITeJZK0fsZskfTkSDGqw5nr7owmTXxOR6vHfLvexkIiMerk5scctZtR6rc1L6guCIIgCELvE0laP9OyZSpAlNoMjklL6OYhqw14d79Domg6lq+4+TFX84LqmV9bVBAEQRAGC5Gk9TOtujulKPVRvYO9e09cN7CA/JarDSTCuKrWECs7vtW+7m2vkyyZiekvy0hsgiAIgjAYiSStn2nq7oznDKPeyslYnbSmQrYl/gPlNORYLbGhxxIbesyBx8LlOMuXi1mdgiAIgtDHRJLWz1gue4H15PRrWCgdS57X1ckRvUPGnjAwb2xB82NmcDgVZ95HrOzY5sdc294AEOPRBEEQBKGPiSStv1E8WA43cqKBHKdMQywzszurUhMUWi4J5ajbekh9NNe21zGCozAK1IzEJQiCIAiDlUjS+iHLFcCz5kG+r/+T5bvrM3LN97dUA+Btmt1p6uQ9cTb5n/6qeR8pEcK1e7HdiiZJbZ1GEARBEIReIpK0fqhpXFpQjhJNGhm55sGLqyvVG5CTYeLFs5r3ce58B8lMkBgrujoFQRAEoa+JJK0fsselSfil/9/enYdHUaQPHP/2nLkmByGQACFcUiAIiqCiiIp4i8p6rLuuoiCgP1A8uQRRREUEF1ZlBUVBRREXj0XRRV3FRUS5BA3QXOFOIAnknsyR6d8fk4RADgbIMUnez/PwPElXV3dNpSEv1VX1FlLo9dXKPY/ke9CgNE+oJXUNAK7m55eeY9/1Nb6QJnjie9ZKm4QQQojGTIK0IGTY/QnOHZoTT1HtBGk5hR6sZbINWFPXUBSRQFHJNhtFHmx7/ourzVVgMldyFSGEEEJUFwnSgpBhc4Dhw4ETT1HtbMGR7yk6Nh/NMLCm/oon4YLScuvB1ZjcObL1hhBCCFFLJEgLQj57JBhewrXCWtsnze31lb7qxOukqElHPK0uLS23p3yNYQnBndi3VtojhBBCNHaWk58iapthi0Qz4M1uS/D9coAin4HZVHOrKQ3Df/1zW/pfs2INI/umD/xfFxT4E6qnLMedeBlYQ2usHUIIIYQ4RkbSgpBhj0QrKiTK7n/9WOCu2RWeea4iPEUGHeLCATDlHTxufzRbZjLmvFTZwFYIIYSoRRKkBSFfcf7OGzcNI5TCGk+yvjktF4CSdQNRX9xD5FdDS8tD936HoZlwt+lfo+0QQgghxDESpAUhw+5PDdW2cDMOnBzOddXo/ZLTcgAwmzQ0VzbmTB1vs+6l5WF7v8WT0AsjtElllxBCCCFENZMgLQgZtqjSrx1aARn5NZsa6kB2IQBtm4RhTVuHhoEnoRcAlty92I7quNteW6NtEEIIIcTxJEgLQiUjaQAROMnMr9nXnSUjde3jwrGkrsXQzHia+zMNhO79DgCXbL0hhBBC1CoJ0oJQyZw0gAjNWeNz0jIL/CN1MaFWrKm/4o3rCtYwAML2foc7RuGLSqrRNgghhBDieBKkBSGjbJCGkyynt4qzz1y204PFpKEV39udeJm/wJ2P/dB6nK0uq9H7CyGEEKI82SctCBnFCdazO93Fr7914k/hthq9X7jdjM1sAk0j54a3S49bU39FM7wUJvSWaF4IIYSoZfK7NwgZtggMNCwRceSbo2o8f6fPB6p5BJrzyPH7ox34CcNkxdW8R43eXwghhBDl1dpImlLKBMwGugMu4H5d13eUKe8FvAJoQBrwN8BdVZ0GSzNhWCOw7vuRS0zhbE+PrrFbGYZBWq6Lc1o4cPwwGnP2Ho7e+Q0A1gM/44rrjmGRLANCCCFEbavNkbRbgBBd13sDY4EZJQVKKQ14E7hP1/U+wNdAUlV1GjrDHon18G9caGziYPEWGTXhUK4Ll9fH0XwP1tS1eJt2AUBzZWNJ/53ChItq7N5CCCGEqFxtzkkrCb7QdX21UqpnmbKOQCbwiFLqHOBLXdd1pdTwKupUyOfzUVBQUOU5TmfV5cEgyhqBSTPj0JwUuL0n/UynK/lAFgCdrWmYnBnkx3ajoKCA0L0/ohk+spucWy/6q65JHwVO+ipw0leBkX4KnPRVYGq3nxyVltTmSFokkF3m+yKlVEmQ2BS4GP+rzf7AlUqpK09Sp0Hz2RwYmgmHVojLW3Nz0vZm+UfpztO2AeBqdj4AIamr8ZntFMZ2q7F7CyGEEKJytRnw5HB8uGjSdb1klnomsEPX9c0ASqmvgfNPUqdCJpOJsLCwgBoU6Hl1QQuNQUMj0uTEXWTUWFsP5/uTt3fxbcVnj8La4hysmomwQ7/iTehFSEQ0ENx9FUyknwInfRU46avANKZ+KirycvRoOl7vqe2j6fMZAHg8+TXRrAajJvrJYrERExOH2Rx46FWbQdpPwABgsVLqIuD3MmW7gAilVIfihQGXAvOAnVXUadAMmwMwcODEW2TU2H3SirMNOEKs/v3RNBOaMxNL5hbyLxxTY/cVQghx+o4eTSckJIzw8Hg0TQu4XlGR/z/mZrO5pprWIFR3PxmGQX5+DkePptO0aULA9WozSPsUuEoptQr/Cs77lFJ/BSJ0XZ+rlBoCfFC8iGCVrutfFq8IPa5OLba3Thn2SAzNzLro6zCl19x9Sv5qe6+eQW7xX3TrgZ8BcLe6uOZuLIQQ4rR5ve5TDtBE3dE0jfDwSPLysk6pXq0Fabqu+4AHTji8tUz5f4ELAqjTKPjsUWhFhWS0GYg7bS9FPgOzqfr/MkaGWmkTXoTJ8IHm/x+D7cAqfNZwvHHdwFWzyd2FEEKcHgnQ6pfT+XnJZrZByrA50AwfSUf+B0C+u2ZSQ6VlFzLC/Amx75xXupGt9cAqPAkXgNlaI/cUQghR/7hcLm67bUCl5evXr2XSpHG12KKGT4K0IFWSGur2vU8DcLSgZka0Nh3MQbmTKYpuByYLpvw0LEd34Gl1SY3cTwghhBCBaRTbWdRHvuIk6zY8WPCSnuciqUn1rlwyDAOTz0VnduFJuB8A6/5VAHhaynw0IYRoSJYtW8rKlStwuVwcOZLJ7bf/hf/9bwUpKTsZMWIUTqeTxYs/xGq1kpjYmtGjn8LtdjN58gRyc3Np2bJV6bV27tzBzJkvYxgGUVFRjBs3qQ4/WcMlQVqQKhlJAwinkIz8U1tmHYhsp4du2i6seClI8E8HtB5Yhc8eVZp5QAghRMNRUFDAjBmv8v333/LRRx8wd+58NmxYx6JFC9mzJ4V33llIWFg4//jHDD7/fAkAbdu2Z/jwESQn/8H69WsBeOmlKYwb9zRt27bjiy8+Y+HCBfTqdWFdfrQGSYK0IOXfgsPPoTk5UgOvO3dk5NPT5N/E1hPvT+ZgO7AKT4uLwCTLs4UQoqE56ywFQESEgzZt2qJpGg6HA5erkLZt2xEWFg5A9+49WLNmNQAXXtgbgC5dumKx+MOGPXtSmDFjKuDfsy0xMam2P0qjIEFakDLsUaVfR+CskTlpu48UEKEVkBamMIc2wZSzD3POXpzdhlT7vYQQQtS9ylcYauzenYLT6SQ0NJTffltPYmJrNM3EH3/8zqWXXs62bVvxev0LzFq3TmLChMnEx8ezadNvZGZm1N6HaEQkSAtSvuKRtPyIdnhcZnxG9W9om5HvZp73TmIv7sSV+F91Arhl0YAQQjQqZrOZwYOH8/DDw9E0E61aJfLAAyMxm828+OKzPPjgEJKS2mC1+lf9P/74OKZMeRqfz5+2cOzYiWRk1OCmno2UZtTAL/+65PEUGVlZVSdGLUlWHtQpRLyFxM3pQM4Fo+n247k8eEkbBl/Uulpv8emG3Uz9726+GN6buAg7jm9HYdv7A5n3/QbF/9uqF30VBKSfAid9FTjpq8A0xn5KS9tDfPypv2KUjAOBqal+qujnFhfnWAf0rOh82YIjWFlCMMx2zK4sws1FHCmo/oUDLfd+zm/2ocQaR8AwsO7/CXfLi0sDNCGEEELUHQnSgphhDSd841wGsIJNB3Oq/fqW1DW4sWKKiMecnYI5P0223hBCCCGChARpQcxXvA2HAycF7qJqv/7ZRVtYbyjQtGP7o8l8NCGEECIoSJAWxAybf4VnpMmJ01O9QZop/xCJHOJ3c2fAv2igKLw5RVFtq/U+QgghhDg9EqQFMSMkGkMz4dAKKfT6qvXaltQ1AOy0dQHD8O+P1vISmY8mhBBCBAkJ0oKYfxsOE5GmQtxF1RukeXPSOGJEkBmhMB/ZhsmZIfPRhBBCiCAiQVoQ86eGMggx+fD5qnerlI0Jd9DT9QZNIiOwHvgJkP3RhBBCiGAiQVoQM2wOMFlY2flZzKZqfA3pK8JiGPgwcVGbGGwHVlHkSMQXmVh99xBCCNGoLFny0RlfY9iwe0lNPXjK9fbs2c3IkcPO+P4nc9NN11Ralpp6kGHD7q3W+0nGgSBm2KPQilxEWg2cHh9en4GlGoI168HV9P7yfrppo2kV1Q3rgZ9xtbu2GloshBCitn2ZfIh//5EW0LklG9hXnh7K76au8dzQpfkptWPBgre59dY/n1IdUTUJ0oJYSWqoa7aM4Q1Gke/yEhVqPePrWlPXYPbmsduIp7lzOyZXtsxHE0IIEbC9e/fwwgvPYrFYMJvN9OjRk5ycbKZPn8qDD45k6tQp5OXlkp2dxYABAxk48DZGjhzGWWcpdu3aSUFBHs899xLx8QnMmfM6v/zyM82bNyc7OwuAw4cPMX36VNxuFzk52dx771D69r2cu+++g8TEJKxWKw899BiTJ0/AMAyaNImtsr3r16/l/ffnY7VaOXz4EDfffCvr169lx45t3H77Xxg48DbWrFnN3Ln/xG6343BEMmbMBBwOB9OmPU9Kyi5atmyF2+3fWP7QoTSmTXsBt9uFzWZn9OjxNdLPEqQFMaN4n7Q2nh0AZBW6qy1I22tOIodwWmavBZAgTQgh6qkbujQPeNSrutIdrVnzC0p14qGHHmPjxg3ExMSwZMlinnhiLLq+lf79r+ayy/qRkZHOyJHDGDjwNgA6d+7CqFGPM2fO63zzzX+45JI+bNy4gbfeehens4A77/wT4H99eeedd9GjR09+/30j8+bNoW/fy3E6ndx77xA6duzEa6/NpH//a7jppoF8991yPv30X1W2+fDhw8yf/wFbt27h6afH8tFHn5Gefpjx45/klltuZdq0F5g9+y3i4prx0UcLee+9t+nRoxdut5u5c+eTlpbGDz98B8Drr8/ittv+TO/el7B27a+88cZrDBv2f2fUpxWRIC2IGTZ/kBZq+PPSped5SIo5w4v6irCkreM3+qABYamr8Ua3wxeRcIYXFkII0VjceOPNLFy4gMcff4jw8AiGDx9RWhYbG8vixR+wYsX3hIWF4/V6S8s6dlQANG/enMzMTFJSdtGpU2dMJhPh4RG0a9eh+BpNWbBgHl9++TmgHXeN1q3bAJCSsotrrrkegHPO6X7SIK1du/ZYLBYcDgctWrTEarXicETidrvIysoiLCycuLhmAHTrdh5vvvlPYmKa0LlzFwDi4+Np1swfDO/atYP33nuHhQsXAGCx1Ew4JQsHgljJSJrNKAQMMvJcZ3xNc+ZWTJ48fi3qSIjZwHrwF//+aEIIIUSAVq5cQffu5zFr1j+54oorWbhwQel8tw8/fI+uXbvx9NPP0a9f/9LjUH4uXOvWSWzZkozP58PpdLJ79y4A3nrrDa699gYmTnyOHj2Ozz1eco2kpCSSkzcBsGXL5pO2uappeNHR0RQU5JORkQHAxo3rSUxMJCmpTek9MjLSSU9PL253Gx588CFee20uTz45nssvv/Kk9z8dMpIWxErmpJkwCMXFkXzPGV/TnLMbw2xntesszrfuxuTJk1edQgghTkmnTmczefJEzGYzJpOJhx56jNTUg0yePJEbb7yZ6dNfZPnyr4iKisJsNpfO5TrRWWcprriiP/fffw9Nm8YRE9MEgCuuuJJZs6bz3nvv0KxZc7KyssrVvf/+B5k0aRzffrucFi1antHn0TSN0aOf4qmnnsRk0oiIcDB27NPExsayadNGhg4dRHx8AtHR0QCMGDGKGTOm4na7cbkKGTXqiTO6f6XtKhvhNgQeT5GRlVVQ5TkFBf7ysLCw2mjSaTPlHiD23QsB6FU4mxsuOIcRl1ZD2qYiNxfO+pkx4V8x3Ps+Gff9hhHWtMJT60tf1TXpp8BJXwVO+iowjbGf0tL2EB+fdMr1qmtOWkNXU/1U0c8tLs6xDuhZ0fkykhbEjOKRtPRO95H1WwTRIdXw4zJ8GCYrmqbR17IZb6SqNEATQggh6pN33nmTdevWlDs+fvykMx5tqwsSpAUxwxaBgUZIeCQeLBSeYWooU+4BYj66mkN9pmH2hdDBnYy7413V1FohhBCibt1331Duu29oXTej2sjCgWCmmTCs4dh3f8tZpgPoh/PO6HLW1DWYXNlsd0VzrrYDq88l89GEEEKIICVBWpAzrGHYMpNpy0HScs5sdac1dQ0+azg/5cfT27QZAw1Pi4uqqaVCCCGEqE4SpAW5kr3SIiikwF10Rteypq7B27wHB3K8XGxOJieqM0ZIdDW0UgghhBDVTYK0IOcr3ist0lSI03P6QZrmysGcuQVPQi+OZmdzrrYDX2vZH00IIYQIVhKkBTnD7k8xEGlykncGI2nmozvAbMOTcAEt8n7HrnkxJV1aXc0UQgjRyCxbtpR//vPVk563fv1aJk0aVwst8rvppmtq/B6TJo1j/fq1lZbfdtsAXK4z34BeVncGOSMkCgNoYnFTUFDEzox82jcNP+XreON7kDF0M2Cis2sxXsOEJ+GCam+vEEKI2mXf+i9CtiwK7OSSrVGr2H0foLDznbg63XZG7RJnToK0IOezR2GYQ2jW7Xq0n+EbPf20gjQMA8x2AC7gD7aYziLeFlHNrRVCCNGYJCf/zqhRD5Kfn8/gwcNwuQr55JOPS1NBTZky7bjzlyz5iBUrvsfr9RIREcHzz7/MN998zc8//4TLVciBA/u5665BXH/9AJKT/2DWrOkYhkFcXDMmTXqO/fv3M3PmyxiGQVRUFOPGTSI0NJRp054nJWUXLVu2qjS7QYk///kWunbtxv79++jRoyf5+Xls2ZJM69ZJTJz4HKmpB3nxxcl4vV5MJhOjRj3BWWd1ZMmSxXzxxWfExjbl6NGjAHi9Xl5++QX279+Hz+dj6NAHy6WxOhMSpAU5wx6J5nPTq/c19DzwO9/o6Qy/OKlc/rMqFXlo8v7FFJw/ClfHW+hk7OA/kX8mvuaaLYQQopa4Ot0W8KhXde+kHxISwssvzyIr6yjDht3LgAG38PLLswgJCWHatOf59defado0DgCfz0d2djYzZ87GZDLx2GMj2bIlGYD8/DxeeeU19u3by5gxj3L99QOYNu15nn32Bdq0acsnn3zM7t27mTFjKuPGPU3btu344ovPWLhwAV27dsPtdjN37nzS0tL44YfvqmxzWloqs2a9QdOmTbnuun7MnTufRx8dzR133Exubi6vvz6TW2+9gz59LmPXrh1Mnfocs2b9k48/XsS77y7CZDIxZMjfAFi69DOioqIZN+5psrOzGDFiGO+/v7ha+hYkSAt6hi0SzfBh3buCuPB41uzNYtvhfFTzwEfBLBnJmPNSMexRmA78ggUfux3nIy87hRBCnIlu3c5F0zRiYpoQHh6BxWJhypRJhIWFsWfPbrp27VZ6rslkwmq18swzTxEaGsrhw4fxer0AdOjQEYBmzZqXjoQdPXqENm38qRD/9KfbAdizJ4UZM6YCUFTkJTExiZSUnXTu3AWA+Ph4mjVrXmWbIyOjiI/3D1OEhobStm07AMLDI3C7XezevZvu3c8D/LlFDx8+xJ49u2nbth02mw2g9H47d+5g06YNbN78R2mbsrOzTrc7y5EgLcgZdn9qqPC1M8nSngPg662HTilIs6b6U2R4Enri/fUNXIaFDUZH7qj+5gohhGhEtmzZDEBmZgb5+XksXvwhS5Z8AcCjj46gbH7wHTu28+OPP/DmmwsoLCwsHY0CKnw71LRpU/bt20tiYmvef38+iYlJtG6dxIQJk4mPj2fTpt/IzMzAYrHw7bf/Af5CRkY66enpVbb5ZG+i2rRpw6ZNv3HJJX3Zvl2nSZNYWrRoye7du3C5CrFYrGzbpnP11deRlNSGZs2acc89g3G5Clmw4G0cjshAu++kJEgLcr7ifdI0dw43nx/PqpSjfJl8mIf7tgv4laf14GqKHIn4IhKwHVjFBuMsYqOq7yESQgjROLlcLh5++AGczgLGjJnA559/wuDBfyM0NBSHw0FGRjoJCS0AaNUqkdDQUIYMuRubzUpsbFMyMioPqJ58cjwvvjgZk8lEbGwsd9zxV5o3j2fKlKfx+fxpEseOnUjr1kls2rSRoUMHER+fQHR09Bl9phEjHmHq1CksWrSQoqIixo2bSExMDPff/wAPPDCY6OgYQkNDAbj55j/x0ktTGDlyGPn5eQwceDsmU/VtnKGVjXIbAo+nyMjKKqjynIICf3lYWFhtNOmMWPf9SPS//0pRRAvS7lrNla+votDrY95fzqVbi5MHWvbtS4lc/iAF3YdS0HMUsfPOYabnT1guHc2d55882Wx96qu6JP0UOOmrwElfBaYx9lNa2h7i45NOuV51z0lrqGqqnyr6ucXFOdYBFa42kJG0IFeScUDz5GOzmLhKxbE0+RBfJqcFFKRp7hzcLXuT33sstj3fo2GwyteFu2NDa7rpQgghRJ1YuXIFixYtLHf89tv/wmWXXVEHLTo9EqQFOcNeEqQ5Abj5nHiWJh9iuZ7O6CvPwmyq5JWntxAsIRR2uYvCs/8CmgnrgVW4sLPRaM8zp7ONhxBCCFEP9OlzGX36XFbXzThjtRakKaVMwGygO+AC7td1fUeZ8seAIUDJC+rhuq7rSqkNQHbxsRRd1++rrTYHg5I5ad4mHcEw6NYikuEXJzFn1R427M+mZ+vocnU0dy7RnwykUN2G87wHQPO/H7ft/4nNtrPxFFqJC7fV5scQQgghxCmqzZG0W4AQXdd7K6UuAmYAN5cp7wHco+v6upIDSqkQAF3XL6/FdgaVktWd7vY3gKahAX/r2Yp31+xj+dbD5YM0n5fI/zyI+ch2vE27lB7WCjKwHNFJiRlMc8N+avusCSGEEKLW1WaQ1gf4GkDX9dVKqRMnyZ0PjFNKxQNf6rr+Iv5RtzCl1PLito7XdX11VTfx+Xylk0gr43RWXR5sfCYbRdkHKMjLBpMVd5GPUIuJr7YcZmTvFljMx1aSxKyejG3vD2RePJm82POhZEJtyg8ArCzqTEyI+aR9VKK+9VVdkX4KnPRV4KSvAtMY+8nnM0ont58Kw/CvijyNqo1KTfWTz2dU8PvXUen5tZlgPZJjry0BipRSZYPERcADQD+gj1LqRqAAmA5cU1y28IQ6jYJhCSFy6/tYs1MAsJlNRNgtFHp9rN2fU3qeY/N7RG55n+wug8lTdx53jZDUX/BZw/k8I55Mp7dW2y+EEKJ+GDXqQfbs2c1XX33BTz/9CMAnn3xcx61qvGoz4Mnh+HDRpOu6F0AppQEzdV3PLv7+S+A84Btgh67rBrBNKZUJJAD7KruJyWQKeBl2vVmubQ0Hdw6hJi/W4jb/uUdLXv7vTpZuPUK/zi3A4yRq8zu42lyNu+8kwkzHLxsOPfQLnhYX4dbNRIdaT/mz15u+qmPST4GTvgqc9FVgGlM/5eRop7U9RMnIUFV1zWYzN954bDbSe++9w+2331np+Q1RIP10Okwm7ZSe09oM0n4CBgCLi+ek/V6mLBL4QynVGcjHP5r2NjAYOAf4P6VUi+LzUmuxzUHBsDkgPxXNnVt67NrOzZjx/U5W7z6Ky+vDbg0l69bP8FkdcEKAZspLxZK1i4z2fwYdmkbIogEhhGhM9u7dwwsvPIvZbC4NwpYtW4rJZCIzM5ObbhrIrbcey0Mzb94cYmNjyc7OJicnm+nTp/LEE2Pr8BM0TrX5uvNToFAptQr4O/CoUuqvSqlhxSNo44Hvgf8BybquLwPmAdFKqZXAR8DgktG3xqRkGw6TO6/0WGSIlW4tIon2ZeH5/EG0wix84fFgK7+1hvXAKgD00HMBiHfYa77RQgghgsaaNb+gVCdeeeU17r77PnJzc8jISGfq1FeYO/cdFi/+gKNHj5SrN2jQECIjoyRAqyO1NpKm67oP/7yysraWKX8PeO+EOm7grzXfuuBWZI/CCmievOOO/+3cprRPf4SEQ/tw5u7HGxJdYX3r/lX47FFs8rQGUmgZJRvZCiFEY3LjjTezcOECnnxyFOHhEVx44UV07dqtNGF4u3btOXBgfx23UpyoNkfSxGkyQmIw0KB4tYn/oI8bdk+hu2knj3tHkhvducK6tp3LCNm2BHdSPw7lewBIaiJBmhBCNCYrV66ge/fz+PvfX+fyy/uxcOG7bN++jaKiIgoLC0lJ2UWrVq0rrNvQ0kfWJxKk1QNGWFOw2CnsclfpsbBfZxC68wu2nf0oX3rO57tt5ZPU2rf/m8j/PIi32bnk9X2eFlEhAJwdX/lyXyGEEA1Pp05nM3fubEaOHMa///0pt956B16vlyeeeJj/+7/7GTRoSKWJydu0acvkyRNrt8ECkLRQ9YJhi0TzFkKRG8w2rPt+JHztLJyd7yS323BYv45F6w9wY5f40jp2fQmO7x7FE9+LnBsXYNgiyMg/gkmDJmGycEAIIRqTli1bMWfOO6V7q23cuIEtW5J59tkXjzvvtdfmAjBkyPDSY6++Oqf2GiqOIyNp9YCveOFA+KopAHhaXkJu3ynkXfYCSU3CcNgtbDucT57Lv6bCvmUxjm8fwdPiIrIHvIdhiwDg+20ZWEymyvN9CiGEECJoSJBWDxg2/+tJ24HVWNLWgclM4Tn3gtmGpmn0V00xgM9+TyUkeSGO/z6OJ/FSsm9YANZj+7Fk5rsxy09cCCEavR49epYbRRPBR35l1wOGPQoAS+ZmIr8eDkWu48oH9UoEwLfubRw/jMGddAXZ178N1uMXCDi9RYTb5A23EEIIUR/Ib+x6wGfzv+40NDO5V78G5uP3OWsZHcrDYd/wmPcd8hL747xuTrlzDMPAU2QQFSI/ciGEEKI+kJG0esCw+1935vcchafFReXKQze8wWO+d/iqqBcLWj5TLkADyC30z1drEi6LBoQQQoj6QIK0esCw+V93GhEJ5crC1r5KxKopFHYYwMvho/l+Z3a5cwB2ZeYD0FyyDQghhBD1ggRp9UDJSFrZ3J0YBmG/vkL4Ly9R2HEguVe9St+OzVm7L5vUbGe5a1iKV3T2SoyujSYLIYRowFwuF0uXfhbQucuWLWXlyhWVlr/33nw2b/6jmlrWsEiQVg8Y1nAMzYTmKh4lMwzCfnmZ8DWvUNjpDnKvnAkmC/EO/2a1837ZV+4aR53+152tJduAEEKIM3TkSGbAQdr11w+gT5/LKi2/++57OfvsrtXUsoZFZpHXB5oJw+ZAc+WAYRD+8/OEbXgD59l/Je/yqaD5Y+2bz4nn5f/u4PvtGUy4uuNxl/h5tz9xriwcEEKIhuXbb//D8uVfBXRuSYonTat6v8yrr76O/v2vqbT83XffZvfuFC69tBc9e16A0+lk7NiJfP31l2zdupmCggLatGnL+PGTmDdvDrGxsbRu3YaFC9/FarWQmnqQfv2uYtCgITz//DNceeXVHDmSyc8//4TLVciBA/u5665BXH/9ADZv/oNXXplGWFgYMTEx2Gx2nnrqmYD7pz6T39j1hGGLxOTOIfynZwnb+BbOroPI6/tcaYAGYDGb6JLgYNPBXLYdzqNjs4jSMv2wzEkTQghRPe65ZzA7d+7gwgt7k5ubyyOPPEF+fh4Oh4OZM2fj8/m4++47SE8/fFy9Q4dSmT//QzweD7fcci2DBg05rjw/P49XXnmNffv2MmbMo1x//QCmT3+RCRMm065de+bMeZ2MjPJpEBsqCdLqCcPmwL7jC7QiFwXdhpDf5xmo4H9C9/RK5InPN/PWz3uYdnOX0uNHC9xoGtgs5lpstRBCiJrWv/81VY56lVWSFspsrr7fBa1bJwFgt4dw9OhRJk0aT1hYGE6nE6/Xe9y57dp1wGKxYLFYsNtDyl2rQwf/W6BmzZrjdrsByMjIoF279gB0734e3323vNraHuxkTlo94bNH+gO0c4dXGqAB9G0fi91iYvWeo8cdzy30Ypd0A0IIIaqBppkwDB8ApuKFaatX/8Thw4d49tkXGDZsBC5XYenr1WP1Tnbd8ic0a9aclJRdACQn/14Nra8/ZCStnijsOgh3u2txdhtS5VOuaRpXqziWJh9iz5ECkpr400IVeHyE22UUTQghxJmLiYnB4/Hich3LgNO5cxfmz5/HsGH3YrPZaNGiZbW8mnz88TG8+OJkQkPDsFotxMU1O+Nr1hfaiVFufefxFBlZWQVVnlNQ4C8PCwur8rz66nCuixvn/sKwi5O4v7d/GPqCGT/SOiaUfw3udUrXauh9VV2knwInfRU46avANMZ+SkvbQ3x80inXq4nXnTVtyZLF9Ot3FTExMcydOxur1cp99w2t0XvWVD9V9HOLi3OsA3pWdL6MpDVAzRx2OsSF88G6Awy+0J/XU9Pg7HhHHbdMCCGEODVNmjThscdGEBoaRkRERKNZ2QkSpDVYHZqGsz09n6+3pnNxmyb4DOgsQZoQQoh65oor+nPFFf3ruhl1QmaSN1DDLvYPp36wbj9bD+UBEGKRH7cQQghRX8hv7QaqVXQosWFWtqXns3aff6Wn+WTLaoQQQggRNCRIa8Cu7tQMw4Cvtvg3E2wb23gm1QohhBD1nQRpDdh9xYsGDuf5NwRs11SCNCGEEKK+kCCtAYsJs3Fey0gANCDcJutEhBBC1J6RI4exZ89uli1bysqVK8qV33RT1ZkSVqz4noyMdDIzM5g+fWpNNTNoSZDWwA3sngCA1Szz0YQQQtSN668fQJ8+l51yvY8//pD8/HxiY5vyxBNja6BlwU2GVhq4vu1jMWkQaq0/GxcKIYQ4NaNHP1Lh8WnTZgLwxhuvsWvXjtI0TSXpl4YPH0n79h345puv+eabr8vVq8z48U9y++13ct5557NlSzKzZ/+D6OgY8vJyyc7OYsCAgQwceFvp+fPmzSE2NpYBAwYybdrzpKTsomXLVqX5OXft2sGrr/4dn88gL8+fsD03N5cdO7YxZcrTTJz4HFOmTGLu3PmsWbOauXP/id1uJzIyinHjnmb7dp2FC9/FarWQmnqQfv2uKpe8vT6SIK2BC7dZeLhvO2LDbXXdFCGEEA3EgAG38NVXX3DeeeezbNkX9OjRk3bt2nPZZf3IyEhn5MhhxwVpJVavXoXb7Wbu3PmkpaXxww/fAZCSsouRIx+lffsOLF/+NcuWLWXMmAl06NCRJ58cj9VqBcAwDKZNe4HZs98iLq4Zixd/yIIF87j44j4cOpTK/Pkf4vF4uOWWayVIE/XDXT1b1XUThBBC1KCTjXw98MBIoPJ0R1dddS1XXXVtwPe78MLezJ49i5ycbDZt2sD06f/gjTdeY8WK7wkLC8fr9VZYLyVlJ507dwEgPj6eZs2aA9C0aTPmz38Lu91OQUEB4eHhFdbPysoiLCy8NH/nueeex5w5s7n44j60a9cBi8WCxWLBbg8J+LMEM5mTJoQQQohTYjKZuOKK/kyfPpVLL72cRYvep2vXbjz99HP069efyvKCJyW1ITl5EwAZGemkp/sTsM+a9TJDhgxnwoRnad++Q2l9k8mEz+crrR8dHU1BQT4ZGRkA/PbbehITWwP+9IcNjYykCSGEEOKU3XDDTdxxx80sWvQpqakHmT79RZYv/4qoqCjMZnPpfLOyLr30cjZt2sjQoYOIj08gOjoagKuvvo6xYx+nSZMmxMU1Izs7C4CuXbsxZcokRo9+CvDPpRs9+imeeupJTCYNhyOS8eOfYdeuHbX1sWuVVlm0W195PEVGVlZBlecUFPjLw8Jk37CTkb4KjPRT4KSvAid9FZjG2E9paXuIj0865XqVve4Ux6upfqro5xYX51gH9KzofHndKYQQQggRhCRIE0IIIYQIQhKkCSGEEEIEIQnShBBCiHqooc0pb+hO5+clQZoQQghRz1gsNvLzcyRQqycMwyA/PweL5dQ2lpctOIQQQoh6JiYmjqNH08nLyzqlej5fyf5jDXBTsWpUE/1ksdiIiYk7tTrVdnchhBBC1Aqz2ULTpgmnXK8xbldyOoKln2otSFNKmYDZQHfABdyv6/qOMuWPAUOA9OJDw4HtVdURQgghhGioanNO2i1AiK7rvYGxwIwTynsA9+i6fnnxHz2AOkIIIYQQDVJtvu7sA3wNoOv6aqXUibvrng+MU0rFA1/quv5iAHXK8fl8pcOUlXE6qy4Xx0hfBUb6KXDSV4GTvgqM9FPgpK8CU7v95Ki0pDaDtEggu8z3RUopi67r3uLvFwGvAznAp0qpGwOoU47dbs1ISmq+p5rbLoQQQghREyrN71WbQVoOx4eLppJgSymlATN1Xc8u/v5L4Lyq6lTh1JZOCCGEEEIEodqck/YTcD2AUuoi4PcyZZHAH0qpiOKArR+w7iR1hBBCCCEaLK22NsIrs7qzG6AB9+FfLBCh6/pcpdTdwMP4V3F+p+v6pIrq6Lq+tVYaLIQQQghRh2otSBNCCCGEEIGTtFBCCCGEEEFIgjQhhBBCiCDU6NJCnSzzgThGKbWBY1ugpOi6fl9dticYKaUuBF7Sdf1ypVQHYD5gAH8AI3Rd99Vl+4LJCX3VA1iKP6sIwD91Xf+o7lpX95RSVuBtoA1gB6YAm5FnqpxK+mo/8kyVo5QyA28CCijCPx9cQ56r41TST1HU8TPV6II0ymQxKF4xOgO4uW6bFHyUUiEAuq5fXsdNCVpKqdHA3UB+8aFXgAm6rv+glHoD/3P1aV21L5hU0Fc9gFd0XZcsIsf8DcjUdf1upVQssAH4DXmmKlJRX01GnqmKDADQdf0SpdTl+P+d0pDn6kQV9dNS6viZaoyvO4/LYgCcNItBI9UdCFNKLVdK/bc4oBXH2wn8qcz35wMrir/+Cuhf6y0KXhX11Q1KqR+VUvOUUpVvud14fAxMLPO9F3mmKlNZX8kzdQJd1z8DhhV/mwQcQp6rcqropzp9phpjkFZhFoO6akwQKwCmA9cADwALpZ+Op+v6EsBT5pCm63rJculc/EPlggr76lfgSV3X+wK7gEl10rAgout6nq7rucW/CP4FTECeqQpV0lfyTFVC13WvUmoB8Cr+/pLnqgIV9FOdP1ONMUg7nSwGjdE24H1d1w1d17cBmUBCHbcp2JWd0+EAsuqoHfXBp7quryv5Gn+GkUZPKZUIfA+8p+v6B8gzVakK+kqeqSrouj4I6Ih/3lVomSJ5rso4oZ+W1/Uz1RiDNMliEJjB+OfroZRqgX8EMrVOWxT8NhTPZQC4DvhfHbYl2P1HKXVB8ddX4s8w0qgppZoDy4Exuq6/XXxYnqkKVNJX8kxVQCl1t1JqXPG3BfgD/7XyXB2vkn76pK6fqcb4+upT4Cql1CqOZT4Q5c0D5iulVuJfATRYRhxP6nHgTaWUDdiCf7hcVOxB4DWllBtI49hckMZsPBADTFRKlcy3GgX8Q56pcirqq8eAmfJMlfMJ8I5S6kfACjyC/1mSf6uOV1E/7aOO/52SjANCCCGEEEGoMb7uFEIIIYQIehKkCSGEEEIEIQnShBBCCCGCkARpQgghhBBBSII0IYQQQogg1Bi34BBCNAJKqd3407tUJFnX9a610AYDuFvX9fdr+l5CiIZHgjQhREP2EjCzguOeCo4JIURQkSBNCNGQ5em6nlbXjRBCiNMhQZoQolFSSrUBUoC7gIn4X43+Cjyk6/rvxedY8O9kPxRIBLYDz+m6vrjMda4DngHOAQ4Dr+u6/nKZW52tlPoBuAj/ruWTy6QyEkKISsnCASFEY/cKMAHohT/R9LdKqagyZU8C44BuwIfAIqXUrQBKqd7AF/jzSJ4LPApMUkoNLXP9EcBs4Gzg3/jT8bSt2Y8khGgIJC2UEKJBKl44kEDF888ewx9YpQAP67r+anGdKGA/8AT+gCwTGKHr+twy1/0IaKfrei+l1IdAgq7rl5cpvwfw6rr+QfHCgRd0XX+quCwGOALcquv6J9X8kYUQDYy87hRCNGSv4x/FOlE6/gTdACtKDuq6nq2U2oL/1eUG/P9G/nRC3R+Bm4q/PgdYVrZQ1/V3Tzh/W5myo0opgNBT+hRCiEZJgjQhREN2RNf1HRUVFI9qQfmRNjPgAworuaa5TJ1AVokWVXBMC6CeEKKRkzlpQojG7vySL4oDN4V/FG074Ab6nHB+H2Bz8ddbgJ5lC5VSU5RSn9VUY4UQjYeMpAkhGrIIpVR8JWUlo1kvKqUOAweBqUAGsFjXdadS6hVgilIqE9gI/Am4FbizuO50YI1SagKwCOgOPAI8XBMfRgjRuMhImhCiIRsDpFbyJ7b4nLn45679gj9wu0LX9fzisonAHPwb4v6OPzi7U9f1jwF0XV+PP3C7HUgGpgHjZYsNIUR1kNWdQohGqcw+aZfqur6yjpsjhBDlyEiaEEIIIUQQkiBNCCGEECIIyetOIYQQQoggJCNpQgghhBBBSII0IYQQQoggJEGaEEIIIUQQkiBNCCGEECIISZAmhBBCCBGEJEgTQgghhAhC/w/XDYCl8UM8UgAAAABJRU5ErkJggg==\\n\",\n \"text/plain\": [\n \"
\"\n ]\n },\n \"metadata\": {\n \"needs_background\": \"light\"\n },\n \"output_type\": \"display_data\"\n }\n ],\n \"source\": [\n \"a = plt.axes(aspect='equal')\\n\",\n \"sns.scatterplot(test_df['MPG'].values, mpg_hat_df['MPG_predictions'].values,\\n\",\n \" s=50)\\n\",\n \"plt.xlabel('True Values [MPG]')\\n\",\n \"plt.ylabel('Predictions [MPG]')\\n\",\n \"lims = [0, 50]\\n\",\n \"plt.xlim(lims)\\n\",\n \"plt.ylim(lims)\\n\",\n \"_ = plt.plot(lims, lims)\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Compare K-fold Cross Validation metrics against hold-out test metrics\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Hold-out Test Metrics\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 19,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"{'loss': 8.303831,\\n\",\n \" 'error': -0.45136052,\\n\",\n \" 'mean_squared_error': 8.303831,\\n\",\n \" 'mean_absolute_error': 2.2274728,\\n\",\n \" 'r2': 0.8558148}\"\n ]\n },\n \"execution_count\": 19,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"test_stats['MPG']\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### K-fold Cross Validation Metrics\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 20,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"text/plain\": [\n \"{'loss_mean': 8.111681,\\n\",\n \" 'loss_std': 2.4598064,\\n\",\n \" 'error_mean': 0.0380627,\\n\",\n \" 'error_std': 0.5965346,\\n\",\n \" 'mean_squared_error_mean': 8.111682,\\n\",\n \" 'mean_squared_error_std': 2.4598064,\\n\",\n \" 'mean_absolute_error_mean': 2.0598435,\\n\",\n \" 'mean_absolute_error_std': 0.2779836,\\n\",\n \" 'r2_mean': 0.8666786,\\n\",\n \" 'r2_std': 0.03552912}\"\n ]\n },\n \"execution_count\": 20,\n \"metadata\": {},\n \"output_type\": \"execute_result\"\n }\n ],\n \"source\": [\n \"kfold_cv_stats['overall']['MPG']\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": []\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.6.9\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 4\n}\n"
},
{
"path": "examples/lbfgs/config.yaml",
"content": "input_features:\n - name: RESOURCE\n type: category\n - name: MGR_ID\n type: category\n - name: ROLE_ROLLUP_1\n type: category\n - name: ROLE_ROLLUP_2\n type: category\n - name: ROLE_DEPTNAME\n type: category\n - name: ROLE_TITLE\n type: category\n - name: ROLE_FAMILY_DESC\n type: category\n - name: ROLE_FAMILY\n type: category\n - name: ROLE_CODE\n type: category\noutput_features:\n - name: ACTION\n type: binary\npreprocessing:\n split:\n type: fixed\ndefaults:\n category:\n encoder:\n type: sparse\ntrainer:\n batch_size: 32769 # entire training set\n train_steps: 1\n steps_per_checkpoint: 1\n learning_rate: 1\n regularization_lambda: 0.0000057\n optimizer:\n type: lbfgs\n max_iter: 100\n tolerance_grad: 0.0001\n history_size: 10\n"
},
{
"path": "examples/lbfgs/model.py",
"content": "import logging\n\nimport pandas as pd\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.datasets import amazon_employee_access_challenge\n\ndf = amazon_employee_access_challenge.load()\n\nmodel = LudwigModel(config=\"config.yaml\", logging_level=logging.INFO)\n\ntraining_statistics, preprocessed_data, output_directory = model.train(\n df,\n skip_save_processed_input=True,\n skip_save_log=True,\n skip_save_progress=True,\n skip_save_training_description=True,\n skip_save_training_statistics=True,\n)\n\n# Predict on unlabeled test\nconfig = model.config\nconfig[\"preprocessing\"] = {}\nmodel.config = config\nunlabeled_test = df[df.split == 2].reset_index(drop=True)\npreds, _ = model.predict(unlabeled_test)\n\n# Save predictions to csv\naction = preds.ACTION_probabilities_True\nsubmission = pd.merge(unlabeled_test.reset_index(drop=True).id.astype(int), action, left_index=True, right_index=True)\nsubmission.rename(columns={\"ACTION_probabilities_True\": \"Action\", \"id\": \"Id\"}, inplace=True)\nsubmission.to_csv(\"submission.csv\", index=False)\n"
},
{
"path": "examples/llama2_7b_finetuning_4bit/README.md",
"content": "# Llama2-7b Fine-Tuning 4bit (QLoRA)\n\n[](https://colab.research.google.com/drive/1c3AO8l_H6V_x37RwQ8V7M6A-RmcBf2tG?usp=sharing)\n\nThis example shows how to fine-tune [Llama2-7b](https://huggingface.co/meta-llama/Llama-2-7b-hf) to follow instructions.\nInstruction tuning is the first step in adapting a general purpose Large Language Model into a chatbot.\n\nThis example uses no distributed training or big data functionality. It is designed to run locally on any machine\nwith GPU availability.\n\n## Prerequisites\n\n- [HuggingFace API Token](https://huggingface.co/docs/hub/security-tokens)\n- Access approval to [Llama2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf)\n- GPU with at least 12 GiB of VRAM (in our tests, we used an Nvidia T4)\n\n## Running\n\n### Command Line\n\nSet your token environment variable from the terminal, then run the API script:\n\n```bash\nexport HUGGING_FACE_HUB_TOKEN=\"\"\n./run_train.sh\n```\n\n### Python API\n\nSet your token environment variable from the terminal, then run the API script:\n\n```bash\nexport HUGGING_FACE_HUB_TOKEN=\"\"\npython train_alpaca.py\n```\n\n## Upload to HuggingFace\n\nYou can upload to the HuggingFace Hub from the command line:\n\n```bash\nludwig upload hf_hub -r / -m \n```\n"
},
{
"path": "examples/llama2_7b_finetuning_4bit/llama2_7b_4bit.yaml",
"content": "model_type: llm\nbase_model: meta-llama/Llama-2-7b-hf\n\nquantization:\n bits: 4\n\nadapter:\n type: lora\n\ninput_features:\n - name: instruction\n type: text\n\noutput_features:\n - name: output\n type: text\n\ntrainer:\n type: finetune\n learning_rate: 0.0003\n batch_size: 2\n gradient_accumulation_steps: 8\n epochs: 3\n learning_rate_scheduler:\n warmup_fraction: 0.01\n\nbackend:\n type: local\n"
},
{
"path": "examples/llama2_7b_finetuning_4bit/run_train.sh",
"content": "#!/usr/bin/env bash\n\n# Fail fast if an error occurs\nset -e\n\n# Get the directory of this script, which contains the config file\nSCRIPT_DIR=$( cd -- \"$( dirname -- \"${BASH_SOURCE[0]}\" )\" &> /dev/null && pwd )\n\n# Train\nludwig train --config ${SCRIPT_DIR}/llama2_7b_4bit.yaml --dataset ludwig://alpaca\n"
},
{
"path": "examples/llama2_7b_finetuning_4bit/train_alpaca.py",
"content": "import logging\nimport os\n\nimport yaml\n\nfrom ludwig.api import LudwigModel\n\nconfig = yaml.safe_load(\"\"\"\nmodel_type: llm\nbase_model: meta-llama/Llama-2-7b-hf\n\nquantization:\n bits: 4\n\nadapter:\n type: lora\n\ninput_features:\n - name: instruction\n type: text\n\noutput_features:\n - name: output\n type: text\n\ntrainer:\n type: finetune\n learning_rate: 0.0003\n batch_size: 2\n gradient_accumulation_steps: 8\n epochs: 3\n learning_rate_scheduler:\n warmup_fraction: 0.01\n\nbackend:\n type: local\n\"\"\")\n\n# Define Ludwig model object that drive model training\nmodel = LudwigModel(config=config, logging_level=logging.INFO)\n\n# initiate model training\n(\n train_stats, # dictionary containing training statistics\n preprocessed_data, # tuple Ludwig Dataset objects of pre-processed training data\n output_directory, # location of training results stored on disk\n) = model.train(\n dataset=\"ludwig://alpaca\",\n experiment_name=\"alpaca_instruct_4bit\",\n model_name=\"llama2_7b\",\n)\n\n# list contents of output directory\nprint(\"contents of output directory:\", output_directory)\nfor item in os.listdir(output_directory):\n print(\"\\t\", item)\n"
},
{
"path": "examples/llm_base_model_dequantization/README.md",
"content": "# Convert quantized base model to fp16\n\nLudwig has utility functions to convert nf4 quantized bitsandbytes base models back to fp16\nfor more efficient inference. This is desireable since inference with bitsandbytes is slow because\nevery forward pass through the model requires dequantizing the model weights from nf4 to fp16 layer\nby layer and then quantizing it back to nf4 to keep memory usage constant.\n\nBy dequantizing the base model in fp16 upfront, you can get the same effect of the quantized weights\nwithout sacrificing on inference performance.\n\n## Visual Illustration\n\n### Without dequantization upfront\n\n| **Request 1:** | **Request 2:** | **Request 3:** |\n| ------------------------------------------ | ------------------------------------------ | ------------------------------------------ |\n| - Quantized bitsandbytes model | - Quantized bitsandbytes model | - Quantized bitsandbytes model |\n| - Dequantization of layer 1 (nf4 to fp16) | - Dequantization of layer 1 (nf4 to fp16) | - Dequantization of layer 1 (nf4 to fp16) |\n| - Forward Pass (using dequantized weights) | - Forward Pass (using dequantized weights) | - Forward Pass (using dequantized weights) |\n| - Quantization of layer 1 (fp16 to nf4) | - Quantization of layer 1 (fp16 to nf4) | - Quantization of layer 1 (fp16 to nf4) |\n| - Dequantization of layer 2 (nf4 to fp16) | - Dequantization of layer 2 (nf4 to fp16) | - Dequantization of layer 2 (nf4 to fp16) |\n| - Forward Pass (using dequantized weights) | - Forward Pass (using dequantized weights) | - Forward Pass (using dequantized weights) |\n| - Quantization of layer 2 (fp16 to nf4) | - Quantization of layer 2 (fp16 to nf4) | - Quantization of layer 2 (fp16 to nf4) |\n| - ... | - ... | - ... |\n| - Final Output | - Final Output | - Final Output |\n\n### With dequantization upfront\n\n| **Request 1:** | **Request 2:** | **Request 3:** |\n| -------------------------------- | -------------------------------- | -------------------------------- |\n| - Dequantized base model in fp16 | - Dequantized base model in fp16 | - Dequantized base model in fp16 |\n| - Forward pass through layer 1 | - Forward pass through layer 1 | - Forward pass through layer 1 |\n| - Forward pass through layer 2 | - Forward pass through layer 2 | - Forward pass through layer 2 |\n| - ... | - ... | - ... |\n| - Final Output | - Final Output | - Final Output |\n\n## Running the example script\n\nThe example `phi_2_dequantization.py` shows how you how you can quantize and then dequantized Phi-2. This process\ncan be repeated for any other base model supported by Ludwig that is quantized using 4 bits nf4 bitsandbytes quantization. You will need a GPU to run the script successfully.\n\nBeneath the surface, this script:\n\n1. Loads the base model in 4 bit nf4 quantization\n1. Dequantizes the model layer by layer back into fp16 in-place.\n1. Write the new dequantized weights to disk at `save_path`\n1. Write the tokenizer to disk at `save_path`\n\nMake sure you update the paths at the top of the file for base model, save path, and huggingface repo ID!\n\n## Bonus\n\nIf desired, you can also use Ludwig to push the new dequantized model weights straight to HuggingFace hub!\n\n```python\nfrom ludwig.utils.hf_utils import upload_folder_to_hfhub\n\nupload_folder_to_hfhub(repo_id=hfhub_repo_id, folder_path=save_path)\n```\n\n### Dequantized base models already on huggingface hub\n\n- [CodeLlama 7b Instruct](https://huggingface.co/arnavgrg/codallama-7b-instruct-nf4-fp16-upscaled)\n- [CodeLlama 13b Instruct](https://huggingface.co/arnavgrg/codellama-13b-instruct-nf4-fp16-upscaled)\n- [CodeLlama 70b Instruct](https://huggingface.co/arnavgrg/codellama-70b-instruct-nf4-fp16-upscaled)\n- [Llama 2 7b](https://huggingface.co/arnavgrg/llama-2-7b-nf4-fp16-upscaled)\n- [Llama 2 7b Chat](https://huggingface.co/arnavgrg/llama-2-7b-chat-nf4-fp16-upscaled)\n- [Llama 2 13b Chat](https://huggingface.co/arnavgrg/llama-2-13b-chat-nf4-fp16-upscaled)\n- [Llama 2 70b Chat](https://huggingface.co/arnavgrg/llama-2-70b-chat-nf4-fp16-upscaled)\n- [Mistral 7b](https://huggingface.co/arnavgrg/mistral-7b-nf4-fp16-upscaled)\n- [Mistral 7b Instruct](https://huggingface.co/arnavgrg/mistral-7b-instruct-nf4-fp16-upscaled)\n- [NousMistral Yarn 7b 128K](https://huggingface.co/arnavgrg/NousResearch-Yarn-Mistral-7b-128k-nf4-fp16-upscaled)\n- [Microsoft Phi-2](https://huggingface.co/arnavgrg/phi-2-nf4-fp16-upscaled)\n- [Zephyr 7b Beta](https://huggingface.co/arnavgrg/zephyr-7b-beta-nf4-fp16-upscaled)\n"
},
{
"path": "examples/llm_base_model_dequantization/phi_2_dequantization.py",
"content": "import logging\nimport os\n\nimport yaml\nfrom huggingface_hub import whoami\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.utils.hf_utils import upload_folder_to_hfhub\n\nhf_username = whoami().get(\"name\")\nbase_model_name = \"microsoft/phi-2\"\ndequantized_path = \"microsoft-phi-2-dequantized\"\nsave_path = \"/home/ray/\" + dequantized_path\nhfhub_repo_id = os.path.join(hf_username, dequantized_path)\n\n\nconfig = yaml.safe_load(f\"\"\"\n model_type: llm\n base_model: {base_model_name}\n\n quantization:\n bits: 4\n\n input_features:\n - name: instruction\n type: text\n\n output_features:\n - name: output\n type: text\n\n trainer:\n type: none\n\n backend:\n type: local\n \"\"\")\n\n# Define Ludwig model object that drive model training\nmodel = LudwigModel(config=config, logging_level=logging.INFO)\nmodel.save_dequantized_base_model(save_path=save_path)\n\n# Optional: Upload to Huggingface Hub\nupload_folder_to_hfhub(repo_id=hfhub_repo_id, folder_path=save_path)\n"
},
{
"path": "examples/llm_few_shot_learning/simple_model_training.py",
"content": "#!/usr/bin/env python\n\n# # Simple Model Training Example\n#\n# This is a simple example of how to use the LLM model type to train\n# a zero shot classification model. It uses the facebook/opt-350m model\n# as the base LLM model.\n\n# Import required libraries\nimport logging\nimport shutil\n\nimport pandas as pd\nimport yaml\n\nfrom ludwig.api import LudwigModel\n\n# clean out prior results\nshutil.rmtree(\"./results\", ignore_errors=True)\n\nreview_label_pairs = [\n {\"review\": \"I loved this movie!\", \"label\": \"positive\"},\n {\"review\": \"The food was okay, but the service was terrible.\", \"label\": \"negative\"},\n {\"review\": \"I can't believe how rude the staff was.\", \"label\": \"negative\"},\n {\"review\": \"This book was a real page-turner.\", \"label\": \"positive\"},\n {\"review\": \"The hotel room was dirty and smelled bad.\", \"label\": \"negative\"},\n {\"review\": \"I had a great experience at this restaurant.\", \"label\": \"positive\"},\n {\"review\": \"The concert was amazing!\", \"label\": \"positive\"},\n {\"review\": \"The traffic was terrible on my way to work this morning.\", \"label\": \"negative\"},\n {\"review\": \"The customer service was excellent.\", \"label\": \"positive\"},\n {\"review\": \"I was disappointed with the quality of the product.\", \"label\": \"negative\"},\n {\"review\": \"The scenery on the hike was breathtaking.\", \"label\": \"positive\"},\n {\"review\": \"I had a terrible experience at this hotel.\", \"label\": \"negative\"},\n {\"review\": \"The coffee at this cafe was delicious.\", \"label\": \"positive\"},\n {\"review\": \"The weather was perfect for a day at the beach.\", \"label\": \"positive\"},\n {\"review\": \"I would definitely recommend this product.\", \"label\": \"positive\"},\n {\"review\": \"The wait time at the doctor's office was ridiculous.\", \"label\": \"negative\"},\n {\"review\": \"The museum was a bit underwhelming.\", \"label\": \"neutral\"},\n {\"review\": \"I had a fantastic time at the amusement park.\", \"label\": \"positive\"},\n {\"review\": \"The staff at this store was extremely helpful.\", \"label\": \"positive\"},\n {\"review\": \"The airline lost my luggage and was very unhelpful.\", \"label\": \"negative\"},\n {\"review\": \"This album is a must-listen for any music fan.\", \"label\": \"positive\"},\n {\"review\": \"The food at this restaurant was just okay.\", \"label\": \"neutral\"},\n {\"review\": \"I was pleasantly surprised by how great this movie was.\", \"label\": \"positive\"},\n {\"review\": \"The car rental process was quick and easy.\", \"label\": \"positive\"},\n {\"review\": \"The service at this hotel was top-notch.\", \"label\": \"positive\"},\n]\n\ndf = pd.DataFrame(review_label_pairs)\ndf[\"split\"] = [0] * 15 + [2] * 10\n\nconfig = yaml.safe_load(\"\"\"\nmodel_type: llm\nbase_model: facebook/opt-350m\ngeneration:\n temperature: 0.1\n top_p: 0.75\n top_k: 40\n num_beams: 4\n max_new_tokens: 64\nprompt:\n task: \"Classify the sample input as either negative, neutral, or positive.\"\n retrieval:\n type: semantic\n k: 3\n model_name: paraphrase-MiniLM-L3-v2\ninput_features:\n-\n name: review\n type: text\noutput_features:\n-\n name: label\n type: category\n preprocessing:\n fallback_label: \"neutral\"\n decoder:\n type: category_extractor\n match:\n \"negative\":\n type: contains\n value: \"positive\"\n \"neural\":\n type: contains\n value: \"neutral\"\n \"positive\":\n type: contains\n value: \"positive\"\npreprocessing:\n split:\n type: fixed\n \"\"\")\n\n# Define Ludwig model object that drive model training\nmodel = LudwigModel(config=config, logging_level=logging.INFO)\n\n# initiate model training\n(\n train_stats, # dictionary containing training statistics\n preprocessed_data, # tuple Ludwig Dataset objects of pre-processed training data\n output_directory, # location of training results stored on disk\n) = model.train(\n dataset=df, experiment_name=\"simple_experiment\", model_name=\"simple_model\", skip_save_processed_input=True\n)\n\ntraining_set, val_set, test_set, _ = preprocessed_data\n\n# batch prediction\npreds, _ = model.predict(test_set, skip_save_predictions=False)\nprint(preds)\n"
},
{
"path": "examples/llm_finetuning/README.md",
"content": "# LLM Fine-tuning\n\nThese examples show you how to fine-tune Large Language Models by taking advantage of model parallelism\nwith [DeepSpeed](https://www.deepspeed.ai/), allowing Ludwig to scale to very large models with billions of\nparameters.\n\nThe task here will be to fine-tune a large billion+ LLM to classify the sentiment of [IMDB movie reviews](https://www.kaggle.com/datasets/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews). As such, we'll be taking a pretrained LLM, attaching a classification head,\nand fine-tuning the weights to improve performance of the LLM on the task. Ludwig will do this for you without no machine learning\ncode, just configuration.\n\n## Prerequisites\n\n- Installed Ludwig with `ludwig[distributed]` dependencies\n- Have a CUDA-enabled version of PyTorch installed\n- Have access to a machine or cluster of machines with multiple GPUs\n- The IMDB dataset used in these examples comes from Kaggle, so make sure you have your credentials set (e.g., `$HOME/.kaggle.kaggle.json`)\n\n## Running DeepSpeed on Ray\n\nThis is the recommended way to use DeepSpeed, which supports auto-batch size tuning and distributed data processing.\nThere is some overhead from using Ray with small datasets (\\<100MB), but in most cases performance should be comparable\nto using native DeepSpeed.\n\nFrom the head node of your Ray cluster:\n\n```bash\n./run_train_dsz3_ray.sh\n```\n\n### Python API\n\nIf you want to run Ludwig programatically (from a notebook or as part of a larger workflow), you can run the following\nPython script using the Ray cluster launcher from your local machine.\n\n```bash\nray submit cluster.yaml train_imdb_ray.py\n```\n\nIf running directly on the Ray head node, you can omit the `ray submit` portion and run like an ordinary Python script:\n\n```bash\npython train_imdb_ray.py\n```\n\n## Running DeepSpeed Native\n\nThis mode is suitable for datasets small enough to fit in memory on a single machine, as it doesn't make use of\ndistributed data processing (requires use of the Ray backend).\n\nThe following example assumes you have 4 GPUs available, but can easily be modified to support your preferred\nsetup.\n\nFrom a terminal on your machine:\n\n```bash\n./run_train_dsz3.sh\n```\n"
},
{
"path": "examples/llm_finetuning/imdb_deepspeed_zero3.yaml",
"content": "input_features:\n - name: review\n type: text\n encoder:\n type: auto_transformer\n pretrained_model_name_or_path: bigscience/bloom-3b\n trainable: true\n adapter: lora\n\noutput_features:\n - name: sentiment\n type: category\n\ntrainer:\n batch_size: 4\n epochs: 3\n gradient_accumulation_steps: 8\n\nbackend:\n type: deepspeed\n zero_optimization:\n stage: 3\n offload_optimizer:\n device: cpu\n pin_memory: true\n"
},
{
"path": "examples/llm_finetuning/imdb_deepspeed_zero3_ray.yaml",
"content": "input_features:\n - name: review\n type: text\n encoder:\n type: auto_transformer\n pretrained_model_name_or_path: bigscience/bloom-3b\n trainable: true\n adapter: lora\n\noutput_features:\n - name: sentiment\n type: category\n\ntrainer:\n batch_size: 4\n epochs: 3\n gradient_accumulation_steps: 8\n\nbackend:\n type: ray\n trainer:\n use_gpu: true\n strategy:\n type: deepspeed\n zero_optimization:\n stage: 3\n offload_optimizer:\n device: cpu\n pin_memory: true\n"
},
{
"path": "examples/llm_finetuning/run_train_dsz3.sh",
"content": "#!/usr/bin/env bash\n\n# Fail fast if an error occurs\nset -e\n\n# Get the directory of this script, which contains the config file\nSCRIPT_DIR=$( cd -- \"$( dirname -- \"${BASH_SOURCE[0]}\" )\" &> /dev/null && pwd )\n\n# Train\ndeepspeed --no_python --no_local_rank --num_gpus 4 ludwig train --config ${SCRIPT_DIR}/imdb_deepspeed_zero3.yaml --dataset ludwig://imdb\n"
},
{
"path": "examples/llm_finetuning/run_train_dsz3_ray.sh",
"content": "#!/usr/bin/env bash\n\n# Fail fast if an error occurs\nset -e\n\n# Get the directory of this script, which contains the config file\nSCRIPT_DIR=$( cd -- \"$( dirname -- \"${BASH_SOURCE[0]}\" )\" &> /dev/null && pwd )\n\n# Train\nludwig train --config ${SCRIPT_DIR}/imdb_deepspeed_zero3_ray.yaml --dataset ludwig://imdb\n"
},
{
"path": "examples/llm_finetuning/train_imdb_ray.py",
"content": "import logging\nimport os\n\nimport yaml\n\nfrom ludwig.api import LudwigModel\n\nconfig = yaml.safe_load(\"\"\"\ninput_features:\n - name: review\n type: text\n\n encoder:\n type: auto_transformer\n pretrained_model_name_or_path: bigscience/bloom-3b\n trainable: true\n adapter:\n type: lora\n\noutput_features:\n - name: sentiment\n type: category\n\ntrainer:\n batch_size: 4\n epochs: 3\n\nbackend:\n type: ray\n trainer:\n use_gpu: true\n strategy:\n type: deepspeed\n zero_optimization:\n stage: 3\n offload_optimizer:\n device: cpu\n pin_memory: true\n\"\"\")\n\n# Define Ludwig model object that drive model training\nmodel = LudwigModel(config=config, logging_level=logging.INFO)\n\n# initiate model training\n(\n train_stats, # dictionary containing training statistics\n preprocessed_data, # tuple Ludwig Dataset objects of pre-processed training data\n output_directory, # location of training results stored on disk\n) = model.train(\n dataset=\"ludwig://imdb\",\n experiment_name=\"imdb_sentiment\",\n model_name=\"bloom3b\",\n)\n\n# list contents of output directory\nprint(\"contents of output directory:\", output_directory)\nfor item in os.listdir(output_directory):\n print(\"\\t\", item)\n"
},
{
"path": "examples/llm_instruction_tuning/train_alpaca_ray.py",
"content": "import logging\nimport os\n\nimport yaml\n\nfrom ludwig.api import LudwigModel\n\nconfig = yaml.safe_load(\"\"\"\nmodel_type: llm\nbase_model: bigscience/bloomz-3b\n\nadapter:\n type: lora\n\ninput_features:\n - name: instruction\n type: text\n\noutput_features:\n - name: output\n type: text\n\ntrainer:\n type: finetune\n batch_size: 4\n epochs: 3\n\nbackend:\n type: ray\n trainer:\n use_gpu: true\n strategy:\n type: deepspeed\n zero_optimization:\n stage: 3\n offload_optimizer:\n device: cpu\n pin_memory: true\n\"\"\")\n\n# Define Ludwig model object that drive model training\nmodel = LudwigModel(config=config, logging_level=logging.INFO)\n\n# initiate model training\n(\n train_stats, # dictionary containing training statistics\n preprocessed_data, # tuple Ludwig Dataset objects of pre-processed training data\n output_directory, # location of training results stored on disk\n) = model.train(\n dataset=\"ludwig://alpaca\",\n experiment_name=\"alpaca_instruct\",\n model_name=\"bloom560m\",\n)\n\n# list contents of output directory\nprint(\"contents of output directory:\", output_directory)\nfor item in os.listdir(output_directory):\n print(\"\\t\", item)\n"
},
{
"path": "examples/llm_text_generation/simple_model_training.py",
"content": "#!/usr/bin/env python\n\n# # Simple Model Training Example\n#\n# This is a simple example of how to use the LLM model type to train\n# a model on a simple question and answer dataset. It uses the\n# facebook/opt-350m model as the base LLM model.\n\n# Import required libraries\nimport logging\nimport shutil\n\nimport pandas as pd\nimport yaml\n\nfrom ludwig.api import LudwigModel\n\n# clean out prior results\nshutil.rmtree(\"./results\", ignore_errors=True)\n\nqa_pairs = [\n {\"Question\": \"What is the capital of Uzbekistan?\", \"Answer\": \"Tashkent\"},\n {\"Question\": \"Who is the founder of Microsoft?\", \"Answer\": \"Bill Gates\"},\n {\"Question\": \"What is the tallest building in the world?\", \"Answer\": \"Burj Khalifa\"},\n {\"Question\": \"What is the currency of Brazil?\", \"Answer\": \"Real\"},\n {\"Question\": \"What is the boiling point of mercury in Celsius?\", \"Answer\": \"-38.83\"},\n {\"Question\": \"What is the most commonly spoken language in the world?\", \"Answer\": \"Mandarin\"},\n {\"Question\": \"What is the diameter of the Earth?\", \"Answer\": \"12,742 km\"},\n {\"Question\": 'Who wrote the novel \"1984\"?', \"Answer\": \"George Orwell\"},\n {\"Question\": \"What is the name of the largest moon of Neptune?\", \"Answer\": \"Triton\"},\n {\"Question\": \"What is the speed of light in meters per second?\", \"Answer\": \"299,792,458 m/s\"},\n {\"Question\": \"What is the smallest country in Africa by land area?\", \"Answer\": \"Seychelles\"},\n {\"Question\": \"What is the largest organ in the human body?\", \"Answer\": \"Skin\"},\n {\"Question\": 'Who directed the film \"The Godfather\"?', \"Answer\": \"Francis Ford Coppola\"},\n {\"Question\": \"What is the name of the smallest planet in our solar system?\", \"Answer\": \"Mercury\"},\n {\"Question\": \"What is the largest lake in Africa?\", \"Answer\": \"Lake Victoria\"},\n {\"Question\": \"What is the smallest country in Asia by land area?\", \"Answer\": \"Maldives\"},\n {\"Question\": \"Who is the current president of Russia?\", \"Answer\": \"Vladimir Putin\"},\n {\"Question\": \"What is the chemical symbol for gold?\", \"Answer\": \"Au\"},\n {\"Question\": \"What is the name of the famous Swiss mountain known for skiing?\", \"Answer\": \"The Matterhorn\"},\n {\"Question\": \"What is the largest flower in the world?\", \"Answer\": \"Rafflesia arnoldii\"},\n]\n\ndf = pd.DataFrame(qa_pairs)\n\nconfig = yaml.safe_load(\"\"\"\n input_features:\n - name: Question\n type: text\n output_features:\n - name: Answer\n type: text\n model_type: llm\n generation:\n temperature: 0.1\n top_p: 0.75\n top_k: 40\n num_beams: 4\n max_new_tokens: 5\n base_model: facebook/opt-350m\n \"\"\")\n\n# Define Ludwig model object that drive model training\nmodel = LudwigModel(config=config, logging_level=logging.INFO)\n\n# initiate model training\n(\n train_stats, # dictionary containing training statistics\n preprocessed_data, # tuple Ludwig Dataset objects of pre-processed training data\n output_directory, # location of training results stored on disk\n) = model.train(\n dataset=df, experiment_name=\"simple_experiment\", model_name=\"simple_model\", skip_save_processed_input=True\n)\n\ntraining_set, val_set, test_set, _ = preprocessed_data\n\n# batch prediction\npreds, _ = model.predict(test_set, skip_save_predictions=False)\nprint(preds)\n"
},
{
"path": "examples/llm_zero_shot_learning/simple_model_training.py",
"content": "#!/usr/bin/env python\n\n# # Simple Model Training Example\n#\n# This is a simple example of how to use the LLM model type to train\n# a zero shot classification model. It uses the facebook/opt-350m model\n# as the base LLM model.\n\n# Import required libraries\nimport logging\nimport shutil\n\nimport pandas as pd\nimport yaml\n\nfrom ludwig.api import LudwigModel\n\n# clean out prior results\nshutil.rmtree(\"./results\", ignore_errors=True)\n\nreview_label_pairs = [\n {\"review\": \"I loved this movie!\", \"label\": \"positive\"},\n {\"review\": \"The food was okay, but the service was terrible.\", \"label\": \"negative\"},\n {\"review\": \"I can't believe how rude the staff was.\", \"label\": \"negative\"},\n {\"review\": \"This book was a real page-turner.\", \"label\": \"positive\"},\n {\"review\": \"The hotel room was dirty and smelled bad.\", \"label\": \"negative\"},\n {\"review\": \"I had a great experience at this restaurant.\", \"label\": \"positive\"},\n {\"review\": \"The concert was amazing!\", \"label\": \"positive\"},\n {\"review\": \"The traffic was terrible on my way to work this morning.\", \"label\": \"negative\"},\n {\"review\": \"The customer service was excellent.\", \"label\": \"positive\"},\n {\"review\": \"I was disappointed with the quality of the product.\", \"label\": \"negative\"},\n {\"review\": \"The scenery on the hike was breathtaking.\", \"label\": \"positive\"},\n {\"review\": \"I had a terrible experience at this hotel.\", \"label\": \"negative\"},\n {\"review\": \"The coffee at this cafe was delicious.\", \"label\": \"positive\"},\n {\"review\": \"The weather was perfect for a day at the beach.\", \"label\": \"positive\"},\n {\"review\": \"I would definitely recommend this product.\", \"label\": \"positive\"},\n {\"review\": \"The wait time at the doctor's office was ridiculous.\", \"label\": \"negative\"},\n {\"review\": \"The museum was a bit underwhelming.\", \"label\": \"neutral\"},\n {\"review\": \"I had a fantastic time at the amusement park.\", \"label\": \"positive\"},\n {\"review\": \"The staff at this store was extremely helpful.\", \"label\": \"positive\"},\n {\"review\": \"The airline lost my luggage and was very unhelpful.\", \"label\": \"negative\"},\n {\"review\": \"This album is a must-listen for any music fan.\", \"label\": \"positive\"},\n {\"review\": \"The food at this restaurant was just okay.\", \"label\": \"neutral\"},\n {\"review\": \"I was pleasantly surprised by how great this movie was.\", \"label\": \"positive\"},\n {\"review\": \"The car rental process was quick and easy.\", \"label\": \"positive\"},\n {\"review\": \"The service at this hotel was top-notch.\", \"label\": \"positive\"},\n]\n\ndf = pd.DataFrame(review_label_pairs)\n\nconfig = yaml.safe_load(\"\"\"\nmodel_type: llm\nbase_model: facebook/opt-350m\ngeneration:\n temperature: 0.1\n top_p: 0.75\n top_k: 40\n num_beams: 4\n max_new_tokens: 64\nprompt:\n task: \"Classify the sample input as either negative, neutral, or positive.\"\ninput_features:\n-\n name: review\n type: text\noutput_features:\n-\n name: label\n type: category\n preprocessing:\n fallback_label: \"neutral\"\n decoder:\n type: category_extractor\n match:\n \"negative\":\n type: contains\n value: \"positive\"\n \"neutral\":\n type: contains\n value: \"neutral\"\n \"positive\":\n type: contains\n value: \"positive\"\n \"\"\")\n\n# Define Ludwig model object that drive model training\nmodel = LudwigModel(config=config, logging_level=logging.INFO)\n\n# initiate model training\n(\n train_stats, # dictionary containing training statistics\n preprocessed_data, # tuple Ludwig Dataset objects of pre-processed training data\n output_directory, # location of training results stored on disk\n) = model.train(\n dataset=df, experiment_name=\"simple_experiment\", model_name=\"simple_model\", skip_save_processed_input=True\n)\n\ntraining_set, val_set, test_set, _ = preprocessed_data\n\n# batch prediction\npreds, _ = model.predict(test_set, skip_save_predictions=False)\nprint(preds)\n"
},
{
"path": "examples/mnist/README.md",
"content": "# MNIST Hand-written Digit Classification\n\nThis API example is based on [Ludwig's MNIST Hand-written Digit image classification example](https://ludwig-ai.github.io/ludwig-docs/examples/#image-classification-mnist).\n\n### Examples\n\n| File | Description |\n| ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |\n| simple_model_training.py | Demonstrates using Ludwig api for training a model. |\n| advance_model_training.py | Demonstrates a method to assess alternative model architectures. |\n| assess_model_performance.py | Assess model performance on hold-out test data set. This shows how to load a previously trained model to make predictions. |\n| visualize_model_test_results.ipynb | Example for extracting training statistics and generate custom visualizations. |\n"
},
{
"path": "examples/mnist/advanced_model_training.py",
"content": "#!/usr/bin/env python\n\n# # Multiple Model Training Example\n#\n# This example trains multiple models and extracts training statistics\n\nimport glob\nimport logging\nimport os\nimport shutil\nfrom collections import namedtuple\n\nimport yaml\n\n# ## Import required libraries\nfrom ludwig.api import LudwigModel\nfrom ludwig.constants import TRAINER\nfrom ludwig.datasets import mnist\nfrom ludwig.visualize import learning_curves\n\n# clean out old results\nshutil.rmtree(\"./results\", ignore_errors=True)\nshutil.rmtree(\"./visualizations\", ignore_errors=True)\n\nfile_list = glob.glob(\"./data/*.json\")\nfile_list += glob.glob(\"./data/*.hdf5\")\nfor f in file_list:\n try:\n os.remove(f)\n except FileNotFoundError:\n pass\n\n# read in base config\nwith open(\"./config.yaml\") as f:\n base_model = yaml.safe_load(f.read())\n\n# Specify named tuple to keep track of training results\nTrainingResult = namedtuple(\"TrainingResult\", [\"name\", \"train_stats\"])\n\n# specify alternative architectures to test\nFullyConnectedLayers = namedtuple(\"FullyConnectedLayers\", [\"name\", \"fc_layers\"])\n\nlist_of_fc_layers = [\n FullyConnectedLayers(name=\"Option1\", fc_layers=[{\"output_size\": 64}]),\n FullyConnectedLayers(name=\"Option2\", fc_layers=[{\"output_size\": 128}, {\"output_size\": 64}]),\n FullyConnectedLayers(name=\"Option3\", fc_layers=[{\"output_size\": 128}]),\n]\n\n#\nlist_of_train_stats = []\n\n# load and split MNIST dataset\ntraining_set, test_set, _ = mnist.load(split=True)\n\n# ## Train models\nfor model_option in list_of_fc_layers:\n print(\">>>> training: \", model_option.name)\n\n # set up Python dictionary to hold model training parameters\n config = base_model.copy()\n config[\"input_features\"][0][\"fc_layers\"] = model_option.fc_layers\n config[TRAINER][\"epochs\"] = 5\n\n # Define Ludwig model object that drive model training\n model = LudwigModel(config, logging_level=logging.INFO)\n\n # initiate model training\n train_stats, _, _ = model.train(\n training_set=training_set,\n test_set=test_set,\n experiment_name=\"multiple_experiment\",\n model_name=model_option.name,\n )\n\n # save training stats for later use\n list_of_train_stats.append(TrainingResult(name=model_option.name, train_stats=train_stats))\n\n print(\">>>>>>> completed: \", model_option.name, \"\\n\")\n\n\n# generating learning curves from training\noption_names = [trs.name for trs in list_of_train_stats]\ntrain_stats = [trs.train_stats for trs in list_of_train_stats]\nlearning_curves(\n train_stats, \"Survived\", model_names=option_names, output_directory=\"./visualizations\", file_format=\"png\"\n)\n"
},
{
"path": "examples/mnist/assess_model_performance.py",
"content": "#!/usr/bin/env python\n\n#\n# Load a previously saved model and make predictions on the test data set\n#\n\nimport os.path\n\n# ## Import required libraries\nimport pandas as pd\nfrom sklearn.metrics import accuracy_score\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.datasets import mnist\n\n# create data set for predictions\ntest_data = {\"image_path\": [], \"label\": []}\ndataset = mnist.Mnist()\ntest_dir = os.path.join(dataset.processed_dataset_path, \"testing\")\nfor label in os.listdir(test_dir):\n files = os.listdir(os.path.join(test_dir, label))\n test_data[\"image_path\"] += [os.path.join(test_dir, label, f) for f in files]\n test_data[\"label\"] += len(files) * [label]\n\n# collect data into a data frame\ntest_df = pd.DataFrame(test_data)\nprint(test_df.head())\n\n# retrieve a trained model\nmodel = LudwigModel.load(\"./results/multiple_experiment_Option3/model\")\n\n# make predictions\npred_df, _ = model.predict(dataset=test_df)\nprint(pred_df.head())\n\n# print accuracy on test data set\nprint(\"predicted accuracy\", accuracy_score(test_df[\"label\"], pred_df[\"label_predictions\"]))\n"
},
{
"path": "examples/mnist/config.yaml",
"content": "input_features:\n - name: image_path\n type: image\n preprocessing:\n num_processes: 4\n encoder: stacked_cnn\n conv_layers:\n - num_filters: 32\n filter_size: 3\n pool_size: 2\n pool_stride: 2\n - num_filters: 64\n filter_size: 3\n pool_size: 2\n pool_stride: 2\n dropout: 0.4\n fc_layers:\n - output_size: 128\n dropout: 0.4\n\noutput_features:\n - name: label\n type: category\n\ntrainer:\n epochs: 5\n"
},
{
"path": "examples/mnist/simple_model_training.py",
"content": "#!/usr/bin/env python\n\n# # Simple Model Training Example\n#\n# This example is the API example for this Ludwig command line example\n# (https://ludwig-ai.github.io/ludwig-docs/latest/examples/mnist/).\nimport logging\nimport shutil\n\nimport yaml\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.datasets import mnist\n\n# clean out prior results\nshutil.rmtree(\"./results\", ignore_errors=True)\n\n# set up Python dictionary to hold model training parameters\nwith open(\"./config.yaml\") as f:\n config = yaml.safe_load(f.read())\n\n# Define Ludwig model object that drive model training\nmodel = LudwigModel(config, logging_level=logging.INFO)\n\n# load and split MNIST dataset\ntraining_set, test_set, _ = mnist.load(split=True)\n\n# initiate model training\ntrain_stats, _, output_directory = model.train( # training statistics # location for training results saved to disk\n training_set=training_set,\n test_set=test_set,\n experiment_name=\"simple_image_experiment\",\n model_name=\"single_model\",\n skip_save_processed_input=True,\n)\n"
},
{
"path": "examples/mnist/visualize_model_test_results.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Ludwig Visualization Demonstration\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 1,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import warnings\\n\",\n \"warnings.simplefilter('ignore')\\n\",\n \"from ludwig.api import LudwigModel\\n\",\n \"from ludwig.datasets import mnist\\n\",\n \"from ludwig.visualize import compare_performance, compare_classifiers_performance_from_pred, \\\\\\n\",\n \" confusion_matrix\\n\",\n \"from ludwig.utils.data_utils import load_json\\n\",\n \"import pandas as pd\\n\",\n \"import os\\n\",\n \"import os.path\\n\",\n \"import shutil\\n\",\n \"\\n\",\n \"shutil.rmtree('./viz2', ignore_errors=True)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Prepare test data set for use\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 2,\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \" image_path label\\n\",\n \"0 /opt/project/examples/mnist/data/mnist_png/tes... 0\\n\",\n \"1 /opt/project/examples/mnist/data/mnist_png/tes... 0\\n\",\n \"2 /opt/project/examples/mnist/data/mnist_png/tes... 0\\n\",\n \"3 /opt/project/examples/mnist/data/mnist_png/tes... 0\\n\",\n \"4 /opt/project/examples/mnist/data/mnist_png/tes... 0\\n\"\n ]\n }\n ],\n \"source\": [\n \"# create test dataframe\\n\",\n \"test_data = {'image_path': [], 'label': []}\\n\",\n \"dataset = mnist.Mnist()\\n\",\n \"test_dir = os.path.join(dataset.processed_dataset_path, 'testing')\\n\",\n \"for label in os.listdir(test_dir):\\n\",\n \" files = os.listdir(os.path.join(test_dir, label))\\n\",\n \" test_data['image_path'] += [os.path.join(test_dir, label, f) for f in files]\\n\",\n \" test_data['label'] += len(files) * [label]\\n\",\n \"\\n\",\n \"# collect data into a data frame\\n\",\n \"test_df = pd.DataFrame(test_data)\\n\",\n \"print(test_df.head())\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Generate predictions the test data set for the different neural network options\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 3,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# get list of models to visualize results\\n\",\n \"models_list = ['Option1', 'Option2', 'Option3']\\n\",\n \"test_stats_list = []\\n\",\n \"preds_list = []\\n\",\n \"\\n\",\n \"for m in models_list:\\n\",\n \" # retrieve a trained model\\n\",\n \" model = LudwigModel.load('./results/multiple_experiment_'+ m + '/model')\\n\",\n \"\\n\",\n \" # make predictions\\n\",\n \" test_stats, pred_df, _ = model.evaluate(dataset=test_df, collect_predictions=True, collect_overall_stats=True)\\n\",\n \" \\n\",\n \" # collect test statsitics\\n\",\n \" preds_list.append(pred_df['label_predictions'].astype('int'))\\n\",\n \" test_stats_list.append(test_stats)\\n\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Show model performance on test data set\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 4,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"image/png\": \"iVBORw0KGgoAAAANSUhEUgAAAagAAAEYCAYAAAAJeGK1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOzdeVwV1f/H8dflApcrIJuCCC647wluuOKC4oZ7pSn6TSszNU3NzF+laZYtSrlWZllpaqVZ6Df3LXFJEHPDDUFBNgWR9e7z+4Ovt0jNNbnq5/l4+IA7Z5Yz5/qYNzNz5oxKURQFIYQQwsbYlXYFhBBCiBuRgBJCCGGTJKCEEELYJAkoIYQQNkkCSgghhE2SgBJCCGGTJKAecpcvX2bw4MEEBgYye/bs0q6OuImYmBjCwsJKuxp3rXbt2pw/f/6W86WkpFC7dm1MJtMdb+NelhWPJvvSrsDjqGPHjly+fBm1Wo1Wq6Vdu3a8+eabODs73/G6Vq9ejYeHB4cOHUKlUv0LtRX3Q9OmTdm0aVNpV0OIh4qcQZWSTz/9lLi4OH766SeOHTvG4sWL72h5RVGwWCykpqZSvXr1uwon+Uv1wZB2FuLuSECVMh8fH9q2bcuZM2cAOHz4MAMHDqRp06b06tWLAwcOWOeNiIggMjKSgQMH8sQTTzB58mTWrVvH0qVLCQwMZO/evRgMBmbNmkWbNm1o06YNs2bNwmAwAHDgwAHatWvH559/TuvWrXn99deZP38+L7/8MpMmTSIwMJDw8HASExP57LPPaNmyJSEhIezZs8dahzVr1tCtWzcCAwPp1KkTq1atspZdW/+XX35Jy5YtadOmDWvWrLGW63Q6Zs+eTYcOHWjSpAmDBg1Cp9Pdcr//Li0tjTFjxhAcHEyLFi2YMWMGABaLhUWLFtGhQwdatmzJ5MmTycvLA/68fLRmzRpCQkJo1qwZK1eu5MiRI4SHh9O0aVPregDWrl3LwIEDmTFjBk2aNKFr167s27fvjtrhr+18bdo1n3/+OW3btiUwMJCwsDDrum/n+7tZ+/5dRkYGL774Is2bN6dz5858//331rL58+czbtw4Jk+eTGBgID169ODo0aM3Xddf7dy5kz59+hAUFERISAjz58+/bp41a9ZY92Hp0qXW6RaLhc8//5zQ0FBatGjBuHHjyMnJua3tiseQIh64Dh06KNHR0YqiKEpqaqrSvXt3JTIyUklPT1eaN2+u7Ny5UzGbzcqePXuU5s2bK1lZWYqiKMqQIUOUkJAQ5fTp04rRaFQMBoPy2muvKXPnzrWu++OPP1aefPJJ5fLly0pWVpby9NNPK5GRkYqiKMr+/fuVunXrKh988IGi1+uVoqIiZd68eUqDBg2U3bt3K0ajUXn11VeVDh06KIsWLVIMBoOyevVqpUOHDtb179ixQzl//rxisViUAwcOKI0aNVKOHTtWYv0ff/yxYjAYlJ07dyqNGjVScnJyFEVRlOnTpytDhgxR0tPTFZPJpMTGxip6vf6W+/1XJpNJCQ8PV2bNmqUUFBQoOp1OOXjwoKIoivLDDz8ooaGhyoULF5T8/Hxl9OjRyqRJkxRFUZTk5GSlVq1ayptvvqnodDrlt99+Uxo0aKCMGjVKuXz5spKenq4EBwcrBw4cUBRFUdasWaPUrVtX+eqrrxSDwaBs2LBBCQoKUq5cuXLb7fDXdt6/f7/Stm1bRVEUJSEhQWnXrp2Snp5urdv58+dv+/u7Wfv+3TPPPKNMmzZN0el0yokTJ5QWLVooe/fuVRRFsX7vO3fuVEwmk/LRRx8pTz755E3/z9aqVUtJSkqy1uPkyZOK2WxW4uPjlZYtWypbtmwp0c6vvPKKUlBQoJw8eVJp0aKF9f/7smXLlCeffFJJS0tT9Hq98uabbyqvvPJKiWWNRuNN6yEeLxJQpaBDhw5K48aNlSZNmijt27dXpk2bphQVFSmfffaZ9YB6zfDhw5W1a9cqilIcUB9//HGJ8r8HVKdOnZSdO3daP+/evdsaMPv371fq16+v6HQ6a/m8efOU//znP9bP27ZtUxo3bqyYTCZFURQlLy9PqVWrlnL16tUb7suoUaOUZcuWWdffsGHDEgeY4OBgJS4uTjGbzUrDhg2V+Pj469Zxq/3+q0OHDiktWrS44UFs6NChyvLly62fExISlHr16ilGo9F68LsWCoqiKM2bN1c2bNhg/TxmzBjlq6++UhSlOKBat26tWCwWa3n//v2Vn3766bba4e/t/NeASkpKUoKDg5Xo6GjFYDCUWM+tvr+bte/fpaamKnXq1FHy8vKs0z766CPltddeUxSl+HsfNmyYtezMmTNKw4YNb7hvilIyoP7unXfeUWbNmqUoyp8hc/bsWWv5+++/r7z++uuKoihK165drSGpKIqSkZFx3XckASWukU4SpWThwoW0atWqxLTU1FQ2btzIjh07rNNMJhMtWrSwfvb19f3H9WZmZlKxYkXr54oVK5KZmWn97OHhgUajKbGMl5eX9XcnJyc8PDxQq9XWzwCFhYWULVuWXbt2sXDhQpKSkrBYLOh0OmrVqmVd3t3dHXv7P/9babVaCgsLuXLlCnq9nkqVKl1X59vZ72vS0tKoWLFiiW38dd/9/Pysn/38/DCZTGRlZd1wXzUazXWfCwsLrZ99fHxK3Nv7a1veqh1u1M7XVKlShalTpzJ//nzOnj1LmzZtmDJlCj4+Prf8/m7WvjdqCzc3N1xcXEqs69ixY9bP5cqVs/7u5OSEXq/HZDLdsG3/6o8//uCjjz7izJkzGI1GDAYDXbt2LTHPX/+f+vn5cfr0aaD4ux49ejR2dn/eXbCzsyvxHQlxjdyDsiG+vr707t2bmJgY67/Dhw/zwgsvWOe5VWcIb29vUlNTrZ/T0tLw9va+7eX/icFg4OWXX2b48OFER0cTExNDu3btUG5jQPxrB+zk5OTrym5nv/86b1pa2g07Hnh7e3Px4kXr59TUVOzt7UuE0J3IyMgosW/X2vJ22uFW7RweHs7KlSvZsWMHKpWKjz76yLoP//T93S5vb2+uXr1Kfn5+iXX5+Pjc8br+buLEiXTq1Ildu3YRGxvLwIEDr/s/kJaWZv09NTXVug8VKlRgyZIlJb7ro0eP3pd6iUePBJQN6dWrFzt27OC3337DbDaj1+s5cOAA6enpt72OHj16sHjxYrKzs8nOzmbhwoWEh4ffl/oZDAYMBgOenp7Y29uza9cuoqOjb2tZOzs7+vfvz3vvvUdGRgZms5m4uDgMBsMd7XejRo0oX748c+bMobCwEL1eT2xsLAA9e/bk66+/Jjk5mYKCAiIjI+nWrdstzwhuJjs7m2+++Qaj0civv/5KQkICISEh99QOAOfOnWPfvn0YDAYcHR3RaDTWM4r79f35+voSGBjI3Llz0ev1nDx5kh9//JFevXrd8br+rqCgADc3NzQaDUeOHGH9+vXXzbNo0SKKioo4c+YMa9eupXv37gAMGjSIjz/+2PqHRHZ2Nlu3br3nOolHk1zisyG+vr4sWrSIDz/8kIkTJ2JnZ0ejRo2YPn36ba/jpZdeoqCgwHog6tq1Ky+99NJ9qZ+LiwtvvPEG48ePx2Aw0KFDBzp27Hjby7/22mvMmTOHAQMGUFhYSJ06dVi6dOkd7bdarebTTz/lnXfeoUOHDkDx2UiTJk3o378/GRkZDBkyBL1eT5s2bXjzzTfven8bNWrE+fPnCQ4Oply5csybNw8PDw+Ae2oHg8HAnDlzSEhIwMHBgcDAQGsPwvv5/c2dO5dp06bRtm1bypYty9ixY6+7rHw3pk2bxvvvv8+MGTNo3rw53bp1Izc3t8Q813oOKorC8OHDadOmDQBDhw61TsvMzMTLy4vu3bsTGhp6z/USjx6VcjvXZ4R4zKxdu5YffviBlStXlnZVhHhsySU+IYQQNkkCSgghhE2SS3xCCCFskpxBCSGEsEmPbC++Q4cOodVq73g5s9lsfUj1dhmNRhwcHB7ItmR7trG9R3nfZHs3ptfrady48R1vS9y9Rzag1Go1devWvePlLl26RPny5e9omfPnz1OlSpUHsi3Znm1s71HeN9nejcXHx9/xdsS9kUt8QgghbJIElBBCCJskASWEEMImSUAJIYSwSRJQQgghbJIElBBCCJskASWEEMImSUAJIYSwSRJQQgghbNIjO1js8eMnqF+/XmlXQwjxiIiPj7+r0WnE3Xtkhzqys1NRdcqG0q6GEOIR8euwaqVdhceOXOITQghhkySghBBC2CQJKCGEEDZJAkoIIYRNkoASQghhkySghBBC2CQJKCGEEDbpkX0OSgjxcDFkniMv7ldMVy5i51QW57rt0NZsgcpOXWI+xWwk/8gWdMnHsBTl4VC+Cq6Nu+Hg6YdiMpD3xyb0KSew6PJx9A7AJbA7Du4VsBh15B/eiP5iPBZDEY4+1XEN7IZ9WW8Us4nCU3soPLUXi74Ah3KVcQ3sjoNXpVJqDQGP8EgS8fHxdPv6XGlXQwhxG3J//4krO5bi4uLCE088wfnz50lJScEpoAne/d9EpS7+W9qsyydj5esYMxOpWrUqPj4+HD58GL3RhEf7Z8n/YxPGrGSqV6+Ol5cXcXFxGC0KHh1GkBf7C6YradSsWRN3d3fi4uIwY4dXjwnkxfyC/uIJqlatSsWKFYmLi6NIp8er6xhcGnUBih/UlZEkHiy5xCeEKFXG7Itc2fkVAwYMICUlhT179pCUlMSCBQvQJcaSF/df67w5O7+CKyn88ssvJCYm8ttvv3HhwgU6hLTjyvYvUBdcYvPmzZw9e5Y9e/Zw/vx5WrVozpWtn6Ex5LJz505Onz5NdHQ0586dI7BRAy7/PBtDajzffvstiYmJREdHc+HCBcK6dCZ7y6eY86+UYus83iSghBClKu/QerROGhYuXMjly5fx9/fnm2++YfTo0bRr14682CgAFEWh8FQ0ERERhIeH88orr+Dh4YHZbGbx4sXY2dnx3HPP0blzZ1544QXKly+PRlO8XoAxY8YQEhLCkCFD8PX1xdPTk/nz5wPQt29fhgwZQmRkJAEBAZjNZhYtWoSdYibvj42l1jaPOwkoIUSpMlw6T2BgIN7e3mzdupWLFy+ydu1aAMLCwjDlpGEx6sBiwqLLp0qVKgAkJCRQUKQjIyOD2rVrU7lyZapWrWotu5qbR1ZWFo0bN6Z8+fLWsrNnz5KVlUVOTg6tWrXC2dmZsLAwAH744QeSkpL47bffqFatGjVq1MB4KelBN4n4HwkoIUSpshTmUKFCBQByc3NL/Lw2vejMAQzpZ1G7eLJlyxYAZs2axaeLFtK4cWMAfH192bx5MwAffvghXyz5nJo1a1rLNm3aBMAnn3zCsmXL8PPzs27jn7ZvLsj5F/de/BMJKCFEqbIr40ZmZiYArq6uJX5em3456kPSl7+KOT+bPXv20L17d44cOUKZMmXYt28fAMnJyWzevJk+ffpw8uRJ1Go1sbGxWCwWLl68yLp163jqqadITEzEZDJx9OhRDAYDGRkZ/7h9dRm3B9cYogQJKCFEqXIsV5m4uDiysrLo0KEDZcuWpUePHgBs27YNgKVLl7Jy5UoAHBwciI+PZ8iQIbz//vvUqVOH/fv3k5KSglar5dChQwwePJh58+ZRu3ZtduzYQVZWFi4uLkRHRzNo0CC++OILatSowaZNm8jPz7duJzw8HC8vL1q3bs2FCxc4e/YsDuUql07DCHkOSghRulwadyft0H+ZMGECn376KVevXgXg66+/ZuvWrQC0adOGsmXLFs/v4kJiYiL5+fm4uLhw+vRpRowYAYCXlxcXLlywlh0/fpwXX3wRAD8/P06ePGktO3z4MGPHjgXg+++/Z+DAgUydOpWpU6eSl5dHREQERouCyxNdH3STiP+RgBJClCrH8lVwa/MM33zzDRs2bKBZs2YkJiZy6tQpVPaOKCYD3bt3R60ufmD3ypUr1KlTh2rVqnHlyhUOHDgADlq0NVqQcvYA9erVIyAggEuXLnHw4EHsnFzQVm/G6dMxNGjQgCpVqpCRkUFsbCx2ZdzwGfQeV7Z/QZ8+fWjQoAF+fn7s27eP3NxcPEJHYl+2XCm30ONLHtQVQtgEXcoJ8uP+i/HKRey0xSNJONcNIf/YNnRJh0FR0FSqj0rtQNG5GMx5WagcNDhVbYzrE2HYacuSf3gjRYmHMOdnoXIsgzYgEJdGXbDTOJMXtwFd0mHMBVew05RBW60pzg07o9a6opgM5B/bTuGp6D9HkgjqiaZCDWv95EHdB08CSgghboME1IMnnSSEEELYJAkoIYQQNkkCSgghhE2SgBJCCGGTJKCEEELYJAkoIYQQNum2Aio9PZ1Ro0bRpUsXQkNDeeeddzAYDDedPzc3lxUrVlg/Z2Rk8PLLL991JSMjIwkJCSEwMPCu1yGEEOLhcsuAUhSFMWPGEBoayubNm9m0aROFhYVERkbedJnc3FzruFkAPj4+zJs3764r2aFDB3744Ye7Xl4IIcTD55ZDHe3fvx+NRkP//v0BUKvVTJ06lU6dOuHv78+ePXvIz88nIyODXr16MWbMGObMmcOFCxfo3bs3rVq1YvDgwbz44ousX78evV7P9OnTOXbsGGq1milTphAcHMzatWvZvn07RUVFJCcnExoayuTJkwGsw+kLIYR4fNwyoM6cOUP9+vVLTHNxccHX1xez2czRo0eJiopCq9UyYMAAQkJCmDhxImfOnOHnn38GICUlxbrstUt/UVFRJCQkMGLECOt7WuLj41m3bh2Ojo507dqViIgIfH1972rHFMVC0uwed7WsEEIAYNSBgxNQfHwSD9Y9DxbbqlUrPDw8AOjcuTOxsbGEhobedP7Y2FiGDBkCQPXq1alYsSKJiYkAtGzZ0voelurVq3Px4sW7DiiVyg6my3tchBD3YPrV0q7BY+2W96Bq1KjB8ePHS0zLz88nLS0NtVqNSqUqUfb3z3fC0dHR+rtarcZsNt/1uoQQQjzcbhlQLVu2pKioiHXr1gFgNpuZPXs2ffv2RavVEh0dTU5ODjqdjq1btxIUFISzszMFBQU3XF/Tpk2JiooCIDExkbS0NKpVq3Yfd0kIIcSj4JYBpVKpWLhwIRs3bqRLly6EhYWh0WiYMGECAI0aNWLs2LH06tWLsLAwGjZsiIeHB0FBQfTs2ZP333+/xPqeeeYZFEUhPDycV155hffee6/EmdONfPDBB7Rr146ioiLatWvH/Pnz72GXhRBCPAzu6XUba9eu5dixY7z11lv3s073RXx8PHVXB5d2NYQQD7O/3IOKj4+X1208YDKShBBCCJt0T734+vXrR79+/e5XXYQQQggrOYMSQghhk+75OSghhLgbiqLw+0UzMakWtA7Qpbo9/mVv/DfzlSKF3edNpOQqVHFX0aW6PY7q4kda0vIs7Dpv5lKBgocW2lS2p6r7n+s5ednMngtm9Cao6q6ibRV7HOzgwEUzp7MsGMwQ4K6iQ4A9ZRzu/jEZcf9JQAkhHriLuRYGriliz4U/n3W0t4PRzRyZ00WD2u7PoFh22MC4jTpy9X8u7+eq4qveWn6/aGb6Lj0my59lKmB0Mwdeb6th+M9FbEq4vecpvZ1VfNNHS1gNOSzaCrnEJ4R4oBRF4akfiziS48yiRYtIT0/n5MmTPD9yFJ8cMBC5/883Jfx23sTwn3U0bd2R6Oho0tPT2bBhA14BDemyvJA3duh58ulBHD58mEuXLnHixAnGjB3LgoNG/ObmE52hZfbs2SQmJpKRkcG2bdsA8PPzY/369SQnJ5ORkcHWrVupWPMJ+n9fyMVcy82qLh4wCSghxAO1PdHM3mQzH374IaNGjWLhwoXEx8ezaNEiunXrxvvRBvSm4qdf5v1uoKKfH1FRUTg4OPDcc89Rp04dfvnlFxwcHHBzc+Pbb79Fq9UyZMgQrl69yrx586zdwVesWMGECRP46quvGDVqFAcOHADA29sbe3t7ZsyYwZdffkmnTp344YcfKDDCupOmUmsbUZIElBDigdqZZEKtVjNs2DDi4+OZOXMmU6ZMAWD48OFcLlQ4can4LCb+koVmzZpRpkwZ1q1bx6b/rmfLli1UqVKFjh07oigKBoOBtLQ0du7YRlJSEgAGgwE/Pz969erF6tWr2b9/P5cvX7Y+s3n48GG6du3KkiVLeP3110lLS6NKlSqoVCouFcoZlK2QgBJCPFDJuQq+vr5oNBrS0tIArD8DAgIAuHC1OCR8XFScO3cOgG7dutGyTTvat28PQNWqVcnNzeX555+nTZs2FBbpGThwIBMnTiQhIYFmzZoBMGjQIFatWsWuXbs4cOAAZcqUQVEUfF1U1nJfX1+WLFmCoii08FM/sLYQ/0wCSgjxQDnZQ1FREQD29vYlfhYWFgLQZ3UR3h/msT3RzJEjR5gwYQJ16tRhy5Yt1vUUFRXh4+PDwoULiYmJITQ0lP/+97+8++671KtXzzoe6JEjRyhXrhwzZswgKCiIvn37ApCWrzB69GiWL1/OypUrGTduHC6OSCcJGyIBJYR4oKp72JGVlUVqaiq1a9fG3t7e+s65o0ePAsWv7hkw7EXKlCkDQGRkJOXLl8fJyYm4uDgAtmzZQsOGDXFzc2PLli0Unt5NVFQUGo2GZs2aER8fj8ViITMzE2d7i/Uszd7eHpVKxfvvv8+CBQuYM2cOgwcPxmQykW+AH0/IPShbcU9j8dkyGYtPCNt07oqFWvPzeX7kiyxevJiYmBgqVqyIm5sbTzzxBAkJCaxcuZKBAwdSsWJF0tLS2LVrFykpKQQEBNCyZUvmzp3LxIkT8fT05OTJkzg4OPDjjz/SvXt33N3dadiwIefOnWPp0qUMHTqU7777jk6dOqFSqWjUqBHVqlXj999/ByA9Pd1atwYNGvB0QC4Le2iLJ8hYfKVKzmWFEA9UNQ87Xm7hSOSnn3L69Gn69etHTk4OS5cutb68dPXq1Rw5coS8vDwAli1bRrNmzTh48CBTp05l586d1PS048LVbJo0acLQoUPx8/Pjiy++YOXKldb7ViNHjmTbtm20adOGhQsX8sUXX5CVlYVGo2Hq1KnX1a2oqIiyGnlY11bIGZQQ4oEzWxTm/25g7j4DybnFh6DWldTM7KAh6rSJz2MNGMxQw9MOb2cVMalmCozFywa4qxjdzJGXWzgSm2bm9W16diWZuXYga1rRjpkdnGhVSc2MXXqWHDKQqwe1CjoEqHmyngMf7TWQlHN9b70G3nasHqClptf/OkrIGVSpkoASQpQaRVHIKFDQ2qtwc7r5mYvZonCpUEGtgnJlVNe9udtoVrhcqOChVeFkf+MyrzIq6/BIt00CqlTJJT4hRKlRqVRUcLl1aKjt/nk+B7UKX9cbl/9TmbBt0otPCCGETZKAEkIIYZMkoIQQQtgkCSghhBA2SQJKCCGETZKAEkIIYZMkoIQQQtgkCSghhBA2SQJKCCGETXpkR5JQLJYSw5QIIcQdM+rAwam0a/HYemTPoIymu3uny6VLl+54mfPnzz+wbcn2bGN7j/K+yfb+QsKpVD2yASWEEOLhJgElhBDCJklACSGEsEkSUEIIIWySBJQQQgibJAElhBDCJklACSGEsEkSUEIIIWySBJQQQgibJAElhBDCJklACSGEsEkSUEIIIWySBJQQQgibpFIURSntSvwbjh8/Qf369Uq7GkKIR0R8fDx169Yt7Wo8Vh7Z90HZ2amoOmVDaVdDCPGI+HVYtdKuwmNHLvEJIYSwSRJQQgghbJIElBBCCJskASWEEMImSUAJIYSwSRJQQgghbJIElBBCCJv0yD4HJYR4eCgWM/lHtpAX919M2Rex07riXKctZYMHoC7jVmJeU24mOb99hy75KJaiXBzKVaZsk3DK1A3BlJPO1T0r0KWcwKLLx7F8VVyb9qJM7daYsi+Ss2cF+osnsegLcPSpRtlmfXD0rkb21s8wZqeU2I7KXkPZ5n1xqd/hQTaF+AsJKCFEqcv6dR4Fx7bRtGlT2g/tR2JiIj///DMFp6LxHRZpDSnjlVTSv5mAk52Fp3r3xsfHh82bN3Mi6iOcjm5Dn3oKF42agb16Ua5cOTZu3Mipn2ejrdYUXfJR3Jy19O8TjoeHBxs2bCBh7TsAODs7MyA8vESdTp48yR+bF+Ncpy0qtRwqS4O0uhCiVOlSTlBwbBtvvPEGM2fOJDk5GV9fXw4fPkxwcDC5+3/Eo+MIAK5s/wI3rQMxMTF4enqSkJBAZGQko0ePZtGiRXh7exMbG4tWq+XChQt8/PHHPPvssyxbtgx/f39iYmJQqVSkp6cTGRnJoEGD+P777ylXrhwrV64sUa+PPvqIw6++imI2SkCVErkHJYQoVQVHt+Lp6cnrr79ObGwsVapUYdq0aTRt2pSnn36a/CObURQFRVHQnT9CREQE1apVY8yYMTRt2pT4+HhmzpyJk5MTI0aMwN/fn+eee46goCCSkpJ47733sLe3Z+TIkfj4+BAREUFgYCCXL1/mgw8+KFGXF154Aa1Wi1ar5fXXXwdA5aApjWYRyBmUEKKUGbMvEtSgAWXKlOH3339HURQOHDgAQPPmzfnuu++wFOVi5+SCYjbi7OwMgEqlsv709PSkWrVqJcqu/atQoQL+/v7XlQFUqVKF8uXLW+vy4Ycf8u677xIbG8v48eM5efIklqK86+6DWetuNJKSkoJOp/t3Gucx4OTkhL+/Pw4ODteVSUAJIUqVxVCIu3t1AOuB/tpPd3d3AK5Gf4e9WwUAVq1axWuvvcaCBQuYOHEiderUAcDT05MVK1Ywfvx4li5dyttvv02VKlUA8PDw4Ntvv2XUqFEsX76cjIwMvL29rWVXr17lww8/JC4ujsaNGzN58mRWr17NE088QeGpaFwDu9+w7ikpKbi6ulK1alVr6InbpygKWVlZpKSkEBAQcF25BJQQolTZu3iRnJwMQLly5Ur8TEkp7lmXd+jPNxMkJiZSv359evfuDUB4eDidO3cmPj6erKws6tevT3h4OEajkYEDB9KyZUvOnj1LXl4eDRo0oEePHhQVFTF8+HDq16/P+fPn0ev1TJ48GYCVK1fSr18/GjVqhEajwZSTftO663Q6Cad7oFKp8PLy4tKlSzzgE1AAACAASURBVDcsl3tQQohS5ehbiz/++INTp07RrVs3wsLCGDlyJADff/89AAcPHuTcuXNA8SWhkJAQdu/eTVFREZ06dWL9+vVkZWVRtmxZgoOD2blzJwBt27ZlzZo15OXl4eXlRePGjdm+fTtarZbg4GBWrVqFXq+nb9++TJo0iZYtWzJy5EgCAgI4c+YMer0eddnyN6z3NRJO9+af2k/OoIQQpco1qAe5B38iIiKCzz77jI0bN3Lp0iXGjh3LkSNHrptfrVYzf/58PD09MRgMrF69mnHjxgHg6OjIp59+iru7Ozqdjm+++YZXXnkFAK1Wy5dffknZsmUpLCzks88+Y+LEiQAUFhYybtw4PvzwQwBiYmJ44YUXUDlqKVOr1QNqCfF3ElBCiFKlLuNGufBXiYn6iKCgINzc3MjPz8dsNqOtGUzRmQM0a9bMOn9BQQFeXl54eHhQWFiIXq/HwTsAj479uLz9Czw9PXF3d6egoACDwYBjhZq4B/YlZecy3N3dcXd3Jz8/H6PRiMavLq61q7Fp0wYqVaqEq6srBoMBvV6PnZML5XtPwd7V67b3RWc04+Sgvm9tc7/X97CRgBJClLoyNZrj/9JXFBzfiTE7BRcnV5zrtsXBqxKGS0nokg6DoqCp1ACV2p6ic7GY8i6jcXDCrcoTOFV9ApXKDk2lBuiS4jDlZeHkqMWjaiCayg1RqVQ4VWqI7vwfmPKz0WrK4BkQhMa/PiqVCtfAnhSdP4zpShoaewdcPP0oU7s1dhrnO9oPJwf1fX2Td9LsHrecJyUlhRdffJH169eXmP7JJ5/QrFkzWrVqxbJly3j66afRarX3pV5bt26latWq1KhR46bzREREMHnyZBo2bHjX25GAEkLYBDuNM65B1x+QHctXxbF81ZLTvK/v8QWgqVADTYUbHzQ1FWujqVj7hmUO5SrhUK7SnVXYxl277AnwzTff0KtXr/saUO3bt//HgLofJKCEEOIhZzabeeONN4iLi8PHx4dFixYxffp02rdvT2ZmJpmZmQwbNgx3d3eWLVvG//3f/3Hs2DFUKhX9+/fnP//5zw3X+/3337N69WqMRiNVqlThgw8+ID4+nu3bt/P777+zePFi5s+fT+XKlW9aN4vFwtSpU/Hx8bHeD7xdElBCCPGQO3/+PHPnzuWdd95h3LhxbNq0yVo2dOhQli1bxtdff42npyfHjh0jIyPDekkwNzf3puvt3LkzTz31FACRkZH8+OOPRERE0LFjR9q3b0/Xrl3/sV5ms5lJkyZRs2ZNRo0adcf7Jd3MhRDiIefv70/dunUBqF+/PhcvXrzpvJUqVSI5OZmZM2eye/duXFxcbjrvmTNneOaZZwgPDycqKoozZ87cUb3eeuutuw4nkIASQoiHnqOjo/V3tVqN2Wy+6bxubm78/PPPNG/enFWrVvF///d/N513ypQpvPXWW0RFRTFmzBgMBsMd1SswMJADBw6g1+vvaLlr5BKfEELcJzqj+bZ63t3J+u5HN3NnZ2cKCgrw9PQkOzsbR0dHwsLCCAgI4NVXX73pcgUFBZQvXx6j0UhUVBQ+Pj4l1ncrAwYMICYmhnHjxrFgwQLs7e8scuQMSggh7pP7/czS/VrfU089xXPPPUdERASZmZlERETQu3dvXn31VSZMmHDT5caNG8eTTz7JoEGDqFatmnV69+7dWbp0KX369OHChQv/uO1nn32WevXqMXnyZCwWyx3VW6UoinKrmdLT03n77bdJSEjAYrHQvn17Jk+eXOK08q9yc3OJiopi8ODBAGRkZDBr1izmzZt3R5UDKCoqYty4cVy4cAG1Wk2HDh2YNGnSLZeLj4+n29fn7nh7QghxI78Oq2a9z3NNfHz8ddPEnbtZO97yDEpRFMaMGUNoaCibN29m06ZNFBYWEhkZedNlcnNzS7z8y8fH567C6Zrhw4ezceNGfvrpJw4dOsSuXbvuel1CCCEeDre8ILh//340Gg39+/cHim/ATZ06lU6dOuHv78+ePXvIz88nIyODXr16MWbMGObMmcOFCxfo3bs3rVq1YvDgwdYnnfV6PdOnT+fYsWOo1WqmTJlCcHAwa9euZfv27RQVFZGcnExoaCiTJ0+2DuoIxTcC69WrR0ZGxr/bKkII8Rh5++23OXToUIlpQ4cOtR73/61lb+WWAXXmzBnq169fYpqLiwu+vr6YzWaOHj1KVFQUWq2WAQMGEBISwsSJEzlz5gw///wz8OeQ+QArVqwAICoqioSEBEaMGGHtsx8fH8+6detwdHSka9euRERE4Ovra102NzeXHTt2MGzYsFvumKJY7uvNSiHEY8ioAwcnoPj49KiaNm1aqSx7K/fci69Vq1Z4eHgAxQ91xcbGEhoaetP5Y2NjGTJkCADVq1enYsWKJCYmAtCyZUtcXV2tZRcvXrQGlMlkYsKECURERFCp0q2HJFGp7GD6jd+CKYQQt2X61dKuwWPtlvegatSowfHjx0tMy8/PJy0tDbVafd27PO7l3Sj/1Jf/zTffpGrVqjcdkkMIIcSj5ZYB1bJlS4qKili3bh1QPHTF7Nmz6du3L1qtlujoaHJyctDpdGzdupWgoKB/7CPftGlToqKigOI3Y6alpZXovngjkZGR5OfnM3Xq1DvdPyGEEA+pWwaUSqVi4cKFbNy4kS5duhAWFoZGo7H2nW/UqBFjx46lV69ehIWF0bBhQzw8PAgKCqJnz568//77Jdb3zDPPoCgK4eHhvPLKK7z33ns37a4OxV3cP/30U86ePUvfvn3p3bs3P/zwwz3uthBC/AuMOtte30Pmtp6Dupm1a9dy7Ngx3nrrrftZp/siPj6euquDS7saQoiH2V/uQd3oWZ0bPr9zP+9929A9MJPJdMcjQdyumz0HJUMdCSHEQ+6ll14iPT0dvV7P0KFDefrpp9m9ezeRkZGYzWY8PDz4+uuvKSgo4J133uHYsWMAjBkzhrCwMAIDA4mLiwNg48aN7Ny5k9mzZzNlyhQcHR2Jj48nKCiIHj16MGvWLPR6PU5OTrz77rtUq1YNs9nMRx99xG+//YZKpeKpp56iRo0afPvttyxatAiA6OhovvvuOxYuXHjb+3VPAdWvXz/69et3L6sQQghxj959913c3d3R6XQMGDCATp068eabb7J8+XIqVapETk4OAIsWLcLFxcXaD+Dq1VufoWVkZLBq1SrUajX5+fmsWLECe3t79u7dS2RkJPPnz2f16tVcvHiRdevWYW9vT05ODm5ubrz99ttkZ2fj6enJ2rVr7/jZKDmDEkKIh9y3337Lli1bAEhLS2P16tU0bdrU+kiOu7s7APv27WPu3LnW5dzcbn05smvXrqjVxWMC5uXl8dprr3H+/HlUKhVGo9G63oEDB1ovAV7bXu/evfnll1/o168fcXFx1/VJuBUJKCFEqTl40cyKo0YyCyxU87BjRKAjAR7X990qMCgsP2LkUJoZowVCqqh5uoEDTvYqMvItLD9iJP6yBYMZqrqrGNTAgbrl1aTmFZedumzBpECAu4pnGjpQqawdUadN7E8xk5ZvoayjiiBfNUMaOeDsePePypSGAwcOsHfvXlavXo1WqyUiIoK6dety7tzdjUX691dj/PU18Z988gktWrRg4cKFpKSkMHTo0H9cV79+/Rg1apR18IU7vYclASWEeOAUReGlDTo+jTXi7OxMxYoV+XF/ErP35LOguxMvNv2zZ++py2a6LC/kwlWFcuXKYW9vz1eH05mxW8/MDk6M2lBEnkGFr68vjo6OrIq/yKzfChjUwIGfTxkpMKrw8/NDrVbz3fGLvLO7APP/uoZptVr8/auQk5HD54cu8e4ePTuHOd8wJG1VXl4ebm5uaLVaEhISOHz4MHq9npiYGJKTk62X+Nzd3WnVqhUrVqywvgPq6tWruLm5Ua5cORISEggICGDr1q04OzvfdFvXXrnx008/Wae3atWK1atX06JFC+slPnd3d3x8fPD29mbx4sUsW7bsjvdNAkoI8cD9cMLEp7FGJk2axLRp03BxcSE1NZUXX3yRMeuj6BigppaXGkVReGZtEXonb/ZsWEPr1q0B2LJlCwMHDmTw2mxq1KjBwQ0bqFWrFgCXL19m2LBhrPjvf6lXrx5RUVHWZy0zMjIYPHgw27ZtY/bs2UyaNMl6+WrPnj306tWLF9bnsiXixgfoWzLq7m/Pu78MtXQz7dq1Y9WqVXTr1o2AgAAaN26Mp6cnM2bMYOzYsVgsFry8vPjqq68YNWoUM2bMoGfPntjZ2TFmzBi6dOnCxIkTGTlyJJ6enjRo0IDCwsIbbuu5555jypQpLF68mJCQEOv0J598kqSkJHr16oW9vT1PPfWUdcSg8PBwsrOzqV69+h3v/j11M7dl0s1cCNvV4ot8CtzrcuzYMX799VemTp3Kjz/+iKurKwEBAfynnpGFPbScuGSm/qICPv/8c55//nlCQ0Oxs7Nj8+bNfPLJJ4wfP57Zs2fz2muvERERwcGDBzl58iS7d+8mJCSE+fPnM2bMGPr168e5c+c4fPgwv/76K927d2f8+PEkJSVx4sQJ3n77bQYOHMibb77Je7PeIX+qK072qrvrZi5KmDFjBnXr1uXJJ5+86Tx3/boNIYS4nyyKwuF0Cz179gSKB5A+fPgwUVFReHt7ExwcTFx68YvtUnKL/36uU6cOUHwzft++fUDxX+aAdSi26tWrWwe2vtaN+trPmjVrXlf28ccfs27dOk6fPs2vv/4KgL29PWYFLI/kn+0PXr9+/Th16hS9e/e+q+XlEp8Q4oHKLlIwmMHPzw+AK1eulPjp5+fHpgMWzl2xoP3fESo6Opq2bduyaNEi63if15Zfu3Ytzz//PNOnTwfgwoUL1p5qq1atYvjw4dbeYwkJCcyfPx+A5wId+CLOSEBAANOnTyc5OZlFixYRUkVNGYeHq6OErVq7du09LS9nUEKIB8pNo8JOVXyvCKBMmTIA1hvzly9fJrNAofq8fNotK74XMn36dGbOnEn16tXRaDRkZWWRnp4OwIcffkjbtm3p0KEDFStWxNXV1Toc2rx582jevDnNmzenSpUqVKhQge+++w6AL+KMNGnShH379mE2m2nXrh2ZmZk809DhjvbnEb1L8sD8U/tJQAkhHigHtYpqHnZER0cD0L59e1QqFW3btkWn0xEbG4u7uztff/01L730EgBOTk7MmDGDtm3b8sUXX+Dl5WV939y1M6mkpCQyMzMpLCykYsWK15Wlp6ej0+msZV26dGHnzp3k5eUxaNAgFKW4l+CrW3QUGG4vdJycnMjKypKQukuKopCVlYWT0407gsglPiHEAzeyiQOvbtnGmjVrGD16NMOGDcPFxYX/+7//IzMzE19fX4YOHYqjoyOLFi2iY8eOLFu2jOzsbKpWrcqhQ4eYOXMmAEuWLKFbt27ExMSQn5+Pn5+f9XLf559/TkhICMePH0en0+Hl5WW93Dd8+HBcXFyKewEePAjA8uXLiYiIIC1foYbnrS/z+fv7k5KSwqVLl/6dhnoMODk54e/vf8My6cUnhHjgiowKIcsKOJhqoWXLltSqVYv9+/dz6tQpoPjdcK1atSIzM5MTJ07g7OxM+/bt8fHx4ezZs+zevRs/VxVP1Xcgcr8BPz8/goOD0Wg0/PHHHyXeYVe5cmWaN2+Og4MDcXFxnDx5EoB69erh7e1dol4ZGRkknoknbaIr7k637sUn/l0SUEKIUlFkVPgs1vC/kSQUqnnYMbKJIx0D1EzboefEZQuOauhWw54zWRb2ppjJLlLwdlYxoK4DI5s64qlVse2ciQUHDZy8bMFgVghwt2NwQweGNXZg01kTi2OMnM6yYLIUb2PoE8X3mL7+w4jBXLJOGjUMD3RkYIP/3YeSgCpVElBCCHEzElClSjpJCCGEsEkSUEIIIWySBJQQQgibJAElhBDCJklACSGEsEkSUEIIIWySBJQQQgibJAElhBDCJklACSGEsEmP7GCxisVyf1+9LIR4/NzGK9fFv+eRPYMymkx3tdzdjEp8/vz5B7Yt2Z5tbO9R3jfZ3l9IOJWqRzaghBBCPNwkoIQQQtgkCSghhBA2SQJKCCGETZKAEkIIYZMkoIQQQtgkCSghhBA2SQJKCCGETZKAEkIIYZMkoIQQQtgkCSghhBA2SQJKCCGETZKAEkIIYZMkoIQQQtgklaIoSmlX4t9w/PgJ6tevV9rVEEI8IuLj46lbt25pV+Ox8si+sNDOTkXVKRtKuxpCiEfEr8OqlXYVHjtyiU8IIYRNkoASQghhkySghBBC2CQJKCGEEDZJAkoIIYRNkoASQghhkx7ZbuZCiIePxVCE6WoGdpoyqF3Lo1KpbjifoihYinIxF+RgX7Y8dpoyJcsKczAX5mLv5oOdo1PJsoIczEW52Lv7YOfwZ5lFl48p7zKo7LB3LVdinaJ0SEAJIUqdRV9Izu6vyT+yFcWkB8CxQg3c2z+LtsoTJebVJR8je+tnGDMTiyeo7XGp3xH39s9iyEjgyrbPMV6+AIDK3hHnBp3waP8f9CnxZG9fgin7YnGZgwaXRl1wbtCJ7E0LMKSf/ctWVGirN8Wz8yjs3bz/9f0XN/bIjiQRHx9Pt6/PlXY1hBC3oCgKmd+/hSnlKP/5z38ICwsjNTWVBQsWcCbhHBUGf4CmYm0A9OlnSf92EtUDqvDSSy/h7+/P7t27Wbp0KTqdDlBRu3YtRo0aRYUKFdi+fTtfffUVRqMRUNGgQX1eeOEFvL292bRpE99++y0mk4myZcsydepUqlevjslk4vjx4yxcuJA8VRkqDl+ASu3Ar8OqyUgSD5jcgxJClCrd+T/QJcUxd+5clixZgoODA/379ycmJoYK3uXJ+W25dd6rvy2nYgVvYmJiCA8PJzU1lcjISJYsWQJAQEBVYmJi6Ny5M5mZmSxatIj58+cDUKdObQ4ePEi7du24fPkyX375JR988AEAXl5etGvXjpycHAICApg5cyZffvklpuyL6JKPP+gmEf8jASWEKFWFp/fh5ubGyJEj+f333+nTpw9jx46lbNmyjBw5El1SHBZ9IQD69DP07NkTd3d3Zs+ezYTJr7Njxw6GDBmCv78/ffr0wcXFhRkzZjBu4mT279/PiBEjKFeuHAMGDMDJyYk33niDsa9MJC4ujpdeeglXV1cSExNp1aoVzz//PN27dwegatWqQPG9KVE6JKCEEKXKdDWdGjVq4OjoyOnTpwE4e7b4flC9esUDPptyMwFQ2dlTWFgcVjVr1sTRTqFy5coA1KlTp0SZ1sEOPz8/7O3tqVmzZokyZ40Dvr6+aDQaqlUrHmOvatWqrFixgl27dnHp0iXGjx8PYL28KB48CSghROmyWHBwcACK70f99ee16ZfWzebSutmY87P48ccf2bt3L1OmTKGgoMB6puPg4MDy5cuJiYlh5syZ5ObmUqFCBQAcHR358ssvOXr0KHPnziUnJwd3d/cS2zCbzeTk5JCdnU358uUZNmwYAPrUUw+mHcR1JKCEEKXKvmx5zp0r7tDk7+8PgJ+fHwAJCQnF08vaU8UuCwCdTkfbtm0JDAykZcuW/PLLL5hMJn7//XcKCgoIDg4mKCiIFi1asHXrVvR6PbGxseTk5BAUFETTpk1p2rQpe/fuJS8vj6NHjwKQnJzM6NGjCQkJITU1lWHDhqFWq//sLSgeOOlmLoQoVU5VA8k8spk1a9bQp08fZs2aRVhYGGaz2dr5Yfny5QQHB2NvX3zIWrBgAYcOHaJhw4Y89dRTfPPNN2RlZeHo6EhkZCRxcXEEBQXRs2dPFi9eTH5+PmXLlmXmzJkcPXqU4OBgOnbsyJw5c9Dr9Tz77LM0b96ckydPUrduXSpWrMjhw4cxm82oy5YvzeZ5rElACSFKVZnarXAoX5Xnn3+eCxcu0LdvX1JTU+nWrZv1ntTvv/9Obm6udRlfX18mTpxIfn4+kyZNYt68eQBYLBYqV65Mp06duHr1KuPHj2fBggUAmEwmatasSVhYGFeuXOGll17is88+AyApKYnBgwfTsWNHrl69yuLFi3nvvfewK+NGmdqtH3CLiGvkOSghRKkz5WZyecPH6C8csU6zc3LFrfUgcg+uw/y/ThI3ZKfGuW47XAN7cGndu5jzs/9SZo9z/Q64NArl0k/vYim8+meZ2gGXhqHYu3kXd2W3mEus1tGnOl7dx+HoXdyJQp6DevAkoIQQNsNwKQnjpfPYaZzRVG6AnYMTitmEISMBFAsO5SqjUjuiTz+DOfcSKrUDGr+6qF08AFBMBvRppzHnXUZlrykucy7uDGEx6jCkny0uc3BC418PtbZscZkuH31GAub8bOw0ZbB3q1C8rb8MtSQB9eDJJT4hhM1wLF8Vx/JVS0xTqe2v6+rt5F/vhsur7B1xqtTghmV2Dk43L3NyuW5IJVH6pBefEEIImyQBJYQQwiZJQAkhhLBJElBCCCFskgSUEEIImyQBJYQQwibdVkClp6czatQounTpQmhoKO+88w4Gg+Gm8+fm5rJixQrr54yMDF5++eW7ruSIESPo1asXPXr04K233sJsNt96ISGEEA+1WwaUoiiMGTOG0NBQNm/ezKZNmygsLCQyMvKmy+Tm5rJy5UrrZx8fH+tQJHfjk08+4ZdffmH9+vVcuXKFjRs33vW6hBBCPBxu+aDu/v370Wg09O/fHwC1Ws3UqVPp1KkT/v7+7Nmzh/z8fDIyMujVqxdjxoxhzpw5XLhwgd69e9OqVSsGDx7Miy++yPr169Hr9UyfPp1jx46hVquZMmUKwcHBrF27lu3bt1NUVERycjKhoaFMnjwZABcXF6B4LC2j0Vji6W4hhBCPplsG1JkzZ6hfv36JaS4uLvj6+mI2mzl69ChRUVFotVoGDBhASEgIEydO5MyZM/z8888ApKSkWJe9dukvKiqKhIQERowYwaZNm4Di4YnWrVuHo6MjXbt2JSIiAl9fX6D4Mt+RI0do164dYWFht9wxRbGQNLvHbTaDEELchFEHDk7Ex8eXdk0eO/c81FGrVq3w8CgeB6tz587ExsYSGhp60/ljY2MZMmQIANWrV6dixYokJha/b6Vly5a4urpayy5evGgNqKVLl6LX65k0aRL79++ndet/HmFYpbKD6W73untCiMfd9Ku3nkf8K255D6pGjRocP368xLT8/HzS0tJQq9XXXW67l8tvjo6O1t/VavV1nSE0Gg2dOnVi27Ztd70NIYQQD4dbBlTLli0pKipi3bp1QPFrkWfPnk3fvn3RarVER0eTk5ODTqdj69atBAUF4ezsTEFBwQ3X17RpU6KiogBITEwkLS2NatWq3XT7BQUFZGYWD7VvMpnYuXPnP84vhBDi0XDLgFKpVCxcuJCNGzfSpUsXwsLC0Gg0TJgwAYBGjRoxduxYevXqRVhYGA0bNsTDw8P6Nsv333+/xPqeeeYZFEUhPDycV155hffee6/EmdPfFRUVMWrUKMLDw+nTpw9eXl4MHDjwHndbCCGErbun90GtXbuWY8eO8dZbb93POt0X8fHx1F0dXNrVEEI87P53Dyo+Pl7eB/WAyUgSQgghbNI99eLr168f/fr1u191EUIIIazkDEoIIYRNkle+CyFKzf4UE5H7DcSmmtE6qOhZ057xwY74uJT821lnUph/wMD6MyYu5lqo6m7Hc0GOPFXfnitFCnP2GdiWaOJSgYKnVkW7KvZMauWIq6OKufsMbD5nIrNAoaanHS82daBbDXsWHjSw4YyJxCsWAPzL2tGnjj2jmjqisZfRamzBPXWSsGXSSUII2/bFIQPPR+nw8vKiS5cuXLlyhS1btlDOyUL0cGeqexaHVJFRIWRZAQdTLbRo0YJq1aoRGxvL6dOn6VbDntNZZpJy7WjXrh1+fn5kZmayc+dO7CwG7O2gwKiidevWVKpUib1793L+/HmcHaDACE2aNKFWrVqoVCpOnTpFbGwsnQLUbI4og921Zzqlk0SpkTMoIcQDl12k8MomHV26dOGnn37Czs4OJycnjh8/TuvWrZmyrYAfniwDwEd7DcSkKfzwww/069ePc+fOUaNGDd5++22mT5+OnZ0d0dF7CA4OJj4+ntq1a3Pq1CmaNGmCzmhk69ZNtG3bluTkZKpUqcL48eNZsGABWq2WmJgYsrKycHV1xdHRkSVLlvDCCy+wJcFMWA05PJY2uQclhHjgvjtqJN8Ac+bMwWQy4efnx5AhQ6hfvz5jxozhxxMmMguKL739etZEmzZtGDBgAJ988gk1a9Zkw4YNTJ06lUqVKuHv709wcDA//fQTv4wNZOnSpdStW5e6devSu3dvOnbsyLRp06hevToxMTG8++67uLm5odfrqVy5MuXKlcPf3x+dTmcdFPvEJXmljy2QgPr/9u4+KKp6DwP4s7yJXkHFF3CCYKbx6nAlF8vCVFReJFl2gVHMEmWKtJobFpNZNtjVqXFqrBmVRhT/8I6KzSCBjQplK23YhQm7I6JoLXYFAeXFi7YIusLu9/7BsOnNal08cLDn8w+we5bnnLO759mz5+z+iGjA/XDFhtGjR2Pq1Kk4deoU2tvb8fXXXwMAZs+eDQAw/7e3oH62CsaNGwcA6OjoANA7pI+npydmzJiBxsZGfPtt7x5U67S/Izo6GtXV1Th79qzjdhaLBUDv17T5+PggLCwMdrsdbW1tWLx4MV5++WV4e3tj3759AIDJ47hpVAPuwxLRgGu/8UvpdHV13fGz7/Id33ej/prAXQOYTCY0NzcjMzMT06ZNg06nc0xrt9thMpnw5JNPYsWKFQgMDMTOnTvR3d2N4uJiWCwWbNy4EU8//TSio6MBAOPHjwcAjBo1Clu2bMG4ceNw/fp1mM1mAEA3d6BUgS8TiGjATRzphkuXLgGAYzSEvp99l+ed7kZq0Q2cbrXj6tWreOyxx/Dhhx/ixx9/dAyIajabMWvWLGRlZSE7OxuPT9diQ63ACAAACfRJREFUw4YNyMjIgE6nQ0NDA8LDw5GdnY3q6moUFRU5bgf0jvYdGBiICRMmoLW1FVu3boWfnx++/KlnQNcH3R0LiogGXPhEN3R1dcFoNEKr1WLu3LlIT08HAMc4ctnZ2TCbzfDz8wMAhIaGYteuXSgqKsK8efPQ0NCA48ePo+9E5KCgIAwb/hcEBwcDgOPySZMmYfv27TAajYiMjER1dTVqamowffp0GAwGhISEQKvVwtfXFyICq9UKb55mrgp8i4+IBlxKqCfWHbNi9erVyM/Ph8lkgs1mw969e7Fnzx4AQEBAACZNmgR3d3cAwM6dOx0jGZw8eRIrV66EzWZDRUUFdu3ahfT0dKSkpAAADhw4gJKSEgBAQUGBY1Tu8vJyPP/88wCAhx56CIWFhY7/f+XKFaxatQqdnZ3QTRoxcCuDfhM/B0VEg6KsvgeGT7vws7V3gFKLxYK2tjY86u+G6hY7vLy84Obmhps3bwIAPD09ERISgu7ubtTV1cF3GPDPxOHYVnkLpjobfH194e/vj7a2Nly7ds2R4+3tjYcffhhdXV1obGzE+BEaBPpqcLLZjhEjRiAoKAidnZ1obm5GT08P/jHXCxvmef8yo/wc1KBhQRHRoGm/Idh98ha+v2zDCA8NEv7qAcNkD5xptePA2W5024CpE9wwZZw7in7oxn+u2uHhpkFEoDtSH/XEaG8NbHbBIXMPSi/Y0Nppx9jhGsx+2AOLQz3w78s2FJ7rQf3Pdgxz1yAy2B1Lp3pihCdQdK4H/2qwoanDjuEeGoSM1mDJ3zwROt79zplkQQ0aFhQR0e9hQQ0aniRBRESqxIIiIiJVYkEREZEqsaCIiEiVWFBERKRKLCgiIlIlFhQREakSC4qIiFSJBUVERKr0wH5ZrNjtjk+AExG5rPsm4On9x9PRfffA7kF197g2nktbW9s936a+vn7AspinjrwHedmY939YToPmgS0oIiIa2lhQRESkSiwoIiJSJRYUERGpEguKiIhUiQVFRESqxIIiIiJVYkEREZEqsaCIiEiVNCIigz0TSqiqqsKwYcMGezaI6AFhtVqh1WoHezb+VB7YgiIioqGNb/EREZEqsaCIiEiVWFBERKRKLCgiIlIlFhQREakSC4qIiFRpSBdUWVkZ4uLiEBsbi9zc3F9df+vWLbz++uuIjY1FSkoKGhsbFc07ceIEkpOTERoaii+++KJfWc7k7d69G/Hx8dDr9UhLS0NTU5OieZ9++in0ej0SExPx7LPP4vz584rm9fnyyy8xefJknD59WrGswsJCREREIDExEYmJiThw4IDLWc7kAUBxcTHi4+Oh0+nwxhtvKJq3adMmx7LFxcXh8ccfVzTv0qVLWL58OZKSkqDX6/HNN98oltXU1IS0tDTo9XosX74czc3NLmcBwLp16zBz5kwkJCTc9XoRwfvvv4/Y2Fjo9XrU1NT0K49+hwxRPT09Eh0dLRcvXhSr1Sp6vV5qa2vvmGbfvn2yfv16ERE5fPiwvPbaa4rmNTQ0yLlz5+TNN9+UkpISl7OczauoqJCuri4REcnLy1N8+To6Ohy/G41GeeGFFxTN68t87rnnJCUlRaqrqxXL+uyzz2Tjxo0u/X9X8i5cuCCJiYly7do1ERG5cuWKonm327Nnj7z99tuK5mVlZUleXp6IiNTW1sr8+fMVy8rIyJDCwkIRESkvL5c1a9a4lNWnsrJSzpw5Izqd7q7Xm0wmSU9PF7vdLidPnpTFixf3K49+25Ddg6qurkZwcDCCgoLg5eUFnU6HY8eO3TFNaWkpkpOTAQBxcXGoqKiAuPi5ZGfyAgMDMWXKFLi59X+1OpMXERGB4cOHAwC0Wm2/Xjk6kzdy5EjH7zdu3IBGo1E0DwC2bt2KlStX9utbQZzNul+cycvPz8eyZcswatQoAMDYsWMVzbvdkSNHfnPv4H7laTQaXL9+HQDQ0dGBCRMmKJb1008/ISIiAkDvc6K/9+2MGTMc98vdHDt2DElJSdBoNNBqtbBYLGhtbe1XJt3dkC2olpYWBAQEOP729/dHS0vLr6aZOHEiAMDDwwM+Pj64evWqYnn3073mFRQUIDIyUvG8vLw8xMTEYPPmzcjKylI0r6amBs3NzZg3b57LOc5mAcDRo0eh1+uxevVqXL58WdG8uro6XLhwAUuXLsWSJUtQVlamaF6fpqYmNDY2OjboSuW9+uqrOHToECIjI7Fq1SqXHyvOZE2ZMgVHjx4FAHz11Vfo7Ox0+XnuyjwFBAQoui34MxuyBUW/+Pzzz3HmzBm8+OKLimctW7YMRqMRa9asQU5OjmI5drsdH3zwAd566y3FMm43f/58lJaW4tChQ3jqqacUz7XZbKivr8fevXvx8ccfY/369bBYLIpmAr17T3FxcXB3d1c8Jzk5GWVlZcjNzcXatWtht9sVyVq7di1OnDiBpKQkVFZWwt/fX/Hlo4ExZAvK39//jre0Wlpa4O/v/6tp+l4J9/T0oKOjA2PGjFEs735yNq+8vBw7duxATk4OvLy8FM/ro9PpYDQaFcvr7OyE2WzGihUrEBUVhaqqKrzyyisunSjhzLKNGTPGsf5SUlL6deDb2cdmVFQUPD09ERQUhJCQENTV1SmW16e4uBg6nc6lnHvJKygowMKFCwEA4eHhsFqtLu3VOLsuP/nkExw8eBCZmZkAAF9f33vOcnWempubFd0W/JkN2YIKCwtDXV0dGhoacOvWLRw5cgRRUVF3TBMVFYWioiIAvWeCRUREuHzcxJm8+8mZvLNnz+Ldd99FTk5Ov45hOJt3+wbUZDIhODhYsTwfHx989913KC0tRWlpKbRaLXJychAWFqbIst1+DKG0tBSPPPKIYssGADExMaisrAQAtLe3o66uDkFBQYrlAb3HaiwWC8LDw13KuZe8iRMnoqKiwpFrtVrh5+enSFZ7e7tj7yw3NxeLFi1yccmcExUVhYMHD0JEUFVVBR8fH5ePsdEfGOyzNPrDZDLJggULJDo6WrZv3y4iIlu2bBGj0SgiIjdv3pSMjAyJiYmRRYsWycWLFxXNO3XqlMyZM0emTZsmTzzxhMTHxyual5aWJjNnzhSDwSAGg0FeeuklRfPee+89iY+PF4PBIKmpqWI2mxXNu11qaqrLZ/E5k/XRRx9JfHy86PV6SU1NlfPnz7uc5Uye3W6XTZs2ycKFCyUhIUEOHz6saJ6IyLZt22Tz5s39ynE2r7a2Vp555hnR6/ViMBjk+PHjimWVlJRIbGysLFiwQN555x2xWq39WrbMzEyZNWuWhIaGypw5cyQ/P1/2798v+/fvF5He+27Dhg0SHR0tCQkJ/Xpc0u/jcBtERKRKQ/YtPiIierCxoIiISJVYUEREpEosKCIiUiUWFBERqRILioiIVIkFRUREqvQ/iOUlr4XY7mYAAAAASUVORK5CYII=\\n\",\n \"text/plain\": [\n \"
\"\n ]\n },\n \"metadata\": {},\n \"output_type\": \"display_data\"\n }\n ],\n \"source\": [\n \"# Plot learning curves for the different models\\n\",\n \"fig = plt.figure(figsize=(10,6))\\n\",\n \"sns.set_style(style='dark')\\n\",\n \"ax = sns.lineplot(x='epoch', y='loss',\\n\",\n \" style='type',\\n\",\n \" hue='model',\\n\",\n \" data=learning_curves)\\n\",\n \"ax.set_title('Learning Curves', fontdict={'fontsize': 16})\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 9,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"fig.savefig('./visualizations/custom_learning_curve.png')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": []\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3 (ipykernel)\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.9.7\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 4\n}\n"
},
{
"path": "examples/titanic/multiple_model_training.py",
"content": "#!/usr/bin/env python\n\n# # Multiple Model Training Example\n#\n# This example trains multiple models and extracts training statistics\n\nimport logging\nimport shutil\n\n# ## Import required libraries\nfrom ludwig.api import LudwigModel\nfrom ludwig.datasets import titanic\nfrom ludwig.visualize import learning_curves\n\n# clean out old results\nshutil.rmtree(\"./results\", ignore_errors=True)\nshutil.rmtree(\"./visualizations\", ignore_errors=True)\n\n# list models to train\nlist_of_model_ids = [\"model1\", \"model2\"]\nlist_of_train_stats = []\n\ntraining_set, _, _ = titanic.load(split=True)\n\n# ## Train models\nfor model_id in list_of_model_ids:\n print(\">>>> training: \", model_id)\n\n # Define Ludwig model object that drive model training\n model = LudwigModel(config=\"./\" + model_id + \"_config.yaml\", logging_level=logging.WARN)\n\n # initiate model training\n train_stats, _, _ = model.train(\n dataset=training_set, experiment_name=\"multiple_model_experiment\", model_name=model_id\n )\n\n # save training stats for later use\n list_of_train_stats.append(train_stats)\n\n print(\">>>>>>> completed: \", model_id, \"\\n\")\n\n# generating learning curves from training\nlearning_curves(\n list_of_train_stats,\n \"Survived\",\n model_names=list_of_model_ids,\n output_directory=\"./visualizations\",\n file_format=\"png\",\n)\n"
},
{
"path": "examples/titanic/simple_model_training.py",
"content": "#!/usr/bin/env python\n\n# # Simple Model Training Example\n#\n# This example is the API example for this Ludwig command line example\n# (https://ludwig-ai.github.io/ludwig-docs/latest/examples/titanic/).\n\n# Import required libraries\nimport logging\nimport os\nimport shutil\n\nimport yaml\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.datasets import titanic\n\n# clean out prior results\nshutil.rmtree(\"./results\", ignore_errors=True)\n\n# Download and prepare the dataset\ntraining_set, test_set, _ = titanic.load(split=True)\n\nconfig = yaml.safe_load(\"\"\"\ninput_features:\n - name: Pclass\n type: category\n - name: Sex\n type: category\n - name: Age\n type: number\n preprocessing:\n missing_value_strategy: fill_with_mean\n - name: SibSp\n type: number\n - name: Parch\n type: number\n - name: Fare\n type: number\n preprocessing:\n missing_value_strategy: fill_with_mean\n - name: Embarked\n type: category\n\noutput_features:\n - name: Survived\n type: binary\n\n\"\"\")\n\n# Define Ludwig model object that drive model training\nmodel = LudwigModel(config=config, logging_level=logging.INFO)\n\n# initiate model training\n(\n train_stats, # dictionary containing training statistics\n preprocessed_data, # tuple Ludwig Dataset objects of pre-processed training data\n output_directory, # location of training results stored on disk\n) = model.train(\n dataset=training_set, experiment_name=\"simple_experiment\", model_name=\"simple_model\", skip_save_processed_input=True\n)\n\n# list contents of output directory\nprint(\"contents of output directory:\", output_directory)\nfor item in os.listdir(output_directory):\n print(\"\\t\", item)\n\n# batch prediction\nmodel.predict(test_set, skip_save_predictions=False)\n"
},
{
"path": "examples/twitter_bots/README.md",
"content": "# Twitter Bots Example\n\nWe'll be using the twitter human-bots dataset which is composed of 37438 rows each corresponding to a Twitter user\naccount. Each row contains 20 feature columns collected via the Twitter API. These features contain multiple data\nmodalities, including the account description and the profile image.\n\nThe target column account_type has two unique values: bot or human. 25013 user accounts were annotated as human\naccounts, the remaining 12425 are bots.\n\n### Preparatory Steps\n\nCreate and download your [Kaggle API Credentials](https://github.com/Kaggle/kaggle-api#api-credentials).\n\nThe Twitter Bots dataset is hosted by Kaggle, Ludwig will need to authenticate you through the Kaggle API to download\nthe dataset.\n\n### Examples\n\nRun `python train_twitter_bots.py` to train a single model.\n\nFor a faster, more lightweight model run `python train_twitter_bots_text_only.py`, which does not use image features.\n\nThis will download the Twitter Bots dataset into the current\ndirectory, train a model, and write results into the following directories:\n\n```\n./outputs/results/\n api_experiment_run/\n./outputs/visualizations/\n confusion_matrix__account_type_top2.png\n confusion_matrix_entropy__account_type_top2.png\n learning_curves_account_type_accuracy.png\n learning_curves_account_type_loss.png\n```\n\nAfter training, the script will generate the following plots:\n\n\n\n\n"
},
{
"path": "examples/twitter_bots/train_twitter_bots.py",
"content": "#!/usr/bin/env python\n\"\"\"Trains model on Twitter Bots dataset using default settings.\"\"\"\n\nimport logging\nimport os\nimport shutil\n\nimport yaml\n\nfrom ludwig import datasets\nfrom ludwig.api import LudwigModel\nfrom ludwig.utils.fs_utils import rename\nfrom ludwig.visualize import confusion_matrix, learning_curves\n\nif __name__ == \"__main__\":\n # Cleans out prior results\n results_dir = os.path.join(\"outputs\", \"results\")\n visualizations_dir = os.path.join(\"outputs\", \"visualizations\")\n shutil.rmtree(results_dir, ignore_errors=True)\n shutil.rmtree(visualizations_dir, ignore_errors=True)\n\n # Loads the dataset\n twitter_bots_dataset = datasets.get_dataset(\"twitter_bots\", cache_dir=\"downloads\")\n training_set, val_set, test_set = twitter_bots_dataset.load(split=True)\n\n # Moves profile images into local directory, so relative paths in the dataset will be resolved.\n if not os.path.exists(\"profile_images\"):\n rename(os.path.join(twitter_bots_dataset.processed_dataset_dir, \"profile_images\"), \"profile_images\")\n\n config = yaml.safe_load(\"\"\"\n input_features:\n - name: default_profile\n type: binary\n - name: default_profile_image\n type: binary\n - name: description\n type: text\n - name: favourites_count\n type: number\n - name: followers_count\n type: number\n - name: friends_count\n type: number\n - name: geo_enabled\n type: binary\n - name: lang\n type: category\n - name: location\n type: category\n - name: profile_background_image_path\n type: category\n - name: profile_image_path\n type: image\n preprocessing:\n num_channels: 3\n - name: statuses_count\n type: number\n - name: verified\n type: binary\n - name: average_tweets_per_day\n type: number\n - name: account_age_days\n type: number\n output_features:\n - name: account_type\n type: binary\n \"\"\")\n\n model = LudwigModel(config, logging_level=logging.INFO)\n\n train_stats, preprocessed_data, output_directory = model.train(dataset=training_set, output_directory=results_dir)\n\n # Generates predictions and performance statistics for the test set.\n test_stats, predictions, output_directory = model.evaluate(\n test_set, collect_predictions=True, collect_overall_stats=True, output_directory=results_dir\n )\n\n confusion_matrix(\n [test_stats],\n model.training_set_metadata,\n \"account_type\",\n top_n_classes=[2],\n model_names=[\"\"],\n normalize=True,\n output_directory=visualizations_dir,\n file_format=\"png\",\n )\n\n # Visualizes learning curves, which show how performance metrics changed over time during training.\n learning_curves(\n train_stats, output_feature_name=\"account_type\", output_directory=visualizations_dir, file_format=\"png\"\n )\n"
},
{
"path": "examples/twitter_bots/train_twitter_bots_text_only.py",
"content": "#!/usr/bin/env python\n\"\"\"Trains twitter bots using tabular and text features only, no images.\"\"\"\n\nimport logging\nimport os\nimport shutil\n\nimport yaml\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.datasets import twitter_bots\nfrom ludwig.visualize import confusion_matrix, learning_curves\n\nif __name__ == \"__main__\":\n # Cleans out prior results\n results_dir = os.path.join(\"outputs\", \"results\")\n visualizations_dir = os.path.join(\"outputs\", \"visualizations\")\n shutil.rmtree(results_dir, ignore_errors=True)\n shutil.rmtree(visualizations_dir, ignore_errors=True)\n\n # Loads the dataset\n training_set, val_set, test_set = twitter_bots.load(split=True)\n\n config = yaml.safe_load(\"\"\"\ninput_features:\n - name: created_at\n type: date\n column: created_at\n - name: default_profile\n type: binary\n column: default_profile\n - name: description\n type: text\n column: description\n - name: favourites_count\n type: number\n column: favourites_count\n - name: followers_count\n type: number\n column: followers_count\n - name: friends_count\n type: number\n column: friends_count\n - name: geo_enabled\n type: binary\n column: geo_enabled\n - name: lang\n type: category\n column: lang\n - name: location\n type: text\n column: location\n - name: screen_name\n type: text\n column: screen_name\n - name: statuses_count\n type: number\n column: statuses_count\n - name: verified\n type: binary\n column: verified\n - name: average_tweets_per_day\n type: number\n column: average_tweets_per_day\n - name: account_age_days\n type: number\n column: account_age_days\noutput_features:\n - name: account_type\n type: category\n column: account_type\ntrainer:\n batch_size: 16\ndefaults:\n text:\n preprocessing:\n tokenizer: space_punct\n max_sequence_length: 16\nmodel_type: ecd\n \"\"\")\n\n model = LudwigModel(config, logging_level=logging.INFO)\n\n train_stats, preprocessed_data, output_directory = model.train(dataset=training_set, output_directory=results_dir)\n\n # Generates predictions and performance statistics for the test set.\n test_stats, predictions, output_directory = model.evaluate(\n test_set, collect_predictions=True, collect_overall_stats=True, output_directory=results_dir\n )\n\n confusion_matrix(\n [test_stats],\n model.training_set_metadata,\n \"account_type\",\n top_n_classes=[2],\n model_names=[\"\"],\n normalize=True,\n output_directory=visualizations_dir,\n file_format=\"png\",\n )\n\n # Visualizes learning curves, which show how performance metrics changed over time during training.\n learning_curves(\n train_stats, output_feature_name=\"account_type\", output_directory=visualizations_dir, file_format=\"png\"\n )\n"
},
{
"path": "examples/wine_quality/README.md",
"content": "# Ludwig Defaults Config Section Example\n\nDemonstrates how to use Ludwig's defaults section introduced in v0.6.\n\n### Preparatory Steps\n\n- Create `data` directory\n- Download [Kaggle wine quality data set](https://www.kaggle.com/rajyellow46/wine-quality) into the `data` directory. Directory should\n appear as follows:\n\n```\nwine_quality/\n data/\n winequalityN.csv\n```\n\n### Description\n\nJupyter notebook `model_defaults_example.ipynb` demonstrates how to use the defaults section of Ludwig.\nKey features demonstrated in the notebook:\n\n- Training data is prepared for use\n- Programmatically create Ludwig config dictionary from the training data dataframe\n- How to define preprocessing, encoder, decoder and loss sub-sections under the defaults section\n"
},
{
"path": "examples/wine_quality/model_defaults_example.ipynb",
"content": "{\n \"cells\": [\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"import pandas as pd \\n\",\n \"import numpy as np\\n\",\n \"\\n\",\n \"import os\\n\",\n \"\\n\",\n \"import shutil\\n\",\n \"from pprint import pprint\\n\",\n \"import logging\\n\",\n \"\\n\",\n \"from ludwig.api import LudwigModel\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Receive data for training\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"train_df = pd.read_csv('./data/winequalityN.csv')\\n\",\n \"train_df['quality'] = train_df['quality'].apply(str)\\n\",\n \"train_df.shape\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# Replace white space in column names with underscore\\n\",\n \"new_col = []\\n\",\n \"for i in range(len(train_df.columns)):\\n\",\n \" new_col.append(train_df.columns[i].replace(' ', '_'))\\n\",\n \" \\n\",\n \"train_df.columns = new_col\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"train_df.head()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"train_df.describe().T\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"train_df.dtypes\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"train_df['quality'].value_counts().sort_index()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"cols = list(set(train_df.columns) - set(['quality']))\\n\",\n \"features = train_df[cols]\\n\",\n \"\\n\",\n \"#extract categorical features\\n\",\n \"categorical_features = []\\n\",\n \"for p in features:\\n\",\n \" if train_df[p].dtype == 'object':\\n\",\n \" categorical_features.append(p)\\n\",\n \" \\n\",\n \"print(\\\"categorical features:\\\", categorical_features, '\\\\n')\\n\",\n \"\\n\",\n \"# get numerical features\\n\",\n \"numerical_features = list(set(features) - set(categorical_features))\\n\",\n \"\\n\",\n \"print(\\\"numerical features:\\\", numerical_features, \\\"\\\\n\\\")\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"for feature in categorical_features:\\n\",\n \" print(f\\\"# of distinct values in categorical feature '{feature}' : {train_df[feature].nunique()}\\\")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Create Ludwig Config\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"# template for config\\n\",\n \"config = {'input_features':[], 'output_features': [], 'trainer':{}}\\n\",\n \"\\n\",\n \"# setup input features for categorical features\\n\",\n \"for p in categorical_features:\\n\",\n \" a_feature = {\\n\",\n \" 'name': p.replace(' ','_'), \\n\",\n \" 'type': 'category'\\n\",\n \" }\\n\",\n \" config['input_features'].append(a_feature)\\n\",\n \"\\n\",\n \"# setup input features for numerical features\\n\",\n \"for p in numerical_features:\\n\",\n \" a_feature = {\\n\",\n \" 'name': p.replace(' ', '_'), \\n\",\n \" 'type': 'number'\\n\",\n \" }\\n\",\n \" config['input_features'].append(a_feature)\\n\",\n \"\\n\",\n \"# set up output variable\\n\",\n \"config['output_features'].append({'name': 'quality', 'type':'category'})\\n\",\n \"\\n\",\n \"# set default preprocessing and encoder for numerical features\\n\",\n \"config['defaults'] = {\\n\",\n \" 'number': {\\n\",\n \" 'preprocessing': {\\n\",\n \" 'missing_value_strategy': 'fill_with_mean', \\n\",\n \" 'normalization': 'zscore'\\n\",\n \" },\\n\",\n \" 'encoder': {\\n\",\n \" 'type': 'dense',\\n\",\n \" 'num_layers': 2\\n\",\n \" },\\n\",\n \" },\\n\",\n \" 'category': {\\n\",\n \" 'encoder': {\\n\",\n \" 'type': 'sparse'\\n\",\n \" },\\n\",\n \" 'decoder': {\\n\",\n \" 'top_k': 2\\n\",\n \" },\\n\",\n \" 'loss': {\\n\",\n \" 'confidence_penalty': 0.1 \\n\",\n \" }\\n\",\n \" }\\n\",\n \"}\\n\",\n \"\\n\",\n \"# set up trainer\\n\",\n \"config['trainer'] = {'epochs': 5}\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pprint(config, indent=2)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Initialize and Train LudwigModel\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"model = LudwigModel(config, backend = 'local', logging_level = logging.INFO)\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### Inspecting Config After Model Initialization\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pprint(model.config['input_features'], indent=2)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"pprint(model.config['output_features'], indent=2)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"eval_stats, train_stats, _, _ = model.experiment(\\n\",\n \" dataset = train_df,\\n\",\n \" experiment_name = 'wine_quality'\\n\",\n \")\"\n ]\n },\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Cleanup\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"try:\\n\",\n \" shutil.rmtree('./results')\\n\",\n \" items = os.listdir('./')\\n\",\n \" for item in items:\\n\",\n \" if item.endswith(\\\".hdf5\\\") or item.endswith(\\\".json\\\") or item == '.lock_preprocessing':\\n\",\n \" os.remove(os.path.join('./', item))\\n\",\n \"except Exception as e:\\n\",\n \" pass \"\n ]\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3.8.13 64-bit\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.8.13\"\n },\n \"orig_nbformat\": 4,\n \"vscode\": {\n \"interpreter\": {\n \"hash\": \"949777d72b0d2535278d3dc13498b2535136f6dfe0678499012e853ee9abcab1\"\n }\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 2\n}\n"
},
{
"path": "examples/wmt15/config_large.yaml",
"content": "input_features:\n - name: en\n type: text\n encoder: bert\n pretrained_model_name_or_path: bert-base-uncased\n\noutput_features:\n - name: fr\n type: text\n tokenizer: french_tokenize\n"
},
{
"path": "examples/wmt15/config_small.yaml",
"content": "input_features:\n - name: en\n type: text\n encoder: embed\n\noutput_features:\n - name: fr\n type: text\n"
},
{
"path": "examples/wmt15/train_nmt.py",
"content": "\"\"\"Sample ludwig training code for training an NMT model (en -> fr) on WMT15 (https://www.statmt.org/wmt15/).\n\nThe dataset is rather large (8GB), which can take several minutes to preprocess.\n\"\"\"\n\nimport logging\nimport shutil\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.datasets import wmt15\n\n# clean out prior results\nshutil.rmtree(\"./results\", ignore_errors=True)\n\n# Download and prepare the dataset\ntraining_set = wmt15.load()\n\nmodel = LudwigModel(config=\"./config_small.yaml\", logging_level=logging.INFO)\n\n(\n train_stats, # dictionary containing training statistics\n preprocessed_data, # tuple Ludwig Dataset objects of pre-processed training data\n output_directory, # location of training results stored on disk\n) = model.train(dataset=training_set, experiment_name=\"simple_experiment\", model_name=\"simple_model\")\n"
},
{
"path": "ludwig/__init__.py",
"content": "# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nimport sys\n\nfrom ludwig.globals import LUDWIG_VERSION as __version__ # noqa\n\nlogging.basicConfig(level=logging.INFO, stream=sys.stdout, format=\"%(message)s\")\n\n# Disable annoying message about NUMEXPR_MAX_THREADS\nlogging.getLogger(\"numexpr\").setLevel(logging.WARNING)\n"
},
{
"path": "ludwig/accounting/__init__.py",
"content": ""
},
{
"path": "ludwig/accounting/used_tokens.py",
"content": "import torch\n\n\ndef get_used_tokens_for_ecd(inputs: dict[str, torch.Tensor], targets: dict[str, torch.Tensor]) -> int:\n \"\"\"Returns the number of used tokens for an ECD model.\n\n The number of used tokens is the total size of the input and output tensors, which corresponds to 1 token for\n binary, category, and number features, and variable number of tokens for text features, for each example in the\n batch.\n\n Args:\n inputs: The input tensors for one forward pass through ecd.\n targets: The target tensors for one forward pass through ecd.\n \"\"\"\n used_tokens = 0\n for input_feature_tensor in inputs.values():\n used_tokens += torch.flatten(input_feature_tensor).shape[0]\n if targets is not None:\n # targets may be None for evaluation.\n for output_feature_tensor in targets.values():\n used_tokens += torch.flatten(output_feature_tensor).shape[0]\n return used_tokens\n\n\ndef get_used_tokens_for_llm(model_inputs: torch.Tensor, tokenizer) -> int:\n \"\"\"Returns the number of used tokens for an LLM model.\n\n Args:\n model_inputs: torch.Tensor with the merged input and target IDs.\n tokenizer: The tokenizer used to encode the inputs.\n\n Returns:\n The total number of non-pad tokens, for all examples in the batch.\n \"\"\"\n return torch.sum(model_inputs != tokenizer.pad_token_id).item()\n"
},
{
"path": "ludwig/api.py",
"content": "# !/usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\"\"\"\nFile name: LudwigModel.py\nAuthor: Piero Molino\nDate created: 5/21/2019\nPython Version: 3+\n\"\"\"\n\nimport copy\nimport dataclasses\nimport logging\nimport os\nimport sys\nimport tempfile\nimport time\nimport traceback\nfrom collections import OrderedDict\nfrom dataclasses import dataclass\nfrom pprint import pformat\nfrom typing import Any, ClassVar\n\nimport numpy as np\nimport pandas as pd\nimport torch\nfrom tabulate import tabulate\n\nfrom ludwig.api_annotations import PublicAPI\nfrom ludwig.backend import Backend, initialize_backend, provision_preprocessing_workers\nfrom ludwig.callbacks import Callback\nfrom ludwig.constants import (\n AUTO,\n BATCH_SIZE,\n EVAL_BATCH_SIZE,\n FALLBACK_BATCH_SIZE,\n FULL,\n HYPEROPT,\n HYPEROPT_WARNING,\n MIN_DATASET_SPLIT_ROWS,\n MODEL_ECD,\n MODEL_LLM,\n TEST,\n TIMESERIES,\n TRAINING,\n VALIDATION,\n)\nfrom ludwig.data.cache.types import CacheableDataset\nfrom ludwig.data.dataset.base import Dataset\nfrom ludwig.data.postprocessing import convert_predictions, postprocess\nfrom ludwig.data.preprocessing import load_metadata, preprocess_for_prediction, preprocess_for_training\nfrom ludwig.datasets import load_dataset_uris\nfrom ludwig.features.feature_registries import update_config_with_metadata, update_config_with_model\nfrom ludwig.globals import (\n LUDWIG_VERSION,\n MODEL_FILE_NAME,\n MODEL_HYPERPARAMETERS_FILE_NAME,\n MODEL_WEIGHTS_FILE_NAME,\n set_disable_progressbar,\n TRAIN_SET_METADATA_FILE_NAME,\n TRAINING_CHECKPOINTS_DIR_PATH,\n)\nfrom ludwig.models.base import BaseModel\nfrom ludwig.models.calibrator import Calibrator\nfrom ludwig.models.inference import InferenceModule, save_ludwig_model_for_inference\nfrom ludwig.models.predictor import (\n calculate_overall_stats,\n print_evaluation_stats,\n save_evaluation_stats,\n save_prediction_outputs,\n)\nfrom ludwig.models.registry import model_type_registry\nfrom ludwig.schema.model_config import ModelConfig\nfrom ludwig.types import ModelConfigDict, TrainingSetMetadataDict\nfrom ludwig.upload import get_upload_registry\nfrom ludwig.utils import metric_utils\nfrom ludwig.utils.backward_compatibility import upgrade_config_dict_to_latest_version\nfrom ludwig.utils.config_utils import get_preprocessing_params\nfrom ludwig.utils.data_utils import (\n figure_data_format,\n generate_kfold_splits,\n load_dataset,\n load_json,\n load_yaml,\n save_json,\n)\nfrom ludwig.utils.dataset_utils import generate_dataset_statistics\nfrom ludwig.utils.defaults import default_random_seed\nfrom ludwig.utils.fs_utils import makedirs, path_exists, upload_output_directory\nfrom ludwig.utils.heuristics import get_auto_learning_rate\nfrom ludwig.utils.llm_utils import create_text_streamer, TextStreamer\nfrom ludwig.utils.misc_utils import (\n get_commit_hash,\n get_file_names,\n get_from_registry,\n get_output_directory,\n set_saved_weights_in_checkpoint_flag,\n)\nfrom ludwig.utils.print_utils import print_boxed\nfrom ludwig.utils.tokenizers import HFTokenizer\nfrom ludwig.utils.torch_utils import DEVICE\nfrom ludwig.utils.trainer_utils import get_training_report\nfrom ludwig.utils.types import DataFrame, TorchDevice\nfrom ludwig.utils.upload_utils import HuggingFaceHub\n\nlogger = logging.getLogger(__name__)\n\n\n@PublicAPI\n@dataclass\nclass EvaluationFrequency: # noqa F821\n \"\"\"Represents the frequency of periodic evaluation of a metric during training. For example:\n\n \"every epoch\"\n frequency: 1, period: EPOCH\n\n \"every 50 steps\".\n frequency: 50, period: STEP\n \"\"\"\n\n frequency: float = 1.0\n period: str = \"epoch\" # One of \"epoch\" or \"step\".\n\n EPOCH: ClassVar[str] = \"epoch\" # One epoch is a single pass through the training set.\n STEP: ClassVar[str] = \"step\" # One step is training on one mini-batch.\n\n\n@PublicAPI\n@dataclass\nclass TrainingStats: # noqa F821\n \"\"\"Training stats were previously represented as a tuple or a dict.\n\n This class replaces those while preserving dict and tuple-like behavior (unpacking, [] access).\n \"\"\"\n\n training: dict[str, Any]\n validation: dict[str, Any]\n test: dict[str, Any]\n evaluation_frequency: EvaluationFrequency = dataclasses.field(default_factory=EvaluationFrequency)\n\n # TODO(daniel): deprecate multiple return value unpacking and dictionary-style element access\n def __iter__(self):\n return iter((self.training, self.test, self.validation))\n\n def __contains__(self, key):\n return (\n (key == TRAINING and self.training)\n or (key == VALIDATION and self.validation)\n or (key == TEST and self.test)\n )\n\n def __getitem__(self, key):\n # Supports dict-style [] element access for compatibility.\n return {TRAINING: self.training, VALIDATION: self.validation, TEST: self.test}[key]\n\n\n@PublicAPI\n@dataclass\nclass PreprocessedDataset: # noqa F821\n training_set: Dataset\n validation_set: Dataset\n test_set: Dataset\n training_set_metadata: TrainingSetMetadataDict\n\n # TODO(daniel): deprecate multiple return value unpacking and indexed access\n def __iter__(self):\n return iter((self.training_set, self.validation_set, self.test_set, self.training_set_metadata))\n\n def __getitem__(self, index):\n return (self.training_set, self.validation_set, self.test_set, self.training_set_metadata)[index]\n\n\n@PublicAPI\n@dataclass\nclass TrainingResults: # noqa F821\n train_stats: TrainingStats\n preprocessed_data: PreprocessedDataset\n output_directory: str\n\n def __iter__(self):\n \"\"\"Supports tuple-style return value unpacking ex.\n\n train_stats, training_set, output_dir = model.train(...)\n \"\"\"\n return iter((self.train_stats, self.preprocessed_data, self.output_directory))\n\n def __getitem__(self, index):\n \"\"\"Provides indexed getter ex.\n\n train_stats = model.train(...)[0]\n \"\"\"\n return (self.train_stats, self.preprocessed_data, self.output_directory)[index]\n\n\n@PublicAPI\nclass LudwigModel:\n \"\"\"Class that allows access to high level Ludwig functionalities.\n\n # Inputs\n\n :param config: (Union[str, dict]) in-memory representation of\n config or string path to a YAML config file.\n :param logging_level: (int) Log level that will be sent to stderr.\n :param backend: (Union[Backend, str]) `Backend` or string name\n of backend to use to execute preprocessing / training steps.\n :param gpus: (Union[str, int, List[int]], default: `None`) GPUs\n to use (it uses the same syntax of CUDA_VISIBLE_DEVICES)\n :param gpu_memory_limit: (float: default: `None`) maximum memory fraction\n [0, 1] allowed to allocate per GPU device.\n :param allow_parallel_threads: (bool, default: `True`) allow Torch\n to use multithreading parallelism to improve performance at the\n cost of determinism.\n\n # Example usage:\n\n ```python\n from ludwig.api import LudwigModel\n ```\n\n Train a model:\n\n ```python\n config = {...}\n ludwig_model = LudwigModel(config)\n train_stats, _, _ = ludwig_model.train(dataset=file_path)\n ```\n\n or\n\n ```python\n train_stats, _, _ = ludwig_model.train(dataset=dataframe)\n ```\n\n If you have already trained a model you can load it and use it to predict\n\n ```python\n ludwig_model = LudwigModel.load(model_dir)\n ```\n\n Predict:\n\n ```python\n predictions, _ = ludwig_model.predict(dataset=file_path)\n ```\n\n or\n\n ```python\n predictions, _ = ludwig_model.predict(dataset=dataframe)\n ```\n\n Evaluation:\n\n ```python\n eval_stats, _, _ = ludwig_model.evaluate(dataset=file_path)\n ```\n\n or\n\n ```python\n eval_stats, _, _ = ludwig_model.evaluate(dataset=dataframe)\n ```\n \"\"\"\n\n def __init__(\n self,\n config: str | dict,\n logging_level: int = logging.ERROR,\n backend: Backend | str | None = None,\n gpus: str | int | list[int] | None = None,\n gpu_memory_limit: float | None = None,\n allow_parallel_threads: bool = True,\n callbacks: list[Callback] | None = None,\n ) -> None:\n \"\"\"Constructor for the Ludwig Model class.\n\n # Inputs\n\n :param config: (Union[str, dict]) in-memory representation of\n config or string path to a YAML config file.\n :param logging_level: (int) Log level that will be sent to stderr.\n :param backend: (Union[Backend, str]) `Backend` or string name\n of backend to use to execute preprocessing / training steps.\n :param gpus: (Union[str, int, List[int]], default: `None`) GPUs\n to use (it uses the same syntax of CUDA_VISIBLE_DEVICES)\n :param gpu_memory_limit: (float: default: `None`) maximum memory fraction\n [0, 1] allowed to allocate per GPU device.\n :param allow_parallel_threads: (bool, default: `True`) allow Torch\n to use multithreading parallelism to improve performance at the\n cost of determinism.\n :param callbacks: (list, default: `None`) a list of\n `ludwig.callbacks.Callback` objects that provide hooks into the\n Ludwig pipeline.\n\n # Return\n\n :return: (None) `None`\n \"\"\"\n # check if config is a path or a dict\n if isinstance(config, str): # assume path\n config_dict = load_yaml(config)\n self.config_fp = config\n else:\n config_dict = copy.deepcopy(config)\n self.config_fp = None # type: ignore [assignment]\n\n self._user_config = upgrade_config_dict_to_latest_version(config_dict)\n\n # Initialize the config object\n self.config_obj = ModelConfig.from_dict(self._user_config)\n\n # setup logging\n self.set_logging_level(logging_level)\n\n # setup Backend\n self.backend = initialize_backend(backend or self._user_config.get(\"backend\"))\n self.callbacks = callbacks if callbacks is not None else []\n\n # setup PyTorch env (GPU allocation, etc.)\n self.backend.initialize_pytorch(\n gpus=gpus, gpu_memory_limit=gpu_memory_limit, allow_parallel_threads=allow_parallel_threads\n )\n\n # setup model\n self.model = None\n self.training_set_metadata: dict[str, dict] | None = None\n\n # online training state\n self._online_trainer = None\n\n # Zero-shot LLM usage.\n if (\n self.config_obj.model_type == MODEL_LLM\n and self.config_obj.trainer.type == \"none\"\n # Category output features require a vocabulary. The LLM LudwigModel should be initialized with\n # model.train(dataset).\n and self.config_obj.output_features[0].type == \"text\"\n ):\n self._initialize_llm()\n\n def _initialize_llm(self, random_seed: int = default_random_seed):\n \"\"\"Initialize the LLM model.\n\n Should only be used in a zero-shot (NoneTrainer) setting.\n \"\"\"\n self.model = LudwigModel.create_model(self.config_obj, random_seed=random_seed)\n\n if self.model.model.device.type == \"cpu\" and torch.cuda.is_available():\n logger.warning(f\"LLM was initialized on {self.model.model.device}. Moving to GPU for inference.\")\n self.model.model.to(torch.device(\"cuda\"))\n\n def train(\n self,\n dataset: str | dict | pd.DataFrame | None = None,\n training_set: str | dict | pd.DataFrame | Dataset | None = None,\n validation_set: str | dict | pd.DataFrame | Dataset | None = None,\n test_set: str | dict | pd.DataFrame | Dataset | None = None,\n training_set_metadata: str | dict | None = None,\n data_format: str | None = None,\n experiment_name: str = \"api_experiment\",\n model_name: str = \"run\",\n model_resume_path: str | None = None,\n skip_save_training_description: bool = False,\n skip_save_training_statistics: bool = False,\n skip_save_model: bool = False,\n skip_save_progress: bool = False,\n skip_save_log: bool = False,\n skip_save_processed_input: bool = False,\n output_directory: str | None = \"results\",\n random_seed: int = default_random_seed,\n **kwargs,\n ) -> TrainingResults:\n \"\"\"This function is used to perform a full training of the model on the specified dataset.\n\n During training if the skip parameters are False\n the model and statistics will be saved in a directory\n `[output_dir]/[experiment_name]_[model_name]_n` where all variables are\n resolved to user specified ones and `n` is an increasing number\n starting from 0 used to differentiate among repeated runs.\n\n # Inputs\n\n :param dataset: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing the entire dataset to be used in the experiment.\n If it has a split column, it will be used for splitting\n (0 for train, 1 for validation, 2 for test),\n otherwise the dataset will be randomly split.\n :param training_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing training data.\n :param validation_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing validation data.\n :param test_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing test data.\n :param training_set_metadata: (Union[str, dict], default: `None`)\n metadata JSON file or loaded metadata. Intermediate preprocessed\n structure containing the mappings of the input dataset created the\n first time an input file is used in the same directory with the\n same name and a '.meta.json' extension.\n :param data_format: (str, default: `None`) format to interpret data\n sources. Will be inferred automatically if not specified. Valid\n formats are `'auto'`, `'csv'`, `'df'`, `'dict'`, `'excel'`,\n `'feather'`, `'fwf'`,\n `'hdf5'` (cache file produced during previous training),\n `'html'` (file containing a single HTML `
`),\n `'json'`, `'jsonl'`, `'parquet'`,\n `'pickle'` (pickled Pandas DataFrame),\n `'sas'`, `'spss'`, `'stata'`, `'tsv'`.\n :param experiment_name: (str, default: `'experiment'`) name for\n the experiment.\n :param model_name: (str, default: `'run'`) name of the model that is\n being used.\n :param model_resume_path: (str, default: `None`) resumes training of\n the model from the path specified. The config is restored.\n In addition to config, training statistics, loss for each\n epoch and the state of the optimizer are restored such that\n training can be effectively continued from a previously interrupted\n training process.\n :param skip_save_training_description: (bool, default: `False`)\n disables saving the description JSON file.\n :param skip_save_training_statistics: (bool, default: `False`)\n disables saving training statistics JSON file.\n :param skip_save_model: (bool, default: `False`) disables\n saving model weights and hyperparameters each time the model\n improves. By default Ludwig saves model weights after each epoch\n the validation metric improves, but if the model is really big\n that can be time consuming. If you do not want to keep\n the weights and just find out what performance a model can get\n with a set of hyperparameters, use this parameter to skip it,\n but the model will not be loadable later on and the returned model\n will have the weights obtained at the end of training, instead of\n the weights of the epoch with the best validation performance.\n :param skip_save_progress: (bool, default: `False`) disables saving\n progress each epoch. By default Ludwig saves weights and stats\n after each epoch for enabling resuming of training, but if\n the model is really big that can be time consuming and will uses\n twice as much space, use this parameter to skip it, but training\n cannot be resumed later on.\n :param skip_save_log: (bool, default: `False`) disables saving\n TensorBoard logs. By default Ludwig saves logs for the TensorBoard,\n but if it is not needed turning it off can slightly increase the\n overall speed.\n :param skip_save_processed_input: (bool, default: `False`) if input\n dataset is provided it is preprocessed and cached by saving an HDF5\n and JSON files to avoid running the preprocessing again. If this\n parameter is `False`, the HDF5 and JSON file are not saved.\n :param output_directory: (str, default: `'results'`) the directory that\n will contain the training statistics, TensorBoard logs, the saved\n model and the training progress files.\n :param random_seed: (int, default: `42`) a random seed that will be\n used anywhere there is a call to a random number generator: data\n splitting, parameter initialization and training set shuffling\n :param kwargs: (dict, default: {}) a dictionary of optional parameters.\n\n # Return\n\n :return: (Tuple[Dict, Union[Dict, pd.DataFrame], str]) tuple containing\n `(training_statistics, preprocessed_data, output_directory)`.\n `training_statistics` is a nested dictionary of dataset -> feature_name -> metric_name -> List of metrics.\n Each metric corresponds to each training checkpoint.\n `preprocessed_data` is the tuple containing these three data sets\n `(training_set, validation_set, test_set)`.\n `output_directory` filepath to where training results are stored.\n \"\"\"\n # Only reset the metadata if the model has not been trained before\n if self.training_set_metadata:\n logger.warning(\n \"This model has been trained before. Its architecture has been defined by the original training set \"\n \"(for example, the number of possible categorical outputs). The current training data will be mapped \"\n \"to this architecture. If you want to change the architecture of the model, please concatenate your \"\n \"new training data with the original and train a new model from scratch.\"\n )\n training_set_metadata = self.training_set_metadata\n\n if self._user_config.get(HYPEROPT):\n print_boxed(\"WARNING\")\n logger.warning(HYPEROPT_WARNING)\n\n # setup directories and file names\n if model_resume_path is not None:\n if path_exists(model_resume_path):\n output_directory = model_resume_path\n if self.backend.is_coordinator():\n logger.info(f\"Model resume path '{model_resume_path}' exists, trying to resume training.\")\n else:\n if self.backend.is_coordinator():\n logger.info(\n f\"Model resume path '{model_resume_path}' does not exist, starting training from scratch\"\n )\n model_resume_path = None\n\n if model_resume_path is None:\n if self.backend.is_coordinator():\n output_directory = get_output_directory(output_directory, experiment_name, model_name)\n else:\n output_directory = None\n\n # if we are skipping all saving,\n # there is no need to create a directory that will remain empty\n should_create_output_directory = not (\n skip_save_training_description\n and skip_save_training_statistics\n and skip_save_model\n and skip_save_progress\n and skip_save_log\n and skip_save_processed_input\n )\n\n output_url = output_directory\n with upload_output_directory(output_directory) as (output_directory, upload_fn):\n train_callbacks = self.callbacks\n if upload_fn is not None:\n # Upload output files (checkpoints, etc.) to remote storage at the end of\n # each epoch and evaluation, in case of failure in the middle of training.\n class UploadOnEpochEndCallback(Callback):\n def on_eval_end(self, trainer, progress_tracker, save_path):\n upload_fn()\n\n def on_epoch_end(self, trainer, progress_tracker, save_path):\n upload_fn()\n\n train_callbacks = train_callbacks + [UploadOnEpochEndCallback()]\n\n description_fn = training_stats_fn = model_dir = None\n if self.backend.is_coordinator():\n if should_create_output_directory:\n makedirs(output_directory, exist_ok=True)\n description_fn, training_stats_fn, model_dir = get_file_names(output_directory)\n\n if isinstance(training_set, Dataset) and training_set_metadata is not None:\n preprocessed_data = (training_set, validation_set, test_set, training_set_metadata)\n else:\n # save description\n if self.backend.is_coordinator():\n description = get_experiment_description(\n self.config_obj.to_dict(),\n dataset=dataset,\n training_set=training_set,\n validation_set=validation_set,\n test_set=test_set,\n training_set_metadata=training_set_metadata,\n data_format=data_format,\n backend=self.backend,\n random_seed=random_seed,\n )\n\n if not skip_save_training_description:\n save_json(description_fn, description)\n\n # print description\n experiment_description = [\n [\"Experiment name\", experiment_name],\n [\"Model name\", model_name],\n [\"Output directory\", output_directory],\n ]\n for key, value in description.items():\n if key != \"config\": # Config is printed separately.\n experiment_description.append([key, pformat(value, indent=4)])\n\n if self.backend.is_coordinator():\n print_boxed(\"EXPERIMENT DESCRIPTION\")\n logger.info(tabulate(experiment_description, tablefmt=\"fancy_grid\"))\n\n print_boxed(\"LUDWIG CONFIG\")\n logger.info(\"User-specified config (with upgrades):\\n\")\n logger.info(pformat(self._user_config, indent=4))\n logger.info(\n \"\\nFull config saved to:\\n\"\n f\"{output_directory}/{experiment_name}/model/model_hyperparameters.json\"\n )\n\n preprocessed_data = self.preprocess( # type: ignore[assignment]\n dataset=dataset,\n training_set=training_set,\n validation_set=validation_set,\n test_set=test_set,\n training_set_metadata=training_set_metadata,\n data_format=data_format,\n experiment_name=experiment_name,\n model_name=model_name,\n model_resume_path=model_resume_path,\n skip_save_training_description=skip_save_training_description,\n skip_save_training_statistics=skip_save_training_statistics,\n skip_save_model=skip_save_model,\n skip_save_progress=skip_save_progress,\n skip_save_log=skip_save_log,\n skip_save_processed_input=skip_save_processed_input,\n output_directory=output_directory,\n random_seed=random_seed,\n **kwargs,\n )\n training_set, validation_set, test_set, training_set_metadata = preprocessed_data\n\n self.training_set_metadata = training_set_metadata\n\n if self.backend.is_coordinator():\n dataset_statistics = generate_dataset_statistics(training_set, validation_set, test_set)\n\n if not skip_save_model:\n # save train set metadata\n os.makedirs(model_dir, exist_ok=True) # type: ignore[arg-type]\n save_json( # type: ignore[arg-type]\n os.path.join(model_dir, TRAIN_SET_METADATA_FILE_NAME), training_set_metadata\n )\n\n logger.info(\"\\nDataset Statistics\")\n logger.info(tabulate(dataset_statistics, headers=\"firstrow\", tablefmt=\"fancy_grid\"))\n\n for callback in self.callbacks:\n callback.on_train_init(\n base_config=self._user_config,\n experiment_directory=output_directory,\n experiment_name=experiment_name,\n model_name=model_name,\n output_directory=output_directory,\n resume_directory=model_resume_path,\n )\n\n # Build model if not provided\n # if it was provided it means it was already loaded\n if not self.model:\n if self.backend.is_coordinator():\n print_boxed(\"MODEL\")\n # update model config with metadata properties derived from training set\n update_config_with_metadata(self.config_obj, training_set_metadata)\n logger.info(\"Warnings and other logs:\")\n self.model = LudwigModel.create_model(self.config_obj, random_seed=random_seed)\n # update config with properties determined during model instantiation\n update_config_with_model(self.config_obj, self.model)\n set_saved_weights_in_checkpoint_flag(self.config_obj)\n\n # auto tune learning rate\n if hasattr(self.config_obj.trainer, \"learning_rate\") and self.config_obj.trainer.learning_rate == AUTO:\n detected_learning_rate = get_auto_learning_rate(self.config_obj)\n self.config_obj.trainer.learning_rate = detected_learning_rate\n\n with self.backend.create_trainer(\n model=self.model,\n config=self.config_obj.trainer,\n resume=model_resume_path is not None,\n skip_save_model=skip_save_model,\n skip_save_progress=skip_save_progress,\n skip_save_log=skip_save_log,\n callbacks=train_callbacks,\n random_seed=random_seed,\n ) as trainer:\n # auto tune batch size\n self._tune_batch_size(trainer, training_set, random_seed=random_seed)\n\n if (\n self.config_obj.model_type == \"LLM\"\n and trainer.config.type == \"none\"\n and self.config_obj.adapter is not None\n and self.config_obj.adapter.pretrained_adapter_weights is not None\n ):\n trainer.model.initialize_adapter() # Load pre-trained adapter weights for inference only\n\n # train model\n if self.backend.is_coordinator():\n print_boxed(\"TRAINING\")\n if not skip_save_model:\n self.save_config(model_dir)\n\n for callback in self.callbacks:\n callback.on_train_start(\n model=self.model,\n config=self.config_obj.to_dict(),\n config_fp=self.config_fp,\n )\n\n try:\n train_stats = trainer.train(\n training_set,\n validation_set=validation_set,\n test_set=test_set,\n save_path=model_dir,\n )\n self.model, train_trainset_stats, train_valiset_stats, train_testset_stats = train_stats\n\n # Calibrates output feature probabilities on validation set if calibration is enabled.\n # Must be done after training, and before final model parameters are saved.\n if self.backend.is_coordinator():\n calibrator = Calibrator(\n self.model,\n self.backend,\n batch_size=trainer.eval_batch_size,\n )\n if calibrator.calibration_enabled():\n if validation_set is None:\n logger.warning(\n \"Calibration uses validation set, but no validation split specified.\"\n \"Will use training set for calibration.\"\n \"Recommend providing a validation set when using calibration.\"\n )\n calibrator.train_calibration(training_set, TRAINING)\n elif len(validation_set) < MIN_DATASET_SPLIT_ROWS:\n logger.warning(\n f\"Validation set size ({len(validation_set)} rows) is too small for calibration.\"\n \"Will use training set for calibration.\"\n f\"Validation set much have at least {MIN_DATASET_SPLIT_ROWS} rows.\"\n )\n calibrator.train_calibration(training_set, TRAINING)\n else:\n calibrator.train_calibration(validation_set, VALIDATION)\n if not skip_save_model:\n self.model.save(model_dir)\n\n # Evaluation Frequency\n if self.config_obj.model_type == MODEL_ECD and self.config_obj.trainer.steps_per_checkpoint:\n evaluation_frequency = EvaluationFrequency(\n self.config_obj.trainer.steps_per_checkpoint, EvaluationFrequency.STEP\n )\n elif self.config_obj.model_type == MODEL_ECD and self.config_obj.trainer.checkpoints_per_epoch:\n evaluation_frequency = EvaluationFrequency(\n 1.0 / self.config_obj.trainer.checkpoints_per_epoch, EvaluationFrequency.EPOCH\n )\n else:\n evaluation_frequency = EvaluationFrequency(1, EvaluationFrequency.EPOCH)\n\n # Unpack train()'s return.\n # The statistics are all nested dictionaries of TrainerMetrics: feature_name -> metric_name ->\n # List[TrainerMetric], with one entry per training checkpoint, according to steps_per_checkpoint.\n # We reduce the dictionary of TrainerMetrics to a simple list of floats for interfacing with Ray\n # Tune.\n train_stats = TrainingStats(\n metric_utils.reduce_trainer_metrics_dict(train_trainset_stats),\n metric_utils.reduce_trainer_metrics_dict(train_valiset_stats),\n metric_utils.reduce_trainer_metrics_dict(train_testset_stats),\n evaluation_frequency,\n )\n\n # save training statistics\n if self.backend.is_coordinator():\n if not skip_save_training_statistics:\n save_json(training_stats_fn, train_stats)\n\n # results of the model with highest validation test performance\n if (\n self.backend.is_coordinator()\n and validation_set is not None\n and not self.config_obj.trainer.skip_all_evaluation\n ):\n print_boxed(\"TRAINING REPORT\")\n training_report = get_training_report(\n trainer.validation_field,\n trainer.validation_metric,\n test_set is not None,\n train_valiset_stats,\n train_testset_stats,\n )\n logger.info(tabulate(training_report, tablefmt=\"fancy_grid\"))\n logger.info(f\"\\nFinished: {experiment_name}_{model_name}\")\n logger.info(f\"Saved to: {output_directory}\")\n finally:\n for callback in self.callbacks:\n callback.on_train_end(output_directory)\n\n self.training_set_metadata = training_set_metadata\n\n if self.is_merge_and_unload_set():\n # For an LLM model trained with a LoRA adapter, merge first, then save the full model.\n self.model.merge_and_unload(progressbar=self.config_obj.adapter.postprocessor.progressbar)\n\n if self.backend.is_coordinator() and not skip_save_model:\n self.model.save_base_model(model_dir)\n elif self.backend.is_coordinator() and not skip_save_model:\n self.model.save(model_dir)\n\n # Synchronize model weights between workers\n self.backend.sync_model(self.model)\n\n print_boxed(\"FINISHED\")\n return TrainingResults(train_stats, preprocessed_data, output_url)\n\n def train_online(\n self,\n dataset: str | dict | pd.DataFrame,\n training_set_metadata: str | dict | None = None,\n data_format: str = \"auto\",\n random_seed: int = default_random_seed,\n ) -> None:\n \"\"\"Performs one epoch of training of the model on `dataset`.\n\n # Inputs\n\n :param dataset: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing the entire dataset to be used in the experiment.\n If it has a split column, it will be used for splitting (0 for train,\n 1 for validation, 2 for test), otherwise the dataset will be\n randomly split.\n :param training_set_metadata: (Union[str, dict], default: `None`)\n metadata JSON file or loaded metadata. Intermediate preprocessed\n structure containing the mappings of the input\n dataset created the first time an input file is used in the same\n directory with the same name and a '.meta.json' extension.\n :param data_format: (str, default: `None`) format to interpret data\n sources. Will be inferred automatically if not specified. Valid\n formats are `'auto'`, `'csv'`, `'df'`, `'dict'`, `'excel'`, `'feather'`,\n `'fwf'`, `'hdf5'` (cache file produced during previous training),\n `'html'` (file containing a single HTML `
`), `'json'`, `'jsonl'`,\n `'parquet'`, `'pickle'` (pickled Pandas DataFrame), `'sas'`, `'spss'`,\n `'stata'`, `'tsv'`.\n :param random_seed: (int, default: `42`) a random seed that is going to be\n used anywhere there is a call to a random number generator: data\n splitting, parameter initialization and training set shuffling\n\n # Return\n\n :return: (None) `None`\n \"\"\"\n training_set_metadata = training_set_metadata or self.training_set_metadata\n preprocessing_params = get_preprocessing_params(self.config_obj)\n\n with provision_preprocessing_workers(self.backend):\n # TODO (Connor): Refactor to use self.config_obj\n training_dataset, _, _, training_set_metadata = preprocess_for_training(\n self.config_obj.to_dict(),\n training_set=dataset,\n training_set_metadata=training_set_metadata,\n data_format=data_format,\n skip_save_processed_input=True,\n preprocessing_params=preprocessing_params,\n backend=self.backend,\n random_seed=random_seed,\n callbacks=self.callbacks,\n )\n\n if not self.training_set_metadata:\n self.training_set_metadata = training_set_metadata\n\n if not self.model:\n update_config_with_metadata(self.config_obj, training_set_metadata)\n self.model = LudwigModel.create_model(self.config_obj, random_seed=random_seed)\n # update config with properties determined during model instantiation\n update_config_with_model(self.config_obj, self.model)\n set_saved_weights_in_checkpoint_flag(self.config_obj)\n\n if not self._online_trainer:\n self._online_trainer = self.backend.create_trainer(\n config=self.config_obj.trainer, model=self.model, random_seed=random_seed\n )\n\n self._tune_batch_size(self._online_trainer, dataset, random_seed=random_seed)\n\n self.model = self._online_trainer.train_online(training_dataset)\n\n def _tune_batch_size(self, trainer, dataset, random_seed: int = default_random_seed):\n \"\"\"Sets AUTO batch-size-related parameters based on the trainer, backend type, and number of workers.\n\n Batch-size related parameters that are set:\n - trainer.batch_size\n - trainer.eval_batch_size\n - trainer.gradient_accumulation_steps\n - trainer.effective_batch_size\n\n The final batch size selected may be non-deterministic even with a fixed random seed since throughput-based\n heuristics may be affected by resources used by other processes running on the machine.\n \"\"\"\n if not self.config_obj.trainer.can_tune_batch_size():\n # Some model types don't have batch sizes to be tuned\n return\n\n # Render the batch size and gradient accumulation steps prior to batch size tuning. This is needed in the event\n # the effective_batch_size and gradient_accumulation_steps are set explicitly, but batch_size is AUTO. In this\n # case, we can infer the batch_size directly without tuning.\n num_workers = self.backend.num_training_workers\n self.config_obj.trainer.update_batch_size_grad_accum(num_workers)\n\n # TODO (ASN): add support for substitute_with_max parameter\n # TODO(travis): detect train and eval batch sizes separately (enable / disable gradients)\n if self.config_obj.trainer.batch_size == AUTO:\n if self.backend.supports_batch_size_tuning():\n tuned_batch_size = trainer.tune_batch_size(\n self.config_obj.to_dict(), dataset, random_seed=random_seed, tune_for_training=True\n )\n else:\n logger.warning(\n f\"Backend {self.backend.BACKEND_TYPE} does not support batch size tuning, \"\n f\"using fallback training batch size {FALLBACK_BATCH_SIZE}.\"\n )\n tuned_batch_size = FALLBACK_BATCH_SIZE\n\n # TODO(travis): pass these in as args to trainer when we call train,\n # to avoid setting state on possibly remote trainer\n self.config_obj.trainer.batch_size = tuned_batch_size\n\n # Re-render the gradient_accumulation_steps to account for the explicit batch size.\n self.config_obj.trainer.update_batch_size_grad_accum(num_workers)\n\n if self.config_obj.trainer.eval_batch_size in {AUTO, None}:\n if self.backend.supports_batch_size_tuning():\n tuned_batch_size = trainer.tune_batch_size(\n self.config_obj.to_dict(), dataset, random_seed=random_seed, tune_for_training=False\n )\n else:\n logger.warning(\n f\"Backend {self.backend.BACKEND_TYPE} does not support batch size tuning, \"\n f\"using fallback eval batch size {FALLBACK_BATCH_SIZE}.\"\n )\n tuned_batch_size = FALLBACK_BATCH_SIZE\n\n self.config_obj.trainer.eval_batch_size = tuned_batch_size\n\n # Update trainer params separate to config params for backends with stateful trainers\n trainer.batch_size = self.config_obj.trainer.batch_size\n trainer.eval_batch_size = self.config_obj.trainer.eval_batch_size\n trainer.gradient_accumulation_steps = self.config_obj.trainer.gradient_accumulation_steps\n\n def save_dequantized_base_model(self, save_path: str) -> None:\n \"\"\"Upscales quantized weights of a model to fp16 and saves the result in a specified folder.\n\n Args:\n save_path (str): The path to the folder where the upscaled model weights will be saved.\n\n Raises:\n ValueError:\n If the model type is not 'llm' or if quantization is not enabled or the number of bits is not 4 or 8.\n RuntimeError:\n If no GPU is available, as GPU is required for quantized models.\n\n Returns:\n None\n \"\"\"\n if self.config_obj.model_type != MODEL_LLM:\n raise ValueError(\n f\"Model type {self.config_obj.model_type} is not supported by this method. Only `llm` model type is \"\n \"supported.\"\n )\n\n if not self.config_obj.quantization:\n raise ValueError(\n \"Quantization is not enabled in your Ludwig model config. \"\n \"To enable quantization, set `quantization` to `{'bits': 4}` or `{'bits': 8}` in your model config.\"\n )\n\n if self.config_obj.quantization.bits != 4:\n raise ValueError(\n \"This method only works with quantized models with 4 bits. \"\n \"Support for 8-bit quantized models will be added in a future release.\"\n )\n\n if not torch.cuda.is_available():\n raise RuntimeError(\"GPU is required for quantized models but no GPU found.\")\n\n # Create the LLM model class instance with the loaded LLM if it hasn't been initialized yet.\n if not self.model:\n self.model = LudwigModel.create_model(self.config_obj)\n\n self.model.save_dequantized_base_model(save_path)\n\n logger.info(\n \"If you want to upload this model to huggingface.co, run the following Python commands: \\n\"\n \"from ludwig.utils.hf_utils import upload_folder_to_hfhub; \\n\"\n f\"upload_folder_to_hfhub(repo_id='desired/huggingface/repo/name', folder_path='{save_path}')\"\n )\n\n def generate(\n self,\n input_strings: str | list[str],\n generation_config: dict | None = None,\n streaming: bool | None = False,\n ) -> str | list[str]:\n \"\"\"A simple generate() method that directly uses the underlying transformers library to generate text.\n\n Args:\n input_strings (Union[str, List[str]]): Input text or list of texts to generate from.\n generation_config (Optional[dict]): Configuration for text generation.\n streaming (Optional[bool]): If True, enable streaming output.\n\n Returns:\n Union[str, List[str]]: Generated text or list of generated texts.\n \"\"\"\n if self.config_obj.model_type != MODEL_LLM:\n raise ValueError(\n f\"Model type {self.config_obj.model_type} is not supported by this method. Only `llm` model type is \"\n \"supported.\"\n )\n if not torch.cuda.is_available():\n # GPU is generally well-advised for working with LLMs and is required for loading quantized models, see\n # https://github.com/ludwig-ai/ludwig/issues/3695.\n raise ValueError(\"GPU is not available.\")\n\n # TODO(Justin): Decide if it's worth folding padding_side handling into llm.py's tokenizer initialization.\n # For batch inference with models like facebook/opt-350m, if the tokenizer padding side is off, HF prints a\n # warning, e.g.:\n # \"A decoder-only architecture is being used, but right-padding was detected! For correct generation results, \"\n # \"please set `padding_side='left'` when initializing the tokenizer.\n padding_side = \"left\" if not self.model.model.config.is_encoder_decoder else \"right\"\n tokenizer = HFTokenizer(self.config_obj.base_model, padding_side=padding_side)\n\n with self.model.use_generation_config(generation_config):\n start_time = time.time()\n tokenized_inputs = tokenizer.tokenizer(input_strings, return_tensors=\"pt\", padding=True)\n input_ids = tokenized_inputs[\"input_ids\"].to(\"cuda\")\n attention_mask = tokenized_inputs[\"attention_mask\"].to(\"cuda\")\n\n if streaming:\n streamer = create_text_streamer(tokenizer.tokenizer)\n outputs = self._generate_streaming_outputs(input_strings, input_ids, attention_mask, streamer)\n else:\n outputs = self._generate_non_streaming_outputs(input_strings, input_ids, attention_mask)\n\n decoded_outputs = tokenizer.tokenizer.batch_decode(outputs, skip_special_tokens=True)\n logger.info(f\"Finished generating in: {(time.time() - start_time):.2f}s.\")\n\n return decoded_outputs[0] if len(decoded_outputs) == 1 else decoded_outputs\n\n def _generate_streaming_outputs(\n self,\n input_strings: str | list[str],\n input_ids: torch.Tensor,\n attention_mask: torch.Tensor,\n streamer: TextStreamer,\n ) -> torch.Tensor:\n \"\"\"Generate streaming outputs for the given input.\n\n Args:\n input_strings (Union[str, List[str]]): Input text or list of texts to generate from.\n input_ids (torch.Tensor): Tensor containing input IDs.\n attention_mask (torch.Tensor): Tensor containing attention masks.\n streamer (Union[TextStreamer, None]): Text streamer instance for streaming output.\n\n Returns:\n torch.Tensor: Concatenated tensor of generated outputs.\n \"\"\"\n outputs = []\n input_strings = input_strings if isinstance(input_strings, list) else [input_strings]\n for i in range(len(input_ids)):\n with torch.no_grad():\n logger.info(f\"Input: {input_strings[i]}\\n\")\n # NOTE: self.model.model.generation_config is not used here because it is the default\n # generation config that the CausalLM was initialized with, rather than the one set within the\n # context manager.\n generated_output = self.model.model.generate(\n input_ids=input_ids[i].unsqueeze(0),\n attention_mask=attention_mask[i].unsqueeze(0),\n generation_config=self.model.generation,\n streamer=streamer,\n )\n logger.info(\"----------------------\")\n outputs.append(generated_output)\n return torch.cat(outputs, dim=0)\n\n def _generate_non_streaming_outputs(\n self,\n _input_strings: str | list[str],\n input_ids: torch.Tensor,\n attention_mask: torch.Tensor,\n ) -> torch.Tensor:\n \"\"\"Generate non-streaming outputs for the given input.\n\n Args:\n _input_strings (Union[str, List[str]]): Unused input parameter.\n input_ids (torch.Tensor): Tensor containing input IDs.\n attention_mask (torch.Tensor): Tensor containing attention masks.\n streamer (Union[TextStreamer, None]): Text streamer instance for streaming output.\n\n Returns:\n torch.Tensor: Tensor of generated outputs.\n \"\"\"\n with torch.no_grad():\n # NOTE: self.model.model.generation_config is not used here because it is the default\n # generation config that the CausalLM was initialized with, rather than the one set within the\n # context manager.\n return self.model.model.generate(\n input_ids=input_ids,\n attention_mask=attention_mask,\n generation_config=self.model.generation,\n )\n\n def predict(\n self,\n dataset: str | dict | pd.DataFrame | None = None,\n data_format: str = None,\n split: str = FULL,\n batch_size: int = 128,\n generation_config: dict | None = None,\n skip_save_unprocessed_output: bool = True,\n skip_save_predictions: bool = True,\n output_directory: str = \"results\",\n return_type: str | dict | pd.DataFrame = pd.DataFrame,\n callbacks: list[Callback] | None = None,\n **kwargs,\n ) -> tuple[dict | pd.DataFrame, str]:\n \"\"\"Using a trained model, make predictions from the provided dataset.\n\n # Inputs\n\n :param dataset: (Union[str, dict, pandas.DataFrame]): source containing the entire dataset to be evaluated.\n :param data_format: (str, default: `None`) format to interpret data sources. Will be inferred automatically\n if not specified. Valid formats are `'auto'`, `'csv'`, `'df'`, `'dict'`, `'excel'`, `'feather'`,\n `'fwf'`, `'hdf5'` (cache file produced during previous training), `'html'` (file containing a single\n HTML `
`), `'json'`, `'jsonl'`, `'parquet'`, `'pickle'` (pickled Pandas DataFrame), `'sas'`,\n `'spss'`, `'stata'`, `'tsv'`.\n :param split: (str, default= `'full'`): if the input dataset contains a split column, this parameter\n indicates which split of the data to use. Possible values are `'full'`, `'training'`, `'validation'`,\n `'test'`.\n :param batch_size: (int, default: 128) size of batch to use when making predictions.\n :param generation_config: (Dict, default: `None`) config for the generation of the\n predictions. If `None`, the config that was used during model training is\n used. This is only used if the model type is LLM. Otherwise, this parameter is\n ignored. See\n [Large Language Models](https://ludwig.ai/latest/configuration/large_language_model/#generation) under\n \"Generation\" for an example generation config.\n :param skip_save_unprocessed_output: (bool, default: `True`) if this parameter is `False`, predictions and\n their probabilities are saved in both raw unprocessed numpy files containing tensors and as\n postprocessed CSV files (one for each output feature). If this parameter is `True`, only the CSV ones\n are saved and the numpy ones are skipped.\n :param skip_save_predictions: (bool, default: `True`) skips saving test predictions CSV files.\n :param output_directory: (str, default: `'results'`) the directory that will contain the training\n statistics, TensorBoard logs, the saved model and the training progress files.\n :param return_type: (Union[str, dict, pandas.DataFrame], default: pd.DataFrame) indicates the format of the\n returned predictions.\n :param callbacks: (Optional[List[Callback]], default: None) optional list of callbacks to use during this\n predict operation. Any callbacks already registered to the model will be preserved.\n\n # Return\n\n :return `(predictions, output_directory)`: (Tuple[Union[dict, pd.DataFrame], str])\n `predictions` predictions from the provided dataset,\n `output_directory` filepath string to where data was stored.\n \"\"\"\n self._check_initialization()\n\n # preprocessing\n start_time = time.time()\n logger.debug(\"Preprocessing\")\n dataset, _ = preprocess_for_prediction( # TODO (Connor): Refactor to use self.config_obj\n self.config_obj.to_dict(),\n dataset=dataset,\n training_set_metadata=self.training_set_metadata,\n data_format=data_format,\n split=split,\n include_outputs=False,\n backend=self.backend,\n callbacks=self.callbacks + (callbacks or []),\n )\n\n logger.debug(\"Predicting\")\n with self.backend.create_predictor(self.model, batch_size=batch_size) as predictor:\n with self.model.use_generation_config(generation_config):\n predictions = predictor.batch_predict(\n dataset,\n )\n\n if self.backend.is_coordinator():\n # if we are skipping all saving,\n # there is no need to create a directory that will remain empty\n should_create_exp_dir = not (skip_save_unprocessed_output and skip_save_predictions)\n if should_create_exp_dir:\n makedirs(output_directory, exist_ok=True)\n\n logger.debug(\"Postprocessing\")\n postproc_predictions = postprocess(\n predictions,\n self.model.output_features,\n self.training_set_metadata,\n output_directory=output_directory,\n backend=self.backend,\n skip_save_unprocessed_output=skip_save_unprocessed_output or not self.backend.is_coordinator(),\n )\n converted_postproc_predictions = convert_predictions(\n postproc_predictions, self.model.output_features, return_type=return_type, backend=self.backend\n )\n if self.backend.is_coordinator():\n if not skip_save_predictions:\n save_prediction_outputs(\n postproc_predictions, self.model.output_features, output_directory, self.backend\n )\n\n logger.info(f\"Saved to: {output_directory}\")\n\n logger.info(f\"Finished predicting in: {(time.time() - start_time):.2f}s.\")\n return converted_postproc_predictions, output_directory\n\n def evaluate(\n self,\n dataset: str | dict | pd.DataFrame | None = None,\n data_format: str | None = None,\n split: str = FULL,\n batch_size: int | None = None,\n skip_save_unprocessed_output: bool = True,\n skip_save_predictions: bool = True,\n skip_save_eval_stats: bool = True,\n collect_predictions: bool = False,\n collect_overall_stats: bool = False,\n output_directory: str = \"results\",\n return_type: str | dict | pd.DataFrame = pd.DataFrame,\n **kwargs,\n ) -> tuple[dict, dict | pd.DataFrame, str]:\n \"\"\"This function is used to predict the output variables given the input variables using the trained model\n and compute test statistics like performance measures, confusion matrices and the like.\n\n # Inputs\n :param dataset: (Union[str, dict, pandas.DataFrame]) source containing\n the entire dataset to be evaluated.\n :param data_format: (str, default: `None`) format to interpret data\n sources. Will be inferred automatically if not specified. Valid\n formats are `'auto'`, `'csv'`, `'df'`, `'dict'`, `'excel'`, `'feather'`,\n `'fwf'`, `'hdf5'` (cache file produced during previous training),\n `'html'` (file containing a single HTML `
`), `'json'`, `'jsonl'`,\n `'parquet'`, `'pickle'` (pickled Pandas DataFrame), `'sas'`, `'spss'`,\n `'stata'`, `'tsv'`.\n :param split: (str, default=`'full'`): if the input dataset contains\n a split column, this parameter indicates which split of the data\n to use. Possible values are `'full'`, `'training'`, `'validation'`, `'test'`.\n :param batch_size: (int, default: None) size of batch to use when making\n predictions. Defaults to model config eval_batch_size\n :param skip_save_unprocessed_output: (bool, default: `True`) if this\n parameter is `False`, predictions and their probabilities are saved\n in both raw unprocessed numpy files containing tensors and as\n postprocessed CSV files (one for each output feature).\n If this parameter is `True`, only the CSV ones are saved and the\n numpy ones are skipped.\n :param skip_save_predictions: (bool, default: `True`) skips saving\n test predictions CSV files.\n :param skip_save_eval_stats: (bool, default: `True`) skips saving\n test statistics JSON file.\n :param collect_predictions: (bool, default: `False`) if `True`\n collects post-processed predictions during eval.\n :param collect_overall_stats: (bool, default: False) if `True`\n collects overall stats during eval.\n :param output_directory: (str, default: `'results'`) the directory that\n will contain the training statistics, TensorBoard logs, the saved\n model and the training progress files.\n :param return_type: (Union[str, dict, pd.DataFrame], default: pandas.DataFrame) indicates\n the format to of the returned predictions.\n\n # Return\n :return: (`evaluation_statistics`, `predictions`, `output_directory`)\n `evaluation_statistics` dictionary containing evaluation performance\n statistics,\n `postprocess_predictions` contains predicted values,\n `output_directory` is location where results are stored.\n \"\"\"\n self._check_initialization()\n\n for callback in self.callbacks:\n callback.on_evaluation_start()\n\n # preprocessing\n logger.debug(\"Preprocessing\")\n dataset, training_set_metadata = preprocess_for_prediction( # TODO (Connor): Refactor to use self.config_obj\n self.config_obj.to_dict(),\n dataset=dataset,\n training_set_metadata=self.training_set_metadata,\n data_format=data_format,\n split=split,\n include_outputs=True,\n backend=self.backend,\n callbacks=self.callbacks,\n )\n\n # Fallback to use eval_batch_size or batch_size if not provided\n if batch_size is None:\n # Requires dictionary getter since some trainer configs may not have a batch_size param\n batch_size = self.config_obj.trainer.to_dict().get(\n EVAL_BATCH_SIZE, None\n ) or self.config_obj.trainer.to_dict().get(BATCH_SIZE, None)\n\n logger.debug(\"Predicting\")\n with self.backend.create_predictor(self.model, batch_size=batch_size) as predictor:\n eval_stats, predictions = predictor.batch_evaluation(\n dataset,\n collect_predictions=collect_predictions or collect_overall_stats,\n )\n\n # calculate the overall metrics\n if collect_overall_stats:\n dataset = dataset.to_df()\n\n overall_stats = calculate_overall_stats(\n self.model.output_features, predictions, dataset, training_set_metadata\n )\n eval_stats = {\n of_name: (\n {**eval_stats[of_name], **overall_stats[of_name]}\n # account for presence of 'combined' key\n if of_name in overall_stats\n else {**eval_stats[of_name]}\n )\n for of_name in eval_stats\n }\n\n if self.backend.is_coordinator():\n # if we are skipping all saving,\n # there is no need to create a directory that will remain empty\n should_create_exp_dir = not (\n skip_save_unprocessed_output and skip_save_predictions and skip_save_eval_stats\n )\n if should_create_exp_dir:\n makedirs(output_directory, exist_ok=True)\n\n if collect_predictions:\n logger.debug(\"Postprocessing\")\n postproc_predictions = postprocess(\n predictions,\n self.model.output_features,\n self.training_set_metadata,\n output_directory=output_directory,\n backend=self.backend,\n skip_save_unprocessed_output=skip_save_unprocessed_output or not self.backend.is_coordinator(),\n )\n else:\n postproc_predictions = predictions # = {}\n\n if self.backend.is_coordinator():\n should_save_predictions = (\n collect_predictions and postproc_predictions is not None and not skip_save_predictions\n )\n if should_save_predictions:\n save_prediction_outputs(\n postproc_predictions, self.model.output_features, output_directory, self.backend\n )\n\n print_evaluation_stats(eval_stats)\n if not skip_save_eval_stats:\n save_evaluation_stats(eval_stats, output_directory)\n\n if should_save_predictions or not skip_save_eval_stats:\n logger.info(f\"Saved to: {output_directory}\")\n\n if collect_predictions:\n postproc_predictions = convert_predictions(\n postproc_predictions, self.model.output_features, return_type=return_type, backend=self.backend\n )\n\n for callback in self.callbacks:\n callback.on_evaluation_end()\n\n return eval_stats, postproc_predictions, output_directory\n\n def forecast(\n self,\n dataset: DataFrame,\n data_format: str | None = None,\n horizon: int = 1,\n output_directory: str | None = None,\n output_format: str = \"parquet\",\n ) -> DataFrame:\n # TODO(travis): WIP\n dataset, _, _, _ = load_dataset_uris(dataset, None, None, None, self.backend)\n if isinstance(dataset, CacheableDataset):\n dataset = dataset.unwrap()\n dataset = load_dataset(dataset, data_format=data_format, df_lib=self.backend.df_engine.df_lib)\n\n window_sizes = [\n feature.preprocessing.window_size\n for feature in self.config_obj.input_features\n if feature.type == TIMESERIES\n ]\n if not window_sizes:\n raise ValueError(\"Forecasting requires at least one input feature of type `timeseries`.\")\n\n # TODO(travis): there's a lot of redundancy in this approach, since we are preprocessing the same DataFrame\n # multiple times with only a small number of features (the horizon) being appended each time.\n # A much better approach would be to only preprocess a single row, but incorporating the row-level embedding\n # over the window_size of rows precending it, then performing the model forward pass on only that row of\n # data.\n max_lookback_window_size = max(window_sizes)\n total_forecasted = 0\n while total_forecasted < horizon:\n # We only need the last `window_size` worth of rows to forecast the next value\n dataset = dataset.tail(max_lookback_window_size)\n\n # Run through preprocessing and prediction to obtain row-wise next values\n # TODO(travis): can optimize the preprocessing part here, since we only need to preprocess / predict\n # the last row, not the last `window_size` rows.\n preds, _ = self.predict(dataset, skip_save_predictions=True, skip_save_unprocessed_output=True)\n\n next_series = {}\n for feature in self.config_obj.output_features:\n if feature.type == TIMESERIES:\n key = f\"{feature.name}_predictions\"\n next_series[feature.column] = pd.Series(preds[key].iloc[-1])\n\n next_preds = pd.DataFrame(next_series)\n dataset = pd.concat([dataset, next_preds], axis=0).reset_index(drop=True)\n total_forecasted += len(next_preds)\n\n horizon_df = dataset.tail(total_forecasted).head(horizon)\n return_cols = [feature.column for feature in self.config_obj.output_features if feature.type == TIMESERIES]\n results_df = horizon_df[return_cols]\n\n if output_directory is not None:\n if self.backend.is_coordinator():\n # TODO(travis): generalize this to support any pandas output format\n if output_format == \"parquet\":\n output_path = os.path.join(output_directory, \"forecast.parquet\")\n results_df.to_parquet(output_path)\n elif output_format == \"csv\":\n output_path = os.path.join(output_directory, \"forecast.csv\")\n results_df.to_csv(output_path)\n else:\n raise ValueError(f\"`output_format` {output_format} not supported. Must be one of [parquet, csv]\")\n logger.info(f\"Saved to: {output_path}\")\n\n return results_df\n\n def experiment(\n self,\n dataset: str | dict | pd.DataFrame | None = None,\n training_set: str | dict | pd.DataFrame | None = None,\n validation_set: str | dict | pd.DataFrame | None = None,\n test_set: str | dict | pd.DataFrame | None = None,\n training_set_metadata: str | dict | None = None,\n data_format: str | None = None,\n experiment_name: str = \"experiment\",\n model_name: str = \"run\",\n model_resume_path: str | None = None,\n eval_split: str = TEST,\n skip_save_training_description: bool = False,\n skip_save_training_statistics: bool = False,\n skip_save_model: bool = False,\n skip_save_progress: bool = False,\n skip_save_log: bool = False,\n skip_save_processed_input: bool = False,\n skip_save_unprocessed_output: bool = False,\n skip_save_predictions: bool = False,\n skip_save_eval_stats: bool = False,\n skip_collect_predictions: bool = False,\n skip_collect_overall_stats: bool = False,\n output_directory: str = \"results\",\n random_seed: int = default_random_seed,\n **kwargs,\n ) -> tuple[dict | None, TrainingStats, PreprocessedDataset, str]:\n \"\"\"Trains a model on a dataset's training and validation splits and uses it to predict on the test split.\n It saves the trained model and the statistics of training and testing.\n\n # Inputs\n :param dataset: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing the entire dataset to be used in the experiment.\n If it has a split column, it will be used for splitting (0 for train,\n 1 for validation, 2 for test), otherwise the dataset will be\n randomly split.\n :param training_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing training data.\n :param validation_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing validation data.\n :param test_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing test data.\n :param training_set_metadata: (Union[str, dict], default: `None`)\n metadata JSON file or loaded metadata. Intermediate preprocessed\n structure containing the mappings of the input\n dataset created the first time an input file is used in the same\n directory with the same name and a '.meta.json' extension.\n :param data_format: (str, default: `None`) format to interpret data\n sources. Will be inferred automatically if not specified. Valid\n formats are `'auto'`, `'csv'`, `'df'`, `'dict'`, `'excel'`, `'feather'`,\n `'fwf'`, `'hdf5'` (cache file produced during previous training),\n `'html'` (file containing a single HTML `
`), `'json'`, `'jsonl'`,\n `'parquet'`, `'pickle'` (pickled Pandas DataFrame), `'sas'`, `'spss'`,\n `'stata'`, `'tsv'`.\n :param experiment_name: (str, default: `'experiment'`) name for\n the experiment.\n :param model_name: (str, default: `'run'`) name of the model that is\n being used.\n :param model_resume_path: (str, default: `None`) resumes training of\n the model from the path specified. The config is restored.\n In addition to config, training statistics and loss for\n epoch and the state of the optimizer are restored such that\n training can be effectively continued from a previously interrupted\n training process.\n :param eval_split: (str, default: `test`) split on which\n to perform evaluation. Valid values are `training`, `validation`\n and `test`.\n :param skip_save_training_description: (bool, default: `False`) disables\n saving the description JSON file.\n :param skip_save_training_statistics: (bool, default: `False`) disables\n saving training statistics JSON file.\n :param skip_save_model: (bool, default: `False`) disables\n saving model weights and hyperparameters each time the model\n improves. By default Ludwig saves model weights after each epoch\n the validation metric improves, but if the model is really big\n that can be time consuming. If you do not want to keep\n the weights and just find out what performance a model can get\n with a set of hyperparameters, use this parameter to skip it,\n but the model will not be loadable later on and the returned model\n will have the weights obtained at the end of training, instead of\n the weights of the epoch with the best validation performance.\n :param skip_save_progress: (bool, default: `False`) disables saving\n progress each epoch. By default Ludwig saves weights and stats\n after each epoch for enabling resuming of training, but if\n the model is really big that can be time consuming and will uses\n twice as much space, use this parameter to skip it, but training\n cannot be resumed later on.\n :param skip_save_log: (bool, default: `False`) disables saving\n TensorBoard logs. By default Ludwig saves logs for the TensorBoard,\n but if it is not needed turning it off can slightly increase the\n overall speed.\n :param skip_save_processed_input: (bool, default: `False`) if input\n dataset is provided it is preprocessed and cached by saving an HDF5\n and JSON files to avoid running the preprocessing again. If this\n parameter is `False`, the HDF5 and JSON file are not saved.\n :param skip_save_unprocessed_output: (bool, default: `False`) by default\n predictions and their probabilities are saved in both raw\n unprocessed numpy files containing tensors and as postprocessed\n CSV files (one for each output feature). If this parameter is True,\n only the CSV ones are saved and the numpy ones are skipped.\n :param skip_save_predictions: (bool, default: `False`) skips saving test\n predictions CSV files\n :param skip_save_eval_stats: (bool, default: `False`) skips saving test\n statistics JSON file\n :param skip_collect_predictions: (bool, default: `False`) skips\n collecting post-processed predictions during eval.\n :param skip_collect_overall_stats: (bool, default: `False`) skips\n collecting overall stats during eval.\n :param output_directory: (str, default: `'results'`) the directory that\n will contain the training statistics, TensorBoard logs, the saved\n model and the training progress files.\n :param random_seed: (int: default: 42) random seed used for weights\n initialization, splits and any other random function.\n\n # Return\n :return: (Tuple[dict, dict, tuple, str))\n `(evaluation_statistics, training_statistics, preprocessed_data, output_directory)`\n `evaluation_statistics` dictionary with evaluation performance\n statistics on the test_set,\n `training_statistics` is a nested dictionary of dataset -> feature_name -> metric_name -> List of metrics.\n Each metric corresponds to each training checkpoint.\n `preprocessed_data` tuple containing preprocessed\n `(training_set, validation_set, test_set)`, `output_directory`\n filepath string to where results are stored.\n \"\"\"\n if self._user_config.get(HYPEROPT):\n print_boxed(\"WARNING\")\n logger.warning(HYPEROPT_WARNING)\n\n train_stats, preprocessed_data, output_directory = self.train(\n dataset=dataset,\n training_set=training_set,\n validation_set=validation_set,\n test_set=test_set,\n training_set_metadata=training_set_metadata,\n data_format=data_format,\n experiment_name=experiment_name,\n model_name=model_name,\n model_resume_path=model_resume_path,\n skip_save_training_description=skip_save_training_description,\n skip_save_training_statistics=skip_save_training_statistics,\n skip_save_model=skip_save_model,\n skip_save_progress=skip_save_progress,\n skip_save_log=skip_save_log,\n skip_save_processed_input=skip_save_processed_input,\n skip_save_unprocessed_output=skip_save_unprocessed_output,\n output_directory=output_directory,\n random_seed=random_seed,\n )\n\n training_set, validation_set, test_set, training_set_metadata = preprocessed_data\n\n eval_set = validation_set\n if eval_split == TRAINING:\n eval_set = training_set\n elif eval_split == VALIDATION:\n eval_set = validation_set\n elif eval_split == TEST:\n eval_set = test_set\n else:\n logger.warning(f\"Eval split {eval_split} not supported. \" f\"Using validation set instead\")\n\n if eval_set is not None:\n trainer_dict = self.config_obj.trainer.to_dict()\n batch_size = trainer_dict.get(EVAL_BATCH_SIZE, trainer_dict.get(BATCH_SIZE, None))\n\n # predict\n try:\n eval_stats, _, _ = self.evaluate(\n eval_set,\n data_format=data_format,\n batch_size=batch_size,\n output_directory=output_directory,\n skip_save_unprocessed_output=skip_save_unprocessed_output,\n skip_save_predictions=skip_save_predictions,\n skip_save_eval_stats=skip_save_eval_stats,\n collect_predictions=not skip_collect_predictions,\n collect_overall_stats=not skip_collect_overall_stats,\n return_type=\"dict\",\n )\n except NotImplementedError:\n logger.warning(\n \"Skipping evaluation as the necessary methods are not \"\n \"supported. Full exception below:\\n\"\n f\"{traceback.format_exc()}\"\n )\n eval_stats = None\n else:\n logger.warning(f\"The evaluation set {eval_set} was not provided. \" f\"Skipping evaluation\")\n eval_stats = None\n\n return eval_stats, train_stats, preprocessed_data, output_directory\n\n def collect_weights(self, tensor_names: list[str] = None, **kwargs) -> list:\n \"\"\"Load a pre-trained model and collect the tensors with a specific name.\n\n # Inputs\n :param tensor_names: (list, default: `None`) List of tensor names to collect\n weights\n\n # Return\n :return: (list) List of tensors\n \"\"\"\n self._check_initialization()\n collected_tensors = self.model.collect_weights(tensor_names)\n return collected_tensors\n\n def collect_activations(\n self,\n layer_names: list[str],\n dataset: str | dict[str, list] | pd.DataFrame,\n data_format: str | None = None,\n split: str = FULL,\n batch_size: int = 128,\n **kwargs,\n ) -> list:\n \"\"\"Loads a pre-trained model model and input data to collect the values of the activations contained in the\n tensors.\n\n # Inputs\n :param layer_names: (list) list of strings for layer names in the model\n to collect activations.\n :param dataset: (Union[str, Dict[str, list], pandas.DataFrame]) source\n containing the data to make predictions.\n :param data_format: (str, default: `None`) format to interpret data\n sources. Will be inferred automatically if not specified. Valid\n formats are `'auto'`, `'csv'`, `'df'`, `'dict'`, `'excel'`, `'feather'`,\n `'fwf'`, `'hdf5'` (cache file produced during previous training),\n `'html'` (file containing a single HTML `
`), `'json'`, `'jsonl'`,\n `'parquet'`, `'pickle'` (pickled Pandas DataFrame), `'sas'`, `'spss'`,\n `'stata'`, `'tsv'`.\n :param split: (str, default= `'full'`): if the input dataset contains\n a split column, this parameter indicates which split of the data\n to use. Possible values are `'full'`, `'training'`, `'validation'`, `'test'`.\n :param batch_size: (int, default: 128) size of batch to use when making\n predictions.\n\n # Return\n :return: (list) list of collected tensors.\n \"\"\"\n self._check_initialization()\n\n # preprocessing\n logger.debug(\"Preprocessing\")\n dataset, training_set_metadata = preprocess_for_prediction( # TODO (Connor): Refactor to use self.config_obj\n self.config_obj.to_dict(),\n dataset=dataset,\n training_set_metadata=self.training_set_metadata,\n data_format=data_format,\n split=split,\n include_outputs=False,\n )\n\n logger.debug(\"Predicting\")\n with self.backend.create_predictor(self.model, batch_size=batch_size) as predictor:\n activations = predictor.batch_collect_activations(\n layer_names,\n dataset,\n )\n\n return activations\n\n def preprocess(\n self,\n dataset: str | dict | pd.DataFrame | None = None,\n training_set: str | dict | pd.DataFrame | None = None,\n validation_set: str | dict | pd.DataFrame | None = None,\n test_set: str | dict | pd.DataFrame | None = None,\n training_set_metadata: str | dict | None = None,\n data_format: str | None = None,\n skip_save_processed_input: bool = True,\n random_seed: int = default_random_seed,\n **kwargs,\n ) -> PreprocessedDataset:\n \"\"\"This function is used to preprocess data.\n\n # Args:\n :param dataset: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing the entire dataset to be used in the experiment.\n If it has a split column, it will be used for splitting\n (0 for train, 1 for validation, 2 for test),\n otherwise the dataset will be randomly split.\n :param training_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing training data.\n :param validation_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing validation data.\n :param test_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing test data.\n :param training_set_metadata: (Union[str, dict], default: `None`)\n metadata JSON file or loaded metadata. Intermediate preprocessed\n structure containing the mappings of the input\n dataset created the first time an input file is used in the same\n directory with the same name and a '.meta.json' extension.\n :param data_format: (str, default: `None`) format to interpret data\n sources. Will be inferred automatically if not specified. Valid\n formats are `'auto'`, `'csv'`, `'df'`, `'dict'`, `'excel'`,\n `'feather'`, `'fwf'`,\n `'hdf5'` (cache file produced during previous training),\n `'html'` (file containing a single HTML `
`),\n `'json'`, `'jsonl'`, `'parquet'`,\n `'pickle'` (pickled Pandas DataFrame),\n `'sas'`, `'spss'`, `'stata'`, `'tsv'`.\n :param skip_save_processed_input: (bool, default: `False`) if input\n dataset is provided it is preprocessed and cached by saving an HDF5\n and JSON files to avoid running the preprocessing again. If this\n parameter is `False`, the HDF5 and JSON file are not saved.\n :param random_seed: (int, default: `42`) a random seed that will be\n used anywhere there is a call to a random number generator: data\n splitting, parameter initialization and training set shuffling\n\n # Returns:\n :return: (PreprocessedDataset) data structure containing\n `(proc_training_set, proc_validation_set, proc_test_set, training_set_metadata)`.\n\n # Raises:\n RuntimeError: An error occurred while preprocessing the data. Examples include training dataset\n being empty after preprocessing, lazy loading not being supported with RayBackend, etc.\n \"\"\"\n print_boxed(\"PREPROCESSING\")\n\n for callback in self.callbacks:\n callback.on_preprocess_start(self.config_obj.to_dict())\n\n preprocessing_params = get_preprocessing_params(self.config_obj)\n\n proc_training_set = proc_validation_set = proc_test_set = None\n try:\n with provision_preprocessing_workers(self.backend):\n # TODO (Connor): Refactor to use self.config_obj\n preprocessed_data = preprocess_for_training(\n self.config_obj.to_dict(),\n dataset=dataset,\n training_set=training_set,\n validation_set=validation_set,\n test_set=test_set,\n training_set_metadata=training_set_metadata,\n data_format=data_format,\n skip_save_processed_input=skip_save_processed_input,\n preprocessing_params=preprocessing_params,\n backend=self.backend,\n random_seed=random_seed,\n callbacks=self.callbacks,\n )\n\n proc_training_set, proc_validation_set, proc_test_set, training_set_metadata = preprocessed_data\n\n return PreprocessedDataset(proc_training_set, proc_validation_set, proc_test_set, training_set_metadata)\n except Exception as e:\n raise RuntimeError(f\"Caught exception during model preprocessing: {str(e)}\") from e\n finally:\n for callback in self.callbacks:\n callback.on_preprocess_end(proc_training_set, proc_validation_set, proc_test_set, training_set_metadata)\n\n @staticmethod\n def load(\n model_dir: str,\n logging_level: int = logging.ERROR,\n backend: Backend | str | None = None,\n gpus: str | int | list[int] | None = None,\n gpu_memory_limit: float | None = None,\n allow_parallel_threads: bool = True,\n callbacks: list[Callback] = None,\n from_checkpoint: bool = False,\n ) -> \"LudwigModel\": # return is an instance of ludwig.api.LudwigModel class\n \"\"\"This function allows for loading pretrained models.\n\n # Inputs\n\n :param model_dir: (str) path to the directory containing the model.\n If the model was trained by the `train` or `experiment` command,\n the model is in `results_dir/experiment_dir/model`.\n :param logging_level: (int, default: 40) log level that will be sent to\n stderr.\n :param backend: (Union[Backend, str]) `Backend` or string name\n of backend to use to execute preprocessing / training steps.\n :param gpus: (Union[str, int, List[int]], default: `None`) GPUs\n to use (it uses the same syntax of CUDA_VISIBLE_DEVICES)\n :param gpu_memory_limit: (float: default: `None`) maximum memory fraction\n [0, 1] allowed to allocate per GPU device.\n :param allow_parallel_threads: (bool, default: `True`) allow Torch\n to use\n multithreading parallelism to improve performance at the cost of\n determinism.\n :param callbacks: (list, default: `None`) a list of\n `ludwig.callbacks.Callback` objects that provide hooks into the\n Ludwig pipeline.\n :param from_checkpoint: (bool, default: `False`) if `True`, the model\n will be loaded from the latest checkpoint (training_checkpoints/)\n instead of the final model weights.\n\n # Return\n\n :return: (LudwigModel) a LudwigModel object\n\n\n # Example usage\n\n ```python\n ludwig_model = LudwigModel.load(model_dir)\n ```\n \"\"\"\n # Initialize PyTorch before calling `broadcast()` to prevent initializing\n # Torch with default parameters\n backend_param = backend\n backend = initialize_backend(backend)\n backend.initialize_pytorch(\n gpus=gpus, gpu_memory_limit=gpu_memory_limit, allow_parallel_threads=allow_parallel_threads\n )\n\n config = backend.broadcast_return(lambda: load_json(os.path.join(model_dir, MODEL_HYPERPARAMETERS_FILE_NAME)))\n\n # Upgrades deprecated fields and adds new required fields in case the config loaded from disk is old.\n config_obj = ModelConfig.from_dict(config)\n\n # Ensure that the original backend is used if it was specified in the config and user requests it\n if backend_param is None and \"backend\" in config:\n # Reset backend from config\n backend = initialize_backend(config.get(\"backend\"))\n\n # initialize model\n ludwig_model = LudwigModel(\n config_obj.to_dict(),\n logging_level=logging_level,\n backend=backend,\n gpus=gpus,\n gpu_memory_limit=gpu_memory_limit,\n allow_parallel_threads=allow_parallel_threads,\n callbacks=callbacks,\n )\n\n # generate model from config\n set_saved_weights_in_checkpoint_flag(config_obj)\n ludwig_model.model = LudwigModel.create_model(config_obj)\n\n # load model weights\n ludwig_model.load_weights(model_dir, from_checkpoint)\n\n # If merge_and_unload was NOT performed before saving (i.e., adapter weights exist),\n # we need to merge them now for inference.\n if ludwig_model.is_merge_and_unload_set():\n weights_save_path = os.path.join(model_dir, MODEL_WEIGHTS_FILE_NAME)\n adapter_config_path = os.path.join(weights_save_path, \"adapter_config.json\")\n if os.path.exists(adapter_config_path):\n ludwig_model.model.merge_and_unload(progressbar=config_obj.adapter.postprocessor.progressbar)\n\n # load train set metadata\n ludwig_model.training_set_metadata = backend.broadcast_return(\n lambda: load_metadata(os.path.join(model_dir, TRAIN_SET_METADATA_FILE_NAME))\n )\n\n return ludwig_model\n\n def load_weights(\n self,\n model_dir: str,\n from_checkpoint: bool = False,\n ) -> None:\n \"\"\"Loads weights from a pre-trained model.\n\n # Inputs\n :param model_dir: (str) filepath string to location of a pre-trained\n model\n :param from_checkpoint: (bool, default: `False`) if `True`, the model\n will be loaded from the latest checkpoint (training_checkpoints/)\n instead of the final model weights.\n\n # Return\n :return: `None`\n\n # Example usage\n\n ```python\n ludwig_model.load_weights(model_dir)\n ```\n \"\"\"\n if self.backend.is_coordinator():\n if from_checkpoint:\n with self.backend.create_trainer(\n model=self.model,\n config=self.config_obj.trainer,\n ) as trainer:\n checkpoint = trainer.create_checkpoint_handle()\n training_checkpoints_path = os.path.join(model_dir, TRAINING_CHECKPOINTS_DIR_PATH)\n trainer.resume_weights_and_optimizer(training_checkpoints_path, checkpoint)\n else:\n self.model.load(model_dir)\n\n self.backend.sync_model(self.model)\n\n def save(self, save_path: str) -> None:\n \"\"\"This function allows to save models on disk.\n\n # Inputs\n\n :param save_path: (str) path to the directory where the model is\n going to be saved. Both a JSON file containing the model\n architecture hyperparameters and checkpoints files containing\n model weights will be saved.\n\n # Return\n\n :return: (None) `None`\n\n # Example usage\n\n ```python\n ludwig_model.save(save_path)\n ```\n \"\"\"\n self._check_initialization()\n\n # save config\n self.save_config(save_path)\n\n # save model weights\n self.model.save(save_path)\n\n # save training set metadata\n training_set_metadata_path = os.path.join(save_path, TRAIN_SET_METADATA_FILE_NAME)\n save_json(training_set_metadata_path, self.training_set_metadata)\n\n @staticmethod\n def upload_to_hf_hub(\n repo_id: str,\n model_path: str,\n repo_type: str = \"model\",\n private: bool = False,\n commit_message: str = \"Upload trained [Ludwig](https://ludwig.ai/latest/) model weights\",\n commit_description: str | None = None,\n ) -> bool:\n \"\"\"Uploads trained model artifacts to the HuggingFace Hub.\n\n # Inputs\n\n :param repo_id: (`str`)\n A namespace (user or an organization) and a repo name separated\n by a `/`.\n :param model_path: (`str`)\n The path of the saved model. This is either (a) the folder where\n the 'model_weights' folder and the 'model_hyperparameters.json' file\n are stored, or (b) the parent of that folder.\n :param private: (`bool`, *optional*, defaults to `False`)\n Whether the model repo should be private.\n :param repo_type: (`str`, *optional*)\n Set to `\"dataset\"` or `\"space\"` if uploading to a dataset or\n space, `None` or `\"model\"` if uploading to a model. Default is\n `None`.\n :param commit_message: (`str`, *optional*)\n The summary / title / first line of the generated commit. Defaults to:\n `f\"Upload {path_in_repo} with huggingface_hub\"`\n :param commit_description: (`str` *optional*)\n The description of the generated commit\n\n # Returns\n\n :return: (bool) True for success, False for failure.\n \"\"\"\n if os.path.exists(os.path.join(model_path, MODEL_FILE_NAME, MODEL_WEIGHTS_FILE_NAME)) and os.path.exists(\n os.path.join(model_path, MODEL_FILE_NAME, MODEL_HYPERPARAMETERS_FILE_NAME)\n ):\n experiment_path = model_path\n elif os.path.exists(os.path.join(model_path, MODEL_WEIGHTS_FILE_NAME)) and os.path.exists(\n os.path.join(model_path, MODEL_HYPERPARAMETERS_FILE_NAME)\n ):\n experiment_path = os.path.dirname(model_path)\n else:\n raise ValueError(\n f\"Can't find 'model_weights' and '{MODEL_HYPERPARAMETERS_FILE_NAME}' either at \"\n f\"'{model_path}' or at '{model_path}/model'\"\n )\n model_service = get_upload_registry()[\"hf_hub\"]\n hub: HuggingFaceHub = model_service()\n hub.login()\n upload_status: bool = hub.upload(\n repo_id=repo_id,\n model_path=experiment_path,\n repo_type=repo_type,\n private=private,\n commit_message=commit_message,\n commit_description=commit_description,\n )\n return upload_status\n\n def save_config(self, save_path: str) -> None:\n \"\"\"Save config to specified location.\n\n # Inputs\n\n :param save_path: (str) filepath string to save config as a\n JSON file.\n\n # Return\n :return: `None`\n \"\"\"\n os.makedirs(save_path, exist_ok=True)\n model_hyperparameters_path = os.path.join(save_path, MODEL_HYPERPARAMETERS_FILE_NAME)\n save_json(model_hyperparameters_path, self.config_obj.to_dict())\n\n def to_torchscript(\n self,\n model_only: bool = False,\n device: TorchDevice | None = None,\n ):\n \"\"\"Converts the trained model to Torchscript.\n\n # Inputs\n\n :param model_only (bool, optional): If True, only the ECD model will be converted to Torchscript. Else,\n preprocessing and postprocessing steps will also be converted to Torchscript. :param device (TorchDevice,\n optional): If None, the model will be converted to Torchscript on the same device to ensure maximum model\n parity.\n\n # Returns\n\n :return: A torch.jit.ScriptModule that can be used to predict on a dictionary of inputs.\n \"\"\"\n if device is None:\n device = DEVICE\n\n self._check_initialization()\n if model_only:\n return self.model.to_torchscript(device)\n else:\n inference_module = InferenceModule.from_ludwig_model(\n self.model, self.config_obj.to_dict(), self.training_set_metadata, device=device\n )\n return torch.jit.script(inference_module)\n\n def save_torchscript(\n self,\n save_path: str,\n model_only: bool = False,\n device: TorchDevice | None = None,\n ):\n \"\"\"Saves the Torchscript model to disk.\n\n # Inputs\n\n :param save_path (str): The path to the directory where the model will be saved.\n :param model_only (bool, optional): If True, only the ECD model will be converted to Torchscript. Else, the\n preprocessing and postprocessing steps will also be converted to Torchscript.\n :param device (TorchDevice, optional): If None, the model will be converted to Torchscript on the same device to\n ensure maximum model parity.\n\n # Return\n\n :return: `None`\n \"\"\"\n if device is None:\n device = DEVICE\n\n save_ludwig_model_for_inference(\n save_path,\n self.model,\n self.config_obj.to_dict(),\n self.training_set_metadata,\n model_only=model_only,\n device=device,\n )\n\n def _check_initialization(self):\n if self.model is None or self._user_config is None or self.training_set_metadata is None:\n raise ValueError(\"Model has not been trained or loaded\")\n\n def free_gpu_memory(self):\n \"\"\"Manually moves the model to CPU to force GPU memory to be freed.\n\n For more context: https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/35\n \"\"\"\n if torch.cuda.is_available():\n self.model.model.to(torch.device(\"cpu\"))\n torch.cuda.empty_cache()\n\n @staticmethod\n def create_model(config_obj: ModelConfig | dict, random_seed: int = default_random_seed) -> BaseModel:\n \"\"\"Instantiates BaseModel object.\n\n # Inputs\n :param config_obj: (Union[Config, dict]) Ludwig config object\n :param random_seed: (int, default: ludwig default random seed) Random seed used for weights initialization,\n splits and any other random function. # Return\n :return: (ludwig.models.BaseModel) Instance of the Ludwig model object.\n \"\"\"\n if isinstance(config_obj, dict):\n config_obj = ModelConfig.from_dict(config_obj)\n model_type = get_from_registry(config_obj.model_type, model_type_registry)\n return model_type(config_obj, random_seed=random_seed)\n\n @staticmethod\n def set_logging_level(logging_level: int) -> None:\n \"\"\"Sets level for log messages.\n\n # Inputs\n\n :param logging_level: (int) Set/Update the logging level. Use logging\n constants like `logging.DEBUG` , `logging.INFO` and `logging.ERROR`.\n\n # Return\n\n :return: `None`\n \"\"\"\n logging.getLogger(\"ludwig\").setLevel(logging_level)\n if logging_level in {logging.WARNING, logging.ERROR, logging.CRITICAL}:\n set_disable_progressbar(True)\n else:\n set_disable_progressbar(False)\n\n @property\n def config(self) -> ModelConfigDict:\n \"\"\"Returns the fully-rendered config of this model including default values.\"\"\"\n return self.config_obj.to_dict()\n\n @config.setter\n def config(self, user_config: ModelConfigDict):\n \"\"\"Updates the config of this model.\n\n WARNING: this can have unexpected results on an already trained model.\n \"\"\"\n self._user_config = user_config\n self.config_obj = ModelConfig.from_dict(self._user_config)\n\n def is_merge_and_unload_set(self) -> bool:\n \"\"\"Check whether the encapsulated model is of type LLM and is configured to merge_and_unload QLoRA weights.\n\n # Return\n\n :return (bool): whether merge_and_unload should be done.\n \"\"\"\n # TODO: In the future, it may be possible to move up the model type check into the BaseModel class.\n return self.config_obj.model_type == MODEL_LLM and self.model.is_merge_and_unload_set()\n\n\n@PublicAPI\ndef kfold_cross_validate(\n num_folds: int,\n config: dict | str,\n dataset: str = None,\n data_format: str = None,\n skip_save_training_description: bool = False,\n skip_save_training_statistics: bool = False,\n skip_save_model: bool = False,\n skip_save_progress: bool = False,\n skip_save_log: bool = False,\n skip_save_processed_input: bool = False,\n skip_save_predictions: bool = False,\n skip_save_eval_stats: bool = False,\n skip_collect_predictions: bool = False,\n skip_collect_overall_stats: bool = False,\n output_directory: str = \"results\",\n random_seed: int = default_random_seed,\n gpus: str | int | list[int] | None = None,\n gpu_memory_limit: float | None = None,\n allow_parallel_threads: bool = True,\n backend: Backend | str | None = None,\n logging_level: int = logging.INFO,\n **kwargs,\n) -> tuple[dict, dict]:\n \"\"\"Performs k-fold cross validation and returns result data structures.\n\n # Inputs\n\n :param num_folds: (int) number of folds to create for the cross-validation\n :param config: (Union[dict, str]) model specification\n required to build a model. Parameter may be a dictionary or string\n specifying the file path to a yaml configuration file. Refer to the\n [User Guide](http://ludwig.ai/user_guide/#model-config)\n for details.\n :param dataset: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing the entire dataset to be used for k_fold processing.\n :param data_format: (str, default: `None`) format to interpret data\n sources. Will be inferred automatically if not specified. Valid\n formats are `'auto'`, `'csv'`, `'df'`, `'dict'`, `'excel'`, `'feather'`,\n `'fwf'`,\n `'html'` (file containing a single HTML `
`), `'json'`, `'jsonl'`,\n `'parquet'`, `'pickle'` (pickled Pandas DataFrame), `'sas'`, `'spss'`,\n `'stata'`, `'tsv'`. Currently `hdf5` format is not supported for\n k_fold cross validation.\n :param skip_save_training_description: (bool, default: `False`) disables\n saving the description JSON file.\n :param skip_save_training_statistics: (bool, default: `False`) disables\n saving training statistics JSON file.\n :param skip_save_model: (bool, default: `False`) disables\n saving model weights and hyperparameters each time the model\n improves. By default Ludwig saves model weights after each epoch\n the validation metric improves, but if the model is really big\n that can be time consuming. If you do not want to keep\n the weights and just find out what performance a model can get\n with a set of hyperparameters, use this parameter to skip it,\n but the model will not be loadable later on and the returned model\n will have the weights obtained at the end of training, instead of\n the weights of the epoch with the best validation performance.\n :param skip_save_progress: (bool, default: `False`) disables saving\n progress each epoch. By default Ludwig saves weights and stats\n after each epoch for enabling resuming of training, but if\n the model is really big that can be time consuming and will uses\n twice as much space, use this parameter to skip it, but training\n cannot be resumed later on.\n :param skip_save_log: (bool, default: `False`) disables saving TensorBoard\n logs. By default Ludwig saves logs for the TensorBoard, but if it\n is not needed turning it off can slightly increase the\n overall speed.\n :param skip_save_processed_input: (bool, default: `False`) if input\n dataset is provided it is preprocessed and cached by saving an HDF5\n and JSON files to avoid running the preprocessing again. If this\n parameter is `False`, the HDF5 and JSON file are not saved.\n :param skip_save_predictions: (bool, default: `False`) skips saving test\n predictions CSV files.\n :param skip_save_eval_stats: (bool, default: `False`) skips saving test\n statistics JSON file.\n :param skip_collect_predictions: (bool, default: `False`) skips collecting\n post-processed predictions during eval.\n :param skip_collect_overall_stats: (bool, default: `False`) skips collecting\n overall stats during eval.\n :param output_directory: (str, default: `'results'`) the directory that\n will contain the training statistics, TensorBoard logs, the saved\n model and the training progress files.\n :param random_seed: (int, default: `42`) Random seed\n used for weights initialization,\n splits and any other random function.\n :param gpus: (list, default: `None`) list of GPUs that are available\n for training.\n :param gpu_memory_limit: (float: default: `None`) maximum memory fraction\n [0, 1] allowed to allocate per GPU device.\n :param allow_parallel_threads: (bool, default: `True`) allow Torch to\n use multithreading parallelism\n to improve performance at the cost of determinism.\n :param backend: (Union[Backend, str]) `Backend` or string name\n of backend to use to execute preprocessing / training steps.\n :param logging_level: (int, default: INFO) log level to send to stderr.\n\n\n # Return\n\n :return: (tuple(kfold_cv_statistics, kfold_split_indices), dict) a tuple of\n dictionaries `kfold_cv_statistics`: contains metrics from cv run.\n `kfold_split_indices`: indices to split training data into\n training fold and test fold.\n \"\"\"\n # if config is a path, convert to dictionary\n if isinstance(config, str): # assume path\n config = load_yaml(config)\n backend = initialize_backend(backend or config.get(\"backend\"))\n\n # check for k_fold\n if num_folds is None:\n raise ValueError(\"k_fold parameter must be specified\")\n\n logger.info(f\"starting {num_folds:d}-fold cross validation\")\n\n # create output_directory if not available\n if not os.path.isdir(output_directory):\n os.mkdir(output_directory)\n\n # prepare data for k-fold processing\n # use Ludwig's utility to facilitate creating a dataframe\n # that is used as the basis for creating folds\n\n dataset, _, _, _ = load_dataset_uris(dataset, None, None, None, backend)\n\n # determine data format of provided dataset\n if not data_format or data_format == \"auto\":\n data_format = figure_data_format(dataset)\n\n data_df = load_dataset(dataset, data_format=data_format, df_lib=backend.df_engine.df_lib)\n\n kfold_cv_stats = {}\n kfold_split_indices = {}\n\n for train_indices, test_indices, fold_num in generate_kfold_splits(data_df, num_folds, random_seed):\n with tempfile.TemporaryDirectory() as temp_dir_name:\n curr_train_df = data_df.iloc[train_indices]\n curr_test_df = data_df.iloc[test_indices]\n\n kfold_split_indices[\"fold_\" + str(fold_num)] = {\n \"training_indices\": train_indices,\n \"test_indices\": test_indices,\n }\n\n # train and validate model on this fold\n logger.info(f\"training on fold {fold_num:d}\")\n\n model = LudwigModel(\n config=config,\n logging_level=logging_level,\n backend=backend,\n gpus=gpus,\n gpu_memory_limit=gpu_memory_limit,\n allow_parallel_threads=allow_parallel_threads,\n )\n eval_stats, train_stats, preprocessed_data, output_directory = model.experiment(\n training_set=curr_train_df,\n test_set=curr_test_df,\n experiment_name=\"cross_validation\",\n model_name=\"fold_\" + str(fold_num),\n skip_save_training_description=skip_save_training_description,\n skip_save_training_statistics=skip_save_training_statistics,\n skip_save_model=skip_save_model,\n skip_save_progress=skip_save_progress,\n skip_save_log=skip_save_log,\n skip_save_processed_input=skip_save_processed_input,\n skip_save_predictions=skip_save_predictions,\n skip_save_eval_stats=skip_save_eval_stats,\n skip_collect_predictions=skip_collect_predictions,\n skip_collect_overall_stats=skip_collect_overall_stats,\n output_directory=os.path.join(temp_dir_name, \"results\"),\n random_seed=random_seed,\n )\n\n # augment the training statistics with scoring metric from\n # the hold out fold\n if dataclasses.is_dataclass(train_stats):\n train_stats_dict = dataclasses.asdict(train_stats)\n elif hasattr(train_stats, \"to_dict\"):\n train_stats_dict = train_stats.to_dict()\n else:\n train_stats_dict = vars(train_stats)\n train_stats_dict[\"fold_eval_stats\"] = eval_stats\n\n # collect training statistics for this fold\n kfold_cv_stats[\"fold_\" + str(fold_num)] = train_stats_dict\n\n # consolidate raw fold metrics across all folds\n raw_kfold_stats = {}\n for fold_name in kfold_cv_stats:\n curr_fold_eval_stats = kfold_cv_stats[fold_name][\"fold_eval_stats\"]\n for of_name in curr_fold_eval_stats:\n if of_name not in raw_kfold_stats:\n raw_kfold_stats[of_name] = {}\n fold_eval_stats_of = curr_fold_eval_stats[of_name]\n\n for metric in fold_eval_stats_of:\n if metric not in {\n \"predictions\",\n \"probabilities\",\n \"confusion_matrix\",\n \"overall_stats\",\n \"per_class_stats\",\n \"roc_curve\",\n \"precision_recall_curve\",\n }:\n if metric not in raw_kfold_stats[of_name]:\n raw_kfold_stats[of_name][metric] = []\n raw_kfold_stats[of_name][metric].append(fold_eval_stats_of[metric])\n\n # calculate overall kfold statistics\n overall_kfold_stats = {}\n for of_name in raw_kfold_stats:\n overall_kfold_stats[of_name] = {}\n for metric in raw_kfold_stats[of_name]:\n mean = np.mean(raw_kfold_stats[of_name][metric])\n std = np.std(raw_kfold_stats[of_name][metric])\n overall_kfold_stats[of_name][metric + \"_mean\"] = mean\n overall_kfold_stats[of_name][metric + \"_std\"] = std\n\n kfold_cv_stats[\"overall\"] = overall_kfold_stats\n\n logger.info(f\"completed {num_folds:d}-fold cross validation\")\n\n return kfold_cv_stats, kfold_split_indices\n\n\ndef _get_compute_description(backend) -> dict:\n \"\"\"Returns the compute description for the backend.\"\"\"\n compute_description = {\"num_nodes\": backend.num_nodes}\n\n if torch.cuda.is_available():\n # Assumption: All nodes are of the same instance type.\n # TODO: fix for Ray where workers may be of different skus\n compute_description.update(\n {\n \"gpus_per_node\": torch.cuda.device_count(),\n \"arch_list\": torch.cuda.get_arch_list(),\n \"gencode_flags\": torch.cuda.get_gencode_flags(),\n \"devices\": {},\n }\n )\n for i in range(torch.cuda.device_count()):\n compute_description[\"devices\"][i] = {\n \"gpu_type\": torch.cuda.get_device_name(i),\n \"device_capability\": torch.cuda.get_device_capability(i),\n \"device_properties\": str(torch.cuda.get_device_properties(i)),\n }\n\n return compute_description\n\n\n@PublicAPI\ndef get_experiment_description(\n config,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n data_format=None,\n backend=None,\n random_seed=None,\n):\n description = OrderedDict()\n description[\"ludwig_version\"] = LUDWIG_VERSION\n description[\"command\"] = \" \".join(sys.argv)\n\n commit_hash = get_commit_hash()\n if commit_hash is not None:\n description[\"commit_hash\"] = commit_hash[:12]\n\n if random_seed is not None:\n description[\"random_seed\"] = random_seed\n\n if isinstance(dataset, str):\n description[\"dataset\"] = dataset\n if isinstance(training_set, str):\n description[\"training_set\"] = training_set\n if isinstance(validation_set, str):\n description[\"validation_set\"] = validation_set\n if isinstance(test_set, str):\n description[\"test_set\"] = test_set\n if training_set_metadata is not None:\n description[\"training_set_metadata\"] = training_set_metadata\n\n # determine data format if not provided or auto\n if not data_format or data_format == \"auto\":\n data_format = figure_data_format(dataset, training_set, validation_set, test_set)\n\n if data_format:\n description[\"data_format\"] = str(data_format)\n\n description[\"config\"] = config\n description[\"torch_version\"] = torch.__version__\n description[\"compute\"] = _get_compute_description(backend)\n\n return description\n"
},
{
"path": "ludwig/api_annotations.py",
"content": "def PublicAPI(*args, **kwargs):\n \"\"\"Annotation for documenting public APIs. Public APIs are classes and methods exposed to end users of Ludwig.\n\n If stability=\"stable\", the APIs will remain backwards compatible across minor Ludwig releases\n (e.g., Ludwig 0.6 -> Ludwig 0.7).\n\n If stability=\"experimental\", the APIs can be used by advanced users who are tolerant to and expect\n breaking changes. This will likely be seen in the case of incremental new feature development.\n\n Args:\n stability: One of {\"stable\", \"experimental\"}\n\n Examples:\n >>> from api_annotations import PublicAPI\n >>> @PublicAPI\n ... def func1(x):\n ... return x\n >>> @PublicAPI(stability=\"experimental\")\n ... def func2(y):\n ... return y\n \"\"\"\n if len(args) == 1 and len(kwargs) == 0 and callable(args[0]):\n return PublicAPI(stability=\"stable\")(args[0])\n\n if \"stability\" in kwargs:\n stability = kwargs[\"stability\"]\n assert stability in [\"stable\", \"experimental\"], stability\n elif kwargs:\n raise ValueError(f\"Unknown kwargs: {kwargs.keys()}\")\n else:\n stability = \"stable\"\n\n def wrap(obj):\n if stability == \"experimental\":\n message = f\"PublicAPI ({stability}): This API is {stability} and may change before becoming stable.\"\n else:\n message = \"PublicAPI: This API is stable across Ludwig releases.\"\n\n _append_doc(obj, message=message)\n _mark_annotated(obj)\n return obj\n\n return wrap\n\n\ndef DeveloperAPI(*args, **kwargs):\n \"\"\"Annotation for documenting developer APIs. Developer APIs are lower-level methods explicitly exposed to\n advanced Ludwig users and library developers. Their interfaces may change across minor Ludwig releases (for\n e.g., Ludwig 0.6.1 and Ludwig 0.6.2).\n\n Examples:\n >>> from api_annotations import DeveloperAPI\n >>> @DeveloperAPI\n ... def func(x):\n ... return x\n \"\"\"\n if len(args) == 1 and len(kwargs) == 0 and callable(args[0]):\n return DeveloperAPI()(args[0])\n\n def wrap(obj):\n _append_doc(obj, message=\"DeveloperAPI: This API may change across minor Ludwig releases.\")\n _mark_annotated(obj)\n return obj\n\n return wrap\n\n\ndef Deprecated(*args, **kwargs):\n \"\"\"Annotation for documenting a deprecated API. Deprecated APIs may be removed in future releases of Ludwig\n (e.g., Ludwig 0.7 to Ludwig 0.8).\n\n Args:\n message: A message to help users understand the reason for the deprecation, and provide a migration path.\n\n Examples:\n >>> from api_annotations import Deprecated\n >>> @Deprecated\n ... def func(x):\n ... return x\n >>> @Deprecated(message=\"g() is deprecated because the API is error prone. Please call h() instead.\")\n ... def g(y):\n ... return y\n \"\"\"\n if len(args) == 1 and len(kwargs) == 0 and callable(args[0]):\n return Deprecated()(args[0])\n\n message = \"**DEPRECATED:** This API is deprecated and may be removed in a future Ludwig release.\"\n\n if \"message\" in kwargs:\n message += \" \" + kwargs[\"message\"]\n del kwargs[\"message\"]\n\n if kwargs:\n raise ValueError(f\"Unknown kwargs: {kwargs.keys()}\")\n\n def inner(obj):\n _append_doc(obj, message=message, directive=\"warning\")\n _mark_annotated(obj)\n return obj\n\n return inner\n\n\ndef _append_doc(obj, message: str, directive: str | None = None) -> str:\n \"\"\"\n Args:\n message: An additional message to append to the end of docstring for a class\n or method that uses one of the API annotations\n directive: A shorter message that provides contexts for the message and indents it.\n For example, this could be something like 'warning' or 'info'.\n \"\"\"\n if not obj.__doc__:\n obj.__doc__ = \"\"\n\n obj.__doc__ = obj.__doc__.rstrip()\n\n indent = _get_indent(obj.__doc__)\n obj.__doc__ += \"\\n\\n\"\n if directive is not None:\n obj.__doc__ += f\"{' ' * indent}.. {directive}::\\n\"\n obj.__doc__ += f\"{' ' * (indent + 4)}{message}\"\n else:\n obj.__doc__ += f\"{' ' * indent}{message}\"\n obj.__doc__ += f\"\\n{' ' * indent}\"\n\n\ndef _mark_annotated(obj) -> None:\n # Set magic token for check_api_annotations linter.\n if hasattr(obj, \"__name__\"):\n obj._annotated = obj.__name__\n\n\ndef _is_annotated(obj) -> bool:\n # Check the magic token exists and applies to this class (not a subclass).\n return hasattr(obj, \"_annotated\") and obj._annotated == obj.__name__\n\n\ndef _get_indent(docstring: str) -> int:\n \"\"\"\n Example:\n >>> def f():\n ... '''Docstring summary.'''\n >>> f.__doc__\n 'Docstring summary.'\n >>> _get_indent(f.__doc__)\n 0\n >>> def g(foo):\n ... '''Docstring summary.\n ...\n ... Args:\n ... foo: Does bar.\n ... '''\n >>> g.__doc__\n 'Docstring summary.\\\\n\\\\n Args:\\\\n foo: Does bar.\\\\n '\n >>> _get_indent(g.__doc__)\n 4\n >>> class A:\n ... def h():\n ... '''Docstring summary.\n ...\n ... Returns:\n ... None.\n ... '''\n >>> A.h.__doc__\n 'Docstring summary.\\\\n\\\\n Returns:\\\\n None.\\\\n '\n >>> _get_indent(A.h.__doc__)\n 8\n \"\"\"\n if not docstring:\n return 0\n\n non_empty_lines = list(filter(bool, docstring.splitlines()))\n if len(non_empty_lines) == 1:\n # Docstring contains summary only.\n return 0\n\n # The docstring summary isn't indented, so check the indentation of the second non-empty line.\n return len(non_empty_lines[1]) - len(non_empty_lines[1].lstrip())\n"
},
{
"path": "ludwig/automl/__init__.py",
"content": "from ludwig.automl.automl import auto_train # noqa\nfrom ludwig.automl.automl import cli_init_config # noqa\nfrom ludwig.automl.automl import create_auto_config # noqa\nfrom ludwig.automl.automl import train_with_config # noqa; noqa\n"
},
{
"path": "ludwig/automl/auto_tune_config.py",
"content": "import copy\nimport logging\nimport math\nfrom collections import OrderedDict\n\nimport psutil\n\ntry:\n import GPUtil\nexcept ImportError:\n raise ImportError(\"GPUtil is not installed. In order to use auto_train please run pip install ludwig[ray]\")\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.backend import initialize_backend\nfrom ludwig.constants import (\n AUTO,\n AUTOML_DEFAULT_TEXT_ENCODER,\n AUTOML_LARGE_TEXT_DATASET,\n AUTOML_MAX_ROWS_PER_CHECKPOINT,\n AUTOML_SMALLER_TEXT_ENCODER,\n AUTOML_SMALLER_TEXT_LENGTH,\n AUTOML_TEXT_ENCODER_MAX_TOKEN_LEN,\n HYPEROPT,\n MINIMUM_BATCH_SIZE,\n PREPROCESSING,\n SPACE,\n TEXT,\n TRAINER,\n)\nfrom ludwig.data.preprocessing import preprocess_for_training\nfrom ludwig.features.feature_registries import update_config_with_metadata\nfrom ludwig.schema.model_config import ModelConfig\nfrom ludwig.utils.automl.utils import get_model_type\nfrom ludwig.utils.torch_utils import initialize_pytorch\n\nlogger = logging.getLogger(__name__)\n\n# maps variable search space that can be modified to minimum permissible value for the range\nRANKED_MODIFIABLE_PARAM_LIST = {\n \"tabnet\": OrderedDict(\n {\n \"trainer.batch_size\": 32,\n \"combiner.size\": 8,\n \"combiner.output_size\": 8,\n }\n ),\n \"concat\": OrderedDict(\n {\n \"trainer.batch_size\": 32,\n \"combiner.output_size\": 64,\n \"combiner.num_fc_layers\": 1,\n }\n ),\n \"tabtransformer\": OrderedDict(\n {\n \"trainer.batch_size\": 32,\n \"combiner.num_heads:\": 4,\n \"combiner.output_size\": 8,\n \"combiner.num_layers\": 4,\n \"combiner.num_fc_layers\": 1,\n }\n ),\n \"text\": OrderedDict( # for single input feature text models e.g. bert and its variants\n {\n \"trainer.batch_size\": 16,\n }\n ),\n}\n\n\nBYTES_PER_MiB = 1048576\nBYTES_PER_WEIGHT = 4 # assumes 32-bit precision = 4 bytes\nBYTES_OPTIMIZER_PER_WEIGHT = 8 # for optimizer m and v vectors\n\n\ndef get_trainingset_metadata(config, dataset, backend):\n _, _, _, training_set_metadata = preprocess_for_training(\n config, dataset=dataset, preprocessing_params=config[PREPROCESSING], backend=backend\n )\n return training_set_metadata\n\n\n# Note: if run in Ray Cluster, this method is run remote with gpu resources requested if available\ndef _get_machine_memory():\n if GPUtil.getGPUs():\n machine_mem = GPUtil.getGPUs()[0].memoryTotal * BYTES_PER_MiB\n else:\n machine_mem = psutil.virtual_memory().total\n return machine_mem\n\n\ndef _get_text_feature_max_length(config, training_set_metadata) -> int:\n \"\"\"Returns max sequence length over text features, subject to preprocessing limit.\"\"\"\n max_length = 0\n for feature in config[\"input_features\"]:\n if feature[\"type\"] == TEXT:\n feature_max_len = training_set_metadata[feature[\"name\"]][\"max_sequence_length\"]\n if feature_max_len > max_length:\n max_length = feature_max_len\n if (\n (\"preprocessing\" in config)\n and (TEXT in config[\"preprocessing\"])\n and (\"max_sequence_length\" in config[\"preprocessing\"][TEXT])\n ):\n limit = config[\"preprocessing\"][TEXT][\"max_sequence_length\"]\n else:\n limit = 256 # Preprocessing default max_sequence_length = 256\n if max_length > limit + 2: # For start and stop symbols.\n max_length = limit + 2\n return max_length\n\n\ndef _get_text_model_memory_usage(config, training_set_metadata, memory_usage) -> int:\n max_feature_token_length = _get_text_feature_max_length(config, training_set_metadata)\n memory_usage = (memory_usage / AUTOML_TEXT_ENCODER_MAX_TOKEN_LEN) * max_feature_token_length\n return memory_usage\n\n\ndef compute_memory_usage(config_obj, training_set_metadata, model_category) -> int:\n update_config_with_metadata(config_obj, training_set_metadata)\n lm = LudwigModel.create_model(config_obj)\n model_size = lm.get_model_size() # number of parameters in model\n batch_size = config_obj.trainer.batch_size\n if batch_size == AUTO:\n # Smallest valid batch size that will allow training to complete\n batch_size = MINIMUM_BATCH_SIZE\n memory_usage = model_size * (BYTES_PER_WEIGHT + BYTES_OPTIMIZER_PER_WEIGHT) * batch_size\n if model_category == TEXT:\n return _get_text_model_memory_usage(config_obj.to_dict(), training_set_metadata, memory_usage)\n else:\n return memory_usage\n\n\ndef sub_new_params(config: dict, new_param_vals: dict):\n new_config = copy.deepcopy(config)\n for param, val in new_param_vals.items():\n config_section = param.split(\".\")[0]\n param_name = param.split(\".\")[1]\n new_config[config_section][param_name] = val\n return new_config\n\n\ndef get_new_params(current_param_values, hyperparam_search_space, params_to_modify):\n for param, _ in params_to_modify.items():\n if param in hyperparam_search_space:\n if hyperparam_search_space[param][SPACE] == \"choice\":\n current_param_values[param] = hyperparam_search_space[param][\"categories\"][-1]\n else:\n current_param_values[param] = hyperparam_search_space[param][\"upper\"]\n return current_param_values\n\n\ndef _update_text_encoder(input_features: list, old_text_encoder: str, new_text_encoder: str) -> None:\n for feature in input_features:\n if feature[\"type\"] == TEXT and feature[\"encoder\"] == old_text_encoder:\n feature[\"encoder\"] = new_text_encoder\n\n\ndef _get_text_feature_min_usable_length(input_features: list, training_set_metadata) -> int:\n \"\"\"Returns min of AUTOML_SMALLER_TEXT_LENGTH and lowest 99th percentile sequence length over text features.\"\"\"\n min_usable_length = AUTOML_SMALLER_TEXT_LENGTH\n for feature in input_features:\n if feature[\"type\"] == TEXT:\n feature_99ptile_len = training_set_metadata[feature[\"name\"]][\"max_sequence_length_99ptile\"]\n if feature_99ptile_len < min_usable_length:\n min_usable_length = feature_99ptile_len\n return round(min_usable_length)\n\n\ndef reduce_text_feature_max_length(config, training_set_metadata) -> bool:\n \"\"\"Reduce max sequence length, when viable, to control its quadratic impact.\"\"\"\n input_features = config[\"input_features\"]\n min_usable_length = _get_text_feature_min_usable_length(input_features, training_set_metadata)\n seq_len_limit = {\"max_sequence_length\": min_usable_length}\n if \"preprocessing\" not in config:\n config[\"preprocessing\"] = {TEXT: seq_len_limit}\n elif (\n (TEXT not in config[\"preprocessing\"])\n or (\"max_sequence_length\" not in config[\"preprocessing\"][TEXT])\n or (min_usable_length < float(config[\"preprocessing\"][TEXT][\"max_sequence_length\"]))\n ):\n config[\"preprocessing\"][TEXT] = seq_len_limit\n else:\n return False\n return True\n\n\n# For hyperparam_search_space comprised solely of choice spaces, compute maximum number of\n# combinations and return that value if it is less than num_samples; else return num_samples.\ndef _update_num_samples(num_samples, hyperparam_search_space):\n max_num_samples = 1\n for param in hyperparam_search_space.keys():\n if hyperparam_search_space[param][SPACE] == \"choice\":\n max_num_samples *= len(hyperparam_search_space[param][\"categories\"])\n else:\n return num_samples\n if max_num_samples < num_samples:\n return max_num_samples\n return num_samples\n\n\n# Note: if run in Ray Cluster, this method is run remote with gpu resources requested if available\ndef memory_tune_config(config, dataset, model_category, row_count, backend):\n backend = initialize_backend(backend)\n\n fits_in_memory = False\n tried_reduce_seq_len = False\n config_obj = ModelConfig.from_dict(config)\n raw_config = config_obj.to_dict()\n training_set_metadata = get_trainingset_metadata(raw_config, dataset, backend)\n modified_hyperparam_search_space = copy.deepcopy(raw_config[HYPEROPT][\"parameters\"])\n current_param_values = {}\n param_list = []\n model_type = get_model_type(raw_config)\n if model_type in RANKED_MODIFIABLE_PARAM_LIST:\n params_to_modify = RANKED_MODIFIABLE_PARAM_LIST[model_type]\n if len(params_to_modify.keys()) > 0:\n param_list = list(params_to_modify.keys())\n max_memory = _get_machine_memory()\n initialize_pytorch()\n\n while param_list:\n # compute memory utilization\n current_param_values = get_new_params(current_param_values, modified_hyperparam_search_space, params_to_modify)\n temp_config = sub_new_params(raw_config, current_param_values)\n config_obj = ModelConfig.from_dict(temp_config)\n mem_use = compute_memory_usage(config_obj, training_set_metadata, model_category)\n if mem_use > max_memory and model_category == TEXT and not tried_reduce_seq_len:\n tried_reduce_seq_len = True\n if reduce_text_feature_max_length(config, training_set_metadata):\n reduce_text_feature_max_length(temp_config, training_set_metadata)\n config_obj = ModelConfig.from_dict(temp_config)\n mem_use = compute_memory_usage(config_obj, training_set_metadata, model_category)\n logger.info(f\"Checking model estimated mem use {mem_use} against memory size {max_memory}\")\n if mem_use <= max_memory:\n fits_in_memory = True\n break\n # check if we have exhausted tuning of current param (e.g. we can no longer reduce the param value)\n param, min_value = param_list[0], params_to_modify[param_list[0]]\n\n if param in modified_hyperparam_search_space.keys():\n param_space = modified_hyperparam_search_space[param][\"space\"]\n if param_space == \"choice\":\n if (\n len(modified_hyperparam_search_space[param][\"categories\"]) >= 2\n and modified_hyperparam_search_space[param][\"categories\"][-2] >= min_value\n ):\n modified_hyperparam_search_space[param][\"categories\"] = modified_hyperparam_search_space[param][\n \"categories\"\n ][:-1]\n else:\n param_list.pop(0) # exhausted reduction of this parameter\n else:\n # reduce by 10%\n upper_bound, lower_bound = (\n modified_hyperparam_search_space[param][\"upper\"],\n modified_hyperparam_search_space[param][\"lower\"],\n )\n reduction_val = (upper_bound - lower_bound) * 0.1\n new_upper_bound = upper_bound - reduction_val\n if (new_upper_bound) > lower_bound and new_upper_bound > min_value:\n modified_hyperparam_search_space[param][\"upper\"] = new_upper_bound\n else:\n param_list.pop(0) # exhausted reduction of this parameter\n else:\n param_list.pop(0) # param not in hyperopt search space\n\n if model_category == TEXT and row_count > AUTOML_LARGE_TEXT_DATASET:\n if \"checkpoints_per_epoch\" not in config[TRAINER] and \"steps_per_checkpoint\" not in config[TRAINER]:\n checkpoints_per_epoch = max(2, math.floor(row_count / AUTOML_MAX_ROWS_PER_CHECKPOINT))\n config[TRAINER][\n \"checkpoints_per_epoch\"\n ] = checkpoints_per_epoch # decrease latency to get model accuracy signal\n if \"evaluate_training_set\" not in config[TRAINER]:\n config[TRAINER][\"evaluate_training_set\"] = False # reduce overhead for increased evaluation frequency\n if not fits_in_memory:\n # Switch to smaller pre-trained model encoder for large datasets.\n _update_text_encoder(config[\"input_features\"], AUTOML_DEFAULT_TEXT_ENCODER, AUTOML_SMALLER_TEXT_ENCODER)\n\n modified_config = copy.deepcopy(config)\n\n modified_config[HYPEROPT][\"parameters\"] = modified_hyperparam_search_space\n modified_config[HYPEROPT][\"executor\"][\"num_samples\"] = _update_num_samples(\n modified_config[HYPEROPT][\"executor\"][\"num_samples\"], modified_hyperparam_search_space\n )\n return modified_config, fits_in_memory\n"
},
{
"path": "ludwig/automl/automl.py",
"content": "\"\"\"automl.py.\n\nDriver script which:\n\n(1) Builds a base config by performing type inference and populating config\n w/default combiner parameters, training parameters, and hyperopt search space\n(2) Tunes config based on resource constraints\n(3) Runs hyperparameter optimization experiment\n\"\"\"\n\nimport argparse\nimport copy\nimport logging\nimport os\nimport warnings\nfrom typing import Any\n\nimport numpy as np\nimport pandas as pd\nimport yaml\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.api_annotations import PublicAPI\nfrom ludwig.automl.base_config import (\n create_default_config,\n DatasetInfo,\n get_dataset_info,\n get_features_config,\n get_reference_configs,\n)\nfrom ludwig.backend import Backend, initialize_backend\nfrom ludwig.constants import (\n AUTO,\n AUTOML_DEFAULT_IMAGE_ENCODER,\n AUTOML_DEFAULT_TABULAR_MODEL,\n AUTOML_DEFAULT_TEXT_ENCODER,\n BINARY,\n CATEGORY,\n ENCODER,\n HYPEROPT,\n IMAGE,\n INPUT_FEATURES,\n NAME,\n NUMBER,\n OUTPUT_FEATURES,\n TABULAR,\n TEXT,\n TRAINER,\n TYPE,\n)\nfrom ludwig.contrib import add_contrib_callback_args\nfrom ludwig.data.cache.types import CacheableDataset\nfrom ludwig.datasets import load_dataset_uris\nfrom ludwig.globals import LUDWIG_VERSION, MODEL_FILE_NAME\nfrom ludwig.hyperopt.run import hyperopt\nfrom ludwig.schema.model_config import ModelConfig\nfrom ludwig.types import ModelConfigDict\nfrom ludwig.utils.automl.ray_utils import _ray_init\nfrom ludwig.utils.automl.utils import _add_transfer_config, get_model_type, set_output_feature_metric\nfrom ludwig.utils.data_utils import load_dataset, use_credentials\nfrom ludwig.utils.defaults import default_random_seed\nfrom ludwig.utils.fs_utils import open_file\nfrom ludwig.utils.heuristics import get_auto_learning_rate\nfrom ludwig.utils.misc_utils import merge_dict\nfrom ludwig.utils.print_utils import print_ludwig\n\ntry:\n import dask.dataframe as dd\n from ray.tune import ExperimentAnalysis\nexcept ImportError as e:\n raise RuntimeError(\"ray is not installed. In order to use auto_train please run pip install ludwig[ray]\") from e\n\n\nlogger = logging.getLogger(__name__)\n\nOUTPUT_DIR = \".\"\nTABULAR_TYPES = {CATEGORY, NUMBER, BINARY}\n\n\nclass AutoTrainResults:\n def __init__(self, experiment_analysis: ExperimentAnalysis, creds: dict[str, Any] = None):\n self._experiment_analysis = experiment_analysis\n self._creds = creds\n\n @property\n def experiment_analysis(self):\n return self._experiment_analysis\n\n @property\n def best_trial_id(self) -> str:\n return self._experiment_analysis.best_trial.trial_id\n\n @property\n def best_model(self) -> LudwigModel | None:\n checkpoint = self._experiment_analysis.best_checkpoint\n if checkpoint is None:\n logger.warning(\"No best model found\")\n return None\n\n # Use credentials context for remote checkpoints that may need custom auth\n with use_credentials(self._creds):\n with checkpoint.as_directory() as ckpt_path:\n model_dir = os.path.join(ckpt_path, MODEL_FILE_NAME)\n if not os.path.isdir(model_dir):\n logger.warning(\n f\"Best checkpoint does not contain model files at {model_dir}. \"\n \"The trial may not have completed a full training epoch.\"\n )\n return None\n # Ray Tune checkpoints contain training_checkpoints/ (from\n # mid-training saves) but not model_weights (only saved after\n # training completes). Load from the training checkpoint.\n return LudwigModel.load(model_dir, from_checkpoint=True)\n\n\n@PublicAPI\ndef auto_train(\n dataset: str | pd.DataFrame | dd.DataFrame,\n target: str,\n time_limit_s: int | float,\n output_directory: str = OUTPUT_DIR,\n tune_for_memory: bool = False,\n user_config: dict = None,\n random_seed: int = default_random_seed,\n use_reference_config: bool = False,\n **kwargs,\n) -> AutoTrainResults:\n \"\"\"Main auto train API that first builds configs for each model type (e.g. concat, tabnet, transformer). Then\n selects model based on dataset attributes. And finally runs a hyperparameter optimization experiment.\n\n All batch and learning rate tuning is done @ training time.\n\n # Inputs\n :param dataset: (str, pd.DataFrame, dd.DataFrame) data source to train over.\n :param target: (str) name of target feature\n :param time_limit_s: (int, float) total time allocated to auto_train. acts\n as the stopping parameter\n :param output_directory: (str) directory into which to write results, defaults to\n current working directory.\n :param tune_for_memory: (bool) refine hyperopt search space for available\n host / GPU memory\n :param user_config: (dict) override automatic selection of specified config items\n :param random_seed: (int, default: `42`) a random seed that will be used anywhere\n there is a call to a random number generator, including\n hyperparameter search sampling, as well as data splitting,\n parameter initialization and training set shuffling\n :param use_reference_config: (bool) refine hyperopt search space by setting first\n search point from reference model config, if any\n :param kwargs: additional keyword args passed down to `ludwig.hyperopt.run.hyperopt`.\n\n # Returns\n :return: (AutoTrainResults) results containing hyperopt experiments and best model\n \"\"\"\n config = create_auto_config(\n dataset,\n target,\n time_limit_s,\n tune_for_memory,\n user_config,\n random_seed,\n use_reference_config=use_reference_config,\n )\n return train_with_config(dataset, config, output_directory=output_directory, random_seed=random_seed, **kwargs)\n\n\n@PublicAPI\ndef create_auto_config(\n dataset: str | pd.DataFrame | dd.DataFrame | DatasetInfo,\n target: str | list[str],\n time_limit_s: int | float,\n tune_for_memory: bool = False,\n user_config: dict = None,\n random_seed: int = default_random_seed,\n imbalance_threshold: float = 0.9,\n use_reference_config: bool = False,\n backend: Backend | str = None,\n) -> ModelConfigDict:\n \"\"\"Returns an auto-generated Ludwig config with the intent of training the best model on given given dataset /\n target in the given time limit.\n\n # Inputs\n :param dataset: (str, pd.DataFrame, dd.DataFrame, DatasetInfo) data source to train over.\n :param target: (str, List[str]) name of target feature\n :param time_limit_s: (int, float) total time allocated to auto_train. acts\n as the stopping parameter\n :param tune_for_memory: (bool) DEPRECATED refine hyperopt search space for available\n host / GPU memory\n :param user_config: (dict) override automatic selection of specified config items\n :param random_seed: (int, default: `42`) a random seed that will be used anywhere\n there is a call to a random number generator, including\n hyperparameter search sampling, as well as data splitting,\n parameter initialization and training set shuffling\n :param imbalance_threshold: (float) maximum imbalance ratio (minority / majority) to perform stratified sampling\n :param use_reference_config: (bool) refine hyperopt search space by setting first\n search point from reference model config, if any\n\n # Return\n :return: (dict) selected model configuration\n \"\"\"\n backend = initialize_backend(backend)\n\n if not isinstance(dataset, DatasetInfo):\n # preload ludwig datasets\n dataset, _, _, _ = load_dataset_uris(dataset, None, None, None, backend)\n if isinstance(dataset, CacheableDataset):\n dataset = dataset.unwrap()\n dataset = load_dataset(dataset, df_lib=backend.df_engine.df_lib)\n\n dataset_info = get_dataset_info(dataset) if not isinstance(dataset, DatasetInfo) else dataset\n features_config = create_features_config(dataset_info, target)\n return create_automl_config_for_features(\n features_config,\n dataset_info,\n target,\n time_limit_s=time_limit_s,\n user_config=user_config,\n random_seed=random_seed,\n imbalance_threshold=imbalance_threshold,\n use_reference_config=use_reference_config,\n backend=backend,\n )\n\n\n@PublicAPI\ndef create_automl_config_for_features(\n features_config: ModelConfigDict,\n dataset_info: DatasetInfo,\n target: str | list[str],\n time_limit_s: int | float,\n tune_for_memory: bool = False,\n user_config: dict = None,\n random_seed: int = default_random_seed,\n imbalance_threshold: float = 0.9,\n use_reference_config: bool = False,\n backend: Backend | str = None,\n) -> ModelConfigDict:\n default_configs = create_default_config(\n features_config, dataset_info, target, time_limit_s, random_seed, imbalance_threshold, backend\n )\n model_config, _, _ = _model_select(dataset_info, default_configs, user_config, use_reference_config)\n\n if tune_for_memory:\n warnings.warn(\"`tune_for_memory=True` is deprecated, `batch_size=auto` will be used instead\")\n\n return model_config\n\n\n@PublicAPI\ndef create_features_config(\n dataset_info: DatasetInfo,\n target_name: str | list[str] = None,\n) -> ModelConfigDict:\n return get_features_config(dataset_info.fields, dataset_info.row_count, target_name)\n\n\n@PublicAPI\ndef train_with_config(\n dataset: str | pd.DataFrame | dd.DataFrame,\n config: dict,\n output_directory: str = OUTPUT_DIR,\n random_seed: int = default_random_seed,\n **kwargs,\n) -> AutoTrainResults:\n \"\"\"Performs hyperparameter optimization with respect to the given config and selects the best model.\n\n # Inputs\n :param dataset: (str) filepath to dataset.\n :param config: (dict) optional Ludwig configuration to use for training, defaults\n to `create_auto_config`.\n :param output_directory: (str) directory into which to write results, defaults to\n current working directory.\n :param random_seed: (int, default: `42`) a random seed that will be used anywhere\n there is a call to a random number generator, including\n hyperparameter search sampling, as well as data splitting,\n parameter initialization and training set shuffling\n :param kwargs: additional keyword args passed down to `ludwig.hyperopt.run.hyperopt`.\n\n # Returns\n :return: (AutoTrainResults) results containing hyperopt experiments and best model\n \"\"\"\n _ray_init()\n\n model_type = get_model_type(config)\n hyperopt_results = _train(\n config, dataset, output_directory=output_directory, model_name=model_type, random_seed=random_seed, **kwargs\n )\n # catch edge case where metric_score is nan\n # TODO (ASN): Decide how we want to proceed if at least one trial has\n # completed\n for trial in hyperopt_results.ordered_trials:\n if isinstance(trial.metric_score, str) or np.isnan(trial.metric_score):\n warnings.warn(\n \"There was an error running the experiment. \"\n \"A trial failed to start. \"\n \"Consider increasing the time budget for experiment. \"\n )\n\n # Extract credentials needed to pull artifacts, if provided\n creds = None\n backend: Backend = initialize_backend(kwargs.get(\"backend\"))\n if backend is not None:\n creds = backend.storage.artifacts.credentials\n\n experiment_analysis = hyperopt_results.experiment_analysis\n return AutoTrainResults(experiment_analysis, creds)\n\n\ndef _model_select(\n dataset_info: DatasetInfo,\n default_configs,\n user_config,\n use_reference_config: bool,\n):\n \"\"\"Performs model selection based on dataset or user specified model.\n\n Note: Current implementation returns tabnet by default for tabular datasets.\n \"\"\"\n fields = dataset_info.fields\n\n base_config = copy.deepcopy(default_configs[\"base_config\"])\n model_category = None\n\n input_features = default_configs[\"base_config\"][\"input_features\"]\n\n # tabular dataset heuristics\n if len(fields) > 3 and all(f[TYPE] in TABULAR_TYPES for f in input_features):\n model_category = TABULAR\n base_config = merge_dict(base_config, default_configs[\"combiner\"][AUTOML_DEFAULT_TABULAR_MODEL])\n\n # override combiner heuristic if explicitly provided by user\n if user_config is not None:\n if \"combiner\" in user_config.keys():\n model_type = user_config[\"combiner\"][\"type\"]\n base_config = merge_dict(base_config, default_configs[\"combiner\"][model_type])\n else:\n # text heuristics\n for i, input_feature in enumerate(input_features):\n base_config_input_feature = base_config[\"input_features\"][i]\n # default text encoder is bert\n if input_feature[TYPE] == TEXT:\n model_category = TEXT\n if ENCODER in input_feature:\n base_config_input_feature[ENCODER][TYPE] = AUTOML_DEFAULT_TEXT_ENCODER\n else:\n base_config_input_feature[ENCODER] = {TYPE: AUTOML_DEFAULT_TEXT_ENCODER}\n # TODO(shreya): Should this hyperopt config param be set here?\n base_config[HYPEROPT][\"executor\"][\"num_samples\"] = 5 # set for small hyperparameter search space\n base_config = merge_dict(base_config, default_configs[TEXT][AUTOML_DEFAULT_TEXT_ENCODER])\n\n # TODO (ASN): add image heuristics\n if input_feature[TYPE] == IMAGE:\n model_category = IMAGE\n if ENCODER in input_feature:\n base_config_input_feature[ENCODER][TYPE] = AUTOML_DEFAULT_IMAGE_ENCODER\n else:\n base_config_input_feature[ENCODER] = {TYPE: AUTOML_DEFAULT_IMAGE_ENCODER}\n\n # Merge combiner config\n base_config = merge_dict(base_config, default_configs[\"combiner\"][\"concat\"])\n\n # Adjust learning rate based on other config settings\n if base_config[TRAINER][\"learning_rate\"] == AUTO:\n # Add a fake output feature to ensure we can load the ModelConfig, as we expect there to be at least\n # one output feature in all cases\n # TODO(travis): less hacky way to do this, we should probably allow ModelConfig to be created without output\n # features\n load_config = copy.deepcopy(base_config)\n if not load_config.get(OUTPUT_FEATURES):\n load_config[OUTPUT_FEATURES] = [{\"name\": \"fake\", \"type\": \"binary\"}]\n base_config[TRAINER][\"learning_rate\"] = get_auto_learning_rate(ModelConfig.from_dict(load_config))\n\n # override and constrain automl config based on user specified values\n if user_config is not None:\n base_config = merge_dict(base_config, user_config)\n\n # remove all parameters from hyperparameter search that user has\n # provided explicit values for\n hyperopt_params = copy.deepcopy(base_config[\"hyperopt\"][\"parameters\"])\n for hyperopt_params in hyperopt_params.keys():\n config_section, param = hyperopt_params.split(\".\")[0], hyperopt_params.split(\".\")[1]\n if config_section in user_config.keys():\n if param in user_config[config_section]:\n del base_config[\"hyperopt\"][\"parameters\"][hyperopt_params]\n\n # if single output feature, set relevant metric and goal if not already set\n base_config = set_output_feature_metric(base_config)\n\n # add as initial trial in the automl search the hyperparameter settings from\n # the best model for a similar dataset and matching model type, if any.\n if use_reference_config:\n ref_configs = get_reference_configs()\n base_config = _add_transfer_config(base_config, ref_configs)\n\n return base_config, model_category, dataset_info.row_count\n\n\ndef _train(\n config: dict,\n dataset: str | pd.DataFrame | dd.DataFrame,\n output_directory: str,\n model_name: str,\n random_seed: int,\n **kwargs,\n):\n hyperopt_results = hyperopt(\n config,\n dataset=dataset,\n output_directory=output_directory,\n model_name=model_name,\n random_seed=random_seed,\n skip_save_log=True, # avoid per-step log overhead by default\n **kwargs,\n )\n return hyperopt_results\n\n\ndef init_config(\n dataset: str,\n target: str | list[str],\n time_limit_s: int | float,\n tune_for_memory: bool = False,\n suggested: bool = False,\n hyperopt: bool = False,\n output: str = None,\n random_seed: int = default_random_seed,\n use_reference_config: bool = False,\n **kwargs,\n):\n config = create_auto_config(\n dataset=dataset,\n target=target,\n time_limit_s=time_limit_s,\n random_seed=random_seed,\n use_reference_config=use_reference_config,\n tune_for_memory=tune_for_memory,\n )\n\n if HYPEROPT in config and not hyperopt:\n del config[HYPEROPT]\n\n if not suggested:\n # Only use inputs and outputs\n minimal_config = {\n INPUT_FEATURES: [{\"name\": f[NAME], \"type\": f[TYPE]} for f in config[INPUT_FEATURES]],\n OUTPUT_FEATURES: [{\"name\": f[NAME], \"type\": f[TYPE]} for f in config[OUTPUT_FEATURES]],\n }\n if hyperopt:\n minimal_config[HYPEROPT] = config[HYPEROPT]\n config = minimal_config\n\n if output is None:\n print(yaml.safe_dump(config, None, sort_keys=False))\n else:\n with open_file(output, \"w\") as f:\n yaml.safe_dump(config, f, sort_keys=False)\n\n\ndef cli_init_config(sys_argv):\n parser = argparse.ArgumentParser(\n description=\"This script initializes a valid config from a dataset.\",\n prog=\"ludwig init_config\",\n usage=\"%(prog)s [options]\",\n )\n parser.add_argument(\n \"-d\",\n \"--dataset\",\n type=str,\n help=\"input data file path\",\n )\n parser.add_argument(\n \"-t\",\n \"--target\",\n type=str,\n help=\"target(s) to predict as output features of the model\",\n action=\"append\",\n required=False,\n )\n parser.add_argument(\n \"--time_limit_s\",\n type=int,\n help=\"time limit to train the model in seconds when using hyperopt\",\n required=False,\n )\n parser.add_argument(\n \"--suggested\",\n type=bool,\n help=\"use suggested config from automl, otherwise only use inferred types and return a minimal config\",\n default=False,\n required=False,\n )\n parser.add_argument(\n \"--hyperopt\",\n type=bool,\n help=\"include automl hyperopt config\",\n default=False,\n required=False,\n )\n parser.add_argument(\n \"--random_seed\",\n type=int,\n help=\"seed for random number generators used in hyperopt to improve repeatability\",\n required=False,\n )\n parser.add_argument(\n \"--use_reference_config\",\n type=bool,\n help=\"refine hyperopt search space by setting first search point from stored reference model config\",\n default=False,\n required=False,\n )\n parser.add_argument(\n \"-o\",\n \"--output\",\n type=str,\n help=\"output initialized YAML config path\",\n required=False,\n )\n\n add_contrib_callback_args(parser)\n args = parser.parse_args(sys_argv)\n\n args.callbacks = args.callbacks or []\n for callback in args.callbacks:\n callback.on_cmdline(\"init_config\", *sys_argv)\n\n print_ludwig(\"Init Config\", LUDWIG_VERSION)\n init_config(**vars(args))\n"
},
{
"path": "ludwig/automl/base_config.py",
"content": "\"\"\"Uses heuristics to build ludwig configuration file:\n\n(1) infer types based on dataset\n(2) populate with\n - default combiner parameters,\n - preprocessing parameters,\n - combiner specific default training parameters,\n - combiner specific hyperopt space\n - feature parameters\n(3) add machineresources\n (base implementation -- # CPU, # GPU)\n\"\"\"\n\nimport logging\nimport os\nfrom dataclasses import dataclass\nfrom typing import Any\n\nimport dask.dataframe as dd\nimport numpy as np\nimport pandas as pd\nimport yaml\nfrom dataclasses_json import dataclass_json, LetterCase\nfrom tqdm import tqdm\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.backend import Backend\nfrom ludwig.constants import (\n COLUMN,\n COMBINER,\n ENCODER,\n EXECUTOR,\n HYPEROPT,\n INPUT_FEATURES,\n PREPROCESSING,\n SCHEDULER,\n SEARCH_ALG,\n SPLIT,\n TEXT,\n TYPE,\n)\nfrom ludwig.types import ModelConfigDict\nfrom ludwig.utils.automl.data_source import DataSource, wrap_data_source\nfrom ludwig.utils.automl.field_info import FieldConfig, FieldInfo, FieldMetadata\nfrom ludwig.utils.automl.type_inference import infer_type, should_exclude\nfrom ludwig.utils.data_utils import load_yaml\nfrom ludwig.utils.misc_utils import merge_dict\nfrom ludwig.utils.system_utils import Resources\n\nlogger = logging.getLogger(__name__)\n\nPATH_HERE = os.path.abspath(os.path.dirname(__file__))\nCONFIG_DIR = os.path.join(PATH_HERE, \"defaults\")\n\nBASE_AUTOML_CONFIG = os.path.join(CONFIG_DIR, \"base_automl_config.yaml\")\nREFERENCE_CONFIGS = os.path.join(CONFIG_DIR, \"reference_configs.yaml\")\n\ncombiner_defaults = {\n \"concat\": os.path.join(CONFIG_DIR, \"combiner/concat_config.yaml\"),\n \"tabnet\": os.path.join(CONFIG_DIR, \"combiner/tabnet_config.yaml\"),\n \"transformer\": os.path.join(CONFIG_DIR, \"combiner/transformer_config.yaml\"),\n}\n\nencoder_defaults = {\"text\": {\"bert\": os.path.join(CONFIG_DIR, \"text/bert_config.yaml\")}}\n\n# Cap for number of distinct values to return.\nMAX_DISTINCT_VALUES_TO_RETURN = 10\n\n\n@DeveloperAPI\n@dataclass_json(letter_case=LetterCase.CAMEL)\n@dataclass\nclass DatasetInfo:\n fields: list[FieldInfo]\n row_count: int\n size_bytes: int = -1\n\n\ndef allocate_experiment_resources(resources: Resources) -> dict:\n \"\"\"Allocates ray trial resources based on available resources.\n\n # Inputs :param resources (dict) specifies all available GPUs, CPUs and associated metadata of the machines\n (i.e. memory)\n\n # Return\n :return: (dict) gpu and cpu resources per trial\n \"\"\"\n # TODO (ASN):\n # (1) expand logic to support multiple GPUs per trial (multi-gpu training)\n # (2) add support for kubernetes namespace (if applicable)\n # (3) add support for smarter allocation based on size of GPU memory\n experiment_resources = {\"cpu_resources_per_trial\": 1}\n gpu_count, cpu_count = resources.gpus, resources.cpus\n if gpu_count > 0:\n experiment_resources.update({\"gpu_resources_per_trial\": 1})\n if cpu_count > 1:\n cpus_per_trial = max(int(cpu_count / gpu_count), 1)\n experiment_resources[\"cpu_resources_per_trial\"] = cpus_per_trial\n\n return experiment_resources\n\n\ndef get_resource_aware_hyperopt_config(\n experiment_resources: dict[str, Any], time_limit_s: int | float, random_seed: int\n) -> dict[str, Any]:\n \"\"\"Returns a Ludwig config with the hyperopt section populated with appropriate parameters.\n\n Hyperopt parameters are intended to be appropriate for the given resources and time limit.\n \"\"\"\n executor = experiment_resources\n executor.update({\"time_budget_s\": time_limit_s})\n if time_limit_s is not None:\n executor.update({SCHEDULER: {\"max_t\": time_limit_s}})\n\n return {\n HYPEROPT: {\n SEARCH_ALG: {\"random_state_seed\": random_seed},\n EXECUTOR: executor,\n },\n }\n\n\ndef _get_stratify_split_config(field_meta: FieldMetadata) -> dict:\n return {\n PREPROCESSING: {\n SPLIT: {\n TYPE: \"stratify\",\n COLUMN: field_meta.name,\n }\n }\n }\n\n\ndef get_default_automl_hyperopt() -> dict[str, Any]:\n \"\"\"Returns general, default settings for hyperopt.\n\n For example:\n - We set a random_state_seed for sample sequence repeatability\n - We use an increased reduction_factor to get more pruning/exploration.\n\n TODO: If settings seem reasonable, consider building this into the hyperopt schema, directly.\n \"\"\"\n return yaml.safe_load(\"\"\"\n search_alg:\n type: variant_generator\n executor:\n type: ray\n num_samples: 10\n time_budget_s: 3600\n scheduler:\n type: async_hyperband\n time_attr: time_total_s\n max_t: 3600\n grace_period: 72\n reduction_factor: 5\n\"\"\")\n\n\ndef create_default_config(\n features_config: ModelConfigDict,\n dataset_info: DatasetInfo,\n target_name: str | list[str],\n time_limit_s: int | float,\n random_seed: int,\n imbalance_threshold: float = 0.9,\n backend: Backend = None,\n) -> dict:\n \"\"\"Returns auto_train configs for three available combiner models. Coordinates the following tasks:\n\n - extracts fields and generates list of FieldInfo objects\n - gets field metadata (i.e avg. words, total non-null entries)\n - builds input_features and output_features section of config\n - for imbalanced datasets, a preprocessing section is added to perform stratified sampling if the imbalance ratio\n is smaller than imbalance_threshold\n - for each combiner, adds default training, hyperopt\n - infers resource constraints and adds gpu and cpu resource allocation per\n trial\n\n # Inputs\n :param dataset_info: (str) filepath Dataset Info object.\n :param target_name: (str, List[str]) name of target feature\n :param time_limit_s: (int, float) total time allocated to auto_train. acts\n as the stopping parameter\n :param random_seed: (int, default: `42`) a random seed that will be used anywhere\n there is a call to a random number generator, including\n hyperparameter search sampling, as well as data splitting,\n parameter initialization and training set shuffling\n :param imbalance_threshold: (float) maximum imbalance ratio (minority / majority) to perform stratified sampling\n :param backend: (Backend) backend to use for training.\n\n # Return\n :return: (dict) dictionaries contain auto train config files for all available\n combiner types\n \"\"\"\n base_automl_config = load_yaml(BASE_AUTOML_CONFIG)\n base_automl_config.update(features_config)\n\n targets = convert_targets(target_name)\n features_metadata = get_field_metadata(dataset_info.fields, dataset_info.row_count, targets)\n\n # Handle expensive features for CPU\n resources = backend.get_available_resources()\n for ifeature in base_automl_config[INPUT_FEATURES]:\n if resources.gpus == 0:\n if ifeature[TYPE] == TEXT:\n # When no GPUs are available, default to the embed encoder, which is fast enough for CPU\n ifeature[ENCODER] = {\"type\": \"embed\"}\n\n # create set of all feature types appearing in the dataset\n feature_types = [[feat[TYPE] for feat in features] for features in features_config.values()]\n feature_types = set(sum(feature_types, []))\n\n model_configs = {}\n\n # update hyperopt config\n experiment_resources = allocate_experiment_resources(resources)\n base_automl_config = merge_dict(\n base_automl_config, get_resource_aware_hyperopt_config(experiment_resources, time_limit_s, random_seed)\n )\n\n # add preprocessing section if single output feature is imbalanced\n outputs_metadata = [f for f in features_metadata if f.mode == \"output\"]\n if len(outputs_metadata) == 1:\n of_meta = outputs_metadata[0]\n is_categorical = of_meta.config.type in [\"category\", \"binary\"]\n is_imbalanced = of_meta.imbalance_ratio < imbalance_threshold\n if is_categorical and is_imbalanced:\n base_automl_config.update(_get_stratify_split_config(of_meta))\n\n model_configs[\"base_config\"] = base_automl_config\n\n # read in all encoder configs\n for feat_type, default_configs in encoder_defaults.items():\n if feat_type in feature_types:\n if feat_type not in model_configs.keys():\n model_configs[feat_type] = {}\n for encoder_name, encoder_config_path in default_configs.items():\n model_configs[feat_type][encoder_name] = load_yaml(encoder_config_path)\n\n # read in all combiner configs\n model_configs[COMBINER] = {}\n for combiner_type, default_config in combiner_defaults.items():\n combiner_config = load_yaml(default_config)\n model_configs[COMBINER][combiner_type] = combiner_config\n\n return model_configs\n\n\n# Read in the score and configuration of a reference model trained by Ludwig for each dataset in a list.\ndef get_reference_configs() -> dict:\n reference_configs = load_yaml(REFERENCE_CONFIGS)\n return reference_configs\n\n\ndef get_dataset_info(df: pd.DataFrame | dd.DataFrame) -> DatasetInfo:\n \"\"\"Constructs FieldInfo objects for each feature in dataset. These objects are used for downstream type\n inference.\n\n # Inputs\n :param df: (Union[pd.DataFrame, dd.DataFrame]) Pandas or Dask dataframe. # Return\n :return: (DatasetInfo) Structure containing list of FieldInfo objects.\n \"\"\"\n source = wrap_data_source(df)\n return get_dataset_info_from_source(source)\n\n\ndef is_field_boolean(source: DataSource, field: str) -> bool:\n \"\"\"Returns a boolean indicating whether the object field should have a bool dtype.\n\n Columns with object dtype that have 3 distinct values of which one is Nan/None is a bool type column.\n \"\"\"\n unique_values = source.df[field].unique()\n if len(unique_values) <= 3:\n for entry in unique_values:\n try:\n if np.isnan(entry):\n continue\n except TypeError:\n # For some field types such as object arrays, np.isnan throws a TypeError\n # In this case, do nothing and proceed to checking if the entry is a bool object\n pass\n if isinstance(entry, bool):\n continue\n return False\n return True\n return False\n\n\n@DeveloperAPI\ndef get_dataset_info_from_source(source: DataSource) -> DatasetInfo:\n \"\"\"Constructs FieldInfo objects for each feature in dataset. These objects are used for downstream type\n inference.\n\n # Inputs\n :param source: (DataSource) A wrapper around a data source, which may represent a pandas or Dask dataframe. # Return\n :return: (DatasetInfo) Structure containing list of FieldInfo objects.\n \"\"\"\n row_count = len(source)\n fields = []\n for field in tqdm(source.columns, desc=\"Analyzing fields\", total=len(source.columns)):\n logger.info(f\"Analyzing field: {field}\")\n dtype = source.get_dtype(field)\n num_distinct_values, distinct_values, distinct_values_balance = source.get_distinct_values(\n field, MAX_DISTINCT_VALUES_TO_RETURN\n )\n nonnull_values = source.get_nonnull_values(field)\n image_values = source.get_image_values(field)\n audio_values = source.get_audio_values(field)\n\n if dtype == \"object\":\n # Check if it is a nullboolean field. We do this since if you read a csv with\n # pandas that has a column of booleans and some missing values, the column is\n # interpreted as object dtype instead of bool\n if is_field_boolean(source, field):\n dtype = \"bool\"\n\n avg_words = None\n if source.is_string_type(dtype):\n try:\n avg_words = source.get_avg_num_tokens(field)\n except AttributeError:\n # Series is not actually a string type despite being an object, e.g., Decimal, Datetime, etc.\n avg_words = None\n\n fields.append(\n FieldInfo(\n name=field,\n dtype=dtype,\n distinct_values=distinct_values,\n num_distinct_values=num_distinct_values,\n distinct_values_balance=distinct_values_balance,\n nonnull_values=nonnull_values,\n image_values=image_values,\n audio_values=audio_values,\n avg_words=avg_words,\n )\n )\n return DatasetInfo(fields=fields, row_count=row_count, size_bytes=source.size_bytes())\n\n\ndef get_features_config(\n fields: list[FieldInfo],\n row_count: int,\n target_name: str | list[str] = None,\n) -> dict:\n \"\"\"Constructs FieldInfo objects for each feature in dataset. These objects are used for downstream type\n inference.\n\n # Inputs\n :param fields: (List[FieldInfo]) FieldInfo objects for all fields in dataset\n :param row_count: (int) total number of entries in original dataset :param target_name (str, List[str]) name of\n target feature # Return\n :return: (dict) section of auto_train config for input_features and output_features\n \"\"\"\n targets = convert_targets(target_name)\n metadata = get_field_metadata(fields, row_count, targets)\n return get_config_from_metadata(metadata, targets)\n\n\ndef convert_targets(target_name: str | list[str] = None) -> set[str]:\n targets = target_name\n if isinstance(targets, str):\n targets = [targets]\n if targets is None:\n targets = []\n return set(targets)\n\n\ndef get_config_from_metadata(metadata: list[FieldMetadata], targets: set[str] = None) -> dict:\n \"\"\"Builds input/output feature sections of auto-train config using field metadata.\n\n # Inputs\n :param metadata: (List[FieldMetadata]) field descriptions :param targets (Set[str]) names of target features #\n Return\n :return: (dict) section of auto_train config for input_features and output_features\n \"\"\"\n config = {\n \"input_features\": [],\n \"output_features\": [],\n }\n\n for field_meta in metadata:\n if field_meta.name in targets:\n config[\"output_features\"].append(field_meta.config.to_dict())\n elif not field_meta.excluded and field_meta.mode == \"input\":\n config[\"input_features\"].append(field_meta.config.to_dict())\n\n return config\n\n\n@DeveloperAPI\ndef get_field_metadata(fields: list[FieldInfo], row_count: int, targets: set[str] = None) -> list[FieldMetadata]:\n \"\"\"Computes metadata for each field in dataset.\n\n # Inputs\n :param fields: (List[FieldInfo]) FieldInfo objects for all fields in dataset\n :param row_count: (int) total number of entries in original dataset :param targets (Set[str]) names of target\n features # Return\n :return: (List[FieldMetadata]) list of objects containing metadata for each field\n \"\"\"\n\n metadata = []\n column_count = len(fields)\n for idx, field in enumerate(fields):\n missing_value_percent = 1 - float(field.nonnull_values) / row_count\n dtype = infer_type(field, missing_value_percent, row_count)\n metadata.append(\n FieldMetadata(\n name=field.name,\n config=FieldConfig(\n name=field.name,\n column=field.name,\n type=dtype,\n ),\n excluded=should_exclude(idx, field, dtype, column_count, row_count, targets),\n mode=infer_mode(field, targets),\n missing_values=missing_value_percent,\n imbalance_ratio=field.distinct_values_balance,\n )\n )\n\n return metadata\n\n\ndef infer_mode(field: FieldInfo, targets: set[str] = None) -> str:\n if field.name in targets:\n return \"output\"\n if field.name.lower() == \"split\":\n return \"split\"\n return \"input\"\n"
},
{
"path": "ludwig/automl/defaults/base_automl_config.yaml",
"content": "trainer:\n batch_size: auto #256\n learning_rate: auto #.001\n # validation_metric: accuracy\n\nhyperopt:\n search_alg:\n # Gives results like default + supports random_state_seed for sample sequence repeatability\n type: variant_generator\n executor:\n type: ray\n num_samples: 10\n time_budget_s: 7200\n scheduler:\n type: async_hyperband\n time_attr: time_total_s\n max_t: 7200\n grace_period: 72\n # Increased over default to get more pruning/exploration\n reduction_factor: 5\n"
},
{
"path": "ludwig/automl/defaults/combiner/concat_config.yaml",
"content": "combiner:\n type: concat\n\nhyperopt:\n # goal: maximize\n parameters:\n combiner.num_fc_layers:\n space: randint\n lower: 1\n upper: 4\n combiner.output_size:\n space: choice\n categories: [128, 256]\n combiner.dropout:\n space: uniform\n lower: 0.0\n upper: 0.1\n # This needs to be loguniform due to invalid schemas created by merging with a choice parameter space. See the\n # comment in ludwig/automl/defaults/text/bert_config.yaml for more information.\n trainer.learning_rate:\n space: loguniform\n lower: 0.00002\n upper: 0.001\n trainer.batch_size:\n space: choice\n categories: [64, 128, 256, 512, 1024]\n"
},
{
"path": "ludwig/automl/defaults/combiner/tabnet_config.yaml",
"content": "combiner:\n type: tabnet\n\ntrainer:\n batch_size: auto\n learning_rate_scaling: sqrt\n learning_rate_scheduler:\n decay: exponential\n decay_steps: 20000\n decay_rate: 0.8\n optimizer:\n type: adam\n\nhyperopt:\n parameters:\n trainer.learning_rate:\n space: loguniform\n lower: 0.00002\n upper: 0.001\n trainer.learning_rate_scheduler.decay_rate:\n space: choice\n categories: [0.8, 0.9, 0.95]\n trainer.learning_rate_scheduler.decay_steps:\n space: choice\n categories: [500, 2000, 8000, 10000, 20000]\n combiner.size:\n space: choice\n categories: [8, 16, 24, 32, 64]\n combiner.output_size:\n space: choice\n categories: [8, 16, 24, 32, 64, 128]\n combiner.num_steps:\n space: choice\n categories: [3, 4, 5, 6, 7, 8, 9, 10]\n combiner.relaxation_factor:\n space: choice\n categories: [1.0, 1.2, 1.5, 2.0]\n combiner.sparsity:\n space: choice\n categories: [0.0, 0.000001, 0.0001, 0.001, 0.01, 0.1]\n combiner.bn_virtual_bs:\n space: choice\n categories: [256, 512, 1024, 2048, 4096]\n combiner.bn_momentum:\n space: choice\n categories: [0.4, 0.3, 0.2, 0.1, 0.05, 0.02]\n"
},
{
"path": "ludwig/automl/defaults/combiner/transformer_config.yaml",
"content": "combiner:\n type: transformer\n\ntrainer:\n batch_size: auto #256\n learning_rate: auto #0.0001\n # validation_metric: accuracy\n\nhyperopt:\n # goal: maximize\n parameters:\n trainer.learning_rate:\n space: loguniform\n lower: 0.00002\n upper: 0.001\n trainer.batch_size:\n space: choice\n categories: [64, 128, 256]\n combiner.num_heads:\n space: choice\n categories: [4]\n combiner.dropout:\n space: uniform\n lower: 0.1\n upper: 0.3\n combiner.num_layers:\n space: randint\n lower: 3\n upper: 4\n combiner.num_fc_layers:\n space: choice\n categories: [1, 2]\n combiner.fc_dropout:\n space: uniform\n lower: 0.1\n upper: 0.5\n"
},
{
"path": "ludwig/automl/defaults/reference_configs.yaml",
"content": "# Record the score and configuration of a reference model trained by Ludwig for specified datasets.\n# This information is useful for Ludwig AutoML hyperparameter transfer learning or for manual experimentation.\ndatasets:\n - name: adult_census_income\n goal: maximize\n metric: accuracy\n validation_metric_score: 0.8682432174682617\n training_rows: 29305\n test_rows: 16281\n validation_rows: 3256\n config:\n output_features:\n - name: income\n type: category\n input_features:\n - name: age\n type: number\n - name: workclass\n type: category\n - name: fnlwgt\n type: number\n - name: education\n type: category\n - name: education-num\n type: number\n - name: marital-status\n type: category\n - name: occupation\n type: category\n - name: relationship\n type: category\n - name: race\n type: category\n - name: sex\n type: category\n - name: capital-gain\n type: number\n - name: capital-loss\n type: number\n - name: hours-per-week\n type: number\n - name: native-country\n type: category\n combiner:\n type: tabnet\n size: 8 # N_a\n output_size: 128 # N_d\n sparsity: 0.0 # lambda_sparse\n bn_momentum: 0.4 # m_B\n num_steps: 3 # N_steps\n relaxation_factor: 1.0 # gamma\n bn_virtual_bs: 4096 # B_v\n trainer:\n batch_size: 256 # B\n eval_batch_size: null # 65536 131072 262144 524288\n epochs: 300\n early_stop: 30\n learning_rate: 0.01\n optimizer:\n type: adam\n learning_rate_scheduler:\n decay: exponential\n decay_steps: 500\n decay_rate: 0.95\n validation_metric: accuracy\n - name: allstate_claims_severity\n goal: minimize\n metric: root_mean_squared_error\n validation_metric_score: 1915.5531005859375\n training_rows: 131726\n test_rows: 37750\n validation_rows: 18842\n config:\n output_features:\n - name: loss\n type: number\n input_features:\n - column: cat1\n name: cat1\n type: category\n - column: cat2\n name: cat2\n type: category\n - column: cat3\n name: cat3\n type: category\n - column: cat4\n name: cat4\n type: category\n - column: cat5\n name: cat5\n type: category\n - column: cat6\n name: cat6\n type: category\n - column: cat7\n name: cat7\n type: category\n - column: cat8\n name: cat8\n type: category\n - column: cat9\n name: cat9\n type: category\n - column: cat10\n name: cat10\n type: category\n - column: cat11\n name: cat11\n type: category\n - column: cat12\n name: cat12\n type: category\n - column: cat13\n name: cat13\n type: category\n - column: cat14\n name: cat14\n type: category\n - column: cat15\n name: cat15\n type: category\n - column: cat16\n name: cat16\n type: category\n - column: cat17\n name: cat17\n type: category\n - column: cat18\n name: cat18\n type: category\n - column: cat19\n name: cat19\n type: category\n - column: cat20\n name: cat20\n type: category\n - column: cat21\n name: cat21\n type: category\n - column: cat22\n name: cat22\n type: category\n - column: cat23\n name: cat23\n type: category\n - column: cat24\n name: cat24\n type: category\n - column: cat25\n name: cat25\n type: category\n - column: cat26\n name: cat26\n type: category\n - column: cat27\n name: cat27\n type: category\n - column: cat28\n name: cat28\n type: category\n - column: cat29\n name: cat29\n type: category\n - column: cat30\n name: cat30\n type: category\n - column: cat31\n name: cat31\n type: category\n - column: cat32\n name: cat32\n type: category\n - column: cat33\n name: cat33\n type: category\n - column: cat34\n name: cat34\n type: category\n - column: cat35\n name: cat35\n type: category\n - column: cat36\n name: cat36\n type: category\n - column: cat37\n name: cat37\n type: category\n - column: cat38\n name: cat38\n type: category\n - column: cat39\n name: cat39\n type: category\n - column: cat40\n name: cat40\n type: category\n - column: cat41\n name: cat41\n type: category\n - column: cat42\n name: cat42\n type: category\n - column: cat43\n name: cat43\n type: category\n - column: cat44\n name: cat44\n type: category\n - column: cat45\n name: cat45\n type: category\n - column: cat46\n name: cat46\n type: category\n - column: cat47\n name: cat47\n type: category\n - column: cat48\n name: cat48\n type: category\n - column: cat49\n name: cat49\n type: category\n - column: cat50\n name: cat50\n type: category\n - column: cat51\n name: cat51\n type: category\n - column: cat52\n name: cat52\n type: category\n - column: cat53\n name: cat53\n type: category\n - column: cat54\n name: cat54\n type: category\n - column: cat55\n name: cat55\n type: category\n - column: cat56\n name: cat56\n type: category\n - column: cat57\n name: cat57\n type: category\n - column: cat58\n name: cat58\n type: category\n - column: cat59\n name: cat59\n type: category\n - column: cat60\n name: cat60\n type: category\n - column: cat61\n name: cat61\n type: category\n - column: cat62\n name: cat62\n type: category\n - column: cat63\n name: cat63\n type: category\n - column: cat64\n name: cat64\n type: category\n - column: cat65\n name: cat65\n type: category\n - column: cat66\n name: cat66\n type: category\n - column: cat67\n name: cat67\n type: category\n - column: cat68\n name: cat68\n type: category\n - column: cat69\n name: cat69\n type: category\n - column: cat70\n name: cat70\n type: category\n - column: cat71\n name: cat71\n type: category\n - column: cat72\n name: cat72\n type: category\n - column: cat73\n name: cat73\n type: category\n - column: cat74\n name: cat74\n type: category\n - column: cat75\n name: cat75\n type: category\n - column: cat76\n name: cat76\n type: category\n - column: cat77\n name: cat77\n type: category\n - column: cat78\n name: cat78\n type: category\n - column: cat79\n name: cat79\n type: category\n - column: cat80\n name: cat80\n type: category\n - column: cat81\n name: cat81\n type: category\n - column: cat82\n name: cat82\n type: category\n - column: cat83\n name: cat83\n type: category\n - column: cat84\n name: cat84\n type: category\n - column: cat85\n name: cat85\n type: category\n - column: cat86\n name: cat86\n type: category\n - column: cat87\n name: cat87\n type: category\n - column: cat88\n name: cat88\n type: category\n - column: cat89\n name: cat89\n type: category\n - column: cat90\n name: cat90\n type: category\n - column: cat91\n name: cat91\n type: category\n - column: cat92\n name: cat92\n type: category\n - column: cat93\n name: cat93\n type: category\n - column: cat94\n name: cat94\n type: category\n - column: cat95\n name: cat95\n type: category\n - column: cat96\n name: cat96\n type: category\n - column: cat97\n name: cat97\n type: category\n - column: cat98\n name: cat98\n type: category\n - column: cat99\n name: cat99\n type: category\n - column: cat100\n name: cat100\n type: category\n - column: cat101\n name: cat101\n type: category\n - column: cat102\n name: cat102\n type: category\n - column: cat103\n name: cat103\n type: category\n - column: cat104\n name: cat104\n type: category\n - column: cat105\n name: cat105\n type: category\n - column: cat106\n name: cat106\n type: category\n - column: cat107\n name: cat107\n type: category\n - column: cat108\n name: cat108\n type: category\n - column: cat109\n name: cat109\n type: category\n - column: cat110\n name: cat110\n type: category\n - column: cat111\n name: cat111\n type: category\n - column: cat112\n name: cat112\n type: category\n - column: cat113\n name: cat113\n type: category\n - column: cat114\n name: cat114\n type: category\n - column: cat115\n name: cat115\n type: category\n - column: cat116\n name: cat116\n type: category\n - column: cont1\n name: cont1\n type: number\n - column: cont2\n name: cont2\n type: number\n - column: cont3\n name: cont3\n type: number\n - column: cont4\n name: cont4\n type: number\n - column: cont5\n name: cont5\n type: number\n - column: cont6\n name: cont6\n type: number\n - column: cont7\n name: cont7\n type: number\n - column: cont8\n name: cont8\n type: number\n - column: cont9\n name: cont9\n type: number\n - column: cont10\n name: cont10\n type: number\n - column: cont11\n name: cont11\n type: number\n - column: cont12\n name: cont12\n type: number\n - column: cont13\n name: cont13\n type: number\n - column: cont14\n name: cont14\n type: number\n combiner:\n type: tabnet\n size: 128 # N_a\n output_size: 8 # N_d\n sparsity: 0.0 # lambda_sparse\n bn_momentum: 0.02 # m_B\n num_steps: 10 # N_steps\n relaxation_factor: 1.0 # gamma\n bn_virtual_bs: 4096 # B_v\n trainer:\n batch_size: 256 # B\n eval_batch_size: null # 65536 131072 262144 524288\n epochs: 300\n early_stop: 30\n learning_rate: 0.01\n optimizer:\n type: adam\n learning_rate_scheduler:\n decay: exponential\n decay_steps: 10000\n decay_rate: 0.9\n validation_metric: root_mean_squared_error\n - name: bnp_claims_management\n goal: maximize\n metric: accuracy\n validation_metric_score: 0.7761691808700562\n training_rows: 80101\n test_rows: 22823\n validation_rows: 11397\n config:\n output_features:\n - name: target\n type: binary\n input_features:\n - name: v1\n type: number\n - name: v2\n type: number\n - name: v3\n type: category\n - name: v4\n type: number\n - name: v5\n type: number\n - name: v6\n type: number\n - name: v7\n type: number\n - name: v8\n type: number\n - name: v9\n type: number\n - name: v10\n type: number\n - name: v11\n type: number\n - name: v12\n type: number\n - name: v13\n type: number\n - name: v14\n type: number\n - name: v15\n type: number\n - name: v16\n type: number\n - name: v17\n type: number\n - name: v18\n type: number\n - name: v19\n type: number\n - name: v20\n type: number\n - name: v21\n type: number\n - name: v22\n type: category\n - name: v23\n type: number\n - name: v24\n type: category\n - name: v25\n type: number\n - name: v26\n type: number\n - name: v27\n type: number\n - name: v28\n type: number\n - name: v29\n type: number\n - name: v30\n type: category\n - name: v31\n type: category\n - name: v32\n type: number\n - name: v33\n type: number\n - name: v34\n type: number\n - name: v35\n type: number\n - name: v36\n type: number\n - name: v37\n type: number\n - name: v38\n type: number\n - name: v39\n type: number\n - name: v40\n type: number\n - name: v41\n type: number\n - name: v42\n type: number\n - name: v43\n type: number\n - name: v44\n type: number\n - name: v45\n type: number\n - name: v46\n type: number\n - name: v47\n type: category\n - name: v48\n type: number\n - name: v49\n type: number\n - name: v50\n type: number\n - name: v51\n type: number\n - name: v52\n type: category\n - name: v53\n type: number\n - name: v54\n type: number\n - name: v55\n type: number\n - name: v56\n type: category\n - name: v57\n type: number\n - name: v58\n type: number\n - name: v59\n type: number\n - name: v60\n type: number\n - name: v61\n type: number\n - name: v62\n type: category\n - name: v63\n type: number\n - name: v64\n type: number\n - name: v65\n type: number\n - name: v66\n type: category\n - name: v67\n type: number\n - name: v68\n type: number\n - name: v69\n type: number\n - name: v70\n type: number\n - name: v71\n type: category\n - name: v72\n type: category\n - name: v73\n type: number\n - name: v74\n type: category\n - name: v75\n type: category\n - name: v76\n type: number\n - name: v77\n type: number\n - name: v78\n type: number\n - name: v79\n type: category\n - name: v80\n type: number\n - name: v81\n type: number\n - name: v82\n type: number\n - name: v83\n type: number\n - name: v84\n type: number\n - name: v85\n type: number\n - name: v86\n type: number\n - name: v87\n type: number\n - name: v88\n type: number\n - name: v89\n type: number\n - name: v90\n type: number\n - name: v91\n type: category\n - name: v92\n type: number\n - name: v93\n type: number\n - name: v94\n type: number\n - name: v95\n type: number\n - name: v96\n type: number\n - name: v97\n type: number\n - name: v98\n type: number\n - name: v99\n type: number\n - name: v100\n type: number\n - name: v101\n type: number\n - name: v102\n type: number\n - name: v103\n type: number\n - name: v104\n type: number\n - name: v105\n type: number\n - name: v106\n type: number\n - name: v107\n type: category\n - name: v108\n type: number\n - name: v109\n type: number\n - name: v110\n type: category\n - name: v111\n type: number\n - name: v112\n type: category\n - name: v113\n type: category\n - name: v114\n type: number\n - name: v115\n type: number\n - name: v116\n type: number\n - name: v117\n type: number\n - name: v118\n type: number\n - name: v119\n type: number\n - name: v120\n type: number\n - name: v121\n type: number\n - name: v122\n type: number\n - name: v123\n type: number\n - name: v124\n type: number\n - name: v125\n type: category\n - name: v126\n type: number\n - name: v127\n type: number\n - name: v128\n type: number\n - name: v129\n type: number\n - name: v130\n type: number\n - name: v131\n type: number\n combiner:\n type: tabnet\n size: 32 # N_a\n output_size: 8 # N_d\n sparsity: 0.0 # lambda_sparse\n bn_momentum: 0.02 # m_B\n num_steps: 3 # N_steps\n relaxation_factor: 1.0 # gamma\n bn_virtual_bs: 256 # B_v\n trainer:\n batch_size: 256 # B\n eval_batch_size: null # 65536 131072 262144 524288\n epochs: 300\n early_stop: 30\n learning_rate: 0.01\n optimizer:\n type: adam\n learning_rate_scheduler:\n decay: exponential\n decay_steps: 2000\n decay_rate: 0.4\n validation_metric: accuracy\n - name: ieee_fraud\n goal: maximize\n metric: accuracy\n validation_metric_score: 0.9836957454681396\n training_rows: 413498\n test_rows: 118039\n validation_rows: 59003\n config:\n output_features:\n - name: isFraud\n type: binary\n input_features:\n - name: TransactionDT\n type: number\n - name: TransactionAmt\n type: number\n - name: ProductCD\n type: category\n - name: card1\n type: number\n - name: card2\n type: number\n - name: card3\n type: number\n - name: card4\n type: category\n - name: card5\n type: number\n - name: card6\n type: category\n - name: addr1\n type: number\n - name: addr2\n type: number\n - name: dist1\n type: number\n - name: dist2\n type: number\n - name: P_emaildomain\n type: category\n - name: R_emaildomain\n type: category\n - name: C1\n type: number\n - name: C2\n type: number\n - name: C3\n type: number\n - name: C4\n type: number\n - name: C5\n type: number\n - name: C6\n type: number\n - name: C7\n type: number\n - name: C8\n type: number\n - name: C9\n type: number\n - name: C10\n type: number\n - name: C11\n type: number\n - name: C12\n type: number\n - name: C13\n type: number\n - name: C14\n type: number\n - name: D1\n type: number\n - name: D2\n type: number\n - name: D3\n type: number\n - name: D4\n type: number\n - name: D5\n type: number\n - name: D6\n type: number\n - name: D7\n type: number\n - name: D8\n type: number\n - name: D9\n type: number\n - name: D10\n type: number\n - name: D11\n type: number\n - name: D12\n type: number\n - name: D13\n type: number\n - name: D14\n type: number\n - name: D15\n type: number\n - name: M1\n type: category\n - name: M2\n type: category\n - name: M3\n type: category\n - name: M4\n type: category\n - name: M5\n type: category\n - name: M6\n type: category\n - name: M7\n type: category\n - name: M8\n type: category\n - name: M9\n type: category\n - name: V1\n type: number\n - name: V2\n type: number\n - name: V3\n type: number\n - name: V4\n type: number\n - name: V5\n type: number\n - name: V6\n type: number\n - name: V7\n type: number\n - name: V8\n type: number\n - name: V9\n type: number\n - name: V10\n type: number\n - name: V11\n type: number\n - name: V12\n type: number\n - name: V13\n type: number\n - name: V14\n type: number\n - name: V15\n type: number\n - name: V16\n type: number\n - name: V17\n type: number\n - name: V18\n type: number\n - name: V19\n type: number\n - name: V20\n type: number\n - name: V21\n type: number\n - name: V22\n type: number\n - name: V23\n type: number\n - name: V24\n type: number\n - name: V25\n type: number\n - name: V26\n type: number\n - name: V27\n type: number\n - name: V28\n type: number\n - name: V29\n type: number\n - name: V30\n type: number\n - name: V31\n type: number\n - name: V32\n type: number\n - name: V33\n type: number\n - name: V34\n type: number\n - name: V35\n type: number\n - name: V36\n type: number\n - name: V37\n type: number\n - name: V38\n type: number\n - name: V39\n type: number\n - name: V40\n type: number\n - name: V41\n type: number\n - name: V42\n type: number\n - name: V43\n type: number\n - name: V44\n type: number\n - name: V45\n type: number\n - name: V46\n type: number\n - name: V47\n type: number\n - name: V48\n type: number\n - name: V49\n type: number\n - name: V50\n type: number\n - name: V51\n type: number\n - name: V52\n type: number\n - name: V53\n type: number\n - name: V54\n type: number\n - name: V55\n type: number\n - name: V56\n type: number\n - name: V57\n type: number\n - name: V58\n type: number\n - name: V59\n type: number\n - name: V60\n type: number\n - name: V61\n type: number\n - name: V62\n type: number\n - name: V63\n type: number\n - name: V64\n type: number\n - name: V65\n type: number\n - name: V66\n type: number\n - name: V67\n type: number\n - name: V68\n type: number\n - name: V69\n type: number\n - name: V70\n type: number\n - name: V71\n type: number\n - name: V72\n type: number\n - name: V73\n type: number\n - name: V74\n type: number\n - name: V75\n type: number\n - name: V76\n type: number\n - name: V77\n type: number\n - name: V78\n type: number\n - name: V79\n type: number\n - name: V80\n type: number\n - name: V81\n type: number\n - name: V82\n type: number\n - name: V83\n type: number\n - name: V84\n type: number\n - name: V85\n type: number\n - name: V86\n type: number\n - name: V87\n type: number\n - name: V88\n type: number\n - name: V89\n type: number\n - name: V90\n type: number\n - name: V91\n type: number\n - name: V92\n type: number\n - name: V93\n type: number\n - name: V94\n type: number\n - name: V95\n type: number\n - name: V96\n type: number\n - name: V97\n type: number\n - name: V98\n type: number\n - name: V99\n type: number\n - name: V100\n type: number\n - name: V101\n type: number\n - name: V102\n type: number\n - name: V103\n type: number\n - name: V104\n type: number\n - name: V105\n type: number\n - name: V106\n type: number\n - name: V107\n type: number\n - name: V108\n type: number\n - name: V109\n type: number\n - name: V110\n type: number\n - name: V111\n type: number\n - name: V112\n type: number\n - name: V113\n type: number\n - name: V114\n type: number\n - name: V115\n type: number\n - name: V116\n type: number\n - name: V117\n type: number\n - name: V118\n type: number\n - name: V119\n type: number\n - name: V120\n type: number\n - name: V121\n type: number\n - name: V122\n type: number\n - name: V123\n type: number\n - name: V124\n type: number\n - name: V125\n type: number\n - name: V126\n type: number\n - name: V127\n type: number\n - name: V128\n type: number\n - name: V129\n type: number\n - name: V130\n type: number\n - name: V131\n type: number\n - name: V132\n type: number\n - name: V133\n type: number\n - name: V134\n type: number\n - name: V135\n type: number\n - name: V136\n type: number\n - name: V137\n type: number\n - name: V138\n type: number\n - name: V139\n type: number\n - name: V140\n type: number\n - name: V141\n type: number\n - name: V142\n type: number\n - name: V143\n type: number\n - name: V144\n type: number\n - name: V145\n type: number\n - name: V146\n type: number\n - name: V147\n type: number\n - name: V148\n type: number\n - name: V149\n type: number\n - name: V150\n type: number\n - name: V151\n type: number\n - name: V152\n type: number\n - name: V153\n type: number\n - name: V154\n type: number\n - name: V155\n type: number\n - name: V156\n type: number\n - name: V157\n type: number\n - name: V158\n type: number\n - name: V159\n type: number\n - name: V160\n type: number\n - name: V161\n type: number\n - name: V162\n type: number\n - name: V163\n type: number\n - name: V164\n type: number\n - name: V165\n type: number\n - name: V166\n type: number\n - name: V167\n type: number\n - name: V168\n type: number\n - name: V169\n type: number\n - name: V170\n type: number\n - name: V171\n type: number\n - name: V172\n type: number\n - name: V173\n type: number\n - name: V174\n type: number\n - name: V175\n type: number\n - name: V176\n type: number\n - name: V177\n type: number\n - name: V178\n type: number\n - name: V179\n type: number\n - name: V180\n type: number\n - name: V181\n type: number\n - name: V182\n type: number\n - name: V183\n type: number\n - name: V184\n type: number\n - name: V185\n type: number\n - name: V186\n type: number\n - name: V187\n type: number\n - name: V188\n type: number\n - name: V189\n type: number\n - name: V190\n type: number\n - name: V191\n type: number\n - name: V192\n type: number\n - name: V193\n type: number\n - name: V194\n type: number\n - name: V195\n type: number\n - name: V196\n type: number\n - name: V197\n type: number\n - name: V198\n type: number\n - name: V199\n type: number\n - name: V200\n type: number\n - name: V201\n type: number\n - name: V202\n type: number\n - name: V203\n type: number\n - name: V204\n type: number\n - name: V205\n type: number\n - name: V206\n type: number\n - name: V207\n type: number\n - name: V208\n type: number\n - name: V209\n type: number\n - name: V210\n type: number\n - name: V211\n type: number\n - name: V212\n type: number\n - name: V213\n type: number\n - name: V214\n type: number\n - name: V215\n type: number\n - name: V216\n type: number\n - name: V217\n type: number\n - name: V218\n type: number\n - name: V219\n type: number\n - name: V220\n type: number\n - name: V221\n type: number\n - name: V222\n type: number\n - name: V223\n type: number\n - name: V224\n type: number\n - name: V225\n type: number\n - name: V226\n type: number\n - name: V227\n type: number\n - name: V228\n type: number\n - name: V229\n type: number\n - name: V230\n type: number\n - name: V231\n type: number\n - name: V232\n type: number\n - name: V233\n type: number\n - name: V234\n type: number\n - name: V235\n type: number\n - name: V236\n type: number\n - name: V237\n type: number\n - name: V238\n type: number\n - name: V239\n type: number\n - name: V240\n type: number\n - name: V241\n type: number\n - name: V242\n type: number\n - name: V243\n type: number\n - name: V244\n type: number\n - name: V245\n type: number\n - name: V246\n type: number\n - name: V247\n type: number\n - name: V248\n type: number\n - name: V249\n type: number\n - name: V250\n type: number\n - name: V251\n type: number\n - name: V252\n type: number\n - name: V253\n type: number\n - name: V254\n type: number\n - name: V255\n type: number\n - name: V256\n type: number\n - name: V257\n type: number\n - name: V258\n type: number\n - name: V259\n type: number\n - name: V260\n type: number\n - name: V261\n type: number\n - name: V262\n type: number\n - name: V263\n type: number\n - name: V264\n type: number\n - name: V265\n type: number\n - name: V266\n type: number\n - name: V267\n type: number\n - name: V268\n type: number\n - name: V269\n type: number\n - name: V270\n type: number\n - name: V271\n type: number\n - name: V272\n type: number\n - name: V273\n type: number\n - name: V274\n type: number\n - name: V275\n type: number\n - name: V276\n type: number\n - name: V277\n type: number\n - name: V278\n type: number\n - name: V279\n type: number\n - name: V280\n type: number\n - name: V281\n type: number\n - name: V282\n type: number\n - name: V283\n type: number\n - name: V284\n type: number\n - name: V285\n type: number\n - name: V286\n type: number\n - name: V287\n type: number\n - name: V288\n type: number\n - name: V289\n type: number\n - name: V290\n type: number\n - name: V291\n type: number\n - name: V292\n type: number\n - name: V293\n type: number\n - name: V294\n type: number\n - name: V295\n type: number\n - name: V296\n type: number\n - name: V297\n type: number\n - name: V298\n type: number\n - name: V299\n type: number\n - name: V300\n type: number\n - name: V301\n type: number\n - name: V302\n type: number\n - name: V303\n type: number\n - name: V304\n type: number\n - name: V305\n type: number\n - name: V306\n type: number\n - name: V307\n type: number\n - name: V308\n type: number\n - name: V309\n type: number\n - name: V310\n type: number\n - name: V311\n type: number\n - name: V312\n type: number\n - name: V313\n type: number\n - name: V314\n type: number\n - name: V315\n type: number\n - name: V316\n type: number\n - name: V317\n type: number\n - name: V318\n type: number\n - name: V319\n type: number\n - name: V320\n type: number\n - name: V321\n type: number\n - name: V322\n type: number\n - name: V323\n type: number\n - name: V324\n type: number\n - name: V325\n type: number\n - name: V326\n type: number\n - name: V327\n type: number\n - name: V328\n type: number\n - name: V329\n type: number\n - name: V330\n type: number\n - name: V331\n type: number\n - name: V332\n type: number\n - name: V333\n type: number\n - name: V334\n type: number\n - name: V335\n type: number\n - name: V336\n type: number\n - name: V337\n type: number\n - name: V338\n type: number\n - name: V339\n type: number\n - name: id_01\n type: number\n - name: id_02\n type: number\n - name: id_03\n type: number\n - name: id_04\n type: number\n - name: id_05\n type: number\n - name: id_06\n type: number\n - name: id_07\n type: number\n - name: id_08\n type: number\n - name: id_09\n type: number\n - name: id_10\n type: number\n - name: id_11\n type: number\n - name: id_12\n type: category\n - name: id_13\n type: number\n - name: id_14\n type: number\n - name: id_15\n type: category\n - name: id_16\n type: category\n - name: id_17\n type: number\n - name: id_18\n type: number\n - name: id_19\n type: number\n - name: id_20\n type: number\n - name: id_21\n type: number\n - name: id_22\n type: number\n - name: id_23\n type: category\n - name: id_24\n type: number\n - name: id_25\n type: number\n - name: id_26\n type: number\n - name: id_27\n type: category\n - name: id_28\n type: category\n - name: id_29\n type: category\n - name: id_30\n type: category\n - name: id_31\n type: text\n - name: id_32\n type: number\n - name: id_33\n type: category\n - name: id_34\n type: category\n - name: id_35\n type: category\n - name: id_36\n type: category\n - name: id_37\n type: category\n - name: id_38\n type: category\n - name: DeviceType\n type: category\n - name: DeviceInfo\n type: category\n combiner:\n type: tabnet\n size: 128 # N_a\n output_size: 24 # N_d\n sparsity: 0.000001 # lambda_sparse\n bn_momentum: 0.02 # m_B\n num_steps: 10 # N_steps\n relaxation_factor: 1.0 # gamma\n bn_virtual_bs: 2048 # B_v\n trainer:\n batch_size: 256 # B\n eval_batch_size: null # 65536 131072 262144 524288\n epochs: 300\n early_stop: 30\n learning_rate: 0.01\n optimizer:\n type: adam\n learning_rate_scheduler:\n decay: exponential\n decay_steps: 10000\n decay_rate: 0.95\n validation_metric: accuracy\n - name: mercedes_benz_greener\n goal: minimize\n metric: root_mean_squared_error\n validation_metric_score: 7.685836315155029\n training_rows: 2969\n test_rows: 840\n validation_rows: 400\n config:\n output_features:\n - name: y\n type: number\n input_features:\n - name: X0\n type: category\n - name: X1\n type: category\n - name: X2\n type: category\n - name: X3\n type: category\n - name: X4\n type: category\n - name: X5\n type: category\n - name: X6\n type: category\n - name: X8\n type: category\n - name: X10\n type: binary\n - name: X11\n type: binary\n - name: X12\n type: binary\n - name: X13\n type: binary\n - name: X14\n type: binary\n - name: X15\n type: binary\n - name: X16\n type: binary\n - name: X17\n type: binary\n - name: X18\n type: binary\n - name: X19\n type: binary\n - name: X20\n type: binary\n - name: X21\n type: binary\n - name: X22\n type: binary\n - name: X23\n type: binary\n - name: X24\n type: binary\n - name: X26\n type: binary\n - name: X27\n type: binary\n - name: X28\n type: binary\n - name: X29\n type: binary\n - name: X30\n type: binary\n - name: X31\n type: binary\n - name: X32\n type: binary\n - name: X33\n type: binary\n - name: X34\n type: binary\n - name: X35\n type: binary\n - name: X36\n type: binary\n - name: X37\n type: binary\n - name: X38\n type: binary\n - name: X39\n type: binary\n - name: X40\n type: binary\n - name: X41\n type: binary\n - name: X42\n type: binary\n - name: X43\n type: binary\n - name: X44\n type: binary\n - name: X45\n type: binary\n - name: X46\n type: binary\n - name: X47\n type: binary\n - name: X48\n type: binary\n - name: X49\n type: binary\n - name: X50\n type: binary\n - name: X51\n type: binary\n - name: X52\n type: binary\n - name: X53\n type: binary\n - name: X54\n type: binary\n - name: X55\n type: binary\n - name: X56\n type: binary\n - name: X57\n type: binary\n - name: X58\n type: binary\n - name: X59\n type: binary\n - name: X60\n type: binary\n - name: X61\n type: binary\n - name: X62\n type: binary\n - name: X63\n type: binary\n - name: X64\n type: binary\n - name: X65\n type: binary\n - name: X66\n type: binary\n - name: X67\n type: binary\n - name: X68\n type: binary\n - name: X69\n type: binary\n - name: X70\n type: binary\n - name: X71\n type: binary\n - name: X73\n type: binary\n - name: X74\n type: binary\n - name: X75\n type: binary\n - name: X76\n type: binary\n - name: X77\n type: binary\n - name: X78\n type: binary\n - name: X79\n type: binary\n - name: X80\n type: binary\n - name: X81\n type: binary\n - name: X82\n type: binary\n - name: X83\n type: binary\n - name: X84\n type: binary\n - name: X85\n type: binary\n - name: X86\n type: binary\n - name: X87\n type: binary\n - name: X88\n type: binary\n - name: X89\n type: binary\n - name: X90\n type: binary\n - name: X91\n type: binary\n - name: X92\n type: binary\n - name: X93\n type: binary\n - name: X94\n type: binary\n - name: X95\n type: binary\n - name: X96\n type: binary\n - name: X97\n type: binary\n - name: X98\n type: binary\n - name: X99\n type: binary\n - name: X100\n type: binary\n - name: X101\n type: binary\n - name: X102\n type: binary\n - name: X103\n type: binary\n - name: X104\n type: binary\n - name: X105\n type: binary\n - name: X106\n type: binary\n - name: X107\n type: binary\n - name: X108\n type: binary\n - name: X109\n type: binary\n - name: X110\n type: binary\n - name: X111\n type: binary\n - name: X112\n type: binary\n - name: X113\n type: binary\n - name: X114\n type: binary\n - name: X115\n type: binary\n - name: X116\n type: binary\n - name: X117\n type: binary\n - name: X118\n type: binary\n - name: X119\n type: binary\n - name: X120\n type: binary\n - name: X122\n type: binary\n - name: X123\n type: binary\n - name: X124\n type: binary\n - name: X125\n type: binary\n - name: X126\n type: binary\n - name: X127\n type: binary\n - name: X128\n type: binary\n - name: X129\n type: binary\n - name: X130\n type: binary\n - name: X131\n type: binary\n - name: X132\n type: binary\n - name: X133\n type: binary\n - name: X134\n type: binary\n - name: X135\n type: binary\n - name: X136\n type: binary\n - name: X137\n type: binary\n - name: X138\n type: binary\n - name: X139\n type: binary\n - name: X140\n type: binary\n - name: X141\n type: binary\n - name: X142\n type: binary\n - name: X143\n type: binary\n - name: X144\n type: binary\n - name: X145\n type: binary\n - name: X146\n type: binary\n - name: X147\n type: binary\n - name: X148\n type: binary\n - name: X150\n type: binary\n - name: X151\n type: binary\n - name: X152\n type: binary\n - name: X153\n type: binary\n - name: X154\n type: binary\n - name: X155\n type: binary\n - name: X156\n type: binary\n - name: X157\n type: binary\n - name: X158\n type: binary\n - name: X159\n type: binary\n - name: X160\n type: binary\n - name: X161\n type: binary\n - name: X162\n type: binary\n - name: X163\n type: binary\n - name: X164\n type: binary\n - name: X165\n type: binary\n - name: X166\n type: binary\n - name: X167\n type: binary\n - name: X168\n type: binary\n - name: X169\n type: binary\n - name: X170\n type: binary\n - name: X171\n type: binary\n - name: X172\n type: binary\n - name: X173\n type: binary\n - name: X174\n type: binary\n - name: X175\n type: binary\n - name: X176\n type: binary\n - name: X177\n type: binary\n - name: X178\n type: binary\n - name: X179\n type: binary\n - name: X180\n type: binary\n - name: X181\n type: binary\n - name: X182\n type: binary\n - name: X183\n type: binary\n - name: X184\n type: binary\n - name: X185\n type: binary\n - name: X186\n type: binary\n - name: X187\n type: binary\n - name: X189\n type: binary\n - name: X190\n type: binary\n - name: X191\n type: binary\n - name: X192\n type: binary\n - name: X194\n type: binary\n - name: X195\n type: binary\n - name: X196\n type: binary\n - name: X197\n type: binary\n - name: X198\n type: binary\n - name: X199\n type: binary\n - name: X200\n type: binary\n - name: X201\n type: binary\n - name: X202\n type: binary\n - name: X203\n type: binary\n - name: X204\n type: binary\n - name: X205\n type: binary\n - name: X206\n type: binary\n - name: X207\n type: binary\n - name: X208\n type: binary\n - name: X209\n type: binary\n - name: X210\n type: binary\n - name: X211\n type: binary\n - name: X212\n type: binary\n - name: X213\n type: binary\n - name: X214\n type: binary\n - name: X215\n type: binary\n - name: X216\n type: binary\n - name: X217\n type: binary\n - name: X218\n type: binary\n - name: X219\n type: binary\n - name: X220\n type: binary\n - name: X221\n type: binary\n - name: X222\n type: binary\n - name: X223\n type: binary\n - name: X224\n type: binary\n - name: X225\n type: binary\n - name: X226\n type: binary\n - name: X227\n type: binary\n - name: X228\n type: binary\n - name: X229\n type: binary\n - name: X230\n type: binary\n - name: X231\n type: binary\n - name: X232\n type: binary\n - name: X233\n type: binary\n - name: X234\n type: binary\n - name: X235\n type: binary\n - name: X236\n type: binary\n - name: X237\n type: binary\n - name: X238\n type: binary\n - name: X239\n type: binary\n - name: X240\n type: binary\n - name: X241\n type: binary\n - name: X242\n type: binary\n - name: X243\n type: binary\n - name: X244\n type: binary\n - name: X245\n type: binary\n - name: X246\n type: binary\n - name: X247\n type: binary\n - name: X248\n type: binary\n - name: X249\n type: binary\n - name: X250\n type: binary\n - name: X251\n type: binary\n - name: X252\n type: binary\n - name: X253\n type: binary\n - name: X254\n type: binary\n - name: X255\n type: binary\n - name: X256\n type: binary\n - name: X257\n type: binary\n - name: X258\n type: binary\n - name: X259\n type: binary\n - name: X260\n type: binary\n - name: X261\n type: binary\n - name: X262\n type: binary\n - name: X263\n type: binary\n - name: X264\n type: binary\n - name: X265\n type: binary\n - name: X266\n type: binary\n - name: X267\n type: binary\n - name: X268\n type: binary\n - name: X269\n type: binary\n - name: X270\n type: binary\n - name: X271\n type: binary\n - name: X272\n type: binary\n - name: X273\n type: binary\n - name: X274\n type: binary\n - name: X275\n type: binary\n - name: X276\n type: binary\n - name: X277\n type: binary\n - name: X278\n type: binary\n - name: X279\n type: binary\n - name: X280\n type: binary\n - name: X281\n type: binary\n - name: X282\n type: binary\n - name: X283\n type: binary\n - name: X284\n type: binary\n - name: X285\n type: binary\n - name: X286\n type: binary\n - name: X287\n type: binary\n - name: X288\n type: binary\n - name: X289\n type: binary\n - name: X290\n type: binary\n - name: X291\n type: binary\n - name: X292\n type: binary\n - name: X293\n type: binary\n - name: X294\n type: binary\n - name: X295\n type: binary\n - name: X296\n type: binary\n - name: X297\n type: binary\n - name: X298\n type: binary\n - name: X299\n type: binary\n - name: X300\n type: binary\n - name: X301\n type: binary\n - name: X302\n type: binary\n - name: X304\n type: binary\n - name: X305\n type: binary\n - name: X306\n type: binary\n - name: X307\n type: binary\n - name: X308\n type: binary\n - name: X309\n type: binary\n - name: X310\n type: binary\n - name: X311\n type: binary\n - name: X312\n type: binary\n - name: X313\n type: binary\n - name: X314\n type: binary\n - name: X315\n type: binary\n - name: X316\n type: binary\n - name: X317\n type: binary\n - name: X318\n type: binary\n - name: X319\n type: binary\n - name: X320\n type: binary\n - name: X321\n type: binary\n - name: X322\n type: binary\n - name: X323\n type: binary\n - name: X324\n type: binary\n - name: X325\n type: binary\n - name: X326\n type: binary\n - name: X327\n type: binary\n - name: X328\n type: binary\n - name: X329\n type: binary\n - name: X330\n type: binary\n - name: X331\n type: binary\n - name: X332\n type: binary\n - name: X333\n type: binary\n - name: X334\n type: binary\n - name: X335\n type: binary\n - name: X336\n type: binary\n - name: X337\n type: binary\n - name: X338\n type: binary\n - name: X339\n type: binary\n - name: X340\n type: binary\n - name: X341\n type: binary\n - name: X342\n type: binary\n - name: X343\n type: binary\n - name: X344\n type: binary\n - name: X345\n type: binary\n - name: X346\n type: binary\n - name: X347\n type: binary\n - name: X348\n type: binary\n - name: X349\n type: binary\n - name: X350\n type: binary\n - name: X351\n type: binary\n - name: X352\n type: binary\n - name: X353\n type: binary\n - name: X354\n type: binary\n - name: X355\n type: binary\n - name: X356\n type: binary\n - name: X357\n type: binary\n - name: X358\n type: binary\n - name: X359\n type: binary\n - name: X360\n type: binary\n - name: X361\n type: binary\n - name: X362\n type: binary\n - name: X363\n type: binary\n - name: X364\n type: binary\n - name: X365\n type: binary\n - name: X366\n type: binary\n - name: X367\n type: binary\n - name: X368\n type: binary\n - name: X369\n type: binary\n - name: X370\n type: binary\n - name: X371\n type: binary\n - name: X372\n type: binary\n - name: X373\n type: binary\n - name: X374\n type: binary\n - name: X375\n type: binary\n - name: X376\n type: binary\n - name: X377\n type: binary\n - name: X378\n type: binary\n - name: X379\n type: binary\n - name: X380\n type: binary\n - name: X382\n type: binary\n - name: X383\n type: binary\n - name: X384\n type: binary\n - name: X385\n type: binary\n combiner:\n type: tabnet\n size: 128 # N_a\n output_size: 8 # N_d\n sparsity: 0.1 # lambda_sparse\n bn_momentum: 0.1 # m_B\n num_steps: 9 # N_steps\n relaxation_factor: 1.0 # gamma\n bn_virtual_bs: 256 # B_v\n trainer:\n batch_size: 256 # B\n eval_batch_size: null # 65536 131072 262144 524288\n epochs: 300\n early_stop: 30\n learning_rate: 0.005\n optimizer:\n type: adam\n learning_rate_scheduler:\n decay: exponential\n decay_steps: 500\n decay_rate: 0.95\n validation_metric: root_mean_squared_error\n - name: otto_group_product\n goal: maximize\n metric: accuracy\n validation_metric_score: 0.7956883907318115\n training_rows: 43459\n test_rows: 12296\n validation_rows: 6123\n config:\n output_features:\n - name: target\n type: category\n input_features:\n - name: feat_1\n type: number\n - name: feat_2\n type: number\n - name: feat_3\n type: number\n - name: feat_4\n type: number\n - name: feat_5\n type: number\n - name: feat_6\n type: number\n - name: feat_7\n type: number\n - name: feat_8\n type: number\n - name: feat_9\n type: number\n - name: feat_10\n type: number\n - name: feat_11\n type: number\n - name: feat_12\n type: number\n - name: feat_13\n type: number\n - name: feat_14\n type: number\n - name: feat_15\n type: number\n - name: feat_16\n type: number\n - name: feat_17\n type: number\n - name: feat_18\n type: number\n - name: feat_19\n type: number\n - name: feat_20\n type: number\n - name: feat_21\n type: category\n - name: feat_22\n type: number\n - name: feat_23\n type: number\n - name: feat_24\n type: number\n - name: feat_25\n type: number\n - name: feat_26\n type: number\n - name: feat_27\n type: number\n - name: feat_28\n type: number\n - name: feat_29\n type: number\n - name: feat_30\n type: number\n - name: feat_31\n type: number\n - name: feat_32\n type: number\n - name: feat_33\n type: number\n - name: feat_34\n type: number\n - name: feat_35\n type: number\n - name: feat_36\n type: number\n - name: feat_37\n type: number\n - name: feat_38\n type: number\n - name: feat_39\n type: number\n - name: feat_40\n type: number\n - name: feat_41\n type: number\n - name: feat_42\n type: number\n - name: feat_43\n type: number\n - name: feat_44\n type: number\n - name: feat_45\n type: number\n - name: feat_46\n type: number\n - name: feat_47\n type: number\n - name: feat_48\n type: number\n - name: feat_49\n type: number\n - name: feat_50\n type: number\n - name: feat_51\n type: number\n - name: feat_52\n type: number\n - name: feat_53\n type: number\n - name: feat_54\n type: number\n - name: feat_55\n type: number\n - name: feat_56\n type: number\n - name: feat_57\n type: number\n - name: feat_58\n type: number\n - name: feat_59\n type: number\n - name: feat_60\n type: number\n - name: feat_61\n type: number\n - name: feat_62\n type: number\n - name: feat_63\n type: number\n - name: feat_64\n type: number\n - name: feat_65\n type: number\n - name: feat_66\n type: number\n - name: feat_67\n type: number\n - name: feat_68\n type: number\n - name: feat_69\n type: number\n - name: feat_70\n type: number\n - name: feat_71\n type: number\n - name: feat_72\n type: number\n - name: feat_73\n type: number\n - name: feat_74\n type: number\n - name: feat_75\n type: number\n - name: feat_76\n type: number\n - name: feat_77\n type: number\n - name: feat_78\n type: number\n - name: feat_79\n type: number\n - name: feat_80\n type: number\n - name: feat_81\n type: number\n - name: feat_82\n type: number\n - name: feat_83\n type: number\n - name: feat_84\n type: number\n - name: feat_85\n type: number\n - name: feat_86\n type: number\n - name: feat_87\n type: number\n - name: feat_88\n type: number\n - name: feat_89\n type: number\n - name: feat_90\n type: number\n - name: feat_91\n type: number\n - name: feat_92\n type: number\n - name: feat_93\n type: number\n combiner:\n type: tabnet\n size: 128 # N_a\n output_size: 128 # N_d\n sparsity: 0.0 # lambda_sparse\n bn_momentum: 0.2 # m_B\n num_steps: 3 # N_steps\n relaxation_factor: 1.0 # gamma\n bn_virtual_bs: 512 # B_v\n trainer:\n batch_size: 256 # B\n eval_batch_size: null # 65536 131072 262144 524288\n epochs: 300\n early_stop: 30\n learning_rate: 0.005\n optimizer:\n type: adam\n learning_rate_scheduler:\n decay: exponential\n decay_steps: 20000\n decay_rate: 0.4\n validation_metric: accuracy\n - name: poker_hand\n goal: maximize\n metric: accuracy\n validation_metric_score: 0.9804078340530396\n training_rows: 22509\n test_rows: 0\n validation_rows: 2501\n config:\n output_features:\n - name: hand\n type: category\n input_features:\n - name: S1\n type: number\n - name: C1\n type: number\n - name: S2\n type: number\n - name: C2\n type: number\n - name: S3\n type: number\n - name: C3\n type: number\n - name: S4\n type: number\n - name: C4\n type: number\n - name: S5\n type: number\n - name: C5\n type: number\n combiner:\n type: tabnet\n size: 16 # N_a\n output_size: 128 # N_d\n sparsity: 0.0 # lambda_sparse\n bn_momentum: 0.02 # m_B\n num_steps: 6 # N_steps\n relaxation_factor: 1.0 # gamma\n bn_virtual_bs: 512 # B_v\n trainer:\n batch_size: 256 # B\n eval_batch_size: null # 65536 131072 262144 524288\n epochs: 300\n early_stop: 30\n learning_rate: 0.01\n optimizer:\n type: adam\n learning_rate_scheduler:\n decay: exponential\n decay_steps: 8000\n decay_rate: 0.8\n validation_metric: accuracy\n - name: porto_seguro_safe_driver\n goal: maximize\n metric: accuracy\n validation_metric_score: 0.9630663394927979\n training_rows: 416779\n test_rows: 118948\n validation_rows: 59485\n config:\n output_features:\n - name: target\n type: binary\n input_features:\n - name: ps_ind_01\n type: category\n - name: ps_ind_02_cat\n type: number\n - name: ps_ind_03\n type: category\n - name: ps_ind_04_cat\n type: category\n - name: ps_ind_05_cat\n type: category\n - name: ps_ind_06_bin\n type: binary\n - name: ps_ind_07_bin\n type: binary\n - name: ps_ind_08_bin\n type: binary\n - name: ps_ind_09_bin\n type: binary\n - name: ps_ind_10_bin\n type: binary\n - name: ps_ind_11_bin\n type: binary\n - name: ps_ind_12_bin\n type: binary\n - name: ps_ind_13_bin\n type: binary\n - name: ps_ind_14\n type: category\n - name: ps_ind_15\n type: category\n - name: ps_ind_16_bin\n type: binary\n - name: ps_ind_17_bin\n type: binary\n - name: ps_ind_18_bin\n type: binary\n - name: ps_reg_01\n type: number\n - name: ps_reg_02\n type: number\n - name: ps_reg_03\n type: number\n - name: ps_car_01_cat\n type: category\n - name: ps_car_02_cat\n type: category\n - name: ps_car_03_cat\n type: category\n - name: ps_car_04_cat\n type: category\n - name: ps_car_05_cat\n type: category\n - name: ps_car_06_cat\n type: category\n - name: ps_car_07_cat\n type: category\n - name: ps_car_08_cat\n type: binary\n - name: ps_car_09_cat\n type: category\n - name: ps_car_10_cat\n type: category\n - name: ps_car_11_cat\n type: number\n - name: ps_car_11\n type: category\n - name: ps_car_12\n type: number\n - name: ps_car_13\n type: number\n - name: ps_car_14\n type: number\n - name: ps_car_15\n type: number\n - name: ps_calc_01\n type: number\n - name: ps_calc_02\n type: number\n - name: ps_calc_03\n type: number\n - name: ps_calc_04\n type: category\n - name: ps_calc_05\n type: category\n - name: ps_calc_06\n type: category\n - name: ps_calc_07\n type: category\n - name: ps_calc_08\n type: category\n - name: ps_calc_09\n type: category\n - name: ps_calc_10\n type: number\n - name: ps_calc_11\n type: number\n - name: ps_calc_12\n type: category\n - name: ps_calc_13\n type: category\n - name: ps_calc_14\n type: number\n - name: ps_calc_15_bin\n type: binary\n - name: ps_calc_16_bin\n type: binary\n - name: ps_calc_17_bin\n type: binary\n - name: ps_calc_18_bin\n type: binary\n - name: ps_calc_19_bin\n type: binary\n - name: ps_calc_20_bin\n type: binary\n combiner:\n type: tabnet\n size: 32 # N_a\n output_size: 32 # N_d\n sparsity: 0.0001 # lambda_sparse\n bn_momentum: 0.4 # m_B\n num_steps: 5 # N_steps\n relaxation_factor: 1.2 # gamma\n bn_virtual_bs: 1024 # B_v\n trainer:\n batch_size: 1024 # B\n eval_batch_size: null # 65536 131072 262144 524288\n epochs: 300\n early_stop: 30\n learning_rate: 0.005\n optimizer:\n type: adam\n learning_rate_scheduler:\n decay: exponential\n decay_steps: 10000\n decay_rate: 0.9\n validation_metric: accuracy\n - name: santander_customer_satisfaction\n goal: maximize\n metric: accuracy\n validation_metric_score: 0.9611535668373108\n training_rows: 53298\n test_rows: 15128\n validation_rows: 7594\n config:\n output_features:\n - name: TARGET\n type: binary\n input_features:\n - name: var3\n type: number\n - name: var15\n type: number\n - name: imp_ent_var16_ult1\n type: number\n - name: imp_op_var39_comer_ult1\n type: number\n - name: imp_op_var39_comer_ult3\n type: number\n - name: imp_op_var40_comer_ult1\n type: number\n - name: imp_op_var40_comer_ult3\n type: number\n - name: imp_op_var40_efect_ult1\n type: number\n - name: imp_op_var40_efect_ult3\n type: number\n - name: imp_op_var40_ult1\n type: number\n - name: imp_op_var41_comer_ult1\n type: number\n - name: imp_op_var41_comer_ult3\n type: number\n - name: imp_op_var41_efect_ult1\n type: number\n - name: imp_op_var41_efect_ult3\n type: number\n - name: imp_op_var41_ult1\n type: number\n - name: imp_op_var39_efect_ult1\n type: number\n - name: imp_op_var39_efect_ult3\n type: number\n - name: imp_op_var39_ult1\n type: number\n - name: imp_sal_var16_ult1\n type: number\n - name: ind_var1_0\n type: binary\n - name: ind_var1\n type: binary\n - name: ind_var2_0\n type: binary\n - name: ind_var2\n type: binary\n - name: ind_var5_0\n type: binary\n - name: ind_var5\n type: binary\n - name: ind_var6_0\n type: binary\n - name: ind_var6\n type: binary\n - name: ind_var8_0\n type: binary\n - name: ind_var8\n type: binary\n - name: ind_var12_0\n type: binary\n - name: ind_var12\n type: binary\n - name: ind_var13_0\n type: binary\n - name: ind_var13_corto_0\n type: binary\n - name: ind_var13_corto\n type: binary\n - name: ind_var13_largo_0\n type: binary\n - name: ind_var13_largo\n type: binary\n - name: ind_var13_medio_0\n type: binary\n - name: ind_var13_medio\n type: binary\n - name: ind_var13\n type: binary\n - name: ind_var14_0\n type: binary\n - name: ind_var14\n type: binary\n - name: ind_var17_0\n type: binary\n - name: ind_var17\n type: binary\n - name: ind_var18_0\n type: binary\n - name: ind_var18\n type: binary\n - name: ind_var19\n type: binary\n - name: ind_var20_0\n type: binary\n - name: ind_var20\n type: binary\n - name: ind_var24_0\n type: binary\n - name: ind_var24\n type: binary\n - name: ind_var25_cte\n type: binary\n - name: ind_var26_0\n type: binary\n - name: ind_var26_cte\n type: binary\n - name: ind_var26\n type: binary\n - name: ind_var25_0\n type: binary\n - name: ind_var25\n type: binary\n - name: ind_var27_0\n type: binary\n - name: ind_var28_0\n type: binary\n - name: ind_var28\n type: binary\n - name: ind_var27\n type: binary\n - name: ind_var29_0\n type: binary\n - name: ind_var29\n type: binary\n - name: ind_var30_0\n type: binary\n - name: ind_var30\n type: binary\n - name: ind_var31_0\n type: binary\n - name: ind_var31\n type: binary\n - name: ind_var32_cte\n type: binary\n - name: ind_var32_0\n type: binary\n - name: ind_var32\n type: binary\n - name: ind_var33_0\n type: binary\n - name: ind_var33\n type: binary\n - name: ind_var34_0\n type: binary\n - name: ind_var34\n type: binary\n - name: ind_var37_cte\n type: binary\n - name: ind_var37_0\n type: binary\n - name: ind_var37\n type: binary\n - name: ind_var39_0\n type: binary\n - name: ind_var40_0\n type: binary\n - name: ind_var40\n type: binary\n - name: ind_var41_0\n type: binary\n - name: ind_var41\n type: binary\n - name: ind_var39\n type: binary\n - name: ind_var44_0\n type: binary\n - name: ind_var44\n type: binary\n - name: ind_var46_0\n type: binary\n - name: ind_var46\n type: binary\n - name: num_var1_0\n type: number\n - name: num_var1\n type: number\n - name: num_var4\n type: category\n - name: num_var5_0\n type: number\n - name: num_var5\n type: number\n - name: num_var6_0\n type: number\n - name: num_var6\n type: number\n - name: num_var8_0\n type: number\n - name: num_var8\n type: number\n - name: num_var12_0\n type: number\n - name: num_var12\n type: number\n - name: num_var13_0\n type: number\n - name: num_var13_corto_0\n type: number\n - name: num_var13_corto\n type: number\n - name: num_var13_largo_0\n type: number\n - name: num_var13_largo\n type: number\n - name: num_var13_medio_0\n type: number\n - name: num_var13_medio\n type: number\n - name: num_var13\n type: number\n - name: num_var14_0\n type: number\n - name: num_var14\n type: number\n - name: num_var17_0\n type: number\n - name: num_var17\n type: number\n - name: num_var18_0\n type: number\n - name: num_var18\n type: number\n - name: num_var20_0\n type: number\n - name: num_var20\n type: number\n - name: num_var24_0\n type: number\n - name: num_var24\n type: number\n - name: num_var26_0\n type: number\n - name: num_var26\n type: number\n - name: num_var25_0\n type: number\n - name: num_var25\n type: number\n - name: num_op_var40_hace2\n type: number\n - name: num_op_var40_hace3\n type: number\n - name: num_op_var40_ult1\n type: number\n - name: num_op_var40_ult3\n type: number\n - name: num_op_var41_hace2\n type: number\n - name: num_op_var41_hace3\n type: number\n - name: num_op_var41_ult1\n type: number\n - name: num_op_var41_ult3\n type: number\n - name: num_op_var39_hace2\n type: number\n - name: num_op_var39_hace3\n type: number\n - name: num_op_var39_ult1\n type: number\n - name: num_op_var39_ult3\n type: number\n - name: num_var27_0\n type: binary\n - name: num_var28_0\n type: binary\n - name: num_var28\n type: binary\n - name: num_var27\n type: binary\n - name: num_var29_0\n type: number\n - name: num_var29\n type: number\n - name: num_var30_0\n type: number\n - name: num_var30\n type: number\n - name: num_var31_0\n type: number\n - name: num_var31\n type: number\n - name: num_var32_0\n type: number\n - name: num_var32\n type: number\n - name: num_var33_0\n type: number\n - name: num_var33\n type: number\n - name: num_var34_0\n type: number\n - name: num_var34\n type: number\n - name: num_var35\n type: number\n - name: num_var37_med_ult2\n type: number\n - name: num_var37_0\n type: number\n - name: num_var37\n type: number\n - name: num_var39_0\n type: number\n - name: num_var40_0\n type: number\n - name: num_var40\n type: number\n - name: num_var41_0\n type: number\n - name: num_var41\n type: binary\n - name: num_var39\n type: number\n - name: num_var42_0\n type: number\n - name: num_var42\n type: number\n - name: num_var44_0\n type: number\n - name: num_var44\n type: number\n - name: num_var46_0\n type: binary\n - name: num_var46\n type: binary\n - name: saldo_var1\n type: number\n - name: saldo_var5\n type: number\n - name: saldo_var6\n type: number\n - name: saldo_var8\n type: number\n - name: saldo_var12\n type: number\n - name: saldo_var13_corto\n type: number\n - name: saldo_var13_largo\n type: number\n - name: saldo_var13_medio\n type: number\n - name: saldo_var13\n type: number\n - name: saldo_var14\n type: number\n - name: saldo_var17\n type: number\n - name: saldo_var18\n type: number\n - name: saldo_var20\n type: number\n - name: saldo_var24\n type: number\n - name: saldo_var26\n type: number\n - name: saldo_var25\n type: number\n - name: saldo_var28\n type: binary\n - name: saldo_var27\n type: binary\n - name: saldo_var29\n type: number\n - name: saldo_var30\n type: number\n - name: saldo_var31\n type: number\n - name: saldo_var32\n type: number\n - name: saldo_var33\n type: number\n - name: saldo_var34\n type: number\n - name: saldo_var37\n type: number\n - name: saldo_var40\n type: number\n - name: saldo_var41\n type: binary\n - name: saldo_var42\n type: number\n - name: saldo_var44\n type: number\n - name: saldo_var46\n type: binary\n - name: var36\n type: number\n - name: delta_imp_amort_var18_1y3\n type: number\n - name: delta_imp_amort_var34_1y3\n type: number\n - name: delta_imp_aport_var13_1y3\n type: number\n - name: delta_imp_aport_var17_1y3\n type: number\n - name: delta_imp_aport_var33_1y3\n type: number\n - name: delta_imp_compra_var44_1y3\n type: number\n - name: delta_imp_reemb_var13_1y3\n type: number\n - name: delta_imp_reemb_var17_1y3\n type: number\n - name: delta_imp_reemb_var33_1y3\n type: number\n - name: delta_imp_trasp_var17_in_1y3\n type: number\n - name: delta_imp_trasp_var17_out_1y3\n type: number\n - name: delta_imp_trasp_var33_in_1y3\n type: number\n - name: delta_imp_trasp_var33_out_1y3\n type: number\n - name: delta_imp_venta_var44_1y3\n type: number\n - name: delta_num_aport_var13_1y3\n type: number\n - name: delta_num_aport_var17_1y3\n type: number\n - name: delta_num_aport_var33_1y3\n type: number\n - name: delta_num_compra_var44_1y3\n type: number\n - name: delta_num_reemb_var13_1y3\n type: number\n - name: delta_num_reemb_var17_1y3\n type: number\n - name: delta_num_reemb_var33_1y3\n type: number\n - name: delta_num_trasp_var17_in_1y3\n type: number\n - name: delta_num_trasp_var17_out_1y3\n type: number\n - name: delta_num_trasp_var33_in_1y3\n type: number\n - name: delta_num_trasp_var33_out_1y3\n type: number\n - name: delta_num_venta_var44_1y3\n type: number\n - name: imp_amort_var18_hace3\n type: binary\n - name: imp_amort_var18_ult1\n type: number\n - name: imp_amort_var34_hace3\n type: binary\n - name: imp_amort_var34_ult1\n type: number\n - name: imp_aport_var13_hace3\n type: number\n - name: imp_aport_var13_ult1\n type: number\n - name: imp_aport_var17_hace3\n type: number\n - name: imp_aport_var17_ult1\n type: number\n - name: imp_aport_var33_hace3\n type: number\n - name: imp_aport_var33_ult1\n type: number\n - name: imp_var7_emit_ult1\n type: number\n - name: imp_var7_recib_ult1\n type: number\n - name: imp_compra_var44_hace3\n type: number\n - name: imp_compra_var44_ult1\n type: number\n - name: imp_reemb_var13_hace3\n type: binary\n - name: imp_reemb_var13_ult1\n type: number\n - name: imp_reemb_var17_hace3\n type: number\n - name: imp_reemb_var17_ult1\n type: number\n - name: imp_reemb_var33_hace3\n type: binary\n - name: imp_reemb_var33_ult1\n type: number\n - name: imp_var43_emit_ult1\n type: number\n - name: imp_trans_var37_ult1\n type: number\n - name: imp_trasp_var17_in_hace3\n type: number\n - name: imp_trasp_var17_in_ult1\n type: number\n - name: imp_trasp_var17_out_hace3\n type: binary\n - name: imp_trasp_var17_out_ult1\n type: number\n - name: imp_trasp_var33_in_hace3\n type: number\n - name: imp_trasp_var33_in_ult1\n type: number\n - name: imp_trasp_var33_out_hace3\n type: binary\n - name: imp_trasp_var33_out_ult1\n type: number\n - name: imp_venta_var44_hace3\n type: number\n - name: imp_venta_var44_ult1\n type: number\n - name: ind_var7_emit_ult1\n type: binary\n - name: ind_var7_recib_ult1\n type: binary\n - name: ind_var10_ult1\n type: binary\n - name: ind_var10cte_ult1\n type: binary\n - name: ind_var9_cte_ult1\n type: binary\n - name: ind_var9_ult1\n type: binary\n - name: ind_var43_emit_ult1\n type: binary\n - name: ind_var43_recib_ult1\n type: binary\n - name: var21\n type: number\n - name: num_var2_0_ult1\n type: binary\n - name: num_var2_ult1\n type: binary\n - name: num_aport_var13_hace3\n type: number\n - name: num_aport_var13_ult1\n type: number\n - name: num_aport_var17_hace3\n type: number\n - name: num_aport_var17_ult1\n type: number\n - name: num_aport_var33_hace3\n type: number\n - name: num_aport_var33_ult1\n type: number\n - name: num_var7_emit_ult1\n type: number\n - name: num_var7_recib_ult1\n type: number\n - name: num_compra_var44_hace3\n type: number\n - name: num_compra_var44_ult1\n type: number\n - name: num_ent_var16_ult1\n type: number\n - name: num_var22_hace2\n type: number\n - name: num_var22_hace3\n type: number\n - name: num_var22_ult1\n type: number\n - name: num_var22_ult3\n type: number\n - name: num_med_var22_ult3\n type: number\n - name: num_med_var45_ult3\n type: number\n - name: num_meses_var5_ult3\n type: category\n - name: num_meses_var8_ult3\n type: category\n - name: num_meses_var12_ult3\n type: category\n - name: num_meses_var13_corto_ult3\n type: category\n - name: num_meses_var13_largo_ult3\n type: category\n - name: num_meses_var13_medio_ult3\n type: number\n - name: num_meses_var17_ult3\n type: category\n - name: num_meses_var29_ult3\n type: category\n - name: num_meses_var33_ult3\n type: category\n - name: num_meses_var39_vig_ult3\n type: category\n - name: num_meses_var44_ult3\n type: category\n - name: num_op_var39_comer_ult1\n type: number\n - name: num_op_var39_comer_ult3\n type: number\n - name: num_op_var40_comer_ult1\n type: number\n - name: num_op_var40_comer_ult3\n type: number\n - name: num_op_var40_efect_ult1\n type: number\n - name: num_op_var40_efect_ult3\n type: number\n - name: num_op_var41_comer_ult1\n type: number\n - name: num_op_var41_comer_ult3\n type: number\n - name: num_op_var41_efect_ult1\n type: number\n - name: num_op_var41_efect_ult3\n type: number\n - name: num_op_var39_efect_ult1\n type: number\n - name: num_op_var39_efect_ult3\n type: number\n - name: num_reemb_var13_hace3\n type: binary\n - name: num_reemb_var13_ult1\n type: number\n - name: num_reemb_var17_hace3\n type: number\n - name: num_reemb_var17_ult1\n type: number\n - name: num_reemb_var33_hace3\n type: binary\n - name: num_reemb_var33_ult1\n type: number\n - name: num_sal_var16_ult1\n type: number\n - name: num_var43_emit_ult1\n type: number\n - name: num_var43_recib_ult1\n type: number\n - name: num_trasp_var11_ult1\n type: number\n - name: num_trasp_var17_in_hace3\n type: number\n - name: num_trasp_var17_in_ult1\n type: number\n - name: num_trasp_var17_out_hace3\n type: binary\n - name: num_trasp_var17_out_ult1\n type: number\n - name: num_trasp_var33_in_hace3\n type: number\n - name: num_trasp_var33_in_ult1\n type: number\n - name: num_trasp_var33_out_hace3\n type: binary\n - name: num_trasp_var33_out_ult1\n type: number\n - name: num_venta_var44_hace3\n type: number\n - name: num_venta_var44_ult1\n type: number\n - name: num_var45_hace2\n type: number\n - name: num_var45_hace3\n type: number\n - name: num_var45_ult1\n type: number\n - name: num_var45_ult3\n type: number\n - name: saldo_var2_ult1\n type: binary\n - name: saldo_medio_var5_hace2\n type: number\n - name: saldo_medio_var5_hace3\n type: number\n - name: saldo_medio_var5_ult1\n type: number\n - name: saldo_medio_var5_ult3\n type: number\n - name: saldo_medio_var8_hace2\n type: number\n - name: saldo_medio_var8_hace3\n type: number\n - name: saldo_medio_var8_ult1\n type: number\n - name: saldo_medio_var8_ult3\n type: number\n - name: saldo_medio_var12_hace2\n type: number\n - name: saldo_medio_var12_hace3\n type: number\n - name: saldo_medio_var12_ult1\n type: number\n - name: saldo_medio_var12_ult3\n type: number\n - name: saldo_medio_var13_corto_hace2\n type: number\n - name: saldo_medio_var13_corto_hace3\n type: number\n - name: saldo_medio_var13_corto_ult1\n type: number\n - name: saldo_medio_var13_corto_ult3\n type: number\n - name: saldo_medio_var13_largo_hace2\n type: number\n - name: saldo_medio_var13_largo_hace3\n type: number\n - name: saldo_medio_var13_largo_ult1\n type: number\n - name: saldo_medio_var13_largo_ult3\n type: number\n - name: saldo_medio_var13_medio_hace2\n type: number\n - name: saldo_medio_var13_medio_hace3\n type: binary\n - name: saldo_medio_var13_medio_ult1\n type: number\n - name: saldo_medio_var13_medio_ult3\n type: number\n - name: saldo_medio_var17_hace2\n type: number\n - name: saldo_medio_var17_hace3\n type: number\n - name: saldo_medio_var17_ult1\n type: number\n - name: saldo_medio_var17_ult3\n type: number\n - name: saldo_medio_var29_hace2\n type: number\n - name: saldo_medio_var29_hace3\n type: number\n - name: saldo_medio_var29_ult1\n type: number\n - name: saldo_medio_var29_ult3\n type: number\n - name: saldo_medio_var33_hace2\n type: number\n - name: saldo_medio_var33_hace3\n type: number\n - name: saldo_medio_var33_ult1\n type: number\n - name: saldo_medio_var33_ult3\n type: number\n - name: saldo_medio_var44_hace2\n type: number\n - name: saldo_medio_var44_hace3\n type: number\n - name: saldo_medio_var44_ult1\n type: number\n - name: saldo_medio_var44_ult3\n type: number\n - name: var38\n type: number\n combiner:\n type: tabnet\n size: 24 # N_a\n output_size: 128 # N_d\n sparsity: 0.001 # lambda_sparse\n bn_momentum: 0.2 # m_B\n num_steps: 7 # N_steps\n relaxation_factor: 1.2 # gamma\n bn_virtual_bs: 256 # B_v\n trainer:\n batch_size: 4096 # B\n eval_batch_size: null # 65536 131072 262144 524288\n epochs: 300\n early_stop: 30\n learning_rate: 0.005\n optimizer:\n type: adam\n learning_rate_scheduler:\n decay: exponential\n decay_steps: 10000\n decay_rate: 0.8\n validation_metric: accuracy\n - name: santander_customer_transaction\n goal: maximize\n metric: accuracy\n validation_metric_score: 0.9150915145874023\n training_rows: 139904\n test_rows: 40098\n validation_rows: 19998\n config:\n output_features:\n - name: target\n type: binary\n input_features:\n - name: var_0\n type: number\n - name: var_1\n type: number\n - name: var_2\n type: number\n - name: var_3\n type: number\n - name: var_4\n type: number\n - name: var_5\n type: number\n - name: var_6\n type: number\n - name: var_7\n type: number\n - name: var_8\n type: number\n - name: var_9\n type: number\n - name: var_10\n type: number\n - name: var_11\n type: number\n - name: var_12\n type: number\n - name: var_13\n type: number\n - name: var_14\n type: number\n - name: var_15\n type: number\n - name: var_16\n type: number\n - name: var_17\n type: number\n - name: var_18\n type: number\n - name: var_19\n type: number\n - name: var_20\n type: number\n - name: var_21\n type: number\n - name: var_22\n type: number\n - name: var_23\n type: number\n - name: var_24\n type: number\n - name: var_25\n type: number\n - name: var_26\n type: number\n - name: var_27\n type: number\n - name: var_28\n type: number\n - name: var_29\n type: number\n - name: var_30\n type: number\n - name: var_31\n type: number\n - name: var_32\n type: number\n - name: var_33\n type: number\n - name: var_34\n type: number\n - name: var_35\n type: number\n - name: var_36\n type: number\n - name: var_37\n type: number\n - name: var_38\n type: number\n - name: var_39\n type: number\n - name: var_40\n type: number\n - name: var_41\n type: number\n - name: var_42\n type: number\n - name: var_43\n type: number\n - name: var_44\n type: number\n - name: var_45\n type: number\n - name: var_46\n type: number\n - name: var_47\n type: number\n - name: var_48\n type: number\n - name: var_49\n type: number\n - name: var_50\n type: number\n - name: var_51\n type: number\n - name: var_52\n type: number\n - name: var_53\n type: number\n - name: var_54\n type: number\n - name: var_55\n type: number\n - name: var_56\n type: number\n - name: var_57\n type: number\n - name: var_58\n type: number\n - name: var_59\n type: number\n - name: var_60\n type: number\n - name: var_61\n type: number\n - name: var_62\n type: number\n - name: var_63\n type: number\n - name: var_64\n type: number\n - name: var_65\n type: number\n - name: var_66\n type: number\n - name: var_67\n type: number\n - name: var_68\n type: number\n - name: var_69\n type: number\n - name: var_70\n type: number\n - name: var_71\n type: number\n - name: var_72\n type: number\n - name: var_73\n type: number\n - name: var_74\n type: number\n - name: var_75\n type: number\n - name: var_76\n type: number\n - name: var_77\n type: number\n - name: var_78\n type: number\n - name: var_79\n type: number\n - name: var_80\n type: number\n - name: var_81\n type: number\n - name: var_82\n type: number\n - name: var_83\n type: number\n - name: var_84\n type: number\n - name: var_85\n type: number\n - name: var_86\n type: number\n - name: var_87\n type: number\n - name: var_88\n type: number\n - name: var_89\n type: number\n - name: var_90\n type: number\n - name: var_91\n type: number\n - name: var_92\n type: number\n - name: var_93\n type: number\n - name: var_94\n type: number\n - name: var_95\n type: number\n - name: var_96\n type: number\n - name: var_97\n type: number\n - name: var_98\n type: number\n - name: var_99\n type: number\n - name: var_100\n type: number\n - name: var_101\n type: number\n - name: var_102\n type: number\n - name: var_103\n type: number\n - name: var_104\n type: number\n - name: var_105\n type: number\n - name: var_106\n type: number\n - name: var_107\n type: number\n - name: var_108\n type: number\n - name: var_109\n type: number\n - name: var_110\n type: number\n - name: var_111\n type: number\n - name: var_112\n type: number\n - name: var_113\n type: number\n - name: var_114\n type: number\n - name: var_115\n type: number\n - name: var_116\n type: number\n - name: var_117\n type: number\n - name: var_118\n type: number\n - name: var_119\n type: number\n - name: var_120\n type: number\n - name: var_121\n type: number\n - name: var_122\n type: number\n - name: var_123\n type: number\n - name: var_124\n type: number\n - name: var_125\n type: number\n - name: var_126\n type: number\n - name: var_127\n type: number\n - name: var_128\n type: number\n - name: var_129\n type: number\n - name: var_130\n type: number\n - name: var_131\n type: number\n - name: var_132\n type: number\n - name: var_133\n type: number\n - name: var_134\n type: number\n - name: var_135\n type: number\n - name: var_136\n type: number\n - name: var_137\n type: number\n - name: var_138\n type: number\n - name: var_139\n type: number\n - name: var_140\n type: number\n - name: var_141\n type: number\n - name: var_142\n type: number\n - name: var_143\n type: number\n - name: var_144\n type: number\n - name: var_145\n type: number\n - name: var_146\n type: number\n - name: var_147\n type: number\n - name: var_148\n type: number\n - name: var_149\n type: number\n - name: var_150\n type: number\n - name: var_151\n type: number\n - name: var_152\n type: number\n - name: var_153\n type: number\n - name: var_154\n type: number\n - name: var_155\n type: number\n - name: var_156\n type: number\n - name: var_157\n type: number\n - name: var_158\n type: number\n - name: var_159\n type: number\n - name: var_160\n type: number\n - name: var_161\n type: number\n - name: var_162\n type: number\n - name: var_163\n type: number\n - name: var_164\n type: number\n - name: var_165\n type: number\n - name: var_166\n type: number\n - name: var_167\n type: number\n - name: var_168\n type: number\n - name: var_169\n type: number\n - name: var_170\n type: number\n - name: var_171\n type: number\n - name: var_172\n type: number\n - name: var_173\n type: number\n - name: var_174\n type: number\n - name: var_175\n type: number\n - name: var_176\n type: number\n - name: var_177\n type: number\n - name: var_178\n type: number\n - name: var_179\n type: number\n - name: var_180\n type: number\n - name: var_181\n type: number\n - name: var_182\n type: number\n - name: var_183\n type: number\n - name: var_184\n type: number\n - name: var_185\n type: number\n - name: var_186\n type: number\n - name: var_187\n type: number\n - name: var_188\n type: number\n - name: var_189\n type: number\n - name: var_190\n type: number\n - name: var_191\n type: number\n - name: var_192\n type: number\n - name: var_193\n type: number\n - name: var_194\n type: number\n - name: var_195\n type: number\n - name: var_196\n type: number\n - name: var_197\n type: number\n - name: var_198\n type: number\n - name: var_199\n type: number\n combiner:\n type: tabnet\n size: 8 # N_a\n output_size: 8 # N_d\n sparsity: 0.0 # lambda_sparse\n bn_momentum: 0.4 # m_B\n num_steps: 3 # N_steps\n relaxation_factor: 2.0 # gamma\n bn_virtual_bs: 256 # B_v\n trainer:\n batch_size: 256 # B\n eval_batch_size: null # 65536 131072 262144 524288\n epochs: 300\n early_stop: 30\n learning_rate: 0.005\n optimizer:\n type: adam\n learning_rate_scheduler:\n decay: exponential\n decay_steps: 20000\n decay_rate: 0.95\n validation_metric: accuracy\n - name: sarcos\n goal: minimize\n metric: root_mean_squared_error\n validation_metric_score: 2.0124664306640625\n training_rows: 40036\n test_rows: 0\n validation_rows: 4448\n config:\n output_features:\n - name: torque_1\n type: number\n input_features:\n - name: position_1\n type: number\n - name: position_2\n type: number\n - name: position_3\n type: number\n - name: position_4\n type: number\n - name: position_5\n type: number\n - name: position_6\n type: number\n - name: position_7\n type: number\n - name: velocity_1\n type: number\n - name: velocity_2\n type: number\n - name: velocity_3\n type: number\n - name: velocity_4\n type: number\n - name: velocity_5\n type: number\n - name: velocity_6\n type: number\n - name: velocity_7\n type: number\n - name: acceleration_1\n type: number\n - name: acceleration_2\n type: number\n - name: acceleration_3\n type: number\n - name: acceleration_4\n type: number\n - name: acceleration_5\n type: number\n - name: acceleration_6\n type: number\n - name: acceleration_7\n type: number\n combiner:\n type: tabnet\n size: 128 # N_a\n output_size: 8 # N_d\n sparsity: 0.000001 # lambda_sparse\n bn_momentum: 0.02 # m_B\n num_steps: 4 # N_steps\n relaxation_factor: 1.2 # gamma\n bn_virtual_bs: 4096 # B_v\n trainer:\n batch_size: 256 # B\n eval_batch_size: null # 65536 131072 262144 524288\n epochs: 300\n early_stop: 30\n learning_rate: 0.005\n optimizer:\n type: adam\n learning_rate_scheduler:\n decay: exponential\n decay_steps: 20000\n decay_rate: 0.4\n validation_metric: root_mean_squared_error\n - name: walmart_recruiting\n goal: maximize\n metric: accuracy\n validation_metric_score: 0.31689465045928955\n training_rows: 453154\n test_rows: 129276\n validation_rows: 64624\n config:\n output_features:\n - name: TripType\n type: category\n input_features:\n - name: VisitNumber\n type: number\n - name: Weekday\n type: category\n - name: Upc\n type: number\n - name: ScanCount\n type: number\n - name: FinelineNumber\n type: number\n combiner:\n type: tabnet\n size: 32 # N_a\n output_size: 128 # N_d\n sparsity: 0.000001 # lambda_sparse\n bn_momentum: 0.4 # m_B\n num_steps: 4 # N_steps\n relaxation_factor: 1.2 # gamma\n bn_virtual_bs: 4096 # B_v\n trainer:\n batch_size: 8192 # B\n eval_batch_size: null # 65536 131072 262144 524288\n epochs: 300\n early_stop: 30\n learning_rate: 0.01\n optimizer:\n type: adam\n learning_rate_scheduler:\n decay: exponential\n decay_steps: 20000\n decay_rate: 0.9\n validation_metric: accuracy\n"
},
{
"path": "ludwig/automl/defaults/text/bert_config.yaml",
"content": "trainer:\n epochs: 10\n learning_rate_scheduler:\n warmup_fraction: 0.1\n decay: linear\n optimizer:\n type: adamw\n use_mixed_precision: true\n\ndefaults:\n text:\n encoder:\n type: bert\n trainable: true\n\nhyperopt:\n # goal: maximize\n parameters:\n # This parameter space was updated to be loguniform because of issues merging with the trainer.learning_rate\n # parameter space in ludwig/automl/defaults/combiner/concat_config.yaml. Doing automl on a text feature would\n # create an invalid combination of loguniform and choice paramters.\n # TODO(jeffkinnison): Add a second pass `merge_dicts` to handle parameter spaces\n trainer.learning_rate:\n space: loguniform\n lower: 0.00002\n upper: 0.00003\n trainer.batch_size:\n space: choice\n categories: [16, 32, 64, 128]\n"
},
{
"path": "ludwig/backend/__init__.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2020 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nimport contextlib\nimport logging\nimport os\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.backend.base import Backend, LocalBackend\n\nlogger = logging.getLogger(__name__)\n\n\n# TODO: remove LOCAL_BACKEND as a global constant, replace with singleton LocalBackend.shared_instance().\nLOCAL_BACKEND = LocalBackend.shared_instance()\n\n\nLOCAL = \"local\"\nDASK = \"dask\"\nDEEPSPEED = \"deepspeed\"\nRAY = \"ray\"\n\nALL_BACKENDS = [LOCAL, DASK, DEEPSPEED, RAY]\n\n\ndef _has_ray():\n # Temporary workaround to prevent tests from automatically using the Ray backend. Taken from\n # https://stackoverflow.com/questions/25188119/test-if-code-is-executed-from-within-a-py-test-session\n if \"PYTEST_CURRENT_TEST\" in os.environ:\n return False\n\n try:\n import ray\n except ImportError:\n return False\n\n if ray.is_initialized():\n return True\n\n try:\n ray.init(\"auto\", ignore_reinit_error=True)\n return True\n except Exception:\n return False\n\n\ndef get_local_backend(**kwargs):\n return LocalBackend(**kwargs)\n\n\ndef create_deepspeed_backend(**kwargs):\n from ludwig.backend.deepspeed import DeepSpeedBackend\n\n return DeepSpeedBackend(**kwargs)\n\n\ndef create_ray_backend(**kwargs):\n from ludwig.backend.ray import RayBackend\n\n return RayBackend(**kwargs)\n\n\nbackend_registry = {\n LOCAL: get_local_backend,\n DEEPSPEED: create_deepspeed_backend,\n RAY: create_ray_backend,\n None: get_local_backend,\n}\n\n\n@DeveloperAPI\ndef create_backend(type, **kwargs):\n if isinstance(type, Backend):\n return type\n\n if type is None and _has_ray():\n type = RAY\n\n return backend_registry[type](**kwargs)\n\n\n@DeveloperAPI\ndef initialize_backend(backend):\n if isinstance(backend, dict):\n backend = create_backend(**backend)\n else:\n backend = create_backend(backend)\n backend.initialize()\n return backend\n\n\n@contextlib.contextmanager\ndef provision_preprocessing_workers(backend):\n if backend.BACKEND_TYPE == RAY:\n with backend.provision_preprocessing_workers():\n yield\n else:\n yield\n"
},
{
"path": "ludwig/backend/base.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2020 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nfrom __future__ import annotations\n\nimport time\nfrom abc import ABC, abstractmethod\nfrom collections.abc import Callable, Generator\nfrom concurrent.futures import ThreadPoolExecutor\nfrom contextlib import contextmanager\nfrom typing import Any, TYPE_CHECKING\n\nimport numpy as np\nimport pandas as pd\nimport psutil\nimport torch\nfrom tqdm import tqdm\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.backend.utils.storage import StorageManager\nfrom ludwig.constants import MODEL_LLM\nfrom ludwig.data.cache.manager import CacheManager\nfrom ludwig.data.dataframe.base import DataFrameEngine\nfrom ludwig.data.dataframe.pandas import PANDAS\nfrom ludwig.data.dataset.base import DatasetManager\nfrom ludwig.data.dataset.pandas import PandasDatasetManager\nfrom ludwig.distributed import init_dist_strategy\nfrom ludwig.distributed.base import DistributedStrategy\nfrom ludwig.models.base import BaseModel\nfrom ludwig.schema.trainer import BaseTrainerConfig\nfrom ludwig.types import HyperoptConfigDict\nfrom ludwig.utils.audio_utils import read_audio_from_path\nfrom ludwig.utils.batch_size_tuner import BatchSizeEvaluator\nfrom ludwig.utils.dataframe_utils import from_batches, to_batches\nfrom ludwig.utils.fs_utils import get_bytes_obj_from_path\nfrom ludwig.utils.misc_utils import get_from_registry\nfrom ludwig.utils.system_utils import Resources\nfrom ludwig.utils.torch_utils import initialize_pytorch\nfrom ludwig.utils.types import DataFrame, Series\n\nif TYPE_CHECKING:\n from ludwig.trainers.base import BaseTrainer\n\n\n@DeveloperAPI\nclass Backend(ABC):\n def __init__(\n self,\n dataset_manager: DatasetManager,\n cache_dir: str | None = None,\n credentials: dict[str, dict[str, Any]] | None = None,\n ):\n credentials = credentials or {}\n self._dataset_manager = dataset_manager\n self._storage_manager = StorageManager(**credentials)\n self._cache_manager = CacheManager(self._dataset_manager, cache_dir)\n\n @property\n def storage(self) -> StorageManager:\n return self._storage_manager\n\n @property\n def cache(self) -> CacheManager:\n return self._cache_manager\n\n @property\n def dataset_manager(self) -> DatasetManager:\n return self._dataset_manager\n\n @abstractmethod\n def initialize(self):\n raise NotImplementedError()\n\n @abstractmethod\n def initialize_pytorch(self, *args, **kwargs):\n raise NotImplementedError()\n\n @contextmanager\n @abstractmethod\n def create_trainer(self, config: BaseTrainerConfig, model: BaseModel, **kwargs) -> Generator:\n raise NotImplementedError()\n\n @abstractmethod\n def sync_model(self, model):\n raise NotImplementedError()\n\n @abstractmethod\n def broadcast_return(self, fn):\n raise NotImplementedError()\n\n @abstractmethod\n def is_coordinator(self):\n raise NotImplementedError()\n\n @property\n @abstractmethod\n def df_engine(self) -> DataFrameEngine:\n raise NotImplementedError()\n\n @property\n @abstractmethod\n def supports_multiprocessing(self):\n raise NotImplementedError()\n\n @abstractmethod\n def read_binary_files(self, column: Series, map_fn: Callable | None = None) -> Series:\n raise NotImplementedError()\n\n @property\n @abstractmethod\n def num_nodes(self) -> int:\n raise NotImplementedError()\n\n @property\n @abstractmethod\n def num_training_workers(self) -> int:\n raise NotImplementedError()\n\n @abstractmethod\n def get_available_resources(self) -> Resources:\n raise NotImplementedError()\n\n @abstractmethod\n def max_concurrent_trials(self, hyperopt_config: HyperoptConfigDict) -> int | None:\n raise NotImplementedError()\n\n @abstractmethod\n def tune_batch_size(self, evaluator_cls: type[BatchSizeEvaluator], dataset_len: int) -> int:\n \"\"\"Returns best batch size (measured in samples / s) on the given evaluator.\n\n The evaluator class will need to be instantiated on each worker in the backend cluster, then call\n `evaluator.select_best_batch_size(dataset_len)`.\n \"\"\"\n raise NotImplementedError()\n\n @abstractmethod\n def batch_transform(\n self, df: DataFrame, batch_size: int, transform_fn: Callable, name: str | None = None\n ) -> DataFrame:\n \"\"\"Applies `transform_fn` to every `batch_size` length batch of `df` and returns the result.\"\"\"\n raise NotImplementedError()\n\n def supports_batch_size_tuning(self) -> bool:\n return True\n\n\nclass LocalPreprocessingMixin:\n @property\n def df_engine(self):\n return PANDAS\n\n @property\n def supports_multiprocessing(self):\n return True\n\n @staticmethod\n def read_binary_files(column: pd.Series, map_fn: Callable | None = None, file_size: int | None = None) -> pd.Series:\n column = column.fillna(np.nan).replace([np.nan], [None]) # normalize NaNs to None\n\n sample_fname = column.head(1).values[0]\n with ThreadPoolExecutor() as executor: # number of threads is inferred\n if isinstance(sample_fname, str):\n if map_fn is read_audio_from_path: # bypass torchaudio issue that no longer takes in file-like objects\n result = executor.map( # type: ignore[misc]\n lambda path: map_fn(path) if path is not None else path, column.values\n )\n else:\n result = executor.map(\n lambda path: get_bytes_obj_from_path(path) if path is not None else path, column.values\n )\n else:\n # If the sample path is not a string, assume the paths has already been read in\n result = column.values\n\n if map_fn is not None and map_fn is not read_audio_from_path:\n result = executor.map(map_fn, result)\n\n return pd.Series(result, index=column.index, name=column.name)\n\n @staticmethod\n def batch_transform(df: DataFrame, batch_size: int, transform_fn: Callable, name: str | None = None) -> DataFrame:\n name = name or \"Batch Transform\"\n batches = to_batches(df, batch_size)\n transform = transform_fn()\n out_batches = [transform(batch.reset_index(drop=True)) for batch in tqdm(batches, desc=name)]\n out_df = from_batches(out_batches).reset_index(drop=True)\n return out_df\n\n\nclass LocalTrainingMixin:\n @staticmethod\n def initialize():\n init_dist_strategy(\"local\")\n\n @staticmethod\n def initialize_pytorch(*args, **kwargs):\n initialize_pytorch(*args, **kwargs)\n\n @staticmethod\n def create_predictor(model: BaseModel, **kwargs):\n from ludwig.models.predictor import get_predictor_cls\n\n return get_predictor_cls(model.type())(model, **kwargs) # type: ignore[call-arg]\n\n def sync_model(self, model):\n pass\n\n @staticmethod\n def broadcast_return(fn):\n return fn()\n\n @staticmethod\n def is_coordinator() -> bool:\n return True\n\n @staticmethod\n def tune_batch_size(evaluator_cls: type[BatchSizeEvaluator], dataset_len: int) -> int:\n evaluator = evaluator_cls()\n return evaluator.select_best_batch_size(dataset_len)\n\n\nclass RemoteTrainingMixin:\n def sync_model(self, model):\n pass\n\n @staticmethod\n def broadcast_return(fn):\n return fn()\n\n @staticmethod\n def is_coordinator() -> bool:\n return True\n\n\n@DeveloperAPI\nclass LocalBackend(LocalPreprocessingMixin, LocalTrainingMixin, Backend):\n BACKEND_TYPE = \"local\"\n\n _shared_instance: LocalBackend\n\n @classmethod\n def shared_instance(cls) -> LocalBackend:\n \"\"\"Returns a shared singleton LocalBackend instance.\"\"\"\n if not hasattr(cls, \"_shared_instance\"):\n cls._shared_instance = cls()\n return cls._shared_instance\n\n def __init__(self, **kwargs) -> None:\n super().__init__(dataset_manager=PandasDatasetManager(self), **kwargs)\n\n @property\n def num_nodes(self) -> int:\n return 1\n\n @property\n def num_training_workers(self) -> int:\n return 1\n\n def get_available_resources(self) -> Resources:\n return Resources(cpus=psutil.cpu_count(), gpus=torch.cuda.device_count())\n\n def max_concurrent_trials(self, hyperopt_config: HyperoptConfigDict) -> int | None:\n # Every trial will be run with Pandas and NO Ray Datasets. Allow Ray Tune to use all the\n # trial resources it wants, because there is no Ray Datasets process to compete with it for CPUs.\n return None\n\n def create_trainer(\n self,\n config: BaseTrainerConfig,\n model: BaseModel,\n **kwargs,\n ) -> BaseTrainer: # type: ignore[override]\n from ludwig.trainers.registry import get_llm_trainers_registry, get_trainers_registry\n\n trainer_cls: type\n if model.type() == MODEL_LLM:\n trainer_cls = get_from_registry(config.type, get_llm_trainers_registry())\n else:\n trainer_cls = get_from_registry(model.type(), get_trainers_registry())\n\n return trainer_cls(config=config, model=model, **kwargs)\n\n\n@DeveloperAPI\nclass DataParallelBackend(LocalPreprocessingMixin, Backend, ABC):\n BACKEND_TYPE = \"deepspeed\"\n\n def __init__(self, **kwargs):\n super().__init__(dataset_manager=PandasDatasetManager(self), **kwargs)\n self._distributed: DistributedStrategy | None = None\n\n @abstractmethod\n def initialize(self):\n pass\n\n def initialize_pytorch(self, *args, **kwargs):\n initialize_pytorch(\n *args, local_rank=self._distributed.local_rank(), local_size=self._distributed.local_size(), **kwargs\n )\n\n def create_trainer(\n self,\n config: BaseTrainerConfig,\n model: BaseModel,\n **kwargs,\n ) -> BaseTrainer: # type: ignore[override]\n from ludwig.trainers.trainer import Trainer\n\n return Trainer(config, model, distributed=self._distributed, **kwargs)\n\n def create_predictor(self, model: BaseModel, **kwargs):\n from ludwig.models.predictor import get_predictor_cls\n\n return get_predictor_cls(model.type())(model, distributed=self._distributed, **kwargs) # type: ignore[call-arg]\n\n def sync_model(self, model):\n # Model weights are only saved on the coordinator, so broadcast\n # to all other ranks\n self._distributed.sync_model(model)\n\n def broadcast_return(self, fn):\n \"\"\"Returns the result of calling `fn` on coordinator, broadcast to all other ranks.\n\n Specifically, `fn` is only executed on coordinator, but its result is returned by every rank by broadcasting the\n return value from coordinator.\n \"\"\"\n result = fn() if self.is_coordinator() else None\n if self._distributed:\n name = f\"broadcast_return_{int(time.time())}\"\n result = self._distributed.broadcast_object(result, name=name)\n return result\n\n def is_coordinator(self):\n return self._distributed.rank() == 0\n\n @property\n def num_nodes(self) -> int:\n return self._distributed.size() // self._distributed.local_size()\n\n @property\n def num_training_workers(self) -> int:\n return self._distributed.size()\n\n def get_available_resources(self) -> Resources:\n # TODO(travis): this double-counts on the same device, it should use a cross-communicator instead\n cpus = torch.as_tensor([psutil.cpu_count()], dtype=torch.int)\n cpus = self._distributed.allreduce(cpus).item()\n\n gpus = torch.as_tensor([torch.cuda.device_count()], dtype=torch.int)\n gpus = self._distributed.allreduce(gpus).item()\n\n return Resources(cpus=cpus, gpus=gpus)\n\n def max_concurrent_trials(self, hyperopt_config: HyperoptConfigDict) -> int | None:\n # Return None since there is no Ray component\n return None\n\n def tune_batch_size(self, evaluator_cls: type[BatchSizeEvaluator], dataset_len: int) -> int:\n evaluator = evaluator_cls()\n return evaluator.select_best_batch_size(dataset_len)\n"
},
{
"path": "ludwig/backend/datasource.py",
"content": "\"\"\"Custom Ray datasource utilities for reading binary files with None handling.\"\"\"\n\nimport logging\nfrom typing import Optional, TYPE_CHECKING\n\nimport pandas as pd\nimport ray\nimport urllib3\n\nfrom ludwig.utils.fs_utils import get_bytes_obj_from_http_path, is_http\n\nif TYPE_CHECKING:\n import pyarrow\n\nlogger = logging.getLogger(__name__)\n\n\ndef read_binary_files_with_index(\n paths_and_idxs: list[tuple[str | None, int]],\n filesystem: Optional[\"pyarrow.fs.FileSystem\"] = None,\n) -> \"ray.data.Dataset\":\n \"\"\"Read binary files into a Ray Dataset, handling None paths and HTTP URLs.\n\n Each row in the resulting dataset has columns:\n - \"data\": the raw bytes of the file (or None if path was None/failed)\n - \"idx\": the original index for reordering\n\n Args:\n paths_and_idxs: List of (path, index) tuples. Path can be None.\n filesystem: PyArrow filesystem for reading non-HTTP files.\n\n Returns:\n A ray.data.Dataset with \"data\" and \"idx\" columns.\n \"\"\"\n\n def _read_file(path: str | None, idx: int) -> dict:\n if path is None:\n return {\"data\": None, \"idx\": idx}\n elif is_http(path):\n try:\n data = get_bytes_obj_from_http_path(path)\n except urllib3.exceptions.HTTPError as e:\n logger.warning(e)\n data = None\n return {\"data\": data, \"idx\": idx}\n else:\n try:\n with filesystem.open_input_stream(path) as f:\n data = f.read()\n except Exception as e:\n logger.warning(f\"Failed to read file {path}: {e}\")\n data = None\n return {\"data\": data, \"idx\": idx}\n\n # Create a dataset from the paths and indices, then map to read files\n records = [{\"path\": p, \"idx\": i} for p, i in paths_and_idxs]\n ds = ray.data.from_items(records)\n\n def read_batch(batch: pd.DataFrame) -> pd.DataFrame:\n results = []\n for _, row in batch.iterrows():\n result = _read_file(row[\"path\"], row[\"idx\"])\n results.append(result)\n return pd.DataFrame(results)\n\n ds = ds.map_batches(read_batch, batch_format=\"pandas\")\n return ds\n"
},
{
"path": "ludwig/backend/deepspeed.py",
"content": "from typing import Any\n\nimport deepspeed\n\nfrom ludwig.backend.base import DataParallelBackend\nfrom ludwig.constants import FALLBACK_BATCH_SIZE\nfrom ludwig.distributed import init_dist_strategy\nfrom ludwig.utils.batch_size_tuner import BatchSizeEvaluator\n\n\nclass DeepSpeedBackend(DataParallelBackend):\n BACKEND_TYPE = \"deepspeed\"\n\n def __init__(\n self,\n zero_optimization: dict[str, Any] | None = None,\n fp16: dict[str, Any] | None = None,\n bf16: dict[str, Any] | None = None,\n compression_training: dict[str, Any] | None = None,\n **kwargs\n ):\n super().__init__(**kwargs)\n self.zero_optimization = zero_optimization\n self.fp16 = fp16\n self.bf16 = bf16\n self.compression_training = compression_training\n\n def initialize(self):\n # Unlike when we use the Ray backend, we need to initialize the `torch.distributed` context so we can\n # broadcast, allgather, etc. before preparing the model within the trainer.\n deepspeed.init_distributed()\n self._distributed = init_dist_strategy(\n self.BACKEND_TYPE,\n zero_optimization=self.zero_optimization,\n fp16=self.fp16,\n bf16=self.bf16,\n compression_training=self.compression_training,\n )\n\n def supports_batch_size_tuning(self) -> bool:\n # TODO(travis): need to fix checkpoint saving/loading for DeepSpeed to enable tuning\n return False\n\n def tune_batch_size(self, evaluator_cls: type[BatchSizeEvaluator], dataset_len: int) -> int:\n return FALLBACK_BATCH_SIZE\n"
},
{
"path": "ludwig/backend/ray.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2020 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nimport contextlib\nimport copy\nimport logging\nimport os\nimport tempfile\nfrom collections.abc import Callable\nfrom functools import partial\nfrom typing import Any, TYPE_CHECKING\n\nimport dask\nimport numpy as np\nimport pandas as pd\nimport ray\nimport ray.train as rt\nimport torch\nimport tqdm\nfrom fsspec.config import conf\nfrom pyarrow.fs import FSSpecHandler, PyFileSystem\nfrom ray import ObjectRef\nfrom ray.train import Checkpoint, RunConfig, ScalingConfig\nfrom ray.train.constants import TRAIN_ENABLE_WORKER_SPREAD_ENV\nfrom ray.train.torch import TorchConfig, TorchTrainer\nfrom ray.util.dask import ray_dask_get\nfrom ray.util.placement_group import placement_group, remove_placement_group\n\nif TYPE_CHECKING:\n from ludwig.api import LudwigModel\n\nfrom ludwig.backend.base import Backend, RemoteTrainingMixin\nfrom ludwig.backend.datasource import read_binary_files_with_index\nfrom ludwig.constants import MODEL_ECD, MODEL_LLM, NAME, PREPROCESSING, PROC_COLUMN, TYPE\nfrom ludwig.data.dataframe.base import DataFrameEngine\n\ntry:\n from ludwig.data.dataset.ray import (\n _SCALAR_TYPES,\n cast_as_tensor_dtype,\n RayDataset,\n RayDatasetManager,\n RayDatasetShard,\n )\nexcept (ImportError, AttributeError):\n _SCALAR_TYPES = cast_as_tensor_dtype = RayDataset = RayDatasetManager = RayDatasetShard = None\nfrom ludwig.models.base import BaseModel\nfrom ludwig.models.ecd import ECD\nfrom ludwig.models.predictor import BasePredictor, get_output_columns, get_predictor_cls\nfrom ludwig.schema.trainer import ECDTrainerConfig\nfrom ludwig.trainers.registry import get_ray_trainers_registry, register_ray_trainer\nfrom ludwig.trainers.trainer import BaseTrainer, RemoteTrainer\nfrom ludwig.utils.data_utils import use_credentials\nfrom ludwig.utils.fs_utils import get_fs_and_path\nfrom ludwig.utils.misc_utils import get_from_registry\nfrom ludwig.utils.system_utils import Resources\nfrom ludwig.utils.torch_utils import get_torch_device, initialize_pytorch\nfrom ludwig.utils.types import Series\n\nlogger = logging.getLogger(__name__)\n\nFIFTEEN_MINS_IN_S = 15 * 60\n\n\ndef _num_nodes() -> int:\n node_resources = [node[\"Resources\"] for node in ray.nodes()]\n return len(node_resources)\n\n\ndef get_trainer_kwargs(**kwargs) -> dict[str, Any]:\n kwargs = copy.deepcopy(kwargs)\n\n # Our goal is to have a worker per resource used for training.\n # The priority is GPUs, but can fall back to CPUs if there are no\n # GPUs available.\n use_gpu = kwargs.get(\"use_gpu\", int(ray.cluster_resources().get(\"GPU\", 0)) > 0)\n if use_gpu:\n num_workers = int(ray.cluster_resources().get(\"GPU\", 0))\n else:\n num_workers = _num_nodes()\n\n # Remove nics if present (legacy option)\n kwargs.pop(\"nics\", None)\n\n defaults = dict(\n backend=TorchConfig(),\n num_workers=num_workers,\n use_gpu=use_gpu,\n resources_per_worker={\n \"CPU\": 0 if use_gpu else 1,\n \"GPU\": 1 if use_gpu else 0,\n },\n )\n return {**defaults, **kwargs}\n\n\ndef _create_dask_engine(**kwargs):\n from ludwig.data.dataframe.dask import DaskEngine\n\n return DaskEngine(**kwargs)\n\n\ndef _create_modin_engine(**kwargs):\n from ludwig.data.dataframe.modin import ModinEngine\n\n return ModinEngine(**kwargs)\n\n\ndef _create_pandas_engine(**kwargs):\n from ludwig.data.dataframe.pandas import PandasEngine\n\n return PandasEngine(**kwargs)\n\n\n_engine_registry = {\n \"dask\": _create_dask_engine,\n \"modin\": _create_modin_engine,\n \"pandas\": _create_pandas_engine,\n}\n\n\ndef _get_df_engine(processor):\n logger.info(f\"Ray processor params: {processor}\")\n if processor is None:\n # TODO ray: find an informed way to set the parallelism, in practice\n # it looks like Dask handles this well on its own most of the time\n return _create_dask_engine()\n\n processor_kwargs = processor.copy()\n\n dtype = processor_kwargs.pop(\"type\", \"dask\")\n engine_cls = _engine_registry.get(dtype)\n\n return engine_cls(**processor_kwargs)\n\n\ndef _make_picklable(obj):\n \"\"\"Recursively convert defaultdicts (which contain unpicklable lambdas) to regular dicts.\"\"\"\n from collections import defaultdict\n\n if isinstance(obj, defaultdict):\n return {k: _make_picklable(v) for k, v in obj.items()}\n elif isinstance(obj, dict):\n return {k: _make_picklable(v) for k, v in obj.items()}\n elif isinstance(obj, tuple) and hasattr(obj, \"_fields\"):\n # NamedTuple: reconstruct with the same field names\n return type(obj)(**{f: _make_picklable(getattr(obj, f)) for f in obj._fields})\n elif isinstance(obj, list):\n return [_make_picklable(item) for item in obj]\n elif isinstance(obj, tuple):\n return tuple(_make_picklable(item) for item in obj)\n return obj\n\n\ndef train_fn(\n executable_kwargs: dict[str, Any] = None,\n model_ref: ObjectRef = None, # noqa: F821\n training_set_metadata: dict[str, Any] = None,\n features: dict[str, dict] = None,\n **kwargs,\n):\n \"\"\"Ray Train worker function for distributed training.\n\n Runs inside each Ray worker process. Loads the model from an object ref, wraps dataset shards, trains, and saves\n results to a Ray checkpoint so the driver can retrieve them (Ray Train 2.x requires a checkpoint for metrics).\n \"\"\"\n # Pin GPU before loading the model to prevent memory leaking onto other devices\n initialize_pytorch()\n\n # Initialize a local distributed strategy so metric modules can sync.\n from ludwig.distributed import init_dist_strategy\n\n init_dist_strategy(\"local\")\n\n train_shard = RayDatasetShard(\n rt.get_dataset_shard(\"train\"),\n features,\n training_set_metadata,\n )\n\n try:\n val_shard = rt.get_dataset_shard(\"val\")\n except KeyError:\n val_shard = None\n\n if val_shard is not None:\n val_shard = RayDatasetShard(\n val_shard,\n features,\n training_set_metadata,\n )\n\n try:\n test_shard = rt.get_dataset_shard(\"test\")\n except KeyError:\n test_shard = None\n\n if test_shard is not None:\n test_shard = RayDatasetShard(\n test_shard,\n features,\n training_set_metadata,\n )\n\n model = ray.get(model_ref)\n # Use Ray Train's device assignment which respects use_gpu setting,\n # rather than get_torch_device() which always picks CUDA if available.\n from ray.train.torch import get_device as ray_get_device\n\n device = ray_get_device()\n model = model.to(device)\n\n trainer = RemoteTrainer(model=model, report_tqdm_to_ray=True, **executable_kwargs)\n results = trainer.train(train_shard, val_shard, test_shard, **kwargs)\n\n if results is not None:\n # only return the model state dict back to the head node.\n trained_model, *args = results\n results = (trained_model.cpu().state_dict(), *args)\n\n torch.cuda.empty_cache()\n\n # Save results to a checkpoint so the driver can retrieve them.\n # In Ray Train 2.x, result.metrics is only populated when a checkpoint is provided.\n train_results = results, trainer.validation_field, trainer.validation_metric\n # Convert defaultdicts to regular dicts so they can be pickled by torch.save.\n train_results = _make_picklable(train_results)\n with tempfile.TemporaryDirectory() as tmpdir:\n torch.save(train_results, os.path.join(tmpdir, \"train_results.pt\"))\n rt.report(metrics={}, checkpoint=Checkpoint.from_directory(tmpdir))\n\n\n@ray.remote\ndef tune_batch_size_fn(\n dataset: RayDataset = None,\n data_loader_kwargs: dict[str, Any] = None,\n executable_kwargs: dict[str, Any] = None,\n model: ECD = None, # noqa: F821\n ludwig_config: dict[str, Any] = None,\n training_set_metadata: dict[str, Any] = None,\n features: dict[str, dict] = None,\n **kwargs,\n) -> int:\n # Pin GPU before loading the model to prevent memory leaking onto other devices\n initialize_pytorch()\n\n try:\n ds = dataset.to_ray_dataset(shuffle=False)\n train_shard = RayDatasetShard(\n ds,\n features,\n training_set_metadata,\n )\n\n device = get_torch_device()\n model = model.to(device)\n\n trainer = RemoteTrainer(model=model, **executable_kwargs)\n return trainer.tune_batch_size(ludwig_config, train_shard, **kwargs)\n finally:\n torch.cuda.empty_cache()\n\n\n@ray.remote\ndef tune_learning_rate_fn(\n dataset: RayDataset,\n config: dict[str, Any],\n data_loader_kwargs: dict[str, Any] = None,\n executable_kwargs: dict[str, Any] = None,\n model: ECD = None, # noqa: F821\n training_set_metadata: dict[str, Any] = None,\n features: dict[str, dict] = None,\n **kwargs,\n) -> float:\n # Pin GPU before loading the model to prevent memory leaking onto other devices\n initialize_pytorch()\n\n try:\n ds = dataset.to_ray_dataset(shuffle=False)\n train_shard = RayDatasetShard(\n ds,\n features,\n training_set_metadata,\n )\n\n device = get_torch_device()\n model = model.to(device)\n\n trainer = RemoteTrainer(model=model, **executable_kwargs)\n return trainer.tune_learning_rate(config, train_shard, **kwargs)\n finally:\n torch.cuda.empty_cache()\n\n\nclass TqdmCallback(rt.UserCallback):\n \"\"\"Class for a custom ray callback that updates tqdm progress bars in the driver process.\"\"\"\n\n def __init__(self) -> None:\n \"\"\"Constructor for TqdmCallback.\"\"\"\n super().__init__()\n self.progess_bars = {}\n\n def after_report(self, run_context, metrics: list[dict], checkpoint=None) -> None:\n \"\"\"Called every time ray.train.report is called from subprocesses.\n\n In Ray 2.x, metrics is a list of metric dicts (one per worker). We look for progress_bar data from the\n coordinator worker.\n \"\"\"\n for result in metrics:\n progress_bar_opts = result.get(\"progress_bar\")\n if not progress_bar_opts:\n continue\n # Skip commands received by non-coordinators\n if not progress_bar_opts[\"is_coordinator\"]:\n continue\n _id = progress_bar_opts[\"id\"]\n action = progress_bar_opts.get(\"action\")\n if action == \"create\":\n progress_bar_config = progress_bar_opts.get(\"config\")\n self.progess_bars[_id] = tqdm.tqdm(**progress_bar_config)\n elif action == \"close\":\n if _id in self.progess_bars:\n self.progess_bars[_id].close()\n elif action == \"update\":\n update_by = progress_bar_opts.get(\"update_by\", 1)\n if _id in self.progess_bars:\n self.progess_bars[_id].update(update_by)\n\n\n@contextlib.contextmanager\ndef spread_env(use_gpu: bool = False, num_workers: int = 1, **kwargs):\n if TRAIN_ENABLE_WORKER_SPREAD_ENV in os.environ:\n # User set this explicitly, so honor their selection\n yield\n return\n\n try:\n if not use_gpu and num_workers > 1:\n # When doing CPU-only training, default to a SPREAD policy to avoid\n # packing too many workers on a single machine\n os.environ[TRAIN_ENABLE_WORKER_SPREAD_ENV] = \"1\"\n yield\n finally:\n if TRAIN_ENABLE_WORKER_SPREAD_ENV in os.environ:\n del os.environ[TRAIN_ENABLE_WORKER_SPREAD_ENV]\n\n\ndef _build_scaling_config(trainer_kwargs: dict[str, Any]) -> ScalingConfig:\n \"\"\"Convert legacy trainer kwargs to a Ray ScalingConfig.\"\"\"\n return ScalingConfig(\n num_workers=trainer_kwargs.get(\"num_workers\", 1),\n use_gpu=trainer_kwargs.get(\"use_gpu\", False),\n resources_per_worker=trainer_kwargs.get(\"resources_per_worker\"),\n )\n\n\ndef run_train_remote(train_loop, trainer_kwargs: dict[str, Any], callbacks=None, datasets=None, train_loop_config=None):\n \"\"\"Run a distributed training function using Ray TorchTrainer.\"\"\"\n resolved_kwargs = get_trainer_kwargs(**trainer_kwargs)\n\n scaling_config = _build_scaling_config(resolved_kwargs)\n torch_config = resolved_kwargs.get(\"backend\", TorchConfig())\n\n run_config_kwargs = {}\n if callbacks:\n run_config_kwargs[\"callbacks\"] = callbacks\n\n with spread_env(**resolved_kwargs):\n torch_trainer = TorchTrainer(\n train_loop_per_worker=train_loop,\n train_loop_config=train_loop_config,\n torch_config=torch_config,\n scaling_config=scaling_config,\n run_config=RunConfig(**run_config_kwargs),\n datasets=datasets,\n )\n result = torch_trainer.fit()\n return result\n\n\n@register_ray_trainer(MODEL_ECD, default=True)\nclass RayTrainerV2(BaseTrainer):\n def __init__(\n self,\n model: BaseModel,\n trainer_kwargs: dict[str, Any],\n data_loader_kwargs: dict[str, Any],\n executable_kwargs: dict[str, Any],\n **kwargs,\n ):\n self.model = model.cpu()\n self.data_loader_kwargs = data_loader_kwargs\n self.executable_kwargs = executable_kwargs\n self.trainer_kwargs = trainer_kwargs\n self._validation_field = None\n self._validation_metric = None\n\n @staticmethod\n def get_schema_cls():\n return ECDTrainerConfig\n\n def train(\n self,\n training_set: RayDataset,\n validation_set: RayDataset | None = None,\n test_set: RayDataset | None = None,\n **kwargs,\n ):\n executable_kwargs = self.executable_kwargs\n\n kwargs = {\n \"training_set_metadata\": training_set.training_set_metadata,\n \"features\": training_set.features,\n **kwargs,\n }\n\n dataset = {\"train\": training_set.to_ray_dataset(shuffle=True)}\n if validation_set is not None:\n dataset[\"val\"] = validation_set.to_ray_dataset(shuffle=False)\n if test_set is not None:\n dataset[\"test\"] = test_set.to_ray_dataset(shuffle=False)\n\n train_loop_config = {\"executable_kwargs\": executable_kwargs, \"model_ref\": ray.put(self.model), **kwargs}\n\n def _train_loop(config):\n train_fn(**config)\n\n result = run_train_remote(\n _train_loop,\n trainer_kwargs=self.trainer_kwargs,\n callbacks=[TqdmCallback()],\n datasets=dataset,\n train_loop_config=train_loop_config,\n )\n\n # Load training results from the checkpoint saved by train_fn\n with result.checkpoint.as_directory() as tmpdir:\n train_results = torch.load(os.path.join(tmpdir, \"train_results.pt\"), weights_only=False)\n results, self._validation_field, self._validation_metric = train_results\n\n # load state dict back into the model\n state_dict, *args = results\n self.model.load_state_dict(state_dict)\n results = (self.model, *args)\n\n return results\n\n def train_online(self, *args, **kwargs):\n # TODO: When this is implemented we also need to update the\n # Tqdm flow to report back the callback\n raise NotImplementedError()\n\n def tune_batch_size(\n self,\n config: dict[str, Any],\n training_set: RayDataset,\n **kwargs,\n ) -> int:\n return ray.get(\n tune_batch_size_fn.options(num_cpus=self.num_cpus, num_gpus=self.num_gpus).remote(\n dataset=training_set,\n data_loader_kwargs=self.data_loader_kwargs,\n executable_kwargs=self.executable_kwargs,\n model=ray.put(self.model),\n ludwig_config=config,\n training_set_metadata=training_set.training_set_metadata,\n features=training_set.features,\n **kwargs,\n )\n )\n\n def tune_learning_rate(self, config, training_set: RayDataset, **kwargs) -> float:\n return ray.get(\n tune_learning_rate_fn.options(num_cpus=self.num_cpus, num_gpus=self.num_gpus).remote(\n dataset=training_set,\n config=config,\n data_loader_kwargs=self.data_loader_kwargs,\n executable_kwargs=self.executable_kwargs,\n model=ray.put(self.model),\n training_set_metadata=training_set.training_set_metadata,\n features=training_set.features,\n **kwargs,\n )\n )\n\n @property\n def validation_field(self):\n return self._validation_field\n\n @property\n def validation_metric(self):\n return self._validation_metric\n\n @property\n def config(self) -> ECDTrainerConfig:\n return self.executable_kwargs[\"config\"]\n\n @property\n def batch_size(self) -> int:\n return self.config.batch_size\n\n @batch_size.setter\n def batch_size(self, value: int):\n self.config.batch_size = value\n\n @property\n def eval_batch_size(self) -> int:\n return self.config.eval_batch_size if self.config.eval_batch_size is not None else self.config.batch_size\n\n @eval_batch_size.setter\n def eval_batch_size(self, value: int):\n self.config.eval_batch_size = value\n\n @property\n def resources_per_worker(self) -> dict[str, Any]:\n trainer_kwargs = get_trainer_kwargs(**self.trainer_kwargs)\n return trainer_kwargs.get(\"resources_per_worker\", {})\n\n @property\n def num_cpus(self) -> int:\n return self.resources_per_worker.get(\"CPU\", 1)\n\n @property\n def num_gpus(self) -> int:\n return self.resources_per_worker.get(\"GPU\", 0)\n\n def set_base_learning_rate(self, learning_rate: float):\n self.config.learning_rate = learning_rate\n\n def shutdown(self):\n pass\n\n\ndef eval_fn(\n predictor_kwargs: dict[str, Any] = None,\n model_ref: ObjectRef = None, # noqa: F821\n training_set_metadata: dict[str, Any] = None,\n features: dict[str, dict] = None,\n **kwargs,\n):\n \"\"\"Ray Train worker function for distributed evaluation.\n\n Runs inside each Ray worker process. Loads the model from an object ref, wraps the eval dataset shard, runs\n prediction and evaluation, and saves results to a Ray checkpoint for driver retrieval.\n \"\"\"\n # Pin GPU before loading the model to prevent memory leaking onto other devices\n initialize_pytorch()\n\n # Initialize a local distributed strategy so metric modules can sync.\n from ludwig.distributed import init_dist_strategy\n\n init_dist_strategy(\"local\")\n\n try:\n eval_shard = RayDatasetShard(\n rt.get_dataset_shard(\"eval\"),\n features,\n training_set_metadata,\n )\n\n model = ray.get(model_ref)\n # Use Ray Train's device assignment which respects use_gpu setting\n from ray.train.torch import get_device as ray_get_device\n\n device = ray_get_device()\n model = model.to(device)\n\n predictor_cls = get_predictor_cls(model.type())\n predictor = predictor_cls(dist_model=model, model=model, report_tqdm_to_ray=True, **predictor_kwargs)\n eval_results = predictor.batch_evaluation(eval_shard, **kwargs)\n\n # Save results to a checkpoint so the driver can retrieve them.\n # In Ray Train 2.x, result.metrics is only populated when a checkpoint is provided.\n eval_results = _make_picklable(eval_results)\n with tempfile.TemporaryDirectory() as tmpdir:\n torch.save(eval_results, os.path.join(tmpdir, \"eval_results.pt\"))\n rt.report(metrics={}, checkpoint=Checkpoint.from_directory(tmpdir))\n finally:\n torch.cuda.empty_cache()\n\n\nclass RayPredictor(BasePredictor):\n def __init__(\n self, model: BaseModel, df_engine: DataFrameEngine, trainer_kwargs, data_loader_kwargs, **predictor_kwargs\n ):\n self.batch_size = predictor_kwargs[\"batch_size\"]\n self.trainer_kwargs = trainer_kwargs\n self.data_loader_kwargs = data_loader_kwargs\n self.predictor_kwargs = predictor_kwargs\n self.actor_handles = []\n self.model = model.cpu()\n self.df_engine = df_engine\n\n def get_trainer_kwargs(self) -> dict[str, Any]:\n return get_trainer_kwargs(**self.trainer_kwargs)\n\n def get_resources_per_worker(self) -> tuple[int, int]:\n trainer_kwargs = self.get_trainer_kwargs()\n resources_per_worker = trainer_kwargs.get(\"resources_per_worker\", {})\n num_gpus = resources_per_worker.get(\"GPU\", 0)\n num_cpus = resources_per_worker.get(\"CPU\", (1 if num_gpus == 0 else 0))\n return num_cpus, num_gpus\n\n def batch_predict(self, dataset: RayDataset, *args, collect_logits: bool = False, **kwargs):\n self._check_dataset(dataset)\n\n predictor_kwargs = self.predictor_kwargs\n output_columns = get_output_columns(self.model.output_features, include_logits=collect_logits)\n batch_predictor = self.get_batch_infer_model(\n self.model,\n predictor_kwargs,\n output_columns,\n dataset.features,\n dataset.training_set_metadata,\n *args,\n collect_logits=collect_logits,\n **kwargs,\n )\n\n columns = [f.proc_column for f in self.model.input_features.values()]\n\n def to_tensors(df: pd.DataFrame) -> pd.DataFrame:\n for c in columns:\n df[c] = cast_as_tensor_dtype(df[c])\n return df\n\n num_cpus, num_gpus = self.get_resources_per_worker()\n\n predictions = dataset.ds.map_batches(to_tensors, batch_format=\"pandas\").map_batches(\n batch_predictor,\n batch_size=self.batch_size,\n compute=ray.data.ActorPoolStrategy(),\n batch_format=\"pandas\",\n num_cpus=num_cpus,\n num_gpus=num_gpus,\n )\n\n predictions = self.df_engine.from_ray_dataset(predictions)\n\n return predictions\n\n def predict_single(self, batch):\n raise NotImplementedError(\"predict_single can only be called on a local predictor\")\n\n def batch_evaluation(\n self,\n dataset: RayDataset,\n collect_predictions: bool = False,\n collect_logits=False,\n **kwargs,\n ):\n # We need to be in a distributed context to collect the aggregated metrics, since it relies on collective\n # communication ops. However, distributed training is not suitable for transforming one big dataset to another.\n # For that we will use Ray Datasets. Therefore, we break this up into two separate steps, and two passes over\n # the dataset. In the future, we can explore ways to combine these into a single step to reduce IO.\n # Collect eval metrics by distributing work across nodes / gpus\n datasets = {\"eval\": dataset.to_ray_dataset(shuffle=False)}\n predictor_kwargs = {\n **self.predictor_kwargs,\n \"collect_predictions\": False,\n }\n eval_loop_config = {\n \"predictor_kwargs\": predictor_kwargs,\n \"model_ref\": ray.put(self.model),\n \"training_set_metadata\": dataset.training_set_metadata,\n \"features\": dataset.features,\n **kwargs,\n }\n\n def _eval_loop(config):\n eval_fn(**config)\n\n result = run_train_remote(\n _eval_loop,\n trainer_kwargs=self.trainer_kwargs,\n datasets=datasets,\n train_loop_config=eval_loop_config,\n )\n\n # Load eval results from the checkpoint saved by eval_fn\n with result.checkpoint.as_directory() as tmpdir:\n eval_stats, _ = torch.load(os.path.join(tmpdir, \"eval_results.pt\"), weights_only=False)\n\n predictions = None\n if collect_predictions:\n # Collect eval predictions by using Ray Datasets to transform partitions of the data in parallel\n predictions = self.batch_predict(dataset, collect_logits=collect_logits)\n\n return eval_stats, predictions\n\n def batch_collect_activations(self, model, *args, **kwargs):\n raise NotImplementedError(\"Ray backend does not support collecting activations at this time.\")\n\n def _check_dataset(self, dataset):\n if not isinstance(dataset, RayDataset):\n raise RuntimeError(f\"Ray backend requires RayDataset for inference, \" f\"found: {type(dataset)}\")\n\n def shutdown(self):\n for handle in self.actor_handles:\n ray.kill(handle)\n self.actor_handles.clear()\n\n def get_batch_infer_model(\n self,\n model: \"LudwigModel\", # noqa: F821\n predictor_kwargs: dict[str, Any],\n output_columns: list[str],\n features: dict[str, dict],\n training_set_metadata: dict[str, Any],\n *args,\n **kwargs,\n ):\n model_ref = ray.put(model)\n _, num_gpus = self.get_resources_per_worker()\n\n class BatchInferModel:\n def __init__(self):\n model = ray.get(model_ref)\n # Respect the GPU setting from resources_per_worker.\n # When num_gpus=0, force CPU even if CUDA is available on the machine,\n # to avoid device mismatches between model outputs and targets.\n if num_gpus > 0:\n device = get_torch_device()\n else:\n device = \"cpu\"\n self.model = model.to(device)\n\n self.output_columns = output_columns\n self.features = features\n self.training_set_metadata = training_set_metadata\n self.reshape_map = {\n f[PROC_COLUMN]: training_set_metadata[f[NAME]].get(\"reshape\") for f in features.values()\n }\n predictor_cls = get_predictor_cls(self.model.type())\n predictor = predictor_cls(dist_model=self.model, model=self.model, **predictor_kwargs)\n self.predict = partial(predictor.predict_single, *args, **kwargs)\n\n def __call__(self, df: pd.DataFrame) -> pd.DataFrame:\n dataset = self._prepare_batch(df)\n predictions = self.predict(batch=dataset).set_index(df.index)\n ordered_predictions = predictions[self.output_columns]\n return ordered_predictions\n\n def _prepare_batch(self, batch: pd.DataFrame) -> dict[str, np.ndarray]:\n res = {}\n for c in self.features.keys():\n if self.features[c][TYPE] not in _SCALAR_TYPES:\n # Ensure columns stacked instead of turned into np.array([np.array, ...], dtype=object) objects\n res[c] = np.stack(batch[c].values)\n else:\n res[c] = batch[c].to_numpy()\n\n for c in self.features.keys():\n reshape = self.reshape_map.get(c)\n if reshape is not None:\n res[c] = res[c].reshape((-1, *reshape))\n\n return res\n\n return BatchInferModel\n\n\nclass RayBackend(RemoteTrainingMixin, Backend):\n BACKEND_TYPE = \"ray\"\n\n def __init__(self, processor=None, trainer=None, loader=None, preprocessor_kwargs=None, **kwargs):\n super().__init__(dataset_manager=RayDatasetManager(self), **kwargs)\n self._preprocessor_kwargs = preprocessor_kwargs or {}\n self._df_engine = _get_df_engine(processor)\n self._distributed_kwargs = trainer or {}\n self._pytorch_kwargs = {}\n self._data_loader_kwargs = loader or {}\n self._preprocessor_pg = None\n\n def initialize(self):\n initialize_ray()\n\n dask.config.set(scheduler=ray_dask_get)\n # Disable placement groups on dask\n dask.config.set(annotations={\"ray_remote_args\": {\"placement_group\": None}})\n # Prevent Dask from converting object-dtype columns to PyArrow strings,\n # which corrupts binary data, numpy arrays, and complex Python objects.\n dask.config.set({\"dataframe.convert-string\": False})\n\n def generate_bundles(self, num_cpu):\n # Ray requires that each bundle be scheduleable on a single node.\n # So a bundle of 320 cpus would never get scheduled. For now a simple heuristic\n # to be used is to just request 1 cpu at a time.\n return [{\"CPU\": 1} for _ in range(int(num_cpu))]\n\n @contextlib.contextmanager\n def provision_preprocessing_workers(self):\n num_cpu = self._preprocessor_kwargs.get(\"num_cpu\")\n if not num_cpu:\n logger.info(\n \"Backend config has num_cpu not set.\" \" provision_preprocessing_workers() is a no-op in this case.\"\n )\n yield\n else:\n bundles = self.generate_bundles(num_cpu)\n logger.info(\"Requesting bundles of %s for preprocessing\", bundles)\n self._preprocessor_pg = placement_group(bundles)\n ready = self._preprocessor_pg.wait(FIFTEEN_MINS_IN_S)\n\n if not ready:\n remove_placement_group(self._preprocessor_pg)\n raise TimeoutError(\n \"Ray timed out in provisioning the placement group for preprocessing.\"\n f\" {num_cpu} CPUs were requested but were unable to be provisioned.\"\n )\n\n logger.info(\"%s CPUs were requested and successfully provisioned\", num_cpu)\n try:\n with dask.config.set(annotations={\"ray_remote_args\": {\"placement_group\": self._preprocessor_pg}}):\n yield\n finally:\n self._release_preprocessing_workers()\n\n def _release_preprocessing_workers(self):\n if self._preprocessor_pg is not None:\n remove_placement_group(self._preprocessor_pg)\n self._preprocessor_pg = None\n\n def initialize_pytorch(self, **kwargs):\n # Make sure we don't claim any GPU resources on the head node\n initialize_pytorch(gpus=-1)\n self._pytorch_kwargs = kwargs\n\n def create_trainer(self, model: BaseModel, **kwargs) -> \"BaseTrainer\": # noqa: F821\n executable_kwargs = {**kwargs, **self._pytorch_kwargs}\n if model.type() == MODEL_LLM:\n from ludwig.trainers.registry import get_llm_ray_trainers_registry\n\n trainer_config = kwargs.get(\"config\")\n trainer_type = trainer_config.type if trainer_config else None\n trainer_cls = get_from_registry(trainer_type, get_llm_ray_trainers_registry())\n else:\n trainer_cls = get_from_registry(model.type(), get_ray_trainers_registry())\n\n # Deep copy to workaround https://github.com/ray-project/ray/issues/24139\n all_kwargs = {\n \"model\": model,\n \"trainer_kwargs\": copy.deepcopy(self._distributed_kwargs),\n \"data_loader_kwargs\": self._data_loader_kwargs,\n \"executable_kwargs\": executable_kwargs,\n }\n all_kwargs.update(kwargs)\n return trainer_cls(**all_kwargs)\n\n def create_predictor(self, model: BaseModel, **kwargs):\n executable_kwargs = {**kwargs, **self._pytorch_kwargs}\n return RayPredictor(\n model,\n self.df_engine,\n copy.deepcopy(self._distributed_kwargs),\n self._data_loader_kwargs,\n **executable_kwargs,\n )\n\n def set_distributed_kwargs(self, **kwargs):\n self._distributed_kwargs = kwargs\n\n @property\n def df_engine(self):\n return self._df_engine\n\n @property\n def supports_multiprocessing(self):\n return False\n\n def check_lazy_load_supported(self, feature):\n if not feature[PREPROCESSING][\"in_memory\"]:\n raise ValueError(\n f\"RayBackend does not support lazy loading of data files at train time. \"\n f\"Set preprocessing config `in_memory: True` for feature {feature[NAME]}\"\n )\n\n def read_binary_files(self, column: Series, map_fn: Callable | None = None, file_size: int | None = None) -> Series:\n column = column.fillna(np.nan).replace([np.nan], [None]) # normalize NaNs to None\n\n # Assume that the list of filenames is small enough to fit in memory. Should be true unless there\n # are literally billions of filenames.\n # TODO(travis): determine if there is a performance penalty to passing in individual files instead of\n # a directory. If so, we can do some preprocessing to determine if it makes sense to read the full directory\n # then filter out files as a postprocessing step (depending on the ratio of included to excluded files in\n # the directory). Based on a preliminary look at how Ray handles directory expansion to files, it looks like\n # there should not be any difference between providing a directory versus a list of files.\n pd_column = self.df_engine.compute(column)\n fnames = pd_column.values.tolist()\n idxs = pd_column.index.tolist()\n\n # Sample a filename to extract the filesystem info\n sample_fname = fnames[0]\n if isinstance(sample_fname, str):\n fs, _ = get_fs_and_path(sample_fname)\n filesystem = PyFileSystem(FSSpecHandler(fs))\n\n paths_and_idxs = list(zip(fnames, idxs))\n ds = read_binary_files_with_index(paths_and_idxs, filesystem=filesystem)\n # Rename \"data\" column to \"value\" for downstream compatibility\n ds = ds.rename_columns({\"data\": \"value\"})\n else:\n # Assume the path has already been read in, so just convert directly to a dataset\n # Name the column \"value\" to match the behavior of the above\n column_df = column.to_frame(name=\"value\")\n column_df[\"idx\"] = column_df.index\n ds = self.df_engine.to_ray_dataset(column_df)\n\n # Collect the Ray Dataset to pandas to avoid Arrow's string coercion\n # for binary/object columns (to_dask() converts bytes to string[pyarrow],\n # corrupting binary data and complex Python objects).\n pdf = ds.to_pandas()\n\n if map_fn is not None:\n with use_credentials(conf):\n pdf[\"value\"] = pdf[\"value\"].map(map_fn)\n\n pdf = pdf.rename(columns={\"value\": column.name})\n if \"idx\" in pdf.columns:\n pdf = pdf.set_index(\"idx\", drop=True)\n pdf.index.name = column.index.name\n\n # Convert to Dask for downstream compatibility.\n # Note: dataframe.convert-string is disabled globally in RayBackend.initialize()\n # to prevent object-dtype columns from being coerced to PyArrow strings.\n df = self.df_engine.from_pandas(pdf)\n return df[column.name]\n\n @property\n def num_nodes(self) -> int:\n if not ray.is_initialized():\n return 1\n return len(ray.nodes())\n\n @property\n def num_training_workers(self) -> int:\n return self._distributed_kwargs.get(\"num_workers\", 1)\n\n def max_concurrent_trials(self, hyperopt_config) -> int | None:\n # Limit concurrency based on available resources to avoid deadlocks between\n # Ray Tune trials and the Ray Datasets used internally for distributed training.\n resources = self.get_available_resources()\n num_cpus_per_trial = self._distributed_kwargs.get(\"resources_per_worker\", {}).get(\"CPU\", 1)\n num_workers = self._distributed_kwargs.get(\"num_workers\", 1)\n cpus_per_trial = num_cpus_per_trial * num_workers\n if cpus_per_trial > 0 and resources.cpus > 0:\n return max(1, int(resources.cpus // cpus_per_trial))\n return None\n\n def tune_batch_size(self, evaluator_cls, dataset_len: int) -> int:\n evaluator = evaluator_cls()\n return evaluator.select_best_batch_size(dataset_len)\n\n def batch_transform(self, df, batch_size: int, transform_fn, name: str | None = None):\n name = name or \"Batch Transform\"\n import dask.dataframe as dd\n\n from ludwig.utils.dataframe_utils import from_batches, to_batches\n\n # Compute Dask DataFrame to pandas before batching, as Dask-expr\n # doesn't support row slicing via integer indexing (df[i:j]).\n npartitions = df.npartitions if hasattr(df, \"npartitions\") else 1\n df = self.df_engine.compute(df)\n batches = to_batches(df, batch_size)\n transform = transform_fn()\n out_batches = [transform(batch.reset_index(drop=True)) for batch in batches]\n out_df = from_batches(out_batches).reset_index(drop=True)\n # Convert back to Dask so downstream code (split, etc.) still works\n return dd.from_pandas(out_df, npartitions=max(1, npartitions))\n\n def get_available_resources(self) -> Resources:\n resources = ray.cluster_resources()\n return Resources(cpus=resources.get(\"CPU\", 0), gpus=resources.get(\"GPU\", 0))\n\n\ndef initialize_ray():\n if not ray.is_initialized():\n try:\n ray.init(\"auto\", ignore_reinit_error=True)\n except ConnectionError:\n init_ray_local()\n\n\ndef init_ray_local():\n logger.info(\"Initializing new Ray cluster...\")\n ray.init(ignore_reinit_error=True)\n"
},
{
"path": "ludwig/backend/utils/__init__.py",
"content": ""
},
{
"path": "ludwig/backend/utils/storage.py",
"content": "import contextlib\nfrom typing import Any, Optional, Union\n\nfrom ludwig.utils import data_utils\n\nCredInputs = Optional[Union[str, dict[str, Any]]]\n\n\nDEFAULTS = \"defaults\"\nARTIFACTS = \"artifacts\"\nDATASETS = \"datasets\"\nCACHE = \"cache\"\n\n\nclass Storage:\n def __init__(self, creds: dict[str, Any] | None):\n self._creds = creds\n\n @contextlib.contextmanager\n def use_credentials(self):\n with data_utils.use_credentials(self._creds):\n yield\n\n @property\n def credentials(self) -> dict[str, Any] | None:\n return self._creds\n\n\nclass StorageManager:\n def __init__(\n self,\n defaults: CredInputs = None,\n artifacts: CredInputs = None,\n datasets: CredInputs = None,\n cache: CredInputs = None,\n ):\n defaults = load_creds(defaults)\n cred_inputs = {\n DEFAULTS: defaults,\n ARTIFACTS: load_creds(artifacts),\n DATASETS: load_creds(datasets),\n CACHE: load_creds(cache),\n }\n\n self.storages = {k: Storage(v if v is not None else defaults) for k, v in cred_inputs.items()}\n\n @property\n def defaults(self) -> Storage:\n return self.storages[DEFAULTS]\n\n @property\n def artifacts(self) -> Storage:\n \"\"\"TODO(travis): Currently used for hyperopt, but should be used for all outputs.\"\"\"\n return self.storages[ARTIFACTS]\n\n @property\n def datasets(self) -> Storage:\n \"\"\"TODO(travis): Should be used to read in datasets.\"\"\"\n return self.storages[DATASETS]\n\n @property\n def cache(self) -> Storage:\n return self.storages[CACHE]\n\n\ndef load_creds(cred: CredInputs) -> dict[str, Any]:\n if isinstance(cred, str):\n cred = data_utils.load_json(cred)\n return cred\n"
},
{
"path": "ludwig/benchmarking/README.md",
"content": "# Ludwig Benchmarking\n\n### Some use cases\n\n- Regression testing for ML experiments across releases and PRs.\n- Model performance testing for experimenting with new features and hyperparameters.\n- Resource usage tracking for the full ML pipeline.\n\n## Ludwig benchmarking CLI and API\n\nTo run benchmarks, run the following command from the command line\n\n```\nludwig benchmark --benchmarking_config path/to/benchmarking/config.yaml\n```\n\nTo use the API\n\n```\nfrom ludwig.benchmarking.benchmark import benchmark\n\nbenchmarking_config_path = \"path/to/benchmarking/config.yaml\"\nbenchmark(benchmarking_config_path)\n```\n\nIn what follows, we describe what the benchmarking config looks for\nmultiple use cases.\n\n## The benchmarking config\n\nThe benchmarking config is where you can specify\n\n1. The datasets you want to run the benchmarks on and their configs.\n1. Whether these experiments are hyperopt or regular train and eval experiments.\n1. The name of the experiment.\n1. A python script to edit the specified Ludwig configs programmatically/on the fly.\n1. The export path of these experiment's artifacts. (remotely or locally)\n1. Whether to use `LudwigProfiler` to track resource\n usage for preprocessing, training, and evaluation of the experiment.\n\nYou can find an example of a benchmarking config in the `examples/` directory.\n\n## Basic Usage\n\nSay you implemented a new feature and would like to test it on several datasets.\nIn this case, this is what the benchmarking config could look like\n\n```\nexperiment_name: SMOTE_test\nhyperopt: false\nexport:\n export_artifacts: true\n export_base_path: s3://benchmarking.us-west-2.ludwig.com/bench/ # include the slash at the end.\nexperiments:\n - dataset_name: ames_housing\n config_path: /home/ray/configs/ames_housing_SMOTE.yaml\n experiment_name: SMOTE_test_with_hyperopt\n hyperopt: true\n - dataset_name: protein\n - ...\n ...\n - dataset_name: mercedes_benz_greener\n config_path: /home/ray/configs/mercedes_benz_greener_SMOTE.yaml\n```\n\nFor each experiment:\n\n- `dataset_name`: name of the dataset in `ludwig.datasets` to run the benchmark on.\n- `config_path` (optional): path to Ludwig config. If not specified, this will load\n the config corresponding to the dataset only containing `input_features` and\n `output_features`.\n\nThis will run `LudwigModel.experiment` on the datasets with their specified configs.\nIf these configs contain a hyperopt section and you'd like to run hyperopt, change\nto `hyperopt: true`.\nYou can specify the same dataset multiple times with different configs.\n\n**Exporting artifacts**\nBy specifying `export_artifacts: true`, this will export the experiment artifacts\nto the `export_base_path`. Once the model is trained and the artifacts are pushed\nto the specified path, you will get a similar message to the following:\n\n```\nUploaded metrics report and experiment config to\n\t s3://benchmarking.us-west-2.ludwig.com/bench/ames_housing/SMOTE_test\n```\n\nThis is the directory structure of the exported artifacts for one of the experiments.\n\n```\ns3://benchmarking.us-west-2.ludwig.com/bench/\n└── ames_housing\n └── SMOTE_test\n ├── config.yaml\n └── experiment_run\n ├── description.json\n ├── model\n │ ├── logs\n │ │ ├── test\n │ │ │ └── events.out.tfevents.1663320893.macbook-pro.lan.8043.2\n │ │ ├── training\n │ │ │ └── events.out.tfevents.1663320893.macbook-pro.lan.8043.0\n │ │ └── validation\n │ │ └── events.out.tfevents.1663320893.macbook-pro.lan.8043.1\n │ ├── model_hyperparameters.json\n │ ├── training_progress.json\n │ └── training_set_metadata.json\n ├── test_statistics.json\n └── training_statistics.json\n```\n\nNote that model checkpoints are not exported. Any other experiments on\nthe `ames_housing` dataset will also live under\n`s3://benchmarking.us-west-2.ludwig.com/bench/ames_housing/`\n\n**Overriding parameters**\nThe benchmarking config's global parameters `experiment_name` and `hyperopt` can be overridden\nif specified within an experiment.\n\n## Programmatically editing Ludwig configs\n\nTo apply some changes to multiple Ludwig configs, you can specify a path to a python script\nthat does this without the need to do manual modifications across many configs. Example:\n\n```\nexperiment_name: logistic_regression_hyperopt\nhyperopt: true\nprocess_config_file_path: /home/ray/process_config.py\nexport:\n export_artifacts: true\n export_base_path: s3://benchmarking.us-west-2.ludwig.com/bench/ # include the slash at the end.\nexperiments:\n - dataset_name: ames_housing\n config_path: /home/ray/configs/ames_housing_SMOTE.yaml\n ...\n```\n\nIn `/home/ray/process_config.py`, define the following function and add custom code to modify\nludwig configs\n\n```\ndef process_config(ludwig_config: dict, experiment_dict: dict) -> dict:\n \"\"\"Modify a Ludwig config.\n\n :param ludwig_config: a Ludwig config.\n :param experiment_dict: a benchmarking config experiment dictionary.\n\n returns: a modified Ludwig config.\n \"\"\"\n\n # code to modify the Ludwig config.\n\n return ludwig_config\n```\n\nView the `examples/` folder for an example `process_config.py`.\n\n## Benchmarking the resource usage with `LudwigProfiler`\n\nTo benchmark the resource usage of the preprocessing, training, and evaluation\nsteps of `LudwigModel.experiment`, you can specify in the benchmarking config\nglobal parameters\n\n```\nprofiler:\n enable: true\n use_torch_profiler: false\n logging_interval: 0.1\n```\n\n- `enable: true` will run benchmarking with `LudwigProfiler`.\n- `use_torch_profiler: false` will skip using the torch profiler.\n- `logging_interval: 0.1` will instruct `LudwigProfiler` to collect\n resource usage information every 0.1 seconds.\n\nNote that profiling is only enabled in the case where `hyperopt: false`.\n`LudwigProfiler` is passed in to `LudwigModel` callbacks. The specific\ncallbacks that will be called are:\n\n- `on_preprocess_(start/end)`\n- `on_train_(start/end)`\n- `on_evaluation_(start/end)`\n\nThis is an example directory output when using the profiler:\n\n```\nfull_bench_with_profiler_with_torch\n├── config.yaml\n├── experiment_run\n├── system_resource_usage\n│ ├── evaluation\n│ │ └── run_0.json\n│ ├── preprocessing\n│ │ └── run_0.json\n│ └── training\n│ └── run_0.json\n└── torch_ops_resource_usage\n ├── evaluation\n │ └── run_0.json\n ├── preprocessing\n │ └── run_0.json\n └── training\n └── run_0.json\n```\n\nThe only difference is the `system_resource_usage` and `torch_ops_resource_usage`.\nThe difference between these two outputs can be found in the `LudwigProfiler` README.\n\n## Parameters and defaults\n\nEach of these parameters can also be specified in the experiments section to override the global value.\nIf not specified, the value of the global parameter will be propagated to the experiments.\n\n- `experiment_name` (required): name of the benchmarking run.\n- `export` (required): dictionary specifying whether to export the experiment artifacts and the export path.\n- `hyperopt` (optional): whether this is a hyperopt run or `LudwigModel.experiment`.\n- `process_config_file_path` (optional): path to python script that will modify configs.\n- `profiler` (optional): dictionary specifying whether to use the profiler and its parameters.\n\n## Comparing experiments\n\nYou can summarize the exported artifacts of two experiments on multiple datasets.\nFor example, if you ran two experiments on the datasets `ames_housing` called\n`small_batch_size` and `big_batch_size` where you varied the batch size,\nyou can create a diff summary of the model performance and resource usage of the two\nexperiments. This is how:\n\n```\nfrom ludwig.benchmarking.summarize import summarize_metrics\n\ndataset_list, metric_diffs, resource_usage_diffs = summarize_metrics(\n bench_config_path = \"path/to/benchmarking_config.yaml\",\n base_experiment = \"small_batch_size\",\n experimental_experiment = \"big_batch_size\",\n download_base_path = \"s3://benchmarking.us-west-2.ludwig.com/bench/\")\n```\n\nThis will print\n\n```\nModel performance metrics for *small_batch_size* vs. *big_batch_size* on dataset *ames_housing*\nOutput Feature Name Metric Name small_batch_size big_batch_size Diff Diff Percentage\nSalePrice mean_absolute_error 180551.609 180425.109 -126.5 -0.07\nSalePrice mean_squared_error 38668763136.0 38618021888.0 -50741248.0 -0.131\nSalePrice r2 -5.399 -5.391 0.008 -0.156\nSalePrice root_mean_squared_error 196643.75 196514.688 -129.062 -0.066\nSalePrice root_mean_squared_percentage_error 1.001 1.001 -0.001 -0.07\nExported a CSV report to summarize_output/performance_metrics/ames_housing/small_batch_size-big_batch_size.csv\n\nResource usage for *small_batch_size* vs. *big_batch_size* on *training* of dataset *ames_housing*\nMetric Name small_batch_size big_batch_size Diff Diff Percentage\naverage_cpu_memory_usage 106.96 Mb 109.43 Mb 2.48 Mb 2.315\naverage_cpu_utilization 1.2966666666666666 1.345 0.04833333333333334 3.728\naverage_global_cpu_memory_available 3.46 Gb 3.46 Gb -1.10 Mb -0.031\naverage_global_cpu_utilization 37.43333333333334 40.49 3.056666666666665 8.166\ndisk_footprint 372736 413696 40960 10.989\nmax_cpu_memory_usage 107.50 Mb 111.93 Mb 4.43 Mb 4.117\nmax_cpu_utilization 1.44 1.67 0.22999999999999998 15.972\nmax_global_cpu_utilization 54.1 60.9 6.799999999999997 12.569\nmin_global_cpu_memory_available 3.46 Gb 3.46 Gb -712.00 Kb -0.02\nnum_cpu 10 10 0 0.0\nnum_oom_events 0 0 0 inf\nnum_runs 1 1 0 0.0\ntorch_cpu_average_memory_used 81.44 Kb 381.15 Kb 299.70 Kb 367.992\ntorch_cpu_max_memory_used 334.26 Kb 2.65 Mb 2.32 Mb 711.877\ntorch_cpu_time 57.400ms 130.199ms 72.799ms 126.828\ntorch_cuda_time 0.000us 0.000us 0.000us inf\ntotal_cpu_memory_size 32.00 Gb 32.00 Gb 0 b 0.0\ntotal_execution_time 334.502ms 1.114s 779.024ms 232.891\nExported a CSV report to summarize_output/resource_usage_metrics/ames_housing/training-small_batch_size-big_batch_size.csv\n\nResource usage for *small_batch_size* vs. *big_batch_size* on *evaluation* of dataset *ames_housing*\n...\nResource usage for *small_batch_size* vs. *big_batch_size* on *preprocessing* of dataset *ames_housing*\n...\n```\n"
},
{
"path": "ludwig/benchmarking/__init__.py",
"content": ""
},
{
"path": "ludwig/benchmarking/artifacts.py",
"content": "import os\nfrom dataclasses import dataclass\nfrom typing import Any\n\nfrom ludwig.globals import MODEL_FILE_NAME\nfrom ludwig.types import ModelConfigDict, TrainingSetMetadataDict\nfrom ludwig.utils.data_utils import load_json, load_yaml\n\n\n@dataclass\nclass BenchmarkingResult:\n # The Ludwig benchmarking config.\n benchmarking_config: dict[str, Any]\n\n # The config for one experiment.\n experiment_config: dict[str, Any]\n\n # The Ludwig config used to run the experiment.\n ludwig_config: ModelConfigDict\n\n # The python script that is used to process the config before being used.\n process_config_file: str\n\n # Loaded `description.json` file.\n description: dict[str, Any]\n\n # Loaded `test_statistics.json` file.\n test_statistics: dict[str, Any]\n\n # Loaded `training_statistics.json` file.\n training_statistics: dict[str, Any]\n\n # Loaded `model_hyperparameters.json` file.\n model_hyperparameters: dict[str, Any]\n\n # Loaded `training_progress.json` file.\n training_progress: dict[str, Any]\n\n # Loaded `training_set_metadata.json` file.\n training_set_metadata: TrainingSetMetadataDict\n\n\ndef build_benchmarking_result(benchmarking_config: dict, experiment_idx: int):\n experiment_config = benchmarking_config[\"experiments\"][experiment_idx]\n process_config_file = \"\"\n if experiment_config[\"process_config_file_path\"]:\n with open(experiment_config[\"process_config_file_path\"]) as f:\n process_config_file = \"\".join(f.readlines())\n experiment_run_path = os.path.join(experiment_config[\"experiment_name\"], \"experiment_run\")\n\n return BenchmarkingResult(\n benchmarking_config=benchmarking_config,\n experiment_config=experiment_config,\n ludwig_config=load_yaml(experiment_config[\"config_path\"]),\n process_config_file=process_config_file,\n description=load_json(os.path.join(experiment_run_path, \"description.json\")),\n test_statistics=load_json(os.path.join(experiment_run_path, \"test_statistics.json\")),\n training_statistics=load_json(os.path.join(experiment_run_path, \"training_statistics.json\")),\n model_hyperparameters=load_json(\n os.path.join(experiment_run_path, MODEL_FILE_NAME, \"model_hyperparameters.json\")\n ),\n training_progress=load_json(os.path.join(experiment_run_path, MODEL_FILE_NAME, \"training_progress.json\")),\n training_set_metadata=load_json(\n os.path.join(experiment_run_path, MODEL_FILE_NAME, \"training_set_metadata.json\")\n ),\n )\n"
},
{
"path": "ludwig/benchmarking/benchmark.py",
"content": "import argparse\nimport importlib\nimport logging\nimport os\nimport shutil\nfrom typing import Any\n\nimport ludwig.datasets\nfrom ludwig.api import LudwigModel\nfrom ludwig.benchmarking.artifacts import BenchmarkingResult, build_benchmarking_result\nfrom ludwig.benchmarking.profiler_callbacks import LudwigProfilerCallback\nfrom ludwig.benchmarking.utils import (\n create_default_config,\n delete_hyperopt_outputs,\n delete_model_checkpoints,\n export_artifacts,\n load_from_module,\n populate_benchmarking_config_with_defaults,\n propagate_global_parameters,\n save_yaml,\n validate_benchmarking_config,\n)\nfrom ludwig.contrib import add_contrib_callback_args\nfrom ludwig.hyperopt.run import hyperopt\nfrom ludwig.utils.data_utils import load_yaml\n\nlogger = logging.getLogger()\n\n\ndef setup_experiment(experiment: dict[str, str]) -> dict[Any, Any]:\n \"\"\"Set up the backend and load the Ludwig config.\n\n Args:\n experiment: dictionary containing the dataset name, config path, and experiment name.\n Returns a Ludwig config.\n \"\"\"\n shutil.rmtree(os.path.join(experiment[\"experiment_name\"]), ignore_errors=True)\n if \"config_path\" not in experiment:\n experiment[\"config_path\"] = create_default_config(experiment)\n model_config = load_yaml(experiment[\"config_path\"])\n\n if experiment[\"process_config_file_path\"]:\n process_config_spec = importlib.util.spec_from_file_location(\n \"process_config_file_path.py\", experiment[\"process_config_file_path\"]\n )\n process_module = importlib.util.module_from_spec(process_config_spec)\n process_config_spec.loader.exec_module(process_module)\n model_config = process_module.process_config(model_config, experiment)\n experiment[\"config_path\"] = experiment[\"config_path\"].replace(\n \".yaml\", \"-\" + experiment[\"experiment_name\"] + \"-modified.yaml\"\n )\n save_yaml(experiment[\"config_path\"], model_config)\n\n return model_config\n\n\ndef benchmark_one(experiment: dict[str, str | dict[str, str]]) -> None:\n \"\"\"Run a Ludwig exepriment and track metrics given a dataset name.\n\n Args:\n experiment: dictionary containing the dataset name, config path, and experiment name.\n \"\"\"\n logger.info(f\"\\nRunning experiment *{experiment['experiment_name']}* on dataset *{experiment['dataset_name']}*\")\n\n # configuring backend and paths\n model_config = setup_experiment(experiment)\n\n # loading dataset\n # dataset_module = importlib.import_module(f\"ludwig.datasets.{experiment['dataset_name']}\")\n dataset_module = ludwig.datasets.get_dataset(experiment[\"dataset_name\"])\n dataset = load_from_module(dataset_module, model_config[\"output_features\"][0])\n\n if experiment[\"hyperopt\"]:\n # run hyperopt\n hyperopt(\n config=model_config,\n dataset=dataset,\n output_directory=experiment[\"experiment_name\"],\n skip_save_model=True,\n skip_save_training_statistics=True,\n skip_save_progress=True,\n skip_save_log=True,\n skip_save_processed_input=True,\n skip_save_unprocessed_output=True,\n skip_save_predictions=True,\n skip_save_training_description=True,\n hyperopt_log_verbosity=0,\n )\n delete_hyperopt_outputs(experiment[\"experiment_name\"])\n else:\n backend = None\n ludwig_profiler_callbacks = None\n if experiment[\"profiler\"][\"enable\"]:\n ludwig_profiler_callbacks = [LudwigProfilerCallback(experiment)]\n # Currently, only local backend is supported with LudwigProfiler.\n backend = \"local\"\n logger.info(\"Currently, only local backend is supported with LudwigProfiler.\")\n # run model and capture metrics\n model = LudwigModel(\n config=model_config, callbacks=ludwig_profiler_callbacks, logging_level=logging.ERROR, backend=backend\n )\n model.experiment(\n dataset=dataset,\n output_directory=experiment[\"experiment_name\"],\n skip_save_processed_input=True,\n skip_save_unprocessed_output=True,\n skip_save_predictions=True,\n skip_collect_predictions=True,\n )\n delete_model_checkpoints(experiment[\"experiment_name\"])\n\n\ndef benchmark(benchmarking_config: dict[str, Any] | str) -> dict[str, tuple[BenchmarkingResult, Exception]]:\n \"\"\"Launch benchmarking suite from a benchmarking config.\n\n Args:\n benchmarking_config: config or config path for the benchmarking tool. Specifies datasets and their\n corresponding Ludwig configs, as well as export options.\n \"\"\"\n if isinstance(benchmarking_config, str):\n benchmarking_config = load_yaml(benchmarking_config)\n validate_benchmarking_config(benchmarking_config)\n benchmarking_config = populate_benchmarking_config_with_defaults(benchmarking_config)\n benchmarking_config = propagate_global_parameters(benchmarking_config)\n\n experiment_artifacts = {}\n for experiment_idx, experiment in enumerate(benchmarking_config[\"experiments\"]):\n dataset_name = experiment[\"dataset_name\"]\n try:\n benchmark_one(experiment)\n experiment_artifacts[dataset_name] = (build_benchmarking_result(benchmarking_config, experiment_idx), None)\n except Exception as e:\n logger.exception(\n f\"Experiment *{experiment['experiment_name']}* on dataset *{experiment['dataset_name']}* failed\"\n )\n experiment_artifacts[dataset_name] = (None, e)\n finally:\n if benchmarking_config[\"export\"][\"export_artifacts\"]:\n export_base_path = benchmarking_config[\"export\"][\"export_base_path\"]\n export_artifacts(experiment, experiment[\"experiment_name\"], export_base_path)\n return experiment_artifacts\n\n\ndef cli(sys_argv):\n parser = argparse.ArgumentParser(\n description=\"This script runs a ludwig experiment on datasets specified in the benchmark config and exports \"\n \"the experiment artifact for each of the datasets following the export parameters specified in\"\n \"the benchmarking config.\",\n prog=\"ludwig benchmark\",\n usage=\"%(prog)s [options]\",\n )\n parser.add_argument(\"--benchmarking_config\", type=str, help=\"The benchmarking config.\")\n add_contrib_callback_args(parser)\n args = parser.parse_args(sys_argv)\n benchmark(args.benchmarking_config)\n"
},
{
"path": "ludwig/benchmarking/examples/benchmarking_config.yaml",
"content": "experiment_name: example_benchmarking_run\nhyperopt: false\nprocess_config_file_path: /home/ray/process_config.py\nprofiler:\n enable: true\n use_torch_profiler: false\n logging_interval: 0.1\nexport:\n export_artifacts: true\n export_base_path: s3://benchmarking.us-west-2.ludwig.com/bench/ # include the slash at the end.\nexperiments:\n - dataset_name: ames_housing\n config_path: /home/ray/anaconda3/lib/python3.12/site-packages/ludwig/benchmarking/configs/ames_housing.yaml\n - dataset_name: protein\n config_path: /home/ray/anaconda3/lib/python3.12/site-packages/ludwig/benchmarking/configs/protein.yaml\n - dataset_name: mercedes_benz_greener\n config_path: /home/ray/anaconda3/lib/python3.12/site-packages/ludwig/benchmarking/configs/mercedes_benz_greener.yaml\n - dataset_name: santander_customer_satisfaction\n config_path: /home/ray/anaconda3/lib/python3.12/site-packages/ludwig/benchmarking/configs/santander_customer_satisfaction.yaml\n - dataset_name: connect4\n config_path: /home/ray/anaconda3/lib/python3.12/site-packages/ludwig/benchmarking/configs/connect4.yaml\n - dataset_name: otto_group_product\n config_path: /home/ray/anaconda3/lib/python3.12/site-packages/ludwig/benchmarking/configs/otto_group_product.yaml\n - dataset_name: bnp_claims_management\n config_path: /home/ray/anaconda3/lib/python3.12/site-packages/ludwig/benchmarking/configs/bnp_claims_management.yaml\n - dataset_name: santander_customer_transaction\n config_path: /home/ray/anaconda3/lib/python3.12/site-packages/ludwig/benchmarking/configs/santander_customer_transaction.yaml\n - dataset_name: allstate_claims_severity\n config_path: /home/ray/anaconda3/lib/python3.12/site-packages/ludwig/benchmarking/configs/allstate_claims_severity.yaml\n - dataset_name: naval\n config_path: /home/ray/anaconda3/lib/python3.12/site-packages/ludwig/benchmarking/configs/naval.yaml\n - dataset_name: sarcos\n config_path: /home/ray/anaconda3/lib/python3.12/site-packages/ludwig/benchmarking/configs/sarcos.yaml\n - dataset_name: walmart_recruiting\n config_path: /home/ray/anaconda3/lib/python3.12/site-packages/ludwig/benchmarking/configs/walmart_recruiting.yaml\n - dataset_name: numerai28pt6\n config_path: /home/ray/anaconda3/lib/python3.12/site-packages/ludwig/benchmarking/configs/numerai28pt6.yaml\n - dataset_name: adult_census_income\n config_path: /home/ray/anaconda3/lib/python3.12/site-packages/ludwig/benchmarking/configs/adult_census_income.yaml\n - dataset_name: amazon_employee_access_challenge\n config_path: /home/ray/anaconda3/lib/python3.12/site-packages/ludwig/benchmarking/configs/amazon_employee_access_challenge.yaml\n - dataset_name: forest_cover\n config_path: /home/ray/anaconda3/lib/python3.12/site-packages/ludwig/benchmarking/configs/forest_cover.yaml\n - dataset_name: mushroom_edibility\n config_path: /home/ray/anaconda3/lib/python3.12/site-packages/ludwig/benchmarking/configs/mushroom_edibility.yaml\n"
},
{
"path": "ludwig/benchmarking/examples/process_config.py",
"content": "\"\"\"This function will take in a Ludwig config, strip away all its parameters except input and output featuresand\nadd some other parameters to run logistic regression hyperopt.\"\"\"\n\n\ndef process_config(ludwig_config: dict, experiment_dict: dict) -> dict:\n \"\"\"Modify a Ludwig config by programmatically adding elements to the config dictionary.\n\n The purpose is to apply changes for all datasets that are the same or are based on the\n attributes of `experiment_dict` (e.g. dataset_name) removing the need to manually apply\n small changes to configs on many datasets.\n\n :param ludwig_config: a Ludwig config.\n :param experiment_dict: a benchmarking config experiment dictionary.\n\n Returns: a modified Ludwig config.\n \"\"\"\n\n # only keep input_features and output_features\n main_config_keys = list(ludwig_config.keys())\n for key in main_config_keys:\n if key not in [\"input_features\", \"output_features\"]:\n del ludwig_config[key]\n\n temp = {\n \"preprocessing\": {\"split\": {\"type\": \"fixed\"}},\n \"trainer\": {\"epochs\": 1024, \"early_stop\": 7, \"eval_batch_size\": 16384, \"evaluate_training_set\": False},\n \"hyperopt\": {\n \"goal\": \"maximize\",\n \"output_feature\": None,\n \"metric\": None,\n \"split\": \"validation\",\n \"parameters\": {\n \"defaults.number.preprocessing.normalization\": {\"space\": \"choice\", \"categories\": [\"zscore\", None]},\n \"defaults.number.preprocessing.missing_value_strategy\": {\n \"space\": \"choice\",\n \"categories\": [\"fill_with_const\", \"fill_with_mean\"],\n },\n \"combiner.type\": {\"space\": \"choice\", \"categories\": [\"tabnet\", \"concat\"]},\n \"trainer.learning_rate_scheduler.decay\": {\"space\": \"choice\", \"categories\": [True, False]},\n \"trainer.learning_rate\": {\"space\": \"loguniform\", \"lower\": 0.0001, \"upper\": 0.1},\n \"trainer.learning_rate_scheduler.decay_rate\": {\"space\": \"uniform\", \"lower\": 0.4, \"upper\": 0.96},\n \"trainer.batch_size\": {\"space\": \"randint\", \"lower\": 32, \"upper\": 2048},\n },\n \"search_alg\": {\"type\": \"variant_generator\"},\n \"executor\": {\"type\": \"ray\", \"num_samples\": 1000},\n \"scheduler\": {\"type\": \"bohb\", \"reduction_factor\": 2},\n },\n }\n\n # add config parameters from temp\n for key, value in temp.items():\n ludwig_config[key] = value\n\n dataset_name_to_metric = {\n \"ames_housing\": \"r2\",\n \"mercedes_benz_greener\": \"r2\",\n \"mushroom_edibility\": \"accuracy\",\n \"amazon_employee_access_challenge\": \"roc_auc\",\n \"naval\": \"r2\",\n \"sarcos\": \"r2\",\n \"protein\": \"r2\",\n \"adult_census_income\": \"accuracy\",\n \"otto_group_product\": \"accuracy\",\n \"santander_customer_satisfaction\": \"accuracy\",\n \"amazon_employee_access\": \"roc_auc\",\n \"numerai28pt6\": \"accuracy\",\n \"bnp_claims_management\": \"accuracy\",\n \"allstate_claims_severity\": \"r2\",\n \"santander_customer_transaction\": \"accuracy\",\n \"connect4\": \"accuracy\",\n \"forest_cover\": \"accuracy\",\n \"ieee_fraud\": \"accuracy\",\n \"porto_seguro_safe_driver\": \"accuracy\",\n \"walmart_recruiting\": \"accuracy\",\n \"poker_hand\": \"accuracy\",\n \"higgs\": \"accuracy\",\n }\n\n # add hyperopt output feature and metric.\n dataset_name = experiment_dict[\"dataset_name\"]\n ludwig_config[\"hyperopt\"][\"metric\"] = dataset_name_to_metric[dataset_name]\n ludwig_config[\"hyperopt\"][\"output_feature\"] = ludwig_config[\"output_features\"][0][\"name\"]\n\n # use sparse encoder for categorical features to mimic logistic regression.\n for i, feature in enumerate(ludwig_config[\"input_features\"]):\n if feature[\"type\"] == \"category\":\n ludwig_config[\"input_features\"][i][\"encoder\"] = \"sparse\"\n for i, feature in enumerate(ludwig_config[\"output_features\"]):\n if feature[\"type\"] == \"category\":\n ludwig_config[\"output_features\"][i][\"encoder\"] = \"sparse\"\n\n # make sure to return the ludwig_config\n return ludwig_config\n"
},
{
"path": "ludwig/benchmarking/profiler.py",
"content": "import contextlib\nimport glob\nimport logging\nimport os\nimport shutil\nimport threading\nimport time\nfrom queue import Empty as EmptyQueueException\nfrom queue import Queue\nfrom subprocess import PIPE, Popen\nfrom typing import Any\nfrom xml.etree.ElementTree import fromstring\n\nimport psutil\nimport torch\nfrom cpuinfo import get_cpu_info\nfrom gpustat.core import GPUStatCollection\n\nfrom ludwig.benchmarking.profiler_dataclasses import profiler_dataclass_to_flat_dict, TorchProfilerMetrics\nfrom ludwig.benchmarking.reporting import get_metrics_from_system_usage_profiler, get_metrics_from_torch_profiler\nfrom ludwig.constants import LUDWIG_TAG\nfrom ludwig.globals import LUDWIG_VERSION\nfrom ludwig.utils.data_utils import save_json\n\nSTOP_MESSAGE = \"stop\"\nlogger = logging.getLogger()\n\n\ndef get_gpu_info():\n \"\"\"Gathers general hardware information about an nvidia GPU.\n\n This function was copied from `experiment_impact_tracker` to get around a Pandas 2.0 breaking change impacting the\n package. https://github.com/Breakend/experiment-impact-\n tracker/blob/master/experiment_impact_tracker/gpu/nvidia.py#L48-L73\n \"\"\"\n p = Popen([\"nvidia-smi\", \"-q\", \"-x\"], stdout=PIPE)\n outs, errors = p.communicate()\n xml = fromstring(outs)\n data = []\n driver_version = xml.findall(\"driver_version\")[0].text\n cuda_version = xml.findall(\"cuda_version\")[0].text\n\n for gpu_id, gpu in enumerate(xml.getiterator(\"gpu\")):\n gpu_data = {}\n name = [x for x in gpu.getiterator(\"product_name\")][0].text\n memory_usage = gpu.findall(\"fb_memory_usage\")[0]\n total_memory = memory_usage.findall(\"total\")[0].text\n\n gpu_data[\"name\"] = name\n gpu_data[\"total_memory\"] = total_memory\n gpu_data[\"driver_version\"] = driver_version\n gpu_data[\"cuda_version\"] = cuda_version\n data.append(gpu_data)\n return data\n\n\ndef monitor(queue: Queue, info: dict[str, Any], logging_interval: int, cuda_is_available: bool) -> None:\n \"\"\"Monitors hardware resource use.\n\n Collects system specific metrics (CPU/CUDA, CPU/CUDA memory) at a `logging_interval` interval and pushes\n results back to the parent process.\n\n Args:\n queue: queue from which we can push and retrieve messages sent to the function targeted by the thread.\n info: dictionary containing system resource usage information about the running process.\n logging_interval: time interval at which we will poll the system for usage metrics.\n cuda_is_available: stores torch.cuda.is_available().\n \"\"\"\n info[\"global_cpu_memory_available\"] = [psutil.virtual_memory().available]\n info[\"global_cpu_utilization\"] = [psutil.cpu_percent()]\n # get the pid of the parent process.\n tracked_process = psutil.Process(os.getpid())\n\n # will return a meaningless 0 value on the first call because `interval` arg is set to None.\n tracked_process.cpu_percent(interval=logging_interval)\n with tracked_process.oneshot():\n info[\"cpu_utilization\"] = [tracked_process.cpu_percent() / info[\"num_cpu\"]]\n info[\"cpu_memory_usage\"] = [tracked_process.memory_full_info().uss]\n try:\n info[\"num_accessible_cpus\"] = len(tracked_process.cpu_affinity())\n except Exception:\n pass\n\n while True:\n try:\n message = queue.get(block=False)\n if isinstance(message, str):\n if message == STOP_MESSAGE:\n # synchronize CUDA to get accurate timing for jobs running on GPU.\n if cuda_is_available:\n torch.cuda.synchronize()\n queue.put(info)\n return\n else:\n queue.put(message)\n except EmptyQueueException:\n pass\n if cuda_is_available:\n gpu_infos = GPUStatCollection.new_query()\n for i, gpu_info in enumerate(gpu_infos):\n gpu_key = f\"cuda_{i}\"\n info[f\"{gpu_key}_memory_used\"].append(gpu_info.memory_used)\n with tracked_process.oneshot():\n info[\"cpu_utilization\"].append(tracked_process.cpu_percent() / info[\"num_cpu\"])\n info[\"cpu_memory_usage\"].append(tracked_process.memory_full_info().uss)\n info[\"global_cpu_memory_available\"].append(psutil.virtual_memory().available)\n info[\"global_cpu_utilization\"].append(psutil.cpu_percent())\n time.sleep(logging_interval)\n\n\nclass LudwigProfiler(contextlib.ContextDecorator):\n \"\"\"Track system resource (hardware and software) usage.\n\n Warning: If `use_torch_profiler=True` while profiling on CUDA, it's not possible to benchmark DataLoaders\n with `num_workers > 0` due to CUDA multiprocessing limitations. See warning under `profile` class\n definition: https://github.com/pytorch/pytorch/blob/master/torch/autograd/profiler.py\n\n Attributes:\n tag: a string tag describing the code block/function that we're tracking.\n (e.g trainer.train, preprocessing, etc.)\n output_dir: path where metrics are saved.\n logging_interval: time interval in seconds at which system is polled for resource usage.\n \"\"\"\n\n def __init__(self, tag: str, use_torch_profiler: bool, output_dir: str, logging_interval: float = 0.1) -> None:\n self.tag = tag\n self._tag = LUDWIG_TAG + self.tag\n self.use_torch_profiler = use_torch_profiler\n self.output_dir = output_dir\n self.logging_interval = logging_interval\n self.cuda_is_available = torch.cuda.is_available()\n self.launched = False\n if self.use_torch_profiler:\n self.profiler_activities = [torch.profiler.ProfilerActivity.CPU]\n if self.cuda_is_available:\n self.profiler_activities.append(torch.profiler.ProfilerActivity.CUDA)\n os.makedirs(os.path.join(self.output_dir), exist_ok=True)\n\n def _init_tracker_info(self):\n \"\"\"Initialize new self.info, self.torch_profiler, and self.torch_record_function instances.\n\n Important to call this in __enter__ if the user decides not to create a new class instance and therefore\n __init__ wouldn't be called.\n \"\"\"\n self.info = {\"code_block_tag\": self.tag}\n if self.use_torch_profiler:\n self.torch_profiler = torch.profiler.profile(activities=self.profiler_activities, profile_memory=True)\n self.torch_record_function = torch.profiler.record_function(self._tag)\n\n def _populate_static_information(self) -> None:\n \"\"\"Populate the report with static software and hardware information.\"\"\"\n self.info[\"ludwig_version\"] = LUDWIG_VERSION\n self.info[\"start_disk_usage\"] = shutil.disk_usage(os.path.expanduser(\"~\")).used\n\n # CPU information\n cpu_info = get_cpu_info()\n self.info[\"cpu_architecture\"] = cpu_info[\"arch\"]\n self.info[\"num_cpu\"] = psutil.cpu_count()\n self.info[\"cpu_name\"] = cpu_info.get(\"brand_raw\", \"unknown\")\n self.info[\"total_cpu_memory_size\"] = psutil.virtual_memory().total\n\n # GPU information\n if self.cuda_is_available:\n gpu_infos = get_gpu_info()\n gpu_usage = GPUStatCollection.new_query()\n for i, gpu_info in enumerate(gpu_infos):\n gpu_key = f\"cuda_{i}\"\n self.info[f\"{gpu_key}_memory_used\"] = [gpu_usage[i].memory_used]\n self.info[f\"{gpu_key}_name\"] = gpu_info[\"name\"]\n self.info[f\"{gpu_key}_total_memory\"] = gpu_info[\"total_memory\"]\n self.info[f\"{gpu_key}_driver_version\"] = gpu_info[\"driver_version\"]\n self.info[f\"{gpu_key}_cuda_version\"] = gpu_info[\"cuda_version\"]\n\n # recording in microseconds to be in line with torch profiler time recording.\n self.info[\"start_time\"] = time.perf_counter_ns() / 1000\n\n def __enter__(self):\n \"\"\"Populate static information and monitors resource usage.\"\"\"\n if self.launched:\n raise RuntimeError(\"LudwigProfiler already launched. You can't use the same instance.\")\n\n self._init_tracker_info()\n self._populate_static_information()\n\n if self.use_torch_profiler:\n # contextlib.ExitStack gracefully handles situations where __enter__ or __exit__ calls throw exceptions.\n with contextlib.ExitStack() as ctx_exit_stack:\n try:\n # Launch torch.profiler to track PyTorch operators.\n ctx_exit_stack.enter_context(self.torch_profiler)\n except RuntimeError:\n # PyTorch profiler is already enabled on this thread.\n # Using the running PyTorch profiler to track events.\n self.torch_profiler = None\n\n ctx_exit_stack.enter_context(self.torch_record_function)\n self._ctx_exit_stack = ctx_exit_stack.pop_all()\n try:\n # Starting thread to monitor system resource usage.\n self.queue = Queue()\n self.t = threading.Thread(\n target=monitor,\n args=(\n self.queue,\n self.info,\n self.logging_interval,\n self.cuda_is_available,\n ),\n )\n self.t.start()\n self.launched = True\n except Exception:\n self.launched = False\n logger.exception(\"Encountered exception when launching tracker thread.\")\n\n return self\n\n def __exit__(self, exc_type, exc_val, exc_tb) -> None:\n \"\"\"Stop profiling, postprocess and export resource usage metrics.\"\"\"\n try:\n self.queue.put(STOP_MESSAGE)\n self.t.join()\n result = self.queue.get()\n # If monitor thread crashed, result may be a string instead of dict\n if isinstance(result, dict):\n self.info = result\n # recording in microseconds to be in line with torch profiler time recording.\n self.info[\"end_time\"] = time.perf_counter_ns() / 1000\n self.info[\"end_disk_usage\"] = shutil.disk_usage(os.path.expanduser(\"~\")).used\n self.launched = False\n except Exception:\n logger.exception(\"Encountered exception when joining tracker thread.\")\n finally:\n if self.use_torch_profiler:\n self._ctx_exit_stack.close()\n self._export_torch_metrics()\n self._export_system_usage_metrics()\n\n def _export_system_usage_metrics(self):\n \"\"\"Export system resource usage metrics (no torch operators).\"\"\"\n system_usage_metrics = get_metrics_from_system_usage_profiler(self.info)\n output_subdir = os.path.join(self.output_dir, \"system_resource_usage\", system_usage_metrics.code_block_tag)\n os.makedirs(output_subdir, exist_ok=True)\n num_prev_runs = len(glob.glob(os.path.join(output_subdir, \"run_*.json\")))\n file_name = os.path.join(output_subdir, f\"run_{num_prev_runs}.json\")\n save_json(file_name, profiler_dataclass_to_flat_dict(system_usage_metrics))\n\n def _reformat_torch_usage_metrics_tags(\n self, torch_usage_metrics: dict[str, Any]\n ) -> dict[str, list[TorchProfilerMetrics]]:\n reformatted_dict = {}\n for key, value in torch_usage_metrics.items():\n assert key.startswith(LUDWIG_TAG)\n reformatted_key = key[len(LUDWIG_TAG) :]\n reformatted_dict[reformatted_key] = value\n return reformatted_dict\n\n def _export_torch_metrics(self):\n \"\"\"Export resource usage metrics of torch operators.\"\"\"\n if self.torch_profiler:\n torch_usage_metrics = get_metrics_from_torch_profiler(self.torch_profiler)\n torch_usage_metrics = self._reformat_torch_usage_metrics_tags(torch_usage_metrics)\n for tag, runs in torch_usage_metrics.items():\n temp_dir = os.path.join(self.output_dir, \"torch_ops_resource_usage\", tag)\n os.makedirs(temp_dir, exist_ok=True)\n for run in runs:\n num_prev_runs = len(glob.glob(os.path.join(temp_dir, \"run_*.json\")))\n save_json(os.path.join(temp_dir, f\"run_{num_prev_runs}.json\"), profiler_dataclass_to_flat_dict(run))\n"
},
{
"path": "ludwig/benchmarking/profiler_callbacks.py",
"content": "from typing import Any\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.benchmarking.profiler import LudwigProfiler\nfrom ludwig.callbacks import Callback\nfrom ludwig.constants import EVALUATION, PREPROCESSING, TRAINING\n\n\n# TODO: Change annotation to PublicAPI once Ludwig 0.7 is released\n@DeveloperAPI\nclass LudwigProfilerCallback(Callback):\n \"\"\"Class that defines the methods necessary to hook into process.\"\"\"\n\n def __init__(self, experiment: dict[str, Any]):\n self.experiment_name = experiment[\"experiment_name\"]\n self.use_torch_profiler = experiment[\"profiler\"][\"use_torch_profiler\"]\n self.logging_interval = experiment[\"profiler\"][\"logging_interval\"]\n self.preprocess_profiler = None\n self.train_profiler = None\n self.evaluation_profiler = None\n\n def on_preprocess_start(self, *args, **kwargs):\n self.preprocess_profiler = LudwigProfiler(\n tag=PREPROCESSING,\n output_dir=self.experiment_name,\n use_torch_profiler=self.use_torch_profiler,\n logging_interval=self.logging_interval,\n )\n self.preprocess_profiler.__enter__()\n\n def on_preprocess_end(self, *args, **kwargs):\n self.preprocess_profiler.__exit__(None, None, None)\n del self.preprocess_profiler\n\n def on_train_start(self, *args, **kwargs):\n self.train_profiler = LudwigProfiler(\n tag=TRAINING,\n output_dir=self.experiment_name,\n use_torch_profiler=self.use_torch_profiler,\n logging_interval=self.logging_interval,\n )\n self.train_profiler.__enter__()\n\n def on_train_end(self, *args, **kwargs):\n self.train_profiler.__exit__(None, None, None)\n del self.train_profiler\n\n def on_evaluation_start(self):\n self.evaluation_profiler = LudwigProfiler(\n tag=EVALUATION,\n output_dir=self.experiment_name,\n use_torch_profiler=self.use_torch_profiler,\n logging_interval=self.logging_interval,\n )\n self.evaluation_profiler.__enter__()\n\n def on_evaluation_end(self):\n self.evaluation_profiler.__exit__(None, None, None)\n del self.evaluation_profiler\n"
},
{
"path": "ludwig/benchmarking/profiler_dataclasses.py",
"content": "import dataclasses\nfrom dataclasses import dataclass\n\nfrom ludwig.utils.data_utils import flatten_dict\n\n\n@dataclass\nclass DeviceUsageMetrics:\n # Max CUDA memory utilization of the code block.\n max_memory_used: float\n\n # Average CUDA memory utilization of the code block.\n average_memory_used: float\n\n\n@dataclass\nclass SystemResourceMetrics:\n # Name of the code block/function to be profiled.\n code_block_tag: str\n\n # Name of the CPU that the code ran on.\n cpu_name: str\n\n # CPU architecture that the code ran on.\n cpu_architecture: str\n\n # Number of CPUs on the machine.\n num_cpu: int\n\n # Total CPU memory size.\n total_cpu_memory_size: float\n\n # Ludwig version in the environment.\n ludwig_version: str\n\n # Total execution time of the code block.\n total_execution_time: float\n\n # The change in disk memory before and after the code block ran.\n disk_footprint: float\n\n # Max CPU utilization of the code block.\n max_cpu_utilization: float\n\n # Max CPU memory (RAM) utilization of the code block.\n max_cpu_memory_usage: float\n\n # Min system-wide CPU memory available (how much physical memory is left).\n min_global_cpu_memory_available: float\n\n # Max system-wide CPU utilization.\n max_global_cpu_utilization: float\n\n # Average CPU utilization of the code block.\n average_cpu_utilization: float\n\n # Average CPU memory (RAM) utilization of the code block.\n average_cpu_memory_usage: float\n\n # Average system-wide CPU memory available (how much physical memory is left).\n average_global_cpu_memory_available: float\n\n # Average system-wide CPU utilization.\n average_global_cpu_utilization: float\n\n # Per device usage. Dictionary containing max and average memory used per device.\n device_usage: dict[str, DeviceUsageMetrics]\n\n\n@dataclass\nclass TorchProfilerMetrics:\n # Time taken by torch ops to execute on the CPU.\n torch_cpu_time: float\n\n # Time taken by torch ops to execute on CUDA devices.\n torch_cuda_time: float\n\n # Number of out of memory events.\n num_oom_events: int\n\n # Per device usage by torch ops. Dictionary containing max and average memory used per device.\n device_usage: dict[str, DeviceUsageMetrics]\n\n\ndef profiler_dataclass_to_flat_dict(data: SystemResourceMetrics | TorchProfilerMetrics) -> dict:\n \"\"\"Returns a flat dictionary representation, with the device_usage key removed.\"\"\"\n nested_dict = dataclasses.asdict(data)\n nested_dict[\"\"] = nested_dict.pop(\"device_usage\")\n return flatten_dict(nested_dict, sep=\"\")\n"
},
{
"path": "ludwig/benchmarking/reporting.py",
"content": "from collections import Counter, defaultdict\nfrom statistics import mean\nfrom typing import Any\n\nimport torch\nfrom torch._C._autograd import _KinetoEvent\nfrom torch.autograd import DeviceType, profiler_util\n\nfrom ludwig.benchmarking.profiler_dataclasses import DeviceUsageMetrics, SystemResourceMetrics, TorchProfilerMetrics\nfrom ludwig.constants import LUDWIG_TAG\n\n\ndef initialize_stats_dict(main_function_events: list[profiler_util.FunctionEvent]) -> dict[str, list]:\n \"\"\"Initialize dictionary which stores resource usage information per tagged code block.\n\n :param main_function_events: list of main function events.\n \"\"\"\n info = {}\n for event_name in [evt.name for evt in main_function_events]:\n info[event_name] = []\n return info\n\n\ndef get_memory_details(kineto_event: _KinetoEvent) -> tuple[str, int]:\n \"\"\"Get device name and number of bytes (de)allocated during an event.\n\n :param kineto_event: a Kineto event instance.\n \"\"\"\n if kineto_event.device_type() in [DeviceType.CPU, DeviceType.MKLDNN, DeviceType.IDEEP]:\n return \"cpu\", kineto_event.nbytes()\n elif kineto_event.device_type() in [DeviceType.CUDA, DeviceType.HIP]:\n return f\"cuda_{kineto_event.device_index()}\", kineto_event.nbytes()\n else:\n raise ValueError(f\"Device {kineto_event.device_type()} is not valid.\")\n\n\ndef get_device_memory_usage(\n kineto_event: _KinetoEvent, memory_events: list[list[_KinetoEvent | bool]]\n) -> dict[str, DeviceUsageMetrics]:\n \"\"\"Get CPU and CUDA memory usage for an event.\n\n :param kineto_event: a Kineto event instance.\n :param memory_events: list of memory events.\n \"\"\"\n mem_records_acc = profiler_util.MemRecordsAcc(memory_events)\n start_us = kineto_event.start_ns() / 1000\n end_us = start_us + kineto_event.duration_ns() / 1000\n records_in_interval = mem_records_acc.in_interval(start_us, end_us)\n memory_so_far = defaultdict(int)\n count_so_far = defaultdict(int)\n average_so_far = defaultdict(float)\n max_so_far = defaultdict(int)\n\n for mem_record in records_in_interval:\n device, nbytes = get_memory_details(mem_record[0])\n memory_so_far[device] += nbytes\n max_so_far[device] = max(max_so_far[device], memory_so_far[device])\n average_so_far[device] = (memory_so_far[device] + (average_so_far[device] * count_so_far[device])) / (\n count_so_far[device] + 1\n )\n count_so_far[device] += 1\n memory_info_per_device = {}\n for device in count_so_far:\n memory_info_per_device[f\"torch_{device}_\"] = DeviceUsageMetrics(\n max_memory_used=max_so_far[device], average_memory_used=average_so_far[device]\n )\n return memory_info_per_device\n\n\ndef get_torch_op_time(events: list[profiler_util.FunctionEvent], attr: str) -> int | float:\n \"\"\"Get time torch operators spent executing for a list of events.\n\n :param events: list of events.\n :param attr: a FunctionEvent attribute. Expecting one of \"cpu_time_total\", \"device_time_total\".\n \"\"\"\n if attr not in [\"cpu_time_total\", \"device_time_total\"]:\n return -1\n\n total = 0\n for e in events:\n # Possible trace_names are torch ops, or tagged code blocks by LudwigProfiler (which are\n # prepended with LUDWIG_TAG).\n if LUDWIG_TAG not in e.trace_name:\n total += getattr(e, attr)\n else:\n total += get_torch_op_time(e.cpu_children, attr)\n return total\n\n\ndef get_device_run_durations(function_event: profiler_util.FunctionEvent) -> tuple[float, float]:\n \"\"\"Get CPU and device run durations for an event.\n\n :param function_event: a function event instance.\n \"\"\"\n torch_cpu_time = get_torch_op_time(function_event.cpu_children, \"cpu_time_total\")\n torch_device_time = get_torch_op_time(function_event.cpu_children, \"device_time_total\")\n return torch_cpu_time, torch_device_time\n\n\ndef get_num_oom_events(kineto_event: _KinetoEvent, out_of_memory_events: list[list[_KinetoEvent | bool]]) -> int:\n oom_records_acc = profiler_util.MemRecordsAcc(out_of_memory_events)\n start_us = kineto_event.start_ns() / 1000\n end_us = start_us + kineto_event.duration_ns() / 1000\n records_in_interval = oom_records_acc.in_interval(start_us, end_us)\n return len(list(records_in_interval))\n\n\ndef get_resource_usage_report(\n main_kineto_events: list[_KinetoEvent],\n main_function_events: list[profiler_util.FunctionEvent],\n memory_events: list[list[_KinetoEvent | bool]],\n out_of_memory_events: list[list[_KinetoEvent | bool]],\n info: dict[str, Any],\n) -> dict[str, list[TorchProfilerMetrics]]:\n \"\"\"Get relevant information from Kineto events and function events exported by the profiler.\n\n :param main_kineto_events: list of main Kineto events.\n :param main_function_events: list of main function events.\n :param memory_events: list of memory events.\n :param out_of_memory_events: list of out of memory events.\n :param info: dictionary used to record resource usage metrics.\n \"\"\"\n main_kineto_events = sorted(\n (evt for evt in main_kineto_events if LUDWIG_TAG in evt.name()), key=lambda x: x.correlation_id()\n )\n main_function_events = sorted((evt for evt in main_function_events if LUDWIG_TAG in evt.name), key=lambda x: x.id)\n\n for kineto_event, function_event in zip(main_kineto_events, main_function_events):\n # Two different instances of `function_event` can have the same name if a the same\n # tagged code block/function was executed more than once.\n memory_info_per_device = get_device_memory_usage(kineto_event, memory_events)\n torch_cpu_time, torch_cuda_time = get_device_run_durations(function_event)\n num_oom_events = get_num_oom_events(kineto_event, out_of_memory_events)\n torch_profiler_metrics = TorchProfilerMetrics(\n torch_cpu_time=torch_cpu_time,\n torch_cuda_time=torch_cuda_time,\n num_oom_events=num_oom_events,\n device_usage=memory_info_per_device,\n )\n info[function_event.name].append(torch_profiler_metrics)\n return info\n\n\ndef get_all_events(kineto_events: list[_KinetoEvent], function_events: profiler_util.EventList) -> tuple[\n list[_KinetoEvent],\n list[profiler_util.FunctionEvent],\n list[list[_KinetoEvent | bool]],\n list[list[_KinetoEvent | bool]],\n]:\n \"\"\"Return main Kineto and function events, memory and OOM events for functions/code blocks tagged in\n LudwigProfiler.\n\n :param kineto_events: list of Kineto Events.\n :param function_events: list of function events.\n \"\"\"\n # LUDWIG_TAG is prepended to LudwigProfiler tags. This edited tag is passed in to `torch.profiler.record_function`\n # so we can easily retrieve events for code blocks wrapped with LudwigProfiler.\n main_function_events = [evt for evt in function_events if LUDWIG_TAG in evt.name]\n main_kineto_events = [event for event in kineto_events if LUDWIG_TAG in event.name()]\n memory_events = [[event, False] for event in kineto_events if profiler_util.MEMORY_EVENT_NAME in event.name()]\n # profiler_util.OUT_OF_MEMORY_EVENT_NAME seems to only be in newer versions of torch.\n out_of_memory_events = [[event, False] for event in kineto_events if \"[OutOfMemory]\" in event.name()]\n return main_kineto_events, main_function_events, memory_events, out_of_memory_events\n\n\ndef get_metrics_from_torch_profiler(profile: torch.profiler.profiler.profile) -> dict[str, list[TorchProfilerMetrics]]:\n \"\"\"Export time and resource usage metrics (CPU and CUDA) from a PyTorch profiler.\n\n The profiler keeps track of *torch operations* being executed in C++. It keeps track\n of what device they're executed on, their execution time, and memory usage.\n We only track the aforementioned metrics, but the torch profiler can keep track of\n the stack trace, FLOPs, and torch modules. Tracking each additional item adds overhead.\n\n The torch profiler surfaces these metrics that are tracked under the hood by `libkineto`.\n More on the Kineto project: https://github.com/pytorch/kineto\n\n :param profile: profiler object that contains all the events that\n were registered during the execution of the wrapped code block.\n \"\"\"\n # events in both of these lists are in chronological order.\n kineto_events = profile.profiler.kineto_results.events()\n function_events = profile.profiler.function_events\n main_kineto_events, main_function_events, memory_events, out_of_memory_events = get_all_events(\n kineto_events, function_events\n )\n\n assert Counter([event.name for event in main_function_events]) == Counter(\n [event.name() for event in main_kineto_events]\n )\n info = initialize_stats_dict(main_function_events)\n info = get_resource_usage_report(\n main_kineto_events, main_function_events, memory_events, out_of_memory_events, info\n )\n return info\n\n\ndef get_metrics_from_system_usage_profiler(system_usage_info: dict) -> SystemResourceMetrics:\n \"\"\"Package system resource usage metrics (no torch operators) in a dataclass.\n\n :param system_usage_info: dictionary containing resource usage information.\n \"\"\"\n device_usage_dict: dict[str, DeviceUsageMetrics] = {}\n for key in system_usage_info:\n if \"cuda_\" in key and \"_memory_used\" in key:\n cuda_device_name = \"_\".join(key.split(\"_\")[:2]) + \"_\"\n max_memory_used = max(system_usage_info[key], default=0)\n average_memory_used = mean(system_usage_info.get(key, [0]))\n device_usage_dict[cuda_device_name] = DeviceUsageMetrics(\n max_memory_used=max_memory_used, average_memory_used=average_memory_used\n )\n return SystemResourceMetrics(\n code_block_tag=system_usage_info[\"code_block_tag\"],\n cpu_name=system_usage_info.get(\"cpu_name\", \"unknown\"),\n cpu_architecture=system_usage_info[\"cpu_architecture\"],\n num_cpu=system_usage_info[\"num_cpu\"],\n total_cpu_memory_size=system_usage_info[\"total_cpu_memory_size\"],\n ludwig_version=system_usage_info[\"ludwig_version\"],\n total_execution_time=system_usage_info[\"end_time\"] - system_usage_info[\"start_time\"],\n disk_footprint=system_usage_info[\"end_disk_usage\"] - system_usage_info[\"start_disk_usage\"],\n max_cpu_utilization=max(system_usage_info[\"cpu_utilization\"], default=0),\n max_cpu_memory_usage=max(system_usage_info[\"cpu_memory_usage\"], default=0),\n min_global_cpu_memory_available=min(system_usage_info[\"global_cpu_memory_available\"], default=0),\n max_global_cpu_utilization=max(system_usage_info[\"global_cpu_utilization\"], default=0),\n average_cpu_utilization=mean(system_usage_info.get(\"cpu_utilization\", [0])),\n average_cpu_memory_usage=mean(system_usage_info.get(\"cpu_memory_usage\", [0])),\n average_global_cpu_memory_available=mean(system_usage_info.get(\"global_cpu_memory_available\", [0])),\n average_global_cpu_utilization=mean(system_usage_info.get(\"global_cpu_utilization\", [0])),\n device_usage=device_usage_dict,\n )\n"
},
{
"path": "ludwig/benchmarking/summarize.py",
"content": "import argparse\nimport logging\nimport os\nimport shutil\n\nfrom ludwig.benchmarking.summary_dataclasses import (\n build_metrics_diff,\n build_resource_usage_diff,\n export_metrics_diff_to_csv,\n export_resource_usage_diff_to_csv,\n MetricsDiff,\n ResourceUsageDiff,\n)\nfrom ludwig.benchmarking.utils import download_artifacts\n\nlogger = logging.getLogger()\n\n\ndef summarize_metrics(\n bench_config_path: str, base_experiment: str, experimental_experiment: str, download_base_path: str\n) -> tuple[list[str], list[MetricsDiff], list[list[ResourceUsageDiff]]]:\n \"\"\"Build metric and resource usage diffs from experiment artifacts.\n\n bench_config_path: bench config file path. Can be the same one that was used to run\n these experiments.\n base_experiment: name of the experiment we're comparing against.\n experimental_experiment: name of the experiment we're comparing.\n download_base_path: base path under which live the stored artifacts of\n the benchmarking experiments.\n \"\"\"\n local_dir, dataset_list = download_artifacts(\n bench_config_path, base_experiment, experimental_experiment, download_base_path\n )\n metric_diffs, resource_usage_diffs = [], []\n for dataset_name in dataset_list:\n try:\n metric_diff = build_metrics_diff(dataset_name, base_experiment, experimental_experiment, local_dir)\n metric_diffs.append(metric_diff)\n\n base_path = os.path.join(local_dir, dataset_name, base_experiment)\n experimental_path = os.path.join(local_dir, dataset_name, experimental_experiment)\n resource_usage_diff = build_resource_usage_diff(\n base_path, experimental_path, base_experiment, experimental_experiment\n )\n resource_usage_diffs.append(resource_usage_diff)\n except Exception:\n logger.exception(f\"Exception encountered while creating diff summary for {dataset_name}.\")\n shutil.rmtree(local_dir, ignore_errors=True)\n export_and_print(dataset_list, metric_diffs, resource_usage_diffs)\n return dataset_list, metric_diffs, resource_usage_diffs\n\n\ndef export_and_print(\n dataset_list: list[str], metric_diffs: list[MetricsDiff], resource_usage_diffs: list[list[ResourceUsageDiff]]\n) -> None:\n \"\"\"Export to CSV and print a diff of performance and resource usage metrics of two experiments.\n\n :param dataset_list: list of datasets for which to print the diffs.\n :param metric_diffs: Diffs for the performance metrics by dataset.\n :param resource_usage_diffs: Diffs for the resource usage metrics per dataset per LudwigProfiler tag.\n \"\"\"\n for dataset_name, experiment_metric_diff in zip(dataset_list, metric_diffs):\n output_path = os.path.join(\"summarize_output\", \"performance_metrics\", dataset_name)\n os.makedirs(output_path, exist_ok=True)\n\n logger.info(\n \"Model performance metrics for *{}* vs. *{}* on dataset *{}*\".format(\n experiment_metric_diff.base_experiment_name,\n experiment_metric_diff.experimental_experiment_name,\n experiment_metric_diff.dataset_name,\n )\n )\n logger.info(experiment_metric_diff.to_string())\n filename = (\n \"-\".join([experiment_metric_diff.base_experiment_name, experiment_metric_diff.experimental_experiment_name])\n + \".csv\"\n )\n export_metrics_diff_to_csv(experiment_metric_diff, os.path.join(output_path, filename))\n\n for dataset_name, experiment_resource_diff in zip(dataset_list, resource_usage_diffs):\n output_path = os.path.join(\"summarize_output\", \"resource_usage_metrics\", dataset_name)\n os.makedirs(output_path, exist_ok=True)\n for tag_diff in experiment_resource_diff:\n logger.info(\n \"Resource usage for *{}* vs. *{}* on *{}* of dataset *{}*\".format(\n tag_diff.base_experiment_name,\n tag_diff.experimental_experiment_name,\n tag_diff.code_block_tag,\n dataset_name,\n )\n )\n logger.info(tag_diff.to_string())\n filename = (\n \"-\".join(\n [tag_diff.code_block_tag, tag_diff.base_experiment_name, tag_diff.experimental_experiment_name]\n )\n + \".csv\"\n )\n export_resource_usage_diff_to_csv(tag_diff, os.path.join(output_path, filename))\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(\n description=\"Summarize the model performance metrics and resource usage metrics of two experiments.\",\n prog=\"python summarize.py\",\n usage=\"%(prog)s [options]\",\n )\n parser.add_argument(\"--benchmarking_config\", type=str, help=\"The benchmarking config.\")\n parser.add_argument(\"--base_experiment\", type=str, help=\"The name of the first experiment.\")\n parser.add_argument(\"--experimental_experiment\", type=str, help=\"The name of the second experiment.\")\n parser.add_argument(\"--download_base_path\", type=str, help=\"The base path to download experiment artifacts from.\")\n args = parser.parse_args()\n summarize_metrics(\n args.benchmarking_config, args.base_experiment, args.experimental_experiment, args.download_base_path\n )\n"
},
{
"path": "ludwig/benchmarking/summary_dataclasses.py",
"content": "import csv\nimport logging\nimport os\nfrom dataclasses import dataclass\nfrom statistics import mean\n\nimport ludwig.modules.metric_modules # noqa: F401\nfrom ludwig.benchmarking.utils import format_memory, format_time\nfrom ludwig.globals import MODEL_FILE_NAME, MODEL_HYPERPARAMETERS_FILE_NAME\nfrom ludwig.modules.metric_registry import get_metric_classes, metric_feature_type_registry # noqa: F401\nfrom ludwig.types import ModelConfigDict\nfrom ludwig.utils.data_utils import load_json\n\nlogger = logging.getLogger()\n\n\n@dataclass\nclass MetricDiff:\n \"\"\"Diffs for a metric.\"\"\"\n\n # Name of the metric.\n name: str\n\n # Value of the metric in base experiment (the one we benchmark against).\n base_value: float\n\n # Value of the metric in the experimental experiment.\n experimental_value: float\n\n # experimental_value - base_value.\n diff: float\n\n # Percentage of change the metric with respect to base_value.\n diff_percentage: float | str\n\n def __post_init__(self):\n \"\"\"Add human-readable string representations to the field.\"\"\"\n\n if \"memory\" in self.name:\n self.base_value_str = format_memory(self.base_value)\n self.experimental_value_str = format_memory(self.experimental_value)\n self.diff_str = format_memory(self.diff)\n elif \"time\" in self.name:\n self.base_value_str = format_time(self.base_value)\n self.experimental_value_str = format_time(self.experimental_value)\n self.diff_str = format_time(self.diff)\n else:\n self.base_value_str = str(self.base_value)\n self.experimental_value_str = str(self.experimental_value)\n self.diff_str = str(self.diff)\n\n\ndef build_diff(name: str, base_value: float, experimental_value: float) -> MetricDiff:\n \"\"\"Build a diff between any type of metric.\n\n :param name: name assigned to the metric to be diff-ed.\n :param base_value: base value of the metric.\n :param experimental_value: experimental value of the metric.\n \"\"\"\n diff = experimental_value - base_value\n diff_percentage = 100 * diff / base_value if base_value != 0 else \"inf\"\n\n return MetricDiff(\n name=name,\n base_value=base_value,\n experimental_value=experimental_value,\n diff=diff,\n diff_percentage=diff_percentage,\n )\n\n\n##############################\n# Resource Usage Dataclasses #\n##############################\n\n\n@dataclass\nclass MetricsSummary:\n \"\"\"Summary of metrics from one experiment.\"\"\"\n\n # Path containing the artifacts for the experiment.\n experiment_local_directory: str\n\n # Full Ludwig config.\n config: ModelConfigDict\n\n # LudwigModel output feature type.\n output_feature_type: str\n\n # LudwigModel output feature name.\n output_feature_name: str\n\n # Dictionary that maps from metric name to their values.\n metric_to_values: dict[str, float | int]\n\n # Names of metrics for the output feature.\n metric_names: set[str]\n\n\n@dataclass\nclass MetricsDiff:\n \"\"\"Store diffs for two experiments.\"\"\"\n\n # Dataset the two experiments are being compared on.\n dataset_name: str\n\n # Name of the base experiment (the one we benchmark against).\n base_experiment_name: str\n\n # Name of the experimental experiment.\n experimental_experiment_name: str\n\n # Path under which all artifacts live on the local machine.\n local_directory: str\n\n # `MetricsSummary` of the base_experiment.\n base_summary: MetricsSummary\n\n # `MetricsSummary` of the experimental_experiment.\n experimental_summary: MetricsSummary\n\n # `List[MetricDiff]` containing diffs for metric of the two experiments.\n metrics: list[MetricDiff]\n\n def to_string(self):\n ret = []\n spacing_str = \"{:<20} {:<33} {:<13} {:<13} {:<13} {:<5}\"\n ret.append(\n spacing_str.format(\n \"Output Feature Name\",\n \"Metric Name\",\n self.base_experiment_name,\n self.experimental_experiment_name,\n \"Diff\",\n \"Diff Percentage\",\n )\n )\n for metric in sorted(self.metrics, key=lambda m: m.name):\n output_feature_name = self.base_summary.output_feature_name\n metric_name = metric.name\n experiment1_val = round(metric.base_value, 3)\n experiment2_val = round(metric.experimental_value, 3)\n diff = round(metric.diff, 3)\n diff_percentage = metric.diff_percentage\n if isinstance(diff_percentage, float):\n diff_percentage = round(metric.diff_percentage, 3)\n ret.append(\n spacing_str.format(\n output_feature_name,\n metric_name,\n experiment1_val,\n experiment2_val,\n diff,\n diff_percentage,\n )\n )\n return \"\\n\".join(ret)\n\n\ndef export_metrics_diff_to_csv(metrics_diff: MetricsDiff, path: str):\n \"\"\"Export metrics report to .csv.\n\n :param metrics_diff: MetricsDiff object containing the diff for two experiments on a dataset.\n :param path: file name of the exported csv.\n \"\"\"\n with open(path, \"w\", newline=\"\") as f:\n writer = csv.DictWriter(\n f,\n fieldnames=[\n \"Dataset Name\",\n \"Output Feature Name\",\n \"Metric Name\",\n metrics_diff.base_experiment_name,\n metrics_diff.experimental_experiment_name,\n \"Diff\",\n \"Diff Percentage\",\n ],\n )\n writer.writeheader()\n\n for metric in sorted(metrics_diff.metrics, key=lambda m: m.name):\n output_feature_name = metrics_diff.base_summary.output_feature_name\n metric_name = metric.name\n experiment1_val = round(metric.base_value, 3)\n experiment2_val = round(metric.experimental_value, 3)\n diff = round(metric.diff, 3)\n diff_percentage = metric.diff_percentage\n if isinstance(diff_percentage, float):\n diff_percentage = round(metric.diff_percentage, 3)\n writer.writerow(\n {\n \"Dataset Name\": metrics_diff.dataset_name,\n \"Output Feature Name\": output_feature_name,\n \"Metric Name\": metric_name,\n metrics_diff.base_experiment_name: experiment1_val,\n metrics_diff.experimental_experiment_name: experiment2_val,\n \"Diff\": diff,\n \"Diff Percentage\": diff_percentage,\n }\n )\n logger.info(f\"Exported a CSV report to {path}\\n\")\n\n\ndef build_metrics_summary(experiment_local_directory: str) -> MetricsSummary:\n \"\"\"Build a metrics summary for an experiment.\n\n :param experiment_local_directory: directory where the experiment artifacts live.\n e.g. local_experiment_repo/ames_housing/some_experiment/\n \"\"\"\n config = load_json(\n os.path.join(experiment_local_directory, \"experiment_run\", MODEL_FILE_NAME, MODEL_HYPERPARAMETERS_FILE_NAME)\n )\n report = load_json(os.path.join(experiment_local_directory, \"experiment_run\", \"test_statistics.json\"))\n output_feature_type: str = config[\"output_features\"][0][\"type\"]\n output_feature_name: str = config[\"output_features\"][0][\"name\"]\n metric_dict = report[output_feature_name]\n full_metric_names = get_metric_classes(output_feature_type)\n metric_to_values: dict[str, float | int] = {\n metric_name: metric_dict[metric_name] for metric_name in full_metric_names if metric_name in metric_dict\n }\n metric_names: set[str] = set(metric_to_values)\n\n return MetricsSummary(\n experiment_local_directory=experiment_local_directory,\n config=config,\n output_feature_name=output_feature_name,\n output_feature_type=output_feature_type,\n metric_to_values=metric_to_values,\n metric_names=metric_names,\n )\n\n\ndef build_metrics_diff(\n dataset_name: str, base_experiment_name: str, experimental_experiment_name: str, local_directory: str\n) -> MetricsDiff:\n \"\"\"Build a MetricsDiff object between two experiments on a dataset.\n\n :param dataset_name: the name of the Ludwig dataset.\n :param base_experiment_name: the name of the base experiment.\n :param experimental_experiment_name: the name of the experimental experiment.\n :param local_directory: the local directory where the experiment artifacts are downloaded.\n \"\"\"\n base_summary: MetricsSummary = build_metrics_summary(\n os.path.join(local_directory, dataset_name, base_experiment_name)\n )\n experimental_summary: MetricsSummary = build_metrics_summary(\n os.path.join(local_directory, dataset_name, experimental_experiment_name)\n )\n\n metrics_in_common = set(base_summary.metric_names).intersection(set(experimental_summary.metric_names))\n\n metrics: list[MetricDiff] = [\n build_diff(name, base_summary.metric_to_values[name], experimental_summary.metric_to_values[name])\n for name in metrics_in_common\n ]\n\n return MetricsDiff(\n dataset_name=dataset_name,\n base_experiment_name=base_experiment_name,\n experimental_experiment_name=experimental_experiment_name,\n local_directory=local_directory,\n base_summary=base_summary,\n experimental_summary=experimental_summary,\n metrics=metrics,\n )\n\n\n##############################\n# Resource Usage Dataclasses #\n##############################\n\n\n@dataclass\nclass ResourceUsageSummary:\n \"\"\"Summary of resource usage metrics from one experiment.\"\"\"\n\n # The tag with which the code block/function is labeled.\n code_block_tag: str\n\n # Dictionary that maps from metric name to their values.\n metric_to_values: dict[str, float | int]\n\n # Names of metrics for the output feature.\n metric_names: set[str]\n\n\n@dataclass\nclass ResourceUsageDiff:\n \"\"\"Store resource usage diffs for two experiments.\"\"\"\n\n # The tag with which the code block/function is labeled.\n code_block_tag: str\n\n # Name of the base experiment (the one we benchmark against).\n base_experiment_name: str\n\n # Name of the experimental experiment.\n experimental_experiment_name: str\n\n # `List[Diff]` containing diffs for metric of the two experiments.\n metrics: list[MetricDiff]\n\n def to_string(self):\n ret = []\n spacing_str = \"{:<36} {:<20} {:<20} {:<20} {:<5}\"\n ret.append(\n spacing_str.format(\n \"Metric Name\",\n self.base_experiment_name,\n self.experimental_experiment_name,\n \"Diff\",\n \"Diff Percentage\",\n )\n )\n for metric in sorted(self.metrics, key=lambda m: m.name):\n diff_percentage = metric.diff_percentage\n if isinstance(metric.diff_percentage, float):\n diff_percentage = round(metric.diff_percentage, 3)\n ret.append(\n spacing_str.format(\n metric.name,\n metric.base_value_str,\n metric.experimental_value_str,\n metric.diff_str,\n diff_percentage,\n )\n )\n return \"\\n\".join(ret)\n\n\ndef export_resource_usage_diff_to_csv(resource_usage_diff: ResourceUsageDiff, path: str):\n \"\"\"Export resource usage metrics report to .csv.\n\n :param resource_usage_diff: ResourceUsageDiff object containing the diff for two experiments on a dataset.\n :param path: file name of the exported csv.\n \"\"\"\n with open(path, \"w\", newline=\"\") as f:\n writer = csv.DictWriter(\n f,\n fieldnames=[\n \"Code Block Tag\",\n \"Metric Name\",\n resource_usage_diff.base_experiment_name,\n resource_usage_diff.experimental_experiment_name,\n \"Diff\",\n \"Diff Percentage\",\n ],\n )\n writer.writeheader()\n\n for metric in sorted(resource_usage_diff.metrics, key=lambda m: m.name):\n diff_percentage = metric.diff_percentage\n if isinstance(metric.diff_percentage, float):\n diff_percentage = round(metric.diff_percentage, 3)\n writer.writerow(\n {\n \"Code Block Tag\": resource_usage_diff.code_block_tag,\n \"Metric Name\": metric.name,\n resource_usage_diff.base_experiment_name: metric.base_value_str,\n resource_usage_diff.experimental_experiment_name: metric.experimental_value_str,\n \"Diff\": metric.diff_str,\n \"Diff Percentage\": diff_percentage,\n }\n )\n logger.info(f\"Exported a CSV report to {path}\\n\")\n\n\ndef average_runs(path_to_runs_dir: str) -> dict[str, int | float]:\n \"\"\"Return average metrics from code blocks/function that ran more than once.\n\n Metrics for code blocks/functions that were executed exactly once will be returned as is.\n\n :param path_to_runs_dir: path to where metrics specific to a tag are stored.\n e.g. resource_usage_out_dir/torch_ops_resource_usage/LudwigModel.evaluate/\n This directory will contain JSON files with the following pattern run_*.json\n \"\"\"\n runs = [load_json(os.path.join(path_to_runs_dir, run)) for run in os.listdir(path_to_runs_dir)]\n # asserting that keys to each of the dictionaries are consistent throughout the runs.\n assert len(runs) == 1 or all(runs[i].keys() == runs[i + 1].keys() for i in range(len(runs) - 1))\n runs_average = {\"num_runs\": len(runs)}\n for key in runs[0]:\n if isinstance(runs[0][key], (int, float)):\n runs_average[key] = mean([run[key] for run in runs])\n return runs_average\n\n\ndef summarize_resource_usage(path: str, tags: list[str] | None = None) -> list[ResourceUsageSummary]:\n \"\"\"Create resource usage summaries for each code block/function that was decorated with ResourceUsageTracker.\n\n Each entry of the list corresponds to the metrics collected from a code block/function run.\n Important: code blocks that ran more than once are averaged.\n\n :param path: corresponds to the `output_dir` argument in a ResourceUsageTracker run.\n :param tags: (optional) list of tags to create summary for. If None, metrics from all tags will be summarized.\n \"\"\"\n summary = dict()\n # metric types: system_resource_usage, torch_ops_resource_usage.\n all_metric_types = {\"system_resource_usage\", \"torch_ops_resource_usage\"}\n for metric_type in all_metric_types.intersection(os.listdir(path)):\n metric_type_path = os.path.join(path, metric_type)\n # code block tags correspond to the `tag` argument in ResourceUsageTracker.\n for code_block_tag in os.listdir(metric_type_path):\n if tags and code_block_tag not in tags:\n continue\n if code_block_tag not in summary:\n summary[code_block_tag] = {}\n run_path = os.path.join(metric_type_path, code_block_tag)\n # Metrics from code blocks/functions that ran more than once are averaged.\n summary[code_block_tag][metric_type] = average_runs(run_path)\n\n summary_list = []\n for code_block_tag, metric_type_dicts in summary.items():\n merged_summary: dict[str, float | int] = {}\n for metrics in metric_type_dicts.values():\n assert \"num_runs\" in metrics\n assert \"num_runs\" not in merged_summary or metrics[\"num_runs\"] == merged_summary[\"num_runs\"]\n merged_summary.update(metrics)\n summary_list.append(\n ResourceUsageSummary(\n code_block_tag=code_block_tag, metric_to_values=merged_summary, metric_names=set(merged_summary)\n )\n )\n return summary_list\n\n\ndef build_resource_usage_diff(\n base_path: str,\n experimental_path: str,\n base_experiment_name: str | None = None,\n experimental_experiment_name: str | None = None,\n) -> list[ResourceUsageDiff]:\n \"\"\"Build and return a ResourceUsageDiff object to diff resource usage metrics between two experiments.\n\n :param base_path: corresponds to the `output_dir` argument in the base ResourceUsageTracker run.\n :param experimental_path: corresponds to the `output_dir` argument in the experimental ResourceUsageTracker run.\n \"\"\"\n base_summary_list = summarize_resource_usage(base_path)\n experimental_summary_list = summarize_resource_usage(experimental_path)\n\n summaries_list = []\n for base_summary in base_summary_list:\n for experimental_summary in experimental_summary_list:\n if base_summary.code_block_tag == experimental_summary.code_block_tag:\n summaries_list.append((base_summary, experimental_summary))\n\n diffs = []\n for base_summary, experimental_summary in summaries_list:\n metrics_in_common = set(base_summary.metric_names).intersection(set(experimental_summary.metric_names))\n metrics: list[MetricDiff] = [\n build_diff(name, base_summary.metric_to_values[name], experimental_summary.metric_to_values[name])\n for name in metrics_in_common\n ]\n diff = ResourceUsageDiff(\n code_block_tag=base_summary.code_block_tag,\n base_experiment_name=base_experiment_name if base_experiment_name else \"experiment_1\",\n experimental_experiment_name=(\n experimental_experiment_name if experimental_experiment_name else \"experiment_2\"\n ),\n metrics=metrics,\n )\n diffs.append(diff)\n return diffs\n"
},
{
"path": "ludwig/benchmarking/utils.py",
"content": "import asyncio\nimport functools\nimport logging\nimport os\nimport shutil\nimport uuid\nfrom concurrent.futures import ThreadPoolExecutor\nfrom types import ModuleType\nfrom typing import Any\n\nimport fsspec\nimport pandas as pd\nimport yaml\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import BINARY, CATEGORY\nfrom ludwig.datasets import model_configs_for_dataset\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\nfrom ludwig.globals import CONFIG_YAML, MODEL_FILE_NAME, MODEL_WEIGHTS_FILE_NAME\nfrom ludwig.utils.data_utils import load_yaml\nfrom ludwig.utils.dataset_utils import get_repeatable_train_val_test_split\nfrom ludwig.utils.defaults import default_random_seed\nfrom ludwig.utils.fs_utils import get_fs_and_path\n\nHYPEROPT_OUTDIR_RETAINED_FILES = [\n \"hyperopt_statistics.json\",\n \"params.json\",\n \"stderr\",\n \"stdout\",\n \"result.json\",\n \"error.txt\",\n]\nlogger = logging.getLogger()\n\n\ndef load_from_module(\n dataset_module: DatasetLoader | ModuleType, output_feature: dict[str, str], subsample_frac: float = 1\n) -> pd.DataFrame:\n \"\"\"Load the ludwig dataset, optionally subsamples it, and returns a repeatable split. A stratified split is\n used for classification datasets.\n\n Args:\n dataset_module: ludwig datasets module (e.g. ludwig.datasets.sst2, ludwig.datasets.ames_housing, etc.)\n subsample_frac: percentage of the total dataset to load.\n \"\"\"\n dataset = dataset_module.load(split=False)\n if subsample_frac < 1:\n dataset = dataset.sample(frac=subsample_frac, replace=False, random_state=default_random_seed)\n\n if output_feature[\"type\"] in [CATEGORY, BINARY]:\n return get_repeatable_train_val_test_split(\n dataset,\n stratify_colname=output_feature[\"name\"],\n random_seed=default_random_seed,\n )\n else:\n return get_repeatable_train_val_test_split(dataset, random_seed=default_random_seed)\n\n\ndef export_artifacts(experiment: dict[str, str], experiment_output_directory: str, export_base_path: str):\n \"\"\"Save the experiment artifacts to the `bench_export_directory`.\n\n Args:\n experiment: experiment dict that contains \"dataset_name\" (e.g. ames_housing),\n \"experiment_name\" (specified by user), and \"config_path\" (path to experiment config.\n Relative to ludwig/benchmarks/configs).\n experiment_output_directory: path where the model, data, and logs of the experiment are saved.\n export_base_path: remote or local path (directory) where artifacts are\n exported. (e.g. s3://benchmarking.us-west-2.ludwig.com/bench/ or your/local/bench/)\n \"\"\"\n protocol, _ = fsspec.core.split_protocol(export_base_path)\n fs, _ = get_fs_and_path(export_base_path)\n try:\n export_full_path = os.path.join(export_base_path, experiment[\"dataset_name\"], experiment[\"experiment_name\"])\n\n # override previous experiment with the same name\n if fs.exists(export_full_path):\n fs.rm(export_full_path, recursive=True)\n fs.put(experiment_output_directory, export_full_path, recursive=True)\n fs.put(\n os.path.join(experiment[\"config_path\"]),\n os.path.join(export_full_path, CONFIG_YAML),\n )\n logger.info(f\"Uploaded experiment artifact to\\n\\t{export_full_path}\")\n except Exception:\n logger.exception(\n f\"Failed to upload experiment artifacts for experiment *{experiment['experiment_name']}* on \"\n f\"dataset {experiment['dataset_name']}\"\n )\n\n\ndef download_artifacts(\n bench_config_path: str,\n base_experiment: str,\n experimental_experiment: str,\n download_base_path: str,\n local_dir: str = \"benchmarking_summaries\",\n) -> tuple[str, list[str]]:\n \"\"\"Download benchmarking artifacts for two experiments.\n\n Args:\n bench_config_path: bench config file path. Can be the same one that was used to run\n these experiments.\n base_experiment: name of the experiment we're comparing against.\n experimental_experiment: name of the experiment we're comparing.\n download_base_path: base path under which live the stored artifacts of\n the benchmarking experiments.\n \"\"\"\n bench_config = load_yaml(bench_config_path)\n protocol, _ = fsspec.core.split_protocol(download_base_path)\n fs, _ = get_fs_and_path(download_base_path)\n os.makedirs(local_dir, exist_ok=True)\n\n coroutines = []\n for experiment in bench_config[\"experiments\"]:\n dataset_name = experiment[\"dataset_name\"]\n for experiment_name in [base_experiment, experimental_experiment]:\n coroutines.append(download_one(fs, download_base_path, dataset_name, experiment_name, local_dir))\n downloaded_names = asyncio.run(asyncio.gather(*coroutines, return_exceptions=True))\n\n dataset_names = [experiment_tuple[0] for experiment_tuple in set(downloaded_names) if experiment_tuple[0]]\n assert (\n len({experiment_tuple[1] for experiment_tuple in downloaded_names}) == 1 and downloaded_names[0][1] == local_dir\n ), \"Experiments not downloaded to the same path\"\n\n return local_dir, dataset_names\n\n\n@DeveloperAPI\nasync def download_one(\n fs, download_base_path: str, dataset_name: str, experiment_name: str, local_dir: str\n) -> tuple[str, str]:\n \"\"\"Download `config.yaml` and `report.json` for an experiment.\n\n Args:\n fs: filesystem to use to download.\n download_base_path: base path under which live the stored artifacts of\n the benchmarking experiments.\n dataset_name: name of the dataset we ran the experiments on.\n experiment_name: name of the experiment (e.g. `v0.5.3_with_bert`)\n local_dir: local directory under which the artifacts will be downloaded.\n \"\"\"\n loop = asyncio.get_running_loop()\n local_experiment_dir = os.path.join(local_dir, dataset_name, experiment_name)\n remote_experiment_directory = os.path.join(download_base_path, dataset_name, experiment_name)\n os.makedirs(local_experiment_dir, exist_ok=True)\n try:\n with ThreadPoolExecutor() as pool:\n func = functools.partial(\n fs.get,\n remote_experiment_directory,\n local_experiment_dir,\n recursive=True,\n )\n await loop.run_in_executor(pool, func)\n except Exception:\n logger.exception(f\"Couldn't download experiment *{experiment_name}* of dataset *{dataset_name}*.\")\n return \"\", local_dir\n return dataset_name, local_dir\n\n\ndef validate_benchmarking_config(benchmarking_config: dict[str, Any]) -> None:\n \"\"\"Validates the parameters of the benchmarking config.\n\n Args:\n benchmarking_config: benchmarking config dictionary.\n\n Raises:\n ValueError if any of the expected parameters is not there.\n \"\"\"\n if \"experiment_name\" not in benchmarking_config and not all(\n \"experiment_name\" in experiment for experiment in benchmarking_config[\"experiments\"]\n ):\n raise ValueError(\"You must either specify a global experiment name or an experiment name for each experiment.\")\n if \"export\" not in benchmarking_config:\n raise ValueError(\"\"\"You must specify export parameters. Example:\n export:\n export_artifacts: true\n export_base_path: s3://benchmarking.us-west-2.ludwig.com/bench/ # include the slash at the end.\n \"\"\")\n if \"experiments\" not in benchmarking_config:\n raise ValueError(\"You must specify a list of experiments.\")\n for experiment in benchmarking_config[\"experiments\"]:\n if \"dataset_name\" not in experiment:\n raise ValueError(\"A Ludwig dataset must be specified.\")\n\n\ndef populate_benchmarking_config_with_defaults(benchmarking_config: dict[str, Any]) -> dict[str, Any]:\n \"\"\"Populates the parameters of the benchmarking config with defaults.\n\n Args:\n benchmarking_config: benchmarking config dictionary.\n \"\"\"\n if \"hyperopt\" not in benchmarking_config:\n benchmarking_config[\"hyperopt\"] = False\n if \"process_config_file_path\" not in benchmarking_config:\n benchmarking_config[\"process_config_file_path\"] = None\n if \"profiler\" not in benchmarking_config:\n benchmarking_config[\"profiler\"] = {\"enable\": False, \"use_torch_profiler\": False, \"logging_interval\": None}\n return benchmarking_config\n\n\ndef propagate_global_parameters(benchmarking_config: dict[str, Any]) -> dict[str, Any]:\n \"\"\"Propagate the global parameters of the benchmarking config to local experiments.\n\n Args:\n benchmarking_config: benchmarking config dictionary.\n \"\"\"\n for experiment in benchmarking_config[\"experiments\"]:\n if \"experiment_name\" not in experiment:\n experiment[\"experiment_name\"] = benchmarking_config[\"experiment_name\"]\n if \"export\" not in experiment:\n experiment[\"export\"] = benchmarking_config[\"export\"]\n if \"hyperopt\" not in experiment:\n experiment[\"hyperopt\"] = benchmarking_config[\"hyperopt\"]\n if \"process_config_file_path\" not in experiment:\n experiment[\"process_config_file_path\"] = benchmarking_config[\"process_config_file_path\"]\n if \"profiler\" not in experiment:\n experiment[\"profiler\"] = benchmarking_config[\"profiler\"]\n return benchmarking_config\n\n\ndef create_default_config(experiment: dict[str, Any]) -> str:\n \"\"\"Create a Ludwig config that only contains input and output features.\n\n Args:\n experiment: experiment dictionary.\n\n Returns:\n path where the default config is saved.\n \"\"\"\n model_config = model_configs_for_dataset(experiment[\"dataset_name\"])[\"default\"]\n\n # only keep input_features and output_features\n main_config_keys = list(model_config.keys())\n for key in main_config_keys:\n if key not in [\"input_features\", \"output_features\"]:\n del model_config[key]\n config_path = f\"{experiment['dataset_name']}-{uuid.uuid4().hex}.yaml\"\n save_yaml(config_path, model_config)\n return config_path\n\n\ndef delete_model_checkpoints(output_directory: str):\n \"\"\"Deletes outputs of the experiment run that we don't want to save with the artifacts.\n\n Args:\n output_directory: output directory of the hyperopt run.\n \"\"\"\n shutil.rmtree(os.path.join(output_directory, MODEL_FILE_NAME, \"training_checkpoints\"), ignore_errors=True)\n if os.path.isfile(os.path.join(output_directory, MODEL_FILE_NAME, MODEL_WEIGHTS_FILE_NAME)):\n os.remove(os.path.join(output_directory, MODEL_FILE_NAME, MODEL_WEIGHTS_FILE_NAME))\n\n\ndef delete_hyperopt_outputs(output_directory: str):\n \"\"\"Deletes outputs of the hyperopt run that we don't want to save with the artifacts.\n\n Args:\n output_directory: output directory of the hyperopt run.\n \"\"\"\n for path, currentDirectory, files in os.walk(output_directory):\n for file in files:\n filename = os.path.join(path, file)\n if file not in HYPEROPT_OUTDIR_RETAINED_FILES:\n os.remove(filename)\n\n\ndef save_yaml(filename, dictionary):\n with open(filename, \"w\") as f:\n yaml.dump(dictionary, f, default_flow_style=False)\n\n\ndef format_time(time_us):\n \"\"\"Defines how to format time in FunctionEvent.\n\n from https://github.com/pytorch/pytorch/blob/master/torch/autograd/profiler_util.py\n \"\"\"\n US_IN_SECOND = 1000.0 * 1000.0\n US_IN_MS = 1000.0\n if time_us >= US_IN_SECOND:\n return f\"{time_us / US_IN_SECOND:.3f}s\"\n if time_us >= US_IN_MS:\n return f\"{time_us / US_IN_MS:.3f}ms\"\n return f\"{time_us:.3f}us\"\n\n\ndef format_memory(nbytes):\n \"\"\"Returns a formatted memory size string.\n\n from https://github.com/pytorch/pytorch/blob/master/torch/autograd/profiler_util.py\n \"\"\"\n KB = 1024\n MB = 1024 * KB\n GB = 1024 * MB\n if abs(nbytes) >= GB:\n return f\"{nbytes * 1.0 / GB:.2f} Gb\"\n elif abs(nbytes) >= MB:\n return f\"{nbytes * 1.0 / MB:.2f} Mb\"\n elif abs(nbytes) >= KB:\n return f\"{nbytes * 1.0 / KB:.2f} Kb\"\n else:\n return str(nbytes) + \" b\"\n"
},
{
"path": "ludwig/callbacks.py",
"content": "# !/usr/bin/env python\n# Copyright (c) 2021 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nfrom abc import ABC\nfrom collections.abc import Callable\nfrom typing import Any\n\nfrom ludwig.api_annotations import PublicAPI\nfrom ludwig.types import HyperoptConfigDict, ModelConfigDict, TrainingSetMetadataDict\n\n\n@PublicAPI\nclass Callback(ABC):\n def on_cmdline(self, cmd: str, *args: list[str]):\n \"\"\"Called when Ludwig is run on the command line with the callback enabled.\n\n :param cmd: The Ludwig subcommand being run, ex. \"train\", \"evaluate\", \"predict\", ...\n :param args: The full list of command-line arguments (sys.argv).\n \"\"\"\n\n def on_preprocess_start(self, config: ModelConfigDict, **kwargs):\n \"\"\"Called before preprocessing starts.\n\n :param config: The config dictionary.\n \"\"\"\n\n def on_preprocess_end(\n self,\n training_set,\n validation_set,\n test_set,\n training_set_metadata: TrainingSetMetadataDict,\n **kwargs,\n ):\n \"\"\"Called after preprocessing ends.\n\n :param training_set: The training set.\n :type training_set: ludwig.dataset.base.Dataset\n :param validation_set: The validation set.\n :type validation_set: ludwig.dataset.base.Dataset\n :param test_set: The test set.\n :type test_set: ludwig.dataset.base.Dataset\n :param training_set_metadata: Values inferred from the training set, including preprocessing settings,\n vocabularies, feature statistics, etc. Same as training_set_metadata.json.\n \"\"\"\n\n def on_hyperopt_init(self, experiment_name: str, **kwargs):\n \"\"\"Called to initialize state before hyperparameter optimization begins.\n\n :param experiment_name: The name of the current experiment.\n \"\"\"\n\n def on_hyperopt_preprocessing_start(self, experiment_name: str, **kwargs):\n \"\"\"Called before data preprocessing for hyperparameter optimization begins.\n\n :param experiment_name: The name of the current experiment.\n \"\"\"\n\n def on_hyperopt_preprocessing_end(self, experiment_name: str, **kwargs):\n \"\"\"Called after data preprocessing for hyperparameter optimization is completed.\n\n :param experiment_name: The name of the current experiment.\n \"\"\"\n\n def on_hyperopt_start(self, experiment_name: str, **kwargs):\n \"\"\"Called before any hyperparameter optimization trials are started.\n\n :param experiment_name: The name of the current experiment.\n \"\"\"\n\n def on_hyperopt_end(self, experiment_name: str, **kwargs):\n \"\"\"Called after all hyperparameter optimization trials are completed.\n\n :param experiment_name: The name of the current experiment.\n \"\"\"\n\n def on_hyperopt_finish(self, experiment_name: str, **kwargs):\n \"\"\"Deprecated.\n\n Use on_hyperopt_end instead.\n \"\"\"\n # TODO(travis): remove in favor of on_hyperopt_end for naming consistency\n\n def on_hyperopt_trial_start(self, parameters: HyperoptConfigDict, **kwargs):\n \"\"\"Called before the start of each hyperparameter optimization trial.\n\n :param parameters: The complete dictionary of parameters for this hyperparameter optimization experiment.\n \"\"\"\n\n def on_hyperopt_trial_end(self, parameters: HyperoptConfigDict, **kwargs):\n \"\"\"Called after the end of each hyperparameter optimization trial.\n\n :param parameters: The complete dictionary of parameters for this hyperparameter optimization experiment.\n \"\"\"\n\n def should_stop_hyperopt(self):\n \"\"\"Returns true if the entire hyperopt run (all trials) should be stopped.\n\n See: https://docs.ray.io/en/latest/tune/api_docs/stoppers.html#ray.tune.Stopper\n \"\"\"\n return False\n\n def on_resume_training(self, is_coordinator: bool, **kwargs):\n pass\n\n def on_train_init(\n self,\n base_config: ModelConfigDict,\n experiment_directory: str,\n experiment_name: str,\n model_name: str,\n output_directory: str,\n resume_directory: str | None,\n **kwargs,\n ):\n \"\"\"Called after preprocessing, but before the creation of the model and trainer objects.\n\n :param base_config: The user-specified config, before the insertion of defaults or inferred values.\n :param experiment_directory: The experiment directory, same as output_directory if no experiment specified.\n :param experiment_name: The experiment name.\n :param model_name: The model name.\n :param output_directory: file path to where training results are stored.\n :param resume_directory: model directory to resume training from, or None.\n \"\"\"\n\n def on_train_start(\n self,\n model,\n config: ModelConfigDict,\n config_fp: str | None,\n **kwargs,\n ):\n \"\"\"Called after creation of trainer, before the start of training.\n\n :param model: The ludwig model.\n :type model: ludwig.utils.torch_utils.LudwigModule\n :param config: The config dictionary.\n :param config_fp: The file path to the config, or none if config was passed to stdin.\n \"\"\"\n\n def on_train_end(self, output_directory: str, **kwargs):\n \"\"\"Called at the end of training, before the model is saved.\n\n :param output_directory: file path to where training results are stored.\n \"\"\"\n\n def on_trainer_train_setup(self, trainer, save_path: str, is_coordinator: bool, **kwargs):\n \"\"\"Called in every trainer (distributed or local) before training starts.\n\n :param trainer: The trainer instance.\n :type trainer: trainer: ludwig.models.Trainer\n :param save_path: The path to the directory model is saved in.\n :param is_coordinator: Is this trainer the coordinator.\n \"\"\"\n\n def on_trainer_train_teardown(self, trainer, progress_tracker, save_path: str, is_coordinator: bool, **kwargs):\n \"\"\"Called in every trainer (distributed or local) after training completes.\n\n :param trainer: The trainer instance.\n :type trainer: ludwig.models.trainer.Trainer\n :param progress_tracker: An object which tracks training progress.\n :type progress_tracker: ludwig.utils.trainer_utils.ProgressTracker\n :param save_path: The path to the directory model is saved in.\n :param is_coordinator: Is this trainer the coordinator.\n \"\"\"\n\n def on_batch_start(self, trainer, progress_tracker, save_path: str, **kwargs):\n \"\"\"Called on coordinator only before each batch.\n\n :param trainer: The trainer instance.\n :type trainer: ludwig.models.trainer.Trainer\n :param progress_tracker: An object which tracks training progress.\n :type progress_tracker: ludwig.utils.trainer_utils.ProgressTracker\n :param save_path: The path to the directory model is saved in.\n \"\"\"\n\n def on_batch_end(self, trainer, progress_tracker, save_path: str, sync_step: bool = True, **kwargs):\n \"\"\"Called on coordinator only after each batch.\n\n :param trainer: The trainer instance.\n :type trainer: ludwig.models.trainer.Trainer\n :param progress_tracker: An object which tracks training progress.\n :type progress_tracker: ludwig.utils.trainer_utils.ProgressTracker\n :param save_path: The path to the directory model is saved in.\n :param sync_step: Whether the model params were updated and synced in this step.\n \"\"\"\n\n def on_eval_start(self, trainer, progress_tracker, save_path: str, **kwargs):\n \"\"\"Called on coordinator at the start of evaluation.\n\n :param trainer: The trainer instance.\n :type trainer: ludwig.models.trainer.Trainer\n :param progress_tracker: An object which tracks training progress.\n :type progress_tracker: ludwig.utils.trainer_utils.ProgressTracker\n :param save_path: The path to the directory model is saved in.\n \"\"\"\n\n def on_eval_end(self, trainer, progress_tracker, save_path: str, **kwargs):\n \"\"\"Called on coordinator at the end of evaluation.\n\n :param trainer: The trainer instance.\n :type trainer: ludwig.models.trainer.Trainer\n :param progress_tracker: An object which tracks training progress.\n :type progress_tracker: ludwig.utils.trainer_utils.ProgressTracker\n :param save_path: The path to the directory model is saved in.\n \"\"\"\n\n def on_epoch_start(self, trainer, progress_tracker, save_path: str, **kwargs):\n \"\"\"Called on coordinator only before the start of each epoch.\n\n :param trainer: The trainer instance.\n :type trainer: ludwig.models.trainer.Trainer\n :param progress_tracker: An object which tracks training progress.\n :type progress_tracker: ludwig.utils.trainer_utils.ProgressTracker\n :param save_path: The path to the directory model is saved in.\n \"\"\"\n\n def on_epoch_end(self, trainer, progress_tracker, save_path: str, **kwargs):\n \"\"\"Called on coordinator only after the end of each epoch.\n\n :param trainer: The trainer instance.\n :type trainer: ludwig.models.trainer.Trainer\n :param progress_tracker: An object which tracks training progress.\n :type progress_tracker: ludwig.utils.trainer_utils.ProgressTracker\n :param save_path: The path to the directory model is saved in.\n \"\"\"\n\n def on_validation_start(self, trainer, progress_tracker, save_path: str, **kwargs):\n \"\"\"Called on coordinator before validation starts.\n\n :param trainer: The trainer instance.\n :type trainer: ludwig.models.trainer.Trainer\n :param progress_tracker: An object which tracks training progress.\n :type progress_tracker: ludwig.utils.trainer_utils.ProgressTracker\n :param save_path: The path to the directory model is saved in.\n \"\"\"\n\n def on_validation_end(self, trainer, progress_tracker, save_path: str, **kwargs):\n \"\"\"Called on coordinator after validation is complete.\n\n :param trainer: The trainer instance.\n :type trainer: ludwig.models.trainer.Trainer\n :param progress_tracker: An object which tracks training progress.\n :type progress_tracker: ludwig.utils.trainer_utils.ProgressTracker\n :param save_path: The path to the directory model is saved in.\n \"\"\"\n\n def on_test_start(self, trainer, progress_tracker, save_path: str, **kwargs):\n \"\"\"Called on coordinator before testing starts.\n\n :param trainer: The trainer instance.\n :type trainer: ludwig.models.trainer.Trainer\n :param progress_tracker: An object which tracks training progress.\n :type progress_tracker: ludwig.utils.trainer_utils.ProgressTracker\n :param save_path: The path to the directory model is saved in.\n \"\"\"\n\n def on_test_end(self, trainer, progress_tracker, save_path: str, **kwargs):\n \"\"\"Called on coordinator after testing ends.\n\n :param trainer: The trainer instance.\n :type trainer: ludwig.models.trainer.Trainer\n :param progress_tracker: An object which tracks training progress.\n :type progress_tracker: ludwig.utils.trainer_utils.ProgressTracker\n :param save_path: The path to the directory model is saved in.\n \"\"\"\n\n def should_early_stop(self, trainer, progress_tracker, is_coordinator, **kwargs):\n # Triggers early stopping if any callback on any worker returns True\n return False\n\n def on_checkpoint(self, trainer, progress_tracker, **kwargs):\n \"\"\"Called after each checkpoint is passed, regardless of whether the model was evaluated or saved at that\n checkpoint.\"\"\"\n\n def on_save_best_checkpoint(self, trainer, progress_tracker, save_path, **kwargs):\n \"\"\"Called on every worker immediately after a new best model is checkpointed.\"\"\"\n\n def on_build_metadata_start(self, df, mode: str, **kwargs):\n \"\"\"Called before building metadata for dataset.\n\n :param df: The dataset.\n :type df: pd.DataFrame\n :param mode: \"prediction\", \"training\", or None.\n \"\"\"\n\n def on_build_metadata_end(self, df, mode, **kwargs):\n \"\"\"Called after building dataset metadata.\n\n :param df: The dataset.\n :type df: pd.DataFrame\n :param mode: \"prediction\", \"training\", or None.\n \"\"\"\n\n def on_build_data_start(self, df, mode, **kwargs):\n \"\"\"Called before build_data, which does preprocessing, handling missing values, adding metadata to\n training_set_metadata.\n\n :param df: The dataset.\n :type df: pd.DataFrame\n :param mode: \"prediction\", \"training\", or None.\n \"\"\"\n\n def on_build_data_end(self, df, mode, **kwargs):\n \"\"\"Called after build_data completes.\n\n :param df: The dataset.\n :type df: pd.DataFrame\n :param mode: \"prediction\", \"training\", or None.\n \"\"\"\n\n def on_evaluation_start(self, **kwargs):\n \"\"\"Called before preprocessing for evaluation.\"\"\"\n\n def on_evaluation_end(self, **kwargs):\n \"\"\"Called after evaluation is complete.\"\"\"\n\n def on_visualize_figure(self, fig, **kwargs):\n \"\"\"Called after a visualization is generated.\n\n :param fig: The figure.\n :type fig: matplotlib.figure.Figure\n \"\"\"\n\n def on_ludwig_end(self, **kwargs):\n \"\"\"Convenience method for any cleanup.\n\n Not yet implemented.\n \"\"\"\n\n def prepare_ray_tune(\n self,\n train_fn: Callable,\n tune_config: dict[str, Any],\n tune_callbacks: list[Callable],\n **kwargs,\n ):\n \"\"\"Configures Ray Tune callback and config.\n\n :param train_fn: The function which runs the experiment trial.\n :param tune_config: The ray tune configuration dictionary.\n :param tune_callbacks: List of callbacks (not used yet).\n :returns: Tuple[Callable, Dict] The train_fn and tune_config, which will be passed to ray tune.\n \"\"\"\n return train_fn, tune_config\n"
},
{
"path": "ludwig/check.py",
"content": "import argparse\nimport logging\nimport tempfile\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import INPUT_FEATURES, OUTPUT_FEATURES, TRAINER\nfrom ludwig.data.dataset_synthesizer import build_synthetic_dataset_df\nfrom ludwig.globals import LUDWIG_VERSION\nfrom ludwig.utils.print_utils import get_logging_level_registry, print_ludwig\n\nNUM_EXAMPLES = 100\n\n\n@DeveloperAPI\ndef check_install(logging_level: int = logging.INFO, **kwargs):\n config = {\n INPUT_FEATURES: [\n {\"name\": \"in1\", \"type\": \"text\"},\n {\"name\": \"in2\", \"type\": \"category\"},\n {\"name\": \"in3\", \"type\": \"number\"},\n ],\n OUTPUT_FEATURES: [{\"name\": \"out1\", \"type\": \"binary\"}],\n TRAINER: {\"epochs\": 2, \"batch_size\": 8},\n }\n\n try:\n df = build_synthetic_dataset_df(NUM_EXAMPLES, config)\n model = LudwigModel(config, logging_level=logging_level)\n with tempfile.TemporaryDirectory() as tmpdir:\n model.train(dataset=df, output_directory=tmpdir)\n except Exception:\n print(\"=== CHECK INSTALL COMPLETE... FAILURE ===\")\n raise\n\n print(\"=== CHECK INSTALL COMPLETE... SUCCESS ===\")\n\n\n@DeveloperAPI\ndef cli(sys_argv):\n parser = argparse.ArgumentParser(\n description=\"This command checks Ludwig installation on a synthetic dataset.\",\n prog=\"ludwig check_install\",\n usage=\"%(prog)s [options]\",\n )\n\n parser.add_argument(\n \"-l\",\n \"--logging_level\",\n default=\"warning\",\n help=\"the level of logging to use\",\n choices=[\"critical\", \"error\", \"warning\", \"info\", \"debug\", \"notset\"],\n )\n\n args = parser.parse_args(sys_argv)\n\n args.logging_level = get_logging_level_registry()[args.logging_level]\n logging.getLogger(\"ludwig\").setLevel(args.logging_level)\n global logger\n logger = logging.getLogger(\"ludwig.check\")\n\n print_ludwig(\"Check Install\", LUDWIG_VERSION)\n check_install(**vars(args))\n"
},
{
"path": "ludwig/cli.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport argparse\nimport sys\n\nimport ludwig.contrib\nfrom ludwig.globals import LUDWIG_VERSION\nfrom ludwig.utils.print_utils import get_logo\n\n\nclass CLI:\n \"\"\"CLI describes a command line interface for interacting with Ludwig.\n\n Functions are described below.\n \"\"\"\n\n def __init__(self):\n parser = argparse.ArgumentParser(\n description=\"ludwig cli runner\",\n usage=f\"\"\"\\n{get_logo(\"ludwig cli\", LUDWIG_VERSION)}\nludwig []\n\nAvailable sub-commands:\n train Trains a model\n predict Predicts using a pretrained model\n evaluate Evaluate a pretrained model's performance\n forecast Forecast the next n data points in a timeseries using a pretrained model\n experiment Runs a full experiment training a model and evaluating it\n hyperopt Perform hyperparameter optimization\n benchmark Run and track experiments on a number of datasets and configs, and export experiment artifacts.\n serve Serves a pretrained model\n visualize Visualizes experimental results\n collect_summary Prints names of weights and layers activations to use with other collect commands\n collect_weights Collects tensors containing a pretrained model weights\n collect_activations Collects tensors for each datapoint using a pretrained model\n datasets Downloads and lists Ludwig-ready datasets\n export_torchscript Exports Ludwig models to Torchscript\n export_triton Exports Ludwig models to Triton\n export_mlflow Exports Ludwig models to MLflow\n export_schema Exports the Ludwig config JSON schema\n preprocess Preprocess data and saves it into HDF5 and JSON format\n synthesize_dataset Creates synthetic data for testing purposes\n init_config Initialize a user config from a dataset and targets\n render_config Renders the fully populated config with all defaults set\n check_install Runs a quick training run on synthetic data to verify installation status\n upload Push trained model artifacts to a registry (e.g., Predibase, HuggingFace Hub)\n\"\"\",\n )\n parser.add_argument(\"command\", help=\"Subcommand to run\")\n # parse_args defaults to [1:] for args, but you need to\n # exclude the rest of the args too, or validation will fail\n args = parser.parse_args(sys.argv[1:2])\n if not hasattr(self, args.command):\n print(\"Unrecognized command\")\n parser.print_help()\n exit(1)\n # use dispatch pattern to invoke method with same name\n getattr(self, args.command)()\n\n def train(self):\n from ludwig import train\n\n train.cli(sys.argv[2:])\n\n def predict(self):\n from ludwig import predict\n\n predict.cli(sys.argv[2:])\n\n def evaluate(self):\n from ludwig import evaluate\n\n evaluate.cli(sys.argv[2:])\n\n def forecast(self):\n from ludwig import forecast\n\n forecast.cli(sys.argv[2:])\n\n def experiment(self):\n from ludwig import experiment\n\n experiment.cli(sys.argv[2:])\n\n def hyperopt(self):\n from ludwig import hyperopt_cli\n\n hyperopt_cli.cli(sys.argv[2:])\n\n def benchmark(self):\n from ludwig.benchmarking import benchmark\n\n benchmark.cli(sys.argv[2:])\n\n def serve(self):\n from ludwig import serve\n\n serve.cli(sys.argv[2:])\n\n def visualize(self):\n from ludwig import visualize\n\n visualize.cli(sys.argv[2:])\n\n def collect_summary(self):\n from ludwig import collect\n\n collect.cli_collect_summary(sys.argv[2:])\n\n def collect_weights(self):\n from ludwig import collect\n\n collect.cli_collect_weights(sys.argv[2:])\n\n def collect_activations(self):\n from ludwig import collect\n\n collect.cli_collect_activations(sys.argv[2:])\n\n def export_torchscript(self):\n from ludwig import export\n\n export.cli_export_torchscript(sys.argv[2:])\n\n def export_triton(self):\n from ludwig import export\n\n export.cli_export_triton(sys.argv[2:])\n\n def export_mlflow(self):\n from ludwig import export\n\n export.cli_export_mlflow(sys.argv[2:])\n\n def export_schema(self):\n from ludwig.schema.export_schema import main as export_schema_main\n\n export_schema_main(sys.argv[2:])\n\n def preprocess(self):\n from ludwig import preprocess\n\n preprocess.cli(sys.argv[2:])\n\n def synthesize_dataset(self):\n from ludwig.data import dataset_synthesizer\n\n dataset_synthesizer.cli(sys.argv[2:])\n\n def init_config(self):\n from ludwig import automl\n\n automl.cli_init_config(sys.argv[2:])\n\n def render_config(self):\n from ludwig.utils import defaults\n\n defaults.cli_render_config(sys.argv[2:])\n\n def check_install(self):\n from ludwig import check\n\n check.cli(sys.argv[2:])\n\n def datasets(self):\n from ludwig import datasets\n\n datasets.cli(sys.argv[2:])\n\n def upload(self):\n from ludwig import upload\n\n upload.cli(sys.argv[2:])\n\n\ndef main():\n ludwig.contrib.preload(sys.argv)\n CLI()\n\n\nif __name__ == \"__main__\":\n main()\n"
},
{
"path": "ludwig/collect.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport argparse\nimport importlib\nimport logging\nimport os\nimport sys\n\nimport numpy as np\nimport torch\nimport torchinfo\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.backend import ALL_BACKENDS, Backend\nfrom ludwig.callbacks import Callback\nfrom ludwig.constants import FULL, TEST, TRAINING, VALIDATION\nfrom ludwig.contrib import add_contrib_callback_args\nfrom ludwig.globals import LUDWIG_VERSION\nfrom ludwig.utils.print_utils import get_logging_level_registry, print_boxed, print_ludwig\nfrom ludwig.utils.strings_utils import make_safe_filename\n\nlogger = logging.getLogger(__name__)\n\n\ndef collect_activations(\n model_path: str,\n layers: list[str],\n dataset: str,\n data_format: str = None,\n split: str = FULL,\n batch_size: int = 128,\n output_directory: str = \"results\",\n gpus: list[str] = None,\n gpu_memory_limit: float | None = None,\n allow_parallel_threads: bool = True,\n callbacks: list[Callback] = None,\n backend: Backend | str = None,\n **kwargs,\n) -> list[str]:\n \"\"\"Uses the pretrained model to collect the tensors corresponding to a datapoint in the dataset. Saves the\n tensors to the experiment directory.\n\n # Inputs\n\n :param model_path: (str) filepath to pre-trained model.\n :param layers: (List[str]) list of strings for layer names in the model\n to collect activations.\n :param dataset: (str) source\n containing the data to make predictions.\n :param data_format: (str, default: `None`) format to interpret data\n sources. Will be inferred automatically if not specified. Valid\n formats are `'auto'`, `'csv'`, `'excel'`, `'feather'`,\n `'fwf'`, `'hdf5'` (cache file produced during previous training),\n `'html'` (file containing a single HTML `
`), `'json'`, `'jsonl'`,\n `'parquet'`, `'pickle'` (pickled Pandas DataFrame), `'sas'`, `'spss'`,\n `'stata'`, `'tsv'`.\n :param split: (str, default: `full`) split on which\n to perform predictions. Valid values are `'training'`, `'validation'`,\n `'test'` and `'full'`.\n :param batch_size: (int, default `128`) size of batches for processing.\n :param output_directory: (str, default: `'results'`) the directory that\n will contain the training statistics, TensorBoard logs, the saved\n model and the training progress files.\n :param gpus: (list, default: `None`) list of GPUs that are available\n for training.\n :param gpu_memory_limit: (float: default: `None`) maximum memory fraction\n [0, 1] allowed to allocate per GPU device.\n :param allow_parallel_threads: (bool, default: `True`) allow PyTorch\n to use multithreading parallelism to improve performance at\n the cost of determinism.\n :param callbacks: (list, default: `None`) a list of\n `ludwig.callbacks.Callback` objects that provide hooks into the\n Ludwig pipeline.\n :param backend: (Union[Backend, str]) `Backend` or string name\n of backend to use to execute preprocessing / training steps.\n\n # Return\n\n :return: (List[str]) list of filepath to `*.npy` files containing\n the activations.\n \"\"\"\n logger.info(f\"Dataset path: {dataset}\")\n logger.info(f\"Model path: {model_path}\")\n logger.info(f\"Output path: {output_directory}\")\n logger.info(\"\\n\")\n\n model = LudwigModel.load(\n model_path,\n gpus=gpus,\n gpu_memory_limit=gpu_memory_limit,\n allow_parallel_threads=allow_parallel_threads,\n callbacks=callbacks,\n backend=backend,\n )\n\n # collect activations\n print_boxed(\"COLLECT ACTIVATIONS\")\n collected_tensors = model.collect_activations(\n layers, dataset, data_format=data_format, split=split, batch_size=batch_size\n )\n\n # saving\n os.makedirs(output_directory, exist_ok=True)\n saved_filenames = save_tensors(collected_tensors, output_directory)\n\n logger.info(f\"Saved to: {output_directory}\")\n return saved_filenames\n\n\ndef collect_weights(model_path: str, tensors: list[str], output_directory: str = \"results\", **kwargs) -> list[str]:\n \"\"\"Loads a pretrained model and collects weights.\n\n # Inputs\n :param model_path: (str) filepath to pre-trained model.\n :param tensors: (list, default: `None`) List of tensor names to collect\n weights\n :param output_directory: (str, default: `'results'`) the directory where\n collected weights will be stored.\n\n # Return\n\n :return: (List[str]) list of filepath to `*.npy` files containing\n the weights.\n \"\"\"\n logger.info(f\"Model path: {model_path}\")\n logger.info(f\"Output path: {output_directory}\")\n logger.info(\"\\n\")\n\n model = LudwigModel.load(model_path)\n\n # collect weights\n print_boxed(\"COLLECT WEIGHTS\")\n collected_tensors = model.collect_weights(tensors)\n\n # saving\n os.makedirs(output_directory, exist_ok=True)\n saved_filenames = save_tensors(collected_tensors, output_directory)\n\n logger.info(f\"Saved to: {output_directory}\")\n return saved_filenames\n\n\ndef save_tensors(collected_tensors, output_directory):\n filenames = []\n for tensor_name, tensor_value in collected_tensors:\n np_filename = os.path.join(output_directory, make_safe_filename(tensor_name) + \".npy\")\n if isinstance(tensor_value, torch.Tensor):\n # Skip non-tensor collected artifacts, e.g. used_tokens.\n np.save(np_filename, tensor_value.detach().cpu().numpy())\n filenames.append(np_filename)\n return filenames\n\n\ndef print_model_summary(model_path: str, **kwargs) -> None:\n \"\"\"Loads a pretrained model and prints names of weights and layers activations.\n\n # Inputs\n :param model_path: (str) filepath to pre-trained model.\n\n # Return\n :return: (`None`)\n \"\"\"\n model = LudwigModel.load(model_path)\n # Move model to CPU for torchinfo summary to avoid device mismatch issues.\n model.model.cpu()\n logger.info(torchinfo.summary(model.model, input_data=[model.model.get_model_inputs()], depth=20))\n\n logger.info(\"\\nModules:\\n\")\n for name, _ in model.model.named_children():\n logger.info(name)\n\n logger.info(\"\\nParameters:\\n\")\n for name, _ in model.model.named_parameters():\n logger.info(name)\n\n\ndef pretrained_summary(pretrained_model: str, **kwargs) -> None:\n \"\"\"Loads a pretrained model from Huggingface or Torchvision models and prints names of layers.\n\n # Inputs\n :param pretrained_model: (str) name of model to load (case sensitive).\n\n # Return\n :return: (`None`)\n \"\"\"\n from transformers import AutoConfig, AutoModel\n\n model = None\n # get access token if available\n token = os.getenv(\"HUGGING_FACE_HUB_TOKEN\")\n if token is None:\n logger.info(\"No token provided. Continuing loading without token access.\")\n elif not token:\n raise ValueError(\"Invalid token provided. Exiting.\")\n else:\n logger.info(\"Valid token provided. Proceeding with token access.\")\n\n # Try to load from transformers/HF\n # TODO -> Fix OOM on large models e.g. llama 3 8B\n try:\n config = AutoConfig.from_pretrained(pretrained_model, token=token, low_cpu_mem_usage=True)\n model = AutoModel.from_config(config=config)\n logger.info(f\"Loaded {pretrained_model} from Hugging Face Transformers.\")\n except Exception as e:\n logger.error(f\"Failed to load {pretrained_model} from Hugging Face Transformers: {e}\")\n\n # Try and load from torchvision-models\n if model is None:\n try:\n module = importlib.import_module(\"torchvision.models\")\n model = getattr(module, pretrained_model)(weights=None)\n except AttributeError:\n logger.error(f\"{pretrained_model} is not a valid torchvision model.\")\n\n if model:\n for name, _ in model.named_parameters():\n logger.info(name)\n else:\n logger.error(f\"Unable to load the model {pretrained_model} from any known source.\")\n\n\ndef cli_collect_activations(sys_argv):\n \"\"\"Command Line Interface to communicate with the collection of tensors and there are several options that can\n specified when calling this function:\n\n --data_csv: Filepath for the input csv\n --data_hdf5: Filepath for the input hdf5 file, if there is a csv file, this\n is not read\n --d: Refers to the dataset type of the file being read, by default is\n *generic*\n --s: Refers to the split of the data, can be one of: train, test,\n validation, full\n --m: Input model that is necessary to collect to the tensors, this is a\n required *option*\n --t: Tensors to collect\n --od: Output directory of the model, defaults to results\n --bs: Batch size\n --g: Number of gpus that are to be used\n --gf: Fraction of each GPUs memory to use.\n --v: Verbose: Defines the logging level that the user will be exposed to\n \"\"\"\n parser = argparse.ArgumentParser(\n description=\"This script loads a pretrained model and uses it collect \"\n \"tensors for each datapoint in the dataset.\",\n prog=\"ludwig collect_activations\",\n usage=\"%(prog)s [options]\",\n )\n\n # ---------------\n # Data parameters\n # ---------------\n parser.add_argument(\"--dataset\", help=\"input data file path\", required=True)\n parser.add_argument(\n \"--data_format\",\n help=\"format of the input data\",\n default=\"auto\",\n choices=[\n \"auto\",\n \"csv\",\n \"excel\",\n \"feather\",\n \"fwf\",\n \"hdf5\",\n \"html\" \"tables\",\n \"json\",\n \"jsonl\",\n \"parquet\",\n \"pickle\",\n \"sas\",\n \"spss\",\n \"stata\",\n \"tsv\",\n ],\n )\n parser.add_argument(\n \"-s\",\n \"--split\",\n default=FULL,\n choices=[TRAINING, VALIDATION, TEST, FULL],\n help=\"the split to obtain the model activations from\",\n )\n\n # ----------------\n # Model parameters\n # ----------------\n parser.add_argument(\"-m\", \"--model_path\", help=\"model to load\", required=True)\n parser.add_argument(\"-lyr\", \"--layers\", help=\"tensors to collect\", nargs=\"+\", required=True)\n\n # -------------------------\n # Output results parameters\n # -------------------------\n parser.add_argument(\n \"-od\", \"--output_directory\", type=str, default=\"results\", help=\"directory that contains the results\"\n )\n\n # ------------------\n # Generic parameters\n # ------------------\n parser.add_argument(\"-bs\", \"--batch_size\", type=int, default=128, help=\"size of batches\")\n\n # ------------------\n # Runtime parameters\n # ------------------\n parser.add_argument(\"-g\", \"--gpus\", type=int, default=0, help=\"list of gpu to use\")\n parser.add_argument(\n \"-gml\",\n \"--gpu_memory_limit\",\n type=float,\n default=None,\n help=\"maximum memory fraction [0, 1] allowed to allocate per GPU device\",\n )\n parser.add_argument(\n \"-dpt\",\n \"--disable_parallel_threads\",\n action=\"store_false\",\n dest=\"allow_parallel_threads\",\n help=\"disable PyTorch from using multithreading for reproducibility\",\n )\n parser.add_argument(\n \"-b\",\n \"--backend\",\n help=\"specifies backend to use for parallel / distributed execution, \" \"defaults to local execution\",\n choices=ALL_BACKENDS,\n )\n parser.add_argument(\n \"-l\",\n \"--logging_level\",\n default=\"info\",\n help=\"the level of logging to use\",\n choices=[\"critical\", \"error\", \"warning\", \"info\", \"debug\", \"notset\"],\n )\n\n add_contrib_callback_args(parser)\n args = parser.parse_args(sys_argv)\n\n args.callbacks = args.callbacks or []\n for callback in args.callbacks:\n callback.on_cmdline(\"collect_activations\", *sys_argv)\n\n args.logging_level = get_logging_level_registry()[args.logging_level]\n logging.getLogger(\"ludwig\").setLevel(args.logging_level)\n global logger\n logger = logging.getLogger(\"ludwig.collect\")\n\n print_ludwig(\"Collect Activations\", LUDWIG_VERSION)\n\n collect_activations(**vars(args))\n\n\ndef cli_collect_weights(sys_argv):\n \"\"\"Command Line Interface to collecting the weights for the model.\n\n --m: Input model that is necessary to collect to the tensors, this is a\n required *option*\n --t: Tensors to collect\n --od: Output directory of the model, defaults to results\n --v: Verbose: Defines the logging level that the user will be exposed to\n \"\"\"\n parser = argparse.ArgumentParser(\n description=\"This script loads a pretrained model \" \"and uses it collect weights.\",\n prog=\"ludwig collect_weights\",\n usage=\"%(prog)s [options]\",\n )\n\n # ----------------\n # Model parameters\n # ----------------\n parser.add_argument(\"-m\", \"--model_path\", help=\"model to load\", required=True)\n parser.add_argument(\"-t\", \"--tensors\", help=\"tensors to collect\", nargs=\"+\", required=True)\n\n # -------------------------\n # Output results parameters\n # -------------------------\n parser.add_argument(\n \"-od\", \"--output_directory\", type=str, default=\"results\", help=\"directory that contains the results\"\n )\n\n # ------------------\n # Runtime parameters\n # ------------------\n parser.add_argument(\n \"-l\",\n \"--logging_level\",\n default=\"info\",\n help=\"the level of logging to use\",\n choices=[\"critical\", \"error\", \"warning\", \"info\", \"debug\", \"notset\"],\n )\n\n add_contrib_callback_args(parser)\n args = parser.parse_args(sys_argv)\n\n args.callbacks = args.callbacks or []\n for callback in args.callbacks:\n callback.on_cmdline(\"collect_weights\", *sys_argv)\n\n args.logging_level = get_logging_level_registry()[args.logging_level]\n logging.getLogger(\"ludwig\").setLevel(args.logging_level)\n global logger\n logger = logging.getLogger(\"ludwig.collect\")\n\n print_ludwig(\"Collect Weights\", LUDWIG_VERSION)\n\n collect_weights(**vars(args))\n\n\ndef cli_collect_summary(sys_argv):\n \"\"\"Command Line Interface to collecting a summary of the model layers and weights.\n\n --m: Input model that is necessary to collect to the tensors\n --pm: Model name in order to fetch from Huggingface or Torchvision\n --v: Verbose: Defines the logging level that the user will be exposed to\n \"\"\"\n parser = argparse.ArgumentParser(\n description=\"This script loads a pretrained model \"\n \"and prints names of weights and layers activations \"\n \"to use with other collect commands\",\n prog=\"ludwig collect_summary\",\n usage=\"%(prog)s [options]\",\n )\n\n # ----------------\n # Model parameters\n # ----------------\n parser.add_argument(\"-m\", \"--model_path\", help=\"model to load\", required=False)\n parser.add_argument(\n \"-pm\", \"--pretrained_model\", help=\"pretrained model to summarize (torchvision and huggingface)\", required=False\n )\n\n # ------------------\n # Runtime parameters\n # ------------------\n parser.add_argument(\n \"-l\",\n \"--logging_level\",\n default=\"info\",\n help=\"the level of logging to use\",\n choices=[\"critical\", \"error\", \"warning\", \"info\", \"debug\", \"notset\"],\n )\n\n add_contrib_callback_args(parser)\n args = parser.parse_args(sys_argv)\n\n args.callbacks = args.callbacks or []\n for callback in args.callbacks:\n callback.on_cmdline(\"collect_summary\", *sys_argv)\n\n args.logging_level = get_logging_level_registry()[args.logging_level]\n logging.getLogger(\"ludwig\").setLevel(args.logging_level)\n global logger\n logger = logging.getLogger(\"ludwig.collect\")\n\n print_ludwig(\"Collect Summary\", LUDWIG_VERSION)\n\n if args.model_path:\n print_model_summary(**vars(args))\n elif args.pretrained_model and not args.model_path:\n pretrained_summary(**vars(args))\n\n\nif __name__ == \"__main__\":\n if len(sys.argv) > 1:\n if sys.argv[1] == \"activations\":\n cli_collect_activations(sys.argv[2:])\n elif sys.argv[1] == \"weights\":\n cli_collect_weights(sys.argv[2:])\n elif sys.argv[1] == \"names\":\n cli_collect_summary(sys.argv[2:])\n else:\n print(\"Unrecognized command\")\n else:\n print(\"Unrecognized command\")\n"
},
{
"path": "ludwig/combiners/__init__.py",
"content": ""
},
{
"path": "ludwig/combiners/combiners.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nfrom abc import ABC\nfrom dataclasses import dataclass\nfrom functools import lru_cache\n\nimport torch\nfrom torch.nn import Linear, ModuleList\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import BINARY, ENCODER_OUTPUT, NUMBER\nfrom ludwig.encoders.registry import get_sequence_encoder_registry\nfrom ludwig.features.base_feature import InputFeature\nfrom ludwig.modules.attention_modules import TransformerStack\nfrom ludwig.modules.embedding_modules import Embed\nfrom ludwig.modules.fully_connected_modules import FCStack\nfrom ludwig.modules.reduction_modules import SequenceReducer\nfrom ludwig.modules.tabnet_modules import TabNet\nfrom ludwig.schema.combiners.base import BaseCombinerConfig\nfrom ludwig.schema.combiners.comparator import ComparatorCombinerConfig\nfrom ludwig.schema.combiners.concat import ConcatCombinerConfig\nfrom ludwig.schema.combiners.project_aggregate import ProjectAggregateCombinerConfig\nfrom ludwig.schema.combiners.sequence import SequenceCombinerConfig\nfrom ludwig.schema.combiners.sequence_concat import SequenceConcatCombinerConfig\nfrom ludwig.schema.combiners.tab_transformer import TabTransformerCombinerConfig\nfrom ludwig.schema.combiners.tabnet import TabNetCombinerConfig\nfrom ludwig.schema.combiners.transformer import TransformerCombinerConfig\nfrom ludwig.utils.misc_utils import get_from_registry\nfrom ludwig.utils.registry import Registry\nfrom ludwig.utils.torch_utils import LudwigModule, sequence_length_3D\nfrom ludwig.utils.torch_utils import sequence_mask as torch_sequence_mask\n\nlogger = logging.getLogger(__name__)\n\n\n@dataclass\nclass Handle:\n \"\"\"This class provides an opaque handle to the input features, preventing them from being registered as state.\n\n This is important because we already reference the `input_features` as an attribute of ECD, so we don't need it to\n appear twice in the state_dict. Furthermore, DeepSpeed will get terribly confused if have the input features set as\n an attribute of the combiner, and lead to shape mismatch errors when we go to load a saved checkpoint.\n \"\"\"\n\n input_features: dict[str, \"InputFeature\"]\n\n\n@DeveloperAPI\nclass Combiner(LudwigModule, ABC):\n \"\"\"Base class for combiners, which implements common properties.\n\n Subclasses will usually override: __init__() to set properties and allocate resources. Should call\n super().__init__(input_features). forward() performs the forward pass given a dictionary of encoder\n outputs. get_schema_cls() must returns the class of the corresponding schema for the combiner type.\n \"\"\"\n\n def __init__(self, input_features: dict[str, \"InputFeature\"]):\n super().__init__()\n self.handle = Handle(input_features)\n\n @property\n def concatenated_shape(self) -> torch.Size:\n # compute the size of the last dimension for the incoming encoder outputs\n # this is required to setup the fully connected layer\n shapes = [\n torch.prod(torch.Tensor([*self.handle.input_features.get(k).output_shape]))\n for k in self.handle.input_features\n ]\n return torch.Size([torch.sum(torch.Tensor(shapes)).type(torch.int32)])\n\n @property\n def input_shape(self) -> dict:\n # input to combiner is a dictionary of the input features encoder\n # outputs, this property returns dictionary of output shapes for each\n # input feature's encoder output shapes.\n return {k: self.handle.input_features.get(k).output_shape for k in self.handle.input_features}\n\n @property\n @lru_cache(maxsize=1)\n def output_shape(self) -> torch.Size:\n pseudo_input = {}\n for k in self.handle.input_features:\n pseudo_input[k] = {\n ENCODER_OUTPUT: torch.rand(\n 2, *self.handle.input_features.get(k).output_shape, dtype=self.input_dtype, device=self.device\n )\n }\n output_tensor = self.forward(pseudo_input)\n return output_tensor[\"combiner_output\"].size()[1:]\n\n\ncombiner_impl_registry = Registry[type[Combiner]]()\n\n\ndef register_combiner(config_cls: type[BaseCombinerConfig]):\n def wrap(cls: type[Combiner]):\n combiner_impl_registry[config_cls] = cls\n return cls\n\n return wrap\n\n\ndef create_combiner(config: BaseCombinerConfig, **kwargs) -> Combiner:\n return combiner_impl_registry[type(config)](config=config, **kwargs)\n\n\n@register_combiner(ConcatCombinerConfig)\nclass ConcatCombiner(Combiner):\n def __init__(self, input_features: dict[str, \"InputFeature\"] = None, config: ConcatCombinerConfig = None, **kwargs):\n super().__init__(input_features)\n self.name = \"ConcatCombiner\"\n logger.debug(f\" {self.name}\")\n\n self.flatten_inputs = config.flatten_inputs\n self.fc_stack = None\n\n # todo future: this may be redundant, check\n fc_layers = config.fc_layers\n if fc_layers is None:\n fc_layers = []\n for i in range(config.num_fc_layers):\n fc_layers.append({\"output_size\": config.output_size})\n self.fc_layers = fc_layers\n\n logger.debug(\" FCStack\")\n self.fc_stack = FCStack(\n first_layer_input_size=self.concatenated_shape[-1],\n layers=config.fc_layers,\n num_layers=config.num_fc_layers,\n default_output_size=config.output_size,\n default_use_bias=config.use_bias,\n default_weights_initializer=config.weights_initializer,\n default_bias_initializer=config.bias_initializer,\n default_norm=config.norm,\n default_norm_params=config.norm_params,\n default_activation=config.activation,\n default_dropout=config.dropout,\n residual=config.residual,\n )\n\n if input_features and len(input_features) == 1 and self.fc_layers is None:\n self.supports_masking = True\n\n def forward(self, inputs: dict) -> dict: # encoder outputs\n encoder_outputs = [inputs[k][ENCODER_OUTPUT] for k in inputs]\n\n # ================ Flatten ================\n if self.flatten_inputs:\n batch_size = encoder_outputs[0].shape[0]\n encoder_outputs = [torch.reshape(eo, [batch_size, -1]) for eo in encoder_outputs]\n\n # ================ Concat ================\n if len(encoder_outputs) > 1:\n hidden = torch.cat(encoder_outputs, 1)\n else:\n hidden = list(encoder_outputs)[0]\n\n # ================ Fully Connected ================\n hidden = self.fc_stack(hidden)\n\n return_data = {\"combiner_output\": hidden}\n\n if len(inputs) == 1:\n # Workaround for including additional tensors from output of input encoders for\n # potential use in decoders, e.g. LSTM state for seq2seq.\n # TODO(Justin): Think about how to make this communication work for multi-sequence\n # features. Other combiners.\n for key, value in [d for d in inputs.values()][0].items():\n if key != ENCODER_OUTPUT:\n return_data[key] = value\n\n return return_data\n\n\n@register_combiner(SequenceConcatCombinerConfig)\nclass SequenceConcatCombiner(Combiner):\n def __init__(\n self, input_features: dict[str, \"InputFeature\"], config: SequenceConcatCombinerConfig = None, **kwargs\n ):\n super().__init__(input_features)\n self.name = \"SequenceConcatCombiner\"\n logger.debug(f\" {self.name}\")\n\n self.reduce_output = config.reduce_output\n self.reduce_sequence = SequenceReducer(\n reduce_mode=config.reduce_output,\n max_sequence_length=self.concatenated_shape[0],\n encoding_size=self.concatenated_shape[1],\n )\n if self.reduce_output is None:\n self.supports_masking = True\n self.main_sequence_feature = config.main_sequence_feature\n\n @property\n def concatenated_shape(self) -> torch.Size:\n # computes the effective shape of the input tensor after combining\n # all the encoder outputs\n # determine max sequence length by finding the first sequence tensor\n # assume all the sequences are of the same size, if not true\n # this will be caught during processing\n seq_size = None\n for k in self.handle.input_features:\n # dim-2 output_shape implies a sequence [seq_size, hidden]\n if len(self.handle.input_features.get(k).output_shape) == 2:\n seq_size = self.handle.input_features.get(k).output_shape[0]\n break\n\n # collect the size of the last dimension for all input feature\n # encoder outputs\n shapes = [\n self.handle.input_features.get(k).output_shape[-1] for k in self.handle.input_features\n ] # output shape not input shape\n return torch.Size([seq_size, sum(shapes)])\n\n def forward(self, inputs: dict) -> dict: # encoder outputs\n if self.main_sequence_feature is None or self.main_sequence_feature not in inputs:\n for if_name, if_outputs in inputs.items():\n # todo: when https://github.com/ludwig-ai/ludwig/issues/810 is closed\n # convert following test from using shape to use explicit\n # if_outputs[TYPE] values for sequence features\n if len(if_outputs[ENCODER_OUTPUT].shape) == 3:\n self.main_sequence_feature = if_name\n break\n\n if self.main_sequence_feature is None:\n raise Exception(\"No sequence feature available for sequence combiner\")\n\n main_sequence_feature_encoding = inputs[self.main_sequence_feature]\n\n representation = main_sequence_feature_encoding[ENCODER_OUTPUT]\n representations = [representation]\n\n sequence_max_length = representation.shape[1]\n sequence_length = sequence_length_3D(representation)\n\n # ================ Concat ================\n for if_name, if_outputs in inputs.items():\n if if_name != self.main_sequence_feature:\n if_representation = if_outputs[ENCODER_OUTPUT]\n if len(if_representation.shape) == 3:\n # The following check makes sense when\n # both representations have a specified\n # sequence length dimension. If they do not,\n # then this check is simply checking if None == None\n # and will not catch discrepancies in the different\n # feature length dimension. Those errors will show up\n # at training time. Possible solutions to this is\n # to enforce a length second dimension in\n # sequential feature placeholders, but that\n # does not work with BucketedBatcher that requires\n # the second dimension to be undefined in order to be\n # able to trim the data points and speed up computation.\n # So for now we are keeping things like this, make sure\n # to write in the documentation that training time\n # dimensions mismatch may occur if the sequential\n # features have different lengths for some data points.\n if if_representation.shape[1] != representation.shape[1]:\n raise ValueError(\n \"The sequence length of the input feature {} \"\n \"is {} and is different from the sequence \"\n \"length of the main sequence feature {} which \"\n \"is {}.\\n Shape of {}: {}, shape of {}: {}.\\n\"\n \"Sequence lengths of all sequential features \"\n \"must be the same in order to be concatenated \"\n \"by the sequence concat combiner. \"\n \"Try to impose the same max sequence length \"\n \"as a preprocessing parameter to both features \"\n \"or to reduce the output of {}.\".format(\n if_name,\n if_representation.shape[1],\n self.main_sequence_feature,\n representation.shape[1],\n if_name,\n if_representation.shape,\n if_name,\n representation.shape,\n if_name,\n )\n )\n # this assumes all sequence representations have the\n # same sequence length, 2nd dimension\n representations.append(if_representation)\n\n elif len(if_representation.shape) == 2:\n multipliers = (1, sequence_max_length, 1)\n tiled_representation = torch.tile(torch.unsqueeze(if_representation, 1), multipliers)\n representations.append(tiled_representation)\n\n else:\n raise ValueError(\n \"The representation of {} has rank {} and cannot be\"\n \" concatenated by a sequence concat combiner. \"\n \"Only rank 2 and rank 3 tensors are supported.\".format(if_name, len(if_representation.shape))\n )\n\n hidden = torch.cat(representations, 2)\n logger.debug(f\" concat_hidden: {hidden}\")\n\n # ================ Mask ================\n sequence_mask = torch_sequence_mask(sequence_length, sequence_max_length)\n hidden = torch.multiply(hidden, torch.unsqueeze(sequence_mask, -1).type(torch.float32))\n\n # ================ Reduce ================\n hidden = self.reduce_sequence(hidden)\n\n return_data = {\"combiner_output\": hidden}\n\n if len(inputs) == 1:\n for key, value in [d for d in inputs.values()][0].items():\n if key != ENCODER_OUTPUT:\n return_data[key] = value\n\n return return_data\n\n\n@register_combiner(SequenceCombinerConfig)\nclass SequenceCombiner(Combiner):\n def __init__(self, input_features: dict[str, \"InputFeature\"], config: SequenceCombinerConfig = None, **kwargs):\n super().__init__(input_features)\n self.name = \"SequenceCombiner\"\n logger.debug(f\" {self.name}\")\n\n self.combiner = SequenceConcatCombiner(\n input_features,\n config=SequenceConcatCombinerConfig(reduce_output=None, main_sequence_feature=config.main_sequence_feature),\n )\n\n logger.debug(\n f\"combiner input shape {self.combiner.concatenated_shape}, \" f\"output shape {self.combiner.output_shape}\"\n )\n\n self.encoder_obj = get_from_registry(config.encoder.type, get_sequence_encoder_registry())(\n should_embed=False,\n reduce_output=config.reduce_output,\n embedding_size=self.combiner.output_shape[1],\n max_sequence_length=self.combiner.output_shape[0],\n **kwargs,\n )\n\n if hasattr(self.encoder_obj, \"supports_masking\") and self.encoder_obj.supports_masking:\n self.supports_masking = True\n\n @property\n def concatenated_shape(self) -> torch.Size:\n # computes the effective shape of the input tensor after combining\n # all the encoder outputs\n # determine max sequence length by finding the first sequence tensor\n # assume all the sequences are of the same size, if not true\n # this will be caught during processing\n seq_size = None\n for k in self.handle.input_features:\n # dim-2 output_shape implies a sequence [seq_size, hidden]\n if len(self.handle.input_features.get(k).output_shape) == 2:\n seq_size = self.handle.input_features.get(k).output_shape[0]\n break\n\n # collect the size of the last dimension for all input feature\n # encoder outputs\n shapes = [\n self.handle.input_features.get(k).output_shape[-1] for k in self.handle.input_features\n ] # output shape not input shape\n return torch.Size([seq_size, sum(shapes)])\n\n def forward(self, inputs: dict) -> dict: # encoder outputs\n # ================ Concat ================\n hidden = self.combiner(inputs)\n\n # ================ Sequence encoding ================\n hidden = self.encoder_obj(hidden[\"combiner_output\"])\n\n return_data = {\"combiner_output\": hidden[ENCODER_OUTPUT]}\n for key, value in hidden.items():\n if key != ENCODER_OUTPUT:\n return_data[key] = value\n\n return return_data\n\n\n@register_combiner(TabNetCombinerConfig)\nclass TabNetCombiner(Combiner):\n def __init__(\n self, input_features: dict[str, \"InputFeature\"], config: TabNetCombinerConfig = None, **kwargs\n ) -> None:\n super().__init__(input_features)\n self.name = \"TabNetCombiner\"\n logger.debug(f\" {self.name}\")\n\n self.tabnet = TabNet(\n self.concatenated_shape[-1],\n config.size,\n config.output_size,\n num_steps=config.num_steps,\n num_total_blocks=config.num_total_blocks,\n num_shared_blocks=config.num_shared_blocks,\n relaxation_factor=config.relaxation_factor,\n bn_epsilon=config.bn_epsilon,\n bn_momentum=config.bn_momentum,\n bn_virtual_bs=config.bn_virtual_bs,\n sparsity=config.sparsity,\n entmax_mode=config.entmax_mode,\n entmax_alpha=config.entmax_alpha,\n )\n\n if config.dropout > 0:\n self.dropout = torch.nn.Dropout(config.dropout)\n else:\n self.dropout = None\n\n @property\n def concatenated_shape(self) -> torch.Size:\n # compute the size of the last dimension for the incoming encoder outputs\n # this is required to setup\n shapes = [\n torch.prod(torch.Tensor([*self.handle.input_features.get(k).output_shape]))\n for k in self.handle.input_features\n ]\n return torch.Size([torch.sum(torch.Tensor(shapes)).type(torch.int32)])\n\n def forward(\n self,\n inputs: torch.Tensor, # encoder outputs\n ) -> dict:\n encoder_outputs = [inputs[k][ENCODER_OUTPUT] for k in inputs]\n\n # ================ Flatten ================\n batch_size = encoder_outputs[0].shape[0]\n encoder_outputs = [torch.reshape(eo, [batch_size, -1]) for eo in encoder_outputs]\n\n # ================ Concat ================\n if len(encoder_outputs) > 1:\n hidden = torch.cat(encoder_outputs, 1)\n else:\n hidden = list(encoder_outputs)[0]\n\n # ================ TabNet ================\n hidden, aggregated_mask, masks = self.tabnet(hidden)\n if self.dropout:\n hidden = self.dropout(hidden)\n\n return_data = {\n \"combiner_output\": hidden,\n \"aggregated_attention_masks\": aggregated_mask,\n \"attention_masks\": masks,\n }\n\n if len(inputs) == 1:\n for key, value in [d for d in inputs.values()][0].items():\n if key != ENCODER_OUTPUT:\n return_data[key] = value\n\n return return_data\n\n @property\n def output_shape(self) -> torch.Size:\n return self.tabnet.output_shape\n\n\n@register_combiner(TransformerCombinerConfig)\nclass TransformerCombiner(Combiner):\n def __init__(\n self, input_features: dict[str, \"InputFeature\"] = None, config: TransformerCombinerConfig = None, **kwargs\n ):\n super().__init__(input_features)\n self.name = \"TransformerCombiner\"\n logger.debug(f\" {self.name}\")\n\n self.reduce_output = config.reduce_output\n self.reduce_sequence = SequenceReducer(\n reduce_mode=config.reduce_output,\n max_sequence_length=len(input_features),\n encoding_size=config.hidden_size,\n )\n if self.reduce_output is None:\n self.supports_masking = True\n\n # max sequence length for Transformer layer is number of input features\n self.max_sequence_length = len(input_features)\n\n logger.debug(\" Projectors\")\n self.projectors = ModuleList(\n # regardless of rank-2 or rank-3 input, torch.prod() calculates size\n # after flattening the encoder output tensor\n [\n Linear(\n torch.prod(torch.Tensor([*input_features.get(inp).output_shape])).type(torch.int32),\n config.hidden_size,\n )\n for inp in input_features\n ]\n )\n\n logger.debug(\" TransformerStack\")\n self.transformer_stack = TransformerStack(\n input_size=config.hidden_size,\n max_sequence_length=self.max_sequence_length,\n hidden_size=config.hidden_size,\n num_heads=config.num_heads,\n output_size=config.transformer_output_size,\n num_layers=config.num_layers,\n dropout=config.dropout,\n )\n\n if self.reduce_output is not None:\n logger.debug(\" FCStack\")\n self.fc_stack = FCStack(\n self.transformer_stack.output_shape[-1],\n layers=config.fc_layers,\n num_layers=config.num_fc_layers,\n default_output_size=config.output_size,\n default_use_bias=config.use_bias,\n default_weights_initializer=config.weights_initializer,\n default_bias_initializer=config.bias_initializer,\n default_norm=config.norm,\n default_norm_params=config.norm_params,\n default_activation=config.fc_activation,\n default_dropout=config.fc_dropout,\n fc_residual=config.fc_residual,\n )\n\n def forward(\n self,\n inputs, # encoder outputs\n ) -> dict:\n encoder_outputs = [inputs[k][ENCODER_OUTPUT] for k in inputs]\n\n # ================ Flatten ================\n batch_size = encoder_outputs[0].shape[0]\n encoder_outputs = [torch.reshape(eo, [batch_size, -1]) for eo in encoder_outputs]\n\n # ================ Project & Concat ================\n projected = [self.projectors[i](eo) for i, eo in enumerate(encoder_outputs)]\n hidden = torch.stack(projected) # shape [num_eo, bs, h]\n hidden = torch.permute(hidden, (1, 0, 2)) # shape [bs, num_eo, h]\n\n # ================ Transformer Layers ================\n hidden = self.transformer_stack(hidden)\n\n # ================ Sequence Reduction ================\n if self.reduce_output is not None:\n hidden = self.reduce_sequence(hidden)\n\n # ================ FC Layers ================\n hidden = self.fc_stack(hidden)\n\n return_data = {\"combiner_output\": hidden}\n\n if len(inputs) == 1:\n for key, value in [d for d in inputs.values()][0].items():\n if key != ENCODER_OUTPUT:\n return_data[key] = value\n\n return return_data\n\n\n@register_combiner(TabTransformerCombinerConfig)\nclass TabTransformerCombiner(Combiner):\n def __init__(\n self, input_features: dict[str, \"InputFeature\"] = None, config: TabTransformerCombinerConfig = None, **kwargs\n ):\n super().__init__(input_features)\n self.name = \"TabTransformerCombiner\"\n logger.debug(f\"Initializing {self.name}\")\n\n self.reduce_output = config.reduce_output\n self.reduce_sequence = SequenceReducer(\n reduce_mode=config.reduce_output, max_sequence_length=len(input_features), encoding_size=config.hidden_size\n )\n self.supports_masking = True\n\n self.embed_input_feature_name = config.embed_input_feature_name\n if self.embed_input_feature_name:\n vocab = [\n i_f\n for i_f in input_features\n if input_features.get(i_f).type() != NUMBER or input_features.get(i_f).type() != BINARY\n ]\n if self.embed_input_feature_name == \"add\":\n self.embed_i_f_name_layer = Embed(vocab, config.hidden_size, force_embedding_size=True)\n projector_size = config.hidden_size\n elif isinstance(self.embed_input_feature_name, int):\n if self.embed_input_feature_name > config.hidden_size:\n raise ValueError(\n \"TabTransformer parameter \"\n \"`embed_input_feature_name` \"\n \"specified integer value ({}) \"\n \"needs to be smaller than \"\n \"`hidden_size` ({}).\".format(self.embed_input_feature_name, config.hidden_size)\n )\n self.embed_i_f_name_layer = Embed(\n vocab,\n self.embed_input_feature_name,\n force_embedding_size=True,\n )\n projector_size = config.hidden_size - self.embed_input_feature_name\n else:\n raise ValueError(\n \"TabTransformer parameter \"\n \"`embed_input_feature_name` \"\n \"should be either None, an integer or `add`, \"\n \"the current value is \"\n \"{}\".format(self.embed_input_feature_name)\n )\n else:\n projector_size = config.hidden_size\n\n logger.debug(\" Projectors\")\n self.unembeddable_features = []\n self.embeddable_features = []\n for i_f in input_features:\n if input_features.get(i_f).type() in {NUMBER, BINARY}:\n self.unembeddable_features.append(i_f)\n else:\n self.embeddable_features.append(i_f)\n\n self.projectors = ModuleList()\n for i_f in self.embeddable_features:\n flatten_size = self.get_flatten_size(input_features.get(i_f).output_shape)\n self.projectors.append(Linear(flatten_size[0], projector_size))\n\n # input to layer_norm are the encoder outputs for unembeddable features,\n # which are number or binary features. These should be 2-dim\n # tensors. Size should be concatenation of these tensors.\n concatenated_unembeddable_encoders_size = 0\n for i_f in self.unembeddable_features:\n concatenated_unembeddable_encoders_size += input_features.get(i_f).output_shape[0]\n\n # Skip LayerNorm when normalizing a single value — LayerNorm(1) always\n # outputs zero which kills gradients for all downstream parameters.\n if concatenated_unembeddable_encoders_size > 1:\n self.layer_norm = torch.nn.LayerNorm(concatenated_unembeddable_encoders_size)\n else:\n self.layer_norm = torch.nn.Identity()\n\n logger.debug(\" TransformerStack\")\n self.transformer_stack = TransformerStack(\n input_size=config.hidden_size,\n max_sequence_length=len(self.embeddable_features),\n hidden_size=config.hidden_size,\n # todo: can we just use projector_size? # hidden_size,\n num_heads=config.num_heads,\n output_size=config.transformer_output_size,\n num_layers=config.num_layers,\n dropout=config.dropout,\n )\n\n logger.debug(\" FCStack\")\n\n # determine input size to fully connected layer based on reducer\n if config.reduce_output == \"concat\":\n fc_input_size = len(self.embeddable_features) * config.hidden_size\n else:\n fc_input_size = self.reduce_sequence.output_shape[-1] if len(self.embeddable_features) > 0 else 0\n self.fc_stack = FCStack(\n fc_input_size + concatenated_unembeddable_encoders_size,\n layers=config.fc_layers,\n num_layers=config.num_fc_layers,\n default_output_size=config.output_size,\n default_use_bias=config.use_bias,\n default_weights_initializer=config.weights_initializer,\n default_bias_initializer=config.bias_initializer,\n default_norm=config.norm,\n default_norm_params=config.norm_params,\n default_activation=config.fc_activation,\n default_dropout=config.fc_dropout,\n fc_residual=config.fc_residual,\n )\n\n self._empty_hidden = torch.empty([1, 0])\n self._embeddable_features_indices = torch.arange(0, len(self.embeddable_features))\n\n # Create empty tensor of shape [1, 0] to use as hidden in case there are no category or numeric/binary features.\n self.register_buffer(\"empty_hidden\", self._empty_hidden)\n self.register_buffer(\"embeddable_features_indices\", self._embeddable_features_indices)\n\n @staticmethod\n def get_flatten_size(output_shape: torch.Size) -> torch.Size:\n size = torch.prod(torch.Tensor([*output_shape]))\n return torch.Size([size.type(torch.int32)])\n\n @property\n def output_shape(self) -> torch.Size:\n return self.fc_stack.output_shape\n\n def forward(\n self,\n inputs: dict, # encoder outputs\n ) -> dict:\n unembeddable_encoder_outputs = [inputs[k][ENCODER_OUTPUT] for k in inputs if k in self.unembeddable_features]\n embeddable_encoder_outputs = [inputs[k][ENCODER_OUTPUT] for k in inputs if k in self.embeddable_features]\n\n batch_size = (\n embeddable_encoder_outputs[0].shape[0]\n if len(embeddable_encoder_outputs) > 0\n else unembeddable_encoder_outputs[0].shape[0]\n )\n\n # ================ Project & Concat embeddables ================\n if len(embeddable_encoder_outputs) > 0:\n # ============== Flatten =================\n embeddable_encoder_outputs = [torch.reshape(eo, [batch_size, -1]) for eo in embeddable_encoder_outputs]\n\n projected = [self.projectors[i](eo) for i, eo in enumerate(embeddable_encoder_outputs)]\n hidden = torch.stack(projected) # num_eo, bs, h\n hidden = torch.permute(hidden, (1, 0, 2)) # bs, num_eo, h\n\n if self.embed_input_feature_name:\n i_f_names_idcs = torch.reshape(\n torch.arange(0, len(embeddable_encoder_outputs), device=self.device), [-1, 1]\n )\n embedded_i_f_names = self.embed_i_f_name_layer(i_f_names_idcs)\n embedded_i_f_names = torch.unsqueeze(embedded_i_f_names, dim=0)\n embedded_i_f_names = torch.tile(embedded_i_f_names, [batch_size, 1, 1])\n if self.embed_input_feature_name == \"add\":\n hidden = hidden + embedded_i_f_names\n else:\n hidden = torch.cat([hidden, embedded_i_f_names], -1)\n\n # ================ Transformer Layers ================\n hidden = self.transformer_stack(hidden)\n\n # ================ Sequence Reduction ================\n hidden = self.reduce_sequence(hidden)\n else:\n # create empty tensor because there are no category features\n hidden = torch.empty([batch_size, 0], device=self.device)\n\n # ================ Concat Skipped ================\n if len(unembeddable_encoder_outputs) > 0:\n unembeddable_encoder_outputs = [torch.reshape(eo, [batch_size, -1]) for eo in unembeddable_encoder_outputs]\n # ================ Flatten ================\n if len(unembeddable_encoder_outputs) > 1:\n unembeddable_hidden = torch.cat(unembeddable_encoder_outputs, -1) # tf.keras.layers.concatenate\n else:\n unembeddable_hidden = list(unembeddable_encoder_outputs)[0]\n unembeddable_hidden = self.layer_norm(unembeddable_hidden)\n\n else:\n # create empty tensor because there are not numeric/binary features\n unembeddable_hidden = torch.tile(self.empty_hidden, [batch_size, 0])\n\n # ================ Concat Skipped and Others ================\n # When reduce_output is None, hidden is 3D [batch, seq, dim] but\n # unembeddable_hidden is 2D [batch, dim]. Expand to match.\n if hidden.dim() == 3 and unembeddable_hidden.dim() == 2:\n unembeddable_hidden = unembeddable_hidden.unsqueeze(1).expand(-1, hidden.size(1), -1)\n hidden = torch.cat([hidden, unembeddable_hidden], -1)\n\n # ================ FC Layers ================\n hidden = self.fc_stack(hidden)\n\n return_data = {\"combiner_output\": hidden}\n\n if len(inputs) == 1:\n for key, value in [d for d in inputs.values()][0].items():\n if key != ENCODER_OUTPUT:\n return_data[key] = value\n\n return return_data\n\n\n@register_combiner(ComparatorCombinerConfig)\nclass ComparatorCombiner(Combiner):\n def __init__(\n self,\n input_features: dict[str, \"InputFeature\"],\n config: ComparatorCombinerConfig = None,\n **kwargs,\n ):\n super().__init__(input_features)\n self.name = \"ComparatorCombiner\"\n logger.debug(f\"Entering {self.name}\")\n\n self.entity_1 = config.entity_1\n self.entity_2 = config.entity_2\n self.required_inputs = set(config.entity_1 + config.entity_2)\n self.output_size = config.output_size\n\n self.fc_stack = None\n\n # todo future: this may be redundant, check\n fc_layers = config.fc_layers\n if fc_layers is None and config.num_fc_layers is not None:\n fc_layers = []\n for _ in range(config.num_fc_layers):\n fc_layers.append({\"output_size\": config.output_size})\n\n if fc_layers is not None:\n logger.debug(\"Setting up FCStack\")\n self.e1_fc_stack = FCStack(\n self.get_entity_shape(config.entity_1)[-1],\n layers=fc_layers,\n num_layers=config.num_fc_layers,\n default_output_size=config.output_size,\n default_use_bias=config.use_bias,\n default_weights_initializer=config.weights_initializer,\n default_bias_initializer=config.bias_initializer,\n default_norm=config.norm,\n default_norm_params=config.norm_params,\n default_activation=config.activation,\n default_dropout=config.dropout,\n )\n self.e2_fc_stack = FCStack(\n self.get_entity_shape(config.entity_2)[-1],\n layers=fc_layers,\n num_layers=config.num_fc_layers,\n default_output_size=config.output_size,\n default_use_bias=config.use_bias,\n default_weights_initializer=config.weights_initializer,\n default_bias_initializer=config.bias_initializer,\n default_norm=config.norm,\n default_norm_params=config.norm_params,\n default_activation=config.activation,\n default_dropout=config.dropout,\n )\n\n self.last_fc_layer_output_size = fc_layers[-1][\"output_size\"]\n\n # todo: set initializer and regularization\n self.register_buffer(\n \"bilinear_weights\",\n torch.randn([self.last_fc_layer_output_size, self.last_fc_layer_output_size], dtype=torch.float32),\n )\n\n def get_entity_shape(self, entity: list) -> torch.Size:\n sizes = [torch.prod(torch.Tensor([*self.handle.input_features.get(k).output_shape])) for k in entity]\n return torch.Size([torch.sum(torch.Tensor(sizes)).type(torch.int32)])\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([2 * self.last_fc_layer_output_size + 2])\n\n def forward(\n self,\n inputs: dict, # encoder outputs\n ) -> dict[str, torch.Tensor]: # encoder outputs\n if inputs.keys() != self.required_inputs:\n raise ValueError(f\"Missing inputs {self.required_inputs - set(inputs.keys())}\")\n\n ############\n # Entity 1 #\n ############\n e1_enc_outputs = [inputs[k][ENCODER_OUTPUT] for k in self.entity_1]\n\n # ================ Flatten ================\n batch_size = e1_enc_outputs[0].shape[0]\n e1_enc_outputs = [torch.reshape(eo, [batch_size, -1]) for eo in e1_enc_outputs]\n\n # ================ Concat ================\n if len(e1_enc_outputs) > 1:\n e1_hidden = torch.cat(e1_enc_outputs, 1)\n else:\n e1_hidden = list(e1_enc_outputs)[0]\n\n # ================ Fully Connected ================\n e1_hidden = self.e1_fc_stack(e1_hidden) # [bs, output_size]\n\n ############\n # Entity 2 #\n ############\n e2_enc_outputs = [inputs[k][ENCODER_OUTPUT] for k in self.entity_2]\n\n # ================ Flatten ================\n batch_size = e2_enc_outputs[0].shape[0]\n e2_enc_outputs = [torch.reshape(eo, [batch_size, -1]) for eo in e2_enc_outputs]\n\n # ================ Concat ================\n if len(e2_enc_outputs) > 1:\n e2_hidden = torch.cat(e2_enc_outputs, 1)\n else:\n e2_hidden = list(e2_enc_outputs)[0]\n\n # ================ Fully Connected ================\n e2_hidden = self.e2_fc_stack(e2_hidden) # [bs, output_size]\n\n ###########\n # Compare #\n ###########\n if e1_hidden.shape != e2_hidden.shape:\n raise ValueError(\n f\"Mismatching shapes among dimensions! \"\n f\"entity1 shape: {e1_hidden.shape} \"\n f\"entity2 shape: {e2_hidden.shape}\"\n )\n\n element_wise_mul = e1_hidden * e2_hidden # [bs, output_size]\n dot_product = torch.sum(element_wise_mul, 1, keepdim=True) # [bs, 1]\n abs_diff = torch.abs(e1_hidden - e2_hidden) # [bs, output_size]\n bilinear_prod = torch.sum(\n torch.mm(e1_hidden, self.bilinear_weights) * e2_hidden, dim=1, keepdim=True\n ) # [bs, 1]\n\n logger.debug(\n \"preparing combiner output by concatenating these tensors: \"\n f\"dot_product: {dot_product.shape}, element_size_mul: {element_wise_mul.shape}\"\n f\", abs_diff: {abs_diff.shape}, bilinear_prod {bilinear_prod.shape}\"\n )\n hidden = torch.cat([dot_product, element_wise_mul, abs_diff, bilinear_prod], 1) # [bs, 2 * output_size + 2]\n\n return {\"combiner_output\": hidden}\n\n\n@register_combiner(ProjectAggregateCombinerConfig)\nclass ProjectAggregateCombiner(Combiner):\n def __init__(\n self, input_features: dict[str, \"InputFeature\"] = None, config: ProjectAggregateCombinerConfig = None, **kwargs\n ):\n super().__init__(input_features)\n self.name = \"ProjectAggregateCombiner\"\n logger.debug(f\" {self.name}\")\n\n logger.debug(\" Projectors\")\n self.projectors = ModuleList(\n # regardless of rank-2 or rank-3 input, torch.prod() calculates size\n # after flattening the encoder output tensor\n [\n Linear(\n torch.prod(torch.Tensor([*input_features.get(inp).output_shape])).type(torch.int32),\n config.projection_size,\n )\n for inp in input_features\n ]\n )\n\n self.fc_stack = None\n\n # todo future: this may be redundant, check\n fc_layers = config.fc_layers\n if fc_layers is None and config.num_fc_layers is not None:\n fc_layers = []\n for i in range(config.num_fc_layers):\n fc_layers.append({\"output_size\": config.output_size})\n\n self.fc_layers = fc_layers\n if self.fc_layers is not None:\n logger.debug(\" FCStack\")\n self.fc_stack = FCStack(\n first_layer_input_size=config.projection_size,\n layers=config.fc_layers,\n num_layers=config.num_fc_layers,\n default_output_size=config.output_size,\n default_use_bias=config.use_bias,\n default_weights_initializer=config.weights_initializer,\n default_bias_initializer=config.bias_initializer,\n default_norm=config.norm,\n default_norm_params=config.norm_params,\n default_activation=config.activation,\n default_dropout=config.dropout,\n residual=config.residual,\n )\n\n if input_features and len(input_features) == 1 and self.fc_layers is None:\n self.supports_masking = True\n\n def forward(self, inputs: dict) -> dict: # encoder outputs\n encoder_outputs = [inputs[k][ENCODER_OUTPUT] for k in inputs]\n\n # ================ Flatten ================\n batch_size = encoder_outputs[0].shape[0]\n encoder_outputs = [torch.reshape(eo, [batch_size, -1]) for eo in encoder_outputs]\n\n # ================ Project ================\n projected = [self.projectors[i](eo) for i, eo in enumerate(encoder_outputs)]\n hidden = torch.stack(projected)\n hidden = torch.permute(hidden, (1, 0, 2)) # shape [bs, num_eo, h]\n\n # ================ Aggregate ================\n hidden = torch.mean(hidden, dim=1)\n\n # ================ Fully Connected ================\n if self.fc_stack is not None:\n hidden = self.fc_stack(hidden)\n\n return_data = {\"combiner_output\": hidden}\n\n if len(inputs) == 1:\n # Workaround for including additional tensors from output of input encoders for\n # potential use in decoders, e.g. LSTM state for seq2seq.\n # TODO(Justin): Think about how to make this communication work for multi-sequence\n # features. Other combiners.\n for key, value in [d for d in inputs.values()][0].items():\n if key != ENCODER_OUTPUT:\n return_data[key] = value\n\n return return_data\n"
},
{
"path": "ludwig/config_sampling/__init__.py",
"content": ""
},
{
"path": "ludwig/config_sampling/explore_schema.py",
"content": "import copy\nimport random\nfrom collections import deque, namedtuple\nfrom typing import Any, Deque\n\nimport pandas as pd\n\nfrom ludwig.config_sampling.parameter_sampling import handle_property_type, ParameterBaseTypes\nfrom ludwig.constants import SEQUENCE, TEXT, TIMESERIES\nfrom ludwig.data.dataset_synthesizer import build_synthetic_dataset_df\nfrom ludwig.schema.model_types.base import ModelConfig\nfrom ludwig.types import ModelConfigDict\nfrom ludwig.utils.misc_utils import merge_dict\n\n# number of examples to generate for synthetic dataset\nNUM_SYNTHETIC_EXAMPLES = 10\n\nConfigOption = namedtuple(\"ConfigOption\", [\"config_option\", \"fully_explored\"])\n\n\ndef explore_properties(\n jsonschema_properties: dict[str, Any],\n parent_parameter_path: str,\n dq: Deque[ConfigOption],\n allow_list: list[str] = [],\n) -> Deque[tuple[dict, bool]]:\n \"\"\"Recursively explores the `properties` part of any subsection of the schema.\n\n Args:\n jsonschema_properties: any properties section of the schema.\n parent_parameter_path: period-delimited list of parent dictionary keys up to the given jsonschema_properties\n (e.g. defaults.number.preprocessing)\n dq: dequeue data structure that stores tuples of (config_options, fully_explored).\n config_options: Dict[str, List], fully_explored: bool is a dictionary is a dictionary of parameter name to\n list of values to explore.\n fully_explored is a boolean value indicating whether all subsections of the properties dictionary have been\n explored.\n allow_list: list of top level keys of the properties dictionary to skip.\n\n Returns:\n A deque of (dict, bool) tuples.\n - The first element of the tuple contains a dictionary of config options, which maps from a ludwig\n config parameter to a list of the values to be explored for that parameter. Here's an example:\n trainer.batch_size: [\"auto\", 2, 43]\n trainer.learning_rate: [\"auto\", 0.1, 0.00002, 0.32424]\n ...\n - The second element of the tuple is whether we've explored this \"config path\"\n fully. This is important to track when recursing into nested structures.\n \"\"\"\n # processed_dq will contain complete config options with all the parameters in the properties dictionary\n # dq will contain configs options that are still being completed.\n processed_dq = deque()\n while dq and not dq[0].fully_explored:\n for parameter_name_or_section, jsonschema_property in jsonschema_properties.items():\n if allow_list and parameter_name_or_section not in allow_list:\n continue\n\n parameter_path = (\n f\"{parent_parameter_path}.{parameter_name_or_section}\"\n if parent_parameter_path\n else parameter_name_or_section\n )\n config_options, _ = dq.popleft()\n\n if \"properties\" in jsonschema_property and \"allOf\" in jsonschema_property:\n for child_item in jsonschema_property[\"allOf\"]:\n expanded_config_options_dq = explore_from_all_of(\n config_options=copy.deepcopy(config_options), item=child_item, key_so_far=parameter_path\n )\n # add returned child config options to the deque to be processed.\n dq.extend(expanded_config_options_dq)\n\n elif \"properties\" in jsonschema_property and \"allOf\" not in jsonschema_property:\n # This is the case where we don't have a list of properties, just a properties\n # dictionary nested inside another.\n child_properties = jsonschema_property[\"properties\"]\n # a new dequeue to be passed to explore parameters from\n raw_entry = deque([ConfigOption(copy.deepcopy(config_options), False)])\n child_config_options_dq = explore_properties(child_properties, parameter_path, raw_entry)\n merged_config_options_dq = merge_dq(config_options, child_config_options_dq)\n # add returned config options to the deque to be processed.\n dq.extend(merged_config_options_dq)\n\n else:\n # this is the base case.\n parameter_samples = get_samples(jsonschema_property)\n if parameter_samples:\n config_options[parameter_path] = parameter_samples\n\n # add config_options back to queue. fully_explored = False because we still didn't finish\n # exploring all the keys in the properties dictionary.\n dq.appendleft(ConfigOption(config_options, False))\n\n # at this point, we finished exploring all keys of the properties dictionary. Add all config options\n # to the processed queue.\n while dq:\n config_options, _ = dq.popleft()\n processed_dq.append(ConfigOption(config_options, True))\n\n return processed_dq\n\n\ndef get_samples(jsonschema_property: dict[str, Any]) -> list[ParameterBaseTypes]:\n \"\"\"Get possible values for a leaf property (no sub-properties).\n\n Args:\n jsonschema_property: leaf property in the schema. Has no sub-properties.\n \"\"\"\n if \"oneOf\" in jsonschema_property:\n temp = []\n for elem in jsonschema_property[\"oneOf\"]:\n temp += get_potential_values(elem)\n return temp\n else:\n return get_potential_values(jsonschema_property)\n\n\ndef merge_dq(config_options: dict[str, Any], child_config_options_dq: Deque[ConfigOption]) -> Deque[ConfigOption]:\n \"\"\"Merge config_options with the child_config_options in the dq.\"\"\"\n dq = deque()\n while child_config_options_dq:\n child_config_options, visited = child_config_options_dq.popleft()\n cfg = merge_dict(child_config_options, config_options)\n dq.append(ConfigOption(cfg, visited))\n return dq\n\n\ndef explore_from_all_of(config_options: dict[str, Any], item: dict[str, Any], key_so_far: str) -> Deque[ConfigOption]:\n \"\"\"Takes a child of `allOf` and calls `explore_properties` on it.\"\"\"\n for parameter_name_or_section in item[\"if\"][\"properties\"]:\n config_options[key_so_far + \".\" + parameter_name_or_section] = item[\"if\"][\"properties\"][\n parameter_name_or_section\n ][\"const\"]\n jsonschema_properties = item[\"then\"][\"properties\"]\n raw_entry = deque([ConfigOption(copy.deepcopy(config_options), False)])\n return explore_properties(jsonschema_properties, parent_parameter_path=key_so_far, dq=raw_entry)\n\n\ndef get_potential_values(item: dict[str, Any]) -> list[ParameterBaseTypes | list[ParameterBaseTypes]]:\n \"\"\"Returns a list of values to explore for a config parameter.\n\n Param:\n item: config parameter-specific dictionary. Considered as a leaf in the schema. Contains type, default, and\n parameter metadata, etc.\n \"\"\"\n temp = []\n item_type = item.get(\"type\")\n if item_type is None:\n # No explicit type — try to infer from enum/const/default\n if \"enum\" in item:\n return [v for v in item[\"enum\"] if v is not None]\n if \"const\" in item:\n return [item[\"const\"]]\n if \"default\" in item:\n return [item[\"default\"]]\n return []\n # Case where we're using OneOf (e.g. to allow batch size 'auto' and integers)\n if isinstance(item_type, list):\n for property_type in item_type:\n temp += handle_property_type(property_type, item)\n else:\n temp += handle_property_type(item_type, item)\n\n # Make sure values are unique. Not using set because some values are unhashable.\n unique_temp = []\n for temp_item in temp:\n if temp_item not in unique_temp:\n unique_temp.append(temp_item)\n return unique_temp\n\n\ndef generate_possible_configs(config_options: dict[str, Any]):\n \"\"\"Generate exhaustive configs from config_options.\n\n This function does not take a cross product of all the options for all the config parameters. It selects parameter\n values independently from each other.\n\n Args:\n config_options: dictionary mapping from ludwig config parameter to all values to be explored.\n Here's an example of what it could look like:\n\n trainer.batch_size: [\"auto\", 2, 43]\n trainer.learning_rate: [\"auto\", 0.1, 0.00002, 0.32424]\n ...\n \"\"\"\n # The number of configs to generate is the max length of the lists of samples over all parameters.\n num_configs = 1\n for parameter_name in config_options:\n if isinstance(config_options[parameter_name], list):\n num_configs = max(num_configs, len(config_options[parameter_name]))\n config_options[parameter_name] = deque(config_options[parameter_name])\n\n for _ in range(num_configs):\n config = {}\n for parameter_name in config_options:\n # if parameter is regular parameter with explored values.\n if config_options[parameter_name] and not isinstance(config_options[parameter_name], str):\n config[parameter_name] = config_options[parameter_name].popleft()\n # case for parameters where we don't have choices such as `encoder.type: parallel_cnn` that\n # cause the downstream parameters to change.\n elif isinstance(config_options[parameter_name], str):\n config[parameter_name] = config_options[parameter_name]\n yield create_nested_dict(config)\n\n\ndef create_nested_dict(flat_dict: dict[str, float | str]) -> ModelConfigDict:\n \"\"\"Generate a nested dict out of a flat dict whose keys are delimited by a delimiter character.\n\n Args:\n flat_dict: potential generated baseline config. Here's an example of what it could look like:\n\n trainer.batch_size: 324\n trainer.learning_rate: 0.0635\n\n The expected output would be\n\n trainer:\n batch_size: 324\n learning_rate: 0.0635\n \"\"\"\n\n def to_nested_format(parameter_name: str, value: str | int | float, delimiter: str = \".\") -> dict[str, Any]:\n # https://stackoverflow.com/a/40401961\n split_parameter_name = parameter_name.split(delimiter)\n for parameter_name_or_section in reversed(split_parameter_name):\n value = {parameter_name_or_section: value}\n return value\n\n config = {}\n for parameter_name_or_section in flat_dict:\n config = merge_dict(\n config, to_nested_format(parameter_name_or_section, copy.deepcopy(flat_dict[parameter_name_or_section]))\n )\n return config\n\n\ndef combine_configs(\n explored: Deque[tuple[dict, bool]], config: ModelConfigDict\n) -> list[tuple[ModelConfigDict, pd.DataFrame]]:\n \"\"\"Merge base config with explored sections.\n\n Args:\n explored: deque containing all the config options.\n config: base Ludwig config to merge the explored configs with.\n \"\"\"\n dataset = build_synthetic_dataset_df(NUM_SYNTHETIC_EXAMPLES, config)\n ret = []\n for config_options, _ in explored:\n for default_config in generate_possible_configs(config_options=config_options):\n merged_config = merge_dict(copy.deepcopy(config), default_config)\n try:\n ModelConfig.from_dict(merged_config)\n ret.append((merged_config, dataset))\n except Exception:\n pass\n return ret\n\n\ndef combine_configs_for_comparator_combiner(\n explored: Deque[tuple], config: ModelConfigDict\n) -> list[tuple[ModelConfigDict, pd.DataFrame]]:\n \"\"\"Merge base config with explored sections.\n\n Completes the entity_1 and entity_2 paramters of the comparator combiner.\n\n Args:\n explored: deque containing all the config options.\n config: base Ludwig config to merge the explored configs with.\n \"\"\"\n dataset = build_synthetic_dataset_df(NUM_SYNTHETIC_EXAMPLES, config)\n ret = []\n for item in explored:\n for default_config in generate_possible_configs(config_options=item[0]):\n merged_config = merge_dict(copy.deepcopy(config), default_config)\n\n # create two random lists for entity1 and entity2\n entity_names = [feature[\"name\"] for feature in config[\"input_features\"]]\n random.shuffle(entity_names)\n entity_1_size = random.randint(1, len(entity_names) - 1)\n merged_config[\"combiner\"][\"entity_1\"] = entity_names[:entity_1_size]\n merged_config[\"combiner\"][\"entity_2\"] = entity_names[entity_1_size:]\n try:\n ModelConfig.from_dict(merged_config)\n ret.append((merged_config, dataset))\n except Exception:\n pass\n return ret\n\n\ndef combine_configs_for_sequence_combiner(\n explored: Deque[tuple], config: ModelConfigDict\n) -> list[tuple[ModelConfigDict, pd.DataFrame]]:\n \"\"\"Merge base config with explored sections.\n\n Uses the right reduce_output strategy for the sequence and sequence_concat combiners.\n\n Args:\n explored: deque containing all the config options.\n config: base Ludwig config to merge the explored configs with.\n \"\"\"\n dataset = build_synthetic_dataset_df(NUM_SYNTHETIC_EXAMPLES, config)\n ret = []\n for item in explored:\n for default_config in generate_possible_configs(config_options=item[0]):\n merged_config = merge_dict(copy.deepcopy(config), default_config)\n for i in range(len(merged_config[\"input_features\"])):\n if merged_config[\"input_features\"][i][\"type\"] in {SEQUENCE, TEXT, TIMESERIES}:\n merged_config[\"input_features\"][0][\"encoder\"] = {\"type\": \"embed\", \"reduce_output\": None}\n try:\n ModelConfig.from_dict(merged_config)\n ret.append((merged_config, dataset))\n except Exception:\n pass\n return ret\n"
},
{
"path": "ludwig/config_sampling/parameter_sampling.py",
"content": "import random\nfrom typing import Any, Union\n\nfrom ludwig.schema.metadata.parameter_metadata import ExpectedImpact\n\n# base types for ludwig config parameters.\nParameterBaseTypes = Union[str, float, int, bool, None]\n\n\ndef handle_property_type(\n property_type: str, item: dict[str, Any], expected_impact: ExpectedImpact = ExpectedImpact.HIGH\n) -> list[ParameterBaseTypes | list[ParameterBaseTypes]]:\n \"\"\"Return possible parameter values for a parameter type.\n\n Args:\n property_type: type of the parameter (e.g. array, number, etc.)\n item: dictionary containing details on the parameter such as default, min and max values.\n expected_impact: threshold expected impact that we'd like to include.\n \"\"\"\n parameter_metadata = item.get(\"parameter_metadata\", None)\n if not parameter_metadata:\n return []\n\n # don't explore internal only parameters.\n if parameter_metadata.get(\"internal_only\", True):\n return []\n\n # don't explore parameters that have expected impact less than HIGH.\n if parameter_metadata.get(\"expected_impact\", ExpectedImpact.LOW) < expected_impact:\n return []\n\n if property_type == \"number\":\n return explore_number(item)\n elif property_type == \"integer\":\n return explore_integer(item)\n elif property_type == \"string\":\n return explore_string(item)\n elif property_type == \"boolean\":\n return explore_boolean()\n elif property_type == \"null\":\n return explore_null()\n elif property_type == \"array\":\n return explore_array(item)\n else:\n return []\n\n\ndef explore_array(item: dict[str, Any]) -> list[list[ParameterBaseTypes]]:\n \"\"\"Return possible parameter values for the `array` parameter type.\n\n Args:\n item: dictionary containing details on the parameter such as default, min and max values.\n \"\"\"\n\n candidates = []\n if \"default\" in item and item[\"default\"]:\n candidates.append(item[\"default\"])\n\n item_choices = []\n maxlen = 0\n\n # In the case where the length of the array isn't defined.\n if not isinstance(item[\"items\"], list):\n return []\n\n for item_of in item[\"items\"]:\n choices = handle_property_type(item_of[\"type\"], item_of)\n maxlen = max(maxlen, len(choices))\n item_choices.append(choices)\n\n # pad to same length\n for i in range(len(item_choices)):\n item_choices[i] = maxlen * item_choices[i]\n item_choices[i] = item_choices[i][:maxlen]\n\n merged = list(zip(*item_choices)) + candidates\n return [list(tup) for tup in merged]\n\n\ndef explore_number(item: dict[str, Any]) -> list[ParameterBaseTypes]:\n \"\"\"Return possible parameter values for the `number` parameter type.\n\n Args:\n item: dictionary containing details on the parameter such as default, min and max values.\n TODO(Wael): Improve logic.\n \"\"\"\n minimum, maximum = 0, 1\n if \"default\" not in item or item[\"default\"] is None:\n candidates = []\n else:\n candidates = [1, 2, item[\"default\"], 2 * (item[\"default\"] + 1), item[\"default\"] // 2, -1 * item[\"default\"]]\n\n if \"minimum\" in item:\n minimum = item[\"minimum\"]\n candidates = [num for num in candidates if num > minimum]\n if \"maximum\" in item:\n maximum = item[\"maximum\"]\n candidates = [num for num in candidates if num < maximum]\n return candidates + [random.random() * 0.99 * maximum]\n\n\ndef explore_integer(item: dict[str, Any]) -> list[ParameterBaseTypes]:\n \"\"\"Return possible parameter values for the `integer` parameter type.\n\n Args:\n item: dictionary containing details on the parameter such as default, min and max values.\n TODO(Wael): Improve logic.\n \"\"\"\n minimum, maximum = 0, 10\n\n if \"default\" not in item or item[\"default\"] is None:\n candidates = []\n else:\n candidates = [item[\"default\"], 2 * (item[\"default\"] + 1), item[\"default\"] // 2, -1 * item[\"default\"]]\n\n if \"minimum\" in item:\n minimum = item[\"minimum\"]\n candidates = [num for num in candidates if num >= item[\"minimum\"]]\n if \"maximum\" in item:\n maximum = item[\"maximum\"]\n candidates = [num for num in candidates if num <= item[\"maximum\"]]\n\n return candidates + [random.randint(minimum, maximum)]\n\n\ndef explore_string(item: dict[str, Any]) -> list[ParameterBaseTypes]:\n \"\"\"Return possible parameter values for the `string` parameter type.\n\n Args:\n item: dictionary containing details on the parameter such as default, min and max values.\n \"\"\"\n\n if \"enum\" in item:\n return item[\"enum\"]\n return [item[\"default\"]]\n\n\ndef explore_boolean() -> list[bool]:\n \"\"\"Return possible parameter values for the `boolean` parameter type (i.e. [True, False])\"\"\"\n return [True, False]\n\n\ndef explore_null() -> list[None]:\n \"\"\"Return possible parameter values for the `null` parameter type (i.e. [None])\"\"\"\n return [None]\n"
},
{
"path": "ludwig/config_validation/__init__.py",
"content": ""
},
{
"path": "ludwig/config_validation/checks.py",
"content": "\"\"\"Checks that are not easily covered by marshmallow JSON schema validation like parameter interdependencies.\"\"\"\n\nfrom abc import ABC, abstractmethod\nfrom collections.abc import Callable\nfrom re import findall\nfrom typing import TYPE_CHECKING\n\nfrom transformers import AutoConfig\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import (\n AUDIO,\n BINARY,\n IMAGE,\n IN_MEMORY,\n MIN_QUANTIZATION_BITS_FOR_MERGE_AND_UNLOAD,\n MODEL_ECD,\n MODEL_LLM,\n SEQUENCE,\n SET,\n TEXT,\n TIMESERIES,\n VECTOR,\n)\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.utils.metric_utils import get_feature_to_metric_names_map_from_feature_collection\nfrom ludwig.utils.misc_utils import merge_dict\n\nif TYPE_CHECKING:\n from ludwig.schema.model_config import ModelConfig\n\n# Set of all sequence feature types.\nSEQUENCE_OUTPUT_FEATURE_TYPES = {SEQUENCE, TEXT, SET, VECTOR}\n\n\nclass ConfigCheckRegistry:\n \"\"\"A registry of configuration checks.\"\"\"\n\n def __init__(self):\n self._registry = []\n\n def register(self, check_fn):\n self._registry.append(check_fn)\n\n def check_config(self, config: \"ModelConfig\") -> None: # noqa: F821\n for check_fn in self._registry:\n check_fn(config)\n\n\n_CONFIG_CHECK_REGISTRY = ConfigCheckRegistry()\n\n\ndef get_config_check_registry():\n \"\"\"Returns the config check registry.\"\"\"\n return _CONFIG_CHECK_REGISTRY\n\n\n@DeveloperAPI\ndef register_config_check(fn) -> Callable:\n \"\"\"Registers a config check function.\"\"\"\n _CONFIG_CHECK_REGISTRY.register(fn)\n\n\nclass ConfigCheck(ABC):\n \"\"\"Checks instances of comprehensive (all parameters and defaults filled in) schema-validated config.\"\"\"\n\n @staticmethod\n @abstractmethod\n def check(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Checks config for validity.\"\"\"\n raise NotImplementedError\n\n\n@register_config_check\ndef check_feature_names_unique(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Checks that all feature names are unique.\"\"\"\n input_features = config.input_features\n input_feature_names = {input_feature.name for input_feature in input_features}\n\n output_features = config.output_features\n output_feature_names = {output_feature.name for output_feature in output_features}\n\n if len(input_feature_names) + len(output_feature_names) != len(input_features) + len(output_features):\n raise ConfigValidationError(\"Feature names must be unique.\")\n\n\n@register_config_check\ndef check_tied_features_valid(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Checks that all tied features are valid.\"\"\"\n input_features = config.input_features\n input_feature_names = {input_feature.name for input_feature in input_features}\n\n for input_feature in input_features:\n if input_feature.tied and input_feature.tied not in input_feature_names:\n raise ConfigValidationError(\n f\"Feature {input_feature.name} is tied to feature {input_feature.tied}, but the \"\n f\"'{input_feature.tied}' feature does not exist.\"\n )\n\n\n@register_config_check\ndef check_training_runway(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Checks that checkpoints_per_epoch and steps_per_checkpoint aren't simultaneously defined.\"\"\"\n if config.model_type == MODEL_ECD:\n if config.trainer.checkpoints_per_epoch != 0 and config.trainer.steps_per_checkpoint != 0:\n raise ConfigValidationError(\n \"It is invalid to specify both trainer.checkpoints_per_epoch AND \"\n \"trainer.steps_per_checkpoint. Please specify one or the other, or specify neither to \"\n \"checkpoint/eval the model every epoch.\"\n )\n\n\n@register_config_check\ndef check_ray_backend_in_memory_preprocessing(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Checks that in memory preprocessing is used with Ray backend.\"\"\"\n if config.backend is None:\n return\n if not hasattr(config.trainer, \"preprocessing\") or not hasattr(config.trainer.preprocessing, IN_MEMORY):\n return\n\n if config.backend.type == \"ray\" and not config.trainer.preprocessing.in_memory:\n raise ConfigValidationError(\n \"RayBackend does not support lazy loading of data files at train time. \"\n \"Set preprocessing config `in_memory: True`\"\n )\n\n for input_feature in config.input_features:\n if input_feature.type == AUDIO or input_feature.type == IMAGE:\n if not input_feature.preprocessing.in_memory and config.backend.type != \"ray\":\n raise ConfigValidationError(\n \"RayBackend does not support lazy loading of data files at train time. \"\n f\"Set preprocessing config `in_memory: True` for input feature {input_feature.name}\"\n )\n\n\ndef check_sequence_concat_combiner_requirements(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Checks that sequence concat combiner has at least one input feature that's sequential.\"\"\"\n if config.model_type != MODEL_ECD:\n return\n if config.combiner != \"sequence_concat\":\n return\n has_sequence_input = False\n for input_feature in config.input_features:\n if input_feature.type in SEQUENCE_OUTPUT_FEATURE_TYPES:\n has_sequence_input = True\n break\n if not has_sequence_input:\n raise ConfigValidationError(\n \"Sequence concat combiner should only be used for at least one sequential input feature.\"\n )\n\n\n@register_config_check\ndef check_comparator_combiner_requirements(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Checks that all of the feature names for entity_1 and entity_2 are valid features.\"\"\"\n if config.model_type != MODEL_ECD:\n return\n if config.combiner.type != \"comparator\":\n return\n\n input_feature_names = [input_feature.name for input_feature in config.input_features]\n for feature_name in config.combiner.entity_1:\n if feature_name not in input_feature_names:\n raise ConfigValidationError(\n f\"Feature {feature_name} in entity_1 for the comparator combiner is not a valid \" \"input feature name.\"\n )\n for feature_name in config.combiner.entity_2:\n if feature_name not in input_feature_names:\n raise ConfigValidationError(\n f\"Feature {feature_name} in entity_2 for the comparator combiner is not a valid \" \"input feature name.\"\n )\n\n if sorted(config.combiner.entity_1 + config.combiner.entity_2) != sorted(input_feature_names):\n raise ConfigValidationError(\"Not all input features are present as entities in the comparator combiner.\")\n\n\n@register_config_check\ndef check_class_balance_preprocessing(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Class balancing is only available for datasets with a single output feature.\"\"\"\n if config.preprocessing.oversample_minority or config.preprocessing.undersample_majority:\n if len(config.output_features) != 1:\n raise ConfigValidationError(\"Class balancing is only available for datasets with a single output feature.\")\n if config.output_features[0].type != BINARY:\n raise ConfigValidationError(\"Class balancing is only supported for binary output features.\")\n\n\n@register_config_check\ndef check_sampling_exclusivity(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Oversample minority and undersample majority are mutually exclusive.\"\"\"\n if config.preprocessing.oversample_minority and config.preprocessing.undersample_majority:\n raise ConfigValidationError(\n \"Oversample minority and undersample majority are mutually exclusive. Specify only one method.\"\n )\n\n\n@register_config_check\ndef check_validation_metric_exists(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Checks that the specified validation metric exists.\"\"\"\n validation_metric_name = config.trainer.validation_metric\n\n # Get all valid metrics.\n feature_to_metric_names_map = get_feature_to_metric_names_map_from_feature_collection(config.output_features)\n all_valid_metrics = set()\n for metric_names in feature_to_metric_names_map.values():\n all_valid_metrics.update(metric_names)\n\n if validation_metric_name not in all_valid_metrics:\n raise ConfigValidationError(\n f\"User-specified trainer.validation_metric '{validation_metric_name}' is not valid. \"\n f\"Available metrics are: {all_valid_metrics}\"\n )\n\n\n@register_config_check\ndef check_splitter(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Checks the validity of the splitter configuration.\"\"\"\n from ludwig.data.split import get_splitter\n\n splitter = get_splitter(**config.preprocessing.split.to_dict())\n splitter.validate(config)\n\n\n@register_config_check\ndef check_hf_tokenizer_requirements(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Checks that the HuggingFace tokenizer has a pretrained_model_name_or_path specified.\"\"\"\n\n for input_feature in config.input_features:\n if input_feature.type == TEXT:\n if input_feature.preprocessing.tokenizer == \"hf_tokenizer\":\n if input_feature.preprocessing.pretrained_model_name_or_path is None:\n raise ConfigValidationError(\n \"Pretrained model name or path must be specified for HuggingFace tokenizer.\"\n )\n\n\n@register_config_check\ndef check_hf_encoder_requirements(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Checks that a HuggingFace encoder has a pretrained_model_name_or_path specified.\"\"\"\n\n for input_feature in config.input_features:\n if input_feature.type == TEXT:\n if hasattr(input_feature.encoder, \"use_pretrained\"):\n if input_feature.preprocessing.pretrained_model_name_or_path is None:\n raise ConfigValidationError(\n \"Pretrained model name or path must be specified for HuggingFace encoder.\"\n )\n\n\n@register_config_check\ndef check_stacked_transformer_requirements(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Checks that the transformer encoder type correctly configures `num_heads` and `hidden_size`\"\"\"\n\n def is_divisible(hidden_size: int, num_heads: int) -> bool:\n \"\"\"Checks that hidden_size is divisible by num_heads.\"\"\"\n return hidden_size % num_heads == 0\n\n sequence_types = [SEQUENCE, TEXT, TIMESERIES]\n\n for input_feature in config.input_features:\n if_type = input_feature.type\n encoder = input_feature.encoder\n if (\n if_type in sequence_types\n and encoder.type == \"transformer\"\n and not is_divisible(encoder.hidden_size, encoder.num_heads)\n ):\n raise ConfigValidationError(\n f\"Input feature {input_feature.name} transformer encoder requires encoder.hidden_size to be divisible \"\n f\"by encoder.num_heads. Found hidden_size {encoder.hidden_size} and num_heads {encoder.num_heads}.\"\n )\n\n\n@register_config_check\ndef check_hyperopt_search_algorithm_dependencies_installed(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Check that the hyperopt search algorithm dependencies are installed.\"\"\"\n if config.hyperopt is None:\n return\n\n try:\n config.hyperopt.search_alg.dependencies_installed()\n except ImportError as e:\n raise ConfigValidationError(e.msg)\n\n\n@register_config_check\ndef check_hyperopt_scheduler_dependencies_installed(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Check that the hyperopt scheduler dependencies are installed.\"\"\"\n if config.hyperopt is None:\n return\n\n try:\n config.hyperopt.executor.scheduler.dependencies_installed()\n except ImportError as e:\n raise ConfigValidationError(e.msg)\n\n\n@register_config_check\ndef check_tagger_decoder_requirements(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Checks that the tagger decoder has at least one sequence, text or timeseries input feature where the\n encoder's reduce_output will produce a 3D shaped output from the combiner.\"\"\"\n # Check if there is a text or sequence output feature using a tagger decoder\n output_feature_with_tagger_decoder = False\n for output_feature in config.output_features:\n if output_feature.type in {TEXT, SEQUENCE} and output_feature.decoder.type == \"tagger\":\n output_feature_with_tagger_decoder = True\n\n if not output_feature_with_tagger_decoder:\n return\n\n # Check that there is at least one sequence, text or timeseries input feature that doesn't reduce the\n # output of the encoder.\n has_sequence_feature = False\n for input_feature in config.input_features:\n if input_feature.type in {SEQUENCE, TEXT, TIMESERIES}:\n has_sequence_feature = True\n if input_feature.encoder.reduce_output is None:\n return\n\n if not has_sequence_feature:\n raise ConfigValidationError(\"Tagger decoder requires at least one text, sequence or timeseries input feature.\")\n else:\n raise ConfigValidationError(\n \"Tagger decoder requires at least one of the text, sequence or timeseries input feature encoders to have \"\n \"`reduce_output` set to `None`.\"\n )\n\n\n@register_config_check\ndef check_hyperopt_parameter_dicts(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Checks for hyperopt parameter dicts against their config objects.\"\"\"\n if config.hyperopt is None:\n return\n\n from ludwig.schema.hyperopt.utils import get_parameter_cls, parameter_config_registry # noqa: F401\n\n for parameter, space in config.hyperopt.parameters.items():\n # skip nested hyperopt parameters\n if parameter != \".\":\n parameter_attribute_path = parameter.split(\".\")\n passed = False\n\n for root in [config, config.input_features, config.output_features]:\n current = root\n for p in parameter_attribute_path:\n try:\n current = current.__getattribute__(p)\n if p == parameter_attribute_path[-1]:\n passed = True\n except AttributeError:\n break\n if passed:\n break\n\n if not passed:\n raise ConfigValidationError(\n f\"The supplied hyperopt parameter {parameter} is not a valid config field. Check the Ludwig \"\n \"docs for the list of valid parameters.\"\n )\n\n try:\n space_cls = get_parameter_cls(space[\"space\"])\n space_cls.from_dict(space)\n except KeyError:\n space_types = \", \".join(parameter_config_registry.keys())\n raise ConfigValidationError(\n f\"Invalid hyperopt parameter space requested for `hyperopt.parameters.{parameter}`. Valid spaces \"\n f\"are {space_types}.\"\n )\n\n\n@register_config_check\ndef check_concat_combiner_requirements(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Checks that if the concat combiner receives a mixture of sequence and non-sequence features, that all\n sequence features are configured with reduce_output to be 2D tensors.\"\"\"\n if config.model_type != MODEL_ECD:\n return\n if config.combiner.type != \"concat\":\n return\n\n has_unreduced_sequence_feature = False\n has_non_sequence_feature = False\n for input_feature in config.input_features:\n if (\n input_feature.type in {SEQUENCE, TEXT, TIMESERIES}\n and hasattr(input_feature.encoder, \"reduce_output\")\n and input_feature.encoder.reduce_output is None\n ):\n has_unreduced_sequence_feature = True\n else:\n has_non_sequence_feature = True\n\n if has_unreduced_sequence_feature and has_non_sequence_feature:\n raise ConfigValidationError(\n \"The concat combiner cannot receive a mix of unreduced sequence features (3D) and non-sequence features \"\n \"(2D). Options: 1) Set reduce_output in sequence feature encoders to a value other than None to ensure 2D \"\n \"encoder outputs, 2) Choose a different combiner like `sequence_concat` which can handle a mix of 2D and \"\n \"3D encoder output shapes, or 3) Remove features to ensure that output shapes from all encoders are the \"\n \"same dimension (all 2D or all 3D).\"\n )\n\n\n@register_config_check\ndef check_hyperopt_nested_parameter_dicts(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Checks that all nested parameters in a hyperopt config exist.\"\"\"\n if config.hyperopt is None or \".\" not in config.hyperopt.parameters:\n return\n\n from ludwig.schema.hyperopt.utils import get_parameter_cls # noqa: F401\n from ludwig.schema.model_types.base import ModelConfig\n\n space = config.hyperopt.parameters[\".\"]\n\n # Build the config that would be produced by each parameter dict to validate subsections that may be in\n config_dict = config.to_dict()\n del config_dict[\"hyperopt\"]\n for category in space[\"categories\"]:\n for i, k in enumerate(category.keys()):\n try:\n config.__getattribute__(k)\n except AttributeError:\n raise ConfigValidationError(f\"Invalid config block {k} in nested hyperopt parameter dict {i}: {space}.\")\n\n category_dict = merge_dict(config_dict, category)\n try:\n ModelConfig.from_dict(category_dict)\n except ConfigValidationError as e:\n raise ConfigValidationError(f\"Invalid config in hyperopt nested parameter config: {category}. {e.message}\")\n\n try:\n space_cls = get_parameter_cls(\"choice\")\n space_cls.from_dict(space)\n except KeyError:\n raise ConfigValidationError(\n f\"Nested hyperparameter search spaces must be of type 'choice'. Requested space type: {space['space']}\"\n )\n\n\n@register_config_check\ndef check_llm_exactly_one_input_text_feature(config: \"ModelConfig\"): # noqa: F821\n if config.model_type != MODEL_LLM:\n return\n\n if len(config.input_features) == 1 and config.input_features[0].type == TEXT:\n return\n else:\n raise ConfigValidationError(\"LLM requires exactly one text input feature.\")\n\n\n@register_config_check\ndef check_llm_finetuning_output_feature_config(config: \"ModelConfig\"): # noqa: F821\n \"\"\"Checks that the output feature config for LLM finetuning is valid.\"\"\"\n if config.model_type != MODEL_LLM:\n return\n\n if config.trainer.type != \"finetune\":\n return\n\n if config.output_features[0].type != TEXT:\n raise ConfigValidationError(\n \"LLM finetuning requires the output feature to be a text feature. If you are trying to use a different \"\n \"output feature type such as category or binary, please change the output feature type to text.\"\n )\n\n\n@register_config_check\ndef check_llm_finetuning_trainer_config(config: \"ModelConfig\"): # noqa: F821\n \"\"\"Ensures that trainer type is finetune if adapter is not None.\"\"\"\n if config.model_type != MODEL_LLM:\n return\n\n if (\n config.trainer.type == \"none\"\n and config.adapter is not None\n and config.adapter.pretrained_adapter_weights is not None\n ):\n # If performing zero-shot, we must specify pretrained adapter weights\n return\n\n if config.adapter is not None and config.trainer.type != \"finetune\":\n raise ConfigValidationError(\"LLM finetuning requires trainer type to be finetune.\")\n\n\n@register_config_check\ndef check_llm_finetuning_backend_config(config: \"ModelConfig\"): # noqa: F821\n \"\"\"Checks that the LLM finetuning using Ray is configured correctly.\n\n DDP strategy is not supported for LLM finetuning because it leads to OOMs since the model is large and DDP strategy\n requires a copy of the model on each GPU.\n \"\"\"\n if config.model_type != MODEL_LLM:\n return\n\n # LLM finetuning is only supported by the finetune trainer type\n if (\n config.trainer.type != \"finetune\"\n and config.adapter is not None\n and config.adapter.pretrained_adapter_weights is not None\n ):\n return\n\n # Using local backend, so skip the checks below\n if not hasattr(config.backend, \"type\"):\n return\n\n backend = config.backend\n if not hasattr(backend.trainer, \"strategy\") or backend.trainer.strategy != \"deepspeed\":\n raise ConfigValidationError(\"LLM finetuning with Ray requires the DeepSpeed strategy.\")\n\n # Deepspeed requires GPU\n if not backend.trainer.use_gpu or backend.trainer.resources_per_worker.GPU < 1:\n raise ConfigValidationError(\"LLM finetuning with DeepSpeed requires GPU.\")\n\n\n@register_config_check\ndef check_llm_finetuning_adalora_config(config: \"ModelConfig\"):\n \"\"\"Checks that the adalora adapter is configured correctly.\n\n We check against PEFT's predefined target module list for ADALORA to see if this target_modules is present there. If\n not, AdaloraModel will run into issues downstream.\n \"\"\"\n if config.model_type != MODEL_LLM:\n return\n\n if not config.adapter:\n return\n\n if config.adapter.type != \"adalora\":\n return\n\n from peft.utils import TRANSFORMERS_MODELS_TO_ADALORA_TARGET_MODULES_MAPPING\n\n model_config = _get_llm_model_config(config.base_model)\n if model_config.model_type not in TRANSFORMERS_MODELS_TO_ADALORA_TARGET_MODULES_MAPPING:\n raise ConfigValidationError(\n f\"Adalora adapter is not supported for {model_config.model_type} model. \"\n f\"Supported model types are: {list(TRANSFORMERS_MODELS_TO_ADALORA_TARGET_MODULES_MAPPING.keys())}. \"\n \"If you know the target modules for your model, please specify them in the config through the \"\n \"`target_modules` key.\"\n )\n\n\n@register_config_check\ndef check_llm_finetuning_adaption_prompt_parameters(config: \"ModelConfig\"):\n \"\"\"Checks that the adaption_prompt adapter is configured correctly.\n\n Adaption prompt is only supported for Llama models.\n \"\"\"\n if config.model_type != MODEL_LLM:\n return\n\n if not config.adapter:\n return\n\n if config.adapter.type != \"adaption_prompt\":\n return\n\n from peft.tuners.adaption_prompt.config import TRANSFORMERS_MODEL_CONFIG\n\n # Adaption Config is currently only supported for Llama model types\n model_config = _get_llm_model_config(config.base_model)\n if model_config.model_type not in TRANSFORMERS_MODEL_CONFIG:\n raise ConfigValidationError(\n f\"Adaption prompt adapter is not supported for {model_config.model_type} model. \"\n f\"Supported model types are: {list(TRANSFORMERS_MODEL_CONFIG.keys())}.\"\n )\n\n\ndef _get_llm_model_config(model_name: str) -> AutoConfig:\n \"\"\"Returns the LLM model config.\"\"\"\n return AutoConfig.from_pretrained(model_name)\n\n\n# TODO(geoffrey, arnav): uncomment this when we have reconciled the config with the backend kwarg in api.py\n# @register_config_check\ndef check_llm_quantization_backend_incompatibility(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Checks that LLM model type with quantization uses the local backend.\"\"\"\n if config.model_type != MODEL_LLM:\n return\n\n if config.quantization is None:\n return\n\n backend_type = None\n if config.backend:\n backend_type = config.backend.get(\"type\", None)\n\n # If backend was explicitly set to Ray, then we need to raise an error\n if backend_type == \"ray\":\n raise ConfigValidationError(f\"LLM with quantization requires the 'local' backend, found: '{backend_type}'\")\n\n # If the backend is not explicitly set, then we need to check if a Ray process is running\n # If a Ray process is running, then we need to raise an error because the backend will be set to Ray\n if config.backend is None:\n try:\n # May not be installed, so we need to catch the ImportError\n import ray\n\n if ray.is_initialized():\n raise ConfigValidationError(\n \"LLM with quantization requires the 'local' backend, but backend will be set \"\n \"to Ray since Ray is already running locally.\"\n )\n except ImportError:\n pass\n\n\n@register_config_check\ndef check_llm_text_encoder_is_not_used_with_ecd(config: \"ModelConfig\") -> None:\n \"\"\"Checks that a pretrained text encoder is not used for ECD models with a text output feature.\"\"\"\n if config.model_type != MODEL_ECD:\n return\n\n if config.input_features[0].type != TEXT:\n return\n\n if config.output_features[0].type != TEXT:\n return\n\n if (\n hasattr(config.input_features[0].encoder, \"pretrained_model_name_or_path\")\n and config.input_features[0].encoder.pretrained_model_name_or_path\n ):\n raise ConfigValidationError(\"Please use the `model_type: llm` for text-to-text models.\")\n\n\n@register_config_check\ndef check_qlora_requirements(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Checks that all the necessary settings are in place for QLoRA.\"\"\"\n if config.model_type != MODEL_LLM or config.trainer.type == \"none\":\n return\n\n if config.quantization and (not config.adapter or config.adapter.type != \"lora\"):\n raise ConfigValidationError(\"Fine-tuning and LLM with quantization requires using the 'lora' adapter\")\n\n\n@register_config_check\ndef check_qlora_merge_and_unload_compatibility(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Checks that model.merge_and_unload() is supported by underlying model.save_pretrained() when merging QLoRA\n layers.\"\"\"\n if config.model_type != MODEL_LLM or config.trainer.type == \"none\":\n return\n\n if not (\n config.adapter\n and config.adapter.type in [\"lora\", \"adalora\"]\n and config.adapter.postprocessor\n and config.adapter.postprocessor.merge_adapter_into_base_model\n and config.quantization\n ):\n return\n\n if config.quantization.bits < MIN_QUANTIZATION_BITS_FOR_MERGE_AND_UNLOAD:\n raise ConfigValidationError(\n f\"\"\"This operation will entail merging LoRA layers on a {config.quantization.bits}-bit \\\nquantized model. Calling \"save_pretrained()\" on that model is currently unsupported. If you want to merge the LoRA \\\nadapter weights into the base model, you need to use 8-bit quantization or do non-quantized based training by removing \\\nthe quantization section from your Ludwig configuration.\"\"\"\n )\n\n\n@register_config_check\ndef check_prompt_requirements(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Checks that prompt's template and task properties are valid, according to the description on the schema.\"\"\"\n if config.model_type != MODEL_LLM:\n return\n\n # TODO: `prompt` by default should be set to null, not a default dict:\n # # If no prompt is provided, no validation necessary:\n # if not config.prompt:\n # return\n from ludwig.schema.llms.prompt import PromptConfig, RetrievalConfig\n\n if config.prompt == PromptConfig():\n return\n\n template = config.prompt.template\n task = config.prompt.task\n retrieval = config.prompt.retrieval\n\n # If template is NOT provided, then task is required for zero/few shot learning:\n if not template and not task:\n raise ConfigValidationError(\"A prompt task is required if no template is provided!\")\n\n template_refs = set(findall(r\"\\{(.*?)\\}\", template)) if isinstance(template, str) else set()\n\n # If a template IS provided (i.e. we are not doing a built-in zero/few-shot learning), then...\n if template:\n # If task is also provided, the template must contain it:\n if task and \"__task__\" not in template_refs:\n raise ConfigValidationError(\n \"When providing a task, you must make sure that the task keyword `{__task__} is \"\n \"present somewhere in the template string!\"\n )\n\n # If retrieval is also provided, the template must reference it:\n # TODO: retrieval by default should be set to null, not a default dict:\n if retrieval and retrieval != RetrievalConfig() and \"__context__\" not in template_refs:\n raise ConfigValidationError(\n \"When providing a retrieval config, you must make sure that the task keyword `{__context__}` is \"\n \"present somewhere in the template string!\"\n )\n\n # Otherwise, the template should at least contain the sample keyword or some input column:\n # TODO: len(template_refs) is a hacky attempt to check that there are references to *something* in the\n # string. The proper validation is to check the references against the features in the user's dataset - but we\n # do not have access to the dataset in this code path right now.\n if not task:\n if len(template_refs) == 0 and \"__sample__\" not in template_refs:\n raise ConfigValidationError(\n \"A template must contain at least one reference to a column or the sample keyword {__sample__} for \"\n \"a JSON-serialized representation of non-output feature columns.\"\n )\n\n # Raise an error if template has a placeholder for the output feature name (column).\n output_feature_col = config.output_features[0].column\n if output_feature_col in template_refs:\n raise ConfigValidationError(\n \"Prompt template should not have a reference to the output feature. The output feature is \"\n \"automatically added to the end of the prompt template merged with the input at training time.\"\n )\n\n\n@register_config_check\ndef check_sample_ratio_and_size_compatible(config: \"ModelConfig\") -> None:\n sample_ratio = config.preprocessing.sample_ratio\n sample_size = config.preprocessing.sample_size\n if sample_size is not None and sample_ratio < 1.0:\n raise ConfigValidationError(\"sample_size cannot be used when sample_ratio < 1.0\")\n"
},
{
"path": "ludwig/config_validation/preprocessing.py",
"content": "def check_global_max_sequence_length_fits_prompt_template(metadata, global_preprocessing_parameters):\n \"\"\"Checks that the prompt template fits within the global max sequence length.\"\"\"\n\n if (\n \"global_max_sequence_length\" in global_preprocessing_parameters\n and global_preprocessing_parameters[\"global_max_sequence_length\"] is not None\n ):\n for feature_name, feature_metadata in metadata.items():\n if (\n \"prompt_template_num_tokens\" in feature_metadata\n and feature_metadata[\"prompt_template_num_tokens\"]\n > global_preprocessing_parameters[\"global_max_sequence_length\"]\n ):\n raise ValueError(\n f'The prompt contains ({feature_metadata[\"prompt_template_num_tokens\"]}) tokens, which is more '\n f\"than the the global_max_sequence_length \"\n f'({global_preprocessing_parameters[\"global_max_sequence_length\"]}), which will remove all unique '\n \"information. Shorten the prompt, or increase the global max sequence length to > \"\n f'({feature_metadata[\"prompt_template_num_tokens\"]}) to include the full prompt.'\n )\n"
},
{
"path": "ludwig/config_validation/validation.py",
"content": "from functools import lru_cache\nfrom threading import Lock\n\nimport jsonschema.exceptions\nfrom jsonschema import Draft7Validator, validate\nfrom jsonschema.validators import extend\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import BASE_MODEL, MODEL_ECD, MODEL_LLM, MODEL_TYPE\nfrom ludwig.error import ConfigValidationError\n\n# TODO(travis): figure out why we need these imports to avoid circular import error\nfrom ludwig.schema.combiners.utils import get_combiner_jsonschema # noqa\nfrom ludwig.schema.features.utils import get_input_feature_jsonschema, get_output_feature_jsonschema # noqa\nfrom ludwig.schema.hyperopt import get_hyperopt_jsonschema # noqa\nfrom ludwig.schema.trainer import get_model_type_jsonschema, get_trainer_jsonschema # noqa\nfrom ludwig.schema.utils import unload_jsonschema_from_marshmallow_class\n\nVALIDATION_LOCK = Lock()\n\n\n@DeveloperAPI\n@lru_cache(maxsize=3)\ndef get_schema(model_type: str = MODEL_ECD):\n # Force populate combiner registry:\n import ludwig.combiners.combiners # noqa: F401\n from ludwig.schema.model_types.base import model_type_schema_registry\n\n cls = model_type_schema_registry[model_type]\n props = unload_jsonschema_from_marshmallow_class(cls)[\"properties\"]\n\n # TODO: Replace with more robust required logic later.\n required = [\"input_features\", \"output_features\"]\n if model_type == MODEL_LLM:\n required += [BASE_MODEL]\n\n return {\n \"type\": \"object\",\n \"properties\": props,\n \"title\": \"model_options\",\n \"description\": \"Settings for Ludwig configuration\",\n \"required\": required,\n \"additionalProperties\": True,\n }\n\n\n@lru_cache(maxsize=1)\ndef get_validator():\n # Manually add support for tuples (pending upstream changes: https://github.com/Julian/jsonschema/issues/148):\n def custom_is_array(checker, instance):\n return isinstance(instance, list) or isinstance(instance, tuple)\n\n # This creates a new class, so cache to prevent a memory leak:\n # https://github.com/python-jsonschema/jsonschema/issues/868\n type_checker = Draft7Validator.TYPE_CHECKER.redefine(\"array\", custom_is_array)\n return extend(Draft7Validator, type_checker=type_checker)\n\n\n@DeveloperAPI\ndef check_schema(updated_config):\n \"\"\"Emulates the pure JSONSchema validation that could be used in an environment without marshmallow.\n\n The incoming config may not be comprehensive, but is assumed to be up to date with the latest ludwig schema.\n \"\"\"\n model_type = updated_config.get(MODEL_TYPE, MODEL_ECD)\n error = None\n with VALIDATION_LOCK:\n try:\n validate(instance=updated_config, schema=get_schema(model_type=model_type), cls=get_validator())\n except jsonschema.exceptions.ValidationError as e:\n # Capture error but don't raise here, otherwise we get the full output from `e`, which contains a dump\n # of the entire schema\n error = e\n\n if error is not None:\n raise ConfigValidationError(f\"Failed to validate JSON schema for config. Error: {error.message}\") from error\n"
},
{
"path": "ludwig/constants.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nINPUT_FEATURES = \"input_features\"\nOUTPUT_FEATURES = \"output_features\"\n\nINPUT = \"input\"\nOUTPUT = \"output\"\nBINARY = \"binary\"\nCATEGORY = \"category\"\nCATEGORY_DISTRIBUTION = \"category_distribution\"\nINT = \"int\"\nFLOAT = \"float\"\nSPACE = \"space\"\nNUMBER = \"number\"\nSET = \"set\"\nBAG = \"bag\"\nTEXT = \"text\"\nSEQUENCE = \"sequence\"\nTIMESERIES = \"timeseries\"\nIMAGE = \"image\"\nAUDIO = \"audio\"\nDATE = \"date\"\nH3 = \"h3\"\nVECTOR = \"vector\"\nHEIGHT = \"height\"\nWIDTH = \"width\"\nINFER_IMAGE_DIMENSIONS = \"infer_image_dimensions\"\nINFER_IMAGE_MAX_HEIGHT = \"infer_image_max_height\"\nINFER_IMAGE_MAX_WIDTH = \"infer_image_max_width\"\nINFER_IMAGE_SAMPLE_SIZE = \"infer_image_sample_size\"\nINFER_IMAGE_NUM_CLASSES = \"infer_image_num_classes\"\nIMAGE_MAX_CLASSES = 128\nNUM_CLASSES = \"num_classes\"\nNUM_CHANNELS = \"num_channels\"\nREQUIRES_EQUAL_DIMENSIONS = \"requires_equal_dimensions\"\nUSE_PRETRAINED = \"use_pretrained\"\nTRAINABLE = \"trainable\"\nCLASS_WEIGHTS = \"class_weights\"\nUSED_TOKENS = \"used_tokens\"\nLOSS = \"loss\"\nROC_AUC = \"roc_auc\"\nEVAL_LOSS = \"eval_loss\"\nTRAIN_MEAN_LOSS = \"train_mean_loss\"\nSEQUENCE_SOFTMAX_CROSS_ENTROPY = \"sequence_softmax_cross_entropy\"\nNEXT_TOKEN_SOFTMAX_CROSS_ENTROPY = \"next_token_softmax_cross_entropy\"\nSOFTMAX_CROSS_ENTROPY = \"softmax_cross_entropy\"\nSIGMOID_CROSS_ENTROPY = \"sigmoid_cross_entropy\"\nBINARY_WEIGHTED_CROSS_ENTROPY = \"binary_weighted_cross_entropy\"\nTHRESHOLD = \"threshold\"\nVALIDATION_METRIC = \"validation_metric\"\nACCURACY = \"accuracy\"\nACCURACY_MICRO = \"accuracy_micro\"\nHITS_AT_K = \"hits_at_k\"\nMEAN_HITS_AT_K = \"mean_hits_at_k\"\nERROR = \"error\"\nABSOLUTE_ERROR = \"absolute_error\"\nSQUARED_ERROR = \"squared_error\"\nMEAN_SQUARED_ERROR = \"mean_squared_error\"\nROOT_MEAN_SQUARED_ERROR = \"root_mean_squared_error\"\nROOT_MEAN_SQUARED_PERCENTAGE_ERROR = \"root_mean_squared_percentage_error\"\nMEAN_ABSOLUTE_ERROR = \"mean_absolute_error\"\nMEAN_ABSOLUTE_PERCENTAGE_ERROR = \"mean_absolute_percentage_error\"\nHUBER = \"huber\"\nCORN = \"corn\"\nR2 = \"r2\"\nEDIT_DISTANCE = \"edit_distance\"\nPERPLEXITY = \"perplexity\"\nNEXT_TOKEN_PERPLEXITY = \"next_token_perplexity\"\nJACCARD = \"jaccard\"\nPRECISION = \"precision\"\nRECALL = \"recall\"\nSPECIFICITY = \"specificity\"\nPREDICTIONS = \"predictions\"\nRESPONSE = \"RESPONSE\"\nTOP_K = \"top_k\"\nTOP_K_PREDICTIONS = \"top_k_predictions\"\nPROBABILITY = \"probability\"\nPROBABILITIES = \"probabilities\"\nSPLIT_PROBABILITIES = \"split_probabilities\"\nTOKEN_ACCURACY = \"token_accuracy\"\nLAST_ACCURACY = \"last_accuracy\"\nSEQUENCE_ACCURACY = \"sequence_accuracy\"\nLAST_PROBABILITIES = \"last_probabilities\"\nLAST_PREDICTIONS = \"last_predictions\"\nLENGTHS = \"lengths\"\nTIED = \"tied\"\nCOMBINED = \"combined\"\n\nPREPROCESSING = \"preprocessing\"\nFILL_WITH_CONST = \"fill_with_const\"\nFILL_WITH_MODE = \"fill_with_mode\"\nFILL_WITH_MEAN = \"fill_with_mean\"\nFILL_WITH_FALSE = \"fill_with_false\"\nFILL_WITH_TRUE = \"fill_with_true\"\nBFILL = \"bfill\"\nFFILL = \"ffill\"\nDROP_ROW = \"drop_row\"\nMISSING_VALUE_STRATEGY = \"missing_value_strategy\"\nMISSING_VALUE_STRATEGY_OPTIONS = [\n FILL_WITH_CONST,\n FILL_WITH_MODE,\n BFILL,\n FFILL,\n DROP_ROW,\n]\n\nCROP_OR_PAD = \"crop_or_pad\"\nINTERPOLATE = \"interpolate\"\nRESIZE_METHODS = [CROP_OR_PAD, INTERPOLATE]\n\n# Special symbols for text.\nSTOP_SYMBOL = \"\"\nSTART_SYMBOL = \"\"\nPADDING_SYMBOL = \"\"\nUNKNOWN_SYMBOL = \"\"\n\nTRAINER = \"trainer\"\nOPTIMIZER = \"optimizer\"\nMETRIC = \"metric\"\nPREDICTION = \"prediction\"\nLOGITS = \"logits\"\nHIDDEN = \"hidden\"\nLAST_HIDDEN = \"last_hidden\"\nENCODER_OUTPUT = \"encoder_output\"\nENCODER_OUTPUT_STATE = \"encoder_output_state\"\nPROJECTION_INPUT = \"projection_input\"\nLEARNING_RATE_SCHEDULER = \"learning_rate_scheduler\"\n\nSEMANTIC = \"semantic\"\n\nRANDOM = \"random\"\nSUM = \"sum\"\nAPPEND = \"append\"\nSEQ_SUM = \"seq_sum\"\nAVG_EXP = \"avg_exp\"\n\nTRAIN = \"train\"\nTRAINING = \"training\"\nVALIDATION = \"validation\"\nTEST = \"test\"\nEVALUATION = \"evaluation\"\nSPLIT = \"split\"\nFORCE_SPLIT = \"force_split\"\nSTRATIFY = \"stratify\"\nFULL = \"full\"\nTRAIN_SPLIT = 0\nVALIDATION_SPLIT = 1\nTEST_SPLIT = 2\nMIN_DATASET_SPLIT_ROWS = 3 # The minimum number of rows in a split. Splits smaller than this size are treated as empty.\n\nMETA = \"meta\"\n\nHYPEROPT = \"hyperopt\"\nSTRATEGY = \"strategy\"\nEXECUTOR = \"executor\"\nMINIMIZE = \"minimize\"\nMAXIMIZE = \"maximize\"\nSAMPLER = \"sampler\"\nNUM_SAMPLES = \"num_samples\"\nSEARCH_ALG = \"search_alg\"\nSCHEDULER = \"scheduler\"\nPARAMETERS = \"parameters\"\nMAX_CONCURRENT_TRIALS = \"max_concurrent_trials\"\nCPU_RESOURCES_PER_TRIAL = \"cpu_resources_per_trial\"\nGPU_RESOURCES_PER_TRIAL = \"gpu_resources_per_trial\"\nGOAL = \"goal\"\nGRID_SEARCH = \"grid_search\"\n\nNAME = \"name\"\nCOLUMN = \"column\"\nTYPE = \"type\"\nACTIVE = \"active\"\n\nRAY = \"ray\"\nIN_MEMORY = \"in_memory\"\n\nPROC_COLUMN = \"proc_column\"\n\nCHECKSUM = \"checksum\"\n\nHDF5 = \"hdf5\"\nPARQUET = \"parquet\"\n\nSRC = \"dataset_src\"\n\nEARLY_STOP = \"early_stop\"\nEPOCHS = \"epochs\"\nBATCH_SIZE = \"batch_size\"\nEVAL_BATCH_SIZE = \"eval_batch_size\"\nEFFECTIVE_BATCH_SIZE = \"effective_batch_size\"\nMAX_BATCH_SIZE = \"max_batch_size\"\nDEFAULT_BATCH_SIZE = \"auto\"\nFALLBACK_BATCH_SIZE = 128\n# The smallest batch size that is supported on Ludwig.\nMINIMUM_BATCH_SIZE = 1\n# 2^40. Used for `max_batch_size` config param. Not a hard constraint for `batch_size` config param.\nMAX_POSSIBLE_BATCH_SIZE = 1099511627776\n# min batch size. Used as a floor for batch size tuning.\nMIN_POSSIBLE_BATCH_SIZE = 1\n# max batch size for dataset is 20% of dataset size\nMAX_BATCH_SIZE_DATASET_FRACTION = 0.2\nMAX_CPU_BATCH_SIZE = 128\nLEARNING_RATE = \"learning_rate\"\nINPUT_SIZE = \"input_size\"\nUSE_BIAS = \"use_bias\"\nBIAS = \"bias\"\nDEFAULT_USE_BIAS = \"default_use_bias\"\nDEFAULT_BIAS = \"default_bias\"\nCONV_USE_BIAS = \"conv_use_bias\"\nCONV_BIAS = \"conv_bias\"\nAUTO = \"auto\"\nCONFIG = \"config\"\n\nCLIP = \"clip\"\nDEPENDENCIES = \"dependencies\"\nREDUCE_INPUT = \"reduce_input\"\nREDUCE_DEPENDENCIES = \"reduce_dependencies\"\n\nBACKEND = \"backend\"\nCOMBINER = \"combiner\"\n\nENCODER = \"encoder\"\nDECODER = \"decoder\"\n\nTRAINABLE = \"trainable\"\n\nDEFAULTS = \"defaults\"\nDEFAULT = \"default\"\nDEFAULT_VALIDATION_METRIC = \"default_validation_metric\"\n\nBALANCE_PERCENTAGE_TOLERANCE = 0.03\nIMBALANCE_DETECTION_RATIO = 0.05\n\nTABULAR = \"tabular\"\nAUTOML_DEFAULT_TABULAR_MODEL = \"tabnet\"\nAUTOML_DEFAULT_TEXT_ENCODER = \"bert\"\nAUTOML_SMALLER_TEXT_ENCODER = \"distilbert\"\nAUTOML_TEXT_ENCODER_MAX_TOKEN_LEN = 512\nAUTOML_SMALLER_TEXT_LENGTH = 128\nAUTOML_LARGE_TEXT_DATASET = 100000\nAUTOML_MAX_ROWS_PER_CHECKPOINT = 350000\nAUTOML_DEFAULT_IMAGE_ENCODER = \"stacked_cnn\"\n\nHYPEROPT_WARNING = (\n \"You are running the ludwig train command but there’s a hyperopt section present in your config. \"\n \"It will be ignored. If you want to run hyperopt you should use the following command: ludwig \"\n \"hyperopt\\n\\n\"\n)\n\nCONTINUE_PROMPT = \"Do you want to continue? \"\n\nDEFAULT_AUDIO_TENSOR_LENGTH = 70000\nAUDIO_FEATURE_KEYS = [\n \"type\",\n \"window_length_in_s\",\n \"window_shift_in_s\",\n \"num_fft_points\",\n \"window_type\",\n \"num_filter_bands\",\n]\n\nBASE_MODEL = \"base_model\"\nMODEL_TYPE = \"model_type\"\nMODEL_ECD = \"ecd\"\nMODEL_LLM = \"llm\"\nDASK_MODULE_NAME = \"dask.dataframe\"\nLUDWIG_VERSION = \"ludwig_version\"\n\nPREPROCESSOR = \"preprocessor\"\nPREDICTOR = \"predictor\"\nPOSTPROCESSOR = \"postprocessor\"\nTARGET_MODULES = \"target_modules\"\n\nGENERATION = \"generation\"\nPROMPT = \"prompt\"\nADAPTER = \"adapter\"\nQUANTIZATION = \"quantization\"\nMIN_QUANTIZATION_BITS_FOR_MERGE_AND_UNLOAD = 8\nPRETRAINED_ADAPTER_WEIGHTS = \"pretrained_adapter_weights\"\nMERGE_ADAPTER_INTO_BASE_MODEL = \"merge_adapter_into_base_model\"\nPROGRESSBAR = \"progressbar\"\n\n# CrossEntropyLoss for LLMs\nIGNORE_INDEX_TOKEN_ID = -100\n\nS3 = \"s3\"\nCACHE = \"cache\"\n\n# If `use_torch_profiler=True` in LudwigProfiler, LUDWIG_TAG is prepended to the specified experiment tag\n# (LudwigProfiler(tag=\"...\", ..)). This edited tag is passed in to `torch.profiler.record_function` so we can\n# retrieve torch ops for the tagged code blocks/functions.\nLUDWIG_TAG = \"[ludwig]\"\n\n# Retry constants\nTRIES = 5\nDELAY = 1\nBACKOFF = 2\nJITTER = (0, 1)\n\n# image support constants\nIMAGENET1K = \"imagenet1k\"\n\nAUGMENTATION = \"augmentation\"\n\nLUDWIG_SCHEMA_VALIDATION_POLICY = \"LUDWIG_SCHEMA_VALIDATION_POLICY\"\n"
},
{
"path": "ludwig/contrib.py",
"content": "# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\"\"\"Module for handling contributed support.\"\"\"\n\nimport argparse\n\nfrom ludwig.contribs import contrib_registry, ContribLoader\n\n\ndef create_load_action(contrib_loader: ContribLoader) -> argparse.Action:\n class LoadContribAction(argparse.Action):\n def __call__(self, parser, namespace, values, option_string):\n items = getattr(namespace, self.dest) or []\n items.append(contrib_loader.load())\n setattr(namespace, self.dest, items)\n\n return LoadContribAction\n\n\ndef add_contrib_callback_args(parser: argparse.ArgumentParser):\n for contrib_name, contrib_loader in contrib_registry.items():\n parser.add_argument(\n f\"--{contrib_name}\",\n dest=\"callbacks\",\n nargs=0,\n action=create_load_action(contrib_loader),\n )\n\n\ndef preload(argv):\n for arg in argv:\n if arg.startswith(\"--\"):\n arg = arg[2:]\n\n if arg in contrib_registry:\n contrib_registry[arg].preload()\n"
},
{
"path": "ludwig/contribs/__init__.py",
"content": "# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\"\"\"All contrib classes must implement the `ludwig.callbacks.Callback` interface.\n\nIf you don't want to handle the call, either provide an empty method with `pass`, or just don't implement the method.\n\"\"\"\n\nfrom abc import ABC, abstractmethod\n\nfrom ludwig.callbacks import Callback\n\n\nclass ContribLoader(ABC):\n @abstractmethod\n def load(self) -> Callback:\n \"\"\"Returns an instantiation of the callback instance, whose callback hooks will be invoked at runtime.\"\"\"\n\n def preload(self):\n \"\"\"Will always be called when Ludwig CLI is invoked, preload gives the callback an opportunity to import or\n create any shared resources.\n\n Importing required 3rd-party libraries should be done here i.e. import wandb. preload is guaranteed to be called\n before any other callback method, and will only be called once per process.\n \"\"\"\n\n\n# Contributors, load your class here:\n\n\nclass AimLoader(ContribLoader):\n def load(self) -> Callback:\n from ludwig.contribs.aim import AimCallback\n\n return AimCallback()\n\n def preload(self):\n import aim # noqa\n\n\nclass CometLoader(ContribLoader):\n def load(self) -> Callback:\n from ludwig.contribs.comet import CometCallback\n\n return CometCallback()\n\n def preload(self):\n import comet_ml # noqa\n\n\nclass WandbLoader(ContribLoader):\n def load(self) -> Callback:\n from ludwig.contribs.wandb import WandbCallback\n\n return WandbCallback()\n\n def preload(self):\n import wandb # noqa\n\n\nclass MlflowLoader(ContribLoader):\n def load(self) -> Callback:\n from ludwig.contribs.mlflow import MlflowCallback\n\n return MlflowCallback()\n\n\ncontrib_registry = {\n # Contributors, add your class here:\n \"comet\": CometLoader(),\n \"wandb\": WandbLoader(),\n \"mlflow\": MlflowLoader(),\n \"aim\": AimLoader(),\n}\n"
},
{
"path": "ludwig/contribs/aim.py",
"content": "import json\nimport logging\n\nfrom ludwig.api_annotations import PublicAPI\nfrom ludwig.callbacks import Callback\nfrom ludwig.utils.data_utils import NumpyEncoder\nfrom ludwig.utils.package_utils import LazyLoader\n\naim = LazyLoader(\"aim\", globals(), \"aim\")\n\nlogger = logging.getLogger(__name__)\n\n\n@PublicAPI\nclass AimCallback(Callback):\n \"\"\"Class that defines the methods necessary to hook into process.\"\"\"\n\n def __init__(self, repo=None):\n self.repo = repo\n\n def on_train_init(\n self,\n base_config,\n experiment_directory,\n experiment_name,\n model_name,\n output_directory,\n resume_directory,\n ):\n logger.info(\"aim.on_train_init() called...\")\n\n try:\n query = f'run.name == \"{model_name}\"'\n if self.repo is None:\n aim_repo = aim.Repo.default_repo()\n else:\n aim_repo = aim.Repo.from_path(self.repo)\n runs_generator = aim_repo.query_runs(query)\n run = next(runs_generator.iter_runs())\n run_hash = run.run.hash\n self.aim_run = aim.Run(run_hash=run_hash, repo=self.repo, experiment=experiment_name)\n except Exception:\n self.aim_run = aim.Run(repo=self.repo, experiment=experiment_name)\n self.aim_run.name = model_name\n\n self.aim_run[\"base_config\"] = self.normalize_config(base_config)\n\n params = dict(name=model_name, dir=experiment_directory)\n self.aim_run[\"params\"] = params\n\n def aim_track(self, progress_tracker):\n logger.info(f\"aim.aim_track() called for epoch {progress_tracker.epoch}, step: {progress_tracker.steps}\")\n\n if self.aim_run:\n for key, value in progress_tracker.log_metrics().items():\n if \"metrics\" in key and \"best\" not in key:\n metrics_dict_name, feature_name, metric_name = key.split(\".\")\n\n self.aim_run.track(\n value,\n name=metric_name,\n context={metrics_dict_name: feature_name},\n epoch=progress_tracker.epoch,\n step=progress_tracker.steps,\n )\n\n def on_trainer_train_teardown(self, trainer, progress_tracker, save_path, is_coordinator: bool):\n pass\n\n def on_train_start(self, model, config, *args, **kwargs):\n logger.info(\"aim.on_train_start() called...\")\n\n config = config.copy()\n del config[\"input_features\"]\n del config[\"output_features\"]\n\n self.aim_run[\"train_config\"] = self.normalize_config(config)\n\n def on_train_end(self, output_directory, *args, **kwargs):\n pass\n\n def on_eval_end(self, trainer, progress_tracker, save_path):\n optimizer_config = {}\n for index, group in enumerate(trainer.optimizer.param_groups):\n for key in group:\n if \"param\" not in key:\n optimizer_config[f\"param_group_{index}_{key}\"] = group[key]\n\n self.aim_run[\"optimizer_config\"] = self.normalize_config(optimizer_config)\n\n self.aim_track(progress_tracker)\n\n def on_ludwig_end(self):\n self.aim_run.close()\n self.aim_run = None\n\n def on_visualize_figure(self, fig):\n logger.info(\"aim.on_visualize_figure() called...\")\n if self.aim_run:\n self.aim_run.track(aim.Figure(fig), name=\"Figure\", context={\"type\": \"Training Figure\"})\n\n @staticmethod\n def normalize_config(config):\n \"\"\"Convert to json string and back again to remove numpy types.\"\"\"\n return json.loads(json.dumps(config, cls=NumpyEncoder))\n"
},
{
"path": "ludwig/contribs/comet.py",
"content": "# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nimport os\nfrom datetime import datetime\n\nfrom ludwig.api_annotations import PublicAPI\nfrom ludwig.callbacks import Callback\nfrom ludwig.utils.package_utils import LazyLoader\n\ncomet_ml = LazyLoader(\"comet_ml\", globals(), \"comet_ml\")\n\nlogger = logging.getLogger(__name__)\n\n\n@PublicAPI\nclass CometCallback(Callback):\n \"\"\"Class that defines the methods necessary to hook into process.\"\"\"\n\n def __init__(self):\n self.cometml_experiment = None\n\n def on_train_init(\n self,\n base_config,\n experiment_directory,\n experiment_name,\n model_name,\n output_directory,\n resume_directory,\n ):\n if self.cometml_experiment:\n # Comet ML already initialized\n return\n\n try:\n self.cometml_experiment = comet_ml.Experiment(log_code=False, project_name=experiment_name)\n except Exception:\n self.cometml_experiment = None\n logger.exception(\"comet_ml.Experiment() had errors. Perhaps you need to define COMET_API_KEY\")\n raise\n\n self.cometml_experiment.set_name(model_name)\n self.cometml_experiment.set_filename(\"Ludwig API\")\n config = comet_ml.get_config()\n self._save_config(config, directory=experiment_directory)\n\n def on_train_start(self, model, config, config_fp, *args, **kwargs):\n if self.cometml_experiment:\n # todo v0.4: currently not clear way to set model graph\n # see: https://github.com/comet-ml/issue-tracking/issues/296\n # if model:\n # self.cometml_experiment.set_model_graph(\n # str(model._graph.as_graph_def()))\n\n if config:\n if config_fp:\n base_name = os.path.basename(config_fp)\n else:\n base_name = \"config.yaml\"\n if \".\" in base_name:\n base_name = base_name.rsplit(\".\", 1)[0] + \".json\"\n else:\n base_name = base_name + \".json\"\n self.cometml_experiment.log_asset_data(config, base_name)\n\n def on_train_end(self, output_directory, *args, **kwargs):\n if self.cometml_experiment:\n self.cometml_experiment.log_asset_folder(output_directory)\n\n def on_eval_end(self, trainer, progress_tracker, save_path):\n \"\"\"Called from ludwig/models/model.py.\"\"\"\n if self.cometml_experiment:\n for key, value in progress_tracker.log_metrics().items():\n self.cometml_experiment.log_metric(key, value)\n\n def on_epoch_end(self, trainer, progress_tracker, save_path):\n \"\"\"Called from ludwig/models/model.py.\"\"\"\n if self.cometml_experiment:\n for key, value in progress_tracker.log_metrics().items():\n self.cometml_experiment.log_metric(key, value)\n\n def on_visualize_figure(self, fig):\n if self.cometml_experiment:\n self.cometml_experiment.log_figure(fig)\n\n def on_cmdline(self, cmd, *args):\n self.cometml_experiment = None\n if cmd in {\"train\", \"experiment\"}:\n # create a new experiment\n try:\n self.cometml_experiment = comet_ml.Experiment(log_code=False)\n except Exception:\n logger.exception(\"comet_ml.Experiment() had errors. Perhaps you need to define COMET_API_KEY\")\n return\n elif cmd in {\"visualize\", \"predict\", \"evaluate\"}:\n # restore from an existing experiment\n try:\n self.cometml_experiment = comet_ml.ExistingExperiment()\n except Exception:\n logger.exception(\"Ignored --comet. No '.comet.config' file\")\n return\n else:\n # unhandled command\n return\n\n cli = self._make_command_line(cmd, args)\n self.cometml_experiment.set_code(cli)\n self.cometml_experiment.set_filename(\"Ludwig CLI\")\n self._log_html(cli)\n config = comet_ml.get_config()\n self._save_config(config)\n\n def _save_config(self, config, directory=\".\"):\n # save the .comet.config here:\n config[\"comet.experiment_key\"] = self.cometml_experiment.id\n config.save(directory=directory)\n\n def _log_html(self, text):\n # log the text to the html tab:\n now = datetime.now()\n timestamp = now.strftime(\"%m/%d/%Y %H:%M:%S\")\n self.cometml_experiment.log_html(f\"
{timestamp}: {text}
\")\n\n def _make_command_line(self, cmd, args):\n # put the commet flag back in:\n arg_str = \" \".join(list(args[:2]) + [\"--comet\"] + list(args[2:]))\n return f\"ludwig {cmd} {arg_str}\"\n"
},
{
"path": "ludwig/contribs/mlflow/__init__.py",
"content": "import logging\nimport os\nimport queue\nimport threading\n\nfrom ludwig.api_annotations import DeveloperAPI, PublicAPI\nfrom ludwig.callbacks import Callback\nfrom ludwig.constants import TRAINER\nfrom ludwig.globals import MODEL_FILE_NAME, MODEL_HYPERPARAMETERS_FILE_NAME, TRAIN_SET_METADATA_FILE_NAME\nfrom ludwig.types import TrainingSetMetadataDict\nfrom ludwig.utils.data_utils import chunk_dict, flatten_dict, save_json, to_json_dict\nfrom ludwig.utils.package_utils import LazyLoader\n\nmlflow = LazyLoader(\"mlflow\", globals(), \"mlflow\")\n\nlogger = logging.getLogger(__name__)\n\n\ndef _get_runs(experiment_id: str):\n return mlflow.tracking.client.MlflowClient().search_runs([experiment_id])\n\n\n@DeveloperAPI\ndef get_or_create_experiment_id(experiment_name, artifact_uri: str = None):\n \"\"\"Gets experiment id from mlflow.\"\"\"\n experiment = mlflow.get_experiment_by_name(experiment_name)\n if experiment is not None:\n return experiment.experiment_id\n return mlflow.create_experiment(name=experiment_name, artifact_location=artifact_uri)\n\n\n# Included for backwards compatibility, Deprecated.\n# TODO(daniel): delete this.\n_get_or_create_experiment_id = get_or_create_experiment_id\n\n\n@PublicAPI\nclass MlflowCallback(Callback):\n def __init__(self, tracking_uri=None, log_artifacts: bool = True):\n self.logged_steps = set()\n\n if tracking_uri:\n mlflow.set_tracking_uri(tracking_uri)\n self.tracking_uri = mlflow.get_tracking_uri()\n\n active_run = mlflow.active_run()\n if active_run is not None:\n # Use experiment already set in the current environment\n self.run = active_run\n self.experiment_id = self.run.info.experiment_id\n self.experiment_name = mlflow.get_experiment(self.experiment_id).name\n self.external_run = True\n else:\n # Will create an experiment at training time\n self.run = None\n self.experiment_id = None\n self.experiment_name = None\n self.external_run = False\n\n self.run_ended = False\n self.training_set_metadata = None\n self.config = None\n self.save_in_background = True\n self.save_fn = None\n self.save_thread = None\n self.log_artifacts = log_artifacts\n\n def get_experiment_id(self, experiment_name):\n return get_or_create_experiment_id(experiment_name)\n\n def on_preprocess_end(\n self,\n training_set: \"Dataset\", # noqa\n validation_set: \"Dataset\", # noqa\n test_set: \"Dataset\", # noqa\n training_set_metadata: TrainingSetMetadataDict,\n ):\n self.training_set_metadata = training_set_metadata\n\n def on_hyperopt_init(self, experiment_name):\n self.experiment_id = self.get_experiment_id(experiment_name)\n self.experiment_name = experiment_name\n\n def on_hyperopt_trial_start(self, parameters):\n # Filter out mlflow params like tracking URI, experiment ID, etc.\n params = {k: v for k, v in parameters.items() if k != \"mlflow\"}\n self._log_params({\"hparam\": params})\n\n # TODO(travis): figure out a good way to support this. The problem with\n # saving artifacts in the background with hyperopt is early stopping. If\n # the scheduler decides to terminate a process, then currently there's no\n # mechanism to detect this a \"flush\" the queue of pending writes before\n # stopping. Should work with Ray Tune team to come up with a solution.\n self.save_in_background = False\n\n def on_train_init(self, base_config, experiment_name, output_directory, resume_directory, **kwargs):\n # Experiment may already have been set during hyperopt init, in\n # which case we don't want to create a new experiment / run, as\n # this should be handled by the executor.\n if self.experiment_id is None:\n mlflow.end_run()\n self.experiment_id = self.get_experiment_id(experiment_name)\n self.experiment_name = experiment_name\n\n active_run = mlflow.active_run()\n if active_run is not None:\n # Currently active run started by Ray Tune MLflow mixin or external run\n self.run = active_run\n else:\n run_id = None\n if resume_directory is not None:\n previous_runs = _get_runs(self.experiment_id)\n if len(previous_runs) > 0:\n run_id = previous_runs[0].info.run_id\n if run_id is not None:\n self.run = mlflow.start_run(run_id=run_id)\n else:\n run_name = os.path.basename(output_directory)\n self.run = mlflow.start_run(experiment_id=self.experiment_id, run_name=run_name)\n\n self.log_config(base_config)\n\n def log_config(self, config):\n if self.log_artifacts:\n mlflow.log_dict(to_json_dict(config), \"config.yaml\")\n\n def on_train_start(self, config, **kwargs):\n self.config = config\n self._log_params({TRAINER: config[TRAINER]})\n\n def on_train_end(self, output_directory):\n if self.log_artifacts:\n _log_artifacts(output_directory)\n if self.run is not None and not self.external_run:\n # Only end runs managed internally to this callback\n mlflow.end_run()\n self.run_ended = True\n\n def on_trainer_train_setup(self, trainer, save_path, is_coordinator):\n if not is_coordinator:\n return\n\n # When running on a remote worker, the model metadata files will only have been\n # saved to the driver process, so re-save it here before uploading.\n training_set_metadata_path = os.path.join(save_path, TRAIN_SET_METADATA_FILE_NAME)\n if not os.path.exists(training_set_metadata_path):\n save_json(training_set_metadata_path, self.training_set_metadata)\n\n model_hyperparameters_path = os.path.join(save_path, MODEL_HYPERPARAMETERS_FILE_NAME)\n if not os.path.exists(model_hyperparameters_path):\n save_json(model_hyperparameters_path, self.config)\n\n if self.save_in_background:\n save_queue = queue.Queue()\n self.save_fn = lambda args: save_queue.put(args)\n self.save_thread = threading.Thread(target=_log_mlflow_loop, args=(save_queue, self.log_artifacts))\n self.save_thread.start()\n else:\n self.save_fn = lambda args: _log_mlflow(*args, self.log_artifacts)\n\n def on_eval_end(self, trainer, progress_tracker, save_path):\n if progress_tracker.steps not in self.logged_steps:\n self.logged_steps.add(progress_tracker.steps)\n # Adds a tuple to the logging queue.\n # True is passed to indicate that the background saving loop should continue.\n self.save_fn((progress_tracker.log_metrics(), progress_tracker.steps, save_path, True))\n\n def on_trainer_train_teardown(self, trainer, progress_tracker, save_path, is_coordinator):\n if is_coordinator:\n if progress_tracker.steps not in self.logged_steps:\n self.logged_steps.add(progress_tracker.steps)\n # Adds a tuple to the logging queue.\n # False is passed to indicate that the background saving loop should break.\n self.save_fn((progress_tracker.log_metrics(), progress_tracker.steps, save_path, False))\n # False ensures that the background saving loop breaks.\n # TODO(Justin): This should probably live in on_ludwig_end, once that's implemented.\n self.save_fn((None, None, None, False))\n\n # Close the save_thread.\n if self.save_thread is not None:\n self.save_thread.join()\n # if self.save_thread.is_alive():\n # logger.warning(\"MLFlow save thread timed out and did not close properly.\")\n\n def on_visualize_figure(self, fig):\n # TODO: need to also include a filename for this figure\n # mlflow.log_figure(fig)\n pass\n\n def prepare_ray_tune(self, train_fn, tune_config, tune_callbacks):\n from functools import wraps\n\n from ray.air.integrations.mlflow import setup_mlflow\n\n mlflow_config = {\n \"experiment_id\": self.experiment_id,\n \"experiment_name\": self.experiment_name,\n \"tracking_uri\": mlflow.get_tracking_uri(),\n }\n\n @wraps(train_fn)\n def wrapper(config, **kwargs):\n setup_mlflow(config, **mlflow_config)\n return train_fn(config, **kwargs)\n\n return wrapper, {\n **tune_config,\n }\n\n def _log_params(self, params):\n flat_params = flatten_dict(params)\n for chunk in chunk_dict(flat_params, chunk_size=100):\n mlflow.log_params(chunk)\n\n def __setstate__(self, d):\n self.__dict__ = d\n if self.tracking_uri:\n mlflow.set_tracking_uri(self.tracking_uri)\n if self.run and not self.run_ended:\n # Run has already been set, but may not be active due to training workers running in a separate\n # process, so resume the run\n mlflow.end_run()\n self.run = mlflow.start_run(run_id=self.run.info.run_id, experiment_id=self.run.info.experiment_id)\n\n\ndef _log_mlflow_loop(q: queue.Queue, log_artifacts: bool = True):\n \"\"\"The save_fn for the background thread that logs to MLFlow when save_in_background is True.\"\"\"\n should_continue = True\n while should_continue:\n elem = q.get()\n log_metrics, steps, save_path, should_continue = elem\n if log_metrics is None:\n # Break out of the loop if we're not going to log anything.\n break\n\n if \"llm_eval_examples\" in log_metrics and log_metrics[\"llm_eval_examples\"] is not None:\n # mlflow.log_dict(log_metrics[\"llm_eval_examples\"], artifact_file=\"llm_eval_examples.json\")\n # Delete the table from the metrics dict so we don't try to log it with the other metrics\n del log_metrics[\"llm_eval_examples\"]\n mlflow.log_metrics(log_metrics, step=steps)\n\n if not q.empty():\n # in other words, don't bother saving the model artifacts\n # if we're about to do it again\n continue\n\n if log_artifacts:\n _log_model(save_path)\n\n\ndef _log_mlflow(log_metrics, steps, save_path, should_continue, log_artifacts: bool = True):\n \"\"\"The save_fn for the MlflowCallback.\n\n This is used when save_in_background is False.\n \"\"\"\n if log_metrics is not None:\n if \"llm_eval_examples\" in log_metrics and log_metrics[\"llm_eval_examples\"] is not None:\n # mlflow.log_dict(log_metrics[\"llm_eval_examples\"], artifact_file=\"llm_eval_examples.json\")\n # Delete the table from the metrics dict so we don't try to log it with the other metrics\n del log_metrics[\"llm_eval_examples\"]\n mlflow.log_metrics(log_metrics, step=steps)\n if log_artifacts:\n _log_model(save_path)\n\n\ndef _log_artifacts(output_directory):\n try:\n contents = os.listdir(output_directory)\n except FileNotFoundError:\n logger.warning(f\"_log_artifacts: output_directory does not exist: {output_directory}\")\n return\n for fname in contents:\n lpath = os.path.join(output_directory, fname)\n if fname == MODEL_FILE_NAME:\n _log_model(lpath)\n else:\n mlflow.log_artifact(lpath)\n\n\ndef _log_model(lpath):\n # Lazy import to avoid requiring this package\n from ludwig.contribs.mlflow.model import log_saved_model\n\n log_saved_model(lpath)\n"
},
{
"path": "ludwig/contribs/mlflow/model.py",
"content": "import logging\nimport os\nimport shutil\nimport tempfile\n\nimport mlflow\nimport yaml\nfrom mlflow import pyfunc\nfrom mlflow.exceptions import MlflowException\nfrom mlflow.models import Model\nfrom mlflow.models.model import MLMODEL_FILE_NAME\nfrom mlflow.models.signature import ModelSignature\nfrom mlflow.models.utils import _save_example, ModelInputExample\nfrom mlflow.tracking._model_registry import DEFAULT_AWAIT_MAX_SLEEP_SECONDS\nfrom mlflow.tracking.artifact_utils import _download_artifact_from_uri\nfrom mlflow.utils.environment import _mlflow_conda_env\nfrom mlflow.utils.model_utils import _get_flavor_configuration\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.globals import MODEL_FILE_NAME, MODEL_HYPERPARAMETERS_FILE_NAME\nfrom ludwig.utils.data_utils import load_json\n\nFLAVOR_NAME = \"ludwig\"\n\n_logger = logging.getLogger(__name__)\n\n\ndef get_default_conda_env():\n \"\"\"\n :return: The default Conda environment for MLflow Models produced by calls to\n :func:`save_model()` and :func:`log_model()`.\n \"\"\"\n import ludwig\n\n # Ludwig is not yet available via the default conda channels, so we install it via pip\n return _mlflow_conda_env(\n additional_conda_deps=None,\n additional_pip_deps=[f\"ludwig=={ludwig.__version__}\"],\n additional_conda_channels=None,\n )\n\n\ndef save_model(\n ludwig_model,\n path,\n conda_env=None,\n mlflow_model=None,\n signature: ModelSignature = None,\n input_example: ModelInputExample = None,\n **kwargs,\n):\n \"\"\"Save a Ludwig model to a path on the local file system.\n\n :param ludwig_model: Ludwig model (an instance of `ludwig.api.LudwigModel`_) to be saved.\n :param path: Local path where the model is to be saved.\n :param conda_env: Either a dictionary representation of a Conda environment or the path to a\n Conda environment yaml file. If provided, this describes the environment\n this model should be run in. At minimum, it should specify the dependencies\n contained in :func:`get_default_conda_env()`. If ``None``, the default\n :func:`get_default_conda_env()` environment is added to the model.\n The following is an *example* dictionary representation of a Conda\n environment::\n\n {\n 'name': 'mlflow-env',\n 'channels': ['defaults'],\n 'dependencies': [\n 'python=3.7.0',\n 'pip': [\n 'ludwig==0.4.0'\n ]\n ]\n }\n\n :param mlflow_model: :py:mod:`mlflow.models.Model` this flavor is being added to.\n\n :param signature: (Experimental) :py:class:`ModelSignature `\n describes model input and output :py:class:`Schema `.\n The model signature can be :py:func:`inferred `\n from datasets with valid model input (e.g. the training dataset with target\n column omitted) and valid model output (e.g. model predictions generated on\n the training dataset), for example:\n\n .. code-block:: python\n\n from mlflow.models.signature import infer_signature\n\n train = df.drop_column(\"target_label\")\n predictions = ... # compute model predictions\n signature = infer_signature(train, predictions)\n :param input_example: (Experimental) Input example provides one or several instances of valid\n model input. The example can be used as a hint of what data to feed the\n model. The given example will be converted to a Pandas DataFrame and then\n serialized to json using the Pandas split-oriented format. Bytes are\n base64-encoded.\n \"\"\"\n import ludwig\n\n path = os.path.abspath(path)\n if os.path.exists(path):\n raise MlflowException(f\"Path '{path}' already exists\")\n model_data_subpath = MODEL_FILE_NAME\n model_data_path = os.path.join(path, model_data_subpath)\n os.makedirs(path)\n if mlflow_model is None:\n mlflow_model = Model()\n if signature is not None:\n mlflow_model.signature = signature\n if input_example is not None:\n _save_example(mlflow_model, input_example, path)\n\n # Save the Ludwig model\n ludwig_model.save(model_data_path)\n\n conda_env_subpath = \"conda.yaml\"\n if conda_env is None:\n conda_env = get_default_conda_env()\n elif not isinstance(conda_env, dict):\n with open(conda_env) as f:\n conda_env = yaml.safe_load(f)\n with open(os.path.join(path, conda_env_subpath), \"w\") as f:\n yaml.safe_dump(conda_env, stream=f, default_flow_style=False)\n\n pyfunc.add_to_model(\n mlflow_model,\n loader_module=\"ludwig.contribs.mlflow.model\",\n data=model_data_subpath,\n env=conda_env_subpath,\n )\n\n schema_keys = {\"name\", \"column\", \"type\"}\n config = ludwig_model.config\n\n mlflow_model.add_flavor(\n FLAVOR_NAME,\n ludwig_version=ludwig.__version__,\n ludwig_schema={\n \"input_features\": [\n {k: v for k, v in feature.items() if k in schema_keys} for feature in config[\"input_features\"]\n ],\n \"output_features\": [\n {k: v for k, v in feature.items() if k in schema_keys} for feature in config[\"output_features\"]\n ],\n },\n data=model_data_subpath,\n )\n mlflow_model.save(os.path.join(path, MLMODEL_FILE_NAME))\n\n\ndef log_model(\n ludwig_model,\n artifact_path,\n conda_env=None,\n registered_model_name=None,\n signature: ModelSignature = None,\n input_example: ModelInputExample = None,\n await_registration_for=DEFAULT_AWAIT_MAX_SLEEP_SECONDS,\n):\n \"\"\"Log a Ludwig model as an MLflow artifact for the current run.\n\n Saves the model locally in MLflow format, then logs it as a run artifact using mlflow.log_artifacts(). This ensures\n the model appears as a run artifact (compatible with MLflow 3.x where Model.log() uses the model registry instead).\n \"\"\"\n with tempfile.TemporaryDirectory() as tmpdir:\n local_path = os.path.join(tmpdir, \"model\")\n save_model(\n ludwig_model,\n path=local_path,\n conda_env=conda_env,\n signature=signature,\n input_example=input_example,\n )\n mlflow.log_artifacts(local_path, artifact_path)\n\n if registered_model_name is not None:\n run_id = mlflow.active_run().info.run_id\n mlflow.register_model(\n f\"runs:/{run_id}/{artifact_path}\",\n registered_model_name,\n await_registration_for=await_registration_for,\n )\n\n\ndef _load_model(path):\n from ludwig.api import LudwigModel\n\n return LudwigModel.load(path, backend=\"local\")\n\n\ndef _load_pyfunc(path):\n \"\"\"Load PyFunc implementation. Called by ``pyfunc.load_pyfunc``.\n\n :param path: Local filesystem path to the MLflow Model with the ``ludwig`` flavor.\n \"\"\"\n return _LudwigModelWrapper(_load_model(path))\n\n\ndef load_model(model_uri):\n \"\"\"Load a Ludwig model from a local file or a run.\n\n :param model_uri: The location, in URI format, of the MLflow model. For example:\n\n - ``/Users/me/path/to/local/model``\n - ``relative/path/to/local/model``\n - ``s3://my_bucket/path/to/model``\n - ``runs://run-relative/path/to/model``\n\n For more information about supported URI schemes, see\n `Referencing Artifacts `_.\n\n :return: A Ludwig model (an instance of `ludwig.api.LudwigModel`_).\n \"\"\"\n local_model_path = _download_artifact_from_uri(artifact_uri=model_uri)\n flavor_conf = _get_flavor_configuration(model_path=local_model_path, flavor_name=FLAVOR_NAME)\n model_data_path = os.path.join(local_model_path, flavor_conf.get(\"data\", \"model\"))\n return _load_model(path=model_data_path)\n\n\nclass _LudwigModelWrapper:\n def __init__(self, ludwig_model):\n self.ludwig_model = ludwig_model\n\n def predict(self, dataframe):\n pred_df, _ = self.ludwig_model.predict(dataframe)\n return pred_df\n\n\ndef export_model(model_path, output_path, registered_model_name=None):\n if registered_model_name:\n if not model_path.startswith(\"runs:/\") or output_path is not None:\n # No run specified, so in order to register the model in mlflow, we need\n # to create a new run and upload the model as an artifact first\n output_path = output_path or MODEL_FILE_NAME\n log_model(\n _CopyModel(model_path),\n artifact_path=output_path,\n registered_model_name=registered_model_name,\n )\n else:\n # Registering a model from an artifact of an existing run\n mlflow.register_model(\n model_path,\n registered_model_name,\n )\n else:\n # No model name means we only want to save the model locally\n save_model(\n _CopyModel(model_path),\n path=output_path,\n )\n\n\n@DeveloperAPI\ndef log_saved_model(lpath):\n \"\"\"Log a saved Ludwig model directory as a proper MLflow model artifact.\"\"\"\n if os.path.isdir(lpath):\n log_model(\n _CopyModel(lpath),\n artifact_path=\"model\",\n )\n elif os.path.isfile(lpath):\n mlflow.log_artifact(lpath, \"model\")\n\n\nclass _CopyModel:\n \"\"\"Get model data without requiring us to read the model weights into memory.\"\"\"\n\n def __init__(self, lpath):\n self.lpath = lpath\n\n def save(self, path):\n shutil.copytree(self.lpath, path)\n\n @property\n def config(self):\n return load_json(os.path.join(self.lpath, MODEL_HYPERPARAMETERS_FILE_NAME))\n"
},
{
"path": "ludwig/contribs/wandb.py",
"content": "# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nimport os\n\nfrom ludwig.api_annotations import PublicAPI\nfrom ludwig.callbacks import Callback\nfrom ludwig.utils.package_utils import LazyLoader\n\nwandb = LazyLoader(\"wandb\", globals(), \"wandb\")\n\nlogger = logging.getLogger(__name__)\n\n\n@PublicAPI\nclass WandbCallback(Callback):\n \"\"\"Class that defines the methods necessary to hook into process.\"\"\"\n\n def on_train_init(\n self,\n base_config,\n experiment_directory,\n experiment_name,\n model_name,\n output_directory,\n resume_directory,\n ):\n logger.info(\"wandb.on_train_init() called...\")\n wandb.init(\n project=os.getenv(\"WANDB_PROJECT\", experiment_name),\n name=model_name,\n sync_tensorboard=True,\n dir=output_directory,\n )\n wandb.save(os.path.join(experiment_directory, \"*\"))\n\n def on_train_start(self, model, config, *args, **kwargs):\n logger.info(\"wandb.on_train_start() called...\")\n config = config.copy()\n del config[\"input_features\"]\n del config[\"output_features\"]\n wandb.config.update(config)\n\n def on_eval_end(self, trainer, progress_tracker, save_path):\n \"\"\"Called from ludwig/models/model.py.\"\"\"\n for key, value in progress_tracker.log_metrics().items():\n wandb.log({key: value})\n\n def on_epoch_end(self, trainer, progress_tracker, save_path):\n \"\"\"Called from ludwig/models/model.py.\"\"\"\n for key, value in progress_tracker.log_metrics().items():\n wandb.log({key: value})\n\n def on_visualize_figure(self, fig):\n logger.info(\"wandb.on_visualize_figure() called...\")\n if wandb.run:\n wandb.log({\"figure\": fig})\n\n def on_train_end(self, output_directory):\n wandb.finish()\n"
},
{
"path": "ludwig/data/__init__.py",
"content": ""
},
{
"path": "ludwig/data/batcher/__init__.py",
"content": ""
},
{
"path": "ludwig/data/batcher/base.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2020 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nfrom abc import ABC, abstractmethod\n\nimport numpy as np\n\n\nclass Batcher(ABC):\n @abstractmethod\n def next_batch(self) -> dict[str, np.ndarray]:\n raise NotImplementedError()\n\n @abstractmethod\n def last_batch(self) -> bool:\n raise NotImplementedError()\n\n @abstractmethod\n def set_epoch(self, epoch: int, batch_size: int):\n raise NotImplementedError()\n"
},
{
"path": "ludwig/data/batcher/bucketed.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport numpy as np\n\nfrom ludwig.data.batcher.base import Batcher\n\n\nclass BucketedBatcher(Batcher):\n def __init__(\n self,\n dataset,\n bucketing_field,\n batch_size=128,\n buckets=10,\n should_shuffle=True,\n ignore_last=False,\n should_trim=False,\n trim_side=\"right\",\n ):\n self.should_shuffle = should_shuffle\n self.bucketing_field = bucketing_field\n self.should_trim = should_trim\n self.trim_side = trim_side\n\n # store our dataset as well\n self.dataset = dataset\n\n field = dataset.get_dataset()[bucketing_field]\n field_lengths = np.apply_along_axis(lambda x: np.sign(x).sum(), 1, field)\n sorted_idcs = np.argsort(field_lengths)\n self.buckets_idcs = []\n datapoints_per_bucket = len(field) // buckets\n for b in range(buckets):\n start = datapoints_per_bucket * b\n end = datapoints_per_bucket * (b + 1) if b < buckets - 1 else len(sorted_idcs)\n self.buckets_idcs.append(sorted_idcs[start:end])\n\n if should_shuffle:\n self.shuffle(self.buckets_idcs)\n\n self.ignore_last = ignore_last\n self.batch_size = batch_size\n self.total_size = min(map(len, dataset.get_dataset().values()))\n self.bucket_sizes = np.array([x for x in map(len, self.buckets_idcs)])\n self.steps_per_epoch = self._compute_steps_per_epoch()\n self.indices = np.array([0] * buckets)\n self.step = 0\n self.epoch = 0\n\n def shuffle(self, buckets_idcs):\n for i in range(len(buckets_idcs)):\n np.random.shuffle(buckets_idcs[i])\n\n def next_batch(self):\n if self.last_batch():\n if self.should_shuffle:\n self.shuffle(self.buckets_idcs)\n self.set_epoch(self.epoch + 1)\n\n if self.ignore_last:\n idcs_below_size = self.indices + self.batch_size < self.bucket_sizes\n else:\n idcs_below_size = self.indices < self.bucket_sizes\n i = np.random.choice(np.arange(0, len(self.buckets_idcs))[idcs_below_size])\n\n selected_bucket = self.buckets_idcs[i]\n selected_idcs = selected_bucket[self.indices[i] : self.indices[i] + self.batch_size]\n\n sub_batch = {}\n for key in self.dataset.get_dataset():\n if key == self.bucketing_field and self.should_trim:\n selected_samples = self.dataset.get(key, selected_idcs)\n max_length = np.sign(selected_samples).sum(axis=1).max()\n if self.trim_side == \"right\":\n sub_batch[key] = selected_samples[:, :max_length]\n elif self.trim_side == \"left\":\n sub_batch[key] = selected_samples[:, -max_length:]\n else:\n raise ValueError(\"Invalid trim side:\", self.trim_side)\n\n else:\n sub_batch[key] = self.dataset.get(key, selected_idcs)\n\n self.indices[i] += self.batch_size\n self.step += 1\n return sub_batch\n\n def last_batch(self):\n return not np.any(self.indices < self.bucket_sizes) or (\n self.ignore_last and not np.any(self.indices + self.batch_size < self.bucket_sizes)\n )\n\n def set_epoch(self, epoch, batch_size):\n self.indices = np.array([0] * len(self.buckets_idcs))\n self.step = 0\n self.epoch = epoch\n self.batch_size = batch_size\n self.steps_per_epoch = self._compute_steps_per_epoch()\n\n def _compute_steps_per_epoch(self) -> int:\n return int(np.sum(np.ceil(self.bucket_sizes / self.batch_size)).item())\n\n\n# dynamic_length_encoders = {\n# 'rnn',\n# 'embed'\n# }\n#\n# todo future: reintroduce the bucketed batcher\n# def initialize_batcher(dataset, batch_size=128, bucketing_field=None,\n# input_features=None, preprocessing=None,\n# should_shuffle=True, ignore_last=False):\n# if bucketing_field is not None:\n# bucketing_feature = [\n# feature for feature in input_features if\n# feature[NAME] == bucketing_field\n# ]\n# if not bucketing_feature:\n# raise ValueError(\n# 'Bucketing field {} not present in input features'.format(\n# bucketing_field\n# )\n# )\n# else:\n# bucketing_feature = bucketing_feature[0]\n# should_trim = bucketing_feature[\n# 'encoder'] in dynamic_length_encoders\n# if 'preprocessing' in bucketing_feature:\n# trim_side = bucketing_feature['preprocessing']['padding']\n# else:\n# trim_side = preprocessing[bucketing_feature[TYPE]]['padding']\n#\n# batcher = BucketedBatcher(\n# dataset,\n# bucketing_field=bucketing_field,\n# batch_size=batch_size,\n# buckets=10,\n# ignore_last=ignore_last,\n# should_shuffle=should_shuffle,\n# should_trim=should_trim,\n# trim_side=trim_side\n# )\n# else:\n# batcher = Batcher(\n# dataset,\n# batch_size,\n# should_shuffle=should_shuffle,\n# ignore_last=ignore_last\n# )\n# return batcher\n"
},
{
"path": "ludwig/data/batcher/iterable.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2020 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nfrom ludwig.data.batcher.base import Batcher\n\n\nclass IterableBatcher(Batcher):\n def __init__(self, dataset, data, steps_per_epoch, ignore_last=False):\n self.dataset = dataset\n self.data = data\n self.data_it = iter(data)\n\n self.ignore_last = ignore_last\n self.steps_per_epoch = steps_per_epoch\n self.step = 0\n\n def next_batch(self):\n if self.last_batch():\n raise StopIteration()\n\n sub_batch = {}\n batch = next(self.data_it)\n for features_name in self.dataset.features:\n sub_batch[features_name] = self.dataset.get(features_name, batch)\n\n self.step += 1\n return sub_batch\n\n def last_batch(self):\n return self.step >= self.steps_per_epoch or (self.ignore_last and self.step + 1 >= self.steps_per_epoch)\n\n def set_epoch(self, epoch, batch_size):\n # TODO ray: implement dynamic batch size\n self.step = 0\n"
},
{
"path": "ludwig/data/batcher/random_access.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nimport math\n\nimport torch\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.data.batcher.base import Batcher\n\nlogger = logging.getLogger(__name__)\n\n\n@DeveloperAPI\nclass RandomAccessBatcher(Batcher):\n def __init__(self, dataset, sampler, batch_size=128, ignore_last=False, augmentation_pipeline=None):\n # store our dataset as well\n self.dataset = dataset\n self.sampler = sampler\n self.sample_it = iter(self.sampler)\n\n self.ignore_last = ignore_last\n self.batch_size = batch_size\n self.total_size = len(sampler)\n self.augmentation_pipeline = augmentation_pipeline\n self.steps_per_epoch = self._compute_steps_per_epoch()\n self.index = 0\n self.step = 0\n\n def next_batch(self):\n if self.last_batch():\n raise StopIteration()\n\n indices = []\n for _ in range(self.batch_size):\n try:\n indices.append(next(self.sample_it))\n self.index += 1\n except StopIteration:\n break\n\n sub_batch = {feature_name: self.dataset.get(feature_name, indices) for feature_name in self.dataset.features}\n\n if self.augmentation_pipeline:\n for feature_name, augmentations in self.augmentation_pipeline.items():\n logger.debug(f\"RandomAccessBatcher applying augmentation pipeline to batch for feature {feature_name}\")\n sub_batch[feature_name] = augmentations(torch.tensor(sub_batch[feature_name]))\n\n self.step += 1\n return sub_batch\n\n def last_batch(self):\n \"\"\"Returns whether we've exhausted all batches for this epoch.\n\n If False, then there is at least 1 more batch available with next_batch().\n \"\"\"\n # If our current index in the dataset exceeds the size of the dataset,\n # we've finished the epoch and can indicate that this is the last batch\n if self.index >= self.total_size:\n return True\n # This avoids the case where batch size > total size and no steps have been done.\n # For e.g., batch size = 128 but the dataset only has 100 rows.\n elif self.ignore_last and self.step:\n # index += batch_size after each epoch. So, if our current index in total dataset is 1 less than the total\n # dataset size, then the last batch will only have 1 row.\n # If this happens, we drop the last batch, unless batch_size is 1.\n if self.batch_size > 1 and self.index - self.total_size == -1:\n logger.info(\"Last batch in epoch only has 1 sample and will be dropped.\")\n return True\n return False\n\n def set_epoch(self, epoch, batch_size):\n self.batch_size = batch_size\n self.steps_per_epoch = self._compute_steps_per_epoch()\n self.index = 0\n self.step = 0\n self.sampler.set_epoch(epoch)\n self.sample_it = iter(self.sampler)\n\n def _compute_steps_per_epoch(self):\n return int(math.ceil(self.total_size / self.batch_size))\n"
},
{
"path": "ludwig/data/batcher/test_batcher.py",
"content": "import logging\n\nimport pandas as pd\nimport yaml\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.data.dataset.pandas import PandasDataset\n\n\ndef test_pandas_size():\n df = pd.DataFrame(\n {\"name\": [\"joe\", \"janice\", \"sara\"], \"mask\": [\"green\", \"black\", \"pink\"], \"weapon\": [\"stick\", \"gun\", \"gun\"]}\n )\n config = yaml.safe_load(\"\"\"\n model_type: llm\n base_model: HuggingFaceH4/tiny-random-LlamaForCausalLM\n input_features:\n - name: name\n type: text\n preprocessing:\n max_sequence_length: 256\n column: name\n output_features:\n - name: weapon\n type: text\n preprocessing:\n max_sequence_length: 256\n column: weapon\n preprocessing:\n split:\n type: random\n probabilities:\n - 1\n - 0\n - 0\n \"\"\")\n model = LudwigModel(config=config, logging_level=logging.INFO)\n data = model.preprocess(df, skip_save_processed_input=False)\n training_set = data[0]\n assert training_set.size == len(df)\n\n # Check if string loading works as well\n # data[0].data_hdf5_fp is the string filepath to the cached data from preprocessing\n data_from_str = PandasDataset(data[0].data_hdf5_fp, data[0].features, None)\n assert data_from_str.size == len(df)\n\n\ndef test_pandas_batcher_use_all_samples():\n df = pd.DataFrame(\n {\"name\": [\"joe\", \"janice\", \"sara\"], \"mask\": [\"green\", \"black\", \"pink\"], \"weapon\": [\"stick\", \"gun\", \"gun\"]}\n )\n config = yaml.safe_load(\"\"\"\n model_type: llm\n base_model: HuggingFaceH4/tiny-random-LlamaForCausalLM\n input_features:\n - name: name\n type: text\n preprocessing:\n max_sequence_length: 256\n column: name\n output_features:\n - name: weapon\n type: text\n preprocessing:\n max_sequence_length: 256\n column: weapon\n preprocessing:\n split:\n type: random\n probabilities:\n - 1\n - 0\n - 0\n \"\"\")\n model = LudwigModel(config=config, logging_level=logging.INFO)\n data = model.preprocess(df, skip_save_processed_input=False)\n training_set = data[0]\n features = training_set.dataset.keys()\n\n batches = []\n with training_set.initialize_batcher(batch_size=1) as batcher:\n while not batcher.last_batch():\n batch = batcher.next_batch()\n batches.append(batch)\n assert (len(batches)) == training_set.size\n\n # Check to see if all items are used exactly once\n for feature in features:\n for i in range(len(training_set.dataset[feature])):\n # Each of the arrays in the line below should contain the vector representation of a feature of sample i\n assert (batches[i][feature].squeeze() == training_set.dataset[feature][i].squeeze()).all()\n\n # Check if string loading works as well\n batches = []\n # data[0].data_hdf5_fp is the string filepath to the cached data from preprocessing\n data_from_str = PandasDataset(data[0].data_hdf5_fp, data[0].features, None)\n features = data_from_str.dataset.keys()\n\n with data_from_str.initialize_batcher(batch_size=1) as batcher:\n while not batcher.last_batch():\n batch = batcher.next_batch()\n batches.append(batch)\n assert (len(batches)) == data_from_str.size\n\n # Check to see if all items are used exactly once\n for feature in features:\n for i in range(len(data_from_str.dataset[feature])):\n # Each of the arrays in the line below should contain the vector representation of a feature of sample i\n assert (batches[i][feature].squeeze() == data_from_str.dataset[feature][i].squeeze()).all()\n"
},
{
"path": "ludwig/data/cache/__init__.py",
"content": ""
},
{
"path": "ludwig/data/cache/manager.py",
"content": "import logging\nimport os\n\nfrom ludwig.constants import CHECKSUM, META, TEST, TRAINING, VALIDATION\nfrom ludwig.data.cache.types import alphanum, CacheableDataset\nfrom ludwig.data.cache.util import calculate_checksum\nfrom ludwig.data.dataset.base import DatasetManager\nfrom ludwig.utils import data_utils\nfrom ludwig.utils.fs_utils import delete, path_exists\n\nlogger = logging.getLogger(__name__)\n\n\nclass DatasetCache:\n def __init__(self, config, checksum, cache_map, dataset_manager):\n self.config = config\n self.checksum = checksum\n self.cache_map = cache_map\n self.dataset_manager = dataset_manager\n\n def get(self):\n training_set_metadata_fp = self.cache_map[META]\n if not path_exists(training_set_metadata_fp):\n return None\n\n try:\n cached_training_set_metadata = data_utils.load_json(training_set_metadata_fp)\n except Exception:\n logger.exception(f\"Failed to load cached training set metadata at {training_set_metadata_fp}\")\n return None\n\n cached_training_set = self.cache_map[TRAINING] if path_exists(self.cache_map[TRAINING]) else None\n if not cached_training_set:\n logger.warning(f\"Failed to load cached training set at {self.cache_map[TRAINING]}\")\n\n cached_validation_set = self.cache_map[VALIDATION] if path_exists(self.cache_map[VALIDATION]) else None\n if not cached_validation_set:\n logger.warning(f\"Failed to load cached validation set at {self.cache_map[VALIDATION]}\")\n\n cached_test_set = self.cache_map[TEST] if path_exists(self.cache_map[TEST]) else None\n if not cached_test_set:\n logger.warning(f\"Failed to load cached test set at {self.cache_map[TEST]}\")\n\n valid = self.checksum == cached_training_set_metadata.get(CHECKSUM) and cached_training_set is not None\n\n return valid, cached_training_set_metadata, cached_training_set, cached_test_set, cached_validation_set\n\n def put(self, training_set, test_set, validation_set, training_set_metadata):\n logger.info(f\"Writing preprocessed training set cache to {self.cache_map[TRAINING]}\")\n training_set = self.dataset_manager.save(\n self.cache_map[TRAINING],\n training_set,\n self.config,\n training_set_metadata,\n TRAINING,\n )\n\n if validation_set is not None:\n logger.info(f\"Writing preprocessed validation set cache to {self.cache_map[VALIDATION]}\")\n validation_set = self.dataset_manager.save(\n self.cache_map[VALIDATION],\n validation_set,\n self.config,\n training_set_metadata,\n VALIDATION,\n )\n\n if test_set is not None:\n logger.info(f\"Writing preprocessed test set cache to {self.cache_map[TEST]}\")\n test_set = self.dataset_manager.save(\n self.cache_map[TEST],\n test_set,\n self.config,\n training_set_metadata,\n TEST,\n )\n\n logger.info(f\"Writing train set metadata to {self.cache_map[META]}\")\n data_utils.save_json(self.cache_map[META], training_set_metadata)\n\n return training_set, test_set, validation_set, training_set_metadata\n\n def delete(self):\n for fname in self.cache_map.values():\n if path_exists(fname):\n # Parquet entries in the cache_ma can be pointers to directories.\n delete(fname, recursive=True)\n\n def get_cached_obj_path(self, cached_obj_name: str) -> str:\n return self.cache_map.get(cached_obj_name)\n\n\nclass CacheManager:\n def __init__(\n self,\n dataset_manager: DatasetManager,\n cache_dir: str | None = None,\n ):\n self._dataset_manager = dataset_manager\n self._cache_dir = cache_dir\n\n def get_dataset_cache(\n self,\n config: dict,\n dataset: CacheableDataset | None = None,\n training_set: CacheableDataset | None = None,\n test_set: CacheableDataset | None = None,\n validation_set: CacheableDataset | None = None,\n ) -> DatasetCache:\n if dataset is not None:\n key = self.get_cache_key(dataset, config)\n cache_map = {\n META: self.get_cache_path(dataset, key, META, \"json\"),\n TRAINING: self.get_cache_path(dataset, key, TRAINING),\n TEST: self.get_cache_path(dataset, key, TEST),\n VALIDATION: self.get_cache_path(dataset, key, VALIDATION),\n }\n return DatasetCache(config, key, cache_map, self._dataset_manager)\n else:\n key = self.get_cache_key(training_set, config)\n cache_map = {\n META: self.get_cache_path(training_set, key, META, \"json\"),\n TRAINING: self.get_cache_path(training_set, key, TRAINING),\n TEST: self.get_cache_path(test_set, key, TEST),\n VALIDATION: self.get_cache_path(validation_set, key, VALIDATION),\n }\n return DatasetCache(config, key, cache_map, self._dataset_manager)\n\n def get_cache_key(self, dataset: CacheableDataset, config: dict) -> str:\n return calculate_checksum(dataset, config)\n\n def get_cache_path(self, dataset: CacheableDataset | None, key: str, tag: str, ext: str | None = None) -> str:\n if self._cache_dir is None and dataset is not None:\n # Use the input dataset filename (minus the extension) as the cache path\n stem = dataset.get_cache_path()\n else:\n # To avoid collisions across different directories, we use the unique checksum\n # as the cache path\n stem = alphanum(key)\n\n ext = ext or self.data_format\n cache_fname = f\"{stem}.{tag}.{ext}\"\n return os.path.join(self.get_cache_directory(dataset), cache_fname)\n\n def get_cache_directory(self, dataset: CacheableDataset | None) -> str:\n if self._cache_dir is None:\n if dataset is None:\n return os.getcwd()\n return dataset.get_cache_directory()\n return self._cache_dir\n\n def can_cache(self, skip_save_processed_input: bool) -> bool:\n return self._dataset_manager.can_cache(skip_save_processed_input)\n\n @property\n def data_format(self) -> str:\n return self._dataset_manager.data_format\n"
},
{
"path": "ludwig/data/cache/types.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nimport os\nimport re\nimport uuid\nfrom abc import ABC, abstractmethod\nfrom dataclasses import dataclass\nfrom pathlib import Path\nfrom typing import Union\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.utils.fs_utils import checksum\nfrom ludwig.utils.types import DataFrame\n\n\ndef alphanum(v):\n \"\"\"Filters a string to only its alphanumeric characters.\"\"\"\n return re.sub(r\"\\W+\", \"\", v)\n\n\n@DeveloperAPI\nclass CacheableDataset(ABC):\n name: str\n checksum: str\n\n @abstractmethod\n def get_cache_path(self) -> str:\n raise NotImplementedError()\n\n @abstractmethod\n def get_cache_directory(self) -> str:\n raise NotImplementedError()\n\n @abstractmethod\n def unwrap(self) -> str | DataFrame:\n raise NotImplementedError()\n\n\n@DeveloperAPI\n@dataclass\nclass CacheableDataframe(CacheableDataset):\n df: DataFrame\n name: str\n checksum: str\n\n def get_cache_path(self) -> str:\n return alphanum(self.name)\n\n def get_cache_directory(self) -> str:\n return os.getcwd()\n\n def unwrap(self) -> str | DataFrame:\n return self.df\n\n\n@DeveloperAPI\n@dataclass\nclass CacheablePath(CacheableDataset):\n path: str\n\n @property\n def name(self) -> str:\n return Path(self.path).stem\n\n @property\n def checksum(self) -> str:\n return checksum(self.path)\n\n def get_cache_path(self) -> str:\n return self.name\n\n def get_cache_directory(self) -> str:\n return os.path.dirname(self.path)\n\n def unwrap(self) -> str | DataFrame:\n return self.path\n\n\nCacheInput = Union[str, DataFrame, CacheableDataset]\n\n\ndef wrap(dataset: CacheInput | None) -> CacheableDataset:\n if dataset is None:\n return None\n\n if isinstance(dataset, CacheableDataset):\n return dataset\n if isinstance(dataset, str):\n return CacheablePath(path=dataset)\n\n # TODO(travis): could try hashing the in-memory dataset, but this is tricky for Dask\n checksum = str(uuid.uuid1())\n name = checksum\n return CacheableDataframe(df=dataset, name=name, checksum=checksum)\n"
},
{
"path": "ludwig/data/cache/util.py",
"content": "import ludwig\nfrom ludwig.constants import DEFAULTS, INPUT_FEATURES, OUTPUT_FEATURES, PREPROCESSING, PROC_COLUMN, TYPE\nfrom ludwig.data.cache.types import CacheableDataset\nfrom ludwig.types import ModelConfigDict\nfrom ludwig.utils.data_utils import hash_dict\n\n\ndef calculate_checksum(original_dataset: CacheableDataset, config: ModelConfigDict):\n \"\"\"Calculates a checksum for a dataset and model config.\n\n The checksum is used to determine if the dataset and model config have changed since the last time the model was\n trained. If either has changed, a different checksum will be produced which will lead to a cache miss and force\n preprocessing to be performed again.\n \"\"\"\n features = config.get(INPUT_FEATURES, []) + config.get(OUTPUT_FEATURES, []) + config.get(\"features\", [])\n info = {\n \"ludwig_version\": ludwig.globals.LUDWIG_VERSION,\n \"dataset_checksum\": original_dataset.checksum,\n \"global_preprocessing\": config.get(PREPROCESSING, {}),\n \"global_defaults\": config.get(DEFAULTS, {}),\n # PROC_COLUMN contains both the feature name and the feature hash that is computed\n # based on each feature's preprocessing parameters and the feature's type.\n # creating a sorted list out of the dict because hash_dict requires all values\n # of the dict to be ordered object to ensure the creation fo the same hash\n \"feature_proc_columns\": sorted({feature[PROC_COLUMN] for feature in features}),\n \"feature_types\": [feature[TYPE] for feature in features],\n \"feature_preprocessing\": [feature.get(PREPROCESSING, {}) for feature in features],\n }\n\n # LLM-specific params\n if \"prompt\" in config:\n info[\"prompt\"] = config[\"prompt\"]\n\n return hash_dict(info, max_length=None).decode(\"ascii\")\n"
},
{
"path": "ludwig/data/concatenate_datasets.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport argparse\nimport logging\n\nimport numpy as np\n\nfrom ludwig.backend import LOCAL_BACKEND\nfrom ludwig.constants import SPLIT\nfrom ludwig.utils.data_utils import read_csv\n\nlogger = logging.getLogger(__name__)\n\n\ndef concatenate_csv(train_csv, vali_csv, test_csv, output_csv):\n concatenated_df = concatenate_files(train_csv, vali_csv, test_csv, read_csv, LOCAL_BACKEND)\n\n logger.info(\"Saving concatenated dataset as csv..\")\n concatenated_df.to_csv(output_csv, encoding=\"utf-8\", index=False)\n logger.info(\"done\")\n\n\ndef concatenate_files(train_fname, vali_fname, test_fname, read_fn, backend):\n df_lib = backend.df_engine.df_lib\n\n logger.info(\"Loading training file...\")\n train_df = read_fn(train_fname, df_lib)\n logger.info(\"done\")\n\n logger.info(\"Loading validation file..\")\n vali_df = read_fn(vali_fname, df_lib) if vali_fname is not None else None\n logger.info(\"done\")\n\n logger.info(\"Loading test file..\")\n test_df = read_fn(test_fname, df_lib) if test_fname is not None else None\n logger.info(\"done\")\n\n logger.info(\"Concatenating files..\")\n concatenated_df = concatenate_df(train_df, vali_df, test_df, backend)\n logger.info(\"done\")\n\n return concatenated_df\n\n\ndef concatenate_df(train_df, vali_df, test_df, backend):\n train_size = len(train_df)\n vali_size = len(vali_df) if vali_df is not None else 0\n\n concatenated_df = backend.df_engine.df_lib.concat(\n [df for df in [train_df, vali_df, test_df] if df is not None], ignore_index=True\n )\n\n def get_split(idx):\n if idx < train_size:\n return 0\n if idx < train_size + vali_size:\n return 1\n return 2\n\n concatenated_df[SPLIT] = concatenated_df.index.to_series().map(get_split).astype(np.int8)\n return concatenated_df\n\n\ndef concatenate_splits(train_df, vali_df, test_df, backend):\n def to_frame(df, split):\n if df is None:\n return None\n\n df = df.index.to_frame(name=SPLIT)\n df[SPLIT] = split\n return df\n\n dfs = [train_df, vali_df, test_df]\n dfs = [to_frame(df, split) for split, df in enumerate(dfs)]\n return backend.df_engine.df_lib.concat([df for df in dfs if df is not None])\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(description=\"Concatenate train validation and test set\")\n\n parser.add_argument(\"-train\", \"--train_csv\", help=\"CSV containing the training set\")\n parser.add_argument(\"-vali\", \"--vali_csv\", help=\"CSV containing the validation set\")\n parser.add_argument(\"-test\", \"--test_csv\", help=\"CSV containing the test set\")\n\n parser.add_argument(\"-o\", \"--output_csv\", help=\"output csv\")\n args = parser.parse_args()\n\n concatenate_csv(args.train_csv, args.vali_csv, args.test_csv, args.output_csv)\n"
},
{
"path": "ludwig/data/dataframe/__init__.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2020 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n"
},
{
"path": "ludwig/data/dataframe/base.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2020 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nfrom abc import ABC, abstractmethod\n\nfrom ludwig.utils.types import DataFrame\n\n\nclass DataFrameEngine(ABC):\n @abstractmethod\n def df_like(self, df, proc_cols):\n raise NotImplementedError()\n\n @abstractmethod\n def parallelize(self, data):\n raise NotImplementedError()\n\n @abstractmethod\n def persist(self, data):\n raise NotImplementedError()\n\n @abstractmethod\n def compute(self, data):\n raise NotImplementedError()\n\n @abstractmethod\n def from_pandas(self, df):\n raise NotImplementedError()\n\n @abstractmethod\n def map_objects(self, series, map_fn, meta=None):\n raise NotImplementedError()\n\n @abstractmethod\n def map_partitions(self, series, map_fn, meta=None):\n raise NotImplementedError()\n\n @abstractmethod\n def map_batches(self, df, map_fn, enable_tensor_extension_casting=True):\n raise NotImplementedError()\n\n @abstractmethod\n def apply_objects(self, series, map_fn, meta=None):\n raise NotImplementedError()\n\n @abstractmethod\n def reduce_objects(self, series, reduce_fn):\n raise NotImplementedError()\n\n @abstractmethod\n def split(self, df, probabilities):\n \"\"\"Splits the input DataFrame into sections with the given proportions.\"\"\"\n raise NotImplementedError()\n\n @abstractmethod\n def to_parquet(self, df, path, index=False):\n \"\"\"Write the input DataFrame to the path in the Parquet format.\n\n Optionally includes the DataFrame index in the Parquet file.\n \"\"\"\n raise NotImplementedError()\n\n @abstractmethod\n def write_predictions(self, df: DataFrame, path: str):\n raise NotImplementedError()\n\n @abstractmethod\n def read_predictions(self, path: str) -> DataFrame:\n raise NotImplementedError()\n\n @abstractmethod\n def to_ray_dataset(self, df):\n raise NotImplementedError()\n\n @property\n @abstractmethod\n def array_lib(self):\n raise NotImplementedError()\n\n @property\n @abstractmethod\n def df_lib(self):\n raise NotImplementedError()\n\n @property\n @abstractmethod\n def partitioned(self):\n raise NotImplementedError()\n\n @abstractmethod\n def set_parallelism(self, parallelism):\n raise NotImplementedError()\n"
},
{
"path": "ludwig/data/dataframe/dask.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2020 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nimport collections\nimport logging\nfrom contextlib import contextmanager\n\nimport dask\nimport dask.array as da\nimport dask.dataframe as dd\nimport ray\nfrom dask.diagnostics import ProgressBar\nfrom packaging import version\nfrom pyarrow.fs import FSSpecHandler, PyFileSystem\nfrom ray.data import Dataset, read_parquet\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.data.dataframe.base import DataFrameEngine\nfrom ludwig.utils.data_utils import get_pa_schema, get_parquet_filename, split_by_slices\nfrom ludwig.utils.dataframe_utils import set_index_name\nfrom ludwig.utils.fs_utils import get_fs_and_path\n\nTMP_COLUMN = \"__TMP_COLUMN__\"\n\n# This is to be compatible with pyarrow.lib.schema\nPandasBlockSchema = collections.namedtuple(\"PandasBlockSchema\", [\"names\", \"types\"])\n\nlogger = logging.getLogger(__name__)\n\n\n_ray_230 = version.parse(ray.__version__) >= version.parse(\"2.3.0\")\n\n\n@DeveloperAPI\ndef set_scheduler(scheduler):\n dask.config.set(scheduler=scheduler)\n\n\n@DeveloperAPI\ndef reset_index_across_all_partitions(df):\n \"\"\"Compute a monotonically increasing index across all partitions.\n\n This differs from dd.reset_index, which computes an independent index for each partition.\n Source: https://stackoverflow.com/questions/61395351/how-to-reset-index-on-concatenated-dataframe-in-dask\n \"\"\"\n # Create temporary column of ones\n df = df.assign(**{TMP_COLUMN: 1})\n\n # Set the index to the cumulative sum of TMP_COLUMN, which we know to be sorted; this improves efficiency.\n df = df.set_index(df[TMP_COLUMN].cumsum() - 1, sorted=True)\n\n # Drop temporary column and ensure the index is not named TMP_COLUMN\n df = df.drop(columns=TMP_COLUMN)\n df = df.map_partitions(lambda pd_df: set_index_name(pd_df, None))\n return df\n\n\n@DeveloperAPI\nclass DaskEngine(DataFrameEngine):\n def __init__(self, parallelism=None, persist=True, _use_ray=True, **kwargs):\n from ray.util.dask import ray_dask_get\n\n self._parallelism = parallelism\n self._persist = persist\n if _use_ray:\n set_scheduler(ray_dask_get)\n\n def set_parallelism(self, parallelism):\n self._parallelism = parallelism\n\n def df_like(self, df: dd.DataFrame, proc_cols: dict[str, dd.Series]):\n \"\"\"Outer joins the given DataFrame with the given processed columns.\n\n NOTE: If any of the processed columns have been repartitioned, the original index is replaced with a\n monotonically increasing index, which is used to define the new divisions and align the various partitions.\n \"\"\"\n # Our goal is to preserve the index of the input dataframe but to drop\n # all its columns. Because to_frame() creates a column from the index,\n # we need to drop it immediately following creation.\n dataset = df.index.to_frame(name=TMP_COLUMN).drop(columns=TMP_COLUMN)\n\n repartitioned_cols = {}\n for k, v in proc_cols.items():\n if v.npartitions == dataset.npartitions:\n # Outer join cols with equal partitions.\n # Dask aligns by index automatically, so no need to force divisions.\n dataset[k] = v\n else:\n # If partitions have changed (e.g. due to conversion from Ray dataset), we handle separately\n repartitioned_cols[k] = v\n\n # Assumes that there is a globally unique index (see preprocessing.build_dataset)\n if repartitioned_cols:\n if not dataset.known_divisions:\n # Sometimes divisions are unknown despite having a usable index– set_index to know divisions\n dataset = dataset.assign(**{TMP_COLUMN: dataset.index})\n dataset = dataset.set_index(TMP_COLUMN, drop=True)\n dataset = dataset.map_partitions(lambda pd_df: set_index_name(pd_df, dataset.index.name))\n\n # Find the divisions of the column with the largest number of partitions\n proc_col_with_max_npartitions = max(repartitioned_cols.values(), key=lambda x: x.npartitions)\n new_divisions = proc_col_with_max_npartitions.divisions\n\n # Repartition all columns to have the same divisions\n dataset = dataset.repartition(divisions=new_divisions)\n repartitioned_cols = {k: v.repartition(divisions=new_divisions) for k, v in repartitioned_cols.items()}\n\n # Outer join the remaining columns\n for k, v in repartitioned_cols.items():\n dataset[k] = v\n\n return dataset\n\n def parallelize(self, data):\n if self.parallelism:\n return data.repartition(npartitions=self.parallelism)\n return data\n\n def persist(self, data):\n # No graph optimizations to prevent dropping custom annotations\n # https://github.com/dask/dask/issues/7036\n return data.persist(optimize_graph=False) if self._persist else data\n\n def concat(self, dfs):\n return self.df_lib.concat(dfs)\n\n def compute(self, data):\n return data.compute()\n\n def from_pandas(self, df):\n parallelism = self._parallelism or 1\n return dd.from_pandas(df, npartitions=parallelism)\n\n def map_objects(self, series, map_fn, meta=None):\n meta = meta if meta is not None else (series.name, \"object\")\n return series.map(map_fn, meta=meta)\n\n def map_partitions(self, series, map_fn, meta=None):\n meta = meta if meta is not None else (series.name, \"object\")\n return series.map_partitions(map_fn, meta=meta)\n\n def map_batches(self, series, map_fn, enable_tensor_extension_casting=True):\n \"\"\"Map a function over batches of a Dask Series.\n\n Args:\n series: Dask Series\n map_fn: Function to apply to each batch\n enable_tensor_extension_casting: Whether to enable tensor extension casting at the end of the Ray Datasets\n map_batches call. This is useful in cases where the output is not supported by the ray Tensor dtype\n extension, such as when the output consists of ragged tensors.\n \"\"\"\n import ray.data\n\n with tensor_extension_casting(enable_tensor_extension_casting):\n ds = ray.data.from_dask(series)\n ds = ds.map_batches(map_fn, batch_format=\"pandas\")\n return ds.to_dask()\n\n def apply_objects(self, df, apply_fn, meta=None):\n meta = meta if meta is not None else (\"result\", \"object\")\n return df.apply(apply_fn, axis=1, meta=meta)\n\n def reduce_objects(self, series, reduce_fn):\n result = series.reduction(reduce_fn, aggregate=reduce_fn, meta=(series.name, \"object\")).compute()\n # The result type depends on the Dask version and what reduce_fn returns.\n # Access the scalar value safely regardless of return type.\n if hasattr(result, \"iloc\"):\n return result.iloc[0]\n return result\n\n def split(self, df, probabilities):\n # Split the DataFrame proprotionately along partitions. This is an inexact solution designed\n # to speed up the split process, as splitting within partitions would be significantly\n # more expensive.\n # TODO(travis): revisit in the future to make this more precise\n\n # First ensure that every split receives at least one partition.\n # If not, we need to increase the number of partitions to satisfy this constraint.\n min_prob = min(probabilities)\n min_partitions = int(1 / min_prob)\n if df.npartitions < min_partitions:\n df = df.repartition(npartitions=min_partitions)\n\n n = df.npartitions\n slices = df.partitions\n return split_by_slices(slices, n, probabilities)\n\n def remove_empty_partitions(self, df):\n # Reference: https://stackoverflow.com/questions/47812785/remove-empty-partitions-in-dask\n ll = list(df.map_partitions(len).compute())\n if all([ll_i > 0 for ll_i in ll]):\n return df\n\n df_delayed = df.to_delayed()\n df_delayed_new = list()\n empty_partition = None\n for ix, n in enumerate(ll):\n if n == 0:\n empty_partition = df.get_partition(ix)\n else:\n df_delayed_new.append(df_delayed[ix])\n if not df_delayed_new:\n # All partitions are empty, return a single empty partition\n return empty_partition\n df = dd.from_delayed(df_delayed_new, meta=empty_partition)\n return df\n\n def to_parquet(self, df, path, index=False):\n schema = get_pa_schema(df)\n with ProgressBar():\n df.to_parquet(\n path,\n engine=\"pyarrow\",\n write_index=index,\n schema=schema,\n name_function=get_parquet_filename,\n )\n\n def write_predictions(self, df: dd.DataFrame, path: str):\n ds = self.to_ray_dataset(df)\n # We disable tensor extension casting here because we are writing out to Parquet and there is no need\n # to cast to the ray Tensor dtype extension before doing so (they will be written out as object dtype as if\n # we were writing to parquet using dask).\n with tensor_extension_casting(False):\n fs, path = get_fs_and_path(path)\n ds.write_parquet(path, filesystem=PyFileSystem(FSSpecHandler(fs)))\n\n def read_predictions(self, path: str) -> dd.DataFrame:\n fs, path = get_fs_and_path(path)\n ds = read_parquet(path, filesystem=PyFileSystem(FSSpecHandler(fs)))\n return self.from_ray_dataset(ds)\n\n def to_ray_dataset(self, df) -> Dataset:\n from ray.data import from_dask\n\n return from_dask(df)\n\n def from_ray_dataset(self, dataset) -> dd.DataFrame:\n # NOTE: When the dataset is an empty MapBatches(BatchInferModel), Ray's native to_dask() raises an IndexError.\n try:\n return dataset.to_dask()\n except IndexError as e:\n logging.warning(\n f\"Encountered an empty Dataset, {dataset.show()} with error {e}. Manually returning an empty dask \"\n \"DataFrame.\"\n )\n return dd.DataFrame.from_dict({}, npartitions=1)\n\n def reset_index(self, df):\n return reset_index_across_all_partitions(df)\n\n @property\n def array_lib(self):\n return da\n\n @property\n def df_lib(self):\n return dd\n\n @property\n def parallelism(self):\n return self._parallelism\n\n @property\n def partitioned(self):\n return True\n\n\n@contextmanager\ndef tensor_extension_casting(enforced: bool):\n \"\"\"This context manager is used to enforce or disable tensor extension casting.\n\n Ray Datasets will automatically cast tensor columns to the ray Tensor dtype extension at the end of\n map_batches calls and before writing to Parquet. This context manager can be used to disable this behavior\n and keep the tensor columns as object dtype. This is useful for writing to Parquet using dask.\n\n Args:\n enforced (bool): Whether to enforce tensor extension casting.\n \"\"\"\n from ray.data.context import DatasetContext\n\n ctx = DatasetContext.get_current()\n prev_enable_tensor_extension_casting = ctx.enable_tensor_extension_casting\n try:\n ctx.enable_tensor_extension_casting = enforced\n yield\n finally:\n ctx.enable_tensor_extension_casting = prev_enable_tensor_extension_casting\n"
},
{
"path": "ludwig/data/dataframe/modin.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nimport os\n\nimport modin.pandas as pd\nimport numpy as np\n\nfrom ludwig.data.dataframe.base import DataFrameEngine\nfrom ludwig.globals import PREDICTIONS_SHAPES_FILE_NAME\nfrom ludwig.utils.data_utils import get_pa_schema, load_json, save_json, split_by_slices\nfrom ludwig.utils.dataframe_utils import flatten_df, unflatten_df\n\n\nclass ModinEngine(DataFrameEngine):\n def __init__(self, **kwargs):\n super().__init__()\n\n def df_like(self, df, proc_cols):\n # df argument unused for pandas, which can instantiate df directly\n return pd.DataFrame(proc_cols)\n\n def parallelize(self, data):\n return data\n\n def persist(self, data):\n return data\n\n def compute(self, data):\n return data\n\n def from_pandas(self, df):\n return pd.DataFrame(df)\n\n def map_objects(self, series, map_fn, meta=None):\n return series.map(map_fn)\n\n def map_batches(self, df, map_fn, enable_tensor_extension_casting=True):\n return map_fn(df)\n\n def map_partitions(self, series, map_fn, meta=None):\n return map_fn(series)\n\n def apply_objects(self, df, apply_fn, meta=None):\n return df.apply(apply_fn, axis=1)\n\n def reduce_objects(self, series, reduce_fn):\n return reduce_fn(series)\n\n def split(self, df, probabilities):\n return split_by_slices(df.iloc, len(df), probabilities)\n\n def remove_empty_partitions(self, df):\n return df\n\n def to_parquet(self, df, path, index=False):\n schema = get_pa_schema(df)\n df.to_parquet(\n path,\n engine=\"pyarrow\",\n index=index,\n schema=schema,\n )\n\n def write_predictions(self, df: pd.DataFrame, path: str):\n df, column_shapes = flatten_df(df, self)\n self.to_parquet(df, path)\n save_json(os.path.join(os.path.dirname(path), PREDICTIONS_SHAPES_FILE_NAME), column_shapes)\n\n def read_predictions(self, path: str) -> pd.DataFrame:\n pred_df = pd.read_parquet(path)\n column_shapes = load_json(os.path.join(os.path.dirname(path), PREDICTIONS_SHAPES_FILE_NAME))\n return unflatten_df(pred_df, column_shapes, self)\n\n def to_ray_dataset(self, df):\n from ray.data import from_modin\n\n return from_modin(df)\n\n def from_ray_dataset(self, dataset) -> pd.DataFrame:\n return dataset.to_modin()\n\n def reset_index(self, df):\n return df.reset_index(drop=True)\n\n @property\n def array_lib(self):\n return np\n\n @property\n def df_lib(self):\n return pd\n\n @property\n def partitioned(self):\n return False\n\n def set_parallelism(self, parallelism):\n pass\n"
},
{
"path": "ludwig/data/dataframe/pandas.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2020 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport os\n\nimport numpy as np\nimport pandas as pd\n\nfrom ludwig.data.dataframe.base import DataFrameEngine\nfrom ludwig.globals import PREDICTIONS_SHAPES_FILE_NAME\nfrom ludwig.utils.data_utils import load_json, save_json, split_by_slices\nfrom ludwig.utils.dataframe_utils import flatten_df, unflatten_df\n\n\nclass PandasEngine(DataFrameEngine):\n def __init__(self, **kwargs):\n super().__init__()\n\n def df_like(self, df, proc_cols):\n # df argument unused for pandas, which can instantiate df directly\n return pd.DataFrame(proc_cols)\n\n def parallelize(self, data):\n return data\n\n def persist(self, data):\n return data\n\n def compute(self, data):\n return data\n\n @staticmethod\n def concat(dfs) -> pd.DataFrame:\n return pd.concat(dfs)\n\n def from_pandas(self, df):\n return df\n\n def map_objects(self, series, map_fn, meta=None):\n return series.map(map_fn)\n\n def map_batches(self, df, map_fn, enable_tensor_extension_casting=True):\n return map_fn(df)\n\n def map_partitions(self, series, map_fn, meta=None):\n return map_fn(series)\n\n def apply_objects(self, df, apply_fn, meta=None):\n return df.apply(apply_fn, axis=1)\n\n def reduce_objects(self, series, reduce_fn):\n return reduce_fn(series)\n\n def split(self, df, probabilities):\n return split_by_slices(df.iloc, len(df), probabilities)\n\n @staticmethod\n def remove_empty_partitions(df: pd.DataFrame) -> pd.DataFrame:\n return df\n\n def to_parquet(self, df, path, index=False):\n df.to_parquet(path, engine=\"pyarrow\", index=index)\n\n def write_predictions(self, df: pd.DataFrame, path: str):\n df, column_shapes = flatten_df(df, self)\n self.to_parquet(df, path)\n save_json(os.path.join(os.path.dirname(path), PREDICTIONS_SHAPES_FILE_NAME), column_shapes)\n\n def read_predictions(self, path: str) -> pd.DataFrame:\n pred_df = pd.read_parquet(path)\n column_shapes = load_json(os.path.join(os.path.dirname(path), PREDICTIONS_SHAPES_FILE_NAME))\n return unflatten_df(pred_df, column_shapes, self)\n\n def to_ray_dataset(self, df):\n from ray.data import from_pandas\n\n return from_pandas(df)\n\n @staticmethod\n def from_ray_dataset(dataset) -> pd.DataFrame:\n return dataset.to_pandas()\n\n @staticmethod\n def reset_index(df) -> pd.DataFrame:\n return df.reset_index(drop=True)\n\n @property\n def array_lib(self):\n return np\n\n @property\n def df_lib(self):\n return pd\n\n @property\n def partitioned(self):\n return False\n\n def set_parallelism(self, parallelism):\n pass\n\n\nPANDAS = PandasEngine()\n"
},
{
"path": "ludwig/data/dataset/__init__.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2020 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\n\ndef get_pandas_dataset_manager(**kwargs):\n from ludwig.data.dataset.pandas import PandasDatasetManager\n\n return PandasDatasetManager(**kwargs)\n\n\ndef get_ray_dataset_manager(**kwargs):\n from ludwig.data.dataset.ray import RayDatasetManager\n\n return RayDatasetManager(**kwargs)\n\n\ndataset_registry = {\n \"hdf5\": get_pandas_dataset_manager,\n \"ray\": get_ray_dataset_manager,\n None: get_pandas_dataset_manager,\n}\n\n\ndef create_dataset_manager(backend, cache_format, **kwargs):\n return dataset_registry[cache_format](backend=backend, **kwargs)\n"
},
{
"path": "ludwig/data/dataset/base.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2020 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nfrom __future__ import annotations\n\nimport contextlib\nfrom abc import ABC, abstractmethod\nfrom collections.abc import Iterable\n\nfrom ludwig.data.batcher.base import Batcher\nfrom ludwig.distributed import DistributedStrategy\nfrom ludwig.features.base_feature import BaseFeature\nfrom ludwig.utils.defaults import default_random_seed\nfrom ludwig.utils.types import DataFrame\n\n\nclass Dataset(ABC):\n @abstractmethod\n def __len__(self) -> int:\n raise NotImplementedError()\n\n @contextlib.contextmanager\n @abstractmethod\n def initialize_batcher(\n self,\n batch_size: int = 128,\n should_shuffle: bool = True,\n random_seed: int = default_random_seed,\n ignore_last: bool = False,\n distributed: DistributedStrategy = None,\n ) -> Batcher:\n raise NotImplementedError()\n\n @abstractmethod\n def to_df(self, features: Iterable[BaseFeature] | None = None) -> DataFrame:\n raise NotImplementedError()\n\n @abstractmethod\n def to_scalar_df(self, features: Iterable[BaseFeature] | None = None) -> DataFrame:\n raise NotImplementedError()\n\n @property\n def in_memory_size_bytes(self) -> int:\n raise NotImplementedError()\n\n\nclass DatasetManager(ABC):\n @abstractmethod\n def create(self, dataset, config, training_set_metadata) -> Dataset:\n raise NotImplementedError()\n\n @abstractmethod\n def save(self, cache_path, dataset, config, training_set_metadata, tag) -> Dataset:\n raise NotImplementedError()\n\n @abstractmethod\n def can_cache(self, skip_save_processed_input) -> bool:\n raise NotImplementedError()\n\n @property\n @abstractmethod\n def data_format(self) -> str:\n raise NotImplementedError()\n"
},
{
"path": "ludwig/data/dataset/pandas.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nfrom __future__ import annotations\n\nimport contextlib\nfrom collections.abc import Iterable\nfrom typing import TYPE_CHECKING\n\nimport numpy as np\nfrom pandas import DataFrame\n\nfrom ludwig.constants import PREPROCESSING, TRAINING\nfrom ludwig.data.batcher.base import Batcher\nfrom ludwig.data.batcher.random_access import RandomAccessBatcher\nfrom ludwig.data.dataset.base import Dataset, DatasetManager\nfrom ludwig.data.sampler import DistributedSampler\nfrom ludwig.distributed import DistributedStrategy\nfrom ludwig.features.base_feature import BaseFeature\nfrom ludwig.utils.data_utils import DATA_TRAIN_HDF5_FP, load_hdf5, save_hdf5\nfrom ludwig.utils.dataframe_utils import from_numpy_dataset, to_numpy_dataset, to_scalar_df\nfrom ludwig.utils.defaults import default_random_seed\nfrom ludwig.utils.fs_utils import download_h5\nfrom ludwig.utils.misc_utils import get_proc_features\n\nif TYPE_CHECKING:\n from ludwig.backend.base import Backend\n\n\nclass PandasDataset(Dataset):\n def __init__(self, dataset, features, data_hdf5_fp):\n self.features = features\n self.data_hdf5_fp = data_hdf5_fp\n\n if isinstance(dataset, str):\n dataset = load_hdf5(dataset)\n self.dataset = to_numpy_dataset(dataset)\n self.size = len(list(self.dataset.values())[0])\n\n def to_df(self, features: Iterable[BaseFeature] | None = None) -> DataFrame:\n \"\"\"Convert the dataset to a Pandas DataFrame.\"\"\"\n if features:\n return from_numpy_dataset({feature.feature_name: self.dataset[feature.proc_column] for feature in features})\n return from_numpy_dataset(self.dataset)\n\n def to_scalar_df(self, features: Iterable[BaseFeature] | None = None) -> DataFrame:\n return to_scalar_df(self.to_df(features))\n\n def get(self, proc_column, idx=None):\n if idx is None:\n idx = range(self.size)\n if (\n self.data_hdf5_fp is None\n or PREPROCESSING not in self.features[proc_column]\n or \"in_memory\" not in self.features[proc_column][\"preprocessing\"]\n ):\n return self.dataset[proc_column][idx]\n if self.features[proc_column][PREPROCESSING][\"in_memory\"]:\n return self.dataset[proc_column][idx]\n\n sub_batch = self.dataset[proc_column][idx]\n\n indices = np.empty((3, len(sub_batch)), dtype=np.int64)\n indices[0, :] = sub_batch\n indices[1, :] = np.arange(len(sub_batch))\n indices = indices[:, np.argsort(indices[0])]\n\n with download_h5(self.data_hdf5_fp) as h5_file:\n im_data = h5_file[proc_column + \"_data\"][indices[0, :], :, :]\n indices[2, :] = np.arange(len(sub_batch))\n indices = indices[:, np.argsort(indices[1])]\n return im_data[indices[2, :]]\n\n def get_dataset(self) -> dict[str, np.ndarray]:\n return self.dataset\n\n def __len__(self):\n return self.size\n\n @property\n def processed_data_fp(self) -> str | None:\n return self.data_hdf5_fp\n\n @property\n def in_memory_size_bytes(self) -> int:\n df = self.to_df()\n return df.memory_usage(deep=True).sum() if df is not None else 0\n\n @contextlib.contextmanager\n def initialize_batcher(\n self,\n batch_size: int = 128,\n should_shuffle: bool = True,\n random_seed: int = default_random_seed,\n ignore_last: bool = False,\n distributed: DistributedStrategy = None,\n augmentation_pipeline=None,\n ) -> Batcher:\n sampler = DistributedSampler(\n len(self), shuffle=should_shuffle, random_seed=random_seed, distributed=distributed\n )\n batcher = RandomAccessBatcher(\n self,\n sampler,\n batch_size=batch_size,\n ignore_last=ignore_last,\n augmentation_pipeline=augmentation_pipeline,\n )\n yield batcher\n\n\nclass PandasDatasetManager(DatasetManager):\n def __init__(self, backend: Backend):\n self.backend: Backend = backend\n\n def create(self, dataset, config, training_set_metadata) -> Dataset:\n return PandasDataset(dataset, get_proc_features(config), training_set_metadata.get(DATA_TRAIN_HDF5_FP))\n\n def save(self, cache_path, dataset, config, training_set_metadata, tag) -> Dataset:\n save_hdf5(cache_path, dataset)\n if tag == TRAINING:\n training_set_metadata[DATA_TRAIN_HDF5_FP] = cache_path\n return dataset\n\n def can_cache(self, skip_save_processed_input) -> bool:\n return self.backend.is_coordinator() and not skip_save_processed_input\n\n @property\n def data_format(self) -> str:\n return \"hdf5\"\n"
},
{
"path": "ludwig/data/dataset/ray.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport contextlib\nimport math\nimport queue\nimport threading\nfrom functools import lru_cache\nfrom typing import Any\n\nimport numpy as np\nimport pandas as pd\nfrom pyarrow.fs import FSSpecHandler, PyFileSystem\nfrom ray.data import Dataset as RayNativeDataset\nfrom ray.data import read_parquet\nfrom ray.data.extensions import TensorArray\n\nfrom ludwig.backend.base import Backend\nfrom ludwig.constants import BINARY, CATEGORY, NAME, NUMBER, TYPE\nfrom ludwig.data.batcher.base import Batcher\nfrom ludwig.data.dataset.base import Dataset, DatasetManager\nfrom ludwig.utils.data_utils import DATA_TRAIN_HDF5_FP, DATA_TRAIN_PARQUET_FP\nfrom ludwig.utils.defaults import default_random_seed\nfrom ludwig.utils.fs_utils import get_fs_and_path\nfrom ludwig.utils.misc_utils import get_proc_features\nfrom ludwig.utils.types import DataFrame, Series\n\n_SCALAR_TYPES = {BINARY, CATEGORY, NUMBER}\n\n\ndef cast_as_tensor_dtype(series: Series) -> Series:\n return TensorArray(series)\n\n\ndef read_remote_parquet(path: str):\n fs, path = get_fs_and_path(path)\n return read_parquet(path, filesystem=PyFileSystem(FSSpecHandler(fs)))\n\n\nclass RayDataset(Dataset):\n \"\"\"Wrapper around ray.data.Dataset.\"\"\"\n\n def __init__(\n self,\n df: str | DataFrame,\n features: dict[str, dict],\n training_set_metadata: dict[str, Any],\n backend: Backend,\n ):\n self.df_engine = backend.df_engine\n self.ds = self.df_engine.to_ray_dataset(df) if not isinstance(df, str) else read_remote_parquet(df)\n self.features = features\n self.training_set_metadata = training_set_metadata\n self.data_hdf5_fp = training_set_metadata.get(DATA_TRAIN_HDF5_FP)\n self.data_parquet_fp = training_set_metadata.get(DATA_TRAIN_PARQUET_FP)\n\n def to_ray_dataset(\n self,\n shuffle: bool = True,\n shuffle_seed: int = default_random_seed,\n ) -> RayNativeDataset:\n \"\"\"Returns a ray.data.Dataset, optionally shuffled.\n\n In modern Ray (2.5+), datasets use lazy execution by default, so there's no need for explicit windowing or\n pipelining.\n \"\"\"\n ds = self.ds\n if shuffle:\n ds = ds.random_shuffle(seed=shuffle_seed)\n return ds\n\n @contextlib.contextmanager\n def initialize_batcher(self, batch_size=128, should_shuffle=True, random_seed=0, ignore_last=False, **kwargs):\n ds = self.ds\n if should_shuffle:\n ds = ds.random_shuffle(seed=random_seed)\n yield RayDatasetBatcher(\n ds,\n self.features,\n self.training_set_metadata,\n batch_size,\n self.size,\n )\n\n def __len__(self):\n return self.ds.count()\n\n @property\n def size(self):\n return len(self)\n\n @property\n def in_memory_size_bytes(self):\n return self.ds.size_bytes() if self.ds is not None else 0\n\n def to_df(self, features=None):\n return self.df_engine.from_ray_dataset(self.ds)\n\n def to_scalar_df(self, features=None):\n from ludwig.utils.dataframe_utils import to_scalar_df\n\n return to_scalar_df(self.to_df(features))\n\n\nclass RayDatasetManager(DatasetManager):\n def __init__(self, backend):\n self.backend = backend\n\n def create(self, dataset: str | DataFrame, config: dict[str, Any], training_set_metadata: dict[str, Any]):\n return RayDataset(dataset, get_proc_features(config), training_set_metadata, self.backend)\n\n def save(\n self,\n cache_path: str,\n dataset: DataFrame,\n config: dict[str, Any],\n training_set_metadata: dict[str, Any],\n tag: str,\n ):\n self.backend.df_engine.to_parquet(dataset, cache_path)\n return cache_path\n\n def can_cache(self, skip_save_processed_input):\n return not skip_save_processed_input\n\n @property\n def data_format(self):\n return \"parquet\"\n\n\nclass RayDatasetShard(Dataset):\n \"\"\"Wraps a Ray DataIterator (from ray.train.get_dataset_shard) for distributed training.\"\"\"\n\n def __init__(\n self,\n dataset_shard,\n features: dict[str, dict],\n training_set_metadata: dict[str, Any],\n ):\n self.dataset_shard = dataset_shard\n self.features = features\n self.training_set_metadata = training_set_metadata\n\n @contextlib.contextmanager\n def initialize_batcher(self, batch_size=128, should_shuffle=True, random_seed=0, ignore_last=False, **kwargs):\n yield RayDatasetShardBatcher(\n self.dataset_shard,\n self.features,\n self.training_set_metadata,\n batch_size,\n self.size,\n )\n\n @lru_cache(1)\n def __len__(self):\n # TODO(travis): find way to avoid calling this, as it's expensive\n # DataIterator doesn't have a direct count method; use iter to count\n count = 0\n for batch in self.dataset_shard.iter_batches(batch_size=4096, batch_format=\"pandas\"):\n count += len(batch)\n return count\n\n @property\n def size(self):\n return len(self)\n\n def to_df(self, features=None):\n raise NotImplementedError(\"RayDatasetShard does not support to_df; use full RayDataset instead.\")\n\n def to_scalar_df(self, features=None):\n raise NotImplementedError(\"RayDatasetShard does not support to_scalar_df; use full RayDataset instead.\")\n\n\nclass _BaseBatcher(Batcher):\n \"\"\"Shared batching logic for preparing batches from pandas DataFrames.\"\"\"\n\n def __init__(\n self,\n features: dict[str, dict],\n training_set_metadata: dict[str, Any],\n batch_size: int,\n samples_per_epoch: int,\n ):\n self.batch_size = batch_size\n self.samples_per_epoch = samples_per_epoch\n self.training_set_metadata = training_set_metadata\n\n self.features = features\n self.columns = list(features.keys())\n self.reshape_map = {\n proc_column: training_set_metadata[feature[NAME]].get(\"reshape\")\n for proc_column, feature in features.items()\n }\n\n self.dataset_batch_iter = None\n self._epoch = 0\n self._next_batch = None\n self._last_batch = False\n self._step = 0\n\n def next_batch(self):\n if self.last_batch():\n raise StopIteration()\n\n batch = self._next_batch\n self._fetch_next_batch()\n self._step += 1\n return batch\n\n def last_batch(self):\n return self._last_batch\n\n def set_epoch(self, epoch, batch_size):\n self.batch_size = batch_size\n if epoch != self._epoch:\n self._fetch_next_epoch()\n self._epoch = epoch\n\n @property\n def step(self):\n return self._step\n\n @property\n def steps_per_epoch(self):\n return math.ceil(self.samples_per_epoch / self.batch_size)\n\n def _fetch_next_batch(self):\n if self.dataset_batch_iter is None:\n self._last_batch = True\n return\n\n self._last_batch = False\n try:\n self._next_batch = next(self.dataset_batch_iter)\n except StopIteration:\n self._last_batch = True\n\n def _fetch_next_epoch(self):\n raise NotImplementedError\n\n def _to_tensors_fn(self):\n columns = self.columns\n features = self.features\n\n def to_tensors(df: pd.DataFrame) -> pd.DataFrame:\n for c in columns:\n # do not convert scalar columns: https://github.com/ray-project/ray/issues/20825\n if features[c][TYPE] not in _SCALAR_TYPES:\n df[c] = cast_as_tensor_dtype(df[c])\n elif features[c][TYPE] == BINARY:\n df[c] = df[c].astype(np.bool_)\n return df\n\n return to_tensors\n\n def _prepare_batch(self, batch: pd.DataFrame) -> dict[str, np.ndarray]:\n res = {}\n for c in self.columns:\n if self.features[c][TYPE] not in _SCALAR_TYPES:\n res[c] = np.stack(batch[c].values)\n else:\n res[c] = batch[c].to_numpy()\n\n for c in self.columns:\n reshape = self.reshape_map.get(c)\n if reshape is not None:\n res[c] = res[c].reshape((-1, *reshape))\n return res\n\n\nclass RayDatasetBatcher(_BaseBatcher):\n \"\"\"Batcher for a full ray.data.Dataset (used by non-distributed/local Ray training).\"\"\"\n\n def __init__(\n self,\n dataset: RayNativeDataset,\n features: dict[str, dict],\n training_set_metadata: dict[str, Any],\n batch_size: int,\n samples_per_epoch: int,\n ):\n self.dataset = dataset\n super().__init__(features, training_set_metadata, batch_size, samples_per_epoch)\n self._fetch_next_epoch()\n\n def _fetch_next_epoch(self):\n \"\"\"Create an async reader over the dataset for one epoch.\"\"\"\n self.dataset_batch_iter = self._create_async_reader(self.dataset)\n self._step = 0\n self._fetch_next_batch()\n\n def _create_async_reader(self, dataset: RayNativeDataset):\n q = queue.Queue(maxsize=100)\n batch_size = self.batch_size\n to_tensors = self._to_tensors_fn()\n\n def producer():\n for batch in dataset.map_batches(to_tensors, batch_format=\"pandas\").iter_batches(\n prefetch_batches=1, batch_size=batch_size, batch_format=\"pandas\"\n ):\n res = self._prepare_batch(batch)\n q.put(res)\n q.put(None)\n\n def async_read():\n t = threading.Thread(target=producer)\n t.start()\n while True:\n batch = q.get(block=True)\n if batch is None:\n break\n yield batch\n t.join()\n\n return async_read()\n\n\nclass RayDatasetShardBatcher(_BaseBatcher):\n \"\"\"Batcher for a Ray DataIterator shard (used in distributed training workers).\"\"\"\n\n def __init__(\n self,\n data_iterator,\n features: dict[str, dict],\n training_set_metadata: dict[str, Any],\n batch_size: int,\n samples_per_epoch: int,\n ):\n self.data_iterator = data_iterator\n super().__init__(features, training_set_metadata, batch_size, samples_per_epoch)\n self._fetch_next_epoch()\n\n def _fetch_next_epoch(self):\n \"\"\"Create an async reader from the DataIterator for one epoch.\"\"\"\n self.dataset_batch_iter = self._create_async_reader()\n self._step = 0\n self._fetch_next_batch()\n\n def _create_async_reader(self):\n q = queue.Queue(maxsize=100)\n batch_size = self.batch_size\n to_tensors = self._to_tensors_fn()\n\n def producer():\n for batch in self.data_iterator.iter_batches(\n batch_size=batch_size,\n batch_format=\"pandas\",\n prefetch_batches=1,\n ):\n batch = to_tensors(batch)\n res = self._prepare_batch(batch)\n q.put(res)\n q.put(None)\n\n def async_read():\n t = threading.Thread(target=producer)\n t.start()\n while True:\n batch = q.get(block=True)\n if batch is None:\n break\n yield batch\n t.join()\n\n return async_read()\n"
},
{
"path": "ludwig/data/dataset_synthesizer.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport argparse\nimport logging\nimport os\nimport random\nimport string\nimport sys\nimport uuid\n\nimport numpy as np\nimport pandas as pd\nimport torch\nimport torchaudio\nimport yaml\nfrom packaging import version\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import (\n AUDIO,\n BAG,\n BINARY,\n CATEGORY,\n CATEGORY_DISTRIBUTION,\n DATE,\n DECODER,\n ENCODER,\n H3,\n IMAGE,\n INPUT_FEATURES,\n NAME,\n NUMBER,\n OUTPUT_FEATURES,\n PREPROCESSING,\n SEQUENCE,\n SET,\n TEXT,\n TIMESERIES,\n TYPE,\n VECTOR,\n)\nfrom ludwig.contrib import add_contrib_callback_args\nfrom ludwig.globals import LUDWIG_VERSION\nfrom ludwig.types import ModelConfigDict\nfrom ludwig.utils.data_utils import save_csv\nfrom ludwig.utils.h3_util import components_to_h3\nfrom ludwig.utils.misc_utils import get_from_registry\nfrom ludwig.utils.print_utils import print_ludwig\n\nlogger = logging.getLogger(__name__)\n\n_TORCH_AUDIO_210 = version.parse(torchaudio.__version__) >= version.parse(\"2.1.0\")\n\nletters = string.ascii_letters\n\nDATETIME_FORMATS = {\n \"%m-%d-%Y\": \"{m:02d}-{d:02d}-{Y:04d}\",\n \"%m-%d-%Y %H:%M:%S\": \"{m:02d}-{d:02d}-{Y:04d} {H:02d}:{M:02d}:{S:02d}\",\n \"%m/%d/%Y\": \"{m:02d}/{d:02d}/{Y:04d}\",\n \"%m/%d/%Y %H:%M:%S\": \"{m:02d}/{d:02d}/{Y:04d} {H:02d}:{M:02d}:{S:02d}\",\n \"%m-%d-%y\": \"{m:02d}-{d:02d}-{y:02d}\",\n \"%m-%d-%y %H:%M:%S\": \"{m:02d}-{d:02d}-{y:02d} {H:02d}:{M:02d}:{S:02d}\",\n \"%m/%d/%y\": \"{m:02d}/{d:02d}/{y:02d}\",\n \"%m/%d/%y %H:%M:%S\": \"{m:02d}/{d:02d}/{y:02d} {H:02d}:{M:02d}:{S:02d}\",\n \"%d-%m-%Y\": \"{d:02d}-{m:02d}-{Y:04d}\",\n \"%d-%m-%Y %H:%M:%S\": \"{d:02d}-{m:02d}-{Y:04d} {H:02d}:{M:02d}:{S:02d}\",\n \"%d/%m/%Y\": \"{d:02d}/{m:02d}/{Y:04d}\",\n \"%d/%m/%Y %H:%M:%S\": \"{d:02d}/{m:02d}/{Y:04d} {H:02d}:{M:02d}:{S:02d}\",\n \"%d-%m-%y\": \"{d:02d}-{m:02d}-{y:02d}\",\n \"%d-%m-%y %H:%M:%S\": \"{d:02d}-{m:02d}-{y:02d} {H:02d}:{M:02d}:{S:02d}\",\n \"%d/%m/%y\": \"{d:02d}/{m:02d}/{y:02d}\",\n \"%d/%m/%y %H:%M:%S\": \"{d:02d}/{m:02d}/{y:02d} {H:02d}:{M:02d}:{S:02d}\",\n \"%y-%m-%d\": \"{y:02d}-{m:02d}-{d:02d}\",\n \"%y-%m-%d %H:%M:%S\": \"{y:02d}-{m:02d}-{d:02d} {H:02d}:{M:02d}:{S:02d}\",\n \"%y/%m/%d\": \"{y:02d}/{m:02d}/{d:02d}\",\n \"%y/%m/%d %H:%M:%S\": \"{y:02d}/{m:02d}/{d:02d} {H:02d}:{M:02d}:{S:02d}\",\n \"%Y-%m-%d\": \"{Y:04d}-{m:02d}-{d:02d}\",\n \"%Y-%m-%d %H:%M:%S\": \"{Y:04d}-{m:02d}-{d:02d} {H:02d}:{M:02d}:{S:02d}\",\n \"%Y/%m/%d\": \"{Y:04d}/{m:02d}/{d:02d}\",\n \"%Y/%m/%d %H:%M:%S\": \"{Y:04d}/{m:02d}/{d:02d} {H:02d}:{M:02d}:{S:02d}\",\n \"%y-%d-%m\": \"{y:02d}-{d:02d}-{m:02d}\",\n \"%y-%d-%m %H:%M:%S\": \"{y:02d}-{d:02d}-{m:02d} {H:02d}:{M:02d}:{S:02d}\",\n \"%y/%d/%m\": \"{y:02d}/{d:02d}/{m:02d}\",\n \"%y/%d/%m %H:%M:%S\": \"{y:02d}/{d:02d}/{m:02d} {H:02d}:{M:02d}:{S:02d}\",\n \"%Y-%d-%m\": \"{Y:04d}-{d:02d}-{m:02d}\",\n \"%Y-%d-%m %H:%M:%S\": \"{Y:04d}-{d:02d}-{m:02d} {H:02d}:{M:02d}:{S:02d}\",\n \"%Y/%d/%m\": \"{Y:04d}/{d:02d}/{m:02d}\",\n \"%Y/%d/%m %H:%M:%S\": \"{Y:04d}/{d:02d}/{m:02d} {H:02d}:{M:02d}:{S:02d}\",\n}\n\n\ndef _get_feature_encoder_or_decoder(feature):\n \"\"\"Returns the nested decoder or encoder dictionary for a feature.\n\n If neither encoder nor decoder is present, creates an empty encoder dict and returns it.\n \"\"\"\n if DECODER in feature:\n return feature[DECODER]\n elif ENCODER in feature:\n return feature[ENCODER]\n else:\n feature[ENCODER] = {}\n return feature[ENCODER]\n\n\ndef generate_string(length):\n sequence = []\n for _ in range(length):\n sequence.append(random.choice(letters))\n return \"\".join(sequence)\n\n\ndef build_vocab(size):\n vocab = []\n for _ in range(size):\n vocab.append(generate_string(random.randint(2, 10)))\n return vocab\n\n\ndef return_none(feature):\n return None\n\n\ndef assign_vocab(feature):\n encoder_or_decoder = _get_feature_encoder_or_decoder(feature)\n encoder_or_decoder[\"idx2str\"] = build_vocab(encoder_or_decoder.get(\"vocab_size\", 10))\n encoder_or_decoder[\"vocab_size\"] = len(encoder_or_decoder[\"idx2str\"])\n\n\ndef build_feature_parameters(features):\n feature_parameters = {}\n for feature in features:\n feature_builder_function = get_from_registry(feature[TYPE], parameters_builders_registry)\n feature_parameters[feature[NAME]] = feature_builder_function(feature)\n return feature_parameters\n\n\nparameters_builders_registry = {\n \"category\": assign_vocab,\n \"text\": assign_vocab,\n \"number\": return_none,\n \"binary\": return_none,\n \"set\": assign_vocab,\n \"bag\": assign_vocab,\n \"sequence\": assign_vocab,\n \"timeseries\": return_none,\n \"image\": return_none,\n \"audio\": return_none,\n \"date\": return_none,\n \"h3\": return_none,\n VECTOR: return_none,\n CATEGORY_DISTRIBUTION: return_none,\n}\n\n\n@DeveloperAPI\ndef build_synthetic_dataset_df(dataset_size: int, config: ModelConfigDict) -> pd.DataFrame:\n for feature in config[OUTPUT_FEATURES]:\n if DECODER not in feature:\n feature[DECODER] = {}\n features = config[INPUT_FEATURES] + config[OUTPUT_FEATURES]\n df = build_synthetic_dataset(dataset_size, features)\n data = [next(df) for _ in range(dataset_size + 1)]\n return pd.DataFrame(data[1:], columns=data[0])\n\n\n@DeveloperAPI\ndef build_synthetic_dataset(dataset_size: int, features: list[dict], outdir: str = \".\"):\n \"\"\"Synthesizes a dataset for testing purposes.\n\n :param dataset_size: (int) size of the dataset\n :param features: (List[dict]) list of features to generate in YAML format.\n Provide a list containing one dictionary for each feature,\n each dictionary must include a name, a type\n and can include some generation parameters depending on the type\n :param outdir: (str) Path to an output directory. Used for saving synthetic image and audio files.\n\n Example content for features:\n\n [\n {name: text_1, type: text, vocab_size: 20, max_len: 20},\n {name: text_2, type: text, vocab_size: 20, max_len: 20},\n {name: category_1, type: category, vocab_size: 10},\n {name: category_2, type: category, vocab_size: 15},\n {name: number_1, type: number},\n {name: number_2, type: number},\n {name: binary_1, type: binary},\n {name: binary_2, type: binary},\n {name: set_1, type: set, vocab_size: 20, max_len: 20},\n {name: set_2, type: set, vocab_size: 20, max_len: 20},\n {name: bag_1, type: bag, vocab_size: 20, max_len: 10},\n {name: bag_2, type: bag, vocab_size: 20, max_len: 10},\n {name: sequence_1, type: sequence, vocab_size: 20, max_len: 20},\n {name: sequence_2, type: sequence, vocab_size: 20, max_len: 20},\n {name: timeseries_1, type: timeseries, max_len: 20},\n {name: timeseries_2, type: timeseries, max_len: 20},\n {name: date_1, type: date},\n {name: date_2, type: date},\n {name: h3_1, type: h3},\n {name: h3_2, type: h3},\n {name: vector_1, type: vector},\n {name: vector_2, type: vector},\n ]\n \"\"\"\n build_feature_parameters(features)\n header = []\n for feature in features:\n header.append(feature[NAME])\n\n yield header\n for _ in range(dataset_size):\n yield generate_datapoint(features=features, outdir=outdir)\n\n\ndef generate_datapoint(features: list[dict], outdir: str) -> str | int | bool:\n \"\"\"Returns a synthetic example containing features specified by the features spec.\n\n `outdir` is only used for generating synthetic image and synthetic audio features. Otherwise, it is unused.\n \"\"\"\n datapoint = []\n for feature in features:\n if \"cycle\" in feature and feature[\"cycle\"] is True and feature[TYPE] in cyclers_registry:\n cycler_function = cyclers_registry[feature[TYPE]]\n feature_value = cycler_function(feature)\n else:\n generator_function = get_from_registry(feature[TYPE], generators_registry)\n feature_value = generator_function(feature=feature, outdir=outdir)\n datapoint.append(feature_value)\n return datapoint\n\n\ndef generate_category(feature, outdir: str | None = None) -> str:\n \"\"\"Returns a random category.\n\n `outdir` is unused.\n \"\"\"\n encoder_or_decoder = _get_feature_encoder_or_decoder(feature)\n return random.choice(encoder_or_decoder[\"idx2str\"])\n\n\ndef generate_number(feature, outdir: str | None = None) -> int:\n \"\"\"Returns a random number.\n\n `outdir` is unused.\n \"\"\"\n return random.uniform(feature[\"min\"] if \"min\" in feature else 0, feature[\"max\"] if \"max\" in feature else 1)\n\n\ndef generate_binary(feature, outdir: str | None = None) -> bool:\n \"\"\"Returns a random boolean.\n\n `outdir` is unused.\n \"\"\"\n choices = feature.get(\"bool2str\", [False, True])\n p = feature[\"prob\"] if \"prob\" in feature else 0.5\n return np.random.choice(choices, p=[1 - p, p])\n\n\ndef generate_sequence(feature, outdir: str | None = None) -> str:\n \"\"\"Returns a random sequence.\n\n `outdir` is unused.\n \"\"\"\n encoder_or_decoder = _get_feature_encoder_or_decoder(feature)\n length = encoder_or_decoder.get(\"max_len\", 10)\n if \"min_len\" in encoder_or_decoder:\n length = random.randint(encoder_or_decoder[\"min_len\"], length)\n sequence = [random.choice(encoder_or_decoder[\"idx2str\"]) for _ in range(length)]\n encoder_or_decoder[\"vocab_size\"] = (\n encoder_or_decoder[\"vocab_size\"] + 4\n ) # For special symbols: START, STOP, PAD, UNK.\n return \" \".join(sequence)\n\n\ndef generate_set(feature, outdir: str | None = None) -> str:\n \"\"\"Returns a random set.\n\n `outdir` is unused.\n \"\"\"\n encoder_or_decoder = _get_feature_encoder_or_decoder(feature)\n elems = []\n for _ in range(random.randint(0, encoder_or_decoder.get(\"max_len\", 3))):\n elems.append(random.choice(encoder_or_decoder[\"idx2str\"]))\n return \" \".join(list(set(elems)))\n\n\ndef generate_bag(feature, outdir: str | None = None) -> str:\n \"\"\"Returns a random bag.\n\n `outdir` is unused.\n \"\"\"\n encoder_or_decoder = _get_feature_encoder_or_decoder(feature)\n elems = []\n for _ in range(random.randint(0, encoder_or_decoder.get(\"max_len\", 3))):\n elems.append(random.choice(encoder_or_decoder[\"idx2str\"]))\n return \" \".join(elems)\n\n\ndef generate_text(feature, outdir: str | None = None) -> str:\n \"\"\"Returns random text.\n\n `outdir` is unused.\n \"\"\"\n encoder_or_decoder = _get_feature_encoder_or_decoder(feature)\n length = encoder_or_decoder.get(\"max_len\", 10)\n text = []\n for _ in range(random.randint(length - int(length * 0.2), length)):\n text.append(random.choice(encoder_or_decoder[\"idx2str\"]))\n return \" \".join(text)\n\n\ndef generate_timeseries(feature, max_len=10, outdir: str | None = None) -> str:\n \"\"\"Returns a random timeseries.\n\n `outdir` is unused.\n \"\"\"\n encoder = _get_feature_encoder_or_decoder(feature)\n series = []\n max_len = encoder.get(\"max_len\", max_len)\n series_len = random.randint(max_len - 2, max_len) # simulates variable length\n for _ in range(series_len):\n series.append(str(random.uniform(encoder.get(\"min\", 0), encoder.get(\"max\", 1))))\n return \" \".join(series)\n\n\ndef generate_audio(feature, outdir: str) -> str:\n \"\"\"Generates random audio and saves it to the outdir.\n\n Returns the path to the directory of saved files.\n \"\"\"\n destination_folder = feature.get(\"destination_folder\", outdir)\n if PREPROCESSING in feature:\n audio_length = feature[PREPROCESSING].get(\"audio_file_length_limit_in_s\", 2)\n else:\n audio_length = feature.get(\"audio_file_length_limit_in_s\", 1)\n sampling_rate = 16000\n num_samples = int(audio_length * sampling_rate)\n audio = np.sin(np.arange(num_samples) / 100 * 2 * np.pi) * 2 * (np.random.random(num_samples) - 0.5)\n audio_tensor = torch.tensor(np.array([audio])).type(torch.float32)\n audio_filename = uuid.uuid4().hex[:10].upper() + \".wav\"\n\n if not os.path.exists(destination_folder):\n os.makedirs(destination_folder)\n audio_dest_path = os.path.join(destination_folder, audio_filename)\n\n try:\n if _TORCH_AUDIO_210:\n torchaudio.save(audio_dest_path, audio_tensor, sample_rate=sampling_rate, backend=\"sox\")\n torchaudio.save(audio_dest_path, audio_tensor, sampling_rate)\n\n except OSError as e:\n raise OSError(f\"Unable to save audio to disk: {e}\")\n\n return audio_dest_path\n\n\ndef generate_image(feature, outdir: str, save_as_numpy: bool = False) -> str:\n \"\"\"Generates random images and saves it to the outdir.\n\n Returns the path to the directory of saved files.\n \"\"\"\n save_as_numpy = feature.get(\"save_as_numpy\", save_as_numpy)\n\n try:\n from torchvision.io import write_png\n except ImportError:\n logger.error(\n \" torchvision is not installed. \"\n \"In order to install all image feature dependencies run \"\n \"pip install ludwig[image]\"\n )\n sys.exit(-1)\n\n # Read num_channels, width, height\n destination_folder = feature.get(\"destination_folder\", outdir)\n if PREPROCESSING in feature:\n height = feature[PREPROCESSING].get(\"height\", 28)\n width = feature[PREPROCESSING].get(\"width\", 28)\n num_channels = feature[PREPROCESSING].get(\"num_channels\", 1)\n else:\n encoder = _get_feature_encoder_or_decoder(feature)\n height = encoder.get(\"height\", 28)\n width = encoder.get(\"width\", 28)\n num_channels = encoder.get(\"num_channels\", 1)\n\n if width <= 0 or height <= 0 or num_channels < 1:\n raise ValueError(\"Invalid arguments for generating images\")\n\n # Create a Random Image\n img = torch.randint(0, 255, (num_channels, width, height), dtype=torch.uint8)\n\n # Generate a unique random filename\n image_filename = uuid.uuid4().hex[:10].upper() + \".png\"\n\n # Save the image to disk either in a specified location/new folder\n if not os.path.exists(destination_folder):\n os.makedirs(destination_folder)\n image_dest_path = os.path.join(destination_folder, image_filename)\n try:\n # save_image(torch.from_numpy(img.astype(\"uint8\")), image_dest_path)\n if save_as_numpy:\n with open(image_dest_path, \"wb\") as f:\n np.save(f, img.detach().cpu().numpy())\n else:\n write_png(img, image_dest_path)\n except OSError as e:\n raise OSError(f\"Unable to save images to disk: {e}\")\n\n return image_dest_path\n\n\ndef generate_datetime(feature, outdir: str | None = None) -> str:\n \"\"\"Generates a random date time, picking a format among different types.\n\n If no format is specified, the first one is used.\n \"\"\"\n if \"datetime_format\" in feature:\n datetime_generation_format = DATETIME_FORMATS[feature[\"datetime_format\"]]\n elif \"preprocessing\" in feature and \"datetime_format\" in feature[\"preprocessing\"]:\n datetime_generation_format = DATETIME_FORMATS[feature[\"preprocessing\"][\"datetime_format\"]]\n else:\n datetime_generation_format = DATETIME_FORMATS[next(iter(DATETIME_FORMATS))]\n\n y = random.randint(1, 99)\n Y = random.randint(1, 9999)\n m = random.randint(1, 12)\n d = random.randint(1, 28)\n H = random.randint(1, 12)\n M = random.randint(1, 59)\n S = random.randint(1, 59)\n\n return datetime_generation_format.format(y=y, Y=Y, m=m, d=d, H=H, M=M, S=S)\n\n\ndef generate_h3(feature, outdir: str | None = None) -> str:\n \"\"\"Returns a random h3.\n\n `outdir` is unused.\n \"\"\"\n resolution = random.randint(0, 15) # valid values [0, 15]\n h3_components = {\n \"mode\": 1, # we can avoid testing other modes\n \"edge\": 0, # only used in other modes\n \"resolution\": resolution,\n \"base_cell\": random.randint(0, 121), # valid values [0, 121]\n # valid values [0, 7]\n \"cells\": [random.randint(0, 7) for _ in range(resolution)],\n }\n\n return components_to_h3(h3_components)\n\n\ndef generate_vector(feature, outdir: str | None = None) -> str:\n \"\"\"Returns a random vector.\n\n `outdir` is unused.\n \"\"\"\n # Space delimited string with floating point numbers\n if PREPROCESSING in feature:\n vector_size = feature[PREPROCESSING].get(\"vector_size\", 10)\n else:\n vector_size = feature.get(\"vector_size\", 10)\n return \" \".join([str(100 * random.random()) for _ in range(vector_size)])\n\n\ndef generate_category_distribution(feature, outdir: str | None = None) -> str:\n \"\"\"Returns a random category distribution.\n\n `outdir` is unused.\n \"\"\"\n # Space delimited string with floating point numbers that sum to 1\n preprocessing = feature.get(PREPROCESSING, {})\n vector_size = len(preprocessing.get(\"vocab\", [\"a\", \"b\", \"c\"]))\n v = np.random.rand(vector_size)\n v = v / v.sum()\n return \" \".join([str(x) for x in v])\n\n\ngenerators_registry = {\n BINARY: generate_binary,\n NUMBER: generate_number,\n CATEGORY: generate_category,\n SET: generate_set,\n BAG: generate_bag,\n SEQUENCE: generate_sequence,\n TEXT: generate_text,\n TIMESERIES: generate_timeseries,\n IMAGE: generate_image,\n AUDIO: generate_audio,\n H3: generate_h3,\n DATE: generate_datetime,\n VECTOR: generate_vector,\n CATEGORY_DISTRIBUTION: generate_category_distribution,\n}\n\ncategory_cycle = 0\n\n\ndef cycle_category(feature):\n global category_cycle\n idx2str = feature[DECODER][\"idx2str\"] if DECODER in feature else feature[ENCODER][\"idx2str\"]\n if category_cycle >= len(idx2str):\n category_cycle = 0\n category = idx2str[category_cycle]\n category_cycle += 1\n return category\n\n\nbinary_cycle = False\n\n\ndef cycle_binary(feature):\n global binary_cycle\n if binary_cycle:\n binary_cycle = False\n return True\n else:\n binary_cycle = True\n return False\n\n\ncyclers_registry = {\"category\": cycle_category, \"binary\": cycle_binary}\n\n\ndef cli_synthesize_dataset(dataset_size: int, features: list[dict], output_path: str, **kwargs) -> None:\n \"\"\"Symthesizes a dataset for testing purposes.\n\n :param dataset_size: (int) size of the dataset\n :param features: (List[dict]) list of features to generate in YAML format.\n Provide a list contaning one dictionary for each feature,\n each dictionary must include a name, a type\n and can include some generation parameters depending on the type\n :param output_path: (str) path where to save the output CSV file\n\n Example content for features:\n\n [\n {name: text_1, type: text, vocab_size: 20, max_len: 20},\n {name: text_2, type: text, vocab_size: 20, max_len: 20},\n {name: category_1, type: category, vocab_size: 10},\n {name: category_2, type: category, vocab_size: 15},\n {name: number_1, type: number},\n {name: number_2, type: number},\n {name: binary_1, type: binary},\n {name: binary_2, type: binary},\n {name: set_1, type: set, vocab_size: 20, max_len: 20},\n {name: set_2, type: set, vocab_size: 20, max_len: 20},\n {name: bag_1, type: bag, vocab_size: 20, max_len: 10},\n {name: bag_2, type: bag, vocab_size: 20, max_len: 10},\n {name: sequence_1, type: sequence, vocab_size: 20, max_len: 20},\n {name: sequence_2, type: sequence, vocab_size: 20, max_len: 20},\n {name: timeseries_1, type: timeseries, max_len: 20},\n {name: timeseries_2, type: timeseries, max_len: 20},\n {name: date_1, type: date},\n {name: date_2, type: date},\n {name: h3_1, type: h3},\n {name: h3_2, type: h3},\n {name: vector_1, type: vector},\n {name: vector_2, type: vector},\n ]\n \"\"\"\n if dataset_size is None or features is None or output_path is None:\n raise ValueError(\n \"Missing one or more required parameters: '--dataset_size', \" \"'--features' or '--output_path'\"\n )\n dataset = build_synthetic_dataset(dataset_size, features)\n save_csv(output_path, dataset)\n\n\ndef cli(sys_argv):\n parser = argparse.ArgumentParser(\n description=\"This script generates a synthetic dataset.\",\n prog=\"ludwig synthesize_dataset\",\n usage=\"%(prog)s [options]\",\n )\n parser.add_argument(\"-od\", \"--output_path\", type=str, help=\"output CSV file path\")\n parser.add_argument(\"-d\", \"--dataset_size\", help=\"size of the dataset\", type=int, default=100)\n parser.add_argument(\n \"-f\",\n \"--features\",\n default=\"[\\\n {name: text_1, type: text, vocab_size: 20, max_len: 20}, \\\n {name: text_2, type: text, vocab_size: 20, max_len: 20}, \\\n {name: category_1, type: category, vocab_size: 10}, \\\n {name: category_2, type: category, vocab_size: 15}, \\\n {name: number_1, type: number}, \\\n {name: number_2, type: number}, \\\n {name: binary_1, type: binary}, \\\n {name: binary_2, type: binary}, \\\n {name: set_1, type: set, vocab_size: 20, max_len: 20}, \\\n {name: set_2, type: set, vocab_size: 20, max_len: 20}, \\\n {name: bag_1, type: bag, vocab_size: 20, max_len: 10}, \\\n {name: bag_2, type: bag, vocab_size: 20, max_len: 10}, \\\n {name: sequence_1, type: sequence, vocab_size: 20, max_len: 20}, \\\n {name: sequence_2, type: sequence, vocab_size: 20, max_len: 20}, \\\n {name: timeseries_1, type: timeseries, max_len: 20}, \\\n {name: timeseries_2, type: timeseries, max_len: 20}, \\\n {name: date_1, type: date}, \\\n {name: date_2, type: date}, \\\n {name: h3_1, type: h3}, \\\n {name: h3_2, type: h3}, \\\n {name: vector_1, type: vector}, \\\n {name: vector_2, type: vector}, \\\n ]\",\n type=yaml.safe_load,\n help=\"list of features to generate in YAML format. \"\n \"Provide a list containing one dictionary for each feature, \"\n \"each dictionary must include a name, a type \"\n \"and can include some generation parameters depending on the type\",\n )\n add_contrib_callback_args(parser)\n args = parser.parse_args(sys_argv)\n\n args.callbacks = args.callbacks or []\n for callback in args.callbacks:\n callback.on_cmdline(\"synthesize_dataset\", *sys_argv)\n\n # No log level parameter this is placeholder if we add at later date\n # args.logging_level = get_logging_level_registry[args.logging_level]\n # logging.getLogger('ludwig').setLevel(\n # args.logging_level\n # )\n # global logger\n # logger = logging.getLogger('ludwig.data.dataset_synthesizer')\n\n print_ludwig(\"Synthesize Dataset\", LUDWIG_VERSION)\n\n cli_synthesize_dataset(**vars(args))\n\n\nif __name__ == \"__main__\":\n cli(sys.argv[1:])\n"
},
{
"path": "ludwig/data/negative_sampling.py",
"content": "import logging\nimport time\nfrom typing import Any\n\nimport numpy as np\nimport pandas as pd\nimport scipy\n\nfrom ludwig.utils.types import DataFrame\n\n\ndef _negative_sample_user(interaction_row: np.array, neg_pos_ratio: int, extra_samples: int) -> tuple[list[int], int]:\n \"\"\"Returns a list of negative item indices for given user-item interactions.\n\n If there are not enough negative items, takes all of them and adds the difference to the extra_samples\n otherwise, samples with replacement.\n\n Params:\n interaction_row: user-item interaction row\n neg_pos_ratio: number of negative samples per positive sample\n extra_samples: number of additional samples to add to the negative sample list\n Returns:\n Tuple of list of negative item indices and number of extra samples\n \"\"\"\n # Find all items that are not interacted with by the user\n neg_items = np.where(interaction_row == 0)[1]\n available_samples = len(neg_items)\n\n # Randomly sample negative items\n npos = interaction_row.shape[1] - len(neg_items)\n samples_required = npos * neg_pos_ratio + extra_samples\n should_sample = samples_required <= available_samples\n\n neg_items = np.random.choice(neg_items, samples_required, replace=False) if should_sample else neg_items\n\n return neg_items.tolist(), max(0, samples_required - available_samples)\n\n\ndef negative_sample(\n df: DataFrame,\n user_id_col: str = \"customer_id\",\n item_id_col: str = \"article_id\",\n label_col: str = \"label\",\n neg_pos_ratio: int = 1,\n neg_val: Any = 0,\n log_pct: int = 0,\n):\n \"\"\"Negative sampling for implicit feedback datasets.\n\n Params:\n df: DataFrame containing user-item interactions\n user_id_col: column name for user ids\n item_id_col: column name for item ids\n label_col: column name for interaction labels (e.g. 1 for positive interaction)\n n_neg: number of negative samples per positive sample\n neg_val: label value for the negative samples\n percent_print: print progress every percent_print percent. 0 to disable\n Returns:\n Input DataFrame with negative samples appended\n\n Source: https://petamind.com/fast-uniform-negative-sampling-for-rating-matrix/\n \"\"\"\n # TODO(joppe): support out of memory negative sampling using Dask\n if not isinstance(df, pd.DataFrame):\n df = df.compute()\n\n # Initialize sparse COOrdinate matrix from users and items in existing interactions\n user_id_cat = df[user_id_col].astype(\"category\").cat\n user_id_codes = user_id_cat.codes.values\n\n item_id_cat = df[item_id_col].astype(\"category\").cat\n item_id_codes = item_id_cat.codes.values\n\n interactions_sparse = scipy.sparse.coo_matrix((df[label_col], (user_id_codes, item_id_codes)))\n\n # Convert to dense user-item matrix so we can iterate\n interactions_dense = interactions_sparse.todense()\n\n nrows = interactions_dense.shape[0]\n niter_log = int(nrows * log_pct / 100)\n start_time = time.time()\n\n user_indices, item_indices = [], []\n extra_samples = 0\n for user_idx, interaction_row in enumerate(interactions_dense):\n if log_pct > 0 and user_idx % niter_log == 0:\n logging.info(\n f\"Negative sampling progress: {float(user_idx) * 100 / nrows:0.0f}% \"\n f\"in {time.time() - start_time:0.2f}s\"\n )\n\n neg_items_for_user, extra_samples = _negative_sample_user(interaction_row, neg_pos_ratio, extra_samples)\n\n # Add to negative user-item pairs\n item_indices += neg_items_for_user\n user_indices += [user_idx] * len(neg_items_for_user)\n\n negative_samples = pd.DataFrame(\n {\n # Map back to original user and item ids\n user_id_col: user_id_cat.categories[user_indices],\n item_id_col: item_id_cat.categories[item_indices],\n label_col: [neg_val] * len(item_indices),\n }\n )\n\n return pd.concat([df[[user_id_col, item_id_col, label_col]], negative_samples])\n"
},
{
"path": "ludwig/data/postprocessing.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport os\nfrom typing import Any, Optional\n\nimport numpy as np\nimport pandas as pd\nimport torch\n\nfrom ludwig.backend import LOCAL_BACKEND\nfrom ludwig.data.utils import convert_to_dict\nfrom ludwig.utils.data_utils import DATAFRAME_FORMATS, DICT_FORMATS\nfrom ludwig.utils.dataframe_utils import to_numpy_dataset\nfrom ludwig.utils.fs_utils import has_remote_protocol, open_file\nfrom ludwig.utils.misc_utils import get_from_registry\nfrom ludwig.utils.strings_utils import make_safe_filename\nfrom ludwig.utils.types import DataFrame\n\n\ndef postprocess(\n predictions,\n output_features,\n training_set_metadata,\n output_directory=\"\",\n backend=LOCAL_BACKEND,\n skip_save_unprocessed_output=False,\n) -> DataFrame:\n if not backend.is_coordinator():\n # Only save unprocessed output on the coordinator\n skip_save_unprocessed_output = True\n\n saved_keys = set()\n if not skip_save_unprocessed_output:\n _save_as_numpy(predictions, output_directory, saved_keys, backend)\n\n def postprocess_batch(df):\n for of_name, output_feature in output_features.items():\n df = output_feature.postprocess_predictions(\n df,\n training_set_metadata[of_name],\n )\n return df\n\n # We disable tensor extension casting here because this step is the final data processing step and\n # we do not expect return to Ray Datasets after this point. The dtype of the predictions will be\n # whatever they would be if we did all postprocessing in Dask.\n predictions = backend.df_engine.map_batches(predictions, postprocess_batch, enable_tensor_extension_casting=False)\n\n # Save any new columns but do not save the original columns again\n if not skip_save_unprocessed_output:\n _save_as_numpy(predictions, output_directory, saved_keys, backend)\n\n return predictions\n\n\ndef _save_as_numpy(predictions, output_directory, saved_keys, backend):\n predictions = predictions[[c for c in predictions.columns if c not in saved_keys]]\n npy_filename = os.path.join(output_directory, \"{}.npy\")\n numpy_predictions = to_numpy_dataset(predictions, backend)\n for k, v in numpy_predictions.items():\n k = k.replace(\"<\", \"[\").replace(\">\", \"]\") # Replace and with [UNK], [PAD]\n if k not in saved_keys:\n if has_remote_protocol(output_directory):\n with open_file(npy_filename.format(make_safe_filename(k)), mode=\"wb\") as f:\n np.save(f, v)\n else:\n np.save(npy_filename.format(make_safe_filename(k)), v)\n saved_keys.add(k)\n\n\ndef convert_dict_to_df(predictions: dict[str, dict[str, list[Any] | torch.Tensor | np.ndarray]]) -> pd.DataFrame:\n \"\"\"Converts a dictionary of predictions into a pandas DataFrame.\n\n Example format of predictions dictionary:\n\n {\n \"binary_C82EB\": {\n \"predictions\": torch.tensor([True, True, True, False]),\n \"probabilities\": torch.tensor([[0.4777, 0.5223], [0.4482, 0.5518], [0.4380, 0.5620], [0.5059, 0.4941]]),\n },\n \"category_1491D\": {\n \"predictions\": [\"NkNUG\", \"NkNUG\", \"NkNUG\", \"NkNUG\"],\n \"probabilities\": torch.tensor(\n [\n [0.1058, 0.4366, 0.1939, 0.2637],\n [0.0816, 0.4807, 0.1978, 0.2399],\n [0.0907, 0.4957, 0.1829, 0.2308],\n [0.0728, 0.5015, 0.1900, 0.2357],\n ]\n ),\n },\n \"num_7B25F\": {\"predictions\": torch.tensor([2.0436, 2.1158, 2.1222, 2.1964])},\n }\n \"\"\"\n output = {}\n for of_name, preds_dict in predictions.items():\n for key, value in preds_dict.items():\n output_key = f\"{of_name}_{key}\"\n if not isinstance(value, list):\n value = value.tolist()\n output[output_key] = value\n return pd.DataFrame.from_dict(output)\n\n\ndef convert_predictions(\n predictions, output_features, return_type=\"dict\", backend: Optional[\"Backend\"] = None # noqa: F821\n):\n convert_fn = get_from_registry(return_type, conversion_registry)\n return convert_fn(predictions, output_features, backend)\n\n\ndef convert_to_df(\n predictions,\n output_features,\n backend: Optional[\"Backend\"] = None, # noqa: F821\n):\n return predictions\n\n\nconversion_registry = {\n **{format: convert_to_dict for format in DICT_FORMATS},\n **{format: convert_to_df for format in DATAFRAME_FORMATS},\n}\n"
},
{
"path": "ludwig/data/preprocessing.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport contextlib\nimport logging\nimport warnings\nfrom abc import ABC, abstractmethod\n\nimport numpy as np\nimport pandas as pd\nimport torch\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.backend import Backend, LOCAL_BACKEND\nfrom ludwig.config_validation.preprocessing import check_global_max_sequence_length_fits_prompt_template\nfrom ludwig.constants import (\n BFILL,\n CHECKSUM,\n COLUMN,\n DEFAULTS,\n DROP_ROW,\n ENCODER,\n FFILL,\n FILL_WITH_CONST,\n FILL_WITH_FALSE,\n FILL_WITH_MEAN,\n FILL_WITH_MODE,\n FILL_WITH_TRUE,\n FULL,\n META,\n MIN_DATASET_SPLIT_ROWS,\n MODEL_ECD,\n NAME,\n NUMBER,\n PREPROCESSING,\n PROC_COLUMN,\n SPLIT,\n SRC,\n TEST,\n TEXT,\n TRAINING,\n TYPE,\n VALIDATION,\n)\nfrom ludwig.data.cache.manager import DatasetCache\nfrom ludwig.data.cache.types import wrap\nfrom ludwig.data.concatenate_datasets import concatenate_df, concatenate_files, concatenate_splits\nfrom ludwig.data.dataset.base import Dataset\nfrom ludwig.data.prompt import format_input_with_prompt, index_column\nfrom ludwig.data.split import get_splitter, split_dataset\nfrom ludwig.data.utils import get_input_and_output_features, set_fixed_split\nfrom ludwig.datasets import load_dataset_uris\nfrom ludwig.features.feature_registries import get_base_type_registry\nfrom ludwig.models.embedder import create_embed_batch_size_evaluator, create_embed_transform_fn\nfrom ludwig.schema.encoders.utils import get_encoder_cls\nfrom ludwig.types import FeatureConfigDict, ModelConfigDict, PreprocessingConfigDict, TrainingSetMetadataDict\nfrom ludwig.utils import data_utils, strings_utils\nfrom ludwig.utils.backward_compatibility import upgrade_metadata\nfrom ludwig.utils.data_utils import (\n CACHEABLE_FORMATS,\n CSV_FORMATS,\n DATA_TEST_PARQUET_FP,\n DATA_TRAIN_HDF5_FP,\n DATA_TRAIN_PARQUET_FP,\n DATA_VALIDATION_PARQUET_FP,\n DATAFRAME_FORMATS,\n DICT_FORMATS,\n EXCEL_FORMATS,\n FEATHER_FORMATS,\n figure_data_format,\n FWF_FORMATS,\n get_split_path,\n HDF5_FORMATS,\n HTML_FORMATS,\n JSON_FORMATS,\n JSONL_FORMATS,\n ORC_FORMATS,\n override_in_memory_flag,\n PARQUET_FORMATS,\n PICKLE_FORMATS,\n read_csv,\n read_excel,\n read_feather,\n read_fwf,\n read_html,\n read_json,\n read_jsonl,\n read_orc,\n read_parquet,\n read_pickle,\n read_sas,\n read_spss,\n read_stata,\n read_tsv,\n sanitize_column_names,\n SAS_FORMATS,\n SPSS_FORMATS,\n STATA_FORMATS,\n TSV_FORMATS,\n)\nfrom ludwig.utils.dataframe_utils import is_dask_series_or_df\nfrom ludwig.utils.defaults import (\n default_prediction_preprocessing_parameters,\n default_random_seed,\n default_training_preprocessing_parameters,\n)\nfrom ludwig.utils.fs_utils import file_lock, path_exists\nfrom ludwig.utils.misc_utils import get_from_registry, merge_dict\nfrom ludwig.utils.types import DataFrame, Series\n\n# Opt-in to future pandas behavior: fillna/ffill/bfill will no longer silently downcast dtypes\npd.set_option(\"future.no_silent_downcasting\", True)\n\nREPARTITIONING_FEATURE_TYPES = {\"image\", \"audio\"}\n\nlogger = logging.getLogger(__name__)\n\n\nclass DataFormatPreprocessor(ABC):\n @staticmethod\n @abstractmethod\n def preprocess_for_training(\n config,\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n callbacks=None,\n ):\n pass\n\n @staticmethod\n @abstractmethod\n def preprocess_for_prediction(\n config, dataset, features, preprocessing_params, training_set_metadata, backend, callbacks\n ):\n pass\n\n @staticmethod\n @abstractmethod\n def prepare_processed_data(\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n ):\n pass\n\n\nclass DictPreprocessor(DataFormatPreprocessor):\n @staticmethod\n def preprocess_for_training(\n config,\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n callbacks=None,\n ):\n num_overrides = override_in_memory_flag(features, True)\n if num_overrides > 0:\n logger.warning(\"Using in_memory = False is not supported \" \"with {} data format.\".format(\"dict\"))\n\n df_engine = backend.df_engine\n if dataset is not None:\n dataset = df_engine.from_pandas(pd.DataFrame(dataset))\n if training_set is not None:\n training_set = df_engine.from_pandas(pd.DataFrame(training_set))\n if validation_set is not None:\n validation_set = df_engine.from_pandas(pd.DataFrame(validation_set))\n if test_set is not None:\n test_set = df_engine.from_pandas(pd.DataFrame(test_set))\n\n return _preprocess_df_for_training(\n config,\n features,\n dataset,\n training_set,\n validation_set,\n test_set,\n training_set_metadata=training_set_metadata,\n preprocessing_params=preprocessing_params,\n backend=backend,\n random_seed=random_seed,\n )\n\n @staticmethod\n def preprocess_for_prediction(\n config, dataset, features, preprocessing_params, training_set_metadata, backend, callbacks\n ):\n dataset, training_set_metadata = build_dataset(\n config,\n pd.DataFrame(dataset),\n features,\n preprocessing_params,\n mode=\"prediction\",\n metadata=training_set_metadata,\n backend=backend,\n callbacks=callbacks,\n )\n return dataset, training_set_metadata, None\n\n\nclass DataFramePreprocessor(DataFormatPreprocessor):\n @staticmethod\n def preprocess_for_training(\n config,\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n callbacks=None,\n ):\n num_overrides = override_in_memory_flag(features, True)\n if num_overrides > 0:\n logger.warning(\"Using in_memory = False is not supported \" \"with {} data format.\".format(\"dataframe\"))\n\n if isinstance(dataset, pd.DataFrame):\n dataset = backend.df_engine.from_pandas(dataset)\n\n return _preprocess_df_for_training(\n config,\n features,\n dataset,\n training_set,\n validation_set,\n test_set,\n training_set_metadata=training_set_metadata,\n preprocessing_params=preprocessing_params,\n backend=backend,\n random_seed=random_seed,\n callbacks=callbacks,\n )\n\n @staticmethod\n def preprocess_for_prediction(\n config, dataset, features, preprocessing_params, training_set_metadata, backend, callbacks\n ):\n if isinstance(dataset, pd.DataFrame):\n dataset = backend.df_engine.from_pandas(dataset)\n\n dataset, training_set_metadata = build_dataset(\n config,\n dataset,\n features,\n preprocessing_params,\n mode=\"prediction\",\n metadata=training_set_metadata,\n backend=backend,\n callbacks=callbacks,\n )\n return dataset, training_set_metadata, None\n\n\nclass CSVPreprocessor(DataFormatPreprocessor):\n @staticmethod\n def preprocess_for_training(\n config,\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n callbacks=None,\n ):\n return _preprocess_file_for_training(\n config,\n features,\n dataset,\n training_set,\n validation_set,\n test_set,\n read_fn=read_csv,\n training_set_metadata=training_set_metadata,\n skip_save_processed_input=skip_save_processed_input,\n preprocessing_params=preprocessing_params,\n backend=backend,\n random_seed=random_seed,\n callbacks=callbacks,\n )\n\n @staticmethod\n def preprocess_for_prediction(\n config, dataset, features, preprocessing_params, training_set_metadata, backend, callbacks\n ):\n dataset_df = read_csv(dataset, df_lib=backend.df_engine.df_lib)\n training_set_metadata[SRC] = dataset\n dataset, training_set_metadata = build_dataset(\n config,\n dataset_df,\n features,\n preprocessing_params,\n mode=\"prediction\",\n metadata=training_set_metadata,\n backend=backend,\n callbacks=callbacks,\n )\n return dataset, training_set_metadata, None\n\n\nclass TSVPreprocessor(DataFormatPreprocessor):\n @staticmethod\n def preprocess_for_training(\n config,\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n callbacks=None,\n ):\n return _preprocess_file_for_training(\n config,\n features,\n dataset,\n training_set,\n validation_set,\n test_set,\n read_fn=read_tsv,\n training_set_metadata=training_set_metadata,\n skip_save_processed_input=skip_save_processed_input,\n preprocessing_params=preprocessing_params,\n backend=backend,\n random_seed=random_seed,\n callbacks=callbacks,\n )\n\n @staticmethod\n def preprocess_for_prediction(\n config, dataset, features, preprocessing_params, training_set_metadata, backend, callbacks\n ):\n dataset_df = read_tsv(dataset, df_lib=backend.df_engine.df_lib)\n training_set_metadata[SRC] = dataset\n dataset, training_set_metadata = build_dataset(\n config,\n dataset_df,\n features,\n preprocessing_params,\n mode=\"prediction\",\n metadata=training_set_metadata,\n backend=backend,\n callbacks=callbacks,\n )\n return dataset, training_set_metadata, None\n\n\nclass JSONPreprocessor(DataFormatPreprocessor):\n @staticmethod\n def preprocess_for_training(\n config,\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n callbacks=None,\n ):\n return _preprocess_file_for_training(\n config,\n features,\n dataset,\n training_set,\n validation_set,\n test_set,\n read_fn=read_json,\n training_set_metadata=training_set_metadata,\n skip_save_processed_input=skip_save_processed_input,\n preprocessing_params=preprocessing_params,\n backend=backend,\n random_seed=random_seed,\n callbacks=callbacks,\n )\n\n @staticmethod\n def preprocess_for_prediction(\n config, dataset, features, preprocessing_params, training_set_metadata, backend, callbacks\n ):\n dataset_df = read_json(dataset, backend.df_engine.df_lib)\n training_set_metadata[SRC] = dataset\n dataset, training_set_metadata = build_dataset(\n config,\n dataset_df,\n features,\n preprocessing_params,\n mode=\"prediction\",\n metadata=training_set_metadata,\n backend=backend,\n callbacks=callbacks,\n )\n return dataset, training_set_metadata, None\n\n\nclass JSONLPreprocessor(DataFormatPreprocessor):\n @staticmethod\n def preprocess_for_training(\n config,\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n callbacks=None,\n ):\n return _preprocess_file_for_training(\n config,\n features,\n dataset,\n training_set,\n validation_set,\n test_set,\n read_fn=read_jsonl,\n training_set_metadata=training_set_metadata,\n skip_save_processed_input=skip_save_processed_input,\n preprocessing_params=preprocessing_params,\n backend=backend,\n random_seed=random_seed,\n callbacks=callbacks,\n )\n\n @staticmethod\n def preprocess_for_prediction(\n config, dataset, features, preprocessing_params, training_set_metadata, backend, callbacks\n ):\n dataset_df = read_jsonl(dataset, backend.df_engine.df_lib)\n training_set_metadata[SRC] = dataset\n dataset, training_set_metadata = build_dataset(\n config,\n dataset_df,\n features,\n preprocessing_params,\n mode=\"prediction\",\n metadata=training_set_metadata,\n backend=backend,\n callbacks=callbacks,\n )\n return dataset, training_set_metadata, None\n\n\nclass ExcelPreprocessor(DataFormatPreprocessor):\n @staticmethod\n def preprocess_for_training(\n config,\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n callbacks=None,\n ):\n return _preprocess_file_for_training(\n config,\n features,\n dataset,\n training_set,\n validation_set,\n test_set,\n read_fn=read_excel,\n training_set_metadata=training_set_metadata,\n skip_save_processed_input=skip_save_processed_input,\n preprocessing_params=preprocessing_params,\n backend=backend,\n random_seed=random_seed,\n callbacks=callbacks,\n )\n\n @staticmethod\n def preprocess_for_prediction(\n config, dataset, features, preprocessing_params, training_set_metadata, backend, callbacks\n ):\n dataset_df = read_excel(dataset, backend.df_engine.df_lib)\n training_set_metadata[SRC] = dataset\n dataset, training_set_metadata = build_dataset(\n config,\n dataset_df,\n features,\n preprocessing_params,\n mode=\"prediction\",\n metadata=training_set_metadata,\n backend=backend,\n callbacks=callbacks,\n )\n return dataset, training_set_metadata, None\n\n\nclass ParquetPreprocessor(DataFormatPreprocessor):\n @staticmethod\n def preprocess_for_training(\n config,\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n callbacks=None,\n ):\n return _preprocess_file_for_training(\n config,\n features,\n dataset,\n training_set,\n validation_set,\n test_set,\n read_fn=read_parquet,\n training_set_metadata=training_set_metadata,\n skip_save_processed_input=skip_save_processed_input,\n preprocessing_params=preprocessing_params,\n backend=backend,\n random_seed=random_seed,\n callbacks=callbacks,\n )\n\n @staticmethod\n def preprocess_for_prediction(\n config, dataset, features, preprocessing_params, training_set_metadata, backend, callbacks\n ):\n dataset_df = read_parquet(dataset, backend.df_engine.df_lib)\n training_set_metadata[SRC] = dataset\n dataset, training_set_metadata = build_dataset(\n config,\n dataset_df,\n features,\n preprocessing_params,\n mode=\"prediction\",\n metadata=training_set_metadata,\n backend=backend,\n callbacks=callbacks,\n )\n return dataset, training_set_metadata, None\n\n @staticmethod\n def prepare_processed_data(\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n ):\n test_set = test_set if test_set and path_exists(test_set) else None\n if test_set and isinstance(test_set, str) and DATA_TEST_PARQUET_FP not in training_set_metadata:\n training_set_metadata[DATA_TEST_PARQUET_FP] = test_set\n\n validation_set = validation_set if validation_set and path_exists(validation_set) else None\n if (\n validation_set\n and isinstance(validation_set, str)\n and DATA_VALIDATION_PARQUET_FP not in training_set_metadata\n ):\n training_set_metadata[DATA_VALIDATION_PARQUET_FP] = validation_set\n\n if training_set and isinstance(training_set, str) and DATA_TRAIN_PARQUET_FP not in training_set_metadata:\n training_set_metadata[DATA_TRAIN_PARQUET_FP] = training_set\n return training_set, test_set, validation_set, training_set_metadata\n\n\nclass PicklePreprocessor(DataFormatPreprocessor):\n @staticmethod\n def preprocess_for_training(\n config,\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n callbacks=None,\n ):\n return _preprocess_file_for_training(\n config,\n features,\n dataset,\n training_set,\n validation_set,\n test_set,\n read_fn=read_pickle,\n training_set_metadata=training_set_metadata,\n skip_save_processed_input=skip_save_processed_input,\n preprocessing_params=preprocessing_params,\n backend=backend,\n random_seed=random_seed,\n callbacks=callbacks,\n )\n\n @staticmethod\n def preprocess_for_prediction(\n config, dataset, features, preprocessing_params, training_set_metadata, backend, callbacks\n ):\n dataset_df = read_pickle(dataset, backend.df_engine.df_lib)\n training_set_metadata[SRC] = dataset\n dataset, training_set_metadata = build_dataset(\n config,\n dataset_df,\n features,\n preprocessing_params,\n mode=\"prediction\",\n metadata=training_set_metadata,\n backend=backend,\n callbacks=callbacks,\n )\n return dataset, training_set_metadata, None\n\n\nclass FatherPreprocessor(DataFormatPreprocessor):\n @staticmethod\n def preprocess_for_training(\n config,\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n callbacks=None,\n ):\n return _preprocess_file_for_training(\n config,\n features,\n dataset,\n training_set,\n validation_set,\n test_set,\n read_fn=read_feather,\n training_set_metadata=training_set_metadata,\n skip_save_processed_input=skip_save_processed_input,\n preprocessing_params=preprocessing_params,\n backend=backend,\n random_seed=random_seed,\n callbacks=callbacks,\n )\n\n @staticmethod\n def preprocess_for_prediction(\n config, dataset, features, preprocessing_params, training_set_metadata, backend, callbacks\n ):\n dataset_df = read_feather(dataset, backend.df_engine.df_lib)\n training_set_metadata[SRC] = dataset\n dataset, training_set_metadata = build_dataset(\n config,\n dataset_df,\n features,\n preprocessing_params,\n mode=\"prediction\",\n metadata=training_set_metadata,\n backend=backend,\n callbacks=callbacks,\n )\n return dataset, training_set_metadata, None\n\n\nclass FWFPreprocessor(DataFormatPreprocessor):\n @staticmethod\n def preprocess_for_training(\n config,\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n callbacks=None,\n ):\n return _preprocess_file_for_training(\n config,\n features,\n dataset,\n training_set,\n validation_set,\n test_set,\n read_fn=read_fwf,\n training_set_metadata=training_set_metadata,\n skip_save_processed_input=skip_save_processed_input,\n preprocessing_params=preprocessing_params,\n backend=backend,\n random_seed=random_seed,\n callbacks=callbacks,\n )\n\n @staticmethod\n def preprocess_for_prediction(\n config, dataset, features, preprocessing_params, training_set_metadata, backend, callbacks\n ):\n dataset_df = read_fwf(dataset, backend.df_engine.df_lib)\n training_set_metadata[SRC] = dataset\n dataset, training_set_metadata = build_dataset(\n config,\n dataset_df,\n features,\n preprocessing_params,\n mode=\"prediction\",\n metadata=training_set_metadata,\n backend=backend,\n callbacks=callbacks,\n )\n return dataset, training_set_metadata, None\n\n\nclass HTMLPreprocessor(DataFormatPreprocessor):\n @staticmethod\n def preprocess_for_training(\n config,\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n callbacks=None,\n ):\n return _preprocess_file_for_training(\n config,\n features,\n dataset,\n training_set,\n validation_set,\n test_set,\n read_fn=read_html,\n training_set_metadata=training_set_metadata,\n skip_save_processed_input=skip_save_processed_input,\n preprocessing_params=preprocessing_params,\n backend=backend,\n random_seed=random_seed,\n callbacks=callbacks,\n )\n\n @staticmethod\n def preprocess_for_prediction(\n config, dataset, features, preprocessing_params, training_set_metadata, backend, callbacks\n ):\n dataset_df = read_html(dataset, backend.df_engine.df_lib)\n training_set_metadata[SRC] = dataset\n dataset, training_set_metadata = build_dataset(\n config,\n dataset_df,\n features,\n preprocessing_params,\n mode=\"prediction\",\n metadata=training_set_metadata,\n backend=backend,\n callbacks=callbacks,\n )\n return dataset, training_set_metadata, None\n\n\nclass ORCPreprocessor(DataFormatPreprocessor):\n @staticmethod\n def preprocess_for_training(\n config,\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n callbacks=None,\n ):\n return _preprocess_file_for_training(\n config,\n features,\n dataset,\n training_set,\n validation_set,\n test_set,\n read_fn=read_orc,\n training_set_metadata=training_set_metadata,\n skip_save_processed_input=skip_save_processed_input,\n preprocessing_params=preprocessing_params,\n backend=backend,\n random_seed=random_seed,\n callbacks=callbacks,\n )\n\n @staticmethod\n def preprocess_for_prediction(\n config, dataset, features, preprocessing_params, training_set_metadata, backend, callbacks\n ):\n dataset_df = read_orc(dataset, backend.df_engine.df_lib)\n training_set_metadata[SRC] = dataset\n dataset, training_set_metadata = build_dataset(\n config,\n dataset_df,\n features,\n preprocessing_params,\n mode=\"prediction\",\n metadata=training_set_metadata,\n backend=backend,\n callbacks=callbacks,\n )\n return dataset, training_set_metadata, None\n\n\nclass SASPreprocessor(DataFormatPreprocessor):\n @staticmethod\n def preprocess_for_training(\n config,\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n callbacks=None,\n ):\n return _preprocess_file_for_training(\n config,\n features,\n dataset,\n training_set,\n validation_set,\n test_set,\n read_fn=read_sas,\n training_set_metadata=training_set_metadata,\n skip_save_processed_input=skip_save_processed_input,\n preprocessing_params=preprocessing_params,\n backend=backend,\n random_seed=random_seed,\n callbacks=callbacks,\n )\n\n @staticmethod\n def preprocess_for_prediction(\n config, dataset, features, preprocessing_params, training_set_metadata, backend, callbacks\n ):\n dataset_df = read_sas(dataset, backend.df_engine.df_lib)\n training_set_metadata[SRC] = dataset\n dataset, training_set_metadata = build_dataset(\n config,\n dataset_df,\n features,\n preprocessing_params,\n mode=\"prediction\",\n metadata=training_set_metadata,\n backend=backend,\n callbacks=callbacks,\n )\n return dataset, training_set_metadata, None\n\n\nclass SPSSPreprocessor(DataFormatPreprocessor):\n @staticmethod\n def preprocess_for_training(\n config,\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n callbacks=None,\n ):\n return _preprocess_file_for_training(\n config,\n features,\n dataset,\n training_set,\n validation_set,\n test_set,\n read_fn=read_spss,\n training_set_metadata=training_set_metadata,\n skip_save_processed_input=skip_save_processed_input,\n preprocessing_params=preprocessing_params,\n backend=backend,\n random_seed=random_seed,\n callbacks=callbacks,\n )\n\n @staticmethod\n def preprocess_for_prediction(\n config, dataset, features, preprocessing_params, training_set_metadata, backend, callbacks\n ):\n dataset_df = read_spss(dataset, backend.df_engine.df_lib)\n training_set_metadata[SRC] = dataset\n dataset, training_set_metadata = build_dataset(\n config,\n dataset_df,\n features,\n preprocessing_params,\n mode=\"prediction\",\n metadata=training_set_metadata,\n backend=backend,\n callbacks=callbacks,\n )\n return dataset, training_set_metadata, None\n\n\nclass StataPreprocessor(DataFormatPreprocessor):\n @staticmethod\n def preprocess_for_training(\n config,\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n callbacks=None,\n ):\n return _preprocess_file_for_training(\n config,\n features,\n dataset,\n training_set,\n validation_set,\n test_set,\n read_fn=read_stata,\n training_set_metadata=training_set_metadata,\n skip_save_processed_input=skip_save_processed_input,\n preprocessing_params=preprocessing_params,\n backend=backend,\n random_seed=random_seed,\n callbacks=callbacks,\n )\n\n @staticmethod\n def preprocess_for_prediction(\n config, dataset, features, preprocessing_params, training_set_metadata, backend, callbacks\n ):\n dataset_df = read_stata(dataset, backend.df_engine.df_lib)\n training_set_metadata[SRC] = dataset\n dataset, training_set_metadata = build_dataset(\n config,\n dataset_df,\n features,\n preprocessing_params,\n mode=\"prediction\",\n metadata=training_set_metadata,\n backend=backend,\n callbacks=callbacks,\n )\n return dataset, training_set_metadata, None\n\n\nclass HDF5Preprocessor(DataFormatPreprocessor):\n @staticmethod\n def preprocess_for_training(\n config,\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n callbacks=None,\n ):\n return HDF5Preprocessor.prepare_processed_data(\n features,\n dataset,\n training_set,\n validation_set,\n test_set,\n training_set_metadata,\n skip_save_processed_input,\n preprocessing_params,\n backend,\n random_seed,\n )\n\n @staticmethod\n def preprocess_for_prediction(\n config, dataset, features, preprocessing_params, training_set_metadata, backend, callbacks\n ):\n hdf5_fp = dataset\n dataset = load_hdf5(dataset, preprocessing_params, backend, split_data=False, shuffle_training=False)\n return dataset, training_set_metadata, hdf5_fp\n\n @staticmethod\n def prepare_processed_data(\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n ):\n if dataset is None and training_set is None:\n raise ValueError(\"One of `dataset` or `training_set` must be not None\")\n not_none_set = dataset if dataset is not None else training_set\n\n if not training_set_metadata:\n raise ValueError(\"When providing HDF5 data, \" \"training_set_metadata must not be None.\")\n\n logger.info(\"Using full hdf5 and json\")\n\n if DATA_TRAIN_HDF5_FP not in training_set_metadata:\n logger.warning(\n \"data_train_hdf5_fp not present in training_set_metadata. \"\n \"Adding it with the current HDF5 file path {}\".format(not_none_set)\n )\n training_set_metadata[DATA_TRAIN_HDF5_FP] = not_none_set\n\n elif training_set_metadata[DATA_TRAIN_HDF5_FP] != not_none_set:\n logger.warning(\n \"data_train_hdf5_fp in training_set_metadata is {}, \"\n \"different from the current HDF5 file path {}. \"\n \"Replacing it\".format(training_set_metadata[DATA_TRAIN_HDF5_FP], not_none_set)\n )\n training_set_metadata[DATA_TRAIN_HDF5_FP] = not_none_set\n\n if dataset is not None:\n training_set, test_set, validation_set = load_hdf5(\n dataset, preprocessing_params, backend, shuffle_training=True\n )\n\n elif training_set is not None:\n kwargs = dict(preprocessing_params=preprocessing_params, backend=backend, split_data=False)\n training_set = load_hdf5(training_set, shuffle_training=True, **kwargs)\n\n if validation_set is not None:\n validation_set = load_hdf5(validation_set, shuffle_training=False, **kwargs)\n\n if test_set is not None:\n test_set = load_hdf5(test_set, shuffle_training=False, **kwargs)\n\n return training_set, test_set, validation_set, training_set_metadata\n\n\ndata_format_preprocessor_registry = {\n **{fmt: DictPreprocessor for fmt in DICT_FORMATS},\n **{fmt: DataFramePreprocessor for fmt in DATAFRAME_FORMATS},\n **{fmt: CSVPreprocessor for fmt in CSV_FORMATS},\n **{fmt: TSVPreprocessor for fmt in TSV_FORMATS},\n **{fmt: JSONPreprocessor for fmt in JSON_FORMATS},\n **{fmt: JSONLPreprocessor for fmt in JSONL_FORMATS},\n **{fmt: ExcelPreprocessor for fmt in EXCEL_FORMATS},\n **{fmt: ParquetPreprocessor for fmt in PARQUET_FORMATS},\n **{fmt: PicklePreprocessor for fmt in PICKLE_FORMATS},\n **{fmt: FWFPreprocessor for fmt in FWF_FORMATS},\n **{fmt: FatherPreprocessor for fmt in FEATHER_FORMATS},\n **{fmt: HTMLPreprocessor for fmt in HTML_FORMATS},\n **{fmt: ORCPreprocessor for fmt in ORC_FORMATS},\n **{fmt: SASPreprocessor for fmt in SAS_FORMATS},\n **{fmt: SPSSPreprocessor for fmt in SPSS_FORMATS},\n **{fmt: StataPreprocessor for fmt in STATA_FORMATS},\n **{fmt: HDF5Preprocessor for fmt in HDF5_FORMATS},\n}\n\n\ndef build_dataset(\n config,\n dataset_df,\n features,\n global_preprocessing_parameters,\n mode,\n metadata=None,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n skip_save_processed_input=False,\n callbacks=None,\n):\n \"\"\"Builds a dataset from a dataframe and a list of features.\n\n Args:\n config: A dictionary containing the Ludwig model configuration\n dataset_df: Pandas or Dask dataframe\n features: List of features\n global_preprocessing_parameters: Global preprocessing parameters\n mode: One of ['training', 'prediction']\n metadata: Training set metadata if available\n backend: Backend\n random_seed: Random seed\n skip_save_processed_input: Whether to skip saving the processed input\n callbacks: List of callbacks\n\n Returns:\n A tuple of (dataset, metadata)\n \"\"\"\n\n df_engine = backend.df_engine\n\n if df_engine.partitioned:\n if any(f[\"type\"] in REPARTITIONING_FEATURE_TYPES for f in features) and dataset_df.npartitions > 1:\n # A globally unique index only matters if you know that there will be a repartition downstream for some\n # particular feature, i.e. for Image and Audio features on a Ray backend.\n # - There is a join operation in `df_like`, and the only way to do the operation is if the partitions across\n # all feature columns are aligned.\n # - In order to align the partitions, we require a way of matching samples to one another across all\n # partitions. Therefore, we must reset_index to create a globally unique index.\n # - If the number of partitions is 1, it is *highly likely* the index is globally unique. Auto-assigned\n # Dask indices in this case are unique, and we pd.concat train, val, and test sets with ignore_index=True\n # If there will NOT be a repartition downstream, then we can skip this step.\n # - In this case, the partitions should remain aligned throughout.\n # - Further, while the indices might not be globally unique, they should be unique within each partition.\n # - These two properties make it possible to do the join op within each partition without a global index.\n logger.warning(\n f\"Dataset has {dataset_df.npartitions} partitions and feature types that cause repartitioning. \"\n f\"Resetting index to ensure globally unique indices.\"\n )\n dataset_df = df_engine.reset_index(dataset_df)\n\n dataset_df = df_engine.parallelize(dataset_df)\n\n # Ensure that column names with non-word characters won't cause problems for downstream operations.\n # NOTE: Must be kept consistent with config sanitization in schema/model_types/base.py.\n dataset_df = sanitize_column_names(dataset_df)\n\n if mode == \"training\":\n sample_ratio = global_preprocessing_parameters[\"sample_ratio\"]\n sample_size = global_preprocessing_parameters[\"sample_size\"]\n dataset_df = _get_sampled_dataset_df(dataset_df, df_engine, sample_ratio, sample_size, random_seed)\n\n # If persisting DataFrames in memory is enabled, we want to do this after\n # each batch of parallel ops in order to avoid redundant computation\n dataset_df = df_engine.persist(dataset_df)\n\n if mode == \"training\":\n default_preprocessing_parameters = default_training_preprocessing_parameters\n elif mode == \"prediction\":\n default_preprocessing_parameters = default_prediction_preprocessing_parameters\n else:\n raise ValueError(f\"Invalid mode {mode}\")\n global_preprocessing_parameters = merge_dict(default_preprocessing_parameters, global_preprocessing_parameters)\n\n split_col = None\n if global_preprocessing_parameters[\"split\"][\"type\"] == \"fixed\":\n if global_preprocessing_parameters[\"split\"][\"column\"] in dataset_df.columns:\n split_col = dataset_df[global_preprocessing_parameters[\"split\"][\"column\"]]\n else:\n logger.warning(\n f\"Specified split column {global_preprocessing_parameters['split']['column']} for fixed \"\n f\"split strategy was not found in dataset.\" # noqa: E713\n )\n\n # update input features with prompt configs during preprocessing (as opposed to during the model forward pass)\n # so that we can compute metadata and build the dataset correctly.\n logger.debug(\"handle text features with prompt parameters\")\n synthesized_dataset_cols = handle_features_with_prompt_config(\n config, dataset_df, features, split_col=split_col, backend=backend\n )\n\n # Get all the unique preprocessing features to compute\n feature_configs = []\n feature_hashes = set()\n for feature in features:\n if feature[PROC_COLUMN] not in feature_hashes:\n feature_configs.append(feature)\n feature_hashes.add(feature[PROC_COLUMN])\n\n dataset_cols = {}\n for feature_config in feature_configs:\n col_name = feature_config[COLUMN]\n dataset_cols[col_name] = (\n synthesized_dataset_cols[col_name] if col_name in synthesized_dataset_cols else dataset_df[col_name]\n )\n\n logger.debug(\"build preprocessing parameters\")\n feature_name_to_preprocessing_parameters = build_preprocessing_parameters(\n dataset_cols, feature_configs, global_preprocessing_parameters, backend, metadata=metadata\n )\n\n # Happens after preprocessing parameters are built, so we can use precomputed fill values.\n logger.debug(\"handle missing values\")\n\n # In some cases, there can be a (temporary) mismatch between the dtype of the column and the type expected by the\n # preprocessing config (e.g., a categorical feature represented as an int-like column). In particular, Dask\n # may raise an error even when there are no missing values in the column itself.\n #\n # Since we immediately cast all columns in accordance with their expected feature types after filling missing\n # values, we work around the above issue by temporarily treating all columns as object dtype.\n for col_key in dataset_cols:\n dataset_cols[col_key] = dataset_cols[col_key].astype(object)\n\n for feature_config in feature_configs:\n preprocessing_parameters = feature_name_to_preprocessing_parameters[feature_config[NAME]]\n handle_missing_values(dataset_cols, feature_config, preprocessing_parameters, backend)\n\n # Happens after missing values are handled to avoid NaN casting issues.\n logger.debug(\"cast columns\")\n cast_columns(dataset_cols, feature_configs, backend)\n\n for callback in callbacks or []:\n callback.on_build_metadata_start(dataset_df, mode)\n\n logger.debug(\"build metadata\")\n metadata: TrainingSetMetadataDict = build_metadata(\n config, metadata, feature_name_to_preprocessing_parameters, dataset_cols, feature_configs, backend\n )\n\n check_global_max_sequence_length_fits_prompt_template(metadata, global_preprocessing_parameters)\n\n for callback in callbacks or []:\n callback.on_build_metadata_end(dataset_df, mode)\n\n for callback in callbacks or []:\n callback.on_build_data_start(dataset_df, mode)\n\n logger.debug(\"build data\")\n proc_cols = build_data(dataset_cols, feature_configs, metadata, backend, skip_save_processed_input)\n\n for callback in callbacks or []:\n callback.on_build_data_end(dataset_df, mode)\n\n # Get any additional columns needed for splitting downstream, otherwise they will not be\n # included in the preprocessed output.\n split_params = global_preprocessing_parameters.get(SPLIT, {})\n if \"type\" not in split_params and SPLIT in dataset_df:\n warnings.warn(\n 'Detected \"split\" column in the data, but using default split type '\n '\"random\". Did you mean to set split type to \"fixed\"?'\n )\n\n splitter = get_splitter(**split_params)\n for column in splitter.required_columns:\n if column not in dataset_df:\n warnings.warn(\n f\"column: '{column}' is required by the dataset splitter with params: {split_params}, but '{column}' \"\n f\"is not present in the `dataset_df` with columns: {dataset_df.columns}. This is acceptable during \"\n \"serving setting where dataset splitting is irrelevant. You may see this warning if, for example, the \"\n \"model was trained with a configuration that used a stratified split on the target column, but for \"\n \"live predictions, a value for the target column is not to be provided.\"\n )\n continue\n proc_cols[column] = dataset_df[column]\n\n # TODO pyarrow: this is needed for caching to work with pyarrow. if removed, the following error is raised:\n # \"pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values\". The data is reshaped when loaded\n # by the batcher in the RayDataset class (see _prepare_batch).\n if not skip_save_processed_input and backend.cache.data_format == \"parquet\":\n for feature in features:\n name = feature[NAME]\n proc_column = feature[PROC_COLUMN]\n reshape = metadata[name].get(\"reshape\")\n if reshape is not None:\n proc_cols[proc_column] = backend.df_engine.map_objects(proc_cols[proc_column], lambda x: x.reshape(-1))\n\n # Implements an outer join of proc_cols\n dataset = backend.df_engine.df_like(dataset_df, proc_cols)\n\n # At this point, there should be no missing values left in the dataframe, unless\n # the DROP_ROW preprocessing option was selected, in which case we need to drop those\n # rows.\n len_dataset_before_drop_rows = len(dataset)\n dataset = dataset.dropna()\n len_dataset_after_drop_rows = len(dataset)\n\n if len_dataset_before_drop_rows != len_dataset_after_drop_rows:\n logger.warning(\n f\"Dropped a total of {len_dataset_before_drop_rows - len_dataset_after_drop_rows} rows out of \"\n f\"{len_dataset_before_drop_rows} due to missing values\"\n )\n\n # NaNs introduced by outer join change dtype of dataset cols (upcast to float64), so we need to cast them back.\n col_name_to_dtype = {}\n for col_name, col in proc_cols.items():\n # if col is a list of list-like objects, we assume the internal dtype of each col[i] remains unchanged.\n if isinstance(col, list) and isinstance(col[0], (list, np.ndarray, torch.Tensor)):\n continue\n dtype = col.dtype\n # Skip non-numpy extension dtypes (e.g. TensorDtype from Ray, ArrowDtype from PyArrow)\n # as they cannot be used with DataFrame.astype() reliably.\n if not isinstance(dtype, np.dtype):\n continue\n col_name_to_dtype[col_name] = dtype\n dataset = dataset.astype(col_name_to_dtype)\n\n # Persist the completed dataset with no NaNs\n dataset = backend.df_engine.persist(dataset)\n\n # Remove partitions that are empty after removing NaNs\n dataset = backend.df_engine.remove_empty_partitions(dataset)\n\n # Embed features with fixed encoders\n dataset = embed_fixed_features(dataset, feature_configs, metadata, backend)\n\n return dataset, metadata\n\n\ndef embed_fixed_features(\n dataset: DataFrame, feature_configs: list[FeatureConfigDict], metadata: TrainingSetMetadataDict, backend: Backend\n) -> DataFrame:\n \"\"\"Transforms every input feature with cacheable encoder embeddings into its encoded form and updates\n metadata.\"\"\"\n # Encode features in bulk at the end\n features_to_encode = get_features_with_cacheable_fixed_embeddings(feature_configs, metadata)\n if not features_to_encode:\n return dataset\n\n logger.info(f\"Cache encoder embeddings for features: {[f[NAME] for f in features_to_encode]}\")\n for feature in features_to_encode:\n # Temporarily set to False to ensure proper encoding\n metadata[feature[NAME]][PREPROCESSING][\"cache_encoder_embeddings\"] = False\n\n batch_size = backend.tune_batch_size(create_embed_batch_size_evaluator(features_to_encode, metadata), len(dataset))\n transform_fn = create_embed_transform_fn(features_to_encode, metadata)\n results = backend.batch_transform(dataset, batch_size, transform_fn, name=\"Caching encoder embeddings\")\n\n for feature in features_to_encode:\n # Set metadata so we know to skip encoding the feature\n metadata[feature[NAME]][PREPROCESSING][\"cache_encoder_embeddings\"] = True\n\n return results\n\n\ndef _get_sampled_dataset_df(dataset_df, df_engine, sample_ratio, sample_size, random_seed):\n df_len = len(dataset_df)\n if sample_ratio < 1.0:\n if not df_engine.partitioned and df_len * sample_ratio < 1:\n raise ValueError(\n f\"sample_ratio {sample_ratio} is too small for dataset of length {df_len}. \"\n f\"Please increase sample_ratio or use a larger dataset.\"\n )\n\n logger.debug(f\"sample {sample_ratio} of data\")\n dataset_df = dataset_df.sample(frac=sample_ratio, random_state=random_seed)\n\n if sample_size:\n if sample_size < df_len:\n # Cannot use 'n' parameter when using dask DataFrames -- only 'frac' is supported\n sample_ratio = sample_size / df_len\n dataset_df = dataset_df.sample(frac=sample_ratio, random_state=random_seed)\n else:\n logger.warning(\"sample_size is larger than dataset size, ignoring sample_size\")\n\n return dataset_df\n\n\ndef get_features_with_cacheable_fixed_embeddings(\n feature_configs: list[FeatureConfigDict], metadata: TrainingSetMetadataDict\n) -> list[FeatureConfigDict]:\n \"\"\"Returns list of features with `cache_encoder_embeddings=True` set in the preprocessing config.\"\"\"\n features_to_encode = []\n for feature_config in feature_configs:\n # deal with encoders that have fixed preprocessing\n if ENCODER in feature_config:\n encoder_params = feature_config[ENCODER]\n if TYPE in encoder_params:\n preprocessing = metadata[feature_config[NAME]][PREPROCESSING]\n if preprocessing.get(\"cache_encoder_embeddings\"):\n # TODO(travis): passing in MODEL_ECD is a hack here that can be removed once we move to using\n # the config object everywhere in preprocessing. Then we won't need to do the lookup on the\n # encoder schema at all.\n encoder_class = get_encoder_cls(MODEL_ECD, feature_config[TYPE], encoder_params[TYPE])\n encoder = encoder_class.from_dict(encoder_params)\n if not encoder.can_cache_embeddings():\n raise ValueError(\n f\"Set `cache_encoder_embeddings=True` for feature {feature_config[NAME]} with \"\n f\"encoder {encoder_params[TYPE]}, but encoder embeddings are not static.\"\n )\n\n # Convert to Ray Datasets, map batches to encode, then convert back to Dask\n features_to_encode.append(feature_config)\n\n return features_to_encode\n\n\ndef cast_columns(dataset_cols, features, backend) -> None:\n \"\"\"Casts columns based on their feature type.\"\"\"\n for feature in features:\n # todo figure out if additional parameters are needed\n # for the cast_column function\n try:\n dataset_cols[feature[COLUMN]] = get_from_registry(feature[TYPE], get_base_type_registry()).cast_column(\n dataset_cols[feature[COLUMN]], backend\n )\n except KeyError as e:\n raise KeyError(\n f\"Feature name {e} specified in the config was not found in dataset with columns: \" # noqa: E713\n + f\"{list(dataset_cols.keys())}\"\n )\n\n\ndef merge_preprocessing(\n feature_config: FeatureConfigDict, global_preprocessing_parameters: PreprocessingConfigDict\n) -> FeatureConfigDict:\n if PREPROCESSING not in feature_config:\n return global_preprocessing_parameters[feature_config[TYPE]]\n\n return merge_dict(global_preprocessing_parameters[feature_config[TYPE]], feature_config[PREPROCESSING])\n\n\ndef build_preprocessing_parameters(\n dataset_cols: dict[str, Series],\n feature_configs: list[FeatureConfigDict],\n global_preprocessing_parameters: PreprocessingConfigDict,\n backend: Backend,\n metadata: TrainingSetMetadataDict | None = None,\n) -> PreprocessingConfigDict:\n if metadata is None:\n metadata = {}\n\n feature_name_to_preprocessing_parameters = {}\n for feature_config in feature_configs:\n feature_name = feature_config[NAME]\n\n # if metadata already exists, we can use it to get preprocessing parameters\n if feature_name in metadata:\n feature_name_to_preprocessing_parameters[feature_name] = metadata[feature_name][PREPROCESSING]\n continue\n\n preprocessing_parameters = feature_config[PREPROCESSING]\n missing_value_strategy = preprocessing_parameters[\"missing_value_strategy\"]\n fill_value = precompute_fill_value(\n dataset_cols, feature_config, missing_value_strategy, preprocessing_parameters, backend\n )\n if fill_value is not None:\n preprocessing_parameters.update({\"computed_fill_value\": fill_value})\n\n # Handle outlier replacement\n outlier_strategy = preprocessing_parameters.get(\"outlier_strategy\")\n if outlier_strategy is not None:\n if outlier_strategy != missing_value_strategy:\n outlier_fill_value = precompute_fill_value(\n dataset_cols, feature_config, outlier_strategy, preprocessing_parameters, backend\n )\n else:\n # Use fill value from missing_value_strategy to avoid redundant computation\n outlier_fill_value = fill_value\n\n if outlier_fill_value is not None:\n preprocessing_parameters.update({\"computed_outlier_fill_value\": outlier_fill_value})\n\n feature_name_to_preprocessing_parameters[feature_name] = preprocessing_parameters\n\n return feature_name_to_preprocessing_parameters\n\n\ndef is_input_feature(feature_config: FeatureConfigDict) -> bool:\n \"\"\"Utility function to check for the presence of encoder in the feature config to determine if the feature is\n an input feature or output feature.\"\"\"\n return ENCODER in feature_config\n\n\ndef build_metadata(\n config: ModelConfigDict,\n metadata: TrainingSetMetadataDict,\n feature_name_to_preprocessing_parameters: dict[str, PreprocessingConfigDict],\n dataset_cols: dict[str, Series],\n feature_configs: list[FeatureConfigDict],\n backend: Backend,\n) -> TrainingSetMetadataDict:\n for feature_config in feature_configs:\n feature_name = feature_config[NAME]\n if feature_name in metadata:\n continue\n\n preprocessing_parameters = feature_name_to_preprocessing_parameters[feature_name]\n\n column = dataset_cols[feature_config[COLUMN]]\n metadata[feature_name] = get_from_registry(feature_config[TYPE], get_base_type_registry()).get_feature_meta(\n config, column, preprocessing_parameters, backend, is_input_feature(feature_config)\n )\n\n metadata[feature_name][PREPROCESSING] = preprocessing_parameters\n\n return metadata\n\n\ndef build_data(\n input_cols: DataFrame,\n feature_configs: list[dict],\n training_set_metadata: dict,\n backend: Backend,\n skip_save_processed_input: bool,\n) -> dict[str, DataFrame]:\n \"\"\"Preprocesses the input dataframe columns, handles missing values, and potentially adds metadata to\n training_set_metadata.\n\n Args:\n input_cols: Input dataframe to be processed.\n feature_configs: List of feature configs.\n training_set_metadata: Training set metadata. Additional fields may be added.\n backend: Backend for data processing.\n skip_save_processed_input: (bool) Whether to skip saving the processed input.\n\n Returns:\n Dictionary of (feature name) -> (processed data).\n \"\"\"\n proc_cols = {}\n for feature_config in feature_configs:\n # TODO(travis): instead of using raw dictionary, this should be loaded into a proper PreprocessingConfig\n # object, so we don't need to hackily check for the presence of added keys.\n preprocessing_parameters = training_set_metadata[feature_config[NAME]][PREPROCESSING]\n\n # Need to run this again here as cast_columns may have introduced new missing values\n handle_missing_values(input_cols, feature_config, preprocessing_parameters, backend)\n\n # For features that support it, we perform outlier removal here using metadata computed on the full dataset\n handle_outliers(\n input_cols, feature_config, preprocessing_parameters, training_set_metadata[feature_config[NAME]], backend\n )\n\n get_from_registry(feature_config[TYPE], get_base_type_registry()).add_feature_data(\n feature_config,\n input_cols,\n proc_cols,\n training_set_metadata,\n preprocessing_parameters,\n backend,\n skip_save_processed_input,\n )\n\n return proc_cols\n\n\ndef balance_data(\n dataset_df: DataFrame,\n output_features: list[dict],\n preprocessing_parameters: dict,\n backend: Backend,\n random_seed: int,\n):\n \"\"\"The purpose of this function is to balance the training dataset using either over-sampling or under-\n sampling.\n\n Args:\n dataset_df: Input dataframe to be over-sampled or under-sampled.\n output_features: List of feature configs.\n preprocessing_parameters: Dictionary of the global preprocessing parameters.\n backend: Backend for data processing.\n random_seed: Integer to seed the random sampling to ensure determinism.\n\n Returns: An over-sampled or under-sampled training dataset.\n \"\"\"\n target = output_features[0][PROC_COLUMN]\n\n if backend.df_engine.partitioned:\n majority_class = backend.df_engine.compute(dataset_df[target].value_counts()).idxmax()\n minority_class = backend.df_engine.compute(dataset_df[target].value_counts()).idxmin()\n else:\n majority_class = dataset_df[target].value_counts().idxmax()\n minority_class = dataset_df[target].value_counts().idxmin()\n majority_df = dataset_df[dataset_df[target] == majority_class]\n minority_df = dataset_df[dataset_df[target] == minority_class]\n\n if preprocessing_parameters[\"oversample_minority\"]:\n sample_fraction = (len(majority_df) * preprocessing_parameters[\"oversample_minority\"]) / len(minority_df)\n minority_df = minority_df.sample(frac=sample_fraction, replace=True, random_state=random_seed)\n elif preprocessing_parameters[\"undersample_majority\"]:\n sample_fraction = int(len(minority_df) / preprocessing_parameters[\"undersample_majority\"]) / len(majority_df)\n majority_df = majority_df.sample(frac=sample_fraction, replace=False, random_state=random_seed)\n\n balanced_df = backend.df_engine.concat([minority_df, majority_df])\n\n return balanced_df\n\n\ndef precompute_fill_value(\n dataset_cols, feature, missing_value_strategy: str, preprocessing_parameters: PreprocessingConfigDict, backend\n):\n \"\"\"Precomputes the fill value for a feature.\n\n NOTE: this is called before NaNs are removed from the dataset. Modifications here must handle NaNs gracefully.\n NOTE: this is called before columns are cast. Modifications here must handle dtype conversion gracefully.\n \"\"\"\n if missing_value_strategy == FILL_WITH_CONST:\n return preprocessing_parameters[\"fill_value\"]\n elif missing_value_strategy == FILL_WITH_MODE:\n # Requires separate handling if Dask since Dask has lazy evaluation\n # Otherwise, dask returns a Dask index structure instead of a value to use as a fill value\n return (\n dataset_cols[feature[COLUMN]].value_counts().index.compute()[0]\n if is_dask_series_or_df(dataset_cols[feature[COLUMN]], backend)\n else dataset_cols[feature[COLUMN]].value_counts().index[0]\n )\n elif missing_value_strategy == FILL_WITH_MEAN:\n if feature[TYPE] != NUMBER:\n raise ValueError(\n f\"Filling missing values with mean is supported \"\n f\"only for number types, not for type {feature[TYPE]}.\",\n )\n return backend.df_engine.compute(dataset_cols[feature[COLUMN]].astype(float).mean())\n elif missing_value_strategy in {FILL_WITH_FALSE, FILL_WITH_TRUE}:\n distinct_values = backend.df_engine.compute(\n dataset_cols[feature[COLUMN]].drop_duplicates().dropna()\n ).values.tolist()\n if len(distinct_values) > 2:\n raise ValueError(\n f\"Missing value strategy `{missing_value_strategy}` \"\n f\"for column {feature[COLUMN]} expects 2 distinct values, \"\n f\"found: {len(distinct_values)} (ex: {distinct_values[:10]})\"\n )\n\n fill_to_bool_value = {FILL_WITH_FALSE: False, FILL_WITH_TRUE: True}\n bool_needed = fill_to_bool_value[missing_value_strategy]\n\n # Determine the False label.\n # Distinct values are sorted in reverse to mirror the selection of the default fallback_true_label (in\n # binary_feature.get_feature_meta) for binary columns with unconventional boolean values, \"human\"/\"bot\".\n for v in sorted(distinct_values, reverse=True):\n fallback_true_label = (\n preprocessing_parameters[\"fallback_true_label\"]\n # By default, preprocessing_parameters.fallback_true_label is None.\n if preprocessing_parameters[\"fallback_true_label\"]\n else \"true\"\n )\n if strings_utils.str2bool(v, fallback_true_label) is bool_needed:\n return v\n raise ValueError(\n f\"Unable to determine {bool_needed} value for column {feature[COLUMN]} \"\n f\"with distinct values: {distinct_values}.\"\n )\n # Otherwise, we cannot precompute the fill value for this dataset\n return None\n\n\n@DeveloperAPI\ndef handle_missing_values(dataset_cols, feature, preprocessing_parameters: PreprocessingConfigDict, backend):\n missing_value_strategy = preprocessing_parameters[\"missing_value_strategy\"]\n computed_fill_value = preprocessing_parameters.get(\"computed_fill_value\")\n _handle_missing_values(dataset_cols, feature, missing_value_strategy, computed_fill_value, backend)\n\n\n@DeveloperAPI\ndef handle_outliers(dataset_cols, feature, preprocessing_parameters: PreprocessingConfigDict, metadata, backend):\n outlier_strategy = preprocessing_parameters.get(\"outlier_strategy\")\n if outlier_strategy is None:\n return\n\n outlier_threshold = preprocessing_parameters[\"outlier_threshold\"]\n computed_fill_value = preprocessing_parameters.get(\"computed_outlier_fill_value\")\n\n # Identify all outliers and set them to NA so they can be removed\n series = dataset_cols[feature[COLUMN]]\n dataset_cols[feature[COLUMN]] = series.mask(\n series.sub(metadata[\"mean\"]).div(metadata[\"std\"]).abs().gt(outlier_threshold)\n )\n\n _handle_missing_values(dataset_cols, feature, outlier_strategy, computed_fill_value, backend)\n\n\ndef _handle_missing_values(\n dataset_cols, feature, missing_value_strategy: str, computed_fill_value: float | None, backend\n):\n if (\n missing_value_strategy in {FILL_WITH_CONST, FILL_WITH_MODE, FILL_WITH_MEAN, FILL_WITH_FALSE, FILL_WITH_TRUE}\n and computed_fill_value is not None\n ):\n dataset_cols[feature[COLUMN]] = dataset_cols[feature[COLUMN]].fillna(\n computed_fill_value,\n )\n elif missing_value_strategy in {BFILL, FFILL}:\n if missing_value_strategy == BFILL:\n dataset_cols[feature[COLUMN]] = dataset_cols[feature[COLUMN]].bfill()\n else:\n dataset_cols[feature[COLUMN]] = dataset_cols[feature[COLUMN]].ffill()\n\n # If the first few rows or last few rows of a dataset is a NaN, it will still be a NaN after ffill or bfill are\n # applied. This causes downstream errors with Dask (https://github.com/ludwig-ai/ludwig/issues/2452)\n # To get around this issue, apply the primary missing value strategy (say bfill) first, and then follow it\n # up with the other missing value strategy (ffill) to ensure all NaNs are filled\n if backend.df_engine.compute(dataset_cols[feature[COLUMN]].isna().sum()) > 0:\n if missing_value_strategy == FFILL:\n dataset_cols[feature[COLUMN]] = dataset_cols[feature[COLUMN]].bfill()\n else:\n dataset_cols[feature[COLUMN]] = dataset_cols[feature[COLUMN]].ffill()\n elif missing_value_strategy == DROP_ROW:\n # Here we only drop from this series, but after preprocessing we'll do a second\n # round of dropping NA values from the entire output dataframe, which will\n # result in the removal of the rows.\n len_before_dropped_rows = len(dataset_cols[feature[COLUMN]])\n dataset_cols[feature[COLUMN]] = dataset_cols[feature[COLUMN]].dropna()\n len_after_dropped_rows = len(dataset_cols[feature[COLUMN]])\n\n if len_before_dropped_rows != len_after_dropped_rows:\n logger.warning(\n f\"DROP_ROW missing value strategy applied. Dropped {len_before_dropped_rows - len_after_dropped_rows} \"\n f\"samples out of {len_before_dropped_rows} from column {feature[COLUMN]}. The rows containing these \"\n f\"samples will ultimately be dropped from the dataset.\"\n )\n else:\n raise ValueError(f\"Invalid missing value strategy {missing_value_strategy}\")\n\n\ndef handle_features_with_prompt_config(\n config: ModelConfigDict,\n dataset_df: DataFrame,\n features: list[FeatureConfigDict],\n backend: Backend,\n split_col: Series | None = None,\n) -> dict[str, Series]:\n \"\"\"Updates (in-place) dataset columns with prompt configurations containing a non-None task parameter.\n\n Dataset columns that are updated here are enriched to have prompts as specified by the prompt configuration.\n\n Args:\n config: Model configuration.\n dataset_df (DataFrame): Input dataset.\n features (List[FeatureConfigDict]): List of feature configurations.\n df_engine (DataFrameEngine): Dataframe engine.\n split_col (Optional[Series], optional): Split column. Defaults to None.\n\n Returns:\n Dict[str, Series]: Modified dataset columns.\n \"\"\"\n dataset_cols = {}\n input_features, output_features = get_input_and_output_features(features)\n for input_feature_config in input_features:\n prompt_config = _get_prompt_config(config, input_feature_config)\n if prompt_config is None:\n continue\n\n input_col_name = input_feature_config[COLUMN]\n if prompt_config[\"retrieval\"][\"type\"] is not None:\n # Ensure that the output features are in the dataset columns saved as part of the index\n # so that they can be retrieved later at lookup time.\n output_feature_col_names = [output_feature_config[COLUMN] for output_feature_config in output_features]\n input_and_output_col_names = set([input_col_name] + output_feature_col_names)\n input_and_output_cols = {\n feature[NAME]: dataset_df[feature[COLUMN]]\n for feature in features\n if feature[NAME] in input_and_output_col_names\n }\n retrieval_model, index_name = index_column(\n prompt_config[\"retrieval\"],\n col_name=input_col_name,\n dataset_cols=input_and_output_cols,\n backend=backend,\n split_col=split_col,\n )\n k = prompt_config[\"retrieval\"][\"k\"]\n\n # NOTE: after indexing the input column, we update the index_name in the prompt config IN PLACE.\n # This ensures that the preprocessing parameters for this feature have an up-to-date index_name\n # when the training set metadata is saved.\n prompt_config[\"retrieval\"][\"index_name\"] = index_name\n else:\n retrieval_model = None\n k = -1\n\n dataset_cols[input_col_name] = format_input_with_prompt(\n input_col_name,\n dataset_df,\n backend,\n prompt_config[\"task\"],\n retrieval_model=retrieval_model,\n k=k,\n template=prompt_config[\"template\"],\n )\n\n return dataset_cols\n\n\ndef _get_prompt_config(config: ModelConfigDict, input_feature_config: dict) -> dict:\n if input_feature_config[TYPE] != TEXT:\n # Prompt config is only applied to text features\n return None\n\n preprocessing = input_feature_config[\"preprocessing\"]\n if _has_prompt_section(preprocessing):\n return preprocessing[\"prompt\"]\n\n if _has_prompt_section(config):\n return config[\"prompt\"]\n\n return None\n\n\ndef _has_prompt_section(config: dict) -> bool:\n return \"prompt\" in config and (config[\"prompt\"][\"template\"] is not None or config[\"prompt\"][\"task\"] is not None)\n\n\ndef load_hdf5(hdf5_file_path, preprocessing_params, backend, split_data=True, shuffle_training=False):\n # TODO dask: this needs to work with DataFrames\n logger.info(f\"Loading data from: {hdf5_file_path}\")\n\n def shuffle(df):\n return df.sample(frac=1).reset_index(drop=True)\n\n dataset = data_utils.load_hdf5(hdf5_file_path)\n if not split_data:\n if shuffle_training:\n dataset = shuffle(dataset)\n return dataset\n\n training_set, validation_set, test_set = split_dataset(dataset, preprocessing_params, backend)\n\n if shuffle_training:\n training_set = shuffle(training_set)\n\n return training_set, test_set, validation_set\n\n\ndef load_metadata(metadata_file_path: str) -> TrainingSetMetadataDict:\n logger.info(f\"Loading metadata from: {metadata_file_path}\")\n training_set_metadata = data_utils.load_json(metadata_file_path)\n # TODO(travis): decouple config from training_set_metadata so we don't need to\n # upgrade it over time.\n training_set_metadata = upgrade_metadata(training_set_metadata)\n return training_set_metadata\n\n\ndef drop_extra_cols(features, dfs):\n retain_cols = list({feature[PROC_COLUMN]: True for feature in features}.keys())\n return tuple(df[retain_cols] if df is not None else df for df in dfs)\n\n\ndef preprocess_for_training(\n config,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n data_format=None,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n callbacks=None,\n) -> tuple[Dataset, Dataset, Dataset, TrainingSetMetadataDict]:\n \"\"\"Returns training, val and test datasets with training set metadata.\"\"\"\n\n # sanity check to make sure some data source is provided\n if dataset is None and training_set is None:\n raise ValueError(\"No training data is provided!\")\n\n # preload ludwig and HF datasets\n dataset, training_set, validation_set, test_set = load_dataset_uris(\n dataset, training_set, validation_set, test_set, backend\n )\n\n # determine data format if not provided or auto\n if not data_format or data_format == \"auto\":\n data_format = figure_data_format(dataset, training_set, validation_set, test_set)\n\n # Wrap dataset into a form we can use to manage within the cache\n dataset = wrap(dataset)\n training_set = wrap(training_set)\n validation_set = wrap(validation_set)\n test_set = wrap(test_set)\n\n try:\n lock_path = backend.cache.get_cache_directory(dataset)\n except (TypeError, ValueError):\n lock_path = None\n with file_lock(lock_path, lock_file=\".lock_preprocessing\"):\n # if training_set_metadata is a string, assume it's a path to load the json\n training_set_metadata = training_set_metadata or {}\n if training_set_metadata and isinstance(training_set_metadata, str):\n training_set_metadata = load_metadata(training_set_metadata)\n\n # setup\n features = config[\"input_features\"] + config[\"output_features\"]\n\n # in case data_format is one of the cacheable formats,\n # check if there's a cached hdf5 file with the same name,\n # and in case move on with the hdf5 branch.\n cached = False\n cache = backend.cache.get_dataset_cache(config, dataset, training_set, test_set, validation_set)\n\n # Unwrap dataset into the form used for preprocessing\n dataset = dataset.unwrap() if dataset is not None else None\n training_set = training_set.unwrap() if training_set is not None else None\n validation_set = validation_set.unwrap() if validation_set is not None else None\n test_set = test_set.unwrap() if test_set is not None else None\n\n if data_format in CACHEABLE_FORMATS:\n with backend.storage.cache.use_credentials():\n # cache.get() returns valid indicating if the checksum for the current config\n # is equal to that from the cached training set metadata, as well as the paths to the\n # cached training set metadata, training set, validation_set, test set\n cache_results = cache.get()\n if cache_results is not None:\n valid, *cache_values = cache_results\n if valid:\n logger.info(_get_cache_hit_message(cache))\n training_set_metadata, training_set, test_set, validation_set = cache_values\n config[\"data_hdf5_fp\"] = training_set\n data_format = backend.cache.data_format\n cached = True\n dataset = None\n else:\n logger.info(\n \"Found cached dataset and meta.json with the same filename \"\n \"of the dataset, but checksums don't match, \"\n \"if saving of processed input is not skipped \"\n \"they will be overridden\"\n )\n cache.delete()\n else:\n logger.info(\n f\"No cached dataset found at {cache.get_cached_obj_path('training')}. \"\n \"Preprocessing the dataset.\"\n )\n\n training_set_metadata[CHECKSUM] = cache.checksum\n data_format_processor = get_from_registry(data_format, data_format_preprocessor_registry)\n\n if cached or data_format == \"hdf5\":\n with backend.storage.cache.use_credentials():\n # Always interpret hdf5 files as preprocessed, even if missing from the cache\n processed = data_format_processor.prepare_processed_data(\n features,\n dataset=dataset,\n training_set=training_set,\n validation_set=validation_set,\n test_set=test_set,\n training_set_metadata=training_set_metadata,\n skip_save_processed_input=skip_save_processed_input,\n preprocessing_params=preprocessing_params,\n backend=backend,\n random_seed=random_seed,\n )\n training_set, test_set, validation_set, training_set_metadata = processed\n else:\n processed = data_format_processor.preprocess_for_training(\n config,\n features,\n dataset=dataset,\n training_set=training_set,\n validation_set=validation_set,\n test_set=test_set,\n training_set_metadata=training_set_metadata,\n skip_save_processed_input=skip_save_processed_input,\n preprocessing_params=preprocessing_params,\n backend=backend,\n random_seed=random_seed,\n callbacks=callbacks,\n )\n training_set, test_set, validation_set, training_set_metadata = processed\n processed = (training_set, test_set, validation_set, training_set_metadata)\n\n # cache the dataset\n if backend.cache.can_cache(skip_save_processed_input):\n with backend.storage.cache.use_credentials():\n logger.debug(\"cache processed data\")\n processed = cache.put(*processed)\n # set cached=True to ensure credentials are used correctly below\n cached = True\n training_set, test_set, validation_set, training_set_metadata = processed\n\n with backend.storage.cache.use_credentials() if cached else contextlib.nullcontext():\n logger.debug(\"create training dataset\")\n training_dataset = backend.dataset_manager.create(training_set, config, training_set_metadata)\n training_set_size = len(training_dataset)\n if training_set_size == 0:\n raise ValueError(\"Training data is empty following preprocessing.\")\n elif training_set_size < MIN_DATASET_SPLIT_ROWS:\n raise ValueError(\n f\"Training dataset has only {training_set_size} rows following preprocessing, need\"\n f\" at least {MIN_DATASET_SPLIT_ROWS} to compute metrics.\"\n )\n\n validation_dataset = None\n if validation_set is not None:\n logger.debug(\"create validation dataset\")\n validation_dataset = backend.dataset_manager.create(validation_set, config, training_set_metadata)\n validation_set_size = len(validation_dataset)\n if validation_set_size == 0:\n logger.warning(\n \"Validation set empty. If this is unintentional, please check the preprocessing configuration.\"\n )\n validation_dataset = None\n elif validation_set_size < MIN_DATASET_SPLIT_ROWS:\n logger.warning(\n f\"Validation set too small to compute metrics. Need at least {MIN_DATASET_SPLIT_ROWS} rows, got\"\n f\" {validation_set_size} after preprocessing.\"\n )\n\n test_dataset = None\n if test_set is not None:\n logger.debug(\"create test dataset\")\n test_dataset = backend.dataset_manager.create(test_set, config, training_set_metadata)\n test_set_size = len(test_dataset)\n if test_set_size == 0:\n logger.warning(\n \"Test set empty. If this is unintentional, please check the preprocessing configuration.\"\n )\n test_dataset = None\n elif test_set_size < MIN_DATASET_SPLIT_ROWS:\n logger.warning(\n f\"Test set too small to compute metrics. Need at least {MIN_DATASET_SPLIT_ROWS} rows, got\"\n f\" {test_set_size} after preprocessing.\"\n )\n\n return (training_dataset, validation_dataset, test_dataset, training_set_metadata)\n\n\ndef _preprocess_file_for_training(\n config,\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n read_fn=read_csv,\n skip_save_processed_input=False,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n callbacks=None,\n):\n \"\"\"Method to pre-process csv data.\n\n :param features: list of all features (input + output)\n :param dataset: path to the data\n :param training_set: training data\n :param validation_set: validation data\n :param test_set: test data\n :param training_set_metadata: train set metadata\n :param skip_save_processed_input: if False, the pre-processed data is saved as .hdf5 files in the same location as\n the csv files with the same names.\n :param preprocessing_params: preprocessing parameters\n :param random_seed: random seed\n :return: training, test, validation datasets, training metadata\n \"\"\"\n if dataset:\n # Use data and ignore _train, _validation and _test.\n # Also ignore data and train set metadata needs preprocessing\n logger.info(\"Using full raw dataset, no hdf5 and json file \" \"with the same name have been found\")\n logger.info(\"Building dataset (it may take a while)\")\n\n dataset_df = read_fn(dataset, backend.df_engine.df_lib)\n training_set_metadata[SRC] = dataset\n\n data, training_set_metadata = build_dataset(\n config,\n dataset_df,\n features,\n preprocessing_params,\n mode=\"training\",\n metadata=training_set_metadata,\n backend=backend,\n random_seed=random_seed,\n skip_save_processed_input=skip_save_processed_input,\n callbacks=callbacks,\n )\n\n elif training_set:\n # use data_train (including _validation and _test if they are present)\n # and ignore data and train set metadata\n # needs preprocessing\n logger.info(\"Using training raw csv, no hdf5 and json \" \"file with the same name have been found\")\n logger.info(\"Building dataset (it may take a while)\")\n\n concatenated_df = concatenate_files(training_set, validation_set, test_set, read_fn, backend)\n training_set_metadata[SRC] = training_set\n\n # Data is pre-split.\n preprocessing_params = set_fixed_split(preprocessing_params)\n\n data, training_set_metadata = build_dataset(\n config,\n concatenated_df,\n features,\n preprocessing_params,\n mode=\"training\",\n metadata=training_set_metadata,\n backend=backend,\n random_seed=random_seed,\n callbacks=callbacks,\n )\n\n else:\n raise ValueError(\"either data or data_train have to be not None\")\n\n logger.debug(\"split train-val-test\")\n training_data, validation_data, test_data = drop_extra_cols(\n features, split_dataset(data, preprocessing_params, backend, random_seed)\n )\n\n if dataset and backend.is_coordinator() and not skip_save_processed_input:\n logger.debug(\"writing split file\")\n splits_df = concatenate_splits(training_data, validation_data, test_data, backend)\n split_fp = get_split_path(dataset or training_set)\n try:\n backend.df_engine.to_parquet(splits_df, split_fp, index=True)\n except Exception as e:\n logger.warning(\n f\"Encountered error: '{e}' while writing data to parquet during saving preprocessed data. \"\n \"Skipping saving processed data.\"\n )\n\n logger.info(\"Building dataset: DONE\")\n if preprocessing_params[\"oversample_minority\"] or preprocessing_params[\"undersample_majority\"]:\n training_data = balance_data(\n training_data, config[\"output_features\"], preprocessing_params, backend, random_seed\n )\n\n return training_data, test_data, validation_data, training_set_metadata\n\n\ndef _preprocess_df_for_training(\n config,\n features,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n preprocessing_params=default_training_preprocessing_parameters,\n backend=LOCAL_BACKEND,\n random_seed=default_random_seed,\n callbacks=None,\n):\n \"\"\"Method to pre-process dataframes.\n\n This doesn't have the option to save the processed data as hdf5 as we don't expect users to do this as the data can\n be processed in memory\n \"\"\"\n if dataset is not None:\n # needs preprocessing\n logger.info(\"Using full dataframe\")\n elif training_set is not None:\n # needs preprocessing\n logger.info(\"Using training dataframe\")\n dataset = concatenate_df(training_set, validation_set, test_set, backend)\n\n # Data is pre-split.\n preprocessing_params = set_fixed_split(preprocessing_params)\n\n logger.info(\"Building dataset (it may take a while)\")\n\n data, training_set_metadata = build_dataset(\n config,\n dataset,\n features,\n preprocessing_params,\n mode=\"training\",\n metadata=training_set_metadata,\n random_seed=random_seed,\n backend=backend,\n callbacks=callbacks,\n )\n\n logger.debug(\"split train-val-test\")\n training_set, validation_set, test_set = drop_extra_cols(\n features, split_dataset(data, preprocessing_params, backend, random_seed)\n )\n\n logger.info(\"Building dataset: DONE\")\n if preprocessing_params[\"oversample_minority\"] or preprocessing_params[\"undersample_majority\"]:\n training_set = balance_data(training_set, config[\"output_features\"], preprocessing_params, backend, random_seed)\n\n return training_set, test_set, validation_set, training_set_metadata\n\n\ndef preprocess_for_prediction(\n config,\n dataset,\n training_set_metadata=None,\n data_format=None,\n split=FULL,\n include_outputs=True,\n backend=LOCAL_BACKEND,\n callbacks=None,\n):\n \"\"\"Preprocesses the dataset to parse it into a format that is usable by the Ludwig core.\n\n Args:\n config: Config dictionary corresponding to Ludwig Model\n dataset: Dataset to be processed\n training_set_metadata: Train set metadata for the input features\n data_format: Format of the data\n split: The split of dataset to return\n include_outputs: Whether to include outputs\n backend: Type of backend to use for preprocessing\n callbacks: Any callbacks passed in\n\n Returns:\n Processed dataset along with updated training set metadata\n \"\"\"\n # Sanity Check to make sure some data source is provided\n if dataset is None:\n raise ValueError(\"No training data is provided!\")\n\n if isinstance(dataset, Dataset):\n return dataset, training_set_metadata\n\n # preload ludwig and HF datasets\n dataset, _, _, _ = load_dataset_uris(dataset, None, None, None, backend)\n\n # determine data format if not provided or auto\n if not data_format or data_format == \"auto\":\n data_format = figure_data_format(dataset)\n\n # manage the in_memory parameter\n if data_format not in HDF5_FORMATS:\n num_overrides = override_in_memory_flag(config[\"input_features\"], True)\n if num_overrides > 0:\n logger.warning(\"Using in_memory = False is not supported \" \"with {} data format.\".format(data_format))\n\n preprocessing_params = {}\n config_defaults = config.get(DEFAULTS, {})\n for feature_type in config_defaults:\n preprocessing_params[feature_type] = config_defaults[feature_type].get(PREPROCESSING, {})\n preprocessing_params[SPLIT] = config.get(PREPROCESSING, {}).get(SPLIT, {})\n\n preprocessing_params = merge_dict(default_prediction_preprocessing_parameters, preprocessing_params)\n\n # if training_set_metadata is a string, assume it's a path to load the json\n if training_set_metadata and isinstance(training_set_metadata, str):\n training_set_metadata = load_metadata(training_set_metadata)\n\n # setup\n output_features = []\n if include_outputs:\n output_features += config[\"output_features\"]\n features = config[\"input_features\"] + output_features\n\n # Check the cache for an already preprocessed dataset. This only\n # applies to scenarios where the user wishes to predict on a split\n # of the full dataset, where we preprocess the whole dataset together\n # during training. If the user wishes to predict on the full dataset,\n # it is assumed they are predicting on unseen data. This is done\n # because the cached data is stored in its split form, and would be\n # expensive to recombine, requiring further caching.\n cached = False\n\n dataset = wrap(dataset)\n cache = backend.cache.get_dataset_cache(config, dataset)\n dataset = dataset.unwrap()\n\n training_set = test_set = validation_set = None\n if data_format in CACHEABLE_FORMATS and split != FULL:\n with backend.storage.cache.use_credentials():\n cache_results = cache.get()\n if cache_results is not None:\n valid, *cache_values = cache_results\n if valid:\n logger.info(_get_cache_hit_message(cache))\n training_set_metadata, training_set, test_set, validation_set = cache_values\n config[\"data_hdf5_fp\"] = training_set\n data_format = backend.cache.data_format\n cached = True\n\n data_format_processor = get_from_registry(data_format, data_format_preprocessor_registry)\n if cached:\n with backend.storage.cache.use_credentials():\n processed = data_format_processor.prepare_processed_data(\n features,\n dataset=dataset,\n training_set=training_set,\n validation_set=validation_set,\n test_set=test_set,\n training_set_metadata=training_set_metadata,\n preprocessing_params=preprocessing_params,\n backend=backend,\n )\n training_set, test_set, validation_set, training_set_metadata = processed\n else:\n processed = data_format_processor.preprocess_for_prediction(\n config, dataset, features, preprocessing_params, training_set_metadata, backend, callbacks\n )\n dataset, training_set_metadata, new_hdf5_fp = processed\n training_set_metadata = training_set_metadata.copy()\n\n if new_hdf5_fp:\n training_set_metadata[DATA_TRAIN_HDF5_FP] = new_hdf5_fp\n\n if split != FULL:\n logger.debug(\"split train-val-test\")\n training_set, validation_set, test_set = drop_extra_cols(\n features, split_dataset(dataset, preprocessing_params, backend)\n )\n\n if split == TRAINING:\n dataset = training_set\n elif split == VALIDATION:\n dataset = validation_set\n elif split == TEST:\n dataset = test_set\n\n config = {\n **config,\n \"output_features\": output_features,\n }\n\n with backend.storage.cache.use_credentials() if cached else contextlib.nullcontext():\n dataset = backend.dataset_manager.create(\n dataset,\n config,\n training_set_metadata,\n )\n\n return dataset, training_set_metadata\n\n\ndef _get_cache_hit_message(cache: DatasetCache) -> str:\n return (\n \"Found cached dataset and meta.json with the same filename of the dataset.\\n\"\n \"Using cached values instead of preprocessing the dataset again.\\n\"\n f\"- Cached training set metadata path: {cache.get_cached_obj_path(META)}\\n\"\n f\"- Cached training set path: {cache.get_cached_obj_path(TRAINING)}\\n\"\n f\"- Cached validation set path: {cache.get_cached_obj_path(VALIDATION)}\\n\"\n f\"- Cached test set path: {cache.get_cached_obj_path(TEST)}\"\n )\n"
},
{
"path": "ludwig/data/prompt.py",
"content": "import json\nimport logging\nimport os\nimport string\nfrom typing import Any, TYPE_CHECKING\n\nimport pandas as pd\n\nif TYPE_CHECKING:\n from ludwig.backend.base import Backend\n\nfrom ludwig.models.retrieval import df_checksum, get_retrieval_model, RetrievalModel\nfrom ludwig.utils.fs_utils import get_default_cache_location, makedirs, path_exists\nfrom ludwig.utils.types import DataFrame, Series\n\nlogger = logging.getLogger(__name__)\n\nCONTEXT = \"__context__\"\nSAMPLE = \"__sample__\"\nTASK = \"__task__\"\n\nDEFAULT_ZERO_SHOT_PROMPT_TEMPLATE = \"\"\"SAMPLE INPUT: {__sample__}\n\nUSER: Complete the following task: {__task__}\n\nASSISTANT:\n\"\"\"\n\n\nDEFAULT_FEW_SHOT_PROMPT_TEMPLATE = \"\"\"Below is relevant context:\n\nCONTEXT: {__context__}\n\nCONTEXT is comprised of labeled samples whose embeddings were similar to that of the sample input. The labels in\nthese samples could aid you in your final prediction. Given this and no prior knowledge, follow the instructions\nbelow.\n\nSAMPLE INPUT: {__sample__}\n\nUSER: Complete the following task: {__task__}\n\nASSISTANT:\n\"\"\"\n\n\ndef index_column(\n retrieval_config: dict[str, Any],\n col_name: str,\n dataset_cols: dict[str, Series],\n backend: \"Backend\",\n split_col: Series | None = None,\n) -> tuple[RetrievalModel, str]:\n \"\"\"Indexes a column for sample retrieval via embedding index lookup.\n\n This function indexes a column and saves the index artifact to disk. If an index name is provided as part of the\n `retrieval_config`, then the index in the ludwig cache with the corresponding name will be loaded instead of being\n built from scratch.\n\n To prevent data leakage, a split column must be provided. This ensures that the retrieval model only ever fetches\n samples from the training set.\n\n To ensure that the index is usable even if the original DataFrame is not available, the columns used to build the\n index are stored as part of the index.\n\n All operations in this function are performed on pandas objects, which means that you may run out of memory if your\n dataset is large.\n\n Args:\n retrieval_config (Dict[str, Any]): The retrieval config from the config object.\n col_name (str): The name of the column to index.\n dataset_cols (Dict[str, Series]): A dictionary mapping column names to their corresponding Series. `col_name`\n must be a key in this dictionary. These columns are stored as part of the index to ensure that the index\n is usable even if the original DataFrame is not available.\n df_engine (DataFrameEngine): The engine used to compute the columns into pandas objects.\n split_col (Optional[Series]): A column that indicates whether a sample is part of the training set. A sample\n is in the training set if the value in this column is 0.\n Returns:\n Tuple[RetrievalModel, str]: A tuple containing the retrieval model and the name of the index.\n \"\"\"\n retrieval_model = get_retrieval_model(\n retrieval_config[\"type\"],\n model_name=retrieval_config[\"model_name\"],\n )\n\n index_name = retrieval_config[\"index_name\"]\n index_cache_directory = os.path.join(get_default_cache_location(), \"index\")\n if not path_exists(index_cache_directory):\n makedirs(index_cache_directory, exist_ok=True)\n\n if index_name is None:\n if split_col is None:\n raise ValueError(\"split column must be provided if using retrieval\")\n split_col = backend.df_engine.compute(split_col).astype(int)\n\n # TODO(geoffrey): add support for Dask DataFrames\n df = pd.DataFrame({name: backend.df_engine.compute(col) for name, col in dataset_cols.items()})\n df = df[split_col == 0] # Ensures that the index is only built on the training set\n\n # Even if index name is not provided, we still want to check if an index for this df already exists in cache\n # If it does, load it and return immediately\n index_hash = df_checksum(df)\n index_name = f\"embedding_index_{index_hash}\"\n if path_exists(os.path.join(index_cache_directory, index_name)):\n logger.info(\n f\"Index for this DataFrame with name '{index_name}' already exists. \"\n f\"Loading index from '{index_cache_directory}'\"\n )\n retrieval_model.load_index(index_name, cache_directory=index_cache_directory)\n return retrieval_model, index_name\n\n # Build index if index name is not provided and index for this df does not already exist in cache\n retrieval_model.create_dataset_index(df, backend, columns_to_index=[col_name])\n logger.info(f\"Saving index to cache directory '{index_cache_directory}' with name '{index_name}'\")\n retrieval_model.save_index(index_name, cache_directory=index_cache_directory)\n else:\n logger.info(f\"Loading index from cache directory '{index_cache_directory}' with name '{index_name}'\")\n retrieval_model.load_index(index_name, cache_directory=index_cache_directory)\n return retrieval_model, index_name\n\n\ndef format_input_with_prompt(\n input_col_name: str,\n dataset_df: DataFrame,\n backend: \"Backend\",\n task_str: str,\n retrieval_model: RetrievalModel | None = None,\n k: int = -1,\n template: str | None = None,\n) -> Series:\n \"\"\"Returns a new Series with the input column data formatted with the prompt.\n\n A prompt can either be zero-shot or few-shot. A zero-shot prompt is comprised of some (unlabeled) input and a task\n to be completed given the input. A few-shot prompt additionally includes some dynamically retrieved context, which\n is retrieved using the `retrieval_model.search` function.\n\n A template can be provided to customize the prompt. The template must be a string with the following fields:\n - __sample__ or at least one column from the input dataset: The input sample.\n - __context__: The context retrieved by the `search_fn` function. Only required if `search_fn` is provided.\n - __task__: The task to be completed given the input. Only required if `task` is set in the prompt config.\n\n Zero-shot example:\n\n Before formatting:\n\n input_col = [\"I am happy\"]\n task_str = \"sentiment analysis\"\n\n After formatting:\n\n input_col = [\"SAMPLE INPUT: I am happy\\n\\nUSER: Complete the following task: sentiment analysis\\n\\nASSISTANT:\"]\n\n Args:\n input_col_name (str): The name of the input column.\n dataset_df (DataFrame): The input dataset.\n backend (Backend): The backend used for map operations.\n task_str (str): The task to be completed given the input.\n retrieval_model (Optional[RetrievalModel]): The retrieval model used to retrieve context. If provided, the\n prompt will be few-shot. If not provided, the prompt will be zero-shot.\n k (int): The number of samples to retrieve. Only required if `retrieval_model` is provided.\n template (Optional[str]): The template to use for the prompt. If not provided, the default will be used.\n\n Returns:\n Series: A new Series with the input column data formatted with the prompt.\n \"\"\"\n # determine if this is a few-shot or zero-shot prompt\n # few-shot prompts require a search function that returns samples from some dataset\n is_few_shot = retrieval_model is not None\n\n # if no template is provided, use the default template\n if template is None:\n if is_few_shot:\n template = DEFAULT_FEW_SHOT_PROMPT_TEMPLATE\n else:\n template = DEFAULT_ZERO_SHOT_PROMPT_TEMPLATE\n\n # ensure that the prompt template has all required fields\n template_fields, field_to_dtype = _get_template_fields(template)\n try:\n _validate_prompt_template(template_fields, task_str, is_few_shot, dataset_df.columns, input_col_name)\n except ValueError as e:\n raise ValueError(f\"template invalid for {'few-shot' if is_few_shot else 'zero-shot'} prompt: {e}\")\n\n def generate_prompt(df: pd.DataFrame):\n if CONTEXT in template_fields:\n df[CONTEXT] = retrieval_model.search(df, backend, k=k, return_data=True)\n if SAMPLE in template_fields:\n # During preprocessing, we're inserting quotes that change the token IDs completely if we\n # don't remove the \" from the string. For parity with expected user output, we need to get rid of them.\n # TODO(Arnav): see if there's a way to only remove them if the entry does't have quotes. This currently\n # removes all \" from the string (even those not added by json.dumps), which is not ideal.\n df[SAMPLE] = df[input_col_name].map(lambda entry: json.dumps(entry, indent=2).strip('\"'))\n if TASK in template_fields:\n df[TASK] = task_str\n\n def generate_prompt_for_row(row):\n kwargs = {col: field_to_dtype[col](row[col]) for col in template_fields}\n return template.format(**kwargs)\n\n return df.apply(generate_prompt_for_row, axis=1)\n\n result = backend.df_engine.map_partitions(dataset_df, generate_prompt, meta=(input_col_name, \"object\"))\n result = backend.df_engine.persist(result) # persist to prevent re-computation\n return result\n\n\ndef _validate_prompt_template(\n template_fields: set[str], task: str | None, is_few_shot: bool, columns: list[str], input_col_name: str\n):\n \"\"\"Validates that the template contains the necessary fields for the prompt.\"\"\"\n if is_few_shot and CONTEXT not in template_fields:\n raise ValueError(f\"Prompt template must contain the '{CONTEXT}' field for few-shot learning\")\n\n if task is not None and TASK not in template_fields:\n raise ValueError(f\"Prompt template must contain the '{TASK}' field if a task is provided\")\n\n if SAMPLE in template_fields:\n if input_col_name not in columns:\n raise ValueError(\n f\"Prompt template contains the '{SAMPLE}' field, \"\n f\"but the input column '{input_col_name}' is not in the dataset\"\n )\n elif not any(col in template_fields for col in columns):\n raise ValueError(\n f\"Prompt template must contain either the '{SAMPLE}' field or one of the columns from the dataset\"\n )\n\n\ndef _get_template_fields(template: str) -> tuple[set[str], dict[str, type]]:\n \"\"\"Returns the fields in the template.\"\"\"\n parsed = [t for t in string.Formatter().parse(template) if t[1] is not None]\n field_set = {field for _, field, _, _ in parsed}\n dtype_map = {field: _get_dtype(format_spec) for _, field, format_spec, _ in parsed}\n return field_set, dtype_map\n\n\ndef _get_dtype(format_spec: str) -> type:\n # We need to prepare data in the row for different formatting options.\n # If you have a number like 0.1234 in the DF and you want to format it like {number:.2f} it will fail if the\n # number is represented as a string in the DF. So we need to cast it to a float before formatting.\n if not format_spec:\n return str\n if \"f\" in format_spec:\n return float\n raise ValueError(f\"Unsupported template format spec: {format_spec}\")\n"
},
{
"path": "ludwig/data/sampler.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2020 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nimport math\n\nimport numpy as np\n\nfrom ludwig.distributed import DistributedStrategy\nfrom ludwig.utils.defaults import default_random_seed\n\n\nclass DistributedSampler:\n \"\"\"Adapted from `torch.utils.data.distributed.DistributedSampler`.\"\"\"\n\n def __init__(\n self,\n dataset_size: int,\n shuffle: bool = True,\n random_seed: int = default_random_seed,\n distributed: DistributedStrategy = None,\n ):\n self.dataset_size = dataset_size\n self.num_replicas = distributed.size() if distributed else 1\n self.rank = distributed.rank() if distributed else 0\n self.epoch = 0\n self.num_samples = int(math.ceil(self.dataset_size * 1.0 / self.num_replicas))\n self.total_size = self.num_samples * self.num_replicas\n self.shuffle = shuffle\n self.random_seed = random_seed\n\n def __iter__(self):\n if self.shuffle:\n # deterministically shuffle based on epoch and seed\n indices = np.random.RandomState(seed=self.random_seed + self.epoch).permutation(self.dataset_size).tolist()\n else:\n indices = list(range(self.dataset_size))\n\n # add extra samples to make it evenly divisible\n indices += indices[: (self.total_size - len(indices))]\n assert len(indices) == self.total_size\n\n # subsample\n indices = indices[self.rank : self.total_size : self.num_replicas]\n assert len(indices) == self.num_samples\n\n return iter(indices)\n\n def __len__(self):\n return self.num_samples\n\n def set_epoch(self, epoch):\n \"\"\"Sets the epoch for this sampler.\n\n When `shuffle=True`, this ensures all replicas use a different random ordering for each epoch. Otherwise, the\n next iteration of this sampler will yield the same ordering.\n\n :param epoch: (int) epoch number\n \"\"\"\n self.epoch = epoch\n"
},
{
"path": "ludwig/data/split.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nimport logging\nfrom abc import ABC, abstractmethod\nfrom typing import TYPE_CHECKING\nfrom zlib import crc32\n\nimport numpy as np\nfrom sklearn.model_selection import train_test_split\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.backend.base import Backend\nfrom ludwig.constants import BINARY, CATEGORY, DATE, MIN_DATASET_SPLIT_ROWS, SPLIT\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.schema.split import (\n DateTimeSplitConfig,\n FixedSplitConfig,\n HashSplitConfig,\n RandomSplitConfig,\n StratifySplitConfig,\n)\nfrom ludwig.types import ModelConfigDict, PreprocessingConfigDict\nfrom ludwig.utils.data_utils import hash_dict, split_dataset_ttv\nfrom ludwig.utils.defaults import default_random_seed\nfrom ludwig.utils.registry import Registry\nfrom ludwig.utils.types import DataFrame\n\nif TYPE_CHECKING:\n from ludwig.schema.model_config import ModelConfig\n\nsplit_registry = Registry()\nlogger = logging.getLogger(__name__)\n\nTMP_SPLIT_COL = \"__SPLIT__\"\nDEFAULT_PROBABILITIES = (0.7, 0.1, 0.2)\n\n\nclass Splitter(ABC):\n @abstractmethod\n def split(\n self, df: DataFrame, backend: Backend, random_seed: int = default_random_seed\n ) -> tuple[DataFrame, DataFrame, DataFrame]:\n pass\n\n def validate(self, config: ModelConfigDict):\n pass\n\n def has_split(self, split_index: int) -> bool:\n return True\n\n @property\n def required_columns(self) -> list[str]:\n \"\"\"Returns the list of columns that are required for splitting.\"\"\"\n return []\n\n\ndef _make_divisions_ensure_minimum_rows(\n divisions: list[int],\n n_examples: int,\n min_val_rows: int = MIN_DATASET_SPLIT_ROWS,\n min_test_rows: int = MIN_DATASET_SPLIT_ROWS,\n) -> list[int]:\n \"\"\"Revises divisions to ensure no dataset split has too few examples.\"\"\"\n result = list(divisions)\n n = [dn - dm for dm, dn in zip((0,) + divisions, divisions + (n_examples,))] # Number of examples in each split.\n if 0 < n[2] < min_test_rows and n[0] > 0:\n # Test set is nonempty but too small, take examples from training set.\n shift = min(min_test_rows - n[2], n[0])\n result = [d - shift for d in result]\n if 0 < n[1] < min_val_rows and n[0] > 0:\n # Validation set is nonempty but too small, take examples from training set.\n result[0] -= min(min_val_rows - n[1], result[0])\n return result\n\n\ndef _split_divisions_with_min_rows(n_rows: int, probabilities: list[float]) -> list[int]:\n \"\"\"Generates splits for a dataset of n_rows into train, validation, and test sets according to split\n probabilities, also ensuring that at least min_val_rows or min_test_rows are present in each nonempty split.\n\n Returns division indices to split on.\n \"\"\"\n d1 = int(np.ceil(probabilities[0] * n_rows))\n if probabilities[-1] > 0:\n n2 = int(probabilities[1] * n_rows)\n d2 = d1 + n2\n else:\n # If the last probability is 0, then use the entire remaining dataset for validation.\n d2 = n_rows\n return _make_divisions_ensure_minimum_rows((d1, d2), n_rows)\n\n\n@split_registry.register(\"random\", default=True)\nclass RandomSplitter(Splitter):\n def __init__(self, probabilities: list[float] = DEFAULT_PROBABILITIES, **kwargs):\n self.probabilities = probabilities\n\n def split(\n self, df: DataFrame, backend: Backend, random_seed: float = default_random_seed\n ) -> tuple[DataFrame, DataFrame, DataFrame]:\n probabilities = self.probabilities\n if not backend.df_engine.partitioned:\n divisions = _split_divisions_with_min_rows(len(df), probabilities)\n shuffled_df = df.sample(frac=1, random_state=random_seed)\n return (\n shuffled_df.iloc[: divisions[0]], # Train\n shuffled_df.iloc[divisions[0] : divisions[1]], # Validation\n shuffled_df.iloc[divisions[1] :], # Test\n )\n\n # The above approach is very inefficient for partitioned backends, which can split by partition.\n # This does not give exact guarantees on split size but is much more efficient for large datasets.\n return df.random_split(self.probabilities, random_state=random_seed)\n\n def has_split(self, split_index: int) -> bool:\n return self.probabilities[split_index] > 0\n\n @staticmethod\n def get_schema_cls():\n return RandomSplitConfig\n\n\n@split_registry.register(\"fixed\")\nclass FixedSplitter(Splitter):\n def __init__(self, column: str = SPLIT, **kwargs):\n self.column = column\n\n def split(\n self, df: DataFrame, backend: Backend, random_seed: float = default_random_seed\n ) -> tuple[DataFrame, DataFrame, DataFrame]:\n df[self.column] = df[self.column].astype(np.int8)\n dfs = split_dataset_ttv(df, self.column)\n train, test, val = tuple(df.drop(columns=self.column) if df is not None else None for df in dfs)\n return train, val, test\n\n @property\n def required_columns(self) -> list[str]:\n return [self.column]\n\n @staticmethod\n def get_schema_cls():\n return FixedSplitConfig\n\n\ndef stratify_split_dataframe(\n df: DataFrame, column: str, probabilities: list[float], backend: Backend, random_seed: float\n) -> tuple[DataFrame, DataFrame, DataFrame]:\n \"\"\"Splits a dataframe into train, validation, and test sets based on the values of a column.\n\n The column must be categorical (including binary). The split is stratified, meaning that the proportion of each\n category in each split is the same as in the original dataset.\n \"\"\"\n\n frac_train, frac_val, frac_test = probabilities\n\n def _safe_stratify(df, column, test_size):\n # Get the examples with cardinality of 1\n df_cadinalities = df.groupby(column)[column].size()\n low_cardinality_elems = df_cadinalities.loc[lambda x: x == 1]\n df_low_card = df[df[column].isin(low_cardinality_elems.index)]\n df = df[~df[column].isin(low_cardinality_elems.index)]\n y = df[[column]]\n\n df_train, df_temp, _, _ = train_test_split(df, y, stratify=y, test_size=test_size, random_state=random_seed)\n\n # concat the examples with cardinality of 1 to the training DF.\n if len(df_low_card.index) > 0:\n df_train = backend.df_engine.concat([df_train, df_low_card])\n\n return df_train, df_temp\n\n df_train, df_temp = _safe_stratify(df, column, 1.0 - frac_train)\n\n relative_frac_test = frac_test / (frac_val + frac_test)\n df_val, df_test = _safe_stratify(df_temp, column, relative_frac_test)\n\n return df_train, df_val, df_test\n\n\n@split_registry.register(\"stratify\")\nclass StratifySplitter(Splitter):\n def __init__(self, column: str, probabilities: list[float] = DEFAULT_PROBABILITIES, **kwargs):\n self.column = column\n self.probabilities = probabilities\n\n def split(\n self, df: DataFrame, backend: Backend, random_seed: float = default_random_seed\n ) -> tuple[DataFrame, DataFrame, DataFrame]:\n if not backend.df_engine.partitioned:\n return stratify_split_dataframe(df, self.column, self.probabilities, backend, random_seed)\n\n # For a partitioned dataset, we can stratify split each partition individually\n # to obtain a global stratified split.\n\n def split_partition(partition: DataFrame) -> DataFrame:\n \"\"\"Splits a single partition into train, val, test.\n\n Returns a single DataFrame with the split column populated. Assumes that the split column is already present\n in the partition and has a default value of 0 (train).\n \"\"\"\n partition = partition.copy()\n _, val, test = stratify_split_dataframe(partition, self.column, self.probabilities, backend, random_seed)\n # Split column defaults to train, so only need to update val and test\n partition.loc[val.index, TMP_SPLIT_COL] = 1\n partition.loc[test.index, TMP_SPLIT_COL] = 2\n return partition\n\n df[TMP_SPLIT_COL] = 0\n df = backend.df_engine.map_partitions(df, split_partition, meta=df)\n\n df_train = df[df[TMP_SPLIT_COL] == 0].drop(columns=TMP_SPLIT_COL)\n df_val = df[df[TMP_SPLIT_COL] == 1].drop(columns=TMP_SPLIT_COL)\n df_test = df[df[TMP_SPLIT_COL] == 2].drop(columns=TMP_SPLIT_COL)\n\n return df_train, df_val, df_test\n\n def validate(self, config: \"ModelConfig\"): # noqa: F821\n features = [f for f in config.input_features] + [f for f in config.output_features]\n feature_cols = {f.column for f in features}\n if self.column not in feature_cols:\n logging.info(\n f\"Stratify column {self.column} is not among the features. \"\n f\"Cannot establish if it is a binary or category feature.\"\n )\n elif [f for f in features if f.column == self.column][0].type not in {BINARY, CATEGORY}:\n raise ConfigValidationError(f\"Feature for stratify column {self.column} must be binary or category\")\n\n def has_split(self, split_index: int) -> bool:\n return self.probabilities[split_index] > 0\n\n @property\n def required_columns(self) -> list[str]:\n return [self.column]\n\n @staticmethod\n def get_schema_cls():\n return StratifySplitConfig\n\n\n@split_registry.register(\"datetime\")\nclass DatetimeSplitter(Splitter):\n def __init__(\n self,\n column: str,\n probabilities: list[float] = DEFAULT_PROBABILITIES,\n datetime_format: str | None = None,\n fill_value: str = \"\",\n **kwargs,\n ):\n self.column = column\n self.probabilities = probabilities\n self.datetime_format = datetime_format\n self.fill_value = fill_value\n\n def split(\n self, df: DataFrame, backend: Backend, random_seed: float = default_random_seed\n ) -> tuple[DataFrame, DataFrame, DataFrame]:\n # In case the split column was preprocessed by Ludwig into a list, convert it back to a\n # datetime string for the sort and split\n def list_to_date_str(x):\n if not isinstance(x, list):\n if not isinstance(x, str):\n # Convert timestamps, etc. to strings and return so it can direct cast to epoch time\n return str(x)\n\n if len(x) != 9:\n # Strings not in the expected format, so assume it's a formatted datetime and return\n return x\n\n return f\"{x[0]}-{x[1]}-{x[2]} {x[5]}:{x[6]}:{x[7]}\"\n\n df[TMP_SPLIT_COL] = backend.df_engine.map_objects(df[self.column], list_to_date_str)\n\n # Convert datetime to int64 to workaround Dask limitation\n # https://github.com/dask/dask/issues/9003\n df[TMP_SPLIT_COL] = backend.df_engine.df_lib.to_datetime(df[TMP_SPLIT_COL]).values.astype(\"int64\")\n\n # Sort by ascending datetime and drop the temporary column\n df = df.sort_values(TMP_SPLIT_COL).drop(columns=TMP_SPLIT_COL)\n\n # Split using different methods based on the underlying df engine.\n # For Pandas, split by row index.\n # For Dask, split by partition, as splitting by row is very inefficient.\n return tuple(backend.df_engine.split(df, self.probabilities))\n\n def validate(self, config: \"ModelConfig\"): # noqa: F821\n features = [f for f in config.input_features] + [f for f in config.output_features]\n feature_cols = {f.column for f in features}\n if self.column not in feature_cols:\n logging.info(\n f\"Datetime split column {self.column} is not among the features. \"\n f\"Cannot establish if it is a valid datetime.\"\n )\n elif [f for f in features if f.column == self.column][0].type not in {DATE}:\n raise ConfigValidationError(f\"Feature for datetime split column {self.column} must be a datetime\")\n\n def has_split(self, split_index: int) -> bool:\n return self.probabilities[split_index] > 0\n\n @property\n def required_columns(self) -> list[str]:\n return [self.column]\n\n @staticmethod\n def get_schema_cls():\n return DateTimeSplitConfig\n\n\n@split_registry.register(\"hash\")\nclass HashSplitter(Splitter):\n def __init__(\n self,\n column: str,\n probabilities: list[float] = DEFAULT_PROBABILITIES,\n **kwargs,\n ):\n self.column = column\n self.probabilities = probabilities\n\n def split(\n self, df: DataFrame, backend: Backend, random_seed: float = default_random_seed\n ) -> tuple[DataFrame, DataFrame, DataFrame]:\n # Maximum value of the hash function crc32\n max_value = 2**32\n thresholds = [v * max_value for v in self.probabilities]\n\n def hash_column(x):\n value = hash_dict({\"value\": x}, max_length=None)\n hash_value = crc32(value)\n if hash_value < thresholds[0]:\n return 0\n elif hash_value < (thresholds[0] + thresholds[1]):\n return 1\n else:\n return 2\n\n df[TMP_SPLIT_COL] = backend.df_engine.map_objects(df[self.column], hash_column).astype(np.int8)\n dfs = split_dataset_ttv(df, TMP_SPLIT_COL)\n train, test, val = tuple(df.drop(columns=TMP_SPLIT_COL) if df is not None else None for df in dfs)\n return train, val, test\n\n def has_split(self, split_index: int) -> bool:\n return self.probabilities[split_index] > 0\n\n @property\n def required_columns(self) -> list[str]:\n return [self.column]\n\n @staticmethod\n def get_schema_cls():\n return HashSplitConfig\n\n\n@DeveloperAPI\ndef get_splitter(type: str | None = None, **kwargs) -> Splitter:\n splitter_cls = split_registry.get(type)\n if splitter_cls is None:\n return ValueError(f\"Invalid split type: {type}\")\n return splitter_cls(**kwargs)\n\n\n@DeveloperAPI\ndef split_dataset(\n df: DataFrame,\n global_preprocessing_parameters: PreprocessingConfigDict,\n backend: Backend,\n random_seed: float = default_random_seed,\n) -> tuple[DataFrame, DataFrame, DataFrame]:\n splitter = get_splitter(**global_preprocessing_parameters.get(SPLIT, {}))\n datasets: tuple[DataFrame, DataFrame, DataFrame] = splitter.split(df, backend, random_seed)\n if len(datasets[0].columns) == 0:\n raise ValueError(\n \"Encountered an empty training set while splitting data. Please double check the preprocessing split \"\n \"configuration.\"\n )\n\n # Remove partitions that are empty after splitting\n datasets = [None if dataset is None else backend.df_engine.remove_empty_partitions(dataset) for dataset in datasets]\n return datasets\n"
},
{
"path": "ludwig/data/split_dataset.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport argparse\nimport random\n\n\ndef split(input_path, output1, output2, split):\n with open(input_path) as file:\n lines = file.readlines()\n\n random.shuffle(lines)\n split_idx = int(len(lines) * split)\n\n with open(output1, \"w\") as f:\n for line in lines[:split_idx]:\n line = line if line.endswith(\"\\n\") else line + \"\\n\"\n f.write(line)\n\n with open(output2, \"w\") as f:\n for line in lines[split_idx:]:\n line = line if line.endswith(\"\\n\") else line + \"\\n\"\n f.write(line)\n\n\nif __name__ == \"__main__\":\n parser = argparse.ArgumentParser(description=\"Split a file based on its lines\")\n\n parser.add_argument(\"-i\", \"--input\", required=True, help=\"input file names\")\n parser.add_argument(\"-o1\", \"--output1\", required=True, help=\"output 1 file name\")\n parser.add_argument(\"-o2\", \"--output2\", required=True, help=\"output 2 file name\")\n parser.add_argument(\"-s\", \"--split\", required=True, type=float, default=0.8, help=\"percentage of the split\")\n\n args = parser.parse_args()\n\n split(args.input, args.output1, args.output2, args.split)\n"
},
{
"path": "ludwig/data/utils.py",
"content": "from typing import Optional\n\nimport numpy as np\n\nfrom ludwig.constants import DECODER, ENCODER, SPLIT\nfrom ludwig.types import FeatureConfigDict, PreprocessingConfigDict\nfrom ludwig.utils.dataframe_utils import is_dask_series_or_df\nfrom ludwig.utils.types import DataFrame\n\n\ndef convert_to_dict(\n predictions: DataFrame,\n output_features: dict[str, FeatureConfigDict],\n backend: Optional[\"Backend\"] = None, # noqa: F821\n):\n \"\"\"Convert predictions from DataFrame format to a dictionary.\"\"\"\n output = {}\n for of_name, output_feature in output_features.items():\n feature_keys = {k for k in predictions.columns if k.startswith(of_name)}\n feature_dict = {}\n for key in feature_keys:\n subgroup = key[len(of_name) + 1 :]\n\n values = predictions[key]\n if is_dask_series_or_df(values, backend):\n values = values.compute()\n try:\n values = np.stack(values.to_numpy())\n except ValueError:\n values = values.to_list()\n\n feature_dict[subgroup] = values\n output[of_name] = feature_dict\n return output\n\n\ndef set_fixed_split(preprocessing_params: PreprocessingConfigDict) -> PreprocessingConfigDict:\n \"\"\"Sets the split policy explicitly to a fixed split.\n\n This potentially overrides the split configuration that the user set or what came from schema defaults.\n \"\"\"\n\n return {\n **preprocessing_params,\n \"split\": {\n \"type\": \"fixed\",\n \"column\": SPLIT,\n },\n }\n\n\ndef get_input_and_output_features(feature_configs):\n \"\"\"Returns a tuple (input_features, output_features) where each element is a list of feature configs.\n\n Determines whether a feature is an input or output feature by checking the presence of the encoder or decoder keys.\n \"\"\"\n input_features = []\n output_features = []\n for feature in feature_configs:\n if ENCODER in feature:\n input_features.append(feature)\n elif DECODER in feature:\n output_features.append(feature)\n return input_features, output_features\n"
},
{
"path": "ludwig/datasets/README.md",
"content": "## Ludwig Datasets API\n\nThe Ludwig Dataset Zoo provides datasets that can be directly plugged into a Ludwig model. For each dataset, we've also\nincluded an example Ludwig config which should train reasonably fast on a current-generation laptop.\n\nThe simplest way to use a dataset is to import it:\n\n```python\nfrom ludwig.datasets import titanic\n\n# Loads into single dataframe with a 'split' column:\ndataset_df = titanic.load()\n\n# Loads into split dataframes:\ntrain_df, test_df, _ = titanic.load(split=True)\n```\n\nThe `ludwig.datasets` API provides functions to list, describe, and get datasets:\n\n______________________________________________________________________\n\n### list_datasets\n\nGets a list of the names of available datasets.\n\n**Example:**\n\n```python\ndataset_names = ludwig.datasets.list_datasets()\n```\n\n______________________________________________________________________\n\n### get_datasets_output_features\n\nIf a specific dataset name is passed in, then returns the output features associated with that dataset. Otherwise,\nreturns an ordered dictionary with dataset names as keys and dictionaries containing the output features for each\ndataset as values.\n\n**Example:**\n\n```python\noutput_features = ludwig.datasets.get_datasets_output_features(dataset=\"titanic\")\n```\n\n______________________________________________________________________\n\n### describe_dataset\n\nGets a human-readable description string for a dataset\n\n**Example:**\n\n```python\nprint(ludwig.datasets.describe_dataset(\"titanic\"))\n```\n\n______________________________________________________________________\n\n### get_dataset\n\nGet a dataset module by name\n\n**Example:**\n\n```python\ntitanic_dataset = ludwig.datasets.get_dataset(\"titanic\")\n```\n\n______________________________________________________________________\n\n### model_configs_for_dataset\n\nGets a dictionary of model configs for the specified dataset. Keys are the config names, and may\ncontain the special keys:\n\n- `default` - The default config for the dataset. Should train to decent performance under 10 minutes on a typical\n laptop without GPU.\n- `best` - The best known config for the dataset. Should be replaced when a better config is found. This is a good\n opportunity for contributions, if you find a better one please check it in and open a PR!\n\n**Example:**\n\n```python\nconfigs = ludwig.datasets.model_configs_for_dataset(\"higgs\")\ndefault_higgs_config = configs[\"default\"]\nbest_higgs_config = configs[\"best\"]\n```\n\n______________________________________________________________________\n\n## Training a model using builtin dataset and config\n\nThis example code trains a model on the Titanic dataset using the default config:\n\n```python\nfrom ludwig.api import LudwigModel\nimport ludwig.datasets\n\ntitanic = ludwig.datasets.get_dataset(\"titanic\")\n\ndataset_df = titanic.load()\n\ntitanic_config = titanic.default_model_config\n\nmodel = LudwigModel(titanic_config)\nmodel.train(dataset_df)\n```\n\nSome datasets are hosted on [Kaggle](https://www.kaggle.com) and require a kaggle account. To use these, you'll need to\n[set up Kaggle credentials](https://www.kaggle.com/docs/api) in your environment. If the dataset is part of a Kaggle\ncompetition, you'll need to accept the terms on the competition page.\n\nTo check programmatically, datasets have an `.is_kaggle_dataset` property.\n\n## Downloading, Processing, and Exporting\n\nDatasets are first downloaded into `LUDWIG_CACHE`, which may be set as an environment variable and defaults to\n`$HOME/.ludwig_cache`.\n\nDatasets are automatically loaded, processed, and re-saved as parquet files. The processed dataset is saved in\nLUDWIG_CACHE.\n\nIf the dataset contains media files including images or audio, media files are saved in subdirectories and referenced by\nrelative paths from the dataset location. To ensure Ludwig can read these files during training, they should be\naccessible from Ludwig's working directory.\n\nTo export the processed dataset, including any media files it depends on, use the `.export` method:\n\n```python\nfrom ludwig.datasets import twitter_bots\n\n# Exports twitter bots dataset and image files to the current working directory.\ntwitter_bots.export(\".\")\n\n# The working directory should now contain:\n# ./twitter_bots.parquet - The twitter bots dataset\n# ./profile_images - Account profile image files\n# ./profile_background_images - Account profile background image files\n```\n"
},
{
"path": "ludwig/datasets/__init__.py",
"content": "import argparse\nimport importlib\nimport logging\nimport os\nfrom collections import OrderedDict\nfrom functools import lru_cache\nfrom io import BytesIO\nfrom typing import Any, Literal\n\nimport yaml\n\nfrom ludwig.api_annotations import DeveloperAPI, PublicAPI\nfrom ludwig.backend.base import Backend\nfrom ludwig.constants import AUDIO, BINARY, CATEGORY, IMAGE, NUMBER, TEST, TEXT, TRAIN, TYPE, VALIDATION\nfrom ludwig.data.cache.types import CacheableDataframe\nfrom ludwig.datasets import configs\nfrom ludwig.datasets.dataset_config import DatasetConfig\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\n\n# PublicAPI\nfrom ludwig.datasets.utils import model_configs_for_dataset # noqa\nfrom ludwig.globals import LUDWIG_VERSION\nfrom ludwig.utils.print_utils import print_ludwig\nfrom ludwig.utils.types import DataFrame\n\nURI_PREFIX = \"ludwig://\"\nHF_PREFIX = \"hf://\"\nSPLITS = [TRAIN, VALIDATION, TEST]\n\n\ndef _load_dataset_config(config_filename: str):\n \"\"\"Loads a dataset config.\"\"\"\n config_path = os.path.join(os.path.dirname(configs.__file__), config_filename)\n with open(config_path) as f:\n return DatasetConfig.from_dict(yaml.safe_load(f))\n\n\n@lru_cache(maxsize=1)\ndef _get_dataset_configs() -> dict[str, DatasetConfig]:\n \"\"\"Returns all dataset configs indexed by name.\"\"\"\n import importlib.resources\n\n config_files = [f.name for f in importlib.resources.files(configs).iterdir() if f.name.endswith(\".yaml\")]\n config_objects = [_load_dataset_config(f) for f in config_files]\n return {c.name: c for c in config_objects}\n\n\ndef _get_dataset_config(dataset_name) -> DatasetConfig:\n \"\"\"Get the config for a dataset.\"\"\"\n configs = _get_dataset_configs()\n if dataset_name not in configs:\n raise AttributeError(f\"No config found for dataset {dataset_name}\")\n return configs[dataset_name]\n\n\n@PublicAPI\ndef get_dataset(dataset_name, cache_dir=None) -> DatasetLoader:\n \"\"\"Gets an instance of the dataset loader for a dataset.\"\"\"\n config = _get_dataset_config(dataset_name)\n class_name = config.loader.split(\".\")[-1]\n module_name = \".\" + \".\".join(config.loader.split(\".\")[:-1])\n loader_module = importlib.import_module(module_name, package=\"ludwig.datasets.loaders\")\n loader_cls = getattr(loader_module, class_name)\n if cache_dir:\n return loader_cls(config, cache_dir=cache_dir)\n return loader_cls(config)\n\n\n@DeveloperAPI\ndef load_dataset_uris(\n dataset: str | DataFrame | None,\n training_set: str | DataFrame | None,\n validation_set: str | DataFrame | None,\n test_set: str | DataFrame | None,\n backend: Backend,\n) -> tuple[\n CacheableDataframe | None,\n CacheableDataframe | None,\n CacheableDataframe | None,\n CacheableDataframe | None,\n]:\n \"\"\"Loads and returns any Ludwig dataset URIs as CacheableDataframes.\n\n Returns the input unmodified for any non-Ludwig datasets.\n \"\"\"\n\n dataset_out, training_set_out, validation_set_out, test_set_out = dataset, training_set, validation_set, test_set\n # Check that any of the datasets begin with the `hf://` prefix denoting a Hugging Face dataset URI\n # Hugging Face datasets should follow the naming convention `hf://--`\n if _is_hf(dataset, training_set):\n return _load_hf_datasets(dataset, training_set, validation_set, test_set, backend)\n\n # Check that any of the datasets begin with the `ludwig://` prefix denoting a Ludwig dataset URI\n if dataset is not None:\n if isinstance(dataset, str) and dataset.startswith(URI_PREFIX):\n dataset_out = _load_cacheable_dataset(dataset, backend)\n return dataset_out, training_set_out, validation_set_out, test_set_out\n if training_set is not None:\n train_df = test_df = val_df = None\n training_set_checksum = None\n if isinstance(training_set, str) and training_set.startswith(URI_PREFIX):\n # For the training set, we only want to use the TRAINING split of the dataset\n dataset_name = training_set[len(URI_PREFIX) :]\n loader = get_dataset(dataset_name)\n train_df, test_df, val_df = loader.load(split=True)\n training_set_checksum = str(loader.get_mtime())\n train_df = backend.df_engine.from_pandas(train_df)\n training_set_out = CacheableDataframe(df=train_df, name=training_set, checksum=training_set_checksum)\n\n if isinstance(validation_set, str) and validation_set.startswith(URI_PREFIX):\n if validation_set == training_set:\n # Reuse the loaded DF from the training split\n val_df = backend.df_engine.from_pandas(val_df)\n validation_set_out = CacheableDataframe(df=val_df, name=validation_set, checksum=training_set_checksum)\n else:\n validation_set_out = _load_cacheable_dataset(validation_set, backend)\n\n if isinstance(test_set, str) and test_set.startswith(URI_PREFIX):\n if test_set == training_set:\n # Reuse the loaded DF from the training split\n test_df = backend.df_engine.from_pandas(test_df)\n test_set_out = CacheableDataframe(df=test_df, name=test_set, checksum=training_set_checksum)\n else:\n test_set_out = _load_cacheable_dataset(test_set, backend)\n\n return dataset_out, training_set_out, validation_set_out, test_set_out\n\n\ndef _is_hf(dataset, training_set):\n dataset_is_hf = dataset is not None and isinstance(dataset, str) and dataset.startswith(HF_PREFIX)\n training_set_is_hf = (\n training_set is not None and isinstance(training_set, str) and training_set.startswith(HF_PREFIX)\n )\n return dataset_is_hf or training_set_is_hf\n\n\ndef _load_hf_datasets(\n dataset: str | DataFrame | None,\n training_set: str | DataFrame | None,\n validation_set: str | DataFrame | None,\n test_set: str | DataFrame | None,\n backend: Backend,\n) -> tuple[\n CacheableDataframe | None,\n CacheableDataframe | None,\n CacheableDataframe | None,\n CacheableDataframe | None,\n]:\n \"\"\"Loads and returns any Hugging Face datasets as CacheableDataframes.\n\n Returns the input unmodified for any non-HF datasets.\n \"\"\"\n dataset_out = dataset\n training_set_out = training_set\n validation_set_out = validation_set\n test_set_out = test_set\n\n # Check that any of the datasets begin with the `hf://` prefix denoting a Hugging Face dataset URI\n # Hugging Face datasets should follow the naming convention `hf://--`\n if dataset is not None:\n if isinstance(dataset, str) and dataset.startswith(HF_PREFIX):\n dataset_out = _load_cacheable_hf_dataset(dataset, backend)\n return dataset_out, training_set_out, validation_set_out, test_set_out\n\n # Because of the conditional logic (_is_hf) in load_dataset_uris, if the above block is not triggered, then\n # training_set must be a string that starts with HF_PREFIX\n train_df = test_df = val_df = None\n loader = get_dataset(\"hugging_face\")\n hf_id, hf_subsample = _get_hf_dataset_and_subsample(training_set)\n train_df, val_df, test_df = loader.load(hf_id, hf_subsample, split=True) # Call hugging_face loader\n train_df = backend.df_engine.from_pandas(train_df)\n training_set_out = CacheableDataframe(df=train_df, name=training_set, checksum=None)\n\n if isinstance(validation_set, str) and validation_set.startswith(HF_PREFIX):\n if validation_set == training_set:\n # Reuse the loaded DF from the training split\n val_df = backend.df_engine.from_pandas(val_df)\n validation_set_out = CacheableDataframe(df=val_df, name=validation_set, checksum=None)\n else: # This handles an edge case -- NOT EXPECTED USER BEHAVIOR\n logging.warning(\n \"A Hugging Face validation set has been passed in that is different from the test set. \"\n \"This is not recommended.\"\n )\n validation_set_out = _load_cacheable_hf_dataset(validation_set, backend, split_set=VALIDATION)\n\n if isinstance(test_set, str) and test_set.startswith(HF_PREFIX):\n if test_set == training_set:\n # Reuse the loaded DF from the training split\n test_df = backend.df_engine.from_pandas(test_df)\n test_set_out = CacheableDataframe(df=test_df, name=test_set, checksum=None)\n else: # This handles an edge case -- NOT EXPECTED USER BEHAVIOR\n logging.warning(\n \"A Hugging Face test set has been passed in that is different from the training set. \"\n \"This is not recommended.\"\n )\n test_set_out = _load_cacheable_hf_dataset(test_set, backend, split_set=TEST)\n\n return dataset_out, training_set_out, validation_set_out, test_set_out\n\n\ndef _load_cacheable_hf_dataset(\n dataset: str, backend: Backend, split_set: Literal[\"train\", \"validation\", \"test\"] | None = None\n) -> CacheableDataframe:\n loader = get_dataset(\"hugging_face\")\n hf_id, hf_subsample = _get_hf_dataset_and_subsample(dataset)\n if split_set:\n train_df, validation_df, test_df = loader.load(hf_id, hf_subsample, split=True)\n df = [train_df, validation_df, test_df][\n SPLITS.index(split_set)\n ] # split_set should be one of TRAIN, VALIDATION, or TEST\n else:\n df = loader.load(hf_id, hf_subsample, split=False)\n df = backend.df_engine.from_pandas(df)\n return CacheableDataframe(df=df, name=dataset, checksum=None)\n\n\ndef _load_cacheable_dataset(dataset: str, backend: Backend) -> CacheableDataframe:\n dataset_name = dataset[len(URI_PREFIX) :]\n loader = get_dataset(dataset_name)\n df = loader.load(split=False)\n df = backend.df_engine.from_pandas(df)\n return CacheableDataframe(df=df, name=dataset, checksum=str(loader.get_mtime()))\n\n\n@PublicAPI\ndef list_datasets() -> list[str]:\n \"\"\"Returns a list of the names of all available datasets.\"\"\"\n return sorted(_get_dataset_configs().keys())\n\n\n@PublicAPI\ndef get_datasets_output_features(\n dataset: str = None, include_competitions: bool = True, include_data_modalities: bool = False\n) -> dict:\n \"\"\"Returns a dictionary with the output features for each dataset. Optionally, you can pass a dataset name\n which will then cause the function to return a dictionary with the output features for that dataset.\n\n Because Hugging Face Datasets are loaded dynamically through a shared connector, they don't have fixed output\n features. As such, we exclude Hugging Face datasets here.\n\n :param dataset: (str) name of the dataset\n :param include_competitions: (bool) whether to include the output features from kaggle competition datasets\n :param include_data_modalities: (bool) whether to include the data modalities associated with the prediction task\n :return: (dict) dictionary with the output features for each dataset or a dictionary with the output features for\n the specified dataset\n \"\"\"\n ordered_configs = OrderedDict(sorted(_get_dataset_configs().items()))\n competition_datasets = []\n hugging_face_datasets = []\n\n for name, config in ordered_configs.items():\n if not include_competitions and config.kaggle_competition:\n competition_datasets.append(name)\n continue\n\n if config.name == \"hugging_face\":\n # There is no output_features attribute for hugging_face datasets\n hugging_face_datasets.append(name)\n continue\n\n ordered_configs[name] = {\"name\": config.name, \"output_features\": config.output_features}\n\n if include_data_modalities:\n column_types = {column[TYPE] for column in config.columns}\n\n data_modalities = set()\n if NUMBER in column_types or CATEGORY in column_types or BINARY in column_types:\n data_modalities.add(\"Tabular\")\n if TEXT in column_types:\n data_modalities.add(\"Text\")\n if IMAGE in column_types:\n data_modalities.add(\"Image\")\n if AUDIO in column_types:\n data_modalities.add(\"Audio\")\n\n ordered_configs[name][\"data_modalities\"] = data_modalities\n\n if dataset:\n return ordered_configs[dataset]\n\n if not include_competitions:\n for competition in competition_datasets:\n del ordered_configs[competition]\n\n del ordered_configs[\"hugging_face\"]\n\n return ordered_configs\n\n\n@PublicAPI\ndef describe_dataset(dataset_name: str) -> str:\n \"\"\"Returns the description of the dataset.\"\"\"\n return _get_dataset_configs()[dataset_name].description\n\n\n@PublicAPI\ndef download_dataset(dataset_name: str, output_dir: str = \".\"):\n \"\"\"Downloads the dataset to the specified directory.\"\"\"\n output_dir = os.path.expanduser(os.path.normpath(output_dir))\n dataset = get_dataset(dataset_name)\n dataset.export(output_dir)\n\n\n@DeveloperAPI\ndef get_buffer(dataset_name: str, kaggle_username: str = None, kaggle_key: str = None) -> BytesIO:\n \"\"\"Returns a byte buffer for the specified dataset.\"\"\"\n try:\n if dataset_name.startswith(HF_PREFIX):\n hf_id, hf_subsample = _get_hf_dataset_and_subsample(dataset_name)\n dataset = get_dataset(\"hugging_face\").load(hf_id, hf_subsample)\n else:\n dataset = get_dataset(dataset_name).load(kaggle_username=kaggle_username, kaggle_key=kaggle_key)\n buffer = BytesIO(dataset.to_parquet())\n return buffer\n except Exception as e:\n logging.error(logging.ERROR, f\"Failed to upload dataset {dataset_name}: {e}\")\n\n\ndef _get_hf_dataset_and_subsample(dataset_name: str) -> tuple[str, str | None]:\n \"\"\"Returns the Hugging Face ID and subsample name from the dataset name.\n\n The dataset name should follow the format \"{HF_PREFIX}{hf_id}--{hf_subsample}\"\n\n Examples (Dataset Name --> HF ID; HF subsample): \"hf://wikisql\" --> \"wikisql\"; None \"hf://ColumbiaNLP/FLUTE\" -->\n \"ColumbiaNLP/FLUTE\"; None \"hf://mstz/adult--income\" --> \"mstz/adult\"; \"income\"\n \"\"\"\n dataset_name = dataset_name[len(HF_PREFIX) :]\n dataset_name = dataset_name.split(\"--\")\n if len(dataset_name) == 1:\n return dataset_name[0], None\n return dataset_name[0], dataset_name[1]\n\n\ndef cli(sys_argv):\n parser = argparse.ArgumentParser(\n description=\"This command downloads and lists Ludwig-ready datasets.\",\n prog=\"ludwig datasets\",\n usage=\"%(prog)s [options]\",\n )\n sub_parsers = parser.add_subparsers(dest=\"command\", help=\"download and list datasets\")\n\n parser_download = sub_parsers.add_parser(\"download\", help=\"download a dataset\")\n parser_download.add_argument(\"dataset\", help=\"dataset to download\")\n parser_download.add_argument(\n \"-o\",\n \"--output_dir\",\n type=str,\n default=\".\",\n help=\"output directory to download into\",\n required=False,\n )\n\n sub_parsers.add_parser(\"list\", help=\"list datasets\")\n\n parser_describe = sub_parsers.add_parser(\"describe\", help=\"describe datasets\")\n parser_describe.add_argument(\"dataset\", help=\"dataset to describe\")\n\n args = parser.parse_args(sys_argv)\n print_ludwig(f\"Datasets {args.command}\", LUDWIG_VERSION)\n\n if args.command == \"list\":\n datasets = list_datasets()\n for ds in datasets:\n print(ds)\n elif args.command == \"describe\":\n print(describe_dataset(args.dataset))\n elif args.command == \"download\":\n download_dataset(args.dataset, args.output_dir)\n else:\n raise ValueError(f\"Unrecognized command: {args.command}\")\n\n\ndef __getattr__(name: str) -> Any:\n \"\"\"Module-level __getattr__ allows us to return an instance of a class. For example:\n\n from ludwig.datasets import titanic\n\n returns an instance of DatasetLoader configured to load titanic.\n\n If you want to download a dataset in a non-default ludwig cache directory, there are two options:\n 1. set the LUDWIG_CACHE environment variable to your desired path before importing the dataset\n 2. Use ludwig.datasets.get_dataset(dataset_name, cache_dir=)\n \"\"\"\n public_methods = {\n \"list_datasets\",\n \"describe_dataset\",\n \"download_dataset\",\n \"cli\",\n \"get_dataset\",\n \"model_configs_for_dataset\",\n }\n if name in public_methods:\n return globals()[name]\n return get_dataset(name)\n"
},
{
"path": "ludwig/datasets/archives.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport gzip\nimport logging\nimport os\nimport shutil\nimport tarfile\nfrom enum import Enum\nfrom zipfile import ZipFile\n\nfrom ludwig.utils.fs_utils import upload_output_directory\n\nlogger = logging.getLogger(__name__)\n\n\nclass ArchiveType(str, Enum):\n \"\"\"The type of file archive.\"\"\"\n\n UNKNOWN = \"unknown\"\n ZIP = \"zip\"\n GZIP = \"gz\"\n TAR = \"tar\"\n TAR_ZIP = \"tar.z\"\n TAR_BZ2 = \"tar.bz2\"\n TAR_GZ = \"tar.gz\"\n\n\ndef infer_archive_type(archive_path):\n \"\"\"Try to infer archive type from file extension.\"\"\"\n # Get the path extension including multiple extensions, ex. \".tar.gz\"\n extension = \".\".join([\"\", *os.path.basename(archive_path).split(\".\")[1:]])\n extension = extension.lower()\n if extension.endswith(\".tar.z\") or extension.endswith(\".tar.zip\"):\n return ArchiveType.TAR_ZIP\n elif extension.endswith(\".tar.bz2\") or extension.endswith(\".tbz2\"):\n return ArchiveType.TAR_BZ2\n elif extension.endswith(\".tar.gz\") or extension.endswith(\".tgz\"):\n return ArchiveType.TAR_GZ\n elif extension.endswith(\".tar\"):\n return ArchiveType.TAR\n elif extension.endswith(\".zip\") or extension.endswith(\".zipx\"):\n return ArchiveType.ZIP\n elif extension.endswith(\".gz\") or extension.endswith(\".gzip\"):\n return ArchiveType.GZIP\n else:\n return ArchiveType.UNKNOWN\n\n\ndef is_archive(path):\n \"\"\"Does this path a supported archive type.\"\"\"\n return infer_archive_type(path) != ArchiveType.UNKNOWN\n\n\ndef list_archive(archive_path, archive_type: ArchiveType | None = None) -> list[str]:\n \"\"\"Return list of files extracted in an archive (without extracting them).\"\"\"\n if archive_type is None:\n archive_type = infer_archive_type(archive_path)\n if archive_type == ArchiveType.UNKNOWN:\n logger.error(\n f\"Could not infer type of archive {archive_path}. May be an unsupported archive type.\"\n \"Specify archive_type in the dataset config if this file has an unknown file extension.\"\n )\n return []\n if archive_type == ArchiveType.ZIP:\n with ZipFile(archive_path) as zfile:\n return zfile.namelist()\n elif archive_type == ArchiveType.GZIP:\n return [\".\".join(archive_path.split(\".\")[:-1])] # Path minus the .gz extension\n elif archive_type in {ArchiveType.TAR, ArchiveType.TAR_ZIP, ArchiveType.TAR_BZ2, ArchiveType.TAR_GZ}:\n with tarfile.open(archive_path) as tar_file:\n return tar_file.getnames()\n else:\n logger.error(f\"Unsupported archive: {archive_path}\")\n return []\n\n\ndef extract_archive(archive_path: str, archive_type: ArchiveType | None = None) -> list[str]:\n \"\"\"Extracts files from archive (into the same directory), returns a list of extracted files.\n\n Args:\n archive_path - The full path to the archive.\n\n Returns A list of the files extracted.\n \"\"\"\n if archive_type is None:\n archive_type = infer_archive_type(archive_path)\n if archive_type == ArchiveType.UNKNOWN:\n logger.error(\n f\"Could not infer type of archive {archive_path}. May be an unsupported archive type.\"\n \"Specify archive_type in the dataset config if this file has an unknown file extension.\"\n )\n return []\n archive_directory = os.path.dirname(archive_path)\n directory_contents_before = os.listdir(archive_directory)\n with upload_output_directory(archive_directory) as (tmpdir, _):\n if archive_type == ArchiveType.ZIP:\n with ZipFile(archive_path) as zfile:\n zfile.extractall(tmpdir)\n elif archive_type == ArchiveType.GZIP:\n gzip_content_file = \".\".join(archive_path.split(\".\")[:-1]) # Path minus the .gz extension\n with gzip.open(archive_path) as gzfile:\n with open(os.path.join(tmpdir, gzip_content_file), \"wb\") as output:\n shutil.copyfileobj(gzfile, output)\n elif archive_type in {ArchiveType.TAR, ArchiveType.TAR_ZIP, ArchiveType.TAR_BZ2, ArchiveType.TAR_GZ}:\n with tarfile.open(archive_path) as tar_file:\n\n def is_within_directory(directory, target):\n abs_directory = os.path.abspath(directory)\n abs_target = os.path.abspath(target)\n\n prefix = os.path.commonprefix([abs_directory, abs_target])\n\n return prefix == abs_directory\n\n def safe_extract(tar, path=\".\", members=None, *, numeric_owner=False):\n for member in tar.getmembers():\n member_path = os.path.join(path, member.name)\n if not is_within_directory(path, member_path):\n raise Exception(\"Attempted Path Traversal in Tar File\")\n\n tar.extractall(path, members, numeric_owner=numeric_owner)\n\n safe_extract(tar_file, path=tmpdir)\n else:\n logger.error(f\"Unsupported archive: {archive_path}\")\n directory_contents_after = set(os.listdir(archive_directory))\n return directory_contents_after.difference(directory_contents_before)\n"
},
{
"path": "ludwig/datasets/configs/__init__.py",
"content": ""
},
{
"path": "ludwig/datasets/configs/adult_census_income.yaml",
"content": "version: 1.0\nname: adult_census_income\ndownload_urls:\n - https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data\n - https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.test\ntrain_filenames: adult.data\ntest_filenames: adult.test\nsha256:\n adult.data: 5b00264637dbfec36bdeaab5676b0b309ff9eb788d63554ca0a249491c86603d\n adult.test: a2a9044bc167a35b2361efbabec64e89d69ce82d9790d2980119aac5fd7e9c05\nloader: adult_census_income.AdultCensusIncomeLoader\ndescription: |\n Predict whether income exceeds $50K/yr based on census data\n https://archive.ics.uci.edu/ml/datasets/adult\ncolumns:\n - name: age\n type: number\n - name: workclass\n type: category\n - name: fnlwgt\n type: category\n - name: education\n type: category\n - name: education-num\n type: category\n - name: marital-status\n type: category\n - name: occupation\n type: category\n - name: relationship\n type: category\n - name: race\n type: category\n - name: sex\n type: category\n - name: capital-gain\n type: number\n - name: capital-loss\n type: number\n - name: hours-per-week\n type: number\n - name: native-country\n type: category\n - name: income\n type: category\noutput_features:\n - name: income\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/ae_price_prediction.yaml",
"content": "version: 1.0\nname: ae_price_prediction\ndownload_urls:\n - https://automl-mm-bench.s3.amazonaws.com/ae_price_prediction/train.pq\n - https://automl-mm-bench.s3.amazonaws.com/ae_price_prediction/test.pq\nsha256:\n test.pq: d05242580e011f3ac5a1a8f0069fd7788ceeacd6b2fb00ca7f409991f998c95e\n train.pq: 181cfebbedd5c6e2bdc6261706103edddfc6eeb4604b8c6ffdc3d084a6e09a4e\ntrain_filenames: train.pq\ntest_filenames: test.pq\ndescription: |\n Innerwear Data from Victoria's Secret and Others\n 600,000+ innerwear product data extracted from popular retail sites\n https://www.kaggle.com/PromptCloudHQ/innerwear-data-from-victorias-secret-and-others\ncolumns:\n - name: product_name\n type: category\n - name: mrp\n type: category\n - name: price\n type: number\n - name: pdp_url\n type: category\n - name: brand_name\n type: category\n - name: product_category\n type: category\n - name: retailer\n type: category\n - name: description\n type: text\n - name: rating\n type: number\n - name: review_count\n type: number\n - name: style_attributes\n type: set\n - name: total_sizes\n type: set\n - name: available_size\n type: set\n - name: color\n type: category\noutput_features:\n - name: price\n type: number\n"
},
{
"path": "ludwig/datasets/configs/agnews.yaml",
"content": "version: 1.0\nname: agnews\ndownload_urls:\n - https://raw.githubusercontent.com/mhjabreel/CharCnn_Keras/master/data/ag_news_csv/train.csv\n - https://raw.githubusercontent.com/mhjabreel/CharCnn_Keras/master/data/ag_news_csv/test.csv\ntrain_filenames: train.csv\ntest_filenames: test.csv\nsha256:\n test.csv: 521465c2428ed7f02f8d6db6ffdd4b5447c1c701962353eb2c40d548c3c85699\n train.csv: 76a0a2d2f92b286371fe4d4044640910a04a803fdd2538e0f3f29a5c6f6b672e\nloader: agnews.AGNewsLoader\ndescription: |\n News articles categorized as \"World\", \"Sports\", \"Business\", and \"Science\".\ncolumns:\n - name: class_index\n type: category\n - name: title\n type: text\n - name: description\n type: text\noutput_features:\n - name: class_index\n type: category\n"
},
{
"path": "ludwig/datasets/configs/allstate_claims_severity.yaml",
"content": "version: 1.0\nname: allstate_claims_severity\nkaggle_competition: allstate-claims-severity\narchive_filenames: allstate-claims-severity.zip\nsha256:\n allstate-claims-severity.zip: 165f7b4bc5ed40f43656dc958da6572143a7e126e2d37bcd41f1299bfbaa68e2\ntrain_filenames: train.csv\ntest_filenames: test.csv\nloader: allstate_claims_severity.AllstateClaimsSeverityLoader\ndescription: |\n Allstate Claims Severity.\n https://www.kaggle.com/c/allstate-claims-severity/overview\noutput_features:\n - name: loss\n type: number\n"
},
{
"path": "ludwig/datasets/configs/alpaca.yaml",
"content": "version: 1.0\nname: alpaca\ndownload_urls: https://raw.githubusercontent.com/tatsu-lab/stanford_alpaca/main/alpaca_data.json\ndataset_filenames: alpaca_data.json\ndescription: |\n Stanford Alpaca instruction-tuning dataset (https://github.com/tatsu-lab/stanford_alpaca) for LLM fine-tuning.\ncolumns:\n - name: instruction\n type: text\n - name: input\n type: text\n - name: output\n type: text\noutput_features:\n - name: output\n type: text\n"
},
{
"path": "ludwig/datasets/configs/amazon_employee_access_challenge.yaml",
"content": "version: 1.0\nname: amazon_employee_access_challenge\nkaggle_competition: amazon-employee-access-challenge\narchive_filenames: amazon-employee-access-challenge.zip\ntrain_filenames: train.csv\ntest_filenames: test.csv\nsha256:\n amazon-employee-access-challenge.zip: bba1cf24bc01f390e7faf3f9cdbebd6267c875d51a36a2c625ce66e0c3e71db7\ndescription: |\n There is a considerable amount of data regarding an employee’s role within an organization and the resources to which\n they have access. Given the data related to current employees and their provisioned access, models can be built that\n automatically determine access privileges as employees enter and leave roles within a company.\n https://www.kaggle.com/c/amazon-employee-access-challenge\noutput_features:\n - name: ACTION\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/amazon_review_polarity.yaml",
"content": "version: 1.0\nname: amazon_review_polarity\ndownload_urls: https://s3.amazonaws.com/fast-ai-nlp/amazon_review_polarity_csv.tgz\ntrain_filenames: amazon_review_polarity_csv/train.csv\ntest_filenames: amazon_review_polarity_csv/test.csv\nsha256:\n amazon_review_polarity_csv.tgz: d2a3ee7a214497a5d1b8eaed7c8d7ba2737de00ada3b0ec46243983efa100361\ndescription: |\n The Amazon Reviews Polarity dataset\n Details:\n 34,686,770 Amazon reviews from 6,643,669 users on 2,441,053\n products, from the Stanford Network Analysis Project (SNAP).\n This dataset contains 600,000 training samples and 130,000\n testing samples in each class.\n Dataset source:\n Character-level Convolutional Networks for Text Classification\n Xiang Zhang et al., 2015\ncolumns:\n - name: label\n type: binary\n - name: review_title\n type: text\n - name: review_text\n type: text\noutput_features:\n - name: label\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/amazon_reviews.yaml",
"content": "version: 1.0\nname: amazon_reviews\ndownload_urls: https://s3.amazonaws.com/fast-ai-nlp/amazon_review_full_csv.tgz\ntrain_filenames: amazon_review_full_csv/train.csv\ntest_filenames: amazon_review_full_csv/test.csv\nsha256:\n amazon_review_full_csv.tgz: 4af62eeee139d0142e0747340b68646d23483d9475c33ea0641ee9175b423443\ndescription: |\n The Amazon Reviews dataset\n Details:\n 34,686,770 Amazon reviews from 6,643,669 users on 2,441,053\n products, from the Stanford Network Analysis Project (SNAP).\n This dataset contains 600,000 training samples and 130,000\n testing samples in each class.\n Dataset source:\n Character-level Convolutional Networks for Text Classification\n Xiang Zhang et al., 2015\ncolumns:\n - name: label\n type: category\n - name: review_title\n type: text\n - name: review_text\n type: text\noutput_features:\n - name: label\n type: category\n"
},
{
"path": "ludwig/datasets/configs/ames_housing.yaml",
"content": "version: 1.0\nname: ames_housing\nkaggle_competition: house-prices-advanced-regression-techniques\narchive_filenames: house-prices-advanced-regression-techniques.zip\ntrain_filenames: train.csv\ntest_filenames: test.csv\nsha256:\n house-prices-advanced-regression-techniques.zip: 65f769a9157a2581671957ed08da8a8162d53e67b4e9970ee856b634deb11d9f\ndescription: |\n The Ames Housing dataset.\n https://www.kaggle.com/c/house-prices-advanced-regression-techniques\noutput_features:\n - name: SalePrice\n type: number\nfallback_mirrors:\n - name: predibase\n download_paths: s3://ludwig-tests/ludwig_backup/house-prices-advanced-regression-techniques.zip\n"
},
{
"path": "ludwig/datasets/configs/bbcnews.yaml",
"content": "version: 1.0\nname: bbcnews\nkaggle_competition: learn-ai-bbc\narchive_filenames: learn-ai-bbc.zip\ntrain_filenames: \"BBC News Train.csv\"\ntest_filenames: \"BBC News Test.csv\"\nsha256:\n learn-ai-bbc.zip: 450dd79c6654248af15d91d94c269fe7e8001effd89389f93c7184aac6699e62\ndescription: |\n BBC News Classification from Kaggle.\n https://www.kaggle.com/competitions/learn-ai-bbc/overview\noutput_features:\n - name: Category\n type: category\n"
},
{
"path": "ludwig/datasets/configs/bnp_claims_management.yaml",
"content": "version: 1.0\nname: bnp_claims_management\nkaggle_competition: bnp-paribas-cardif-claims-management\narchive_filenames: bnp-paribas-cardif-claims-management.zip\ntrain_filenames: train.csv\ntest_filenames: test.csv\nsha256:\n bnp-paribas-cardif-claims-management.zip: c01a11ceae565bc95ec30a1ef4c9ffe4aa27e07d6e433776e90a4d5474f3e95d\ndescription: |\n The BNP Paribas Cardif Claims Management dataset.\n https://www.kaggle.com/c/bnp-paribas-cardif-claims-management\noutput_features:\n - name: target\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/bookprice_prediction.yaml",
"content": "version: 1.0\nname: bookprice_prediction\ndownload_urls:\n - https://automl-mm-bench.s3.amazonaws.com/machine_hack_competitions/predict_the_price_of_books/train.csv\n - https://automl-mm-bench.s3.amazonaws.com/machine_hack_competitions/predict_the_price_of_books/test.csv\nsha256:\n test.csv: 75bcc853efe734a53764127428e005bb9eb7585ad3dc1dce2eb284fa04313c1b\n train.csv: dd978b591e623f9c5d4f9ade0f237200597afcad2c6417eb1e764698f1afcfcf\ntrain_filenames: train.csv\ntest_filenames: test.csv\ndescription: |\n Here we explore a database of books of different genres, from thousands of authors.\n In this challenge, participants are required to use the dataset to build a\n Machine Learning model to predict the price of books based on a given set of features.\n https://machinehack.com/hackathons/predict_the_price_of_books/overview\ncolumns:\n - name: Title\n type: category\n - name: Author\n type: category\n - name: Edition\n type: category\n - name: Reviews\n type: number\n - name: Ratings\n type: number\n - name: Synopsis\n type: text\n - name: Genre\n type: category\n - name: BookCategory\n type: category\n - name: Price\n type: number\noutput_features:\n - name: Price\n type: number\n"
},
{
"path": "ludwig/datasets/configs/california_house_price.yaml",
"content": "version: 1.0\nname: california_house_price\ndownload_urls:\n - https://automl-mm-bench.s3.amazonaws.com/kaggle-california-house-prices/train.csv\n - https://automl-mm-bench.s3.amazonaws.com/kaggle-california-house-prices/test.csv\nsha256:\n test.csv: b5bb9ed6e56cbdd0a410e186a19c6fe137c2ffbb50ba6b0808540434a8123dc6\n train.csv: 907d45804e622fb136a9d55bde97269f421fb9b8f7c9f34416672cf7078ee94b\ntrain_filenames: train.csv\ntest_filenames: test.csv\ndescription: |\n Predict house sale prices based on the house information, such as # of bedrooms,\n living areas, locations, near-by schools, and the seller summary. The data consist\n of houses sold in California in 2020, with houses in the test dataset sold after\n the ones in the training dataset.\n https://www.kaggle.com/c/california-house-prices\ncolumns:\n - name: Address\n type: category\n - name: Sold Price\n type: number\n - name: Summary\n type: text\n - name: Type\n type: category\n - name: Year built\n type: number\n - name: Heating\n type: category\n - name: Cooling\n type: category\n - name: Parking\n type: category\n - name: Lot\n type: number\n - name: Bedrooms\n type: number\n - name: Bathrooms\n type: number\n - name: Full bathrooms\n type: number\n - name: Total interior livable area\n type: number\n - name: Total spaces\n type: number\n - name: Garage spaces\n type: number\n - name: Region\n type: category\n - name: Elementary School\n type: category\n - name: Elementary School Score\n type: number\n - name: Elementary School Distance\n type: number\n - name: Middle School\n type: category\n - name: Middle School Score\n type: number\n - name: Middle School Distance\n type: number\n - name: High School\n type: category\n - name: High School Score\n type: number\n - name: High School Distance\n type: number\n - name: Flooring\n type: set\n - name: Heating features\n type: set\n - name: Cooling features\n type: set\n - name: Appliances included\n type: set\n - name: Laundry features\n type: set\n - name: Parking features\n type: set\n - name: Tax assessed value\n type: number\n - name: Annual tax amount\n type: number\n - name: Listed On\n type: date\n - name: Listed Price\n type: number\n - name: Last Sold On\n type: date\n - name: Last Sold Price\n type: number\n - name: City\n type: category\n - name: Zip\n type: category\n - name: State\n type: category\noutput_features:\n - name: Sold Price\n type: number\n"
},
{
"path": "ludwig/datasets/configs/camseq.yaml",
"content": "version: 1.0\nname: camseq\nkaggle_dataset_id: carlolepelaars/camseq-semantic-segmentation\narchive_filenames: camseq-semantic-segmentation.zip\nsha256:\n camseq-semantic-segmentation.zip: ea3aeba2661d9b3e3ea406668e7d9240cb2ba0c7e374914bb6d866147faff502\nloader: camseq.CamseqLoader\npreserve_paths:\n - images\n - masks\ndescription: |\n CamSeq01 Cambridge Labeled Objects in Video\n https://www.kaggle.com/datasets/carlolepelaars/camseq-semantic-segmentation\ncolumns:\n - name: image_path\n type: image\n - name: mask_path\n type: image\noutput_features:\n - name: mask_path\n type: image\n"
},
{
"path": "ludwig/datasets/configs/code_alpaca.yaml",
"content": "version: 1.0\nname: code_alpaca\ndownload_urls: https://raw.githubusercontent.com/sahil280114/codealpaca/master/data/code_alpaca_20k.json\ntrain_filenames: code_alpaca_20k.json\nloader: code_alpaca_loader.CodeAlpacaLoader\ndescription: |\n This dataset, created by sahil280114, aims to build and share an instruction-following LLaMA model for code generation. The repo containing\n this dataset is fully based on Stanford Alpaca, and only changes the data used for training.\ncolumns:\n - name: instruction\n type: text\n - name: input\n type: text\n - name: output\n type: text\noutput_features:\n - name: output\n type: text\n"
},
{
"path": "ludwig/datasets/configs/connect4.yaml",
"content": "version: 1.0\nname: connect4\nkaggle_dataset_id: tbrewer/connect-4\narchive_filenames: connect-4.zip\ndataset_filenames: c4_game_database.csv\nsha256:\n connect-4.zip: 46c33c47f2664948a4abe53bafee92a602773f31db615bc8bd239e1f98a3d2cf\ndescription: |\n Each row represents the end results of a Connect-4 game.\n Columns 1-42 are the positions on the grid from left to right, top to bottom. Each element in these columns represent to player's piece : 1, and -1, 0 marks an empty cell.\n Column 43 marks the winner of the game : -1, 1, and 0 for tie games.\ncolumns:\n - name: pos_01\n type: number\n - name: pos_02\n type: number\n - name: pos_03\n type: number\n - name: pos_04\n type: number\n - name: pos_05\n type: number\n - name: pos_06\n type: number\n - name: pos_07\n type: number\n - name: pos_08\n type: number\n - name: pos_09\n type: number\n - name: pos_10\n type: number\n - name: pos_11\n type: number\n - name: pos_12\n type: number\n - name: pos_13\n type: number\n - name: pos_14\n type: number\n - name: pos_15\n type: number\n - name: pos_16\n type: number\n - name: pos_17\n type: number\n - name: pos_18\n type: number\n - name: pos_19\n type: number\n - name: pos_20\n type: number\n - name: pos_21\n type: number\n - name: pos_22\n type: number\n - name: pos_23\n type: number\n - name: pos_24\n type: number\n - name: pos_25\n type: number\n - name: pos_26\n type: number\n - name: pos_27\n type: number\n - name: pos_28\n type: number\n - name: pos_29\n type: number\n - name: pos_30\n type: number\n - name: pos_31\n type: number\n - name: pos_32\n type: number\n - name: pos_33\n type: number\n - name: pos_34\n type: number\n - name: pos_35\n type: number\n - name: pos_36\n type: number\n - name: pos_37\n type: number\n - name: pos_38\n type: number\n - name: pos_39\n type: number\n - name: pos_40\n type: number\n - name: pos_41\n type: number\n - name: pos_42\n type: number\n - name: winner\n type: number\noutput_features:\n - name: winner\n type: category\n"
},
{
"path": "ludwig/datasets/configs/consumer_complaints.yaml",
"content": "version: 1.0\nname: consumer_complaints\nkaggle_dataset_id: selener/consumer-complaint-database\narchive_filenames: consumer-complaint-database.zip\ndataset_filenames: rows.csv\nloader: consumer_complaints_loader.ConsumerComplaintsLoader\ndescription: |\n The dataset contains different information of complaints that customers have made about a multiple products and\n services in the financial sector, such us Credit Reports, Student Loans, Money Transfer, etc. The date of each\n complaint ranges from November 2011 to May 2019.\ncolumns:\n - name: Date received\n type: Date\n - name: Product\n type: text\n - name: Sub-product\n type: text\n - name: Issue\n type: text\n - name: Sub-issue\n type: text\n - name: Consumer complaint narrative\n type: text\n - name: Company public response\n type: text\n - name: Company\n type: text\n - name: State\n type: category\n - name: ZIP code\n type: category\n - name: Tags\n type: category\n - name: Consumer consent provided?\n type: text\n - name: Submitted via\n type: category\n - name: Date sent to company\n type: date\n - name: Company response to consumer\n type: text\n - name: Timely response?\n type: binary\n - name: Consumer disputed?\n type: binary\n - name: Complaint ID\n type: number\noutput_features:\n - name: Issue\n type: text\n"
},
{
"path": "ludwig/datasets/configs/consumer_complaints_generation.yaml",
"content": "version: 1.0\nname: consumer_complaints_generation\ndownload_urls: https://predibase-public-us-west-2.s3.us-west-2.amazonaws.com/datasets/consumer_complaints_gen_tutorial.csv\ntrain_filenames: consumer_complaints_gen_tutorial.csv\ndescription: |\n The dataset contains different information of complaints that customers have made about a multiple products and\n services in the financial sector, such us Credit Reports, Student Loans, Money Transfer, etc. The date of each\n complaint ranges from November 2011 to May 2019. The dataset has been modified to be used for text generation.\n We have added a structured JSON field that contains a company generated response to the raised complaint. The idea\n is to fine-tune an LLM to generate this output JSON field.\ncolumns:\n - name: Complaint ID\n type: number\n - name: Date received\n type: Date\n - name: Product\n type: text\n - name: Issue\n type: text\n - name: Complaint\n type: text\n - name: Company Response\n type: text\n - name: Structured JSON Output\n type: text\noutput_features:\n - name: Structured JSON Output\n type: text\n"
},
{
"path": "ludwig/datasets/configs/creditcard_fraud.yaml",
"content": "version: 1.0\nname: creditcard_fraud\nkaggle_dataset_id: mlg-ulb/creditcardfraud\narchive_filenames: creditcardfraud.zip\nsha256:\n creditcardfraud.zip: a0360ce715992212e9ac72d8ccdca97f4be87dc1fdf2bed011358f7ab409a28a\nloader: creditcard_fraud.CreditCardFraudLoader\ndescription: |\n The Machine Learning Group ULB Dataset\n https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud\ncolumns:\n - name: Time\n type: number\n - name: V1\n type: number\n - name: V2\n type: number\n - name: V3\n type: number\n - name: V4\n type: number\n - name: V5\n type: number\n - name: V6\n type: number\n - name: V7\n type: number\n - name: V8\n type: number\n - name: V9\n type: number\n - name: V10\n type: number\n - name: V11\n type: number\n - name: V12\n type: number\n - name: V13\n type: number\n - name: V14\n type: number\n - name: V15\n type: number\n - name: V16\n type: number\n - name: V17\n type: number\n - name: V18\n type: number\n - name: V19\n type: number\n - name: V20\n type: number\n - name: V21\n type: number\n - name: V22\n type: number\n - name: V23\n type: number\n - name: V24\n type: number\n - name: V25\n type: number\n - name: V26\n type: number\n - name: V27\n type: number\n - name: V28\n type: number\n - name: Amount\n type: number\n - name: Class\n type: number\noutput_features:\n - name: Class\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/customer_churn_prediction.yaml",
"content": "version: 1.0\nname: customer_churn_prediction\nkaggle_competition: customer-churn-prediction-2020\narchive_filenames: customer-churn-prediction-2020.zip\ntrain_filenames: train.csv\ntest_filenames: test.csv\nsha256:\n customer-churn-prediction-2020.zip: fb5cbc787081a6a559592230c657a0520a181447da6eb2adc34a3aebbe8ed9ca\ndescription: |\n Dataset from a Kaggle competition that is about predicting whether a customer will change\n telecommunications provider, something known as \"churning\".\n https://www.kaggle.com/c/customer-churn-prediction-2020\noutput_features:\n - name: churn\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/data_scientist_salary.yaml",
"content": "version: 1.0\nname: data_scientist_salary\ndownload_urls:\n - https://automl-mm-bench.s3.amazonaws.com/machine_hack_competitions/predict_the_data_scientists_salary_in_india_hackathon/train.csv\n - https://automl-mm-bench.s3.amazonaws.com/machine_hack_competitions/predict_the_data_scientists_salary_in_india_hackathon/test.csv\nsha256:\n test.csv: 244c215f4a03cae4b107e76c7fe94269728450cabf44c943415211ce7d6437df\n train.csv: 99d6aa80505ac1311e97f402d5723996119e859c7f3fce261350462148debe3d\ntrain_filenames: train.csv\ntest_filenames: test.csv\ndescription: |\n The training data and test data comprise of 19802 samples and of 6601 samples each from the\n Analytics India Annual Salary Study.\n https://machinehack.com/hackathons/predict_the_data_scientists_salary_in_india_hackathon/overview\ncolumns:\n - name: experience\n type: category\n - name: job_description\n type: text\n - name: job_desig\n type: category\n - name: job_type\n type: category\n - name: key_skills\n type: set\n - name: location\n type: category\n - name: salary\n type: category\noutput_features:\n - name: salary\n type: category\n"
},
{
"path": "ludwig/datasets/configs/dbpedia.yaml",
"content": "version: 1.0\nname: dbpedia\ndownload_urls: https://s3.amazonaws.com/fast-ai-nlp/dbpedia_csv.tgz\ntrain_filenames: dbpedia_csv/train.csv\ntest_filenames: dbpedia_csv/test.csv\nsha256:\n dbpedia_csv.tgz: 42db5221ddedddb673a4cabcc5f3a7d869714c878bcfe4ba94b29d14aa38e417\ndescription: |\n The DBPedia Ontology dataset.\n\n Details:\n 40,000 training samples and 5,000 testing samples from 14\n nonoverlapping classes from DBpedia 2014.\n Dataset source:\n Character-level Convolutional Networks for Text Classification\n Xiang Zhang et al., 2015\ncolumns:\n - name: label\n type: category\n - name: title\n type: category\n - name: content\n type: text\noutput_features:\n - name: label\n type: category\n"
},
{
"path": "ludwig/datasets/configs/electricity.yaml",
"content": "version: 1.0\nname: electricity\ndownload_urls: https://raw.githubusercontent.com/nimz/electricity_demand/master/elecdemand.csv\nsha256:\n elecdemand.csv: 4fd3c8a4b8168f34703b55313c5341f8e8385810a54f1a1cdf6987c1904c9698\ndescription: |\n Electricity demand dataset. Half-hourly electricity demand in Victoria, Australia during 2014, along with\n Melbourne temperatures.\n\n Source textbook:\n Forecasting: Principles and Practice\n Rob J Hyndman and George Athanasopoulos\ncolumns:\n - name: Demand\n type: number\n - name: WorkDay\n type: binary\n - name: Temperature\n type: number\noutput_features:\n - name: Demand\n type: number\n"
},
{
"path": "ludwig/datasets/configs/ethos_binary.yaml",
"content": "version: 1.1\nname: ethos_binary\ndownload_urls:\n - https://raw.githubusercontent.com/intelligence-csd-auth-gr/Ethos-Hate-Speech-Dataset/master/ethos/ethos_data/Ethos_Dataset_Binary.csv\nsha256:\n Ethos_Dataset_Binary.csv: 0cd0050c2592afcb5eca5876df485ca15cda9d7d16fe32c269857260fd10d96c\nloader: ethos_binary.EthosBinaryLoader\ndescription: |\n The Ethos Hate Speech Dataset.\n\n Source Paper:\n ETHOS: an Online Hate Speech Detection Dataset\n Ioannis Mollas and Zoe Chrysopoulou and Stamatis Karlos and\n Grigorios Tsoumakas\ncolumns:\n - name: comment\n type: text\n - name: isHate\n type: binary\noutput_features:\n - name: isHate\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/fake_job_postings2.yaml",
"content": "version: 1.0\nname: fake_job_postings2\ndownload_urls:\n - https://automl-mm-bench.s3.amazonaws.com/fake_job_postings2/train.csv\n - https://automl-mm-bench.s3.amazonaws.com/fake_job_postings2/test.csv\nsha256:\n test.csv: a5296f49129d440434e6274bb892a1320fe1dd4c26d5a1b085786d5ea1133dd8\n train.csv: b6568e415ad49cb7bd23848dfbb8d381f9de590e133a5075abbf4c1a7c7c1711\ntrain_filenames: train.csv\ntest_filenames: test.csv\ndescription: |\n This dataset contains 18K job descriptions out of which about 800 are fake.\n The data consists of both textual information and meta-information about the jobs.\n This dataset is \"fake_job_postings2\" in the AutoGluon paper.\n https://www.kaggle.com/datasets/shivamb/real-or-fake-fake-jobposting-prediction\ncolumns:\n - name: title\n type: category\n - name: salary_range\n type: category\n - name: description\n type: text\n - name: required_experience\n type: category\n - name: required_education\n type: category\n - name: fraudulent\n type: binary\noutput_features:\n - name: fraudulent\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/fever.yaml",
"content": "version: 1.0\nname: fever\ndownload_urls:\n - https://fever.ai/download/fever/train.jsonl\n - https://fever.ai/download/fever/paper_dev.jsonl\n - https://fever.ai/download/fever/paper_test.jsonl\nsha256:\n train.jsonl: eba7e8f87076753f8494718b9a857827af7bf73e76c9e4b75420207d26e588b6\n paper_test.jsonl: fb7b0280a0adc2302bbb29bfb7af37274fa585de3171bcf908f180642d11d88e\n paper_dev.jsonl: 41158707810008747946bf23471e82df53e77a513524b9e3ec1c2e674ef5ef8c\ntrain_filenames: train.jsonl\ntest_filenames: paper_test.jsonl\nvalidation_filenames: paper_dev.jsonl\ncolumn_types:\n evidence: str\ndescription: |\n FEVER: a Large-scale Dataset for Fact Extraction and VERification\ncolumns:\n - name: id\n type: category\n - name: verifiable\n type: category\n - name: label\n type: category\n - name: label\n type: category\n - name: claim\n type: text\n - name: evidence\n type: category\n - name: label\n type: category\noutput_features:\n - name: label\n type: category\n"
},
{
"path": "ludwig/datasets/configs/flickr8k.yaml",
"content": "version: 1.0\nname: flickr8k\ndownload_urls:\n - https://github.com/jbrownlee/Datasets/releases/download/Flickr8k/Flickr8k_Dataset.zip\n - https://github.com/jbrownlee/Datasets/releases/download/Flickr8k/Flickr8k_text.zip\ndataset_filenames: flickr8k_dataset.csv\npreserve_paths: Flicker8k_Dataset\nsha256:\n Flickr8k_Dataset.zip: 61e4b111d32b24a55b69dafd91f4c3aec07391b7b9217face15dd35d517fe6de\n Flickr8k_text.zip: 4992ddc8110e9aa49da5bf698522b0c8f11c448814a488584ee6bf040e5137e7\nloader: flickr8k.Flickr8kLoader\ndescription: |\n A new benchmark collection for sentence-based image description and search,\n consisting of 8,000 images that are each paired with five different\n captions which provide clear descriptions of the salient entities and\n events. The images were chosen from six different Flickr groups, and tend\n not to contain any well-known people or locations, but were manually\n selected to depict a variety of scenes and situations.\noutput_features:\n - name: caption0\n type: text\n - name: caption1\n type: text\n - name: caption2\n type: text\n - name: caption3\n type: text\n - name: caption4\n type: text\n"
},
{
"path": "ludwig/datasets/configs/forest_cover.yaml",
"content": "version: 1.0\nname: forest_cover\ndownload_urls: https://archive.ics.uci.edu/ml/machine-learning-databases/covtype/covtype.data.gz\nsha256:\n covtype.data.gz: 614360d0257557dd1792834a85a1cdebfadc3c4f30b011d56afee7ffb5b15771\ndataset_filenames: covtype.data\nloader: forest_cover.ForestCoverLoader\ndescription: |\n The Forest Cover Type dataset.\n Predicting forest cover type from cartographic variables only.\n https://archive.ics.uci.edu/ml/datasets/covertype\ncolumns:\n - name: Elevation\n type: number\n - name: Aspect\n type: number\n - name: Slope\n type: number\n - name: Horizontal_Distance_To_Hydrology\n type: number\n - name: Vertical_Distance_To_Hydrology\n type: number\n - name: Horizontal_Distance_To_Roadways\n type: number\n - name: Hillshade_9am\n type: number\n - name: Hillshade_Noon\n type: number\n - name: Hillshade_3pm\n type: number\n - name: Horizontal_Distance_To_Fire_Points\n type: number\n - name: Wilderness_Area\n type: category\n - name: Soil_Type\n type: category\n - name: Cover_Type\n type: category\noutput_features:\n - name: Cover_Type\n type: category\n"
},
{
"path": "ludwig/datasets/configs/goemotions.yaml",
"content": "version: 1.0\nname: goemotions\ndownload_urls:\n - https://raw.githubusercontent.com/google-research/google-research/master/goemotions/data/train.tsv\n - https://raw.githubusercontent.com/google-research/google-research/master/goemotions/data/dev.tsv\n - https://raw.githubusercontent.com/google-research/google-research/master/goemotions/data/test.tsv\ntrain_filenames: train.tsv\nvalidation_filenames: dev.tsv\ntest_filenames: test.tsv\nsha256:\n train.tsv: 1c254a142be5c00e80d819b9ae1bbd36d94b2eeb8f4b1271846508d57e57d9c5\n dev.tsv: 575489c079c9de1097062a01738f998590d6b7ead66dd1c9fd1d2ba01fd8bc62\n test.tsv: 0587b2dd8b27b97352adbfc3fb083d46005c8946657fdc2b1ca8b1cc7f1f8be4\nloader: goemotions.GoEmotionsLoader\ndescription: |\n GoEmotions: A Dataset for Fine-Grained Emotion Classification.\n https://ai.googleblog.com/2021/10/goemotions-dataset-for-fine-grained.html\ncolumns:\n - name: text\n type: text\n - name: emotion_ids\n type: category\n - name: comment_id\n type: category\noutput_features:\n - name: emotion_ids\n type: category\n"
},
{
"path": "ludwig/datasets/configs/goodbooks_books.yaml",
"content": "version: 1.0\nname: goodbooks_books\ndownload_urls:\n - https://github.com/zygmuntz/goodbooks-10k/releases/download/v1.0/goodbooks-10k.zip\nsha256:\n goodbooks-10k.zip: 261b97b56db61f3fb2ce5aadbb13704d30179fcc986c17ace665a0af9ed00731\ndataset_filenames: books.csv\ndescription: |\n goodbooks_books is a multimodal dataset of 10K books, taken from the goodreads dataset.\n The Goodbooks-10K dataset contains six million ratings for ten thousand most popular (with most ratings) books.\n The dataset also contains:\n books marked to read by the users\n book metadata (author, year, etc.)\n tags/shelves/genres\n https://github.com/zygmuntz/goodbooks-10k\ncolumns:\n - name: book_id\n type: category\n - name: goodreads_book_id\n type: category\n - name: best_book_id\n type: category\n - name: work_id\n type: category\n - name: books_count\n type: number\n - name: isbn\n type: category\n - name: isbn13\n type: category\n - name: authors\n type: category\n - name: original_publication_year\n type: category\n - name: original_title\n type: category\n - name: title\n type: category\n - name: language_code\n type: category\n - name: average_rating\n type: number\n - name: ratings_count\n type: number\n - name: work_ratings_count\n type: number\n - name: work_text_reviews_count\n type: number\n - name: ratings_1\n type: number\n - name: ratings_2\n type: number\n - name: ratings_3\n type: number\n - name: ratings_4\n type: number\n - name: ratings_5\n type: number\n - name: image_url\n type: image\n - name: small_image_url\n type: image\noutput_features:\n - name: average_rating\n type: number\n - name: ratings_1\n type: number\n - name: ratings_2\n type: number\n - name: ratings_3\n type: number\n - name: ratings_4\n type: number\n - name: ratings_5\n type: number\n"
},
{
"path": "ludwig/datasets/configs/google_qa_answer_type_reason_explanation.yaml",
"content": "version: 1.0\nname: google_qa_answer_type_reason_explanation\ndownload_urls:\n - https://automl-mm-bench.s3.amazonaws.com/google_quest_qa/train.pq\n - https://automl-mm-bench.s3.amazonaws.com/google_quest_qa/dev.pq\nsha256:\n train.pq: 92274286ffb759c96bfca77001c10eb323b3531db3a0e178813db9b82e80a12a\n dev.pq: 2e66450215b94dc404eadc7dde83a1eabad9640d946863c298aa2d42c998ed84\ntrain_filenames: train.pq\ntest_filenames: dev.pq\ndescription: |\n Google QUEST Q&A Labeling\n Improving automated understanding of complex question answer content.\n The data for this competition includes questions and answers from various StackExchange properties.\n https://www.kaggle.com/c/google-quest-challenge/data\n Note: this is the same dataset as `google_quest_qa`. It is duplicated here to have a one-to-one mapping\n with the benchmarking datasets in https://arxiv.org/pdf/2111.02705.pdf\n In this paper, the column `answer_type_reason_explanation` is used as the output feature.\ncolumns:\n - name: qa_id\n type: category\n - name: question_title\n type: text\n - name: question_body\n type: text\n - name: question_user_name\n type: category\n - name: question_user_page\n type: category\n - name: answer\n type: text\n - name: answer_user_name\n type: category\n - name: answer_user_page\n type: category\n - name: url\n type: category\n - name: category\n type: category\n - name: host\n type: category\n - name: question_asker_intent_understanding\n type: number\n - name: question_body_critical\n type: number\n - name: question_conversational\n type: number\n - name: question_expect_short_answer\n type: number\n - name: question_fact_seeking\n type: number\n - name: question_has_commonly_accepted_answer\n type: number\n - name: question_interestingness_others\n type: number\n - name: question_interestingness_self\n type: number\n - name: question_multi_intent\n type: number\n - name: question_not_really_a_question\n type: number\n - name: question_opinion_seeking\n type: number\n - name: question_type_choice\n type: number\n - name: question_type_compare\n type: number\n - name: question_type_consequence\n type: number\n - name: question_type_definition\n type: number\n - name: question_type_entity\n type: number\n - name: question_type_instructions\n type: number\n - name: question_type_procedure\n type: number\n - name: question_type_reason_explanation\n type: number\n - name: question_type_spelling\n type: number\n - name: question_well_written\n type: number\n - name: answer_helpful\n type: number\n - name: answer_level_of_information\n type: number\n - name: answer_plausible\n type: number\n - name: answer_relevance\n type: number\n - name: answer_satisfaction\n type: number\n - name: answer_type_instructions\n type: number\n - name: answer_type_procedure\n type: number\n - name: answer_type_reason_explanation\n type: number\n - name: answer_well_written\n type: number\noutput_features:\n - name: answer_type_reason_explanation\n type: number\n"
},
{
"path": "ludwig/datasets/configs/google_qa_question_type_reason_explanation.yaml",
"content": "version: 1.0\nname: google_qa_question_type_reason_explanation\ndownload_urls:\n - https://automl-mm-bench.s3.amazonaws.com/google_quest_qa/train.pq\n - https://automl-mm-bench.s3.amazonaws.com/google_quest_qa/dev.pq\nsha256:\n train.pq: 92274286ffb759c96bfca77001c10eb323b3531db3a0e178813db9b82e80a12a\n dev.pq: 2e66450215b94dc404eadc7dde83a1eabad9640d946863c298aa2d42c998ed84\ntrain_filenames: train.pq\ntest_filenames: dev.pq\ndescription: |\n Google QUEST Q&A Labeling\n Improving automated understanding of complex question answer content.\n The data for this competition includes questions and answers from various StackExchange properties.\n https://www.kaggle.com/c/google-quest-challenge/data\n Note: this is the same dataset as `google_quest_qa`. It is duplicated here to have a one-to-one mapping\n with the benchmarking datasets in https://arxiv.org/pdf/2111.02705.pdf\n In this paper, the column `question_type_reason_explanation` is used as the output feature.\ncolumns:\n - name: qa_id\n type: category\n - name: question_title\n type: text\n - name: question_body\n type: text\n - name: question_user_name\n type: category\n - name: question_user_page\n type: category\n - name: answer\n type: text\n - name: answer_user_name\n type: category\n - name: answer_user_page\n type: category\n - name: url\n type: category\n - name: category\n type: category\n - name: host\n type: category\n - name: question_asker_intent_understanding\n type: number\n - name: question_body_critical\n type: number\n - name: question_conversational\n type: number\n - name: question_expect_short_answer\n type: number\n - name: question_fact_seeking\n type: number\n - name: question_has_commonly_accepted_answer\n type: number\n - name: question_interestingness_others\n type: number\n - name: question_interestingness_self\n type: number\n - name: question_multi_intent\n type: number\n - name: question_not_really_a_question\n type: number\n - name: question_opinion_seeking\n type: number\n - name: question_type_choice\n type: number\n - name: question_type_compare\n type: number\n - name: question_type_consequence\n type: number\n - name: question_type_definition\n type: number\n - name: question_type_entity\n type: number\n - name: question_type_instructions\n type: number\n - name: question_type_procedure\n type: number\n - name: question_type_reason_explanation\n type: number\n - name: question_type_spelling\n type: number\n - name: question_well_written\n type: number\n - name: answer_helpful\n type: number\n - name: answer_level_of_information\n type: number\n - name: answer_plausible\n type: number\n - name: answer_relevance\n type: number\n - name: answer_satisfaction\n type: number\n - name: answer_type_instructions\n type: number\n - name: answer_type_procedure\n type: number\n - name: answer_type_reason_explanation\n type: number\n - name: answer_well_written\n type: number\noutput_features:\n - name: question_type_reason_explanation\n type: number\n"
},
{
"path": "ludwig/datasets/configs/google_quest_qa.yaml",
"content": "version: 1.0\nname: google_quest_qa\ndownload_urls:\n - https://automl-mm-bench.s3.amazonaws.com/google_quest_qa/train.pq\n - https://automl-mm-bench.s3.amazonaws.com/google_quest_qa/dev.pq\n - https://automl-mm-bench.s3.amazonaws.com/google_quest_qa/test.pq\nsha256:\n test.pq: cb1bb5f32374d83ad4ef7feb4e443c9376cdd919cda40057732ef500e9a4ecf3\n train.pq: 92274286ffb759c96bfca77001c10eb323b3531db3a0e178813db9b82e80a12a\n dev.pq: 2e66450215b94dc404eadc7dde83a1eabad9640d946863c298aa2d42c998ed84\ntrain_filenames: train.pq\nvalidation_filenames: dev.pq\ntest_filenames: test.pq\ndescription: |\n Google QUEST Q&A Labeling\n Improving automated understanding of complex question answer content.\n The data for this competition includes questions and answers from various StackExchange properties.\n https://www.kaggle.com/c/google-quest-challenge/data\ncolumns:\n - name: qa_id\n type: category\n - name: question_title\n type: text\n - name: question_body\n type: text\n - name: question_user_name\n type: category\n - name: question_user_page\n type: category\n - name: answer\n type: text\n - name: answer_user_name\n type: category\n - name: answer_user_page\n type: category\n - name: url\n type: category\n - name: category\n type: category\n - name: host\n type: category\n - name: question_asker_intent_understanding\n type: number\n - name: question_body_critical\n type: number\n - name: question_conversational\n type: number\n - name: question_expect_short_answer\n type: number\n - name: question_fact_seeking\n type: number\n - name: question_has_commonly_accepted_answer\n type: number\n - name: question_interestingness_others\n type: number\n - name: question_interestingness_self\n type: number\n - name: question_multi_intent\n type: number\n - name: question_not_really_a_question\n type: number\n - name: question_opinion_seeking\n type: number\n - name: question_type_choice\n type: number\n - name: question_type_compare\n type: number\n - name: question_type_consequence\n type: number\n - name: question_type_definition\n type: number\n - name: question_type_entity\n type: number\n - name: question_type_instructions\n type: number\n - name: question_type_procedure\n type: number\n - name: question_type_reason_explanation\n type: number\n - name: question_type_spelling\n type: number\n - name: question_well_written\n type: number\n - name: answer_helpful\n type: number\n - name: answer_level_of_information\n type: number\n - name: answer_plausible\n type: number\n - name: answer_relevance\n type: number\n - name: answer_satisfaction\n type: number\n - name: answer_type_instructions\n type: number\n - name: answer_type_procedure\n type: number\n - name: answer_type_reason_explanation\n type: number\n - name: answer_well_written\n type: number\noutput_features:\n - name: question_type_reason_explanation\n type: category\n"
},
{
"path": "ludwig/datasets/configs/higgs.yaml",
"content": "version: 1.0\nname: higgs\ndownload_urls: https://archive.ics.uci.edu/ml/machine-learning-databases/00280/HIGGS.csv.gz\nsha256:\n HIGGS.csv.gz: ea302c18164d4e3d916a1e2e83a9a8d07069fa6ebc7771e4c0540d54e593b698\ncolumn_types:\n label: int32\nloader: higgs.HiggsLoader\ndescription: |\n The Higgs Boson dataset.\n\n This is a classification problem to distinguish between a signal process\n which produces Higgs bosons and a background process which does not.\n\n https://archive.ics.uci.edu/ml/datasets/HIGGS\ncolumns:\n - name: label\n type: binary\n - name: lepton_pT\n type: number\n - name: lepton_eta\n type: number\n - name: lepton_phi\n type: number\n - name: missing_energy_magnitude\n type: number\n - name: missing_energy_phi\n type: number\n - name: jet_1_pt\n type: number\n - name: jet_1_eta\n type: number\n - name: jet_1_phi\n type: number\n - name: jet_1_b-tag\n type: number\n - name: jet_2_pt\n type: number\n - name: jet_2_eta\n type: number\n - name: jet_2_phi\n type: number\n - name: jet_2_b-tag\n type: number\n - name: jet_3_pt\n type: number\n - name: jet_3_eta\n type: number\n - name: jet_3_phi\n type: number\n - name: jet_3_b-tag\n type: number\n - name: jet_4_pt\n type: number\n - name: jet_4_eta\n type: number\n - name: jet_4_phi\n type: number\n - name: jet_4_b-tag\n type: number\n - name: m_jj\n type: number\n - name: m_jjj\n type: number\n - name: m_lv\n type: number\n - name: m_jlv\n type: number\n - name: m_bb\n type: number\n - name: m_wbb\n type: number\n - name: m_wwbb\n type: number\noutput_features:\n - name: label\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/hugging_face.yaml",
"content": "version: 1.0\nname: hugging_face\nloader: hugging_face.HFLoader\ndescription: |\n Hugging Face Datasets\n"
},
{
"path": "ludwig/datasets/configs/ieee_fraud.yaml",
"content": "version: 1.0\nname: ieee_fraud\nkaggle_competition: ieee-fraud-detection\narchive_filenames: ieee-fraud-detection.zip\nsha256:\n ieee-fraud-detection.zip: 4cc646da09d0a9b265983ffed775b1f9ee15af5266586df610e04d6adae0b829\ntrain_filenames:\n - train_identity.csv\n - train_transaction.csv\ntest_filenames:\n - test_identity.csv\n - test_transaction.csv\nloader: ieee_fraud.IEEEFraudLoader\ndescription: |\n The IEEE-CIS Fraud Detection Dataset\n https://www.kaggle.com/c/ieee-fraud-detection/overview.\noutput_features:\n - name: isFraud\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/imbalanced_insurance.yaml",
"content": "version: 1.0\nname: imbalaced_insurance\nkaggle_dataset_id: arashnic/imbalanced-data-practice\narchive_filenames: imbalanced-data-practice.zip\nsha256:\n imbalanced-data-practice.zip: 33c7d15cbdb7cc151c1d5e920a8a613b015c19222f90d4eac04ca8cfc5416847\ndataset_filenames: aug_train.csv\nloader: split_loaders.RandomSplitLoader\ndescription: |\n Health Insurance Cross Sell Prediction\n Predict Health Insurance Owners' who will be interested in Vehicle Insurance\n https://www.kaggle.com/datasets/arashnic/imbalanced-data-practice\ncolumns:\n - name: id\n type: category\n - name: Gender\n type: binary\n - name: Age\n type: number\n - name: Driving_License\n type: binary\n - name: Region_Code\n type: category\n - name: Previously_Insured\n type: binary\n - name: Vehicle_Age\n type: category\n - name: Vehicle_Damage\n type: binary\n - name: Annual_Premium\n type: number\n - name: Policy_Sales_Channel\n type:\n - name: Vintage\n type:\n - name: Response\n type:\noutput_features:\n - name: Response\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/imdb.yaml",
"content": "version: 1.0\nname: imdb\nkaggle_dataset_id: lakshmi25npathi/imdb-dataset-of-50k-movie-reviews\narchive_filenames: imdb-dataset-of-50k-movie-reviews.zip\nsha256:\n imdb-dataset-of-50k-movie-reviews.zip: 73a235bc5fc4df57bb5d517afa480fe6bfd4e2afc25dc5e5867fc87f2d25614d\ndescription: |\n IMDB dataset having 50K movie reviews for natural language processing or Text analytics.\n https://www.kaggle.com/datasets/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews\ncolumns:\n - name: review\n type: text\n - name: sentiment\n type: category\noutput_features:\n - name: sentiment\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/imdb_genre_prediction.yaml",
"content": "version: 1.0\nname: imdb_genre_prediction\ndownload_urls:\n - https://automl-mm-bench.s3.amazonaws.com/imdb_genre_prediction/train.csv\n - https://automl-mm-bench.s3.amazonaws.com/imdb_genre_prediction/test.csv\nsha256:\n test.csv: 5bca7b6ca34f4057e2a4920d6034f481055bd03061bb0128c87d6c99a6b4661f\n train.csv: b63f1f6fcad17f644d9266891a01d0f0e1187c277ccf6eecb80af72b92b0b621\ntrain_filenames: train.csv\ntest_filenames: test.csv\ndescription: |\n A data set of 1,000 most popular movies on IMDB in the last 10 years. The data points included are:\n Title, Genre, Description, Director, Actors, Year, Runtime, Rating, Votes, Revenue, Metascrore\n https://www.kaggle.com/PromptCloudHQ/imdb-data\ncolumns:\n - name: Rank\n type: number\n - name: Title\n type: category\n - name: Description\n type: text\n - name: Director\n type: category\n - name: Actors\n type: set\n - name: Year\n type: category\n - name: Runtime (Minutes)\n type: number\n - name: Rating\n type: Number\n - name: Votes\n type: number\n - name: Revenue (Millions)\n type: number\n - name: Metascore\n type: number\n - name: Genre_is_Drama\n type: binary\noutput_features:\n- name: Genre_is_Drama\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/insurance_lite.yaml",
"content": "version: 1.0\nname: insurance_lite\nkaggle_dataset_id: infernape/fast-furious-and-insured\narchive_filenames: fast-furious-and-insured.zip\nsha256:\n fast-furious-and-insured.zip: 3b88ada517aa88d9c9187121d7ef42f4b5539808677a2b0827b989ca0fa19600\ndataset_filenames: Fast_Furious_Insured/train.csv\npreserve_paths: Fast_Furious_Insured\nloader: insurance_lite.InsuranceLiteLoader\ndescription: |\n The dataset consists of parameters such as the images of damaged cars,\n the price of the cars and their insurance claim, and the like.\n Predict the insurance claim for the cars that are provided in the dataset.\ncolumns:\n - name: image_path\n type: image\n - name: insurance_company\n type: category\n - name: cost_of_vehicle\n type: number\n - name: min_coverage\n type: number\n - name: expiry_date\n type: date\n - name: max_coverage\n type: number\n - name: condition\n type: binary\n - name: amount\n type: number\noutput_features:\n - name: amount\n type: number\n"
},
{
"path": "ludwig/datasets/configs/iris.yaml",
"content": "version: 1.0\nname: iris\ndownload_urls: https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data\nsha256:\n iris.data: 6f608b71a7317216319b4d27b4d9bc84e6abd734eda7872b71a458569e2656c0\ndescription: |\n Iris Dataset\n https://archive.ics.uci.edu/ml/datasets/Iris\ncolumns:\n - name: sepal_length_cm\n type: number\n - name: sepal_width_cm\n type: number\n - name: petal_length_cm\n type: number\n - name: petal_width_cm\n type: number\n - name: class\n type: category\noutput_features:\n - name: class\n type: category\n"
},
{
"path": "ludwig/datasets/configs/irony.yaml",
"content": "version: 1.0\nname: irony\ndownload_urls: https://raw.githubusercontent.com/bwallace/ACL-2014-irony/master/irony-labeled.csv\nsha256:\n irony-labeled.csv: 11f4d0964bd9c5c8363de2920612f5d926a4e6b3a8ab9187da2c33cfc0fdd02b\ndescription: |\n The Reddit Irony dataset.\n Source Paper: Humans Require Context to Infer Ironic Intent (so Computers Probably do, too)\n Byron C Wallace, Do Kook Choe, Laura Kertz, and Eugene Charniak\ncolumns:\n - name: comment_text\n type: text\n - name: label\n type: binary\noutput_features:\n - name: label\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/jc_penney_products.yaml",
"content": "version: 1.0\nname: jc_penney_products\ndownload_urls:\n - https://automl-mm-bench.s3.amazonaws.com/jc_penney_products/train.csv\n - https://automl-mm-bench.s3.amazonaws.com/jc_penney_products/test.csv\nsha256:\n test.csv: 458fb13b07701897fbc0d88481823b90e884e92a42e65eeba816cdf3523b2e85\n train.csv: e9e3d3da627dc544d01f4c27b1d023288c68e55ce2db2593fb7b2268a6b9b020\ntrain_filenames: train.csv\ntest_filenames: test.csv\ndescription: |\n JCPenney products\n 20,000 product listings from JCPenney\n https://www.kaggle.com/PromptCloudHQ/all-jc-penny-products\ncolumns:\n - name: name_title\n type: category\n - name: description\n type: text\n - name: sale_price\n type: number\n - name: average_product_rating\n type: number\n - name: brand\n type: category\n - name: total_number_reviews\n type: number\noutput_features:\n- name: sale_price\n type: number\n"
},
{
"path": "ludwig/datasets/configs/jigsaw_unintended_bias.yaml",
"content": "version: 1.0\nname: jigsaw_unintended_bias\ndownload_urls:\n - https://automl-mm-bench.s3.amazonaws.com/jigsaw_unintended_bias/train.pq\n - https://automl-mm-bench.s3.amazonaws.com/jigsaw_unintended_bias/dev.pq\n - https://automl-mm-bench.s3.amazonaws.com/jigsaw_unintended_bias/test.pq\nsha256:\n test.pq: e9f3fd6fa83ddea2af8d21e93eb677b2fa5686c9b8ae38e6293f7c3306f66fad\n train.pq: 30bedd5bbd5b2277b8bffa4ed3a02ce6ef7c838aa5c1338908b5ad599a6a9888\n dev.pq: 57e1e3a06733fb83ad9ca46839ed8afd7d670e5e5f5c7f0026b748d760457d57\ntrain_filenames: train.pq\nvalidation_filenames: dev.pq\ntest_filenames: test.pq\ndescription: |\n A dataset labeled for identity mentions and optimizing a metric designed to measure unintended bias.\n Disclaimer: The dataset for this competition contains text that may be considered profane, vulgar, or offensive.\n https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification\ncolumns:\n - name: id\n type: category\n - name: target\n type: binary\n - name: comment_text\n type: text\n - name: severe_toxicity\n type: number\n - name: obscene\n type: number\n - name: identity_attack\n type: number\n - name: insult\n type: number\n - name: threat\n type: number\n - name: asian\n type: number\n - name: atheist\n type: number\n - name: bisexual\n type: number\n - name: black\n type: number\n - name: buddhist\n type: number\n - name: christian\n type: number\n - name: female\n type: number\n - name: heterosexual\n type: number\n - name: hindu\n type: number\n - name: homosexual_gay_or_lesbian\n type: number\n - name: intellectual_or_learning_disability\n type: number\n - name: jewish\n type: number\n - name: latino\n type: number\n - name: male\n type: number\n - name: muslim\n type: number\n - name: other_disability\n type: number\n - name: other_gender\n type: number\n - name: other_race_or_ethnicity\n type: number\n - name: other_religion\n type: number\n - name: other_sexual_orientation\n type: number\n - name: physical_disability\n type: number\n - name: psychiatric_or_mental_illness\n type: number\n - name: transgender\n type: number\n - name: white\n type: number\n - name: created_date\n type: date\n - name: publication_id\n type: category\n - name: parent_id\n type: category\n - name: article_id\n type: category\n - name: rating\n type: category\n - name: funny\n type: number\n - name: wow\n type: number\n - name: sad\n type: number\n - name: likes\n type: number\n - name: disagree\n type: number\n - name: sexual_explicit\n type: number\n - name: identity_annotator_count\n type: number\n - name: toxicity_annotator_count\n type: number\noutput_features:\n- name: target\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/jigsaw_unintended_bias100k.yaml",
"content": "version: 1.0\nname: jigsaw_unintended_bias100K\ndownload_urls:\n - https://automl-mm-bench.s3.amazonaws.com/jigsaw_unintended_bias100K/train.pq\n - https://automl-mm-bench.s3.amazonaws.com/jigsaw_unintended_bias100K/test.pq\nsha256:\n test.pq: f7a0ec60ac89ffdb94919bf95e514057588a444c90ebdcb8ac90dfb0bfec3d48\n train.pq: 48916c037b0a20167f6e9176cc1eedcb0e6ef942beeedb7dc02f19dfebac0229\ntrain_filenames: train.pq\ntest_filenames: test.pq\ndescription: |\n A dataset labeled for identity mentions and optimizing a metric designed to measure unintended bias.\n Disclaimer: The dataset for this competition contains text that may be considered profane, vulgar, or offensive.\n https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification\ncolumns:\n - name: id\n type: category\n - name: target\n type: binary\n - name: comment_text\n type: text\n - name: severe_toxicity\n type: number\n - name: obscene\n type: number\n - name: identity_attack\n type: number\n - name: insult\n type: number\n - name: threat\n type: number\n - name: asian\n type: number\n - name: atheist\n type: number\n - name: bisexual\n type: number\n - name: black\n type: number\n - name: buddhist\n type: number\n - name: christian\n type: number\n - name: female\n type: number\n - name: heterosexual\n type: number\n - name: hindu\n type: number\n - name: homosexual_gay_or_lesbian\n type: number\n - name: intellectual_or_learning_disability\n type: number\n - name: jewish\n type: number\n - name: latino\n type: number\n - name: male\n type: number\n - name: muslim\n type: number\n - name: other_disability\n type: number\n - name: other_gender\n type: number\n - name: other_race_or_ethnicity\n type: number\n - name: other_religion\n type: number\n - name: other_sexual_orientation\n type: number\n - name: physical_disability\n type: number\n - name: psychiatric_or_mental_illness\n type: number\n - name: transgender\n type: number\n - name: white\n type: number\n - name: created_date\n type: date\n - name: publication_id\n type: category\n - name: parent_id\n type: category\n - name: article_id\n type: category\n - name: rating\n type: category\n - name: funny\n type: number\n - name: wow\n type: number\n - name: sad\n type: number\n - name: likes\n type: number\n - name: disagree\n type: number\n - name: sexual_explicit\n type: number\n - name: identity_annotator_count\n type: number\n - name: toxicity_annotator_count\n type: number\noutput_features:\n- name: target\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/kdd_appetency.yaml",
"content": "version: 1.0\nname: kdd_appetency\ndownload_urls:\n - https://kdd.org/cupfiles/KDDCupData/2009/orange_small_train.data.zip\n - https://kdd.org/cupfiles/KDDCupData/2009/orange_small_test.data.zip\n - https://kdd.org/cupfiles/KDDCupData/2009/orange_small_train_appetency.labels\n - https://raw.githubusercontent.com/catboost/benchmarks/master/quality_benchmarks/prepare_appetency_churn_upselling/appetency/stratified_train_idx_appetency.txt\n - https://raw.githubusercontent.com/catboost/benchmarks/master/quality_benchmarks/prepare_appetency_churn_upselling/appetency/stratified_test_idx_appetency.txt\nsha256:\n orange_small_test.data.zip: 440ac8a350144c14f4d6947c096ad675ee84aa27b4b742071662696e333cec53\n orange_small_train.data.zip: 31ccb810bdbb71c16e079326443166dc3dfbf73cd358fc4a4ce7440fb1bc6040\n orange_small_train_appetency.labels: edbfa40e7513804cf25c3f8b3c8f4a6cf5c77116cffc2f87ef770351250a963c\n stratified_train_idx_appetency.txt: 9c6bf7da6209653e13d9a1d2ef90e4afafe0ecac0eb843c8025816a445c625d9\n stratified_test_idx_appetency.txt: b80fb8dcf43cd028f4b8affeab65299d580a7e5432ebbe639527dc8177f8764a\ndataset_filenames: orange_small_train.data\nloader: kdd_loader.KDDAppetencyLoader\ndescription: |\n The KDD Cup 2009 Appetency dataset.\n https://www.kdd.org/kdd-cup/view/kdd-cup-2009/Data\ncolumns:\n - name: Var1\n type: number\n - name: Var2\n type: number\n - name: Var3\n type: number\n - name: Var4\n type: number\n - name: Var5\n type: number\n - name: Var6\n type: number\n - name: Var7\n type: number\n - name: Var8\n type: number\n - name: Var9\n type: number\n - name: Var10\n type: number\n - name: Var11\n type: number\n - name: Var12\n type: number\n - name: Var13\n type: number\n - name: Var14\n type: number\n - name: Var15\n type: number\n - name: Var16\n type: number\n - name: Var17\n type: number\n - name: Var18\n type: number\n - name: Var19\n type: number\n - name: Var20\n type: number\n - name: Var21\n type: number\n - name: Var22\n type: number\n - name: Var23\n type: number\n - name: Var24\n type: number\n - name: Var25\n type: number\n - name: Var26\n type: number\n - name: Var27\n type: number\n - name: Var28\n type: number\n - name: Var29\n type: number\n - name: Var30\n type: number\n - name: Var31\n type: number\n - name: Var32\n type: number\n - name: Var33\n type: number\n - name: Var34\n type: number\n - name: Var35\n type: number\n - name: Var36\n type: number\n - name: Var37\n type: number\n - name: Var38\n type: number\n - name: Var39\n type: number\n - name: Var40\n type: number\n - name: Var41\n type: number\n - name: Var42\n type: number\n - name: Var43\n type: number\n - name: Var44\n type: number\n - name: Var45\n type: number\n - name: Var46\n type: number\n - name: Var47\n type: number\n - name: Var48\n type: number\n - name: Var49\n type: number\n - name: Var50\n type: number\n - name: Var51\n type: number\n - name: Var52\n type: number\n - name: Var53\n type: number\n - name: Var54\n type: number\n - name: Var55\n type: number\n - name: Var56\n type: number\n - name: Var57\n type: number\n - name: Var58\n type: number\n - name: Var59\n type: number\n - name: Var60\n type: number\n - name: Var61\n type: number\n - name: Var62\n type: number\n - name: Var63\n type: number\n - name: Var64\n type: number\n - name: Var65\n type: number\n - name: Var66\n type: number\n - name: Var67\n type: number\n - name: Var68\n type: number\n - name: Var69\n type: number\n - name: Var70\n type: number\n - name: Var71\n type: number\n - name: Var72\n type: number\n - name: Var73\n type: number\n - name: Var74\n type: number\n - name: Var75\n type: number\n - name: Var76\n type: number\n - name: Var77\n type: number\n - name: Var78\n type: number\n - name: Var79\n type: number\n - name: Var80\n type: number\n - name: Var81\n type: number\n - name: Var82\n type: number\n - name: Var83\n type: number\n - name: Var84\n type: number\n - name: Var85\n type: number\n - name: Var86\n type: number\n - name: Var87\n type: number\n - name: Var88\n type: number\n - name: Var89\n type: number\n - name: Var90\n type: number\n - name: Var91\n type: number\n - name: Var92\n type: number\n - name: Var93\n type: number\n - name: Var94\n type: number\n - name: Var95\n type: number\n - name: Var96\n type: number\n - name: Var97\n type: number\n - name: Var98\n type: number\n - name: Var99\n type: number\n - name: Var100\n type: number\n - name: Var101\n type: number\n - name: Var102\n type: number\n - name: Var103\n type: number\n - name: Var104\n type: number\n - name: Var105\n type: number\n - name: Var106\n type: number\n - name: Var107\n type: number\n - name: Var108\n type: number\n - name: Var109\n type: number\n - name: Var110\n type: number\n - name: Var111\n type: number\n - name: Var112\n type: number\n - name: Var113\n type: number\n - name: Var114\n type: number\n - name: Var115\n type: number\n - name: Var116\n type: number\n - name: Var117\n type: number\n - name: Var118\n type: number\n - name: Var119\n type: number\n - name: Var120\n type: number\n - name: Var121\n type: number\n - name: Var122\n type: number\n - name: Var123\n type: number\n - name: Var124\n type: number\n - name: Var125\n type: number\n - name: Var126\n type: number\n - name: Var127\n type: number\n - name: Var128\n type: number\n - name: Var129\n type: number\n - name: Var130\n type: number\n - name: Var131\n type: number\n - name: Var132\n type: number\n - name: Var133\n type: number\n - name: Var134\n type: number\n - name: Var135\n type: number\n - name: Var136\n type: number\n - name: Var137\n type: number\n - name: Var138\n type: number\n - name: Var139\n type: number\n - name: Var140\n type: number\n - name: Var141\n type: number\n - name: Var142\n type: number\n - name: Var143\n type: number\n - name: Var144\n type: number\n - name: Var145\n type: number\n - name: Var146\n type: number\n - name: Var147\n type: number\n - name: Var148\n type: number\n - name: Var149\n type: number\n - name: Var150\n type: number\n - name: Var151\n type: number\n - name: Var152\n type: number\n - name: Var153\n type: number\n - name: Var154\n type: number\n - name: Var155\n type: number\n - name: Var156\n type: number\n - name: Var157\n type: number\n - name: Var158\n type: number\n - name: Var159\n type: number\n - name: Var160\n type: number\n - name: Var161\n type: number\n - name: Var162\n type: number\n - name: Var163\n type: number\n - name: Var164\n type: number\n - name: Var165\n type: number\n - name: Var166\n type: number\n - name: Var167\n type: number\n - name: Var168\n type: number\n - name: Var169\n type: number\n - name: Var170\n type: number\n - name: Var171\n type: number\n - name: Var172\n type: number\n - name: Var173\n type: number\n - name: Var174\n type: number\n - name: Var175\n type: number\n - name: Var176\n type: number\n - name: Var177\n type: number\n - name: Var178\n type: number\n - name: Var179\n type: number\n - name: Var180\n type: number\n - name: Var181\n type: number\n - name: Var182\n type: number\n - name: Var183\n type: number\n - name: Var184\n type: number\n - name: Var185\n type: number\n - name: Var186\n type: number\n - name: Var187\n type: number\n - name: Var188\n type: number\n - name: Var189\n type: number\n - name: Var190\n type: number\n - name: Var191\n type: category\n - name: Var192\n type: category\n - name: Var193\n type: category\n - name: Var194\n type: category\n - name: Var195\n type: category\n - name: Var196\n type: category\n - name: Var197\n type: category\n - name: Var198\n type: category\n - name: Var199\n type: category\n - name: Var200\n type: category\n - name: Var201\n type: category\n - name: Var202\n type: category\n - name: Var203\n type: category\n - name: Var204\n type: category\n - name: Var205\n type: category\n - name: Var206\n type: category\n - name: Var207\n type: category\n - name: Var208\n type: category\n - name: Var209\n type: number\n - name: Var210\n type: category\n - name: Var211\n type: category\n - name: Var212\n type: category\n - name: Var213\n type: category\n - name: Var214\n type: category\n - name: Var215\n type: category\n - name: Var216\n type: category\n - name: Var217\n type: category\n - name: Var218\n type: category\n - name: Var219\n type: category\n - name: Var220\n type: category\n - name: Var221\n type: category\n - name: Var222\n type: category\n - name: Var223\n type: category\n - name: Var224\n type: category\n - name: Var225\n type: category\n - name: Var226\n type: category\n - name: Var227\n type: category\n - name: Var228\n type: category\n - name: Var229\n type: category\n - name: Var230\n type: number\n - name: target\n type: binary\noutput_features:\n - name: target\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/kdd_churn.yaml",
"content": "version: 1.0\nname: kdd_churn\ndownload_urls:\n - https://kdd.org/cupfiles/KDDCupData/2009/orange_small_train.data.zip\n - https://kdd.org/cupfiles/KDDCupData/2009/orange_small_test.data.zip\n - https://kdd.org/cupfiles/KDDCupData/2009/orange_small_train_churn.labels\n - https://raw.githubusercontent.com/catboost/benchmarks/master/quality_benchmarks/prepare_appetency_churn_upselling/churn/stratified_train_idx_churn.txt\n - https://raw.githubusercontent.com/catboost/benchmarks/master/quality_benchmarks/prepare_appetency_churn_upselling/churn/stratified_test_idx_churn.txt\nsha256:\n orange_small_test.data.zip: 440ac8a350144c14f4d6947c096ad675ee84aa27b4b742071662696e333cec53\n orange_small_train.data.zip: 31ccb810bdbb71c16e079326443166dc3dfbf73cd358fc4a4ce7440fb1bc6040\n orange_small_train_churn.labels: fe8891cc574bd55a214514e522a5bed1eec2c3f347a49a36e51620009e7b6f5b\n stratified_train_idx_churn.txt: 34f9880959ced6f668b25f879fdd388b3826efeca0df03f5a2a5494ce6795406\n stratified_test_idx_churn.txt: 1675a62cd49c43535eedee3b746f65f8c6a4ebd7f4d0da04e442fd658a408042\ndataset_filenames: orange_small_train.data\nloader: kdd_loader.KDDChurnLoader\ndescription: |\n The KDD Cup 2009 Churn dataset.\n https://www.kdd.org/kdd-cup/view/kdd-cup-2009/Data\ncolumns:\n - name: Var1\n type: number\n - name: Var2\n type: number\n - name: Var3\n type: number\n - name: Var4\n type: number\n - name: Var5\n type: number\n - name: Var6\n type: number\n - name: Var7\n type: number\n - name: Var8\n type: number\n - name: Var9\n type: number\n - name: Var10\n type: number\n - name: Var11\n type: number\n - name: Var12\n type: number\n - name: Var13\n type: number\n - name: Var14\n type: number\n - name: Var15\n type: number\n - name: Var16\n type: number\n - name: Var17\n type: number\n - name: Var18\n type: number\n - name: Var19\n type: number\n - name: Var20\n type: number\n - name: Var21\n type: number\n - name: Var22\n type: number\n - name: Var23\n type: number\n - name: Var24\n type: number\n - name: Var25\n type: number\n - name: Var26\n type: number\n - name: Var27\n type: number\n - name: Var28\n type: number\n - name: Var29\n type: number\n - name: Var30\n type: number\n - name: Var31\n type: number\n - name: Var32\n type: number\n - name: Var33\n type: number\n - name: Var34\n type: number\n - name: Var35\n type: number\n - name: Var36\n type: number\n - name: Var37\n type: number\n - name: Var38\n type: number\n - name: Var39\n type: number\n - name: Var40\n type: number\n - name: Var41\n type: number\n - name: Var42\n type: number\n - name: Var43\n type: number\n - name: Var44\n type: number\n - name: Var45\n type: number\n - name: Var46\n type: number\n - name: Var47\n type: number\n - name: Var48\n type: number\n - name: Var49\n type: number\n - name: Var50\n type: number\n - name: Var51\n type: number\n - name: Var52\n type: number\n - name: Var53\n type: number\n - name: Var54\n type: number\n - name: Var55\n type: number\n - name: Var56\n type: number\n - name: Var57\n type: number\n - name: Var58\n type: number\n - name: Var59\n type: number\n - name: Var60\n type: number\n - name: Var61\n type: number\n - name: Var62\n type: number\n - name: Var63\n type: number\n - name: Var64\n type: number\n - name: Var65\n type: number\n - name: Var66\n type: number\n - name: Var67\n type: number\n - name: Var68\n type: number\n - name: Var69\n type: number\n - name: Var70\n type: number\n - name: Var71\n type: number\n - name: Var72\n type: number\n - name: Var73\n type: number\n - name: Var74\n type: number\n - name: Var75\n type: number\n - name: Var76\n type: number\n - name: Var77\n type: number\n - name: Var78\n type: number\n - name: Var79\n type: number\n - name: Var80\n type: number\n - name: Var81\n type: number\n - name: Var82\n type: number\n - name: Var83\n type: number\n - name: Var84\n type: number\n - name: Var85\n type: number\n - name: Var86\n type: number\n - name: Var87\n type: number\n - name: Var88\n type: number\n - name: Var89\n type: number\n - name: Var90\n type: number\n - name: Var91\n type: number\n - name: Var92\n type: number\n - name: Var93\n type: number\n - name: Var94\n type: number\n - name: Var95\n type: number\n - name: Var96\n type: number\n - name: Var97\n type: number\n - name: Var98\n type: number\n - name: Var99\n type: number\n - name: Var100\n type: number\n - name: Var101\n type: number\n - name: Var102\n type: number\n - name: Var103\n type: number\n - name: Var104\n type: number\n - name: Var105\n type: number\n - name: Var106\n type: number\n - name: Var107\n type: number\n - name: Var108\n type: number\n - name: Var109\n type: number\n - name: Var110\n type: number\n - name: Var111\n type: number\n - name: Var112\n type: number\n - name: Var113\n type: number\n - name: Var114\n type: number\n - name: Var115\n type: number\n - name: Var116\n type: number\n - name: Var117\n type: number\n - name: Var118\n type: number\n - name: Var119\n type: number\n - name: Var120\n type: number\n - name: Var121\n type: number\n - name: Var122\n type: number\n - name: Var123\n type: number\n - name: Var124\n type: number\n - name: Var125\n type: number\n - name: Var126\n type: number\n - name: Var127\n type: number\n - name: Var128\n type: number\n - name: Var129\n type: number\n - name: Var130\n type: number\n - name: Var131\n type: number\n - name: Var132\n type: number\n - name: Var133\n type: number\n - name: Var134\n type: number\n - name: Var135\n type: number\n - name: Var136\n type: number\n - name: Var137\n type: number\n - name: Var138\n type: number\n - name: Var139\n type: number\n - name: Var140\n type: number\n - name: Var141\n type: number\n - name: Var142\n type: number\n - name: Var143\n type: number\n - name: Var144\n type: number\n - name: Var145\n type: number\n - name: Var146\n type: number\n - name: Var147\n type: number\n - name: Var148\n type: number\n - name: Var149\n type: number\n - name: Var150\n type: number\n - name: Var151\n type: number\n - name: Var152\n type: number\n - name: Var153\n type: number\n - name: Var154\n type: number\n - name: Var155\n type: number\n - name: Var156\n type: number\n - name: Var157\n type: number\n - name: Var158\n type: number\n - name: Var159\n type: number\n - name: Var160\n type: number\n - name: Var161\n type: number\n - name: Var162\n type: number\n - name: Var163\n type: number\n - name: Var164\n type: number\n - name: Var165\n type: number\n - name: Var166\n type: number\n - name: Var167\n type: number\n - name: Var168\n type: number\n - name: Var169\n type: number\n - name: Var170\n type: number\n - name: Var171\n type: number\n - name: Var172\n type: number\n - name: Var173\n type: number\n - name: Var174\n type: number\n - name: Var175\n type: number\n - name: Var176\n type: number\n - name: Var177\n type: number\n - name: Var178\n type: number\n - name: Var179\n type: number\n - name: Var180\n type: number\n - name: Var181\n type: number\n - name: Var182\n type: number\n - name: Var183\n type: number\n - name: Var184\n type: number\n - name: Var185\n type: number\n - name: Var186\n type: number\n - name: Var187\n type: number\n - name: Var188\n type: number\n - name: Var189\n type: number\n - name: Var190\n type: number\n - name: Var191\n type: category\n - name: Var192\n type: category\n - name: Var193\n type: category\n - name: Var194\n type: category\n - name: Var195\n type: category\n - name: Var196\n type: category\n - name: Var197\n type: category\n - name: Var198\n type: category\n - name: Var199\n type: category\n - name: Var200\n type: category\n - name: Var201\n type: category\n - name: Var202\n type: category\n - name: Var203\n type: category\n - name: Var204\n type: category\n - name: Var205\n type: category\n - name: Var206\n type: category\n - name: Var207\n type: category\n - name: Var208\n type: category\n - name: Var209\n type: number\n - name: Var210\n type: category\n - name: Var211\n type: category\n - name: Var212\n type: category\n - name: Var213\n type: category\n - name: Var214\n type: category\n - name: Var215\n type: category\n - name: Var216\n type: category\n - name: Var217\n type: category\n - name: Var218\n type: category\n - name: Var219\n type: category\n - name: Var220\n type: category\n - name: Var221\n type: category\n - name: Var222\n type: category\n - name: Var223\n type: category\n - name: Var224\n type: category\n - name: Var225\n type: category\n - name: Var226\n type: category\n - name: Var227\n type: category\n - name: Var228\n type: category\n - name: Var229\n type: category\n - name: Var230\n type: number\n - name: target\n type: binary\noutput_features:\n - name: target\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/kdd_upselling.yaml",
"content": "version: 1.0\nname: kdd_upselling\ndownload_urls:\n - https://kdd.org/cupfiles/KDDCupData/2009/orange_small_train.data.zip\n - https://kdd.org/cupfiles/KDDCupData/2009/orange_small_test.data.zip\n - https://kdd.org/cupfiles/KDDCupData/2009/orange_small_train_upselling.labels\n - https://raw.githubusercontent.com/catboost/benchmarks/master/quality_benchmarks/prepare_appetency_churn_upselling/upselling/stratified_train_idx_upselling.txt\n - https://raw.githubusercontent.com/catboost/benchmarks/master/quality_benchmarks/prepare_appetency_churn_upselling/upselling/stratified_test_idx_upselling.txt\nsha256:\n orange_small_test.data.zip: 440ac8a350144c14f4d6947c096ad675ee84aa27b4b742071662696e333cec53\n orange_small_train.data.zip: 31ccb810bdbb71c16e079326443166dc3dfbf73cd358fc4a4ce7440fb1bc6040\n orange_small_train_upselling.labels: 86effe68394fe1ab21c2d855f74adf70f442990aa95dfe5c97340fc924440e68\n stratified_train_idx_upselling.txt: 659060717872177d607fbb157e8d2142c719912771d1716da11ccdd6ff915a05\n stratified_test_idx_upselling.txt: 64cb66ef559b4ccff096e0d7c150c7d019321ffd6cef2362c195a56c56effcb7\ndataset_filenames: orange_small_train.data\nloader: kdd_loader.KDDUpsellingLoader\ndescription: |\n The KDD Cup 2009 Upselling dataset.\n https://www.kdd.org/kdd-cup/view/kdd-cup-2009/Data\ncolumns:\n - name: Var1\n type: number\n - name: Var2\n type: number\n - name: Var3\n type: number\n - name: Var4\n type: number\n - name: Var5\n type: number\n - name: Var6\n type: number\n - name: Var7\n type: number\n - name: Var8\n type: number\n - name: Var9\n type: number\n - name: Var10\n type: number\n - name: Var11\n type: number\n - name: Var12\n type: number\n - name: Var13\n type: number\n - name: Var14\n type: number\n - name: Var15\n type: number\n - name: Var16\n type: number\n - name: Var17\n type: number\n - name: Var18\n type: number\n - name: Var19\n type: number\n - name: Var20\n type: number\n - name: Var21\n type: number\n - name: Var22\n type: number\n - name: Var23\n type: number\n - name: Var24\n type: number\n - name: Var25\n type: number\n - name: Var26\n type: number\n - name: Var27\n type: number\n - name: Var28\n type: number\n - name: Var29\n type: number\n - name: Var30\n type: number\n - name: Var31\n type: number\n - name: Var32\n type: number\n - name: Var33\n type: number\n - name: Var34\n type: number\n - name: Var35\n type: number\n - name: Var36\n type: number\n - name: Var37\n type: number\n - name: Var38\n type: number\n - name: Var39\n type: number\n - name: Var40\n type: number\n - name: Var41\n type: number\n - name: Var42\n type: number\n - name: Var43\n type: number\n - name: Var44\n type: number\n - name: Var45\n type: number\n - name: Var46\n type: number\n - name: Var47\n type: number\n - name: Var48\n type: number\n - name: Var49\n type: number\n - name: Var50\n type: number\n - name: Var51\n type: number\n - name: Var52\n type: number\n - name: Var53\n type: number\n - name: Var54\n type: number\n - name: Var55\n type: number\n - name: Var56\n type: number\n - name: Var57\n type: number\n - name: Var58\n type: number\n - name: Var59\n type: number\n - name: Var60\n type: number\n - name: Var61\n type: number\n - name: Var62\n type: number\n - name: Var63\n type: number\n - name: Var64\n type: number\n - name: Var65\n type: number\n - name: Var66\n type: number\n - name: Var67\n type: number\n - name: Var68\n type: number\n - name: Var69\n type: number\n - name: Var70\n type: number\n - name: Var71\n type: number\n - name: Var72\n type: number\n - name: Var73\n type: number\n - name: Var74\n type: number\n - name: Var75\n type: number\n - name: Var76\n type: number\n - name: Var77\n type: number\n - name: Var78\n type: number\n - name: Var79\n type: number\n - name: Var80\n type: number\n - name: Var81\n type: number\n - name: Var82\n type: number\n - name: Var83\n type: number\n - name: Var84\n type: number\n - name: Var85\n type: number\n - name: Var86\n type: number\n - name: Var87\n type: number\n - name: Var88\n type: number\n - name: Var89\n type: number\n - name: Var90\n type: number\n - name: Var91\n type: number\n - name: Var92\n type: number\n - name: Var93\n type: number\n - name: Var94\n type: number\n - name: Var95\n type: number\n - name: Var96\n type: number\n - name: Var97\n type: number\n - name: Var98\n type: number\n - name: Var99\n type: number\n - name: Var100\n type: number\n - name: Var101\n type: number\n - name: Var102\n type: number\n - name: Var103\n type: number\n - name: Var104\n type: number\n - name: Var105\n type: number\n - name: Var106\n type: number\n - name: Var107\n type: number\n - name: Var108\n type: number\n - name: Var109\n type: number\n - name: Var110\n type: number\n - name: Var111\n type: number\n - name: Var112\n type: number\n - name: Var113\n type: number\n - name: Var114\n type: number\n - name: Var115\n type: number\n - name: Var116\n type: number\n - name: Var117\n type: number\n - name: Var118\n type: number\n - name: Var119\n type: number\n - name: Var120\n type: number\n - name: Var121\n type: number\n - name: Var122\n type: number\n - name: Var123\n type: number\n - name: Var124\n type: number\n - name: Var125\n type: number\n - name: Var126\n type: number\n - name: Var127\n type: number\n - name: Var128\n type: number\n - name: Var129\n type: number\n - name: Var130\n type: number\n - name: Var131\n type: number\n - name: Var132\n type: number\n - name: Var133\n type: number\n - name: Var134\n type: number\n - name: Var135\n type: number\n - name: Var136\n type: number\n - name: Var137\n type: number\n - name: Var138\n type: number\n - name: Var139\n type: number\n - name: Var140\n type: number\n - name: Var141\n type: number\n - name: Var142\n type: number\n - name: Var143\n type: number\n - name: Var144\n type: number\n - name: Var145\n type: number\n - name: Var146\n type: number\n - name: Var147\n type: number\n - name: Var148\n type: number\n - name: Var149\n type: number\n - name: Var150\n type: number\n - name: Var151\n type: number\n - name: Var152\n type: number\n - name: Var153\n type: number\n - name: Var154\n type: number\n - name: Var155\n type: number\n - name: Var156\n type: number\n - name: Var157\n type: number\n - name: Var158\n type: number\n - name: Var159\n type: number\n - name: Var160\n type: number\n - name: Var161\n type: number\n - name: Var162\n type: number\n - name: Var163\n type: number\n - name: Var164\n type: number\n - name: Var165\n type: number\n - name: Var166\n type: number\n - name: Var167\n type: number\n - name: Var168\n type: number\n - name: Var169\n type: number\n - name: Var170\n type: number\n - name: Var171\n type: number\n - name: Var172\n type: number\n - name: Var173\n type: number\n - name: Var174\n type: number\n - name: Var175\n type: number\n - name: Var176\n type: number\n - name: Var177\n type: number\n - name: Var178\n type: number\n - name: Var179\n type: number\n - name: Var180\n type: number\n - name: Var181\n type: number\n - name: Var182\n type: number\n - name: Var183\n type: number\n - name: Var184\n type: number\n - name: Var185\n type: number\n - name: Var186\n type: number\n - name: Var187\n type: number\n - name: Var188\n type: number\n - name: Var189\n type: number\n - name: Var190\n type: number\n - name: Var191\n type: category\n - name: Var192\n type: category\n - name: Var193\n type: category\n - name: Var194\n type: category\n - name: Var195\n type: category\n - name: Var196\n type: category\n - name: Var197\n type: category\n - name: Var198\n type: category\n - name: Var199\n type: category\n - name: Var200\n type: category\n - name: Var201\n type: category\n - name: Var202\n type: category\n - name: Var203\n type: category\n - name: Var204\n type: category\n - name: Var205\n type: category\n - name: Var206\n type: category\n - name: Var207\n type: category\n - name: Var208\n type: category\n - name: Var209\n type: number\n - name: Var210\n type: category\n - name: Var211\n type: category\n - name: Var212\n type: category\n - name: Var213\n type: category\n - name: Var214\n type: category\n - name: Var215\n type: category\n - name: Var216\n type: category\n - name: Var217\n type: category\n - name: Var218\n type: category\n - name: Var219\n type: category\n - name: Var220\n type: category\n - name: Var221\n type: category\n - name: Var222\n type: category\n - name: Var223\n type: category\n - name: Var224\n type: category\n - name: Var225\n type: category\n - name: Var226\n type: category\n - name: Var227\n type: category\n - name: Var228\n type: category\n - name: Var229\n type: category\n - name: Var230\n type: number\n - name: target\n type: binary\noutput_features:\n - name: target\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/kick_starter_funding.yaml",
"content": "version: 1.0\nname: kick_starter_funding\ndownload_urls:\n - https://automl-mm-bench.s3.amazonaws.com/kick_starter_funding/train.csv\n - https://automl-mm-bench.s3.amazonaws.com/kick_starter_funding/test.csv\nsha256:\n test.csv: 13c2d4b74ac8d1e258659b5f5fa74526b9d27e305f6c29ad7e853dfeeb01983c\n train.csv: 3120b69f30bbc08c68940ab9e5d85d6cc2fbc9a65e8a24c66739179b6a60150e\ntrain_filenames: train.csv\ntest_filenames: test.csv\ndescription: |\n Funding Successful Projects on Kickstarter\n Predict if a project will get successfully funded or not using labeled data\n https://www.kaggle.com/codename007/funding-successful-projects\ncolumns:\n - name: name\n type: category\n - name: desc\n type: text\n - name: goal\n type: number\n - name: keywords\n type: category\n - name: disable_communication\n type: binary\n - name: country\n type: category\n - name: currency\n type: category\n - name: deadline\n type: number\n - name: created_at\n type: number\n - name: final_status\n type: binary\noutput_features:\n - name: final_status\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/melbourne_airbnb.yaml",
"content": "version: 1.0\nname: melbourne_airbnb\ndownload_urls:\n - https://automl-mm-bench.s3.amazonaws.com/airbnb_melbourne/train.pq\n - https://automl-mm-bench.s3.amazonaws.com/airbnb_melbourne/test.pq\nsha256:\n test.pq: 9fe965cfdbd24ee9af7a004a7dc8c4e535a7ffceb722dce00f8ea90a54f95aa9\n train.pq: c158c0f497ef355ba9d5de0de7556f6eb7f9bc343a67c4c681b014f6c7412e48\ntrain_filenames: train.pq\ntest_filenames: test.pq\ndescription: |\n Melbourne Airbnb Open Data\n Detailed and summarized data of Airbnb activity in Melbourne, VIC, Australia\n https://www.kaggle.com/tylerx/melbourne-airbnb-open-data\ncolumns:\n - name: id\n type: number\n - name: listing_url\n type: category\n - name: scrape_id\n type: number\n - name: last_scraped\n type: date\n - name: text\n type: category\n - name: summary\n type: text\n - name: space\n type: text\n - name: description\n type: text\n - name: neighborhood_overview\n type: text\n - name: notes\n type: text\n - name: transit\n type: text\n - name: access\n type: text\n - name: interaction\n type: text\n - name: house_rules\n type: text\n - name: picture_url\n type: category\n - name: host_id\n type: category\n - name: host_url\n type: category\n - name: host_name\n type: category\n - name: host_since\n type: date\n - name: host_location\n type: category\n - name: host_about\n type: text\n - name: host_response_time\n type: category\n - name: host_response_rate\n type: category\n - name: host_is_superhost\n type: binary\n - name: host_thumbnail_url\n type: category\n - name: host_picture_url\n type: category\n - name: host_neighborhood\n type: category\n - name: host_verifications\n type: set\n - name: host_has_profile_pic\n type: binary\n - name: host_identity_verified\n type: binary\n - name: street\n type: category\n - name: neighborhood\n type: category\n - name: city\n type: category\n - name: suburb\n type: category\n - name: state\n type: category\n - name: zipcode\n type: category\n - name: smart_location\n type: category\n - name: country_code\n type: category\n - name: country\n type: category\n - name: latitude\n type: number\n - name: longitude\n type: number\n - name: is_location_exact\n type: binary\n - name: property_type\n type: category\n - name: room_type\n type: category\n - name: accommodates\n type: number\n - name: bathrooms\n type: number\n - name: bedrooms\n type: number\n - name: beds\n type: number\n - name: bed_type\n type: category\n - name: amenities\n type: set\n - name: price\n type: number\n - name: weekly_price\n type: number\n - name: monthly_price\n type: number\n - name: security_deposit\n type: number\n - name: cleaning_fee\n type: number\n - name: guests_included\n type: number\n - name: extra_people\n type: number\n - name: minimum_nights\n type: number\n - name: maximum_nights\n type: number\n - name: calendar_updated\n type: category\n - name: has_availability\n type: binary\n - name: availability_30\n type: number\n - name: availability_60\n type: number\n - name: availability_90\n type: number\n - name: availability_365\n type: number\n - name: calendar_last_scraped\n type: date\n - name: number_of_reviews\n type: number\n - name: first_review\n type: date\n - name: last_review\n type: date\n - name: review_scores_rating\n type: number\n - name: review_scores_accuracy\n type: number\n - name: review_scores_cleanliness\n type: number\n - name: review_scores_checkin\n type: number\n - name: review_scores_communication\n type: number\n - name: review_scores_location\n type: number\n - name: review_scores_value\n type: number\n - name: requires_license\n type: binary\n - name: license\n type: category\n - name: instant_bookable\n type: binary\n - name: cancellation_policy\n type: category\n - name: require_guest_profile_picture\n type: binary\n - name: require_guest_phone_verification\n type: binary\n - name: calculated_host_listings_count\n type: number\n - name: reviews_per_month\n type: number\n - name: price_label\n type: number\n - name: host_verifications_jumio\n type: binary\n - name: host_verifications_government_id\n type: binary\n - name: host_verifications_kba\n type: binary\n - name: host_verifications_zhima_selfie\n type: binary\n - name: host_verifications_facebook\n type: binary\n - name: host_verifications_work_email\n type: binary\n - name: host_verifications_google\n type: binary\n - name: host_verifications_sesame\n type: binary\n - name: host_verifications_manual_online\n type: binary\n - name: host_verifications_manual_offline\n type: binary\n - name: host_verifications_offline_government_id\n type: binary\n - name: host_verifications_selfie\n type: binary\n - name: host_verifications_reviews\n type: binary\n - name: host_verifications_identity_manual\n type: binary\n - name: host_verifications_sesame_offline\n type: binary\n - name: host_verifications_weibo\n type: binary\n - name: host_verifications_email\n type: binary\n - name: host_verifications_sent_id\n type: binary\n - name: host_verifications_phone\n type: binary\noutput_features:\n - name: price_label\n type: category\n"
},
{
"path": "ludwig/datasets/configs/mercari_price_suggestion.yaml",
"content": "version: 1.0\nname: mercari_price_suggestion\ndownload_urls:\n - https://automl-mm-bench.s3.amazonaws.com/mercari_price_suggestion/train.pq\n - https://automl-mm-bench.s3.amazonaws.com/mercari_price_suggestion/dev.pq\n - https://automl-mm-bench.s3.amazonaws.com/mercari_price_suggestion/test.pq\nsha256:\n test.pq: 05fed940f5545e6a470ca595d014a02b173fd3362ca5bc5c458d02640b892a57\n train.pq: a0613b77714ebb9f8927cf6bff2092af8143f4a66a64e45e9c3bf9d18604cfe3\n dev.pq: f7284b86adde0354f30ee2c2b7a7a55dc895d202b4291138e807c8f3eaacb6b0\ntrain_filenames: train.pq\nvalidation_filenames: dev.pq\ntest_filenames: test.pq\ndescription: |\n Predict product price based on details like product category name, brand name, and item condition.\n We have converted price to log price by log(1 + price).\n https://www.kaggle.com/c/mercari-price-suggestion-challenge\ncolumns:\n - name: train_id\n type: category\n - name: name\n type: category\n - name: item_condition_id\n type: category\n - name: category_name\n type: category\n - name: brand_name\n type: category\n - name: price\n type: number\n - name: shipping\n type: binary\n - name: item_description\n type: text\n - name: log_price\n type: number\n - name: cat1\n type: category\n - name: cat2\n type: category\n - name: cat3\n type: category\noutput_features:\n - name: log_price\n type: number\n"
},
{
"path": "ludwig/datasets/configs/mercari_price_suggestion100K.yaml",
"content": "version: 1.0\nname: mercari_price_suggestion100K\ndownload_urls:\n - https://automl-mm-bench.s3.amazonaws.com/mercari_price_suggestion100K/train.pq\n - https://automl-mm-bench.s3.amazonaws.com/mercari_price_suggestion100K/test.pq\nsha256:\n test.pq: 60431577bd6cb433bae287ced2edc7a557497b66b1fe90e2fbec6ffc24bf35eb\n train.pq: f60063847d9b828f1e9366eb69fa53774771b53291586d1cce506c931b7173f4\ntrain_filenames: train.pq\ntest_filenames: test.pq\ndescription: |\n Predict product price based on details like product category name, brand name, and item condition.\n We have converted price to log price by log(1 + price).\n https://www.kaggle.com/c/mercari-price-suggestion-challenge\ncolumns:\n - name: train_id\n type: category\n - name: name\n type: category\n - name: item_condition_id\n type: category\n - name: category_name\n type: category\n - name: brand_name\n type: category\n - name: price\n type: number\n - name: shipping\n type: binary\n - name: item_description\n type: text\n - name: log_price\n type: number\n - name: cat1\n type: category\n - name: cat2\n type: category\n - name: cat3\n type: category\noutput_features:\n - name: log_price\n type: number\n"
},
{
"path": "ludwig/datasets/configs/mercedes_benz_greener.yaml",
"content": "version: 1.0\nname: mercedes_benz_greener\nkaggle_competition: mercedes-benz-greener-manufacturing\narchive_filenames: mercedes-benz-greener-manufacturing.zip\nsha256:\n mercedes-benz-greener-manufacturing.zip: 91143716085345a84dc4991b8eb1d5ff80d8aa134930de946b3b24be0f2e5d1a\ntrain_filenames: train.csv\ntest_filenames: test.csv\ndescription: |\n The Mercedes-Benz Greener Manufacturing dataset.\n https://www.kaggle.com/c/mercedes-benz-greener-manufacturing\noutput_features:\n - name: y\n type: number\nfallback_mirrors:\n - name: predibase\n download_paths: s3://ludwig-tests/ludwig_backup/mercedes-benz-greener-manufacturing.zip\n"
},
{
"path": "ludwig/datasets/configs/mnist.yaml",
"content": "version: 1.0\nname: mnist\ndownload_urls:\n - https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz\n - https://ossci-datasets.s3.amazonaws.com/mnist/train-labels-idx1-ubyte.gz\n - https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz\n - https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz\nsha256:\n t10k-images-idx3-ubyte.gz: 8d422c7b0a1c1c79245a5bcf07fe86e33eeafee792b84584aec276f5a2dbc4e6\n train-images-idx3-ubyte.gz: 440fcabf73cc546fa21475e81ea370265605f56be210a4024d2ca8f203523609\n train-labels-idx1-ubyte.gz: 3552534a0a558bbed6aed32b30c495cca23d567ec52cac8be1a0730e8010255c\n t10k-labels-idx1-ubyte.gz: f7ae60f92e00ec6debd23a6088c31dbd2371eca3ffa0defaefb259924204aec6\npreserve_paths:\n - training\n - testing\nloader: mnist.MNISTLoader\ndescription: |\n The MNIST database of handwritten digits, available from this page,\n has a training set of 60,000 examples, and a test set of 10,000 examples.\n It is a subset of a larger set available from NIST. The digits have been\n size-normalized and centered in a fixed-size image.\n It is a good database for people who want to try learning techniques and\n pattern recognition methods on real-world data while spending minimal\n efforts on preprocessing and formatting.\n http://yann.lecun.com/exdb/mnist/\ncolumns:\n - name: image_path\n type: image\n - name: label\n type: category\noutput_features:\n - name: label\n type: category\n"
},
{
"path": "ludwig/datasets/configs/mushroom_edibility.yaml",
"content": "version: 1.0\nname: mushroom_edibility\ndownload_urls: http://archive.ics.uci.edu/ml/machine-learning-databases/mushroom/agaricus-lepiota.data\nsha256:\n agaricus-lepiota.data: e65d082030501a3ebcbcd7c9f7c71aa9d28fdfff463bf4cf4716a3fe13ac360e\ntrain_filenames: agaricus-lepiota.data\ndescription: |\n This data set includes descriptions of hypothetical samples corresponding\n to 23 species of gilled mushrooms in the Agaricus and Lepiota Family (pp. 500-525).\n Each species is identified as definitely edible, definitely poisonous,\n or of unknown edibility and not recommended. This latter class was combined with\n the poisonous one.\ncolumns:\n - name: class\n type: category\n - name: cap-shape\n type: category\n - name: cap-surface\n type: category\n - name: cap-color\n type: category\n - name: bruises?\n type: category\n - name: odor\n type: category\n - name: gill-attachment\n type: category\n - name: gill-spacing\n type: category\n - name: gill-size\n type: category\n - name: gill-color\n type: category\n - name: stalk-shape\n type: category\n - name: stalk-root\n type: category\n - name: stalk-surface-above-ring\n type: category\n - name: stalk-surface-below-ring\n type: category\n - name: stalk-color-above-ring\n type: category\n - name: stalk-color-below-ring\n type: category\n - name: veil-type\n type: category\n - name: veil-color\n type: category\n - name: ring-number\n type: category\n - name: ring-type\n type: category\n - name: spore-print-color\n type: category\n - name: population\n type: category\n - name: habitat\n type: category\noutput_features:\n - name: class\n type: category\n"
},
{
"path": "ludwig/datasets/configs/naval.yaml",
"content": "version: 1.0\nname: naval\ndownload_urls: http://archive.ics.uci.edu/ml/machine-learning-databases/00316/UCI%20CBM%20Dataset.zip\nsha256:\n UCI%20CBM%20Dataset.zip: 91a3815da80b5ab7e2d5b82ac82f1c2cbf89182c7a65bcdf240db1e014423cb9\ndataset_filenames: UCI CBM Dataset/data.txt\nloader: naval.NavalLoader\ndescription: |\n Condition Based Maintenance of Naval Propulsion Plants Data Set\n http://archive.ics.uci.edu/ml/datasets/condition+based+maintenance+of+naval+propulsion+plants\ncolumns:\n - name: lp\n type: number\n - name: v\n type: number\n - name: gtt\n type: number\n - name: gtn\n type: number\n - name: ggn\n type: number\n - name: ts\n type: number\n - name: tp\n type: number\n - name: t48\n type: number\n - name: t1\n type: number\n - name: t2\n type: number\n - name: p48\n type: number\n - name: p1\n type: number\n - name: p2\n type: number\n - name: pexh\n type: number\n - name: tic\n type: number\n - name: mf\n type: number\n - name: gtcdsc\n type: number\n - name: gttdsc\n type: number\noutput_features:\n- name: gtcdsc\n type: number\n"
},
{
"path": "ludwig/datasets/configs/news_channel.yaml",
"content": "version: 1.0\nname: news_channel\ndownload_urls:\n - https://automl-mm-bench.s3.amazonaws.com/news_channel/train.csv\n - https://automl-mm-bench.s3.amazonaws.com/news_channel/test.csv\nsha256:\n test.csv: d48e7261dce69964eb1163c89e05261b8732c676b10de9b40339b2d95559c9c3\n train.csv: 46e433fcf070ec684cfaf30bada482a73637e8dd954edc3e1fe860de8e661055\ntrain_filenames: train.csv\ntest_filenames: test.csv\ndescription: |\n Online News Popularity Data Set\n This dataset summarizes a heterogeneous set of features about articles\n published by Mashable in a period of two years. The goal is to predict\n the number of shares in social networks (popularity).\n https://archive.ics.uci.edu/ml/datasets/online+news+popularity\ncolumns: # Most lot of these columns have a leading space\n - name: n_tokens_content\n type: number\n - name: n_unique_tokens\n type: number\n - name: n_non_stop_words\n type: number\n - name: n_non_stop_unique_tokens\n type: number\n - name: num_hrefs\n type: number\n - name: num_self_hrefs\n type: number\n - name: num_imgs\n type: number\n - name: num_videos\n type: number\n - name: average_token_length\n type: number\n - name: num_keywords\n type: number\n - name: global_subjectivity\n type: number\n - name: global_sentiment_polarity\n type: number\n - name: global_rate_positive_words\n type: number\n - name: global_rate_negative_words\n type: number\n - name: rate_positive_words\n type: number\n - name: rate_negative_words\n type: number\n - name: article_title\n type: text\n - name: channel\n type: category\noutput_features:\n- name: channel\n type: category\n"
},
{
"path": "ludwig/datasets/configs/news_popularity2.yaml",
"content": "version: 1.0\nname: news_popularity2\ndownload_urls:\n - https://automl-mm-bench.s3.amazonaws.com/news_popularity2/train.csv\n - https://automl-mm-bench.s3.amazonaws.com/news_popularity2/test.csv\nsha256:\n test.csv: 276effa981456e187fb1fc07abd8556d240e1a110fc5c096f2ad75a4082d1ccb\n train.csv: 3673a07b87dbe09a9073e5ab83241681f561984269a9dc5411018fd9bca70b71\ntrain_filenames: train.csv\ntest_filenames: test.csv\ndescription: |\n Online News Popularity Data Set\n This dataset summarizes a heterogeneous set of features about articles\n published by Mashable in a period of two years. The goal is to predict\n the number of shares in social networks (popularity).\n https://archive.ics.uci.edu/ml/datasets/online+news+popularity\ncolumns:\n - name: n_tokens_content\n type: number\n - name: average_token_length\n type: number\n - name: num_keywords\n type: number\n - name: log_shares\n type: number\n - name: article_title\n type: text\noutput_features:\n- name: log_shares\n type: number\n"
},
{
"path": "ludwig/datasets/configs/noshow_appointments.yaml",
"content": "version: 1.0\nname: noshow_appointments\nkaggle_dataset_id: joniarroba/noshowappointments\narchive_filenames: noshowappointments.zip\nsha256:\n noshowappointments.zip: 4b4f258837029bd4e61ed4c9bab2ce8a3b8a299d1a4f5bdabcc98967d5e29a43\nloader: split_loaders.RandomSplitLoader\ndescription: |\n 110.527 medical appointments its 14 associated variables (characteristics).\n The most important one if the patient show-up or no-show to the appointment.\n https://www.kaggle.com/datasets/joniarroba/noshowappointments\ncolumns:\n - name: PatientId\n type: category\n - name: AppointmentID\n type: category\n - name: Gender\n type: binary\n - name: ScheduledDay\n type: date\n - name: AppointmentDay\n type: date\n - name: Age\n type: number\n - name: Neighbourhood\n type: category\n - name: Scholarship\n type: binary\n - name: Hipertension\n type: binary\n - name: Diabetes\n type: binary\n - name: Alcoholism\n type: binary\n - name: Handcap\n type: binary\n - name: SMS_received\n type: binary\n - name: No-show\n type: binary\noutput_features:\n - name: No-show\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/numerai28pt6.yaml",
"content": "version: 1.0\nname: numerai28pt6\nkaggle_dataset_id: numerai/encrypted-stock-market-data-from-numerai\narchive_filenames: encrypted-stock-market-data-from-numerai.zip\nsha256:\n encrypted-stock-market-data-from-numerai.zip: cc0714c5f4c8ac6b212f7569641c5110bd2296547af434cba77184ebb03f304b\ndescription: |\n Encrypted Stock Market Data from Numerai dataset from Kaggle.\ncolumns:\n - name: feature1\n type: number\n - name: feature2\n type: number\n - name: feature3\n type: number\n - name: feature4\n type: number\n - name: feature5\n type: number\n - name: feature6\n type: number\n - name: feature7\n type: number\n - name: feature8\n type: number\n - name: feature9\n type: number\n - name: feature10\n type: number\n - name: feature11\n type: number\n - name: feature12\n type: number\n - name: feature13\n type: number\n - name: feature14\n type: number\n - name: feature15\n type: number\n - name: feature16\n type: number\n - name: feature17\n type: number\n - name: feature18\n type: number\n - name: feature19\n type: number\n - name: feature20\n type: number\n - name: feature21\n type: number\n - name: target\n type: binary\noutput_features:\n - name: target\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/ohsumed_7400.yaml",
"content": "version: 1.0\nname: ohsumed_7400\nkaggle_dataset_id: weipengfei/ohr8r52\narchive_filenames: ohr8r52.zip\nsha256:\n ohr8r52.zip: 93c7a8817a32b994d93267506ad766281764ba9382e3f4f9d978544cebab6ca4\ntrain_filenames: oh/oh-train-stemmed.csv\nvalidation_filenames: oh/oh-dev-stemmed.csv\ntest_filenames: oh/oh-test-stemmed.csv\ndescription: |\n Ohsumed corpus is extracted from MEDLINE database. MEDLINE is designed for multi-label classification, we remove the\n text with two or more labels.\n https://www.kaggle.com/datasets/weipengfei/ohr8r52\ncolumns:\n - name: text\n type: text\n - name: edge\n type: text\n - name: intent\n type: category\noutput_features:\n - name: intent\n type: category\n"
},
{
"path": "ludwig/datasets/configs/ohsumed_cmu.yaml",
"content": "version: 1.0\nname: ohsumed_cmu\ndownload_urls: http://boston.lti.cs.cmu.edu/classes/95-865-K/HW/HW2/ohsumed-allcats-6.zip\nsha256:\n ohsumed-allcats-6.zip: 3f2f6c4e27faaac1c8dc179a121bed92d6adbdf91a1e11d2d124f7bd963798da\ndescription: |\n OHSUMED is a well-known medical abstracts dataset. It contains 348,566 references,\n and is still used for research and development.\n\n This is a subset of OHSUMED containing 6 categories, from this CMU course:\n http://boston.lti.cs.cmu.edu/classes/95-865-K/HW/HW2/\ncolumns:\n - name: text\n type: text\n - name: class\n type: category\noutput_features:\n - name: class\n type: category\n"
},
{
"path": "ludwig/datasets/configs/otto_group_product.yaml",
"content": "version: 1.0\nname: otto_group_product\nkaggle_competition: otto-group-product-classification-challenge\narchive_filenames: otto-group-product-classification-challenge.zip\nsha256:\n otto-group-product-classification-challenge.zip: 81d1fa5805036772b7a2a2425311fdc7b1568af4fbb42f0ec8f9661d0d21ce42\ntrain_filenames: train.csv\ntest_filenames: test.csv\ndescription: |\n The Otto Group Product Classification Challenge\n https://www.kaggle.com/c/otto-group-product-classification-challenge/overview\noutput_features:\n - name: target\n type: category\n"
},
{
"path": "ludwig/datasets/configs/poker_hand.yaml",
"content": "version: 1.0\nname: poker_hand\ndownload_urls:\n - http://archive.ics.uci.edu/ml/machine-learning-databases/poker/poker-hand-training-true.data\n - http://archive.ics.uci.edu/ml/machine-learning-databases/poker/poker-hand-testing.data\ntrain_filenames: poker-hand-training-true.data\ntest_filenames: poker-hand-testing.data\nsha256:\n poker-hand-testing.data: 3cd75958e19dd321ed5ca3f7f154c0f6aad544aab9f37731ac545b5f66b232c7\n poker-hand-training-true.data: 37becdf87d5f8cbf2b91d6471e965a25b86cb4a6d878c0f94a4025969fca464f\ndescription: |\n Each record is an example of a hand consisting of five playing cards\n drawn from a standard deck of 52. Each card is described using two\n attributes (suit and rank), for a total of 10 predictive attributes.\n There is one Class attribute that describes the \"Poker Hand\". The\n order of cards is important, which is why there are 480 possible\n Royal Flush hands as compared to 4.\n https://archive.ics.uci.edu/ml/datasets/Poker+Hand\ncolumns:\n - name: S1\n type: number\n - name: C1\n type: number\n - name: S2\n type: number\n - name: C2\n type: number\n - name: S3\n type: number\n - name: C3\n type: number\n - name: S4\n type: number\n - name: C4\n type: number\n - name: S5\n type: number\n - name: C5\n type: number\n - name: hand\n type: category\noutput_features:\n - name: hand\n type: category\n"
},
{
"path": "ludwig/datasets/configs/porto_seguro_safe_driver.yaml",
"content": "version: 1.0\nname: porto_seguro_safe_driver\nkaggle_competition: porto-seguro-safe-driver-prediction\narchive_filenames: porto-seguro-safe-driver-prediction.zip\nsha256:\n porto-seguro-safe-driver-prediction.zip: 53dd7b67b9b3df088c4e0814cba7317d3bc8f76094c726471c8f91e84f61ccdc\ntrain_filenames: train.csv\ntest_filenames: test.csv\ndescription: |\n Predict the probability that an auto insurance policy holder files a claim.\n https://www.kaggle.com/competitions/porto-seguro-safe-driver-prediction\noutput_features:\n - name: target\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/product_sentiment_machine_hack.yaml",
"content": "version: 1.0\nname: product_sentiment_machine_hack\ndownload_urls:\n - https://automl-mm-bench.s3.amazonaws.com/machine_hack_product_sentiment/train.csv\n - https://automl-mm-bench.s3.amazonaws.com/machine_hack_product_sentiment/dev.csv\nsha256:\n dev.csv: 33adff4dba7d9322397b398900c20f678d3fffc5d87b0ea825d9aa497a343150\n train.csv: 85a229e162b6d8c4839d1b27f834c36ae5e244fd027534fe62a888d4f536f0ef\ntrain_filenames: train.csv\ntest_filenames: dev.csv\ndescription: |\n We challenge the machinehackers community to develop a machine learning model\n to accurately classify various products into 4 different classes of sentiments\n based on the raw text review provided by the user.\n https://www.machinehack.com/hackathons/product_sentiment_classification_weekend_hackathon_19/overview\ncolumns:\n - name: Text_ID\n type: category\n - name: Product_Description\n type: text\n - name: Product_Type\n type: category\n - name: Sentiment\n type: category\noutput_features:\n- name: Sentiment\n type: category\n"
},
{
"path": "ludwig/datasets/configs/protein.yaml",
"content": "version: 1.0\nname: protein\ndownload_urls: http://archive.ics.uci.edu/ml/machine-learning-databases/00265/CASP.csv\nsha256:\n CASP.csv: 4277cfcb4e91a181746cbc654f001b57951c9e6a80f4f795fdb5c807e0848f40\ndescription: |\n Physicochemical Properties of Protein Tertiary Structure Data Set.\n https://archive.ics.uci.edu/ml/datasets/Physicochemical+Properties+of+Protein+Tertiary+Structure\ncolumns:\n - name: RMSD\n type: number\n - name: F1\n type: number\n - name: F2\n type: number\n - name: F3\n type: number\n - name: F4\n type: number\n - name: F5\n type: number\n - name: F6\n type: number\n - name: F7\n type: number\n - name: F8\n type: number\n - name: F9\n type: number\noutput_features:\n - name: RMSD\n type: number\n"
},
{
"path": "ludwig/datasets/configs/reuters_cmu.yaml",
"content": "version: 1.0\nname: reuters_cmu\ndownload_urls: http://boston.lti.cs.cmu.edu/classes/95-865-K/HW/HW2/reuters-allcats-6.zip\nsha256:\n reuters-allcats-6.zip: 304ae223f9ca35f7ce9066c9d31558c06ed5c72cd91faa885f82b928b2aa6f34\ndescription: |\n Reuters-21578 is a well-known newswire dataset containing 21,578 documents.\n\n This is a subset of Reuters-21578 using only 6 categories, from this CMU course:\n http://boston.lti.cs.cmu.edu/classes/95-865-K/HW/HW2/\ncolumns:\n - name: text\n type: text\n - name: class\n type: category\noutput_features:\n - name: class\n type: category\n"
},
{
"path": "ludwig/datasets/configs/reuters_r8.yaml",
"content": "version: 1.0\nname: reuters_r8\nkaggle_dataset_id: weipengfei/ohr8r52\narchive_filenames: ohr8r52.zip\nsha256:\n ohr8r52.zip: 93c7a8817a32b994d93267506ad766281764ba9382e3f4f9d978544cebab6ca4\ntrain_filenames: r8/r8-train-stemmed.csv\nvalidation_filenames: r8/r8-dev-stemmed.csv\ntest_filenames: r8/r8-test-stemmed.csv\ndescription: |\n Reuters R8 subset of Reuters 21578 dataset from Kaggle.\ncolumns:\n - name: text\n type: text\n - name: edge\n type: text\n - name: intent\n type: category\noutput_features:\n - name: intent\n type: category\n"
},
{
"path": "ludwig/datasets/configs/rossman_store_sales.yaml",
"content": "version: 1.0\nname: rossman_store_sales\nkaggle_competition: rossmann-store-sales\narchive_filenames: rossmann-store-sales.zip\nsha256:\n rossmann-store-sales.zip: 52ce715e02dc70cac16b14548580d656997f5d43ce3544220d5e574d26483cf3\nloader: rossman_store_sales.RossmanStoreSalesLoader\ndescription: |\n The Rossmann Store Sales dataset.\n Using the time split from the catboost benchmark\n https://github.com/catboost/benchmarks/tree/master/kaggle/rossmann-store-sales\n that is used in the TabNet paper,\n because the test set does not contain sales ground truth.\noutput_features:\n - name: Sales\n type: number\n"
},
{
"path": "ludwig/datasets/configs/santander_customer_satisfaction.yaml",
"content": "version: 1.0\nname: santander_customer_satisfaction\nkaggle_competition: santander-customer-satisfaction\narchive_filenames: santander-customer-satisfaction.zip\nsha256:\n santander-customer-satisfaction.zip: d4c2d068d8041af168d82d0eef7ad0b53ddd1d7fca9aba4e5d88fa1f957ee594\ntrain_filenames: train.csv\ntest_filenames: test.csv\ndescription: |\n Santander Customer Satisfaction Prediction.\n https://www.kaggle.com/c/santander-customer-satisfaction/overview\noutput_features:\n - name: TARGET\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/santander_customer_transaction.yaml",
"content": "version: 1.0\nname: santander_customer_transaction\nkaggle_competition: santander-customer-transaction-prediction\narchive_filenames: santander-customer-transaction-prediction.zip\nsha256:\n santander-customer-transaction-prediction.zip: b3a56d036b493a9cf0695018c968baba1ba7ef8c39d842cc5626e72f13c0ec69\ntrain_filenames: train.csv\ntest_filenames: test.csv\ndescription: |\n Santander Customer Transaction Prediction.\n https://www.kaggle.com/c/santander-customer-transaction-prediction/overview\noutput_features:\n - name: target\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/santander_value_prediction.yaml",
"content": "version: 1.0\nname: santander_value_prediction\nkaggle_competition: santander-value-prediction-challenge\narchive_filenames: santander-value-prediction-challenge.zip\nsha256:\n santander-value-prediction-challenge.zip: a8b44a0403bff6ab42f2bd1da8d9cbaf98f1fd4b9ea7a86e47491ac996384bf4\ntrain_filenames: train.csv\nloader: santander_value_prediction.SantanderValuePredictionLoader\ndescription: |\n The Santander Value Prediction Challenge dataset.\n https://www.kaggle.com/c/santander-value-prediction-challenge\noutput_features:\n - name: target\n type: number\n"
},
{
"path": "ludwig/datasets/configs/sarcastic_headlines.yaml",
"content": "version: 1.0\nname: sarcastic_headlines\ntrain_filenames: Sarcasm_Headlines_Dataset.json\narchive_filenames: news-headlines-dataset-for-sarcasm-detection.zip\nsha256:\n news-headlines-dataset-for-sarcasm-detection.zip: 3728f0fbce563536c3c67ab92e343e3ebcdc5cf1feaf4980c3abd4e54109eb51\nkaggle_dataset_id: rmisra/news-headlines-dataset-for-sarcasm-detection\ndescription: A dataset to determine if a news headline is sarcastic or serious.\nloader: sarcastic_headlines.SarcasticHeadlinesLoader\ncolumns:\n - name: article_link\n type: category\n - name: headline\n type: text\n - name: is_sarcastic\n type: binary\noutput_features:\n - name: is_sarcastic\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/sarcos.yaml",
"content": "version: 1.0\nname: sarcos\ndownload_urls:\n - http://www.gaussianprocess.org/gpml/data/sarcos_inv.mat\n - http://www.gaussianprocess.org/gpml/data/sarcos_inv_test.mat\nsha256:\n sarcos_inv_test.mat: 161a59b5c3b4f4b404584323f181607b2acbe620eb134dc720760dc3f38f5cec\n sarcos_inv.mat: b8a249733253ba6097372fedee7696833fcf30de42037d5b4a7227f21a6d1d97\ntrain_filenames: sarcos_inv.mat\ntest_filenames: sarcos_inv_test.mat\nloader: sarcos.SarcosLoader\ndescription: |\n The data relates to an inverse dynamics problem for a seven\n degrees-of-freedom SARCOS anthropomorphic robot arm.\n The task is to map from a 21-dimensional input space\n (7 joint positions, 7 joint velocities, 7 joint accelerations)\n to the corresponding 7 joint torques.\n http://gaussianprocess.org/gpml/data/\ncolumns:\n - name: position_1\n type: number\n - name: position_2\n type: number\n - name: position_3\n type: number\n - name: position_4\n type: number\n - name: position_5\n type: number\n - name: position_6\n type: number\n - name: position_7\n type: number\n - name: velocity_1\n type: number\n - name: velocity_2\n type: number\n - name: velocity_3\n type: number\n - name: velocity_4\n type: number\n - name: velocity_5\n type: number\n - name: velocity_6\n type: number\n - name: velocity_7\n type: number\n - name: acceleration_1\n type: number\n - name: acceleration_2\n type: number\n - name: acceleration_3\n type: number\n - name: acceleration_4\n type: number\n - name: acceleration_5\n type: number\n - name: acceleration_6\n type: number\n - name: acceleration_7\n type: number\n - name: torque_1\n type: number\n - name: torque_2\n type: number\n - name: torque_3\n type: number\n - name: torque_4\n type: number\n - name: torque_5\n type: number\n - name: torque_6\n type: number\n - name: torque_7\n type: number\noutput_features:\n- name: torque_1\n type: number\nfallback_mirrors:\n - name: predibase\n download_paths:\n - s3://ludwig-tests/ludwig_backup/sarcos_inv.mat\n - s3://ludwig-tests/ludwig_backup/sarcos_inv_test.mat\n"
},
{
"path": "ludwig/datasets/configs/sst2.yaml",
"content": "version: 1.0\nname: sst2\ndownload_urls: https://nlp.stanford.edu/~socherr/stanfordSentimentTreebank.zip\nsha256:\n stanfordSentimentTreebank.zip: 3f5209483b46bbf129cacbbbe6ae02fe780407034f61cf6342b7833257c3f1db\ntrain_filenames: train.csv\nvalidation_filenames: dev.csv\ntest_filenames: test.csv\nloader: sst.SST2Loader\ndescription: |\n The SST2 dataset.\n\n This dataset is constructed using the Stanford Sentiment Treebank Dataset.\n This dataset contains binary labels (positive or negative) for each sample.\n\n The original dataset specified 5 labels:\n very negative, negative, neutral, positive, very positive with\n the following cutoffs:\n [0, 0.2], (0.2, 0.4], (0.4, 0.6], (0.6, 0.8], (0.8, 1.0]\n\n In the construction of this dataset, we remove all neutral phrases\n and assign a negative label if the original rating falls\n into the following range: [0, 0.4] and a positive label\n if the original rating is between (0.6, 1.0].\ncolumns:\n - name: sentence\n type: text\n - name: label\n type: binary\noutput_features:\n - name: label\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/sst3.yaml",
"content": "version: 1.0\nname: sst3\ndownload_urls: https://nlp.stanford.edu/~socherr/stanfordSentimentTreebank.zip\nsha256:\n stanfordSentimentTreebank.zip: 3f5209483b46bbf129cacbbbe6ae02fe780407034f61cf6342b7833257c3f1db\ntrain_filenames: train.csv\nvalidation_filenames: dev.csv\ntest_filenames: test.csv\nloader: sst.SST3Loader\ndescription: |\n The SST3 dataset.\n\n This dataset is constructed using the Stanford Sentiment Treebank Dataset.\n The original dataset contains five labels (very negative, negative, neutral,\n positive, very positive) for each sample.\n\n In this dataset, the 3 labels negative, neutral, positive have the following cutoffs:\n [0, 0.4], (0.4, 0.6], (0.6, 1.0]\ncolumns:\n - name: sentence\n type: text\n - name: label\n type: category\noutput_features:\n - name: label\n type: category\n"
},
{
"path": "ludwig/datasets/configs/sst5.yaml",
"content": "version: 1.0\nname: sst5\ndownload_urls: https://nlp.stanford.edu/~socherr/stanfordSentimentTreebank.zip\nsha256:\n stanfordSentimentTreebank.zip: 3f5209483b46bbf129cacbbbe6ae02fe780407034f61cf6342b7833257c3f1db\ntrain_filenames: train.csv\nvalidation_filenames: dev.csv\ntest_filenames: test.csv\nloader: sst.SST5Loader\ndescription: |\n The SST5 dataset.\n\n This dataset is constructed using the Stanford Sentiment Treebank Dataset.\n This dataset contains five labels (very negative, negative, neutral,\n positive, very positive) for each sample.\n\n In the original dataset, the 5 labels: very negative, negative, neutral, positive,\n and very positive have the following cutoffs:\n [0, 0.2], (0.2, 0.4], (0.4, 0.6], (0.6, 0.8], (0.8, 1.0]\ncolumns:\n - name: sentence\n type: text\n - name: label\n type: category\noutput_features:\n - name: label\n type: category\n"
},
{
"path": "ludwig/datasets/configs/synthetic_fraud.yaml",
"content": "version: 1.0\nname: synthetic_fraud\nkaggle_dataset_id: ealaxi/paysim1\narchive_filenames: paysim1.zip\nsha256:\n paysim1.zip: f7eef9ffad5cfa64a034143a5c9b30491d189420b273d5ad5723ca40b596613d\ndescription: |\n The Synthetic Financial Datasets For Fraud Detection dataset.\n https://www.kaggle.com/ealaxi/paysim1\ncolumns:\n - name: step\n type: category\n - name: type\n type: category\n - name: amount\n type: number\n - name: nameOrig\n type: category\n - name: oldbalanceOrg\n type: number\n - name: newbalanceOrig\n type: number\n - name: nameDest\n type: category\n - name: oldbalanceDest\n type: number\n - name: newbalanceDest\n type: number\n - name: isFraud\n type: binary\n - name: isFlaggedFraud\n type: binary\noutput_features:\n - name: isFraud\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/talkingdata_adtrack_fraud.yaml",
"content": "version: 1.0\nname: talkingdata_adtrack_fraud_detection\nkaggle_competition: talkingdata-adtracking-fraud-detection\narchive_filenames: talkingdata-adtracking-fraud-detection.zip\nsha256:\n talkingdata-adtracking-fraud-detection.zip: 4441bea984e936db153aba30627b222cb1685021efb887bd22d78771fb793735\ntrain_filenames: train.csv\ndescription: |\n TalkingData AdTracking Fraud Detection Challenge.\n https://www.kaggle.com/competitions/talkingdata-adtracking-fraud-detection/overview\noutput_features:\n - name: is_attributed\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/telco_customer_churn.yaml",
"content": "version: 1.0\nname: telco_customer_churn\nkaggle_dataset_id: blastchar/telco-customer-churn\narchive_filenames: telco-customer-churn.zip\ndataset_filenames: WA_Fn-UseC_-Telco-Customer-Churn.csv\nsha256:\n telco-customer-churn.zip: cf7e6dcd8a238ecaa841a7d133142525453992d8d5e3ef6d1e5f0d359e7bf444\ndescription: |\n The Telco customer churn data contains information about a fictional telco company\n that provided home phone and Internet services to customers. Each row represents a\n customer, each column contains customer’s attributes described on the column Metadata.\n https://www.kaggle.com/datasets/blastchar/telco-customer-churn\ncolumns:\n - name: customerID\n type: category\n - name: gender\n type: binary\n - name: SeniorCitizen\n type: binary\n - name: Partner\n type: binary\n - name: Dependents\n type: binary\n - name: tenure\n type: number\n - name: PhoneService\n type: binary\n - name: MultipleLines\n type: category\n - name: InternetService\n type: category\n - name: OnlineSecurity\n type: category\n - name: OnlineBackup\n type: category\n - name: DeviceProtection\n type: category\n - name: TechSupport\n type: category\n - name: StreamingTV\n type: category\n - name: StreamingMovies\n type: category\n - name: Contract\n type: category\n - name: PaperlessBilling\n type: binary\n - name: PaymentMethod\n type: category\n - name: MonthlyCharges\n type: number\n - name: TotalCharges\n type: number\n - name: Churn\n type: binary\noutput_features:\n - name: Churn\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/temperature.yaml",
"content": "version: 1.0\nname: temperature\nkaggle_dataset_id: selfishgene/historical-hourly-weather-data\narchive_filenames: historical-hourly-weather-data.zip\nsha256:\n historical-hourly-weather-data.zip: db40ffce67318f366115b82a6f693d6dc82c808f23514e2ddae56c0434f606d7\ndataset_filenames: temperature.csv\ndescription: |\n Hourly temperature dataset from Kaggle\n https://www.kaggle.com/selfishgene/historical-hourly-weather-data\ncolumns:\n - name: datetime\n type: date\n - name: Vancouver\n type: number\n - name: Portland\n type: number\n - name: San Francisco\n type: number\n - name: Seattle\n type: number\n - name: Los Angeles\n type: number\n - name: San Diego\n type: number\n - name: Las Vegas\n type: number\n - name: Phoenix\n type: number\n - name: Albuquerque\n type: number\n - name: Denver\n type: number\n - name: San Antonio\n type: number\n - name: Dallas\n type: number\n - name: Houston\n type: number\n - name: Kansas City\n type: number\n - name: Minneapolis\n type: number\n - name: Saint Louis\n type: number\n - name: Chicago\n type: number\n - name: Nashville\n type: number\n - name: Indianapolis\n type: number\n - name: Atlanta\n type: number\n - name: Detroit\n type: number\n - name: Jacksonville\n type: number\n - name: Charlotte\n type: number\n - name: Miami\n type: number\n - name: Pittsburgh\n type: number\n - name: Toronto\n type: number\n - name: Philadelphia\n type: number\n - name: New York\n type: number\n - name: Montreal\n type: number\n - name: Boston\n type: number\n - name: Beersheba\n type: number\n - name: Tel Aviv District\n type: number\n - name: Eilat\n type: number\n - name: Haifa\n type: number\n - name: Nahariyya\n type: number\n - name: Jerusalem\n type: number\noutput_features:\n - name: San Francisco\n type: number\n"
},
{
"path": "ludwig/datasets/configs/titanic.yaml",
"content": "version: 1.0\nname: titanic\nkaggle_competition: titanic\narchive_filenames: titanic.zip\nsha256:\n titanic.zip: bb1bda464cc6819d412b41d34be69fd89d26b372dc24c09421c3dbca1b0dbe9f\ntrain_filenames: train.csv\ntest_filenames: test.csv\ndescription: |\n The Titanic dataset: use machine learning to create a model\n that predicts which passengers survived the Titanic shipwreck.\n https://www.kaggle.com/c/titanic\noutput_features:\n - name: Survived\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/twitter_bots.yaml",
"content": "version: 1.0\nname: twitter_bots\nkaggle_dataset_id: danieltreiman/twitter-human-bots-dataset\narchive_filenames: twitter-human-bots-dataset.zip\ndataset_filenames: twitter_human_bots_dataset.csv\nsha256:\n twitter-human-bots-dataset.zip: 16ffaad719ebb9688231844a80f92901c5efb1ff96eafeb869dc5de07b323cdd\npreserve_paths:\n - profile_images\n - profile_background_images\ndescription: |\n A dataset for Twitter Bot account detection.\n https://www.kaggle.com/datasets/davidmartngutirrez/twitter-bots-accounts\ncolumns:\n - name: created_at\n type: date\n - name: default_profile\n type: binary\n - name: default_profile_image\n type: binary\n - name: description\n type: text\n - name: favourites_count\n type: number\n - name: followers_count\n type: number\n - name: friends_count\n type: number\n - name: geo_enabled\n type: binary\n - name: id\n type: category\n - name: lang\n type: category\n - name: location\n type: category\n - name: profile_background_image_url\n type: category\n - name: profile_image_url\n type: category\n - name: screen_name\n type: category\n - name: statuses_count\n type: number\n - name: verified\n type: binary\n - name: average_tweets_per_day\n type: number\n - name: account_age_days\n type: number\n - name: account_type\n type: category\n - name: profile_image_path\n type: image\n - name: profile_background_image_path\n type: image\noutput_features:\n - name: account_type\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/walmart_recruiting.yaml",
"content": "version: 1.0\nname: walmart_recruiting\nkaggle_competition: walmart-recruiting-trip-type-classification\narchive_filenames: walmart-recruiting-trip-type-classification.zip\nsha256:\n walmart-recruiting-trip-type-classification.zip: 4c0ad71034d0b907e018adcb00c7b2835d2c30abe770fde5ce8719d7b89d4de6\ntrain_filenames: train.csv\ndescription: |\n Walmart Recruiting: Trip Type Classification\n https://www.kaggle.com/c/walmart-recruiting-trip-type-classification\noutput_features:\n - name: TripType\n type: category\n"
},
{
"path": "ludwig/datasets/configs/wine_reviews.yaml",
"content": "version: 1.0\nname: wine_reviews\ndownload_urls:\n - https://automl-mm-bench.s3.amazonaws.com/wine_reviews/train.csv\n - https://automl-mm-bench.s3.amazonaws.com/wine_reviews/test.csv\nsha256:\n test.csv: c862d1af572659406ab39356a25c7d5e9b7c8570a89e069311fca1abb6bf1849\n train.csv: c54101bb07571a3df0723e93a5f7c48123dd792b316396db4404a04bcf1809cb\ntrain_filenames: train.csv\ntest_filenames: test.csv\ndescription: |\n Wine Reviews\n 130k wine reviews with variety, location, winery, price, and description\n https://www.kaggle.com/datasets/zynicide/wine-reviews\ncolumns:\n - name: country\n type: category\n - name: description\n type: text\n - name: points\n type: number\n - name: price\n type: number\n - name: province\n type: category\n - name: variety\n type: category\noutput_features:\n - name: points\n type: number\n"
},
{
"path": "ludwig/datasets/configs/wmt15.yaml",
"content": "version: 1.0\nname: wmt15\nkaggle_dataset_id: dhruvildave/en-fr-translation-dataset\narchive_filenames: en-fr-translation-dataset.zip\nsha256:\n en-fr-translation-dataset.zip: 5fb911b327f2f36ea32315b4754f6aef95e6830562eec7054d31d614dd53d93c\ndescription: |\n French/English parallel texts for training translation models.\n Over 22.5 million sentences in French and English.\n https://www.kaggle.com/dhruvildave/en-fr-translation-dataset\noutput_features:\n - name: en\n type: text\n"
},
{
"path": "ludwig/datasets/configs/women_clothing_review.yaml",
"content": "version: 1.0\nname: women_clothing_review\ndownload_urls:\n - https://automl-mm-bench.s3.amazonaws.com/women_clothing_review/train.pq\n - https://automl-mm-bench.s3.amazonaws.com/women_clothing_review/test.pq\nsha256:\n test.pq: 477de72fe7e672ef87e1eca00de312f55ba884a9b80fbd04fa79c0d0159e5593\n train.pq: 1b3d248397cee76a6ccff814560f29ae3d66eeb26a6e97ac0837e021629bc740\ntrain_filenames: train.pq\ntest_filenames: test.pq\ndescription: |\n Women's E-Commerce Clothing Reviews\n 23,000 Customer Reviews and Ratings\n https://www.kaggle.com/nicapotato/womens-ecommerce-clothing-reviews\ncolumns:\n - name: Clothing ID\n type: category\n - name: Age\n type: number\n - name: Title\n type: text\n - name: Review Text\n type: text\n - name: Rating\n type: number\n - name: Recommended IND\n type: binary\n - name: Positive Feedback Count\n type: number\n - name: Division Name\n type: category\n - name: Department Name\n type: category\n - name: Class Name\n type: category\noutput_features:\n - name: Rating\n type: number\n"
},
{
"path": "ludwig/datasets/configs/yahoo_answers.yaml",
"content": "version: 1.0\nname: yahoo_answers\ndownload_urls: https://s3.amazonaws.com/fast-ai-nlp/yahoo_answers_csv.tgz\nsha256:\n yahoo_answers_csv.tgz: 2d4277855faf8b35259009425fa8f7fe1888b5644b47165508942d000f4c96ae\ntrain_filenames: yahoo_answers_csv/train.csv\ntest_filenames: yahoo_answers_csv/test.csv\ndescription: |\n The Yahoo Answers dataset\n Details:\n The 10 largest main categories from the Yahoo! Answers \\\n Comprehensive Questions and Answers version 1.0 dataset. \\\n Each class contains 140,000 training samples and 5,000 \\\n testing samples.\n Dataset source:\n Character-level Convolutional Networks for Text Classification\n Xiang Zhang et al., 2015\n https://arxiv.org/abs/1509.01626\ncolumns:\n - name: label\n type: category\n - name: question_title\n type: text\n - name: question\n type: text\n - name: best_answer\n type: text\noutput_features:\n - name: label\n type: category\n"
},
{
"path": "ludwig/datasets/configs/yelp_review_polarity.yaml",
"content": "version: 1.0\nname: yelp_review_polarity\ndownload_urls: https://s3.amazonaws.com/fast-ai-nlp/yelp_review_polarity_csv.tgz\nsha256:\n yelp_review_polarity_csv.tgz: 528f22e286cad085948acbc3bea7e58188416546b0e364d0ae4ca0ce666abe35\ntrain_filenames: yelp_review_polarity_csv/train.csv\ntest_filenames: yelp_review_polarity_csv/test.csv\ndescription: |\n The Yelp Polarity dataset\n Details:\n 1,569,264 samples from the Yelp Dataset Challenge 2015.\n This subset has 280,000 training samples and 19,000 test samples\n in each polarity.\n Dataset source:\n Character-level Convolutional Networks for Text Classification\n Xiang Zhang et al., 2015\ncolumns:\n - name: label\n type: binary\n - name: text\n type: text\noutput_features:\n - name: label\n type: binary\n"
},
{
"path": "ludwig/datasets/configs/yelp_reviews.yaml",
"content": "version: 1.0\nname: yelp_reviews\ndownload_urls: https://s3.amazonaws.com/fast-ai-nlp/yelp_review_full_csv.tgz\nsha256:\n yelp_review_full_csv.tgz: 56006b0a17a370f1e366504b1f2c3e3754e4a3dda17d3e718a885c552869a559\ntrain_filenames: yelp_review_full_csv/train.csv\ntest_filenames: yelp_review_full_csv/test.csv\ndescription: |\n The Yelp Reviews dataset\n Details:\n 1,569,264 samples from the Yelp Dataset Challenge 2015.\n This subset has 130,000 training samples and 10,000\n testing samples in each star rating.\n Dataset source:\n Character-level Convolutional Networks for Text Classification\n Xiang Zhang et al., 2015\ncolumns:\n - name: label\n type: category\n - name: text\n type: text\noutput_features:\n - name: label\n type: category\n"
},
{
"path": "ludwig/datasets/configs/yosemite.yaml",
"content": "version: 1.0\nname: yosemite\ndownload_urls: https://raw.githubusercontent.com/ourownstory/neuralprophet-data/main/datasets_raw/yosemite_temps.csv\nsha256:\n yosemite_temps.csv: c0ec9f2cb4bbf0bc53f7bfd2e39f88ae21e43b7b8912b2d1eb8185055f9510e2\ndescription: |\n Yosemite temperatures dataset.\n As found in https://github.com/ourownstory/neural_prophet\ncolumns:\n - name: ds\n type: date\n - name: y\n type: number\noutput_features:\n - name: y\n type: number\n"
},
{
"path": "ludwig/datasets/dataset_config.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nfrom dataclasses import dataclass, field\n\nfrom dataclasses_json import dataclass_json\n\n\n@dataclass_json\n@dataclass\nclass DatasetFallbackMirror:\n # Name of the mirror\n name: str\n\n # List of paths to download from. Must map 1:1 to DatasetConfig.download_urls or to the archive_filenames\n # that we get from Kaggle.\n download_paths: str | list[str]\n\n\n@dataclass_json\n@dataclass\nclass DatasetConfig:\n \"\"\"The configuration of a Ludwig dataset.\"\"\"\n\n # The version of the dataset.\n version: str\n\n # The name of the dataset. Make this a valid python module name, should not contain spaces or dashes.\n name: str\n\n # The readable description of the dataset\n description: str = \"\"\n\n # Fallback mirrors. Paths must be in local/remote filesystems.\n fallback_mirrors: list[DatasetFallbackMirror] | None = None\n\n # Optional. The (suggested) output features for this dataset. Helps users discover new datasets and filter for\n # relevance to a specific machine learning setting.\n output_features: list[dict] = field(default_factory=list)\n\n # The kaggle competition this dataset belongs to, or None if this dataset is not hosted by a Kaggle competition.\n kaggle_competition: str | None = None\n\n # The kaggle dataset ID, or None if this dataset if not hosted by Kaggle.\n kaggle_dataset_id: str | None = None\n\n # The list of URLs to download.\n download_urls: str | list[str] = field(default_factory=list)\n\n # The list of file archives which will be downloaded. If download_urls contains a filename with extension, for\n # example https://domain.com/archive.zip, then archive_filenames does not need to be specified.\n archive_filenames: str | list[str] = field(default_factory=list)\n\n # The names of files in the dataset (after extraction). Glob-style patterns are supported, see\n # https://docs.python.org/3/library/glob.html\n dataset_filenames: str | list[str] = field(default_factory=list)\n\n # If the dataset contains separate files for training, testing, or validation. Glob-style patterns are supported,\n # see https://docs.python.org/3/library/glob.html\n train_filenames: str | list[str] = field(default_factory=list)\n validation_filenames: str | list[str] = field(default_factory=list)\n test_filenames: str | list[str] = field(default_factory=list)\n\n # If the dataset contains additional referenced files or directories (ex. images or audio) list them here and they\n # will be copied to the same location as the processed dataset. Glob-style patterns are supported,\n # see https://docs.python.org/3/library/glob.html\n preserve_paths: str | list[str] = field(default_factory=list)\n\n # Optionally verify integrity of the dataset by providing sha256 checksums for important files. Maps filename to\n # sha256 digest. Use `sha256sum ` on linux, `shasum -a 256 ` on Mac to get checksums.\n # If verification fails, loading the dataset will fail with a ValueError.\n # If no sha256 digests are in the config, a warning is logged and the dataset will load without verification.\n sha256: dict[str, str] = field(default_factory=dict)\n\n # List of column names, for datasets which do not have column names. If specified, will override the column names\n # already present in the dataset.\n columns: list[dict] = field(default_factory=list)\n\n # Optional dictionary which maps column name to column type. Column's will be converted to the requested type, or\n # will be inferred from the dataset by default.\n column_types: dict[str, str] = field(default_factory=dict)\n\n # The loader module and class to use, relative to ludwig.datasets.loaders. Only change this if the dataset requires\n # processing which is not handled by the default loader.\n loader: str = \"dataset_loader.DatasetLoader\"\n"
},
{
"path": "ludwig/datasets/kaggle.py",
"content": "import os\nfrom contextlib import contextmanager\n\nfrom ludwig.utils.fs_utils import upload_output_directory\n\n\ndef create_kaggle_client():\n # Need to import here to prevent Kaggle from authenticating on import\n from kaggle import api\n\n return api\n\n\n@contextmanager\ndef update_env(**kwargs):\n override_env = {k: v for k, v in kwargs.items() if v is not None}\n old = os.environ.copy()\n try:\n os.environ.update(override_env)\n yield\n finally:\n os.environ = old\n\n\ndef download_kaggle_dataset(\n download_directory: str,\n kaggle_dataset_id: str | None = None,\n kaggle_competition: str | None = None,\n kaggle_username: str | None = None,\n kaggle_key: str | None = None,\n):\n \"\"\"Download all files in a kaggle dataset. One of kaggle_dataset_id,\n\n If the user has not specified creds in the kaggle.json file we lookup the passed in username and the api key and\n perform authentication.\n \"\"\"\n with update_env(KAGGLE_USERNAME=kaggle_username, KAGGLE_KEY=kaggle_key):\n # Call authenticate explicitly to pick up new credentials if necessary\n api = create_kaggle_client()\n api.authenticate()\n with upload_output_directory(download_directory) as (tmpdir, _):\n if kaggle_competition:\n api.competition_download_files(kaggle_competition, path=tmpdir)\n else:\n api.dataset_download_files(kaggle_dataset_id, path=tmpdir)\n return [os.path.join(download_directory, f) for f in os.listdir(download_directory)]\n"
},
{
"path": "ludwig/datasets/loaders/__init__.py",
"content": ""
},
{
"path": "ludwig/datasets/loaders/adult_census_income.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport pandas as pd\n\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\n\n\nclass AdultCensusIncomeLoader(DatasetLoader):\n def load_file_to_dataframe(self, file_path: str) -> pd.DataFrame:\n if file_path.endswith(\".test\"):\n # The test file contains the line \"|1x3 Cross validator\" before the CSV content.\n return pd.read_csv(file_path, skiprows=1)\n return super().load_file_to_dataframe(file_path)\n\n def transform_dataframe(self, dataframe: pd.DataFrame) -> pd.DataFrame:\n processed_df = super().transform_dataframe(dataframe)\n processed_df[\"income\"] = processed_df[\"income\"].str.rstrip(\".\")\n processed_df[\"income\"] = processed_df[\"income\"].str.strip()\n return processed_df\n"
},
{
"path": "ludwig/datasets/loaders/agnews.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport pandas as pd\n\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\n\n\nclass AGNewsLoader(DatasetLoader):\n def transform_dataframe(self, dataframe: pd.DataFrame) -> pd.DataFrame:\n processed_df = super().transform_dataframe(dataframe)\n # Maps class_index to class name.\n class_names = [\"\", \"world\", \"sports\", \"business\", \"sci_tech\"]\n # Adds new column 'class' by mapping class indexes to strings.\n processed_df[\"class\"] = processed_df.class_index.apply(lambda i: class_names[i])\n # Agnews has no validation split, only train and test (0, 2). For convenience, we'll designate the first 5% of\n # each class from the training set as the validation set.\n val_set_n = int((len(processed_df) * 0.05) // len(class_names)) # rows from each class in validation set.\n for ci in range(1, 5):\n # For each class, reassign the first val_set_n rows of the training set to validation set.\n train_rows = processed_df[(processed_df.split == 0) & (processed_df.class_index == ci)].index\n processed_df.loc[train_rows[:val_set_n], \"split\"] = 1\n return processed_df\n"
},
{
"path": "ludwig/datasets/loaders/allstate_claims_severity.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport os\n\nimport pandas as pd\n\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\n\n\nclass AllstateClaimsSeverityLoader(DatasetLoader):\n def load_file_to_dataframe(self, file_path: str) -> pd.DataFrame:\n if os.path.basename(file_path) == \"train.csv\":\n # train.csv has been updated with quoted test rows at the end; don't load these, only load the original\n # training set.\n return pd.read_csv(file_path, nrows=188319)\n if os.path.basename(file_path) == \"test.csv\":\n # we limit the loaded rows for the same reason as the training set.\n return pd.read_csv(file_path, nrows=125547)\n super().load_file_to_dataframe(file_path)\n"
},
{
"path": "ludwig/datasets/loaders/camseq.py",
"content": "# Copyright (c) 2023 Aizen Corp.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport os\n\nimport pandas as pd\n\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\nfrom ludwig.utils.fs_utils import makedirs\n\n\nclass CamseqLoader(DatasetLoader):\n def transform_files(self, file_paths: list[str]) -> list[str]:\n if not os.path.exists(self.processed_dataset_dir):\n os.makedirs(self.processed_dataset_dir)\n\n # move images and masks into separate directories\n source_dir = self.raw_dataset_dir\n images_dir = os.path.join(source_dir, \"images\")\n masks_dir = os.path.join(source_dir, \"masks\")\n makedirs(images_dir, exist_ok=True)\n makedirs(masks_dir, exist_ok=True)\n\n data_files = []\n for f in os.listdir(source_dir):\n if f.endswith(\"_L.png\"): # masks\n dest_file = os.path.join(masks_dir, f)\n elif f.endswith(\".png\"): # images\n dest_file = os.path.join(images_dir, f)\n else:\n continue\n source_file = os.path.join(source_dir, f)\n os.replace(source_file, dest_file)\n data_files.append(dest_file)\n\n return super().transform_files(data_files)\n\n def load_unprocessed_dataframe(self, file_paths: list[str]) -> pd.DataFrame:\n \"\"\"Creates a dataframe of image paths and mask paths.\"\"\"\n images_dir = os.path.join(self.processed_dataset_dir, \"images\")\n masks_dir = os.path.join(self.processed_dataset_dir, \"masks\")\n images = []\n masks = []\n for f in os.listdir(images_dir):\n images.append(os.path.join(images_dir, f))\n mask_f = f[:-4] + \"_L.png\"\n masks.append(os.path.join(masks_dir, mask_f))\n\n return pd.DataFrame({\"image_path\": images, \"mask_path\": masks})\n"
},
{
"path": "ludwig/datasets/loaders/code_alpaca_loader.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nimport pandas as pd\n\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\n\n\nclass CodeAlpacaLoader(DatasetLoader):\n \"\"\"The Code Alpaca dataset.\"\"\"\n\n def load_file_to_dataframe(self, file_path: str) -> pd.DataFrame:\n \"\"\"Loads a file into a dataframe.\"\"\"\n df = pd.read_json(file_path)\n return df\n"
},
{
"path": "ludwig/datasets/loaders/consumer_complaints_loader.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\nimport pandas as pd\n\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\n\n\nclass ConsumerComplaintsLoader(DatasetLoader):\n \"\"\"The Consumer Complaints dataset.\"\"\"\n\n def load_file_to_dataframe(self, file_path: str) -> pd.DataFrame:\n \"\"\"Loads a file into a dataframe.\"\"\"\n\n consumer_complaints_df = pd.read_csv(file_path)\n consumer_complaints_df = preprocess_df(consumer_complaints_df)\n\n return consumer_complaints_df\n\n\ndef preprocess_df(df):\n \"\"\"Preprocesses the dataframe.\n\n - Remove all rows with missing values in the following columns:\n - Consumer complaint narrative\n - Issue\n - Product\n\n Args:\n df (pd.DataFrame): The dataframe to preprocess.\n\n Returns:\n pd.DataFrame: The preprocessed dataframe.\n \"\"\"\n return df.dropna(subset=[\"Consumer complaint narrative\", \"Issue\", \"Product\"])\n"
},
{
"path": "ludwig/datasets/loaders/creditcard_fraud.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport pandas as pd\n\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\n\n\nclass CreditCardFraudLoader(DatasetLoader):\n def transform_dataframe(self, dataframe: pd.DataFrame) -> pd.DataFrame:\n processed_df = super().transform_dataframe(dataframe)\n # Train/Test split like https://www.kaggle.com/competitions/1056lab-fraud-detection-in-credit-card/overview\n processed_df = processed_df.sort_values(by=[\"Time\"])\n processed_df.loc[:198365, \"split\"] = 0\n processed_df.loc[198365:, \"split\"] = 2\n processed_df.split = processed_df.split.astype(int)\n return processed_df\n"
},
{
"path": "ludwig/datasets/loaders/dataset_loader.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nfrom __future__ import annotations\n\nimport glob\nimport hashlib\nimport logging\nimport os\nimport shutil\nimport urllib\nfrom enum import Enum\nfrom urllib.parse import urlparse\n\nimport pandas as pd\nfrom tqdm import tqdm\n\nfrom ludwig.api_annotations import DeveloperAPI, PublicAPI\nfrom ludwig.constants import SPLIT\nfrom ludwig.datasets.archives import extract_archive, is_archive, list_archive\nfrom ludwig.datasets.dataset_config import DatasetConfig, DatasetFallbackMirror\nfrom ludwig.datasets.kaggle import download_kaggle_dataset\nfrom ludwig.datasets.utils import model_configs_for_dataset\nfrom ludwig.utils.fs_utils import get_default_cache_location, get_fs_and_path\nfrom ludwig.utils.strings_utils import make_safe_filename\n\nlogger = logging.getLogger(__name__)\n\n\n@DeveloperAPI\nclass TqdmUpTo(tqdm):\n \"\"\"Provides progress bar for `urlretrieve`.\n\n Taken from: https://gist.github.com/leimao/37ff6e990b3226c2c9670a2cd1e4a6f5\n \"\"\"\n\n def update_to(self, b=1, bsize=1, tsize=None):\n \"\"\"\n b : int, optional\n Number of blocks transferred so far [default: 1].\n bsize : int, optional\n Size of each block (in tqdm units) [default: 1].\n tsize : int, optional\n Total size (in tqdm units). If [default: None] remains unchanged.\n \"\"\"\n if tsize is not None:\n self.total = tsize # noqa W0201\n self.update(b * bsize - self.n) # will also set self.n = b * bsize\n\n\ndef _list_of_strings(list_or_string: str | list[str]) -> list[str]:\n \"\"\"Helper function to accept single string or lists in config.\"\"\"\n return [list_or_string] if isinstance(list_or_string, str) else list_or_string\n\n\ndef _glob_multiple(pathnames: list[str], root_dir: str = None, recursive: bool = True) -> set[str]:\n \"\"\"Recursive glob multiple patterns, returns set of matches.\n\n Note: glob's root_dir argument was added in python 3.10, not using it for compatibility.\n \"\"\"\n if root_dir:\n pathnames = [os.path.join(root_dir, p) for p in pathnames]\n return set().union(*[glob.glob(p, recursive=recursive) for p in pathnames])\n\n\ndef _sha256_digest(file_path) -> str:\n \"\"\"Returns the sha256 digest for the specified file.\"\"\"\n hash = hashlib.sha256()\n buffer = bytearray(hash.block_size * 1024) # Attempts to read in multiples of the hash block size (64KB).\n mv = memoryview(buffer)\n with open(file_path, \"rb\", buffering=0) as f:\n for bytes_read in iter(lambda: f.readinto(mv), 0):\n hash.update(mv[:bytes_read])\n return hash.hexdigest()\n\n\n@PublicAPI\nclass DatasetState(int, Enum):\n \"\"\"The state of the dataset.\"\"\"\n\n NOT_LOADED = 0\n DOWNLOADED = 1\n EXTRACTED = 2\n TRANSFORMED = 3\n\n\n@PublicAPI\nclass DatasetLoader:\n \"\"\"Base class that defines the default pipeline for loading a ludwig dataset.\n\n Clients will typically call load(), which processes the dataset according to the config.\n\n A dataset is processed in 4 phases:\n 1. Download - The dataset files are downloaded to the cache.\n 2. Verify - Hashes of downloaded files are verified.\n 3. Extract - The dataset files are extracted from an archive (may be a no-op if data is not archived).\n 4. Transform - The dataset is transformed into a format usable for training and is ready to load.\n a. Transform Files (Files -> Files)\n b. Load Dataframe (Files -> DataFrame)\n c. Transform Dataframe (DataFrame -> DataFrame)\n d. Save Processed (DataFrame -> File)\n\n The download and extract phases are run for each URL based on the URL type and file extension. After extraction, the\n full set of downloaded and extracted files are collected and passed as a list to the transform stage.\n\n The transform phase offers customization points for datasets which require preprocessing before they are usable for\n training.\n \"\"\"\n\n def __init__(self, config: DatasetConfig, cache_dir: str | None = None):\n \"\"\"Constructor.\"\"\"\n self.config = config\n self.cache_dir = cache_dir if cache_dir else get_default_cache_location()\n\n @property\n def name(self):\n \"\"\"The name of the dataset.\"\"\"\n return self.config.name\n\n @property\n def version(self):\n \"\"\"The version of the dataset.\"\"\"\n return self.config.version\n\n @property\n def is_kaggle_dataset(self) -> bool:\n return self.config.kaggle_dataset_id or self.config.kaggle_competition\n\n @property\n def download_dir(self) -> str:\n \"\"\"Directory where all dataset artifacts are saved.\"\"\"\n return os.path.join(self.cache_dir, f\"{self.name}_{self.version}\")\n\n @property\n def raw_dataset_dir(self) -> str:\n \"\"\"Save path for raw data downloaded from the web.\"\"\"\n return os.path.join(self.download_dir, \"raw\")\n\n @property\n def processed_dataset_dir(self) -> str:\n \"\"\"Save path for processed data.\"\"\"\n return os.path.join(self.download_dir, \"processed\")\n\n @property\n def processed_dataset_filename(self) -> str:\n \"\"\"Filename for processed data.\"\"\"\n return f\"{make_safe_filename(self.config.name)}.parquet\"\n\n @property\n def processed_dataset_path(self) -> str:\n \"\"\"Save path to processed dataset file.\"\"\"\n return os.path.join(self.processed_dataset_dir, self.processed_dataset_filename)\n\n @property\n def processed_temp_dir(self) -> str:\n \"\"\"Save path for processed temp data.\"\"\"\n return os.path.join(self.download_dir, \"_processed\")\n\n @property\n def state(self) -> DatasetState:\n \"\"\"Dataset state.\"\"\"\n if os.path.exists(self.processed_dataset_path):\n return DatasetState.TRANSFORMED\n if all([os.path.exists(os.path.join(self.raw_dataset_dir, filename)) for filename in self.download_filenames]):\n archive_filenames = [f for f in self.download_filenames if is_archive(f)]\n if archive_filenames:\n # Check to see if archive has been extracted.\n extracted_files = [\n f for a in archive_filenames for f in list_archive(os.path.join(self.raw_dataset_dir, a))\n ]\n if all(os.path.exists(os.path.join(self.raw_dataset_dir, ef)) for ef in extracted_files):\n return DatasetState.EXTRACTED\n else:\n return DatasetState.DOWNLOADED\n # If none of the dataset download files are archives, skip extraction phase.\n return DatasetState.EXTRACTED\n return DatasetState.NOT_LOADED\n\n @property\n def download_urls(self) -> list[str]:\n return _list_of_strings(self.config.download_urls)\n\n @property\n def download_filenames(self) -> list[str]:\n \"\"\"Filenames for downloaded files inferred from download_urls.\"\"\"\n if self.config.archive_filenames:\n return _list_of_strings(self.config.archive_filenames)\n return [os.path.basename(urlparse(url).path) for url in self.download_urls]\n\n @staticmethod\n def get_mirror_download_paths(mirror: DatasetFallbackMirror):\n \"\"\"Filenames for downloaded files inferred from mirror download_paths.\"\"\"\n return _list_of_strings(mirror.download_paths)\n\n def get_mirror_download_filenames(self, mirror: DatasetFallbackMirror):\n \"\"\"Filenames for downloaded files inferred from mirror download_paths.\"\"\"\n if self.config.archive_filenames:\n return _list_of_strings(self.config.archive_filenames)\n return [os.path.basename(path) for path in mirror.download_paths]\n\n def description(self) -> str:\n \"\"\"Returns human-readable description of the dataset.\"\"\"\n return f\"{self.config.name} {self.config.version}\\n{self.config.description}\"\n\n @property\n def model_configs(self) -> dict[str, dict]:\n \"\"\"Returns a dictionary of built-in model configs for this dataset.\"\"\"\n return model_configs_for_dataset(self.config.name)\n\n @property\n def best_model_config(self) -> dict | None:\n \"\"\"Returns the best built-in model config for this dataset, or None.\"\"\"\n return self.model_configs.get(\"best\")\n\n @property\n def default_model_config(self) -> dict | None:\n \"\"\"Returns the default built-in model config for this dataset.\n\n This is a good first model which should train in under 10m on a current laptop without GPU acceleration.\n \"\"\"\n return self.model_configs.get(\"default\")\n\n def _get_preserved_paths(self, root_dir=None):\n \"\"\"Gets list of files to preserve when exporting dataset, not including self.processed_dataset_path.\n\n Returns paths relative to the dataset root directory.\n \"\"\"\n root_dir = root_dir if root_dir else self.processed_dataset_dir\n preserved_paths = _glob_multiple(_list_of_strings(self.config.preserve_paths), root_dir=root_dir)\n return [os.path.relpath(p, start=root_dir) for p in preserved_paths]\n\n def export(self, output_directory: str) -> None:\n \"\"\"Exports the dataset (and any files required by it) into the specified directory.\"\"\"\n self._download_and_process()\n os.makedirs(output_directory, exist_ok=True)\n shutil.copy2(self.processed_dataset_path, os.path.join(output_directory, self.processed_dataset_filename))\n preserve_paths = self._get_preserved_paths()\n for relative_path in preserve_paths:\n source = os.path.join(self.processed_dataset_dir, relative_path)\n destination = os.path.join(output_directory, relative_path)\n if os.path.isdir(source):\n shutil.copytree(source, destination, symlinks=False, dirs_exist_ok=True)\n else:\n shutil.copy2(source, destination)\n\n def _download_and_process(self, kaggle_username: str | None = None, kaggle_key: str | None = None):\n \"\"\"Loads the dataset, downloaded and processing it if needed.\n\n If dataset is already processed, does nothing.\n \"\"\"\n if self.state == DatasetState.NOT_LOADED:\n try:\n self.download(kaggle_username=kaggle_username, kaggle_key=kaggle_key)\n except Exception as e:\n logger.warning(\n f\"Finding fallback mirrors to download the dataset. Downloading from \"\n f\"the original source failed with the following error {e}.\"\n )\n if not self.config.fallback_mirrors:\n logger.exception(f\"No fallback mirror found. Failed to download dataset {self.config.name}.\")\n else:\n self.download_from_fallback_mirrors()\n self.verify()\n if self.state == DatasetState.DOWNLOADED:\n # Extract dataset\n try:\n self.extract()\n except Exception:\n logger.exception(\"Failed to extract dataset\")\n if self.state == DatasetState.EXTRACTED:\n # Transform dataset\n try:\n self.transform()\n except Exception:\n logger.exception(\"Failed to transform dataset\")\n\n def load(\n self, kaggle_username: str | None = None, kaggle_key: str | None = None, split: bool = False\n ) -> pd.DataFrame | list[pd.DataFrame, pd.DataFrame, pd.DataFrame]:\n \"\"\"Loads the dataset, downloaded and processing it if needed.\n\n Note: This method is also responsible for splitting the data, returning a single dataframe if split=False, and a\n 3-tuple of train, val, test if split=True.\n\n :param kaggle_username: (str) username on Kaggle platform\n :param kaggle_key: (str) dataset key on Kaggle platform\n :param split: (bool) splits dataset along 'split' column if present. The split column should always have values\n 0: train, 1: validation, 2: test.\n \"\"\"\n self._download_and_process(kaggle_username=kaggle_username, kaggle_key=kaggle_key)\n if self.state == DatasetState.TRANSFORMED:\n dataset_df = self.load_transformed_dataset()\n if split:\n return self.split(dataset_df)\n else:\n return dataset_df\n\n def download(self, kaggle_username: str | None = None, kaggle_key: str | None = None) -> list[str]:\n if not os.path.exists(self.raw_dataset_dir):\n os.makedirs(self.raw_dataset_dir)\n if self.is_kaggle_dataset:\n return download_kaggle_dataset(\n self.raw_dataset_dir,\n kaggle_dataset_id=self.config.kaggle_dataset_id,\n kaggle_competition=self.config.kaggle_competition,\n kaggle_username=kaggle_username,\n kaggle_key=kaggle_key,\n )\n else:\n for url, filename in zip(self.download_urls, self.download_filenames):\n downloaded_file_path = os.path.join(self.raw_dataset_dir, filename)\n with TqdmUpTo(unit=\"B\", unit_scale=True, unit_divisor=1024, miniters=1, desc=filename) as t:\n urllib.request.urlretrieve(url, downloaded_file_path, t.update_to)\n\n def download_from_fallback_mirrors(self):\n for mirror in self.config.fallback_mirrors:\n logger.info(f\"Attempting download from mirror {mirror.name}.\")\n try:\n download_paths = self.get_mirror_download_paths(mirror)\n filenames = self.get_mirror_download_filenames(mirror)\n for path, filename in zip(download_paths, filenames):\n downloaded_file_path = os.path.join(self.raw_dataset_dir, filename)\n with TqdmUpTo(unit=\"B\", unit_scale=True, unit_divisor=1024, miniters=1, desc=filename):\n fs, path = get_fs_and_path(path)\n fs.get(path, downloaded_file_path)\n return\n except Exception:\n logger.exception(f\"Download from mirror `{mirror.name}` failed.\")\n\n def verify(self) -> None:\n \"\"\"Verifies checksums for dataset.\"\"\"\n for filename, sha256sum in self.config.sha256.items():\n digest = _sha256_digest(os.path.join(self.raw_dataset_dir, filename))\n if digest != sha256sum:\n raise ValueError(f\"Checksum mismatch for file {filename} of {self.config.name} dataset\")\n if not self.config.sha256:\n logger.warning(f\"No sha256 digest provided for dataset {self.config.name}, cannot verify.\")\n logger.info(\"Contents:\")\n for filename in os.listdir(self.raw_dataset_dir):\n path = os.path.join(self.raw_dataset_dir, filename)\n if not os.path.isdir(path):\n digest = _sha256_digest(path)\n logger.info(f\" {filename}: {digest}\")\n\n def extract(self) -> list[str]:\n extracted_files = set()\n for download_filename in self.download_filenames:\n download_path = os.path.join(self.raw_dataset_dir, download_filename)\n if is_archive(download_path):\n extracted_files.update(extract_archive(download_path))\n # If the archive contains archives, extract those too. For example, bnp_claims_management.\n archive_contents = extracted_files.copy()\n for extracted_file in archive_contents:\n extracted_path = os.path.join(self.raw_dataset_dir, extracted_file)\n if is_archive(extracted_path):\n try:\n extracted_files.update(extract_archive(extracted_path))\n except RuntimeError as e:\n logger.warning(f\"Error extracting {extracted_file}\" + str(e))\n return list(extracted_files)\n\n def transform(self) -> None:\n data_filenames = [\n os.path.join(self.raw_dataset_dir, f) for f in os.listdir(self.raw_dataset_dir) if not is_archive(f)\n ]\n transformed_files = self.transform_files(data_filenames)\n unprocessed_dataframe = self.load_unprocessed_dataframe(transformed_files)\n transformed_dataframe = self.transform_dataframe(unprocessed_dataframe)\n self.save_processed(transformed_dataframe)\n\n def transform_files(self, file_paths: list[str]) -> list[str]:\n \"\"\"Transform data files before loading to dataframe.\n\n Subclasses should override this method to process files before loading dataframe, calling the base class\n implementation after transformation if the results of transformation are needed by preserve_paths.\n \"\"\"\n data_files = [p for p in file_paths if not os.path.isdir(p)]\n if not os.path.exists(self.processed_dataset_dir):\n os.makedirs(self.processed_dataset_dir)\n # Moves any preserved paths (ex. image directories) into processed directory to avoid unnecessary copy.\n for rel_path in self._get_preserved_paths(self.raw_dataset_dir):\n source_path = os.path.join(self.raw_dataset_dir, rel_path)\n dest_path = os.path.join(self.processed_dataset_dir, rel_path)\n if os.path.exists(source_path) and not os.path.exists(dest_path):\n os.replace(source_path, dest_path)\n return data_files\n\n def load_file_to_dataframe(self, file_path: str) -> pd.DataFrame:\n \"\"\"Loads a file into a dataframe.\n\n Subclasses may override this method to support other input formats (json, jsonl, tsv, csv, parquet)\n \"\"\"\n file_extension = os.path.splitext(file_path)[-1].lower()\n if file_extension == \".json\":\n return pd.read_json(file_path)\n elif file_extension == \".jsonl\":\n return pd.read_json(file_path, lines=True)\n elif file_extension == \".tsv\":\n return pd.read_table(file_path)\n elif file_extension in {\".csv\", \".data\"}:\n return pd.read_csv(file_path)\n elif file_extension in {\".parquet\", \".pq\", \".pqt\"}:\n return pd.read_parquet(file_path)\n else:\n raise ValueError(f\"Unsupported dataset file type: {file_extension}\")\n\n def load_files_to_dataframe(self, file_paths: list[str], root_dir=None) -> pd.DataFrame:\n \"\"\"Loads a file or list of files and returns a dataframe.\n\n Subclasses may override this method to change the loader's behavior for groups of files.\n \"\"\"\n if root_dir:\n file_paths = [os.path.join(root_dir, path) for path in file_paths]\n dataframes = [self.load_file_to_dataframe(path) for path in file_paths]\n try:\n if self.config.columns:\n column_names = [column[\"name\"] for column in self.config.columns]\n\n set_cols_dfs = []\n for df in dataframes:\n # Split column is not included in configs, add in if pre-set split is present\n if SPLIT in df.columns:\n column_names.append(SPLIT)\n\n # If the number of columns in the dataframe does not match the number of columns in the config,\n # then the dataframe likely has an extra column that we don't want - i.e. \"Unnamed: 0\".\n if len(column_names) != len(df.columns):\n df = df[column_names]\n set_cols_dfs.append(df.set_axis(column_names, axis=1))\n return pd.concat(set_cols_dfs, ignore_index=True)\n else:\n return pd.concat(dataframes, ignore_index=True)\n except ValueError as e:\n logger.warning(f\"Error setting column names: {e}\")\n return pd.concat(dataframes, ignore_index=True)\n\n def load_unprocessed_dataframe(self, file_paths: list[str]) -> pd.DataFrame:\n \"\"\"Load dataset files into a dataframe.\n\n Will use the list of data files in the dataset directory as a default if all of config's dataset_filenames,\n train_filenames, validation_filenames, test_filenames are empty.\n \"\"\"\n dataset_paths = _glob_multiple(_list_of_strings(self.config.dataset_filenames), root_dir=self.raw_dataset_dir)\n train_paths = _glob_multiple(_list_of_strings(self.config.train_filenames), root_dir=self.raw_dataset_dir)\n validation_paths = _glob_multiple(\n _list_of_strings(self.config.validation_filenames), root_dir=self.raw_dataset_dir\n )\n test_paths = _glob_multiple(_list_of_strings(self.config.test_filenames), root_dir=self.raw_dataset_dir)\n if self.config.name == \"hugging_face\":\n dataframes = self._get_dataframe_with_fixed_splits_from_hf()\n else:\n dataframes = self._get_dataframe_with_fixed_splits(\n train_paths, validation_paths, test_paths, dataset_paths, file_paths\n )\n return pd.concat(dataframes, ignore_index=True)\n\n def _get_dataframe_with_fixed_splits_from_hf(self):\n dataframes = []\n splits = [\"train\", \"validation\", \"test\"]\n data_dict = self.load_hf_to_dict(\n self.config.huggingface_dataset_id, self.config.huggingface_subset\n ) # This function is defined in the Hugging Face dataloader\n for split_type in splits:\n if split_type in data_dict:\n # We don't have to do anything if split not in data_dict because we just concatenate the dataframes\n # in the end anyway.\n data_dict[split_type][SPLIT] = splits.index(split_type) # Add \"split\" column (0, 1, or 2)\n dataframes.append(data_dict[split_type])\n return dataframes\n\n def _get_dataframe_with_fixed_splits(self, train_paths, validation_paths, test_paths, dataset_paths, file_paths):\n dataframes = []\n if len(train_paths) > 0:\n train_df = self.load_files_to_dataframe(train_paths)\n train_df[SPLIT] = 0\n dataframes.append(train_df)\n if len(validation_paths) > 0:\n validation_df = self.load_files_to_dataframe(validation_paths)\n validation_df[SPLIT] = 1\n dataframes.append(validation_df)\n if len(test_paths) > 0:\n test_df = self.load_files_to_dataframe(test_paths)\n test_df[SPLIT] = 2\n dataframes.append(test_df)\n # If we have neither train/validation/test files nor dataset_paths in the config,\n # use data files in root dir.\n if len(dataset_paths) == len(dataframes) == 0:\n dataset_paths = file_paths\n if len(dataset_paths) > 0:\n dataframes.append(self.load_files_to_dataframe(dataset_paths))\n return dataframes\n\n def transform_dataframe(self, dataframe: pd.DataFrame) -> pd.DataFrame:\n \"\"\"Transforms a dataframe of the entire dataset.\n\n Subclasses should override this method if transformation of the dataframe is needed.\n \"\"\"\n for column_name, type in self.config.column_types.items():\n dataframe[column_name] = dataframe[column_name].astype(type)\n return dataframe\n\n def save_processed(self, dataframe: pd.DataFrame) -> None:\n \"\"\"Saves transformed dataframe as a flat file ludwig can load for training.\"\"\"\n if not os.path.exists(self.processed_dataset_dir):\n os.makedirs(self.processed_dataset_dir)\n dataframe.to_parquet(self.processed_dataset_path, engine=\"pyarrow\")\n\n def load_transformed_dataset(self) -> pd.DataFrame:\n \"\"\"Load processed dataset into a dataframe.\"\"\"\n return pd.read_parquet(self.processed_dataset_path)\n\n def get_mtime(self) -> float:\n \"\"\"Last modified time of the processed dataset after downloading successfully.\"\"\"\n return os.path.getmtime(self.processed_dataset_path)\n\n @staticmethod\n def split(dataset: pd.DataFrame) -> tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame]:\n if SPLIT in dataset:\n dataset[SPLIT] = pd.to_numeric(dataset[SPLIT])\n training_set = dataset[dataset[SPLIT] == 0].drop(columns=[SPLIT])\n val_set = dataset[dataset[SPLIT] == 1].drop(columns=[SPLIT])\n test_set = dataset[dataset[SPLIT] == 2].drop(columns=[SPLIT])\n return training_set, test_set, val_set\n else:\n raise ValueError(f\"The dataset does not a '{SPLIT}' column, load with `split=False`\")\n"
},
{
"path": "ludwig/datasets/loaders/ethos_binary.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport pandas as pd\n\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\n\n\nclass EthosBinaryLoader(DatasetLoader):\n def load_file_to_dataframe(self, file_path: str) -> pd.DataFrame:\n # This dataset uses ; seperator instead of ,\n return pd.read_csv(file_path, sep=\";\")\n\n def transform_dataframe(self, dataframe: pd.DataFrame) -> pd.DataFrame:\n processed_df = super().transform_dataframe(dataframe)\n # convert float labels (0.0, 1.0) to binary labels\n processed_df[\"isHate\"] = processed_df[\"isHate\"] >= 0.5\n processed_df[\"isHate\"] = processed_df[\"isHate\"].astype(int)\n return processed_df\n"
},
{
"path": "ludwig/datasets/loaders/flickr8k.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport os\nimport re\nfrom collections import defaultdict\n\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\n\n\nclass Flickr8kLoader(DatasetLoader):\n def transform_files(self, file_paths: list[str]) -> list[str]:\n # create a dictionary matching image_path --> list of captions\n image_to_caption = defaultdict(list)\n with open(f\"{self.raw_dataset_dir}/Flickr8k.token.txt\") as captions_file:\n image_to_caption = defaultdict(list)\n for line in captions_file:\n line = line.split(\"#\")\n # the regex is to format the string to fit properly in a csv\n line[1] = line[1].strip(\"\\n01234.\\t \")\n line[1] = re.sub('\"', '\"\"', line[1])\n line[1] = '\"' + line[1] + '\"'\n image_to_caption[line[0]].append(line[1])\n # create csv file with 7 columns: image_path, 5 captions, and split\n with open(os.path.join(self.raw_dataset_dir, \"flickr8k_dataset.csv\"), \"w\") as output_file:\n output_file.write(\"image_path,caption0,caption1,caption2,\")\n output_file.write(\"caption3,caption4,split\\n\")\n splits = [\"train\", \"dev\", \"test\"]\n for i in range(len(splits)):\n split = splits[i]\n with open(f\"{self.raw_dataset_dir}/Flickr_8k.{split}Images.txt\") as split_file:\n for image_name in split_file:\n image_name = image_name.strip(\"\\n\")\n if image_name in image_to_caption:\n output_file.write(\n \"{},{},{},{},{},{},{}\\n\".format(\n # Note: image folder is named Flicker8k_Dataset\n f\"{self.raw_dataset_dir}/Flicker8k_Dataset/{image_name}\",\n *image_to_caption[image_name],\n i,\n )\n )\n return super().transform_files(file_paths)\n"
},
{
"path": "ludwig/datasets/loaders/forest_cover.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\n\nfrom ludwig.datasets.dataset_config import DatasetConfig\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\n\n\nclass ForestCoverLoader(DatasetLoader):\n def __init__(self, config: DatasetConfig, cache_dir: str | None = None, use_tabnet_split=True):\n super().__init__(config, cache_dir=cache_dir)\n self.use_tabnet_split = use_tabnet_split\n\n def transform_dataframe(self, dataframe: pd.DataFrame) -> pd.DataFrame:\n df = super().transform_dataframe(dataframe)\n # Elevation quantitative meters Elevation in meters\n # Aspect quantitative azimuth Aspect in degrees azimuth\n # Slope quantitative degrees Slope in degrees\n # Horizontal_Distance_To_Hydrology quantitative meters Horz Dist to nearest surface water features # noqa: E501\n # Vertical_Distance_To_Hydrology quantitative meters Vert Dist to nearest surface water features # noqa: E501\n # Horizontal_Distance_To_Roadways quantitative meters Horz Dist to nearest roadway # noqa: E501\n # Hillshade_9am quantitative 0 to 255 index Hillshade index at 9am, summer solstice # noqa: E501\n # Hillshade_Noon quantitative 0 to 255 index Hillshade index at noon, summer soltice # noqa: E501\n # Hillshade_3pm quantitative 0 to 255 index Hillshade index at 3pm, summer solstice # noqa: E501\n # Horizontal_Distance_To_Fire_Points quantitative meters Horz Dist to nearest wildfire ignition points # noqa: E501\n # Wilderness_Area (4 binary columns) qualitative 0 (absence) or 1 (presence) Wilderness area designation # noqa: E501\n # Soil_Type (40 binary columns) qualitative 0 (absence) or 1 (presence) Soil Type designation\n # Cover_Type (7 types) integer 1 to 7 Forest Cover Type designation # noqa: E501\n\n # Map the 40 soil types to a single integer instead of 40 binary columns\n st_cols = [\n \"Soil_Type_1\",\n \"Soil_Type_2\",\n \"Soil_Type_3\",\n \"Soil_Type_4\",\n \"Soil_Type_5\",\n \"Soil_Type_6\",\n \"Soil_Type_7\",\n \"Soil_Type_8\",\n \"Soil_Type_9\",\n \"Soil_Type_10\",\n \"Soil_Type_11\",\n \"Soil_Type_12\",\n \"Soil_Type_13\",\n \"Soil_Type_14\",\n \"Soil_Type_15\",\n \"Soil_Type_16\",\n \"Soil_Type_17\",\n \"Soil_Type_18\",\n \"Soil_Type_19\",\n \"Soil_Type_20\",\n \"Soil_Type_21\",\n \"Soil_Type_22\",\n \"Soil_Type_23\",\n \"Soil_Type_24\",\n \"Soil_Type_25\",\n \"Soil_Type_26\",\n \"Soil_Type_27\",\n \"Soil_Type_28\",\n \"Soil_Type_29\",\n \"Soil_Type_30\",\n \"Soil_Type_31\",\n \"Soil_Type_32\",\n \"Soil_Type_33\",\n \"Soil_Type_34\",\n \"Soil_Type_35\",\n \"Soil_Type_36\",\n \"Soil_Type_37\",\n \"Soil_Type_38\",\n \"Soil_Type_39\",\n \"Soil_Type_40\",\n ]\n st_vals = []\n for _, row in df[st_cols].iterrows():\n st_vals.append(row.to_numpy().nonzero()[0].item(0))\n df = df.drop(columns=st_cols)\n df[\"Soil_Type\"] = st_vals\n\n # Map the 4 wilderness areas to a single integer\n # instead of 4 binary columns\n wa_cols = [\"Wilderness_Area_1\", \"Wilderness_Area_2\", \"Wilderness_Area_3\", \"Wilderness_Area_4\"]\n wa_vals = []\n for _, row in df[wa_cols].iterrows():\n wa_vals.append(row.to_numpy().nonzero()[0].item(0))\n df = df.drop(columns=wa_cols)\n df[\"Wilderness_Area\"] = wa_vals\n\n if not self.use_tabnet_split:\n # first 11340 records used for training data subset\n # next 3780 records used for validation data subset\n # last 565892 records used for testing data subset\n df[\"split\"] = [0] * 11340 + [1] * 3780 + [2] * 565892\n else:\n # Split used in the tabNet paper\n # https://github.com/google-research/google-research/blob/master/tabnet/download_prepare_covertype.py\n train_val_indices, test_indices = train_test_split(range(len(df)), test_size=0.2, random_state=0)\n train_indices, val_indices = train_test_split(train_val_indices, test_size=0.2 / 0.6, random_state=0)\n\n df[\"split\"] = 0\n df.loc[val_indices, \"split\"] = 1\n df.loc[test_indices, \"split\"] = 2\n\n return df\n"
},
{
"path": "ludwig/datasets/loaders/goemotions.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport pandas as pd\n\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\n\n\nclass GoEmotionsLoader(DatasetLoader):\n def transform_dataframe(self, dataframe: pd.DataFrame) -> pd.DataFrame:\n processed_df = super().transform_dataframe(dataframe)\n # Format emotion IDs as space-delimited string (Set).\n processed_df[\"emotion_ids\"] = processed_df[\"emotion_ids\"].apply(lambda e_id: \" \".join(e_id.split(\",\")))\n return processed_df\n"
},
{
"path": "ludwig/datasets/loaders/higgs.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nimport pandas as pd\n\nfrom ludwig.datasets.dataset_config import DatasetConfig\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\n\n\nclass HiggsLoader(DatasetLoader):\n def __init__(self, config: DatasetConfig, cache_dir: str | None = None, add_validation_set=True):\n super().__init__(config, cache_dir)\n self.add_validation_set = add_validation_set\n\n def load_file_to_dataframe(self, file_path: str) -> pd.DataFrame:\n \"\"\"Loads a file into a dataframe.\"\"\"\n return pd.read_csv(file_path, header=None)\n\n def transform_dataframe(self, dataframe: pd.DataFrame) -> pd.DataFrame:\n processed_df = super().transform_dataframe(dataframe)\n if self.add_validation_set:\n processed_df[\"split\"] = [0] * 10000000 + [1] * 500000 + [2] * 500000\n else:\n processed_df[\"split\"] = [0] * 10500000 + [2] * 500000\n return processed_df\n"
},
{
"path": "ludwig/datasets/loaders/hugging_face.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nfrom __future__ import annotations\n\nimport logging\n\nimport datasets\nimport pandas as pd\n\nfrom ludwig.constants import TEST, TRAIN, VALIDATION\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\n\nSPLITS = [TRAIN, VALIDATION, TEST]\nlogger = logging.getLogger(__name__)\n\n\nclass HFLoader(DatasetLoader):\n \"\"\"HFLoader differs from all other DatasetLoaders because of how it loads data through the Hugging Face\n datasets API instead of saving any files to the cache.\n\n The config for HFLoader contains two unique parameters, huggingface_dataset_id and huggingface_subsample, that\n identify which dataset and which subsample of that dataset to load in.\n \"\"\"\n\n @staticmethod\n def load_hf_to_dict(hf_id: str | None = None, hf_subsample: str | None = None) -> dict[str, pd.DataFrame]:\n \"\"\"Returns a map of split -> pd.DataFrame for the given HF dataset.\n\n :param hf_id: (str) path to dataset on HuggingFace platform\n :param hf_subsample: (str) name of dataset configuration on HuggingFace platform\n \"\"\"\n dataset_dict: dict[str, datasets.Dataset] = datasets.load_dataset(path=hf_id, name=hf_subsample)\n pandas_dict = {}\n for split in dataset_dict:\n # Convert from HF DatasetDict type to a dictionary of pandas dataframes\n pandas_dict[split] = dataset_dict[split].to_pandas()\n return pandas_dict\n\n # TODO(Alex): Standardize load() signature as interface method in DatasetLoader and adhere to it in all subclasses.\n def load(\n self, hf_id: str | None = None, hf_subsample: str | None = None, split: bool = False\n ) -> pd.DataFrame | list[pd.DataFrame, pd.DataFrame, pd.DataFrame]:\n \"\"\"When load() is called, HFLoader calls the datasets API to return all of the data in a HuggingFace\n DatasetDict, converts it to a dictionary of pandas dataframes, and returns either three dataframes\n containing train, validation, and test data or one dataframe that is the concatenation of all three\n depending on whether `split` is set to True or False.\n\n :param split: (bool) directive for how to interpret if dataset contains validation or test set (see below)\n\n Note that some datasets may not provide a validation set or a test set. In this case:\n - If split is True, the DataFrames corresponding to the missing sets are initialized to be empty\n - If split is False, the \"split\" column in the resulting DataFrame will reflect the fact that there is no\n validation/test split (i.e., there will be no 1s/2s)\n\n A train set should always be provided by Hugging Face.\n\n :param hf_id: (str) path to dataset on HuggingFace platform\n :param hf_subsample: (str) name of dataset configuration on HuggingFace platform\n \"\"\"\n self.config.huggingface_dataset_id = hf_id\n self.config.huggingface_subsample = hf_subsample\n pandas_dict = self.load_hf_to_dict(\n hf_id=hf_id,\n hf_subsample=hf_subsample,\n )\n if split: # For each split, either return the appropriate dataframe or an empty dataframe\n for spl in SPLITS:\n if spl not in pandas_dict:\n logger.warning(f\"No {spl} set found in provided Hugging Face dataset. Skipping {spl} set.\")\n train_df = pandas_dict[TRAIN] if TRAIN in pandas_dict else pd.DataFrame()\n validation_df = pandas_dict[VALIDATION] if VALIDATION in pandas_dict else pd.DataFrame()\n test_df = pandas_dict[TEST] if TEST in pandas_dict else pd.DataFrame()\n\n return train_df, validation_df, test_df\n else:\n dataset_list = []\n for spl in pandas_dict:\n pandas_dict[spl][\"split\"] = SPLITS.index(spl) # Add a column containing 0s, 1s, and 2s denoting splits\n dataset_list.append(pandas_dict[spl])\n return pd.concat(dataset_list)\n"
},
{
"path": "ludwig/datasets/loaders/ieee_fraud.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport os\n\nimport pandas as pd\n\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\n\n\nclass IEEEFraudLoader(DatasetLoader):\n \"\"\"The IEEE-CIS Fraud Detection Dataset https://www.kaggle.com/c/ieee-fraud-detection/overview.\"\"\"\n\n def load_unprocessed_dataframe(self, file_paths: list[str]) -> pd.DataFrame:\n \"\"\"Load dataset files into a dataframe.\"\"\"\n train_files = {\"train_identity.csv\", \"train_transaction.csv\"}\n test_files = {\"test_identity.csv\", \"test_transaction.csv\"}\n\n train_dfs, test_dfs = {}, {}\n\n for filename in train_files.union(test_files):\n split_name = os.path.splitext(filename)[0]\n file_df = self.load_file_to_dataframe(os.path.join(self.raw_dataset_dir, filename))\n if filename in train_files:\n train_dfs[split_name] = file_df\n elif filename in test_files:\n test_dfs[split_name] = file_df\n\n # Merge on TransactionID\n final_train = pd.merge(\n train_dfs[\"train_transaction\"], train_dfs[\"train_identity\"], on=\"TransactionID\", how=\"left\"\n )\n return final_train\n"
},
{
"path": "ludwig/datasets/loaders/insurance_lite.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport os\n\nimport pandas as pd\n\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\n\n\nclass InsuranceLiteLoader(DatasetLoader):\n \"\"\"Health Insurance Cross Sell Prediction Predict Health Insurance Owners' who will be interested in Vehicle\n Insurance https://www.kaggle.com/datasets/arashnic/imbalanced-data-practice.\"\"\"\n\n def transform_dataframe(self, dataframe: pd.DataFrame) -> pd.DataFrame:\n df = super().transform_dataframe(dataframe)\n # Make image paths relative to dataset root directory\n df[\"image_path\"] = df[\"image_path\"].apply(\n lambda x: os.path.join(\"Fast_Furious_Insured\", \"trainImages\", os.path.basename(x))\n )\n return df\n"
},
{
"path": "ludwig/datasets/loaders/kdd_loader.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport os\n\nimport pandas as pd\n\nfrom ludwig.datasets.dataset_config import DatasetConfig\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\n\n\nclass KDDCup2009Loader(DatasetLoader):\n def __init__(self, config: DatasetConfig, cache_dir: str | None = None, task_name=\"\", include_test_download=False):\n super().__init__(config, cache_dir=cache_dir)\n self.task_name = task_name\n self.include_test_download = include_test_download\n\n def load_file_to_dataframe(self, file_path: str) -> pd.DataFrame:\n \"\"\"Loads a file into a dataframe.\"\"\"\n return pd.read_csv(file_path, sep=\"\\t\")\n\n def transform_dataframe(self, dataframe: pd.DataFrame) -> pd.DataFrame:\n train_df = super().transform_dataframe(dataframe)\n train_df = process_categorical_features(train_df, categorical_features)\n train_df = process_number_features(train_df, categorical_features)\n\n targets = (\n pd.read_csv(os.path.join(self.raw_dataset_dir, f\"orange_small_train_{self.task_name}.labels\"), header=None)[\n 0\n ]\n .astype(str)\n .apply(lambda x: \"true\" if x == \"1\" else \"false\")\n )\n\n train_idcs = pd.read_csv(\n os.path.join(self.raw_dataset_dir, f\"stratified_train_idx_{self.task_name}.txt\"), header=None\n )[0]\n\n val_idcs = pd.read_csv(\n os.path.join(self.raw_dataset_dir, f\"stratified_test_idx_{self.task_name}.txt\"), header=None\n )[0]\n\n processed_train_df = train_df.iloc[train_idcs].copy()\n processed_train_df[\"target\"] = targets.iloc[train_idcs]\n processed_train_df[\"split\"] = 0\n\n processed_val_df = train_df.iloc[val_idcs].copy()\n processed_val_df[\"target\"] = targets.iloc[val_idcs]\n processed_val_df[\"split\"] = 1\n\n if self.include_test_download:\n test_df = self.load_file_to_dataframe(os.path.join(self.raw_dataset_dir, \"orange_small_test.data\"))\n test_df[\"target\"] = \"\" # no ground truth labels for test download\n test_df[\"split\"] = 2\n df = pd.concat([processed_train_df, processed_val_df, test_df])\n else:\n df = pd.concat([processed_train_df, processed_val_df])\n\n return df\n\n\ndef process_categorical_features(df, categorical_features):\n for i in categorical_features:\n df.iloc[:, i].fillna(\"\", inplace=True)\n return df\n\n\ndef process_number_features(df, categorical_features):\n for i, column in enumerate(df.columns):\n if i not in categorical_features:\n df[column].astype(float, copy=False)\n return df\n\n\ncategorical_features = {\n 190,\n 191,\n 192,\n 193,\n 194,\n 195,\n 196,\n 197,\n 198,\n 199,\n 200,\n 201,\n 202,\n 203,\n 204,\n 205,\n 206,\n 207,\n 209,\n 210,\n 211,\n 212,\n 213,\n 214,\n 215,\n 216,\n 217,\n 218,\n 219,\n 220,\n 221,\n 222,\n 223,\n 224,\n 225,\n 226,\n 227,\n 228,\n}\n\n\nclass KDDAppetencyLoader(KDDCup2009Loader):\n \"\"\"The KDD Cup 2009 Appetency dataset.\n\n https://www.kdd.org/kdd-cup/view/kdd-cup-2009/Data\n \"\"\"\n\n def __init__(self, config: DatasetConfig, cache_dir: str | None = None, include_test_download=False):\n super().__init__(\n config, cache_dir=cache_dir, task_name=\"appetency\", include_test_download=include_test_download\n )\n\n\nclass KDDChurnLoader(KDDCup2009Loader):\n \"\"\"The KDD Cup 2009 Churn dataset.\n\n https://www.kdd.org/kdd-cup/view/kdd-cup-2009/Data\n \"\"\"\n\n def __init__(self, config: DatasetConfig, cache_dir: str | None = None, include_test_download=False):\n super().__init__(config, cache_dir=cache_dir, task_name=\"churn\", include_test_download=include_test_download)\n\n\nclass KDDUpsellingLoader(KDDCup2009Loader):\n \"\"\"The KDD Cup 2009 Upselling dataset.\n\n https://www.kdd.org/kdd-cup/view/kdd-cup-2009/Data\n \"\"\"\n\n def __init__(self, config: DatasetConfig, cache_dir: str | None = None, include_test_download=False):\n super().__init__(\n config, cache_dir=cache_dir, task_name=\"upselling\", include_test_download=include_test_download\n )\n"
},
{
"path": "ludwig/datasets/loaders/mnist.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nimport os\nimport struct\nfrom multiprocessing.pool import ThreadPool\n\nimport numpy as np\nimport pandas as pd\nimport torch\n\nfrom ludwig.datasets.dataset_config import DatasetConfig\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\nfrom ludwig.utils.fs_utils import makedirs\n\nlogger = logging.getLogger(__name__)\nNUM_LABELS = 10\n\n\nclass MNISTLoader(DatasetLoader):\n def __init__(self, config: DatasetConfig, cache_dir: str | None = None):\n try:\n from torchvision.io import write_png\n\n self.write_png = write_png\n except ImportError:\n logger.error(\n \"torchvision is not installed. \"\n \"In order to install all image feature dependencies run \"\n \"pip install ludwig[image]\"\n )\n raise\n super().__init__(config, cache_dir)\n\n def transform_files(self, file_paths: list[str]) -> list[str]:\n for dataset in [\"training\", \"testing\"]:\n labels, images = self.read_source_dataset(dataset, self.raw_dataset_dir)\n self.write_output_dataset(labels, images, os.path.join(self.raw_dataset_dir, dataset))\n return super().transform_files(file_paths)\n\n def load_unprocessed_dataframe(self, file_paths: list[str]) -> pd.DataFrame:\n \"\"\"Load dataset files into a dataframe.\"\"\"\n return self.output_training_and_test_data()\n\n def read_source_dataset(self, dataset=\"training\", path=\".\"):\n \"\"\"Create a directory for training and test and extract all the images and labels to this destination.\n\n :args: dataset (str) : the label for the dataset path (str): the raw dataset path\n :returns: A tuple of the label for the image, the file array, the size and rows and columns for the image\n \"\"\"\n if dataset == \"training\":\n fname_img = os.path.join(path, \"train-images-idx3-ubyte\")\n fname_lbl = os.path.join(path, \"train-labels-idx1-ubyte\")\n elif dataset == \"testing\":\n fname_img = os.path.join(path, \"t10k-images-idx3-ubyte\")\n fname_lbl = os.path.join(path, \"t10k-labels-idx1-ubyte\")\n else:\n raise ValueError(\"dataset must be 'testing' or 'training'\")\n\n with open(fname_lbl, \"rb\") as flbl:\n struct.unpack(\">II\", flbl.read(8))\n lbl = np.frombuffer(flbl.read(), dtype=np.uint8)\n\n with open(fname_img, \"rb\") as fimg:\n magic_nr, size, rows, cols = struct.unpack(\">IIII\", fimg.read(16))\n img = np.frombuffer(fimg.read(), dtype=np.uint8)\n img = img.reshape((size, rows, cols))\n\n return lbl, img\n\n def write_output_dataset(self, labels, images, output_dir):\n \"\"\"Create output directories where we write out the images.\n\n :args: labels (str) : the labels for the image data (np.array) : the binary array corresponding to the image\n output_dir (str) : the output directory that we need to write to path (str): the raw dataset path\n :returns: A tuple of the label for the image, the file array, the size and rows and columns for the image\n \"\"\"\n # create child image output directories\n output_dirs = [os.path.join(output_dir, str(i)) for i in range(NUM_LABELS)]\n\n for output_dir in output_dirs:\n makedirs(output_dir, exist_ok=True)\n\n def write_processed_image(t):\n i, label = t\n output_filename = os.path.join(output_dirs[label], str(i) + \".png\")\n torch_image = torch.from_numpy(images[i].copy()).view(1, 28, 28)\n self.write_png(torch_image, output_filename)\n\n # write out image data\n tasks = list(enumerate(labels))\n pool = ThreadPool(NUM_LABELS)\n pool.map(write_processed_image, tasks)\n pool.close()\n pool.join()\n\n def output_training_and_test_data(self):\n \"\"\"Creates a combined (training and test) dataframe by iterating through all the images and labels.\"\"\"\n dataframes = []\n for name in [\"training\", \"testing\"]:\n labels = []\n paths = []\n splits = []\n for i in range(NUM_LABELS):\n label_dir = f\"{name}/{i}\"\n img_dir = os.path.join(self.processed_dataset_dir, label_dir)\n for file in os.listdir(img_dir):\n if file.endswith(\".png\"):\n labels.append(str(i))\n paths.append(os.path.join(img_dir, file))\n splits.append(0 if name == \"training\" else 2)\n dataframes.append(pd.DataFrame({\"image_path\": paths, \"label\": labels, \"split\": splits}))\n return pd.concat(dataframes, ignore_index=True)\n"
},
{
"path": "ludwig/datasets/loaders/naval.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport pandas as pd\n\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\n\n\nclass NavalLoader(DatasetLoader):\n def load_file_to_dataframe(self, file_path: str) -> pd.DataFrame:\n \"\"\"Loads a file into a dataframe.\"\"\"\n return pd.read_csv(file_path, header=None, sep=\" \")\n"
},
{
"path": "ludwig/datasets/loaders/rossman_store_sales.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport calendar\nimport os\n\nimport numpy as np\nimport pandas as pd\n\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\n\n\nclass RossmanStoreSalesLoader(DatasetLoader):\n \"\"\"The Rossmann Store Sales dataset.\"\"\"\n\n def load_unprocessed_dataframe(self, file_paths: list[str]) -> pd.DataFrame:\n \"\"\"Load dataset files into a dataframe.\"\"\"\n\n stores_df = pd.read_csv(os.path.join(self.raw_dataset_dir, \"store.csv\"))\n\n train_df = pd.read_csv(os.path.join(self.raw_dataset_dir, \"train.csv\"), low_memory=False)\n train_df = preprocess_df(train_df, stores_df)\n\n train_df[\"split\"] = -1\n train_df.loc[train_df[\"Year\"] == 2014, \"split\"] = 0\n train_df.loc[train_df[\"Year\"] == 2015, \"split\"] = 2\n train_df.drop(train_df[train_df[\"split\"] == -1].index, inplace=True)\n return train_df\n\n\ndef preprocess_dates(df):\n # Make integer Year,Month,Day columns instead of Date\n dates = np.array([[int(v) for v in s.split(\"-\")] for s in df[\"Date\"]])\n df = df.drop([\"Date\"], axis=1)\n df[\"Year\"] = dates[:, 0]\n df[\"Month\"] = dates[:, 1]\n df[\"Day\"] = dates[:, 2]\n return df\n\n\nmonth_abbrs = calendar.month_abbr[1:]\nmonth_abbrs[8] = \"Sept\"\n\n\ndef preprocess_stores(df, stores_df):\n # join data in df with stores df\n df = df.join(stores_df, on=\"Store\", rsuffix=\"_right\")\n df = df.drop([\"Store_right\"], axis=1)\n\n promo2_start_months = [(s.split(\",\") if not pd.isnull(s) else []) for s in df[\"PromoInterval\"]]\n\n for month_abbr in month_abbrs:\n df[\"Promo2Start_\" + month_abbr] = np.array(\n [(1 if month_abbr in s else 0) for s in promo2_start_months], dtype=np.int8\n )\n df = df.drop([\"PromoInterval\"], axis=1)\n\n return df\n\n\nint_columns = [\n \"Store\",\n \"DayOfWeek\",\n \"Sales\",\n \"Customers\",\n \"Open\",\n \"Promo\",\n \"SchoolHoliday\",\n \"Year\",\n \"Month\",\n \"Day\",\n \"CompetitionDistance\",\n \"CompetitionOpenSinceMonth\",\n \"CompetitionOpenSinceYear\",\n \"Promo2\",\n \"Promo2SinceWeek\",\n \"Promo2SinceYear\",\n]\n\n\ndef preprocess_df(df, stores_df):\n df = preprocess_dates(df)\n df = preprocess_stores(df, stores_df)\n\n for column in int_columns:\n df[column] = pd.to_numeric(df[column].fillna(0), downcast=\"integer\")\n\n df[\"StateHoliday\"] = df[\"StateHoliday\"].astype(str)\n df.loc[df[\"StateHoliday\"] == \"0\", \"StateHoliday\"] = \"No\"\n\n return df\n"
},
{
"path": "ludwig/datasets/loaders/santander_value_prediction.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport pandas as pd\n\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\n\n\nclass SantanderValuePredictionLoader(DatasetLoader):\n \"\"\"The Santander Value Prediction Challenge dataset.\n\n https://www.kaggle.com/c/santander-value-prediction-challenge\n \"\"\"\n\n def transform_dataframe(self, dataframe: pd.DataFrame) -> pd.DataFrame:\n processed_df = super().transform_dataframe(dataframe)\n # Ensure feature column names are strings (some are numeric); keep special names as is\n processed_df.columns = [\"C\" + str(col) for col in processed_df.columns]\n processed_df.rename(columns={\"CID\": \"ID\", \"Ctarget\": \"target\", \"Csplit\": \"split\"}, inplace=True)\n return processed_df\n"
},
{
"path": "ludwig/datasets/loaders/sarcastic_headlines.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nimport pandas as pd\n\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\n\n\nclass SarcasticHeadlinesLoader(DatasetLoader):\n def load_file_to_dataframe(self, file_path: str) -> pd.DataFrame:\n \"\"\"Loads a file into a dataframe.\"\"\"\n return pd.read_json(file_path, lines=True)\n"
},
{
"path": "ludwig/datasets/loaders/sarcos.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport os\n\nimport pandas as pd\nfrom scipy.io import loadmat\n\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\nfrom ludwig.utils.fs_utils import open_file\n\n\nclass SarcosLoader(DatasetLoader):\n \"\"\"The Sarcos dataset.\n\n Details:\n The data relates to an inverse dynamics problem for a seven\n degrees-of-freedom SARCOS anthropomorphic robot arm. The\n task is to map from a 21-dimensional input space (7 joint\n positions, 7 joint velocities, 7 joint accelerations) to the\n corresponding 7 joint torques. There are 44,484 training\n examples and 4,449 test examples. The first 21 columns are\n the input variables, and the 22nd column is used as the target\n variable.\n\n Dataset source:\n Locally Weighted Projection RegressionL: An O(n) Algorithm for\n Incremental Real Time Learning in High Dimensional Space,\n S. Vijayakumar and S. Schaal, Proc ICML 2000.\n http://www.gaussianprocess.org/gpml/data/\n \"\"\"\n\n def load_file_to_dataframe(self, file_path: str) -> pd.DataFrame:\n \"\"\"Loads a file into a dataframe.\"\"\"\n with open_file(file_path) as f:\n mat = loadmat(f)\n file_df = pd.DataFrame(mat[os.path.basename(file_path).split(\".\")[0]])\n return file_df\n\n def transform_dataframe(self, dataframe: pd.DataFrame) -> pd.DataFrame:\n processed_df = super().transform_dataframe(dataframe)\n columns = []\n columns += [f\"position_{i}\" for i in range(1, 8)]\n columns += [f\"velocity_{i}\" for i in range(1, 8)]\n columns += [f\"acceleration_{i}\" for i in range(1, 8)]\n columns += [f\"torque_{i}\" for i in range(1, 8)]\n columns += [\"split\"]\n\n processed_df.columns = columns\n return processed_df\n"
},
{
"path": "ludwig/datasets/loaders/split_loaders.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport numpy as np\nimport pandas as pd\n\nfrom ludwig.constants import SPLIT\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\n\n\nclass RandomSplitLoader(DatasetLoader):\n \"\"\"Adds a random split column to the dataset, with fixed proportions of:\n\n train: 70%\n validation: 10%\n test: 20%\n .\n \"\"\"\n\n def transform_dataframe(self, dataframe: pd.DataFrame) -> pd.DataFrame:\n df = super().transform_dataframe(dataframe)\n df[SPLIT] = np.random.choice(3, len(df), p=(0.7, 0.1, 0.2)).astype(np.int8)\n return df\n"
},
{
"path": "ludwig/datasets/loaders/sst.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport os\n\nimport pandas as pd\n\nfrom ludwig.datasets.dataset_config import DatasetConfig\nfrom ludwig.datasets.loaders.dataset_loader import DatasetLoader\n\n\nclass SSTLoader(DatasetLoader):\n \"\"\"The SST dataset.\n\n This dataset is constructed using the Stanford Sentiment Treebank Dataset.\n This dataset contains binary labels (positive or negative) for each sample.\n\n The original dataset specified 5 labels:\n very negative, negative, neutral, positive, very positive with\n the following cutoffs:\n [0, 0.2], (0.2, 0.4], (0.4, 0.6], (0.6, 0.8], (0.8, 1.0]\n \"\"\"\n\n def __init__(\n self,\n config: DatasetConfig,\n cache_dir: str | None = None,\n include_subtrees=False,\n discard_neutral=False,\n convert_parentheses=True,\n remove_duplicates=False,\n ):\n super().__init__(config, cache_dir=cache_dir)\n self.include_subtrees = include_subtrees\n self.discard_neutral = discard_neutral\n self.convert_parentheses = convert_parentheses\n self.remove_duplicates = remove_duplicates\n\n @staticmethod\n def get_sentiment_label(id2sent, phrase_id):\n raise NotImplementedError\n\n def transform_files(self, file_paths: list[str]) -> list[str]:\n # maybe this should be\n\n \"\"\"Load dataset files into a dataframe.\"\"\"\n sentences_df = pd.read_csv(\n os.path.join(self.raw_dataset_dir, \"stanfordSentimentTreebank/datasetSentences.txt\"),\n sep=\"\\t\",\n )\n\n sentences_df[\"sentence\"] = sentences_df[\"sentence\"].apply(format_text)\n\n datasplit_df = pd.read_csv(\n os.path.join(self.raw_dataset_dir, \"stanfordSentimentTreebank/datasetSplit.txt\"), sep=\",\"\n )\n\n phrase2id = {}\n with open(os.path.join(self.raw_dataset_dir, \"stanfordSentimentTreebank/dictionary.txt\")) as f:\n Lines = f.readlines()\n for line in Lines:\n if line:\n split_line = line.split(\"|\")\n phrase = split_line[0]\n phrase2id[phrase] = int(split_line[1])\n\n id2sent = {}\n with open(os.path.join(self.raw_dataset_dir, \"stanfordSentimentTreebank/sentiment_labels.txt\")) as f:\n Lines = f.readlines()\n for line in Lines:\n if line:\n split_line = line.split(\"|\")\n try:\n id2sent[int(split_line[0])] = float(split_line[1])\n except ValueError:\n pass\n\n trees_pointers = None\n trees_phrases = None\n\n if self.include_subtrees:\n trees_pointers = []\n with open(os.path.join(self.raw_dataset_dir, \"stanfordSentimentTreebank/STree.txt\")) as f:\n Lines = f.readlines()\n for line in Lines:\n if line:\n trees_pointers.append([int(s.strip()) for s in line.split(\"|\")])\n\n trees_phrases = []\n with open(os.path.join(self.raw_dataset_dir, \"stanfordSentimentTreebank/SOStr.txt\")) as f:\n Lines = f.readlines()\n for line in Lines:\n if line:\n trees_phrases.append([s.strip() for s in line.split(\"|\")])\n\n splits = {\"train\": 1, \"test\": 2, \"dev\": 3}\n\n generated_csv_filenames = []\n for split_name, split_id in splits.items():\n sentence_idcs = get_sentence_idcs_in_split(datasplit_df, split_id)\n\n pairs = []\n if split_name == \"train\" and self.include_subtrees:\n phrases = []\n for sentence_idx in sentence_idcs:\n # trees_pointers and trees_phrases are 0 indexed\n # while sentence_idx starts from 1\n # so we need to decrease sentence_idx value\n sentence_idx -= 1\n subtrees = sentence_subtrees(sentence_idx, trees_pointers, trees_phrases)\n\n sentence_idx += 1\n sentence_phrase = list(sentences_df[sentences_df[\"sentence_index\"] == sentence_idx][\"sentence\"])[0]\n\n sentence_phrase = convert_parentheses(sentence_phrase)\n label = self.get_sentiment_label(id2sent, phrase2id[sentence_phrase])\n # filter @ sentence level\n # For SST-2, check subtrees only if sentence is not neutral\n if not self.discard_neutral or label != -1:\n for phrase in subtrees:\n label = self.get_sentiment_label(id2sent, phrase2id[phrase])\n if not self.discard_neutral or label != -1:\n if not self.convert_parentheses:\n phrase = convert_parentheses_back(phrase)\n phrase = phrase.replace(\"\\xa0\", \" \")\n pairs.append([phrase, label])\n else:\n phrases = get_sentences_with_idcs(sentences_df, sentence_idcs)\n for phrase in phrases:\n phrase = convert_parentheses(phrase)\n label = self.get_sentiment_label(id2sent, phrase2id[phrase])\n if not self.discard_neutral or label != -1:\n if not self.convert_parentheses:\n phrase = convert_parentheses_back(phrase)\n phrase = phrase.replace(\"\\xa0\", \" \")\n pairs.append([phrase, label])\n\n final_csv = pd.DataFrame(pairs)\n final_csv.columns = [\"sentence\", \"label\"]\n if self.remove_duplicates:\n final_csv = final_csv.drop_duplicates(subset=[\"sentence\"])\n csv_filename = os.path.join(self.raw_dataset_dir, f\"{split_name}.csv\")\n generated_csv_filenames.append(csv_filename)\n final_csv.to_csv(csv_filename, index=False)\n\n return super().transform_files(generated_csv_filenames)\n\n\nclass SST2Loader(SSTLoader):\n \"\"\"The SST2 dataset.\n\n This dataset is constructed using the Stanford Sentiment Treebank Dataset.\n This dataset contains binary labels (positive or negative) for each sample.\n\n The original dataset specified 5 labels:\n very negative, negative, neutral, positive, very positive with\n the following cutoffs:\n [0, 0.2], (0.2, 0.4], (0.4, 0.6], (0.6, 0.8], (0.8, 1.0]\n\n In the construction of this dataset, we remove all neutral phrases\n and assign a negative label if the original rating falls\n into the following range: [0, 0.4] and a positive label\n if the original rating is between (0.6, 1.0].\n \"\"\"\n\n def __init__(\n self,\n config: DatasetConfig,\n cache_dir: str | None = None,\n include_subtrees=False,\n convert_parentheses=True,\n remove_duplicates=False,\n ):\n super().__init__(\n config,\n cache_dir=cache_dir,\n include_subtrees=include_subtrees,\n discard_neutral=True,\n convert_parentheses=convert_parentheses,\n remove_duplicates=remove_duplicates,\n )\n\n def get_sentiment_label(self, id2sent, phrase_id):\n sentiment = id2sent[phrase_id]\n if sentiment <= 0.4: # negative\n return 0\n elif sentiment > 0.6: # positive\n return 1\n return -1 # neutral\n\n\nclass SST3Loader(SSTLoader):\n \"\"\"The SST3 dataset.\n\n This dataset is constructed using the Stanford Sentiment Treebank Dataset.\n This dataset contains five labels (very negative, negative, neutral,\n positive, very positive) for each sample.\n\n In the original dataset, the 5 labels: very negative, negative, neutral, positive,\n and very positive have the following cutoffs:\n [0, 0.4], (0.4, 0.6], (0.6, 1.0]\n\n This class pulls in an array of mixins for different types of functionality\n which belongs in the workflow for ingesting and transforming\n training data into a destination dataframe that can be use by Ludwig.\n \"\"\"\n\n def __init__(\n self,\n config: DatasetConfig,\n cache_dir: str | None = None,\n include_subtrees=False,\n convert_parentheses=True,\n remove_duplicates=False,\n ):\n super().__init__(\n config,\n cache_dir=cache_dir,\n include_subtrees=include_subtrees,\n convert_parentheses=convert_parentheses,\n remove_duplicates=remove_duplicates,\n )\n\n def get_sentiment_label(self, id2sent, phrase_id):\n sentiment = id2sent[phrase_id]\n if sentiment <= 0.4:\n return \"negative\"\n elif sentiment <= 0.6:\n return \"neutral\"\n elif sentiment <= 1.0:\n return \"positive\"\n return \"neutral\"\n\n\nclass SST5Loader(SSTLoader):\n \"\"\"The SST5 dataset.\n\n This dataset is constructed using the Stanford Sentiment Treebank Dataset.\n This dataset contains five labels (very negative, negative, neutral,\n positive, very positive) for each sample.\n\n In the original dataset, the 5 labels: very negative, negative, neutral, positive,\n and very positive have the following cutoffs:\n [0, 0.2], (0.2, 0.4], (0.4, 0.6], (0.6, 0.8], (0.8, 1.0]\n\n This class pulls in an array of mixins for different types of functionality\n which belongs in the workflow for ingesting and transforming\n training data into a destination dataframe that can be use by Ludwig.\n \"\"\"\n\n def __init__(\n self,\n config: DatasetConfig,\n cache_dir: str | None = None,\n include_subtrees=False,\n convert_parentheses=True,\n remove_duplicates=False,\n ):\n super().__init__(\n config,\n cache_dir=cache_dir,\n include_subtrees=include_subtrees,\n convert_parentheses=convert_parentheses,\n remove_duplicates=remove_duplicates,\n )\n\n def get_sentiment_label(self, id2sent, phrase_id):\n sentiment = id2sent[phrase_id]\n if sentiment <= 0.2:\n return \"very_negative\"\n elif sentiment <= 0.4:\n return \"negative\"\n elif sentiment <= 0.6:\n return \"neutral\"\n elif sentiment <= 0.8:\n return \"positive\"\n elif sentiment <= 1.0:\n return \"very_positive\"\n return \"neutral\"\n\n\ndef format_text(text: str):\n \"\"\"Formats text by decoding into utf-8.\"\"\"\n return \" \".join([w.encode(\"latin1\").decode(\"utf-8\") for w in text.strip().split(\" \")])\n\n\ndef convert_parentheses(text: str):\n \"\"\"Replaces -LRB- and -RRB- tokens present in SST with ( and )\"\"\"\n return text.replace(\"-LRB-\", \"(\").replace(\"-RRB-\", \")\")\n\n\ndef convert_parentheses_back(text: str):\n \"\"\"Replaces ( and ) tokens with -LRB- and -RRB-\"\"\"\n return text.replace(\"(\", \"-LRB-\").replace(\")\", \"-RRB-\")\n\n\ndef get_sentence_idcs_in_split(datasplit: pd.DataFrame, split_id: int):\n \"\"\"Given a dataset split is (1 for train, 2 for test, 3 for dev), returns the set of corresponding sentence\n indices in sentences_df.\"\"\"\n return set(datasplit[datasplit[\"splitset_label\"] == split_id][\"sentence_index\"])\n\n\ndef get_sentences_with_idcs(sentences: pd.DataFrame, sentences_idcs: set[int]):\n \"\"\"Given a set of sentence indices, returns the corresponding sentences texts in sentences.\"\"\"\n criterion = sentences[\"sentence_index\"].map(lambda x: x in sentences_idcs)\n return sentences[criterion][\"sentence\"].tolist()\n\n\ndef sentence_subtrees(sentence_idx, trees_pointers, trees_phrases):\n tree_pointers = trees_pointers[sentence_idx]\n tree_phrases = trees_phrases[sentence_idx]\n tree = SSTTree(tree_pointers, tree_phrases)\n return tree.subtrees()\n\n\ndef visit_postorder(node, visit_list):\n if node:\n visit_postorder(node.left, visit_list)\n visit_postorder(node.right, visit_list)\n visit_list.append(node.val)\n\n\nclass SSTTree:\n class Node:\n def __init__(self, key, val=None):\n self.left = None\n self.right = None\n self.key = key\n self.val = val\n\n def create_node(self, parent, i):\n if self.nodes[i] is not None:\n # already created\n return\n self.nodes[i] = self.Node(i)\n\n if parent[i] == -1:\n # is root\n self.root = self.nodes[i]\n return\n\n if self.nodes[parent[i]] is None:\n # parent not yet created\n self.create_node(parent, parent[i])\n\n # assign current node to parent\n parent = self.nodes[parent[i]]\n if parent.left is None:\n parent.left = self.nodes[i]\n else:\n parent.right = self.nodes[i]\n\n def create_tree(self, parents, tree_phrases):\n n = len(parents)\n self.nodes = [None for i in range(n)]\n self.root = [None]\n for i in range(n):\n self.create_node(parents, i)\n for i, phrase in enumerate(tree_phrases):\n self.nodes[i].val = phrase\n for node in self.nodes:\n if node.val is None:\n node.val = \" \".join((node.left.val, node.right.val))\n\n def __init__(self, tree_pointers, tree_phrases):\n self.create_tree([int(elem) - 1 for elem in tree_pointers], tree_phrases)\n\n def subtrees(self):\n visit_list = []\n visit_postorder(self.root, visit_list)\n return visit_list\n"
},
{
"path": "ludwig/datasets/model_configs/__init__.py",
"content": ""
},
{
"path": "ludwig/datasets/model_configs/adult_census_income_default.yaml",
"content": "output_features:\n - name: income\n type: category\ninput_features:\n - name: age\n type: number\n - name: workclass\n type: category\n - name: fnlwgt\n type: number\n - name: education\n type: category\n - name: education-num\n type: number\n - name: marital-status\n type: category\n - name: occupation\n type: category\n - name: relationship\n type: category\n - name: race\n type: category\n - name: sex\n type: category\n - name: capital-gain\n type: number\n - name: capital-loss\n type: number\n - name: hours-per-week\n type: number\n - name: native-country\n type: category\ncombiner:\n type: concat\n num_fc_layers: 3\n fc_size: 128\n dropout: 0.1\ntraining:\n batch_size: 256\n learning_rate: .001\n epochs: 1\n steps_per_checkpoint: 1\n"
},
{
"path": "ludwig/datasets/model_configs/allstate_claims_severity_default.yaml",
"content": "output_features:\n - name: loss\n type: number\ninput_features:\n - name: cat1\n type: category\n - name: cat2\n type: category\n - name: cat3\n type: category\n - name: cat4\n type: category\n - name: cat5\n type: category\n - name: cat6\n type: category\n - name: cat7\n type: category\n - name: cat8\n type: category\n - name: cat9\n type: category\n - name: cat10\n type: category\n - name: cat11\n type: category\n - name: cat12\n type: category\n - name: cat13\n type: category\n - name: cat14\n type: category\n - name: cat15\n type: category\n - name: cat16\n type: category\n - name: cat17\n type: category\n - name: cat18\n type: category\n - name: cat19\n type: category\n - name: cat20\n type: category\n - name: cat21\n type: category\n - name: cat22\n type: category\n - name: cat23\n type: category\n - name: cat24\n type: category\n - name: cat25\n type: category\n - name: cat26\n type: category\n - name: cat27\n type: category\n - name: cat28\n type: category\n - name: cat29\n type: category\n - name: cat30\n type: category\n - name: cat31\n type: category\n - name: cat32\n type: category\n - name: cat33\n type: category\n - name: cat34\n type: category\n - name: cat35\n type: category\n - name: cat36\n type: category\n - name: cat37\n type: category\n - name: cat38\n type: category\n - name: cat39\n type: category\n - name: cat40\n type: category\n - name: cat41\n type: category\n - name: cat42\n type: category\n - name: cat43\n type: category\n - name: cat44\n type: category\n - name: cat45\n type: category\n - name: cat46\n type: category\n - name: cat47\n type: category\n - name: cat48\n type: category\n - name: cat49\n type: category\n - name: cat50\n type: category\n - name: cat51\n type: category\n - name: cat52\n type: category\n - name: cat53\n type: category\n - name: cat54\n type: category\n - name: cat55\n type: category\n - name: cat56\n type: category\n - name: cat57\n type: category\n - name: cat58\n type: category\n - name: cat59\n type: category\n - name: cat60\n type: category\n - name: cat61\n type: category\n - name: cat62\n type: category\n - name: cat63\n type: category\n - name: cat64\n type: category\n - name: cat65\n type: category\n - name: cat66\n type: category\n - name: cat67\n type: category\n - name: cat68\n type: category\n - name: cat69\n type: category\n - name: cat70\n type: category\n - name: cat71\n type: category\n - name: cat72\n type: category\n - name: cat73\n type: category\n - name: cat74\n type: category\n - name: cat75\n type: category\n - name: cat76\n type: category\n - name: cat77\n type: category\n - name: cat78\n type: category\n - name: cat79\n type: category\n - name: cat80\n type: category\n - name: cat81\n type: category\n - name: cat82\n type: category\n - name: cat83\n type: category\n - name: cat84\n type: category\n - name: cat85\n type: category\n - name: cat86\n type: category\n - name: cat87\n type: category\n - name: cat88\n type: category\n - name: cat89\n type: category\n - name: cat90\n type: category\n - name: cat91\n type: category\n - name: cat92\n type: category\n - name: cat93\n type: category\n - name: cat94\n type: category\n - name: cat95\n type: category\n - name: cat96\n type: category\n - name: cat97\n type: category\n - name: cat98\n type: category\n - name: cat99\n type: category\n - name: cat100\n type: category\n - name: cat101\n type: category\n - name: cat102\n type: category\n - name: cat103\n type: category\n - name: cat104\n type: category\n - name: cat105\n type: category\n - name: cat106\n type: category\n - name: cat107\n type: category\n - name: cat108\n type: category\n - name: cat109\n type: category\n - name: cat110\n type: category\n - name: cat111\n type: category\n - name: cat112\n type: category\n - name: cat113\n type: category\n - name: cat114\n type: category\n - name: cat115\n type: category\n - name: cat116\n type: category\n - name: cont1\n type: number\n - name: cont2\n type: number\n - name: cont3\n type: number\n - name: cont4\n type: number\n - name: cont5\n type: number\n - name: cont6\n type: number\n - name: cont7\n type: number\n - name: cont8\n type: number\n - name: cont9\n type: number\n - name: cont10\n type: number\n - name: cont11\n type: number\n - name: cont12\n type: number\n - name: cont13\n type: number\n - name: cont14\n type: number\ncombiner:\n type: concat\n num_fc_layers: 3\n fc_size: 128\n dropout: 0.1\ntraining:\n batch_size: 256\n learning_rate: .001\n epochs: 1\n"
},
{
"path": "ludwig/datasets/model_configs/ames_housing_default.yaml",
"content": "output_features:\n - name: SalePrice\n type: number\ninput_features:\n - name: MSSubClass\n type: category\n - name: MSZoning\n type: category\n - name: LotFrontage\n type: number\n - name: LotArea\n type: number\n - name: Street\n type: category\n - name: Alley\n type: category\n - name: LotShape\n type: category\n - name: LandContour\n type: category\n - name: Utilities\n type: category\n - name: LotConfig\n type: category\n - name: LandSlope\n type: category\n - name: Neighborhood\n type: category\n - name: Condition1\n type: category\n - name: Condition2\n type: category\n - name: BldgType\n type: category\n - name: HouseStyle\n type: category\n - name: OverallQual\n type: category\n - name: OverallCond\n type: category\n - name: YearBuilt\n type: number\n - name: YearRemodAdd\n type: number\n - name: RoofStyle\n type: category\n - name: RoofMatl\n type: category\n - name: Exterior1st\n type: category\n - name: Exterior2nd\n type: category\n - name: MasVnrType\n type: category\n - name: MasVnrArea\n type: number\n - name: ExterQual\n type: category\n - name: ExterCond\n type: category\n - name: Foundation\n type: category\n - name: BsmtQual\n type: category\n - name: BsmtCond\n type: category\n - name: BsmtExposure\n type: category\n - name: BsmtFinType1\n type: category\n - name: BsmtFinSF1\n type: number\n - name: BsmtFinType2\n type: category\n - name: BsmtFinSF2\n type: number\n - name: BsmtUnfSF\n type: number\n - name: TotalBsmtSF\n type: number\n - name: Heating\n type: category\n - name: HeatingQC\n type: category\n - name: CentralAir\n type: binary\n - name: Electrical\n type: category\n - name: 1stFlrSF\n type: number\n - name: 2ndFlrSF\n type: number\n - name: LowQualFinSF\n type: number\n - name: GrLivArea\n type: number\n - name: BsmtFullBath\n type: number\n - name: BsmtHalfBath\n type: number\n - name: FullBath\n type: number\n - name: HalfBath\n type: number\n - name: BedroomAbvGr\n type: number\n - name: KitchenAbvGr\n type: number\n - name: KitchenQual\n type: category\n - name: TotRmsAbvGrd\n type: number\n - name: Functional\n type: category\n - name: Fireplaces\n type: number\n - name: FireplaceQu\n type: category\n - name: GarageType\n type: category\n - name: GarageYrBlt\n type: number\n - name: GarageFinish\n type: category\n - name: GarageCars\n type: number\n - name: GarageArea\n type: number\n - name: GarageQual\n type: category\n - name: GarageCond\n type: category\n - name: PavedDrive\n type: category\n - name: WoodDeckSF\n type: number\n - name: OpenPorchSF\n type: number\n - name: EnclosedPorch\n type: number\n - name: 3SsnPorch\n type: number\n - name: ScreenPorch\n type: number\n - name: PoolArea\n type: number\n - name: PoolQC\n type: category\n - name: Fence\n type: category\n - name: MiscFeature\n type: category\n - name: MiscVal\n type: number\n - name: MoSold\n type: category\n - name: YrSold\n type: number\n - name: SaleType\n type: category\n - name: SaleCondition\n type: category\ncombiner:\n type: concat\n num_fc_layers: 3\n fc_size: 128\n dropout: 0.1\ntraining:\n batch_size: 256\n learning_rate: .001\n epochs: 1\n"
},
{
"path": "ludwig/datasets/model_configs/bnp_claims_management_default.yaml",
"content": "output_features:\n - name: target\n type: binary\ninput_features:\n - name: v1\n type: number\n - name: v2\n type: number\n - name: v3\n type: category\n - name: v4\n type: number\n - name: v5\n type: number\n - name: v6\n type: number\n - name: v7\n type: number\n - name: v8\n type: number\n - name: v9\n type: number\n - name: v10\n type: number\n - name: v11\n type: number\n - name: v12\n type: number\n - name: v13\n type: number\n - name: v14\n type: number\n - name: v15\n type: number\n - name: v16\n type: number\n - name: v17\n type: number\n - name: v18\n type: number\n - name: v19\n type: number\n - name: v20\n type: number\n - name: v21\n type: number\n - name: v22\n type: category\n - name: v23\n type: number\n - name: v24\n type: category\n - name: v25\n type: number\n - name: v26\n type: number\n - name: v27\n type: number\n - name: v28\n type: number\n - name: v29\n type: number\n - name: v30\n type: category\n - name: v31\n type: category\n - name: v32\n type: number\n - name: v33\n type: number\n - name: v34\n type: number\n - name: v35\n type: number\n - name: v36\n type: number\n - name: v37\n type: number\n - name: v38\n type: number\n - name: v39\n type: number\n - name: v40\n type: number\n - name: v41\n type: number\n - name: v42\n type: number\n - name: v43\n type: number\n - name: v44\n type: number\n - name: v45\n type: number\n - name: v46\n type: number\n - name: v47\n type: category\n - name: v48\n type: number\n - name: v49\n type: number\n - name: v50\n type: number\n - name: v51\n type: number\n - name: v52\n type: category\n - name: v53\n type: number\n - name: v54\n type: number\n - name: v55\n type: number\n - name: v56\n type: category\n - name: v57\n type: number\n - name: v58\n type: number\n - name: v59\n type: number\n - name: v60\n type: number\n - name: v61\n type: number\n - name: v62\n type: number\n - name: v63\n type: number\n - name: v64\n type: number\n - name: v65\n type: number\n - name: v66\n type: category\n - name: v67\n type: number\n - name: v68\n type: number\n - name: v69\n type: number\n - name: v70\n type: number\n - name: v71\n type: category\n - name: v72\n type: number\n - name: v73\n type: number\n - name: v74\n type: category\n - name: v75\n type: category\n - name: v76\n type: number\n - name: v77\n type: number\n - name: v78\n type: number\n - name: v79\n type: category\n - name: v80\n type: number\n - name: v81\n type: number\n - name: v82\n type: number\n - name: v83\n type: number\n - name: v84\n type: number\n - name: v85\n type: number\n - name: v86\n type: number\n - name: v87\n type: number\n - name: v88\n type: number\n - name: v89\n type: number\n - name: v90\n type: number\n - name: v91\n type: category\n - name: v92\n type: number\n - name: v93\n type: number\n - name: v94\n type: number\n - name: v95\n type: number\n - name: v96\n type: number\n - name: v97\n type: number\n - name: v98\n type: number\n - name: v99\n type: number\n - name: v100\n type: number\n - name: v101\n type: number\n - name: v102\n type: number\n - name: v103\n type: number\n - name: v104\n type: number\n - name: v105\n type: number\n - name: v106\n type: number\n - name: v107\n type: category\n - name: v108\n type: number\n - name: v109\n type: number\n - name: v110\n type: category\n - name: v111\n type: number\n - name: v112\n type: category\n - name: v113\n type: category\n - name: v114\n type: number\n - name: v115\n type: number\n - name: v116\n type: number\n - name: v117\n type: number\n - name: v118\n type: number\n - name: v119\n type: number\n - name: v120\n type: number\n - name: v121\n type: number\n - name: v122\n type: number\n - name: v123\n type: number\n - name: v124\n type: number\n - name: v125\n type: category\n - name: v126\n type: number\n - name: v127\n type: number\n - name: v128\n type: number\n - name: v129\n type: number\n - name: v130\n type: number\n - name: v131\n type: number\ncombiner:\n type: concat\n num_fc_layers: 3\n fc_size: 128\n dropout: 0.1\ntraining:\n batch_size: 256\n learning_rate: .001\n epochs: 1\n"
},
{
"path": "ludwig/datasets/model_configs/forest_cover_default.yaml",
"content": "output_features:\n - name: Cover_Type\n type: category\ninput_features:\n - name: Elevation\n type: number\n - name: Aspect\n type: number\n - name: Slope\n type: number\n - name: Horizontal_Distance_To_Hydrology\n type: number\n - name: Vertical_Distance_To_Hydrology\n type: number\n - name: Horizontal_Distance_To_Roadways\n type: number\n - name: Hillshade_9am\n type: number\n - name: Hillshade_Noon\n type: number\n - name: Hillshade_3pm\n type: number\n - name: Horizontal_Distance_To_Fire_Points\n type: number\n - name: Wilderness_Area\n type: category\n - name: Soil_Type\n type: category\ncombiner:\n type: concat\n num_fc_layers: 3\n fc_size: 128\n dropout: 0.1\ntraining:\n batch_size: 256\n learning_rate: .001\n epochs: 1\n"
},
{
"path": "ludwig/datasets/model_configs/higgs_best.yaml",
"content": "output_features:\n - name: label\n type: binary\n weight_regularization: null\ninput_features:\n - name: lepton_pT\n type: number\n - name: lepton_eta\n type: number\n - name: lepton_phi\n type: number\n - name: missing_energy_magnitude\n type: number\n - name: missing_energy_phi\n type: number\n - name: jet_1_pt\n type: number\n - name: jet_1_eta\n type: number\n - name: jet_1_phi\n type: number\n - name: jet_1_b-tag\n type: number\n - name: jet_2_pt\n type: number\n - name: jet_2_eta\n type: number\n - name: jet_2_phi\n type: number\n - name: jet_2_b-tag\n type: number\n - name: jet_3_pt\n type: number\n - name: jet_3_eta\n type: number\n - name: jet_3_phi\n type: number\n - name: jet_3_b-tag\n type: number\n - name: jet_4_pt\n type: number\n - name: jet_4_eta\n type: number\n - name: jet_4_phi\n type: number\n - name: jet_4_b-tag\n type: number\n - name: m_jj\n type: number\n - name: m_jjj\n type: number\n - name: m_lv\n type: number\n - name: m_jlv\n type: number\n - name: m_bb\n type: number\n - name: m_wbb\n type: number\n - name: m_wwbb\n type: number\ncombiner:\n type: tabnet\n bn_momentum: 0.95\n bn_virtual_bs: 1024\n dropout: 0.05252744300130521\n fc_size: 128\n num_fc_layers: 3\n num_steps: 3\n output_size: 128\n relaxation_factor: 1.5\n size: 32\n sparsity: 0.0001\ntraining:\n batch_size: 8192\n learning_rate: 0.01\n shuffle_buffer_size: 1000000\n should_shuffle: true\n eval_batch_size: 500000 #4096 # 65536 131072 262144 524288\n epochs: 300\n early_stop: 30\n optimizer:\n type: adam\n learning_rate_scheduler:\n decay: exponential\n decay_rate: 0.8\n decay_steps: 20000\n regularization_lambda: 1\n validation_field: label\n"
},
{
"path": "ludwig/datasets/model_configs/higgs_default.yaml",
"content": "output_features:\n - name: label\n type: binary\n weight_regularization: null\ninput_features:\n - name: lepton_pT\n type: number\n - name: lepton_eta\n type: number\n - name: lepton_phi\n type: number\n - name: missing_energy_magnitude\n type: number\n - name: missing_energy_phi\n type: number\n - name: jet_1_pt\n type: number\n - name: jet_1_eta\n type: number\n - name: jet_1_phi\n type: number\n - name: jet_1_b-tag\n type: number\n - name: jet_2_pt\n type: number\n - name: jet_2_eta\n type: number\n - name: jet_2_phi\n type: number\n - name: jet_2_b-tag\n type: number\n - name: jet_3_pt\n type: number\n - name: jet_3_eta\n type: number\n - name: jet_3_phi\n type: number\n - name: jet_3_b-tag\n type: number\n - name: jet_4_pt\n type: number\n - name: jet_4_eta\n type: number\n - name: jet_4_phi\n type: number\n - name: jet_4_b-tag\n type: number\n - name: m_jj\n type: number\n - name: m_jjj\n type: number\n - name: m_lv\n type: number\n - name: m_jlv\n type: number\n - name: m_bb\n type: number\n - name: m_wbb\n type: number\n - name: m_wwbb\n type: number\ncombiner:\n type: concat\n num_fc_layers: 3\n fc_size: 128\n dropout: 0.1\ntraining:\n batch_size: 256\n learning_rate: .001\n epochs: 1\n"
},
{
"path": "ludwig/datasets/model_configs/ieee_fraud_default.yaml",
"content": "output_features:\n - name: isFraud\n type: binary\ninput_features:\n - name: TransactionDT\n type: number\n - name: TransactionAmt\n type: number\n - name: ProductCD\n type: category\n - name: card1\n type: number\n - name: card2\n type: number\n - name: card3\n type: number\n - name: card4\n type: category\n - name: card5\n type: number\n - name: card6\n type: category\n - name: addr1\n type: number\n - name: addr2\n type: number\n - name: dist1\n type: number\n - name: dist2\n type: number\n - name: P_emaildomain\n type: category\n - name: R_emaildomain\n type: number\n - name: C1\n type: number\n - name: C2\n type: number\n - name: C3\n type: number\n - name: C4\n type: number\n - name: C5\n type: number\n - name: C6\n type: number\n - name: C7\n type: number\n - name: C8\n type: number\n - name: C9\n type: number\n - name: C10\n type: number\n - name: C11\n type: number\n - name: C12\n type: number\n - name: C13\n type: number\n - name: C14\n type: number\n - name: D1\n type: number\n - name: D2\n type: number\n - name: D3\n type: number\n - name: D4\n type: number\n - name: D5\n type: number\n - name: D6\n type: number\n - name: D7\n type: number\n - name: D8\n type: number\n - name: D9\n type: number\n - name: D10\n type: number\n - name: D11\n type: number\n - name: D12\n type: number\n - name: D13\n type: number\n - name: D14\n type: number\n - name: D15\n type: number\n - name: M1\n type: category\n - name: M2\n type: category\n - name: M3\n type: category\n - name: M4\n type: category\n - name: M5\n type: category\n - name: M6\n type: category\n - name: M7\n type: category\n - name: M8\n type: category\n - name: M9\n type: category\n - name: V1\n type: number\n - name: V2\n type: number\n - name: V3\n type: number\n - name: V4\n type: number\n - name: V5\n type: number\n - name: V6\n type: number\n - name: V7\n type: number\n - name: V8\n type: number\n - name: V9\n type: number\n - name: V10\n type: number\n - name: V11\n type: number\n - name: V12\n type: number\n - name: V13\n type: number\n - name: V14\n type: number\n - name: V15\n type: number\n - name: V16\n type: number\n - name: V17\n type: number\n - name: V18\n type: number\n - name: V19\n type: number\n - name: V20\n type: number\n - name: V21\n type: number\n - name: V22\n type: number\n - name: V23\n type: number\n - name: V24\n type: number\n - name: V25\n type: number\n - name: V26\n type: number\n - name: V27\n type: number\n - name: V28\n type: number\n - name: V29\n type: number\n - name: V30\n type: number\n - name: V31\n type: number\n - name: V32\n type: number\n - name: V33\n type: number\n - name: V34\n type: number\n - name: V35\n type: number\n - name: V36\n type: number\n - name: V37\n type: number\n - name: V38\n type: number\n - name: V39\n type: number\n - name: V40\n type: number\n - name: V41\n type: number\n - name: V42\n type: number\n - name: V43\n type: number\n - name: V44\n type: number\n - name: V45\n type: number\n - name: V46\n type: number\n - name: V47\n type: number\n - name: V48\n type: number\n - name: V49\n type: number\n - name: V50\n type: number\n - name: V51\n type: number\n - name: V52\n type: number\n - name: V53\n type: number\n - name: V54\n type: number\n - name: V55\n type: number\n - name: V56\n type: number\n - name: V57\n type: number\n - name: V58\n type: number\n - name: V59\n type: number\n - name: V60\n type: number\n - name: V61\n type: number\n - name: V62\n type: number\n - name: V63\n type: number\n - name: V64\n type: number\n - name: V65\n type: number\n - name: V66\n type: number\n - name: V67\n type: number\n - name: V68\n type: number\n - name: V69\n type: number\n - name: V70\n type: number\n - name: V71\n type: number\n - name: V72\n type: number\n - name: V73\n type: number\n - name: V74\n type: number\n - name: V75\n type: number\n - name: V76\n type: number\n - name: V77\n type: number\n - name: V78\n type: number\n - name: V79\n type: number\n - name: V80\n type: number\n - name: V81\n type: number\n - name: V82\n type: number\n - name: V83\n type: number\n - name: V84\n type: number\n - name: V85\n type: number\n - name: V86\n type: number\n - name: V87\n type: number\n - name: V88\n type: number\n - name: V89\n type: number\n - name: V90\n type: number\n - name: V91\n type: number\n - name: V92\n type: number\n - name: V93\n type: number\n - name: V94\n type: number\n - name: V95\n type: number\n - name: V96\n type: number\n - name: V97\n type: number\n - name: V98\n type: number\n - name: V99\n type: number\n - name: V100\n type: number\n - name: V101\n type: number\n - name: V102\n type: number\n - name: V103\n type: number\n - name: V104\n type: number\n - name: V105\n type: number\n - name: V106\n type: number\n - name: V107\n type: number\n - name: V108\n type: number\n - name: V109\n type: number\n - name: V110\n type: number\n - name: V111\n type: number\n - name: V112\n type: number\n - name: V113\n type: number\n - name: V114\n type: number\n - name: V115\n type: number\n - name: V116\n type: number\n - name: V117\n type: number\n - name: V118\n type: number\n - name: V119\n type: number\n - name: V120\n type: number\n - name: V121\n type: number\n - name: V122\n type: number\n - name: V123\n type: number\n - name: V124\n type: number\n - name: V125\n type: number\n - name: V126\n type: number\n - name: V127\n type: number\n - name: V128\n type: number\n - name: V129\n type: number\n - name: V130\n type: number\n - name: V131\n type: number\n - name: V132\n type: number\n - name: V133\n type: number\n - name: V134\n type: number\n - name: V135\n type: number\n - name: V136\n type: number\n - name: V137\n type: number\n - name: V138\n type: number\n - name: V139\n type: number\n - name: V140\n type: number\n - name: V141\n type: number\n - name: V142\n type: number\n - name: V143\n type: number\n - name: V144\n type: number\n - name: V145\n type: number\n - name: V146\n type: number\n - name: V147\n type: number\n - name: V148\n type: number\n - name: V149\n type: number\n - name: V150\n type: number\n - name: V151\n type: number\n - name: V152\n type: number\n - name: V153\n type: number\n - name: V154\n type: number\n - name: V155\n type: number\n - name: V156\n type: number\n - name: V157\n type: number\n - name: V158\n type: number\n - name: V159\n type: number\n - name: V160\n type: number\n - name: V161\n type: number\n - name: V162\n type: number\n - name: V163\n type: number\n - name: V164\n type: number\n - name: V165\n type: number\n - name: V166\n type: number\n - name: V167\n type: number\n - name: V168\n type: number\n - name: V169\n type: number\n - name: V170\n type: number\n - name: V171\n type: number\n - name: V172\n type: number\n - name: V173\n type: number\n - name: V174\n type: number\n - name: V175\n type: number\n - name: V176\n type: number\n - name: V177\n type: number\n - name: V178\n type: number\n - name: V179\n type: number\n - name: V180\n type: number\n - name: V181\n type: number\n - name: V182\n type: number\n - name: V183\n type: number\n - name: V184\n type: number\n - name: V185\n type: number\n - name: V186\n type: number\n - name: V187\n type: number\n - name: V188\n type: number\n - name: V189\n type: number\n - name: V190\n type: number\n - name: V191\n type: number\n - name: V192\n type: number\n - name: V193\n type: number\n - name: V194\n type: number\n - name: V195\n type: number\n - name: V196\n type: number\n - name: V197\n type: number\n - name: V198\n type: number\n - name: V199\n type: number\n - name: V200\n type: number\n - name: V201\n type: number\n - name: V202\n type: number\n - name: V203\n type: number\n - name: V204\n type: number\n - name: V205\n type: number\n - name: V206\n type: number\n - name: V207\n type: number\n - name: V208\n type: number\n - name: V209\n type: number\n - name: V210\n type: number\n - name: V211\n type: number\n - name: V212\n type: number\n - name: V213\n type: number\n - name: V214\n type: number\n - name: V215\n type: number\n - name: V216\n type: number\n - name: V217\n type: number\n - name: V218\n type: number\n - name: V219\n type: number\n - name: V220\n type: number\n - name: V221\n type: number\n - name: V222\n type: number\n - name: V223\n type: number\n - name: V224\n type: number\n - name: V225\n type: number\n - name: V226\n type: number\n - name: V227\n type: number\n - name: V228\n type: number\n - name: V229\n type: number\n - name: V230\n type: number\n - name: V231\n type: number\n - name: V232\n type: number\n - name: V233\n type: number\n - name: V234\n type: number\n - name: V235\n type: number\n - name: V236\n type: number\n - name: V237\n type: number\n - name: V238\n type: number\n - name: V239\n type: number\n - name: V240\n type: number\n - name: V241\n type: number\n - name: V242\n type: number\n - name: V243\n type: number\n - name: V244\n type: number\n - name: V245\n type: number\n - name: V246\n type: number\n - name: V247\n type: number\n - name: V248\n type: number\n - name: V249\n type: number\n - name: V250\n type: number\n - name: V251\n type: number\n - name: V252\n type: number\n - name: V253\n type: number\n - name: V254\n type: number\n - name: V255\n type: number\n - name: V256\n type: number\n - name: V257\n type: number\n - name: V258\n type: number\n - name: V259\n type: number\n - name: V260\n type: number\n - name: V261\n type: number\n - name: V262\n type: number\n - name: V263\n type: number\n - name: V264\n type: number\n - name: V265\n type: number\n - name: V266\n type: number\n - name: V267\n type: number\n - name: V268\n type: number\n - name: V269\n type: number\n - name: V270\n type: number\n - name: V271\n type: number\n - name: V272\n type: number\n - name: V273\n type: number\n - name: V274\n type: number\n - name: V275\n type: number\n - name: V276\n type: number\n - name: V277\n type: number\n - name: V278\n type: number\n - name: V279\n type: number\n - name: V280\n type: number\n - name: V281\n type: number\n - name: V282\n type: number\n - name: V283\n type: number\n - name: V284\n type: number\n - name: V285\n type: number\n - name: V286\n type: number\n - name: V287\n type: number\n - name: V288\n type: number\n - name: V289\n type: number\n - name: V290\n type: number\n - name: V291\n type: number\n - name: V292\n type: number\n - name: V293\n type: number\n - name: V294\n type: number\n - name: V295\n type: number\n - name: V296\n type: number\n - name: V297\n type: number\n - name: V298\n type: number\n - name: V299\n type: number\n - name: V300\n type: number\n - name: V301\n type: number\n - name: V302\n type: number\n - name: V303\n type: number\n - name: V304\n type: number\n - name: V305\n type: number\n - name: V306\n type: number\n - name: V307\n type: number\n - name: V308\n type: number\n - name: V309\n type: number\n - name: V310\n type: number\n - name: V311\n type: number\n - name: V312\n type: number\n - name: V313\n type: number\n - name: V314\n type: number\n - name: V315\n type: number\n - name: V316\n type: number\n - name: V317\n type: number\n - name: V318\n type: number\n - name: V319\n type: number\n - name: V320\n type: number\n - name: V321\n type: number\n - name: V322\n type: number\n - name: V323\n type: number\n - name: V324\n type: number\n - name: V325\n type: number\n - name: V326\n type: number\n - name: V327\n type: number\n - name: V328\n type: number\n - name: V329\n type: number\n - name: V330\n type: number\n - name: V331\n type: number\n - name: V332\n type: number\n - name: V333\n type: number\n - name: V334\n type: number\n - name: V335\n type: number\n - name: V336\n type: number\n - name: V337\n type: number\n - name: V338\n type: number\n - name: V339\n type: number\n - name: id_01\n type: number\n - name: id_02\n type: number\n - name: id_03\n type: number\n - name: id_04\n type: number\n - name: id_05\n type: number\n - name: id_06\n type: number\n - name: id_07\n type: number\n - name: id_08\n type: number\n - name: id_09\n type: number\n - name: id_10\n type: number\n - name: id_11\n type: number\n - name: id_12\n type: number\n - name: id_13\n type: number\n - name: id_14\n type: number\n - name: id_15\n type: number\n - name: id_16\n type: number\n - name: id_17\n type: number\n - name: id_18\n type: number\n - name: id_19\n type: number\n - name: id_20\n type: number\n - name: id_21\n type: number\n - name: id_22\n type: number\n - name: id_23\n type: number\n - name: id_24\n type: number\n - name: id_25\n type: number\n - name: id_26\n type: number\n - name: id_27\n type: number\n - name: id_28\n type: number\n - name: id_29\n type: number\n - name: id_30\n type: number\n - name: id_31\n type: number\n - name: id_32\n type: number\n - name: id_33\n type: number\n - name: id_34\n type: number\n - name: id_35\n type: number\n - name: id_36\n type: number\n - name: id_37\n type: number\n - name: id_38\n type: number\n - name: DeviceType\n type: number\n - name: DeviceInfo\n type: number\ncombiner:\n type: concat\n num_fc_layers: 3\n fc_size: 128\n dropout: 0.1\ntraining:\n batch_size: 256\n learning_rate: .001\n epochs: 1\n"
},
{
"path": "ludwig/datasets/model_configs/mercedes_benz_greener_default.yaml",
"content": "output_features:\n - name: y\n type: number\ninput_features:\n - name: X0\n type: category\n - name: X1\n type: category\n - name: X2\n type: category\n - name: X3\n type: category\n - name: X4\n type: category\n - name: X5\n type: category\n - name: X6\n type: category\n - name: X8\n type: category\n - name: X10\n type: binary\n - name: X11\n type: binary\n - name: X12\n type: binary\n - name: X13\n type: binary\n - name: X14\n type: binary\n - name: X15\n type: binary\n - name: X16\n type: binary\n - name: X17\n type: binary\n - name: X18\n type: binary\n - name: X19\n type: binary\n - name: X20\n type: binary\n - name: X21\n type: binary\n - name: X22\n type: binary\n - name: X23\n type: binary\n - name: X24\n type: binary\n - name: X26\n type: binary\n - name: X27\n type: binary\n - name: X28\n type: binary\n - name: X29\n type: binary\n - name: X30\n type: binary\n - name: X31\n type: binary\n - name: X32\n type: binary\n - name: X33\n type: binary\n - name: X34\n type: binary\n - name: X35\n type: binary\n - name: X36\n type: binary\n - name: X37\n type: binary\n - name: X38\n type: binary\n - name: X39\n type: binary\n - name: X40\n type: binary\n - name: X41\n type: binary\n - name: X42\n type: binary\n - name: X43\n type: binary\n - name: X44\n type: binary\n - name: X45\n type: binary\n - name: X46\n type: binary\n - name: X47\n type: binary\n - name: X48\n type: binary\n - name: X49\n type: binary\n - name: X50\n type: binary\n - name: X51\n type: binary\n - name: X52\n type: binary\n - name: X53\n type: binary\n - name: X54\n type: binary\n - name: X55\n type: binary\n - name: X56\n type: binary\n - name: X57\n type: binary\n - name: X58\n type: binary\n - name: X59\n type: binary\n - name: X60\n type: binary\n - name: X61\n type: binary\n - name: X62\n type: binary\n - name: X63\n type: binary\n - name: X64\n type: binary\n - name: X65\n type: binary\n - name: X66\n type: binary\n - name: X67\n type: binary\n - name: X68\n type: binary\n - name: X69\n type: binary\n - name: X70\n type: binary\n - name: X71\n type: binary\n - name: X73\n type: binary\n - name: X74\n type: binary\n - name: X75\n type: binary\n - name: X76\n type: binary\n - name: X77\n type: binary\n - name: X78\n type: binary\n - name: X79\n type: binary\n - name: X80\n type: binary\n - name: X81\n type: binary\n - name: X82\n type: binary\n - name: X83\n type: binary\n - name: X84\n type: binary\n - name: X85\n type: binary\n - name: X86\n type: binary\n - name: X87\n type: binary\n - name: X88\n type: binary\n - name: X89\n type: binary\n - name: X90\n type: binary\n - name: X91\n type: binary\n - name: X92\n type: binary\n - name: X93\n type: binary\n - name: X94\n type: binary\n - name: X95\n type: binary\n - name: X96\n type: binary\n - name: X97\n type: binary\n - name: X98\n type: binary\n - name: X99\n type: binary\n - name: X100\n type: binary\n - name: X101\n type: binary\n - name: X102\n type: binary\n - name: X103\n type: binary\n - name: X104\n type: binary\n - name: X105\n type: binary\n - name: X106\n type: binary\n - name: X107\n type: binary\n - name: X108\n type: binary\n - name: X109\n type: binary\n - name: X110\n type: binary\n - name: X111\n type: binary\n - name: X112\n type: binary\n - name: X113\n type: binary\n - name: X114\n type: binary\n - name: X115\n type: binary\n - name: X116\n type: binary\n - name: X117\n type: binary\n - name: X118\n type: binary\n - name: X119\n type: binary\n - name: X120\n type: binary\n - name: X122\n type: binary\n - name: X123\n type: binary\n - name: X124\n type: binary\n - name: X125\n type: binary\n - name: X126\n type: binary\n - name: X127\n type: binary\n - name: X128\n type: binary\n - name: X129\n type: binary\n - name: X130\n type: binary\n - name: X131\n type: binary\n - name: X132\n type: binary\n - name: X133\n type: binary\n - name: X134\n type: binary\n - name: X135\n type: binary\n - name: X136\n type: binary\n - name: X137\n type: binary\n - name: X138\n type: binary\n - name: X139\n type: binary\n - name: X140\n type: binary\n - name: X141\n type: binary\n - name: X142\n type: binary\n - name: X143\n type: binary\n - name: X144\n type: binary\n - name: X145\n type: binary\n - name: X146\n type: binary\n - name: X147\n type: binary\n - name: X148\n type: binary\n - name: X150\n type: binary\n - name: X151\n type: binary\n - name: X152\n type: binary\n - name: X153\n type: binary\n - name: X154\n type: binary\n - name: X155\n type: binary\n - name: X156\n type: binary\n - name: X157\n type: binary\n - name: X158\n type: binary\n - name: X159\n type: binary\n - name: X160\n type: binary\n - name: X161\n type: binary\n - name: X162\n type: binary\n - name: X163\n type: binary\n - name: X164\n type: binary\n - name: X165\n type: binary\n - name: X166\n type: binary\n - name: X167\n type: binary\n - name: X168\n type: binary\n - name: X169\n type: binary\n - name: X170\n type: binary\n - name: X171\n type: binary\n - name: X172\n type: binary\n - name: X173\n type: binary\n - name: X174\n type: binary\n - name: X175\n type: binary\n - name: X176\n type: binary\n - name: X177\n type: binary\n - name: X178\n type: binary\n - name: X179\n type: binary\n - name: X180\n type: binary\n - name: X181\n type: binary\n - name: X182\n type: binary\n - name: X183\n type: binary\n - name: X184\n type: binary\n - name: X185\n type: binary\n - name: X186\n type: binary\n - name: X187\n type: binary\n - name: X189\n type: binary\n - name: X190\n type: binary\n - name: X191\n type: binary\n - name: X192\n type: binary\n - name: X194\n type: binary\n - name: X195\n type: binary\n - name: X196\n type: binary\n - name: X197\n type: binary\n - name: X198\n type: binary\n - name: X199\n type: binary\n - name: X200\n type: binary\n - name: X201\n type: binary\n - name: X202\n type: binary\n - name: X203\n type: binary\n - name: X204\n type: binary\n - name: X205\n type: binary\n - name: X206\n type: binary\n - name: X207\n type: binary\n - name: X208\n type: binary\n - name: X209\n type: binary\n - name: X210\n type: binary\n - name: X211\n type: binary\n - name: X212\n type: binary\n - name: X213\n type: binary\n - name: X214\n type: binary\n - name: X215\n type: binary\n - name: X216\n type: binary\n - name: X217\n type: binary\n - name: X218\n type: binary\n - name: X219\n type: binary\n - name: X220\n type: binary\n - name: X221\n type: binary\n - name: X222\n type: binary\n - name: X223\n type: binary\n - name: X224\n type: binary\n - name: X225\n type: binary\n - name: X226\n type: binary\n - name: X227\n type: binary\n - name: X228\n type: binary\n - name: X229\n type: binary\n - name: X230\n type: binary\n - name: X231\n type: binary\n - name: X232\n type: binary\n - name: X233\n type: binary\n - name: X234\n type: binary\n - name: X235\n type: binary\n - name: X236\n type: binary\n - name: X237\n type: binary\n - name: X238\n type: binary\n - name: X239\n type: binary\n - name: X240\n type: binary\n - name: X241\n type: binary\n - name: X242\n type: binary\n - name: X243\n type: binary\n - name: X244\n type: binary\n - name: X245\n type: binary\n - name: X246\n type: binary\n - name: X247\n type: binary\n - name: X248\n type: binary\n - name: X249\n type: binary\n - name: X250\n type: binary\n - name: X251\n type: binary\n - name: X252\n type: binary\n - name: X253\n type: binary\n - name: X254\n type: binary\n - name: X255\n type: binary\n - name: X256\n type: binary\n - name: X257\n type: binary\n - name: X258\n type: binary\n - name: X259\n type: binary\n - name: X260\n type: binary\n - name: X261\n type: binary\n - name: X262\n type: binary\n - name: X263\n type: binary\n - name: X264\n type: binary\n - name: X265\n type: binary\n - name: X266\n type: binary\n - name: X267\n type: binary\n - name: X268\n type: binary\n - name: X269\n type: binary\n - name: X270\n type: binary\n - name: X271\n type: binary\n - name: X272\n type: binary\n - name: X273\n type: binary\n - name: X274\n type: binary\n - name: X275\n type: binary\n - name: X276\n type: binary\n - name: X277\n type: binary\n - name: X278\n type: binary\n - name: X279\n type: binary\n - name: X280\n type: binary\n - name: X281\n type: binary\n - name: X282\n type: binary\n - name: X283\n type: binary\n - name: X284\n type: binary\n - name: X285\n type: binary\n - name: X286\n type: binary\n - name: X287\n type: binary\n - name: X288\n type: binary\n - name: X289\n type: binary\n - name: X290\n type: binary\n - name: X291\n type: binary\n - name: X292\n type: binary\n - name: X293\n type: binary\n - name: X294\n type: binary\n - name: X295\n type: binary\n - name: X296\n type: binary\n - name: X297\n type: binary\n - name: X298\n type: binary\n - name: X299\n type: binary\n - name: X300\n type: binary\n - name: X301\n type: binary\n - name: X302\n type: binary\n - name: X304\n type: binary\n - name: X305\n type: binary\n - name: X306\n type: binary\n - name: X307\n type: binary\n - name: X308\n type: binary\n - name: X309\n type: binary\n - name: X310\n type: binary\n - name: X311\n type: binary\n - name: X312\n type: binary\n - name: X313\n type: binary\n - name: X314\n type: binary\n - name: X315\n type: binary\n - name: X316\n type: binary\n - name: X317\n type: binary\n - name: X318\n type: binary\n - name: X319\n type: binary\n - name: X320\n type: binary\n - name: X321\n type: binary\n - name: X322\n type: binary\n - name: X323\n type: binary\n - name: X324\n type: binary\n - name: X325\n type: binary\n - name: X326\n type: binary\n - name: X327\n type: binary\n - name: X328\n type: binary\n - name: X329\n type: binary\n - name: X330\n type: binary\n - name: X331\n type: binary\n - name: X332\n type: binary\n - name: X333\n type: binary\n - name: X334\n type: binary\n - name: X335\n type: binary\n - name: X336\n type: binary\n - name: X337\n type: binary\n - name: X338\n type: binary\n - name: X339\n type: binary\n - name: X340\n type: binary\n - name: X341\n type: binary\n - name: X342\n type: binary\n - name: X343\n type: binary\n - name: X344\n type: binary\n - name: X345\n type: binary\n - name: X346\n type: binary\n - name: X347\n type: binary\n - name: X348\n type: binary\n - name: X349\n type: binary\n - name: X350\n type: binary\n - name: X351\n type: binary\n - name: X352\n type: binary\n - name: X353\n type: binary\n - name: X354\n type: binary\n - name: X355\n type: binary\n - name: X356\n type: binary\n - name: X357\n type: binary\n - name: X358\n type: binary\n - name: X359\n type: binary\n - name: X360\n type: binary\n - name: X361\n type: binary\n - name: X362\n type: binary\n - name: X363\n type: binary\n - name: X364\n type: binary\n - name: X365\n type: binary\n - name: X366\n type: binary\n - name: X367\n type: binary\n - name: X368\n type: binary\n - name: X369\n type: binary\n - name: X370\n type: binary\n - name: X371\n type: binary\n - name: X372\n type: binary\n - name: X373\n type: binary\n - name: X374\n type: binary\n - name: X375\n type: binary\n - name: X376\n type: binary\n - name: X377\n type: binary\n - name: X378\n type: binary\n - name: X379\n type: binary\n - name: X380\n type: binary\n - name: X382\n type: binary\n - name: X383\n type: binary\n - name: X384\n type: binary\n - name: X385\n type: binary\ncombiner:\n type: concat\n num_fc_layers: 3\n fc_size: 128\n dropout: 0.1\ntraining:\n batch_size: 256\n learning_rate: .001\n epochs: 1\n"
},
{
"path": "ludwig/datasets/model_configs/mnist_default.yaml",
"content": "output_features:\n - name: label\n type: category\ninput_features:\n - name: image_path\n type: image\n preprocessing:\n num_processes: 4\n encoder: stacked_cnn\n conv_layers:\n - num_filters: 32\n filter_size: 3\n pool_size: 2\n pool_stride: 2\n - num_filters: 64\n filter_size: 3\n pool_size: 2\n pool_stride: 2\n dropout: 0.4\n fc_layers:\n - output_size: 128\n dropout: 0.4\ntrainer:\n epochs: 1\n"
},
{
"path": "ludwig/datasets/model_configs/mushroom_edibility_default.yaml",
"content": "output_features:\n - name: class\n type: category\ninput_features:\n - name: cap-shape\n type: category\n - name: cap-surface\n type: category\n - name: cap-color\n type: category\n - name: bruises?\n type: category\n - name: odor\n type: category\n - name: gill-attachment\n type: category\n - name: gill-spacing\n type: category\n - name: gill-size\n type: category\n - name: gill-color\n type: category\n - name: stalk-shape\n type: category\n - name: stalk-root\n type: category\n - name: stalk-surface-above-ring\n type: category\n - name: stalk-surface-below-ring\n type: category\n - name: stalk-color-above-ring\n type: category\n - name: stalk-color-below-ring\n type: category\n - name: veil-type\n type: category\n - name: veil-color\n type: category\n - name: ring-number\n type: category\n - name: ring-type\n type: category\n - name: spore-print-color\n type: category\n - name: population\n type: category\n - name: habitat\n type: category\ncombiner:\n type: concat\n num_fc_layers: 3\n fc_size: 128\n dropout: 0.1\ntraining:\n batch_size: 256\n learning_rate: .001\n epochs: 1\n"
},
{
"path": "ludwig/datasets/model_configs/otto_group_product_default.yaml",
"content": "output_features:\n - name: target\n type: category\ninput_features:\n - name: feat_1\n type: number\n - name: feat_2\n type: number\n - name: feat_3\n type: number\n - name: feat_4\n type: number\n - name: feat_5\n type: number\n - name: feat_6\n type: number\n - name: feat_7\n type: number\n - name: feat_8\n type: number\n - name: feat_9\n type: number\n - name: feat_10\n type: number\n - name: feat_11\n type: number\n - name: feat_12\n type: number\n - name: feat_13\n type: number\n - name: feat_14\n type: number\n - name: feat_15\n type: number\n - name: feat_16\n type: number\n - name: feat_17\n type: number\n - name: feat_18\n type: number\n - name: feat_19\n type: number\n - name: feat_20\n type: number\n - name: feat_21\n type: number\n - name: feat_22\n type: number\n - name: feat_23\n type: number\n - name: feat_24\n type: number\n - name: feat_25\n type: number\n - name: feat_26\n type: number\n - name: feat_27\n type: number\n - name: feat_28\n type: number\n - name: feat_29\n type: number\n - name: feat_30\n type: number\n - name: feat_31\n type: number\n - name: feat_32\n type: number\n - name: feat_33\n type: number\n - name: feat_34\n type: number\n - name: feat_35\n type: number\n - name: feat_36\n type: number\n - name: feat_37\n type: number\n - name: feat_38\n type: number\n - name: feat_39\n type: number\n - name: feat_40\n type: number\n - name: feat_41\n type: number\n - name: feat_42\n type: number\n - name: feat_43\n type: number\n - name: feat_44\n type: number\n - name: feat_45\n type: number\n - name: feat_46\n type: number\n - name: feat_47\n type: number\n - name: feat_48\n type: number\n - name: feat_49\n type: number\n - name: feat_50\n type: number\n - name: feat_51\n type: number\n - name: feat_52\n type: number\n - name: feat_53\n type: number\n - name: feat_54\n type: number\n - name: feat_55\n type: number\n - name: feat_56\n type: number\n - name: feat_57\n type: number\n - name: feat_58\n type: number\n - name: feat_59\n type: number\n - name: feat_60\n type: number\n - name: feat_61\n type: number\n - name: feat_62\n type: number\n - name: feat_63\n type: number\n - name: feat_64\n type: number\n - name: feat_65\n type: number\n - name: feat_66\n type: number\n - name: feat_67\n type: number\n - name: feat_68\n type: number\n - name: feat_69\n type: number\n - name: feat_70\n type: number\n - name: feat_71\n type: number\n - name: feat_72\n type: number\n - name: feat_73\n type: number\n - name: feat_74\n type: number\n - name: feat_75\n type: number\n - name: feat_76\n type: number\n - name: feat_77\n type: number\n - name: feat_78\n type: number\n - name: feat_79\n type: number\n - name: feat_80\n type: number\n - name: feat_81\n type: number\n - name: feat_82\n type: number\n - name: feat_83\n type: number\n - name: feat_84\n type: number\n - name: feat_85\n type: number\n - name: feat_86\n type: number\n - name: feat_87\n type: number\n - name: feat_88\n type: number\n - name: feat_89\n type: number\n - name: feat_90\n type: number\n - name: feat_91\n type: number\n - name: feat_92\n type: number\n - name: feat_93\n type: number\ncombiner:\n type: concat\n num_fc_layers: 3\n fc_size: 128\n dropout: 0.1\ntraining:\n batch_size: 256\n learning_rate: .001\n epochs: 1\n"
},
{
"path": "ludwig/datasets/model_configs/poker_hand_default.yaml",
"content": "output_features:\n - name: hand\n type: category\ninput_features:\n - name: S1\n type: category\n - name: C1\n type: category\n - name: S2\n type: category\n - name: C2\n type: category\n - name: S3\n type: category\n - name: C3\n type: category\n - name: S4\n type: category\n - name: C4\n type: category\n - name: S5\n type: category\n - name: C5\n type: category\ncombiner:\n type: concat\n num_fc_layers: 3\n fc_size: 128\n dropout: 0.1\ntraining:\n batch_size: 256\n learning_rate: .001\n epochs: 1\n"
},
{
"path": "ludwig/datasets/model_configs/porto_seguro_safe_driver_default.yaml",
"content": "output_features:\n - name: target\n type: binary\ninput_features:\n - name: ps_ind_01\n type: number\n - name: ps_ind_02_cat\n type: category\n - name: ps_ind_03\n type: number\n - name: ps_ind_04_cat\n type: category\n - name: ps_ind_05_cat\n type: category\n - name: ps_ind_06_bin\n type: binary\n - name: ps_ind_07_bin\n type: binary\n - name: ps_ind_08_bin\n type: binary\n - name: ps_ind_09_bin\n type: binary\n - name: ps_ind_10_bin\n type: binary\n - name: ps_ind_11_bin\n type: binary\n - name: ps_ind_12_bin\n type: binary\n - name: ps_ind_13_bin\n type: binary\n - name: ps_ind_14\n type: number\n - name: ps_ind_15\n type: number\n - name: ps_ind_16_bin\n type: binary\n - name: ps_ind_17_bin\n type: binary\n - name: ps_ind_18_bin\n type: binary\n - name: ps_reg_01\n type: number\n - name: ps_reg_02\n type: number\n - name: ps_reg_03\n type: number\n - name: ps_car_01_cat\n type: category\n - name: ps_car_02_cat\n type: category\n - name: ps_car_03_cat\n type: category\n - name: ps_car_04_cat\n type: category\n - name: ps_car_05_cat\n type: category\n - name: ps_car_06_cat\n type: category\n - name: ps_car_07_cat\n type: category\n - name: ps_car_08_cat\n type: category\n - name: ps_car_09_cat\n type: category\n - name: ps_car_10_cat\n type: category\n - name: ps_car_11_cat\n type: category\n - name: ps_car_11\n type: number\n - name: ps_car_12\n type: number\n - name: ps_car_13\n type: number\n - name: ps_car_14\n type: number\n - name: ps_car_15\n type: number\n - name: ps_calc_01\n type: number\n - name: ps_calc_02\n type: number\n - name: ps_calc_03\n type: number\n - name: ps_calc_04\n type: number\n - name: ps_calc_05\n type: number\n - name: ps_calc_06\n type: number\n - name: ps_calc_07\n type: number\n - name: ps_calc_08\n type: number\n - name: ps_calc_09\n type: number\n - name: ps_calc_10\n type: number\n - name: ps_calc_11\n type: number\n - name: ps_calc_12\n type: number\n - name: ps_calc_13\n type: number\n - name: ps_calc_14\n type: number\n - name: ps_calc_15_bin\n type: binary\n - name: ps_calc_16_bin\n type: binary\n - name: ps_calc_17_bin\n type: binary\n - name: ps_calc_18_bin\n type: binary\n - name: ps_calc_19_bin\n type: binary\n - name: ps_calc_20_bin\n type: binary\ncombiner:\n type: concat\n num_fc_layers: 3\n fc_size: 128\n dropout: 0.1\ntraining:\n batch_size: 256\n learning_rate: .001\n epochs: 1\n"
},
{
"path": "ludwig/datasets/model_configs/synthetic_fraud_default.yaml",
"content": "output_features:\n - name: isFraud\n type: binary\ninput_features:\n - name: step\n type: number\n - name: type\n type: category\n - name: amount\n type: number\n - name: oldbalanceOrg\n type: number\n - name: newbalanceOrig\n type: number\n - name: oldbalanceDest\n type: number\n - name: newbalanceDest\n type: number\ncombiner:\n type: concat\n num_fc_layers: 3\n fc_size: 128\n dropout: 0.1\ntraining:\n batch_size: 256\n learning_rate: .001\n epochs: 1\n"
},
{
"path": "ludwig/datasets/model_configs/titanic_default.yaml",
"content": "output_features:\n - name: Survived\n type: binary\ninput_features:\n - name: Pclass\n type: category\n - name: Sex\n type: category\n - name: Age\n type: number\n preprocessing:\n missing_value_strategy: fill_with_mean\n - name: SibSp\n type: number\n - name: Parch\n type: number\n - name: Fare\n type: number\n preprocessing:\n missing_value_strategy: fill_with_mean\n - name: Embarked\n type: category\ntraining:\n batch_size: 256\n epochs: 1\n"
},
{
"path": "ludwig/datasets/utils.py",
"content": "import os\nfrom functools import lru_cache\n\nimport yaml\n\nfrom ludwig.api_annotations import PublicAPI\nfrom ludwig.datasets import model_configs\n\n\n@PublicAPI\ndef model_configs_for_dataset(dataset_name: str) -> dict[str, dict]:\n \"\"\"Returns a dictionary of built-in model configs for the specified dataset.\n\n Maps config name to ludwig config dict.\n \"\"\"\n return _get_model_configs(dataset_name)\n\n\n@lru_cache(maxsize=3)\ndef _get_model_configs(dataset_name: str) -> dict[str, dict]:\n \"\"\"Returns all model configs for the specified dataset.\n\n Model configs are named _.yaml\n \"\"\"\n import importlib.resources\n\n config_filenames = [\n f.name\n for f in importlib.resources.files(model_configs).iterdir()\n if f.name.endswith(\".yaml\") and f.name.startswith(dataset_name)\n ]\n configs = {}\n for config_filename in config_filenames:\n basename = os.path.splitext(config_filename)[0]\n config_name = basename[len(dataset_name) + 1 :]\n configs[config_name] = _load_model_config(config_filename)\n return configs\n\n\ndef _load_model_config(model_config_filename: str):\n \"\"\"Loads a model config.\"\"\"\n model_config_path = os.path.join(os.path.dirname(model_configs.__file__), model_config_filename)\n with open(model_config_path) as f:\n return yaml.safe_load(f)\n"
},
{
"path": "ludwig/decoders/__init__.py",
"content": "# register all decoders\nimport ludwig.decoders.generic_decoders # noqa\nimport ludwig.decoders.image_decoders # noqa\nimport ludwig.decoders.llm_decoders # noqa\nimport ludwig.decoders.sequence_decoders # noqa\nimport ludwig.decoders.sequence_tagger # noqa\n"
},
{
"path": "ludwig/decoders/base.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2020 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nfrom abc import ABC, abstractmethod\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.utils.torch_utils import LudwigModule\n\n\n@DeveloperAPI\nclass Decoder(LudwigModule, ABC):\n @abstractmethod\n def forward(self, inputs, mask=None):\n raise NotImplementedError\n\n @property\n def name(self):\n return self.__class__.__name__\n"
},
{
"path": "ludwig/decoders/generic_decoders.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nfrom functools import partial\n\nimport torch\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import BINARY, CATEGORY, CATEGORY_DISTRIBUTION, LOSS, NUMBER, SET, TIMESERIES, TYPE, VECTOR\nfrom ludwig.decoders.base import Decoder\nfrom ludwig.decoders.registry import register_decoder\nfrom ludwig.schema.decoders.base import ClassifierConfig, PassthroughDecoderConfig, ProjectorConfig, RegressorConfig\nfrom ludwig.utils.torch_utils import Dense, get_activation\n\nlogger = logging.getLogger(__name__)\n\n\n@DeveloperAPI\n# TODO(Arnav): Re-enable once we add DotProduct Combiner: https://github.com/ludwig-ai/ludwig/issues/3150\n# @register_decoder(\"passthrough\", [BINARY, CATEGORY, NUMBER, SET, VECTOR, SEQUENCE, TEXT])\nclass PassthroughDecoder(Decoder):\n def __init__(self, input_size: int = 1, num_classes: int = None, decoder_config=None, **kwargs):\n super().__init__()\n self.config = decoder_config\n\n logger.debug(f\" {self.name}\")\n self.input_size = input_size\n self.num_classes = num_classes\n\n def forward(self, inputs, **kwargs):\n return inputs\n\n @staticmethod\n def get_schema_cls():\n return PassthroughDecoderConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.input_size])\n\n @property\n def output_shape(self) -> torch.Size:\n return self.input_shape\n\n\n@DeveloperAPI\n@register_decoder(\"regressor\", [BINARY, NUMBER])\nclass Regressor(Decoder):\n def __init__(\n self,\n input_size,\n use_bias=True,\n weights_initializer=\"xavier_uniform\",\n bias_initializer=\"zeros\",\n decoder_config=None,\n **kwargs,\n ):\n super().__init__()\n self.config = decoder_config\n\n logger.debug(f\" {self.name}\")\n\n logger.debug(\" Dense\")\n\n self.dense = Dense(\n input_size=input_size,\n output_size=1,\n use_bias=use_bias,\n weights_initializer=weights_initializer,\n bias_initializer=bias_initializer,\n )\n\n @staticmethod\n def get_schema_cls():\n return RegressorConfig\n\n @property\n def input_shape(self):\n return self.dense.input_shape\n\n def forward(self, inputs, **kwargs):\n return self.dense(inputs)\n\n\n@DeveloperAPI\n@register_decoder(\"projector\", [VECTOR, TIMESERIES])\nclass Projector(Decoder):\n def __init__(\n self,\n input_size,\n output_size,\n use_bias=True,\n weights_initializer=\"xavier_uniform\",\n bias_initializer=\"zeros\",\n activation=None,\n multiplier=1.0,\n clip=None,\n decoder_config=None,\n **kwargs,\n ):\n super().__init__()\n self.config = decoder_config\n\n logger.debug(f\" {self.name}\")\n\n logger.debug(\" Dense\")\n self.dense = Dense(\n input_size=input_size,\n output_size=output_size,\n use_bias=use_bias,\n weights_initializer=weights_initializer,\n bias_initializer=bias_initializer,\n )\n\n self.activation = get_activation(activation)\n self.multiplier = multiplier\n\n if clip is not None:\n if isinstance(clip, (list, tuple)) and len(clip) == 2:\n self.clip = partial(torch.clip, min=clip[0], max=clip[1])\n else:\n raise ValueError(\n \"The clip parameter of {} is {}. \"\n \"It must be a list or a tuple of length 2.\".format(self.feature_name, self.clip)\n )\n else:\n self.clip = None\n\n @staticmethod\n def get_schema_cls():\n return ProjectorConfig\n\n @property\n def input_shape(self):\n return self.dense.input_shape\n\n def forward(self, inputs, **kwargs):\n values = self.activation(self.dense(inputs)) * self.multiplier\n if self.clip:\n values = self.clip(values)\n return values\n\n\n@DeveloperAPI\n@register_decoder(\"classifier\", [CATEGORY, CATEGORY_DISTRIBUTION, SET])\nclass Classifier(Decoder):\n def __init__(\n self,\n input_size,\n num_classes,\n use_bias=True,\n weights_initializer=\"xavier_uniform\",\n bias_initializer=\"zeros\",\n decoder_config=None,\n **kwargs,\n ):\n super().__init__()\n self.config = decoder_config\n\n logger.debug(f\" {self.name}\")\n\n logger.debug(\" Dense\")\n self.num_classes = num_classes\n self.dense = Dense(\n input_size=input_size,\n output_size=num_classes,\n use_bias=use_bias,\n weights_initializer=weights_initializer,\n bias_initializer=bias_initializer,\n )\n\n self.sampled_loss = False\n if LOSS in kwargs and TYPE in kwargs[LOSS] and kwargs[LOSS][TYPE] is not None:\n self.sampled_loss = kwargs[LOSS][TYPE].startswith(\"sampled\")\n\n @staticmethod\n def get_schema_cls():\n return ClassifierConfig\n\n @property\n def input_shape(self):\n return self.dense.input_shape\n\n def forward(self, inputs, **kwargs):\n return self.dense(inputs)\n"
},
{
"path": "ludwig/decoders/image_decoders.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Aizen Corp.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\n\nimport torch\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import ENCODER_OUTPUT_STATE, HIDDEN, IMAGE, LOGITS, PREDICTIONS\nfrom ludwig.decoders.base import Decoder\nfrom ludwig.decoders.registry import register_decoder\nfrom ludwig.modules.convolutional_modules import UNetUpStack\nfrom ludwig.schema.decoders.image_decoders import ImageDecoderConfig, UNetDecoderConfig\n\nlogger = logging.getLogger(__name__)\n\n\n@DeveloperAPI\n@register_decoder(\"unet\", IMAGE)\nclass UNetDecoder(Decoder):\n def __init__(\n self,\n input_size: int,\n height: int,\n width: int,\n num_channels: int = 1,\n num_classes: int = 2,\n conv_norm: str | None = None,\n decoder_config=None,\n **kwargs,\n ):\n super().__init__()\n self.config = decoder_config\n self.num_classes = num_classes\n\n logger.debug(f\" {self.name}\")\n if num_classes < 2:\n raise ValueError(f\"Invalid `num_classes` {num_classes} for unet decoder\")\n if height % 16 or width % 16:\n raise ValueError(f\"Invalid `height` {height} or `width` {width} for unet decoder\")\n\n self.unet = UNetUpStack(\n img_height=height,\n img_width=width,\n out_channels=num_classes,\n norm=conv_norm,\n )\n\n self.input_reshape = list(self.unet.input_shape)\n self.input_reshape.insert(0, -1)\n self._output_shape = (height, width)\n\n def forward(self, combiner_outputs: dict[str, torch.Tensor], target: torch.Tensor):\n hidden = combiner_outputs[HIDDEN]\n skips = combiner_outputs[ENCODER_OUTPUT_STATE]\n\n # unflatten combiner outputs\n hidden = hidden.reshape(self.input_reshape)\n\n logits = self.unet(hidden, skips)\n predictions = logits.argmax(dim=1).squeeze(1).byte()\n\n return {LOGITS: logits, PREDICTIONS: predictions}\n\n def get_prediction_set(self):\n return {LOGITS, PREDICTIONS}\n\n @staticmethod\n def get_schema_cls() -> type[ImageDecoderConfig]:\n return UNetDecoderConfig\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size(self._output_shape)\n\n @property\n def input_shape(self) -> torch.Size:\n return self.unet.input_shape\n"
},
{
"path": "ludwig/decoders/llm_decoders.py",
"content": "import logging\nimport re\nfrom typing import Any\n\nimport torch\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import CATEGORY, LOGITS, PREDICTIONS, PROBABILITIES, TEXT\nfrom ludwig.decoders.base import Decoder\nfrom ludwig.decoders.registry import register_decoder\nfrom ludwig.decoders.utils import extract_generated_tokens\nfrom ludwig.schema.decoders.llm_decoders import CategoryExtractorDecoderConfig, TextExtractorDecoderConfig\nfrom ludwig.utils.strings_utils import get_tokenizer\n\nlogger = logging.getLogger(__name__)\n\n\n# TODO(Arnav): Refactor to split into strategies like splitters\nclass Matcher:\n def __init__(self, match: dict[str, dict[str, Any]]):\n self.match = match\n\n def contains(self, decoded_input: str, value: str) -> bool:\n return value in decoded_input\n\n def regex(self, decoded_input: str, regex_pattern: str) -> bool:\n \"\"\"Perform a regex match on a given text using a specified regex pattern.\n\n Parameters:\n text (str): The text to perform the match on.\n regex_pattern (str): The regex pattern to use for the match.\n\n Returns:\n A list of match objects.\n \"\"\"\n # Compile the regex pattern\n matches = []\n try:\n regex = re.compile(regex_pattern)\n # Perform the match\n matches = regex.findall(decoded_input)\n except Exception:\n logger.warning(f\"Regex pattern {regex_pattern} could not be compiled.\")\n # If there is a match, matches is a non-empty list, so we can use this\n # to infer if there was a match or not and return a bool\n return len(matches) > 0\n\n def __call__(self, decoded_input: str) -> str | None:\n # Greedy match on first label that matches the input\n for label, label_def in self.match.items():\n label_def_type = label_def[\"type\"]\n label_def_value = label_def[\"value\"]\n\n if label_def_type == \"contains\":\n is_match = self.contains(decoded_input, label_def_value)\n elif label_def_type == \"regex\":\n is_match = self.regex(decoded_input, label_def_value)\n else:\n raise ValueError(\n f\"{label_def_type} is not a valid match `type`. Ludwig \"\n \"currently supports `contains` and `regex` match types.\"\n )\n\n if is_match:\n return label\n return None\n\n\n@DeveloperAPI\n@register_decoder(\"text_extractor\", [TEXT])\nclass TextExtractorDecoder(Decoder):\n def __init__(\n self,\n input_size: int,\n decoder_config=None,\n **kwargs,\n ):\n super().__init__()\n self.config = decoder_config\n self.input_size = input_size\n\n # Tokenizer\n self.tokenizer_type = self.config.tokenizer\n self.pretrained_model_name_or_path = self.config.pretrained_model_name_or_path\n self.vocab_file = self.config.vocab_file\n\n # Load tokenizer required for decoding the output from the generate\n # function of the text input feature for LLMs.\n self.tokenizer = get_tokenizer(self.tokenizer_type, self.vocab_file, self.pretrained_model_name_or_path)\n if hasattr(self.tokenizer, \"tokenizer\"):\n # Transformer Tokenizers\n self.tokenizer_vocab_size = self.tokenizer.tokenizer.vocab_size\n else:\n # TorchText Tokenizers\n self.tokenizer_vocab_size = len(self.tokenizer.vocab)\n\n # Maximum number of new tokens that will be generated\n # TODO(geoffrey): figure out where self.max_sequence_length is used– if not used, we might consider removing it.\n # It's confusing to have both this and `max_new_tokens` as a mandatory param in the `forward` function.\n self.max_sequence_length = self.config.max_new_tokens\n\n @staticmethod\n def get_schema_cls():\n return TextExtractorDecoderConfig\n\n @property\n def input_shape(self):\n return self.input_size\n\n def get_prediction_set(self):\n return {LOGITS, PREDICTIONS, PROBABILITIES}\n\n def forward(self, inputs: list[torch.Tensor], input_lengths: list[int], max_new_tokens: int):\n # Extract the sequences tensor from the LLMs forward pass\n generated_outputs = extract_generated_tokens(\n raw_generated_output_sequences=inputs,\n input_lengths=input_lengths,\n max_new_tokens=max_new_tokens,\n pad_sequence=True,\n )\n # Stack the predictions for each example in the batch. The padding should ensure they are all the same shape.\n for output in generated_outputs:\n if output.shape[0] > max_new_tokens:\n raise ValueError(\n f\"Output {output} is longer than the max_new_tokens {max_new_tokens} during decoding. \"\n f\"This should never happen– please file an issue on GitHub.\"\n )\n\n generated_outputs = torch.stack(generated_outputs, dim=0)\n outputs_device = generated_outputs.device\n\n return {\n PREDICTIONS: generated_outputs,\n # TODO(Arnav): Add support for probabilities and logits\n PROBABILITIES: torch.zeros((len(generated_outputs), max_new_tokens, self.tokenizer_vocab_size)).to(\n outputs_device\n ),\n LOGITS: torch.zeros((len(generated_outputs), max_new_tokens, self.tokenizer_vocab_size)).to(outputs_device),\n }\n\n\n@DeveloperAPI\n@register_decoder(\"category_extractor\", [CATEGORY])\nclass CategoryExtractorDecoder(Decoder):\n def __init__(\n self,\n decoder_config=None,\n **kwargs,\n ):\n super().__init__()\n self.config = decoder_config\n\n self.input_size = self.config.input_size\n self.fallback_label = self.config.fallback_label\n self.str2idx = self.config.str2idx\n self.vocab_size = len(self.config.str2idx)\n\n # Create Matcher object to perform matching on the decoded output\n self.matcher = Matcher(self.config.match)\n\n # Tokenizer\n self.tokenizer_type = self.config.tokenizer\n self.pretrained_model_name_or_path = self.config.pretrained_model_name_or_path\n self.vocab_file = self.config.vocab_file\n\n # Load tokenizer required for decoding the output from the generate\n # function of the text input feature for LLMs.\n self.tokenizer = get_tokenizer(self.tokenizer_type, self.vocab_file, self.pretrained_model_name_or_path)\n\n @staticmethod\n def get_schema_cls():\n return CategoryExtractorDecoderConfig\n\n @property\n def input_shape(self):\n return self.input_size\n\n def get_prediction_set(self):\n return {LOGITS, PREDICTIONS, PROBABILITIES}\n\n def forward(self, inputs: list[torch.Tensor], input_lengths: list[int], max_new_tokens: int):\n # Extract the sequences tensor from the LLMs forward pass\n generated_outputs = extract_generated_tokens(\n raw_generated_output_sequences=inputs,\n input_lengths=input_lengths,\n max_new_tokens=max_new_tokens,\n pad_sequence=False,\n )\n outputs_device = generated_outputs[0].device\n\n # Decode generated outputs from the LLM's generate function.\n decoded_outputs = self.tokenizer.tokenizer.batch_decode(generated_outputs, skip_special_tokens=True)\n\n # Parse labels based on matching criteria and return probability vectors\n matched_labels = []\n probabilities = []\n logits = []\n for output in decoded_outputs:\n output = output.lower() # Convert to lowercase for matching\n\n matched_label = self.matcher(output)\n idx = self.str2idx[matched_label] if matched_label in self.str2idx else self.str2idx[self.fallback_label]\n\n # Append the index of the matched label\n matched_labels.append(idx)\n\n # Append the probability vector for the matched label\n probability_vec = [0] * self.vocab_size\n probability_vec[idx] = 1\n probabilities.append(probability_vec)\n\n # TODO(Arnav): Figure out how to compute logits. For now, we return\n # a tensor of zeros.\n logits.append([0] * self.vocab_size)\n\n return {\n PREDICTIONS: torch.tensor(matched_labels, device=outputs_device),\n PROBABILITIES: torch.tensor(probabilities, dtype=torch.float32, device=outputs_device),\n LOGITS: torch.tensor(logits, dtype=torch.float32, device=outputs_device),\n }\n"
},
{
"path": "ludwig/decoders/registry.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.decoders.base import Decoder\nfrom ludwig.utils.registry import Registry\n\n_decoder_registry = Registry()\n\n\n@DeveloperAPI\ndef get_decoder_registry() -> Registry:\n return _decoder_registry\n\n\n@DeveloperAPI\ndef register_decoder(name: str, features: str | list[str]):\n if isinstance(features, str):\n features = [features]\n\n def wrap(cls):\n for feature in features:\n feature_registry = get_decoder_registry().get(feature, {})\n feature_registry[name] = cls\n get_decoder_registry()[feature] = feature_registry\n return cls\n\n return wrap\n\n\n@DeveloperAPI\ndef get_decoder_cls(feature: str, name: str) -> type[Decoder]:\n return get_decoder_registry()[feature][name]\n\n\n@DeveloperAPI\ndef get_decoder_classes(feature: str) -> dict[str, type[Decoder]]:\n return get_decoder_registry()[feature]\n"
},
{
"path": "ludwig/decoders/sequence_decoder_utils.py",
"content": "\"\"\"Utility functions related to sequence decoders.\"\"\"\n\nimport torch\n\nfrom ludwig.constants import ENCODER_OUTPUT_STATE, HIDDEN\nfrom ludwig.modules.reduction_modules import SequenceReducer\n\n\ndef repeat_2D_tensor(tensor, k):\n \"\"\"Repeats a 2D-tensor k times over the first dimension.\n\n For example:\n Input: Tensor of [batch_size, state_size], k=2\n Output: Tensor of [k, batch_size, state_size]\n \"\"\"\n if len(tensor.size()) > 2:\n raise ValueError(\"Cannot repeat a non-2D tensor with this method.\")\n return tensor.repeat(k, 1, 1)\n\n\ndef get_rnn_init_state(\n combiner_outputs: dict[str, torch.Tensor], sequence_reducer: SequenceReducer, num_layers: int\n) -> torch.Tensor:\n \"\"\"Computes the hidden state that the RNN decoder should start with.\n\n Args:\n combiner_outputs: Dictionary of tensors from the outputs of the combiner and other output features.\n sequence_reducer: SequenceReducer to reduce rank-3 to rank-2.\n num_layers: Number of layers the decoder uses.\n\n Returns:\n Tensor of [num_layers, batch_size, hidden_size].\n \"\"\"\n if ENCODER_OUTPUT_STATE not in combiner_outputs:\n # Use the combiner's hidden state.\n encoder_output_state = combiner_outputs[HIDDEN]\n else:\n # Use the encoder's output state.\n encoder_output_state = combiner_outputs[ENCODER_OUTPUT_STATE]\n if isinstance(encoder_output_state, tuple):\n if len(encoder_output_state) == 2:\n # LSTM encoder. Use the hidden state and ignore the cell state.\n encoder_output_state = encoder_output_state[0]\n elif len(encoder_output_state) == 4:\n # Bi-directional LSTM encoder. Use the average of hidden states and ignore cell state.\n encoder_output_state = torch.mean([encoder_output_state[0], encoder_output_state[2]])\n else:\n raise ValueError(\n f\"Invalid sequence decoder inputs with keys: {combiner_outputs.keys()} with extracted encoder \"\n + f\"state: {encoder_output_state.size()} that was invalid. Please double check the compatibility \"\n + \"of your encoder and decoder.\"\n )\n\n if len(encoder_output_state.size()) > 3:\n raise ValueError(\"Init state for RNN decoders only works for 1d or 2d tensors (encoder_output).\")\n\n if len(encoder_output_state.size()) == 3:\n # Reduce to [batch_size, hidden_size].\n encoder_output_state = sequence_reducer(encoder_output_state)\n\n return repeat_2D_tensor(encoder_output_state, num_layers)\n\n\ndef get_lstm_init_state(\n combiner_outputs: dict[str, torch.Tensor], sequence_reducer: SequenceReducer, num_layers: int\n) -> tuple[torch.Tensor, torch.Tensor]:\n \"\"\"Returns the states that the LSTM decoder should start with.\n\n Args:\n combiner_outputs: Dictionary of tensors from the outputs of the combiner and other output features.\n sequence_reducer: SequenceReducer to reduce rank-3 to rank-2.\n num_layers: Number of layers the decoder uses.\n\n Returns:\n Tuple of 2 tensors (decoder hidden state, decoder cell state), each [num_layers, batch_size, hidden_size].\n \"\"\"\n if ENCODER_OUTPUT_STATE not in combiner_outputs:\n # Use the combiner's hidden state.\n decoder_hidden_state = combiner_outputs[HIDDEN]\n decoder_cell_state = torch.clone(decoder_hidden_state)\n else:\n # Use the encoder's output state.\n encoder_output_state = combiner_outputs[ENCODER_OUTPUT_STATE]\n if not isinstance(encoder_output_state, tuple):\n decoder_hidden_state = encoder_output_state\n decoder_cell_state = decoder_hidden_state\n else:\n if len(encoder_output_state) == 2:\n # The encoder was probably an LSTM.\n decoder_hidden_state, decoder_cell_state = encoder_output_state\n elif len(encoder_output_state) == 4:\n # The encoder was probably a bi-LSTM.\n # Use the average of the encoder's hidden states for hidden state.\n # Use the average of the encoder's cell states for cell state.\n decoder_hidden_state = torch.mean([encoder_output_state[0], encoder_output_state[2]])\n decoder_cell_state = torch.mean([encoder_output_state[1], encoder_output_state[3]])\n else:\n raise ValueError(\n f\"Invalid sequence decoder inputs with keys: {combiner_outputs.keys()} with extracted encoder \"\n + f\"state: {encoder_output_state} that was invalid. Please double check the compatibility of your \"\n + \"encoder and decoder.\"\n )\n\n # Check rank and reduce if necessary.\n if len(decoder_hidden_state.size()) > 3 or len(decoder_cell_state.size()) > 3:\n raise ValueError(\n f\"Invalid sequence decoder inputs with keys: {combiner_outputs.keys()} with extracted encoder \"\n + f\"state: {decoder_hidden_state.size()} that was invalid. Please double check the compatibility \"\n + \"of your encoder and decoder.\"\n )\n if len(decoder_hidden_state.size()) == 3:\n decoder_hidden_state = sequence_reducer(decoder_hidden_state)\n if len(decoder_cell_state.size()) == 3:\n decoder_cell_state = sequence_reducer(decoder_cell_state)\n\n # Repeat over the number of layers.\n return repeat_2D_tensor(decoder_hidden_state, num_layers), repeat_2D_tensor(decoder_cell_state, num_layers)\n"
},
{
"path": "ludwig/decoders/sequence_decoders.py",
"content": "# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\n\nimport torch\nimport torch.nn as nn\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import LOGITS, PREDICTIONS, PROBABILITIES, SEQUENCE, TEXT\nfrom ludwig.decoders.base import Decoder\nfrom ludwig.decoders.registry import register_decoder\nfrom ludwig.decoders.sequence_decoder_utils import get_lstm_init_state, get_rnn_init_state\nfrom ludwig.modules.reduction_modules import SequenceReducer\nfrom ludwig.schema.decoders.sequence_decoders import SequenceGeneratorDecoderConfig\nfrom ludwig.utils import strings_utils\n\nlogger = logging.getLogger(__name__)\n\n\n@DeveloperAPI\nclass RNNDecoder(nn.Module):\n \"\"\"GRU or RNN-based decoder.\"\"\"\n\n def __init__(self, hidden_size: int, vocab_size: int, cell_type: str, num_layers: int = 1):\n super().__init__()\n self.hidden_size = hidden_size\n self.vocab_size = vocab_size\n self.embedding = nn.Embedding(vocab_size, hidden_size)\n if cell_type == \"gru\":\n self.rnn = nn.GRU(hidden_size, hidden_size, num_layers=num_layers, batch_first=True)\n else:\n self.rnn = nn.RNN(hidden_size, hidden_size, num_layers=num_layers, batch_first=True)\n self.out = nn.Linear(hidden_size, vocab_size)\n\n # Have the embedding and projection share weights.\n # This is a trick used by the Transformer, and seems to attain better loss.\n # See section 3.4 of https://arxiv.org/pdf/1706.03762.pdf.\n self.out.weight = self.embedding.weight\n\n def forward(self, input: torch.Tensor, hidden: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:\n \"\"\"Runs a single decoding time step.\n\n Modeled off of https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html.\n\n Args:\n input: [batch_size] tensor with the previous step's predicted symbol.\n hidden: [batch_size, hidden_size] tensor with the previous step's hidden state.\n\n Returns:\n Tuple of two tensors:\n - output: [batch_size, 1, vocab_size] tensor with the logits.\n - hidden: [num_layers, batch_size, hidden_size] tensor with the hidden state for the next time step.\n \"\"\"\n # Unsqueeze predicted tokens.\n input = input.unsqueeze(1).to(torch.int)\n output = self.embedding(input)\n output, hidden = self.rnn(output, hidden)\n output_logits = self.out(output)\n return output_logits, hidden\n\n\n@DeveloperAPI\nclass LSTMDecoder(nn.Module):\n \"\"\"LSTM-based decoder.\"\"\"\n\n def __init__(self, hidden_size: int, vocab_size: int, num_layers: int = 1):\n super().__init__()\n self.hidden_size = hidden_size\n self.vocab_size = vocab_size\n self.embedding = nn.Embedding(vocab_size, hidden_size)\n self.lstm = nn.LSTM(hidden_size, hidden_size, batch_first=True, num_layers=num_layers)\n self.out = nn.Linear(hidden_size, vocab_size)\n\n # Have the embedding and projection share weights.\n # This is a trick used by the Transformer, and seems to attain better loss.\n # See section 3.4 of https://arxiv.org/pdf/1706.03762.pdf.\n self.out.weight = self.embedding.weight\n\n def forward(\n self, input: torch.Tensor, hidden_state: torch.Tensor, cell_state: torch.Tensor\n ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:\n \"\"\"Runs a single decoding time step.\n\n Modeled off of https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html.\n\n Args:\n input: [batch_size] tensor with the previous step's predicted symbol.\n hidden_state: [batch_size, hidden_size] tensor with the previous step's hidden state.\n cell_state: [batch_size, hidden_size] tensor with the previous step's cell state.\n\n Returns:\n Tuple of 3 tensors:\n - output: [batch_size, vocab_size] tensor with the logits.\n - hidden_state: [batch_size, hidden_size] tensor with the hidden state for the next time step.\n - cell_state: [batch_size, hidden_size] tensor with the cell state for the next time step.\n \"\"\"\n # Unsqueeze predicted tokens.\n input = input.unsqueeze(1).to(torch.int)\n output = self.embedding(input)\n output, (hidden_state, cell_state) = self.lstm(output, (hidden_state, cell_state))\n output_logits = self.out(output)\n return output_logits, hidden_state, cell_state\n\n\n@DeveloperAPI\nclass SequenceRNNDecoder(nn.Module):\n \"\"\"RNN-based decoder over multiple time steps.\"\"\"\n\n def __init__(\n self,\n hidden_size: int,\n vocab_size: int,\n max_sequence_length: int,\n cell_type: str,\n num_layers: int = 1,\n reduce_input=\"sum\",\n ):\n super().__init__()\n self.hidden_size = hidden_size\n self.vocab_size = vocab_size\n self.rnn_decoder = RNNDecoder(hidden_size, vocab_size, cell_type, num_layers=num_layers)\n self.max_sequence_length = max_sequence_length\n self.reduce_sequence = SequenceReducer(reduce_mode=reduce_input)\n self.num_layers = num_layers\n\n self.register_buffer(\"logits\", torch.zeros([max_sequence_length, vocab_size]))\n self.register_buffer(\"decoder_input\", torch.Tensor([strings_utils.SpecialSymbol.START.value]))\n\n def forward(self, combiner_outputs: dict[str, torch.Tensor], target: torch.Tensor):\n \"\"\"Runs max_sequence_length RNN decoding time steps.\n\n Args:\n combiner_outputs: Dictionary of tensors from the outputs of the combiner and other output features.\n target: Tensor [batch_size, max_sequence_length] with target symbols.\n\n Returns:\n Tensor of logits [batch_size, max_sequence_length, vocab_size].\n \"\"\"\n # Prepare the encoder output state.\n decoder_hidden = get_rnn_init_state(combiner_outputs, self.reduce_sequence, self.num_layers)\n\n batch_size = decoder_hidden.size()[1]\n\n # Tensor to store decoder output logits.\n logits = self.logits.unsqueeze(0).repeat(batch_size, 1, 1)\n\n # Initialize the decoder with start symbols.\n decoder_input = self.decoder_input.repeat(batch_size)\n\n # Unsqueeze to account for extra multilayer dimension.\n # decoder_hidden = encoder_output_state.unsqueeze(0)\n\n # Decode until max length.\n for di in range(self.max_sequence_length):\n decoder_output, decoder_hidden = self.rnn_decoder(decoder_input, decoder_hidden)\n\n # decoder_output: [batch_size, 1, vocab_size]\n # Squeeze out the multilayer dimension and save logits.\n logits[:, di, :] = decoder_output.squeeze(1)\n\n # Determine inputs for next time step.\n # Using teacher forcing causes the model to converge faster but when the trained network is exploited, it\n # may be unstable: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.378.4095&rep=rep1&type=pdf.\n # TODO: Use a configurable ratio for how often to use teacher forcing during training.\n if target is None:\n _, topi = decoder_output.topk(1)\n # Squeeze out multilayer and vocabulary dimensions.\n decoder_input = topi.squeeze(1).squeeze(1).detach() # detach from history as input\n else:\n # Teacher forcing.\n decoder_input = target[:, di]\n\n return logits\n\n\n@DeveloperAPI\nclass SequenceLSTMDecoder(nn.Module):\n \"\"\"LSTM-based decoder over multiple time steps.\"\"\"\n\n def __init__(\n self,\n hidden_size: int,\n vocab_size: int,\n max_sequence_length: int,\n reduce_input: str = \"sum\",\n num_layers: int = 1,\n ):\n super().__init__()\n self.hidden_size = hidden_size\n self.vocab_size = vocab_size\n self.lstm_decoder = LSTMDecoder(hidden_size, vocab_size, num_layers)\n self.max_sequence_length = max_sequence_length\n self.reduce_sequence = SequenceReducer(reduce_mode=reduce_input)\n self.num_layers = num_layers\n\n self.register_buffer(\"logits\", torch.zeros([max_sequence_length, vocab_size]))\n self.register_buffer(\"decoder_input\", torch.Tensor([strings_utils.SpecialSymbol.START.value]))\n\n def forward(self, combiner_outputs: dict[str, torch.Tensor], target: torch.Tensor) -> torch.Tensor:\n \"\"\"Runs max_sequence_length LSTM decoding time steps.\n\n Args:\n combiner_outputs: Dictionary of tensors from the outputs of the combiner and other output features.\n target: Tensor [batch_size, max_sequence_length] with target symbols.\n\n Returns:\n Tensor of logits [batch_size, max_sequence_length, vocab_size].\n \"\"\"\n # Prepare the decoder initial state.\n decoder_hidden, decoder_cell_state = get_lstm_init_state(\n combiner_outputs, self.reduce_sequence, self.num_layers\n )\n batch_size = decoder_hidden.size()[1]\n\n # Initialize the decoder with start symbols.\n decoder_input = self.decoder_input.repeat(batch_size)\n\n # Tensor to store decoder output logits.\n logits = self.logits.unsqueeze(0).repeat(batch_size, 1, 1)\n\n # Decode until max length.\n for di in range(self.max_sequence_length):\n decoder_output, decoder_hidden, decoder_cell_state = self.lstm_decoder(\n decoder_input, decoder_hidden, decoder_cell_state\n )\n\n # decoder_output: [batch_size, 1, vocab_size]\n # Squeeze out the multilayer dimension and save logits.\n logits[:, di, :] = decoder_output.squeeze(1)\n\n # Determine inputs for next time step.\n # Using teacher forcing causes the model to converge faster but when the trained network is exploited, it\n # may be unstable: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.378.4095&rep=rep1&type=pdf.\n # TODO: Use a configurable ratio for how often to use teacher forcing during training.\n if target is None:\n _, topi = decoder_output.topk(1)\n # Squeeze out multilayer and vocabulary dimensions.\n decoder_input = topi.squeeze(1).squeeze(1).detach() # detach from history as input\n else:\n # Teacher forcing.\n decoder_input = target[:, di]\n\n return logits\n\n\n@DeveloperAPI\n@register_decoder(\"generator\", [SEQUENCE, TEXT])\nclass SequenceGeneratorDecoder(Decoder):\n \"\"\"Dispatcher for different sequence generator decoders.\"\"\"\n\n def __init__(\n self,\n vocab_size: int,\n max_sequence_length: int,\n cell_type: str = \"gru\",\n input_size: int = 256,\n reduce_input: str = \"sum\",\n num_layers: int = 1,\n decoder_config=None,\n **kwargs,\n ):\n \"\"\"\n Args:\n vocab_size: Vocab size.\n max_sequence_length: Maximum sequence length.\n cell_type: Type of RNN cell to use. 'rnn', 'gru', or 'lstm'.\n input_size: Size of incoming combiner output.\n reduce_input: Mode with which to reduce incoming combiner output, if needed.\n num_layers: Number of layers for the RNN deecoders.\n \"\"\"\n super().__init__()\n self.config = decoder_config\n\n self.vocab_size = vocab_size\n self.input_size = input_size\n self.max_sequence_length = max_sequence_length\n if cell_type == \"lstm\":\n self.rnn_decoder = SequenceLSTMDecoder(\n hidden_size=input_size,\n vocab_size=vocab_size,\n max_sequence_length=max_sequence_length,\n reduce_input=reduce_input,\n num_layers=num_layers,\n )\n else:\n self.rnn_decoder = SequenceRNNDecoder(\n hidden_size=input_size,\n vocab_size=vocab_size,\n max_sequence_length=max_sequence_length,\n cell_type=cell_type,\n reduce_input=reduce_input,\n num_layers=num_layers,\n )\n\n def forward(\n self, combiner_outputs: dict[str, torch.Tensor], target: torch.Tensor = None\n ) -> dict[str, torch.Tensor]:\n \"\"\"Decodes combiner_outputs into a sequence.\n\n Args:\n combiner_outputs: Dictionary of tensors from the outputs of the combiner and other output features.\n target: Tensor [batch_size, max_sequence_length] with target symbols.\n\n Returns:\n Dictionary of tensors of logits [batch_size, max_sequence_length, vocab_size].\n \"\"\"\n logits = self.rnn_decoder(combiner_outputs, target)\n return {LOGITS: logits}\n\n def get_prediction_set(self):\n return {LOGITS, PREDICTIONS, PROBABILITIES}\n\n @staticmethod\n def get_schema_cls():\n return SequenceGeneratorDecoderConfig\n\n @property\n def input_shape(self):\n # Dummy implementation.\n return torch.Size([1])\n\n @property\n def output_shape(self):\n return torch.Size([self.max_sequence_length, self.vocab_size])\n"
},
{
"path": "ludwig/decoders/sequence_tagger.py",
"content": "import logging\n\nimport torch\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import HIDDEN, LOGITS, PREDICTIONS, PROBABILITIES, SEQUENCE, TEXT\nfrom ludwig.decoders.base import Decoder\nfrom ludwig.decoders.registry import register_decoder\nfrom ludwig.modules.attention_modules import MultiHeadSelfAttention\nfrom ludwig.schema.decoders.sequence_decoders import SequenceTaggerDecoderConfig\nfrom ludwig.utils.torch_utils import Dense\n\nlogger = logging.getLogger(__name__)\n\n\n@DeveloperAPI\n@register_decoder(\"tagger\", [SEQUENCE, TEXT])\nclass SequenceTaggerDecoder(Decoder):\n def __init__(\n self,\n input_size: int,\n vocab_size: int,\n max_sequence_length: int,\n use_attention: bool = False,\n use_bias: bool = True,\n attention_embedding_size: int = 256,\n attention_num_heads: int = 8,\n decoder_config=None,\n **kwargs,\n ):\n super().__init__()\n self.config = decoder_config\n\n self.vocab_size = vocab_size\n self.max_sequence_length = max_sequence_length\n self.input_size = input_size\n self.use_attention = use_attention\n if use_attention:\n logger.debug(\" MultiHeadSelfAttention\")\n self.self_attention = MultiHeadSelfAttention(\n input_size=input_size, hidden_size=attention_embedding_size, num_heads=attention_num_heads\n )\n # Adjust the input size to the final projection layer.\n input_size = self.self_attention.output_shape[0]\n self.projection_layer = Dense(input_size=input_size, output_size=vocab_size, use_bias=use_bias)\n\n def forward(self, inputs: dict[str, torch.Tensor], target: torch.Tensor = None) -> dict[str, torch.Tensor]:\n \"\"\"Decodes the inputs into a sequence.\n\n Args:\n inputs: Dictionary of tensors from the outputs of the combiner and other output features.\n target: Tensor [batch_size, max_sequence_length] with predictions.\n\n Returns:\n Dictionary of tensors with logits [batch_size, max_sequence_length, vocab_size].\n \"\"\"\n hidden = inputs[HIDDEN]\n if len(hidden.size()) != 3:\n raise ValueError(\n f\"Decoder inputs rank is {len(hidden.size())}, but should be 3: \"\n + \"[batch_size x max_sequence_length x hidden_size] in when using a tagger sequential decoder. \"\n + \"Consider setting reduce_output to None if a sequential encoder / combiner is used.\"\n )\n if list(hidden.shape[1:]) != [self.max_sequence_length, self.input_size]:\n raise ValueError(\n \"Sequence tagger decoder inputs (hidden) should be [batch_size, self.max_sequence_length, \"\n + f\"input_size], or [batch_size, {self.max_sequence_length}, {self.input_size}]. However, the \"\n + f\"inputs (hidden) was instead: {list(hidden.size())}. \"\n + \"The encoder is not length preserving. Please check its configuration.\"\n )\n\n if self.use_attention:\n hidden = self.self_attention(hidden)\n\n logits = self.projection_layer(hidden)\n return {LOGITS: logits}\n\n def get_prediction_set(self):\n return {LOGITS, PROBABILITIES, PREDICTIONS}\n\n @staticmethod\n def get_schema_cls():\n return SequenceTaggerDecoderConfig\n\n @property\n def input_shape(self):\n # Dummy implementation.\n return torch.Size([1])\n\n @property\n def output_shape(self):\n return torch.Size([self.max_sequence_length, self.vocab_size])\n"
},
{
"path": "ludwig/decoders/utils.py",
"content": "import torch\nfrom torch import Tensor\n\n\ndef extract_generated_tokens(\n raw_generated_output_sequences: list[Tensor],\n input_lengths: list[int],\n max_new_tokens: int,\n pad_sequence: bool,\n) -> list[Tensor]:\n \"\"\"Extracts the generated tokens from the raw output sequences of the language model.\n\n Args:\n raw_generated_output_sequences: The raw output sequences of the language model.\n Represented as a list to handle variable length sequences.\n input_lengths: The length of the inputs to the language model.\n max_new_tokens: The maximum number of new tokens that were generated. Used to\n pad the generated sequences to the max_new_tokens.\n pad_sequence: Whether to pad the generated sequences to the max_new_tokens.\n\n Returns:\n The generated tokens.\n \"\"\"\n if len(raw_generated_output_sequences) != len(input_lengths):\n raise ValueError(\n f\"The number of raw_generated_output_sequences ({len(raw_generated_output_sequences)}) \"\n f\"must be the same as the number of input_lengths ({len(input_lengths)}).\"\n )\n\n generated_outputs = []\n for idx, input_length in enumerate(input_lengths):\n # Remove the input sequence from the generated sequence\n generated_sequence = raw_generated_output_sequences[idx][input_length:]\n\n # Pad the sequence if it is shorter than the max_new_tokens for downstream metric computation\n if pad_sequence and generated_sequence.size()[0] < max_new_tokens:\n generated_sequence = torch.nn.functional.pad(\n generated_sequence, (0, max_new_tokens - generated_sequence.size()[0]), \"constant\", 0\n )\n generated_outputs.append(generated_sequence)\n return generated_outputs\n"
},
{
"path": "ludwig/distributed/__init__.py",
"content": "from typing import Any\n\nfrom ludwig.distributed.base import DistributedStrategy, LocalStrategy\n\n\ndef load_ddp():\n from ludwig.distributed.ddp import DDPStrategy\n\n return DDPStrategy\n\n\ndef load_fsdp():\n from ludwig.distributed.fsdp import FSDPStrategy\n\n return FSDPStrategy\n\n\ndef load_deepspeed():\n from ludwig.distributed.deepspeed import DeepSpeedStrategy\n\n return DeepSpeedStrategy\n\n\ndef load_local():\n return LocalStrategy\n\n\nSTRATEGIES = {\n \"ddp\": load_ddp,\n \"fsdp\": load_fsdp,\n \"deepspeed\": load_deepspeed,\n \"local\": load_local,\n}\n\n\n_current_strategy: DistributedStrategy = None\n\n\ndef init_dist_strategy(strategy: str | dict[str, Any], **kwargs) -> DistributedStrategy:\n global _current_strategy\n if isinstance(strategy, dict):\n dtype = strategy.pop(\"type\", None)\n obj = get_dist_strategy(dtype)(**strategy)\n else:\n obj = get_dist_strategy(strategy)(**kwargs)\n _current_strategy = obj\n return obj\n\n\ndef get_current_dist_strategy() -> DistributedStrategy:\n if _current_strategy is None:\n raise RuntimeError(\"Distributed strategy not initialized\")\n return _current_strategy\n\n\ndef get_dist_strategy(strategy: str | dict[str, Any]) -> type[DistributedStrategy]:\n name = strategy\n if isinstance(strategy, dict):\n name = strategy[\"type\"]\n return STRATEGIES[name]()\n\n\ndef get_default_strategy_name() -> str:\n return \"ddp\"\n"
},
{
"path": "ludwig/distributed/base.py",
"content": "from __future__ import annotations\n\nimport contextlib\nfrom abc import ABC, abstractmethod\nfrom collections.abc import Callable\nfrom typing import Any, TYPE_CHECKING\n\nimport torch\nfrom torch import nn\nfrom torch.optim import Optimizer\n\nfrom ludwig.modules.optimization_modules import create_optimizer\nfrom ludwig.utils.torch_utils import get_torch_device\n\nif TYPE_CHECKING:\n from ray.train.backend import BackendConfig\n from ray.train.data_parallel_trainer import DataParallelTrainer\n\n from ludwig.models.base import BaseModel\n from ludwig.modules.lr_scheduler import LRScheduler\n from ludwig.schema.trainer import ECDTrainerConfig\n from ludwig.utils.checkpoint_utils import Checkpoint\n\n\nclass DistributedStrategy(ABC):\n \"\"\"Interface that wraps a distributed training framework (DDP, FSDP, DeepSpeed).\n\n Distributed strategies modify the model and/or optimizer to coordinate gradient updates among multiple workers\n running in parallel. In most cases, these are using collective communication libraries pass messages between\n processes.\n \"\"\"\n\n @abstractmethod\n def prepare(\n self,\n model: nn.Module,\n trainer_config: ECDTrainerConfig,\n base_learning_rate: float,\n ) -> tuple[nn.Module, Optimizer]:\n \"\"\"Modifies the model to support distributed training and creates the optimizer.\n\n Args:\n model: The model to wrap for distributed training.\n trainer_config: The trainer configuration, which includes optimizer params.\n base_learning_rate: The base learning rate to init the optimizer, which may be scaled by the strategy.\n\n Returns:\n A tuple of the wrapped model and the optimizer.\n \"\"\"\n\n def prepare_for_inference(self, model: nn.Module) -> nn.Module:\n return model\n\n def to_device(self, model: BaseModel, device: torch.device | None = None) -> nn.Module:\n return model.to_device(device if device is not None else get_torch_device())\n\n def backward(self, loss: torch.Tensor, model: nn.Module):\n loss.backward()\n\n def step(self, optimizer: Optimizer, *args, **kwargs):\n optimizer.step(*args, **kwargs)\n\n def zero_grad(self, optimizer: Optimizer):\n optimizer.zero_grad()\n\n def set_batch_size(self, model: nn.Module, batch_size: int):\n pass\n\n @abstractmethod\n def size(self) -> int:\n pass\n\n @abstractmethod\n def rank(self) -> int:\n pass\n\n @abstractmethod\n def local_size(self) -> int:\n pass\n\n @abstractmethod\n def local_rank(self) -> int:\n pass\n\n def is_coordinator(self) -> bool:\n return self.rank() == 0\n\n @abstractmethod\n def barrier(self):\n pass\n\n @abstractmethod\n def allreduce(self, t: torch.Tensor) -> torch.Tensor:\n pass\n\n @abstractmethod\n def broadcast(self, t: torch.Tensor) -> torch.Tensor:\n pass\n\n @abstractmethod\n def sync_model(self, model: nn.Module):\n pass\n\n @abstractmethod\n def sync_optimizer(self, optimizer: Optimizer):\n pass\n\n @abstractmethod\n def broadcast_object(self, v: Any, name: str | None = None) -> Any:\n pass\n\n @abstractmethod\n def wait_optimizer_synced(self, optimizer: Optimizer):\n pass\n\n @abstractmethod\n @contextlib.contextmanager\n def prepare_model_update(self, model: nn.Module, should_step: bool):\n pass\n\n @abstractmethod\n @contextlib.contextmanager\n def prepare_optimizer_update(self, optimizer: Optimizer):\n pass\n\n @classmethod\n @abstractmethod\n def is_available(cls) -> bool:\n pass\n\n @classmethod\n @abstractmethod\n def gather_all_tensors_fn(cls) -> Callable | None:\n pass\n\n @classmethod\n @abstractmethod\n def get_ray_trainer_backend(cls, **kwargs) -> Any | None:\n pass\n\n @classmethod\n @abstractmethod\n def get_trainer_cls(cls, backend_config: BackendConfig) -> tuple[type[DataParallelTrainer], dict[str, Any]]:\n pass\n\n @abstractmethod\n def shutdown(self):\n pass\n\n def return_first(self, fn: Callable) -> Callable:\n \"\"\"Wraps function so results are only returned by the first (coordinator) rank.\n\n The purpose of this function is to reduce network overhead.\n \"\"\"\n\n def wrapped(*args, **kwargs):\n res = fn(*args, **kwargs)\n return res if self.rank() == 0 else None\n\n return wrapped\n\n def allow_gradient_accumulation(self) -> bool:\n return True\n\n def allow_mixed_precision(self) -> bool:\n return True\n\n def allow_clip_gradients(self) -> bool:\n return True\n\n def prepare_before_load(self) -> bool:\n \"\"\"True if we need to call `prepare` again before loading a checkpoint.\"\"\"\n return False\n\n @classmethod\n def is_model_parallel(cls) -> bool:\n return False\n\n def create_checkpoint_handle(\n self,\n dist_model: nn.Module,\n model: nn.Module,\n optimizer: Optimizer | None = None,\n scheduler: LRScheduler | None = None,\n ) -> Checkpoint:\n from ludwig.utils.checkpoint_utils import MultiNodeCheckpoint\n\n return MultiNodeCheckpoint(self, model, optimizer, scheduler)\n\n @classmethod\n def extract_model_for_serialization(cls, model: nn.Module) -> nn.Module | tuple[nn.Module, list[dict]]:\n return model\n\n @classmethod\n def replace_model_from_serialization(cls, state: nn.Module | tuple[nn.Module, list[dict]]) -> nn.Module:\n assert isinstance(state, nn.Module)\n return state\n\n\nclass LocalStrategy(DistributedStrategy):\n def prepare(\n self,\n model: nn.Module,\n trainer_config: ECDTrainerConfig,\n base_learning_rate: float,\n ) -> tuple[nn.Module, Optimizer]:\n return model, create_optimizer(model, trainer_config.optimizer, base_learning_rate)\n\n def size(self) -> int:\n return 1\n\n def rank(self) -> int:\n return 0\n\n def local_size(self) -> int:\n return 0\n\n def local_rank(self) -> int:\n return 0\n\n def barrier(self):\n pass\n\n def allreduce(self, t: torch.Tensor) -> torch.Tensor:\n return t\n\n def broadcast(self, t: torch.Tensor) -> torch.Tensor:\n return t\n\n def sync_model(self, model: nn.Module):\n pass\n\n def sync_optimizer(self, optimizer: Optimizer):\n pass\n\n def broadcast_object(self, v: Any, name: str | None = None) -> Any:\n return v\n\n def wait_optimizer_synced(self, optimizer: Optimizer):\n pass\n\n @contextlib.contextmanager\n def prepare_model_update(self, model: nn.Module, should_step: bool):\n yield\n\n @contextlib.contextmanager\n def prepare_optimizer_update(self, optimizer: Optimizer):\n yield\n\n @classmethod\n def is_available(cls) -> bool:\n # While this strategy is always an option, it is not \"distributed\" which is the meaning of availability\n # in this context.\n return False\n\n @classmethod\n def gather_all_tensors_fn(cls) -> Callable | None:\n return None\n\n @classmethod\n def get_ray_trainer_backend(cls, **kwargs) -> Any | None:\n return None\n\n @classmethod\n def get_trainer_cls(cls, backend_config: BackendConfig) -> tuple[type[DataParallelTrainer], dict[str, Any]]:\n raise ValueError(\"Cannot construct a trainer from a local strategy.\")\n\n def shutdown(self):\n pass\n"
},
{
"path": "ludwig/distributed/ddp.py",
"content": "import contextlib\nimport logging\nimport os\nimport socket\nfrom collections.abc import Callable\nfrom typing import Any, Optional, TYPE_CHECKING, Union\n\nimport torch\nimport torch.distributed as dist\nfrom ray.train.backend import BackendConfig\nfrom ray.train.data_parallel_trainer import DataParallelTrainer\nfrom ray.train.torch import TorchTrainer\nfrom torch import nn\nfrom torch.nn.parallel import DistributedDataParallel as DDP\nfrom torch.optim import Optimizer\nfrom torchmetrics.utilities.distributed import gather_all_tensors\n\nfrom ludwig.distributed.base import DistributedStrategy\nfrom ludwig.modules.optimization_modules import create_optimizer\nfrom ludwig.utils.torch_utils import get_torch_device\n\nif TYPE_CHECKING:\n from ludwig.models.base import BaseModel\n from ludwig.modules.lr_scheduler import LRScheduler\n from ludwig.schema.trainer import ECDTrainerConfig\n from ludwig.utils.checkpoint_utils import Checkpoint\n\n\nclass DDPStrategy(DistributedStrategy):\n def __init__(self):\n self._local_rank, self._local_size = local_rank_and_size()\n self._log_on_init()\n\n def _log_on_init(self):\n logging.info(\"Using DDP strategy\")\n\n def prepare(\n self,\n model: nn.Module,\n trainer_config: \"ECDTrainerConfig\",\n base_learning_rate: float,\n ) -> tuple[nn.Module, Optimizer]:\n return DDP(model), create_optimizer(model, trainer_config.optimizer, base_learning_rate)\n\n def size(self) -> int:\n return dist.get_world_size()\n\n def rank(self) -> int:\n return dist.get_rank()\n\n def local_size(self) -> int:\n return self._local_size\n\n def local_rank(self) -> int:\n return self._local_rank\n\n def barrier(self):\n return dist.barrier()\n\n def allreduce(self, t: torch.Tensor) -> torch.Tensor:\n dist.all_reduce(t)\n return t\n\n def broadcast(self, t: torch.Tensor) -> torch.Tensor:\n dist.broadcast(t)\n return t\n\n def sync_model(self, model: nn.Module):\n # TODO(travis): open question if this is needed to ensure all workers using same weights\n pass\n\n def sync_optimizer(self, optimizer: Optimizer):\n # TODO(travis): open question if this is needed to ensure all workers using same optimizer state\n pass\n\n def broadcast_object(self, v: Any, name: str | None = None) -> Any:\n output = [v]\n dist.broadcast_object_list(output)\n return output[0]\n\n def wait_optimizer_synced(self, optimizer: Optimizer):\n pass\n\n @contextlib.contextmanager\n def prepare_model_update(self, model: nn.Module, should_step: bool):\n if should_step:\n yield\n else:\n # Prevents DDP from syncing gradients during accumulation step\n with model.no_sync():\n yield\n\n @contextlib.contextmanager\n def prepare_optimizer_update(self, optimizer: Optimizer):\n yield\n\n @classmethod\n def is_available(cls) -> bool:\n return dist.is_available() and dist.is_initialized()\n\n @classmethod\n def gather_all_tensors_fn(cls) -> Callable | None:\n return gather_all_tensors\n\n @classmethod\n def get_ray_trainer_backend(cls, **kwargs) -> Any | None:\n from ray.train.torch import TorchConfig\n\n return TorchConfig()\n\n @classmethod\n def get_trainer_cls(cls, backend_config: BackendConfig) -> tuple[type[DataParallelTrainer], dict[str, Any]]:\n return TorchTrainer, dict(torch_config=backend_config)\n\n def shutdown(self):\n # TODO(travis): currently Ray handles this for us, but is subject to hangs if one of the workers raises an\n # exception and the other makes a collective op. We should figure out a way to make this safe to call\n # multiple times. It looks like there is a fix we can make use of when we upgrade to Ray 2.1:\n # https://discuss.ray.io/t/torchtrainer-hangs-when-only-1-worker-raises-error/7447/11\n # dist.destroy_process_group()\n pass\n\n def create_checkpoint_handle(\n self,\n dist_model: nn.Module,\n model: nn.Module,\n optimizer: Optimizer | None = None,\n scheduler: Optional[\"LRScheduler\"] = None,\n ) -> \"Checkpoint\":\n from ludwig.utils.checkpoint_utils import MultiNodeCheckpoint\n\n return MultiNodeCheckpoint(self, model, optimizer, scheduler)\n\n def to_device(self, model: Union[\"BaseModel\", DDP], device: torch.device | None = None) -> nn.Module:\n try:\n return model.to_device(device if device is not None else get_torch_device())\n except AttributeError:\n # Model is already wrapped in DistributedDataParallel, so it has already been moved to device\n return model\n\n\ndef local_rank_and_size() -> tuple[int, int]:\n # DeepSpeed CLI and other tools may set these environment variables for us.\n local_rank, local_size = os.environ.get(\"LOCAL_RANK\"), os.environ.get(\"LOCAL_SIZE\")\n if local_rank is not None and local_size is not None:\n return int(local_rank), int(local_size)\n\n # Gather the rank and hostnames from every worker so we can count up how many belong to the same host, which\n # constitutes the local group.\n rank = dist.get_rank()\n host = socket.gethostname()\n output = [None for _ in range(dist.get_world_size())]\n dist.all_gather_object(output, (rank, host))\n\n # Every time we find a worker with the same host, we increment the size counter.\n # The local rank is determined by the world rank relative to the other workers on the same host, so every time\n # we see a worker on our host with a lower rank, we increment the rank counter.\n local_size = 0\n local_rank = 0\n for other_rank, other_host in output:\n if other_host == host:\n local_size += 1\n if other_rank < rank:\n local_rank += 1\n\n return local_rank, local_size\n"
},
{
"path": "ludwig/distributed/deepspeed.py",
"content": "import logging\nimport os\nimport warnings\nfrom collections.abc import Mapping\nfrom typing import Any, Optional, TYPE_CHECKING\n\nimport deepspeed\nimport deepspeed.comm\nimport torch\nfrom deepspeed.utils.zero_to_fp32 import get_fp32_state_dict_from_zero_checkpoint\nfrom packaging import version\nfrom torch import nn\nfrom torch.optim.optimizer import Optimizer\n\nfrom ludwig.constants import MIN_POSSIBLE_BATCH_SIZE\nfrom ludwig.distributed.ddp import DDPStrategy\nfrom ludwig.modules.optimization_modules import get_optimizer_class_and_kwargs\nfrom ludwig.utils.checkpoint_utils import Checkpoint\nfrom ludwig.utils.model_utils import extract_tensors, replace_tensors\n\n_deepspeed_0101 = version.parse(deepspeed.__version__) >= version.parse(\"0.10.1\")\n\n\nif TYPE_CHECKING:\n from ludwig.modules.lr_scheduler import LRScheduler\n from ludwig.schema.trainer import ECDTrainerConfig\n\n\nDEFAULT_ZERO_OPTIMIZATION = {\n \"stage\": \"auto\",\n \"stage3_gather_16bit_weights_on_model_save\": \"auto\",\n \"offload_optimizer\": {\"device\": \"auto\"},\n \"offload_param\": {\"device\": \"auto\"},\n}\n\n# Filter out warnings about DeepSpeed use of deprecated methods. Can remove on upgrade to DeepSpeed 0.9.\nwarnings.filterwarnings(\n action=\"ignore\",\n category=UserWarning,\n module=\"torch.distributed.distributed_c10d\",\n)\n\n\nclass DeepSpeedStrategy(DDPStrategy):\n def __init__(\n self,\n zero_optimization: dict[str, Any] | None = None,\n fp16: dict[str, Any] | None = None,\n bf16: dict[str, Any] | None = None,\n compression_training: dict[str, Any] | None = None,\n **kwargs\n ):\n # If we're initializing from a `deepspeed` CLI command, deepspeed will have already been initialized, as\n # indicated by the presence of the LOCAL_RANK var. Otherwise, we're initializing from Ray / torchrun, and will\n # need to set this var ourselves, then init DeepSpeed here.\n local_rank, local_size = os.environ.get(\"LOCAL_RANK\"), os.environ.get(\"LOCAL_SIZE\")\n init_deepspeed = local_rank is None or local_size is None\n\n super().__init__(**kwargs)\n self.zero_optimization = zero_optimization or DEFAULT_ZERO_OPTIMIZATION\n self.fp16 = fp16\n self.bf16 = bf16\n self.compression_training = compression_training\n\n if init_deepspeed:\n os.environ[\"LOCAL_RANK\"] = str(self.local_rank())\n os.environ[\"LOCAL_SIZE\"] = str(self.local_size())\n os.environ[\"RANK\"] = str(self.rank())\n os.environ[\"WORLD_SIZE\"] = str(self.size())\n deepspeed.init_distributed()\n\n def _log_on_init(self):\n logging.info(\"Using DeepSpeed strategy\")\n\n def prepare(\n self,\n model: nn.Module,\n trainer_config: \"ECDTrainerConfig\",\n base_learning_rate: float,\n ) -> tuple[nn.Module, Optimizer]:\n # If `batch_size=auto`, we set to MIN_POSSIBLE_BATCH_SIZE temporarily until auto-tuning adjusts it`\n # We can really set it to be whatever we want, as it will be overridden by the auto-tuning.\n batch_size = (\n trainer_config.batch_size if isinstance(trainer_config.batch_size, int) else MIN_POSSIBLE_BATCH_SIZE\n )\n # Paged and 8-bit optimizers are not supported by Deepspeed - just whatever is supported\n # by torch.optim.Optimizer. https://www.deepspeed.ai/docs/config-json/#optimizer-parameters.\n if trainer_config.optimizer.is_paged or trainer_config.optimizer.is_8bit:\n raise ValueError(\"Cannot use a paged or 8-bit optimizer with DeepSpeed.\")\n optimizer_cls, optimizer_kwargs = get_optimizer_class_and_kwargs(trainer_config.optimizer, base_learning_rate)\n ds_config = {\n \"amp\": {\n \"enabled\": trainer_config.use_mixed_precision,\n },\n \"optimizer\": {\"type\": optimizer_cls.__name__, \"params\": optimizer_kwargs},\n \"zero_optimization\": self.zero_optimization,\n \"gradient_clipping\": trainer_config.gradient_clipping.clipglobalnorm,\n \"train_micro_batch_size_per_gpu\": batch_size,\n \"gradient_accumulation_steps\": trainer_config.gradient_accumulation_steps,\n \"steps_per_print\": trainer_config.steps_per_checkpoint or 10000,\n }\n\n # DeepSpeed doesn't like passing these params as None values\n if self.fp16 is not None:\n ds_config[\"fp16\"] = self.fp16\n if self.bf16 is not None:\n ds_config[\"bf16\"] = self.bf16\n if self.compression_training is not None:\n ds_config[\"compression_training\"] = self.compression_training\n\n model_engine, optimizer, _, _ = deepspeed.initialize(\n model=model,\n model_parameters=model.parameters(),\n lr_scheduler=None, # Don't let DeepSpeed manage the learning rate scheduler\n config=ds_config,\n dist_init_required=False,\n )\n\n if hasattr(optimizer, \"optimizer\"):\n # Zero-3 wraps the optimizer\n optimizer = optimizer.optimizer\n\n return model_engine, optimizer\n\n def prepare_for_inference(self, model: nn.Module) -> nn.Module:\n ds_config = {}\n model_engine = deepspeed.init_inference(model=model, config=ds_config)\n return model_engine\n\n def to_device(self, model: nn.Module, device: torch.device | None = None) -> nn.Module:\n return model\n\n def backward(self, loss: torch.Tensor, model: nn.Module):\n # See: https://github.com/huggingface/accelerate/blob/main/src/accelerate/utils/deepspeed.py\n # runs backpropagation and handles mixed precision\n model.backward(loss)\n\n # Deepspeed's `engine.step` performs the following operations:\n # - gradient accumulation check\n # - gradient clipping\n # - optimizer step\n # - zero grad\n # - checking overflow\n # - lr_scheduler step (only if engine.lr_scheduler is not None)\n model.step()\n # and this plugin overrides the above calls with no-ops when Accelerate runs under\n # Deepspeed, but allows normal functionality for non-Deepspeed cases thus enabling a simple\n # training loop that works transparently under many training regimes.\n\n def step(self, optimizer: Optimizer, *args, **kwargs):\n # Handled by `self.backward(loss)`\n pass\n\n def zero_grad(self, optimizer: Optimizer):\n # Handled by `self.backward(loss)`\n pass\n\n def set_batch_size(self, model: nn.Module, batch_size: int):\n # Adapted from:\n # https://github.com/microsoft/DeepSpeed/blob/7ce371b139521b1ebbf052f0496b1a16397c1d19/deepspeed/runtime/engine.py#L422 # noqa: E501\n model._config.micro_batch_size_per_gpu = batch_size\n model._config.train_batch_size = batch_size * self.size() * model._config.gradient_accumulation_steps\n\n def barrier(self):\n deepspeed.comm.barrier()\n\n def allow_gradient_accumulation(self) -> bool:\n \"\"\"DeepSpeed handles gradient accumulation internally.\"\"\"\n return False\n\n def allow_mixed_precision(self) -> bool:\n \"\"\"DeepSpeed handles mixed precision internally.\"\"\"\n return False\n\n def allow_clip_gradients(self) -> bool:\n \"\"\"DeepSpeed handles gradient clipping internally.\"\"\"\n return False\n\n def prepare_before_load(self) -> bool:\n \"\"\"DeepSpeed requires the engine to be re-initialized before loading.\n\n https://deepspeed.readthedocs.io/en/latest/model-checkpointing.html#loading-training-checkpoints\n \"\"\"\n return True\n\n @classmethod\n def is_model_parallel(cls) -> bool:\n return True\n\n def create_checkpoint_handle(\n self,\n dist_model: nn.Module,\n model: nn.Module,\n optimizer: Optimizer | None = None,\n scheduler: Optional[\"LRScheduler\"] = None,\n ) -> Checkpoint:\n return DeepSpeedCheckpoint(self, dist_model, optimizer, scheduler)\n\n @classmethod\n def extract_model_for_serialization(cls, model: nn.Module) -> nn.Module | tuple[nn.Module, list[dict]]:\n return extract_tensors(model)\n\n @classmethod\n def replace_model_from_serialization(cls, state: nn.Module | tuple[nn.Module, list[dict]]) -> nn.Module:\n assert isinstance(state, tuple)\n model, model_weights = state\n replace_tensors(model, model_weights, torch.device(\"cpu\"))\n return model\n\n\nclass DeepSpeedCheckpoint(Checkpoint):\n def prepare(self, directory: str):\n if self.distributed.local_rank() == 0:\n # Checkpoints need to be written on every rank, but the directory only needs to be created once per node.\n super().prepare(directory)\n\n def load(self, save_path: str, device: torch.device | None = None) -> bool:\n \"\"\"Load a checkpoint.\n\n For DeepSpeed, we need every worker to independently load back the model weights, as the checkpoints themselves\n may be sharded (when using DeepSpeed Zero3).\n\n https://deepspeed.readthedocs.io/en/latest/model-checkpointing.html#loading-training-checkpoints\n \"\"\"\n # NOTE(geoffrey): `load_module_strict=False` because this code path is frequently used to load models trained\n # using adapter-based fine-tuning, where the checkpoints only contain the adapter weights, and not the full\n # model weights. This may lead to silent, unexpected behavior for resuming full model fine-tuning,\n # where all the model weights *must* be loaded in.\n # TODO(geoffrey): Add a boolean arg to function to control load_module_strict behavior.\n _, client_state = self.model.load_checkpoint(\n save_path, load_lr_scheduler_states=False, load_module_strict=False\n )\n self.global_step = self._get_global_step(client_state, save_path)\n if self.scheduler is not None and \"scheduler_state\" in client_state:\n self.scheduler.load_state_dict(client_state[\"scheduler_state\"])\n return True\n\n def save(self, save_path: str, global_step: int):\n client_state = {\n \"global_step\": global_step,\n }\n if self.scheduler is not None:\n client_state[\"scheduler_state\"] = self.scheduler.state_dict()\n\n kwargs = {}\n if _deepspeed_0101:\n kwargs[\"exclude_frozen_parameters\"] = True\n\n self.model.save_checkpoint(save_path, client_state=client_state, **kwargs)\n\n def get_state_for_inference(self, save_path: str, device: torch.device | None = None) -> Mapping[str, Any]:\n if self.model.zero_optimization_stage() == 3:\n return get_fp32_state_dict_from_zero_checkpoint(save_path)\n\n self.model.load_checkpoint(\n save_path, load_optimizer_states=False, load_lr_scheduler_states=False, load_module_only=True\n )\n return self.model.module.cpu().state_dict()\n"
},
{
"path": "ludwig/distributed/fsdp.py",
"content": "import logging\nfrom typing import TYPE_CHECKING\n\nimport torch\nfrom torch import nn\nfrom torch.distributed.fsdp import FullyShardedDataParallel as FSDP\nfrom torch.optim import Optimizer\n\nfrom ludwig.distributed.ddp import DDPStrategy\nfrom ludwig.modules.optimization_modules import create_optimizer\n\nif TYPE_CHECKING:\n from ludwig.schema.trainer import ECDTrainerConfig\n\n\nclass FSDPStrategy(DDPStrategy):\n def _log_on_init(self):\n logging.info(\"Using FSDP strategy\")\n\n def prepare(\n self,\n model: nn.Module,\n trainer_config: \"ECDTrainerConfig\",\n base_learning_rate: float,\n ) -> tuple[nn.Module, Optimizer]:\n return FSDP(model), create_optimizer(model, trainer_config.optimizer, base_learning_rate)\n\n def to_device(self, model: nn.Module, device: torch.device | None = None) -> nn.Module:\n return model\n\n @classmethod\n def is_model_parallel(cls) -> bool:\n return True\n"
},
{
"path": "ludwig/encoders/__init__.py",
"content": "# register all encoders\nimport ludwig.encoders.bag_encoders\nimport ludwig.encoders.category_encoders\nimport ludwig.encoders.date_encoders\nimport ludwig.encoders.generic_encoders\nimport ludwig.encoders.h3_encoders\nimport ludwig.encoders.image\nimport ludwig.encoders.sequence_encoders\nimport ludwig.encoders.set_encoders\nimport ludwig.encoders.text_encoders # noqa\n"
},
{
"path": "ludwig/encoders/bag_encoders.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nfrom typing import Any\n\nimport torch\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import BAG, ENCODER_OUTPUT\nfrom ludwig.encoders.base import Encoder\nfrom ludwig.encoders.registry import register_encoder\nfrom ludwig.encoders.types import EncoderOutputDict\nfrom ludwig.modules.embedding_modules import EmbedWeighted\nfrom ludwig.modules.fully_connected_modules import FCStack\nfrom ludwig.schema.encoders.bag_encoders import BagEmbedWeightedConfig\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\n\nlogger = logging.getLogger(__name__)\n\n\n@DeveloperAPI\n@register_encoder(\"embed\", BAG)\nclass BagEmbedWeightedEncoder(Encoder):\n def __init__(\n self,\n vocab: list[str],\n embedding_size: int = 50,\n representation: str = \"dense\",\n embeddings_trainable: bool = True,\n pretrained_embeddings: str | None = None,\n force_embedding_size: bool = False,\n embeddings_on_cpu: bool = False,\n fc_layers=None,\n num_fc_layers: int = 0,\n output_size: int = 10,\n use_bias: bool = True,\n weights_initializer: str = \"xavier_uniform\",\n bias_initializer: str = \"zeros\",\n norm: str | None = None,\n norm_params: dict[str, Any] | None = None,\n activation: str = \"relu\",\n dropout: float = 0.0,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n\n logger.debug(\" EmbedWeighted\")\n self.embed_weighted = EmbedWeighted(\n vocab,\n embedding_size,\n representation=representation,\n embeddings_trainable=embeddings_trainable,\n pretrained_embeddings=pretrained_embeddings,\n force_embedding_size=force_embedding_size,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=weights_initializer,\n )\n logger.debug(\" FCStack\")\n self.fc_stack = FCStack(\n self.embed_weighted.output_shape[-1],\n layers=fc_layers,\n num_layers=num_fc_layers,\n default_output_size=output_size,\n default_use_bias=use_bias,\n default_weights_initializer=weights_initializer,\n default_bias_initializer=bias_initializer,\n default_norm=norm,\n default_norm_params=norm_params,\n default_activation=activation,\n default_dropout=dropout,\n )\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return BagEmbedWeightedConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([len(self.vocab)])\n\n @property\n def output_shape(self) -> torch.Size:\n return self.fc_stack.output_shape\n\n def forward(self, inputs: torch.Tensor) -> EncoderOutputDict:\n \"\"\"\n :param inputs: The inputs fed into the encoder.\n Shape: [batch x vocab size], type torch.int32\n\n :param return: embeddings of shape [batch x embed size], type torch.float32\n \"\"\"\n hidden = self.embed_weighted(inputs)\n hidden = self.fc_stack(hidden)\n\n return {ENCODER_OUTPUT: hidden}\n"
},
{
"path": "ludwig/encoders/base.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2020 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nfrom abc import ABC, abstractmethod\n\nfrom torch import nn\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.utils.torch_utils import LudwigModule\n\n\n@DeveloperAPI\nclass Encoder(LudwigModule, ABC):\n @abstractmethod\n def forward(self, inputs, training=None, mask=None):\n raise NotImplementedError\n\n def get_embedding_layer(self) -> nn.Module:\n \"\"\"Returns layer that embeds inputs, used for computing explanations.\n\n Captum adds an evaluation hook to this module returned by this function. The hook copies the module's return\n with .clone(). The module returned by this function must return a tensor, not a dictionary of tensors.\n \"\"\"\n return next(self.children())\n\n @property\n def name(self) -> str:\n return self.__class__.__name__\n"
},
{
"path": "ludwig/encoders/category_encoders.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\n\nimport torch\nfrom torch import nn\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import CATEGORY, ENCODER_OUTPUT\nfrom ludwig.encoders.base import Encoder\nfrom ludwig.encoders.registry import register_encoder\nfrom ludwig.encoders.types import EncoderOutputDict\nfrom ludwig.modules.embedding_modules import Embed\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.category_encoders import (\n CategoricalEmbedConfig,\n CategoricalOneHotEncoderConfig,\n CategoricalPassthroughEncoderConfig,\n CategoricalSparseConfig,\n)\n\nlogger = logging.getLogger(__name__)\n\n\n@DeveloperAPI\n@register_encoder(\"passthrough\", [CATEGORY])\nclass CategoricalPassthroughEncoder(Encoder):\n def __init__(self, input_size=1, encoder_config=None, **kwargs):\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n self.input_size = input_size\n self.identity = nn.Identity()\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n \"\"\"\n :param inputs: The inputs fed into the encoder.\n Shape: [batch x 1]\n \"\"\"\n return {\"encoder_output\": self.identity(inputs.float())}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return CategoricalPassthroughEncoderConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.input_size])\n\n @property\n def output_shape(self) -> torch.Size:\n return self.input_shape\n\n def get_embedding_layer(self) -> nn.Module:\n return self.identity\n\n\n@DeveloperAPI\n@register_encoder(\"dense\", CATEGORY)\nclass CategoricalEmbedEncoder(Encoder):\n def __init__(\n self,\n vocab: list[str],\n embedding_size: int = 50,\n embeddings_trainable: bool = True,\n pretrained_embeddings: str | None = None,\n embeddings_on_cpu: bool = False,\n dropout: float = 0.0,\n embedding_initializer: str | dict | None = None,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n\n logger.debug(\" Embed\")\n self.embed = Embed(\n vocab=vocab,\n embedding_size=embedding_size,\n representation=\"dense\",\n embeddings_trainable=embeddings_trainable,\n pretrained_embeddings=pretrained_embeddings,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=embedding_initializer,\n )\n self.embedding_size = self.embed.embedding_size\n\n def forward(self, inputs: torch.Tensor) -> EncoderOutputDict:\n \"\"\"\n :param inputs: The inputs fed into the encoder.\n Shape: [batch x 1], type torch.int32\n\n :param return: embeddings of shape [batch x embed size], type torch.float32\n \"\"\"\n embedded = self.embed(inputs)\n return {ENCODER_OUTPUT: embedded}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return CategoricalEmbedConfig\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.embedding_size])\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([1])\n\n\n@DeveloperAPI\n@register_encoder(\"sparse\", CATEGORY)\nclass CategoricalSparseEncoder(Encoder):\n def __init__(\n self,\n vocab: list[str],\n embeddings_trainable: bool = False,\n pretrained_embeddings: str | None = None,\n embeddings_on_cpu: bool = False,\n dropout: float = 0.0,\n embedding_initializer: str | dict | None = None,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n\n logger.debug(\" Embed\")\n self.embed = Embed(\n vocab=vocab,\n embedding_size=len(vocab),\n representation=\"sparse\",\n embeddings_trainable=embeddings_trainable,\n pretrained_embeddings=pretrained_embeddings,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=embedding_initializer,\n )\n self.embedding_size = self.embed.embedding_size\n\n def forward(self, inputs: torch.Tensor) -> EncoderOutputDict:\n \"\"\"\n :param inputs: The inputs fed into the encoder.\n Shape: [batch x 1], type torch.int32\n\n :param return: embeddings of shape [batch x embed size], type torch.float32\n \"\"\"\n embedded = self.embed(inputs)\n return {ENCODER_OUTPUT: embedded}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return CategoricalSparseConfig\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.embedding_size])\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([1])\n\n\n@DeveloperAPI\n@register_encoder(\"onehot\", [CATEGORY])\nclass CategoricalOneHotEncoder(Encoder):\n def __init__(\n self,\n vocab: list[str],\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n self.vocab_size = len(vocab)\n self.identity = nn.Identity()\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n \"\"\"\n :param inputs: The inputs fed into the encoder.\n Shape: [batch, 1] or [batch]\n \"\"\"\n t = inputs.reshape(-1).long()\n # the output of this must be a float so that it can be concatenated with other\n # encoder outputs and passed to dense layers in the combiner, decoder, etc.\n outputs = self.identity(torch.nn.functional.one_hot(t, num_classes=self.vocab_size).float())\n return {\"encoder_output\": outputs}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return CategoricalOneHotEncoderConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([1])\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.vocab_size])\n\n def get_embedding_layer(self) -> nn.Module:\n return self.identity\n"
},
{
"path": "ludwig/encoders/date_encoders.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\n\nimport torch\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import DATE, ENCODER_OUTPUT\nfrom ludwig.encoders.base import Encoder\nfrom ludwig.encoders.registry import register_encoder\nfrom ludwig.encoders.types import EncoderOutputDict\nfrom ludwig.modules.embedding_modules import Embed\nfrom ludwig.modules.fully_connected_modules import FCStack\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.date_encoders import DateEmbedConfig, DateWaveConfig\nfrom ludwig.utils import torch_utils\n\nlogger = logging.getLogger(__name__)\n\n# Year, month, day, weekday, yearday, hour, minute, seconds, second_of_day.\n# TODO: Share this constant with date_feature.DATE_VECTOR_SIZE.\nDATE_INPUT_SIZE = 9\n\n\n@DeveloperAPI\n@register_encoder(\"embed\", DATE)\nclass DateEmbed(Encoder):\n def __init__(\n self,\n embedding_size: int = 10,\n embeddings_on_cpu: bool = False,\n fc_layers: list[dict] | None = None,\n num_fc_layers: int = 0,\n output_size: int = 10,\n use_bias: bool = True,\n weights_initializer: str = \"xavier_uniform\",\n bias_initializer: str = \"zeros\",\n norm: str | None = None,\n norm_params: dict | None = None,\n activation: str = \"relu\",\n dropout: float = 0,\n encoder_config=None,\n **kwargs,\n ):\n \"\"\"\n :param embedding_size: The maximum embedding size, the actual size\n will be `min(vocabulary_size, embedding_size)` for `dense`\n representations and exactly `vocabulary_size` for the `sparse`\n encoding, where `vocabulary_size` is the number of different\n strings appearing in the training set in the column the feature\n is named after (plus 1 for ``).\n :type embedding_size: Integer\n :param embeddings_on_cpu: by default embeddings matrices are stored\n on GPU memory if a GPU is used, as it allows for faster access,\n but in some cases the embedding matrix may be really big and\n this parameter forces the placement of the embedding matrix in\n regular memory and the CPU is used to resolve them, slightly\n slowing down the process as a result of data transfer between\n CPU and GPU memory.\n :param fc_layers: list of dictionaries containing the parameters of\n all the fully connected layers.\n :type fc_layers: List\n :param num_fc_layers: Number of stacked fully connected layers.\n :type num_fc_layers: Integer\n :param output_size: Size of each layer.\n :type output_size: Integer\n :param use_bias: bool determines where to use a bias vector.\n :type use_bias: bool\n :param weights_initializer: Initializer for the weights (aka kernel)\n matrix.\n :type weights_initializer: string\n :param bias_initializer: Initializer for the bias vector.\n :type bias_initializer: string\n :param norm: type of normalization to use 'batch' or 'layer'.\n :type norm: string, default None\n :param norm_params: parameters to pass to normalization function.\n :type norm_params: dictionary\n :param activation: Activation function to use.\n :type activation: string\n :param dropout: determines if there should be a dropout layer before\n returning the encoder output.\n :type dropout: float\n \"\"\"\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n\n logger.debug(\" year FCStack\")\n self.year_fc = FCStack(\n first_layer_input_size=1,\n num_layers=1,\n default_output_size=1,\n default_use_bias=use_bias,\n default_weights_initializer=weights_initializer,\n default_bias_initializer=bias_initializer,\n default_norm=None,\n default_norm_params=None,\n default_activation=None,\n default_dropout=dropout,\n )\n\n logger.debug(\" month Embed\")\n self.embed_month = Embed(\n [str(i) for i in range(12)],\n embedding_size,\n representation=\"dense\",\n embeddings_trainable=True,\n pretrained_embeddings=None,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=weights_initializer,\n )\n\n logger.debug(\" day Embed\")\n self.embed_day = Embed(\n [str(i) for i in range(31)],\n embedding_size,\n representation=\"dense\",\n embeddings_trainable=True,\n pretrained_embeddings=None,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=weights_initializer,\n )\n\n logger.debug(\" weekday Embed\")\n self.embed_weekday = Embed(\n [str(i) for i in range(7)],\n embedding_size,\n representation=\"dense\",\n embeddings_trainable=True,\n pretrained_embeddings=None,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=weights_initializer,\n )\n\n logger.debug(\" yearday Embed\")\n self.embed_yearday = Embed(\n [str(i) for i in range(366)],\n embedding_size,\n representation=\"dense\",\n embeddings_trainable=True,\n pretrained_embeddings=None,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=weights_initializer,\n )\n\n logger.debug(\" hour Embed\")\n self.embed_hour = Embed(\n [str(i) for i in range(24)],\n embedding_size,\n representation=\"dense\",\n embeddings_trainable=True,\n pretrained_embeddings=None,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=weights_initializer,\n )\n\n logger.debug(\" minute Embed\")\n self.embed_minute = Embed(\n [str(i) for i in range(60)],\n embedding_size,\n representation=\"dense\",\n embeddings_trainable=True,\n pretrained_embeddings=None,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=weights_initializer,\n )\n\n logger.debug(\" second Embed\")\n self.embed_second = Embed(\n [str(i) for i in range(60)],\n embedding_size,\n representation=\"dense\",\n embeddings_trainable=True,\n pretrained_embeddings=None,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=weights_initializer,\n )\n\n # Summed sizes of all of the embeddings.\n fc_layer_input_size = (\n self.year_fc.output_shape[0]\n + self.embed_month.output_shape[0]\n + self.embed_day.output_shape[0]\n + self.embed_weekday.output_shape[0]\n + self.embed_yearday.output_shape[0]\n + self.embed_hour.output_shape[0]\n + self.embed_minute.output_shape[0]\n + self.embed_second.output_shape[0]\n + 1 # for periodic_second_of_day.\n )\n\n logger.debug(\" FCStack\")\n self.fc_stack = FCStack(\n first_layer_input_size=fc_layer_input_size,\n layers=fc_layers,\n num_layers=num_fc_layers,\n default_output_size=output_size,\n default_use_bias=use_bias,\n default_weights_initializer=weights_initializer,\n default_bias_initializer=bias_initializer,\n default_norm=norm,\n default_norm_params=norm_params,\n default_activation=activation,\n default_dropout=dropout,\n )\n\n def forward(self, inputs: torch.Tensor) -> EncoderOutputDict:\n \"\"\"\n :param inputs: The input vector fed into the encoder.\n Shape: [batch x DATE_INPUT_SIZE], type torch.int8\n :type inputs: Tensor\n \"\"\"\n # ================ Embeddings ================\n input_vector = inputs.type(torch.int)\n\n scaled_year = self.year_fc(input_vector[:, 0:1].type(torch.float))\n embedded_month = self.embed_month(input_vector[:, 1:2] - 1)\n embedded_day = self.embed_day(input_vector[:, 2:3] - 1)\n embedded_weekday = self.embed_weekday(input_vector[:, 3:4])\n embedded_yearday = self.embed_yearday(input_vector[:, 4:5] - 1)\n embedded_hour = self.embed_hour(input_vector[:, 5:6])\n embedded_minute = self.embed_minute(input_vector[:, 6:7])\n embedded_second = self.embed_second(input_vector[:, 7:8])\n periodic_second_of_day = torch_utils.periodic(input_vector[:, 8:9].type(torch.float), 86400)\n\n hidden = torch.cat(\n [\n scaled_year,\n embedded_month,\n embedded_day,\n embedded_weekday,\n embedded_yearday,\n embedded_hour,\n embedded_minute,\n embedded_second,\n periodic_second_of_day,\n ],\n dim=1,\n )\n\n # ================ FC Stack ================\n # logger.debug(' flatten hidden: {0}'.format(hidden))\n\n hidden = self.fc_stack(hidden)\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return DateEmbedConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([DATE_INPUT_SIZE])\n\n @property\n def output_shape(self) -> torch.Size:\n return self.fc_stack.output_shape\n\n\n@DeveloperAPI\n@register_encoder(\"wave\", DATE)\nclass DateWave(Encoder):\n def __init__(\n self,\n fc_layers: list[FCStack] | None = None,\n num_fc_layers: int = 1,\n output_size: int = 10,\n use_bias: bool = True,\n weights_initializer: str = \"xavier_uniform\",\n bias_initializer: str = \"zeros\",\n norm: str | None = None,\n norm_params: dict | None = None,\n activation: str = \"relu\",\n dropout: float = 0,\n encoder_config=None,\n **kwargs,\n ):\n \"\"\"\n :param fc_layers: list of dictionaries containing the parameters of\n all the fully connected layers.\n :type fc_layers: List\n :param num_fc_layers: Number of stacked fully connected layers.\n :type num_fc_layers: Integer\n :param output_size: Size of each layer.\n :type output_size: Integer\n :param use_bias: bool determines where to use a bias vector.\n :type use_bias: bool\n :param weights_initializer: Initializer for the weights (aka kernel)\n matrix.\n :type weights_initializer: string\n :param bias_initializer: Initializer for the bias vector.\n :type bias_initializer: string\n :param norm: type of normalization to use 'batch' or 'layer'.\n :type norm: string, default None\n :param norm_params: parameters to pass to normalization function.\n :type norm_params: dictionary\n :param activation: Activation function to use.\n :type activation: string\n :param dropout: determines if there should be a dropout layer before\n returning the encoder output.\n :type dropout: float\n \"\"\"\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n\n logger.debug(\" year FCStack\")\n self.year_fc = FCStack(\n first_layer_input_size=1,\n num_layers=1,\n default_output_size=1,\n default_use_bias=use_bias,\n default_weights_initializer=weights_initializer,\n default_bias_initializer=bias_initializer,\n default_norm=None,\n default_norm_params=None,\n default_activation=None,\n default_dropout=dropout,\n )\n\n # Summed sizes of all of the embeddings.\n # Additional 8 for periodic_[month, day, ..., second_of_day].\n fc_layer_input_size = self.year_fc.output_shape[0] + 8\n\n logger.debug(\" FCStack\")\n self.fc_stack = FCStack(\n first_layer_input_size=fc_layer_input_size,\n layers=fc_layers,\n num_layers=num_fc_layers,\n default_output_size=output_size,\n default_use_bias=use_bias,\n default_weights_initializer=weights_initializer,\n default_bias_initializer=bias_initializer,\n default_norm=norm,\n default_norm_params=norm_params,\n default_activation=activation,\n default_dropout=dropout,\n )\n\n def forward(self, inputs: torch.Tensor) -> EncoderOutputDict:\n \"\"\"\n :param inputs: The input vector fed into the encoder.\n Shape: [batch x DATE_INPUT_SIZE], type torch.int8\n :type inputs: Tensor\n \"\"\"\n # ================ Embeddings ================\n input_vector = inputs.type(torch.float)\n scaled_year = self.year_fc(input_vector[:, 0:1])\n periodic_month = torch_utils.periodic(input_vector[:, 1:2], 12)\n periodic_day = torch_utils.periodic(input_vector[:, 2:3], 31)\n periodic_weekday = torch_utils.periodic(input_vector[:, 3:4], 7)\n periodic_yearday = torch_utils.periodic(input_vector[:, 4:5], 366)\n periodic_hour = torch_utils.periodic(input_vector[:, 5:6], 24)\n periodic_minute = torch_utils.periodic(input_vector[:, 6:7], 60)\n periodic_second = torch_utils.periodic(input_vector[:, 7:8], 60)\n periodic_second_of_day = torch_utils.periodic(input_vector[:, 8:9], 86400)\n\n hidden = torch.cat(\n [\n scaled_year,\n periodic_month,\n periodic_day,\n periodic_weekday,\n periodic_yearday,\n periodic_hour,\n periodic_minute,\n periodic_second,\n periodic_second_of_day,\n ],\n dim=1,\n )\n\n # ================ FC Stack ================\n # logger.debug(' flatten hidden: {0}'.format(hidden))\n\n hidden = self.fc_stack(hidden)\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return DateWaveConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([DATE_INPUT_SIZE])\n\n @property\n def output_shape(self) -> torch.Size:\n return self.fc_stack.output_shape\n"
},
{
"path": "ludwig/encoders/generic_encoders.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\n\nimport torch\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import BINARY, ENCODER_OUTPUT, NUMBER, TEXT, TIMESERIES, VECTOR\nfrom ludwig.encoders.base import Encoder\nfrom ludwig.encoders.registry import register_encoder\nfrom ludwig.encoders.types import EncoderOutputDict\nfrom ludwig.modules.fully_connected_modules import FCStack\nfrom ludwig.schema.encoders.base import BaseEncoderConfig, DenseEncoderConfig, PassthroughEncoderConfig\n\nlogger = logging.getLogger(__name__)\n\n\n@DeveloperAPI\n@register_encoder(\"passthrough\", [BINARY, NUMBER, TEXT, VECTOR])\nclass PassthroughEncoder(Encoder):\n def __init__(self, input_size=1, encoder_config=None, **kwargs):\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n self.input_size = input_size\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n \"\"\"\n :param inputs: The inputs fed into the encoder.\n Shape: [batch x 1], type tf.float32\n \"\"\"\n return {ENCODER_OUTPUT: inputs}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return PassthroughEncoderConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.input_size])\n\n @property\n def output_shape(self) -> torch.Size:\n return self.input_shape\n\n\n@DeveloperAPI\n@register_encoder(\"dense\", [BINARY, NUMBER, VECTOR, TIMESERIES])\nclass DenseEncoder(Encoder):\n def __init__(\n self,\n input_size,\n fc_layers=None,\n num_layers=1,\n output_size=256,\n use_bias=True,\n weights_initializer=\"xavier_uniform\",\n bias_initializer=\"zeros\",\n norm=None,\n norm_params=None,\n activation=\"relu\",\n dropout=0,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n self.input_size = input_size\n\n logger.debug(\" FCStack\")\n self.fc_stack = FCStack(\n first_layer_input_size=input_size,\n layers=fc_layers,\n num_layers=num_layers,\n default_output_size=output_size,\n default_use_bias=use_bias,\n default_weights_initializer=weights_initializer,\n default_bias_initializer=bias_initializer,\n default_norm=norm,\n default_norm_params=norm_params,\n default_activation=activation,\n default_dropout=dropout,\n )\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n \"\"\"\n :param inputs: The inputs fed into the encoder.\n Shape: [batch x 1], type tf.float32\n \"\"\"\n return {ENCODER_OUTPUT: self.fc_stack(inputs)}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return DenseEncoderConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.input_size])\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.fc_stack.layers[-1][\"output_size\"]])\n"
},
{
"path": "ludwig/encoders/h3_encoders.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\n\nimport torch\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import ENCODER_OUTPUT, ENCODER_OUTPUT_STATE, H3\nfrom ludwig.encoders.base import Encoder\nfrom ludwig.encoders.registry import register_encoder\nfrom ludwig.encoders.types import EncoderOutputDict\nfrom ludwig.modules.embedding_modules import Embed, EmbedSequence\nfrom ludwig.modules.fully_connected_modules import FCStack\nfrom ludwig.modules.initializer_modules import get_initializer\nfrom ludwig.modules.recurrent_modules import RecurrentStack\nfrom ludwig.modules.reduction_modules import SequenceReducer\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.h3_encoders import H3EmbedConfig, H3RNNConfig, H3WeightedSumConfig\nfrom ludwig.utils import torch_utils\n\nlogger = logging.getLogger(__name__)\n\n# TODO: Share this with h3_feature.H3_VECTOR_LENGTH\nH3_INPUT_SIZE = 19\n\n\n@DeveloperAPI\n@register_encoder(\"embed\", H3)\nclass H3Embed(Encoder):\n def __init__(\n self,\n embedding_size: int = 10,\n embeddings_on_cpu: bool = False,\n fc_layers: list | None = None,\n num_fc_layers: int = 0,\n output_size: int = 10,\n use_bias: bool = True,\n weights_initializer: str = \"xavier_uniform\",\n bias_initializer: str = \"zeros\",\n norm: str = None,\n norm_params: dict = None,\n activation: str = \"relu\",\n dropout: float = 0,\n reduce_output: str = \"sum\",\n encoder_config=None,\n **kwargs,\n ):\n \"\"\"\n :param embedding_size: it is the maximum embedding size, the actual\n size will be `min(vocabulary_size, embedding_size)`\n for `dense` representations and exactly `vocabulary_size`\n for the `sparse` encoding, where `vocabulary_size` is\n the number of different strings appearing in the training set\n in the column the feature is named after (plus 1 for\n ``).\n :type embedding_size: Integer\n :param embeddings_on_cpu: by default embeddings matrices are stored\n on GPU memory if a GPU is used, as it allows\n for faster access, but in some cases the embedding matrix\n may be really big and this parameter forces the placement\n of the embedding matrix in regular memory and the CPU is used\n to resolve them, slightly slowing down the process\n as a result of data transfer between CPU and GPU memory.\n :param dropout: determines if there should be a dropout layer before\n returning the encoder output.\n :type dropout: Boolean\n \"\"\"\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n\n self.embedding_size = embedding_size\n self.reduce_output = reduce_output\n self.reduce_sequence = SequenceReducer(reduce_mode=reduce_output)\n\n logger.debug(\" mode Embed\")\n self.embed_mode = Embed(\n [str(i) for i in range(3)],\n embedding_size,\n representation=\"dense\",\n embeddings_trainable=True,\n pretrained_embeddings=None,\n force_embedding_size=True,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=weights_initializer,\n )\n\n logger.debug(\" edge Embed\")\n self.embed_edge = Embed(\n [str(i) for i in range(7)],\n embedding_size,\n representation=\"dense\",\n embeddings_trainable=True,\n pretrained_embeddings=None,\n force_embedding_size=True,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=weights_initializer,\n )\n\n logger.debug(\" resolution Embed\")\n self.embed_resolution = Embed(\n [str(i) for i in range(16)],\n embedding_size,\n representation=\"dense\",\n embeddings_trainable=True,\n pretrained_embeddings=None,\n force_embedding_size=True,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=weights_initializer,\n )\n\n logger.debug(\" base cell Embed\")\n self.embed_base_cell = Embed(\n [str(i) for i in range(122)],\n embedding_size,\n representation=\"dense\",\n embeddings_trainable=True,\n pretrained_embeddings=None,\n force_embedding_size=True,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=weights_initializer,\n )\n\n logger.debug(\" cells Embed\")\n self.embed_cells = EmbedSequence(\n [str(i) for i in range(8)],\n embedding_size,\n max_sequence_length=(H3_INPUT_SIZE - 4),\n representation=\"dense\",\n embeddings_trainable=True,\n pretrained_embeddings=None,\n force_embedding_size=True,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=weights_initializer,\n )\n\n logger.debug(\" FCStack\")\n self.fc_stack = FCStack(\n first_layer_input_size=embedding_size,\n layers=fc_layers,\n num_layers=num_fc_layers,\n default_output_size=output_size,\n default_use_bias=use_bias,\n default_weights_initializer=weights_initializer,\n default_bias_initializer=bias_initializer,\n default_norm=norm,\n default_norm_params=norm_params,\n default_activation=activation,\n default_dropout=dropout,\n )\n\n def forward(self, inputs: torch.Tensor) -> EncoderOutputDict:\n \"\"\"\n :param inputs: The input vector fed into the encoder.\n Shape: [batch x H3_INPUT_SIZE], type torch.int8\n :type inputs: Tensor\n \"\"\"\n input_vector = inputs.int()\n\n # ================ Embeddings ================\n embedded_mode = self.embed_mode(input_vector[:, 0:1]).unsqueeze(1)\n embedded_edge = self.embed_edge(input_vector[:, 1:2]).unsqueeze(1)\n embedded_resolution = self.embed_resolution(input_vector[:, 2:3]).unsqueeze(1)\n embedded_base_cell = self.embed_base_cell(input_vector[:, 3:4]).unsqueeze(1)\n embedded_cells = self.embed_cells(input_vector[:, 4:])\n\n # ================ Masking ================\n # Mask out cells beyond the resolution of interest.\n resolution = input_vector[:, 2]\n mask = torch.unsqueeze(torch_utils.sequence_mask(resolution, 15), dim=-1).float()\n # Batch size X 15(max resolution) X embedding size\n masked_embedded_cells = embedded_cells * mask\n\n # ================ Reduce ================\n # Batch size X H3_INPUT_SIZE X embedding size\n concatenated = torch.cat(\n [embedded_mode, embedded_edge, embedded_resolution, embedded_base_cell, masked_embedded_cells], dim=1\n )\n\n hidden = self.reduce_sequence(concatenated)\n\n # ================ FC Stack ================\n # logger.debug(' flatten hidden: {0}'.format(hidden))\n hidden = self.fc_stack(hidden)\n\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return H3EmbedConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([H3_INPUT_SIZE])\n\n @property\n def output_shape(self) -> torch.Size:\n return self.fc_stack.output_shape\n\n\n@DeveloperAPI\n@register_encoder(\"weighted_sum\", H3)\nclass H3WeightedSum(Encoder):\n def __init__(\n self,\n embedding_size: int = 10,\n embeddings_on_cpu: bool = False,\n should_softmax: bool = False,\n fc_layers: list | None = None,\n num_fc_layers: int = 0,\n output_size: int = 10,\n use_bias: bool = True,\n weights_initializer: str = \"xavier_uniform\",\n bias_initializer: str = \"zeros\",\n norm: str | None = None,\n norm_params: dict = None,\n activation: str = \"relu\",\n dropout: float = 0,\n encoder_config=None,\n **kwargs,\n ):\n \"\"\"\n :param embedding_size: it is the maximum embedding size, the actual\n size will be `min(vocabulary_size, embedding_size)`\n for `dense` representations and exactly `vocabulary_size`\n for the `sparse` encoding, where `vocabulary_size` is\n the number of different strings appearing in the training set\n in the column the feature is named after (plus 1 for\n ``).\n :type embedding_size: Integer\n :param embeddings_on_cpu: by default embeddings matrices are stored\n on GPU memory if a GPU is used, as it allows\n for faster access, but in some cases the embedding matrix\n may be really big and this parameter forces the placement\n of the embedding matrix in regular memory and the CPU is used\n to resolve them, slightly slowing down the process\n as a result of data transfer between CPU and GPU memory.\n :param dropout: determines if there should be a dropout layer before\n returning the encoder output.\n :type dropout: Boolean\n \"\"\"\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n\n self.should_softmax = should_softmax\n self.sum_sequence_reducer = SequenceReducer(reduce_mode=\"sum\")\n\n self.h3_embed = H3Embed(\n embedding_size,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n weights_initializer=weights_initializer,\n bias_initializer=bias_initializer,\n reduce_output=\"None\",\n )\n\n self.register_buffer(\n \"aggregation_weights\", torch.Tensor(get_initializer(weights_initializer)([H3_INPUT_SIZE, 1]))\n )\n\n logger.debug(\" FCStack\")\n self.fc_stack = FCStack(\n first_layer_input_size=self.h3_embed.output_shape[0],\n layers=fc_layers,\n num_layers=num_fc_layers,\n default_output_size=output_size,\n default_use_bias=use_bias,\n default_weights_initializer=weights_initializer,\n default_bias_initializer=bias_initializer,\n default_norm=norm,\n default_norm_params=norm_params,\n default_activation=activation,\n default_dropout=dropout,\n )\n\n def forward(self, inputs: torch.Tensor) -> EncoderOutputDict:\n \"\"\"\n :param inputs: The input vector fed into the encoder.\n Shape: [batch x H3_INPUT_SIZE], type torch.int8\n :type inputs: Tensor\n \"\"\"\n # ================ Embeddings ================\n input_vector = inputs\n embedded_h3 = self.h3_embed(input_vector)\n\n # ================ Weighted Sum ================\n if self.should_softmax:\n weights = torch.softmax(self.aggregation_weights, dim=None)\n else:\n weights = self.aggregation_weights\n\n hidden = self.sum_sequence_reducer(embedded_h3[ENCODER_OUTPUT] * weights)\n\n # ================ FC Stack ================\n # logger.debug(' flatten hidden: {0}'.format(hidden))\n hidden = self.fc_stack(hidden)\n\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return H3WeightedSumConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([H3_INPUT_SIZE])\n\n @property\n def output_shape(self) -> torch.Size:\n return self.fc_stack.output_shape\n\n\n@DeveloperAPI\n@register_encoder(\"rnn\", H3)\nclass H3RNN(Encoder):\n def __init__(\n self,\n embedding_size: int = 10,\n embeddings_on_cpu: bool = False,\n num_layers: int = 1,\n hidden_size: int = 10,\n cell_type: str = \"rnn\",\n bidirectional: bool = False,\n activation: str = \"tanh\",\n recurrent_activation: str = \"sigmoid\",\n use_bias: bool = True,\n unit_forget_bias: bool = True,\n weights_initializer: str = \"xavier_uniform\",\n recurrent_initializer: str = \"orthogonal\",\n bias_initializer: str = \"zeros\",\n dropout: float = 0.0,\n recurrent_dropout: float = 0.0,\n reduce_output: str = \"last\",\n encoder_config=None,\n **kwargs,\n ):\n \"\"\"\n :param embedding_size: it is the maximum embedding size, the actual\n size will be `min(vocabulary_size, embedding_size)`\n for `dense` representations and exactly `vocabulary_size`\n for the `sparse` encoding, where `vocabulary_size` is\n the number of different strings appearing in the training set\n in the column the feature is named after (plus 1 for\n ``).\n :type embedding_size: Integer\n :param embeddings_on_cpu: by default embeddings matrices are stored\n on GPU memory if a GPU is used, as it allows\n for faster access, but in some cases the embedding matrix\n may be really big and this parameter forces the placement\n of the embedding matrix in regular memory and the CPU is used\n to resolve them, slightly slowing down the process\n as a result of data transfer between CPU and GPU memory.\n :param num_layers: the number of stacked recurrent layers.\n :type num_layers: Integer\n :param cell_type: the type of recurrent cell to use.\n Available values are: `rnn`, `lstm`, `lstm_block`, `lstm`,\n `ln`, `lstm_cudnn`, `gru`, `gru_block`, `gru_cudnn`.\n For reference about the differences between the cells please\n refer to PyTorch's documentation. We suggest to use the\n `block` variants on CPU and the `cudnn` variants on GPU\n because of their increased speed.\n :type cell_type: str\n :param hidden_size: the size of the state of the rnn.\n :type hidden_size: Integer\n :param bidirectional: if `True` two recurrent networks will perform\n encoding in the forward and backward direction and\n their outputs will be concatenated.\n :type bidirectional: Boolean\n :param activation: Activation function to use.\n :type activation: string\n :param recurrent_activation: Activation function to use for the\n recurrent step.\n :type recurrent_activation: string\n :param use_bias: bool determines where to use a bias vector\n :type use_bias: bool\n :param unit_forget_bias: if True add 1 to the bias forget gate at\n initialization.\n :type unit_forget_bias: bool\n :param weights_initializer: Initializer for the weights (aka kernel)\n matrix\n :type weights_initializer: string\n :param recurrent_initializer: Initializer for the recurrent weights\n matrix\n :type recurrent_initializer: string\n :param bias_initializer: Initializer for the bias vector\n :type bias_initializer: string\n :param dropout: determines if there should be a dropout layer before\n returning the encoder output.\n :type dropout: float\n :param recurrent_dropout: Dropout rate for the RNN encoder of the H3 embeddings.\n :type recurrent_dropout: float\n \"\"\"\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n\n self.embedding_size = embedding_size\n\n self.h3_embed = H3Embed(\n embedding_size,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n weights_initializer=weights_initializer,\n bias_initializer=bias_initializer,\n reduce_output=\"None\",\n )\n\n logger.debug(\" RecurrentStack\")\n self.recurrent_stack = RecurrentStack(\n input_size=self.h3_embed.output_shape[0],\n max_sequence_length=H3_INPUT_SIZE,\n hidden_size=hidden_size,\n cell_type=cell_type,\n num_layers=num_layers,\n bidirectional=bidirectional,\n use_bias=use_bias,\n dropout=recurrent_dropout,\n )\n\n def forward(self, inputs: torch.Tensor) -> EncoderOutputDict:\n \"\"\"\n :param inputs: The input vector fed into the encoder.\n Shape: [batch x H3_INPUT_SIZE], type torch.int8\n :type inputs: Tensor\n \"\"\"\n\n # ================ Embeddings ================\n embedded_h3 = self.h3_embed(inputs)\n\n # ================ RNN ================\n hidden, final_state = self.recurrent_stack(embedded_h3[ENCODER_OUTPUT])\n\n return {ENCODER_OUTPUT: hidden, ENCODER_OUTPUT_STATE: final_state}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return H3RNNConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([H3_INPUT_SIZE])\n\n @property\n def output_shape(self) -> torch.Size:\n return self.recurrent_stack.output_shape\n"
},
{
"path": "ludwig/encoders/image/__init__.py",
"content": "import ludwig.encoders.image.base\nimport ludwig.encoders.image.timm # noqa\nimport ludwig.encoders.image.torchvision # noqa\n"
},
{
"path": "ludwig/encoders/image/base.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nfrom typing import Any\n\nimport torch\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import ENCODER_OUTPUT, ENCODER_OUTPUT_STATE, IMAGE\nfrom ludwig.encoders.base import Encoder\nfrom ludwig.encoders.registry import register_encoder\nfrom ludwig.encoders.types import EncoderOutputDict\nfrom ludwig.modules.convolutional_modules import Conv2DStack, ResNet, UNetDownStack\nfrom ludwig.modules.fully_connected_modules import FCStack\nfrom ludwig.modules.mlp_mixer_modules import MLPMixer\nfrom ludwig.schema.encoders.image.base import (\n ImageEncoderConfig,\n MLPMixerConfig,\n ResNetConfig,\n Stacked2DCNNConfig,\n UNetEncoderConfig,\n ViTConfig,\n)\nfrom ludwig.utils.torch_utils import FreezeModule\n\nlogger = logging.getLogger(__name__)\n\n\n@DeveloperAPI\nclass ImageEncoder(Encoder):\n pass\n\n\n@DeveloperAPI\n@register_encoder(\"stacked_cnn\", IMAGE)\nclass Stacked2DCNN(ImageEncoder):\n def __init__(\n self,\n height: int,\n width: int,\n conv_layers: list[dict] | None = None,\n num_conv_layers: int | None = None,\n num_channels: int = None,\n out_channels: int = 32,\n kernel_size: int | tuple[int] = 3,\n stride: int | tuple[int] = 1,\n padding: int | tuple[int] | str = \"valid\",\n dilation: int | tuple[int] = 1,\n conv_use_bias: bool = True,\n padding_mode: str = \"zeros\",\n conv_norm: str | None = None,\n conv_norm_params: dict[str, Any] | None = None,\n conv_activation: str = \"relu\",\n conv_dropout: int = 0,\n pool_function: str = \"max\",\n pool_kernel_size: int | tuple[int] = 2,\n pool_stride: int | tuple[int] = None,\n pool_padding: int | tuple[int] = 0,\n pool_dilation: int | tuple[int] = 1,\n groups: int = 1,\n fc_layers: list[dict] | None = None,\n num_fc_layers: int | None = 1,\n output_size: int = 128,\n fc_use_bias: bool = True,\n fc_weights_initializer: str = \"xavier_uniform\",\n fc_bias_initializer: str = \"zeros\",\n fc_norm: str | None = None,\n fc_norm_params: dict[str, Any] | None = None,\n fc_activation: str = \"relu\",\n fc_dropout: float = 0,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n\n # map parameter input feature config names to internal names\n img_height = height\n img_width = width\n first_in_channels = num_channels\n\n self._input_shape = (first_in_channels, img_height, img_width)\n\n if first_in_channels is None:\n raise ValueError(\"first_in_channels must not be None.\")\n\n logger.debug(\" Conv2DStack\")\n self.conv_stack_2d = Conv2DStack(\n img_height=img_height,\n img_width=img_width,\n layers=conv_layers,\n num_layers=num_conv_layers,\n first_in_channels=first_in_channels,\n default_out_channels=out_channels,\n default_kernel_size=kernel_size,\n default_stride=stride,\n default_padding=padding,\n default_dilation=dilation,\n default_groups=groups,\n default_use_bias=conv_use_bias,\n default_padding_mode=padding_mode,\n default_norm=conv_norm,\n default_norm_params=conv_norm_params,\n default_activation=conv_activation,\n default_dropout=conv_dropout,\n default_pool_function=pool_function,\n default_pool_kernel_size=pool_kernel_size,\n default_pool_stride=pool_stride,\n default_pool_padding=pool_padding,\n default_pool_dilation=pool_dilation,\n )\n out_channels, img_height, img_width = self.conv_stack_2d.output_shape\n first_fc_layer_input_size = out_channels * img_height * img_width\n\n self.flatten = torch.nn.Flatten()\n\n logger.debug(\" FCStack\")\n self.fc_stack = FCStack(\n first_layer_input_size=first_fc_layer_input_size,\n layers=fc_layers,\n num_layers=num_fc_layers,\n default_output_size=output_size,\n default_use_bias=fc_use_bias,\n default_weights_initializer=fc_weights_initializer,\n default_bias_initializer=fc_bias_initializer,\n default_norm=fc_norm,\n default_norm_params=fc_norm_params,\n default_activation=fc_activation,\n default_dropout=fc_dropout,\n )\n\n def forward(self, inputs: torch.Tensor) -> EncoderOutputDict:\n \"\"\"\n :param inputs: The inputs fed into the encoder.\n Shape: [batch x channels x height x width], type torch.uint8\n \"\"\"\n\n hidden = self.conv_stack_2d(inputs)\n hidden = self.flatten(hidden)\n outputs = self.fc_stack(hidden)\n\n return {ENCODER_OUTPUT: outputs}\n\n @staticmethod\n def get_schema_cls() -> type[ImageEncoderConfig]:\n return Stacked2DCNNConfig\n\n @property\n def output_shape(self) -> torch.Size:\n return self.fc_stack.output_shape\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size(self._input_shape)\n\n\n@DeveloperAPI\n@register_encoder(\"_resnet_legacy\", IMAGE)\nclass ResNetEncoder(ImageEncoder):\n def __init__(\n self,\n height: int,\n width: int,\n resnet_size: int = 50,\n num_channels: int = 3,\n out_channels: int = 16,\n kernel_size: int | tuple[int] = 3,\n conv_stride: int | tuple[int] = 1,\n first_pool_kernel_size: int | tuple[int] = None,\n first_pool_stride: int | tuple[int] = None,\n batch_norm_momentum: float = 0.1,\n batch_norm_epsilon: float = 0.001,\n fc_layers: list[dict] | None = None,\n num_fc_layers: int | None = 1,\n output_size: int = 256,\n use_bias: bool = True,\n weights_initializer: str = \"xavier_uniform\",\n bias_initializer: str = \"zeros\",\n norm: str | None = None,\n norm_params: dict[str, Any] | None = None,\n activation: str = \"relu\",\n dropout: float = 0,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n # map parameter input feature config names to internal names\n img_height = height\n img_width = width\n first_in_channels = num_channels\n\n self._input_shape = (first_in_channels, img_height, img_width)\n\n logger.debug(\" ResNet\")\n self.resnet = ResNet(\n img_height=img_height,\n img_width=img_width,\n first_in_channels=first_in_channels,\n out_channels=out_channels,\n resnet_size=resnet_size,\n kernel_size=kernel_size,\n conv_stride=conv_stride,\n first_pool_kernel_size=first_pool_kernel_size,\n first_pool_stride=first_pool_stride,\n batch_norm_momentum=batch_norm_momentum,\n batch_norm_epsilon=batch_norm_epsilon,\n )\n first_fc_layer_input_size = self.resnet.output_shape[0]\n\n logger.debug(\" FCStack\")\n self.fc_stack = FCStack(\n first_layer_input_size=first_fc_layer_input_size,\n layers=fc_layers,\n num_layers=num_fc_layers,\n default_output_size=output_size,\n default_use_bias=use_bias,\n default_weights_initializer=weights_initializer,\n default_bias_initializer=bias_initializer,\n default_norm=norm,\n default_norm_params=norm_params,\n default_activation=activation,\n default_dropout=dropout,\n )\n\n def forward(self, inputs: torch.Tensor) -> EncoderOutputDict:\n hidden = self.resnet(inputs)\n axes = [2, 3]\n hidden = torch.mean(hidden, axes)\n hidden = self.fc_stack(hidden)\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[ImageEncoderConfig]:\n return ResNetConfig\n\n @property\n def output_shape(self) -> torch.Size:\n return self.fc_stack.output_shape\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size(self._input_shape)\n\n\n@DeveloperAPI\n@register_encoder(\"mlp_mixer\", IMAGE)\nclass MLPMixerEncoder(ImageEncoder):\n def __init__(\n self,\n height: int,\n width: int,\n num_channels: int = None,\n patch_size: int = 16,\n embed_size: int = 512,\n token_size: int = 2048,\n channel_dim: int = 256,\n num_layers: int = 8,\n dropout: float = 0.0,\n avg_pool: bool = True,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n # map parameter input feature config names to internal names\n img_height = height\n img_width = width\n in_channels = num_channels\n\n if num_channels is None:\n raise RuntimeError(\"num_channels must not be None\")\n\n self._input_shape = (in_channels, img_height, img_width)\n\n logger.debug(\" MLPMixer\")\n self.mlp_mixer = MLPMixer(\n img_height=img_height,\n img_width=img_width,\n in_channels=in_channels,\n patch_size=patch_size,\n embed_size=embed_size,\n token_size=token_size,\n channel_dim=channel_dim,\n num_layers=num_layers,\n dropout=dropout,\n avg_pool=avg_pool,\n )\n\n self._output_shape = self.mlp_mixer.output_shape\n\n def forward(self, inputs: torch.Tensor) -> EncoderOutputDict:\n hidden = self.mlp_mixer(inputs)\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[ImageEncoderConfig]:\n return MLPMixerConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size(self._input_shape)\n\n @property\n def output_shape(self) -> torch.Size:\n return self._output_shape\n\n\n@DeveloperAPI\n@register_encoder(\"_vit_legacy\", IMAGE)\nclass ViTEncoder(ImageEncoder):\n def __init__(\n self,\n height: int,\n width: int,\n num_channels: int = 3,\n use_pretrained: bool = True,\n pretrained_model: str = \"google/vit-base-patch16-224\",\n saved_weights_in_checkpoint: bool = False,\n hidden_size: int = 768,\n num_hidden_layers: int = 12,\n num_attention_heads: int = 12,\n intermediate_size: int = 3072,\n hidden_act: str = \"gelu\",\n hidden_dropout_prob: float = 0.1,\n attention_probs_dropout_prob: float = 0.1,\n initializer_range: float = 0.02,\n layer_norm_eps: float = 1e-12,\n gradient_checkpointing: bool = False,\n patch_size: int = 16,\n trainable: bool = True,\n encoder_config=None,\n **kwargs,\n ):\n \"\"\"Creates a ViT encoder using transformers.ViTModel.\n\n use_pretrained: If True, uses a pretrained transformer based on the\n pretrained_model argument.\n pretrained: If str, expects the path to a pretrained model or the id of\n a model on huggingface.co, and ignores the configuration provided in\n the arguments.\n \"\"\"\n super().__init__()\n self.config = encoder_config\n\n try:\n from transformers import ViTConfig, ViTModel\n except ModuleNotFoundError:\n raise RuntimeError(\n \" transformers is not installed. \"\n \"In order to install all image feature dependencies run \"\n \"pip install ludwig[image]\"\n )\n\n # map parameter input feature config names to internal names\n img_height = height\n img_width = width\n in_channels = num_channels\n\n img_width = img_width or img_height\n if img_width != img_height:\n raise ValueError(\"img_height and img_width should be identical.\")\n self._input_shape = (in_channels, img_height, img_width)\n\n config_dict: dict\n if use_pretrained and not saved_weights_in_checkpoint:\n config_dict = {\n \"pretrained_model_name_or_path\": pretrained_model,\n }\n transformer = ViTModel.from_pretrained(**config_dict)\n else:\n config_dict = {\n \"image_size\": img_height,\n \"num_channels\": in_channels,\n \"patch_size\": patch_size,\n \"hidden_size\": hidden_size,\n \"num_hidden_layers\": num_hidden_layers,\n \"num_attention_heads\": num_attention_heads,\n \"intermediate_size\": intermediate_size,\n \"hidden_act\": hidden_act,\n \"hidden_dropout_prob\": hidden_dropout_prob,\n \"attention_probs_dropout_prob\": attention_probs_dropout_prob,\n \"initializer_range\": initializer_range,\n \"layer_norm_eps\": layer_norm_eps,\n \"gradient_checkpointing\": gradient_checkpointing,\n }\n config = ViTConfig(**config_dict)\n transformer = ViTModel(config)\n\n self.transformer = FreezeModule(transformer, frozen=not trainable)\n\n self._output_shape = (transformer.config.hidden_size,)\n\n def forward(self, inputs: torch.Tensor, head_mask: torch.Tensor | None = None) -> EncoderOutputDict:\n output = self.transformer.module(inputs, head_mask=head_mask)\n return {ENCODER_OUTPUT: output.pooler_output}\n\n @staticmethod\n def get_schema_cls() -> type[ImageEncoderConfig]:\n return ViTConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size(self._input_shape)\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size(self._output_shape)\n\n\n@DeveloperAPI\n@register_encoder(\"unet\", IMAGE)\nclass UNetEncoder(ImageEncoder):\n def __init__(\n self,\n height: int,\n width: int,\n num_channels: int = 3,\n conv_norm: str | None = None,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n if height % 16 or width % 16:\n raise ValueError(f\"Invalid `height` {height} or `width` {width} for unet encoder\")\n\n self.unet = UNetDownStack(\n img_height=height,\n img_width=width,\n in_channels=num_channels,\n norm=conv_norm,\n )\n\n def forward(self, inputs: torch.Tensor) -> EncoderOutputDict:\n hidden, skips = self.unet(inputs)\n return {ENCODER_OUTPUT: hidden, ENCODER_OUTPUT_STATE: skips}\n\n @staticmethod\n def get_schema_cls() -> type[ImageEncoderConfig]:\n return UNetEncoderConfig\n\n @property\n def output_shape(self) -> torch.Size:\n return self.unet.output_shape\n\n @property\n def input_shape(self) -> torch.Size:\n return self.unet.input_shape\n"
},
{
"path": "ludwig/encoders/image/timm.py",
"content": "import logging\n\nimport torch\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import ENCODER_OUTPUT, IMAGE\nfrom ludwig.encoders.image.base import ImageEncoder\nfrom ludwig.encoders.registry import register_encoder\nfrom ludwig.encoders.types import EncoderOutputDict\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.image.timm import (\n TimmCAFormerEncoderConfig,\n TimmConvFormerEncoderConfig,\n TimmEncoderConfig,\n TimmPoolFormerEncoderConfig,\n)\n\nlogger = logging.getLogger(__name__)\n\n\ndef _get_timm():\n try:\n import timm\n except ImportError:\n raise ImportError(\"timm is required for this encoder. Install it with: pip install timm\")\n return timm\n\n\n@DeveloperAPI\n@register_encoder(\"timm\", IMAGE)\nclass TimmEncoder(ImageEncoder):\n \"\"\"Wraps any model from the timm (pytorch-image-models) library as a Ludwig image encoder.\n\n This provides access to hundreds of pretrained vision models including MetaFormer variants\n (CAFormer, ConvFormer, PoolFormer), ConvNeXt V2, EfficientFormer, and many more.\n\n Usage in Ludwig config:\n encoder:\n type: timm\n model_name: caformer_s18.sail_in22k_ft_in1k\n use_pretrained: true\n trainable: true\n \"\"\"\n\n def __init__(\n self,\n model_name: str = \"caformer_s18\",\n use_pretrained: bool = True,\n trainable: bool = True,\n saved_weights_in_checkpoint: bool = False,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n\n timm = _get_timm()\n\n pretrained = use_pretrained and not saved_weights_in_checkpoint\n if pretrained:\n logger.info(f\"Instantiating timm image encoder '{model_name}' with pretrained weights.\")\n else:\n logger.info(f\"Instantiating timm image encoder '{model_name}' without pretrained weights.\")\n\n # num_classes=0 removes the classification head, returning pooled features\n self.model = timm.create_model(model_name, pretrained=pretrained, num_classes=0)\n\n # Get the model's expected input config for input_shape\n data_config = timm.data.resolve_model_data_config(self.model)\n self._input_size = data_config[\"input_size\"] # (C, H, W)\n\n # Compute output dim by running a dummy forward\n with torch.no_grad():\n dummy = torch.zeros(1, *self._input_size)\n out = self.model(dummy)\n self._output_dim = out.shape[-1]\n\n for p in self.model.parameters():\n p.requires_grad_(trainable)\n\n def forward(self, inputs: torch.Tensor) -> EncoderOutputDict:\n return {ENCODER_OUTPUT: self.model(inputs)}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TimmEncoderConfig\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self._output_dim])\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size(self._input_size)\n\n\n@DeveloperAPI\n@register_encoder(\"caformer\", IMAGE)\nclass TimmCAFormerEncoder(TimmEncoder):\n \"\"\"CAFormer encoder — hybrid Conv+Attention MetaFormer achieving SOTA accuracy on ImageNet.\n\n Variants: s18 (26M, 83.6%), s36 (39M, 84.5%), m36 (56M, 85.2%), b36 (99M, 85.5%).\n \"\"\"\n\n def __init__(self, model_name: str = \"caformer_s18\", **kwargs):\n super().__init__(model_name=model_name, **kwargs)\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TimmCAFormerEncoderConfig\n\n\n@DeveloperAPI\n@register_encoder(\"convformer\", IMAGE)\nclass TimmConvFormerEncoder(TimmEncoder):\n \"\"\"ConvFormer encoder — pure CNN MetaFormer that outperforms ConvNeXt.\"\"\"\n\n def __init__(self, model_name: str = \"convformer_s18\", **kwargs):\n super().__init__(model_name=model_name, **kwargs)\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TimmConvFormerEncoderConfig\n\n\n@DeveloperAPI\n@register_encoder(\"poolformer\", IMAGE)\nclass TimmPoolFormerEncoder(TimmEncoder):\n \"\"\"PoolFormer encoder — MetaFormer using simple average pooling as token mixer.\"\"\"\n\n def __init__(self, model_name: str = \"poolformerv2_s12\", **kwargs):\n super().__init__(model_name=model_name, **kwargs)\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TimmPoolFormerEncoderConfig\n"
},
{
"path": "ludwig/encoders/image/torchvision.py",
"content": "import logging\nimport os\nfrom abc import abstractmethod\n\nimport torch\nimport torchvision.models as tvm\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import ENCODER_OUTPUT, IMAGE\nfrom ludwig.encoders.image.base import ImageEncoder\nfrom ludwig.encoders.registry import register_encoder\nfrom ludwig.encoders.types import EncoderOutputDict\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.image.torchvision import (\n TVAlexNetEncoderConfig,\n TVConvNeXtEncoderConfig,\n TVDenseNetEncoderConfig,\n TVEfficientNetEncoderConfig,\n TVGoogLeNetEncoderConfig,\n TVInceptionV3EncoderConfig,\n TVMaxVitEncoderConfig,\n TVMNASNetEncoderConfig,\n TVMobileNetV2EncoderConfig,\n TVMobileNetV3EncoderConfig,\n TVRegNetEncoderConfig,\n TVResNetEncoderConfig,\n TVResNeXtEncoderConfig,\n TVShuffleNetV2EncoderConfig,\n TVSqueezeNetEncoderConfig,\n TVSwinTransformerEncoderConfig,\n TVVGGEncoderConfig,\n TVViTEncoderConfig,\n TVWideResNetEncoderConfig,\n)\nfrom ludwig.utils.image_utils import register_torchvision_model_variants, torchvision_model_registry, TVModelVariant\n\nlogger = logging.getLogger(__name__)\n\n\n@DeveloperAPI\nclass TVBaseEncoder(ImageEncoder):\n def __init__(\n self,\n model_variant: str | int = None,\n use_pretrained: bool = True,\n saved_weights_in_checkpoint: bool = False,\n model_cache_dir: str | None = None,\n trainable: bool = True,\n **kwargs,\n ):\n super().__init__()\n\n logger.debug(f\" {self.name}\")\n # map parameter input feature config names to internal names\n self.model_variant = model_variant\n self.use_pretrained = use_pretrained\n self.model_cache_dir = model_cache_dir\n\n # remove any Ludwig specific keyword parameters\n kwargs.pop(\"encoder_config\", None)\n kwargs.pop(\"type\", None)\n kwargs.pop(\"skip\", None)\n\n # cache pre-trained models if requested\n # based on https://github.com/pytorch/vision/issues/616#issuecomment-428637564\n if self.model_cache_dir is not None:\n os.environ[\"TORCH_HOME\"] = self.model_cache_dir\n\n # retrieve function to create requested model\n self.create_model = torchvision_model_registry[self.torchvision_model_type][\n self.model_variant\n ].create_model_function\n\n # get weight specification\n if use_pretrained and not saved_weights_in_checkpoint:\n weights_specification = torchvision_model_registry[self.torchvision_model_type][\n self.model_variant\n ].model_weights.DEFAULT\n logger.info(\n f\"Instantiating torchvision image encoder '{self.torchvision_model_type}' with pretrained weights: \"\n f\"{weights_specification}.\"\n )\n else:\n weights_specification = None\n if saved_weights_in_checkpoint:\n logger.info(\n f\"Instantiating torchvision image encoder: '{self.torchvision_model_type}' \"\n \"with weights saved in the checkpoint.\"\n )\n else:\n logger.info(\n f\"Instantiating torchvision image encoder: '{self.torchvision_model_type}' \"\n \"with no pretrained weights.\"\n )\n\n # get torchvision transforms object\n transforms_obj = torchvision_model_registry[self.torchvision_model_type][\n self.model_variant\n ].model_weights.DEFAULT.transforms()\n\n # capture key attributes from torchvision transform for later use\n self.num_channels = len(transforms_obj.mean)\n self.normalize_mean = transforms_obj.mean\n self.normalize_std = transforms_obj.std\n self.crop_size = transforms_obj.crop_size\n\n logger.debug(f\" {self.torchvision_model_type}\")\n # create pretrained model with pretrained weights or None for untrained model\n self.model = self.create_model(weights=weights_specification, **kwargs)\n\n # remove final classification layer\n self._remove_softmax_layer()\n\n # freeze parameters if requested\n for p in self.model.parameters():\n p.requires_grad_(trainable)\n\n def forward(self, inputs: torch.Tensor) -> EncoderOutputDict:\n return {ENCODER_OUTPUT: self.model(inputs)}\n\n @abstractmethod\n def _remove_softmax_layer(self):\n \"\"\"Model specific method that allows the final softmax layer to be implemented in the Ludwig Decoder\n component. The model specific implementation should change the final softmax layer in the torchvision\n model architecture to torch.nn.Identity(). This allows the output tensor from the preceding layer to be\n passed to the Ludwig Combiner and then to the Decoder.\n\n Returns: None\n \"\"\"\n raise NotImplementedError()\n\n @property\n def output_shape(self) -> torch.Size:\n # create synthetic image and run through forward method\n inputs = torch.randn([1, *self.input_shape])\n output = self.model(inputs)\n return torch.Size(output.shape[1:])\n\n @property\n def input_shape(self) -> torch.Size:\n # expected shape after all pre-processing\n # len(transforms_obj.mean) determines the number of channels\n # transforms_obj.crop_size determines the height and width of image\n # [num_channels, height, width]\n return torch.Size([self.num_channels, *(2 * self.crop_size)])\n\n\n@DeveloperAPI\n@register_torchvision_model_variants(\n [\n TVModelVariant(variant_id=\"base\", create_model_function=tvm.alexnet, model_weights=tvm.AlexNet_Weights),\n ]\n)\n@register_encoder(\"alexnet\", IMAGE)\nclass TVAlexNetEncoder(TVBaseEncoder):\n # specify base model type\n torchvision_model_type: str = \"alexnet\"\n\n def __init__(\n self,\n **kwargs,\n ):\n logger.debug(f\" {self.name}\")\n super().__init__(**kwargs)\n\n # TODO: discussion w/ justin\n # @property\n # def get_torchvision_model_type(self):\n # return \"alexnet\"\n\n def _remove_softmax_layer(self):\n self.model.classifier[-1] = torch.nn.Identity()\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TVAlexNetEncoderConfig\n\n\n@DeveloperAPI\n@register_torchvision_model_variants(\n [\n TVModelVariant(\n variant_id=\"tiny\", create_model_function=tvm.convnext_tiny, model_weights=tvm.ConvNeXt_Tiny_Weights\n ),\n TVModelVariant(\n variant_id=\"small\", create_model_function=tvm.convnext_small, model_weights=tvm.ConvNeXt_Small_Weights\n ),\n TVModelVariant(\n variant_id=\"base\", create_model_function=tvm.convnext_base, model_weights=tvm.ConvNeXt_Base_Weights\n ),\n TVModelVariant(\n variant_id=\"large\", create_model_function=tvm.convnext_large, model_weights=tvm.ConvNeXt_Large_Weights\n ),\n ]\n)\n@register_encoder(\"convnext\", IMAGE)\nclass TVConvNeXtEncoder(TVBaseEncoder):\n # specify base torchvision model\n torchvision_model_type: str = \"convnext\"\n\n def __init__(\n self,\n **kwargs,\n ):\n logger.debug(f\" {self.name}\")\n super().__init__(**kwargs)\n\n def _remove_softmax_layer(self) -> None:\n self.model.classifier[-1] = torch.nn.Identity()\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TVConvNeXtEncoderConfig\n\n\n@DeveloperAPI\n@register_torchvision_model_variants(\n [\n TVModelVariant(121, tvm.densenet121, tvm.DenseNet121_Weights),\n TVModelVariant(161, tvm.densenet161, tvm.DenseNet161_Weights),\n TVModelVariant(169, tvm.densenet169, tvm.DenseNet169_Weights),\n TVModelVariant(201, tvm.densenet201, tvm.DenseNet201_Weights),\n ]\n)\n@register_encoder(\"densenet\", IMAGE)\nclass TVDenseNetEncoder(TVBaseEncoder):\n # specify base torchvision model\n torchvision_model_type: str = \"densenet\"\n\n def __init__(\n self,\n **kwargs,\n ):\n logger.debug(f\" {self.name}\")\n super().__init__(**kwargs)\n\n def _remove_softmax_layer(self) -> None:\n self.model.classifier = torch.nn.Identity()\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TVDenseNetEncoderConfig\n\n\n@DeveloperAPI\n@register_torchvision_model_variants(\n [\n TVModelVariant(\"b0\", tvm.efficientnet_b0, tvm.EfficientNet_B0_Weights),\n TVModelVariant(\"b1\", tvm.efficientnet_b1, tvm.EfficientNet_B1_Weights),\n TVModelVariant(\"b2\", tvm.efficientnet_b2, tvm.EfficientNet_B2_Weights),\n TVModelVariant(\"b3\", tvm.efficientnet_b3, tvm.EfficientNet_B3_Weights),\n TVModelVariant(\"b4\", tvm.efficientnet_b4, tvm.EfficientNet_B4_Weights),\n TVModelVariant(\"b5\", tvm.efficientnet_b5, tvm.EfficientNet_B5_Weights),\n TVModelVariant(\"b6\", tvm.efficientnet_b6, tvm.EfficientNet_B6_Weights),\n TVModelVariant(\"b7\", tvm.efficientnet_b7, tvm.EfficientNet_B7_Weights),\n TVModelVariant(\"v2_s\", tvm.efficientnet_v2_s, tvm.EfficientNet_V2_S_Weights),\n TVModelVariant(\"v2_m\", tvm.efficientnet_v2_m, tvm.EfficientNet_V2_M_Weights),\n TVModelVariant(\"v2_l\", tvm.efficientnet_v2_l, tvm.EfficientNet_V2_L_Weights),\n ]\n)\n@register_encoder(\"efficientnet\", IMAGE)\nclass TVEfficientNetEncoder(TVBaseEncoder):\n # specify base torchvision model\n torchvision_model_type: str = \"efficientnet\"\n\n def __init__(\n self,\n **kwargs,\n ):\n logger.debug(f\" {self.name}\")\n super().__init__(**kwargs)\n\n def _remove_softmax_layer(self) -> None:\n self.model.classifier[-1] = torch.nn.Identity()\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TVEfficientNetEncoderConfig\n\n\n@DeveloperAPI\n@register_torchvision_model_variants(\n [\n TVModelVariant(\"base\", tvm.googlenet, tvm.GoogLeNet_Weights),\n ]\n)\n@register_encoder(\"googlenet\", IMAGE)\nclass TVGoogLeNetEncoder(TVBaseEncoder):\n # specify base torchvision model\n torchvision_model_type: str = \"googlenet\"\n\n def __init__(\n self,\n **kwargs,\n ):\n logger.debug(f\" {self.name}\")\n super().__init__(**kwargs)\n\n # if auxiliary network exists, eliminate auxiliary network\n # to resolve issue when loading a saved model which does not\n # contain the auxiliary network\n if self.model.aux_logits:\n self.model.aux_logits = False\n self.model.aux1 = None\n self.model.aux2 = None\n\n def _remove_softmax_layer(self) -> None:\n self.model.fc = torch.nn.Identity()\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TVGoogLeNetEncoderConfig\n\n\n@DeveloperAPI\n@register_torchvision_model_variants(\n [\n TVModelVariant(\"base\", tvm.inception_v3, tvm.Inception_V3_Weights),\n ]\n)\n@register_encoder(\"inceptionv3\", IMAGE)\nclass TVInceptionV3Encoder(TVBaseEncoder):\n # specify base torchvision model\n torchvision_model_type: str = \"inceptionv3\"\n\n def __init__(\n self,\n **kwargs,\n ):\n logger.debug(f\" {self.name}\")\n super().__init__(**kwargs)\n\n # if auxiliary network exists, eliminate auxiliary network\n # to resolve issue when loading a saved model which does not\n # contain the auxiliary network\n if self.model.aux_logits:\n self.model.aux_logits = False\n self.model.AuxLogits = None\n\n def _remove_softmax_layer(self) -> None:\n self.model.fc = torch.nn.Identity()\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TVInceptionV3EncoderConfig\n\n\n@DeveloperAPI\n@register_torchvision_model_variants(\n [\n TVModelVariant(\"t\", tvm.maxvit_t, tvm.MaxVit_T_Weights),\n ]\n)\n@register_encoder(\"maxvit\", IMAGE)\nclass TVMaxVitEncoder(TVBaseEncoder):\n # specify base torchvision model\n torchvision_model_type: str = \"maxvit\"\n\n def __init__(\n self,\n **kwargs,\n ):\n logger.debug(f\" {self.name}\")\n super().__init__(**kwargs)\n\n def _remove_softmax_layer(self) -> None:\n self.model.classifier[-1] = torch.nn.Identity()\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TVMaxVitEncoderConfig\n\n\n@DeveloperAPI\n@register_torchvision_model_variants(\n [\n TVModelVariant(\"0_5\", tvm.mnasnet0_5, tvm.mnasnet.MNASNet0_5_Weights),\n TVModelVariant(\"0_75\", tvm.mnasnet0_75, tvm.mnasnet.MNASNet0_75_Weights),\n TVModelVariant(\"1_0\", tvm.mnasnet1_0, tvm.mnasnet.MNASNet1_0_Weights),\n TVModelVariant(\"1_3\", tvm.mnasnet1_3, tvm.mnasnet.MNASNet1_3_Weights),\n ]\n)\n@register_encoder(\"mnasnet\", IMAGE)\nclass TVMNASNetEncoder(TVBaseEncoder):\n # specify base torchvision model\n torchvision_model_type: str = \"mnasnet\"\n\n def __init__(\n self,\n **kwargs,\n ):\n logger.debug(f\" {self.name}\")\n super().__init__(**kwargs)\n\n def _remove_softmax_layer(self) -> None:\n self.model.classifier[-1] = torch.nn.Identity()\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TVMNASNetEncoderConfig\n\n\n@DeveloperAPI\n@register_torchvision_model_variants(\n [\n TVModelVariant(\"base\", tvm.mobilenet_v2, tvm.MobileNet_V2_Weights),\n ]\n)\n@register_encoder(\"mobilenetv2\", IMAGE)\nclass TVMobileNetV2Encoder(TVBaseEncoder):\n # specify base torchvision model\n torchvision_model_type: str = \"mobilenetv2\"\n\n def __init__(\n self,\n **kwargs,\n ):\n logger.debug(f\" {self.name}\")\n super().__init__(**kwargs)\n\n def _remove_softmax_layer(self) -> None:\n self.model.classifier[-1] = torch.nn.Identity()\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TVMobileNetV2EncoderConfig\n\n\n@DeveloperAPI\n@register_torchvision_model_variants(\n [\n TVModelVariant(\"small\", tvm.mobilenet_v3_small, tvm.MobileNet_V3_Small_Weights),\n TVModelVariant(\"large\", tvm.mobilenet_v3_large, tvm.MobileNet_V3_Large_Weights),\n ]\n)\n@register_encoder(\"mobilenetv3\", IMAGE)\nclass TVMobileNetV3Encoder(TVBaseEncoder):\n # specify base torchvision model\n torchvision_model_type: str = \"mobilenetv3\"\n\n def __init__(\n self,\n **kwargs,\n ):\n logger.debug(f\" {self.name}\")\n super().__init__(**kwargs)\n\n def _remove_softmax_layer(self) -> None:\n self.model.classifier[-1] = torch.nn.Identity()\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TVMobileNetV3EncoderConfig\n\n\n@DeveloperAPI\n@register_torchvision_model_variants(\n [\n TVModelVariant(\"x_16gf\", tvm.regnet_x_16gf, tvm.RegNet_X_16GF_Weights),\n TVModelVariant(\"x_1_6gf\", tvm.regnet_x_1_6gf, tvm.RegNet_X_1_6GF_Weights),\n TVModelVariant(\"x_32gf\", tvm.regnet_x_32gf, tvm.RegNet_X_32GF_Weights),\n TVModelVariant(\"x_3_2gf\", tvm.regnet_x_3_2gf, tvm.RegNet_X_3_2GF_Weights),\n TVModelVariant(\"x_400mf\", tvm.regnet_x_400mf, tvm.RegNet_X_400MF_Weights),\n TVModelVariant(\"x_800mf\", tvm.regnet_x_800mf, tvm.RegNet_X_800MF_Weights),\n TVModelVariant(\"x_8gf\", tvm.regnet_x_8gf, tvm.RegNet_X_8GF_Weights),\n TVModelVariant(\"y_128gf\", tvm.regnet_y_128gf, tvm.RegNet_Y_128GF_Weights),\n TVModelVariant(\"y_16gf\", tvm.regnet_y_16gf, tvm.RegNet_Y_16GF_Weights),\n TVModelVariant(\"y_1_6gf\", tvm.regnet_y_1_6gf, tvm.RegNet_Y_1_6GF_Weights),\n TVModelVariant(\"y_32gf\", tvm.regnet_y_32gf, tvm.RegNet_Y_32GF_Weights),\n TVModelVariant(\"y_3_2gf\", tvm.regnet_y_3_2gf, tvm.RegNet_Y_3_2GF_Weights),\n TVModelVariant(\"y_400mf\", tvm.regnet_y_400mf, tvm.RegNet_Y_400MF_Weights),\n TVModelVariant(\"y_800mf\", tvm.regnet_y_800mf, tvm.RegNet_Y_800MF_Weights),\n TVModelVariant(\"y_8gf\", tvm.regnet_y_8gf, tvm.RegNet_Y_8GF_Weights),\n ]\n)\n@register_encoder(\"regnet\", IMAGE)\nclass TVRegNetEncoder(TVBaseEncoder):\n # specify base torchvision model\n torchvision_model_type: str = \"regnet\"\n\n def __init__(\n self,\n **kwargs,\n ):\n logger.debug(f\" {self.name}\")\n super().__init__(**kwargs)\n\n def _remove_softmax_layer(self) -> None:\n self.model.fc = torch.nn.Identity()\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TVRegNetEncoderConfig\n\n\n@DeveloperAPI\n@register_torchvision_model_variants(\n [\n TVModelVariant(18, tvm.resnet18, tvm.ResNet18_Weights),\n TVModelVariant(34, tvm.resnet34, tvm.ResNet34_Weights),\n TVModelVariant(50, tvm.resnet50, tvm.ResNet50_Weights),\n TVModelVariant(101, tvm.resnet101, tvm.ResNet101_Weights),\n TVModelVariant(152, tvm.resnet152, tvm.ResNet152_Weights),\n ]\n)\n@register_encoder(\"resnet\", IMAGE)\nclass TVResNetEncoder(TVBaseEncoder):\n # specify base torchvision model\n torchvision_model_type: str = \"resnet\"\n\n def __init__(\n self,\n **kwargs,\n ):\n logger.debug(f\" {self.name}\")\n super().__init__(**kwargs)\n\n def _remove_softmax_layer(self) -> None:\n self.model.fc = torch.nn.Identity()\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TVResNetEncoderConfig\n\n\n@DeveloperAPI\n@register_torchvision_model_variants(\n [\n TVModelVariant(\"50_32x4d\", tvm.resnext50_32x4d, tvm.ResNeXt50_32X4D_Weights),\n TVModelVariant(\"101_328xd\", tvm.resnext101_32x8d, tvm.ResNeXt101_32X8D_Weights),\n TVModelVariant(\"101_64x4d\", tvm.resnext101_64x4d, tvm.ResNeXt101_64X4D_Weights),\n ]\n)\n@register_encoder(\"resnext\", IMAGE)\nclass TVResNeXtEncoder(TVBaseEncoder):\n # specify base torchvision model\n torchvision_model_type: str = \"resnext\"\n\n def __init__(\n self,\n **kwargs,\n ):\n logger.debug(f\" {self.name}\")\n super().__init__(**kwargs)\n\n def _remove_softmax_layer(self) -> None:\n self.model.fc = torch.nn.Identity()\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TVResNeXtEncoderConfig\n\n\n@DeveloperAPI\n@register_torchvision_model_variants(\n [\n TVModelVariant(\"x0_5\", tvm.shufflenet_v2_x0_5, tvm.ShuffleNet_V2_X0_5_Weights),\n TVModelVariant(\"x1_0\", tvm.shufflenet_v2_x1_0, tvm.ShuffleNet_V2_X1_0_Weights),\n TVModelVariant(\"x1_5\", tvm.shufflenet_v2_x1_5, tvm.ShuffleNet_V2_X1_5_Weights),\n TVModelVariant(\"x2_0\", tvm.shufflenet_v2_x2_0, tvm.ShuffleNet_V2_X2_0_Weights),\n ]\n)\n@register_encoder(\"shufflenet_v2\", IMAGE)\nclass TVShuffleNetV2Encoder(TVBaseEncoder):\n # specify base torchvision model\n torchvision_model_type: str = \"shufflenet_v2\"\n\n def __init__(\n self,\n **kwargs,\n ):\n logger.debug(f\" {self.name}\")\n super().__init__(**kwargs)\n\n def _remove_softmax_layer(self) -> None:\n self.model.fc = torch.nn.Identity()\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TVShuffleNetV2EncoderConfig\n\n\n@DeveloperAPI\n@register_torchvision_model_variants(\n [\n TVModelVariant(\"1_0\", tvm.squeezenet1_0, tvm.SqueezeNet1_0_Weights),\n TVModelVariant(\"1_1\", tvm.squeezenet1_1, tvm.SqueezeNet1_1_Weights),\n ]\n)\n@register_encoder(\"squeezenet\", IMAGE)\nclass TVSqueezeNetEncoder(TVBaseEncoder):\n # specify base torchvision model\n torchvision_model_type: str = \"squeezenet\"\n\n def __init__(\n self,\n **kwargs,\n ):\n logger.debug(f\" {self.name}\")\n super().__init__(**kwargs)\n\n def _remove_softmax_layer(self) -> None:\n # SqueezeNet does not have a final nn.Linear() layer\n # Use flatten output from last AdaptiveAvgPool2d layer\n # as encoder output.\n pass\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TVSqueezeNetEncoderConfig\n\n\n@DeveloperAPI\n@register_torchvision_model_variants(\n [\n TVModelVariant(\"t\", tvm.swin_t, tvm.Swin_T_Weights),\n TVModelVariant(\"s\", tvm.swin_s, tvm.Swin_S_Weights),\n TVModelVariant(\"b\", tvm.swin_b, tvm.Swin_B_Weights),\n ]\n)\n@register_encoder(\"swin_transformer\", IMAGE)\nclass TVSwinTransformerEncoder(TVBaseEncoder):\n # specify base torchvision model\n torchvision_model_type: str = \"swin_transformer\"\n\n def __init__(\n self,\n **kwargs,\n ):\n logger.debug(f\" {self.name}\")\n super().__init__(**kwargs)\n\n def _remove_softmax_layer(self) -> None:\n self.model.head = torch.nn.Identity()\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TVSwinTransformerEncoderConfig\n\n\n@DeveloperAPI\n@register_torchvision_model_variants(\n [\n TVModelVariant(11, tvm.vgg11, tvm.VGG11_Weights),\n TVModelVariant(\"11_bn\", tvm.vgg11_bn, tvm.VGG11_BN_Weights),\n TVModelVariant(13, tvm.vgg13, tvm.VGG13_Weights),\n TVModelVariant(\"13_bn\", tvm.vgg13_bn, tvm.VGG13_BN_Weights),\n TVModelVariant(16, tvm.vgg16, tvm.VGG16_Weights),\n TVModelVariant(\"16_bn\", tvm.vgg16_bn, tvm.VGG16_BN_Weights),\n TVModelVariant(19, tvm.vgg19, tvm.VGG19_Weights),\n TVModelVariant(\"19_bn\", tvm.vgg19_bn, tvm.VGG19_BN_Weights),\n ]\n)\n@register_encoder(\"vgg\", IMAGE)\nclass TVVGGEncoder(TVBaseEncoder):\n # specify base torchvison model\n torchvision_model_type: str = \"vgg\"\n\n def __init__(\n self,\n **kwargs,\n ):\n logger.debug(f\" {self.name}\")\n super().__init__(**kwargs)\n\n def _remove_softmax_layer(self) -> None:\n self.model.classifier[-1] = torch.nn.Identity()\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TVVGGEncoderConfig\n\n\n@DeveloperAPI\n@register_torchvision_model_variants(\n [\n TVModelVariant(\"b_16\", tvm.vit_b_16, tvm.ViT_B_16_Weights),\n TVModelVariant(\"b_32\", tvm.vit_b_32, tvm.ViT_B_32_Weights),\n TVModelVariant(\"l_16\", tvm.vit_l_16, tvm.ViT_L_16_Weights),\n TVModelVariant(\"l_32\", tvm.vit_l_32, tvm.ViT_L_32_Weights),\n TVModelVariant(\"h_14\", tvm.vit_h_14, tvm.ViT_H_14_Weights),\n ]\n)\n@register_encoder(\"vit\", IMAGE)\nclass TVViTEncoder(TVBaseEncoder):\n # specify base torchvision model\n torchvision_model_type: str = \"vit\"\n\n def __init__(\n self,\n **kwargs,\n ):\n logger.debug(f\" {self.name}\")\n\n # Depending on model variant and weight specification, the expected image size\n # will vary. This code determines at run time what the expected image size will be\n # and adds to the kwargs dictionary the parameter that specifies the image size.\n # this is needed only if not using pretrained weights. If pre-trained weights are\n # specified, then the correct image size is set.\n if not kwargs[\"use_pretrained\"]:\n weights_specification = torchvision_model_registry[self.torchvision_model_type][\n kwargs[\"model_variant\"]\n ].model_weights.DEFAULT\n kwargs[\"image_size\"] = weights_specification.transforms.keywords[\"crop_size\"]\n\n super().__init__(**kwargs)\n\n def _remove_softmax_layer(self) -> None:\n self.model.heads[-1] = torch.nn.Identity()\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TVViTEncoderConfig\n\n\n@DeveloperAPI\n@register_torchvision_model_variants(\n [\n TVModelVariant(\"50_2\", tvm.wide_resnet50_2, tvm.Wide_ResNet50_2_Weights),\n TVModelVariant(\"101_2\", tvm.wide_resnet101_2, tvm.Wide_ResNet101_2_Weights),\n ]\n)\n@register_encoder(\"wide_resnet\", IMAGE)\nclass TVWideResNetEncoder(TVBaseEncoder):\n # specify base torchvision model\n torchvision_model_type: str = \"wide_resnet\"\n\n def __init__(\n self,\n **kwargs,\n ):\n logger.debug(f\" {self.name}\")\n super().__init__(**kwargs)\n\n def _remove_softmax_layer(self) -> None:\n self.model.fc = torch.nn.Identity()\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TVWideResNetEncoderConfig\n"
},
{
"path": "ludwig/encoders/registry.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.encoders.base import Encoder\nfrom ludwig.utils.registry import Registry\n\n_encoder_registry = Registry()\n_sequence_encoder_registry = Registry()\n\n\n@DeveloperAPI\ndef get_encoder_registry() -> Registry:\n return _encoder_registry\n\n\n@DeveloperAPI\ndef get_sequence_encoder_registry() -> Registry:\n return _sequence_encoder_registry\n\n\ndef register_sequence_encoder(name: str):\n def wrap(cls):\n get_sequence_encoder_registry()[name] = cls\n return cls\n\n return wrap\n\n\ndef register_encoder(name: str, features: str | list[str]):\n if isinstance(features, str):\n features = [features]\n\n def update_registry(registry_getter_fn, cls, feature):\n feature_registry = registry_getter_fn().get(feature, {})\n feature_registry[name] = cls\n registry_getter_fn()[feature] = feature_registry\n\n def wrap(cls):\n for feature in features:\n update_registry(get_encoder_registry, cls, feature)\n return cls\n\n return wrap\n\n\ndef get_encoder_cls(feature: str, name: str) -> type[Encoder]:\n return get_encoder_registry()[feature][name]\n\n\ndef get_encoder_classes(feature: str) -> dict[str, type[Encoder]]:\n return get_encoder_registry()[feature]\n"
},
{
"path": "ludwig/encoders/sequence_encoders.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\n\nimport torch\nfrom torch import nn\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import AUDIO, ENCODER_OUTPUT, ENCODER_OUTPUT_STATE, SEQUENCE, TEXT, TIMESERIES\nfrom ludwig.encoders.base import Encoder\nfrom ludwig.encoders.registry import register_encoder, register_sequence_encoder\nfrom ludwig.encoders.types import EncoderOutputDict\nfrom ludwig.modules.attention_modules import TransformerStack\nfrom ludwig.modules.convolutional_modules import Conv1DStack, ParallelConv1D, ParallelConv1DStack\nfrom ludwig.modules.embedding_modules import EmbedSequence, TokenAndPositionEmbedding\nfrom ludwig.modules.fully_connected_modules import FCStack\nfrom ludwig.modules.recurrent_modules import RecurrentStack\nfrom ludwig.modules.reduction_modules import SequenceReducer\nfrom ludwig.schema.encoders.sequence_encoders import (\n ParallelCNNConfig,\n SequenceEmbedConfig,\n SequenceEncoderConfig,\n SequencePassthroughConfig,\n StackedCNNConfig,\n StackedCNNRNNConfig,\n StackedParallelCNNConfig,\n StackedRNNConfig,\n StackedTransformerConfig,\n)\n\nlogger = logging.getLogger(__name__)\n\n\nclass SequenceEncoder(Encoder):\n pass\n\n\n@DeveloperAPI\n@register_encoder(\"passthrough\", [SEQUENCE, TEXT, TIMESERIES])\nclass SequencePassthroughEncoder(SequenceEncoder):\n def __init__(\n self,\n reduce_output: str = None,\n max_sequence_length: int = 256,\n encoding_size: int = None,\n encoder_config=None,\n **kwargs,\n ):\n \"\"\"\n :param reduce_output: defines how to reduce the output tensor along\n the `s` sequence length dimension if the rank of the tensor\n is greater than 2. Available values are: `sum`,\n `mean` or `avg`, `max`, `concat` (concatenates along\n the first dimension), `last` (returns the last vector of the\n first dimension) and `None` or `null` (which does not reduce\n and returns the full tensor).\n :param max_sequence_length: The maximum sequence length.\n :param encoding_size: The size of the encoding vector, or None if sequence elements are scalars.\n \"\"\"\n super().__init__()\n self.config = encoder_config\n self.max_sequence_length = max_sequence_length\n\n logger.debug(f\" {self.name}\")\n\n self.reduce_output = reduce_output\n self.reduce_sequence = SequenceReducer(\n reduce_mode=reduce_output, max_sequence_length=max_sequence_length, encoding_size=encoding_size\n )\n if self.reduce_output is None:\n self.supports_masking = True\n\n def forward(self, input_sequence: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n \"\"\"\n :param input_sequence: The input sequence fed into the encoder.\n Shape: [batch x sequence length], type torch.int32 or\n [batch x sequence length x encoding size], type torch.float32\n :type input_sequence: Tensor\n :param mask: Sequence mask (not yet implemented).\n Shape: [batch x sequence length]\n :type mask: Tensor\n \"\"\"\n input_sequence = input_sequence.type(torch.float32)\n while len(input_sequence.shape) < 3:\n input_sequence = input_sequence.unsqueeze(-1)\n hidden = self.reduce_sequence(input_sequence)\n\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[SequenceEncoderConfig]:\n return SequencePassthroughConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n return self.input_shape\n\n\n@DeveloperAPI\n@register_encoder(\"embed\", [SEQUENCE, TEXT])\nclass SequenceEmbedEncoder(SequenceEncoder):\n def __init__(\n self,\n vocab,\n max_sequence_length,\n representation=\"dense\",\n embedding_size=256,\n embeddings_trainable=True,\n pretrained_embeddings=None,\n embeddings_on_cpu=False,\n weights_initializer=None,\n dropout=0,\n reduce_output=\"sum\",\n encoder_config=None,\n **kwargs,\n ):\n \"\"\"\n :param vocab: Vocabulary of the input feature to encode\n :type vocab: List\n :param max_sequence_length: The maximum sequence length.\n :type max_sequence_length: int\n :param representation: the possible values are `dense` and `sparse`.\n `dense` means the embeddings are initialized randomly,\n `sparse` means they are initialized to be one-hot encodings.\n :type representation: str (one of 'dense' or 'sparse')\n :param embedding_size: it is the maximum embedding size, the actual\n size will be `min(vocabulary_size, embedding_size)`\n for `dense` representations and exactly `vocabulary_size`\n for the `sparse` encoding, where `vocabulary_size` is\n the number of different strings appearing in the training set\n in the column the feature is named after (plus 1 for ``).\n :type embedding_size: Integer\n :param embeddings_trainable: If `True` embeddings are trained during\n the training process, if `False` embeddings are fixed.\n It may be useful when loading pretrained embeddings\n for avoiding finetuning them. This parameter has effect only\n for `representation` is `dense` as `sparse` one-hot encodings\n are not trainable.\n :type embeddings_trainable: Boolean\n :param pretrained_embeddings: by default `dense` embeddings\n are initialized randomly, but this parameter allows to specify\n a path to a file containing embeddings in the GloVe format.\n When the file containing the embeddings is loaded, only the\n embeddings with labels present in the vocabulary are kept,\n the others are discarded. If the vocabulary contains strings\n that have no match in the embeddings file, their embeddings\n are initialized with the average of all other embedding plus\n some random noise to make them different from each other.\n This parameter has effect only if `representation` is `dense`.\n :type pretrained_embeddings: str (filepath)\n :param embeddings_on_cpu: by default embeddings matrices are stored\n on GPU memory if a GPU is used, as it allows\n for faster access, but in some cases the embedding matrix\n may be really big and this parameter forces the placement\n of the embedding matrix in regular memory and the CPU is used\n to resolve them, slightly slowing down the process\n as a result of data transfer between CPU and GPU memory.\n :type embeddings_on_cpu: Boolean\n :param weights_initializer: the initializer to use. If `None`, the default\n initialized of each variable is used (`xavier_uniform`\n in most cases). Options are: `constant`, `identity`, `zeros`,\n `ones`, `orthogonal`, `normal`, `uniform`,\n `truncated_normal`, `variance_scaling`, `xavier_normal`,\n `xavier_uniform`, `xavier_normal`,\n `he_normal`, `he_uniform`, `lecun_normal`, `lecun_uniform`.\n Alternatively it is possible to specify a dictionary with\n a key `type` that identifies the type of initializer and\n other keys for its parameters, e.g.\n `{type: normal, mean: 0, stddev: 0}`.\n To know the parameters of each initializer, please refer to\n PyTorch's documentation.\n :type weights_initializer: str\n :param dropout: Tensor (torch.float) The dropout probability.\n :type dropout: Tensor\n :param reduce_output: defines how to reduce the output tensor along\n the `s` sequence length dimension if the rank of the tensor\n is greater than 2. Available values are: `sum`,\n `mean` or `avg`, `max`, `concat` (concatenates along\n the first dimension), `last` (returns the last vector of the\n first dimension) and `None` or `null` (which does not reduce\n and returns the full tensor).\n :type reduce_output: str\n \"\"\"\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n self.embedding_size = embedding_size\n self.max_sequence_length = max_sequence_length\n\n self.reduce_output = reduce_output\n if self.reduce_output is None:\n self.supports_masking = True\n\n logger.debug(\" EmbedSequence\")\n self.embed_sequence = EmbedSequence(\n vocab,\n embedding_size,\n max_sequence_length=max_sequence_length,\n representation=representation,\n embeddings_trainable=embeddings_trainable,\n pretrained_embeddings=pretrained_embeddings,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=weights_initializer,\n )\n\n self.reduce_sequence = SequenceReducer(\n reduce_mode=reduce_output,\n max_sequence_length=max_sequence_length,\n encoding_size=self.embed_sequence.output_shape[-1],\n )\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n \"\"\"\n :param inputs: The input sequence fed into the encoder.\n Shape: [batch x sequence length], type torch.int32\n :param mask: Input mask (unused, not yet implemented in EmbedSequence)\n \"\"\"\n embedded_sequence = self.embed_sequence(inputs, mask=mask)\n hidden = self.reduce_sequence(embedded_sequence)\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[SequenceEncoderConfig]:\n return SequenceEmbedConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n return self.reduce_sequence.output_shape\n\n\n@DeveloperAPI\n@register_sequence_encoder(\"parallel_cnn\")\n@register_encoder(\"parallel_cnn\", [AUDIO, SEQUENCE, TEXT, TIMESERIES])\nclass ParallelCNN(SequenceEncoder):\n def __init__(\n self,\n should_embed=True,\n vocab=None,\n representation=\"dense\",\n embedding_size=256,\n max_sequence_length=None,\n embeddings_trainable=True,\n pretrained_embeddings=None,\n embeddings_on_cpu=False,\n conv_layers=None,\n num_conv_layers=None,\n filter_size=3,\n num_filters=256,\n pool_function=\"max\",\n pool_size=None,\n fc_layers=None,\n num_fc_layers=None,\n output_size=256,\n use_bias=True,\n weights_initializer=\"xavier_uniform\",\n bias_initializer=\"zeros\",\n norm=None,\n norm_params=None,\n activation=\"relu\",\n dropout=0,\n reduce_output=\"max\",\n encoder_config=None,\n **kwargs,\n ):\n # todo: revise docstring\n \"\"\"\n :param should_embed: If True the input sequence is expected\n to be made of integers and will be mapped into embeddings\n :type should_embed: Boolean\n :param vocab: Vocabulary of the input feature to encode\n :type vocab: List\n :param representation: the possible values are `dense` and `sparse`.\n `dense` means the embeddings are initialized randomly,\n `sparse` means they are initialized to be one-hot encodings.\n :type representation: Str (one of 'dense' or 'sparse')\n :param embedding_size: it is the maximum embedding size, the actual\n size will be `min(vocabulary_size, embedding_size)`\n for `dense` representations and exactly `vocabulary_size`\n for the `sparse` encoding, where `vocabulary_size` is\n the number of different strings appearing in the training set\n in the column the feature is named after (plus 1 for ``).\n :type embedding_size: Integer\n :param embeddings_trainable: If `True` embeddings are trained during\n the training process, if `False` embeddings are fixed.\n It may be useful when loading pretrained embeddings\n for avoiding finetuning them. This parameter has effect only\n for `representation` is `dense` as `sparse` one-hot encodings\n are not trainable.\n :type embeddings_trainable: Boolean\n :param pretrained_embeddings: by default `dense` embeddings\n are initialized randomly, but this parameter allows to specify\n a path to a file containing embeddings in the GloVe format.\n When the file containing the embeddings is loaded, only the\n embeddings with labels present in the vocabulary are kept,\n the others are discarded. If the vocabulary contains strings\n that have no match in the embeddings file, their embeddings\n are initialized with the average of all other embedding plus\n some random noise to make them different from each other.\n This parameter has effect only if `representation` is `dense`.\n :type pretrained_embeddings: str (filepath)\n :param embeddings_on_cpu: by default embeddings matrices are stored\n on GPU memory if a GPU is used, as it allows\n for faster access, but in some cases the embedding matrix\n may be really big and this parameter forces the placement\n of the embedding matrix in regular memroy and the CPU is used\n to resolve them, slightly slowing down the process\n as a result of data transfer between CPU and GPU memory.\n :param conv_layers: it is a list of dictionaries containing\n the parameters of all the convolutional layers. The length\n of the list determines the number of parallel convolutional\n layers and the content of each dictionary determines\n the parameters for a specific layer. The available parameters\n for each layer are: `filter_size`, `num_filters`, `pool`,\n `norm`, and `activation`. If any of those values\n is missing from the dictionary, the default one specified\n as a parameter of the encoder will be used instead. If both\n `conv_layers` and `num_conv_layers` are `None`, a default\n list will be assigned to `conv_layers` with the value\n `[{filter_size: 2}, {filter_size: 3}, {filter_size: 4},\n {filter_size: 5}]`.\n :type conv_layers: List\n :param num_conv_layers: if `conv_layers` is `None`, this is\n the number of parallel convolutional layers.\n :type num_conv_layers: Integer\n :param filter_size: if a `filter_size` is not already specified in\n `conv_layers` this is the default `filter_size` that\n will be used for each layer. It indicates how wide is\n the 1d convolutional filter.\n :type filter_size: Integer\n :param num_filters: if a `num_filters` is not already specified in\n `conv_layers` this is the default `num_filters` that\n will be used for each layer. It indicates the number\n of filters, and by consequence the output channels of\n the 1d convolution.\n :type num_filters: Integer\n :param pool_size: if a `pool_size` is not already specified\n in `conv_layers` this is the default `pool_size` that\n will be used for each layer. It indicates the size of\n the max pooling that will be performed along the `s` sequence\n dimension after the convolution operation.\n :type pool_size: Integer\n :param fc_layers: it is a list of dictionaries containing\n the parameters of all the fully connected layers. The length\n of the list determines the number of stacked fully connected\n layers and the content of each dictionary determines\n the parameters for a specific layer. The available parameters\n for each layer are: `output_size`, `norm` and `activation`.\n If any of those values is missing from\n the dictionary, the default one specified as a parameter of\n the encoder will be used instead. If both `fc_layers` and\n `num_fc_layers` are `None`, a default list will be assigned\n to `fc_layers` with the value\n `[{output_size: 512}, {output_size: 256}]`\n (only applies if `reduce_output` is not `None`).\n :type fc_layers: List\n :param num_fc_layers: if `fc_layers` is `None`, this is the number\n of stacked fully connected layers (only applies if\n `reduce_output` is not `None`).\n :type num_fc_layers: Integer\n :param output_size: if a `output_size` is not already specified in\n `fc_layers` this is the default `output_size` that will be used\n for each layer. It indicates the size of the output\n of a fully connected layer.\n :type output_size: Integer\n :param norm: if a `norm` is not already specified in `conv_layers`\n or `fc_layers` this is the default `norm` that will be used\n for each layer. It indicates the norm of the output.\n :type norm: str\n :param activation: Default activation function to use\n :type activation: Str\n :param dropout: determines if there should be a dropout layer before\n returning the encoder output.\n :type dropout: Boolean\n :param initializer: the initializer to use. If `None` it uses\n `xavier_uniform`. Options are: `constant`, `identity`,\n `zeros`, `ones`, `orthogonal`, `normal`, `uniform`,\n `truncated_normal`, `variance_scaling`, `xavier_normal`,\n `xavier_uniform`, `xavier_normal`,\n `he_normal`, `he_uniform`, `lecun_normal`, `lecun_uniform`.\n Alternatively it is possible to specify a dictionary with\n a key `type` that identifies the type of initializer and\n other keys for its parameters,\n e.g. `{type: normal, mean: 0, stddev: 0}`.\n To know the parameters of each initializer, please refer\n to PyTorch's documentation.\n :type initializer: str\n :param reduce_output: defines how to reduce the output tensor of\n the convolutional layers along the `s` sequence length\n dimension if the rank of the tensor is greater than 2.\n Available values are: `sum`, `mean` or `avg`, `max`, `concat`\n (concatenates along the first dimension), `last` (returns\n the last vector of the first dimension) and `None` or `null`\n (which does not reduce and returns the full tensor).\n :type reduce_output: str\n \"\"\"\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n\n self.max_sequence_length = max_sequence_length\n\n if conv_layers is not None and num_conv_layers is None:\n # use custom-defined layers\n self.conv_layers = conv_layers\n self.num_conv_layers = len(conv_layers)\n elif conv_layers is None and num_conv_layers is not None:\n # generate num_conv_layers with default parameters\n self.conv_layers = None\n self.num_conv_layers = num_conv_layers\n elif conv_layers is None and num_conv_layers is None:\n # use default layers with varying filter sizes\n self.conv_layers = [{\"filter_size\": 2}, {\"filter_size\": 3}, {\"filter_size\": 4}, {\"filter_size\": 5}]\n self.num_conv_layers = 4\n else:\n raise ValueError(\"Invalid layer parametrization, use either conv_layers or num_conv_layers\")\n\n # The user is expected to provide fc_layers or num_fc_layers\n # The following logic handles the case where the user either provides\n # both or neither.\n if fc_layers is None and num_fc_layers is None:\n # use default layers with varying filter sizes\n fc_layers = [{\"output_size\": 512}, {\"output_size\": 256}]\n num_fc_layers = 2\n elif fc_layers is not None and num_fc_layers is not None:\n raise ValueError(\"Invalid layer parametrization, use either fc_layers or num_fc_layers only. Not both.\")\n\n self.should_embed = should_embed\n self.embed_sequence = None\n\n if self.should_embed:\n logger.debug(\" EmbedSequence\")\n self.embed_sequence = EmbedSequence(\n vocab,\n embedding_size,\n max_sequence_length=max_sequence_length,\n representation=representation,\n embeddings_trainable=embeddings_trainable,\n pretrained_embeddings=pretrained_embeddings,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=weights_initializer,\n )\n\n logger.debug(\" ParallelConv1D\")\n in_channels = self.embed_sequence.output_shape[-1] if self.should_embed else embedding_size\n self.parallel_conv1d = ParallelConv1D(\n in_channels=in_channels,\n max_sequence_length=self.max_sequence_length,\n layers=self.conv_layers,\n default_num_filters=num_filters,\n default_filter_size=filter_size,\n default_use_bias=use_bias,\n default_weights_initializer=weights_initializer,\n default_bias_initializer=bias_initializer,\n default_norm=norm,\n default_norm_params=norm_params,\n default_activation=activation,\n default_dropout=dropout,\n default_pool_function=pool_function,\n default_pool_size=pool_size,\n default_pool_padding=\"same\",\n )\n\n self.reduce_output = reduce_output\n self.reduce_sequence = SequenceReducer(\n reduce_mode=reduce_output,\n max_sequence_length=max_sequence_length,\n encoding_size=self.parallel_conv1d.output_shape[-1],\n )\n if self.reduce_output is not None:\n logger.debug(\" FCStack\")\n self.fc_stack = FCStack(\n self.reduce_sequence.output_shape[-1],\n layers=fc_layers,\n num_layers=num_fc_layers,\n default_output_size=output_size,\n default_use_bias=use_bias,\n default_weights_initializer=weights_initializer,\n default_bias_initializer=bias_initializer,\n default_norm=norm,\n default_norm_params=norm_params,\n default_activation=activation,\n default_dropout=dropout,\n )\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n \"\"\"\n :param inputs: The input sequence fed into the encoder.\n Shape: [batch x sequence length], type torch.int32\n :param mask: Input mask (unused, not yet implemented)\n \"\"\"\n # ================ Embeddings ================\n if self.should_embed:\n embedded_sequence = self.embed_sequence(inputs, mask=mask)\n else:\n embedded_sequence = inputs\n while len(embedded_sequence.shape) < 3:\n embedded_sequence = embedded_sequence.unsqueeze(-1)\n embedded_sequence = embedded_sequence.to(dtype=torch.float)\n\n # shape=(?, sequence_length, embedding_size)\n hidden = embedded_sequence\n\n # ================ Conv Layers ================\n hidden = self.parallel_conv1d(hidden, mask=mask)\n\n # ================ Sequence Reduction ================\n if self.reduce_output is not None:\n hidden = self.reduce_sequence(hidden)\n\n # ================ FC Layers ================\n hidden = self.fc_stack(hidden, mask=mask)\n\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[SequenceEncoderConfig]:\n return ParallelCNNConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is not None:\n return self.fc_stack.output_shape\n return self.parallel_conv1d.output_shape\n\n\n@DeveloperAPI\n@register_sequence_encoder(\"stacked_cnn\")\n@register_encoder(\"stacked_cnn\", [AUDIO, SEQUENCE, TEXT, TIMESERIES])\nclass StackedCNN(SequenceEncoder):\n def __init__(\n self,\n should_embed=True,\n vocab=None,\n representation=\"dense\",\n embedding_size=256,\n max_sequence_length=None,\n embeddings_trainable=True,\n pretrained_embeddings=None,\n embeddings_on_cpu=False,\n conv_layers=None,\n num_conv_layers=None,\n num_filters=256,\n filter_size=5,\n strides=1,\n # todo: assess how to specify padding for equivalent to 'same'\n padding=\"same\",\n dilation_rate=1,\n pool_function=\"max\",\n pool_size=None,\n pool_strides=None,\n # todo: determine how to pool_padding equivalent of 'same'\n pool_padding=\"same\",\n fc_layers=None,\n num_fc_layers=None,\n output_size=256,\n use_bias=True,\n weights_initializer=\"xavier_uniform\",\n bias_initializer=\"zeros\",\n norm=None,\n norm_params=None,\n activation=\"relu\",\n dropout=0,\n reduce_output=\"max\",\n encoder_config=None,\n **kwargs,\n ):\n # todo: fixup docstring\n \"\"\"\n :param should_embed: If True the input sequence is expected\n to be made of integers and will be mapped into embeddings\n :type should_embed: Boolean\n :param vocab: Vocabulary of the input feature to encode\n :type vocab: List\n :param representation: the possible values are `dense` and `sparse`.\n `dense` means the embeddings are initialized randomly,\n `sparse` means they are initialized to be one-hot encodings.\n :type representation: Str (one of 'dense' or 'sparse')\n :param embedding_size: it is the maximum embedding size, the actual\n size will be `min(vocabulary_size, embedding_size)`\n for `dense` representations and exactly `vocabulary_size`\n for the `sparse` encoding, where `vocabulary_size` is\n the number of different strings appearing in the training set\n in the column the feature is named after (plus 1 for ``).\n :type embedding_size: Integer\n :param embeddings_trainable: If `True` embeddings are trained during\n the training process, if `False` embeddings are fixed.\n It may be useful when loading pretrained embeddings\n for avoiding finetuning them. This parameter has effect only\n for `representation` is `dense` as `sparse` one-hot encodings\n are not trainable.\n :type embeddings_trainable: Boolean\n :param pretrained_embeddings: by default `dense` embeddings\n are initialized randomly, but this parameter allows to specify\n a path to a file containing embeddings in the GloVe format.\n When the file containing the embeddings is loaded, only the\n embeddings with labels present in the vocabulary are kept,\n the others are discarded. If the vocabulary contains strings\n that have no match in the embeddings file, their embeddings\n are initialized with the average of all other embedding plus\n some random noise to make them different from each other.\n This parameter has effect only if `representation` is `dense`.\n :type pretrained_embeddings: str (filepath)\n :param embeddings_on_cpu: by default embeddings matrices are stored\n on GPU memory if a GPU is used, as it allows\n for faster access, but in some cases the embedding matrix\n may be really big and this parameter forces the placement\n of the embedding matrix in regular memroy and the CPU is used\n to resolve them, slightly slowing down the process\n as a result of data transfer between CPU and GPU memory.\n :param conv_layers: it is a list of dictionaries containing\n the parameters of all the convolutional layers. The length\n of the list determines the number of parallel convolutional\n layers and the content of each dictionary determines\n the parameters for a specific layer. The available parameters\n for each layer are: `filter_size`, `num_filters`, `pool`,\n `norm` and `activation`. If any of those values\n is missing from the dictionary, the default one specified\n as a parameter of the encoder will be used instead. If both\n `conv_layers` and `num_conv_layers` are `None`, a default\n list will be assigned to `conv_layers` with the value\n `[{filter_size: 2}, {filter_size: 3}, {filter_size: 4},\n {filter_size: 5}]`.\n :type conv_layers: List\n :param num_conv_layers: if `conv_layers` is `None`, this is\n the number of stacked convolutional layers.\n :type num_conv_layers: Integer\n :param filter_size: if a `filter_size` is not already specified in\n `conv_layers` this is the default `filter_size` that\n will be used for each layer. It indicates how wide is\n the 1d convolutional filter.\n :type filter_size: Integer\n :param num_filters: if a `num_filters` is not already specified in\n `conv_layers` this is the default `num_filters` that\n will be used for each layer. It indicates the number\n of filters, and by consequence the output channels of\n the 1d convolution.\n :type num_filters: Integer\n :param pool_size: if a `pool_size` is not already specified\n in `conv_layers` this is the default `pool_size` that\n will be used for each layer. It indicates the size of\n the max pooling that will be performed along the `s` sequence\n dimension after the convolution operation.\n :type pool_size: Integer\n :param fc_layers: it is a list of dictionaries containing\n the parameters of all the fully connected layers. The length\n of the list determines the number of stacked fully connected\n layers and the content of each dictionary determines\n the parameters for a specific layer. The available parameters\n for each layer are: `output_size`, `norm` and `activation`.\n If any of those values is missing from\n the dictionary, the default one specified as a parameter of\n the encoder will be used instead. If both `fc_layers` and\n `num_fc_layers` are `None`, a default list will be assigned\n to `fc_layers` with the value\n `[{output_size: 512}, {output_size: 256}]`\n (only applies if `reduce_output` is not `None`).\n :type fc_layers: List\n :param num_fc_layers: if `fc_layers` is `None`, this is the number\n of stacked fully connected layers (only applies if\n `reduce_output` is not `None`).\n :type num_fc_layers: Integer\n :param output_size: if a `output_size` is not already specified in\n `fc_layers` this is the default `output_size` that will be used\n for each layer. It indicates the size of the output\n of a fully connected layer.\n :type output_size: Integer\n :param norm: if a `norm` is not already specified in `conv_layers`\n or `fc_layers` this is the default `norm` that will be used\n for each layer. It indicates the norm of the output.\n :type norm: str\n :param activation: Default activation function to use\n :type activation: Str\n :param dropout: determines if there should be a dropout layer before\n returning the encoder output.\n :type dropout: Boolean\n :param initializer: the initializer to use. If `None` it uses\n `xavier_uniform`. Options are: `constant`, `identity`,\n `zeros`, `ones`, `orthogonal`, `normal`, `uniform`,\n `truncated_normal`, `variance_scaling`, `xavier_normal`,\n `xavier_uniform`, `xavier_normal`,\n `he_normal`, `he_uniform`, `lecun_normal`, `lecun_uniform`.\n Alternatively it is possible to specify a dictionary with\n a key `type` that identifies the type of initializer and\n other keys for its parameters,\n e.g. `{type: normal, mean: 0, stddev: 0}`.\n To know the parameters of each initializer, please refer\n to PyTorch's documentation.\n :type initializer: str\n :param reduce_output: defines how to reduce the output tensor of\n the convolutional layers along the `s` sequence length\n dimension if the rank of the tensor is greater than 2.\n Available values are: `sum`, `mean` or `avg`, `max`, `concat`\n (concatenates along the first dimension), `last` (returns\n the last vector of the first dimension) and `None` or `null`\n (which does not reduce and returns the full tensor).\n :type reduce_output: str\n \"\"\"\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n\n if conv_layers is not None and num_conv_layers is None:\n # use custom-defined layers\n self.conv_layers = conv_layers\n self.num_conv_layers = len(conv_layers)\n elif conv_layers is None and num_conv_layers is not None:\n # generate num_conv_layers with default parameters\n self.conv_layers = None\n self.num_conv_layers = num_conv_layers\n elif conv_layers is None and num_conv_layers is None:\n # use default layers with varying filter sizes\n self.conv_layers = [\n {\n \"filter_size\": 7,\n \"pool_size\": 3,\n },\n {\n \"filter_size\": 7,\n \"pool_size\": 3,\n },\n {\n \"filter_size\": 3,\n \"pool_size\": None,\n },\n {\n \"filter_size\": 3,\n \"pool_size\": None,\n },\n {\n \"filter_size\": 3,\n \"pool_size\": None,\n },\n {\n \"filter_size\": 3,\n \"pool_size\": 3,\n },\n ]\n self.num_conv_layers = 6\n else:\n raise ValueError(\"Invalid layer parametrization, use either conv_layers or \" \"num_conv_layers\")\n\n # The user is expected to provide fc_layers or num_fc_layers\n # The following logic handles the case where the user either provides\n # both or neither.\n if fc_layers is None and num_fc_layers is None:\n # use default layers with varying filter sizes\n fc_layers = [{\"output_size\": 512}, {\"output_size\": 256}]\n num_fc_layers = 2\n elif fc_layers is not None and num_fc_layers is not None:\n raise ValueError(\"Invalid layer parametrization, use either fc_layers or \" \"num_fc_layers only. Not both.\")\n\n self.max_sequence_length = max_sequence_length\n self.num_filters = num_filters\n self.should_embed = should_embed\n self.embed_sequence = None\n\n if self.should_embed:\n logger.debug(\" EmbedSequence\")\n self.embed_sequence = EmbedSequence(\n vocab,\n embedding_size,\n max_sequence_length=self.max_sequence_length,\n representation=representation,\n embeddings_trainable=embeddings_trainable,\n pretrained_embeddings=pretrained_embeddings,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=weights_initializer,\n )\n\n logger.debug(\" Conv1DStack\")\n in_channels = self.embed_sequence.output_shape[-1] if self.should_embed else embedding_size\n self.conv1d_stack = Conv1DStack(\n in_channels=in_channels,\n max_sequence_length=max_sequence_length,\n layers=self.conv_layers,\n default_num_filters=num_filters,\n default_filter_size=filter_size,\n default_strides=strides,\n default_padding=padding,\n default_dilation_rate=dilation_rate,\n default_use_bias=use_bias,\n default_weights_initializer=weights_initializer,\n default_bias_initializer=bias_initializer,\n default_norm=norm,\n default_norm_params=norm_params,\n default_activation=activation,\n default_dropout=dropout,\n default_pool_function=pool_function,\n default_pool_size=pool_size,\n default_pool_strides=pool_strides,\n default_pool_padding=pool_padding,\n )\n\n self.reduce_output = reduce_output\n self.reduce_sequence = SequenceReducer(\n reduce_mode=reduce_output,\n max_sequence_length=self.conv1d_stack.output_shape[-2],\n encoding_size=self.conv1d_stack.output_shape[-1],\n )\n if self.reduce_output is not None:\n logger.debug(\" FCStack\")\n self.fc_stack = FCStack(\n self.reduce_sequence.output_shape[-1],\n layers=fc_layers,\n num_layers=num_fc_layers,\n default_output_size=output_size,\n default_use_bias=use_bias,\n default_weights_initializer=weights_initializer,\n default_bias_initializer=bias_initializer,\n default_norm=norm,\n default_norm_params=norm_params,\n default_activation=activation,\n default_dropout=dropout,\n )\n\n @staticmethod\n def get_schema_cls() -> type[SequenceEncoderConfig]:\n return StackedCNNConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is None:\n return self.conv1d_stack.output_shape\n return self.fc_stack.output_shape\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n \"\"\"\n :param inputs: The input sequence fed into the encoder.\n Shape: [batch x sequence length], type torch.int32\n :param mask: Input mask (unused, not yet implemented)\n \"\"\"\n # ================ Embeddings ================\n if self.should_embed:\n embedded_sequence = self.embed_sequence(inputs, mask=mask)\n else:\n embedded_sequence = inputs\n while len(embedded_sequence.shape) < 3:\n embedded_sequence = embedded_sequence.unsqueeze(-1)\n\n # shape=(?, sequence_length, embedding_size)\n hidden = embedded_sequence\n\n # ================ Conv Layers ================\n hidden = self.conv1d_stack(hidden, mask=mask)\n\n # ================ Sequence Reduction ================\n if self.reduce_output is not None:\n hidden = self.reduce_sequence(hidden)\n\n # ================ FC Layers ================\n hidden = self.fc_stack(hidden, mask=mask)\n\n # no reduction: hidden [batch_size, seq_size, num_filters]\n # with reduction: hidden [batch_size, output_size]\n return {ENCODER_OUTPUT: hidden}\n\n\n@DeveloperAPI\n@register_sequence_encoder(\"stacked_parallel_cnn\")\n@register_encoder(\"stacked_parallel_cnn\", [AUDIO, SEQUENCE, TEXT, TIMESERIES])\nclass StackedParallelCNN(SequenceEncoder):\n def __init__(\n self,\n should_embed=True,\n vocab=None,\n representation=\"dense\",\n embedding_size=256,\n max_sequence_length=None,\n embeddings_trainable=True,\n pretrained_embeddings=None,\n embeddings_on_cpu=False,\n stacked_layers=None,\n num_stacked_layers=None,\n filter_size=3,\n num_filters=256,\n pool_function=\"max\",\n pool_size=None,\n fc_layers=None,\n num_fc_layers=None,\n output_size=256,\n use_bias=True,\n weights_initializer=\"xavier_uniform\",\n bias_initializer=\"zeros\",\n norm=None,\n norm_params=None,\n activation=\"relu\",\n dropout=0,\n reduce_output=\"max\",\n encoder_config=None,\n **kwargs,\n ):\n # todo: review docstring\n \"\"\"\n :param should_embed: If True the input sequence is expected\n to be made of integers and will be mapped into embeddings\n :type should_embed: Boolean\n :param vocab: Vocabulary of the input feature to encode\n :type vocab: List\n :param representation: the possible values are `dense` and `sparse`.\n `dense` means the embeddings are initialized randomly,\n `sparse` means they are initialized to be one-hot encodings.\n :type representation: Str (one of 'dense' or 'sparse')\n :param embedding_size: it is the maximum embedding size, the actual\n size will be `min(vocabulary_size, embedding_size)`\n for `dense` representations and exactly `vocabulary_size`\n for the `sparse` encoding, where `vocabulary_size` is\n the number of different strings appearing in the training set\n in the column the feature is named after (plus 1 for ``).\n :type embedding_size: Integer\n :param embeddings_trainable: If `True` embeddings are trained during\n the training process, if `False` embeddings are fixed.\n It may be useful when loading pretrained embeddings\n for avoiding finetuning them. This parameter has effect only\n for `representation` is `dense` as `sparse` one-hot encodings\n are not trainable.\n :type embeddings_trainable: Boolean\n :param pretrained_embeddings: by default `dense` embeddings\n are initialized randomly, but this parameter allows to specify\n a path to a file containing embeddings in the GloVe format.\n When the file containing the embeddings is loaded, only the\n embeddings with labels present in the vocabulary are kept,\n the others are discarded. If the vocabulary contains strings\n that have no match in the embeddings file, their embeddings\n are initialized with the average of all other embedding plus\n some random noise to make them different from each other.\n This parameter has effect only if `representation` is `dense`.\n :type pretrained_embeddings: str (filepath)\n :param embeddings_on_cpu: by default embeddings matrices are stored\n on GPU memory if a GPU is used, as it allows\n for faster access, but in some cases the embedding matrix\n may be really big and this parameter forces the placement\n of the embedding matrix in regular memroy and the CPU is used\n to resolve them, slightly slowing down the process\n as a result of data transfer between CPU and GPU memory.\n :param stacked_layers: it is a of lists of list of dictionaries\n containing the parameters of the stack of\n parallel convolutional layers. The length of the list\n determines the number of stacked parallel\n convolutional layers, length of the sub-lists determines\n the number of parallel conv layers and the content\n of each dictionary determines the parameters for\n a specific layer. The available parameters for each layer are:\n `filter_size`, `num_filters`, `pool_size`, `norm` and\n `activation`. If any of those values\n is missing from the dictionary, the default one specified\n as a parameter of the encoder will be used instead. If both\n `stacked_layers` and `num_stacked_layers` are `None`,\n a default list will be assigned to `stacked_layers` with\n the value `[[{filter_size: 2}, {filter_size: 3},\n {filter_size: 4}, {filter_size: 5}], [{filter_size: 2},\n {filter_size: 3}, {filter_size: 4}, {filter_size: 5}],\n [{filter_size: 2}, {filter_size: 3}, {filter_size: 4},\n {filter_size: 5}]]`.\n :type stacked_layers: List\n :param num_stacked_layers: if `stacked_layers` is `None`, this is\n the number of elements in the stack of\n parallel convolutional layers.\n :type num_stacked_layers: Integer\n :param filter_size: if a `filter_size` is not already specified in\n `conv_layers` this is the default `filter_size` that\n will be used for each layer. It indicates how wide is\n the 1d convolutional filter.\n :type filter_size: Integer\n :param num_filters: if a `num_filters` is not already specified in\n `conv_layers` this is the default `num_filters` that\n will be used for each layer. It indicates the number\n of filters, and by consequence the output channels of\n the 1d convolution.\n :type num_filters: Integer\n :param pool_size: if a `pool_size` is not already specified\n in `conv_layers` this is the default `pool_size` that\n will be used for each layer. It indicates the size of\n the max pooling that will be performed along the `s` sequence\n dimension after the convolution operation.\n :type pool_size: Integer\n :param fc_layers: it is a list of dictionaries containing\n the parameters of all the fully connected layers. The length\n of the list determines the number of stacked fully connected\n layers and the content of each dictionary determines\n the parameters for a specific layer. The available parameters\n for each layer are: `output_size`, `norm` and `activation`.\n If any of those values is missing from\n the dictionary, the default one specified as a parameter of\n the encoder will be used instead. If both `fc_layers` and\n `num_fc_layers` are `None`, a default list will be assigned\n to `fc_layers` with the value\n `[{output_size: 512}, {output_size: 256}]`\n (only applies if `reduce_output` is not `None`).\n :type fc_layers: List\n :param num_fc_layers: if `fc_layers` is `None`, this is the number\n of stacked fully connected layers (only applies if\n `reduce_output` is not `None`).\n :type num_fc_layers: Integer\n :param output_size: if a `output_size` is not already specified in\n `fc_layers` this is the default `output_size` that will be used\n for each layer. It indicates the size of the output\n of a fully connected layer.\n :type output_size: Integer\n :param norm: if a `norm` is not already specified in `conv_layers`\n or `fc_layers` this is the default `norm` that will be used\n for each layer. It indicates the norm of the output.\n :type norm: str\n :param activation: Default activation function to use\n :type activation: Str\n :param dropout: determines if there should be a dropout layer before\n returning the encoder output.\n :type dropout: Boolean\n :param initializer: the initializer to use. If `None` it uses\n `xavier_uniform`. Options are: `constant`, `identity`,\n `zeros`, `ones`, `orthogonal`, `normal`, `uniform`,\n `truncated_normal`, `variance_scaling`, `xavier_normal`,\n `xavier_uniform`, `xavier_normal`,\n `he_normal`, `he_uniform`, `lecun_normal`, `lecun_uniform`.\n Alternatively it is possible to specify a dictionary with\n a key `type` that identifies the type of initializer and\n other keys for its parameters,\n e.g. `{type: normal, mean: 0, stddev: 0}`.\n To know the parameters of each initializer, please refer\n to PyTorch's documentation.\n :type initializer: str\n :param reduce_output: defines how to reduce the output tensor of\n the convolutional layers along the `s` sequence length\n dimension if the rank of the tensor is greater than 2.\n Available values are: `sum`, `mean` or `avg`, `max`, `concat`\n (concatenates along the first dimension), `last` (returns\n the last vector of the first dimension) and `None` or `null`\n (which does not reduce and returns the full tensor).\n :type reduce_output: str\n \"\"\"\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n\n self.max_sequence_length = max_sequence_length\n self.embedding_size = embedding_size\n\n if stacked_layers is not None and num_stacked_layers is None:\n # use custom-defined layers\n self.stacked_layers = stacked_layers\n self.num_stacked_layers = len(stacked_layers)\n elif stacked_layers is None and num_stacked_layers is not None:\n # generate num_conv_layers with default parameters\n self.stacked_layers = None\n self.num_stacked_layers = num_stacked_layers\n elif stacked_layers is None and num_stacked_layers is None:\n # use default layers with varying filter sizes\n self.stacked_layers = [\n [{\"filter_size\": 2}, {\"filter_size\": 3}, {\"filter_size\": 4}, {\"filter_size\": 5}],\n [{\"filter_size\": 2}, {\"filter_size\": 3}, {\"filter_size\": 4}, {\"filter_size\": 5}],\n [{\"filter_size\": 2}, {\"filter_size\": 3}, {\"filter_size\": 4}, {\"filter_size\": 5}],\n ]\n self.num_stacked_layers = 6\n else:\n raise ValueError(\"Invalid layer parametrization, use either stacked_layers or\" \" num_stacked_layers\")\n\n # The user is expected to provide fc_layers or num_fc_layers\n # The following logic handles the case where the user either provides\n # both or neither.\n if fc_layers is None and num_fc_layers is None:\n # use default layers with varying filter sizes\n fc_layers = [{\"output_size\": 512}, {\"output_size\": 256}]\n num_fc_layers = 2\n elif fc_layers is not None and num_fc_layers is not None:\n raise ValueError(\"Invalid layer parametrization, use either fc_layers or \" \"num_fc_layers only. Not both.\")\n\n self.should_embed = should_embed\n self.embed_sequence = None\n\n if self.should_embed:\n logger.debug(\" EmbedSequence\")\n self.embed_sequence = EmbedSequence(\n vocab,\n embedding_size,\n max_sequence_length=self.max_sequence_length,\n representation=representation,\n embeddings_trainable=embeddings_trainable,\n pretrained_embeddings=pretrained_embeddings,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=weights_initializer,\n )\n\n in_channels = self.embed_sequence.output_shape[-1] if self.should_embed else embedding_size\n logger.debug(\" ParallelConv1DStack\")\n self.parallel_conv1d_stack = ParallelConv1DStack(\n in_channels=in_channels,\n stacked_layers=self.stacked_layers,\n max_sequence_length=max_sequence_length,\n default_num_filters=num_filters,\n default_filter_size=filter_size,\n default_use_bias=use_bias,\n default_weights_initializer=weights_initializer,\n default_bias_initializer=bias_initializer,\n default_norm=norm,\n default_norm_params=norm_params,\n default_activation=activation,\n default_dropout=dropout,\n default_pool_function=pool_function,\n default_pool_size=pool_size,\n )\n\n self.reduce_output = reduce_output\n self.reduce_sequence = SequenceReducer(\n reduce_mode=reduce_output,\n max_sequence_length=self.parallel_conv1d_stack.output_shape[-2],\n encoding_size=self.parallel_conv1d_stack.output_shape[-1],\n )\n if self.reduce_output is not None:\n logger.debug(\" FCStack\")\n self.fc_stack = FCStack(\n self.reduce_sequence.output_shape[-1],\n layers=fc_layers,\n num_layers=num_fc_layers,\n default_output_size=output_size,\n default_use_bias=use_bias,\n default_weights_initializer=weights_initializer,\n default_bias_initializer=bias_initializer,\n default_norm=norm,\n default_norm_params=norm_params,\n default_activation=activation,\n default_dropout=dropout,\n )\n\n @staticmethod\n def get_schema_cls() -> type[SequenceEncoderConfig]:\n return StackedParallelCNNConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is not None:\n return self.fc_stack.output_shape\n return self.parallel_conv1d_stack.output_shape\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n \"\"\"\n :param inputs: The input sequence fed into the encoder.\n Shape: [batch x sequence length], type torch.int32\n :param mask: Input mask (unused, not yet implemented)\n \"\"\"\n # ================ Embeddings ================\n if self.should_embed:\n embedded_sequence = self.embed_sequence(inputs, mask=mask)\n else:\n embedded_sequence = inputs\n while len(embedded_sequence.shape) < 3:\n embedded_sequence = embedded_sequence.unsqueeze(-1)\n\n # shape=(?, sequence_length, embedding_size)\n hidden = embedded_sequence\n\n # ================ Conv Layers ================\n hidden = self.parallel_conv1d_stack(hidden, mask=mask)\n\n # ================ Sequence Reduction ================\n if self.reduce_output is not None:\n hidden = self.reduce_sequence(hidden)\n\n # ================ FC Layers ================\n hidden = self.fc_stack(hidden, mask=mask)\n\n # no reduction: hidden [batch_size, seq_size, num_filter]\n # with reduction: hidden [batch_size, output_size]\n return {ENCODER_OUTPUT: hidden}\n\n\n@DeveloperAPI\n@register_sequence_encoder(\"rnn\")\n@register_encoder(\"rnn\", [AUDIO, SEQUENCE, TEXT, TIMESERIES])\nclass StackedRNN(SequenceEncoder):\n def __init__(\n self,\n should_embed=True,\n vocab=None,\n representation=\"dense\",\n embedding_size=256,\n embeddings_trainable=True,\n pretrained_embeddings=None,\n embeddings_on_cpu=False,\n num_layers=1,\n max_sequence_length=None,\n state_size=256,\n cell_type=\"rnn\",\n bidirectional=False,\n activation=\"tanh\",\n recurrent_activation=\"sigmoid\",\n unit_forget_bias=True,\n recurrent_initializer=\"orthogonal\",\n dropout=0.0,\n recurrent_dropout=0.0,\n fc_layers=None,\n num_fc_layers=0,\n output_size=256,\n use_bias=True,\n weights_initializer=\"xavier_uniform\",\n bias_initializer=\"zeros\",\n norm=None,\n norm_params=None,\n fc_activation=\"relu\",\n fc_dropout=0,\n reduce_output=\"last\",\n encoder_config=None,\n **kwargs,\n ):\n # todo: fix up docstring\n \"\"\"\n :param should_embed: If True the input sequence is expected\n to be made of integers and will be mapped into embeddings\n :type should_embed: Boolean\n :param vocab: Vocabulary of the input feature to encode\n :type vocab: List\n :param representation: the possible values are `dense` and `sparse`.\n `dense` means the embeddings are initialized randomly,\n `sparse` means they are initialized to be one-hot encodings.\n :type representation: Str (one of 'dense' or 'sparse')\n :param embedding_size: it is the maximum embedding size, the actual\n size will be `min(vocabulary_size, embedding_size)`\n for `dense` representations and exactly `vocabulary_size`\n for the `sparse` encoding, where `vocabulary_size` is\n the number of different strings appearing in the training set\n in the column the feature is named after (plus 1 for ``).\n :type embedding_size: Integer\n :param embeddings_trainable: If `True` embeddings are trained during\n the training process, if `False` embeddings are fixed.\n It may be useful when loading pretrained embeddings\n for avoiding finetuning them. This parameter has effect only\n for `representation` is `dense` as `sparse` one-hot encodings\n are not trainable.\n :type embeddings_trainable: Boolean\n :param pretrained_embeddings: by default `dense` embeddings\n are initialized randomly, but this parameter allows to specify\n a path to a file containing embeddings in the GloVe format.\n When the file containing the embeddings is loaded, only the\n embeddings with labels present in the vocabulary are kept,\n the others are discarded. If the vocabulary contains strings\n that have no match in the embeddings file, their embeddings\n are initialized with the average of all other embedding plus\n some random noise to make them different from each other.\n This parameter has effect only if `representation` is `dense`.\n :type pretrained_embeddings: str (filepath)\n :param embeddings_on_cpu: by default embeddings matrices are stored\n on GPU memory if a GPU is used, as it allows\n for faster access, but in some cases the embedding matrix\n may be really big and this parameter forces the placement\n of the embedding matrix in regular memroy and the CPU is used\n to resolve them, slightly slowing down the process\n as a result of data transfer between CPU and GPU memory.\n :param conv_layers: it is a list of dictionaries containing\n the parameters of all the convolutional layers. The length\n of the list determines the number of parallel convolutional\n layers and the content of each dictionary determines\n the parameters for a specific layer. The available parameters\n for each layer are: `filter_size`, `num_filters`, `pool`,\n `norm`, `activation` and `regularize`. If any of those values\n is missing from the dictionary, the default one specified\n as a parameter of the encoder will be used instead. If both\n `conv_layers` and `num_conv_layers` are `None`, a default\n list will be assigned to `conv_layers` with the value\n `[{filter_size: 2}, {filter_size: 3}, {filter_size: 4},\n {filter_size: 5}]`.\n :type conv_layers: List\n :param num_conv_layers: if `conv_layers` is `None`, this is\n the number of stacked convolutional layers.\n :type num_conv_layers: Integer\n :param filter_size: if a `filter_size` is not already specified in\n `conv_layers` this is the default `filter_size` that\n will be used for each layer. It indicates how wide is\n the 1d convolutional filter.\n :type filter_size: Integer\n :param num_filters: if a `num_filters` is not already specified in\n `conv_layers` this is the default `num_filters` that\n will be used for each layer. It indicates the number\n of filters, and by consequence the output channels of\n the 1d convolution.\n :type num_filters: Integer\n :param pool_size: if a `pool_size` is not already specified\n in `conv_layers` this is the default `pool_size` that\n will be used for each layer. It indicates the size of\n the max pooling that will be performed along the `s` sequence\n dimension after the convolution operation.\n :type pool_size: Integer\n :param num_rec_layers: the number of stacked recurrent layers.\n :type num_rec_layers: Integer\n :param cell_type: the type of recurrent cell to use.\n Available values are: `rnn`, `lstm`, `gru`.\n For reference about the differences between the cells please\n refer to PyTorch's documentation.\n :type cell_type: str\n :param state_size: the size of the state of the rnn.\n :type state_size: Integer\n :param bidirectional: if `True` two recurrent networks will perform\n encoding in the forward and backward direction and\n their outputs will be concatenated.\n :type bidirectional: Boolean\n :param dropout: determines if there should be a dropout layer before\n returning the encoder output.\n :type dropout: Boolean\n :param recurrent_dropout: Dropout rate for the recurrent stack.\n :type recurrent_dropout: float\n :param initializer: the initializer to use. If `None` it uses\n `xavier_uniform`. Options are: `constant`, `identity`,\n `zeros`, `ones`, `orthogonal`, `normal`, `uniform`,\n `truncated_normal`, `variance_scaling`, `xavier_normal`,\n `xavier_uniform`, `xavier_normal`,\n `he_normal`, `he_uniform`, `lecun_normal`, `lecun_uniform`.\n Alternatively it is possible to specify a dictionary with\n a key `type` that identifies the type of initializer and\n other keys for its parameters,\n e.g. `{type: normal, mean: 0, stddev: 0}`.\n To know the parameters of each initializer, please refer\n to PyTorch's documentation.\n :type initializer: str\n :param reduce_output: defines how to reduce the output tensor of\n the convolutional layers along the `s` sequence length\n dimension if the rank of the tensor is greater than 2.\n Available values are: `sum`, `mean` or `avg`, `max`, `concat`\n (concatenates along the first dimension), `last` (returns\n the last vector of the first dimension) and `None` or `null`\n (which does not reduce and returns the full tensor).\n :type reduce_output: str\n \"\"\"\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n\n self.max_sequence_length = max_sequence_length\n self.hidden_size = state_size\n self.embedding_size = embedding_size\n\n self.should_embed = should_embed\n self.embed_sequence = None\n\n if self.should_embed:\n logger.debug(\" EmbedSequence\")\n self.embed_sequence = EmbedSequence(\n vocab,\n embedding_size,\n max_sequence_length=self.max_sequence_length,\n representation=representation,\n embeddings_trainable=embeddings_trainable,\n pretrained_embeddings=pretrained_embeddings,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=weights_initializer,\n )\n\n logger.debug(\" RecurrentStack\")\n input_size = self.embed_sequence.output_shape[-1] if self.should_embed else embedding_size\n self.recurrent_stack = RecurrentStack(\n input_size=input_size,\n hidden_size=state_size,\n cell_type=cell_type,\n max_sequence_length=max_sequence_length,\n num_layers=num_layers,\n bidirectional=bidirectional,\n activation=activation,\n recurrent_activation=recurrent_activation,\n use_bias=use_bias,\n unit_forget_bias=unit_forget_bias,\n weights_initializer=weights_initializer,\n recurrent_initializer=recurrent_initializer,\n bias_initializer=bias_initializer,\n dropout=recurrent_dropout,\n )\n\n self.reduce_output = reduce_output\n self.reduce_sequence = SequenceReducer(\n reduce_mode=reduce_output,\n max_sequence_length=self.recurrent_stack.output_shape[-2],\n encoding_size=self.recurrent_stack.output_shape[-1], # state_size\n )\n if self.reduce_output is None:\n self.supports_masking = True\n else:\n logger.debug(\" FCStack\")\n self.fc_stack = FCStack(\n self.reduce_sequence.output_shape[-1],\n layers=fc_layers,\n num_layers=num_fc_layers,\n default_output_size=output_size,\n default_use_bias=use_bias,\n default_weights_initializer=weights_initializer,\n default_bias_initializer=bias_initializer,\n default_norm=norm,\n default_norm_params=norm_params,\n default_activation=fc_activation,\n default_dropout=fc_dropout,\n )\n\n @staticmethod\n def get_schema_cls() -> type[SequenceEncoderConfig]:\n return StackedRNNConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is not None:\n return self.fc_stack.output_shape\n return self.recurrent_stack.output_shape\n\n def input_dtype(self):\n return torch.int32\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n \"\"\"\n :param inputs: The input sequence fed into the encoder.\n Shape: [batch x sequence length], type torch.int32\n :param mask: Input mask (unused, not yet implemented)\n \"\"\"\n # ================ Embeddings ================\n if self.should_embed:\n embedded_sequence = self.embed_sequence(inputs, mask=mask)\n else:\n embedded_sequence = inputs\n while len(embedded_sequence.shape) < 3:\n embedded_sequence = embedded_sequence.unsqueeze(-1)\n\n # shape=(?, sequence_length, embedding_size)\n hidden = embedded_sequence\n\n # ================ Recurrent Layers ================\n hidden, final_state = self.recurrent_stack(hidden, mask=mask)\n\n # ================ Sequence Reduction ================\n if self.reduce_output is not None:\n hidden = self.reduce_sequence(hidden)\n\n # ================ FC Layers ================\n hidden = self.fc_stack(hidden, mask=mask)\n\n return {ENCODER_OUTPUT: hidden, ENCODER_OUTPUT_STATE: final_state}\n\n\n@DeveloperAPI\n@register_sequence_encoder(\"cnnrnn\")\n@register_encoder(\"cnnrnn\", [AUDIO, SEQUENCE, TEXT, TIMESERIES])\nclass StackedCNNRNN(SequenceEncoder):\n def __init__(\n self,\n should_embed=True,\n vocab=None,\n max_sequence_length=None,\n representation=\"dense\",\n embedding_size=256,\n embeddings_trainable=True,\n pretrained_embeddings=None,\n embeddings_on_cpu=False,\n conv_layers=None,\n num_conv_layers=None,\n num_filters=256,\n filter_size=5,\n strides=1,\n padding=\"same\",\n dilation_rate=1,\n conv_activation=\"relu\",\n conv_dropout=0.0,\n pool_function=\"max\",\n pool_size=2,\n pool_strides=None,\n pool_padding=\"same\",\n num_rec_layers=1,\n state_size=256,\n cell_type=\"rnn\",\n bidirectional=False,\n activation=\"tanh\",\n recurrent_activation=\"sigmoid\",\n unit_forget_bias=True,\n recurrent_initializer=\"orthogonal\",\n dropout=0.0,\n recurrent_dropout=0.0,\n fc_layers=None,\n num_fc_layers=0,\n output_size=256,\n use_bias=True,\n weights_initializer=\"xavier_uniform\",\n bias_initializer=\"zeros\",\n norm=None,\n norm_params=None,\n fc_activation=\"relu\",\n fc_dropout=0,\n reduce_output=\"last\",\n encoder_config=None,\n **kwargs,\n ):\n # todo: fix up docstring\n \"\"\"\n :param should_embed: If True the input sequence is expected\n to be made of integers and will be mapped into embeddings\n :type should_embed: Boolean\n :param vocab: Vocabulary of the input feature to encode\n :type vocab: List\n :param representation: the possible values are `dense` and `sparse`.\n `dense` means the embeddings are initialized randomly,\n `sparse` means they are initialized to be one-hot encodings.\n :type representation: Str (one of 'dense' or 'sparse')\n :param embedding_size: it is the maximum embedding size, the actual\n size will be `min(vocabulary_size, embedding_size)`\n for `dense` representations and exactly `vocabulary_size`\n for the `sparse` encoding, where `vocabulary_size` is\n the number of different strings appearing in the training set\n in the column the feature is named after (plus 1 for ``).\n :type embedding_size: Integer\n :param embeddings_trainable: If `True` embeddings are trained during\n the training process, if `False` embeddings are fixed.\n It may be useful when loading pretrained embeddings\n for avoiding finetuning them. This parameter has effect only\n for `representation` is `dense` as `sparse` one-hot encodings\n are not trainable.\n :type embeddings_trainable: Boolean\n :param pretrained_embeddings: by default `dense` embeddings\n are initialized randomly, but this parameter allows to specify\n a path to a file containing embeddings in the GloVe format.\n When the file containing the embeddings is loaded, only the\n embeddings with labels present in the vocabulary are kept,\n the others are discarded. If the vocabulary contains strings\n that have no match in the embeddings file, their embeddings\n are initialized with the average of all other embedding plus\n some random noise to make them different from each other.\n This parameter has effect only if `representation` is `dense`.\n :type pretrained_embeddings: str (filepath)\n :param embeddings_on_cpu: by default embeddings matrices are stored\n on GPU memory if a GPU is used, as it allows\n for faster access, but in some cases the embedding matrix\n may be really big and this parameter forces the placement\n of the embedding matrix in regular memroy and the CPU is used\n to resolve them, slightly slowing down the process\n as a result of data transfer between CPU and GPU memory.\n :param num_layers: the number of stacked recurrent layers.\n :type num_layers: Integer\n :param cell_type: the type of recurrent cell to use.\n Available values are: `rnn`, `lstm`, `gru`.\n For reference about the differences between the cells please\n refer to PyTorch's documentation.\n :type cell_type: str\n :param state_size: the size of the state of the rnn.\n :type state_size: Integer\n :param bidirectional: if `True` two recurrent networks will perform\n encoding in the forward and backward direction and\n their outputs will be concatenated.\n :type bidirectional: Boolean\n :param dropout: determines if there should be a dropout layer before\n returning the encoder output.\n :type dropout: Boolean\n :param recurrent_dropout: Dropout rate for the recurrent stack.\n :type recurrent_dropout: float\n :param initializer: the initializer to use. If `None` it uses\n `xavier_uniform`. Options are: `constant`, `identity`,\n `zeros`, `ones`, `orthogonal`, `normal`, `uniform`,\n `truncated_normal`, `variance_scaling`, `xavier_normal`,\n `xavier_uniform`, `xavier_normal`,\n `he_normal`, `he_uniform`, `lecun_normal`, `lecun_uniform`.\n Alternatively it is possible to specify a dictionary with\n a key `type` that identifies the type of initializer and\n other keys for its parameters,\n e.g. `{type: normal, mean: 0, stddev: 0}`.\n To know the parameters of each initializer, please refer\n to PyTorch's documentation.\n :type initializer: str\n :param reduce_output: defines how to reduce the output tensor of\n the convolutional layers along the `s` sequence length\n dimension if the rank of the tensor is greater than 2.\n Available values are: `sum`, `mean` or `avg`, `max`, `concat`\n (concatenates along the first dimension), `last` (returns\n the last vector of the first dimension) and `None` or `null`\n (which does not reduce and returns the full tensor).\n :type reduce_output: str\n \"\"\"\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n\n if conv_layers is not None and num_conv_layers is None:\n # use custom-defined layers\n self.conv_layers = conv_layers\n self.num_conv_layers = len(conv_layers)\n elif conv_layers is None and num_conv_layers is not None:\n # generate num_conv_layers with default parameters\n self.conv_layers = None\n self.num_conv_layers = num_conv_layers\n elif conv_layers is None and num_conv_layers is None:\n # use default layers with varying filter sizes\n self.conv_layers = [{\"pool_size\": 3}, {\"pool_size\": None}]\n self.num_conv_layers = 2\n else:\n raise ValueError(\"Invalid layer parametrization, use either conv_layers or \" \"num_conv_layers\")\n\n self.max_sequence_length = max_sequence_length\n self.should_embed = should_embed\n self.embed_sequence = None\n\n if self.should_embed:\n logger.debug(\" EmbedSequence\")\n self.embed_sequence = EmbedSequence(\n vocab,\n embedding_size,\n max_sequence_length=self.max_sequence_length,\n representation=representation,\n embeddings_trainable=embeddings_trainable,\n pretrained_embeddings=pretrained_embeddings,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=weights_initializer,\n )\n\n logger.debug(\" Conv1DStack\")\n in_channels = self.embed_sequence.output_shape[-1] if self.should_embed else embedding_size\n self.conv1d_stack = Conv1DStack(\n in_channels=in_channels,\n max_sequence_length=max_sequence_length,\n layers=self.conv_layers,\n default_num_filters=num_filters,\n default_filter_size=filter_size,\n default_strides=strides,\n default_padding=padding,\n default_dilation_rate=dilation_rate,\n default_use_bias=use_bias,\n default_weights_initializer=weights_initializer,\n default_bias_initializer=bias_initializer,\n default_norm=norm,\n default_norm_params=norm_params,\n default_activation=conv_activation,\n default_dropout=conv_dropout,\n default_pool_function=pool_function,\n default_pool_size=pool_size,\n default_pool_strides=pool_strides,\n default_pool_padding=pool_padding,\n )\n\n logger.debug(\" RecurrentStack\")\n self.recurrent_stack = RecurrentStack(\n input_size=self.conv1d_stack.output_shape[1],\n hidden_size=state_size,\n max_sequence_length=self.conv1d_stack.output_shape[0],\n cell_type=cell_type,\n num_layers=num_rec_layers,\n bidirectional=bidirectional,\n activation=activation,\n recurrent_activation=recurrent_activation,\n use_bias=use_bias,\n unit_forget_bias=unit_forget_bias,\n weights_initializer=weights_initializer,\n recurrent_initializer=recurrent_initializer,\n bias_initializer=bias_initializer,\n dropout=recurrent_dropout,\n )\n\n self.reduce_output = reduce_output\n self.reduce_sequence = SequenceReducer(\n reduce_mode=reduce_output,\n max_sequence_length=self.recurrent_stack.output_shape[-2],\n encoding_size=self.recurrent_stack.output_shape[-1], # State size\n )\n if self.reduce_output is not None:\n logger.debug(\" FCStack\")\n self.fc_stack = FCStack(\n self.reduce_sequence.output_shape[-1],\n layers=fc_layers,\n num_layers=num_fc_layers,\n default_output_size=output_size,\n default_use_bias=use_bias,\n default_weights_initializer=weights_initializer,\n default_bias_initializer=bias_initializer,\n default_norm=norm,\n default_norm_params=norm_params,\n default_activation=fc_activation,\n default_dropout=fc_dropout,\n )\n\n @staticmethod\n def get_schema_cls() -> type[SequenceEncoderConfig]:\n return StackedCNNRNNConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is not None:\n return self.fc_stack.output_shape\n return self.recurrent_stack.output_shape\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n \"\"\"\n :param inputs: The input sequence fed into the encoder.\n Shape: [batch x sequence length], type torch.int32\n :param mask: Input mask (unused, not yet implemented)\n \"\"\"\n # ================ Embeddings ================\n if self.should_embed:\n embedded_sequence = self.embed_sequence(inputs, mask=mask)\n else:\n embedded_sequence = inputs\n while len(embedded_sequence.shape) < 3:\n embedded_sequence = embedded_sequence.unsqueeze(-1)\n\n # shape=(?, sequence_length, embedding_size)\n hidden = embedded_sequence\n\n # ================ Conv Layers ================\n hidden = self.conv1d_stack(hidden, mask=mask)\n\n # ================ Recurrent Layers ================\n hidden, final_state = self.recurrent_stack(hidden)\n\n # ================ Sequence Reduction ================\n if self.reduce_output is not None:\n hidden = self.reduce_sequence(hidden)\n\n # ================ FC Layers ================\n hidden = self.fc_stack(hidden, mask=mask)\n\n # no reduction: hidden [batch_size, seq_size, state_size]\n # with reduction: hidden [batch_size, seq_size, output_size]\n # final_state: if rnn/gru [batch_size, state_size]\n # lstm ([batch_size, state_size], [batch_size, state_size])\n return {ENCODER_OUTPUT: hidden, ENCODER_OUTPUT_STATE: final_state}\n\n\n@DeveloperAPI\n@register_sequence_encoder(\"transformer\")\n@register_encoder(\"transformer\", [SEQUENCE, TEXT, TIMESERIES])\nclass StackedTransformer(SequenceEncoder):\n def __init__(\n self,\n max_sequence_length,\n should_embed=True,\n vocab=None,\n representation=\"dense\",\n embedding_size=256,\n embeddings_trainable=True,\n pretrained_embeddings=None,\n embeddings_on_cpu=False,\n num_layers=1,\n hidden_size=256,\n num_heads=8,\n transformer_output_size=256,\n dropout=0.1,\n fc_layers=None,\n num_fc_layers=0,\n output_size=256,\n use_bias=True,\n weights_initializer=\"xavier_uniform\",\n bias_initializer=\"zeros\",\n norm=None,\n norm_params=None,\n fc_activation=\"relu\",\n fc_dropout=0,\n reduce_output=\"last\",\n encoder_config=None,\n **kwargs,\n ):\n # todo: update docstring as needed\n \"\"\"\n :param should_embed: If True the input sequence is expected\n to be made of integers and will be mapped into embeddings\n :type should_embed: Boolean\n :param vocab: Vocabulary of the input feature to encode\n :type vocab: List\n :param representation: the possible values are `dense` and `sparse`.\n `dense` means the embeddings are initialized randomly,\n `sparse` means they are initialized to be one-hot encodings.\n :type representation: Str (one of 'dense' or 'sparse')\n :param embedding_size: it is the maximum embedding size, the actual\n size will be `min(vocabulary_size, embedding_size)`\n for `dense` representations and exactly `vocabulary_size`\n for the `sparse` encoding, where `vocabulary_size` is\n the number of different strings appearing in the training set\n in the column the feature is named after (plus 1 for ``).\n :type embedding_size: Integer\n :param embeddings_trainable: If `True` embeddings are trained during\n the training process, if `False` embeddings are fixed.\n It may be useful when loading pretrained embeddings\n for avoiding finetuning them. This parameter has effect only\n for `representation` is `dense` as `sparse` one-hot encodings\n are not trainable.\n :type embeddings_trainable: Boolean\n :param pretrained_embeddings: by default `dense` embeddings\n are initialized randomly, but this parameter allows to specify\n a path to a file containing embeddings in the GloVe format.\n When the file containing the embeddings is loaded, only the\n embeddings with labels present in the vocabulary are kept,\n the others are discarded. If the vocabulary contains strings\n that have no match in the embeddings file, their embeddings\n are initialized with the average of all other embedding plus\n some random noise to make them different from each other.\n This parameter has effect only if `representation` is `dense`.\n :type pretrained_embeddings: str (filepath)\n :param embeddings_on_cpu: by default embeddings matrices are stored\n on GPU memory if a GPU is used, as it allows\n for faster access, but in some cases the embedding matrix\n may be really big and this parameter forces the placement\n of the embedding matrix in regular memroy and the CPU is used\n to resolve them, slightly slowing down the process\n as a result of data transfer between CPU and GPU memory.\n :param conv_layers: it is a list of dictionaries containing\n the parameters of all the convolutional layers. The length\n of the list determines the number of parallel convolutional\n layers and the content of each dictionary determines\n the parameters for a specific layer. The available parameters\n for each layer are: `filter_size`, `num_filters`, `pool`,\n `norm`, `activation` and `regularize`. If any of those values\n is missing from the dictionary, the default one specified\n as a parameter of the encoder will be used instead. If both\n `conv_layers` and `num_conv_layers` are `None`, a default\n list will be assigned to `conv_layers` with the value\n `[{filter_size: 2}, {filter_size: 3}, {filter_size: 4},\n {filter_size: 5}]`.\n :type conv_layers: List\n :param num_conv_layers: if `conv_layers` is `None`, this is\n the number of stacked convolutional layers.\n :type num_conv_layers: Integer\n :param filter_size: if a `filter_size` is not already specified in\n `conv_layers` this is the default `filter_size` that\n will be used for each layer. It indicates how wide is\n the 1d convolutional filter.\n :type filter_size: Integer\n :param num_filters: if a `num_filters` is not already specified in\n `conv_layers` this is the default `num_filters` that\n will be used for each layer. It indicates the number\n of filters, and by consequence the output channels of\n the 1d convolution.\n :type num_filters: Integer\n :param pool_size: if a `pool_size` is not already specified\n in `conv_layers` this is the default `pool_size` that\n will be used for each layer. It indicates the size of\n the max pooling that will be performed along the `s` sequence\n dimension after the convolution operation.\n :type pool_size: Integer\n :param num_rec_layers: the number of stacked recurrent layers.\n :type num_rec_layers: Integer\n :param cell_type: the type of recurrent cell to use.\n Available values are: `rnn`, `lstm`, `lstm_block`, `lstm`,\n `ln`, `lstm_cudnn`, `gru`, `gru_block`, `gru_cudnn`.\n For reference about the differences between the cells please\n refer to PyTorch's documentation. We suggest to use the\n `block` variants on CPU and the `cudnn` variants on GPU\n because of their increased speed.\n :type cell_type: str\n :param state_size: the size of the state of the rnn.\n :type state_size: Integer\n :param bidirectional: if `True` two recurrent networks will perform\n encoding in the forward and backward direction and\n their outputs will be concatenated.\n :type bidirectional: Boolean\n :param dropout: determines if there should be a dropout layer before\n returning the encoder output.\n :type dropout: Boolean\n :param initializer: the initializer to use. If `None` it uses\n `xavier_uniform`. Options are: `constant`, `identity`,\n `zeros`, `ones`, `orthogonal`, `normal`, `uniform`,\n `truncated_normal`, `variance_scaling`, `xavier_normal`,\n `xavier_uniform`, `xavier_normal`,\n `he_normal`, `he_uniform`, `lecun_normal`, `lecun_uniform`.\n Alternatively it is possible to specify a dictionary with\n a key `type` that identifies the type of initializer and\n other keys for its parameters,\n e.g. `{type: normal, mean: 0, stddev: 0}`.\n To know the parameters of each initializer, please refer\n to PyTorch's documentation.\n :type initializer: str\n :param reduce_output: defines how to reduce the output tensor of\n the convolutional layers along the `s` sequence length\n dimension if the rank of the tensor is greater than 2.\n Available values are: `sum`, `mean` or `avg`, `max`, `concat`\n (concatenates along the first dimension), `last` (returns\n the last vector of the first dimension) and `None` or `null`\n (which does not reduce and returns the full tensor).\n :type reduce_output: str\n \"\"\"\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n\n self.max_sequence_length = max_sequence_length\n\n self.should_embed = should_embed\n self.should_project = False\n self.embed_sequence = None\n\n if self.should_embed:\n logger.debug(\" EmbedSequence\")\n self.embed_sequence = TokenAndPositionEmbedding(\n max_sequence_length=max_sequence_length,\n vocab=vocab,\n embedding_size=embedding_size,\n representation=representation,\n embeddings_trainable=embeddings_trainable,\n pretrained_embeddings=pretrained_embeddings,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=weights_initializer,\n )\n # If vocab size is smaller than embedding size, embedding layer will use len(vocab) as embedding_size.\n used_embedding_size = self.embed_sequence.output_shape[-1]\n if used_embedding_size != hidden_size:\n logger.debug(\" project_to_embed_size\")\n self.project_to_hidden_size = nn.Linear(self.embed_sequence.output_shape[-1], hidden_size)\n self.should_project = True\n else:\n logger.debug(\" project_to_embed_size\")\n self.project_to_hidden_size = nn.Linear(embedding_size, hidden_size)\n self.should_project = True\n\n logger.debug(\" TransformerStack\")\n self.transformer_stack = TransformerStack(\n input_size=hidden_size,\n max_sequence_length=max_sequence_length,\n hidden_size=hidden_size,\n num_heads=num_heads,\n output_size=transformer_output_size,\n num_layers=num_layers,\n dropout=dropout,\n )\n\n self.reduce_output = reduce_output\n self.reduce_sequence = SequenceReducer(\n reduce_mode=reduce_output,\n max_sequence_length=self.transformer_stack.output_shape[-2],\n encoding_size=self.transformer_stack.output_shape[-1], # hidden_size\n )\n if self.reduce_output is None:\n self.supports_masking = True\n else:\n logger.debug(\" FCStack\")\n self.fc_stack = FCStack(\n self.reduce_sequence.output_shape[-1],\n layers=fc_layers,\n num_layers=num_fc_layers,\n default_output_size=output_size,\n default_use_bias=use_bias,\n default_weights_initializer=weights_initializer,\n default_bias_initializer=bias_initializer,\n default_norm=norm,\n default_norm_params=norm_params,\n default_activation=fc_activation,\n default_dropout=fc_dropout,\n )\n\n @staticmethod\n def get_schema_cls() -> type[SequenceEncoderConfig]:\n return StackedTransformerConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is not None:\n return self.fc_stack.output_shape\n return self.transformer_stack.output_shape\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n \"\"\"\n :param inputs: The input sequence fed into the encoder.\n Shape: [batch x sequence length], type torch.int32\n :param mask: Input mask (unused, not yet implemented)\n \"\"\"\n # ================ Embeddings ================\n if self.should_embed:\n embedded_sequence = self.embed_sequence(inputs, mask=mask)\n else:\n embedded_sequence = inputs\n while len(embedded_sequence.shape) < 3:\n embedded_sequence = embedded_sequence.unsqueeze(-1)\n\n # shape=(?, sequence_length, embedding_size)\n if self.should_project:\n hidden = self.project_to_hidden_size(embedded_sequence)\n else:\n hidden = embedded_sequence\n # shape=(?, sequence_length, hidden)\n\n # ================ Transformer Layers ================\n hidden = self.transformer_stack(hidden, mask=mask)\n\n # ================ Sequence Reduction ================\n if self.reduce_output is not None:\n hidden = self.reduce_sequence(hidden)\n\n # ================ FC Layers ================\n hidden = self.fc_stack(hidden, mask=mask)\n\n return {ENCODER_OUTPUT: hidden}\n"
},
{
"path": "ludwig/encoders/set_encoders.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nfrom typing import Any\n\nimport torch\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import ENCODER_OUTPUT, SET\nfrom ludwig.encoders.base import Encoder\nfrom ludwig.encoders.registry import register_encoder\nfrom ludwig.encoders.types import EncoderOutputDict\nfrom ludwig.modules.embedding_modules import EmbedSet\nfrom ludwig.modules.fully_connected_modules import FCStack\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.set_encoders import SetSparseEncoderConfig\n\nlogger = logging.getLogger(__name__)\n\n\n@DeveloperAPI\n@register_encoder(\"embed\", SET)\nclass SetSparseEncoder(Encoder):\n def __init__(\n self,\n vocab: list[str],\n representation: str = \"dense\",\n embedding_size: int = 50,\n embeddings_trainable: bool = True,\n pretrained_embeddings: str | None = None,\n embeddings_on_cpu: bool = False,\n fc_layers=None,\n num_fc_layers: int = 0,\n output_size: int = 10,\n use_bias: bool = True,\n weights_initializer: str = \"xavier_uniform\",\n bias_initializer: str = \"zeros\",\n norm: str | None = None,\n norm_params: dict[str, Any] | None = None,\n activation: str = \"relu\",\n dropout: float = 0.0,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n self.config = encoder_config\n\n logger.debug(f\" {self.name}\")\n\n self.vocab_size = len(vocab)\n\n logger.debug(\" Embed\")\n self.embed = EmbedSet(\n vocab,\n embedding_size,\n representation=representation,\n embeddings_trainable=embeddings_trainable,\n pretrained_embeddings=pretrained_embeddings,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=weights_initializer,\n )\n\n logger.debug(\" FCStack\")\n # TODO(shreya): Make sure this is updated when FCStack is updated\n self.fc_stack = FCStack(\n first_layer_input_size=self.embed.output_shape[-1],\n layers=fc_layers,\n num_layers=num_fc_layers,\n default_output_size=output_size,\n default_use_bias=use_bias,\n default_weights_initializer=weights_initializer,\n default_bias_initializer=bias_initializer,\n default_norm=norm,\n default_norm_params=norm_params,\n default_activation=activation,\n default_dropout=dropout,\n )\n\n def forward(self, inputs: torch.Tensor) -> EncoderOutputDict:\n \"\"\"\n Params:\n inputs: The inputs fed into the encoder.\n Shape: [batch x vocab_size], type tf.int32.\n\n Returns:\n Embeddings of shape [batch x vocab_size x embed size], type float32.\n \"\"\"\n hidden = self.embed(inputs)\n hidden = self.fc_stack(hidden)\n\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return SetSparseEncoderConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.vocab_size])\n\n @property\n def output_shape(self) -> torch.Size:\n return self.fc_stack.output_shape\n"
},
{
"path": "ludwig/encoders/text_encoders.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport inspect\nimport logging\nfrom collections.abc import Callable\nfrom typing import Any, TYPE_CHECKING, TypeVar\n\nimport numpy as np\nimport torch\nfrom torch import nn\nfrom transformers import AutoConfig\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import ENCODER_OUTPUT, TEXT\nfrom ludwig.encoders.base import Encoder\nfrom ludwig.encoders.registry import register_encoder\nfrom ludwig.encoders.types import EncoderOutputDict\nfrom ludwig.modules.reduction_modules import SequenceReducer\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.sequence_encoders import SequenceEncoderConfig\nfrom ludwig.schema.encoders.text_encoders import (\n ALBERTConfig,\n AutoTransformerConfig,\n BERTConfig,\n CamemBERTConfig,\n CTRLConfig,\n DebertaV2Config,\n DistilBERTConfig,\n ELECTRAConfig,\n FlauBERTConfig,\n GPT2Config,\n GPTConfig,\n LLMEncoderConfig,\n LongformerConfig,\n MT5Config,\n RoBERTaConfig,\n T5Config,\n TfIdfEncoderConfig,\n TransformerXLConfig,\n XLMConfig,\n XLMRoBERTaConfig,\n XLNetConfig,\n)\nfrom ludwig.schema.llms.peft import adapter_registry, BaseAdapterConfig\nfrom ludwig.utils.data_utils import clear_data_cache\nfrom ludwig.utils.hf_utils import load_pretrained_hf_model_with_hub_fallback\nfrom ludwig.utils.llm_utils import get_context_len, initialize_adapter, load_pretrained_from_config\nfrom ludwig.utils.tokenizers import HFTokenizer\nfrom ludwig.utils.torch_utils import FreezeModule\n\nif TYPE_CHECKING:\n from transformers import PretrainedConfig, PreTrainedModel\n\n from ludwig.schema.encoders.text_encoders import HFEncoderConfig\n\nlogger = logging.getLogger(__name__)\n\n\ndef _cls_pooled_error_message(encoder: str):\n # TODO(Arnav): Remove this once we have reduce_output options set for\n # each encoder type in the schema\n raise ValueError(f\"reduce_output cannot be cls_pooled for {encoder}\")\n\n\nclass HFTextEncoder(Encoder):\n def _init_config(self, transformer, schema_keys: list[str], encoder_config: SequenceEncoderConfig):\n \"\"\"Creates a config object for the encoder using the transformer model and the passed-in encoder config.\n\n The transformer's config is only known after it is instantiated, so we must update the\n encoder config with the values from the transformer config.\n\n Args:\n transformer: The transformer model.\n schema_keys: The keys in the encoder config schema. We only want to update the encoder config\n with the values from the transformer config that are in the schema.\n encoder_config: The existing encoder config containing defaults and user-specified values.\n If the values in this config differ from the transformer's config, the transformer's config\n values will override this config's values.\n Returns:\n A new encoder config object with the updated values from the transformer config.\n \"\"\"\n transformer_config = transformer.config.to_dict()\n final_hf_config_params = {k: v for k, v in transformer_config.items() if k in schema_keys}\n encoder_config_dict = encoder_config.to_dict()\n encoder_config_dict.update(final_hf_config_params)\n return self.get_schema_cls().from_dict(encoder_config_dict)\n\n def _init_transformer_from_scratch(\n self, hf_model_cls: type, hf_config_cls: type, hf_config_params: dict[str, Any], vocab_size: int\n ):\n \"\"\"Initializes the transformer model from scratch. This is in contrast to loading a pre-trained model.\n\n Args:\n hf_model_cls: The HuggingFace model class.\n hf_config_cls: The HuggingFace config class.\n hf_config_params: The HuggingFace config parameters exposed through the Ludwig schema.\n vocab_size: The vocab size of the dataset. Because we are training from scratch, we can resize the\n token embeddings table freely.\n Returns:\n The transformer model.\n \"\"\"\n config = hf_config_cls(**hf_config_params)\n transformer = hf_model_cls(config)\n self._maybe_resize_token_embeddings(transformer, vocab_size)\n return transformer\n\n def _maybe_resize_token_embeddings(self, transformer, vocab_size: int) -> None:\n \"\"\"Resizes the token embeddings if the vocab size is different from the transformer's vocab size.\n\n This should only happen if we are instantiating a model from scratch (i.e. not loading from a pretrained model\n or checkpoint). Pretrained models update the vocab size stored in the config. This means if we are loading a\n pretrained model from a checkpoint, the config vocab size should match the model's vocab size.\n\n It is important that pretrained models update the vocab size stored in the config because sometimes the\n pretrained models will have an embeddings table that is a different size than the vocab size. Examples:\n\n CamemBERT: https://github.com/huggingface/tokenizers/issues/900#issue-1122256698\n T5: https://github.com/huggingface/transformers/issues/4875#issue-635471552\n\n Args:\n transformer: The transformer model.\n vocab_size: The vocab size of the dataset.\n \"\"\"\n if vocab_size != transformer.config.vocab_size:\n transformer.resize_token_embeddings(vocab_size)\n\n def _wrap_transformer(\n self, transformer: nn.Module, adapter: BaseAdapterConfig | dict | None, trainable: bool\n ) -> nn.Module:\n if adapter is not None:\n from peft import get_peft_model\n\n if isinstance(adapter, dict):\n adapter_cls = adapter_registry[adapter[\"type\"]]\n adapter = adapter_cls.model_validate(adapter)\n peft_config = adapter.to_config()\n transformer = get_peft_model(transformer, peft_config)\n\n logger.info(\"==================================================\")\n logger.info(\"Trainable Parameter Summary For Fine-Tuning:\")\n transformer.print_trainable_parameters()\n logger.info(\"==================================================\")\n return FreezeModule(transformer, frozen=not trainable)\n\n def get_embedding_layer(self) -> nn.Module:\n return next(self.transformer.module.children())\n\n\nHFModelT = TypeVar(\"HFModelT\", bound=\"PreTrainedModel\")\nHFConfigT = TypeVar(\"HFConfigT\", bound=\"PretrainedConfig\")\nConfigT = TypeVar(\"ConfigT\", bound=\"HFEncoderConfig\")\n\n\nclass HFTextEncoderImpl(HFTextEncoder):\n def __init__(\n self,\n model_cls: type[HFModelT],\n config_cls: type[HFConfigT],\n schema_cls: type[ConfigT],\n max_sequence_length: int,\n use_pretrained: bool,\n pretrained_model_name_or_path: str,\n saved_weights_in_checkpoint: bool,\n reduce_output: str,\n trainable: bool,\n adapter: BaseAdapterConfig | None,\n pretrained_kwargs: dict,\n encoder_config: ConfigT | None,\n **kwargs,\n ):\n super().__init__()\n\n # TODO(travis): get_hf_config_param_names should be implemented as abstract in HFEncoderConfig\n vocab_size = kwargs[\"vocab_size\"]\n hf_config_params = {k: v for k, v in kwargs.items() if k in schema_cls.get_hf_config_param_names()}\n if use_pretrained and not saved_weights_in_checkpoint:\n pretrained_kwargs = pretrained_kwargs or {}\n transformer, _ = load_pretrained_hf_model_with_hub_fallback(\n model_cls, pretrained_model_name_or_path, **pretrained_kwargs\n )\n else:\n transformer = self._init_transformer_from_scratch(model_cls, config_cls, hf_config_params, vocab_size)\n\n if encoder_config is not None:\n self.config = self._init_config(transformer, hf_config_params.keys(), encoder_config)\n else:\n self.config = None\n\n self.reduce_output = reduce_output\n if not self.reduce_output == \"cls_pooled\":\n self.reduce_sequence = SequenceReducer(reduce_mode=reduce_output)\n self.transformer = self._wrap_transformer(transformer, adapter, trainable)\n self.max_sequence_length = max_sequence_length\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n if mask is not None:\n mask = mask.to(torch.int32)\n transformer_outputs = self.transformer.module(\n input_ids=inputs,\n attention_mask=mask,\n token_type_ids=torch.zeros_like(inputs),\n )\n if self.reduce_output == \"cls_pooled\":\n hidden = transformer_outputs[\"pooler_output\"]\n else:\n hidden = transformer_outputs[\"last_hidden_state\"][:, 1:-1, :] # bos + [sent] + sep\n hidden = self.reduce_sequence(hidden, self.reduce_output)\n return {ENCODER_OUTPUT: hidden}\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is None:\n return torch.Size([self.max_sequence_length - 2, self.transformer.module.config.hidden_size])\n if self.reduce_output == \"concat\":\n return torch.Size(\n [\n (self.max_sequence_length - 2) * self.transformer.module.config.hidden_size,\n ]\n )\n elif self.reduce_output == \"concat\":\n # add the -2 to account of start and end tokens.\n return torch.Size([self.transformer.module.config.hidden_size * (self.max_sequence_length - 2)])\n return torch.Size([self.transformer.module.config.hidden_size])\n\n @property\n def input_dtype(self) -> torch.dtype:\n return torch.int32\n\n\n@DeveloperAPI\n@register_encoder(\"albert\", TEXT)\nclass ALBERTEncoder(HFTextEncoder):\n DEFAULT_MODEL_NAME = \"albert-base-v2\"\n\n def __init__(\n self,\n max_sequence_length,\n use_pretrained: bool = True,\n pretrained_model_name_or_path: str = DEFAULT_MODEL_NAME,\n saved_weights_in_checkpoint: bool = False,\n trainable: bool = False,\n adapter: BaseAdapterConfig | None = None,\n reduce_output: str = \"cls_pooled\",\n vocab_size: int = 30000,\n embedding_size: int = 128,\n hidden_size: int = 4096,\n num_hidden_layers: int = 12,\n num_hidden_groups: int = 1,\n num_attention_heads: int = 64,\n intermediate_size: int = 16384,\n inner_group_num: int = 1,\n hidden_act: str = \"gelu_new\",\n hidden_dropout_prob: float = 0,\n attention_probs_dropout_prob: float = 0,\n max_position_embeddings: int = 512,\n type_vocab_size: int = 2,\n initializer_range: float = 0.02,\n layer_norm_eps: float = 1e-12,\n classifier_dropout_prob: float = 0.1,\n position_embedding_type: str = \"absolute\",\n pad_token_id: int = 0,\n bos_token_id: int = 2,\n eos_token_id: int = 3,\n pretrained_kwargs: dict = None,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n\n from transformers import AlbertConfig, AlbertModel\n\n hf_config_params = dict(\n vocab_size=vocab_size,\n embedding_size=embedding_size,\n hidden_size=hidden_size,\n num_hidden_layers=num_hidden_layers,\n num_hidden_groups=num_hidden_groups,\n num_attention_heads=num_attention_heads,\n intermediate_size=intermediate_size,\n inner_group_num=inner_group_num,\n hidden_act=hidden_act,\n hidden_dropout_prob=hidden_dropout_prob,\n attention_probs_dropout_prob=attention_probs_dropout_prob,\n max_position_embeddings=max_position_embeddings,\n type_vocab_size=type_vocab_size,\n initializer_range=initializer_range,\n layer_norm_eps=layer_norm_eps,\n classifier_dropout_prob=classifier_dropout_prob,\n position_embedding_type=position_embedding_type,\n pad_token_id=pad_token_id,\n bos_token_id=bos_token_id,\n eos_token_id=eos_token_id,\n )\n\n if use_pretrained and not saved_weights_in_checkpoint:\n pretrained_kwargs = pretrained_kwargs or {}\n transformer, _ = load_pretrained_hf_model_with_hub_fallback(\n AlbertModel, pretrained_model_name_or_path, **pretrained_kwargs\n )\n else:\n transformer = self._init_transformer_from_scratch(AlbertModel, AlbertConfig, hf_config_params, vocab_size)\n\n if encoder_config is not None:\n self.config = self._init_config(transformer, hf_config_params.keys(), encoder_config)\n else:\n self.config = None\n\n self.reduce_output = reduce_output\n if not self.reduce_output == \"cls_pooled\":\n self.reduce_sequence = SequenceReducer(reduce_mode=reduce_output)\n self.transformer = self._wrap_transformer(transformer, adapter, trainable)\n self.max_sequence_length = max_sequence_length\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n if mask is not None:\n mask = mask.to(torch.int32)\n transformer_outputs = self.transformer.module(\n input_ids=inputs,\n attention_mask=mask,\n token_type_ids=torch.zeros_like(inputs),\n )\n if self.reduce_output == \"cls_pooled\":\n hidden = transformer_outputs[1]\n else:\n hidden = transformer_outputs[0][:, 1:-1, :]\n hidden = self.reduce_sequence(hidden, self.reduce_output)\n\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return ALBERTConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is None:\n # Subtract 2 to remove CLS and PAD tokens added by BERT tokenizer.\n return torch.Size(\n [\n self.max_sequence_length - 2,\n self.transformer.module.config.hidden_size,\n ]\n )\n elif self.reduce_output == \"concat\":\n # add the -2 to account of start and end tokens.\n return torch.Size([self.transformer.module.config.hidden_size * (self.max_sequence_length - 2)])\n return torch.Size([self.transformer.module.config.hidden_size])\n\n @property\n def input_dtype(self) -> torch.dtype:\n return torch.int32\n\n\n@DeveloperAPI\n@register_encoder(\"mt5\", TEXT)\nclass MT5Encoder(HFTextEncoder):\n DEFAULT_MODEL_NAME = \"google/mt5-base\"\n\n def __init__(\n self,\n max_sequence_length: int,\n use_pretrained: bool = True,\n pretrained_model_name_or_path: str = DEFAULT_MODEL_NAME,\n saved_weights_in_checkpoint: bool = False,\n trainable: bool = False,\n adapter: BaseAdapterConfig | None = None,\n reduce_output: str = \"sum\",\n vocab_size: int = 250112,\n d_model: int = 512,\n d_kv: int = 64,\n d_ff: int = 1024,\n num_layers: int = 8,\n num_decoder_layers: int = None,\n num_heads: int = 6,\n relative_attention_num_buckets: int = 32,\n dropout_rate: float = 0.1,\n layer_norm_epsilon: float = 1e-06,\n initializer_factor: float = 1.0,\n feed_forward_proj: str = \"gated-gelu\",\n is_encoder_decoder: bool = True,\n use_cache: bool = True,\n tokenizer_class: str = \"T5Tokenizer\",\n tie_word_embeddings: bool = False,\n pad_token_id: int = 0,\n eos_token_id: int = 1,\n decoder_start_token_id: int = 0,\n pretrained_kwargs: dict = None,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n\n from transformers import MT5Config, MT5EncoderModel\n\n hf_config_params = dict(\n vocab_size=vocab_size,\n d_model=d_model,\n d_kv=d_kv,\n d_ff=d_ff,\n num_layers=num_layers,\n num_decoder_layers=num_decoder_layers,\n num_heads=num_heads,\n relative_attention_num_buckets=relative_attention_num_buckets,\n dropout_rate=dropout_rate,\n layer_norm_epsilon=layer_norm_epsilon,\n initializer_factor=initializer_factor,\n feed_forward_proj=feed_forward_proj,\n is_encoder_decoder=is_encoder_decoder,\n use_cache=use_cache,\n tokenizer_class=tokenizer_class,\n tie_word_embeddings=tie_word_embeddings,\n pad_token_id=pad_token_id,\n eos_token_id=eos_token_id,\n decoder_start_token_id=decoder_start_token_id,\n )\n\n if use_pretrained and not saved_weights_in_checkpoint:\n pretrained_kwargs = pretrained_kwargs or {}\n transformer, _ = load_pretrained_hf_model_with_hub_fallback(\n MT5EncoderModel, pretrained_model_name_or_path, **pretrained_kwargs\n )\n else:\n transformer = self._init_transformer_from_scratch(MT5EncoderModel, MT5Config, hf_config_params, vocab_size)\n\n if encoder_config is not None:\n self.config = self._init_config(transformer, hf_config_params.keys(), encoder_config)\n else:\n self.config = None\n\n self.reduce_output = reduce_output\n if reduce_output == \"cls_pooled\":\n _cls_pooled_error_message(self.__class__.__name__)\n self.reduce_sequence = SequenceReducer(reduce_mode=reduce_output)\n self.transformer = self._wrap_transformer(transformer, adapter, trainable)\n self.max_sequence_length = max_sequence_length\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n if mask is not None:\n mask = mask.to(torch.int32)\n transformer_outputs = self.transformer.module(\n input_ids=inputs,\n attention_mask=mask,\n )\n hidden = transformer_outputs[0][:, 1:-1, :]\n hidden = self.reduce_sequence(hidden, self.reduce_output)\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return MT5Config\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is None:\n # Subtract 2 to remove CLS and PAD tokens added by MT5 tokenizer.\n return torch.Size(\n [\n self.max_sequence_length - 2,\n self.transformer.module.config.hidden_size,\n ]\n )\n elif self.reduce_output == \"concat\":\n # add the -2 to account of start and end tokens.\n return torch.Size([self.transformer.module.config.hidden_size * (self.max_sequence_length - 2)])\n return torch.Size([self.transformer.module.config.hidden_size])\n\n @property\n def input_dtype(self) -> torch.dtype:\n return torch.int32\n\n\n@DeveloperAPI\n@register_encoder(\"xlmroberta\", TEXT)\nclass XLMRoBERTaEncoder(HFTextEncoder):\n DEFAULT_MODEL_NAME = \"xlm-roberta-base\"\n\n def __init__(\n self,\n max_sequence_length: int,\n use_pretrained: bool = True,\n pretrained_model_name_or_path: str = DEFAULT_MODEL_NAME,\n saved_weights_in_checkpoint: bool = False,\n reduce_output: str = \"cls_pooled\",\n trainable: bool = False,\n adapter: BaseAdapterConfig | None = None,\n vocab_size: int = None,\n pad_token_id: int = 1,\n bos_token_id: int = 0,\n eos_token_id: int = 2,\n max_position_embeddings: int = 514,\n type_vocab_size: int = 1,\n add_pooling_layer: bool = True,\n pretrained_kwargs: dict = None,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n\n from transformers import XLMRobertaConfig, XLMRobertaModel\n\n hf_config_params = dict(\n pad_token_id=pad_token_id,\n bos_token_id=bos_token_id,\n eos_token_id=eos_token_id,\n max_position_embeddings=max_position_embeddings,\n type_vocab_size=type_vocab_size,\n )\n\n if use_pretrained and not saved_weights_in_checkpoint:\n pretrained_kwargs = pretrained_kwargs or {}\n transformer, _ = load_pretrained_hf_model_with_hub_fallback(\n XLMRobertaModel, pretrained_model_name_or_path, **pretrained_kwargs\n )\n else:\n transformer = self._init_transformer_from_scratch(\n XLMRobertaModel, XLMRobertaConfig, hf_config_params, vocab_size\n )\n\n if encoder_config is not None:\n self.config = self._init_config(transformer, hf_config_params.keys(), encoder_config)\n else:\n self.config = None\n\n self.reduce_output = reduce_output\n if not self.reduce_output == \"cls_pooled\":\n self.reduce_sequence = SequenceReducer(reduce_mode=reduce_output)\n self.transformer = self._wrap_transformer(transformer, adapter, trainable)\n self.max_sequence_length = max_sequence_length\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n if mask is not None:\n mask = mask.to(torch.int32)\n transformer_outputs = self.transformer.module(\n input_ids=inputs,\n attention_mask=mask,\n token_type_ids=torch.zeros_like(inputs),\n )\n if self.reduce_output == \"cls_pooled\":\n hidden = transformer_outputs[1]\n else:\n hidden = transformer_outputs[0][:, 1:-1, :]\n hidden = self.reduce_sequence(hidden, self.reduce_output)\n\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return XLMRoBERTaConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is None:\n # Subtract 2 to remove CLS and PAD tokens added by XLMRoberta tokenizer.\n return torch.Size(\n [\n self.max_sequence_length - 2,\n self.transformer.module.config.hidden_size,\n ]\n )\n elif self.reduce_output == \"concat\":\n # add the -2 to account of start and end tokens.\n return torch.Size([self.transformer.module.config.hidden_size * (self.max_sequence_length - 2)])\n return torch.Size([self.transformer.module.config.hidden_size])\n\n @property\n def input_dtype(self) -> torch.dtype:\n return torch.int32\n\n\n@DeveloperAPI\n@register_encoder(\"bert\", TEXT)\nclass BERTEncoder(HFTextEncoder):\n DEFAULT_MODEL_NAME = \"bert-base-uncased\"\n\n def __init__(\n self,\n max_sequence_length: int,\n use_pretrained: bool = True,\n pretrained_model_name_or_path: str = DEFAULT_MODEL_NAME,\n saved_weights_in_checkpoint: bool = False,\n trainable: bool = False,\n adapter: BaseAdapterConfig | None = None,\n reduce_output: str = \"cls_pooled\",\n vocab_size: int = 30522,\n hidden_size: int = 768,\n num_hidden_layers: int = 12,\n num_attention_heads: int = 12,\n intermediate_size: int = 3072,\n hidden_act: str | Callable = \"gelu\",\n hidden_dropout_prob: float = 0.1,\n attention_probs_dropout_prob: float = 0.1,\n max_position_embeddings: int = 512,\n type_vocab_size: int = 2,\n initializer_range: float = 0.02,\n layer_norm_eps: float = 1e-12,\n pad_token_id: int = 0,\n gradient_checkpointing: bool = False,\n position_embedding_type: str = \"absolute\",\n classifier_dropout: float = None,\n pretrained_kwargs: dict = None,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n\n from transformers import BertConfig, BertModel\n\n hf_config_params = dict(\n vocab_size=vocab_size,\n hidden_size=hidden_size,\n num_hidden_layers=num_hidden_layers,\n num_attention_heads=num_attention_heads,\n intermediate_size=intermediate_size,\n hidden_act=hidden_act,\n hidden_dropout_prob=hidden_dropout_prob,\n attention_probs_dropout_prob=attention_probs_dropout_prob,\n max_position_embeddings=max_position_embeddings,\n type_vocab_size=type_vocab_size,\n initializer_range=initializer_range,\n layer_norm_eps=layer_norm_eps,\n pad_token_id=pad_token_id,\n gradient_checkpointing=gradient_checkpointing,\n position_embedding_type=position_embedding_type,\n classifier_dropout=classifier_dropout,\n )\n\n if use_pretrained and not saved_weights_in_checkpoint:\n pretrained_kwargs = pretrained_kwargs or {}\n transformer, _ = load_pretrained_hf_model_with_hub_fallback(\n BertModel, pretrained_model_name_or_path, **pretrained_kwargs\n )\n else:\n transformer = self._init_transformer_from_scratch(BertModel, BertConfig, hf_config_params, vocab_size)\n\n if encoder_config is not None:\n self.config = self._init_config(transformer, hf_config_params.keys(), encoder_config)\n else:\n self.config = None\n\n self.reduce_output = reduce_output\n if not self.reduce_output == \"cls_pooled\":\n self.reduce_sequence = SequenceReducer(reduce_mode=reduce_output)\n\n self.transformer = self._wrap_transformer(transformer, adapter, trainable)\n\n self.max_sequence_length = max_sequence_length\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n if mask is not None:\n mask = mask.to(torch.int32)\n transformer_outputs = self.transformer.module(\n input_ids=inputs,\n attention_mask=mask,\n token_type_ids=torch.zeros_like(inputs),\n )\n if self.reduce_output == \"cls_pooled\":\n hidden = transformer_outputs[1]\n else:\n hidden = transformer_outputs[0][:, 1:-1, :]\n hidden = self.reduce_sequence(hidden, self.reduce_output)\n\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return BERTConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n # TODO(shreya): Confirm that this is it\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is None:\n # Subtract 2 to remove CLS and PAD tokens added by BERT tokenizer.\n return torch.Size(\n [\n self.max_sequence_length - 2,\n self.transformer.module.config.hidden_size,\n ]\n )\n elif self.reduce_output == \"concat\":\n # add the -2 to account of start and end tokens.\n return torch.Size([self.transformer.module.config.hidden_size * (self.max_sequence_length - 2)])\n return torch.Size([self.transformer.module.config.hidden_size])\n\n @property\n def input_dtype(self) -> torch.dtype:\n return torch.int32\n\n\n@DeveloperAPI\n@register_encoder(\"xlm\", TEXT)\nclass XLMEncoder(HFTextEncoder):\n DEFAULT_MODEL_NAME = \"xlm-mlm-en-2048\"\n\n def __init__(\n self,\n max_sequence_length: int,\n use_pretrained: bool = True,\n pretrained_model_name_or_path: str = DEFAULT_MODEL_NAME,\n saved_weights_in_checkpoint: bool = False,\n trainable: bool = False,\n adapter: BaseAdapterConfig | None = None,\n reduce_output: str = \"sum\",\n vocab_size: int = 30145,\n emb_dim: int = 2048,\n n_layers: int = 12,\n n_heads: int = 16,\n dropout: float = 0.1,\n attention_dropout: float = 0.1,\n gelu_activation: bool = True,\n sinusoidal_embeddings: bool = False,\n causal: bool = False,\n asm: bool = False,\n n_langs: int = 1,\n use_lang_emb: bool = True,\n max_position_embeddings: int = 512,\n embed_init_std: float = 2048**-0.5,\n layer_norm_eps: float = 1e-12,\n init_std: float = 0.02,\n bos_index: int = 0,\n eos_index: int = 1,\n pad_index: int = 2,\n unk_index: int = 3,\n mask_index: int = 5,\n is_encoder: bool = True,\n start_n_top: int = 5,\n end_n_top: int = 5,\n mask_token_id: int = 0,\n lang_id: int = 0,\n pad_token_id: int = 2,\n bos_token_id: int = 0,\n pretrained_kwargs: dict = None,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n\n from transformers import XLMConfig, XLMModel\n\n hf_config_params = dict(\n vocab_size=vocab_size,\n emb_dim=emb_dim,\n n_layers=n_layers,\n n_heads=n_heads,\n dropout=dropout,\n attention_dropout=attention_dropout,\n gelu_activation=gelu_activation,\n sinusoidal_embeddings=sinusoidal_embeddings,\n causal=causal,\n asm=asm,\n n_langs=n_langs,\n use_lang_emb=use_lang_emb,\n max_position_embeddings=max_position_embeddings,\n embed_init_std=embed_init_std,\n layer_norm_eps=layer_norm_eps,\n init_std=init_std,\n bos_index=bos_index,\n eos_index=eos_index,\n pad_index=pad_index,\n unk_index=unk_index,\n mask_index=mask_index,\n is_encoder=is_encoder,\n start_n_top=start_n_top,\n end_n_top=end_n_top,\n mask_token_id=mask_token_id,\n lang_id=lang_id,\n pad_token_id=pad_token_id,\n bos_token_id=bos_token_id,\n )\n\n if use_pretrained and not saved_weights_in_checkpoint:\n pretrained_kwargs = pretrained_kwargs or {}\n transformer, _ = load_pretrained_hf_model_with_hub_fallback(\n XLMModel, pretrained_model_name_or_path, **pretrained_kwargs\n )\n else:\n transformer = self._init_transformer_from_scratch(XLMModel, XLMConfig, hf_config_params, vocab_size)\n\n self.config = self._init_config(transformer, hf_config_params, encoder_config)\n\n self.transformer = self._wrap_transformer(transformer, adapter, trainable)\n self.reduce_output = reduce_output\n if self.reduce_output == \"cls_pooled\":\n _cls_pooled_error_message(self.__class__.__name__)\n self.reduce_sequence = SequenceReducer(reduce_mode=reduce_output)\n self.max_sequence_length = max_sequence_length\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n if mask is not None:\n mask = mask.to(torch.int32)\n transformer_outputs = self.transformer.module(\n input_ids=inputs,\n attention_mask=mask,\n token_type_ids=torch.zeros_like(inputs),\n )\n hidden = transformer_outputs[0]\n hidden = self.reduce_sequence(hidden, self.reduce_output)\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return XLMConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n # TODO(shreya): Confirm that this is it\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is None:\n # Subtract 2 to remove CLS and PAD tokens added by BERT tokenizer.\n return torch.Size(\n [\n self.max_sequence_length - 2,\n self.transformer.module.config.hidden_size,\n ]\n )\n elif self.reduce_output == \"concat\":\n # add the -2 to account of start and end tokens.\n return torch.Size([self.transformer.module.config.hidden_size * (self.max_sequence_length - 2)])\n return torch.Size([self.transformer.module.config.hidden_size])\n\n @property\n def input_dtype(self) -> torch.dtype:\n return torch.int32\n\n\n@DeveloperAPI\n@register_encoder(\"gpt\", TEXT)\nclass GPTEncoder(HFTextEncoder):\n DEFAULT_MODEL_NAME = \"openai-gpt\"\n\n def __init__(\n self,\n max_sequence_length: int,\n reduce_output: str = \"sum\",\n use_pretrained: bool = True,\n pretrained_model_name_or_path: str = DEFAULT_MODEL_NAME,\n saved_weights_in_checkpoint: bool = False,\n trainable: bool = False,\n adapter: BaseAdapterConfig | None = None,\n vocab_size: int = 30522,\n n_positions: int = 40478,\n n_ctx: int = 512,\n n_embd: int = 768,\n n_layer: int = 12,\n n_head: int = 12,\n afn: str = \"gelu\",\n resid_pdrop: float = 0.1,\n embd_pdrop: float = 0.1,\n attn_pdrop: float = 0.1,\n layer_norm_epsilon: float = 1e-5,\n initializer_range: float = 0.02,\n pretrained_kwargs: dict = None,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n\n from transformers import OpenAIGPTConfig, OpenAIGPTModel\n\n hf_config_params = dict(\n vocab_size=vocab_size,\n n_positions=n_positions,\n n_ctx=n_ctx,\n n_embd=n_embd,\n n_layer=n_layer,\n n_head=n_head,\n afn=afn,\n resid_pdrop=resid_pdrop,\n embd_pdrop=embd_pdrop,\n attn_pdrop=attn_pdrop,\n layer_norm_epsilon=layer_norm_epsilon,\n initializer_range=initializer_range,\n )\n\n if use_pretrained and not saved_weights_in_checkpoint:\n pretrained_kwargs = pretrained_kwargs or {}\n transformer, _ = load_pretrained_hf_model_with_hub_fallback(\n OpenAIGPTModel, pretrained_model_name_or_path, **pretrained_kwargs\n )\n else:\n transformer = self._init_transformer_from_scratch(\n OpenAIGPTModel, OpenAIGPTConfig, hf_config_params, vocab_size\n )\n\n if encoder_config is not None:\n self.config = self._init_config(transformer, hf_config_params.keys(), encoder_config)\n else:\n self.config = None\n\n self.reduce_output = reduce_output\n if self.reduce_output == \"cls_pooled\":\n _cls_pooled_error_message(self.__class__.__name__)\n self.reduce_sequence = SequenceReducer(reduce_mode=reduce_output)\n self.transformer = self._wrap_transformer(transformer, adapter, trainable)\n self.max_sequence_length = max_sequence_length\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n if mask is not None:\n mask = mask.to(torch.int32)\n transformer_outputs = self.transformer.module(\n input_ids=inputs,\n attention_mask=mask,\n token_type_ids=torch.zeros_like(inputs),\n )\n hidden = transformer_outputs[0]\n hidden = self.reduce_sequence(hidden, self.reduce_output)\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return GPTConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is None:\n return torch.Size([self.max_sequence_length, self.transformer.module.config.hidden_size])\n elif self.reduce_output == \"concat\":\n return torch.Size([self.transformer.module.config.hidden_size * self.max_sequence_length])\n return torch.Size([self.transformer.module.config.hidden_size])\n\n @property\n def input_dtype(self) -> torch.dtype:\n return torch.int32\n\n\n@DeveloperAPI\n@register_encoder(\"gpt2\", TEXT)\nclass GPT2Encoder(HFTextEncoder):\n DEFAULT_MODEL_NAME = \"gpt2\"\n\n def __init__(\n self,\n max_sequence_length: int,\n use_pretrained: bool = True,\n pretrained_model_name_or_path: str = DEFAULT_MODEL_NAME,\n reduce_output: str = \"sum\",\n trainable: bool = False,\n adapter: BaseAdapterConfig | None = None,\n vocab_size: int = 50257,\n n_positions: int = 1024,\n n_ctx: int = 1024,\n n_embd: int = 768,\n n_layer: int = 12,\n n_head: int = 12,\n n_inner: int | None = None,\n activation_function: str = \"gelu\",\n resid_pdrop: float = 0.1,\n embd_pdrop: float = 0.1,\n attn_pdrop: float = 0.1,\n layer_norm_epsilon: float = 1e-5,\n initializer_range: float = 0.02,\n scale_attn_weights: bool = True,\n pretrained_kwargs: dict = None,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n\n from transformers import GPT2Config, GPT2Model\n\n hf_config_params = dict(\n vocab_size=vocab_size,\n n_positions=n_positions,\n n_ctx=n_ctx,\n n_embd=n_embd,\n n_layer=n_layer,\n n_head=n_head,\n n_inner=n_inner,\n activation_function=activation_function,\n resid_pdrop=resid_pdrop,\n embd_pdrop=embd_pdrop,\n attn_pdrop=attn_pdrop,\n layer_norm_epsilon=layer_norm_epsilon,\n initializer_range=initializer_range,\n scale_attn_weights=scale_attn_weights,\n )\n\n if use_pretrained:\n pretrained_kwargs = pretrained_kwargs or {}\n transformer, _ = load_pretrained_hf_model_with_hub_fallback(\n GPT2Model, pretrained_model_name_or_path, **pretrained_kwargs\n )\n else:\n transformer = self._init_transformer_from_scratch(GPT2Model, GPT2Config, hf_config_params, vocab_size)\n\n if encoder_config is not None:\n self.config = self._init_config(transformer, hf_config_params.keys(), encoder_config)\n else:\n self.config = None\n\n self.transformer = self._wrap_transformer(transformer, adapter, trainable)\n self.max_sequence_length = max_sequence_length\n self.reduce_output = reduce_output\n if self.reduce_output == \"cls_pooled\":\n _cls_pooled_error_message(self.__class__.__name__)\n self.reduce_sequence = SequenceReducer(reduce_mode=reduce_output)\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n if mask is not None:\n mask = mask.to(torch.int32)\n transformer_outputs = self.transformer.module(\n input_ids=inputs,\n attention_mask=mask,\n token_type_ids=torch.zeros_like(inputs),\n )\n hidden = transformer_outputs[0]\n hidden = self.reduce_sequence(hidden, self.reduce_output)\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return GPT2Config\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is None:\n return torch.Size([self.max_sequence_length, self.transformer.module.config.hidden_size])\n elif self.reduce_output == \"concat\":\n return torch.Size([self.transformer.module.config.hidden_size * (self.max_sequence_length)])\n return torch.Size([self.transformer.module.config.hidden_size])\n\n @property\n def input_dtype(self) -> torch.dtype:\n return torch.int32\n\n\n@DeveloperAPI\n@register_encoder(\"deberta\", TEXT)\nclass DeBERTaEncoder(HFTextEncoderImpl):\n def __init__(self, *args, **kwargs):\n from transformers import DebertaV2Config as _DebertaV2Config\n from transformers import DebertaV2Model\n\n super().__init__(DebertaV2Model, _DebertaV2Config, DebertaV2Config, *args, **kwargs)\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return DebertaV2Config\n\n\n@DeveloperAPI\n@register_encoder(\"roberta\", TEXT)\nclass RoBERTaEncoder(HFTextEncoder):\n DEFAULT_MODEL_NAME = \"roberta-base\"\n\n def __init__(\n self,\n max_sequence_length,\n use_pretrained: bool = True,\n pretrained_model_name_or_path: str = DEFAULT_MODEL_NAME,\n saved_weights_in_checkpoint: bool = False,\n reduce_output: str = \"cls_pooled\",\n trainable: bool = False,\n adapter: BaseAdapterConfig | None = None,\n vocab_size: int = None,\n pad_token_id: int = 1,\n bos_token_id: int = 0,\n eos_token_id: int = 2,\n max_position_embeddings: int = 514,\n type_vocab_size: int = 1,\n pretrained_kwargs: dict = None,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n\n from transformers import RobertaConfig, RobertaModel\n\n hf_config_params = dict(\n pad_token_id=pad_token_id,\n bos_token_id=bos_token_id,\n eos_token_id=eos_token_id,\n max_position_embeddings=max_position_embeddings,\n type_vocab_size=type_vocab_size,\n )\n\n if use_pretrained and not saved_weights_in_checkpoint:\n pretrained_kwargs = pretrained_kwargs or {}\n transformer, _ = load_pretrained_hf_model_with_hub_fallback(\n RobertaModel, pretrained_model_name_or_path, **pretrained_kwargs\n )\n else:\n transformer = self._init_transformer_from_scratch(RobertaModel, RobertaConfig, hf_config_params, vocab_size)\n\n if encoder_config is not None:\n self.config = self._init_config(transformer, hf_config_params.keys(), encoder_config)\n else:\n self.config = None\n\n self.transformer = self._wrap_transformer(transformer, adapter, trainable)\n self.max_sequence_length = max_sequence_length\n self.reduce_output = reduce_output\n if not self.reduce_output == \"cls_pooled\":\n self.reduce_sequence = SequenceReducer(reduce_mode=reduce_output)\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n if mask is not None:\n mask = mask.to(torch.int32)\n transformer_outputs = self.transformer.module(\n input_ids=inputs,\n attention_mask=mask,\n token_type_ids=torch.zeros_like(inputs),\n )\n if self.reduce_output == \"cls_pooled\":\n hidden = transformer_outputs[1]\n else:\n hidden = transformer_outputs[0][:, 1:-1, :] # bos + [sent] + sep\n hidden = self.reduce_sequence(hidden, self.reduce_output)\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return RoBERTaConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is None:\n return torch.Size([self.max_sequence_length - 2, self.transformer.module.config.hidden_size])\n elif self.reduce_output == \"concat\":\n # add the -2 to account of start and end tokens.\n return torch.Size([self.transformer.module.config.hidden_size * (self.max_sequence_length - 2)])\n return torch.Size([self.transformer.module.config.hidden_size])\n\n @property\n def input_dtype(self) -> torch.dtype:\n return torch.int32\n\n\n@DeveloperAPI\n@register_encoder(\"transformer_xl\", TEXT)\nclass TransformerXLEncoder(HFTextEncoder):\n DEFAULT_MODEL_NAME = \"transfo-xl-wt103\"\n\n def __init__(\n self,\n max_sequence_length: int,\n use_pretrained: bool = True,\n pretrained_model_name_or_path: str = DEFAULT_MODEL_NAME,\n saved_weights_in_checkpoint: bool = False,\n reduce_output: str = \"sum\",\n trainable: bool = False,\n adapter: BaseAdapterConfig | None = None,\n vocab_size: int = 267735,\n cutoffs: list[int] = [20000, 40000, 200000],\n d_model: int = 1024,\n d_embed: int = 1024,\n n_head: int = 16,\n d_head: int = 64,\n d_inner: int = 4096,\n div_val: int = 4,\n pre_lnorm: bool = False,\n n_layer: int = 18,\n mem_len: int = 1600,\n clamp_len: int = 1000,\n same_length: bool = True,\n proj_share_all_but_first: bool = True,\n attn_type: int = 0,\n sample_softmax: int = -1,\n adaptive: bool = True,\n dropout: float = 0.1,\n dropatt: float = 0.0,\n untie_r: bool = True,\n init: str = \"normal\",\n init_range: float = 0.01,\n proj_init_std: float = 0.01,\n init_std: float = 0.02,\n layer_norm_epsilon: float = 1e-5,\n eos_token_id: int = 0,\n pretrained_kwargs: dict = None,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n\n from transformers import TransfoXLConfig, TransfoXLModel\n\n hf_config_params = dict(\n vocab_size=vocab_size,\n cutoffs=cutoffs,\n d_model=d_model,\n d_embed=d_embed,\n n_head=n_head,\n d_head=d_head,\n d_inner=d_inner,\n div_val=div_val,\n pre_lnorm=pre_lnorm,\n n_layer=n_layer,\n mem_len=mem_len,\n clamp_len=clamp_len,\n same_length=same_length,\n proj_share_all_but_first=proj_share_all_but_first,\n attn_type=attn_type,\n sample_softmax=sample_softmax,\n adaptive=adaptive,\n dropout=dropout,\n dropatt=dropatt,\n untie_r=untie_r,\n init=init,\n init_range=init_range,\n proj_init_std=proj_init_std,\n init_std=init_std,\n layer_norm_epsilon=layer_norm_epsilon,\n eos_token_id=eos_token_id,\n )\n\n if use_pretrained and not saved_weights_in_checkpoint:\n pretrained_kwargs = pretrained_kwargs or {}\n transformer, _ = load_pretrained_hf_model_with_hub_fallback(\n TransfoXLModel, pretrained_model_name_or_path, **pretrained_kwargs\n )\n else:\n config = TransfoXLConfig(**hf_config_params)\n transformer = TransfoXLModel(config)\n\n if encoder_config is not None:\n self.config = self._init_config(transformer, hf_config_params.keys(), encoder_config)\n else:\n self.config = None\n\n self.reduce_output = reduce_output\n if self.reduce_output == \"cls_pooled\":\n _cls_pooled_error_message(self.__class__.__name__)\n self.reduce_sequence = SequenceReducer(reduce_mode=reduce_output)\n self.transformer = self._wrap_transformer(transformer, adapter, trainable)\n self.max_sequence_length = max_sequence_length\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n transformer_outputs = self.transformer.module(inputs)\n hidden = transformer_outputs[0]\n hidden = self.reduce_sequence(hidden, self.reduce_output)\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TransformerXLConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is None:\n return torch.Size([self.max_sequence_length, self.transformer.module.config.d_model])\n elif self.reduce_output == \"concat\":\n # add the -2 to account of start and end tokens.\n return torch.Size([self.transformer.module.config.d_model * self.max_sequence_length])\n return torch.Size([self.transformer.module.config.d_model])\n\n @property\n def input_dtype(self) -> torch.dtype:\n return torch.int32\n\n\n@DeveloperAPI\n@register_encoder(\"xlnet\", TEXT)\nclass XLNetEncoder(HFTextEncoder):\n DEFAULT_MODEL_NAME = \"xlnet-base-cased\"\n\n def __init__(\n self,\n max_sequence_length: int,\n use_pretrained: bool = True,\n pretrained_model_name_or_path: str = DEFAULT_MODEL_NAME,\n saved_weights_in_checkpoint: bool = False,\n reduce_output: str = \"sum\",\n trainable: bool = False,\n adapter: BaseAdapterConfig | None = None,\n vocab_size: int = 32000,\n d_model: int = 1024,\n n_layer: int = 24,\n n_head: int = 16,\n d_inner: int = 4096,\n ff_activation: str = \"gelu\",\n untie_r: bool = True,\n attn_type: str = \"bi\",\n initializer_range: float = 0.02,\n layer_norm_eps: float = 1e-12,\n dropout: float = 0.1,\n mem_len: int | None = 512,\n reuse_len: int | None = None,\n use_mems_eval: bool = True,\n use_mems_train: bool = False,\n bi_data: bool = False,\n clamp_len: int = -1,\n same_length: bool = False,\n summary_type: str = \"last\",\n summary_use_proj: bool = True,\n summary_activation: str = \"tanh\",\n summary_last_dropout: float = 0.1,\n start_n_top: int = 5,\n end_n_top: int = 5,\n pad_token_id: int = 5,\n bos_token_id: int = 1,\n eos_token_id: int = 2,\n pretrained_kwargs: dict = None,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n\n from transformers import XLNetConfig, XLNetModel\n\n hf_config_params = dict(\n vocab_size=vocab_size,\n d_model=d_model,\n n_layer=n_layer,\n n_head=n_head,\n d_inner=d_inner,\n ff_activation=ff_activation,\n untie_r=untie_r,\n attn_type=attn_type,\n initializer_range=initializer_range,\n layer_norm_eps=layer_norm_eps,\n dropout=dropout,\n mem_len=mem_len,\n reuse_len=reuse_len,\n use_mems_eval=use_mems_eval,\n use_mems_train=use_mems_train,\n bi_data=bi_data,\n clamp_len=clamp_len,\n same_length=same_length,\n summary_type=summary_type,\n summary_use_proj=summary_use_proj,\n summary_activation=summary_activation,\n summary_last_dropout=summary_last_dropout,\n start_n_top=start_n_top,\n end_n_top=end_n_top,\n pad_token_id=pad_token_id,\n bos_token_id=bos_token_id,\n eos_token_id=eos_token_id,\n )\n\n if use_pretrained and not saved_weights_in_checkpoint:\n pretrained_kwargs = pretrained_kwargs or {}\n transformer, _ = load_pretrained_hf_model_with_hub_fallback(\n XLNetModel, pretrained_model_name_or_path, **pretrained_kwargs\n )\n else:\n transformer = self._init_transformer_from_scratch(XLNetModel, XLNetConfig, hf_config_params, vocab_size)\n\n if encoder_config is not None:\n self.config = self._init_config(transformer, hf_config_params.keys(), encoder_config)\n else:\n self.config = None\n\n self.max_sequence_length = max_sequence_length\n self.reduce_output = reduce_output\n if self.reduce_output == \"cls_pooled\":\n _cls_pooled_error_message(self.__class__.__name__)\n self.reduce_sequence = SequenceReducer(reduce_mode=reduce_output)\n self.transformer = self._wrap_transformer(transformer, adapter, trainable)\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n if mask is not None:\n mask = mask.to(torch.int32)\n transformer_outputs = self.transformer.module(\n input_ids=inputs,\n attention_mask=mask,\n token_type_ids=torch.zeros_like(inputs),\n )\n hidden = transformer_outputs[0]\n hidden = self.reduce_sequence(hidden, self.reduce_output)\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return XLNetConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is None:\n return torch.Size([self.max_sequence_length, self.transformer.module.config.d_model])\n elif self.reduce_output == \"concat\":\n return torch.Size([self.transformer.module.config.d_model * self.max_sequence_length])\n return torch.Size([self.transformer.module.config.d_model])\n\n @property\n def input_dtype(self) -> torch.dtype:\n return torch.int32\n\n\n@DeveloperAPI\n@register_encoder(\"distilbert\", TEXT)\nclass DistilBERTEncoder(HFTextEncoder):\n DEFAULT_MODEL_NAME = \"distilbert-base-uncased\"\n\n def __init__(\n self,\n max_sequence_length: int,\n pretrained_model_name_or_path: str = DEFAULT_MODEL_NAME,\n saved_weights_in_checkpoint: bool = False,\n reduce_output: str = \"sum\",\n trainable: bool = False,\n adapter: BaseAdapterConfig | None = None,\n use_pretrained: bool = True,\n vocab_size: int = 30522,\n max_position_embeddings: int = 512,\n sinusoidal_pos_embds: bool = False,\n n_layers: int = 6,\n n_heads: int = 12,\n dim: int = 768,\n hidden_dim: int = 3072,\n dropout: float = 0.1,\n attention_dropout: float = 0.1,\n activation: str | Callable = \"gelu\",\n initializer_range: float = 0.02,\n qa_dropout: float = 0.1,\n seq_classif_dropout: float = 0.2,\n pretrained_kwargs: dict = None,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n\n from transformers import DistilBertConfig, DistilBertModel\n\n hf_config_params = dict(\n vocab_size=vocab_size,\n max_position_embeddings=max_position_embeddings,\n sinusoidal_pos_embds=sinusoidal_pos_embds,\n n_layers=n_layers,\n n_heads=n_heads,\n dim=dim,\n hidden_dim=hidden_dim,\n dropout=dropout,\n attention_dropout=attention_dropout,\n activation=activation,\n initializer_range=initializer_range,\n qa_dropout=qa_dropout,\n seq_classif_dropout=seq_classif_dropout,\n )\n\n if use_pretrained and not saved_weights_in_checkpoint:\n pretrained_kwargs = pretrained_kwargs or {}\n transformer, _ = load_pretrained_hf_model_with_hub_fallback(\n DistilBertModel, pretrained_model_name_or_path, **pretrained_kwargs\n )\n else:\n transformer = self._init_transformer_from_scratch(\n DistilBertModel, DistilBertConfig, hf_config_params, vocab_size\n )\n\n if encoder_config is not None:\n self.config = self._init_config(transformer, hf_config_params.keys(), encoder_config)\n else:\n self.config = None\n\n self.transformer = self._wrap_transformer(transformer, adapter, trainable)\n self.reduce_output = reduce_output\n if self.reduce_output == \"cls_pooled\":\n _cls_pooled_error_message(self.__class__.__name__)\n self.max_sequence_length = max_sequence_length\n self.reduce_sequence = SequenceReducer(reduce_mode=reduce_output)\n self.last_inputs = None\n self.last_hidden = None\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n if mask is not None:\n mask = mask.to(torch.int32)\n\n transformer_outputs = self.transformer.module(\n input_ids=inputs,\n attention_mask=mask,\n )\n hidden = transformer_outputs[0][:, 1:-1, :]\n self.last_inputs = inputs\n self.last_hidden = hidden\n hidden = self.reduce_sequence(hidden, self.reduce_output)\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return DistilBERTConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is None:\n # Subtract 2 to remove CLS and PAD tokens added by BERT tokenizer.\n return torch.Size([self.max_sequence_length - 2, self.transformer.module.config.dim])\n elif self.reduce_output == \"concat\":\n # add the -2 to account of start and end tokens.\n return torch.Size([self.transformer.module.config.dim * (self.max_sequence_length - 2)])\n return torch.Size([self.transformer.module.config.dim])\n\n @property\n def input_dtype(self) -> torch.dtype:\n return torch.int32\n\n\n@DeveloperAPI\n@register_encoder(\"ctrl\", TEXT)\nclass CTRLEncoder(HFTextEncoder):\n DEFAULT_MODEL_NAME = \"ctrl\"\n\n def __init__(\n self,\n max_sequence_length: int,\n use_pretrained: bool = True,\n pretrained_model_name_or_path: str = DEFAULT_MODEL_NAME,\n saved_weights_in_checkpoint: bool = False,\n reduce_output: str = \"sum\",\n trainable: bool = False,\n adapter: BaseAdapterConfig | None = None,\n vocab_size: int = 246534,\n n_positions: int = 256,\n n_ctx: int = 256,\n n_embd: int = 1280,\n dff: int = 8192,\n n_layer: int = 48,\n n_head: int = 16,\n resid_pdrop: float = 0.1,\n embd_pdrop: float = 0.1,\n attn_pdrop: float = 0.1,\n layer_norm_epsilon: float = 1e-6,\n initializer_range: float = 0.02,\n pretrained_kwargs: dict = None,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n\n from transformers import CTRLConfig, CTRLModel\n\n hf_config_params = dict(\n vocab_size=vocab_size,\n n_positions=n_positions,\n n_ctx=n_ctx,\n n_embd=n_embd,\n dff=dff,\n n_layer=n_layer,\n n_head=n_head,\n resid_pdrop=resid_pdrop,\n embd_pdrop=embd_pdrop,\n attn_pdrop=attn_pdrop,\n layer_norm_epsilon=layer_norm_epsilon,\n initializer_range=initializer_range,\n )\n\n if use_pretrained and not saved_weights_in_checkpoint:\n pretrained_kwargs = pretrained_kwargs or {}\n transformer, _ = load_pretrained_hf_model_with_hub_fallback(\n CTRLModel, pretrained_model_name_or_path, **pretrained_kwargs\n )\n self.vocab_size = transformer.config.vocab_size\n else:\n transformer = self._init_transformer_from_scratch(CTRLModel, CTRLConfig, hf_config_params, vocab_size)\n self.vocab_size = vocab_size\n\n if encoder_config is not None:\n self.config = self._init_config(transformer, hf_config_params.keys(), encoder_config)\n else:\n self.config = None\n\n self.max_sequence_length = max_sequence_length\n self.transformer = self._wrap_transformer(transformer, adapter, trainable)\n self.reduce_output = reduce_output\n if self.reduce_output == \"cls_pooled\":\n _cls_pooled_error_message(self.__class__.__name__)\n self.reduce_sequence = SequenceReducer(reduce_mode=reduce_output)\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n if mask is not None:\n mask = mask.to(torch.int32)\n transformer_outputs = self.transformer.module(\n input_ids=inputs,\n attention_mask=mask,\n token_type_ids=torch.zeros_like(inputs),\n )\n hidden = transformer_outputs[0]\n hidden = self.reduce_sequence(hidden, self.reduce_output)\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls():\n return CTRLConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is None:\n return torch.Size([self.max_sequence_length, self.transformer.module.config.n_embd])\n elif self.reduce_output == \"concat\":\n # add the -2 to account of start and end tokens.\n return torch.Size([self.transformer.module.config.n_embd * (self.max_sequence_length - 2)])\n return torch.Size([self.transformer.module.config.n_embd])\n\n @property\n def input_dtype(self) -> torch.dtype:\n return torch.int32\n\n\n@DeveloperAPI\n@register_encoder(\"camembert\", TEXT)\nclass CamemBERTEncoder(HFTextEncoder):\n DEFAULT_MODEL_NAME = \"camembert-base\"\n\n def __init__(\n self,\n max_sequence_length: int,\n use_pretrained: bool = True,\n pretrained_model_name_or_path: str = DEFAULT_MODEL_NAME,\n saved_weights_in_checkpoint: bool = False,\n reduce_output: str = \"cls-pooled\",\n trainable: bool = False,\n adapter: BaseAdapterConfig | None = None,\n vocab_size: int = 30522,\n hidden_size: int = 768,\n num_hidden_layers: int = 12,\n num_attention_heads: int = 12,\n intermediate_size: int = 3072,\n hidden_act: str | Callable = \"gelu\",\n hidden_dropout_prob: float = 0.1,\n attention_probs_dropout_prob: float = 0.1,\n max_position_embeddings: int = 512,\n type_vocab_size: int = 2,\n initializer_range: float = 0.02,\n layer_norm_eps: float = 1e-12,\n pad_token_id: int = 0,\n gradient_checkpointing: bool = False,\n position_embedding_type: str = \"absolute\",\n classifier_dropout: float = None,\n pretrained_kwargs: dict = None,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n\n from transformers import CamembertConfig, CamembertModel\n\n hf_config_params = dict(\n vocab_size=vocab_size,\n hidden_size=hidden_size,\n num_hidden_layers=num_hidden_layers,\n num_attention_heads=num_attention_heads,\n intermediate_size=intermediate_size,\n hidden_act=hidden_act,\n hidden_dropout_prob=hidden_dropout_prob,\n attention_probs_dropout_prob=attention_probs_dropout_prob,\n max_position_embeddings=max_position_embeddings,\n type_vocab_size=type_vocab_size,\n initializer_range=initializer_range,\n layer_norm_eps=layer_norm_eps,\n pad_token_id=pad_token_id,\n gradient_checkpointing=gradient_checkpointing,\n position_embedding_type=position_embedding_type,\n classifier_dropout=classifier_dropout,\n )\n\n if use_pretrained and not saved_weights_in_checkpoint:\n pretrained_kwargs = pretrained_kwargs or {}\n transformer, _ = load_pretrained_hf_model_with_hub_fallback(\n CamembertModel, pretrained_model_name_or_path, **pretrained_kwargs\n )\n else:\n transformer = self._init_transformer_from_scratch(\n CamembertModel, CamembertConfig, hf_config_params, vocab_size\n )\n\n if encoder_config is not None:\n self.config = self._init_config(transformer, hf_config_params.keys(), encoder_config)\n else:\n self.config = None\n\n self.transformer = self._wrap_transformer(transformer, adapter, trainable)\n self.reduce_output = reduce_output\n if not self.reduce_output == \"cls_pooled\":\n self.reduce_sequence = SequenceReducer(reduce_mode=reduce_output)\n self.max_sequence_length = max_sequence_length\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n if mask is not None:\n mask = mask.to(torch.int32)\n transformer_outputs = self.transformer.module(\n input_ids=inputs,\n attention_mask=mask,\n token_type_ids=torch.zeros_like(inputs),\n )\n if self.reduce_output == \"cls_pooled\":\n hidden = transformer_outputs[1]\n else:\n hidden = transformer_outputs[0][:, 1:-1, :]\n hidden = self.reduce_sequence(hidden, self.reduce_output)\n\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return CamemBERTConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is None:\n # Subtract 2 to remove CLS and PAD tokens added by BERT tokenizer.\n return torch.Size(\n [\n self.max_sequence_length - 2,\n self.transformer.module.config.hidden_size,\n ]\n )\n elif self.reduce_output == \"concat\":\n # add the -2 to account of start and end tokens.\n return torch.Size([self.transformer.module.config.hidden_size * (self.max_sequence_length - 2)])\n return torch.Size([self.transformer.module.config.hidden_size])\n\n @property\n def input_dtype(self) -> torch.dtype:\n return torch.int32\n\n\n@DeveloperAPI\n@register_encoder(\"t5\", TEXT)\nclass T5Encoder(HFTextEncoder):\n DEFAULT_MODEL_NAME = \"t5-small\"\n\n def __init__(\n self,\n max_sequence_length: int,\n use_pretrained: bool = True,\n pretrained_model_name_or_path: str = DEFAULT_MODEL_NAME,\n saved_weights_in_checkpoint: bool = False,\n reduce_output: str = \"sum\",\n trainable: bool = False,\n adapter: BaseAdapterConfig | None = None,\n vocab_size: int = 32128,\n d_model: int = 512,\n d_kv: int = 64,\n d_ff: int = 2048,\n num_layers: int = 6,\n num_decoder_layers: int | None = None,\n num_heads: int = 8,\n relative_attention_num_buckets: int = 32,\n dropout_rate: float = 0.1,\n layer_norm_eps: float = 1e-6,\n initializer_factor: float = 1,\n feed_forward_proj: str = \"relu\",\n pretrained_kwargs: dict = None,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n\n from transformers import T5Config, T5Model\n\n hf_config_params = dict(\n vocab_size=vocab_size,\n d_model=d_model,\n d_kv=d_kv,\n d_ff=d_ff,\n num_layers=num_layers,\n num_decoder_layers=num_decoder_layers,\n num_heads=num_heads,\n relative_attention_num_buckets=relative_attention_num_buckets,\n dropout_rate=dropout_rate,\n layer_norm_eps=layer_norm_eps,\n initializer_factor=initializer_factor,\n feed_forward_proj=feed_forward_proj,\n )\n\n if use_pretrained and not saved_weights_in_checkpoint:\n pretrained_kwargs = pretrained_kwargs or {}\n transformer, _ = load_pretrained_hf_model_with_hub_fallback(\n T5Model, pretrained_model_name_or_path, **pretrained_kwargs\n )\n else:\n transformer = self._init_transformer_from_scratch(T5Model, T5Config, hf_config_params, vocab_size)\n\n if encoder_config is not None:\n self.config = self._init_config(transformer, hf_config_params.keys(), encoder_config)\n else:\n self.config = None\n\n self.max_sequence_length = max_sequence_length\n self.reduce_output = reduce_output\n if self.reduce_output == \"cls_pooled\":\n _cls_pooled_error_message(self.__class__.__name__)\n self.reduce_sequence = SequenceReducer(reduce_mode=reduce_output)\n self.transformer = self._wrap_transformer(transformer, adapter, trainable)\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n if mask is not None:\n mask = mask.to(torch.int32)\n transformer_outputs = self.transformer.module(\n inputs,\n decoder_input_ids=inputs,\n attention_mask=mask,\n )\n hidden = transformer_outputs[0][:, 0:-1, :] # [eos token]\n hidden = self.reduce_sequence(hidden, self.reduce_output)\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return T5Config\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is None:\n # Subtract 1 to remove EOS token added by T5 tokenizer.\n return torch.Size(\n [\n self.max_sequence_length - 1,\n self.transformer.module.config.hidden_size,\n ]\n )\n elif self.reduce_output == \"concat\":\n # add the -1 to account of start and end tokens.\n return torch.Size([self.transformer.module.config.hidden_size * (self.max_sequence_length - 1)])\n return torch.Size([self.transformer.module.config.d_model])\n\n @property\n def input_dtype(self) -> torch.dtype:\n return torch.int32\n\n\n@DeveloperAPI\n@register_encoder(\"flaubert\", TEXT)\nclass FlauBERTEncoder(HFTextEncoder):\n DEFAULT_MODEL_NAME = \"flaubert/flaubert_small_cased\"\n\n def __init__(\n self,\n max_sequence_length: int,\n use_pretrained: bool,\n pretrained_model_name_or_path: str = DEFAULT_MODEL_NAME,\n saved_weights_in_checkpoint: bool = False,\n reduce_output: str = \"sum\",\n trainable: bool = False,\n adapter: BaseAdapterConfig | None = None,\n vocab_size: int = 30145,\n pre_norm: bool = False,\n layerdrop: float = 0.0,\n emb_dim: int = 2048,\n n_layers: int = 12,\n n_heads: int = 16,\n dropout: float = 0.1,\n attention_dropout: float = 0.1,\n gelu_activation: bool = True,\n sinusoidal_embeddings: bool = False,\n causal: bool = False,\n asm: bool = False,\n n_langs: int = 1,\n use_lang_emb: bool = True,\n max_position_embeddings: int = 512,\n embed_init_std: float = 2048**-0.5,\n init_std: int = 0.02,\n layer_norm_eps: float = 1e-12,\n bos_index: int = 0,\n eos_index: int = 1,\n pad_index: int = 2,\n unk_index: int = 3,\n mask_index: int = 5,\n is_encoder: bool = True,\n mask_token_id: int = 0,\n lang_id: int = 1,\n pretrained_kwargs: dict = None,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n\n from transformers import FlaubertConfig, FlaubertModel\n\n hf_config_params = dict(\n vocab_size=vocab_size,\n pre_norm=pre_norm,\n layerdrop=layerdrop,\n emb_dim=emb_dim,\n n_layers=n_layers,\n n_heads=n_heads,\n dropout=dropout,\n attention_dropout=dropout,\n gelu_activation=gelu_activation,\n sinusoidal_embeddings=sinusoidal_embeddings,\n causal=causal,\n asm=asm,\n n_langs=n_langs,\n use_lang_emb=use_lang_emb,\n max_position_embeddings=max_position_embeddings,\n embed_init_std=embed_init_std,\n init_std=init_std,\n layer_norm_eps=layer_norm_eps,\n bos_index=bos_index,\n eos_index=eos_index,\n pad_index=pad_index,\n unk_index=unk_index,\n mask_index=mask_index,\n is_encoder=is_encoder,\n mask_token_id=mask_token_id,\n lang_id=lang_id,\n )\n\n if use_pretrained and not saved_weights_in_checkpoint:\n pretrained_kwargs = pretrained_kwargs or {}\n transformer, _ = load_pretrained_hf_model_with_hub_fallback(\n FlaubertModel, pretrained_model_name_or_path, **pretrained_kwargs\n )\n else:\n transformer = self._init_transformer_from_scratch(\n FlaubertModel, FlaubertConfig, hf_config_params, vocab_size\n )\n\n if encoder_config is not None:\n self.config = self._init_config(transformer, hf_config_params.keys(), encoder_config)\n else:\n self.config = None\n\n self.max_sequence_length = max_sequence_length\n self.reduce_output = reduce_output\n if self.reduce_output == \"cls_pooled\":\n _cls_pooled_error_message(self.__class__.__name__)\n self.reduce_sequence = SequenceReducer(reduce_mode=reduce_output)\n self.transformer = self._wrap_transformer(transformer, adapter, trainable)\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n if mask is not None:\n mask = mask.to(torch.int32)\n transformer_outputs = self.transformer.module(\n input_ids=inputs,\n attention_mask=mask,\n token_type_ids=torch.zeros_like(inputs),\n )\n hidden = transformer_outputs[0][:, 1:-1, :]\n hidden = self.reduce_sequence(hidden, self.reduce_output)\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return FlauBERTConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is None:\n # Subtract 2 to remove CLS and PAD tokens added by tokenizer.\n return torch.Size(\n [\n self.max_sequence_length - 2,\n self.transformer.module.config.hidden_size,\n ]\n )\n elif self.reduce_output == \"concat\":\n # add the -2 to account of start and end tokens.\n return torch.Size([self.transformer.module.config.hidden_size * (self.max_sequence_length - 2)])\n return torch.Size([self.transformer.module.config.emb_dim])\n\n @property\n def input_dtype(self) -> torch.dtype:\n return torch.int32\n\n\n@DeveloperAPI\n@register_encoder(\"electra\", TEXT)\nclass ELECTRAEncoder(HFTextEncoder):\n DEFAULT_MODEL_NAME = \"google/electra-small-discriminator\"\n\n def __init__(\n self,\n max_sequence_length: int,\n use_pretrained: bool = True,\n pretrained_model_name_or_path: str = DEFAULT_MODEL_NAME,\n saved_weights_in_checkpoint: bool = False,\n reduce_output: str = \"sum\",\n trainable: bool = False,\n adapter: BaseAdapterConfig | None = None,\n vocab_size: int = 30522,\n embedding_size: int = 128,\n hidden_size: int = 256,\n num_hidden_layers: int = 12,\n num_attention_heads: int = 4,\n intermediate_size: int = 1024,\n hidden_act: str | Callable = \"gelu\",\n hidden_dropout_prob: float = 0.1,\n attention_probs_dropout_prob: float = 0.1,\n max_position_embeddings: int = 512,\n type_vocab_size: int = 2,\n initializer_range: float = 0.02,\n layer_norm_eps: float = 1e-12,\n position_embedding_type: str = \"absolute\",\n classifier_dropout: float | None = None,\n pretrained_kwargs: dict = None,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n\n from transformers import ElectraConfig, ElectraModel\n\n hf_config_params = dict(\n vocab_size=vocab_size,\n embedding_size=embedding_size,\n hidden_size=hidden_size,\n num_hidden_layers=num_hidden_layers,\n num_attention_heads=num_attention_heads,\n intermediate_size=intermediate_size,\n hidden_act=hidden_act,\n hidden_dropout_prob=hidden_dropout_prob,\n attention_probs_dropout_prob=attention_probs_dropout_prob,\n max_position_embeddings=max_position_embeddings,\n type_vocab_size=type_vocab_size,\n initializer_range=initializer_range,\n layer_norm_eps=layer_norm_eps,\n position_embedding_type=position_embedding_type,\n classifier_dropout=classifier_dropout,\n )\n\n if use_pretrained and not saved_weights_in_checkpoint:\n pretrained_kwargs = pretrained_kwargs or {}\n transformer, _ = load_pretrained_hf_model_with_hub_fallback(\n ElectraModel, pretrained_model_name_or_path, **pretrained_kwargs\n )\n else:\n transformer = self._init_transformer_from_scratch(ElectraModel, ElectraConfig, hf_config_params, vocab_size)\n\n if encoder_config is not None:\n self.config = self._init_config(transformer, hf_config_params.keys(), encoder_config)\n else:\n self.config = None\n\n self.max_sequence_length = max_sequence_length\n self.reduce_output = reduce_output\n if self.reduce_output == \"cls_pooled\":\n _cls_pooled_error_message(self.__class__.__name__)\n self.reduce_sequence = SequenceReducer(reduce_mode=reduce_output)\n self.transformer = self._wrap_transformer(transformer, adapter, trainable)\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n if mask is not None:\n mask = mask.to(torch.int32)\n transformer_outputs = self.transformer.module(\n input_ids=inputs,\n attention_mask=mask,\n token_type_ids=torch.zeros_like(inputs),\n )\n hidden = transformer_outputs[0][:, 1:-1, :]\n hidden = self.reduce_sequence(hidden, self.reduce_output)\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return ELECTRAConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is None:\n # Subtract 2 to remove CLS and PAD tokens added by tokenizer.\n return torch.Size(\n [\n self.max_sequence_length - 2,\n self.transformer.module.config.hidden_size,\n ]\n )\n elif self.reduce_output == \"concat\":\n # add the -2 to account of start and end tokens.\n return torch.Size([self.transformer.module.config.hidden_size * (self.max_sequence_length - 2)])\n return torch.Size([self.transformer.module.config.hidden_size])\n\n @property\n def input_dtype(self) -> torch.dtype:\n return torch.int32\n\n\n@DeveloperAPI\n@register_encoder(\"longformer\", TEXT)\nclass LongformerEncoder(HFTextEncoder):\n DEFAULT_MODEL_NAME = \"allenai/longformer-base-4096\"\n\n def __init__(\n self,\n max_sequence_length: int,\n use_pretrained: bool = True,\n attention_window: list[int] | int = 512,\n sep_token_id: int = 2,\n pretrained_model_name_or_path: str = DEFAULT_MODEL_NAME,\n saved_weights_in_checkpoint: bool = False,\n reduce_output: str | None = \"cls_pooled\",\n trainable: bool = False,\n adapter: BaseAdapterConfig | None = None,\n vocab_size: int = 50265,\n num_tokens: int | None = None,\n pretrained_kwargs: dict = None,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n\n from transformers import LongformerConfig, LongformerModel\n\n hf_config_params = dict(\n attention_window=attention_window,\n sep_token_id=sep_token_id,\n vocab_size=vocab_size,\n **kwargs,\n )\n\n if use_pretrained and not saved_weights_in_checkpoint:\n pretrained_kwargs = pretrained_kwargs or {}\n transformer, _ = load_pretrained_hf_model_with_hub_fallback(\n LongformerModel, pretrained_model_name_or_path, **pretrained_kwargs\n )\n else:\n transformer = self._init_transformer_from_scratch(\n LongformerModel, LongformerConfig, hf_config_params, vocab_size\n )\n\n if encoder_config is not None:\n self.config = self._init_config(transformer, hf_config_params.keys(), encoder_config)\n else:\n self.config = None\n\n self.reduce_output = reduce_output\n if not self.reduce_output == \"cls_pooled\":\n self.reduce_sequence = SequenceReducer(reduce_mode=reduce_output)\n self.transformer = self._wrap_transformer(transformer, adapter, trainable)\n self.max_sequence_length = max_sequence_length\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n if mask is not None:\n mask = mask.to(torch.int32)\n transformer_outputs = self.transformer.module(\n input_ids=inputs,\n attention_mask=mask,\n token_type_ids=torch.zeros_like(inputs),\n )\n if self.reduce_output == \"cls_pooled\":\n hidden = transformer_outputs[1]\n else:\n hidden = transformer_outputs[0][:, 1:-1, :] # bos + [sent] + sep\n hidden = self.reduce_sequence(hidden, self.reduce_output)\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return LongformerConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is None:\n # Subtract 2 to remove CLS and PAD tokens added by Longformer (== Roberta) tokenizer.\n return torch.Size(\n [\n self.max_sequence_length - 2,\n self.transformer.module.config.hidden_size,\n ]\n )\n elif self.reduce_output == \"concat\":\n # add the -2 to account of start and end tokens.\n return torch.Size([self.transformer.module.config.hidden_size * (self.max_sequence_length - 2)])\n return torch.Size([self.transformer.module.config.hidden_size])\n\n @property\n def input_dtype(self) -> torch.dtype:\n return torch.int32\n\n\n@DeveloperAPI\n@register_encoder(\"auto_transformer\", TEXT)\nclass AutoTransformerEncoder(HFTextEncoder):\n DEFAULT_MODEL_NAME = None\n\n def __init__(\n self,\n pretrained_model_name_or_path: str,\n max_sequence_length: int,\n reduce_output: str = \"sum\",\n trainable: bool = False,\n adapter: BaseAdapterConfig | None = None,\n vocab_size: int | None = None,\n pretrained_kwargs: dict = None,\n encoder_config=None,\n **kwargs,\n ):\n super().__init__()\n\n from transformers import AutoModel\n\n pretrained_kwargs = pretrained_kwargs or {}\n transformer, _ = load_pretrained_hf_model_with_hub_fallback(\n AutoModel, pretrained_model_name_or_path, **pretrained_kwargs\n )\n self._maybe_resize_token_embeddings(transformer, vocab_size)\n\n self.config = self._init_config(transformer, [], encoder_config)\n\n # Precompute the set of params that are included in the forward signature of the AutoModel implementation so\n # we can filter out unused params during the `forward` call.\n self.forward_kwargs = set(inspect.signature(transformer.forward).parameters.keys())\n\n self.transformer = self._wrap_transformer(transformer, adapter, trainable)\n self.reduce_output = reduce_output\n if self.reduce_output != \"cls_pooled\":\n self.reduce_sequence = SequenceReducer(\n reduce_mode=reduce_output, encoding_size=self.transformer.module.config.hidden_size\n )\n self.max_sequence_length = max_sequence_length\n\n def _maybe_resize_token_embeddings(self, transformer, vocab_size: int | None = None):\n \"\"\"Overridden because AutoModel should use its own vocab size unless vocab size is explicitly specified.\"\"\"\n if vocab_size is not None:\n transformer.resize_token_embeddings(vocab_size)\n self.vocab_size = vocab_size\n else:\n self.vocab_size = transformer.config.vocab_size\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n if mask is not None:\n mask = mask.to(torch.int32)\n\n # The forward signature of AutoModel is not consistent across implementations, so we need to make sure we're\n # only passing in params included in the forward signature.\n kwargs = dict(\n input_ids=inputs,\n attention_mask=mask,\n token_type_ids=torch.zeros_like(inputs),\n )\n kwargs = {k: v for k, v in kwargs.items() if k in self.forward_kwargs}\n\n transformer_outputs = self.transformer.module(**kwargs)\n if self.reduce_output == \"cls_pooled\":\n # this works only if the user know that the specific model\n # they want to use has the same outputs of\n # the BERT base class call() function\n hidden = transformer_outputs[\"pooler_output\"]\n else:\n hidden = transformer_outputs[\"last_hidden_state\"]\n hidden = self.reduce_sequence(hidden, self.reduce_output)\n return {ENCODER_OUTPUT: hidden}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return AutoTransformerConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n if self.reduce_output is None:\n # TODO(justin): This may need to be conditioned on which AutoModel gets chosen.\n return torch.Size([self.max_sequence_length, self.transformer.module.config.hidden_size])\n if self.reduce_output == \"concat\":\n return torch.Size(\n [\n self.max_sequence_length * self.transformer.module.config.hidden_size,\n ]\n )\n elif self.reduce_output == \"concat\":\n # add the -2 to account of start and end tokens.\n return torch.Size([self.transformer.module.config.hidden_size * (self.max_sequence_length - 2)])\n return torch.Size([self.transformer.module.config.hidden_size])\n\n @property\n def input_dtype(self) -> torch.dtype:\n return torch.int32\n\n\n@DeveloperAPI\n@register_encoder(\"tf_idf\", [TEXT])\nclass TfIdfEncoder(Encoder):\n def __init__(\n self,\n max_sequence_length: int,\n encoder_config=None,\n str2idf=None,\n vocab=None,\n vocab_size: int = None,\n **kwargs,\n ):\n super().__init__()\n self.config = encoder_config\n self.max_sequence_length = max_sequence_length\n self.vocab_size = vocab_size\n\n logger.debug(f\" {self.name}\")\n\n # Convert mapping of token -> frequency to a dense array\n idf = np.zeros(vocab_size)\n for i, s in enumerate(vocab):\n idf[i] = str2idf[s]\n self.register_buffer(\"idf\", torch.from_numpy(idf).float().unsqueeze(0))\n\n def forward(self, t: torch.Tensor, mask: torch.Tensor | None = None) -> EncoderOutputDict:\n # Compute the term frequency within each row\n tf = torch.stack([t_i.bincount(minlength=self.vocab_size) for t_i in torch.unbind(t.long())])\n\n # Normalize the term frequency by the number of tokens in each row\n tf = tf / tf.sum(dim=1).unsqueeze(-1)\n\n # Multiply the term frequency by the inverse document frequency\n tfidf = tf * self.idf\n\n return {ENCODER_OUTPUT: tfidf}\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return TfIdfEncoderConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.vocab_size])\n\n def get_embedding_layer(self) -> nn.Module:\n return self\n\n\n@DeveloperAPI\n@register_encoder(\"llm\", [TEXT])\nclass LLMEncoder(Encoder):\n # Per-adapter type prefixes for parameter names in the state dict, taken from\n # https://github.com/huggingface/peft/blob/0f1e9091cc975eb5458cc163bf1843a34fb42b76/src/peft/utils/save_and_load.py#L173C9-L180\n ADAPTER_PARAM_NAME_PREFIX = {\n \"adalora\": \"lora_\",\n \"ia3\": \"ia3_\",\n \"lora\": \"lora_\",\n }\n\n def __init__(self, encoder_config: LLMEncoderConfig = None, **kwargs):\n super().__init__()\n self.register_load_state_dict_post_hook(self.remove_missing_non_adapter_keys)\n\n self.config = encoder_config\n\n self.adapter_is_initialized = False\n\n self.model_name = self.config.base_model\n self.model_config = AutoConfig.from_pretrained(self.config.base_model)\n\n self.model = load_pretrained_from_config(self.config, model_config=self.model_config)\n self.curr_device = next(self.model.parameters()).device\n logger.info(\"Done.\")\n\n self.context_len = get_context_len(self.model_config)\n\n # TODO(Arnav): This needs be more flexible to account for RoPE Scaling\n # When merging input IDs and target IDs for LLM fine-tuning, we want to make sure that the merged tensor is\n # not longer than the global maximum sequence length. This is provided in the preprocessing config. We never\n # want to exceed the maximum possible context length so we also check for that.\n if self.config.max_sequence_length:\n max_sequence_length = self.config.max_sequence_length\n self.max_sequence_length = (\n max_sequence_length if max_sequence_length <= self.context_len else self.context_len\n )\n else:\n self.max_sequence_length = self.context_len\n\n # Initialize tokenizer\n self.tokenizer = HFTokenizer(self.config.base_model).tokenizer\n\n self.attention_masks = None\n\n clear_data_cache()\n\n # Because we use the last hidden state as encoder output rather than the logits, the final module of the model\n # has input pass through but no gradient update in the backward pass. This can lead to a DDP error. Freezing\n # the module prevents this from happening. This is done at initialization to prevent \"unused parameters\" errors\n # from happening when the encoder is used before `prepare_for_training` is called, for example during batch\n # size tuning.\n out_module = list(self.model.modules())[-1]\n out_module.requires_grad_(requires_grad=False)\n\n @staticmethod\n def get_schema_cls() -> type[BaseEncoderConfig]:\n return LLMEncoderConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length, self.model_config.hidden_size])\n\n def get_embedding_layer(self) -> nn.Module:\n return self\n\n def initialize_adapter(self):\n \"\"\"If an adapter config is provided, we want to wrap the model with a PEFT model for fine-tuning.\"\"\"\n if self.config.adapter:\n self.model = initialize_adapter(self.model, self.config)\n\n logger.info(\"==================================================\")\n logger.info(\"Trainable Parameter Summary For LLM Encoder Fine-Tuning\")\n logger.info(f\"Fine-tuning with adapter: {self.config.adapter.type}\")\n self.model.print_trainable_parameters()\n logger.info(\"==================================================\")\n\n self.adapter_is_initialized = True\n\n def prepare_for_training(self):\n # TODO: this implementation will not work if resuming from a previous checkpoint. Need to fix this.\n if self.config.quantization:\n self.prepare_for_quantized_training()\n self.initialize_adapter()\n\n def prepare_for_quantized_training(self):\n from peft import prepare_model_for_kbit_training\n\n self.model = prepare_model_for_kbit_training(self.model, use_gradient_checkpointing=False)\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None):\n # Get the hidden state of the last layer and return it as the text encoding\n model_outputs = self.model(input_ids=inputs, output_hidden_states=True).hidden_states[-1]\n\n return {ENCODER_OUTPUT: model_outputs.type(torch.float32)}\n\n def _save_to_state_dict(self, destination: dict, prefix: str, keep_vars: bool):\n # This is called by `torch.nn.Module.state_dict()` under the hood. `state_dict()` does additional work to\n # prep the dictionary, get submodule state, and run hooks. Overriding this method only impacts the\n # contents of the state_dict.\n # The three args to this method are supplied by Module.state_dict\n # https://github.com/pytorch/pytorch/blob/8739d1e3f9b08f4282fe79fc8dacd781d16913ff/torch/nn/modules/module.py#L1824\n if self.config.adapter and self.adapter_is_initialized:\n # get_peft_model_state_dict geneates a state dict that only contains the adapter weights\n from peft.utils.save_and_load import get_peft_model_state_dict\n\n sd = get_peft_model_state_dict(self.model)\n destination.update(sd)\n\n else:\n super()._save_to_state_dict(destination, prefix=prefix, keep_vars=keep_vars)\n\n def state_dict(self, *args, destination=None, prefix=\"\", keep_vars=False):\n destination = super().state_dict(destination, prefix=prefix, keep_vars=keep_vars)\n\n if self.config.adapter and self.adapter_is_initialized:\n adapter_type_prefix = self.ADAPTER_PARAM_NAME_PREFIX[self.config.adapter.type]\n exclude_model_keys = [k for k in destination.keys() if adapter_type_prefix not in k]\n\n for k in exclude_model_keys:\n del destination[k]\n\n return destination\n\n def _load_from_state_dict(\n self, state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs\n ):\n # Call this first to make sure torch can do its usual load. In the adapter case, this should essentially be a\n # no-op, but the adapter weights will be collected in `unexpected_keys` because PEFT changes the parameter\n # names under the hood.\n\n super()._load_from_state_dict(\n state_dict, prefix, local_metadata, strict, missing_keys, unexpected_keys, error_msgs\n )\n\n if self.config.adapter and self.adapter_is_initialized:\n # When using an adapter, only the adapter weights are saved, and so we only want to load those weights.\n # Under the hood, PEFT alters the names of the parameters, which leads to an \"unexpected keys\" error when\n # using strict mode. This block uses PEFT's version of `load_state_dict` to handle loading in weights.\n from peft.utils.save_and_load import set_peft_model_state_dict\n\n adapter_type_prefix = self.ADAPTER_PARAM_NAME_PREFIX[self.config.adapter.type]\n peft_model_state_dict = {k: v for k, v in state_dict.items() if adapter_type_prefix in k}\n set_peft_model_state_dict(self.model, peft_model_state_dict)\n\n def remove_missing_non_adapter_keys(self, module, incompatible_keys):\n \"\"\"Update the missing and unexpected keys lists to reflect custom adapter state load logic.\n\n This method should never return anything unless the underlying torch hook logic is updated. Any changes to the\n lists in `incompatible_keys` must be made in-place.\n\n Args:\n module: The torch module with newly loaded state\n incompatible_keys: A tuple with the lists of missing and unexpected keys that were recorded while loading\n \"\"\"\n # If no adapter was used, `LLMEncoder.load_state_dict` should use the default `torch.Module.load_state_dict`\n # code path to load weights and no modification should be necessary.\n if self.config.adapter and self.adapter_is_initialized:\n adapter_type_prefix = self.ADAPTER_PARAM_NAME_PREFIX[self.config.adapter.type]\n missing_keys, unexpected_keys = incompatible_keys\n\n # The state dict uses fully qualified parameter names, but this function does not have access to the\n # fully qualified names or a prefix to recreate them. Iterate over the missing keys and greedily select the\n # first non-adapter key that shares a suffix with a model parameter name.\n sample_missing_key = \"\"\n sample_model_key = \"\"\n for k in missing_keys:\n # Exclude any adapter weight--those should not be missing. Let torch handle that downstream.\n if adapter_type_prefix not in k:\n sample_model_keys = [p for p, _ in self.named_parameters() if p in k]\n if sample_model_keys:\n sample_model_key = sample_model_keys[0]\n sample_missing_key = k\n break\n sd_prefix = sample_missing_key.replace(sample_model_key, \"\")\n\n # When loading the adapter weights in strict mode, torch will register the base model weights as missing\n # from the state dict and raise an exception. The base model weights are intended to be excluded, so the\n # missing_keys list is updated post-load to avoid the error.\n for k, _ in self.named_parameters():\n full_name = f\"{sd_prefix}{k}\"\n if full_name in missing_keys and adapter_type_prefix not in full_name:\n missing_keys.remove(full_name)\n\n # peft changes the adapter parameter names under the hood to include the adapter name. When retreiving the\n # adapter state dict, however, the name is not included. This causes the adpater weights to be recorded as\n # unexpected parameters. `LLMEncoder._load_from_state_dict` loads the adapter parameters using a peft\n # utility that accounts for the updated names, so here we remove any adapter parameters from the unexpected\n # keys list to avoid errors.\n from peft.utils.save_and_load import get_peft_model_state_dict\n\n sd = get_peft_model_state_dict(self.model)\n for k in sd.keys():\n if k in unexpected_keys:\n unexpected_keys.remove(k)\n"
},
{
"path": "ludwig/encoders/types.py",
"content": "from typing import TypedDict\n\nimport torch\n\n\nclass EncoderOutputDict(TypedDict, total=False):\n encoder_output: torch.Tensor\n encoder_output_state: torch.Tensor # only used by sequence and h3 encoders\n attentions: torch.Tensor # only used by the vit legacy encoder\n"
},
{
"path": "ludwig/error.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nfrom ludwig.api_annotations import PublicAPI\n\n\n@PublicAPI\nclass LudwigError(Exception):\n \"\"\"Base class for all custom exceptions raised by the Ludwig framework.\"\"\"\n\n def __reduce__(self):\n \"\"\"Docs: https://docs.python.org/3/library/pickle.html#object.__reduce__.\"\"\"\n raise NotImplementedError(\n \"Implement __reduce__ for all subclasses of LudwigError as it's necessary for \"\n \"serialization by Ray. See https://github.com/ludwig-ai/ludwig/pull/2695.\"\n )\n\n\n@PublicAPI\nclass InputDataError(LudwigError, ValueError):\n \"\"\"Exception raised for errors in the input data.\n\n Appropriate for data which is not convertible to the input feature type, columns with all missing values,\n categorical columns with only one category, etc...\n\n Attributes:\n column - The name of the input column which caused the error\n feature_type - The Ludwig feature type which caused the error (number, binary, category...).\n message - An error message describing the situation.\n \"\"\"\n\n def __init__(self, column_name: str, feature_type: str, message: str):\n self.column_name = column_name\n self.feature_type = feature_type\n self.message = message\n super().__init__(message)\n\n def __str__(self):\n return f'Column \"{self.column_name}\" as {self.feature_type} feature: {self.message}'\n\n def __reduce__(self):\n return type(self), (self.column_name, self.feature_type, self.message)\n\n\n@PublicAPI\nclass ConfigValidationError(LudwigError, ValueError):\n \"\"\"Exception raised for errors in the Ludwig configuration.\n\n Appropriate for bad configuration values, missing required configuration values, etc...\n\n Attributes:\n message - An error message describing the situation.\n \"\"\"\n\n def __init__(self, message: str):\n self.message = message\n super().__init__(message)\n\n def __reduce__(self):\n return type(self), (self.message,)\n"
},
{
"path": "ludwig/evaluate.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport argparse\nimport logging\nimport sys\n\nimport pandas as pd\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.backend import ALL_BACKENDS, Backend, initialize_backend\nfrom ludwig.callbacks import Callback\nfrom ludwig.constants import FULL, TEST, TRAINING, VALIDATION\nfrom ludwig.contrib import add_contrib_callback_args\nfrom ludwig.globals import LUDWIG_VERSION\nfrom ludwig.utils.print_utils import get_logging_level_registry, print_ludwig\n\nlogger = logging.getLogger(__name__)\n\n\ndef evaluate_cli(\n model_path: str,\n dataset: str | dict | pd.DataFrame = None,\n data_format: str = None,\n split: str = FULL,\n batch_size: int = 128,\n skip_save_unprocessed_output: bool = False,\n skip_save_predictions: bool = False,\n skip_save_eval_stats: bool = False,\n skip_collect_predictions: bool = False,\n skip_collect_overall_stats: bool = False,\n output_directory: str = \"results\",\n gpus: str | int | list[int] = None,\n gpu_memory_limit: float | None = None,\n allow_parallel_threads: bool = True,\n callbacks: list[Callback] = None,\n backend: Backend | str = None,\n logging_level: int = logging.INFO,\n **kwargs,\n) -> None:\n \"\"\"Loads pre-trained model and evaluates its performance by comparing the predictions against ground truth.\n\n # Inputs\n\n :param model_path: (str) filepath to pre-trained model.\n :param dataset: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing the entire dataset to be used in the evaluation.\n :param data_format: (str, default: `None`) format to interpret data\n sources. Will be inferred automatically if not specified. Valid\n formats are `'auto'`, `'csv'`, `'excel'`, `'feather'`,\n `'fwf'`, `'hdf5'` (cache file produced during previous training),\n `'html'` (file containing a single HTML `
`), `'json'`, `'jsonl'`,\n `'parquet'`, `'pickle'` (pickled Pandas DataFrame), `'sas'`, `'spss'`,\n `'stata'`, `'tsv'`.\n :param split: (str, default: `full`) split on which\n to perform predictions. Valid values are `'training'`, `'validation'`,\n `'test'` and `'full'`.\n :param batch_size: (int, default `128`) size of batches for processing.\n :param skip_save_unprocessed_output: (bool, default: `False`) by default\n predictions and their probabilities are saved in both raw\n unprocessed numpy files containing tensors and as postprocessed\n CSV files (one for each output feature). If this parameter is True,\n only the CSV ones are saved and the numpy ones are skipped.\n :param skip_save_predictions: (bool, default: `False`) skips saving test\n predictions CSV files\n :param skip_save_eval_stats: (bool, default: `False`) skips saving test\n statistics JSON file\n :param skip_collect_predictions: (bool, default: `False`) skips\n collecting post-processed predictions during eval.\n :param skip_collect_overall_stats: (bool, default: `False`) skips\n collecting overall stats during eval.\n :param output_directory: (str, default: `'results'`) the directory that\n will contain the training statistics, TensorBoard logs, the saved\n model and the training progress files.\n :param gpus: (list, default: `None`) list of GPUs that are available\n for training.\n :param gpu_memory_limit: (float: default: `None`) maximum memory fraction\n [0, 1] allowed to allocate per GPU device.\n :param allow_parallel_threads: (bool, default: `True`) allow PyTorch\n to use multithreading parallelism to improve performance at\n the cost of determinism.\n :param callbacks: (list, default: `None`) a list of\n `ludwig.callbacks.Callback` objects that provide hooks into the\n Ludwig pipeline.\n :param backend: (Union[Backend, str]) `Backend` or string name\n of backend to use to execute preprocessing / training steps.\n :param logging_level: (int) Log level that will be sent to stderr.\n\n # Returns\n\n :return: (`None`)\n \"\"\"\n model = LudwigModel.load(\n model_path,\n logging_level=logging_level,\n backend=backend,\n gpus=gpus,\n gpu_memory_limit=gpu_memory_limit,\n allow_parallel_threads=allow_parallel_threads,\n callbacks=callbacks,\n )\n model.evaluate(\n dataset=dataset,\n data_format=data_format,\n batch_size=batch_size,\n split=split,\n skip_save_unprocessed_output=skip_save_unprocessed_output,\n skip_save_predictions=skip_save_predictions,\n skip_save_eval_stats=skip_save_eval_stats,\n collect_predictions=not skip_collect_predictions,\n collect_overall_stats=not skip_collect_overall_stats,\n output_directory=output_directory,\n return_type=\"dict\",\n )\n\n\ndef cli(sys_argv):\n parser = argparse.ArgumentParser(\n description=\"This script loads a pretrained model \"\n \"and evaluates its performance by comparing\"\n \"its predictions with ground truth.\",\n prog=\"ludwig evaluate\",\n usage=\"%(prog)s [options]\",\n )\n\n # ---------------\n # Data parameters\n # ---------------\n parser.add_argument(\"--dataset\", help=\"input data file path\", required=True)\n parser.add_argument(\n \"--data_format\",\n help=\"format of the input data\",\n default=\"auto\",\n choices=[\n \"auto\",\n \"csv\",\n \"excel\",\n \"feather\",\n \"fwf\",\n \"hdf5\",\n \"html\" \"tables\",\n \"json\",\n \"jsonl\",\n \"parquet\",\n \"pickle\",\n \"sas\",\n \"spss\",\n \"stata\",\n \"tsv\",\n ],\n )\n parser.add_argument(\n \"-s\", \"--split\", default=FULL, choices=[TRAINING, VALIDATION, TEST, FULL], help=\"the split to test the model on\"\n )\n\n # ----------------\n # Model parameters\n # ----------------\n parser.add_argument(\"-m\", \"--model_path\", help=\"model to load\", required=True)\n\n # -------------------------\n # Output results parameters\n # -------------------------\n parser.add_argument(\n \"-od\", \"--output_directory\", type=str, default=\"results\", help=\"directory that contains the results\"\n )\n parser.add_argument(\n \"-ssuo\",\n \"--skip_save_unprocessed_output\",\n help=\"skips saving intermediate NPY output files\",\n action=\"store_true\",\n default=False,\n )\n parser.add_argument(\n \"-sses\",\n \"--skip_save_eval_stats\",\n help=\"skips saving intermediate JSON eval statistics\",\n action=\"store_true\",\n default=False,\n )\n parser.add_argument(\n \"-scp\", \"--skip_collect_predictions\", help=\"skips collecting predictions\", action=\"store_true\", default=False\n )\n parser.add_argument(\n \"-scos\",\n \"--skip_collect_overall_stats\",\n help=\"skips collecting overall stats\",\n action=\"store_true\",\n default=False,\n )\n\n # ------------------\n # Generic parameters\n # ------------------\n parser.add_argument(\"-bs\", \"--batch_size\", type=int, default=128, help=\"size of batches\")\n\n # ------------------\n # Runtime parameters\n # ------------------\n parser.add_argument(\"-g\", \"--gpus\", type=int, default=0, help=\"list of gpu to use\")\n parser.add_argument(\n \"-gml\",\n \"--gpu_memory_limit\",\n type=float,\n default=None,\n help=\"maximum memory fraction [0, 1] allowed to allocate per GPU device\",\n )\n parser.add_argument(\n \"-dpt\",\n \"--disable_parallel_threads\",\n action=\"store_false\",\n dest=\"allow_parallel_threads\",\n help=\"disable PyTorch from using multithreading for reproducibility\",\n )\n parser.add_argument(\n \"-b\",\n \"--backend\",\n help=\"specifies backend to use for parallel / distributed execution, \" \"defaults to local execution\",\n choices=ALL_BACKENDS,\n )\n parser.add_argument(\n \"-l\",\n \"--logging_level\",\n default=\"info\",\n help=\"the level of logging to use\",\n choices=[\"critical\", \"error\", \"warning\", \"info\", \"debug\", \"notset\"],\n )\n\n add_contrib_callback_args(parser)\n args = parser.parse_args(sys_argv)\n args.evaluate_performance = True\n\n args.callbacks = args.callbacks or []\n for callback in args.callbacks:\n callback.on_cmdline(\"evaluate\", *sys_argv)\n\n args.logging_level = get_logging_level_registry()[args.logging_level]\n logging.getLogger(\"ludwig\").setLevel(args.logging_level)\n global logger\n logger = logging.getLogger(\"ludwig.test_performance\")\n\n backend = initialize_backend(args.backend)\n if backend.is_coordinator():\n print_ludwig(\"Evaluate\", LUDWIG_VERSION)\n logger.info(f\"Dataset path: {args.dataset}\")\n logger.info(f\"Model path: {args.model_path}\")\n logger.info(\"\")\n\n evaluate_cli(**vars(args))\n\n\nif __name__ == \"__main__\":\n cli(sys.argv[1:])\n"
},
{
"path": "ludwig/experiment.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport argparse\nimport logging\nimport os\nimport sys\n\nimport pandas as pd\n\nfrom ludwig.api import kfold_cross_validate, LudwigModel\nfrom ludwig.backend import ALL_BACKENDS, Backend, initialize_backend\nfrom ludwig.callbacks import Callback\nfrom ludwig.constants import CONTINUE_PROMPT, FULL, HYPEROPT, HYPEROPT_WARNING, TEST, TRAINING, VALIDATION\nfrom ludwig.contrib import add_contrib_callback_args\nfrom ludwig.globals import LUDWIG_VERSION\nfrom ludwig.utils.data_utils import load_config_from_str, load_yaml, save_json\nfrom ludwig.utils.defaults import default_random_seed\nfrom ludwig.utils.print_utils import get_logging_level_registry, print_ludwig, query_yes_no\n\nlogger = logging.getLogger(__name__)\n\n\ndef experiment_cli(\n config: str | dict,\n dataset: str | dict | pd.DataFrame = None,\n training_set: str | dict | pd.DataFrame = None,\n validation_set: str | dict | pd.DataFrame = None,\n test_set: str | dict | pd.DataFrame = None,\n training_set_metadata: str | dict = None,\n data_format: str = None,\n experiment_name: str = \"experiment\",\n model_name: str = \"run\",\n model_load_path: str = None,\n model_resume_path: str = None,\n eval_split: str = TEST,\n skip_save_training_description: bool = False,\n skip_save_training_statistics: bool = False,\n skip_save_model: bool = False,\n skip_save_progress: bool = False,\n skip_save_log: bool = False,\n skip_save_processed_input: bool = False,\n skip_save_unprocessed_output: bool = False,\n skip_save_predictions: bool = False,\n skip_save_eval_stats: bool = False,\n skip_collect_predictions: bool = False,\n skip_collect_overall_stats: bool = False,\n output_directory: str = \"results\",\n gpus: str | int | list[int] = None,\n gpu_memory_limit: float | None = None,\n allow_parallel_threads: bool = True,\n callbacks: list[Callback] = None,\n backend: Backend | str = None,\n random_seed: int = default_random_seed,\n logging_level: int = logging.INFO,\n **kwargs,\n):\n \"\"\"Trains a model on a dataset's training and validation splits and uses it to predict on the test split. It\n saves the trained model and the statistics of training and testing.\n\n # Inputs\n\n :param config: (Union[str, dict]) in-memory representation of\n config or string path to a YAML config file.\n :param dataset: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing the entire dataset to be used in the experiment.\n If it has a split column, it will be used for splitting (0 for train,\n 1 for validation, 2 for test), otherwise the dataset will be\n randomly split.\n :param training_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing training data.\n :param validation_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing validation data.\n :param test_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing test data.\n :param training_set_metadata: (Union[str, dict], default: `None`)\n metadata JSON file or loaded metadata. Intermediate preprocessed\n structure containing the mappings of the input\n dataset created the first time an input file is used in the same\n directory with the same name and a '.meta.json' extension.\n :param data_format: (str, default: `None`) format to interpret data\n sources. Will be inferred automatically if not specified. Valid\n formats are `'auto'`, `'csv'`, `'excel'`, `'feather'`,\n `'fwf'`, `'hdf5'` (cache file produced during previous training),\n `'html'` (file containing a single HTML `
`), `'json'`, `'jsonl'`,\n `'parquet'`, `'pickle'` (pickled Pandas DataFrame), `'sas'`, `'spss'`,\n `'stata'`, `'tsv'`.\n :param experiment_name: (str, default: `'experiment'`) name for\n the experiment.\n :param model_name: (str, default: `'run'`) name of the model that is\n being used.\n :param model_load_path: (str, default: `None`) if this is specified the\n loaded model will be used as initialization\n (useful for transfer learning).\n :param model_resume_path: (str, default: `None`) resumes training of\n the model from the path specified. The config is restored.\n In addition to config, training statistics and loss for\n epoch and the state of the optimizer are restored such that\n training can be effectively continued from a previously interrupted\n training process.\n :param eval_split: (str, default: `test`) split on which\n to perform evaluation. Valid values are `training`, `validation`\n and `test`.\n :param skip_save_training_description: (bool, default: `False`) disables\n saving the description JSON file.\n :param skip_save_training_statistics: (bool, default: `False`) disables\n saving training statistics JSON file.\n :param skip_save_model: (bool, default: `False`) disables\n saving model weights and hyperparameters each time the model\n improves. By default Ludwig saves model weights after each epoch\n the validation metric improves, but if the model is really big\n that can be time consuming. If you do not want to keep\n the weights and just find out what performance a model can get\n with a set of hyperparameters, use this parameter to skip it,\n but the model will not be loadable later on and the returned model\n will have the weights obtained at the end of training, instead of\n the weights of the epoch with the best validation performance.\n :param skip_save_progress: (bool, default: `False`) disables saving\n progress each epoch. By default Ludwig saves weights and stats\n after each epoch for enabling resuming of training, but if\n the model is really big that can be time consuming and will uses\n twice as much space, use this parameter to skip it, but training\n cannot be resumed later on.\n :param skip_save_log: (bool, default: `False`) disables saving\n TensorBoard logs. By default Ludwig saves logs for the TensorBoard,\n but if it is not needed turning it off can slightly increase the\n overall speed.\n :param skip_save_processed_input: (bool, default: `False`) if input\n dataset is provided it is preprocessed and cached by saving an HDF5\n and JSON files to avoid running the preprocessing again. If this\n parameter is `False`, the HDF5 and JSON file are not saved.\n :param skip_save_unprocessed_output: (bool, default: `False`) by default\n predictions and their probabilities are saved in both raw\n unprocessed numpy files containing tensors and as postprocessed\n CSV files (one for each output feature). If this parameter is True,\n only the CSV ones are saved and the numpy ones are skipped.\n :param skip_save_predictions: (bool, default: `False`) skips saving test\n predictions CSV files\n :param skip_save_eval_stats: (bool, default: `False`) skips saving test\n statistics JSON file\n :param skip_collect_predictions: (bool, default: `False`) skips\n collecting post-processed predictions during eval.\n :param skip_collect_overall_stats: (bool, default: `False`) skips\n collecting overall stats during eval.\n :param output_directory: (str, default: `'results'`) the directory that\n will contain the training statistics, TensorBoard logs, the saved\n model and the training progress files.\n :param gpus: (list, default: `None`) list of GPUs that are available\n for training.\n :param gpu_memory_limit: (float: default: `None`) maximum memory fraction\n [0, 1] allowed to allocate per GPU device.\n :param allow_parallel_threads: (bool, default: `True`) allow PyTorch\n to use multithreading parallelism to improve performance at\n the cost of determinism.\n :param callbacks: (list, default: `None`) a list of\n `ludwig.callbacks.Callback` objects that provide hooks into the\n Ludwig pipeline.\n :param backend: (Union[Backend, str]) `Backend` or string name\n of backend to use to execute preprocessing / training steps.\n :param random_seed: (int: default: 42) random seed used for weights\n initialization, splits and any other random function.\n :param logging_level: (int) Log level that will be sent to stderr.\n\n # Return\n :return: (Tuple[LudwigModel, dict, dict, tuple, str)):\n `(model, evaluation_statistics, training_statistics, preprocessed_data, output_directory)`\n `model` LudwigModel instance\n `evaluation_statistics` dictionary with evaluation performance\n statistics on the test_set,\n `training_statistics` is a nested dictionary of dataset -> feature_name -> metric_name -> List of metrics.\n Each metric corresponds to each training checkpoint.\n `preprocessed_data` tuple containing preprocessed\n `(training_set, validation_set, test_set)`, `output_directory`\n filepath string to where results are stored.\n \"\"\"\n if HYPEROPT in config:\n if not query_yes_no(HYPEROPT_WARNING + CONTINUE_PROMPT):\n exit(1)\n\n if isinstance(config, str):\n config = load_yaml(config)\n backend = initialize_backend(backend or config.get(\"backend\"))\n\n if model_load_path:\n model = LudwigModel.load(\n model_load_path,\n logging_level=logging_level,\n backend=backend,\n gpus=gpus,\n gpu_memory_limit=gpu_memory_limit,\n allow_parallel_threads=allow_parallel_threads,\n callbacks=callbacks,\n )\n else:\n model = LudwigModel(\n config=config,\n logging_level=logging_level,\n backend=backend,\n gpus=gpus,\n gpu_memory_limit=gpu_memory_limit,\n allow_parallel_threads=allow_parallel_threads,\n callbacks=callbacks,\n )\n eval_stats, train_stats, preprocessed_data, output_directory = model.experiment(\n dataset=dataset,\n training_set=training_set,\n validation_set=validation_set,\n test_set=test_set,\n training_set_metadata=training_set_metadata,\n data_format=data_format,\n experiment_name=experiment_name,\n model_name=model_name,\n model_resume_path=model_resume_path,\n eval_split=eval_split,\n skip_save_training_description=skip_save_training_description,\n skip_save_training_statistics=skip_save_training_statistics,\n skip_save_model=skip_save_model,\n skip_save_progress=skip_save_progress,\n skip_save_log=skip_save_log,\n skip_save_processed_input=skip_save_processed_input,\n skip_save_unprocessed_output=skip_save_unprocessed_output,\n skip_save_predictions=skip_save_predictions,\n skip_save_eval_stats=skip_save_eval_stats,\n skip_collect_predictions=skip_collect_predictions,\n skip_collect_overall_stats=skip_collect_overall_stats,\n output_directory=output_directory,\n random_seed=random_seed,\n )\n\n return model, eval_stats, train_stats, preprocessed_data, output_directory\n\n\ndef kfold_cross_validate_cli(\n k_fold,\n config=None,\n dataset=None,\n data_format=None,\n output_directory=\"results\",\n random_seed=default_random_seed,\n skip_save_k_fold_split_indices=False,\n **kwargs,\n):\n \"\"\"Wrapper function to performs k-fold cross validation.\n\n # Inputs\n :param k_fold: (int) number of folds to create for the cross-validation\n :param config: (Union[str, dict], default: None) a dictionary or file path containing model configuration. Refer to\n the [User Guide] (http://ludwig.ai/user_guide/#model-config) for details.\n :param dataset: (string, default: None)\n :param output_directory: (string, default: 'results')\n :param random_seed: (int) Random seed used k-fold splits.\n :param skip_save_k_fold_split_indices: (boolean, default: False) Disables saving k-fold split indices\n :return: None\n \"\"\"\n\n kfold_cv_stats, kfold_split_indices = kfold_cross_validate(\n k_fold,\n config=config,\n dataset=dataset,\n data_format=data_format,\n output_directory=output_directory,\n random_seed=random_seed,\n )\n\n # save k-fold cv statistics\n save_json(os.path.join(output_directory, \"kfold_training_statistics.json\"), kfold_cv_stats)\n\n # save k-fold split indices\n if not skip_save_k_fold_split_indices:\n save_json(os.path.join(output_directory, \"kfold_split_indices.json\"), kfold_split_indices)\n\n\ndef cli(sys_argv):\n parser = argparse.ArgumentParser(\n description=\"This script trains and evaluates a model\", prog=\"ludwig experiment\", usage=\"%(prog)s [options]\"\n )\n\n # ----------------------------\n # Experiment naming parameters\n # ----------------------------\n parser.add_argument(\"--output_directory\", type=str, default=\"results\", help=\"directory that contains the results\")\n parser.add_argument(\"--experiment_name\", type=str, default=\"experiment\", help=\"experiment name\")\n parser.add_argument(\"--model_name\", type=str, default=\"run\", help=\"name for the model\")\n\n # ---------------\n # Data parameters\n # ---------------\n parser.add_argument(\n \"--dataset\",\n help=\"input data file path. \"\n \"If it has a split column, it will be used for splitting \"\n \"(0: train, 1: validation, 2: test), \"\n \"otherwise the dataset will be randomly split\",\n )\n parser.add_argument(\"--training_set\", help=\"input train data file path\")\n parser.add_argument(\"--validation_set\", help=\"input validation data file path\")\n parser.add_argument(\"--test_set\", help=\"input test data file path\")\n\n parser.add_argument(\n \"--training_set_metadata\",\n help=\"input metadata JSON file path. An intermediate preprocessed file \"\n \"containing the mappings of the input file created \"\n \"the first time a file is used, in the same directory \"\n \"with the same name and a .json extension\",\n )\n\n parser.add_argument(\n \"--data_format\",\n help=\"format of the input data\",\n default=\"auto\",\n choices=[\n \"auto\",\n \"csv\",\n \"excel\",\n \"feather\",\n \"fwf\",\n \"hdf5\",\n \"html\" \"tables\",\n \"json\",\n \"jsonl\",\n \"parquet\",\n \"pickle\",\n \"sas\",\n \"spss\",\n \"stata\",\n \"tsv\",\n ],\n )\n\n parser.add_argument(\n \"-es\",\n \"--eval_split\",\n default=TEST,\n choices=[TRAINING, VALIDATION, TEST, FULL],\n help=\"the split to evaluate the model on\",\n )\n\n parser.add_argument(\n \"-sspi\",\n \"--skip_save_processed_input\",\n help=\"skips saving intermediate HDF5 and JSON files\",\n action=\"store_true\",\n default=False,\n )\n parser.add_argument(\n \"-ssuo\",\n \"--skip_save_unprocessed_output\",\n help=\"skips saving intermediate NPY output files\",\n action=\"store_true\",\n default=False,\n )\n\n # -----------------\n # K-fold parameters\n # -----------------\n parser.add_argument(\n \"-kf\", \"--k_fold\", type=int, default=None, help=\"number of folds for a k-fold cross validation run \"\n )\n parser.add_argument(\n \"-skfsi\",\n \"--skip_save_k_fold_split_indices\",\n action=\"store_true\",\n default=False,\n help=\"disables saving indices generated to split training data set \"\n \"for the k-fold cross validation run, but if it is not needed \"\n \"turning it off can slightly increase the overall speed\",\n )\n\n # ----------------\n # Model parameters\n # ----------------\n config = parser.add_mutually_exclusive_group(required=True)\n config.add_argument(\n \"-c\",\n \"--config\",\n type=load_yaml,\n help=\"Path to the YAML file containing the model configuration\",\n )\n config.add_argument(\n \"-cs\",\n \"--config_str\",\n dest=\"config\",\n type=load_config_from_str,\n help=\"JSON or YAML serialized string of the model configuration\",\n )\n\n parser.add_argument(\"-mlp\", \"--model_load_path\", help=\"path of a pretrained model to load as initialization\")\n parser.add_argument(\"-mrp\", \"--model_resume_path\", help=\"path of the model directory to resume training of\")\n parser.add_argument(\n \"-sstd\",\n \"--skip_save_training_description\",\n action=\"store_true\",\n default=False,\n help=\"disables saving the description JSON file\",\n )\n parser.add_argument(\n \"-ssts\",\n \"--skip_save_training_statistics\",\n action=\"store_true\",\n default=False,\n help=\"disables saving training statistics JSON file\",\n )\n parser.add_argument(\n \"-sstp\",\n \"--skip_save_predictions\",\n help=\"skips saving test predictions CSV files\",\n action=\"store_true\",\n default=False,\n )\n parser.add_argument(\n \"-sstes\",\n \"--skip_save_eval_stats\",\n help=\"skips saving eval statistics JSON file\",\n action=\"store_true\",\n default=False,\n )\n parser.add_argument(\n \"-ssm\",\n \"--skip_save_model\",\n action=\"store_true\",\n default=False,\n help=\"disables saving model weights and hyperparameters each time \"\n \"the model improves. \"\n \"By default Ludwig saves model weights after each epoch \"\n \"the validation metric improves, but if the model is really big \"\n \"that can be time consuming. If you do not want to keep \"\n \"the weights and just find out what performance a model can get \"\n \"with a set of hyperparameters, use this parameter to skip it,\"\n \"but the model will not be loadable later on\",\n )\n parser.add_argument(\n \"-ssp\",\n \"--skip_save_progress\",\n action=\"store_true\",\n default=False,\n help=\"disables saving progress each epoch. By default Ludwig saves \"\n \"weights and stats after each epoch for enabling resuming \"\n \"of training, but if the model is really big that can be \"\n \"time consuming and will uses twice as much space, use \"\n \"this parameter to skip it, but training cannot be resumed \"\n \"later on\",\n )\n parser.add_argument(\n \"-ssl\",\n \"--skip_save_log\",\n action=\"store_true\",\n default=False,\n help=\"disables saving TensorBoard logs. By default Ludwig saves \"\n \"logs for the TensorBoard, but if it is not needed turning it off \"\n \"can slightly increase the overall speed\",\n )\n\n # ------------------\n # Runtime parameters\n # ------------------\n parser.add_argument(\n \"-rs\",\n \"--random_seed\",\n type=int,\n default=42,\n help=\"a random seed that is going to be used anywhere there is a call \"\n \"to a random number generator: data splitting, parameter \"\n \"initialization and training set shuffling\",\n )\n parser.add_argument(\"-g\", \"--gpus\", nargs=\"+\", type=int, default=None, help=\"list of GPUs to use\")\n parser.add_argument(\n \"-gml\",\n \"--gpu_memory_limit\",\n type=float,\n default=None,\n help=\"maximum memory fraction [0, 1] allowed to allocate per GPU device\",\n )\n parser.add_argument(\n \"-dpt\",\n \"--disable_parallel_threads\",\n action=\"store_false\",\n dest=\"allow_parallel_threads\",\n help=\"disable PyTorch from using multithreading for reproducibility\",\n )\n parser.add_argument(\n \"-b\",\n \"--backend\",\n help=\"specifies backend to use for parallel / distributed execution, \" \"defaults to local execution\",\n choices=ALL_BACKENDS,\n )\n parser.add_argument(\n \"-l\",\n \"--logging_level\",\n default=\"info\",\n help=\"the level of logging to use\",\n choices=[\"critical\", \"error\", \"warning\", \"info\", \"debug\", \"notset\"],\n )\n\n add_contrib_callback_args(parser)\n args = parser.parse_args(sys_argv)\n\n args.callbacks = args.callbacks or []\n for callback in args.callbacks:\n callback.on_cmdline(\"experiment\", *sys_argv)\n\n args.logging_level = get_logging_level_registry()[args.logging_level]\n logging.getLogger(\"ludwig\").setLevel(args.logging_level)\n global logger\n logger = logging.getLogger(\"ludwig.experiment\")\n\n args.backend = initialize_backend(args.backend or args.config.get(\"backend\"))\n if args.backend.is_coordinator():\n print_ludwig(\"Experiment\", LUDWIG_VERSION)\n\n if args.k_fold is None:\n experiment_cli(**vars(args))\n else:\n kfold_cross_validate_cli(**vars(args))\n\n\nif __name__ == \"__main__\":\n cli(sys.argv[1:])\n"
},
{
"path": "ludwig/explain/__init__.py",
"content": ""
},
{
"path": "ludwig/explain/captum.py",
"content": "import copy\nimport gc\nimport logging\nfrom collections import defaultdict\nfrom dataclasses import dataclass\n\nimport numpy as np\nimport numpy.typing as npt\nimport pandas as pd\nimport torch\nfrom captum.attr import LayerIntegratedGradients, TokenReferenceBase\nfrom captum.attr._utils.input_layer_wrapper import InputIdentity\nfrom torch.autograd import Variable\nfrom tqdm import tqdm\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.api_annotations import PublicAPI\nfrom ludwig.constants import (\n BINARY,\n CATEGORY,\n DATE,\n IMAGE,\n INPUT_FEATURES,\n MINIMUM_BATCH_SIZE,\n NAME,\n NUMBER,\n PREPROCESSING,\n SEQUENCE,\n SET,\n TEXT,\n UNKNOWN_SYMBOL,\n)\nfrom ludwig.data.preprocessing import preprocess_for_prediction\nfrom ludwig.explain.explainer import Explainer\nfrom ludwig.explain.explanation import ExplanationsResult\nfrom ludwig.explain.util import get_pred_col, replace_layer_with_copy\nfrom ludwig.features.feature_utils import LudwigFeatureDict\nfrom ludwig.models.ecd import ECD\nfrom ludwig.utils.torch_utils import DEVICE\n\nlogger = logging.getLogger(__name__)\n\n# These types as provided as integer values and passed through an embedding layer that breaks integrated gradients.\n# As such, we need to take care to encode them before handing them to the explainer.\nEMBEDDED_TYPES = {SEQUENCE, TEXT, CATEGORY, SET, DATE}\n\n\n@dataclass\nclass ExplanationRunConfig:\n \"\"\"Mutable state containing runtime configuration for explanation process.\n\n This is useful for updating the batch size used during explanation so it can be propagated across calls to\n `get_total_attribution`.\n \"\"\"\n\n batch_size: int\n\n\ndef retry_with_halved_batch_size(run_config: ExplanationRunConfig):\n \"\"\"Function wrapper that retries an fn with a halved batch size.\n\n We want to maintain as large of a batch size as possible to maximize throughput. However, calculating explanations\n requires significantly more memory, and the original batch sized used during training may be too large and cause a\n CUDA OOM error, for example, if using GPUs.\n\n Will raise an error if a non-OOM error is raised, or if the batch size is reduced below 1 and the fn still fails.\n \"\"\"\n\n def retry_with_halved_batch_size_fn(fn):\n def retry_with_halved_batch_size_wrapper(*args, **kwargs):\n latest_error = None\n while run_config.batch_size >= MINIMUM_BATCH_SIZE:\n try:\n return fn(*args, **kwargs)\n except RuntimeError as e:\n latest_error = e\n # PyTorch only generates Runtime errors for CUDA OOM.\n gc.collect()\n if \"CUDA out of memory\" in str(e) or isinstance(e, torch.cuda.OutOfMemoryError):\n logger.exception(f\"OOM at batch_size={run_config.batch_size}, halving and trying again\")\n run_config.batch_size //= 2\n else:\n # Not a CUDA error\n raise\n\n raise RuntimeError(\n f\"Ran into latest error {latest_error} during explanation. \"\n \"If a CUDA out of memory error, then the batch size could not be reduced any further.\"\n )\n\n return retry_with_halved_batch_size_wrapper\n\n return retry_with_halved_batch_size_fn\n\n\nclass WrapperModule(torch.nn.Module):\n \"\"\"Model used by the explainer to generate predictions.\n\n Unlike Ludwig's ECD class, this wrapper takes individual args as inputs to the forward function. We derive the order\n of these args from the order of the input_feature keys in ECD, which is guaranteed to be consistent (Python\n dictionaries are ordered consistently), so we can map back to the input feature dictionary as a second step within\n this wrapper.\n \"\"\"\n\n def __init__(self, model: ECD, target: str):\n super().__init__()\n self.model = model\n self.target = target\n self.input_maps = LudwigFeatureDict()\n self.input_maps.update(\n {\n arg_name: InputIdentity(arg_name)\n for arg_name in self.model.input_features.keys()\n if self.model.input_features.get(arg_name).type() not in EMBEDDED_TYPES\n }\n )\n\n def forward(self, *args):\n # Add back the dictionary structure so it conforms to ECD format.\n input_features: LudwigFeatureDict = self.model.input_features\n inputs = {\n # Send the input through the identity layer so that we can use the output of the layer for attribution.\n # Except for text/category features where we use the embedding layer for attribution.\n feat_name: (\n feat_input\n if input_features.get(feat_name).type() in EMBEDDED_TYPES\n else self.input_maps.get(feat_name)(feat_input)\n )\n for feat_name, feat_input in zip(input_features.keys(), args)\n }\n\n outputs = self.model(inputs)\n\n # At this point we only have the raw logits, but to make explainability work we need the probabilities\n # and predictions as well, so derive them.\n predictions = {}\n for of_name in self.model.output_features:\n predictions[of_name] = self.model.output_features.get(of_name).predictions(outputs, of_name)\n\n pred_t = get_pred_col(predictions, self.target)\n\n # If the target feature is a non-scalar type (vector, set, etc.), sum it to get a scalar value.\n # https://github.com/pytorch/captum/issues/377\n if len(pred_t.shape) > 1 and self.model.output_features.get(self.target).type() not in {\n CATEGORY,\n NUMBER,\n BINARY,\n }:\n pred_t = torch.sum(pred_t.reshape(pred_t.shape[0], -1), dim=1)\n\n return pred_t\n\n\n@PublicAPI(stability=\"experimental\")\nclass IntegratedGradientsExplainer(Explainer):\n def explain(self) -> ExplanationsResult:\n \"\"\"Explain the model's predictions using Integrated Gradients.\n\n # Return\n\n :return: ExplanationsResult containing the explanations.\n `global_explanations`: (Explanation) Aggregate explanation for the entire input data.\n\n `row_explanations`: (List[Explanation]) A list of explanations, one for each row in the input data. Each\n explanation contains the integrated gradients for each label in the target feature's vocab with respect to\n each input feature.\n\n `expected_values`: (List[float]) of length [output feature cardinality] Average convergence delta for each\n label in the target feature's vocab.\n \"\"\"\n\n # TODO(travis): add back skip encoders at the end in finally. Shouldn't be an issue in most cases as we\n # typically perform explanations on a loaded model and don't use it to predict afterwards.\n self.model.model.unskip()\n self.model.model.to(DEVICE)\n\n input_features: LudwigFeatureDict = self.model.model.input_features\n run_config = ExplanationRunConfig(batch_size=self.model.config_obj.trainer.batch_size)\n\n get_input_tensors_with_retry = retry_with_halved_batch_size(run_config)(get_input_tensors)\n get_total_attribution_with_retry = retry_with_halved_batch_size(run_config)(get_total_attribution)\n\n # Convert input data into embedding tensors from the output of the model encoders.\n inputs_encoded = get_input_tensors_with_retry(self.model, self.inputs_df, run_config)\n sample_encoded = get_input_tensors_with_retry(self.model, self.sample_df, run_config)\n baseline = get_baseline(self.model, sample_encoded)\n\n # Compute attribution for each possible output feature label separately.\n expected_values = []\n for target_idx in tqdm(range(self.vocab_size), desc=\"Explain\"):\n total_attribution, feat_to_token_attributions, total_attribution_global = get_total_attribution_with_retry(\n self.model,\n self.target_feature_name,\n target_idx if self.is_category_target else None,\n inputs_encoded,\n baseline,\n len(self.inputs_df),\n run_config,\n )\n\n # Aggregate token attributions\n feat_to_token_attributions_global = {}\n for feat_name, token_attributions in feat_to_token_attributions.items():\n token_attributions_global = defaultdict(float)\n # sum attributions for each token\n for token, token_attribution in (ta for tas in token_attributions for ta in tas):\n token_attributions_global[token] += abs(token_attribution)\n # divide by number of samples to get average attribution per token\n token_attributions_global = {\n token: token_attribution / max(0, len(token_attributions))\n for token, token_attribution in token_attributions_global.items()\n }\n # convert to list of tuples and sort by attribution\n token_attributions_global = sorted(token_attributions_global.items(), key=lambda x: x[1], reverse=True)\n # keep only top 100 tokens\n token_attributions_global = token_attributions_global[:100]\n feat_to_token_attributions_global[feat_name] = token_attributions_global\n\n self.global_explanation.add(\n input_features.keys(), total_attribution_global, feat_to_token_attributions_global\n )\n\n for i, (feature_attributions, explanation) in enumerate(zip(total_attribution, self.row_explanations)):\n # Add the feature attributions to the explanation object for this row.\n explanation.add(\n input_features.keys(),\n feature_attributions,\n {k: v[i] for k, v in feat_to_token_attributions.items()},\n )\n\n # TODO(travis): for force plots, need something similar to SHAP E[X]\n expected_values.append(0.0)\n\n if self.is_binary_target:\n # For binary targets, we only need to compute attribution for the positive class (see below).\n break\n\n # For binary targets, add an extra attribution for the negative class (false).\n if self.is_binary_target:\n le_true = self.global_explanation.label_explanations[0]\n negated_attributions = le_true.to_array() * -1\n negated_token_attributions = {\n fa.feature_name: [(t, -a) for t, a in fa.token_attributions]\n for fa in le_true.feature_attributions\n if fa.token_attributions is not None\n }\n # Prepend the negative class to the list of label explanations.\n self.global_explanation.add(\n input_features.keys(), negated_attributions, negated_token_attributions, prepend=True\n )\n\n for explanation in self.row_explanations:\n le_true = explanation.label_explanations[0]\n negated_attributions = le_true.to_array() * -1\n negated_token_attributions = {\n fa.feature_name: [(t, -a) for t, a in fa.token_attributions]\n for fa in le_true.feature_attributions\n if fa.token_attributions is not None\n }\n # Prepend the negative class to the list of label explanations.\n explanation.add(input_features.keys(), negated_attributions, negated_token_attributions, prepend=True)\n\n # TODO(travis): for force plots, need something similar to SHAP E[X]\n expected_values.append(0.0)\n\n return ExplanationsResult(self.global_explanation, self.row_explanations, expected_values)\n\n\ndef get_input_tensors(\n model: LudwigModel, input_set: pd.DataFrame, run_config: ExplanationRunConfig\n) -> list[torch.Tensor]:\n \"\"\"Convert the input data into a list of variables, one for each input feature.\n\n # Inputs\n\n :param model: The LudwigModel to use for encoding.\n :param input_set: The input data to encode of shape [batch size, num input features]. # Return\n :return: A list of variables, one for each input feature. Shape of each variable is [batch size, embedding size].\n \"\"\"\n # Ignore sample_ratio and sample_size from the model config, since we want to explain all the data.\n sample_ratio_bak = model.config_obj.preprocessing.sample_ratio\n sample_size_bak = model.config_obj.preprocessing.sample_size\n model.config_obj.preprocessing.sample_ratio = 1.0\n model.config_obj.preprocessing.sample_size = None\n\n config = model.config_obj.to_dict()\n training_set_metadata = copy.deepcopy(model.training_set_metadata)\n for feature in config[INPUT_FEATURES]:\n preprocessing = training_set_metadata[feature[NAME]][PREPROCESSING]\n if preprocessing.get(\"cache_encoder_embeddings\"):\n preprocessing[\"cache_encoder_embeddings\"] = False\n\n # Convert raw input data into preprocessed tensor data\n dataset, _ = preprocess_for_prediction(\n config,\n dataset=input_set,\n training_set_metadata=training_set_metadata,\n data_format=\"auto\",\n split=\"full\",\n include_outputs=False,\n backend=model.backend,\n callbacks=model.callbacks,\n )\n\n # Restore sample_ratio and sample_size\n model.config_obj.preprocessing.sample_ratio = sample_ratio_bak\n model.config_obj.preprocessing.sample_size = sample_size_bak\n\n # Make sure the number of rows in the preprocessed dataset matches the number of rows in the input data\n assert (\n dataset.to_df().shape[0] == input_set.shape[0]\n ), f\"Expected {input_set.shape[0]} rows in preprocessed dataset, but got {dataset.to_df().shape[0]}\"\n\n # Convert dataset into a dict of tensors, and split each tensor into batches to control GPU memory usage\n inputs = {\n name: torch.from_numpy(dataset.dataset[feature.proc_column]).split(run_config.batch_size)\n for name, feature in model.model.input_features.items()\n }\n\n # Dict of lists to list of dicts\n input_batches = [dict(zip(inputs, t)) for t in zip(*inputs.values())]\n\n # List of dicts to dict of lists\n preproc_inputs = {k: torch.cat([d[k] for d in input_batches]) for k in input_batches[0]}\n\n data_to_predict = [v for _, v in preproc_inputs.items()]\n tensors = []\n for t in data_to_predict:\n # TODO(travis): Consider changing to `if not torch.is_floating_point(t.dtype)` to simplify, then handle bool\n # case in this block.\n if t.dtype == torch.int8 or t.dtype == torch.int16 or t.dtype == torch.int32 or t.dtype == torch.int64:\n # Don't wrap input into a variable if it's an integer type, since it will be used as an index into the\n # embedding table. We explain the output of the embedding table, not the input to the embedding table using\n # LayerIntegratedGradients.\n tensors.append(t)\n else:\n # Wrap input into a variable so torch will track the gradient and LayerIntegratedGradients can explain it.\n if t.dtype == torch.bool:\n t = t.to(torch.float32)\n tensors.append(Variable(t, requires_grad=True))\n\n return tensors\n\n\ndef get_baseline(model: LudwigModel, sample_encoded: list[Variable]) -> list[torch.Tensor]:\n # TODO(travis): pre-compute this during training from the full training dataset.\n input_features: LudwigFeatureDict = model.model.input_features\n\n baselines = []\n for sample_input, (name, feature) in zip(sample_encoded, input_features.items()):\n metadata = model.training_set_metadata[name]\n if feature.type() == TEXT:\n PAD_IND = metadata.get(\"pad_idx\", metadata.get(\"word_pad_idx\"))\n token_reference = TokenReferenceBase(reference_token_idx=PAD_IND)\n baseline = token_reference.generate_reference(sequence_length=sample_input.shape[1], device=DEVICE)\n elif feature.type() == CATEGORY:\n most_popular_token = max(metadata[\"str2freq\"], key=metadata[\"str2freq\"].get)\n most_popular_tok_idx = metadata[\"str2idx\"].get(most_popular_token)\n\n # If an unknown is defined, use that as the baseline index, else use the most popular token\n baseline_tok_idx = metadata[\"str2idx\"].get(UNKNOWN_SYMBOL, most_popular_tok_idx)\n baseline = torch.tensor(baseline_tok_idx, device=DEVICE)\n elif feature.type() == IMAGE:\n baseline = torch.zeros_like(sample_input[0], device=DEVICE)\n else:\n # For a robust baseline, we take the mean of all samples from the training data.\n baseline = torch.mean(sample_input.float(), dim=0)\n baselines.append(baseline.unsqueeze(0))\n\n return baselines\n\n\ndef get_total_attribution(\n model: LudwigModel,\n target_feature_name: str,\n target_idx: int | None,\n feature_inputs: list[Variable],\n baseline: list[torch.Tensor],\n nsamples: int,\n run_config: ExplanationRunConfig,\n) -> tuple[npt.NDArray[np.float64], dict[str, list[list[tuple[str, float]]]]]:\n \"\"\"Compute the total attribution for each input feature for each row in the input data.\n\n Args:\n model: The Ludwig model to explain.\n target_feature_name: The name of the target feature to explain.\n target_idx: The index of the target feature label to explain if the target feature is a category.\n feature_inputs: The preprocessed input data as a list of tensors of length [num_features].\n baseline: The baseline input data as a list of tensors of length [num_features].\n nsamples: The total number of samples in the input data.\n\n Returns:\n The token-attribution pair for each token in the input feature for each row in the input data. The members of\n the output tuple are structured as follows:\n\n `total_attribution_rows`: (npt.NDArray[np.float64]) of shape [num_rows, num_features]\n The total attribution for each input feature for each row in the input data.\n\n `feat_to_token_attributions`: (Dict[str, List[List[Tuple[str, float]]]]) with values of shape\n [num_rows, seq_len, 2]\n\n `total_attribution_global`: (npt.NDArray[np.float64]) of shape [num_features]\n The attribution for each input feature aggregated across all input data.\n \"\"\"\n input_features: LudwigFeatureDict = model.model.input_features\n\n # Configure the explainer, which includes wrapping the model so its interface conforms to\n # the format expected by Captum.\n model.model.zero_grad()\n explanation_model = WrapperModule(model.model, target_feature_name)\n\n layers = []\n for feat_name, feat in input_features.items():\n if feat.type() in EMBEDDED_TYPES:\n # Get embedding layer from encoder, which is the first child of the encoder.\n target_layer = feat.encoder_obj.get_embedding_layer()\n\n # If the current layer matches any layer in the list, make a deep copy of the layer.\n if len(layers) > 0 and any(target_layer == layer for layer in layers):\n # Replace the layer with a deep copy of the layer to ensure that the attributions unique for each input\n # feature that uses a shared layer.\n # Recommended here: https://github.com/pytorch/captum/issues/794#issuecomment-1093021638\n replace_layer_with_copy(feat, target_layer)\n target_layer = feat.encoder_obj.get_embedding_layer() # get the new copy\n else:\n # Get the wrapped input layer.\n target_layer = explanation_model.input_maps.get(feat_name)\n\n layers.append(target_layer)\n\n explainer = LayerIntegratedGradients(explanation_model, layers)\n\n feature_inputs_splits = [ipt.split(run_config.batch_size) for ipt in feature_inputs]\n baseline = [t.to(DEVICE) for t in baseline]\n\n total_attribution_rows = None\n total_attribution_global = None\n feat_to_token_attributions = defaultdict(list)\n for input_batch in zip(*feature_inputs_splits):\n input_batch = [ipt.to(DEVICE) for ipt in input_batch]\n attribution = explainer.attribute(\n tuple(input_batch),\n baselines=tuple(baseline),\n target=target_idx,\n # https://captum.ai/docs/faq#i-am-facing-out-of-memory-oom-errors-when-using-captum-how-do-i-resolve-this\n internal_batch_size=run_config.batch_size,\n )\n\n attributions_reduced = []\n for a in attribution:\n a_reduced = a.detach().cpu()\n if a_reduced.ndim == 2 or a_reduced.ndim == 3:\n # Reduces category-level attributions of shape [batch_size, embedding_dim] by summing over the\n # embedding dimension to get attributions of shape [batch_size].\n # Reduces token-level attributions of shape [batch_size, sequence_length, embedding_dim] by summing\n # over the embedding dimension to get attributions of shape [batch_size, sequence_length]. We keep\n # the sequence dimension so we can map the attributions to the tokens.\n a_reduced = a_reduced.sum(dim=-1)\n elif a_reduced.ndim == 4:\n # Reduce pixel-level attributions of shape [batch_size, num_channels, height, width] by summing\n # over the channel and spatial dimensions to get attributions of shape [batch_size].\n a_reduced = a_reduced.sum(dim=(1, 2, 3))\n attributions_reduced.append(a_reduced)\n\n for inputs, attrs, (name, feat) in zip(input_batch, attributions_reduced, input_features.items()):\n if feat.type() == TEXT:\n tok_attrs = get_token_attributions(model, name, inputs.detach().cpu(), attrs)\n feat_to_token_attributions[name].append(tok_attrs)\n\n # Reduce attribution to [num_input_features, batch_size] by summing over the sequence dimension (if present).\n attribution = [a.sum(dim=-1) if a.ndim == 2 else a for a in attributions_reduced]\n attribution = np.stack(attribution)\n\n # Transpose to [batch_size, num_input_features]\n attribution = attribution.T\n\n if total_attribution_rows is not None:\n total_attribution_rows = np.concatenate([total_attribution_rows, attribution], axis=0)\n else:\n total_attribution_rows = attribution\n\n if total_attribution_global is not None:\n total_attribution_global += attribution.sum(axis=0)\n else:\n total_attribution_global = attribution.sum(axis=0)\n\n total_attribution_global /= nsamples\n\n feat_to_token_attributions = {k: [e for lst in v for e in lst] for k, v in feat_to_token_attributions.items()}\n\n return total_attribution_rows, feat_to_token_attributions, total_attribution_global\n\n\ndef get_token_attributions(\n model: LudwigModel,\n feature_name: str,\n input_ids: torch.Tensor,\n token_attributions: torch.Tensor,\n) -> list[list[tuple[str, float]]]:\n \"\"\"Convert token-level attributions to an array of token-attribution pairs of shape.\n\n [batch_size, sequence_length, 2].\n\n Args:\n model: The LudwigModel used to generate the attributions.\n feature_name: The name of the feature for which the attributions were generated.\n input_ids: The input ids of shape [batch_size, sequence_length].\n token_attributions: The token-level attributions of shape [batch_size, sequence_length].\n\n Returns:\n An array of token-attribution pairs of shape [batch_size, sequence_length, 2].\n \"\"\"\n assert (\n input_ids.dtype == torch.int8\n or input_ids.dtype == torch.int16\n or input_ids.dtype == torch.int32\n or input_ids.dtype == torch.int64\n )\n\n # Normalize token-level attributions to visualize the relative importance of each token.\n norm = torch.linalg.norm(token_attributions, dim=1)\n # Safe divide by zero by setting the norm to 1 if the norm is 0.\n norm = torch.where(norm == 0, torch.ones_like(norm), norm)\n token_attributions = token_attributions / norm.unsqueeze(-1)\n\n # map input ids to input tokens via the vocabulary\n feature = model.training_set_metadata[feature_name]\n vocab = feature.get(\"idx2str\", feature.get(\"word_idx2str\"))\n idx2str = np.vectorize(lambda idx: vocab[idx])\n input_tokens = idx2str(input_ids)\n\n # add attribution to the input tokens\n tok_attrs = [\n list(zip(t, a)) for t, a in zip(input_tokens, token_attributions.tolist())\n ] # [batch_size, sequence_length, 2]\n\n return tok_attrs\n"
},
{
"path": "ludwig/explain/captum_ray.py",
"content": "from collections import defaultdict\nfrom typing import Any\n\nimport numpy as np\nimport pandas as pd\nimport ray\nfrom torch.autograd import Variable\nfrom tqdm import tqdm\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.api_annotations import PublicAPI\nfrom ludwig.explain.captum import (\n ExplanationRunConfig,\n get_baseline,\n get_input_tensors,\n get_total_attribution,\n IntegratedGradientsExplainer,\n retry_with_halved_batch_size,\n)\nfrom ludwig.explain.explanation import ExplanationsResult\nfrom ludwig.features.feature_utils import LudwigFeatureDict\nfrom ludwig.utils.torch_utils import get_torch_device\n\n\n@PublicAPI(stability=\"experimental\")\nclass RayIntegratedGradientsExplainer(IntegratedGradientsExplainer):\n def __init__(self, *args, resources_per_task: dict[str, Any] = None, num_workers: int = 1, **kwargs):\n super().__init__(*args, **kwargs)\n self.resources_per_task = resources_per_task or {}\n self.num_workers = num_workers\n\n def explain(self) -> ExplanationsResult:\n \"\"\"Explain the model's predictions using Integrated Gradients.\n\n # Return\n\n :return: ExplanationsResult containing the explanations.\n `global_explanations`: (Explanation) Aggregate explanation for the entire input data.\n\n `row_explanations`: (List[Explanation]) A list of explanations, one for each row in the input data. Each\n explanation contains the integrated gradients for each label in the target feature's vocab with respect to\n each input feature.\n\n `expected_values`: (List[float]) of length [output feature cardinality] Average convergence delta for each\n label in the target feature's vocab.\n \"\"\"\n self.model.model.cpu()\n input_features: LudwigFeatureDict = self.model.model.input_features\n model_ref = ray.put(self.model)\n run_config = ExplanationRunConfig(batch_size=self.model.config_obj.trainer.batch_size)\n\n # Convert input data into embedding tensors from the output of the model encoders.\n inputs_encoded_ref = get_input_tensors_task.options(**self.resources_per_task).remote(\n model_ref, ray.put(self.inputs_df), run_config\n )\n sample_encoded_ref = get_input_tensors_task.options(**self.resources_per_task).remote(\n model_ref, ray.put(self.sample_df), run_config\n )\n\n inputs_encoded, run_config = ray.get(inputs_encoded_ref)\n sample_encoded, run_config = ray.get(sample_encoded_ref)\n baseline = get_baseline(self.model, sample_encoded)\n\n inputs_encoded_ref = ray.put(inputs_encoded)\n baseline_ref = ray.put(baseline)\n\n if self.is_category_target:\n # Evenly divide the list of labels among the desired number of workers (Ray tasks).\n # For example, 4 GPUs -> 4 workers. We do this instead of creating nlabels tasks because\n # there is significant overhead to spawning a Ray task.\n target_splits = split_list(list(range(self.vocab_size)), self.num_workers)\n else:\n # No target index to compare against exists for number features.\n # For binary targets, we only need to compute attribution for the positive class (see below).\n # May need to revisit in the future for additional feature types.\n target_splits = [[None]]\n\n # Compute attribution for each possible output feature label separately.\n attrs_refs = []\n for target_indices in target_splits:\n attrs_ref = get_total_attribution_task.options(**self.resources_per_task).remote(\n model_ref,\n self.target_feature_name,\n target_indices,\n inputs_encoded_ref,\n baseline_ref,\n len(self.inputs_df),\n run_config,\n )\n attrs_refs.append(attrs_ref)\n\n # Await the completion of our Ray tasks, then merge the results.\n expected_values = []\n for attrs_ref in tqdm(attrs_refs, desc=\"Explain\"):\n attrs = ray.get(attrs_ref)\n for total_attribution, feat_to_token_attributions, total_attribution_global in attrs:\n # Aggregate token attributions\n feat_to_token_attributions_global = {}\n for feat_name, token_attributions in feat_to_token_attributions.items():\n token_attributions_global = defaultdict(float)\n # sum attributions for each token\n for token, token_attribution in (ta for tas in token_attributions for ta in tas):\n token_attributions_global[token] += token_attribution\n # divide by number of samples to get average attribution per token\n token_attributions_global = {\n token: token_attribution / max(0, len(token_attributions))\n for token, token_attribution in token_attributions_global.items()\n }\n # convert to list of tuples and sort by attribution\n token_attributions_global = sorted(\n token_attributions_global.items(), key=lambda x: x[1], reverse=True\n )\n # keep only top 100 tokens\n token_attributions_global = token_attributions_global[:100]\n feat_to_token_attributions_global[feat_name] = token_attributions_global\n\n self.global_explanation.add(\n input_features.keys(), total_attribution_global, feat_to_token_attributions_global\n )\n\n for i, (feature_attributions, explanation) in enumerate(zip(total_attribution, self.row_explanations)):\n # Add the feature attributions to the explanation object for this row.\n explanation.add(\n input_features.keys(),\n feature_attributions,\n {k: v[i] for k, v in feat_to_token_attributions.items()},\n )\n\n # TODO(travis): for force plots, need something similar to SHAP E[X]\n expected_values.append(0.0)\n\n # For binary targets, add an extra attribution for the negative class (false).\n if self.is_binary_target:\n le_true = self.global_explanation.label_explanations[0]\n negated_attributions = le_true.to_array() * -1\n negated_token_attributions = {\n fa.feature_name: [(t, -a) for t, a in fa.token_attributions]\n for fa in le_true.feature_attributions\n if fa.token_attributions is not None\n }\n # Prepend the negative class to the list of label explanations.\n self.global_explanation.add(\n input_features.keys(), negated_attributions, negated_token_attributions, prepend=True\n )\n\n for explanation in self.row_explanations:\n le_true = explanation.label_explanations[0]\n negated_attributions = le_true.to_array() * -1\n negated_token_attributions = {\n fa.feature_name: [(t, -a) for t, a in fa.token_attributions]\n for fa in le_true.feature_attributions\n if fa.token_attributions is not None\n }\n # Prepend the negative class to the list of label explanations.\n explanation.add(input_features.keys(), negated_attributions, negated_token_attributions, prepend=True)\n\n # TODO(travis): for force plots, need something similar to SHAP E[X]\n expected_values.append(0.0)\n\n return ExplanationsResult(self.global_explanation, self.row_explanations, expected_values)\n\n\n@ray.remote(max_calls=1)\ndef get_input_tensors_task(\n model: LudwigModel, df: pd.DataFrame, run_config: ExplanationRunConfig\n) -> tuple[list[Variable], ExplanationRunConfig]:\n model.model.unskip()\n model.model.to(get_torch_device())\n try:\n get_total_attribution_with_retry = retry_with_halved_batch_size(run_config)(get_input_tensors)\n return get_total_attribution_with_retry(model, df, run_config), run_config\n finally:\n model.model.cpu()\n\n\n@ray.remote(max_calls=1)\ndef get_total_attribution_task(\n model: LudwigModel,\n target_feature_name: str,\n target_indices: list[int | None],\n inputs_encoded: list[Variable],\n baseline: list[Variable],\n nsamples: int,\n run_config: ExplanationRunConfig,\n) -> list[np.array]:\n model.model.unskip()\n model.model.to(get_torch_device())\n try:\n get_total_attribution_with_retry = retry_with_halved_batch_size(run_config)(get_total_attribution)\n return [\n get_total_attribution_with_retry(\n model=model,\n target_feature_name=target_feature_name,\n target_idx=target_idx,\n feature_inputs=inputs_encoded,\n baseline=baseline,\n nsamples=nsamples,\n run_config=run_config,\n )\n for target_idx in tqdm(target_indices, desc=\"Explain\")\n ]\n finally:\n model.model.cpu()\n\n\ndef split_list(v, n):\n \"\"\"Splits a list into n roughly equal sub-lists.\n\n Source: https://stackoverflow.com/a/2135920\n \"\"\"\n k, m = divmod(len(v), n)\n return (v[i * k + min(i, m) : (i + 1) * k + min(i + 1, m)] for i in range(n))\n"
},
{
"path": "ludwig/explain/explainer.py",
"content": "from abc import ABCMeta, abstractmethod\n\nimport pandas as pd\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import BINARY, CATEGORY, TYPE\nfrom ludwig.explain.explanation import Explanation, ExplanationsResult\nfrom ludwig.explain.util import prepare_data\n\n\n@DeveloperAPI\nclass Explainer(metaclass=ABCMeta):\n def __init__(\n self,\n model: LudwigModel,\n inputs_df: pd.DataFrame,\n sample_df: pd.DataFrame,\n target: str,\n ):\n \"\"\"Constructor for the explainer.\n\n # Inputs\n\n :param model: (LudwigModel) The LudwigModel to explain.\n :param inputs_df: (pd.DataFrame) The input data to explain.\n :param sample_df: (pd.DataFrame) A sample of the ground truth data.\n :param target: (str) The name of the target to explain.\n \"\"\"\n self.model = model\n self.inputs_df = inputs_df\n self.sample_df = sample_df\n self.target = target\n self.inputs_df, self.sample_df, self.feature_cols, self.target_feature_name = prepare_data(\n model, inputs_df, sample_df, target\n )\n\n self.global_explanation = Explanation(self.target_feature_name)\n self.row_explanations = [Explanation(self.target_feature_name) for _ in self.inputs_df.index]\n\n # Lookup from column name to output feature\n config = self.model.config\n self.output_feature_map = {feature[\"column\"]: feature for feature in config[\"output_features\"]}\n\n @property\n def is_binary_target(self) -> bool:\n \"\"\"Whether the target is binary.\"\"\"\n return self.output_feature_map[self.target_feature_name][TYPE] == BINARY\n\n @property\n def is_category_target(self) -> bool:\n \"\"\"Whether the target is categorical.\"\"\"\n return self.output_feature_map[self.target_feature_name][TYPE] == CATEGORY\n\n @property\n def vocab_size(self) -> int:\n \"\"\"The vocab size of the target feature.\n\n For regression (number) this is 1, for binary it is 2, and for category it is the vocab size.\n \"\"\"\n if self.is_category_target:\n return self.model.training_set_metadata[self.target_feature_name][\"vocab_size\"]\n elif self.is_binary_target:\n return 2\n return 1\n\n @abstractmethod\n def explain(self) -> ExplanationsResult:\n \"\"\"Explain the model's predictions.\n\n # Return\n\n :return: ExplanationsResult containing the explanations.\n \"\"\"\n"
},
{
"path": "ludwig/explain/explanation.py",
"content": "from dataclasses import dataclass, field\n\nimport numpy as np\nimport numpy.typing as npt\n\nfrom ludwig.api_annotations import DeveloperAPI, PublicAPI\n\n\n@DeveloperAPI\n@dataclass\nclass FeatureAttribution:\n \"\"\"Stores the attribution for a single input feature.\"\"\"\n\n # The name of the input feature.\n feature_name: str\n\n # The scalar attribution for the input feature.\n attribution: float\n\n # (Optional) The attribution for each token in the input feature as an array of shape (seq_len, 2).\n token_attributions: list[tuple[str, float]] = None\n\n\n@DeveloperAPI\n@dataclass\nclass LabelExplanation:\n \"\"\"Stores the feature attributions for a single label in the target feature's vocab.\"\"\"\n\n # The attribution for each input feature.\n feature_attributions: list[FeatureAttribution] = field(default_factory=list)\n\n def add(self, feature_name: str, attribution: float, token_attributions: list[tuple[str, float]] = None):\n \"\"\"Add the attribution for a single input feature.\"\"\"\n self.feature_attributions.append(FeatureAttribution(feature_name, attribution, token_attributions))\n\n def to_array(self) -> npt.NDArray[np.float64]:\n \"\"\"Convert the explanation to a 1D array of shape (num_features,).\"\"\"\n return np.array([fa.attribution for fa in self.feature_attributions])\n\n\n@DeveloperAPI\n@dataclass\nclass Explanation:\n \"\"\"Stores the explanations for a single row of input data.\n\n Contains the feature attributions for each label in the target feature's vocab.\n \"\"\"\n\n target: str\n\n # The explanations for each label in the vocab of the target feature.\n label_explanations: list[LabelExplanation] = field(default_factory=list)\n\n def add(\n self,\n feat_names: list[str],\n feat_attributions: npt.NDArray[np.float64],\n feat_to_token_attributions: dict[str, list[tuple[str, float]]] = None,\n prepend: bool = False,\n ):\n \"\"\"Add the feature attributions for a single label.\"\"\"\n assert len(feat_names) == len(\n feat_attributions\n ), f\"Expected {len(feat_names)} feature attributions, got {len(feat_attributions)}\"\n if len(self.label_explanations) > 0:\n # Check that the feature attributions are the same shape as existing explanations.\n assert self.label_explanations[0].to_array().shape == feat_attributions.shape, (\n f\"Expected feature attributions of shape {self.label_explanations[0].to_array().shape}, \"\n f\"got {feat_attributions.shape}\"\n )\n\n le = LabelExplanation()\n for i, feat_name in enumerate(feat_names):\n le.add(\n feat_name,\n feat_attributions[i],\n feat_to_token_attributions.get(feat_name) if feat_to_token_attributions else None,\n )\n self.label_explanations.insert(0, le) if prepend else self.label_explanations.append(le)\n\n def to_array(self) -> npt.NDArray[np.float64]:\n \"\"\"Convert the explanation to a 2D array of shape (num_labels, num_features).\"\"\"\n return np.array([le.to_array() for le in self.label_explanations])\n\n\n@PublicAPI(stability=\"experimental\")\n@dataclass\nclass ExplanationsResult:\n # Aggregate explanation for the entire input data.\n global_explanation: Explanation # GlobalExplanation\n\n # A list of explanations, one for each row in the input data.\n # Each explanation contains the feature attributions for each label in the target feature's vocab.\n row_explanations: list[Explanation]\n\n # Expected value for each label in the target feature's vocab.\n expected_values: list[float]\n"
},
{
"path": "ludwig/explain/util.py",
"content": "from copy import deepcopy\n\nimport pandas as pd\nimport torch\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.constants import COLUMN, INPUT_FEATURES, PREPROCESSING, SPLIT\nfrom ludwig.data.split import get_splitter\nfrom ludwig.features.base_feature import BaseFeature\n\n\ndef filter_cols(df, cols):\n cols = {c.lower() for c in cols}\n retain_cols = [c for c in df.columns if c.lower() in cols]\n return df[retain_cols]\n\n\ndef prepare_data(model: LudwigModel, inputs_df: pd.DataFrame, sample_df: pd.DataFrame, target: str):\n config = model.config\n feature_cols = [feature[COLUMN] for feature in config[INPUT_FEATURES]]\n if SPLIT in config.get(PREPROCESSING, {}):\n # Keep columns required for Ludwig preprocessing\n splitter = get_splitter(**config[PREPROCESSING][SPLIT])\n feature_cols += splitter.required_columns\n target_feature_name = get_feature_name(model, target)\n\n inputs_df = filter_cols(inputs_df, feature_cols)\n if sample_df is not None:\n sample_df = filter_cols(sample_df, feature_cols)\n\n return inputs_df, sample_df, feature_cols, target_feature_name\n\n\ndef get_pred_col(preds, target):\n t = target.lower()\n for c in preds.keys():\n if c.lower() == t:\n if \"probabilities\" in preds[c]:\n return preds[c][\"probabilities\"]\n else:\n return preds[c][\"predictions\"]\n raise ValueError(f\"Unable to find target column {t} in {preds.keys()}\")\n\n\ndef get_feature_name(model: LudwigModel, target: str) -> str:\n t = target.lower()\n for c in model.training_set_metadata.keys():\n if c.lower() == t:\n return c\n raise ValueError(f\"Unable to find target column {t} in {model.training_set_metadata.keys()}\")\n\n\ndef get_absolute_module_key_from_submodule(module: torch.nn.Module, submodule: torch.nn.Module):\n \"\"\"Get the absolute module key for each param in the target layer.\n\n Assumes that the keys in the submodule are relative to the module.\n\n We find the params from the submodule in the module by comparing the data\n pointers, since the data returned by named_parameters is by reference.\n More information on checking if tensors point to the same place in storage can be found here:\n https://discuss.pytorch.org/t/any-way-to-check-if-two-tensors-have-the-same-base/44310/2\n \"\"\"\n absolute_keys = []\n for module_key, module_param in module.named_parameters():\n for _, submodule_param in submodule.named_parameters():\n if submodule_param.data_ptr() == module_param.data_ptr():\n absolute_keys.append(module_key)\n break\n return absolute_keys\n\n\ndef replace_layer_with_copy(feat: BaseFeature, target_layer: torch.nn.Module):\n \"\"\"Replaces a layer in a feature with a copy of the layer in-place.\n\n This is useful in a tied weights scenario, where a single encoder may be used by multiple features. If we leave\n as-is, Captum complains about the resulting computation graph. The solution is to create an identical\n (deep) copy of the layer fed into Captum: https://github.com/pytorch/captum/issues/794#issuecomment-1093021638\n\n This is safe to do during the explain step because we are essentially running inference, and no model artifacts are\n being saved during the explain step.\n\n TODO(geoffrey): if a user ever wants to train immediately after explain (i.e. w/o loading weights from the disk),\n we might want to implement this as a context so that we can restore the original encoder object at the end.\n Will defer this implementation for now because that scenario seems unlikely.\n\n At a high-level the approach is the following:\n 1. Create a deep-copy of the entire encoder object and set it as the feature's encoder object\n 2. Replace the tensors in the copied encoder object with the tensors from the original encoder object, except for\n the tensors in the target layer. We want to explain these tensors, so we want to keep them as deep copies.\n\n This approach ensures that at most 2 copies of the encoder object are in memory at any given time.\n \"\"\"\n with torch.no_grad():\n # Get the original encoder object and a mapping from param names to the params themselves.\n orig_encoder_obj = feat.encoder_obj\n orig_encoder_obj_state_dict = orig_encoder_obj.state_dict()\n\n # Deep copy the original encoder object and set the copy as this feature's encoder object.\n copy_encoder_obj = deepcopy(orig_encoder_obj)\n feat.encoder_obj = copy_encoder_obj\n\n # We have to get the absolute module key in order to do string matching because the target_layer keys are\n # relative to itself. If we were to leave it as-is and attempt to suffix match, we may get duplicates for\n # common layers i.e. \"LayerNorm.weight\" and \"LayerNorm.bias\". Getting the absolute module key ensures we\n # use values like \"transformer.module.embedding.LayerNorm.weight\" instead.\n keys_to_keep_copy = get_absolute_module_key_from_submodule(orig_encoder_obj, target_layer)\n\n # Get the tensors to keep from the copied encoder object. These are the tensors in the target layer.\n for key, param in copy_encoder_obj.named_parameters():\n if key not in keys_to_keep_copy:\n param.data = orig_encoder_obj_state_dict[key].data\n"
},
{
"path": "ludwig/export.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport argparse\nimport logging\nimport os\nimport sys\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.contrib import add_contrib_callback_args\nfrom ludwig.globals import LUDWIG_VERSION\nfrom ludwig.utils.print_utils import get_logging_level_registry, print_ludwig\nfrom ludwig.utils.triton_utils import export_triton as utils_export_triton\n\nlogger = logging.getLogger(__name__)\n\n\ndef export_torchscript(\n model_path: str, model_only: bool = False, output_path: str | None = None, device: str | None = None, **kwargs\n) -> None:\n \"\"\"Exports a model to torchscript.\n\n # Inputs\n\n :param model_path: (str) filepath to pre-trained model.\n :param model_only: (bool, default: `False`) If true, scripts and exports the model only.\n :param output_path: directory to store torchscript. If `None`, defaults to model_path\n\n # Return\n :returns: (`None`)\n \"\"\"\n logger.info(f\"Model path: {model_path}\")\n logger.info(f\"Saving model only: {model_only}\")\n\n if output_path is None:\n logger.info(\"output_path is None, defaulting to model_path\")\n output_path = model_path\n logger.info(f\"Output path: {output_path}\")\n logger.info(\"\\n\")\n\n model = LudwigModel.load(model_path)\n os.makedirs(output_path, exist_ok=True)\n model.save_torchscript(output_path, model_only=model_only, device=device)\n\n logger.info(f\"Saved to: {output_path}\")\n\n\ndef export_triton(model_path, output_path=\"model_repository\", model_name=\"ludwig_model\", model_version=1, **kwargs):\n \"\"\"Exports a model in torchscript format with config for Triton serving.\n\n # Inputs\n\n :param model_path: (str) filepath to pre-trained model.\n :param output_path: (str, default: `'model_repository'`) directory to store the\n triton models.\n :param model_name: (str, default: `'ludwig_model'`) save triton under this name.\n :param model_name: (int, default: `1`) save model under this verison.\n\n # Return\n\n :returns: (`None`)\n \"\"\"\n logger.info(f\"Model path: {model_path}\")\n logger.info(f\"Output path: {output_path}\")\n logger.info(f\"Model name: {model_name}\")\n logger.info(f\"Model version: {model_version}\")\n logger.info(\"\\n\")\n\n model = LudwigModel.load(model_path)\n os.makedirs(output_path, exist_ok=True)\n\n utils_export_triton(model=model, output_path=output_path, model_name=model_name, model_version=model_version)\n\n logger.info(f\"Saved to: {output_path}\")\n\n\ndef export_mlflow(model_path, output_path=\"mlflow\", registered_model_name=None, **kwargs):\n \"\"\"Exports a model to MLflow.\n\n # Inputs\n\n :param model_path: (str) filepath to pre-trained model.\n :param output_path: (str, default: `'mlflow'`) directory to store the\n mlflow model.\n :param registered_model_name: (str, default: `None`) save mlflow under this\n name in the model registry. Saved locally if `None`.\n\n # Return\n\n :returns: (`None`)\n \"\"\"\n logger.info(f\"Model path: {model_path}\")\n logger.info(f\"Output path: {output_path}\")\n logger.info(\"\\n\")\n\n from ludwig.contribs.mlflow.model import export_model\n\n export_model(model_path, output_path, registered_model_name)\n\n logger.info(f\"Saved to: {output_path}\")\n\n\ndef cli_export_torchscript(sys_argv):\n parser = argparse.ArgumentParser(\n description=\"This script loads a pretrained model \" \"and saves it as torchscript.\",\n prog=\"ludwig export_torchscript\",\n usage=\"%(prog)s [options]\",\n )\n\n # ----------------\n # Model parameters\n # ----------------\n parser.add_argument(\"-m\", \"--model_path\", help=\"model to load\", required=True)\n parser.add_argument(\n \"-mo\",\n \"--model_only\",\n help=\"Script and export the model only.\",\n action=\"store_true\",\n )\n parser.add_argument(\n \"-d\",\n \"--device\",\n type=str,\n help=(\n 'Device to use for torchscript tracing (e.g. \"cuda\" or \"cpu\"). Ideally, this is the same as the device '\n \"used when the model is loaded.\"\n ),\n default=None,\n )\n\n # -----------------\n # Output parameters\n # -----------------\n parser.add_argument(\n \"-op\",\n \"--output_path\",\n type=str,\n help=\"path where to save the export model. If not specified, defaults to model_path.\",\n default=None,\n )\n\n # ------------------\n # Runtime parameters\n # ------------------\n parser.add_argument(\n \"-l\",\n \"--logging_level\",\n default=\"info\",\n help=\"the level of logging to use\",\n choices=[\"critical\", \"error\", \"warning\", \"info\", \"debug\", \"notset\"],\n )\n\n add_contrib_callback_args(parser)\n args = parser.parse_args(sys_argv)\n\n args.callbacks = args.callbacks or []\n for callback in args.callbacks:\n callback.on_cmdline(\"export_torchscript\", *sys_argv)\n\n args.logging_level = get_logging_level_registry()[args.logging_level]\n logging.getLogger(\"ludwig\").setLevel(args.logging_level)\n global logger\n logger = logging.getLogger(\"ludwig.export\")\n\n print_ludwig(\"Export Torchscript\", LUDWIG_VERSION)\n\n export_torchscript(**vars(args))\n\n\ndef cli_export_triton(sys_argv):\n parser = argparse.ArgumentParser(\n description=\"This script loads a pretrained model \" \"and saves it as torchscript for Triton.\",\n prog=\"ludwig export_triton\",\n usage=\"%(prog)s [options]\",\n )\n\n # ----------------\n # Model parameters\n # ----------------\n parser.add_argument(\"-m\", \"--model_path\", help=\"model to load\", required=True)\n parser.add_argument(\"-mn\", \"--model_name\", help=\"model name\", default=\"ludwig_model\")\n parser.add_argument(\"-mv\", \"--model_version\", type=int, help=\"model version\", default=1)\n\n # -----------------\n # Output parameters\n # -----------------\n parser.add_argument(\"-op\", \"--output_path\", type=str, help=\"path where to save the export model\", required=True)\n\n # ------------------\n # Runtime parameters\n # ------------------\n parser.add_argument(\n \"-l\",\n \"--logging_level\",\n default=\"info\",\n help=\"the level of logging to use\",\n choices=[\"critical\", \"error\", \"warning\", \"info\", \"debug\", \"notset\"],\n )\n\n add_contrib_callback_args(parser)\n args = parser.parse_args(sys_argv)\n\n args.callbacks = args.callbacks or []\n for callback in args.callbacks:\n callback.on_cmdline(\"export_triton\", *sys_argv)\n\n args.logging_level = get_logging_level_registry()[args.logging_level]\n logging.getLogger(\"ludwig\").setLevel(args.logging_level)\n global logger\n logger = logging.getLogger(\"ludwig.export\")\n\n print_ludwig(\"Export Triton\", LUDWIG_VERSION)\n\n export_triton(**vars(args))\n\n\ndef cli_export_mlflow(sys_argv):\n parser = argparse.ArgumentParser(\n description=\"This script loads a pretrained model \" \"and saves it as an MLFlow model.\",\n prog=\"ludwig export_mlflow\",\n usage=\"%(prog)s [options]\",\n )\n\n # ----------------\n # Model parameters\n # ----------------\n parser.add_argument(\"-m\", \"--model_path\", help=\"model to load\", required=True)\n parser.add_argument(\n \"-mn\",\n \"--registered_model_name\",\n help=\"model name to upload to in MLflow model registry\",\n )\n\n # -----------------\n # Output parameters\n # -----------------\n parser.add_argument(\n \"-op\", \"--output_path\", type=str, help=\"path where to save the exported model\", default=\"mlflow\"\n )\n\n # ------------------\n # Runtime parameters\n # ------------------\n parser.add_argument(\n \"-l\",\n \"--logging_level\",\n default=\"info\",\n help=\"the level of logging to use\",\n choices=[\"critical\", \"error\", \"warning\", \"info\", \"debug\", \"notset\"],\n )\n\n add_contrib_callback_args(parser)\n args = parser.parse_args(sys_argv)\n\n args.callbacks = args.callbacks or []\n for callback in args.callbacks:\n callback.on_cmdline(\"export_mlflow\", *sys_argv)\n\n args.logging_level = get_logging_level_registry()[args.logging_level]\n logging.getLogger(\"ludwig\").setLevel(args.logging_level)\n global logger\n logger = logging.getLogger(\"ludwig.export\")\n\n print_ludwig(\"Export MLFlow\", LUDWIG_VERSION)\n\n export_mlflow(**vars(args))\n\n\nif __name__ == \"__main__\":\n if len(sys.argv) > 1:\n if sys.argv[1] == \"savedmodel\":\n cli_export_torchscript(sys.argv[2:])\n elif sys.argv[1] == \"mlflow\":\n cli_export_mlflow(sys.argv[2:])\n elif sys.argv[1] == \"triton\":\n cli_export_triton(sys.argv[2:])\n else:\n print(\"Unrecognized command\")\n else:\n print(\"Unrecognized command\")\n"
},
{
"path": "ludwig/features/__init__.py",
"content": ""
},
{
"path": "ludwig/features/audio_feature.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nimport os\n\nimport numpy as np\nimport torch\nimport torchaudio\nfrom packaging import version\n\nfrom ludwig.constants import AUDIO, AUDIO_FEATURE_KEYS, COLUMN, NAME, PREPROCESSING, PROC_COLUMN, SRC, TYPE\nfrom ludwig.features.base_feature import BaseFeatureMixin\nfrom ludwig.features.sequence_feature import SequenceInputFeature\nfrom ludwig.schema.features.audio_feature import AudioInputFeatureConfig\nfrom ludwig.types import FeatureMetadataDict, ModelConfigDict, PreprocessingConfigDict, TrainingSetMetadataDict\nfrom ludwig.utils.audio_utils import (\n calculate_mean,\n calculate_var,\n get_default_audio,\n get_fbank,\n get_group_delay,\n get_length_in_samp,\n get_max_length_stft_based,\n get_non_symmetric_length,\n get_phase_stft_magnitude,\n get_stft_magnitude,\n is_torch_audio_tuple,\n read_audio_from_bytes_obj,\n read_audio_from_path,\n)\nfrom ludwig.utils.data_utils import get_abs_path\nfrom ludwig.utils.fs_utils import has_remote_protocol\nfrom ludwig.utils.misc_utils import set_default_value\nfrom ludwig.utils.types import TorchscriptPreprocessingInput\n\nlogger = logging.getLogger(__name__)\n\n_TORCH_200 = version.parse(torch.__version__) >= version.parse(\"2.0.0\")\n\n\nclass _AudioPreprocessing(torch.nn.Module):\n audio_feature_dict: dict[str, float | int | str]\n\n def __init__(self, metadata: TrainingSetMetadataDict):\n super().__init__()\n self.audio_feature_dict = {\n key: value\n for key, value in metadata[\"preprocessing\"].items()\n if key in AUDIO_FEATURE_KEYS and value is not None\n }\n self.feature_dim = metadata[\"feature_dim\"]\n self.max_length = metadata[\"max_length\"]\n self.padding_value = metadata[\"preprocessing\"][\"padding_value\"]\n self.normalization_type = metadata[\"preprocessing\"][\"norm\"]\n\n def forward(self, v: TorchscriptPreprocessingInput) -> torch.Tensor:\n if not torch.jit.isinstance(v, list[tuple[torch.Tensor, int]]):\n raise ValueError(f\"Unsupported input: {v}\")\n\n processed_audio_matrix = []\n for audio, sampling_rate_in_hz in v:\n processed_audio = AudioFeatureMixin._transform_to_feature(\n audio,\n sampling_rate_in_hz,\n self.audio_feature_dict,\n self.feature_dim,\n self.max_length,\n self.padding_value,\n self.normalization_type,\n )\n processed_audio_matrix.append(processed_audio)\n return torch.stack(processed_audio_matrix)\n\n\nclass AudioFeatureMixin(BaseFeatureMixin):\n @staticmethod\n def type():\n return AUDIO\n\n @staticmethod\n def cast_column(column, backend):\n return column\n\n @staticmethod\n def get_feature_meta(\n config: ModelConfigDict,\n column,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n is_input_feature: bool,\n ) -> FeatureMetadataDict:\n first_audio_file_path = column.head(1).iloc[0]\n _, sampling_rate_in_hz = torchaudio.load(first_audio_file_path)\n\n feature_dim = AudioFeatureMixin._get_feature_dim(preprocessing_parameters, sampling_rate_in_hz)\n audio_file_length_limit_in_s = preprocessing_parameters[\"audio_file_length_limit_in_s\"]\n max_length = AudioFeatureMixin._get_max_length_feature(\n preprocessing_parameters, sampling_rate_in_hz, audio_file_length_limit_in_s\n )\n return {\n \"feature_dim\": feature_dim,\n \"sampling_rate_in_hz\": sampling_rate_in_hz,\n \"max_length\": max_length,\n \"reshape\": (max_length, feature_dim),\n }\n\n @staticmethod\n def _get_feature_dim(preprocessing_parameters: PreprocessingConfigDict, sampling_rate_in_hz):\n feature_type = preprocessing_parameters[TYPE]\n\n if feature_type == \"raw\":\n feature_dim = 1\n elif feature_type == \"stft_phase\":\n feature_dim_symmetric = get_length_in_samp(\n preprocessing_parameters[\"window_length_in_s\"], sampling_rate_in_hz\n )\n feature_dim = 2 * get_non_symmetric_length(feature_dim_symmetric)\n elif feature_type in [\"stft\", \"group_delay\"]:\n feature_dim_symmetric = get_length_in_samp(\n preprocessing_parameters[\"window_length_in_s\"], sampling_rate_in_hz\n )\n feature_dim = get_non_symmetric_length(feature_dim_symmetric)\n elif feature_type == \"fbank\":\n feature_dim = preprocessing_parameters[\"num_filter_bands\"]\n else:\n raise ValueError(f\"{feature_type} is not recognized.\")\n\n return feature_dim\n\n @staticmethod\n def _process_in_memory(\n column,\n audio_feature_dict,\n feature_dim,\n max_length,\n padding_value,\n normalization_type,\n audio_file_length_limit_in_s,\n backend,\n ):\n df_engine = backend.df_engine\n if _TORCH_200:\n # Read audio from path if the version of torch is >= 2.0.0.\n raw_audio = backend.read_binary_files(column, map_fn=read_audio_from_path)\n else:\n raw_audio = backend.read_binary_files(column, map_fn=read_audio_from_bytes_obj)\n\n try:\n default_audio = get_default_audio([audio for audio in raw_audio if is_torch_audio_tuple(audio)])\n except RuntimeError as e:\n raise RuntimeError(f\"Unable to process audio files provided: {e}\") from e\n\n raw_audio = df_engine.map_objects(raw_audio, lambda row: row if is_torch_audio_tuple(row) else default_audio)\n processed_audio = df_engine.map_objects(\n raw_audio,\n lambda row: AudioFeatureMixin._transform_to_feature(\n audio=row[0],\n sampling_rate_in_hz=row[1],\n audio_feature_dict=audio_feature_dict,\n feature_dim=feature_dim,\n max_length=max_length,\n padding_value=padding_value,\n normalization_type=normalization_type,\n ).numpy(), # non-torchscript preprocessing requires np.ndarray\n )\n\n audio_stats = df_engine.map_objects(\n raw_audio,\n lambda row: AudioFeatureMixin._get_stats(\n audio=row[0],\n sampling_rate_in_hz=row[1],\n max_length_in_s=audio_file_length_limit_in_s,\n ),\n )\n\n def reduce(series):\n merged_stats = None\n for audio_stats in series:\n if merged_stats is None:\n merged_stats = audio_stats.copy()\n else:\n AudioFeatureMixin._merge_stats(merged_stats, audio_stats)\n return merged_stats\n\n merged_stats = df_engine.reduce_objects(audio_stats, reduce)\n merged_stats[\"mean\"] = calculate_mean(merged_stats[\"sum\"], merged_stats[\"count\"])\n merged_stats[\"var\"] = calculate_var(merged_stats[\"sum\"], merged_stats[\"sum2\"], merged_stats[\"count\"])\n merged_stats[\"std\"] = np.sqrt(merged_stats[\"var\"] / float(merged_stats[\"count\"]))\n print_statistics = (\n \"{} audio files loaded.\\n\"\n \"Statistics of audio file lengths:\\n\"\n \"- mean: {:.4f}\\n\"\n \"- std: {:.4f}\\n\"\n \"- max: {:.4f}\\n\"\n \"- min: {:.4f}\\n\"\n \"- cropped audio_files: {}\\n\"\n \"Max length was given as {}s\"\n ).format(\n merged_stats[\"count\"],\n merged_stats[\"mean\"],\n merged_stats[\"std\"],\n merged_stats[\"max\"],\n merged_stats[\"min\"],\n merged_stats[\"cropped\"],\n audio_file_length_limit_in_s,\n )\n logger.debug(print_statistics)\n return processed_audio\n\n @staticmethod\n def _transform_to_feature(\n audio: torch.Tensor,\n sampling_rate_in_hz: int,\n audio_feature_dict: dict[str, float | int | str],\n feature_dim: int,\n max_length: int,\n padding_value: float,\n normalization_type: str | None = None,\n type_key: str = TYPE,\n ):\n feature_type: str = str(audio_feature_dict[type_key])\n if feature_type == \"raw\":\n audio_feature = torch.unsqueeze(audio[0], dim=-1)\n elif feature_type in [\"stft\", \"stft_phase\", \"group_delay\", \"fbank\"]:\n audio_feature = AudioFeatureMixin._get_2D_feature(\n audio, feature_type, audio_feature_dict, sampling_rate_in_hz\n )\n audio_feature = torch.transpose(audio_feature, 0, 1)\n else:\n raise ValueError(f\"{feature_type} is not recognized.\")\n\n # Outer conditional is type refinement from Union[str, None] to str\n if normalization_type is not None:\n if normalization_type == \"per_file\":\n mean = torch.mean(audio_feature, dim=0)\n std = torch.std(audio_feature, dim=0)\n audio_feature = torch.divide((audio_feature - mean), std + 1.0e-10)\n elif normalization_type == \"global\":\n raise ValueError(\"not implemented yet\")\n\n feature_length = audio_feature.shape[0]\n broadcast_feature_length = min(feature_length, max_length)\n audio_feature_padded = torch.full(\n (max_length, feature_dim), padding_value, dtype=torch.float32, device=audio_feature.device\n )\n audio_feature_padded[:broadcast_feature_length, :] = audio_feature[:max_length, :]\n\n return audio_feature_padded\n\n @staticmethod\n def _get_stats(audio, sampling_rate_in_hz, max_length_in_s):\n audio_length_in_s = audio.shape[-1] / float(sampling_rate_in_hz)\n return {\n \"count\": 1,\n \"sum\": audio_length_in_s,\n \"sum2\": audio_length_in_s * audio_length_in_s,\n \"min\": audio_length_in_s,\n \"max\": audio_length_in_s,\n \"cropped\": 1 if audio_length_in_s > max_length_in_s else 0,\n }\n\n @staticmethod\n def _merge_stats(merged_stats, audio_stats):\n merged_stats[\"count\"] += audio_stats[\"count\"]\n merged_stats[\"sum\"] += audio_stats[\"sum\"]\n merged_stats[\"sum2\"] += audio_stats[\"sum2\"]\n merged_stats[\"min\"] = min(merged_stats[\"min\"], audio_stats[\"min\"])\n merged_stats[\"max\"] = max(merged_stats[\"max\"], audio_stats[\"max\"])\n merged_stats[\"cropped\"] += audio_stats[\"cropped\"]\n\n @staticmethod\n def _get_2D_feature(\n audio: torch.Tensor,\n feature_type: str,\n audio_feature_dict: dict[str, float | int | str],\n sampling_rate_in_hz: int,\n ) -> torch.Tensor:\n window_length_in_s = audio_feature_dict[\"window_length_in_s\"]\n window_shift_in_s = audio_feature_dict[\"window_shift_in_s\"]\n assert torch.jit.isinstance(window_length_in_s, float)\n assert torch.jit.isinstance(window_shift_in_s, float)\n\n window_length_in_samp = get_length_in_samp(window_length_in_s, sampling_rate_in_hz)\n\n if \"num_fft_points\" in audio_feature_dict:\n num_fft_points = audio_feature_dict[\"num_fft_points\"]\n assert torch.jit.isinstance(num_fft_points, int)\n\n if num_fft_points < window_length_in_samp:\n raise ValueError(\n \"num_fft_points: {} < window length in \"\n \"samples: {} (corresponds to window length\"\n \" in s: {}\".format(num_fft_points, window_length_in_s, window_length_in_samp)\n )\n else:\n num_fft_points = window_length_in_samp\n\n if \"window_type\" in audio_feature_dict:\n window_type = audio_feature_dict[\"window_type\"]\n assert torch.jit.isinstance(window_type, str)\n else:\n window_type = \"hamming\"\n\n if feature_type == \"stft_phase\":\n return get_phase_stft_magnitude(\n audio, sampling_rate_in_hz, window_length_in_s, window_shift_in_s, num_fft_points, window_type\n )\n elif feature_type == \"stft\":\n return get_stft_magnitude(\n audio, sampling_rate_in_hz, window_length_in_s, window_shift_in_s, num_fft_points, window_type\n )\n elif feature_type == \"group_delay\":\n return get_group_delay(\n audio, sampling_rate_in_hz, window_length_in_s, window_shift_in_s, num_fft_points, window_type\n )\n elif feature_type == \"fbank\":\n num_filter_bands = audio_feature_dict[\"num_filter_bands\"]\n assert torch.jit.isinstance(num_filter_bands, int)\n\n return get_fbank(\n audio,\n sampling_rate_in_hz,\n window_length_in_s,\n window_shift_in_s,\n num_fft_points,\n window_type,\n num_filter_bands,\n )\n else:\n raise ValueError(f'feature_type \"{feature_type}\" is not recognized.')\n\n @staticmethod\n def add_feature_data(\n feature_config,\n input_df,\n proc_df,\n metadata,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n skip_save_processed_input,\n ):\n set_default_value(feature_config[\"preprocessing\"], \"in_memory\", preprocessing_parameters[\"in_memory\"])\n\n name = feature_config[NAME]\n column = input_df[feature_config[COLUMN]]\n\n num_audio_files = len(column)\n if num_audio_files == 0:\n raise ValueError(\"There are no audio files in the dataset provided.\")\n\n first_audio_entry = next(iter(column))\n logger.debug(f\"Detected audio feature type is {type(first_audio_entry)}\")\n\n if not isinstance(first_audio_entry, str) and not isinstance(first_audio_entry, torch.Tensor):\n raise ValueError(\n \"Invalid audio feature data type. Detected type is {}, \"\n \"expected either string for local/remote file path or Torch Tensor.\".format(type(first_audio_entry))\n )\n\n src_path = None\n if SRC in metadata:\n if isinstance(first_audio_entry, str) and not has_remote_protocol(first_audio_entry):\n src_path = os.path.dirname(os.path.abspath(metadata.get(SRC)))\n abs_path_column = backend.df_engine.map_objects( # This gets the CSV file path\n column, lambda row: get_abs_path(src_path, row) if isinstance(row, str) else row\n )\n\n num_audio_utterances = len(input_df[feature_config[COLUMN]])\n padding_value = preprocessing_parameters[\"padding_value\"]\n normalization_type = preprocessing_parameters[\"norm\"]\n\n feature_dim = metadata[name][\"feature_dim\"]\n max_length = metadata[name][\"max_length\"]\n audio_feature_dict = {\n key: value\n for key, value in preprocessing_parameters.items()\n if key in AUDIO_FEATURE_KEYS and value is not None\n }\n audio_file_length_limit_in_s = preprocessing_parameters[\"audio_file_length_limit_in_s\"]\n\n if num_audio_utterances == 0:\n raise ValueError(\"There are no audio files in the dataset provided.\")\n\n if feature_config[PREPROCESSING][\"in_memory\"]:\n audio_features = AudioFeatureMixin._process_in_memory(\n abs_path_column,\n audio_feature_dict,\n feature_dim,\n max_length,\n padding_value,\n normalization_type,\n audio_file_length_limit_in_s,\n backend,\n )\n proc_df[feature_config[PROC_COLUMN]] = audio_features\n\n return proc_df\n\n @staticmethod\n def _get_max_length_feature(\n preprocessing_parameters: PreprocessingConfigDict, sampling_rate_in_hz, audio_length_limit_in_s\n ):\n feature_type = preprocessing_parameters[TYPE]\n audio_length_limit_in_samp = audio_length_limit_in_s * sampling_rate_in_hz\n\n if not audio_length_limit_in_samp.is_integer():\n raise ValueError(\n \"Audio_file_length_limit has to be chosen \"\n \"so that {} (in s) * {} (sampling rate in Hz) \"\n \"is an integer.\".format(audio_length_limit_in_s, sampling_rate_in_hz)\n )\n audio_length_limit_in_samp = int(audio_length_limit_in_samp)\n\n if feature_type == \"raw\":\n return audio_length_limit_in_samp\n elif feature_type in [\"stft\", \"stft_phase\", \"group_delay\", \"fbank\"]:\n window_length_in_s = preprocessing_parameters[\"window_length_in_s\"]\n window_shift_in_s = preprocessing_parameters[\"window_shift_in_s\"]\n return get_max_length_stft_based(\n audio_length_limit_in_samp, window_length_in_s, window_shift_in_s, sampling_rate_in_hz\n )\n else:\n raise ValueError(f\"{feature_type} is not recognized.\")\n\n\nclass AudioInputFeature(AudioFeatureMixin, SequenceInputFeature):\n def __init__(self, input_feature_config: AudioInputFeatureConfig, encoder_obj=None, **kwargs):\n super().__init__(input_feature_config, encoder_obj=encoder_obj, **kwargs)\n\n if not getattr(self.encoder_obj.config, \"embedding_size\", None):\n raise ValueError(\"embedding_size has to be defined - \" 'check \"update_config_with_metadata()\"')\n if not getattr(self.encoder_obj.config, \"max_sequence_length\", None):\n raise ValueError(\"max_sequence_length has to be defined - \" 'check \"update_config_with_metadata()\"')\n\n def forward(self, inputs, mask=None):\n assert isinstance(inputs, torch.Tensor)\n assert inputs.dtype == torch.float32\n assert len(inputs.shape) == 3, f\"expected 3D shape, found: {inputs.shape}\"\n\n encoder_output = self.encoder_obj(inputs, mask=mask)\n\n return encoder_output\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.encoder_obj.config.max_sequence_length, self.encoder_obj.config.embedding_size])\n\n @property\n def input_dtype(self):\n return torch.float32\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n feature_config.encoder.max_sequence_length = feature_metadata[\"max_length\"]\n feature_config.encoder.embedding_size = feature_metadata[\"feature_dim\"]\n feature_config.encoder.should_embed = False\n\n @staticmethod\n def create_preproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n return _AudioPreprocessing(metadata)\n\n @staticmethod\n def get_schema_cls():\n return AudioInputFeatureConfig\n"
},
{
"path": "ludwig/features/bag_feature.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nfrom collections import Counter\n\nimport numpy as np\nimport torch\n\nfrom ludwig.constants import BAG, COLUMN, NAME, PROC_COLUMN\nfrom ludwig.features.base_feature import BaseFeatureMixin, InputFeature\nfrom ludwig.features.feature_utils import set_str_to_idx\nfrom ludwig.features.set_feature import _SetPreprocessing\nfrom ludwig.schema.features.bag_feature import BagInputFeatureConfig\nfrom ludwig.types import FeatureMetadataDict, ModelConfigDict, PreprocessingConfigDict, TrainingSetMetadataDict\nfrom ludwig.utils.strings_utils import create_vocabulary\n\nlogger = logging.getLogger(__name__)\n\n\nclass BagFeatureMixin(BaseFeatureMixin):\n @staticmethod\n def type():\n return BAG\n\n @staticmethod\n def cast_column(column, backend):\n return column.astype(str)\n\n @staticmethod\n def get_feature_meta(\n config: ModelConfigDict,\n column,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n is_input_feature: bool,\n ) -> FeatureMetadataDict:\n vocabulary = create_vocabulary(\n column,\n preprocessing_parameters[\"tokenizer\"],\n num_most_frequent=preprocessing_parameters[\"most_common\"],\n lowercase=preprocessing_parameters[\"lowercase\"],\n processor=backend.df_engine,\n )\n return {\n \"idx2str\": vocabulary.vocab,\n \"str2idx\": vocabulary.str2idx,\n \"str2freq\": vocabulary.str2freq,\n \"vocab_size\": len(vocabulary.str2idx),\n \"max_set_size\": vocabulary.max_sequence_length,\n }\n\n @staticmethod\n def feature_data(column, metadata, preprocessing_parameters: PreprocessingConfigDict, backend):\n def to_vector(set_str):\n bag_vector = np.zeros((len(metadata[\"str2idx\"]),), dtype=np.float32)\n col_counter = Counter(set_str_to_idx(set_str, metadata[\"str2idx\"], preprocessing_parameters[\"tokenizer\"]))\n\n bag_vector[list(col_counter.keys())] = list(col_counter.values())\n return bag_vector\n\n return backend.df_engine.map_objects(column, to_vector)\n\n @staticmethod\n def add_feature_data(\n feature_config,\n input_df,\n proc_df,\n metadata,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n skip_save_processed_input,\n ):\n proc_df[feature_config[PROC_COLUMN]] = BagFeatureMixin.feature_data(\n input_df[feature_config[COLUMN]],\n metadata[feature_config[NAME]],\n preprocessing_parameters,\n backend,\n )\n return proc_df\n\n\nclass BagInputFeature(BagFeatureMixin, InputFeature):\n def __init__(self, input_feature_config: BagInputFeatureConfig, encoder_obj=None, **kwargs):\n super().__init__(input_feature_config, **kwargs)\n\n if encoder_obj:\n self.encoder_obj = encoder_obj\n else:\n self.encoder_obj = self.initialize_encoder(input_feature_config.encoder)\n\n def forward(self, inputs):\n assert isinstance(inputs, torch.Tensor)\n # assert inputs.dtype == tf.bool # this fails\n\n encoder_output = self.encoder_obj(inputs)\n\n return encoder_output\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([len(self.encoder_obj.config.vocab)])\n\n @property\n def output_shape(self) -> torch.Size:\n return self.encoder_obj.output_shape\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n feature_config.encoder.vocab = feature_metadata[\"idx2str\"]\n\n @staticmethod\n def get_schema_cls():\n return BagInputFeatureConfig\n\n @staticmethod\n def create_preproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n return _SetPreprocessing(metadata, is_bag=True)\n"
},
{
"path": "ludwig/features/base_feature.py",
"content": "# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nfrom abc import ABC, abstractmethod, abstractstaticmethod\nfrom dataclasses import dataclass\nfrom typing import Any\n\nimport torch\nfrom torch import Tensor\n\nfrom ludwig.constants import (\n ENCODER_OUTPUT,\n ENCODER_OUTPUT_STATE,\n HIDDEN,\n LENGTHS,\n LOGITS,\n LOSS,\n PREDICTIONS,\n PROBABILITIES,\n)\nfrom ludwig.decoders.registry import get_decoder_cls\nfrom ludwig.encoders.registry import get_encoder_cls\nfrom ludwig.features.feature_utils import get_input_size_with_dependencies\nfrom ludwig.modules.fully_connected_modules import FCStack\nfrom ludwig.modules.loss_modules import create_loss\nfrom ludwig.modules.metric_modules import LossMetric, LudwigMetric, MeanMetric\nfrom ludwig.modules.metric_registry import get_metric_classes, get_metric_cls, get_metric_tensor_input\nfrom ludwig.modules.reduction_modules import SequenceReducer\nfrom ludwig.schema.features.base import BaseFeatureConfig, BaseOutputFeatureConfig\nfrom ludwig.types import (\n FeatureConfigDict,\n FeatureMetadataDict,\n ModelConfigDict,\n PreprocessingConfigDict,\n TrainingSetMetadataDict,\n)\nfrom ludwig.utils import output_feature_utils\nfrom ludwig.utils.calibration import CalibrationModule\nfrom ludwig.utils.torch_utils import LudwigModule\nfrom ludwig.utils.types import DataFrame, TorchscriptPreprocessingInput\n\nlogger = logging.getLogger(__name__)\n\n\nclass BaseFeatureMixin(ABC):\n \"\"\"Parent class for feature mixins.\n\n Feature mixins support preprocessing functionality shared across input and output features.\n \"\"\"\n\n @abstractstaticmethod\n def type() -> str:\n \"\"\"Returns the type of feature this mixin supports.\"\"\"\n raise NotImplementedError\n\n @abstractstaticmethod\n def cast_column(column: DataFrame, backend) -> DataFrame:\n \"\"\"Returns a copy of the dataset column for the given feature, potentially after a type cast.\n\n Args:\n column: Pandas column of values.\n backend: (Union[Backend, str]) Backend to use for feature data processing.\n \"\"\"\n raise NotImplementedError\n\n @abstractstaticmethod\n def get_feature_meta(\n config: ModelConfigDict,\n column: DataFrame,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n is_input_feature: bool,\n ) -> FeatureMetadataDict:\n \"\"\"Returns a dictionary of feature metadata.\n\n Args:\n config: Ludwig model config dict.\n column: Pandas column of values.\n preprocessing_parameters: Preprocessing configuration for this feature.\n backend: (Union[Backend, str]) Backend to use for feature data processing.\n \"\"\"\n raise NotImplementedError\n\n @abstractstaticmethod\n def add_feature_data(\n feature_config: FeatureConfigDict,\n input_df: DataFrame,\n proc_df: dict[str, DataFrame],\n metadata: TrainingSetMetadataDict,\n preprocessing_parameters: PreprocessingConfigDict,\n backend, # Union[Backend, str]\n skip_save_processed_input: bool,\n ) -> None:\n \"\"\"Runs preprocessing on the input_df and stores results in the proc_df and metadata dictionaries.\n\n Args:\n feature_config: Feature configuration.\n input_df: Pandas column of values.\n proc_df: Dict of processed columns of data. Feature data is added to this.\n metadata: Metadata returned by get_feature_meta(). Additional information may be added to this.\n preprocessing_parameters: Preprocessing configuration for this feature.\n backend: (Union[Backend, str]) Backend to use for feature data processing.\n skip_save_processed_input: Whether to skip saving the processed input.\n \"\"\"\n raise NotImplementedError\n\n\n@dataclass\nclass ModuleWrapper:\n \"\"\"Used to prevent the PredictModule from showing up an attribute on the feature module.\n\n This is necessary to avoid inflight errors from DeepSpeed. These errors occur when DeepSpeed believes that a param\n is still in the process of being processed asynchronously (allgathered, etc.).\n \"\"\"\n\n module: torch.nn.Module\n\n\nclass PredictModule(torch.nn.Module):\n \"\"\"Base class for all modules that convert model outputs to predictions.\n\n Explicit member variables needed here for scripting, as Torchscript will not be able to recognize global variables\n during scripting.\n \"\"\"\n\n def __init__(self):\n super().__init__()\n self.predictions_key = PREDICTIONS\n self.probabilities_key = PROBABILITIES\n self.logits_key = LOGITS\n\n\nclass BaseFeature:\n \"\"\"Base class for all features.\n\n Note that this class is not-cooperative (does not forward kwargs), so when constructing feature class hierarchies,\n there should be only one parent class that derives from base feature. Other functionality should be put into mixin\n classes to avoid the diamond pattern.\n \"\"\"\n\n def __init__(self, feature: BaseFeatureConfig):\n super().__init__()\n\n if not feature.name:\n raise ValueError(\"Missing feature name\")\n self.feature_name = feature.name\n\n if not feature.column:\n feature.column = self.feature_name\n self.column = feature.column\n\n self.proc_column = feature.proc_column\n\n\nclass InputFeature(BaseFeature, LudwigModule, ABC):\n \"\"\"Parent class for all input features.\"\"\"\n\n def create_sample_input(self, batch_size: int = 2):\n # Used by get_model_inputs(), which is used for tracing-based torchscript generation.\n return torch.rand([batch_size, *self.input_shape]).to(self.input_dtype)\n\n def unskip(self) -> \"InputFeature\":\n \"\"\"Convert feature using passthrough wrapper back to full encoder.\"\"\"\n return self\n\n @staticmethod\n @abstractmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n pass\n\n def update_config_after_module_init(self, feature_config):\n \"\"\"Updates the config after the torch.nn.Module objects have been initialized.\"\"\"\n\n def initialize_encoder(self, encoder_config):\n encoder_cls = get_encoder_cls(self.type(), encoder_config.type)\n encoder_schema = encoder_cls.get_schema_cls().Schema()\n encoder_params_dict = encoder_schema.dump(encoder_config)\n return encoder_cls(encoder_config=encoder_config, **encoder_params_dict)\n\n @classmethod\n def get_preproc_input_dtype(cls, metadata: TrainingSetMetadataDict) -> str:\n return \"string\"\n\n @staticmethod\n def create_preproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n raise NotImplementedError(\"Torchscript tracing not supported for feature\")\n\n\nclass OutputFeature(BaseFeature, LudwigModule, ABC):\n \"\"\"Parent class for all output features.\"\"\"\n\n def __init__(\n self,\n feature: BaseOutputFeatureConfig,\n other_output_features: dict[str, \"OutputFeature\"],\n *args,\n **kwargs,\n ):\n \"\"\"Defines defaults, overwrites them based on the feature dictionary, and sets up dependencies.\n\n Any output feature can depend on one or more other output features. The `other_output_features` input dictionary\n should contain entries for any dependent output features, which is accomplished by constructing output features\n in topographically sorted order. Attributes of any dependent output features are used to properly initialize\n this feature's sizes.\n \"\"\"\n super().__init__(feature)\n\n # List of names of metrics that this OutputFeature computes.\n self.metric_names = []\n self.loss = feature.loss\n self.reduce_input = feature.reduce_input\n self.reduce_dependencies = feature.reduce_dependencies\n\n # List of feature names that this output feature is dependent on.\n self.dependencies = feature.dependencies\n\n logger.debug(\" output feature fully connected layers\")\n logger.debug(\" FCStack\")\n\n self.input_size = get_input_size_with_dependencies(feature.input_size, self.dependencies, other_output_features)\n feature.input_size = self.input_size\n\n self.fc_stack = FCStack(\n first_layer_input_size=self.input_size,\n layers=feature.decoder.fc_layers,\n num_layers=feature.decoder.num_fc_layers,\n default_output_size=feature.decoder.fc_output_size,\n default_use_bias=feature.decoder.fc_use_bias,\n default_weights_initializer=feature.decoder.fc_weights_initializer,\n default_bias_initializer=feature.decoder.fc_bias_initializer,\n default_norm=feature.decoder.fc_norm,\n default_norm_params=feature.decoder.fc_norm_params,\n default_activation=feature.decoder.fc_activation,\n default_dropout=feature.decoder.fc_dropout,\n )\n self._calibration_module = self.create_calibration_module(feature)\n self._prediction_module = ModuleWrapper(self.create_predict_module())\n\n # set up two sequence reducers, one for inputs and other for dependencies\n self.reduce_sequence_input = SequenceReducer(reduce_mode=self.reduce_input)\n if self.dependencies:\n self.dependency_reducers = torch.nn.ModuleDict()\n # todo: re-evaluate need for separate handling of `attention` reducer\n # currently this code does not support `attention`\n for dependency in self.dependencies:\n self.dependency_reducers[dependency] = SequenceReducer(reduce_mode=self.reduce_dependencies)\n\n def create_sample_output(self, batch_size: int = 2):\n output_shape = self.output_shape\n shape = [batch_size, *self.output_shape] if output_shape != torch.Size([1]) else [batch_size]\n return torch.rand(shape).to(self.get_output_dtype())\n\n @abstractmethod\n def get_prediction_set(self):\n \"\"\"Returns the set of tensor keys returned by this feature's PredictModule.\n\n TODO(Justin): Move this to the PredictModule.\n \"\"\"\n raise NotImplementedError(\"OutputFeature is missing implementation for get_prediction_set.\")\n\n @classmethod\n @abstractmethod\n def get_output_dtype(cls):\n \"\"\"Returns the Tensor data type feature outputs.\"\"\"\n\n def initialize_decoder(self, decoder_config):\n # Input to the decoder is the output feature's FC hidden layer.\n decoder_config.input_size = self.fc_stack.output_shape[-1]\n decoder_cls = get_decoder_cls(self.type(), decoder_config.type)\n decoder_schema = decoder_cls.get_schema_cls().Schema()\n decoder_params_dict = decoder_schema.dump(decoder_config)\n return decoder_cls(decoder_config=decoder_config, **decoder_params_dict)\n\n def train_loss(self, targets: Tensor, predictions: dict[str, Tensor], feature_name):\n loss_class = type(self.train_loss_function)\n prediction_key = output_feature_utils.get_feature_concat_name(feature_name, loss_class.get_loss_inputs())\n return self.train_loss_function(predictions[prediction_key], targets)\n\n def eval_loss(self, targets: Tensor, predictions: dict[str, Tensor]):\n loss_class = type(self.train_loss_function)\n prediction_key = loss_class.get_loss_inputs()\n if isinstance(self.eval_loss_metric, MeanMetric):\n # MeanMetric's forward() implicitly updates the running average.\n # For MeanMetrics, we use get_current_value() to compute the loss without changing the state. All metrics\n # are updated at the BaseModel level as part of update_metrics().\n return self.eval_loss_metric.get_current_value(predictions[prediction_key].detach(), targets)\n return self.eval_loss_metric(predictions[prediction_key].detach(), targets)\n\n def _setup_loss(self):\n self.train_loss_function = create_loss(self.loss)\n self._eval_loss_metric = ModuleWrapper(get_metric_cls(self.type(), self.loss.type)(config=self.loss))\n\n def _setup_metrics(self):\n kwargs = {}\n for name, cls in get_metric_classes(self.type()).items():\n if cls.can_report(self) and isinstance(cls, LossMetric):\n kwargs[name] = cls(config=self.loss, **self.metric_kwargs())\n elif cls.can_report(self):\n kwargs[name] = cls(**self.metric_kwargs())\n self._metric_functions = {\n LOSS: self.eval_loss_metric,\n **kwargs,\n }\n self.metric_names = sorted(list(self._metric_functions.keys()))\n\n def create_calibration_module(self, feature: BaseOutputFeatureConfig) -> CalibrationModule:\n \"\"\"Creates and returns a CalibrationModule that converts logits to a probability distribution.\"\"\"\n return None\n\n @property\n def eval_loss_metric(self) -> LudwigMetric:\n return self._eval_loss_metric.module\n\n @property\n def calibration_module(self) -> torch.nn.Module:\n \"\"\"Returns the CalibrationModule used to convert logits to a probability distribution.\"\"\"\n return self._calibration_module\n\n @abstractmethod\n def create_predict_module(self) -> PredictModule:\n \"\"\"Creates and returns a `nn.Module` that converts raw model outputs (logits) to predictions.\n\n This module is needed when generating the Torchscript model using scripting.\n \"\"\"\n raise NotImplementedError()\n\n @property\n def prediction_module(self) -> PredictModule:\n \"\"\"Returns the PredictModule used to convert model outputs to predictions.\"\"\"\n return self._prediction_module.module\n\n def predictions(self, all_decoder_outputs: dict[str, torch.Tensor], feature_name: str) -> dict[str, torch.Tensor]:\n \"\"\"Computes actual predictions from the outputs of feature decoders.\n\n TODO(Justin): Consider refactoring this to accept feature-specific decoder outputs.\n\n Args:\n all_decoder_outputs: A dictionary of {feature name}::{tensor_name} -> output tensor.\n Returns:\n Dictionary of tensors with predictions as well as any additional tensors that may be\n necessary for computing evaluation metrics.\n \"\"\"\n return self.prediction_module(all_decoder_outputs, feature_name)\n\n @abstractmethod\n def logits(self, combiner_outputs: dict[str, torch.Tensor], target=None, **kwargs) -> dict[str, torch.Tensor]:\n \"\"\"Unpacks and feeds combiner_outputs to the decoder. Invoked as part of the output feature's forward pass.\n\n If target is not None, then we are in training.\n\n Args:\n combiner_outputs: Dictionary of tensors from the combiner's forward pass.\n Returns:\n Dictionary of decoder's output tensors (non-normalized), as well as any additional\n tensors that may be necessary for computing predictions or evaluation metrics.\n \"\"\"\n raise NotImplementedError(\"OutputFeature is missing logits() implementation.\")\n\n def metric_kwargs(self) -> dict[str, Any]:\n \"\"\"Returns arguments that are used to instantiate an instance of each metric class.\"\"\"\n return {}\n\n def update_metrics(self, targets: Tensor, predictions: dict[str, Tensor]) -> None:\n \"\"\"Updates metrics with the given targets and predictions.\n\n Args:\n targets: Tensor with target values for this output feature.\n predictions: Dict of tensors returned by predictions().\n \"\"\"\n for metric_name, metric_fn in self._metric_functions.items():\n prediction_key = get_metric_tensor_input(metric_name)\n metric_fn = metric_fn.to(predictions[prediction_key].device)\n metric_fn.update(predictions[prediction_key].detach(), targets)\n\n def get_metrics(self):\n metric_vals = {}\n for metric_name, metric_fn in self._metric_functions.items():\n try:\n computed_metric = metric_fn.compute()\n except Exception as e:\n logger.exception(f\"Caught exception computing metric: {metric_name} with error: {e}.\")\n continue\n\n # Metrics from torchmetrics can be a straightforward tensor.\n if isinstance(computed_metric, Tensor):\n metric_vals[metric_name] = computed_metric.detach().cpu().numpy().item()\n else:\n # Metrics from torchmetrics can be a dict of tensors.\n # For example, ROUGE is returned as a dictionary of tensors.\n # Unpack.\n for sub_metric_name, metric in computed_metric.items():\n metric_vals[sub_metric_name] = metric.detach().cpu().numpy().item()\n return metric_vals\n\n def reset_metrics(self):\n for _, metric_fn in self._metric_functions.items():\n if metric_fn is not None:\n metric_fn.reset()\n\n def forward(\n self,\n combiner_outputs: dict[str, torch.Tensor],\n other_output_feature_outputs: dict[str, torch.Tensor],\n mask: torch.Tensor | None = None,\n target: torch.Tensor | None = None,\n ) -> dict[str, torch.Tensor]:\n \"\"\"Forward pass that takes in output from the combiner, and passes it through to the decoder.\n\n Args:\n combiner_outputs: Dict of outputs from the combiner.\n other_output_feature_outputs: Dict of tensors from other output features. Used for resolving dependencies.\n mask: (Unused). Tensor for masking.\n target: Tensor with targets. During training, targets != None. During prediction, targets = None.\n\n Returns:\n Dict of output tensors, with at least 'last_hidden' and 'logits' as keys, as well as any additional tensor\n results from the decoder.\n \"\"\"\n # extract the combined hidden layer\n combiner_hidden = combiner_outputs[\"combiner_output\"]\n hidden = self.prepare_decoder_inputs(combiner_hidden, other_output_feature_outputs, mask=mask)\n\n # ================ Predictions ================\n logits_input = {HIDDEN: hidden}\n # pass supplemental data from encoders to decoder\n if ENCODER_OUTPUT_STATE in combiner_outputs:\n logits_input[ENCODER_OUTPUT_STATE] = combiner_outputs[ENCODER_OUTPUT_STATE]\n if LENGTHS in combiner_outputs:\n logits_input[LENGTHS] = combiner_outputs[LENGTHS]\n\n logits = self.logits(logits_input, target=target)\n\n # For binary and number features, self.logits() is a tensor.\n # There are two special cases where self.logits() is a dict:\n # categorical\n # keys: logits, projection_input\n # sequence\n # keys: logits\n # TODO(Justin): Clean this up.\n if isinstance(logits, Tensor):\n logits = {\"logits\": logits}\n\n # For multi-class features, we must choose a consistent tuple subset.\n return {\n # last_hidden used for dependencies processing\n \"last_hidden\": hidden,\n **logits,\n }\n\n @abstractmethod\n def postprocess_predictions(\n self,\n result: dict[str, Tensor],\n metadata: TrainingSetMetadataDict,\n ):\n raise NotImplementedError\n\n @classmethod\n def get_postproc_output_dtype(cls, metadata: TrainingSetMetadataDict) -> str:\n return \"string\"\n\n @staticmethod\n def create_postproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n raise NotImplementedError(\"Torchscript tracing not supported for feature\")\n\n @staticmethod\n @abstractmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n pass\n\n @staticmethod\n @abstractmethod\n def calculate_overall_stats(predictions, targets, train_set_metadata):\n pass\n\n def output_specific_fully_connected(self, inputs, mask=None):\n feature_hidden = inputs\n original_feature_hidden = inputs\n\n # flatten inputs\n if len(original_feature_hidden.shape) > 2:\n feature_hidden = torch.reshape(feature_hidden, (-1, list(feature_hidden.shape)[-1]))\n\n # pass it through fc_stack\n feature_hidden = self.fc_stack(feature_hidden, mask=mask)\n feature_hidden_size = feature_hidden.shape[-1]\n\n # reshape back to original first and second dimension\n if len(original_feature_hidden.shape) > 2:\n sequence_length = original_feature_hidden.shape[1]\n feature_hidden = torch.reshape(feature_hidden, (-1, sequence_length, feature_hidden_size))\n\n return feature_hidden\n\n def prepare_decoder_inputs(\n self, combiner_hidden: Tensor, other_output_features: dict[str, Tensor], mask=None\n ) -> Tensor:\n \"\"\"Takes the combiner output and the outputs of other outputs features computed so far and performs:\n\n - reduction of combiner outputs (if needed)\n - concatenating the outputs of dependent features (if needed)\n - output_specific fully connected layers (if needed)\n\n Args:\n combiner_hidden: hidden state of the combiner\n other_output_features: output tensors from other output features\n \"\"\"\n # ================ Reduce Inputs ================\n feature_hidden = combiner_hidden\n if self.reduce_input is not None and len(combiner_hidden.shape) > 2:\n feature_hidden = self.reduce_sequence_input(combiner_hidden)\n\n # ================ Concat Dependencies ================\n if self.dependencies:\n feature_hidden = output_feature_utils.concat_dependencies(\n self.column, self.dependencies, self.dependency_reducers, feature_hidden, other_output_features\n )\n\n # ================ Output-wise Fully Connected ================\n feature_hidden = self.output_specific_fully_connected(feature_hidden, mask=mask)\n\n return feature_hidden\n\n\nclass PassthroughPreprocModule(torch.nn.Module):\n \"\"\"Combines preprocessing and encoding into a single module for TorchScript inference.\n\n For encoder outputs that were cached during preprocessing, the encoder is simply the identity function in the ECD\n module. As such, we need this module to apply the encoding that would normally be done during preprocessing for\n realtime inference.\n \"\"\"\n\n def __init__(self, preproc: torch.nn.Module, encoder: torch.nn.Module):\n self.preproc = preproc\n self.encoder = encoder\n\n def forward(self, v: TorchscriptPreprocessingInput) -> torch.Tensor:\n preproc_v = self.preproc(v)\n return self.encoder(preproc_v)\n\n\ndef create_passthrough_input_feature(feature: InputFeature, config: BaseFeatureConfig) -> InputFeature:\n \"\"\"Creates a shim input feature that acts as a transparent identifiy function on the input data.\n\n Used when the feature's encoder embeddings were cached in preprocessing. This way, we don't need to make any changes\n to the underlying interface in such cases other than to swap the feature that would normally do the encoding with\n this one.\n \"\"\"\n\n class _InputPassthroughFeature(InputFeature):\n def __init__(self, config: BaseFeatureConfig):\n super().__init__(config)\n\n def forward(self, inputs, mask=None):\n assert isinstance(inputs, torch.Tensor)\n return {ENCODER_OUTPUT: inputs}\n\n @property\n def input_dtype(self):\n # Doesn't matter as combiner will need to cast them to float32 anyway\n return torch.float32\n\n @property\n def input_shape(self):\n return feature.encoder_obj.output_shape\n\n @property\n def output_shape(self) -> torch.Size:\n return feature.encoder_obj.output_shape\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n return feature.update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs)\n\n @staticmethod\n def get_schema_cls():\n return feature.get_schema_cls()\n\n @staticmethod\n def create_preproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n return PassthroughPreprocModule(feature.create_preproc_module(metadata), feature)\n\n @staticmethod\n def type():\n return feature.type()\n\n def unskip(self) -> InputFeature:\n return feature\n\n @property\n def encoder_obj(self) -> torch.nn.Module:\n return feature.encoder_obj\n\n return _InputPassthroughFeature(config)\n"
},
{
"path": "ludwig/features/binary_feature.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\n\nimport numpy as np\nimport torch\n\nfrom ludwig.constants import BINARY, COLUMN, HIDDEN, LOGITS, NAME, PREDICTIONS, PROBABILITIES, PROBABILITY, PROC_COLUMN\nfrom ludwig.error import InputDataError\nfrom ludwig.features.base_feature import BaseFeatureMixin, InputFeature, OutputFeature, PredictModule\nfrom ludwig.schema.features.binary_feature import BinaryInputFeatureConfig, BinaryOutputFeatureConfig\nfrom ludwig.types import (\n FeatureConfigDict,\n FeatureMetadataDict,\n FeaturePostProcessingOutputDict,\n ModelConfigDict,\n PreprocessingConfigDict,\n TrainingSetMetadataDict,\n)\nfrom ludwig.utils import calibration, output_feature_utils, strings_utils\nfrom ludwig.utils.eval_utils import (\n average_precision_score,\n ConfusionMatrix,\n precision_recall_curve,\n roc_auc_score,\n roc_curve,\n)\nfrom ludwig.utils.types import DataFrame, TorchscriptPreprocessingInput\n\nlogger = logging.getLogger(__name__)\n\n\nclass _BinaryPreprocessing(torch.nn.Module):\n def __init__(self, metadata: TrainingSetMetadataDict):\n super().__init__()\n str2bool = metadata.get(\"str2bool\")\n self.str2bool = str2bool or {v: True for v in strings_utils.BOOL_TRUE_STRS}\n self.should_lower = str2bool is None\n\n def forward(self, v: TorchscriptPreprocessingInput) -> torch.Tensor:\n if torch.jit.isinstance(v, list[tuple[torch.Tensor, int]]):\n raise ValueError(f\"Unsupported input: {v}\")\n\n if torch.jit.isinstance(v, list[torch.Tensor]):\n v = torch.stack(v)\n\n if torch.jit.isinstance(v, torch.Tensor):\n return v.to(dtype=torch.float32)\n\n v = [s.strip() for s in v]\n if self.should_lower:\n v = [s.lower() for s in v]\n indices = [self.str2bool.get(s, False) for s in v]\n return torch.tensor(indices, dtype=torch.float32)\n\n\nclass _BinaryPostprocessing(torch.nn.Module):\n def __init__(self, metadata: TrainingSetMetadataDict):\n super().__init__()\n bool2str = metadata.get(\"bool2str\")\n self.bool2str = {i: v for i, v in enumerate(bool2str)} if bool2str is not None else None\n self.predictions_key = PREDICTIONS\n self.probabilities_key = PROBABILITIES\n\n def forward(self, preds: dict[str, torch.Tensor], feature_name: str) -> FeaturePostProcessingOutputDict:\n predictions = output_feature_utils.get_output_feature_tensor(preds, feature_name, self.predictions_key)\n probabilities = output_feature_utils.get_output_feature_tensor(preds, feature_name, self.probabilities_key)\n\n if self.bool2str is not None:\n predictions = predictions.to(dtype=torch.int32)\n predictions = [self.bool2str.get(pred, self.bool2str[0]) for pred in predictions]\n\n probabilities = torch.stack([1 - probabilities, probabilities], dim=-1)\n\n return {\n self.predictions_key: predictions,\n self.probabilities_key: probabilities,\n }\n\n\nclass _BinaryPredict(PredictModule):\n def __init__(self, threshold, calibration_module=None):\n super().__init__()\n self.threshold = threshold\n self.calibration_module = calibration_module\n\n def forward(self, inputs: dict[str, torch.Tensor], feature_name: str) -> dict[str, torch.Tensor]:\n logits = output_feature_utils.get_output_feature_tensor(inputs, feature_name, self.logits_key)\n\n if self.calibration_module is not None:\n probabilities = self.calibration_module(logits)\n else:\n probabilities = torch.sigmoid(logits)\n\n predictions = probabilities >= self.threshold\n return {\n self.probabilities_key: probabilities,\n self.predictions_key: predictions,\n self.logits_key: logits,\n }\n\n\nclass BinaryFeatureMixin(BaseFeatureMixin):\n @staticmethod\n def type():\n return BINARY\n\n @staticmethod\n def cast_column(column, backend):\n \"\"\"Cast column of dtype object to bool.\n\n Unchecked casting to boolean when given a column of dtype object converts all non-empty cells to True. We check\n the values of the column directly and manually determine the best dtype to use.\n \"\"\"\n values = backend.df_engine.compute(column.drop_duplicates())\n\n if strings_utils.values_are_pandas_numbers(values):\n # If numbers, convert to float so it can be converted to bool\n column = column.astype(float).astype(bool)\n elif strings_utils.values_are_pandas_bools(values):\n # If booleans, manually assign boolean values\n column = backend.df_engine.map_objects(\n column, lambda x: x.lower() in strings_utils.PANDAS_TRUE_STRS\n ).astype(bool)\n else:\n # If neither numbers or booleans, they are strings (objects)\n column = column.astype(object)\n return column\n\n @staticmethod\n def get_feature_meta(\n config: ModelConfigDict,\n column: DataFrame,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n is_input_feature: bool,\n ) -> FeatureMetadataDict:\n if column.dtype != object:\n return {}\n\n distinct_values = backend.df_engine.compute(column.drop_duplicates())\n if len(distinct_values) > 2:\n raise InputDataError(\n column.name, BINARY, f\"expects 2 distinct values, found {distinct_values.values.tolist()}\"\n )\n if preprocessing_parameters[\"fallback_true_label\"]:\n fallback_true_label = preprocessing_parameters[\"fallback_true_label\"]\n else:\n fallback_true_label = sorted(distinct_values)[0]\n preprocessing_parameters[\"fallback_true_label\"] = fallback_true_label\n\n try:\n str2bool = {v: strings_utils.str2bool(v) for v in distinct_values}\n except Exception as e:\n logger.warning(\n f\"Binary feature {column.name} has at least 1 unconventional boolean value: {e}. \"\n f\"We will now interpret {fallback_true_label} as 1 and the other values as 0. \"\n f\"If this is incorrect, please use the category feature type or \"\n f\"manually specify the true value with `preprocessing.fallback_true_label`.\"\n )\n str2bool = {v: strings_utils.str2bool(v, fallback_true_label) for v in distinct_values}\n\n bool2str = [k for k, v in sorted(str2bool.items(), key=lambda item: item[1])]\n return {\"str2bool\": str2bool, \"bool2str\": bool2str, \"fallback_true_label\": fallback_true_label}\n\n @staticmethod\n def add_feature_data(\n feature_config: FeatureConfigDict,\n input_df: DataFrame,\n proc_df: dict[str, DataFrame],\n metadata: TrainingSetMetadataDict,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n skip_save_processed_input: bool,\n ) -> None:\n column = input_df[feature_config[COLUMN]]\n\n if column.dtype == object:\n metadata = metadata[feature_config[NAME]]\n if \"str2bool\" in metadata:\n column = backend.df_engine.map_objects(column, lambda x: metadata[\"str2bool\"][str(x)])\n else:\n # No predefined mapping from string to bool, so compute it directly\n column = backend.df_engine.map_objects(column, strings_utils.str2bool)\n\n proc_df[feature_config[PROC_COLUMN]] = column.astype(np.bool_)\n\n return proc_df\n\n\nclass BinaryInputFeature(BinaryFeatureMixin, InputFeature):\n def __init__(self, input_feature_config: BinaryInputFeatureConfig, encoder_obj=None, **kwargs):\n super().__init__(input_feature_config, **kwargs)\n input_feature_config.encoder.input_size = self.input_shape[-1]\n\n if encoder_obj:\n self.encoder_obj = encoder_obj\n else:\n self.encoder_obj = self.initialize_encoder(input_feature_config.encoder)\n\n def forward(self, inputs):\n assert isinstance(inputs, torch.Tensor)\n assert inputs.dtype in [torch.bool, torch.int64, torch.float32]\n assert len(inputs.shape) == 1 or (len(inputs.shape) == 2 and inputs.shape[1] == 1)\n\n if len(inputs.shape) == 1:\n inputs = inputs[:, None]\n\n # Inputs to the binary encoder could be of dtype torch.bool. Linear layer\n # weights are of dtype torch.float32. The inputs and the weights need to\n # be of the same dtype.\n if inputs.dtype == torch.bool:\n inputs = inputs.type(torch.float32)\n\n encoder_outputs = self.encoder_obj(inputs)\n return encoder_outputs\n\n @property\n def input_dtype(self):\n return torch.bool\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([1])\n\n @property\n def output_shape(self) -> torch.Size:\n return self.encoder_obj.output_shape\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n pass\n\n @staticmethod\n def get_schema_cls():\n return BinaryInputFeatureConfig\n\n def create_sample_input(self, batch_size: int = 2):\n return torch.rand([batch_size]) > 0.5\n\n @classmethod\n def get_preproc_input_dtype(cls, metadata: TrainingSetMetadataDict) -> str:\n return \"string\" if metadata.get(\"str2bool\") else \"int32\"\n\n @staticmethod\n def create_preproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n return _BinaryPreprocessing(metadata)\n\n\nclass BinaryOutputFeature(BinaryFeatureMixin, OutputFeature):\n def __init__(\n self,\n output_feature_config: BinaryOutputFeatureConfig | dict,\n output_features: dict[str, OutputFeature],\n **kwargs,\n ):\n self.threshold = output_feature_config.threshold\n super().__init__(output_feature_config, output_features, **kwargs)\n self.decoder_obj = self.initialize_decoder(output_feature_config.decoder)\n self._setup_loss()\n self._setup_metrics()\n\n def logits(self, inputs, **kwargs):\n hidden = inputs[HIDDEN]\n return self.decoder_obj(hidden)\n\n def create_calibration_module(self, feature: BinaryOutputFeatureConfig) -> torch.nn.Module:\n \"\"\"Creates the appropriate calibration module based on the feature config.\n\n Today, only one type of calibration (\"temperature_scaling\") is available, but more options may be supported in\n the future.\n \"\"\"\n if feature.calibration:\n calibration_cls = calibration.get_calibration_cls(BINARY, \"temperature_scaling\")\n return calibration_cls(binary=True)\n return None\n\n def create_predict_module(self) -> PredictModule:\n # A lot of code assumes output features have a prediction module, but if we are using a passthrough\n # decoder then there is no threshold.\n threshold = getattr(self, \"threshold\", 0.5)\n return _BinaryPredict(threshold, calibration_module=self.calibration_module)\n\n def get_prediction_set(self):\n return {PREDICTIONS, PROBABILITIES, LOGITS}\n\n @classmethod\n def get_output_dtype(cls):\n return torch.bool\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([1])\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([1])\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n pass\n\n @staticmethod\n def calculate_overall_stats(predictions, targets, train_set_metadata):\n overall_stats = {}\n confusion_matrix = ConfusionMatrix(targets, predictions[PREDICTIONS], labels=[\"False\", \"True\"])\n overall_stats[\"confusion_matrix\"] = confusion_matrix.cm.tolist()\n overall_stats[\"overall_stats\"] = confusion_matrix.stats()\n overall_stats[\"per_class_stats\"] = confusion_matrix.per_class_stats()\n fpr, tpr, thresholds = roc_curve(targets, predictions[PROBABILITIES])\n overall_stats[\"roc_curve\"] = {\n \"false_positive_rate\": fpr.tolist(),\n \"true_positive_rate\": tpr.tolist(),\n }\n overall_stats[\"roc_auc_macro\"] = roc_auc_score(targets, predictions[PROBABILITIES], average=\"macro\")\n overall_stats[\"roc_auc_micro\"] = roc_auc_score(targets, predictions[PROBABILITIES], average=\"micro\")\n ps, rs, thresholds = precision_recall_curve(targets, predictions[PROBABILITIES])\n overall_stats[\"precision_recall_curve\"] = {\n \"precisions\": ps.tolist(),\n \"recalls\": rs.tolist(),\n }\n overall_stats[\"average_precision_macro\"] = average_precision_score(\n targets, predictions[PROBABILITIES], average=\"macro\"\n )\n overall_stats[\"average_precision_micro\"] = average_precision_score(\n targets, predictions[PROBABILITIES], average=\"micro\"\n )\n overall_stats[\"average_precision_samples\"] = average_precision_score(\n targets, predictions[PROBABILITIES], average=\"samples\"\n )\n\n return overall_stats\n\n def postprocess_predictions(\n self,\n result,\n metadata,\n ):\n class_names = [\"False\", \"True\"]\n if \"bool2str\" in metadata:\n class_names = metadata[\"bool2str\"]\n\n predictions_col = f\"{self.feature_name}_{PREDICTIONS}\"\n if predictions_col in result:\n if \"bool2str\" in metadata:\n result[predictions_col] = result[predictions_col].map(\n lambda pred: metadata[\"bool2str\"][pred],\n )\n\n probabilities_col = f\"{self.feature_name}_{PROBABILITIES}\"\n if probabilities_col in result:\n false_col = f\"{probabilities_col}_{class_names[0]}\"\n true_col = f\"{probabilities_col}_{class_names[1]}\"\n prob_col = f\"{self.feature_name}_{PROBABILITY}\"\n\n result = result.assign(\n **{\n false_col: lambda x: 1 - x[probabilities_col],\n true_col: lambda x: x[probabilities_col],\n prob_col: np.where(\n result[probabilities_col] > 0.5, result[probabilities_col], 1 - result[probabilities_col]\n ),\n probabilities_col: result[probabilities_col].map(lambda x: [1 - x, x]),\n },\n )\n\n return result\n\n @staticmethod\n def get_schema_cls():\n return BinaryOutputFeatureConfig\n\n @classmethod\n def get_postproc_output_dtype(cls, metadata: TrainingSetMetadataDict) -> str:\n return \"string\" if metadata.get(\"bool2str\") else \"int32\"\n\n @staticmethod\n def create_postproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n return _BinaryPostprocessing(metadata)\n\n def metric_kwargs(self) -> dict:\n \"\"\"Returns arguments that are used to instantiate an instance of each metric class.\"\"\"\n return {\"task\": \"binary\"}\n"
},
{
"path": "ludwig/features/category_feature.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nfrom typing import Any\n\nimport numpy as np\nimport torch\n\nfrom ludwig.constants import (\n CATEGORY,\n CATEGORY_DISTRIBUTION,\n COLUMN,\n HIDDEN,\n LOGITS,\n NAME,\n PREDICTIONS,\n PREPROCESSING,\n PROBABILITIES,\n PROBABILITY,\n PROC_COLUMN,\n PROJECTION_INPUT,\n)\nfrom ludwig.error import InputDataError\nfrom ludwig.features.base_feature import BaseFeatureMixin, InputFeature, OutputFeature, PredictModule\nfrom ludwig.features.vector_feature import VectorFeatureMixin\nfrom ludwig.schema.features.category_feature import (\n CategoryDistributionOutputFeatureConfig,\n CategoryInputFeatureConfig,\n CategoryOutputFeatureConfig,\n)\nfrom ludwig.schema.features.loss.loss import CORNLossConfig\nfrom ludwig.types import (\n FeatureMetadataDict,\n FeaturePostProcessingOutputDict,\n ModelConfigDict,\n PreprocessingConfigDict,\n TrainingSetMetadataDict,\n)\nfrom ludwig.utils import calibration, output_feature_utils\nfrom ludwig.utils.eval_utils import ConfusionMatrix\nfrom ludwig.utils.math_utils import int_type, softmax\nfrom ludwig.utils.strings_utils import create_vocabulary_single_token, UNKNOWN_SYMBOL\nfrom ludwig.utils.types import TorchscriptPreprocessingInput\n\nlogger = logging.getLogger(__name__)\n\n\nclass _CategoryPreprocessing(torch.nn.Module):\n def __init__(self, metadata: TrainingSetMetadataDict):\n super().__init__()\n self.str2idx = metadata[\"str2idx\"]\n if UNKNOWN_SYMBOL in self.str2idx:\n self.unk = self.str2idx[UNKNOWN_SYMBOL]\n else:\n # self.unk is set to 0 to comply with Torchscript type tracing and will\n # likely not be used during training, but potentially during inference\n self.unk = 0\n\n def forward(self, v: TorchscriptPreprocessingInput) -> torch.Tensor:\n if not torch.jit.isinstance(v, list[str]):\n raise ValueError(f\"Unsupported input: {v}\")\n\n indices = [self.str2idx.get(s.strip(), self.unk) for s in v]\n return torch.tensor(indices, dtype=torch.int32)\n\n\nclass _CategoryPostprocessing(torch.nn.Module):\n def __init__(self, metadata: TrainingSetMetadataDict):\n super().__init__()\n self.idx2str = {i: v for i, v in enumerate(metadata[\"idx2str\"])}\n self.unk = UNKNOWN_SYMBOL\n self.predictions_key = PREDICTIONS\n self.probabilities_key = PROBABILITIES\n\n def forward(self, preds: dict[str, torch.Tensor], feature_name: str) -> FeaturePostProcessingOutputDict:\n predictions = output_feature_utils.get_output_feature_tensor(preds, feature_name, self.predictions_key)\n probabilities = output_feature_utils.get_output_feature_tensor(preds, feature_name, self.probabilities_key)\n\n inv_preds = [self.idx2str.get(pred, self.unk) for pred in predictions]\n\n return {\n self.predictions_key: inv_preds,\n self.probabilities_key: probabilities,\n }\n\n\nclass _CategoryPredict(PredictModule):\n def __init__(self, calibration_module=None, use_cumulative_probs=False):\n super().__init__()\n self.calibration_module = calibration_module\n\n # Derive the label from the cumulative probability distribution of the ordered category logits.\n # Taken from CORN loss implementation:\n # https://github.com/Raschka-research-group/coral-pytorch/blob/main/coral_pytorch/dataset.py#L123\n self.use_cumulative_probs = use_cumulative_probs\n\n def forward(self, inputs: dict[str, torch.Tensor], feature_name: str) -> dict[str, torch.Tensor]:\n logits = output_feature_utils.get_output_feature_tensor(inputs, feature_name, self.logits_key)\n\n if self.use_cumulative_probs:\n if self.calibration_module is not None:\n probabilities = self.calibration_module(logits)\n else:\n probabilities = torch.sigmoid(logits)\n probabilities = torch.cumprod(probabilities, dim=1)\n\n predict_levels = probabilities > 0.5\n predictions = torch.sum(predict_levels, dim=1)\n else:\n if self.calibration_module is not None:\n probabilities = self.calibration_module(logits)\n else:\n probabilities = torch.softmax(logits, -1)\n predictions = torch.argmax(probabilities, -1)\n\n predictions = predictions.long()\n\n # EXPECTED SHAPE OF RETURNED TENSORS\n # predictions: [batch_size]\n # probabilities: [batch_size, num_classes]\n # logits: [batch_size, num_classes]\n return {self.predictions_key: predictions, self.probabilities_key: probabilities, self.logits_key: logits}\n\n\nclass CategoryFeatureMixin(BaseFeatureMixin):\n @staticmethod\n def type():\n return CATEGORY\n\n @staticmethod\n def cast_column(column, backend):\n return column.astype(str)\n\n @staticmethod\n def get_feature_meta(\n config: ModelConfigDict,\n column,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n is_input_feature: bool,\n ) -> FeatureMetadataDict:\n idx2str, str2idx, str2freq = create_vocabulary_single_token(\n column,\n num_most_frequent=preprocessing_parameters[\"most_common\"],\n processor=backend.df_engine,\n )\n\n if \"vocab\" in preprocessing_parameters and preprocessing_parameters[\"vocab\"]: # Check that vocab is non-empty\n # If vocab was explciitly provided, override the inferred vocab\n idx2str = preprocessing_parameters[\"vocab\"]\n str2idx = {s: i for i, s in enumerate(idx2str)}\n str2freq = {k: str2freq.get(k, 0) for k in idx2str}\n\n if \"fallback_label\" in preprocessing_parameters:\n # This is a category output feature for LLMs\n # Check if the fallback label is in the vocab, if not add it.\n if preprocessing_parameters[\"fallback_label\"] not in str2idx:\n str2idx[preprocessing_parameters[\"fallback_label\"]] = len(str2idx)\n idx2str.append(preprocessing_parameters[\"fallback_label\"])\n str2freq[preprocessing_parameters[\"fallback_label\"]] = 0\n\n vocab_size = len(str2idx)\n if not is_input_feature and vocab_size <= 1:\n # Category output feature with vocab size 1\n raise InputDataError(\n column.name,\n CATEGORY,\n f\"\"\"\n At least 2 distinct values are required for category output features, but column\n only contains {str(idx2str)}.\n \"\"\",\n )\n if vocab_size <= 1:\n # Category input feature with vocab size 1\n logger.info(\n f\"Input feature '{column.name}' contains only 1 distinct value {str(idx2str)}. This is not useful\"\n \" for machine learning models because this feature has zero variance. Consider removing this feature\"\n \" from your input features.\"\n )\n return {\"idx2str\": idx2str, \"str2idx\": str2idx, \"str2freq\": str2freq, \"vocab_size\": vocab_size}\n\n @staticmethod\n def feature_data(backend, column, metadata):\n def __replace_token_with_idx(value: Any, metadata: TrainingSetMetadataDict, fallback_symbol_idx: int) -> int:\n stripped_value = value.strip()\n if stripped_value in metadata[\"str2idx\"]:\n return metadata[\"str2idx\"][stripped_value]\n logger.warning(f\"\"\"\n Encountered unknown symbol '{stripped_value}' for '{column.name}' during category\n feature preprocessing. This should never happen during training. If this happens during\n inference, this may be an indication that not all possible symbols were present in your\n training set. Consider re-splitting your data to ensure full representation, or setting\n preprocessing.most_common parameter to be smaller than this feature's total vocabulary\n size, {len(metadata[\"str2idx\"])}, which will ensure that the model is architected and\n trained with an UNKNOWN symbol. Returning the index for the most frequent symbol,\n {metadata[\"idx2str\"][fallback_symbol_idx]}, instead.\n \"\"\")\n return fallback_symbol_idx\n\n # No unknown symbol in Metadata from preprocessing means that all values\n # should be mappable to vocabulary\n if UNKNOWN_SYMBOL not in metadata[\"str2idx\"]:\n # If no unknown is defined, just use the most popular token's index as the fallback index\n most_popular_token = max(metadata[\"str2freq\"], key=metadata[\"str2freq\"].get)\n most_popular_token_idx = metadata[\"str2idx\"].get(most_popular_token)\n return backend.df_engine.map_objects(\n column,\n lambda x: __replace_token_with_idx(x, metadata, most_popular_token_idx),\n meta=(column.name, int),\n ).astype(int_type(metadata[\"vocab_size\"]))\n else:\n return backend.df_engine.map_objects(\n column,\n lambda x: (\n metadata[\"str2idx\"][x.strip()]\n if x.strip() in metadata[\"str2idx\"]\n else metadata[\"str2idx\"][UNKNOWN_SYMBOL]\n ),\n meta=(column.name, int),\n ).astype(int_type(metadata[\"vocab_size\"]))\n\n @staticmethod\n def add_feature_data(\n feature_config,\n input_df,\n proc_df,\n metadata,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n skip_save_processed_input,\n ):\n proc_df[feature_config[PROC_COLUMN]] = CategoryFeatureMixin.feature_data(\n backend,\n input_df[feature_config[COLUMN]],\n metadata[feature_config[NAME]],\n )\n\n return proc_df\n\n\nclass CategoryDistributionFeatureMixin(VectorFeatureMixin):\n @staticmethod\n def type():\n return CATEGORY_DISTRIBUTION\n\n @staticmethod\n def get_feature_meta(\n config: ModelConfigDict,\n column,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n is_input_feature: bool,\n ) -> FeatureMetadataDict:\n idx2str = preprocessing_parameters[\"vocab\"]\n str2idx = {s: i for i, s in enumerate(idx2str)}\n return {\n \"preprocessing\": preprocessing_parameters,\n \"idx2str\": idx2str,\n \"str2idx\": str2idx,\n \"vocab_size\": len(idx2str),\n }\n\n\nclass CategoryInputFeature(CategoryFeatureMixin, InputFeature):\n def __init__(self, input_feature_config: CategoryInputFeatureConfig, encoder_obj=None, **kwargs):\n super().__init__(input_feature_config, **kwargs)\n\n if encoder_obj:\n self.encoder_obj = encoder_obj\n else:\n self.encoder_obj = self.initialize_encoder(input_feature_config.encoder)\n\n def forward(self, inputs):\n assert isinstance(inputs, torch.Tensor)\n assert inputs.dtype in (torch.int8, torch.int16, torch.int32, torch.int64)\n assert len(inputs.shape) == 1 or (len(inputs.shape) == 2 and inputs.shape[1] == 1)\n\n inputs = inputs.reshape(-1, 1)\n if inputs.dtype == torch.int8 or inputs.dtype == torch.int16:\n inputs = inputs.type(torch.int)\n encoder_output = self.encoder_obj(inputs)\n\n return encoder_output\n\n @property\n def input_dtype(self):\n return torch.int32\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([1])\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size(self.encoder_obj.output_shape)\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n feature_config.encoder.vocab = feature_metadata[\"idx2str\"]\n feature_config.encoder.skip = feature_metadata[PREPROCESSING].get(\"cache_encoder_embeddings\", False)\n\n @staticmethod\n def get_schema_cls():\n return CategoryInputFeatureConfig\n\n @staticmethod\n def create_preproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n return _CategoryPreprocessing(metadata)\n\n\nclass CategoryOutputFeature(CategoryFeatureMixin, OutputFeature):\n def __init__(\n self,\n output_feature_config: CategoryOutputFeatureConfig | dict,\n output_features: dict[str, OutputFeature],\n **kwargs,\n ):\n self.num_classes = output_feature_config.num_classes\n self.top_k = output_feature_config.top_k\n\n # TODO(travis): make this more general to other cumulative loss functions\n self.use_cumulative_probs = isinstance(output_feature_config.loss, CORNLossConfig)\n\n super().__init__(output_feature_config, output_features, **kwargs)\n if hasattr(output_feature_config.decoder, \"num_classes\"):\n output_feature_config.decoder.num_classes = output_feature_config.num_classes\n self.decoder_obj = self.initialize_decoder(output_feature_config.decoder)\n self._setup_loss()\n self._setup_metrics()\n\n def logits(self, inputs, **kwargs): # hidden\n hidden = inputs[HIDDEN]\n\n # EXPECTED SHAPES FOR RETURNED TENSORS\n # logits: shape [batch_size, num_classes]\n # hidden: shape [batch_size, size of final fully connected layer]\n return {LOGITS: self.decoder_obj(hidden), PROJECTION_INPUT: hidden}\n\n def create_calibration_module(self, feature: CategoryOutputFeatureConfig) -> torch.nn.Module:\n \"\"\"Creates the appropriate calibration module based on the feature config.\n\n Today, only one type of calibration (\"temperature_scaling\") is available, but more options may be supported in\n the future.\n \"\"\"\n if feature.calibration:\n calibration_cls = calibration.get_calibration_cls(CATEGORY, \"temperature_scaling\")\n return calibration_cls(num_classes=self.num_classes)\n return None\n\n def create_predict_module(self) -> PredictModule:\n return _CategoryPredict(\n calibration_module=self.calibration_module, use_cumulative_probs=self.use_cumulative_probs\n )\n\n def get_prediction_set(self):\n return {PREDICTIONS, PROBABILITIES, LOGITS}\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.input_size])\n\n @classmethod\n def get_output_dtype(cls):\n return torch.int64\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([1])\n\n def metric_kwargs(self):\n return {\"top_k\": self.top_k, \"num_classes\": self.num_classes}\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n feature_config.num_classes = feature_metadata[\"vocab_size\"]\n feature_config.top_k = min(feature_config.num_classes, feature_config.top_k)\n\n # If labels are provided, then this is a classification task for LLMs\n if hasattr(feature_config.preprocessing, \"vocab\"):\n # Enrich the feature config's decoder with str2idx\n feature_config.decoder.str2idx = feature_metadata[\"str2idx\"]\n\n if isinstance(feature_config.loss.class_weights, (list, tuple)):\n if len(feature_config.loss.class_weights) != feature_config.num_classes:\n raise ValueError(\n f\"The length of class_weights ({len(feature_config.loss.class_weights)}) is not compatible with \"\n f\"the number of classes ({feature_config.num_classes}) for feature {feature_config.column}. \"\n \"Check the metadata JSON file to see the classes \"\n \"and their order and consider there needs to be a weight \"\n \"for the class too.\"\n )\n\n if isinstance(feature_config.loss.class_weights, dict):\n if feature_metadata[\"str2idx\"].keys() != feature_config.loss.class_weights.keys():\n raise ValueError(\n f\"The class_weights keys ({feature_config.loss.class_weights.keys()}) are not compatible with \"\n f'the classes ({feature_metadata[\"str2idx\"].keys()}) of feature {feature_config.column}. '\n \"Check the metadata JSON file to see the classes \"\n \"and consider there needs to be a weight \"\n \"for the class too.\"\n )\n else:\n class_weights = feature_config.loss.class_weights\n idx2str = feature_metadata[\"idx2str\"]\n class_weights_list = [class_weights[s] for s in idx2str]\n feature_config.loss.class_weights = class_weights_list\n\n if feature_config.loss.class_similarities_temperature > 0:\n if feature_config.loss.class_similarities is not None:\n similarities = feature_config.loss.class_similarities\n temperature = feature_config.loss.class_similarities_temperature\n\n curr_row = 0\n first_row_length = 0\n is_first_row = True\n for row in similarities:\n if is_first_row:\n first_row_length = len(row)\n is_first_row = False\n curr_row += 1\n else:\n curr_row_length = len(row)\n if curr_row_length != first_row_length:\n raise ValueError(\n \"The length of row {} of the class_similarities \"\n \"of {} is {}, different from the length of \"\n \"the first row {}. All rows must have \"\n \"the same length.\".format(\n curr_row, feature_config.column, curr_row_length, first_row_length\n )\n )\n else:\n curr_row += 1\n all_rows_length = first_row_length\n\n if all_rows_length != len(similarities):\n raise ValueError(\n \"The class_similarities matrix of {} has \"\n \"{} rows and {} columns, \"\n \"their number must be identical.\".format(\n feature_config.column, len(similarities), all_rows_length\n )\n )\n\n if all_rows_length != feature_config.num_classes:\n raise ValueError(\n f\"The size of the class_similarities matrix of {feature_config.column} is \"\n f\"{all_rows_length}, different from the number of classes ({feature_config.num_classes}). \"\n \"Check the metadata JSON file to see the classes \"\n \"and their order and \"\n \"consider class too.\"\n )\n\n similarities = np.array(similarities, dtype=np.float32)\n for i in range(len(similarities)):\n similarities[i, :] = softmax(similarities[i, :], temperature=temperature)\n\n feature_config.loss.class_similarities = similarities\n else:\n raise ValueError(\n \"class_similarities_temperature > 0, \"\n \"but no class_similarities are provided \"\n \"for feature {}\".format(feature_config.column)\n )\n\n @staticmethod\n def calculate_overall_stats(predictions, targets, train_set_metadata):\n overall_stats = {}\n confusion_matrix = ConfusionMatrix(targets, predictions[PREDICTIONS], labels=train_set_metadata[\"idx2str\"])\n overall_stats[\"confusion_matrix\"] = confusion_matrix.cm.tolist()\n overall_stats[\"overall_stats\"] = confusion_matrix.stats()\n overall_stats[\"per_class_stats\"] = confusion_matrix.per_class_stats()\n\n return overall_stats\n\n def postprocess_predictions(\n self,\n predictions,\n metadata,\n ):\n predictions_col = f\"{self.feature_name}_{PREDICTIONS}\"\n if predictions_col in predictions:\n if \"idx2str\" in metadata:\n predictions[predictions_col] = predictions[predictions_col].map(lambda pred: metadata[\"idx2str\"][pred])\n\n probabilities_col = f\"{self.feature_name}_{PROBABILITIES}\"\n if probabilities_col in predictions:\n prob_col = f\"{self.feature_name}_{PROBABILITY}\"\n predictions[prob_col] = predictions[probabilities_col].map(max)\n predictions[probabilities_col] = predictions[probabilities_col].map(lambda pred: pred.tolist())\n if \"idx2str\" in metadata:\n for i, label in enumerate(metadata[\"idx2str\"]):\n key = f\"{probabilities_col}_{label}\"\n\n # Use default param to force a capture before the loop completes, see:\n # https://stackoverflow.com/questions/2295290/what-do-lambda-function-closures-capture\n predictions[key] = predictions[probabilities_col].map(\n lambda prob, i=i: prob[i],\n )\n\n top_k_col = f\"{self.feature_name}_predictions_top_k\"\n if top_k_col in predictions:\n if \"idx2str\" in metadata:\n predictions[top_k_col] = predictions[top_k_col].map(\n lambda pred_top_k: [metadata[\"idx2str\"][pred] for pred in pred_top_k]\n )\n\n return predictions\n\n @staticmethod\n def get_schema_cls():\n return CategoryOutputFeatureConfig\n\n @staticmethod\n def create_postproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n return _CategoryPostprocessing(metadata)\n\n\nclass CategoryDistributionOutputFeature(CategoryDistributionFeatureMixin, CategoryOutputFeature):\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.input_size])\n\n @classmethod\n def get_output_dtype(cls):\n return torch.float32\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.num_classes])\n\n @staticmethod\n def get_schema_cls():\n return CategoryDistributionOutputFeatureConfig\n"
},
{
"path": "ludwig/features/date_feature.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nfrom datetime import date, datetime\n\nimport numpy as np\nimport torch\n\nfrom ludwig.constants import COLUMN, DATE, PROC_COLUMN\nfrom ludwig.features.base_feature import BaseFeatureMixin, InputFeature\nfrom ludwig.schema.features.date_feature import DateInputFeatureConfig\nfrom ludwig.types import (\n FeatureConfigDict,\n FeatureMetadataDict,\n ModelConfigDict,\n PreprocessingConfigDict,\n TrainingSetMetadataDict,\n)\nfrom ludwig.utils.date_utils import create_vector_from_datetime_obj, parse_datetime\nfrom ludwig.utils.types import DataFrame, TorchscriptPreprocessingInput\n\nlogger = logging.getLogger(__name__)\n\nDATE_VECTOR_LENGTH = 9\n\n\nclass _DatePreprocessing(torch.nn.Module):\n def __init__(self, metadata: TrainingSetMetadataDict):\n super().__init__()\n\n def forward(self, v: TorchscriptPreprocessingInput) -> torch.Tensor:\n if torch.jit.isinstance(v, list[torch.Tensor]):\n v = torch.stack(v)\n\n if torch.jit.isinstance(v, torch.Tensor):\n return v.to(dtype=torch.int)\n else:\n raise ValueError(f\"Unsupported input: {v}\")\n\n\nclass DateFeatureMixin(BaseFeatureMixin):\n @staticmethod\n def type():\n return DATE\n\n @staticmethod\n def cast_column(column, backend):\n return column\n\n @staticmethod\n def get_feature_meta(\n config: ModelConfigDict,\n column,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n is_input_feature: bool,\n ) -> FeatureMetadataDict:\n return {\"preprocessing\": preprocessing_parameters}\n\n @staticmethod\n def date_to_list(date_value, datetime_format, preprocessing_parameters):\n try:\n if isinstance(date_value, datetime):\n datetime_obj = date_value\n elif isinstance(date_value, date):\n datetime_obj = datetime.combine(date=date_value, time=datetime.min.time())\n elif isinstance(date_value, str) and datetime_format is not None:\n try:\n datetime_obj = datetime.strptime(date_value, datetime_format)\n except ValueError:\n datetime_obj = parse_datetime(date_value)\n else:\n datetime_obj = parse_datetime(date_value)\n except Exception as e:\n logger.error(\n f\"Error parsing date: '{date_value}' with error '{e}' \"\n \"Please provide a datetime format that parses it \"\n \"in the preprocessing section of the date feature \"\n \"in the config. \"\n \"The preprocessing fill in value will be used.\"\n \"For more details: \"\n \"https://ludwig-ai.github.io/ludwig-docs/latest/configuration/features/date_features/#date-features-preprocessing\" # noqa\n )\n fill_value = preprocessing_parameters[\"fill_value\"]\n if fill_value != \"\":\n datetime_obj = parse_datetime(fill_value)\n else:\n datetime_obj = datetime.now()\n\n return create_vector_from_datetime_obj(datetime_obj)\n\n @staticmethod\n def add_feature_data(\n feature_config: FeatureConfigDict,\n input_df: DataFrame,\n proc_df: dict[str, DataFrame],\n metadata: TrainingSetMetadataDict,\n preprocessing_parameters: PreprocessingConfigDict,\n backend, # Union[Backend, str]\n skip_save_processed_input: bool,\n ) -> None:\n datetime_format = preprocessing_parameters[\"datetime_format\"]\n proc_df[feature_config[PROC_COLUMN]] = backend.df_engine.map_objects(\n input_df[feature_config[COLUMN]],\n lambda x: np.array(\n DateFeatureMixin.date_to_list(x, datetime_format, preprocessing_parameters), dtype=np.int32\n ),\n )\n return proc_df\n\n\nclass DateInputFeature(DateFeatureMixin, InputFeature):\n def __init__(self, input_feature_config: DateInputFeatureConfig, encoder_obj=None, **kwargs):\n super().__init__(input_feature_config, **kwargs)\n\n if encoder_obj:\n self.encoder_obj = encoder_obj\n else:\n self.encoder_obj = self.initialize_encoder(input_feature_config.encoder)\n\n def forward(self, inputs):\n assert isinstance(inputs, torch.Tensor), type(inputs)\n assert inputs.dtype in [torch.int16, torch.int32, torch.int64, torch.float32], inputs.dtype\n inputs_encoded = self.encoder_obj(inputs)\n return inputs_encoded\n\n @property\n def input_dtype(self):\n return torch.int32\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([DATE_VECTOR_LENGTH])\n\n @property\n def output_shape(self) -> torch.Size:\n return self.encoder_obj.output_shape\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n pass\n\n def create_sample_input(self, batch_size: int = 2):\n date = [2013, 2, 26, 1, 57, 0, 0, 0, 0]\n return torch.Tensor([date for _ in range(batch_size)]).type(torch.int32)\n\n @staticmethod\n def get_schema_cls():\n return DateInputFeatureConfig\n\n @staticmethod\n def create_preproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n return _DatePreprocessing(metadata)\n"
},
{
"path": "ludwig/features/feature_registries.py",
"content": "# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nfrom typing import Any, TYPE_CHECKING\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import (\n AUDIO,\n BAG,\n BINARY,\n CATEGORY,\n CATEGORY_DISTRIBUTION,\n DATE,\n H3,\n IMAGE,\n NUMBER,\n SEQUENCE,\n SET,\n TEXT,\n TIMESERIES,\n VECTOR,\n)\nfrom ludwig.features.audio_feature import AudioFeatureMixin, AudioInputFeature\nfrom ludwig.features.bag_feature import BagFeatureMixin, BagInputFeature\nfrom ludwig.features.binary_feature import BinaryFeatureMixin, BinaryInputFeature, BinaryOutputFeature\nfrom ludwig.features.category_feature import (\n CategoryDistributionFeatureMixin,\n CategoryDistributionOutputFeature,\n CategoryFeatureMixin,\n CategoryInputFeature,\n CategoryOutputFeature,\n)\nfrom ludwig.features.date_feature import DateFeatureMixin, DateInputFeature\nfrom ludwig.features.h3_feature import H3FeatureMixin, H3InputFeature\nfrom ludwig.features.image_feature import ImageFeatureMixin, ImageInputFeature, ImageOutputFeature\nfrom ludwig.features.number_feature import NumberFeatureMixin, NumberInputFeature, NumberOutputFeature\nfrom ludwig.features.sequence_feature import SequenceFeatureMixin, SequenceInputFeature, SequenceOutputFeature\nfrom ludwig.features.set_feature import SetFeatureMixin, SetInputFeature, SetOutputFeature\nfrom ludwig.features.text_feature import TextFeatureMixin, TextInputFeature, TextOutputFeature\nfrom ludwig.features.timeseries_feature import TimeseriesFeatureMixin, TimeseriesInputFeature, TimeseriesOutputFeature\nfrom ludwig.features.vector_feature import VectorFeatureMixin, VectorInputFeature, VectorOutputFeature\nfrom ludwig.utils.misc_utils import get_from_registry\n\nif TYPE_CHECKING:\n from ludwig.models.base import BaseModel\n from ludwig.schema.model_types.base import ModelConfig\n\n\n@DeveloperAPI\ndef get_base_type_registry() -> dict:\n return {\n TEXT: TextFeatureMixin,\n CATEGORY: CategoryFeatureMixin,\n SET: SetFeatureMixin,\n BAG: BagFeatureMixin,\n BINARY: BinaryFeatureMixin,\n NUMBER: NumberFeatureMixin,\n SEQUENCE: SequenceFeatureMixin,\n TIMESERIES: TimeseriesFeatureMixin,\n IMAGE: ImageFeatureMixin,\n AUDIO: AudioFeatureMixin,\n H3: H3FeatureMixin,\n DATE: DateFeatureMixin,\n VECTOR: VectorFeatureMixin,\n CATEGORY_DISTRIBUTION: CategoryDistributionFeatureMixin,\n }\n\n\n@DeveloperAPI\ndef get_input_type_registry() -> dict:\n return {\n TEXT: TextInputFeature,\n NUMBER: NumberInputFeature,\n BINARY: BinaryInputFeature,\n CATEGORY: CategoryInputFeature,\n SET: SetInputFeature,\n SEQUENCE: SequenceInputFeature,\n IMAGE: ImageInputFeature,\n AUDIO: AudioInputFeature,\n TIMESERIES: TimeseriesInputFeature,\n BAG: BagInputFeature,\n H3: H3InputFeature,\n DATE: DateInputFeature,\n VECTOR: VectorInputFeature,\n }\n\n\n@DeveloperAPI\ndef get_output_type_registry() -> dict:\n return {\n CATEGORY: CategoryOutputFeature,\n BINARY: BinaryOutputFeature,\n NUMBER: NumberOutputFeature,\n SEQUENCE: SequenceOutputFeature,\n SET: SetOutputFeature,\n TEXT: TextOutputFeature,\n TIMESERIES: TimeseriesOutputFeature,\n VECTOR: VectorOutputFeature,\n CATEGORY_DISTRIBUTION: CategoryDistributionOutputFeature,\n IMAGE: ImageOutputFeature,\n }\n\n\ndef update_config_with_metadata(config_obj: \"ModelConfig\", training_set_metadata: dict[str, Any]):\n # populate input features fields depending on data\n for input_feature in config_obj.input_features:\n feature = get_from_registry(input_feature.type, get_input_type_registry())\n feature.update_config_with_metadata(input_feature, training_set_metadata[input_feature.name])\n\n # populate output features fields depending on data\n for output_feature in config_obj.output_features:\n feature = get_from_registry(output_feature.type, get_output_type_registry())\n feature.update_config_with_metadata(output_feature, training_set_metadata[output_feature.name])\n\n\ndef update_config_with_model(config_obj: \"ModelConfig\", model: \"BaseModel\"):\n \"\"\"Updates the config with the final input feature params given a model.\n\n This function should only be called to update the config after the model is initialized. Currently only implemented\n for input features because it is only relevant for HuggingFace text encoders. HuggingFace text encoders only know\n their final config after class initialization.\n \"\"\"\n for input_feature in config_obj.input_features:\n model_input_feature = model.input_features.get(input_feature.name)\n model_input_feature.update_config_after_module_init(input_feature)\n"
},
{
"path": "ludwig/features/feature_utils.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport re\n\nimport numpy as np\nimport torch\n\nfrom ludwig.constants import NAME, PREPROCESSING, SEQUENCE, TEXT, TIMESERIES, TYPE\nfrom ludwig.utils.data_utils import hash_dict\nfrom ludwig.utils.strings_utils import get_tokenizer_from_registry, UNKNOWN_SYMBOL\n\nSEQUENCE_TYPES = {SEQUENCE, TEXT, TIMESERIES}\nFEATURE_NAME_SUFFIX = \"__ludwig\"\nFEATURE_NAME_SUFFIX_LENGTH = len(FEATURE_NAME_SUFFIX)\n\n\ndef should_regularize(regularize_layers):\n regularize = False\n if isinstance(regularize_layers, bool) and regularize_layers:\n regularize = True\n elif isinstance(regularize_layers, (list, tuple)) and regularize_layers and regularize_layers[-1]:\n regularize = True\n return regularize\n\n\ndef set_str_to_idx(set_string, feature_dict, tokenizer_name):\n try:\n tokenizer = get_tokenizer_from_registry(tokenizer_name)()\n except ValueError:\n raise Exception(f\"Tokenizer {tokenizer_name} not supported\")\n\n out = [feature_dict.get(item, feature_dict[UNKNOWN_SYMBOL]) for item in tokenizer(set_string)]\n\n return np.array(out, dtype=np.int32)\n\n\ndef compute_token_probabilities(\n probabilities: list | tuple | np.ndarray,\n) -> np.ndarray:\n \"\"\"Gets the maximum probability per timestep.\n\n Args:\n probabilities: An iterable of iterables or np.ndarray with shape (sequence_length, num_classes)\n where each inner iterable or np.ndarray is the probability distribution for a single timestep.\n Returns:\n An np.ndarray with shape (sequence_length,) containing the maximum probability for each timestep.\n \"\"\"\n if isinstance(probabilities, (list, tuple)):\n if not hasattr(probabilities[0], \"__len__\"):\n raise ValueError(\n \"Received token probabilities as a flat 1D list. Expected list of list of probabilities \"\n \"(sequence_length, vocab_size).\"\n )\n max_probs = []\n for timestep_probs in probabilities:\n max_probs.append(np.max(timestep_probs))\n max_probs = np.array(max_probs)\n elif isinstance(probabilities, np.ndarray):\n if len(probabilities.shape) != 2:\n raise ValueError(\n f\"Received token probabilities with non 2D shape: {probabilities.shape}. Expected shape: \"\n \"(sequence_length, vocab_size).\"\n )\n max_probs = np.max(probabilities, axis=-1)\n else:\n raise ValueError(f\"probabilities type must be in [list, tuple, np.ndarray]. Got {type(probabilities)}\")\n return max_probs\n\n\ndef compute_sequence_probability(\n sequence_probabilities: np.ndarray,\n max_sequence_length: int | None = None,\n return_log_prob: bool = True,\n) -> float:\n \"\"\"Computes the sequence level probability.\n\n Args:\n sequence_probabilities: An iterable of iterables or np.ndarray with shape (sequence_length,)\n max_sequence_length: The maximum sequence length to use. If None, uses the first dim of `sequence_probabilities`\n return_log_prob: Whether to return the log probability. Defaults to True.\n \"\"\"\n if max_sequence_length is None:\n max_sequence_length = sequence_probabilities.shape[0]\n\n sequence_probabilities = sequence_probabilities[:max_sequence_length]\n\n if return_log_prob:\n return np.sum(np.log(np.clip(sequence_probabilities, 1e-10, 1.0)))\n else:\n return np.prod(sequence_probabilities)\n\n\ndef sanitize(name):\n \"\"\"Replaces invalid id characters.\"\"\"\n return re.sub(\"\\\\W|^(?=\\\\d)\", \"_\", name)\n\n\ndef compute_feature_hash(feature: dict) -> str:\n \"\"\"This function computes a hash for each feature based on the preprocessing dictionary associated with each\n feature, as well as the feature's type.\n\n Args:\n feature: Feature dictionary\n\n Returns: Feature hash name\n \"\"\"\n feature_data = dict(\n preprocessing=feature.get(PREPROCESSING, {}),\n type=feature[TYPE],\n )\n return sanitize(feature[NAME]) + \"_\" + hash_dict(feature_data).decode(\"ascii\")\n\n\ndef get_input_size_with_dependencies(\n combiner_output_size: int, dependencies: list[str], other_output_features # Dict[str, \"OutputFeature\"]\n):\n \"\"\"Returns the input size for the first layer of this output feature's FC stack, accounting for dependencies on\n other output features.\n\n In the forward pass, the hidden states of any dependent output features get concatenated with the combiner's output.\n If this output feature depends on other output features, then the input size for this feature's FCStack is the sum\n of the output sizes of other output features + the combiner's output size.\n \"\"\"\n input_size_with_dependencies = combiner_output_size\n for feature_name in dependencies:\n if other_output_features[feature_name].fc_stack.num_layers:\n input_size_with_dependencies += other_output_features[feature_name].fc_stack.output_shape[-1]\n else:\n # 0-layer FCStack. Use the output feature's input size.\n input_size_with_dependencies += other_output_features[feature_name].input_size\n return input_size_with_dependencies\n\n\ndef get_module_dict_key_from_name(name: str, feature_name_suffix: str = FEATURE_NAME_SUFFIX) -> str:\n \"\"\"Returns a key that's guaranteed to be compatible with torch.\"\"\"\n key = name.replace(\".\", \"__ludwig_punct_period__\")\n return key + feature_name_suffix\n\n\ndef get_name_from_module_dict_key(key: str, feature_name_suffix_length: int = FEATURE_NAME_SUFFIX_LENGTH) -> str:\n \"\"\"Reverse of get_module_dict_key_from_name.\"\"\"\n name = key.replace(\"__ludwig_punct_period__\", \".\")\n return name[:-feature_name_suffix_length]\n\n\nclass LudwigFeatureDict(torch.nn.Module):\n \"\"\"Torch ModuleDict wrapper that permits keys with any name.\n\n Torch's ModuleDict implementation doesn't allow certain keys to be used if they conflict with existing class\n attributes, e.g.\n\n > torch.nn.ModuleDict({'type': torch.nn.Module()}) # Raises KeyError.\n\n This class is a simple wrapper around torch's ModuleDict that mitigates possible conflicts by using a key-suffixing\n protocol.\n\n This is also tracked in Pytorch: https://github.com/pytorch/pytorch/issues/71203.\n \"\"\"\n\n def __init__(self):\n super().__init__()\n self.module_dict = torch.nn.ModuleDict()\n self.internal_key_to_original_name_map = {}\n\n def get(self, key) -> torch.nn.Module:\n return self.module_dict[get_module_dict_key_from_name(key)]\n\n def set(self, key: str, module: torch.nn.Module) -> None:\n module_dict_key_name = get_module_dict_key_from_name(key)\n self.internal_key_to_original_name_map[module_dict_key_name] = key\n self.module_dict[module_dict_key_name] = module\n\n def __len__(self) -> int:\n return len(self.module_dict)\n\n def __next__(self) -> None:\n return next(iter(self))\n\n def __iter__(self) -> None:\n return iter(self.keys())\n\n def keys(self) -> list[str]:\n return [\n get_name_from_module_dict_key(feature_name)\n for feature_name in self.internal_key_to_original_name_map.keys()\n ]\n\n def values(self) -> list[torch.nn.Module]:\n return [module for _, module in self.module_dict.items()]\n\n def items(self) -> list[tuple[str, torch.nn.Module]]:\n return [\n (get_name_from_module_dict_key(feature_name), module) for feature_name, module in self.module_dict.items()\n ]\n\n def update(self, modules: dict[str, torch.nn.Module]) -> None:\n for feature_name, module in modules.items():\n self.set(feature_name, module)\n"
},
{
"path": "ludwig/features/h3_feature.py",
"content": "# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\n\nimport numpy as np\nimport torch\n\nfrom ludwig.constants import COLUMN, H3, PROC_COLUMN\nfrom ludwig.features.base_feature import BaseFeatureMixin, InputFeature\nfrom ludwig.schema.features.h3_feature import H3InputFeatureConfig\nfrom ludwig.types import FeatureMetadataDict, ModelConfigDict, PreprocessingConfigDict, TrainingSetMetadataDict\nfrom ludwig.utils.h3_util import h3_to_components\nfrom ludwig.utils.types import TorchscriptPreprocessingInput\n\nlogger = logging.getLogger(__name__)\n\nMAX_H3_RESOLUTION = 15\nH3_VECTOR_LENGTH = MAX_H3_RESOLUTION + 4\nH3_PADDING_VALUE = 7\n\n\nclass _H3Preprocessing(torch.nn.Module):\n def __init__(self, metadata: TrainingSetMetadataDict):\n super().__init__()\n self.max_h3_resolution = MAX_H3_RESOLUTION\n self.h3_padding_value = H3_PADDING_VALUE\n self.computed_fill_value = float(metadata[\"preprocessing\"][\"computed_fill_value\"])\n\n def forward(self, v: TorchscriptPreprocessingInput) -> torch.Tensor:\n if torch.jit.isinstance(v, list[torch.Tensor]):\n v = torch.stack(v)\n\n if not torch.jit.isinstance(v, torch.Tensor):\n raise ValueError(f\"Unsupported input: {v}\")\n\n v = torch.nan_to_num(v, nan=self.computed_fill_value)\n v = v.long()\n\n outputs: list[torch.Tensor] = []\n for v_i in v:\n components = h3_to_components(v_i)\n header: list[int] = [\n components.mode,\n components.edge,\n components.resolution,\n components.base_cell,\n ]\n cells_padding: list[int] = [self.h3_padding_value] * (self.max_h3_resolution - len(components.cells))\n output = torch.tensor(header + components.cells + cells_padding, dtype=torch.uint8, device=v.device)\n outputs.append(output)\n\n return torch.stack(outputs)\n\n\nclass H3FeatureMixin(BaseFeatureMixin):\n @staticmethod\n def type():\n return H3\n\n @staticmethod\n def cast_column(column, backend):\n try:\n return column.astype(int)\n except ValueError:\n logger.warning(\"H3Feature could not be read as int directly. Reading as float and converting to int.\")\n return column.astype(float).astype(int)\n\n @staticmethod\n def get_feature_meta(\n config: ModelConfigDict,\n column,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n is_input_feature: bool,\n ) -> FeatureMetadataDict:\n return {}\n\n @staticmethod\n def h3_to_list(h3_int):\n components = h3_to_components(h3_int)\n header = [components.mode, components.edge, components.resolution, components.base_cell]\n cells_padding = [H3_PADDING_VALUE] * (MAX_H3_RESOLUTION - len(components.cells))\n return header + components.cells + cells_padding\n\n @staticmethod\n def add_feature_data(\n feature_config,\n input_df,\n proc_df,\n metadata,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n skip_save_processed_input,\n ):\n column = input_df[feature_config[COLUMN]]\n if column.dtype == object:\n column = backend.df_engine.map_objects(column, int)\n column = backend.df_engine.map_objects(column, H3FeatureMixin.h3_to_list)\n\n proc_df[feature_config[PROC_COLUMN]] = backend.df_engine.map_objects(\n column, lambda x: np.array(x, dtype=np.uint8)\n )\n return proc_df\n\n\nclass H3InputFeature(H3FeatureMixin, InputFeature):\n def __init__(self, input_feature_config: H3InputFeatureConfig, encoder_obj=None, **kwargs):\n super().__init__(input_feature_config, **kwargs)\n\n if encoder_obj:\n self.encoder_obj = encoder_obj\n else:\n self.encoder_obj = self.initialize_encoder(input_feature_config.encoder)\n\n def forward(self, inputs):\n assert isinstance(inputs, torch.Tensor)\n assert inputs.dtype in [torch.uint8, torch.int64]\n assert len(inputs.shape) == 2\n\n inputs_encoded = self.encoder_obj(inputs)\n\n return inputs_encoded\n\n @property\n def input_dtype(self):\n return torch.uint8\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([H3_VECTOR_LENGTH])\n\n @property\n def output_shape(self) -> torch.Size:\n return self.encoder_obj.output_shape\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n pass\n\n @staticmethod\n def create_preproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n return _H3Preprocessing(metadata)\n\n @staticmethod\n def get_schema_cls():\n return H3InputFeatureConfig\n"
},
{
"path": "ludwig/features/image_feature.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nimport os\nimport warnings\nfrom collections import Counter\nfrom collections.abc import Callable\nfrom dataclasses import dataclass\nfrom functools import partial\nfrom typing import Any\n\nimport numpy as np\nimport torch\nfrom torchvision import transforms\nfrom torchvision.transforms import functional as F\nfrom torchvision.transforms.functional import normalize\n\nfrom ludwig.constants import (\n CHECKSUM,\n COLUMN,\n ENCODER,\n HEIGHT,\n IMAGE,\n IMAGENET1K,\n INFER_IMAGE_DIMENSIONS,\n INFER_IMAGE_MAX_HEIGHT,\n INFER_IMAGE_MAX_WIDTH,\n INFER_IMAGE_NUM_CLASSES,\n INFER_IMAGE_SAMPLE_SIZE,\n LOGITS,\n NAME,\n NUM_CHANNELS,\n PREDICTIONS,\n PREPROCESSING,\n PROC_COLUMN,\n REQUIRES_EQUAL_DIMENSIONS,\n SRC,\n TRAINING,\n TYPE,\n WIDTH,\n)\nfrom ludwig.data.cache.types import wrap\nfrom ludwig.encoders.base import Encoder\nfrom ludwig.encoders.image.torchvision import TVModelVariant\nfrom ludwig.features.base_feature import BaseFeatureMixin, InputFeature, OutputFeature, PredictModule\nfrom ludwig.schema.features.augmentation.base import BaseAugmentationConfig\nfrom ludwig.schema.features.augmentation.image import (\n AutoAugmentationConfig,\n RandomBlurConfig,\n RandomBrightnessConfig,\n RandomContrastConfig,\n RandomHorizontalFlipConfig,\n RandomRotateConfig,\n RandomVerticalFlipConfig,\n)\nfrom ludwig.schema.features.image_feature import ImageInputFeatureConfig, ImageOutputFeatureConfig\nfrom ludwig.types import (\n FeatureMetadataDict,\n FeaturePostProcessingOutputDict,\n ModelConfigDict,\n PreprocessingConfigDict,\n TrainingSetMetadataDict,\n)\nfrom ludwig.utils import output_feature_utils\nfrom ludwig.utils.augmentation_utils import get_augmentation_op, register_augmentation_op\nfrom ludwig.utils.data_utils import get_abs_path\nfrom ludwig.utils.dataframe_utils import is_dask_series_or_df\nfrom ludwig.utils.fs_utils import has_remote_protocol, upload_h5\nfrom ludwig.utils.image_utils import (\n get_class_mask_from_image,\n get_gray_default_image,\n get_image_from_class_mask,\n get_unique_channels,\n grayscale,\n num_channels_in_image,\n read_image_from_bytes_obj,\n read_image_from_path,\n resize_image,\n ResizeChannels,\n torchvision_model_registry,\n)\nfrom ludwig.utils.misc_utils import set_default_value\nfrom ludwig.utils.types import Series, TorchscriptPreprocessingInput\n\n# constants used for Ludwig image preprocessing\nIMAGENET1K_MEAN = [0.485, 0.456, 0.406]\nIMAGENET1K_STD = [0.229, 0.224, 0.225]\n\n\nlogger = logging.getLogger(__name__)\n\n\n###\n# Image specific augmentation operations\n###\n@register_augmentation_op(name=\"auto_augmentation\", features=IMAGE)\nclass AutoAugment(torch.nn.Module):\n def __init__(self, config: AutoAugmentationConfig):\n super().__init__()\n self.auto_augmentation_method = config.method\n self.augmentation_method = self.get_augmentation_method()\n\n def get_augmentation_method(self):\n if self.auto_augmentation_method == \"trivial_augment\":\n return transforms.TrivialAugmentWide()\n if self.auto_augmentation_method == \"auto_augment\":\n return transforms.AutoAugment()\n if self.auto_augmentation_method == \"rand_augment\":\n return transforms.RandAugment()\n raise ValueError(f\"Unsupported auto-augmentation method: {self.auto_augmentation_method}\")\n\n def forward(self, imgs: torch.Tensor) -> torch.Tensor:\n method = self.augmentation_method\n uint8imgs = imgs.to(torch.uint8)\n augmented_imgs = method(uint8imgs)\n\n return augmented_imgs.to(torch.float32)\n\n\n@register_augmentation_op(name=\"random_vertical_flip\", features=IMAGE)\nclass RandomVFlip(torch.nn.Module):\n def __init__(\n self,\n config: RandomVerticalFlipConfig,\n ):\n super().__init__()\n\n def forward(self, imgs):\n if torch.rand(1) < 0.5:\n imgs = F.vflip(imgs)\n\n return imgs\n\n\n@register_augmentation_op(name=\"random_horizontal_flip\", features=IMAGE)\nclass RandomHFlip(torch.nn.Module):\n def __init__(\n self,\n config: RandomHorizontalFlipConfig,\n ):\n super().__init__()\n\n def forward(self, imgs):\n if torch.rand(1) < 0.5:\n imgs = F.hflip(imgs)\n\n return imgs\n\n\n@register_augmentation_op(name=\"random_rotate\", features=IMAGE)\nclass RandomRotate(torch.nn.Module):\n def __init__(self, config: RandomRotateConfig):\n super().__init__()\n self.degree = config.degree\n\n def forward(self, imgs):\n if torch.rand(1) < 0.5:\n # map angle to interval (-degree, +degree)\n angle = (torch.rand(1) * 2 * self.degree - self.degree).item()\n return F.rotate(imgs, angle)\n else:\n return imgs\n\n\n@register_augmentation_op(name=\"random_contrast\", features=IMAGE)\nclass RandomContrast(torch.nn.Module):\n def __init__(self, config: RandomContrastConfig):\n super().__init__()\n self.min_contrast = config.min\n self.contrast_adjustment_range = config.max - config.min\n\n def forward(self, imgs):\n if torch.rand(1) < 0.5:\n # random contrast adjustment\n adjust_factor = (torch.rand(1) * self.contrast_adjustment_range + self.min_contrast).item()\n return F.adjust_contrast(imgs, adjust_factor)\n else:\n return imgs\n\n\n@register_augmentation_op(name=\"random_brightness\", features=IMAGE)\nclass RandomBrightness(torch.nn.Module):\n def __init__(self, config: RandomBrightnessConfig):\n super().__init__()\n self.min_brightness = config.min\n self.brightness_adjustment_range = config.max - config.min\n\n def forward(self, imgs):\n if torch.rand(1) < 0.5:\n # random contrast adjustment\n adjust_factor = (torch.rand(1) * self.brightness_adjustment_range + self.min_brightness).item()\n return F.adjust_brightness(imgs, adjust_factor)\n else:\n return imgs\n\n\n@register_augmentation_op(name=\"random_blur\", features=IMAGE)\nclass RandomBlur(torch.nn.Module):\n def __init__(self, config: RandomBlurConfig):\n super().__init__()\n self.kernel_size = [config.kernel_size, config.kernel_size]\n\n def forward(self, imgs):\n if torch.rand(1) < 0.5:\n imgs = F.gaussian_blur(imgs, self.kernel_size)\n\n return imgs\n\n\nclass ImageAugmentation(torch.nn.Module):\n def __init__(\n self,\n augmentation_list: list[BaseAugmentationConfig],\n normalize_mean: list[float] | None = None,\n normalize_std: list[float] | None = None,\n ):\n super().__init__()\n\n logger.debug(f\"Creating augmentation pipeline: {augmentation_list}\")\n\n self.normalize_mean = normalize_mean\n self.normalize_std = normalize_std\n\n if self.training:\n self.augmentation_steps = torch.nn.Sequential()\n for aug_config in augmentation_list:\n try:\n aug_op = get_augmentation_op(IMAGE, aug_config.type)\n self.augmentation_steps.append(aug_op(aug_config))\n except KeyError:\n raise ValueError(f\"Invalid augmentation operation specification: {aug_config}\")\n else:\n # TODO: should this raise an exception if not in training mode?\n self.augmentation_steps = None\n\n def forward(self, imgs):\n if self.augmentation_steps:\n # convert from float to uint8 values - this is required for the augmentation\n imgs = self._convert_back_to_uint8(imgs)\n\n logger.debug(\"Executing augmentation pipeline steps: %s\", self.augmentation_steps)\n imgs = self.augmentation_steps(imgs)\n\n # convert back to float32 values and renormalize if needed\n imgs = self._renormalize_image(imgs)\n\n return imgs\n\n # function to partially undo the TorchVision ImageClassification transformation.\n # back out the normalization step and convert from float32 to uint8 dtype\n # to make the tensor displayable as an image\n # crop size remains the same\n def _convert_back_to_uint8(self, images):\n if self.normalize_mean:\n mean = torch.as_tensor(self.normalize_mean, dtype=torch.float32).view(-1, 1, 1)\n std = torch.as_tensor(self.normalize_std, dtype=torch.float32).view(-1, 1, 1)\n return images.mul(std).add(mean).mul(255.0).type(torch.uint8)\n else:\n return images.mul(255.0).type(torch.uint8)\n\n # function to redo part of the TorchVision ImageClassification transformation.\n # convert uint8 to float32\n # apply the imagenet1k normalization\n def _renormalize_image(self, images):\n if self.normalize_mean:\n mean = torch.as_tensor(self.normalize_mean, dtype=torch.float32).view(-1, 1, 1)\n std = torch.as_tensor(self.normalize_std, dtype=torch.float32).view(-1, 1, 1)\n return images.type(torch.float32).div(255.0).sub(mean).div(std)\n else:\n return images.type(torch.float32).div(255.0)\n\n\n@dataclass\nclass ImageTransformMetadata:\n height: int\n width: int\n num_channels: int\n\n\ndef _get_torchvision_transform(\n torchvision_parameters: TVModelVariant,\n) -> tuple[torch.nn.Module, ImageTransformMetadata]:\n \"\"\"Returns a torchvision transform that is compatible with the model variant.\n\n Note that the raw torchvision transform is not returned. Instead, a Sequential module that includes\n image resizing is returned. This is because the raw torchvision transform assumes that the input image has\n three channels, which is not always the case with images input into Ludwig.\n\n Args:\n torchvision_parameters: The parameters for the torchvision model variant.\n Returns:\n (torchvision_transform, transform_metadata): A torchvision transform and the metadata for the transform.\n \"\"\"\n torchvision_transform_raw = torchvision_parameters.model_weights.DEFAULT.transforms()\n torchvision_transform = torch.nn.Sequential(\n ResizeChannels(num_channels=3),\n torchvision_transform_raw,\n )\n transform_metadata = ImageTransformMetadata(\n height=torchvision_transform_raw.crop_size[0],\n width=torchvision_transform_raw.crop_size[0],\n num_channels=len(torchvision_transform_raw.mean),\n )\n return (torchvision_transform, transform_metadata)\n\n\ndef _get_torchvision_parameters(model_type: str, model_variant: str) -> TVModelVariant:\n return torchvision_model_registry.get(model_type).get(model_variant)\n\n\ndef is_torchvision_encoder(encoder_obj: Encoder) -> bool:\n # TODO(travis): do this through an interface rather than conditional logic\n from ludwig.encoders.image.torchvision import TVBaseEncoder\n\n return isinstance(encoder_obj, TVBaseEncoder)\n\n\nclass _ImagePreprocessing(torch.nn.Module):\n \"\"\"Torchscript-enabled version of preprocessing done by ImageFeatureMixin.add_feature_data.\"\"\"\n\n def __init__(\n self,\n metadata: TrainingSetMetadataDict,\n torchvision_transform: torch.nn.Module | None = None,\n transform_metadata: ImageTransformMetadata | None = None,\n ):\n super().__init__()\n\n self.resize_method = metadata[\"preprocessing\"][\"resize_method\"]\n self.torchvision_transform = torchvision_transform\n if transform_metadata is not None:\n self.height = transform_metadata.height\n self.width = transform_metadata.width\n self.num_channels = transform_metadata.num_channels\n self.channel_class_map = torch.Tensor([])\n else:\n self.height = metadata[\"preprocessing\"][\"height\"]\n self.width = metadata[\"preprocessing\"][\"width\"]\n self.num_channels = metadata[\"preprocessing\"][\"num_channels\"]\n self.channel_class_map = torch.ByteTensor(metadata[\"preprocessing\"][\"channel_class_map\"])\n\n def forward(self, v: TorchscriptPreprocessingInput) -> torch.Tensor:\n \"\"\"Takes a list of images and adjusts the size and number of channels as specified in the metadata.\n\n If `v` is already a torch.Tensor, we assume that the images are already preprocessed to be the same size.\n \"\"\"\n # Nested conditional is a workaround to short-circuit boolean evaluation.\n if not torch.jit.isinstance(v, list[torch.Tensor]):\n if not torch.jit.isinstance(v, torch.Tensor):\n raise ValueError(f\"Unsupported input: {v}\")\n\n if self.torchvision_transform is not None:\n # perform pre-processing for torchvision pretrained model encoders\n if torch.jit.isinstance(v, list[torch.Tensor]):\n imgs = [self.torchvision_transform(img) for img in v]\n else:\n # convert batch of image tensors to a list and then run torchvision pretrained\n # model transforms on each image\n imgs = [self.torchvision_transform(img) for img in torch.unbind(v)]\n\n # collect the list of images into a batch\n imgs_stacked = torch.stack(imgs)\n else:\n # perform pre-processing for Ludwig defined image encoders\n if torch.jit.isinstance(v, list[torch.Tensor]):\n imgs = [resize_image(img, (self.height, self.width), self.resize_method) for img in v]\n imgs_stacked = torch.stack(imgs)\n else:\n imgs_stacked = v\n\n _, num_channels, height, width = imgs_stacked.shape\n\n # Ensure images are the size expected by the model\n if height != self.height or width != self.width:\n imgs_stacked = resize_image(imgs_stacked, (self.height, self.width), self.resize_method)\n\n # Ensures images have the number of channels expected by the model\n if num_channels != self.num_channels:\n if self.num_channels == 1:\n imgs_stacked = grayscale(imgs_stacked)\n elif num_channels < self.num_channels:\n extra_channels = self.num_channels - num_channels\n imgs_stacked = torch.nn.functional.pad(imgs_stacked, [0, 0, 0, 0, 0, extra_channels])\n else:\n raise ValueError(\n f\"Number of channels cannot be reconciled. metadata.num_channels = \"\n f\"{self.num_channels}, but imgs.shape[1] = {num_channels}\"\n )\n\n # Create class-masked images if required\n if self.channel_class_map.shape[0]:\n masks = []\n for img in imgs_stacked:\n mask = get_class_mask_from_image(self.channel_class_map, img)\n masks.append(mask)\n imgs_stacked = torch.stack(masks)\n else:\n imgs_stacked = imgs_stacked.type(torch.float32) / 255\n\n return imgs_stacked\n\n\nclass _ImagePostprocessing(torch.nn.Module):\n def __init__(self):\n super().__init__()\n self.logits_key = LOGITS\n self.predictions_key = PREDICTIONS\n\n def forward(self, preds: dict[str, torch.Tensor], feature_name: str) -> FeaturePostProcessingOutputDict:\n predictions = output_feature_utils.get_output_feature_tensor(preds, feature_name, self.predictions_key)\n logits = output_feature_utils.get_output_feature_tensor(preds, feature_name, self.logits_key)\n\n return {self.predictions_key: predictions, self.logits_key: logits}\n\n\nclass _ImagePredict(PredictModule):\n def forward(self, inputs: dict[str, torch.Tensor], feature_name: str) -> dict[str, torch.Tensor]:\n predictions = output_feature_utils.get_output_feature_tensor(inputs, feature_name, self.predictions_key)\n logits = output_feature_utils.get_output_feature_tensor(inputs, feature_name, self.logits_key)\n\n return {self.predictions_key: predictions, self.logits_key: logits}\n\n\nclass ImageFeatureMixin(BaseFeatureMixin):\n @staticmethod\n def type():\n return IMAGE\n\n @staticmethod\n def cast_column(column, backend):\n return column\n\n @staticmethod\n def get_feature_meta(\n config: ModelConfigDict,\n column,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n is_input_feature: bool,\n ) -> FeatureMetadataDict:\n return {PREPROCESSING: preprocessing_parameters}\n\n @staticmethod\n def _read_image_if_bytes_obj_and_resize(\n img_entry: bytes | torch.Tensor | np.ndarray | str,\n img_width: int,\n img_height: int,\n should_resize: bool,\n num_channels: int,\n resize_method: str,\n user_specified_num_channels: bool,\n standardize_image: str,\n channel_class_map: torch.Tensor,\n ) -> np.ndarray | None:\n \"\"\":param img_entry Union[bytes, torch.Tensor, np.ndarray, str]: if str file path to the image else\n torch.Tensor of the image itself :param img_width: expected width of the image :param img_height: expected\n height of the image :param should_resize: Should the image be resized? :param resize_method: type of\n resizing method :param num_channels: expected number of channels in the first image :param\n user_specified_num_channels: did the user specify num channels? :param standardize_image: specifies whether\n to standarize image with imagenet1k specifications :param channel_class_map: A tensor mapping channel\n values to classes, where dim=0 is the class :return: image object as a numpy array.\n\n Helper method to read and resize an image according to model definition. If the user doesn't specify a number of\n channels, we use the first image in the dataset as the source of truth. If any image in the dataset doesn't have\n the same number of channels as the first image, raise an exception.\n\n If the user specifies a number of channels, we try to convert all the images to the specifications by dropping\n channels/padding 0 channels\n \"\"\"\n\n if isinstance(img_entry, bytes):\n img = read_image_from_bytes_obj(img_entry, num_channels)\n elif isinstance(img_entry, str):\n img = read_image_from_path(img_entry, num_channels)\n elif isinstance(img_entry, np.ndarray):\n img = torch.from_numpy(np.array(img_entry, copy=True)).permute(2, 0, 1)\n else:\n img = img_entry\n\n if not isinstance(img, torch.Tensor):\n warnings.warn(f\"Image with value {img} cannot be read\")\n return None\n\n img_num_channels = num_channels_in_image(img)\n # Convert to grayscale if needed.\n if num_channels == 1 and img_num_channels != 1:\n img = grayscale(img)\n img_num_channels = 1\n\n if should_resize:\n img = resize_image(img, (img_height, img_width), resize_method)\n\n if user_specified_num_channels:\n # Number of channels is specified by the user\n # img_padded = np.zeros((img_height, img_width, num_channels),\n # dtype=np.uint8)\n # min_num_channels = min(num_channels, img_num_channels)\n # img_padded[:, :, :min_num_channels] = img[:, :, :min_num_channels]\n # img = img_padded\n if num_channels > img_num_channels:\n extra_channels = num_channels - img_num_channels\n img = torch.nn.functional.pad(img, [0, 0, 0, 0, 0, extra_channels])\n\n if img_num_channels != num_channels:\n logger.warning(\n \"Image has {} channels, where as {} \"\n \"channels are expected. Dropping/adding channels \"\n \"with 0s as appropriate\".format(img_num_channels, num_channels)\n )\n else:\n # If the image isn't like the first image, raise exception\n if img_num_channels != num_channels:\n raise ValueError(\n \"Image has {} channels, unlike the first image, which \"\n \"has {} channels. Make sure all the images have the same \"\n \"number of channels or use the num_channels property in \"\n \"image preprocessing\".format(img_num_channels, num_channels)\n )\n\n if img.shape[1] != img_height or img.shape[2] != img_width:\n raise ValueError(\n \"Images are not of the same size. \"\n \"Expected size is {}, \"\n \"current image size is {}.\"\n \"Images are expected to be all of the same size \"\n \"or explicit image width and height are expected \"\n \"to be provided. \"\n \"Additional information: \"\n \"https://ludwig-ai.github.io/ludwig-docs/latest/configuration/features/image_features\"\n \"#image-features-preprocessing\".format([img_height, img_width, num_channels], img.shape)\n )\n\n # Create class-masked image if required\n if channel_class_map.shape[0]:\n img = get_class_mask_from_image(channel_class_map, img)\n else:\n # casting and rescaling\n img = img.type(torch.float32) / 255\n\n if standardize_image == IMAGENET1K:\n img = normalize(img, mean=IMAGENET1K_MEAN, std=IMAGENET1K_STD)\n\n return img.numpy()\n\n @staticmethod\n def _read_image_with_pretrained_transform(\n img_entry: bytes | torch.Tensor | np.ndarray,\n transform_fn: Callable,\n ) -> np.ndarray | None:\n if isinstance(img_entry, bytes):\n img = read_image_from_bytes_obj(img_entry)\n elif isinstance(img_entry, str):\n img = read_image_from_path(img_entry)\n elif isinstance(img_entry, np.ndarray):\n img = torch.from_numpy(img_entry).permute(2, 0, 1)\n else:\n img = img_entry\n\n if not isinstance(img, torch.Tensor):\n warnings.warn(f\"Image with value {img} cannot be read\")\n return None\n\n img = transform_fn(img)\n\n return img.numpy()\n\n @staticmethod\n def _set_image_and_height_equal_for_encoder(\n width: int, height: int, preprocessing_parameters: dict, encoder_type: str\n ) -> tuple[int, int]:\n \"\"\"Some pretrained image encoders require images with the same dimension, or images with a specific width\n and heigh values. The returned width and height are set based on compatibility with the downstream encoder\n using the encoder parameters for the feature.\n\n Args:\n width: Represents the width of the image. This is either specified in the user config, or inferred using\n a sample of images.\n height: Represents the height of the image. This is either specified in the user config, or inferred using\n a sample of images.\n preprocessing_parameters: Parameters defining how the image feature should be preprocessed\n encoder_type: The name of the encoder\n\n Return:\n (width, height) Updated width and height so that they are equal\n \"\"\"\n\n if preprocessing_parameters[REQUIRES_EQUAL_DIMENSIONS] and height != width:\n width = height = min(width, height)\n # Update preprocessing parameters dictionary to reflect new height and width values\n preprocessing_parameters[\"width\"] = width\n preprocessing_parameters[\"height\"] = height\n logger.info(\n f\"Set image feature height and width to {width} to be compatible with\" f\" {encoder_type} encoder.\"\n )\n return width, height\n\n @staticmethod\n def _infer_image_size(\n image_sample: list[torch.Tensor],\n max_height: int,\n max_width: int,\n preprocessing_parameters: dict,\n encoder_type: str,\n ) -> tuple[int, int]:\n \"\"\"Infers the size to use from a group of images. The returned height will be the average height of images\n in image_sample rounded to the nearest integer, or max_height. Likewise for width.\n\n Args:\n image_sample: Sample of images to use to infer image size. Must be formatted as [channels, height, width].\n max_height: Maximum height.\n max_width: Maximum width.\n preprocessing_parameters: Parameters defining how the image feature should be preprocessed\n encoder_type: The name of the encoder\n\n Return:\n (height, width) The inferred height and width.\n \"\"\"\n\n height_avg = sum(x.shape[1] for x in image_sample) / len(image_sample)\n width_avg = sum(x.shape[2] for x in image_sample) / len(image_sample)\n height = min(int(round(height_avg)), max_height)\n width = min(int(round(width_avg)), max_width)\n\n # Update height and width if the downstream encoder requires images\n # with the same dimension or specific width and height values\n width, height = ImageFeatureMixin._set_image_and_height_equal_for_encoder(\n width, height, preprocessing_parameters, encoder_type\n )\n\n logger.debug(f\"Inferring height: {height} and width: {width}\")\n return height, width\n\n @staticmethod\n def _infer_number_of_channels(image_sample: list[torch.Tensor]):\n \"\"\"Infers the channel depth to use from a group of images.\n\n We make the assumption that the majority of datasets scraped from the web will be RGB, so if we get a mixed bag\n of images we should default to that. However, if the majority of the sample images have a specific channel depth\n (other than 3) this is probably intentional so we keep it, but log an info message.\n \"\"\"\n n_images = len(image_sample)\n channel_frequency = Counter([num_channels_in_image(x) for x in image_sample])\n if channel_frequency[1] > n_images / 2:\n # If the majority of images in sample are 1 channel, use 1.\n num_channels = 1\n elif channel_frequency[2] > n_images / 2:\n # If the majority of images in sample are 2 channel, use 2.\n num_channels = 2\n elif channel_frequency[4] > n_images / 2:\n # If the majority of images in sample are 4 channel, use 4.\n num_channels = 4\n else:\n # Default case: use 3 channels.\n num_channels = 3\n logger.info(f\"Inferring num_channels from the first {n_images} images.\")\n logger.info(\"\\n\".join([f\" images with {k} channels: {v}\" for k, v in sorted(channel_frequency.items())]))\n if num_channels == max(channel_frequency, key=channel_frequency.get):\n logger.info(\n f\"Using {num_channels} channels because it is the majority in sample. If an image with\"\n f\" a different depth is read, will attempt to convert to {num_channels} channels.\"\n )\n else:\n logger.info(f\"Defaulting to {num_channels} channels.\")\n logger.info(\n \"To explicitly set the number of channels, define num_channels in the preprocessing dictionary of \"\n \"the image input feature config.\"\n )\n return num_channels\n\n @staticmethod\n def _infer_image_num_classes(\n image_sample: list[torch.Tensor],\n num_channels: int,\n num_classes: int,\n ) -> torch.Tensor:\n \"\"\"Infers the number of channel classes from a group of images (for image segmentation). The returned\n tensor contains the channel value for each class, where dim=0 is the class.\n\n Args:\n image_sample: Sample of images to use to infer image size. Must be formatted as [channels, height, width].\n num_channels: Expected number of channels\n num_classes: Expected number of channel classes or None\n\n Return:\n channel_class_map: A tensor mapping channel values to classes, where dim=0 is the class.\n \"\"\"\n n_images = len(image_sample)\n logger.info(f\"Inferring num_classes from the first {n_images} images.\")\n channel_class_map = get_unique_channels(image_sample, num_channels, num_classes)\n\n inferred_num_classes = channel_class_map.shape[0]\n if num_classes:\n if num_classes < inferred_num_classes:\n raise ValueError(\n f\"Images inferred num classes {inferred_num_classes} exceeds `num_classes` {num_classes}.\"\n )\n elif num_classes > inferred_num_classes:\n logger.warning(\n \"Images inferred num classes {} does not match `num_classes` {}. \"\n \"Using inferred num classes {}.\".format(inferred_num_classes, num_classes, inferred_num_classes)\n )\n\n return channel_class_map\n\n @staticmethod\n def _finalize_preprocessing_parameters(\n preprocessing_parameters: dict,\n encoder_type: str,\n column: Series,\n ) -> tuple:\n \"\"\"Helper method to determine the height, width and number of channels for preprocessing the image data.\n\n This is achieved by looking at the parameters provided by the user. When there are some missing parameters, we\n fall back on to the first image in the dataset. The assumption being that all the images in the data are\n expected be of the same size with the same number of channels.\n\n Args:\n preprocessing_parameters: Parameters defining how the image feature should be preprocessed\n encoder_type: The name of the encoder\n column: The data itself. Can be a Pandas, Modin or Dask series.\n \"\"\"\n\n explicit_height_width = preprocessing_parameters[HEIGHT] or preprocessing_parameters[WIDTH]\n explicit_num_channels = NUM_CHANNELS in preprocessing_parameters and preprocessing_parameters[NUM_CHANNELS]\n\n if preprocessing_parameters[INFER_IMAGE_DIMENSIONS] and not (explicit_height_width and explicit_num_channels):\n sample_size = min(len(column), preprocessing_parameters[INFER_IMAGE_SAMPLE_SIZE])\n else:\n sample_size = 1 # Take first image\n\n sample = []\n sample_num_bytes = []\n failed_entries = []\n for image_entry in column.head(sample_size):\n if isinstance(image_entry, bytes):\n image = read_image_from_bytes_obj(image_entry)\n elif isinstance(image_entry, str):\n # Tries to read image as PNG or numpy file from the path.\n image, num_bytes = read_image_from_path(image_entry, return_num_bytes=True)\n if num_bytes is not None:\n sample_num_bytes.append(num_bytes)\n else:\n image = image_entry\n\n if isinstance(image, torch.Tensor):\n sample.append(image)\n elif isinstance(image, np.ndarray):\n sample.append(torch.from_numpy(image).permute(2, 0, 1))\n else:\n failed_entries.append(image_entry)\n if len(sample) == 0:\n failed_entries_repr = \"\\n\\t- \".join(failed_entries)\n raise ValueError(\n f\"Images dimensions cannot be inferred. Failed to read {sample_size} images as samples:\"\n f\"\\n\\t- {failed_entries_repr}.\"\n )\n\n should_resize = False\n if explicit_height_width:\n should_resize = True\n try:\n height = int(preprocessing_parameters[HEIGHT])\n width = int(preprocessing_parameters[WIDTH])\n # Update height and width if the downstream encoder requires images\n # with the same dimension or specific width and height values\n width, height = ImageFeatureMixin._set_image_and_height_equal_for_encoder(\n width, height, preprocessing_parameters, encoder_type\n )\n except ValueError as e:\n raise ValueError(\"Image height and width must be set and have \" \"positive integer values: \" + str(e))\n if height <= 0 or width <= 0:\n raise ValueError(\"Image height and width must be positive integers\")\n else:\n # User hasn't specified height and width.\n # Default to inferring from sample or first image.\n if preprocessing_parameters[INFER_IMAGE_DIMENSIONS]:\n should_resize = True\n height, width = ImageFeatureMixin._infer_image_size(\n sample,\n max_height=preprocessing_parameters[INFER_IMAGE_MAX_HEIGHT],\n max_width=preprocessing_parameters[INFER_IMAGE_MAX_WIDTH],\n preprocessing_parameters=preprocessing_parameters,\n encoder_type=encoder_type,\n )\n else:\n raise ValueError(\n \"Explicit image width/height are not set, infer_image_dimensions is false, \"\n \"and first image cannot be read, so image dimensions are unknown\"\n )\n\n if explicit_num_channels:\n # User specified num_channels in the model/feature config\n user_specified_num_channels = True\n num_channels = preprocessing_parameters[NUM_CHANNELS]\n else:\n user_specified_num_channels = False\n if preprocessing_parameters[INFER_IMAGE_DIMENSIONS]:\n user_specified_num_channels = True\n num_channels = ImageFeatureMixin._infer_number_of_channels(sample)\n elif len(sample) > 0:\n num_channels = num_channels_in_image(sample[0])\n else:\n raise ValueError(\n \"Explicit image num channels is not set, infer_image_dimensions is false, \"\n \"and first image cannot be read, so image num channels is unknown\"\n )\n\n assert isinstance(num_channels, int), ValueError(\"Number of image channels needs to be an integer\")\n\n average_file_size = np.mean(sample_num_bytes) if sample_num_bytes else None\n\n standardize_image = preprocessing_parameters[\"standardize_image\"]\n if standardize_image == \"imagenet1k\" and num_channels != 3:\n warnings.warn(\n f\"'standardize_image=imagenet1k' is defined only for 'num_channels=3' but \"\n f\"detected 'num_channels={num_channels}'. For this situation setting 'standardize_image=None'.\",\n RuntimeWarning,\n )\n standardize_image = None\n\n if preprocessing_parameters[INFER_IMAGE_NUM_CLASSES] or preprocessing_parameters[\"num_classes\"]:\n channel_class_map = ImageFeatureMixin._infer_image_num_classes(\n sample, num_channels, preprocessing_parameters[\"num_classes\"]\n )\n else:\n channel_class_map = torch.Tensor([])\n\n return (\n should_resize,\n width,\n height,\n num_channels,\n user_specified_num_channels,\n average_file_size,\n standardize_image,\n channel_class_map,\n )\n\n @staticmethod\n def add_feature_data(\n feature_config,\n input_df,\n proc_df,\n metadata,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n skip_save_processed_input,\n ):\n set_default_value(feature_config[PREPROCESSING], \"in_memory\", preprocessing_parameters[\"in_memory\"])\n\n name = feature_config[NAME]\n column = input_df[feature_config[COLUMN]]\n encoder_type = feature_config[ENCODER][TYPE] if ENCODER in feature_config.keys() else None\n\n src_path = None\n if SRC in metadata:\n src_path = os.path.dirname(os.path.abspath(metadata.get(SRC)))\n abs_path_column = backend.df_engine.map_objects(\n column,\n lambda row: get_abs_path(src_path, row) if isinstance(row, str) and not has_remote_protocol(row) else row,\n )\n\n # determine if specified encoder is a torchvision model\n model_type = feature_config[ENCODER].get(\"type\", None) if ENCODER in feature_config.keys() else None\n model_variant = feature_config[ENCODER].get(\"model_variant\") if ENCODER in feature_config.keys() else None\n if model_variant:\n torchvision_parameters = _get_torchvision_parameters(model_type, model_variant)\n else:\n torchvision_parameters = None\n\n if torchvision_parameters:\n logger.warning(\n f\"Using the transforms specified for the torchvision model {model_type} {model_variant} \"\n f\"This includes setting the number of channels is 3 and resizing the image to the needs of the model.\"\n )\n\n torchvision_transform, transform_metadata = _get_torchvision_transform(torchvision_parameters)\n\n # torchvision_parameters is not None\n # perform torchvision model transformations\n read_image_if_bytes_obj_and_resize = partial(\n ImageFeatureMixin._read_image_with_pretrained_transform,\n transform_fn=torchvision_transform,\n )\n average_file_size = None\n\n # save weight specification in preprocessing section\n preprocessing_parameters[\"torchvision_model_default_weights\"] = (\n f\"{torchvision_parameters.model_weights.DEFAULT}\"\n )\n\n # add torchvision model id to preprocessing section for torchscript\n preprocessing_parameters[\"torchvision_model_type\"] = model_type\n preprocessing_parameters[\"torchvision_model_variant\"] = model_variant\n\n # get required setup parameters for in_memory = False processing\n height = transform_metadata.height\n width = transform_metadata.width\n num_channels = transform_metadata.num_channels\n channel_class_map = torch.Tensor([])\n else:\n # torchvision_parameters is None\n # perform Ludwig specified transformations\n (\n should_resize,\n width,\n height,\n num_channels,\n user_specified_num_channels,\n average_file_size,\n standardize_image,\n channel_class_map,\n ) = ImageFeatureMixin._finalize_preprocessing_parameters(\n preprocessing_parameters, encoder_type, abs_path_column\n )\n\n metadata[name][PREPROCESSING][\"height\"] = height\n metadata[name][PREPROCESSING][\"width\"] = width\n metadata[name][PREPROCESSING][\"num_channels\"] = num_channels\n metadata[name][PREPROCESSING][\"num_classes\"] = channel_class_map.shape[0]\n metadata[name][PREPROCESSING][\"channel_class_map\"] = channel_class_map.tolist()\n\n read_image_if_bytes_obj_and_resize = partial(\n ImageFeatureMixin._read_image_if_bytes_obj_and_resize,\n img_width=width,\n img_height=height,\n should_resize=should_resize,\n num_channels=num_channels,\n resize_method=preprocessing_parameters[\"resize_method\"],\n user_specified_num_channels=user_specified_num_channels,\n standardize_image=standardize_image,\n channel_class_map=channel_class_map,\n )\n\n # TODO: alternatively use get_average_image() for unreachable images\n if channel_class_map.shape[0]:\n default_image = get_gray_default_image(1, height, width).squeeze(0)\n metadata[name][\"reshape\"] = (height, width)\n else:\n default_image = get_gray_default_image(num_channels, height, width)\n metadata[name][\"reshape\"] = (num_channels, height, width)\n\n in_memory = feature_config[PREPROCESSING][\"in_memory\"]\n if in_memory or skip_save_processed_input:\n proc_col = backend.read_binary_files(\n abs_path_column, map_fn=read_image_if_bytes_obj_and_resize, file_size=average_file_size\n )\n\n num_failed_image_reads = (\n proc_col.isna().sum().compute() if is_dask_series_or_df(proc_col, backend) else proc_col.isna().sum()\n )\n\n proc_col = backend.df_engine.map_objects(\n proc_col, lambda row: default_image if not isinstance(row, np.ndarray) else row\n )\n\n proc_df[feature_config[PROC_COLUMN]] = proc_col\n else:\n num_images = len(abs_path_column)\n num_failed_image_reads = 0\n\n data_fp = backend.cache.get_cache_path(wrap(metadata.get(SRC)), metadata.get(CHECKSUM), TRAINING)\n with upload_h5(data_fp) as h5_file:\n # todo future add multiprocessing/multithreading\n image_dataset = h5_file.create_dataset(\n feature_config[PROC_COLUMN] + \"_data\", (num_images, num_channels, height, width), dtype=np.float32\n )\n for i, img_entry in enumerate(abs_path_column):\n res = read_image_if_bytes_obj_and_resize(img_entry)\n if isinstance(res, np.ndarray):\n image_dataset[i, :height, :width, :] = res\n else:\n logger.warning(f\"Failed to read image {img_entry} while preprocessing feature `{name}`. \")\n image_dataset[i, :height, :width, :] = default_image\n num_failed_image_reads += 1\n h5_file.flush()\n\n proc_df[feature_config[PROC_COLUMN]] = np.arange(num_images)\n\n if num_failed_image_reads > 0:\n logger.warning(\n f\"Failed to read {num_failed_image_reads} images while preprocessing feature `{name}`. \"\n \"Using default image for these rows in the dataset.\"\n )\n\n return proc_df\n\n\nclass ImageInputFeature(ImageFeatureMixin, InputFeature):\n def __init__(self, input_feature_config: ImageInputFeatureConfig, encoder_obj=None, **kwargs):\n super().__init__(input_feature_config, **kwargs)\n\n if encoder_obj:\n self.encoder_obj = encoder_obj\n else:\n self.encoder_obj = self.initialize_encoder(input_feature_config.encoder)\n\n # set up for augmentation if it is enabled\n if input_feature_config.augmentation:\n # assume no image normalize is required\n normalize_mean = normalize_std = None\n\n # determine if specified encoder is a torchvision model\n if is_torchvision_encoder(self.encoder_obj):\n # encoder is a torchvision model\n normalize_mean = self.encoder_obj.normalize_mean\n normalize_std = self.encoder_obj.normalize_std\n else:\n # encoder is a Ludwig encoder, determine if standardize_image is set to IMAGENET1K\n if input_feature_config.preprocessing.standardize_image == IMAGENET1K:\n normalize_mean = IMAGENET1K_MEAN\n normalize_std = IMAGENET1K_STD\n\n # create augmentation pipeline object\n self.augmentation_pipeline = ImageAugmentation(\n input_feature_config.augmentation,\n normalize_mean,\n normalize_std,\n )\n\n def forward(self, inputs: torch.Tensor) -> torch.Tensor:\n assert isinstance(inputs, torch.Tensor), f\"inputs to image feature must be a torch tensor, got {type(inputs)}\"\n assert inputs.dtype in [torch.float32], f\"inputs to image feature must be a float32 tensor, got {inputs.dtype}\"\n\n inputs_encoded = self.encoder_obj(inputs)\n\n return inputs_encoded\n\n @property\n def input_dtype(self):\n return torch.float32\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size(self.encoder_obj.input_shape)\n\n @property\n def output_shape(self) -> torch.Size:\n return self.encoder_obj.output_shape\n\n def update_config_after_module_init(self, feature_config):\n if is_torchvision_encoder(self.encoder_obj):\n # update feature preprocessing parameters to reflect used in torchvision pretrained model\n # Note: image height and width is determined by the encoder crop_size attribute. Source of this\n # attribute is from the torchvision.transforms._presets.ImageClassification class. This class stores\n # crop_size as a single element list. the single element in this list is used to set both the height\n # and width of an image.\n feature_config.preprocessing.height = self.encoder_obj.crop_size[0]\n feature_config.preprocessing.width = self.encoder_obj.crop_size[0]\n feature_config.preprocessing.num_channels = self.encoder_obj.num_channels\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n for key in [\"height\", \"width\", \"num_channels\", \"standardize_image\"]:\n if hasattr(feature_config.encoder, key):\n setattr(feature_config.encoder, key, feature_metadata[PREPROCESSING][key])\n\n @staticmethod\n def get_schema_cls():\n return ImageInputFeatureConfig\n\n @staticmethod\n def create_preproc_module(metadata: dict[str, Any]) -> torch.nn.Module:\n model_type = metadata[\"preprocessing\"].get(\"torchvision_model_type\")\n model_variant = metadata[\"preprocessing\"].get(\"torchvision_model_variant\")\n if model_variant:\n torchvision_parameters = _get_torchvision_parameters(model_type, model_variant)\n else:\n torchvision_parameters = None\n\n if torchvision_parameters:\n torchvision_transform, transform_metadata = _get_torchvision_transform(torchvision_parameters)\n else:\n torchvision_transform = None\n transform_metadata = None\n\n return _ImagePreprocessing(\n metadata, torchvision_transform=torchvision_transform, transform_metadata=transform_metadata\n )\n\n def get_augmentation_pipeline(self):\n return self.augmentation_pipeline\n\n\nclass ImageOutputFeature(ImageFeatureMixin, OutputFeature):\n def __init__(\n self,\n output_feature_config: ImageOutputFeatureConfig | dict,\n output_features: dict[str, OutputFeature],\n **kwargs,\n ):\n super().__init__(output_feature_config, output_features, **kwargs)\n self.decoder_obj = self.initialize_decoder(output_feature_config.decoder)\n self._setup_loss()\n self._setup_metrics()\n\n def logits(self, inputs: dict[str, torch.Tensor], target=None, **kwargs):\n return self.decoder_obj(inputs, target=target)\n\n def metric_kwargs(self):\n return dict(num_outputs=self.output_shape[0])\n\n def create_predict_module(self) -> PredictModule:\n return _ImagePredict()\n\n def get_prediction_set(self):\n return self.decoder_obj.get_prediction_set()\n\n @classmethod\n def get_output_dtype(cls):\n return torch.float32\n\n @property\n def output_shape(self) -> torch.Size:\n return self.decoder_obj.output_shape\n\n @property\n def input_shape(self) -> torch.Size:\n return self.decoder_obj.input_shape\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n for key in [\"height\", \"width\", \"num_channels\", \"num_classes\", \"standardize_image\"]:\n if hasattr(feature_config.decoder, key):\n setattr(feature_config.decoder, key, feature_metadata[PREPROCESSING][key])\n\n @staticmethod\n def calculate_overall_stats(predictions, targets, metadata):\n # no overall stats, just return empty dictionary\n return {}\n\n def postprocess_predictions(\n self,\n result,\n metadata,\n ):\n predictions_col = f\"{self.feature_name}_{PREDICTIONS}\"\n\n if predictions_col in result:\n channel_class_map = torch.ByteTensor(metadata[PREPROCESSING][\"channel_class_map\"])\n\n if channel_class_map.shape[0]:\n\n def class_mask2img(row):\n pred = row[predictions_col]\n return get_image_from_class_mask(channel_class_map, pred)\n\n result[predictions_col] = result.apply(class_mask2img, axis=1)\n\n return result\n\n @staticmethod\n def create_postproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n return _ImagePostprocessing(metadata)\n\n @staticmethod\n def get_schema_cls():\n return ImageOutputFeatureConfig\n"
},
{
"path": "ludwig/features/number_feature.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport copy\nimport logging\nfrom abc import ABC, abstractmethod\nfrom typing import Any\n\nimport numpy as np\nimport pandas as pd\nimport torch\nfrom torch import nn\n\nfrom ludwig.constants import COLUMN, HIDDEN, LOGITS, NAME, NUMBER, PREDICTIONS, PROC_COLUMN\nfrom ludwig.features.base_feature import BaseFeatureMixin, InputFeature, OutputFeature, PredictModule\nfrom ludwig.schema.features.number_feature import NumberInputFeatureConfig, NumberOutputFeatureConfig\nfrom ludwig.types import (\n FeatureMetadataDict,\n FeaturePostProcessingOutputDict,\n ModelConfigDict,\n PreprocessingConfigDict,\n TrainingSetMetadataDict,\n)\nfrom ludwig.utils import output_feature_utils\nfrom ludwig.utils.misc_utils import get_from_registry\nfrom ludwig.utils.types import TorchscriptPreprocessingInput\n\nlogger = logging.getLogger(__name__)\n\n\nclass NumberTransformer(nn.Module, ABC):\n @abstractmethod\n def transform(self, x: np.ndarray) -> np.ndarray:\n pass\n\n @abstractmethod\n def inverse_transform(self, x: np.ndarray) -> np.ndarray:\n pass\n\n @abstractmethod\n def transform_inference(self, x: torch.Tensor) -> torch.Tensor:\n pass\n\n @abstractmethod\n def inverse_transform_inference(self, x: torch.Tensor) -> torch.Tensor:\n pass\n\n @staticmethod\n @abstractmethod\n def fit_transform_params(column: np.ndarray, backend: Any) -> dict[str, Any]:\n pass\n\n\nclass ZScoreTransformer(NumberTransformer):\n def __init__(self, mean: float = None, std: float = None, **kwargs: dict):\n super().__init__()\n self.mu = float(mean) if mean is not None else mean\n self.sigma = float(std) if std is not None else std\n self.feature_name = kwargs.get(NAME, \"\")\n if self.sigma == 0:\n raise RuntimeError(\n f\"Cannot apply zscore normalization to `{self.feature_name}` since it has a standard deviation of 0. \"\n f\"This is most likely because `{self.feature_name}` has a constant value of {self.mu} for all rows in \"\n \"the dataset. Consider removing this feature from your Ludwig config since it is not useful for \"\n \"your machine learning model.\"\n )\n\n def transform(self, x: np.ndarray) -> np.ndarray:\n return (x - self.mu) / self.sigma\n\n def inverse_transform(self, x: np.ndarray) -> np.ndarray:\n return x * self.sigma + self.mu\n\n def transform_inference(self, x: torch.Tensor) -> torch.Tensor:\n return (x - self.mu) / self.sigma\n\n def inverse_transform_inference(self, x: torch.Tensor) -> torch.Tensor:\n return x * self.sigma + self.mu\n\n @staticmethod\n def fit_transform_params(column: np.ndarray, backend: \"Backend\") -> dict[str, Any]: # noqa\n compute = backend.df_engine.compute\n return {\n \"mean\": compute(column.astype(np.float32).mean()),\n \"std\": compute(column.astype(np.float32).std()),\n }\n\n\nclass MinMaxTransformer(NumberTransformer):\n def __init__(self, min: float = None, max: float = None, **kwargs: dict):\n super().__init__()\n self.min_value = float(min) if min is not None else min\n self.max_value = float(max) if max is not None else max\n if self.min_value is None or self.max_value is None:\n self.range = None\n else:\n self.range = self.max_value - self.min_value\n\n def transform(self, x: np.ndarray) -> np.ndarray:\n return (x - self.min_value) / self.range\n\n def inverse_transform(self, x: np.ndarray) -> np.ndarray:\n if self.range is None:\n raise ValueError(\"Numeric transformer needs to be instantiated with \" \"min and max values.\")\n return x * self.range + self.min_value\n\n def transform_inference(self, x: torch.Tensor) -> torch.Tensor:\n return (x - self.min_value) / self.range\n\n def inverse_transform_inference(self, x: torch.Tensor) -> torch.Tensor:\n if self.range is None:\n raise ValueError(\"Numeric transformer needs to be instantiated with \" \"min and max values.\")\n return x * self.range + self.min_value\n\n @staticmethod\n def fit_transform_params(column: np.ndarray, backend: \"Backend\") -> dict[str, Any]: # noqa\n compute = backend.df_engine.compute\n return {\n \"min\": compute(column.astype(np.float32).min()),\n \"max\": compute(column.astype(np.float32).max()),\n }\n\n\nclass InterQuartileTransformer(NumberTransformer):\n def __init__(self, q1: float = None, q2: float = None, q3: float = None, **kwargs: dict):\n super().__init__()\n self.q1 = float(q1) if q1 is not None else q1\n self.q2 = float(q2) if q2 is not None else q2\n self.q3 = float(q3) if q3 is not None else q3\n if self.q1 is None or self.q3 is None:\n self.interquartile_range = None\n else:\n self.interquartile_range = self.q3 - self.q1\n self.feature_name = kwargs.get(NAME, \"\")\n if self.interquartile_range == 0:\n raise RuntimeError(\n f\"Cannot apply InterQuartileNormalization to `{self.feature_name}` since\"\n \"the interquartile range is 0, which will result in a ZeroDivisionError.\"\n )\n\n def transform(self, x: np.ndarray) -> np.ndarray:\n return (x - self.q2) / self.interquartile_range\n\n def inverse_transform(self, x: np.ndarray) -> np.ndarray:\n return x * self.interquartile_range + self.q2\n\n def transform_inference(self, x: torch.Tensor) -> torch.Tensor:\n return (x - self.q2) / self.interquartile_range\n\n def inverse_transform_inference(self, x: torch.Tensor) -> torch.Tensor:\n return x * self.interquartile_range + self.q2\n\n @staticmethod\n def fit_transform_params(column: np.ndarray, backend: \"Backend\") -> dict[str, Any]: # noqa\n # backend.df_engine.compute is not used here because `percentile` is not parallelized in dask.\n # We compute the percentile directly.\n return {\n \"q1\": np.percentile(column.astype(np.float32), 25),\n \"q2\": np.percentile(column.astype(np.float32), 50),\n \"q3\": np.percentile(column.astype(np.float32), 75),\n }\n\n\nclass Log1pTransformer(NumberTransformer):\n def __init__(self, **kwargs: dict):\n super().__init__()\n self.feature_name = kwargs.get(NAME, \"\")\n\n def transform(self, x: np.ndarray) -> np.ndarray:\n if np.any(x <= 0):\n raise ValueError(\n f\"One or more values in the `{self.feature_name}` feature are non-positive. \"\n \"log1p normalization is defined only for positive values.\"\n )\n return np.log1p(x)\n\n def inverse_transform(self, x: np.ndarray) -> np.ndarray:\n return np.expm1(x)\n\n def transform_inference(self, x: torch.Tensor) -> torch.Tensor:\n return torch.log1p(x)\n\n def inverse_transform_inference(self, x: torch.Tensor) -> torch.Tensor:\n return torch.expm1(x)\n\n @staticmethod\n def fit_transform_params(column: np.ndarray, backend: \"Backend\") -> dict[str, Any]: # noqa\n return {}\n\n\nclass IdentityTransformer(NumberTransformer):\n def __init__(self, **kwargs):\n super().__init__()\n\n def transform(self, x: np.ndarray) -> np.ndarray:\n return x\n\n def inverse_transform(self, x: np.ndarray) -> np.ndarray:\n return x\n\n def transform_inference(self, x: torch.Tensor) -> torch.Tensor:\n return x\n\n def inverse_transform_inference(self, x: torch.Tensor) -> torch.Tensor:\n return x\n\n @staticmethod\n def fit_transform_params(column: np.ndarray, backend: \"Backend\") -> dict[str, Any]: # noqa\n return {}\n\n\nnumeric_transformation_registry = {\n \"minmax\": MinMaxTransformer,\n \"zscore\": ZScoreTransformer,\n \"log1p\": Log1pTransformer,\n \"iq\": InterQuartileTransformer,\n None: IdentityTransformer,\n}\n\n\ndef get_transformer(metadata, preprocessing_parameters) -> NumberTransformer:\n return get_from_registry(\n preprocessing_parameters.get(\"normalization\", None),\n numeric_transformation_registry,\n )(**metadata)\n\n\nclass _OutlierReplacer(torch.nn.Module):\n def __init__(self, metadata: TrainingSetMetadataDict):\n super().__init__()\n self.zscore_transformer = ZScoreTransformer(**metadata)\n self.outlier_threshold = metadata[\"preprocessing\"].get(\"outlier_threshold\")\n self.computed_outlier_fill_value = float(metadata[\"preprocessing\"][\"computed_outlier_fill_value\"])\n\n def forward(self, v: torch.Tensor) -> torch.Tensor:\n outliers = self.zscore_transformer.transform_inference(v).abs().gt(self.outlier_threshold)\n v_masked = torch.masked_fill(v, outliers, torch.nan)\n\n v = torch.nan_to_num(v_masked, nan=self.computed_outlier_fill_value)\n return v.to(dtype=torch.float32)\n\n\nclass _NumberPreprocessing(torch.nn.Module):\n def __init__(self, metadata: TrainingSetMetadataDict):\n super().__init__()\n self.computed_fill_value = float(metadata[\"preprocessing\"][\"computed_fill_value\"])\n self.numeric_transformer = get_transformer(metadata, metadata[\"preprocessing\"])\n\n # Optional outlier replacement\n self.outlier_replacer = None\n if metadata[\"preprocessing\"].get(\"outlier_strategy\") is not None:\n self.outlier_replacer = _OutlierReplacer(metadata)\n\n def forward(self, v: TorchscriptPreprocessingInput) -> torch.Tensor:\n if not torch.jit.isinstance(v, torch.Tensor):\n raise ValueError(f\"Unsupported input: {v}\")\n\n v = torch.nan_to_num(v, nan=self.computed_fill_value)\n v = v.to(dtype=torch.float32)\n\n # Handle outliers if needed\n if self.outlier_replacer is not None:\n v = self.outlier_replacer(v)\n\n return self.numeric_transformer.transform_inference(v)\n\n\nclass _NumberPostprocessing(torch.nn.Module):\n def __init__(self, metadata: TrainingSetMetadataDict):\n super().__init__()\n self.numeric_transformer = get_transformer(metadata, metadata[\"preprocessing\"])\n self.predictions_key = PREDICTIONS\n\n def forward(self, preds: dict[str, torch.Tensor], feature_name: str) -> FeaturePostProcessingOutputDict:\n predictions = output_feature_utils.get_output_feature_tensor(preds, feature_name, self.predictions_key)\n\n return {self.predictions_key: self.numeric_transformer.inverse_transform_inference(predictions)}\n\n\nclass _NumberPredict(PredictModule):\n def __init__(self, clip):\n super().__init__()\n self.clip = clip\n\n def forward(self, inputs: dict[str, torch.Tensor], feature_name: str) -> dict[str, torch.Tensor]:\n logits = output_feature_utils.get_output_feature_tensor(inputs, feature_name, self.logits_key)\n predictions = logits\n\n if self.clip is not None:\n predictions = torch.clamp(logits, self.clip[0], self.clip[1])\n logger.debug(f\" clipped_predictions: {predictions}\")\n\n return {self.predictions_key: predictions, self.logits_key: logits}\n\n\nclass NumberFeatureMixin(BaseFeatureMixin):\n @staticmethod\n def type():\n return NUMBER\n\n @staticmethod\n def cast_column(column, backend):\n return backend.df_engine.df_lib.to_numeric(column, errors=\"coerce\").astype(np.float32)\n\n @staticmethod\n def get_feature_meta(\n config: ModelConfigDict,\n column,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n is_input_feature: bool,\n ) -> FeatureMetadataDict:\n numeric_transformer: NumberTransformer = get_from_registry(\n preprocessing_parameters.get(\"normalization\", None),\n numeric_transformation_registry,\n )\n\n params = numeric_transformer.fit_transform_params(column, backend)\n\n # Ensure mean and std are computed if we're removing outliers\n outlier_strategy = preprocessing_parameters.get(\"outlier_strategy\")\n if outlier_strategy is not None and (\"mean\" not in params or \"std\" not in params):\n params.update(ZScoreTransformer.fit_transform_params(column, backend))\n\n return params\n\n @staticmethod\n def add_feature_data(\n feature_config,\n input_df,\n proc_df,\n metadata,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n skip_save_processed_input,\n ):\n # Had to replace normalize() function due to issue #1911\n # this comment is to provide context for the change.\n # original code\n # def normalize(series: pd.Series) -> pd.Series:\n # series = series.copy()\n # numeric_transformer = get_transformer(metadata[feature_config[NAME]], preprocessing_parameters)\n # series.update(numeric_transformer.transform(series.values))\n # return series\n\n def normalize(series: pd.Series) -> pd.Series:\n _feature_metadata = copy.deepcopy(metadata[feature_config[NAME]])\n _feature_metadata.update({NAME: feature_config[NAME]})\n\n # retrieve request numeric transformer\n numeric_transformer = get_transformer(_feature_metadata, preprocessing_parameters)\n\n # transform input numeric values with specified transformer\n transformed_values = numeric_transformer.transform(series.values)\n\n # return transformed values with same index values as original series.\n return pd.Series(transformed_values, index=series.index)\n\n input_series = input_df[feature_config[COLUMN]].astype(np.float32)\n proc_df[feature_config[PROC_COLUMN]] = backend.df_engine.map_partitions(\n input_series, normalize, meta=input_series\n )\n\n return proc_df\n\n\nclass NumberInputFeature(NumberFeatureMixin, InputFeature):\n def __init__(self, input_feature_config: NumberInputFeatureConfig, encoder_obj=None, **kwargs):\n super().__init__(input_feature_config, **kwargs)\n input_feature_config.encoder.input_size = self.input_shape[-1]\n\n if encoder_obj:\n self.encoder_obj = encoder_obj\n else:\n self.encoder_obj = self.initialize_encoder(input_feature_config.encoder)\n\n def forward(self, inputs):\n assert isinstance(inputs, torch.Tensor)\n assert inputs.dtype == torch.float32 or inputs.dtype == torch.float64\n assert len(inputs.shape) == 1 or (len(inputs.shape) == 2 and inputs.shape[1] == 1)\n\n if len(inputs.shape) == 1:\n inputs = inputs[:, None]\n inputs_encoded = self.encoder_obj(inputs)\n\n return inputs_encoded\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([1])\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size(self.encoder_obj.output_shape)\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n pass\n\n @staticmethod\n def get_schema_cls():\n return NumberInputFeatureConfig\n\n def create_sample_input(self, batch_size: int = 2):\n return torch.rand([batch_size])\n\n @classmethod\n def get_preproc_input_dtype(cls, metadata: TrainingSetMetadataDict) -> str:\n return \"float32\"\n\n @staticmethod\n def create_preproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n return _NumberPreprocessing(metadata)\n\n\nclass NumberOutputFeature(NumberFeatureMixin, OutputFeature):\n def __init__(\n self,\n output_feature_config: NumberOutputFeatureConfig | dict,\n output_features: dict[str, OutputFeature],\n **kwargs,\n ):\n self.clip = output_feature_config.clip\n super().__init__(output_feature_config, output_features, **kwargs)\n self.decoder_obj = self.initialize_decoder(output_feature_config.decoder)\n self._setup_loss()\n self._setup_metrics()\n\n def logits(self, inputs, **kwargs): # hidden\n hidden = inputs[HIDDEN]\n return self.decoder_obj(hidden)\n\n def create_predict_module(self) -> PredictModule:\n if getattr(self, \"clip\", None) and not (isinstance(self.clip, (list, tuple)) and len(self.clip) == 2):\n raise ValueError(\n f\"The clip parameter of {self.feature_name} is {self.clip}. \"\n f\"It must be a list or a tuple of length 2.\"\n )\n return _NumberPredict(getattr(self, \"clip\", None))\n\n def get_prediction_set(self):\n return {PREDICTIONS, LOGITS}\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.decoder_obj.config.input_size])\n\n @classmethod\n def get_output_dtype(cls):\n return torch.float32\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([1])\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n pass\n\n @staticmethod\n def calculate_overall_stats(predictions, targets, metadata):\n # no overall stats, just return empty dictionary\n return {}\n\n def postprocess_predictions(\n self,\n predictions,\n metadata,\n ):\n predictions_col = f\"{self.feature_name}_{PREDICTIONS}\"\n if predictions_col in predictions:\n # as needed convert predictions make to original value space\n numeric_transformer = get_from_registry(\n metadata[\"preprocessing\"].get(\"normalization\", None),\n numeric_transformation_registry,\n )(**metadata)\n predictions[predictions_col] = predictions[predictions_col].map(\n lambda pred: numeric_transformer.inverse_transform(pred)\n )\n\n return predictions\n\n @staticmethod\n def get_schema_cls():\n return NumberOutputFeatureConfig\n\n @classmethod\n def get_postproc_output_dtype(cls, metadata: TrainingSetMetadataDict) -> str:\n return \"float32\"\n\n @staticmethod\n def create_postproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n return _NumberPostprocessing(metadata)\n"
},
{
"path": "ludwig/features/sequence_feature.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nimport logging\nfrom functools import partial\n\nimport numpy as np\nimport torch\n\nfrom ludwig.constants import (\n COLUMN,\n LAST_PREDICTIONS,\n LENGTHS,\n NAME,\n PREDICTIONS,\n PROBABILITIES,\n PROBABILITY,\n PROC_COLUMN,\n SEQUENCE,\n)\nfrom ludwig.features.base_feature import BaseFeatureMixin, InputFeature, OutputFeature, PredictModule\nfrom ludwig.features.feature_utils import compute_sequence_probability, compute_token_probabilities\nfrom ludwig.schema.features.sequence_feature import SequenceInputFeatureConfig, SequenceOutputFeatureConfig\nfrom ludwig.types import (\n FeatureMetadataDict,\n FeaturePostProcessingOutputDict,\n ModelConfigDict,\n PreprocessingConfigDict,\n TrainingSetMetadataDict,\n)\nfrom ludwig.utils import output_feature_utils\nfrom ludwig.utils.math_utils import softmax\nfrom ludwig.utils.strings_utils import (\n build_sequence_matrix,\n create_vocabulary,\n SpecialSymbol,\n START_SYMBOL,\n STOP_SYMBOL,\n UNKNOWN_SYMBOL,\n)\nfrom ludwig.utils.tokenizers import get_tokenizer_from_registry\nfrom ludwig.utils.types import TorchscriptPreprocessingInput\n\nlogger = logging.getLogger(__name__)\n\n\nclass _SequencePreprocessing(torch.nn.Module):\n \"\"\"Torchscript-enabled version of preprocessing done by SequenceFeatureMixin.add_feature_data.\"\"\"\n\n def __init__(self, metadata: TrainingSetMetadataDict):\n super().__init__()\n self.lowercase = metadata[\"preprocessing\"][\"lowercase\"]\n self.tokenizer_type = metadata[\"preprocessing\"][\"tokenizer\"]\n self.tokenizer = get_tokenizer_from_registry(self.tokenizer_type)(\n pretrained_model_name_or_path=metadata[\"preprocessing\"].get(\"pretrained_model_name_or_path\", None)\n )\n\n if not isinstance(self.tokenizer, torch.nn.Module):\n raise ValueError(f\"tokenizer must be a torch.nn.Module, got {self.tokenizer}\")\n\n self.padding_symbol = metadata[\"preprocessing\"][\"padding_symbol\"]\n self.unknown_symbol = metadata[\"preprocessing\"][\"unknown_symbol\"]\n self.start_symbol = START_SYMBOL\n self.stop_symbol = STOP_SYMBOL\n self.max_sequence_length = int(metadata[\"max_sequence_length\"])\n self.unit_to_id = metadata[\"str2idx\"]\n self.computed_fill_value = metadata[\"preprocessing\"][\"computed_fill_value\"]\n\n def forward(self, v: TorchscriptPreprocessingInput) -> torch.Tensor:\n \"\"\"Takes a list of strings and returns a tensor of token ids.\"\"\"\n if not torch.jit.isinstance(v, list[str]):\n raise ValueError(f\"Unsupported input: {v}\")\n\n futures: list[torch.jit.Future[torch.Tensor]] = []\n for sequence in v:\n futures.append(\n torch.jit.fork(\n self._process_sequence,\n sequence,\n )\n )\n\n sequence_matrix = []\n for future in futures:\n sequence_matrix.append(torch.jit.wait(future))\n\n return torch.stack(sequence_matrix)\n\n def _process_sequence(self, sequence: str) -> torch.Tensor:\n sequence = self.computed_fill_value if sequence == \"nan\" else sequence\n\n # If tokenizer is HF, we defer lowercase transformation to the tokenizer.\n if self.lowercase and self.tokenizer_type != \"hf_tokenizer\":\n sequence_str: str = sequence.lower()\n else:\n sequence_str: str = sequence\n\n sequence_vector = torch.full([self.max_sequence_length], self.unit_to_id[self.padding_symbol])\n\n if self.tokenizer_type == \"hf_tokenizer\":\n # Handles start, stop, and unknown symbols implicitly\n unit_sequence = self.tokenizer(sequence)\n assert torch.jit.isinstance(unit_sequence, list[int])\n # Ensures that the sequence lengths are aligned between the input and output tensors.\n sequence_length = min(len(unit_sequence), self.max_sequence_length)\n sequence_vector[:sequence_length] = torch.tensor(unit_sequence)[:sequence_length]\n return sequence_vector\n\n # If tokenizer is not HF, we manually convert tokens to IDs and insert start, stop, and unknown symbols.\n unit_sequence = self.tokenizer(sequence_str)\n assert torch.jit.isinstance(unit_sequence, list[str])\n\n sequence_vector[0] = self.unit_to_id[self.start_symbol]\n if len(unit_sequence) + 1 < self.max_sequence_length:\n sequence_length = len(unit_sequence)\n sequence_vector[len(unit_sequence) + 1] = self.unit_to_id[self.stop_symbol]\n else:\n sequence_length = self.max_sequence_length - 1\n\n for i in range(sequence_length):\n curr_unit = unit_sequence[i]\n if curr_unit in self.unit_to_id:\n curr_id = self.unit_to_id[curr_unit]\n else:\n curr_id = self.unit_to_id[self.unknown_symbol]\n sequence_vector[i + 1] = curr_id\n return sequence_vector\n\n\nclass _SequencePostprocessing(torch.nn.Module):\n def __init__(self, metadata: TrainingSetMetadataDict):\n super().__init__()\n self.max_sequence_length = int(metadata[\"max_sequence_length\"])\n self.idx2str = metadata[\"idx2str\"]\n self.unknown_symbol = UNKNOWN_SYMBOL\n self.predictions_key = PREDICTIONS\n self.probabilities_key = PROBABILITIES\n self.probability_key = PROBABILITY\n\n def forward(self, preds: dict[str, torch.Tensor], feature_name: str) -> FeaturePostProcessingOutputDict:\n pred_predictions = output_feature_utils.get_output_feature_tensor(preds, feature_name, self.predictions_key)\n pred_probabilities = output_feature_utils.get_output_feature_tensor(preds, feature_name, self.probabilities_key)\n\n predictions: list[list[str]] = []\n for sequence in pred_predictions:\n sequence_predictions: list[str] = []\n for i in range(self.max_sequence_length):\n unit_id = int(sequence[i].item())\n if unit_id < len(self.idx2str):\n unit_prediction = self.idx2str[unit_id]\n else:\n unit_prediction = self.unknown_symbol\n sequence_predictions.append(unit_prediction)\n predictions.append(sequence_predictions)\n\n probabilities, _ = torch.max(pred_probabilities, dim=-1)\n probability = torch.sum(torch.log(probabilities.clamp(min=1e-10)), dim=-1)\n\n return {\n self.predictions_key: predictions,\n self.probabilities_key: probabilities,\n self.probability_key: probability,\n }\n\n\nclass _SequencePredict(PredictModule):\n def forward(self, inputs: dict[str, torch.Tensor], feature_name: str) -> dict[str, torch.Tensor]:\n logits = output_feature_utils.get_output_feature_tensor(inputs, feature_name, self.logits_key)\n probabilities = torch.softmax(logits, -1)\n predictions = torch.argmax(logits, -1)\n\n # predictions: [batch_size, sequence_length]\n # probabilities: [batch_size, sequence_length, vocab_size]\n # logits: [batch_size, sequence_length, vocab_size]\n return {self.predictions_key: predictions, self.probabilities_key: probabilities, self.logits_key: logits}\n\n\nclass SequenceFeatureMixin(BaseFeatureMixin):\n @staticmethod\n def type():\n return SEQUENCE\n\n @staticmethod\n def cast_column(column, backend):\n return column.astype(str)\n\n @staticmethod\n def get_feature_meta(\n config: ModelConfigDict,\n column,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n is_input_feature: bool,\n ) -> FeatureMetadataDict:\n vocabulary = create_vocabulary(\n column,\n preprocessing_parameters[\"tokenizer\"],\n lowercase=preprocessing_parameters[\"lowercase\"],\n num_most_frequent=preprocessing_parameters[\"most_common\"],\n vocab_file=preprocessing_parameters[\"vocab_file\"],\n unknown_symbol=preprocessing_parameters[\"unknown_symbol\"],\n padding_symbol=preprocessing_parameters[\"padding_symbol\"],\n ngram_size=preprocessing_parameters[\"ngram_size\"],\n processor=backend.df_engine,\n )\n logger.info(\n f\"Max length of feature '{column.name}': {vocabulary.max_sequence_length} (without start and stop symbols)\"\n )\n\n # Use sequence_length if provided, otherwise use max length found in dataset.\n if preprocessing_parameters[\"sequence_length\"] is not None:\n logger.info(\n f\"Setting max length to sequence_length={preprocessing_parameters['sequence_length']} provided in \"\n f\"preprocessing parameters\"\n )\n max_sequence_length = preprocessing_parameters[\"sequence_length\"]\n else:\n max_sequence_length = vocabulary.max_sequence_length\n logger.info(f\"Setting max length using dataset: {max_sequence_length} (including start and stop symbols)\")\n\n # If max_sequence_length is None, then use the max length found in the dataset.\n if (\n preprocessing_parameters[\"max_sequence_length\"] is not None\n and preprocessing_parameters[\"max_sequence_length\"] < max_sequence_length\n ):\n logger.info(\n f\"Truncating max length with max_sequence_length={preprocessing_parameters['max_sequence_length']} \"\n f\"from preprocessing parameters\"\n )\n max_sequence_length = preprocessing_parameters[\"max_sequence_length\"]\n\n logger.info(f\"Max sequence length is {max_sequence_length} for feature '{column.name}'\")\n return {\n \"idx2str\": vocabulary.vocab,\n \"str2idx\": vocabulary.str2idx,\n \"str2freq\": vocabulary.str2freq,\n \"vocab_size\": len(vocabulary.vocab),\n \"max_sequence_length\": max_sequence_length,\n }\n\n @staticmethod\n def feature_data(column, metadata, preprocessing_parameters: PreprocessingConfigDict, backend):\n sequence_data = build_sequence_matrix(\n sequences=column,\n inverse_vocabulary=metadata[\"str2idx\"],\n tokenizer_type=preprocessing_parameters[\"tokenizer\"],\n length_limit=metadata[\"max_sequence_length\"],\n padding_symbol=preprocessing_parameters[\"padding_symbol\"],\n padding=preprocessing_parameters[\"padding\"],\n unknown_symbol=preprocessing_parameters[\"unknown_symbol\"],\n lowercase=preprocessing_parameters[\"lowercase\"],\n tokenizer_vocab_file=preprocessing_parameters[\"vocab_file\"],\n processor=backend.df_engine,\n )\n return sequence_data\n\n @staticmethod\n def add_feature_data(\n feature_config,\n input_df,\n proc_df,\n metadata,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n skip_save_processed_input,\n ):\n sequence_data = SequenceInputFeature.feature_data(\n input_df[feature_config[COLUMN]],\n metadata[feature_config[NAME]],\n preprocessing_parameters,\n backend,\n )\n proc_df[feature_config[PROC_COLUMN]] = sequence_data\n return proc_df\n\n\nclass SequenceInputFeature(SequenceFeatureMixin, InputFeature):\n def __init__(self, input_feature_config: SequenceInputFeatureConfig, encoder_obj=None, **kwargs):\n super().__init__(input_feature_config, **kwargs)\n\n if encoder_obj:\n self.encoder_obj = encoder_obj\n else:\n self.encoder_obj = self.initialize_encoder(input_feature_config.encoder)\n\n def forward(self, inputs: torch.Tensor, mask=None):\n assert isinstance(inputs, torch.Tensor)\n assert inputs.dtype in [torch.int8, inputs.dtype, torch.int16, torch.int32, torch.int64]\n assert len(inputs.shape) == 2\n inputs_exp = inputs.type(torch.int32)\n inputs_mask = torch.not_equal(inputs, SpecialSymbol.PADDING.value)\n lengths = torch.sum(inputs_mask.type(torch.int32), dim=1)\n encoder_output = self.encoder_obj(inputs_exp, mask=inputs_mask)\n encoder_output[LENGTHS] = lengths\n return encoder_output\n\n @property\n def input_dtype(self):\n return torch.int32\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n feature_config.encoder.vocab = feature_metadata[\"idx2str\"]\n feature_config.encoder.vocab_size = len(feature_metadata[\"idx2str\"])\n feature_config.encoder.max_sequence_length = feature_metadata[\"max_sequence_length\"]\n\n @staticmethod\n def get_schema_cls():\n return SequenceInputFeatureConfig\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.encoder_obj.config.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n return self.encoder_obj.output_shape\n\n @staticmethod\n def create_preproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n return _SequencePreprocessing(metadata)\n\n\nclass SequenceOutputFeature(SequenceFeatureMixin, OutputFeature):\n def __init__(\n self,\n output_feature_config: SequenceOutputFeatureConfig | dict,\n output_features: dict[str, OutputFeature],\n **kwargs,\n ):\n super().__init__(output_feature_config, output_features, **kwargs)\n self.decoder_obj = self.initialize_decoder(output_feature_config.decoder)\n self._setup_loss()\n self._setup_metrics()\n\n def logits(self, inputs: dict[str, torch.Tensor], target=None):\n return self.decoder_obj(inputs, target=target)\n\n def create_predict_module(self) -> PredictModule:\n return _SequencePredict()\n\n def get_prediction_set(self):\n return self.decoder_obj.get_prediction_set()\n\n @classmethod\n def get_output_dtype(cls):\n return torch.int32\n\n @property\n def input_shape(self) -> torch.Size:\n # Dummy implementation.\n return torch.Size([1])\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.decoder_obj.config.max_sequence_length])\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n feature_config.decoder.vocab_size = feature_metadata[\"vocab_size\"]\n feature_config.decoder.max_sequence_length = feature_metadata[\"max_sequence_length\"]\n if isinstance(feature_config.loss.class_weights, (list, tuple)):\n if len(feature_config.loss.class_weights) != feature_config.decoder.vocab_size:\n raise ValueError(\n f\"The length of class_weights ({len(feature_config.loss.class_weights)}) is not compatible with \"\n f\"the number of classes ({feature_config.decoder.vocab_size}) for feature {feature_config.column}. \"\n \"Check the metadata JSON file to see the classes \"\n \"and their order and consider there needs to be a weight \"\n \"for the and class too.\"\n )\n\n if isinstance(feature_config.loss.class_weights, dict):\n if feature_metadata[\"str2idx\"].keys() != feature_config.loss.class_weights.keys():\n raise ValueError(\n f\"The class_weights keys ({feature_config.loss.class_weights.keys()}) are not compatible with \"\n f'the classes ({feature_metadata[\"str2idx\"].keys()}) of feature {feature_config.column}. '\n \"Check the metadata JSON file to see the classes \"\n \"and consider there needs to be a weight \"\n \"for the class too.\"\n )\n else:\n class_weights = feature_config.loss.class_weights\n idx2str = feature_metadata[\"idx2str\"]\n class_weights_list = [class_weights[s] for s in idx2str]\n feature_config.loss.class_weights = class_weights_list\n\n if feature_config.loss.class_similarities_temperature > 0:\n if feature_config.loss.class_similarities is not None:\n similarities = feature_config.loss.class_similarities\n temperature = feature_config.loss.class_similarities_temperature\n\n curr_row = 0\n first_row_length = 0\n is_first_row = True\n for row in similarities:\n if is_first_row:\n first_row_length = len(row)\n is_first_row = False\n curr_row += 1\n else:\n curr_row_length = len(row)\n if curr_row_length != first_row_length:\n raise ValueError(\n \"The length of row {} of the class_similarities \"\n \"of {} is {}, different from the length of \"\n \"the first row {}. All rows must have \"\n \"the same length.\".format(\n curr_row, feature_config.column, curr_row_length, first_row_length\n )\n )\n else:\n curr_row += 1\n all_rows_length = first_row_length\n\n if all_rows_length != len(similarities):\n raise ValueError(\n f\"The class_similarities matrix of {feature_config.column} has \"\n f\"{len(similarities)} rows and {all_rows_length} columns, \"\n \"their number must be identical.\"\n )\n\n if all_rows_length != feature_config.decoder.vocab_size:\n raise ValueError(\n f\"The size of the class_similarities matrix of {feature_config.column} is \"\n f\"{all_rows_length}, different from the number of classes \"\n f\"({feature_config.decoder.vocab_size}). Check the metadata JSON file to see the classes \"\n \"and their order and \"\n \"consider and class too.\"\n )\n\n similarities = np.array(similarities, dtype=np.float32)\n for i in range(len(similarities)):\n similarities[i, :] = softmax(similarities[i, :], temperature=temperature)\n feature_config.loss.class_similarities = similarities\n else:\n raise ValueError(\n \"class_similarities_temperature > 0, \"\n \"but no class_similarities are provided \"\n f\"for feature {feature_config.column}\"\n )\n\n @staticmethod\n def calculate_overall_stats(predictions, targets, train_set_metadata):\n # TODO(Justin): Add a confusion matrix, see\n # https://github.com/ludwig-ai/ludwig/blob/tf-legacy/ludwig/features/sequence_feature.py#L411\n return {}\n\n def postprocess_predictions(\n self,\n result,\n metadata,\n ):\n predictions_col = f\"{self.feature_name}_{PREDICTIONS}\"\n lengths_col = f\"{self.feature_name}_{LENGTHS}\"\n if predictions_col in result:\n if \"idx2str\" in metadata:\n\n def idx2str(row):\n pred = row[predictions_col]\n length = metadata[\"max_sequence_length\"]\n return [\n metadata[\"idx2str\"][token] if token < len(metadata[\"idx2str\"]) else UNKNOWN_SYMBOL\n for token in [pred[i] for i in range(length)]\n ]\n\n result[predictions_col] = result.apply(idx2str, axis=1)\n\n last_preds_col = f\"{self.feature_name}_{LAST_PREDICTIONS}\"\n if last_preds_col in result:\n if \"idx2str\" in metadata:\n\n def last_idx2str(last_pred):\n if last_pred < len(metadata[\"idx2str\"]):\n return metadata[\"idx2str\"][last_pred]\n return UNKNOWN_SYMBOL\n\n result[last_preds_col] = result[last_preds_col].map(last_idx2str)\n\n probs_col = f\"{self.feature_name}_{PROBABILITIES}\"\n prob_col = f\"{self.feature_name}_{PROBABILITY}\"\n if probs_col in result:\n # currently does not return full probabilties because usually it is huge:\n # dataset x length x classes\n # TODO: add a mechanism for letting the user decide to save it\n result[probs_col] = result[probs_col].map(compute_token_probabilities)\n result[prob_col] = result[probs_col].map(\n partial(\n compute_sequence_probability,\n max_sequence_length=metadata[\"max_sequence_length\"],\n return_log_prob=True,\n )\n )\n\n if lengths_col in result:\n del result[lengths_col]\n\n return result\n\n @staticmethod\n def create_postproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n return _SequencePostprocessing(metadata)\n\n @staticmethod\n def get_schema_cls():\n return SequenceOutputFeatureConfig\n"
},
{
"path": "ludwig/features/set_feature.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nfrom typing import Any\n\nimport numpy as np\nimport torch\n\nfrom ludwig.constants import COLUMN, HIDDEN, LOGITS, NAME, PREDICTIONS, PROBABILITIES, PROC_COLUMN, SET\nfrom ludwig.features.base_feature import BaseFeatureMixin, InputFeature, OutputFeature, PredictModule\nfrom ludwig.features.feature_utils import set_str_to_idx\nfrom ludwig.schema.features.set_feature import SetInputFeatureConfig, SetOutputFeatureConfig\nfrom ludwig.types import (\n FeatureMetadataDict,\n FeaturePostProcessingOutputDict,\n ModelConfigDict,\n PreprocessingConfigDict,\n TrainingSetMetadataDict,\n)\nfrom ludwig.utils import output_feature_utils\nfrom ludwig.utils.strings_utils import create_vocabulary, UNKNOWN_SYMBOL\nfrom ludwig.utils.tokenizers import get_tokenizer_from_registry, TORCHSCRIPT_COMPATIBLE_TOKENIZERS\nfrom ludwig.utils.types import TorchscriptPreprocessingInput\n\nlogger = logging.getLogger(__name__)\n\n\nclass _SetPreprocessing(torch.nn.Module):\n \"\"\"Torchscript-enabled version of preprocessing done by SetFeatureMixin.add_feature_data.\n\n If is_bag is true, forward returns a vector for each sample indicating counts of each token. Else, forward returns a\n multi-hot vector for each sample indicating presence of each token.\n \"\"\"\n\n def __init__(self, metadata: TrainingSetMetadataDict, is_bag: bool = False):\n super().__init__()\n if metadata[\"preprocessing\"][\"tokenizer\"] not in TORCHSCRIPT_COMPATIBLE_TOKENIZERS:\n raise ValueError(\n f\"{metadata['preprocessing']['tokenizer']} is not supported by torchscript. Please use \"\n f\"one of {TORCHSCRIPT_COMPATIBLE_TOKENIZERS}.\"\n )\n\n self.lowercase = metadata[\"preprocessing\"][\"lowercase\"]\n self.tokenizer = get_tokenizer_from_registry(metadata[\"preprocessing\"][\"tokenizer\"])()\n self.vocab_size = metadata[\"vocab_size\"]\n self.unknown_symbol = UNKNOWN_SYMBOL\n self.unit_to_id = metadata[\"str2idx\"]\n self.is_bag = is_bag\n\n def forward(self, v: TorchscriptPreprocessingInput) -> torch.Tensor:\n \"\"\"Takes a list of strings and returns a tensor of counts for each token.\"\"\"\n if not torch.jit.isinstance(v, list[str]):\n raise ValueError(f\"Unsupported input: {v}\")\n\n if self.lowercase:\n sequences = [sequence.lower() for sequence in v]\n else:\n sequences = v\n\n unit_sequences = self.tokenizer(sequences)\n # refines type of unit_sequences from Any to List[List[str]]\n assert torch.jit.isinstance(unit_sequences, list[list[str]]), \"unit_sequences is not a list of lists.\"\n\n set_matrix = torch.zeros(len(unit_sequences), self.vocab_size, dtype=torch.float32)\n for sample_idx, unit_sequence in enumerate(unit_sequences):\n sequence_length = len(unit_sequence)\n for i in range(sequence_length):\n curr_unit = unit_sequence[i]\n if curr_unit in self.unit_to_id:\n curr_id = self.unit_to_id[curr_unit]\n else:\n curr_id = self.unit_to_id[self.unknown_symbol]\n\n if self.is_bag:\n set_matrix[sample_idx][curr_id] += 1\n else:\n set_matrix[sample_idx][curr_id] = 1\n\n return set_matrix\n\n\nclass _SetPostprocessing(torch.nn.Module):\n \"\"\"Torchscript-enabled version of postprocessing done by SetFeatureMixin.add_feature_data.\"\"\"\n\n def __init__(self, metadata: TrainingSetMetadataDict):\n super().__init__()\n self.idx2str = {i: v for i, v in enumerate(metadata[\"idx2str\"])}\n self.predictions_key = PREDICTIONS\n self.probabilities_key = PROBABILITIES\n self.unk = UNKNOWN_SYMBOL\n\n def forward(self, preds: dict[str, torch.Tensor], feature_name: str) -> FeaturePostProcessingOutputDict:\n predictions = output_feature_utils.get_output_feature_tensor(preds, feature_name, self.predictions_key)\n probabilities = output_feature_utils.get_output_feature_tensor(preds, feature_name, self.probabilities_key)\n\n inv_preds: list[list[str]] = []\n filtered_probs: list[torch.Tensor] = []\n for sample_idx, sample in enumerate(predictions):\n sample_preds: list[str] = []\n pos_sample_idxs: list[int] = []\n pos_class_idxs: list[int] = []\n for class_idx, is_positive in enumerate(sample):\n if is_positive == 1:\n sample_preds.append(self.idx2str.get(class_idx, self.unk))\n pos_sample_idxs.append(sample_idx)\n pos_class_idxs.append(class_idx)\n inv_preds.append(sample_preds)\n filtered_probs.append(probabilities[pos_sample_idxs, pos_class_idxs])\n\n return {\n self.predictions_key: inv_preds,\n self.probabilities_key: filtered_probs,\n }\n\n\nclass _SetPredict(PredictModule):\n def __init__(self, threshold):\n super().__init__()\n self.threshold = threshold\n\n def forward(self, inputs: dict[str, torch.Tensor], feature_name: str) -> dict[str, torch.Tensor]:\n logits = output_feature_utils.get_output_feature_tensor(inputs, feature_name, self.logits_key)\n probabilities = torch.sigmoid(logits)\n\n predictions = torch.greater_equal(probabilities, self.threshold)\n predictions = predictions.type(torch.int64)\n\n return {self.predictions_key: predictions, self.probabilities_key: probabilities, self.logits_key: logits}\n\n\nclass SetFeatureMixin(BaseFeatureMixin):\n @staticmethod\n def type():\n return SET\n\n @staticmethod\n def cast_column(column, backend):\n return column.astype(str)\n\n @staticmethod\n def get_feature_meta(\n config: ModelConfigDict,\n column,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n is_input_feature: bool,\n ) -> FeatureMetadataDict:\n vocabulary = create_vocabulary(\n column,\n preprocessing_parameters[\"tokenizer\"],\n num_most_frequent=preprocessing_parameters[\"most_common\"],\n lowercase=preprocessing_parameters[\"lowercase\"],\n add_special_symbols=False,\n processor=backend.df_engine,\n )\n return {\n \"idx2str\": vocabulary.vocab,\n \"str2idx\": vocabulary.str2idx,\n \"str2freq\": vocabulary.str2freq,\n \"vocab_size\": len(vocabulary.str2idx),\n \"max_set_size\": vocabulary.max_sequence_length,\n }\n\n @staticmethod\n def feature_data(column, metadata, preprocessing_parameters: PreprocessingConfigDict, backend):\n def to_dense(x):\n feature_vector = set_str_to_idx(x, metadata[\"str2idx\"], preprocessing_parameters[\"tokenizer\"])\n\n set_vector = np.zeros((len(metadata[\"str2idx\"]),))\n set_vector[feature_vector] = 1\n return set_vector.astype(np.bool_)\n\n return backend.df_engine.map_objects(column, to_dense)\n\n @staticmethod\n def add_feature_data(\n feature_config,\n input_df,\n proc_df,\n metadata,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n skip_save_processed_input,\n ):\n proc_df[feature_config[PROC_COLUMN]] = SetFeatureMixin.feature_data(\n input_df[feature_config[COLUMN]],\n metadata[feature_config[NAME]],\n preprocessing_parameters,\n backend,\n )\n return proc_df\n\n\nclass SetInputFeature(SetFeatureMixin, InputFeature):\n def __init__(self, input_feature_config: SetInputFeatureConfig, encoder_obj=None, **kwargs):\n super().__init__(input_feature_config, **kwargs)\n\n if encoder_obj:\n self.encoder_obj = encoder_obj\n else:\n self.encoder_obj = self.initialize_encoder(input_feature_config.encoder)\n\n def forward(self, inputs):\n assert isinstance(inputs, torch.Tensor)\n assert inputs.dtype in [torch.bool, torch.int64, torch.float32]\n\n encoder_output = self.encoder_obj(inputs)\n\n return encoder_output\n\n @property\n def input_dtype(self):\n return torch.bool\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([len(self.encoder_obj.config.vocab)])\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n feature_config.encoder.vocab = feature_metadata[\"idx2str\"]\n\n @staticmethod\n def get_schema_cls():\n return SetInputFeatureConfig\n\n @property\n def output_shape(self) -> torch.Size:\n return self.encoder_obj.output_shape\n\n @staticmethod\n def create_preproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n return _SetPreprocessing(metadata)\n\n\nclass SetOutputFeature(SetFeatureMixin, OutputFeature):\n def __init__(\n self,\n output_feature_config: SetOutputFeatureConfig | dict,\n output_features: dict[str, OutputFeature],\n **kwargs,\n ):\n self.threshold = output_feature_config.threshold\n super().__init__(output_feature_config, output_features, **kwargs)\n self.decoder_obj = self.initialize_decoder(output_feature_config.decoder)\n self._setup_loss()\n self._setup_metrics()\n\n def logits(self, inputs, **kwargs): # hidden\n hidden = inputs[HIDDEN]\n return self.decoder_obj(hidden)\n\n def metric_kwargs(self) -> dict[str, Any]:\n return {\"threshold\": self.threshold}\n\n def create_predict_module(self) -> PredictModule:\n return _SetPredict(self.threshold)\n\n def get_prediction_set(self):\n return {PREDICTIONS, PROBABILITIES, LOGITS}\n\n @classmethod\n def get_output_dtype(cls):\n return torch.bool\n\n @property\n def input_shape(self) -> torch.Size:\n return self.decoder_obj.input_shape\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.decoder_obj.config.num_classes])\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n feature_config.decoder.num_classes = feature_metadata[\"vocab_size\"]\n if isinstance(feature_config.loss.class_weights, (list, tuple)):\n if len(feature_config.loss.class_weights) != feature_config.decoder.num_classes:\n raise ValueError(\n f\"The length of class_weights ({len(feature_config.loss.class_weights)}) is not compatible with \"\n f\"the number of classes ({feature_config.decoder.num_classes}) for feature {feature_config.name}. \"\n \"Check the metadata JSON file to see the classes \"\n \"and their order and consider there needs to be a weight \"\n \"for the and class too.\"\n )\n\n if isinstance(feature_config.loss.class_weights, dict):\n if feature_metadata[\"str2idx\"].keys() != feature_config.loss.class_weights.keys():\n raise ValueError(\n f\"The class_weights keys ({feature_config.loss.class_weights.keys()}) are not compatible with \"\n f'the classes ({feature_metadata[\"str2idx\"].keys()}) of feature {feature_config.name}. '\n \"Check the metadata JSON file to see the classes \"\n \"and consider there needs to be a weight \"\n \"for the and class too.\"\n )\n else:\n class_weights = feature_config.loss.class_weights\n idx2str = feature_metadata[\"idx2str\"]\n class_weights_list = [class_weights[s] for s in idx2str]\n feature_config.loss.class_weights = class_weights_list\n\n @staticmethod\n def calculate_overall_stats(predictions, targets, train_set_metadata):\n # no overall stats, just return empty dictionary\n return {}\n\n def postprocess_predictions(\n self,\n result,\n metadata,\n ):\n predictions_col = f\"{self.feature_name}_{PREDICTIONS}\"\n if predictions_col in result:\n\n def idx2str(pred_set):\n return [metadata[\"idx2str\"][i] for i, pred in enumerate(pred_set) if pred]\n\n result[predictions_col] = result[predictions_col].map(idx2str)\n\n probabilities_col = f\"{self.feature_name}_{PROBABILITIES}\"\n if probabilities_col in result:\n\n def get_prob(prob_set):\n # Cast to float32 because empty np.array objects are np.float64, causing mismatch errors during saving.\n return np.array([prob for prob in prob_set if prob >= self.threshold], dtype=np.float32)\n\n result[probabilities_col] = result[probabilities_col].map(get_prob)\n\n return result\n\n @staticmethod\n def create_postproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n return _SetPostprocessing(metadata)\n\n @staticmethod\n def get_schema_cls():\n return SetOutputFeatureConfig\n"
},
{
"path": "ludwig/features/text_feature.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nfrom functools import partial\n\nimport numpy as np\nimport torch\nfrom torch import Tensor\nfrom transformers import PreTrainedTokenizer\n\nfrom ludwig.constants import (\n COLUMN,\n IGNORE_INDEX_TOKEN_ID,\n LAST_PREDICTIONS,\n LENGTHS,\n NAME,\n PREDICTIONS,\n PREPROCESSING,\n PROBABILITIES,\n PROBABILITY,\n PROC_COLUMN,\n RESPONSE,\n TEXT,\n)\nfrom ludwig.features.base_feature import BaseFeatureMixin, OutputFeature\nfrom ludwig.features.feature_utils import compute_sequence_probability, compute_token_probabilities\nfrom ludwig.features.sequence_feature import (\n _SequencePostprocessing,\n _SequencePreprocessing,\n SequenceInputFeature,\n SequenceOutputFeature,\n)\nfrom ludwig.modules.metric_registry import get_metric_tensor_input\nfrom ludwig.schema.features.text_feature import TextInputFeatureConfig, TextOutputFeatureConfig\nfrom ludwig.types import FeatureMetadataDict, ModelConfigDict, PreprocessingConfigDict, TrainingSetMetadataDict\nfrom ludwig.utils.math_utils import softmax\nfrom ludwig.utils.strings_utils import (\n build_sequence_matrix,\n create_vocabulary,\n get_tokenizer,\n SpecialSymbol,\n UNKNOWN_SYMBOL,\n Vocabulary,\n)\n\nlogger = logging.getLogger(__name__)\n\n\ndef get_decoded_targets_and_predictions(\n targets: Tensor,\n predictions: dict[str, Tensor],\n tokenizer: PreTrainedTokenizer,\n) -> tuple[list[str], list[str]]:\n \"\"\"Returns the decoded targets and predictions, accounting for IGNORE_INDEX_TOKEN_ID.\"\"\"\n # Ensure targets and predictions are on the same device\n pred_tensor = predictions[PREDICTIONS]\n if targets.device != pred_tensor.device:\n targets = targets.to(pred_tensor.device)\n sanitized_targets = torch.where(targets != IGNORE_INDEX_TOKEN_ID, targets, tokenizer.pad_token_id)\n sanitized_predictions = torch.where(\n targets != IGNORE_INDEX_TOKEN_ID,\n pred_tensor,\n tokenizer.pad_token_id,\n )\n decoded_targets = tokenizer.batch_decode(sanitized_targets, skip_special_tokens=True)\n decoded_predictions = tokenizer.batch_decode(sanitized_predictions, skip_special_tokens=True)\n return decoded_targets, decoded_predictions\n\n\ndef _get_metadata_reconciled_max_sequence_length(\n preprocessing_parameters: dict, vocabulary: Vocabulary\n) -> tuple[int, int]:\n \"\"\"Reconciles the different ways sequence length can be specified in preprocessing parameters.\n\n If the max sequence length is explicitly specified, we use the minimum of the true maximum sequence length and\n the explicitly specified value. If the explicitly specified value is less than the true maximum sequence length, we\n log a warning.\n\n If the max sequence length is not specified, we use the true maximum sequence length.\n\n Returns:\n Tuple(max_sequence_length, sequence_length_99ptile).\n \"\"\"\n # For sequence features with a fixed length specified by `sequence_length`, use this as the max_sequence_length.\n if preprocessing_parameters[\"sequence_length\"] is not None:\n return preprocessing_parameters[\"sequence_length\"], preprocessing_parameters[\"sequence_length\"]\n\n # Max sequence length is explicitly set. Use this as the max_sequence_length.\n if preprocessing_parameters[\"max_sequence_length\"] is not None:\n if preprocessing_parameters[\"max_sequence_length\"] < vocabulary.max_sequence_length:\n logger.warning(\n f\"The max sequence length of the data, {vocabulary.max_sequence_length}, is longer than the max \"\n f\"sequence length set in the config, {preprocessing_parameters['max_sequence_length']}. Note that this \"\n \"will truncate all examples to max_sequence_length=\"\n f\"{preprocessing_parameters['max_sequence_length']}.\"\n )\n return (\n min(vocabulary.max_sequence_length, preprocessing_parameters[\"max_sequence_length\"]),\n min(vocabulary.sequence_length_99ptile, preprocessing_parameters[\"max_sequence_length\"]),\n )\n\n # Max sequence length is None. Use the max sequence length of the data.\n return vocabulary.max_sequence_length, vocabulary.sequence_length_99ptile\n\n\nclass TextFeatureMixin(BaseFeatureMixin):\n @staticmethod\n def type():\n return TEXT\n\n @staticmethod\n def cast_column(column, backend):\n return column.astype(str)\n\n @staticmethod\n def get_feature_meta(\n config: ModelConfigDict,\n column,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n is_input_feature: bool,\n ) -> FeatureMetadataDict:\n \"\"\"Returns all metadata for the given text feature.\n\n Raises:\n ValueError, if the tokenized prompt template is longer than the max sequence length.\n \"\"\"\n prompt_template = config.get(\"prompt\", {}).get(\"template\", \"\")\n vocabulary: Vocabulary = create_vocabulary(\n column,\n tokenizer_type=preprocessing_parameters[\"tokenizer\"],\n num_most_frequent=preprocessing_parameters[\"most_common\"],\n lowercase=preprocessing_parameters[\"lowercase\"],\n vocab_file=preprocessing_parameters[\"vocab_file\"],\n unknown_symbol=preprocessing_parameters[\"unknown_symbol\"],\n padding_symbol=preprocessing_parameters[\"padding_symbol\"],\n pretrained_model_name_or_path=preprocessing_parameters[\"pretrained_model_name_or_path\"],\n ngram_size=preprocessing_parameters[\"ngram_size\"],\n compute_idf=preprocessing_parameters[\"compute_idf\"],\n processor=backend.df_engine,\n prompt_template=prompt_template,\n )\n # Note: The vocabulary's max_sequence_length includes the prompt template, which is merged into the column prior\n # to computing feature metadata.\n logger.info(\n f\"Max length of feature '{column.name}': {vocabulary.max_sequence_length} (without start and stop symbols)\"\n )\n\n max_sequence_length, max_sequence_length_99ptile = _get_metadata_reconciled_max_sequence_length(\n preprocessing_parameters, vocabulary\n )\n\n if is_input_feature and max_sequence_length < vocabulary.prompt_template_num_tokens:\n raise ValueError(\n f\"The input feature's max sequence length ({max_sequence_length}) is shorter than the prompt template \"\n f\"length ({vocabulary.prompt_template_num_tokens}). This will truncate all unique information. \"\n \"Consider making the template shorter or increasing the input feature's max sequence length to a \"\n f\"value >> {vocabulary.prompt_template_num_tokens}.\"\n )\n\n logger.info(f\"Max sequence length is {max_sequence_length} for feature '{column.name}'\")\n\n return {\n \"idx2str\": vocabulary.vocab,\n \"str2idx\": vocabulary.str2idx,\n \"str2freq\": vocabulary.str2freq,\n \"str2idf\": vocabulary.str2idf,\n \"vocab_size\": len(vocabulary.vocab),\n \"max_sequence_length\": max_sequence_length,\n \"max_sequence_length_99ptile\": max_sequence_length_99ptile,\n \"pad_idx\": vocabulary.pad_idx,\n \"padding_symbol\": vocabulary.padding_symbol,\n \"unknown_symbol\": vocabulary.unknown_symbol,\n \"prompt_template_num_tokens\": vocabulary.prompt_template_num_tokens,\n }\n\n @staticmethod\n def feature_data(column, metadata, preprocessing_parameters: PreprocessingConfigDict, backend) -> np.ndarray:\n # TODO(1891): Remove backward compatibility hack once all models have been retrained with Ludwig after\n # https://github.com/ludwig-ai/ludwig/pull/1859.\n prefix = \"\"\n padding_symbol_metadata_key = \"padding_symbol\"\n unknown_symbol_metadata_key = \"unknown_symbol\"\n if \"str2idx\" not in metadata:\n prefix = \"word_\"\n padding_symbol_metadata_key = \"word_pad_symbol\"\n unknown_symbol_metadata_key = \"word_unk_symbol\"\n\n # ensure preprocessing param values match the metadata determined from dataset\n preprocessing_parameters[\"padding_symbol\"] = metadata[padding_symbol_metadata_key]\n preprocessing_parameters[\"unknown_symbol\"] = metadata[unknown_symbol_metadata_key]\n if preprocessing_parameters[\"fill_value\"] == UNKNOWN_SYMBOL:\n preprocessing_parameters[\"fill_value\"] = preprocessing_parameters[\"unknown_symbol\"]\n if (\n \"computed_fill_value\" in preprocessing_parameters\n and preprocessing_parameters[\"computed_fill_value\"] == UNKNOWN_SYMBOL\n ):\n preprocessing_parameters[\"computed_fill_value\"] = preprocessing_parameters[\"unknown_symbol\"]\n\n sequences = column\n\n return build_sequence_matrix(\n sequences=sequences,\n inverse_vocabulary=metadata[f\"{prefix}str2idx\"],\n tokenizer_type=preprocessing_parameters[f\"{prefix}tokenizer\"],\n length_limit=metadata[f\"{prefix}max_sequence_length\"],\n padding_symbol=metadata[padding_symbol_metadata_key],\n padding=preprocessing_parameters[\"padding\"],\n unknown_symbol=metadata[unknown_symbol_metadata_key],\n lowercase=preprocessing_parameters[\"lowercase\"],\n tokenizer_vocab_file=preprocessing_parameters[f\"{prefix}vocab_file\"],\n pretrained_model_name_or_path=preprocessing_parameters[\"pretrained_model_name_or_path\"],\n processor=backend.df_engine,\n )\n\n @staticmethod\n def add_feature_data(\n feature_config,\n input_df,\n proc_df,\n metadata,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n skip_save_processed_input,\n ):\n proc_df[feature_config[PROC_COLUMN]] = TextFeatureMixin.feature_data(\n input_df[feature_config[COLUMN]],\n metadata[feature_config[NAME]],\n preprocessing_parameters,\n backend,\n )\n return proc_df\n\n\nclass TextInputFeature(TextFeatureMixin, SequenceInputFeature):\n def __init__(self, input_feature_config: TextInputFeatureConfig, encoder_obj=None, **kwargs):\n super().__init__(input_feature_config, encoder_obj=encoder_obj, **kwargs)\n\n def forward(self, inputs, mask=None):\n assert isinstance(inputs, torch.Tensor)\n assert (\n inputs.dtype == torch.int8\n or inputs.dtype == torch.int16\n or inputs.dtype == torch.int32\n or inputs.dtype == torch.int64\n )\n assert len(inputs.shape) == 2\n\n inputs_mask = torch.not_equal(inputs, SpecialSymbol.PADDING.value)\n\n inputs_exp = inputs.type(torch.int32)\n lengths = torch.sum(inputs_mask.type(torch.int32), dim=1)\n encoder_output = self.encoder_obj(inputs_exp, mask=inputs_mask)\n encoder_output[LENGTHS] = lengths\n\n return encoder_output\n\n @property\n def input_dtype(self):\n return torch.int32\n\n @property\n def input_shape(self):\n return torch.Size([self.encoder_obj.config.max_sequence_length])\n\n def update_config_after_module_init(self, feature_config):\n feature_config.encoder = self.encoder_obj.config\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n feature_config.encoder.vocab = feature_metadata[\"idx2str\"]\n feature_config.encoder.vocab_size = len(feature_metadata[\"idx2str\"])\n feature_config.encoder.max_sequence_length = feature_metadata[\"max_sequence_length\"]\n feature_config.encoder.pad_idx = feature_metadata[\"pad_idx\"]\n feature_config.encoder.num_tokens = len(feature_metadata[\"idx2str\"])\n feature_config.encoder.str2freq = feature_metadata[\"str2freq\"]\n feature_config.encoder.str2idf = feature_metadata[\"str2idf\"]\n feature_config.encoder.skip = feature_metadata[PREPROCESSING].get(\"cache_encoder_embeddings\", False)\n\n @staticmethod\n def get_schema_cls():\n return TextInputFeatureConfig\n\n @property\n def output_shape(self) -> torch.Size:\n return self.encoder_obj.output_shape\n\n @staticmethod\n def create_preproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n return _SequencePreprocessing(metadata)\n\n\nclass TextOutputFeature(TextFeatureMixin, SequenceOutputFeature):\n def __init__(\n self,\n output_feature_config: TextOutputFeatureConfig | dict,\n output_features: dict[str, OutputFeature],\n **kwargs,\n ):\n super().__init__(output_feature_config, output_features, **kwargs)\n\n @classmethod\n def get_output_dtype(cls):\n return torch.int32\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.decoder_obj.config.max_sequence_length])\n\n def update_metrics(\n self,\n targets: Tensor,\n predictions: dict[str, Tensor],\n tokenizer: PreTrainedTokenizer | None = None,\n ) -> None:\n \"\"\"Updates metrics with the given targets and predictions.\n\n If decoded_targets and decoded_predictions are provided, as through LLM model types, then additional\n response-based metrics like BLEU and ROUGE are also computed.\n\n Args:\n targets: Tensor with target values for this output feature.\n predictions: Dict of tensors returned by predictions().\n \"\"\"\n if tokenizer is not None:\n # Decode the targets and predictions to compute response-based metrics using the initialized tokenizer.\n decoded_targets, decoded_predictions = get_decoded_targets_and_predictions(targets, predictions, tokenizer)\n\n for metric_name, metric_fn in self._metric_functions.items():\n prediction_key = get_metric_tensor_input(metric_name)\n try:\n if prediction_key == RESPONSE:\n if tokenizer is not None:\n # RESPONSE metrics cannot be computed if decoded texts are not provided.\n # Decoded texts are only provided using the LLM model type.\n if decoded_targets is not None and decoded_predictions is not None:\n # Move metric function to the device of the predictions.\n # For CUDA, it can be computed on any of the GPUs since it uses allgather to collect\n # the results from all GPUs and compute the final metric.\n # We use 'predictions' as the key since it is always present in the predictions dict.\n device = \"cuda\" if predictions[\"predictions\"].is_cuda else \"cpu\"\n metric_fn = metric_fn.to(device)\n if metric_name == \"bleu\":\n # BLEU takes in targets as a list.\n metric_fn.update(decoded_predictions, [decoded_targets])\n else:\n metric_fn.update(decoded_predictions, decoded_targets)\n else:\n metric_fn = metric_fn.to(predictions[prediction_key].device)\n metric_fn.update(predictions[prediction_key].detach(), targets)\n except Exception as e:\n logger.info(f\"Ran into error when calculating metric {metric_name}. Skipping. The error is: {e}\")\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n feature_config.decoder.vocab_size = feature_metadata[\"vocab_size\"]\n feature_config.decoder.max_sequence_length = feature_metadata[\"max_sequence_length\"]\n if isinstance(feature_config.loss.class_weights, (list, tuple)):\n # [0, 0] for UNK and PAD\n feature_config.loss.class_weights = [0, 0] + feature_config.loss.class_weights\n if len(feature_config.loss.class_weights) != feature_config.decoder.vocab_size:\n raise ValueError(\n f\"The length of class_weights ({len(feature_config.loss.class_weights)}) is not compatible with \"\n f\"the number of classes ({feature_config.decoder.vocab_size})\"\n )\n\n if isinstance(feature_config.loss.class_weights, dict):\n if feature_metadata[\"str2idx\"].keys() != feature_config.loss.class_weights.keys():\n raise ValueError(\n f\"The class_weights keys ({feature_config.loss.class_weights.keys()}) are not compatible with \"\n f'the classes ({feature_metadata[\"str2idx\"].keys()}) of feature {feature_config.column}. '\n \"Check the metadata JSON file to see the classes \"\n \"and consider there needs to be a weight \"\n \"for the class too.\"\n )\n else:\n class_weights = feature_config.loss.class_weights\n idx2str = feature_metadata[\"idx2str\"]\n class_weights_list = [class_weights[s] for s in idx2str]\n feature_config.loss.class_weights = class_weights_list\n\n if feature_config.loss.class_similarities_temperature > 0:\n if feature_config.class_similarities:\n distances = feature_config.class_similarities\n temperature = feature_config.loss.class_similarities_temperature\n for i in range(len(distances)):\n distances[i, :] = softmax(distances[i, :], temperature=temperature)\n feature_config.loss.class_similarities = distances\n else:\n raise ValueError(\n \"class_similarities_temperature > 0,\"\n \"but no class similarities are provided \"\n \"for feature {}\".format(feature_config.column)\n )\n\n @staticmethod\n def calculate_overall_stats(\n predictions,\n targets,\n train_set_metadata,\n ):\n return {}\n\n def postprocess_predictions(\n self,\n result,\n metadata,\n ):\n # todo: refactor to reuse SequenceOutputFeature.postprocess_predictions\n predictions_col = f\"{self.feature_name}_{PREDICTIONS}\"\n\n tokenizer = None\n if metadata[\"preprocessing\"][\"tokenizer\"] == \"hf_tokenizer\":\n tokenizer = get_tokenizer(\n metadata[\"preprocessing\"][\"tokenizer\"],\n metadata[\"preprocessing\"][\"vocab_file\"],\n metadata[\"preprocessing\"][\"pretrained_model_name_or_path\"],\n )\n\n if predictions_col in result:\n token_col = result[predictions_col]\n\n def idx2str(pred):\n if tokenizer is None:\n return [\n metadata[\"idx2str\"][token] if token < len(metadata[\"idx2str\"]) else UNKNOWN_SYMBOL\n for token in pred\n ]\n # Decode each token ID individually. In transformers 5.x, batch_decode\n # on a 1D array treats it as a single sequence rather than individual tokens.\n return [tokenizer.tokenizer.decode([int(token_id)], skip_special_tokens=True) for token_id in pred]\n\n result[predictions_col] = token_col.map(idx2str)\n\n # Add additional response column that represents the predicted text output\n # as a single string instead of a list of tokens.\n def idx2response(pred):\n if tokenizer is None:\n # This works because we treat each word as a token.\n return \" \".join(\n [\n metadata[\"idx2str\"][token] if token < len(metadata[\"idx2str\"]) else UNKNOWN_SYMBOL\n for token in pred\n ]\n )\n return tokenizer.tokenizer.decode(pred, skip_special_tokens=True)\n\n result[f\"{self.feature_name}_response\"] = token_col.map(idx2response)\n\n last_preds_col = f\"{self.feature_name}_{LAST_PREDICTIONS}\"\n if last_preds_col in result:\n\n def last_idx2str(last_pred):\n if last_pred < len(metadata[\"idx2str\"]):\n return metadata[\"idx2str\"][last_pred]\n return UNKNOWN_SYMBOL\n\n result[last_preds_col] = result[last_preds_col].map(last_idx2str)\n\n probs_col = f\"{self.feature_name}_{PROBABILITIES}\"\n prob_col = f\"{self.feature_name}_{PROBABILITY}\"\n\n # \"Summarizes\" the `result`'s probability-related output:\n # - result[probs_col]:\n # Each row is now a list of \"max\" probabilities. Each element is the probability of the argmax token for\n # the given time step.\n #\n # Note that we intentionally do not return full list of probabilties for each time step because the output\n # of postprocess_predictions is saved to disk and the full probability distribution can be huge,\n # especially for large vocab sizes:\n # dataset_size x sequence_length x vocab_size\n #\n # TODO: Add a mechanism that lets the user save the full probability distribution if they want.\n # - result[prob_col]:\n # Each row is the overall probability of the sequence. This is the product of the max probabilities over\n # all time steps.\n if probs_col in result:\n # result[probs_col]: From PredictModule, each row has a list of size (sequence_length) of a list of\n # probabiltiies of (vocab_size). compute_token_probabilities gets the maximum probability per timestep.\n result[probs_col] = result[probs_col].map(compute_token_probabilities)\n result[prob_col] = result[probs_col].map(\n partial(\n compute_sequence_probability,\n max_sequence_length=metadata[\"max_sequence_length\"],\n return_log_prob=True,\n ),\n )\n\n lengths_col = f\"{self.feature_name}_{LENGTHS}\"\n if lengths_col in result:\n del result[lengths_col]\n\n return result\n\n @staticmethod\n def create_postproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n return _SequencePostprocessing(metadata)\n\n @staticmethod\n def get_schema_cls():\n return TextOutputFeatureConfig\n"
},
{
"path": "ludwig/features/timeseries_feature.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nfrom typing import TYPE_CHECKING\n\nimport numpy as np\nimport torch\n\nfrom ludwig.constants import COLUMN, HIDDEN, LOGITS, NAME, PREDICTIONS, PROC_COLUMN, TIMESERIES\nfrom ludwig.features.base_feature import BaseFeatureMixin, OutputFeature, PredictModule\nfrom ludwig.features.sequence_feature import SequenceInputFeature\nfrom ludwig.features.vector_feature import _VectorPostprocessing, _VectorPredict\nfrom ludwig.schema.features.timeseries_feature import TimeseriesInputFeatureConfig, TimeseriesOutputFeatureConfig\nfrom ludwig.types import FeatureMetadataDict, ModelConfigDict, PreprocessingConfigDict, TrainingSetMetadataDict\nfrom ludwig.utils.tokenizers import get_tokenizer_from_registry, TORCHSCRIPT_COMPATIBLE_TOKENIZERS\nfrom ludwig.utils.types import Series, TorchscriptPreprocessingInput\n\nif TYPE_CHECKING:\n from ludwig.backend.base import Backend\n\nlogger = logging.getLogger(__name__)\n\n\ndef create_time_delay_embedding(\n series: Series, window_size: int, horizon: int, padding_value: int, backend: \"Backend\"\n) -> Series:\n \"\"\"Time delay embedding from:\n\n https://towardsdatascience.com/machine-learning-for-forecasting-transformations-and-feature-extraction-bbbea9de0ac2\n\n Args:\n series: Column-major timeseries data.\n window_size: Size of the lookback sliding window for timeseries inputs.\n horizon: Size of the forward-looking horizon for timeseries outputs.\n padding_value: Value to pad out the window when there is not enough data around the observation.\n\n Returns:\n A column of timeseries window arrays in row-major format for training.\n \"\"\"\n # Replace default fill value of \"\" with nan as we will be assuming numeric values here\n series = series.replace(\"\", np.nan)\n\n # Create the list of shifts we want to perform over the series.\n # For backwards looking shifts, we want to include the current element, while for forward looking shifts we do not.\n # Example:\n # window_size=3, horizon=0 --> shift_offsets=[2, 1, 0]\n # window_size=0, horizon=2 --> shift_offsets=[-1, -2]\n shift_offsets = list(range(window_size - 1, -(horizon + 1), -1))\n shifts = [series.shift(i) for i in shift_offsets]\n df = backend.df_engine.df_lib.concat(shifts, axis=1)\n df.columns = [f\"__tmp_column_{j}\" for j in shift_offsets]\n return df.apply(lambda x: np.nan_to_num(np.array(x.tolist()).astype(np.float32), nan=padding_value), axis=1)\n\n\nclass _TimeseriesPreprocessing(torch.nn.Module):\n \"\"\"Torchscript-enabled version of preprocessing done by TimeseriesFeatureMixin.add_feature_data.\"\"\"\n\n def __init__(self, metadata: TrainingSetMetadataDict):\n super().__init__()\n if metadata[\"preprocessing\"][\"tokenizer\"] not in TORCHSCRIPT_COMPATIBLE_TOKENIZERS:\n raise ValueError(\n f\"{metadata['preprocessing']['tokenizer']} is not supported by torchscript. Please use \"\n f\"one of {TORCHSCRIPT_COMPATIBLE_TOKENIZERS}.\"\n )\n self.tokenizer = get_tokenizer_from_registry(metadata[\"preprocessing\"][\"tokenizer\"])()\n self.padding = metadata[\"preprocessing\"][\"padding\"]\n self.padding_value = float(metadata[\"preprocessing\"][\"padding_value\"])\n self.max_timeseries_length = int(metadata[\"max_timeseries_length\"])\n self.computed_fill_value = metadata[\"preprocessing\"][\"computed_fill_value\"]\n\n def _process_str_sequence(self, sequence: list[str], limit: int) -> torch.Tensor:\n float_sequence = [float(s) for s in sequence[:limit]]\n return torch.tensor(float_sequence)\n\n def _nan_to_fill_value(self, v: torch.Tensor) -> torch.Tensor:\n if v.isnan().any():\n tokenized_fill_value = self.tokenizer(self.computed_fill_value)\n # refines type of sequences from Any to List[str]\n assert torch.jit.isinstance(tokenized_fill_value, list[str])\n return self._process_str_sequence(tokenized_fill_value, self.max_timeseries_length)\n return v\n\n def forward_list_of_tensors(self, v: list[torch.Tensor]) -> torch.Tensor:\n v = [self._nan_to_fill_value(v_i) for v_i in v]\n\n if self.padding == \"right\":\n timeseries_matrix = torch.nn.utils.rnn.pad_sequence(v, batch_first=True, padding_value=self.padding_value)\n timeseries_matrix = timeseries_matrix[:, : self.max_timeseries_length]\n else:\n reversed_timeseries = [torch.flip(v_i[: self.max_timeseries_length], dims=(0,)) for v_i in v]\n reversed_timeseries_padded = torch.nn.utils.rnn.pad_sequence(\n reversed_timeseries, batch_first=True, padding_value=self.padding_value\n )\n timeseries_matrix = torch.flip(reversed_timeseries_padded, dims=(1,))\n return timeseries_matrix\n\n def forward_list_of_strs(self, v: list[str]) -> torch.Tensor:\n v = [self.computed_fill_value if s == \"nan\" else s for s in v]\n\n sequences = self.tokenizer(v)\n # refines type of sequences from Any to List[List[str]]\n assert torch.jit.isinstance(sequences, list[list[str]]), \"sequences is not a list of lists.\"\n\n timeseries_matrix = torch.full(\n [len(sequences), self.max_timeseries_length], self.padding_value, dtype=torch.float32\n )\n for sample_idx, str_sequence in enumerate(sequences):\n limit = min(len(str_sequence), self.max_timeseries_length)\n float_sequence = self._process_str_sequence(str_sequence, limit)\n if self.padding == \"right\":\n timeseries_matrix[sample_idx][:limit] = float_sequence\n else: # if self.padding == 'left\n timeseries_matrix[sample_idx][self.max_timeseries_length - limit :] = float_sequence\n return timeseries_matrix\n\n def forward(self, v: TorchscriptPreprocessingInput) -> torch.Tensor:\n \"\"\"Takes a list of float values and creates a padded torch.Tensor.\"\"\"\n if torch.jit.isinstance(v, list[torch.Tensor]):\n return self.forward_list_of_tensors(v)\n if torch.jit.isinstance(v, list[str]):\n return self.forward_list_of_strs(v)\n raise ValueError(f\"Unsupported input: {v}\")\n\n\nclass TimeseriesFeatureMixin(BaseFeatureMixin):\n @staticmethod\n def type():\n return TIMESERIES\n\n @staticmethod\n def cast_column(column, backend):\n return column\n\n @staticmethod\n def get_feature_meta(\n config: ModelConfigDict,\n column,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n is_input_feature: bool,\n ) -> FeatureMetadataDict:\n window_size = preprocessing_parameters.get(\"window_size\", 0) or preprocessing_parameters.get(\"horizon\", 0)\n if window_size > 0:\n # Column-major data\n return {\"max_timeseries_length\": window_size}\n\n column = column.astype(str)\n tokenizer = get_tokenizer_from_registry(preprocessing_parameters[\"tokenizer\"])()\n max_length = 0\n for timeseries in column:\n processed_line = tokenizer(timeseries)\n max_length = max(max_length, len(processed_line))\n max_length = min(preprocessing_parameters[\"timeseries_length_limit\"], max_length)\n\n return {\"max_timeseries_length\": max_length}\n\n @staticmethod\n def build_matrix(timeseries, tokenizer_name, length_limit, padding_value, padding, backend):\n tokenizer = get_tokenizer_from_registry(tokenizer_name)()\n\n ts_vectors = backend.df_engine.map_objects(\n timeseries, lambda ts: np.nan_to_num(np.array(tokenizer(ts)).astype(np.float32), nan=padding_value)\n )\n\n max_length = backend.df_engine.compute(ts_vectors.map(len).max())\n if max_length < length_limit:\n logger.debug(f\"max length of {tokenizer_name}: {max_length} < limit: {length_limit}\")\n max_length = length_limit\n\n def pad(vector):\n padded = np.full((max_length,), padding_value, dtype=np.float32)\n limit = min(vector.shape[0], max_length)\n if padding == \"right\":\n padded[:limit] = vector[:limit]\n else: # if padding == 'left\n padded[max_length - limit :] = vector[:limit]\n return padded\n\n return backend.df_engine.map_objects(ts_vectors, pad)\n\n @staticmethod\n def feature_data(column, metadata, preprocessing_parameters: PreprocessingConfigDict, backend):\n padding_value = preprocessing_parameters[\"padding_value\"]\n\n window_size = preprocessing_parameters.get(\"window_size\", 0)\n horizon = preprocessing_parameters.get(\"horizon\", 0)\n if window_size > 0 or horizon > 0:\n # Column-major data. Convert the column into the row-major embedding\n return create_time_delay_embedding(column, window_size, horizon, padding_value, backend)\n\n timeseries_data = TimeseriesFeatureMixin.build_matrix(\n column,\n preprocessing_parameters[\"tokenizer\"],\n metadata[\"max_timeseries_length\"],\n padding_value,\n preprocessing_parameters[\"padding\"],\n backend,\n )\n return timeseries_data\n\n @staticmethod\n def add_feature_data(\n feature_config,\n input_df,\n proc_df,\n metadata,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n skip_save_processed_input,\n ):\n proc_df[feature_config[PROC_COLUMN]] = TimeseriesFeatureMixin.feature_data(\n input_df[feature_config[COLUMN]].astype(str),\n metadata[feature_config[NAME]],\n preprocessing_parameters,\n backend,\n )\n return proc_df\n\n\nclass TimeseriesInputFeature(TimeseriesFeatureMixin, SequenceInputFeature):\n def __init__(self, input_feature_config: TimeseriesInputFeatureConfig, encoder_obj=None, **kwargs):\n # add required sequence encoder parameters for time series\n input_feature_config.encoder.embedding_size = 1\n input_feature_config.encoder.should_embed = False\n\n # SequenceInputFeauture's constructor initializes the encoder.\n super().__init__(input_feature_config, encoder_obj=encoder_obj, **kwargs)\n\n def forward(self, inputs, mask=None):\n assert isinstance(inputs, torch.Tensor)\n assert inputs.dtype in [torch.float16, torch.float32, torch.float64]\n assert len(inputs.shape) == 2\n\n inputs_exp = inputs.type(torch.float32)\n encoder_output = self.encoder_obj(inputs_exp, mask=mask)\n\n return encoder_output\n\n @property\n def input_shape(self) -> torch.Size:\n return self.encoder_obj.input_shape\n\n @property\n def input_dtype(self):\n return torch.float32\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n feature_config.encoder.input_size = feature_metadata[\"max_timeseries_length\"]\n feature_config.encoder.max_sequence_length = feature_metadata[\"max_timeseries_length\"]\n\n @staticmethod\n def get_schema_cls():\n return TimeseriesInputFeatureConfig\n\n @staticmethod\n def create_preproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n return _TimeseriesPreprocessing(metadata)\n\n\nclass TimeseriesOutputFeature(TimeseriesFeatureMixin, OutputFeature):\n def __init__(\n self,\n output_feature_config: TimeseriesOutputFeatureConfig | dict,\n output_features: dict[str, OutputFeature],\n **kwargs,\n ):\n self.horizon = output_feature_config.horizon\n super().__init__(output_feature_config, output_features, **kwargs)\n output_feature_config.decoder.output_size = self.horizon\n\n self.decoder_obj = self.initialize_decoder(output_feature_config.decoder)\n self._setup_loss()\n self._setup_metrics()\n\n def logits(self, inputs, **kwargs): # hidden\n hidden = inputs[HIDDEN]\n return self.decoder_obj(hidden)\n\n def loss_kwargs(self):\n return self.loss.to_dict()\n\n def metric_kwargs(self):\n return dict(num_outputs=self.output_shape[0])\n\n def create_predict_module(self) -> PredictModule:\n return _VectorPredict()\n\n def get_prediction_set(self):\n return {PREDICTIONS, LOGITS}\n\n @classmethod\n def get_output_dtype(cls):\n return torch.float32\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.horizon])\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.input_size])\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n feature_config.horizon = feature_metadata[\"max_timeseries_length\"]\n\n @staticmethod\n def calculate_overall_stats(predictions, targets, train_set_metadata):\n # no overall stats, just return empty dictionary\n return {}\n\n def postprocess_predictions(\n self,\n result,\n metadata,\n ):\n predictions_col = f\"{self.feature_name}_{PREDICTIONS}\"\n if predictions_col in result:\n result[predictions_col] = result[predictions_col].map(lambda pred: pred.tolist())\n return result\n\n @staticmethod\n def create_postproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n return _VectorPostprocessing()\n\n @staticmethod\n def get_schema_cls():\n return TimeseriesOutputFeatureConfig\n"
},
{
"path": "ludwig/features/vector_feature.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\n\nimport numpy as np\nimport torch\n\nfrom ludwig.constants import COLUMN, HIDDEN, LOGITS, NAME, PREDICTIONS, PROC_COLUMN, VECTOR\nfrom ludwig.features.base_feature import InputFeature, OutputFeature, PredictModule\nfrom ludwig.schema.features.vector_feature import VectorInputFeatureConfig, VectorOutputFeatureConfig\nfrom ludwig.types import (\n FeatureMetadataDict,\n FeaturePostProcessingOutputDict,\n ModelConfigDict,\n PreprocessingConfigDict,\n TrainingSetMetadataDict,\n)\nfrom ludwig.utils import output_feature_utils\nfrom ludwig.utils.types import TorchscriptPreprocessingInput\n\nlogger = logging.getLogger(__name__)\n\n\nclass _VectorPreprocessing(torch.nn.Module):\n def forward(self, v: TorchscriptPreprocessingInput) -> torch.Tensor:\n if torch.jit.isinstance(v, torch.Tensor):\n out = v\n elif torch.jit.isinstance(v, list[torch.Tensor]):\n out = torch.stack(v)\n elif torch.jit.isinstance(v, list[str]):\n vectors = []\n for sample in v:\n vector = torch.tensor([float(x) for x in sample.split()], dtype=torch.float32)\n vectors.append(vector)\n out = torch.stack(vectors)\n else:\n raise ValueError(f\"Unsupported input: {v}\")\n\n if out.isnan().any():\n raise ValueError(\"Scripted NaN handling not implemented for Vector feature\")\n return out\n\n\nclass _VectorPostprocessing(torch.nn.Module):\n def __init__(self):\n super().__init__()\n self.predictions_key = PREDICTIONS\n self.logits_key = LOGITS\n\n def forward(self, preds: dict[str, torch.Tensor], feature_name: str) -> FeaturePostProcessingOutputDict:\n predictions = output_feature_utils.get_output_feature_tensor(preds, feature_name, self.predictions_key)\n logits = output_feature_utils.get_output_feature_tensor(preds, feature_name, self.logits_key)\n\n return {self.predictions_key: predictions, self.logits_key: logits}\n\n\nclass _VectorPredict(PredictModule):\n def forward(self, inputs: dict[str, torch.Tensor], feature_name: str) -> dict[str, torch.Tensor]:\n logits = output_feature_utils.get_output_feature_tensor(inputs, feature_name, self.logits_key)\n\n return {self.predictions_key: logits, self.logits_key: logits}\n\n\nclass VectorFeatureMixin:\n @staticmethod\n def type():\n return VECTOR\n\n @staticmethod\n def cast_column(column, backend):\n return column\n\n @staticmethod\n def get_feature_meta(\n config: ModelConfigDict,\n column,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n is_input_feature: bool,\n ) -> FeatureMetadataDict:\n return {\"preprocessing\": preprocessing_parameters}\n\n @staticmethod\n def add_feature_data(\n feature_config,\n input_df,\n proc_df,\n metadata,\n preprocessing_parameters: PreprocessingConfigDict,\n backend,\n skip_save_processed_input,\n ):\n \"\"\"Expects all the vectors to be of the same size.\n\n The vectors need to be whitespace delimited strings. Missing values are not handled.\n \"\"\"\n if len(input_df[feature_config[COLUMN]]) == 0:\n raise ValueError(\"There are no vectors in the dataset provided\")\n\n # Convert the string of features into a numpy array\n try:\n proc_df[feature_config[PROC_COLUMN]] = backend.df_engine.map_objects(\n input_df[feature_config[COLUMN]], lambda x: np.array(x.split(), dtype=np.float32)\n )\n except ValueError:\n logger.error(\n \"Unable to read the vector data. Make sure that all the vectors\"\n \" are of the same size and do not have missing/null values.\"\n )\n raise\n\n # Determine vector size\n vector_size = backend.df_engine.compute(proc_df[feature_config[PROC_COLUMN]].map(len).max())\n vector_size_param = preprocessing_parameters.get(\"vector_size\")\n if vector_size_param is not None:\n # TODO(travis): do we even need a user param for vector size if we're going to auto-infer it in all\n # cases? Is this only useful as a sanity check for the user to make sure their data conforms to\n # expectations?\n if vector_size != vector_size_param:\n raise ValueError(\n \"The user provided value for vector size ({}) does not \"\n \"match the value observed in the data: {}\".format(preprocessing_parameters, vector_size)\n )\n else:\n logger.debug(f\"Detected vector size: {vector_size}\")\n\n metadata[feature_config[NAME]][\"vector_size\"] = vector_size\n return proc_df\n\n\nclass VectorInputFeature(VectorFeatureMixin, InputFeature):\n def __init__(self, input_feature_config: VectorInputFeatureConfig, encoder_obj=None, **kwargs):\n super().__init__(input_feature_config, **kwargs)\n\n # input_feature_config.encoder.input_size = input_feature_config.encoder.vector_size\n if encoder_obj:\n self.encoder_obj = encoder_obj\n else:\n self.encoder_obj = self.initialize_encoder(input_feature_config.encoder)\n\n def forward(self, inputs: torch.Tensor) -> torch.Tensor:\n assert isinstance(inputs, torch.Tensor)\n assert inputs.dtype in [torch.float32, torch.float64]\n assert len(inputs.shape) == 2\n\n inputs_encoded = self.encoder_obj(inputs)\n\n return inputs_encoded\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.encoder_obj.config.input_size])\n\n @property\n def output_shape(self) -> torch.Size:\n return self.encoder_obj.output_shape\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n feature_config.encoder.input_size = feature_metadata[\"vector_size\"]\n\n @staticmethod\n def create_preproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n return _VectorPreprocessing()\n\n @staticmethod\n def get_schema_cls():\n return VectorInputFeatureConfig\n\n\nclass VectorOutputFeature(VectorFeatureMixin, OutputFeature):\n def __init__(\n self,\n output_feature_config: VectorOutputFeatureConfig | dict,\n output_features: dict[str, OutputFeature],\n **kwargs,\n ):\n self.vector_size = output_feature_config.vector_size\n super().__init__(output_feature_config, output_features, **kwargs)\n output_feature_config.decoder.output_size = self.vector_size\n\n self.decoder_obj = self.initialize_decoder(output_feature_config.decoder)\n self._setup_loss()\n self._setup_metrics()\n\n def logits(self, inputs, **kwargs): # hidden\n hidden = inputs[HIDDEN]\n return self.decoder_obj(hidden)\n\n def metric_kwargs(self):\n return dict(num_outputs=self.output_shape[0])\n\n def create_predict_module(self) -> PredictModule:\n return _VectorPredict()\n\n def get_prediction_set(self):\n return {PREDICTIONS, LOGITS}\n\n @classmethod\n def get_output_dtype(cls):\n return torch.float32\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.vector_size])\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.input_size])\n\n @staticmethod\n def update_config_with_metadata(feature_config, feature_metadata, *args, **kwargs):\n feature_config.vector_size = feature_metadata[\"vector_size\"]\n\n @staticmethod\n def calculate_overall_stats(predictions, targets, train_set_metadata):\n # no overall stats, just return empty dictionary\n return {}\n\n def postprocess_predictions(\n self,\n result,\n metadata,\n ):\n predictions_col = f\"{self.feature_name}_{PREDICTIONS}\"\n if predictions_col in result:\n result[predictions_col] = result[predictions_col].map(lambda pred: pred.tolist())\n return result\n\n @staticmethod\n def create_postproc_module(metadata: TrainingSetMetadataDict) -> torch.nn.Module:\n return _VectorPostprocessing()\n\n @staticmethod\n def get_schema_cls():\n return VectorOutputFeatureConfig\n"
},
{
"path": "ludwig/forecast.py",
"content": "import argparse\nimport logging\nimport sys\n\nimport pandas as pd\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.backend import ALL_BACKENDS, Backend, initialize_backend\nfrom ludwig.callbacks import Callback\nfrom ludwig.contrib import add_contrib_callback_args\nfrom ludwig.globals import LUDWIG_VERSION\nfrom ludwig.utils.print_utils import get_logging_level_registry, print_ludwig\n\nlogger = logging.getLogger(__name__)\n\n\ndef forecast_cli(\n model_path: str,\n dataset: str | dict | pd.DataFrame = None,\n data_format: str | None = None,\n horizon: int = 1,\n output_directory: str | None = None,\n output_format: str = \"parquet\",\n callbacks: list[Callback] = None,\n backend: Backend | str = None,\n logging_level: int = logging.INFO,\n **kwargs,\n) -> None:\n \"\"\"Loads pre-trained model to forecast on the provided dataset.\n\n # Inputs\n\n :param model_path: (str) filepath to pre-trained model.\n :param dataset: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing the entire dataset to be used in the prediction.\n :param data_format: (str, default: `None`) format to interpret data\n sources. Will be inferred automatically if not specified.\n :param horizon: How many samples into the future to forecast.\n :param output_directory: (str, default: `'results'`) the directory that\n will contain the forecasted values.\n :param output_format: (str) format of the output dataset.\n :param callbacks: (list, default: `None`) a list of\n `ludwig.callbacks.Callback` objects that provide hooks into the\n Ludwig pipeline.\n :param backend: (Union[Backend, str]) `Backend` or string name\n of backend to use to execute preprocessing / training steps.\n :param logging_level: (int) Log level that will be sent to stderr.\n\n # Returns\n\n :return: ('None')\n \"\"\"\n model = LudwigModel.load(\n model_path,\n logging_level=logging_level,\n backend=backend,\n callbacks=callbacks,\n )\n model.forecast(\n dataset=dataset,\n data_format=data_format,\n horizon=horizon,\n output_directory=output_directory,\n output_format=output_format,\n )\n\n\ndef cli(sys_argv):\n parser = argparse.ArgumentParser(\n description=\"This script loads a pretrained model and uses it to forecast\",\n prog=\"ludwig forecast\",\n usage=\"%(prog)s [options]\",\n )\n\n parser.add_argument(\n \"-n\", \"--horizon\", help=\"horizon, or number of steps in the future to forecast\", type=int, default=1\n )\n\n # ---------------\n # Data parameters\n # ---------------\n parser.add_argument(\"--dataset\", help=\"input data file path\", required=True)\n parser.add_argument(\n \"--data_format\",\n help=\"format of the input data\",\n default=\"auto\",\n choices=[\n \"auto\",\n \"csv\",\n \"excel\",\n \"feather\",\n \"fwf\",\n \"hdf5\",\n \"html\",\n \"tables\",\n \"json\",\n \"jsonl\",\n \"parquet\",\n \"pickle\",\n \"sas\",\n \"spss\",\n \"stata\",\n \"tsv\",\n ],\n )\n\n # ----------------\n # Model parameters\n # ----------------\n parser.add_argument(\"-m\", \"--model_path\", help=\"model to load\", required=True)\n\n # -------------------------\n # Output results parameters\n # -------------------------\n parser.add_argument(\n \"-od\", \"--output_directory\", type=str, default=\"results\", help=\"directory that contains the results\"\n )\n\n parser.add_argument(\n \"-of\",\n \"--output_format\",\n help=\"format to write the output dataset\",\n default=\"parquet\",\n choices=[\n \"csv\",\n \"parquet\",\n ],\n )\n\n parser.add_argument(\n \"-b\",\n \"--backend\",\n help=\"specifies backend to use for parallel / distributed execution, \" \"defaults to local execution\",\n choices=ALL_BACKENDS,\n )\n\n parser.add_argument(\n \"-l\",\n \"--logging_level\",\n default=\"info\",\n help=\"the level of logging to use\",\n choices=[\"critical\", \"error\", \"warning\", \"info\", \"debug\", \"notset\"],\n )\n\n add_contrib_callback_args(parser)\n args = parser.parse_args(sys_argv)\n\n args.callbacks = args.callbacks or []\n for callback in args.callbacks:\n callback.on_cmdline(\"forecast\", *sys_argv)\n\n args.logging_level = get_logging_level_registry()[args.logging_level]\n logging.getLogger(\"ludwig\").setLevel(args.logging_level)\n global logger\n logger = logging.getLogger(\"ludwig.forecast\")\n\n args.backend = initialize_backend(args.backend)\n if args.backend.is_coordinator():\n print_ludwig(\"Forecast\", LUDWIG_VERSION)\n logger.info(f\"Dataset path: {args.dataset}\")\n logger.info(f\"Model path: {args.model_path}\")\n logger.info(\"\")\n\n forecast_cli(**vars(args))\n\n\nif __name__ == \"__main__\":\n cli(sys.argv[1:])\n"
},
{
"path": "ludwig/globals.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nLUDWIG_VERSION = \"0.11.2\"\n\nMODEL_FILE_NAME = \"model\"\nMODEL_WEIGHTS_FILE_NAME = \"model_weights\"\nMODEL_HYPERPARAMETERS_FILE_NAME = \"model_hyperparameters.json\"\nTRAIN_SET_METADATA_FILE_NAME = \"training_set_metadata.json\"\nTRAINING_PROGRESS_TRACKER_FILE_NAME = \"training_progress.json\"\nTRAINING_CHECKPOINTS_DIR_PATH = \"training_checkpoints\"\n\nTEST_STATISTICS_FILE_NAME = \"test_statistics.json\"\n\nDESCRIPTION_FILE_NAME = \"description.json\"\n\nPREDICTIONS_PARQUET_FILE_NAME = \"predictions.parquet\"\nPREDICTIONS_SHAPES_FILE_NAME = \"predictions.shapes.json\"\n\nTRAINING_PREPROC_FILE_NAME = \"training.hdf5\"\n\nHYPEROPT_STATISTICS_FILE_NAME = \"hyperopt_statistics.json\"\n\nCONFIG_YAML = \"config.yaml\"\n\nDISABLE_PROGRESSBAR = False\n\n\ndef set_disable_progressbar(value):\n global DISABLE_PROGRESSBAR\n DISABLE_PROGRESSBAR = value\n\n\ndef is_progressbar_disabled():\n return DISABLE_PROGRESSBAR\n"
},
{
"path": "ludwig/hyperopt/__init__.py",
"content": ""
},
{
"path": "ludwig/hyperopt/execution.py",
"content": "import contextlib\nimport copy\nimport datetime\nimport glob\nimport json\nimport logging\nimport os\nimport shutil\nimport sys\nimport tempfile\nimport threading\nimport time\nimport traceback\nimport uuid\nfrom collections.abc import Callable\nfrom functools import lru_cache\nfrom inspect import signature\nfrom pathlib import Path\nfrom typing import Any\n\nimport ray\nfrom ray import tune\nfrom ray.tune import ExperimentAnalysis, PlacementGroupFactory, register_trainable, Stopper\nfrom ray.tune.schedulers.resource_changing_scheduler import DistributeResources, ResourceChangingScheduler\nfrom ray.tune.search import BasicVariantGenerator, ConcurrencyLimiter, SEARCH_ALG_IMPORT\nfrom ray.tune.utils import wait_for_gpu\nfrom ray.util.queue import Queue as RayQueue\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.backend import initialize_backend, RAY\nfrom ludwig.backend.ray import initialize_ray\nfrom ludwig.callbacks import Callback\nfrom ludwig.constants import MAXIMIZE, TEST, TRAINER, TRAINING, TYPE, VALIDATION\nfrom ludwig.hyperopt.results import HyperoptResults, TrialResults\nfrom ludwig.hyperopt.search_algos import get_search_algorithm\nfrom ludwig.hyperopt.utils import load_json_values, substitute_parameters\nfrom ludwig.modules.metric_modules import get_best_function\nfrom ludwig.schema.model_types.utils import merge_with_defaults\nfrom ludwig.utils import metric_utils\nfrom ludwig.utils.data_utils import hash_dict, NumpyEncoder\nfrom ludwig.utils.defaults import default_random_seed\nfrom ludwig.utils.fs_utils import has_remote_protocol, safe_move_file\nfrom ludwig.utils.misc_utils import get_from_registry\n\nlogger = logging.getLogger(__name__)\n\n\ndef _patch_bohb_configspace_conversion():\n \"\"\"Monkey-patch TuneBOHB.convert_search_space for ConfigSpace 1.x compatibility.\n\n ConfigSpace 1.x removed the `q` (quantization) parameter from hyperparameter classes.\n Ray Tune's BOHB integration still passes `q=...`, so we patch the converter to drop it.\n \"\"\"\n try:\n # Check if ConfigSpace 1.x (no 'q' parameter)\n import inspect\n import math\n\n import ConfigSpace\n from ray.tune.search.bohb.bohb_search import TuneBOHB\n from ray.tune.search.sample import Categorical, Float, Integer, LogUniform, Normal, Quantized, Uniform\n from ray.tune.search.variant_generator import parse_spec_vars\n from ray.tune.utils import flatten_dict\n\n sig = inspect.signature(ConfigSpace.UniformFloatHyperparameter.__init__)\n if \"q\" in sig.parameters:\n return # Old ConfigSpace, no patching needed\n\n @staticmethod\n def convert_search_space(spec):\n resolved_vars, domain_vars, grid_vars = parse_spec_vars(spec)\n if grid_vars:\n raise ValueError(\n \"Grid search parameters cannot be automatically converted \" \"to a TuneBOHB search space.\"\n )\n spec = flatten_dict(spec, prevent_delimiter=True)\n resolved_vars, domain_vars, grid_vars = parse_spec_vars(spec)\n\n def resolve_value(par, domain):\n quantize = None\n sampler = domain.get_sampler()\n if isinstance(sampler, Quantized):\n quantize = sampler.q\n sampler = sampler.sampler\n\n if isinstance(domain, Float):\n if isinstance(sampler, LogUniform):\n lower = domain.lower\n upper = domain.upper\n if quantize:\n lower = math.ceil(domain.lower / quantize) * quantize\n upper = math.floor(domain.upper / quantize) * quantize\n return ConfigSpace.UniformFloatHyperparameter(par, lower=lower, upper=upper, log=True)\n elif isinstance(sampler, Uniform):\n lower = domain.lower\n upper = domain.upper\n if quantize:\n lower = math.ceil(domain.lower / quantize) * quantize\n upper = math.floor(domain.upper / quantize) * quantize\n return ConfigSpace.UniformFloatHyperparameter(par, lower=lower, upper=upper, log=False)\n elif isinstance(sampler, Normal):\n return ConfigSpace.hyperparameters.NormalFloatHyperparameter(\n par, mu=sampler.mean, sigma=sampler.sd, log=False\n )\n elif isinstance(domain, Integer):\n if isinstance(sampler, LogUniform):\n lower = domain.lower\n upper = domain.upper\n if quantize:\n lower = math.ceil(domain.lower / quantize) * quantize\n upper = math.floor(domain.upper / quantize) * quantize\n else:\n upper -= 1\n return ConfigSpace.UniformIntegerHyperparameter(par, lower=lower, upper=upper, log=True)\n elif isinstance(sampler, Uniform):\n lower = domain.lower\n upper = domain.upper\n if quantize:\n lower = math.ceil(domain.lower / quantize) * quantize\n upper = math.floor(domain.upper / quantize) * quantize\n else:\n upper -= 1\n return ConfigSpace.UniformIntegerHyperparameter(par, lower=lower, upper=upper, log=False)\n elif isinstance(domain, Categorical):\n if isinstance(sampler, Uniform):\n return ConfigSpace.CategoricalHyperparameter(par, choices=domain.categories)\n\n raise ValueError(\n \"TuneBOHB does not support parameters of type \"\n \"`{}` with samplers of type `{}`\".format(type(domain).__name__, type(domain.sampler).__name__)\n )\n\n cs = ConfigSpace.ConfigurationSpace()\n for path, domain in domain_vars:\n par = \"/\".join(str(p) for p in path)\n value = resolve_value(par, domain)\n cs.add_hyperparameter(value)\n return cs\n\n TuneBOHB.convert_search_space = convert_search_space\n logger.info(\"Patched TuneBOHB.convert_search_space for ConfigSpace 1.x compatibility\")\n\n except ImportError:\n pass # BOHB not installed\n\n\n_patch_bohb_configspace_conversion()\n\n\ntry:\n from ludwig.backend.ray import RayBackend\n\n # TODO: refactor this into an interface\n def _is_ray_backend(backend) -> bool:\n if isinstance(backend, str):\n return backend == RAY\n return isinstance(backend, RayBackend)\n\nexcept ImportError as e:\n logger.warning(\n f\"ImportError (execution.py) failed to import RayBackend with error: \\n\\t{e}. \"\n \"The LocalBackend will be used instead. If you want to use the RayBackend, please install ludwig[distributed].\"\n )\n\n class RayBackend:\n pass\n\n def _is_ray_backend(backend) -> bool:\n return False\n\n\ndef identity(x):\n return x\n\n\ndef _get_relative_checkpoints_dir_parts(path: Path):\n return path.parts[-2:]\n\n\n# Follwing disabled at the moment, expect to be re-enabled pending https://github.com/ludwig-ai/ludwig/issues/2039\ndef ray_resource_allocation_function(\n trial_runner: \"trial_runner.TrialRunner\", # noqa\n trial: \"Trial\", # noqa\n result: dict[str, Any],\n scheduler: \"ResourceChangingScheduler\",\n):\n \"\"\"Determine resources to allocate to running trials.\"\"\"\n pgf = DistributeResources(trial_runner, trial, result, scheduler)\n # restore original base trial resources\n\n # create bundles\n if scheduler.base_trial_resources.required_resources.get(\"GPU\", 0):\n bundles = [{\"CPU\": 1, \"GPU\": 1}] * int(pgf.required_resources[\"GPU\"])\n else:\n bundles = [{\"CPU\": 1}] * (int(pgf.required_resources[\"CPU\"] - 0.001))\n # we can't set Trial actor's CPUs to 0 so we just go very low\n bundles = [{\"CPU\": 0.001}] + bundles\n pgf = PlacementGroupFactory(bundles)\n return pgf\n\n\ndef _create_tune_checkpoint(save_path):\n \"\"\"Create a Ray Tune Checkpoint from a model save path.\"\"\"\n\n def ignore_dot_files(src, files):\n return [f for f in files if f.startswith(\".\")]\n\n tmpdir = tempfile.mkdtemp()\n checkpoint_model = os.path.join(tmpdir, \"model\")\n if os.path.exists(save_path):\n copy_id = uuid.uuid4()\n tmp_dst = f\"{checkpoint_model}.{copy_id}.tmp\"\n shutil.copytree(save_path, tmp_dst, ignore=ignore_dot_files)\n try:\n os.rename(tmp_dst, checkpoint_model)\n except Exception:\n shutil.rmtree(tmp_dst)\n\n return tune.Checkpoint.from_directory(tmpdir)\n\n\nclass RayTuneExecutor:\n def __init__(\n self,\n parameters: dict,\n output_feature: str,\n metric: str,\n goal: str,\n split: str,\n search_alg: dict | None = None,\n cpu_resources_per_trial: int = None,\n gpu_resources_per_trial: int = None,\n kubernetes_namespace: str = None,\n time_budget_s: int | float | datetime.timedelta = None,\n max_concurrent_trials: int | None = None,\n num_samples: int = 1,\n scheduler: dict | None = None,\n **kwargs,\n ) -> None:\n if ray is None:\n raise ImportError(\"ray module is not installed. To install it, try running pip install ray\")\n self.output_feature = output_feature\n self.metric = metric\n self.split = split\n initialize_ray()\n self.search_space, self.decode_ctx = self._get_search_space(parameters)\n self.num_samples = num_samples\n self.goal = goal\n self.search_algorithm = get_search_algorithm(search_alg)\n self.scheduler = None if scheduler is None else tune.create_scheduler(scheduler[TYPE], **scheduler)\n self.output_feature = output_feature\n self.metric = metric\n self.split = split\n self.trial_id = 0\n self.cpu_resources_per_trial = cpu_resources_per_trial\n self.gpu_resources_per_trial = gpu_resources_per_trial\n self.kubernetes_namespace = kubernetes_namespace\n self.time_budget_s = time_budget_s\n self.max_concurrent_trials = max_concurrent_trials\n self.sync_config = None\n self.sync_client = None\n # Head node is the node to which all checkpoints are synced if running on a K8s cluster.\n self.head_node_ip = ray.util.get_node_ip_address()\n\n def _get_search_space(self, parameters: dict) -> tuple[dict, dict]:\n \"\"\"Encode search space parameters as JSON with context for decoding.\"\"\"\n config = {}\n ctx = {}\n for param, values in parameters.items():\n # Encode list and dict types as JSON encoded strings to\n # workaround type limitations of the underlying frameworks\n values = self.encode_values(param, values, ctx)\n\n param_search_type = values[\"space\"].lower()\n if hasattr(tune, param_search_type):\n param_search_space = getattr(tune, param_search_type)\n else:\n raise ValueError(f\"'{param_search_type}' is not a supported Ray Tune search space\")\n\n param_search_input_args = {}\n param_search_space_sig = signature(param_search_space)\n for arg in param_search_space_sig.parameters.values():\n if arg.name in values:\n param_search_input_args[arg.name] = values[arg.name]\n else:\n if arg.default is arg.empty:\n raise ValueError(f\"Parameter '{arg}' not defined for {param}\")\n config[param] = param_search_space(**param_search_input_args)\n return config, ctx\n\n @staticmethod\n def encode_values(param: str, values: dict, ctx: dict) -> dict:\n \"\"\"JSON encodes any search spaces whose values are lists / dicts.\n\n Only applies to grid search and choice options. See here for details:\n\n https://docs.ray.io/en/master/tune/api_docs/search_space.html#random-distributions-api\n \"\"\"\n values = values.copy()\n for key in [\"values\", \"categories\"]:\n if key in values and not isinstance(values[key][0], (int, float)):\n values[key] = [json.dumps(v) for v in values[key]]\n ctx[param] = json.loads\n return values\n\n @staticmethod\n def decode_values(config: dict, ctx: dict) -> dict:\n \"\"\"Decode config values with the decode function in the context.\n\n Uses the identity function if no encoding is needed.\n \"\"\"\n return {key: ctx.get(key, identity)(value) for key, value in config.items()}\n\n def _has_metric(self, stats, split):\n if not stats:\n return False\n\n if split is not None:\n if split not in stats:\n return False\n stats = stats[split]\n\n if self.output_feature not in stats:\n return False\n stats = stats[self.output_feature]\n\n if self.metric not in stats:\n return False\n stats = stats[self.metric]\n return len(stats) > 0\n\n def _has_eval_metric(self, stats):\n if stats is None:\n return False\n\n if self.output_feature not in stats:\n return False\n stats = stats[self.output_feature]\n\n for metric_part in self.metric.split(\".\"):\n if not isinstance(stats, dict) or metric_part not in stats:\n return False\n stats = stats[metric_part]\n return isinstance(stats, float)\n\n def get_metric_score(self, train_stats) -> float:\n if self._has_metric(train_stats, VALIDATION):\n logger.info(\"Returning metric score from training (validation) statistics\")\n return self.get_metric_score_from_train_stats(train_stats, VALIDATION)\n elif self._has_metric(train_stats, TRAINING):\n logger.info(\"Returning metric score from training split statistics, \" \"as no validation was given\")\n return self.get_metric_score_from_train_stats(train_stats, TRAINING)\n else:\n raise RuntimeError(\"Unable to obtain metric score from missing training (validation) statistics\")\n\n def get_metric_score_from_eval_stats(self, eval_stats) -> float | list:\n stats = eval_stats[self.output_feature]\n for metric_part in self.metric.split(\".\"):\n if isinstance(stats, dict):\n if metric_part in stats:\n stats = stats[metric_part]\n else:\n raise ValueError(f\"Evaluation statistics do not contain the metric {self.metric}\")\n else:\n raise ValueError(f\"Evaluation statistics do not contain the metric {self.metric}\")\n\n if not isinstance(stats, float):\n raise ValueError(f\"The metric {self.metric} in evaluation statistics is not a numerical value: {stats}\")\n return stats\n\n def get_metric_score_from_train_stats(self, train_stats, select_split=None) -> float:\n select_split = select_split or VALIDATION\n\n # grab the results of the model with highest validation test performance\n train_valiset_stats = train_stats[select_split]\n\n validation_field_result = train_valiset_stats[self.output_feature]\n best_function = get_best_function(self.metric)\n\n # results of the model with highest validation test performance\n epoch_best_validation_metric, best_validation_metric = best_function(\n enumerate(validation_field_result[self.metric]), key=lambda pair: pair[1]\n )\n\n return best_validation_metric\n\n def sort_hyperopt_results(self, hyperopt_results):\n return sorted(\n hyperopt_results, key=lambda hp_res: hp_res.metric_score, reverse=self.hyperopt_sampler.goal == MAXIMIZE\n )\n\n @property\n def _cpu_resources_per_trial_non_none(self):\n return self.cpu_resources_per_trial if self.cpu_resources_per_trial is not None else 1\n\n @property\n def _gpu_resources_per_trial_non_none(self):\n return self.gpu_resources_per_trial if self.gpu_resources_per_trial is not None else 0\n\n def _get_remote_checkpoint_dir(self, trial_dir: Path) -> str | tuple[str, str] | None:\n \"\"\"Get the path to remote checkpoint directory.\"\"\"\n if self.sync_config is None:\n return None\n\n if self.sync_config.upload_dir is not None:\n # Cloud storage sync config\n remote_checkpoint_dir = os.path.join(\n self.sync_config.upload_dir, *_get_relative_checkpoints_dir_parts(trial_dir)\n )\n return remote_checkpoint_dir\n elif self.kubernetes_namespace is not None:\n # Kubernetes sync config. Returns driver node name and path.\n # When running on kubernetes, each trial is rsynced to the node running the main process.\n node_name = self._get_kubernetes_node_address_by_ip()(self.head_node_ip)\n return (node_name, trial_dir)\n else:\n logger.warning(\n \"Checkpoint syncing disabled as syncing is only supported to remote cloud storage or on Kubernetes \"\n \"clusters is supported. To use syncing, set the kubernetes_namespace in the config or use a cloud URI \"\n \"as the output directory.\"\n )\n return None\n\n @lru_cache(maxsize=1)\n def _get_kubernetes_node_address_by_ip(self) -> Callable:\n \"\"\"Returns a method to get the node name by IP address within a K8s cluster.\"\"\"\n assert self.kubernetes_namespace is not None\n from ray.tune.integration.kubernetes import KubernetesSyncer\n\n # Initialized with null local and remote directories as we only need to use get_node_address_by_ip.\n kubernetes_syncer = KubernetesSyncer(None, None)\n\n return kubernetes_syncer.get_node_address_by_ip\n\n # For specified [stopped] trial, remove checkpoint marker on any partial checkpoints\n @staticmethod\n def _remove_partial_checkpoints(trial_path: str):\n marker_paths = glob.glob(os.path.join(glob.escape(trial_path), \"checkpoint_*/.is_checkpoint\"))\n for marker_path in marker_paths:\n chkpt_dir = os.path.dirname(marker_path)\n metadata_file = glob.glob(os.path.join(glob.escape(chkpt_dir), \"*.tune_metadata\"))\n # glob.glob: filenames starting with a dot are special cases\n # that are not matched by '*' and '?' patterns.\n metadata_file += glob.glob(os.path.join(glob.escape(chkpt_dir), \".tune_metadata\"))\n metadata_file = list(set(metadata_file)) # avoid duplication\n if len(metadata_file) < 1:\n # Remove checkpoint marker on incomplete directory\n os.remove(marker_path)\n\n @contextlib.contextmanager\n def _get_best_model_path(self, trial_or_path, analysis: ExperimentAnalysis) -> str:\n # Accept either a Trial object or a path string\n from ray.tune.experiment.trial import Trial\n\n if isinstance(trial_or_path, str):\n trial_path = trial_or_path\n else:\n trial_path = trial_or_path.local_path\n\n remote_checkpoint_dir = self._get_remote_checkpoint_dir(Path(trial_path))\n if remote_checkpoint_dir is not None and self.sync_client is not None:\n self.sync_client.sync_down(remote_checkpoint_dir, trial_path)\n self.sync_client.wait_or_retry()\n self._remove_partial_checkpoints(trial_path) # needed by get_best_checkpoint\n\n # get_best_checkpoint requires a Trial object in Ray 2.x\n if isinstance(trial_or_path, Trial):\n trial = trial_or_path\n else:\n # Try to find the trial by matching its path\n trial = None\n for t in analysis.trials:\n if t.local_path and t.local_path.rstrip(\"/\") == trial_path.rstrip(\"/\"):\n trial = t\n break\n\n try:\n if trial is not None:\n checkpoint = analysis.get_best_checkpoint(trial)\n else:\n checkpoint = None\n except Exception:\n logger.warning(\n f\"Cannot get best model path for {trial_path} due to exception below:\" f\"\\n{traceback.format_exc()}\"\n )\n yield None\n return\n\n if checkpoint is not None:\n with checkpoint.as_directory() as path:\n yield path\n else:\n yield checkpoint\n\n @staticmethod\n def _evaluate_best_model(\n trial,\n trial_path,\n best_model_path,\n dataset,\n data_format,\n skip_save_unprocessed_output,\n skip_save_predictions,\n skip_save_eval_stats,\n gpus,\n gpu_memory_limit,\n allow_parallel_threads,\n backend,\n debug,\n ):\n model_path = os.path.join(best_model_path, \"model\")\n if not os.path.isdir(model_path):\n logger.warning(\n f\"Best model path {model_path} does not exist or is incomplete. \"\n \"This can happen when time budget expires mid-checkpoint. Skipping evaluation.\"\n )\n return\n best_model = LudwigModel.load(\n model_path,\n backend=backend,\n gpus=gpus,\n gpu_memory_limit=gpu_memory_limit,\n allow_parallel_threads=allow_parallel_threads,\n from_checkpoint=True,\n )\n if best_model.config[TRAINER][\"eval_batch_size\"]:\n batch_size = best_model.config[TRAINER][\"eval_batch_size\"]\n else:\n batch_size = best_model.config[TRAINER][\"batch_size\"]\n try:\n eval_stats, _, _ = best_model.evaluate(\n dataset=dataset,\n data_format=data_format,\n batch_size=batch_size,\n output_directory=trial_path,\n skip_save_unprocessed_output=skip_save_unprocessed_output,\n skip_save_predictions=skip_save_predictions,\n skip_save_eval_stats=skip_save_eval_stats,\n collect_predictions=False,\n collect_overall_stats=True,\n return_type=\"dict\",\n debug=debug,\n )\n trial[\"eval_stats\"] = json.dumps(eval_stats, cls=NumpyEncoder)\n except NotImplementedError:\n logger.warning(\n \"Skipping evaluation as the necessary methods are not \"\n \"supported. Full exception below:\\n\"\n f\"{traceback.format_exc()}\"\n )\n\n def _run_experiment(\n self,\n config,\n checkpoint_dir,\n hyperopt_dict,\n decode_ctx,\n is_using_ray_backend=False,\n ):\n # Ray Tune redirects stdout/stderr through a Tee object that may not\n # implement isatty(), which ray.data's progress bar code requires.\n # Patch it to avoid AttributeError.\n for stream in (sys.stdout, sys.stderr):\n if not hasattr(stream, \"isatty\"):\n stream.isatty = lambda: False\n\n for gpu_id in ray.get_gpu_ids():\n # Previous trial may not have freed its memory yet, so wait to avoid OOM\n wait_for_gpu(gpu_id)\n\n # Some config values may be JSON encoded as strings, so decode them here\n config = self.decode_values(config, decode_ctx)\n\n # Remove mlflow injected config parameters: https://github.com/ludwig-ai/ludwig/issues/2288\n if \"mlflow\" in config:\n del config[\"mlflow\"]\n\n trial_id = tune.get_context().get_trial_id()\n trial_dir = Path(tune.get_context().get_trial_dir())\n\n modified_config = substitute_parameters(copy.deepcopy(hyperopt_dict[\"config\"]), config)\n\n modified_config = merge_with_defaults(modified_config)\n\n hyperopt_dict[\"config\"] = modified_config\n hyperopt_dict[\"experiment_name \"] = f'{hyperopt_dict[\"experiment_name\"]}_{trial_id}'\n hyperopt_dict[\"output_directory\"] = str(trial_dir)\n\n tune_executor = self\n if is_using_ray_backend:\n ray_queue = RayQueue(actor_options={\"num_cpus\": 0})\n else:\n ray_queue = None\n\n def report(progress_tracker, save_path=None):\n # The progress tracker's metrics are nested dictionaries of TrainerMetrics: feature_name -> metric_name ->\n # List[TrainerMetric], with one entry per training checkpoint, according to steps_per_checkpoint.\n # We reduce the dictionary of TrainerMetrics to a simple list of floats for interfacing with Ray Tune.\n train_stats = {\n TRAINING: metric_utils.reduce_trainer_metrics_dict(progress_tracker.train_metrics),\n VALIDATION: metric_utils.reduce_trainer_metrics_dict(progress_tracker.validation_metrics),\n TEST: metric_utils.reduce_trainer_metrics_dict(progress_tracker.test_metrics),\n }\n\n metric_score = tune_executor.get_metric_score(train_stats)\n report_kwargs = {\n \"metrics\": {\n \"parameters\": json.dumps(config, cls=NumpyEncoder),\n \"metric_score\": metric_score,\n \"training_stats\": json.dumps(train_stats, cls=NumpyEncoder),\n \"eval_stats\": \"{}\",\n \"trial_id\": tune.get_context().get_trial_id(),\n \"trial_dir\": str(tune.get_context().get_trial_dir()),\n }\n }\n if save_path is not None:\n report_kwargs[\"checkpoint\"] = _create_tune_checkpoint(save_path)\n tune.report(**report_kwargs)\n\n class RayTuneReportCallback(Callback):\n def __init__(self):\n super().__init__()\n self.last_steps = 0\n self.resume_ckpt_dir = None\n\n def _get_remote_checkpoint_dir(self) -> str | tuple[str, str] | None:\n # sync client has to be recreated to avoid issues with serialization\n return tune_executor._get_remote_checkpoint_dir(trial_dir)\n\n def _checkpoint_progress(self, trainer, progress_tracker, save_path) -> None:\n \"\"\"Checkpoints the progress tracker.\"\"\"\n if is_using_ray_backend:\n # Pass the save_path directly through the queue. On single-node clusters,\n # the trial driver and training workers share the same filesystem.\n # For multi-node, the checkpoint should be on shared storage.\n ray_queue.put((progress_tracker, save_path))\n return\n # For non-Ray backend, report metrics + checkpoint together\n report(progress_tracker, save_path=save_path)\n\n def on_train_start(self, model, config: dict[str, Any], config_fp: str | None):\n if is_using_ray_backend and checkpoint_dir:\n # Store the checkpoint directory path for syncing to the trainer worker.\n self.resume_ckpt_dir = checkpoint_dir\n\n def on_trainer_train_setup(self, trainer, save_path, is_coordinator):\n # Check local rank before manipulating files, as otherwise there will be a race condition\n # between multiple workers running on the same node.\n if self.resume_ckpt_dir is not None and trainer.local_rank == 0:\n # Resume from a previous checkpoint by syncing files from the checkpoint\n # directory to the save_path.\n ckpt_path = self.resume_ckpt_dir\n # Attempt an atomic move from the ckpt_path to the save_path\n # This may first require removing the existing save_path\n tmp_path = save_path + \".tmp\"\n if os.path.exists(save_path):\n os.rename(save_path, tmp_path)\n\n try:\n model_path = os.path.join(ckpt_path, \"model\")\n if os.path.exists(model_path):\n safe_move_file(model_path, save_path)\n elif os.path.exists(ckpt_path):\n safe_move_file(ckpt_path, save_path)\n except Exception:\n # Rollback from partial changes. Remove the save_path\n # and move the original save_path back.\n if os.path.exists(save_path):\n shutil.rmtree(save_path)\n if os.path.exists(tmp_path):\n os.rename(tmp_path, save_path)\n raise\n\n # Cleanup the backup save_path as it's no longer needed\n if os.path.exists(tmp_path):\n shutil.rmtree(tmp_path)\n\n # Sync all workers here before continuing to training\n trainer.barrier()\n\n def on_eval_end(self, trainer, progress_tracker, save_path):\n progress_tracker.tune_checkpoint_num += 1\n self.last_steps = progress_tracker.steps\n self._checkpoint_progress(trainer, progress_tracker, save_path)\n\n def on_trainer_train_teardown(self, trainer, progress_tracker, save_path, is_coordinator):\n if is_coordinator and progress_tracker.steps > self.last_steps:\n # Note: Calling tune.report in both on_eval_end() and here can cause multiprocessing issues\n # for some ray samplers if not steps have happened since the last eval.\n self._checkpoint_progress(trainer, progress_tracker, save_path)\n\n callbacks = hyperopt_dict.get(\"callbacks\") or []\n hyperopt_dict[\"callbacks\"] = callbacks + [RayTuneReportCallback()]\n\n # set tune resources\n if is_using_ray_backend:\n resources = tune.get_context().get_trial_resources()\n # check if we are using at least 1 gpu per trial\n use_gpu = bool(self._gpu_resources_per_trial_non_none)\n # get the resources assigned to the current trial\n num_gpus = resources.required_resources.get(\"GPU\", 0)\n num_cpus = resources.required_resources.get(\"CPU\", 1) if num_gpus == 0 else 0\n\n distributed_kwargs = {\n \"num_workers\": int(num_gpus) if use_gpu else 1,\n \"use_gpu\": use_gpu,\n \"resources_per_worker\": {\n \"CPU\": num_cpus,\n \"GPU\": 1 if use_gpu else 0,\n },\n }\n hyperopt_dict[\"backend\"].set_distributed_kwargs(**distributed_kwargs)\n\n logger.debug(f\"Trial distributed kwargs: {distributed_kwargs}\")\n\n stats = []\n thread_error = [None] # Use list to allow mutation from nested function\n\n def _run():\n try:\n train_stats, eval_stats = run_experiment(\n **hyperopt_dict,\n model_resume_path=checkpoint_dir,\n parameters=config,\n )\n stats.append((train_stats, eval_stats))\n except Exception as e:\n thread_error[0] = e\n logger.error(f\"Error in hyperopt trial thread: {e}\")\n\n if is_using_ray_backend:\n # We have to pull the results to the trial actor\n # from worker actors, as the Tune session is running\n # only on the trial actor\n thread = threading.Thread(target=_run)\n thread.daemon = True\n thread.start()\n\n def check_queue():\n qsize = ray_queue.qsize()\n if qsize:\n results = ray_queue.get_nowait_batch(qsize)\n for progress_tracker, save_path in results:\n report(progress_tracker, save_path=save_path)\n\n while thread.is_alive():\n thread.join(timeout=0)\n check_queue()\n time.sleep(0.1)\n thread.join()\n check_queue()\n else:\n # remove threading overhead\n _run()\n\n if thread_error[0] is not None:\n raise RuntimeError(f\"Experiment failed: {thread_error[0]}\") from thread_error[0]\n if not stats:\n raise RuntimeError(\"Experiment did not complete.\")\n train_stats, eval_stats = stats.pop()\n\n metric_score = self.get_metric_score(train_stats)\n tune.report(\n metrics={\n \"parameters\": json.dumps(config, cls=NumpyEncoder),\n \"metric_score\": metric_score,\n \"training_stats\": json.dumps(train_stats, cls=NumpyEncoder),\n \"eval_stats\": json.dumps(eval_stats, cls=NumpyEncoder),\n \"trial_id\": tune.get_context().get_trial_id(),\n \"trial_dir\": str(tune.get_context().get_trial_dir()),\n }\n )\n\n def execute(\n self,\n config,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n data_format=None,\n experiment_name=\"hyperopt\",\n model_name=\"run\",\n resume=None,\n skip_save_training_description=False,\n skip_save_training_statistics=False,\n skip_save_model=False,\n skip_save_progress=False,\n skip_save_log=False,\n skip_save_processed_input=True,\n skip_save_unprocessed_output=False,\n skip_save_predictions=False,\n skip_save_eval_stats=False,\n output_directory=\"results\",\n gpus=None,\n gpu_memory_limit=None,\n allow_parallel_threads=True,\n callbacks=None,\n tune_callbacks=None,\n backend=None,\n random_seed=default_random_seed,\n debug=False,\n hyperopt_log_verbosity=3,\n **kwargs,\n ) -> HyperoptResults:\n if isinstance(dataset, str) and not has_remote_protocol(dataset) and not os.path.isabs(dataset):\n dataset = os.path.abspath(dataset)\n\n # Ray Tune / PyArrow requires absolute paths or URIs for storage_path\n if not has_remote_protocol(output_directory) and not os.path.isabs(output_directory):\n output_directory = os.path.abspath(output_directory)\n\n if isinstance(backend, str):\n backend = initialize_backend(backend)\n\n if gpus is not None:\n raise ValueError(\n \"Parameter `gpus` is not supported when using Ray Tune. \"\n \"Configure GPU resources with Ray and set `gpu_resources_per_trial` in your \"\n \"hyperopt config.\"\n )\n\n if gpu_memory_limit is None and 0 < self._gpu_resources_per_trial_non_none < 1:\n # Enforce fractional GPU utilization\n gpu_memory_limit = self.gpu_resources_per_trial\n\n hyperopt_dict = dict(\n config=config,\n dataset=dataset,\n training_set=training_set,\n validation_set=validation_set,\n test_set=test_set,\n training_set_metadata=training_set_metadata,\n data_format=data_format,\n experiment_name=experiment_name,\n model_name=model_name,\n eval_split=self.split,\n skip_save_training_description=skip_save_training_description,\n skip_save_training_statistics=skip_save_training_statistics,\n skip_save_model=skip_save_model,\n skip_save_progress=skip_save_progress,\n skip_save_log=skip_save_log,\n skip_save_processed_input=skip_save_processed_input,\n skip_save_unprocessed_output=skip_save_unprocessed_output,\n skip_save_predictions=skip_save_predictions,\n skip_save_eval_stats=skip_save_eval_stats,\n output_directory=output_directory,\n gpus=gpus,\n gpu_memory_limit=gpu_memory_limit,\n allow_parallel_threads=allow_parallel_threads,\n callbacks=callbacks,\n backend=backend,\n random_seed=random_seed,\n debug=debug,\n )\n\n mode = \"min\" if self.goal != MAXIMIZE else \"max\"\n metric = \"metric_score\"\n # if random seed not set, use Ludwig seed\n self.search_algorithm.check_for_random_seed(random_seed)\n if self.search_algorithm.search_alg_dict is not None:\n if TYPE not in self.search_algorithm.search_alg_dict:\n candiate_search_algs = [search_alg for search_alg in SEARCH_ALG_IMPORT.keys()]\n logger.warning(\n \"WARNING: search_alg type parameter missing, using 'variant_generator' as default. \"\n f\"These are possible values for the type parameter: {candiate_search_algs}.\"\n )\n search_alg = None\n else:\n search_alg_type = self.search_algorithm.search_alg_dict[TYPE]\n search_alg = tune.create_searcher(\n search_alg_type, metric=metric, mode=mode, **self.search_algorithm.search_alg_dict\n )\n else:\n search_alg = None\n\n if self.max_concurrent_trials:\n assert (\n self.max_concurrent_trials > 0\n ), f\"`max_concurrent_trials` must be greater than 0, got {self.max_concurrent_trials}\"\n if isinstance(search_alg, BasicVariantGenerator) or search_alg is None:\n search_alg = BasicVariantGenerator(max_concurrent=self.max_concurrent_trials)\n elif isinstance(search_alg, ConcurrencyLimiter):\n raise ValueError(\n \"You have specified `max_concurrent_trials`, but the search \"\n \"algorithm is already a `ConcurrencyLimiter`. FIX THIS \"\n \"by setting `max_concurrent_trials=None`.\"\n )\n else:\n search_alg = ConcurrencyLimiter(search_alg, max_concurrent=self.max_concurrent_trials)\n\n resources_per_trial = {\n \"cpu\": self._cpu_resources_per_trial_non_none,\n \"gpu\": self._gpu_resources_per_trial_non_none,\n }\n\n def run_experiment_trial(config, local_hyperopt_dict, checkpoint_dir=None):\n return self._run_experiment(\n config,\n checkpoint_dir,\n local_hyperopt_dict,\n self.decode_ctx,\n _is_ray_backend(backend),\n )\n\n tune_config = {}\n _tune_callbacks = list(tune_callbacks or [])\n for callback in callbacks or []:\n run_experiment_trial, tune_config = callback.prepare_ray_tune(\n run_experiment_trial,\n tune_config,\n _tune_callbacks,\n )\n tune_callbacks = _tune_callbacks\n\n if _is_ray_backend(backend):\n # for now, we do not do distributed training on cpu (until spread scheduling is implemented for Ray Train)\n # but we do want to enable it when GPUs are specified\n resources_per_trial = PlacementGroupFactory(\n [{}] + ([{\"CPU\": 0, \"GPU\": 1}] * self._gpu_resources_per_trial_non_none)\n if self._gpu_resources_per_trial_non_none\n else [{}] + [{\"CPU\": self._cpu_resources_per_trial_non_none}]\n )\n\n if has_remote_protocol(output_directory):\n # In Ray 2.x, remote storage is handled via RunConfig storage_path\n self.sync_config = tune.SyncConfig()\n self.sync_client = None\n # output_directory will be used as storage_path\n elif self.kubernetes_namespace:\n logger.warning(\n \"Kubernetes-specific syncing is no longer supported in Ray 2.x. \"\n \"Use cloud storage (S3, GCS) as the output directory instead.\"\n )\n\n run_experiment_trial_params = tune.with_parameters(run_experiment_trial, local_hyperopt_dict=hyperopt_dict)\n\n @ray.remote\n def _register(name, trainable):\n register_trainable(name, trainable)\n\n ray.get(_register.remote(f\"trainable_func_f{hash_dict(config).decode('ascii')}\", run_experiment_trial_params))\n\n # Note that resume=\"AUTO\" will attempt to resume the experiment if possible, and\n # otherwise will start a new experiment:\n # https://docs.ray.io/en/latest/tune/tutorials/tune-stopping.html\n should_resume = \"AUTO\" if resume is None else resume\n\n # If the output directory is an S3 path and AWS_ENDPOINT_URL is set,\n # configure a custom S3 filesystem for Ray Tune. We use fsspec's s3fs\n # wrapped in PyArrow's FSSpecHandler because PyArrow's native S3 C++\n # client doesn't read AWS_ENDPOINT_URL and its chunked transfer encoding\n # is incompatible with some S3-compatible stores (e.g. MinIO).\n storage_filesystem = None\n if output_directory and str(output_directory).startswith(\"s3://\"):\n endpoint_url = os.environ.get(\"AWS_ENDPOINT_URL\")\n if endpoint_url:\n import pyarrow.fs\n import s3fs\n\n s3 = s3fs.S3FileSystem(\n endpoint_url=endpoint_url,\n key=os.environ.get(\"AWS_ACCESS_KEY_ID\"),\n secret=os.environ.get(\"AWS_SECRET_ACCESS_KEY\"),\n )\n storage_filesystem = pyarrow.fs.PyFileSystem(pyarrow.fs.FSSpecHandler(s3))\n # When storage_filesystem is set, storage_path must be a plain\n # path (bucket/key...), not a URI (s3://bucket/key...).\n output_directory = str(output_directory).removeprefix(\"s3://\")\n\n try:\n analysis = tune.run(\n f\"trainable_func_f{hash_dict(config).decode('ascii')}\",\n name=experiment_name,\n config={\n **self.search_space,\n **tune_config,\n },\n scheduler=self.scheduler,\n search_alg=search_alg,\n num_samples=self.num_samples,\n checkpoint_config=tune.CheckpointConfig(num_to_keep=1),\n max_failures=1, # retry a trial failure once\n resources_per_trial=resources_per_trial,\n time_budget_s=self.time_budget_s,\n sync_config=self.sync_config,\n storage_path=output_directory,\n storage_filesystem=storage_filesystem,\n metric=metric,\n mode=mode,\n trial_name_creator=lambda trial: f\"trial_{trial.trial_id}\",\n trial_dirname_creator=lambda trial: f\"trial_{trial.trial_id}\",\n callbacks=tune_callbacks,\n stop=CallbackStopper(callbacks),\n verbose=hyperopt_log_verbosity,\n resume=should_resume,\n log_to_file=True,\n )\n except Exception as e:\n # Explicitly raise a RuntimeError if an error is encountered during a Ray trial.\n # NOTE: Cascading the exception with \"raise _ from e\" still results in hanging.\n raise RuntimeError(f\"Encountered Ray Tune error: {e}\")\n\n if \"metric_score\" in analysis.results_df.columns:\n ordered_trials = analysis.results_df.sort_values(\"metric_score\", ascending=self.goal != MAXIMIZE)\n\n # Catch nans in edge case where the trial doesn't complete\n temp_ordered_trials = []\n for kwargs in ordered_trials.to_dict(orient=\"records\"):\n for key in [\"parameters\", \"training_stats\", \"eval_stats\"]:\n if isinstance(kwargs[key], float):\n kwargs[key] = {}\n temp_ordered_trials.append(kwargs)\n\n # Trials w/empty eval_stats fields & non-empty training_stats fields ran intermediate\n # tune.report call(s) but were terminated before reporting eval_stats from post-train\n # evaluation (e.g., trial stopped due to time budget or relatively poor performance.)\n # For any such trials, run model evaluation for the best model in that trial & record\n # results in ordered_trials which is returned & is persisted in hyperopt_statistics.json.\n for trial in temp_ordered_trials:\n if trial[\"eval_stats\"] == \"{}\" and trial[\"training_stats\"] != \"{}\":\n # Evaluate the best model on the eval_split, which is validation_set\n if validation_set is not None and validation_set.size > 0:\n trial_path = trial[\"trial_dir\"]\n with self._get_best_model_path(trial_path, analysis) as best_model_path:\n if best_model_path is not None:\n try:\n self._evaluate_best_model(\n trial,\n trial_path,\n best_model_path,\n validation_set,\n data_format,\n skip_save_unprocessed_output,\n skip_save_predictions,\n skip_save_eval_stats,\n gpus,\n gpu_memory_limit,\n allow_parallel_threads,\n backend,\n debug,\n )\n except Exception:\n logger.warning(\n f\"Failed to evaluate best model for trial {trial_path}. \"\n \"This can happen with incomplete checkpoints from early stopping. \"\n f\"Full exception:\\n{traceback.format_exc()}\"\n )\n else:\n logger.warning(\"Skipping evaluation as no model checkpoints were available\")\n else:\n logger.warning(\"Skipping evaluation as no validation set was provided\")\n\n ordered_trials = [TrialResults.from_dict(load_json_values(kwargs)) for kwargs in temp_ordered_trials]\n else:\n logger.warning(\"No trials reported results; check if time budget lower than epoch latency\")\n ordered_trials = []\n\n return HyperoptResults(ordered_trials=ordered_trials, experiment_analysis=analysis)\n\n\nclass CallbackStopper(Stopper):\n \"\"\"Ray Tune Stopper that triggers the entire job to stop if one callback returns True.\"\"\"\n\n def __init__(self, callbacks: list[Callback] | None):\n self.callbacks = callbacks or []\n\n def __call__(self, trial_id, result):\n return False\n\n def stop_all(self):\n for callback in self.callbacks:\n if callback.should_stop_hyperopt():\n return True\n return False\n\n\ndef get_build_hyperopt_executor(executor_type):\n return get_from_registry(executor_type, executor_registry)\n\n\nexecutor_registry = {\"ray\": RayTuneExecutor}\n\n\ndef set_values(params: dict[str, Any], model_dict: dict[str, Any]):\n for key, value in params.items():\n if isinstance(value, dict):\n for sub_key, sub_value in value.items():\n if key not in model_dict:\n model_dict[key] = dict()\n model_dict[key][sub_key] = sub_value\n else:\n model_dict[key] = value\n\n\ndef run_experiment(\n config,\n parameters=None,\n dataset=None,\n training_set=None,\n validation_set=None,\n test_set=None,\n training_set_metadata=None,\n data_format=None,\n experiment_name=\"hyperopt\",\n model_name=\"run\",\n model_resume_path=None,\n eval_split=VALIDATION,\n skip_save_training_description=False,\n skip_save_training_statistics=False,\n skip_save_model=False,\n skip_save_progress=False,\n skip_save_log=False,\n skip_save_processed_input=False,\n skip_save_unprocessed_output=False,\n skip_save_predictions=False,\n skip_save_eval_stats=False,\n output_directory=\"results\",\n gpus=None,\n gpu_memory_limit=None,\n allow_parallel_threads=True,\n callbacks=None,\n backend=None,\n random_seed=default_random_seed,\n debug=False,\n **kwargs,\n):\n for callback in callbacks or []:\n callback.on_hyperopt_trial_start(parameters)\n\n # Collect training and validation losses and metrics\n # & append it to `results`\n model = LudwigModel(\n config=config,\n backend=backend,\n gpus=gpus,\n gpu_memory_limit=gpu_memory_limit,\n allow_parallel_threads=allow_parallel_threads,\n callbacks=callbacks,\n )\n\n eval_stats, train_stats, _, _ = model.experiment(\n dataset=dataset,\n training_set=training_set,\n validation_set=validation_set,\n test_set=test_set,\n training_set_metadata=training_set_metadata,\n data_format=data_format,\n experiment_name=experiment_name,\n model_name=model_name,\n model_resume_path=model_resume_path,\n eval_split=eval_split,\n skip_save_training_description=skip_save_training_description,\n skip_save_training_statistics=skip_save_training_statistics,\n skip_save_model=skip_save_model,\n skip_save_progress=skip_save_progress,\n skip_save_log=skip_save_log,\n skip_save_processed_input=skip_save_processed_input,\n skip_save_unprocessed_output=skip_save_unprocessed_output,\n skip_save_predictions=skip_save_predictions,\n skip_save_eval_stats=skip_save_eval_stats,\n output_directory=output_directory,\n skip_collect_predictions=True,\n skip_collect_overall_stats=False,\n random_seed=random_seed,\n debug=debug,\n )\n\n for callback in callbacks or []:\n callback.on_hyperopt_trial_end(parameters)\n\n return train_stats, eval_stats\n\n\ndef _run_experiment_unary(kwargs):\n \"\"\"Unary function is needed by Fiber to map a list of args.\"\"\"\n return run_experiment(**kwargs)\n"
},
{
"path": "ludwig/hyperopt/results.py",
"content": "# !/usr/bin/env python\n\nfrom dataclasses import dataclass\nfrom typing import Any\n\nfrom dataclasses_json import dataclass_json\n\ntry:\n from ray.tune import ExperimentAnalysis\nexcept ImportError:\n ExperimentAnalysis = Any\n\n\n@dataclass_json\n@dataclass\nclass TrialResults:\n parameters: dict\n metric_score: float\n training_stats: dict\n eval_stats: dict\n\n\n@dataclass\nclass HyperoptResults:\n ordered_trials: list[TrialResults]\n experiment_analysis: ExperimentAnalysis\n"
},
{
"path": "ludwig/hyperopt/run.py",
"content": "import copy\nimport logging\nimport os\nfrom pprint import pformat\n\nimport pandas as pd\nimport yaml\nfrom tabulate import tabulate\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.backend import Backend, initialize_backend, LocalBackend\nfrom ludwig.callbacks import Callback\nfrom ludwig.constants import (\n AUTO,\n COMBINED,\n EXECUTOR,\n GOAL,\n HYPEROPT,\n LOSS,\n MAX_CONCURRENT_TRIALS,\n METRIC,\n NAME,\n OUTPUT_FEATURES,\n PARAMETERS,\n PREPROCESSING,\n SEARCH_ALG,\n SPLIT,\n TEST,\n TRAINING,\n TYPE,\n VALIDATION,\n)\nfrom ludwig.data.split import get_splitter\nfrom ludwig.hyperopt.results import HyperoptResults\nfrom ludwig.hyperopt.utils import (\n log_warning_if_all_grid_type_parameters,\n print_hyperopt_results,\n save_hyperopt_stats,\n should_tune_preprocessing,\n update_hyperopt_params_with_defaults,\n)\nfrom ludwig.schema.model_config import ModelConfig\nfrom ludwig.utils.backward_compatibility import upgrade_config_dict_to_latest_version\nfrom ludwig.utils.dataset_utils import generate_dataset_statistics\nfrom ludwig.utils.defaults import default_random_seed\nfrom ludwig.utils.fs_utils import makedirs, open_file\n\ntry:\n from ray.tune import Callback as TuneCallback\n\n from ludwig.backend.ray import RayBackend\nexcept ImportError:\n TuneCallback = object\n\n class RayBackend:\n pass\n\n\nlogger = logging.getLogger(__name__)\n\n\ndef hyperopt(\n config: str | dict,\n dataset: str | dict | pd.DataFrame = None,\n training_set: str | dict | pd.DataFrame = None,\n validation_set: str | dict | pd.DataFrame = None,\n test_set: str | dict | pd.DataFrame = None,\n training_set_metadata: str | dict = None,\n data_format: str = None,\n experiment_name: str = \"hyperopt\",\n model_name: str = \"run\",\n resume: bool | None = None,\n skip_save_training_description: bool = False,\n skip_save_training_statistics: bool = False,\n skip_save_model: bool = False,\n skip_save_progress: bool = False,\n skip_save_log: bool = False,\n skip_save_processed_input: bool = True,\n skip_save_unprocessed_output: bool = False,\n skip_save_predictions: bool = False,\n skip_save_eval_stats: bool = False,\n skip_save_hyperopt_statistics: bool = False,\n output_directory: str = \"results\",\n gpus: str | int | list[int] = None,\n gpu_memory_limit: float | None = None,\n allow_parallel_threads: bool = True,\n callbacks: list[Callback] = None,\n tune_callbacks: list[TuneCallback] = None,\n backend: Backend | str = None,\n random_seed: int = default_random_seed,\n hyperopt_log_verbosity: int = 3,\n **kwargs,\n) -> HyperoptResults:\n \"\"\"This method performs an hyperparameter optimization.\n\n # Inputs\n\n :param config: (Union[str, dict]) config which defines\n the different parameters of the model, features, preprocessing and\n training. If `str`, filepath to yaml configuration file.\n :param dataset: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing the entire dataset to be used in the experiment.\n If it has a split column, it will be used for splitting (0 for train,\n 1 for validation, 2 for test), otherwise the dataset will be\n randomly split.\n :param training_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing training data.\n :param validation_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing validation data.\n :param test_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing test data.\n :param training_set_metadata: (Union[str, dict], default: `None`)\n metadata JSON file or loaded metadata. Intermediate preprocessed\n structure containing the mappings of the input\n dataset created the first time an input file is used in the same\n directory with the same name and a '.meta.json' extension.\n :param data_format: (str, default: `None`) format to interpret data\n sources. Will be inferred automatically if not specified. Valid\n formats are `'auto'`, `'csv'`, `'df'`, `'dict'`, `'excel'`, `'feather'`,\n `'fwf'`, `'hdf5'` (cache file produced during previous training),\n `'html'` (file containing a single HTML `
`), `'json'`, `'jsonl'`,\n `'parquet'`, `'pickle'` (pickled Pandas DataFrame), `'sas'`, `'spss'`,\n `'stata'`, `'tsv'`.\n :param experiment_name: (str, default: `'experiment'`) name for\n the experiment.\n :param model_name: (str, default: `'run'`) name of the model that is\n being used.\n :param resume: (bool) If true, continue hyperopt from the state of the previous\n run in the output directory with the same experiment name. If false, will create\n new trials, ignoring any previous state, even if they exist in the output_directory.\n By default, will attempt to resume if there is already an existing experiment with\n the same name, and will create new trials if not.\n :param skip_save_training_description: (bool, default: `False`) disables\n saving the description JSON file.\n :param skip_save_training_statistics: (bool, default: `False`) disables\n saving training statistics JSON file.\n :param skip_save_model: (bool, default: `False`) disables\n saving model weights and hyperparameters each time the model\n improves. By default Ludwig saves model weights after each epoch\n the validation metric improves, but if the model is really big\n that can be time consuming. If you do not want to keep\n the weights and just find out what performance a model can get\n with a set of hyperparameters, use this parameter to skip it,\n but the model will not be loadable later on and the returned model\n will have the weights obtained at the end of training, instead of\n the weights of the epoch with the best validation performance.\n :param skip_save_progress: (bool, default: `False`) disables saving\n progress each epoch. By default Ludwig saves weights and stats\n after each epoch for enabling resuming of training, but if\n the model is really big that can be time consuming and will uses\n twice as much space, use this parameter to skip it, but training\n cannot be resumed later on.\n :param skip_save_log: (bool, default: `False`) disables saving\n TensorBoard logs. By default Ludwig saves logs for the TensorBoard,\n but if it is not needed turning it off can slightly increase the\n overall speed.\n :param skip_save_processed_input: (bool, default: `False`) if input\n dataset is provided it is preprocessed and cached by saving an HDF5\n and JSON files to avoid running the preprocessing again. If this\n parameter is `False`, the HDF5 and JSON file are not saved.\n :param skip_save_unprocessed_output: (bool, default: `False`) by default\n predictions and their probabilities are saved in both raw\n unprocessed numpy files containing tensors and as postprocessed\n CSV files (one for each output feature). If this parameter is True,\n only the CSV ones are saved and the numpy ones are skipped.\n :param skip_save_predictions: (bool, default: `False`) skips saving test\n predictions CSV files.\n :param skip_save_eval_stats: (bool, default: `False`) skips saving test\n statistics JSON file.\n :param skip_save_hyperopt_statistics: (bool, default: `False`) skips saving\n hyperopt stats file.\n :param output_directory: (str, default: `'results'`) the directory that\n will contain the training statistics, TensorBoard logs, the saved\n model and the training progress files.\n :param gpus: (list, default: `None`) list of GPUs that are available\n for training.\n :param gpu_memory_limit: (float: default: `None`) maximum memory fraction\n [0, 1] allowed to allocate per GPU device.\n :param allow_parallel_threads: (bool, default: `True`) allow PyTorch\n to use multithreading parallelism to improve performance at\n the cost of determinism.\n :param callbacks: (list, default: `None`) a list of\n `ludwig.callbacks.Callback` objects that provide hooks into the\n Ludwig pipeline.\n :param backend: (Union[Backend, str]) `Backend` or string name\n of backend to use to execute preprocessing / training steps.\n :param random_seed: (int: default: 42) random seed used for weights\n initialization, splits and any other random function.\n :param hyperopt_log_verbosity: (int: default: 3) controls verbosity of\n ray tune log messages. Valid values: 0 = silent, 1 = only status updates,\n 2 = status and brief trial results, 3 = status and detailed trial results.\n\n # Return\n\n :return: (List[dict]) List of results for each trial, ordered by\n descending performance on the target metric.\n \"\"\"\n from ludwig.hyperopt.execution import get_build_hyperopt_executor, RayTuneExecutor\n\n # check if config is a path or a dict\n if isinstance(config, str): # assume path\n with open_file(config, \"r\") as def_file:\n config_dict = yaml.safe_load(def_file)\n else:\n config_dict = config\n\n if HYPEROPT not in config_dict:\n raise ValueError(\"Hyperopt Section not present in config\")\n\n # backwards compatibility\n upgraded_config = upgrade_config_dict_to_latest_version(config_dict)\n\n # Initialize config object\n config_obj = ModelConfig.from_dict(upgraded_config)\n\n # Retain pre-merged config for hyperopt schema generation\n premerged_config = copy.deepcopy(upgraded_config)\n\n # Get full config with defaults\n full_config = config_obj.to_dict() # TODO (Connor): Refactor to use config object\n\n hyperopt_config = full_config[HYPEROPT]\n\n # Explicitly default to a local backend to avoid picking up Ray\n # backend from the environment.\n backend = backend or config_dict.get(\"backend\") or \"local\"\n backend = initialize_backend(backend)\n\n update_hyperopt_params_with_defaults(hyperopt_config)\n\n # Check if all features are grid type parameters and log UserWarning if needed\n log_warning_if_all_grid_type_parameters(hyperopt_config)\n\n # Infer max concurrent trials\n if hyperopt_config[EXECUTOR].get(MAX_CONCURRENT_TRIALS) == AUTO:\n hyperopt_config[EXECUTOR][MAX_CONCURRENT_TRIALS] = backend.max_concurrent_trials(hyperopt_config)\n logger.info(f\"Setting max_concurrent_trials to {hyperopt_config[EXECUTOR][MAX_CONCURRENT_TRIALS]}\")\n\n # Print hyperopt config\n logger.info(\"Hyperopt Config\")\n logger.info(pformat(hyperopt_config, indent=4))\n logger.info(\"\\n\")\n\n search_alg = hyperopt_config[SEARCH_ALG]\n executor = hyperopt_config[EXECUTOR]\n parameters = hyperopt_config[PARAMETERS]\n split = hyperopt_config[SPLIT]\n output_feature = hyperopt_config[\"output_feature\"]\n metric = hyperopt_config[METRIC]\n goal = hyperopt_config[GOAL]\n\n ######################\n # check validity of output_feature / metric/ split combination\n ######################\n splitter = get_splitter(**full_config[PREPROCESSING][\"split\"])\n if split == TRAINING:\n if training_set is None and not splitter.has_split(0):\n raise ValueError(\n 'The data for the specified split for hyperopt \"{}\" '\n \"was not provided, \"\n \"or the split amount specified in the preprocessing section \"\n \"of the config is not greater than 0\".format(split)\n )\n elif split == VALIDATION:\n if validation_set is None and not splitter.has_split(1):\n raise ValueError(\n 'The data for the specified split for hyperopt \"{}\" '\n \"was not provided, \"\n \"or the split amount specified in the preprocessing section \"\n \"of the config is not greater than 0\".format(split)\n )\n elif split == TEST:\n if test_set is None and not splitter.has_split(2):\n raise ValueError(\n 'The data for the specified split for hyperopt \"{}\" '\n \"was not provided, \"\n \"or the split amount specified in the preprocessing section \"\n \"of the config is not greater than 0\".format(split)\n )\n else:\n raise ValueError(\n 'unrecognized hyperopt split \"{}\". ' \"Please provide one of: {}\".format(split, {TRAINING, VALIDATION, TEST})\n )\n if output_feature == COMBINED:\n if metric != LOSS:\n raise ValueError('The only valid metric for \"combined\" output feature is \"loss\"')\n else:\n output_feature_names = {of[NAME] for of in full_config[OUTPUT_FEATURES]}\n if output_feature not in output_feature_names:\n raise ValueError(\n 'The output feature specified for hyperopt \"{}\" '\n \"cannot be found in the config. \"\n 'Available ones are: {} and \"combined\"'.format(output_feature, output_feature_names)\n )\n\n hyperopt_executor = get_build_hyperopt_executor(executor[TYPE])(\n parameters, output_feature, metric, goal, split, search_alg=search_alg, **executor\n )\n\n # Explicitly default to a local backend to avoid picking up Ray\n # backend from the environment.\n backend = backend or config_dict.get(\"backend\") or \"local\"\n backend = initialize_backend(backend)\n if not (\n isinstance(backend, LocalBackend)\n or (isinstance(hyperopt_executor, RayTuneExecutor) and isinstance(backend, RayBackend))\n ):\n raise ValueError(\n \"Hyperopt requires using a `local` backend at this time, or \" \"`ray` backend with `ray` executor.\"\n )\n\n for callback in callbacks or []:\n callback.on_hyperopt_init(experiment_name)\n\n if not should_tune_preprocessing(full_config):\n # preprocessing is not being tuned, so generate it once before starting trials\n for callback in callbacks or []:\n callback.on_hyperopt_preprocessing_start(experiment_name)\n\n model = LudwigModel(\n config=full_config,\n backend=backend,\n gpus=gpus,\n gpu_memory_limit=gpu_memory_limit,\n allow_parallel_threads=allow_parallel_threads,\n callbacks=callbacks,\n )\n\n training_set, validation_set, test_set, training_set_metadata = model.preprocess(\n dataset=dataset,\n training_set=training_set,\n validation_set=validation_set,\n test_set=test_set,\n training_set_metadata=training_set_metadata,\n data_format=data_format,\n skip_save_processed_input=skip_save_processed_input,\n random_seed=random_seed,\n )\n dataset = None\n\n dataset_statistics = generate_dataset_statistics(training_set, validation_set, test_set)\n\n logger.info(\"\\nDataset Statistics\")\n logger.info(tabulate(dataset_statistics, headers=\"firstrow\", tablefmt=\"fancy_grid\"))\n\n for callback in callbacks or []:\n callback.on_hyperopt_preprocessing_end(experiment_name)\n\n for callback in callbacks or []:\n callback.on_hyperopt_start(experiment_name)\n\n hyperopt_results = hyperopt_executor.execute(\n premerged_config,\n dataset=dataset,\n training_set=training_set,\n validation_set=validation_set,\n test_set=test_set,\n training_set_metadata=training_set_metadata,\n data_format=data_format,\n experiment_name=experiment_name,\n model_name=model_name,\n resume=resume,\n skip_save_training_description=skip_save_training_description,\n skip_save_training_statistics=skip_save_training_statistics,\n skip_save_model=skip_save_model,\n skip_save_progress=skip_save_progress,\n skip_save_log=skip_save_log,\n skip_save_processed_input=skip_save_processed_input,\n skip_save_unprocessed_output=skip_save_unprocessed_output,\n skip_save_predictions=skip_save_predictions,\n skip_save_eval_stats=skip_save_eval_stats,\n output_directory=output_directory,\n gpus=gpus,\n gpu_memory_limit=gpu_memory_limit,\n allow_parallel_threads=allow_parallel_threads,\n callbacks=callbacks,\n tune_callbacks=tune_callbacks,\n backend=backend,\n random_seed=random_seed,\n hyperopt_log_verbosity=hyperopt_log_verbosity,\n **kwargs,\n )\n\n if backend.is_coordinator():\n print_hyperopt_results(hyperopt_results)\n\n if not skip_save_hyperopt_statistics:\n with backend.storage.artifacts.use_credentials():\n results_directory = os.path.join(output_directory, experiment_name)\n makedirs(results_directory, exist_ok=True)\n\n hyperopt_stats = {\n \"hyperopt_config\": hyperopt_config,\n \"hyperopt_results\": [t.to_dict() for t in hyperopt_results.ordered_trials],\n }\n\n save_hyperopt_stats(hyperopt_stats, results_directory)\n logger.info(f\"Hyperopt stats saved to: {results_directory}\")\n\n for callback in callbacks or []:\n callback.on_hyperopt_end(experiment_name)\n callback.on_hyperopt_finish(experiment_name)\n\n logger.info(\"Finished hyperopt\")\n\n return hyperopt_results\n"
},
{
"path": "ludwig/hyperopt/search_algos.py",
"content": "import logging\nfrom abc import ABC\nfrom importlib import import_module\n\nfrom ludwig.constants import TYPE\nfrom ludwig.utils.misc_utils import get_from_registry\n\nlogger = logging.getLogger(__name__)\n\n\ndef _is_package_installed(package_name: str, search_algo_name: str) -> bool:\n try:\n import_module(package_name)\n return True\n except ImportError:\n raise ImportError(\n f\"Search algorithm {search_algo_name} requires package {package_name}, however package is not installed.\"\n \" Please refer to Ray Tune documentation for packages required for this search algorithm.\"\n )\n\n\nclass SearchAlgorithm(ABC):\n def __init__(self, search_alg_dict: dict) -> None:\n self.search_alg_dict = search_alg_dict\n self.random_seed_attribute_name = None\n\n def check_for_random_seed(self, ludwig_random_seed: int) -> None:\n if self.random_seed_attribute_name not in self.search_alg_dict:\n self.search_alg_dict[self.random_seed_attribute_name] = ludwig_random_seed\n\n\nclass BasicVariantSA(SearchAlgorithm):\n def __init__(self, search_alg_dict: dict) -> None:\n super().__init__(search_alg_dict)\n self.random_seed_attribute_name = \"random_state\"\n\n\nclass HyperoptSA(SearchAlgorithm):\n def __init__(self, search_alg_dict: dict) -> None:\n _is_package_installed(\"hyperopt\", \"hyperopt\")\n super().__init__(search_alg_dict)\n self.random_seed_attribute_name = \"random_state_seed\"\n\n\nclass BOHBSA(SearchAlgorithm):\n def __init__(self, search_alg_dict: dict) -> None:\n _is_package_installed(\"hpbandster\", \"bohb\")\n _is_package_installed(\"ConfigSpace\", \"bohb\")\n super().__init__(search_alg_dict)\n self.random_seed_attribute_name = \"seed\"\n\n\nclass AxSA(SearchAlgorithm):\n def __init__(self, search_alg_dict: dict) -> None:\n _is_package_installed(\"sqlalchemy\", \"ax\")\n _is_package_installed(\"ax\", \"ax\")\n super().__init__(search_alg_dict)\n\n # override parent method, this search algorithm does not support\n # setting random seed\n def check_for_random_seed(self, ludwig_random_seed: int) -> None:\n pass\n\n\nclass BayesOptSA(SearchAlgorithm):\n def __init__(self, search_alg_dict: dict) -> None:\n _is_package_installed(\"bayes_opt\", \"bayesopt\")\n super().__init__(search_alg_dict)\n self.random_seed_attribute_name = \"random_state\"\n\n\nclass BlendsearchSA(SearchAlgorithm):\n def __init__(self, search_alg_dict: dict) -> None:\n _is_package_installed(\"flaml\", \"blendsearch\")\n super().__init__(search_alg_dict)\n\n # override parent method, this search algorithm does not support\n # setting random seed\n def check_for_random_seed(self, ludwig_random_seed: int) -> None:\n pass\n\n\nclass CFOSA(SearchAlgorithm):\n def __init__(self, search_alg_dict: dict) -> None:\n _is_package_installed(\"flaml\", \"cfo\")\n super().__init__(search_alg_dict)\n self.random_seed_attribute_name = \"seed\"\n\n # override parent method, this search algorithm does not support\n # setting random seed\n def check_for_random_seed(self, ludwig_random_seed: int) -> None:\n pass\n\n\nclass DragonflySA(SearchAlgorithm):\n def __init__(self, search_alg_dict: dict) -> None:\n _is_package_installed(\"dragonfly\", \"dragonfly\")\n super().__init__(search_alg_dict)\n self.random_seed_attribute_name = \"random_state_seed\"\n\n\nclass HEBOSA(SearchAlgorithm):\n def __init__(self, search_alg_dict: dict) -> None:\n _is_package_installed(\"hebo\", \"hebo\")\n super().__init__(search_alg_dict)\n self.random_seed_attribute_name = \"random_state_seed\"\n\n\nclass SkoptSA(SearchAlgorithm):\n def __init__(self, search_alg_dict: dict) -> None:\n _is_package_installed(\"skopt\", \"skopt\")\n super().__init__(search_alg_dict)\n\n # override parent method, this search algorithm does not support\n # setting random seed\n def check_for_random_seed(self, ludwig_random_seed: int) -> None:\n pass\n\n\nclass NevergradSA(SearchAlgorithm):\n def __init__(self, search_alg_dict: dict) -> None:\n _is_package_installed(\"nevergrad\", \"nevergrad\")\n super().__init__(search_alg_dict)\n\n # override parent method, this search algorithm does not support\n # setting random seed\n def check_for_random_seed(self, ludwig_random_seed: int) -> None:\n pass\n\n\nclass OptunaSA(SearchAlgorithm):\n def __init__(self, search_alg_dict: dict) -> None:\n _is_package_installed(\"optuna\", \"optuna\")\n super().__init__(search_alg_dict)\n self.random_seed_attribute_name = \"seed\"\n\n\nclass ZooptSA(SearchAlgorithm):\n def __init__(self, search_alg_dict: dict) -> None:\n _is_package_installed(\"zoopt\", \"zoopt\")\n super().__init__(search_alg_dict)\n\n # override parent method, this search algorithm does not support\n # setting random seed\n def check_for_random_seed(self, ludwig_random_seed: int) -> None:\n pass\n\n\ndef get_search_algorithm(search_algo):\n search_algo_name = search_algo.get(TYPE, None)\n return get_from_registry(search_algo_name, search_algo_registry)(search_algo)\n\n\nsearch_algo_registry = {\n None: BasicVariantSA,\n \"variant_generator\": BasicVariantSA,\n \"random\": BasicVariantSA,\n \"hyperopt\": HyperoptSA,\n \"bohb\": BOHBSA,\n \"ax\": AxSA,\n \"bayesopt\": BayesOptSA,\n \"blendsearch\": BlendsearchSA,\n \"cfo\": CFOSA,\n \"dragonfly\": DragonflySA,\n \"hebo\": HEBOSA,\n \"skopt\": SkoptSA,\n \"nevergrad\": NevergradSA,\n \"optuna\": OptunaSA,\n \"zoopt\": ZooptSA,\n}\n"
},
{
"path": "ludwig/hyperopt/utils.py",
"content": "import copy\nimport dataclasses\nimport json\nimport logging\nimport os\nimport warnings\nfrom typing import Any\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import (\n AUTO,\n COMBINED,\n EXECUTOR,\n GOAL,\n GRID_SEARCH,\n HYPEROPT,\n INPUT_FEATURES,\n LOSS,\n MAX_CONCURRENT_TRIALS,\n METRIC,\n MINIMIZE,\n NAME,\n NUM_SAMPLES,\n OUTPUT_FEATURES,\n PARAMETERS,\n PREPROCESSING,\n RAY,\n SPACE,\n SPLIT,\n TYPE,\n VALIDATION,\n)\nfrom ludwig.globals import HYPEROPT_STATISTICS_FILE_NAME\nfrom ludwig.hyperopt.results import HyperoptResults, TrialResults\nfrom ludwig.types import HyperoptConfigDict, ModelConfigDict\nfrom ludwig.utils.data_utils import save_json\nfrom ludwig.utils.misc_utils import (\n get_class_attributes,\n get_from_registry,\n merge_dict,\n set_default_value,\n set_default_values,\n)\nfrom ludwig.utils.print_utils import print_boxed\n\nlogger = logging.getLogger(__name__)\n\n\ndef print_hyperopt_results(hyperopt_results: HyperoptResults):\n print_boxed(\"HYPEROPT RESULTS\", print_fun=logger.info)\n for trial_results in hyperopt_results.ordered_trials:\n if not isinstance(trial_results.metric_score, str):\n logger.info(f\"score: {trial_results.metric_score:.6f} | parameters: {trial_results.parameters}\")\n logger.info(\"\")\n\n\ndef save_hyperopt_stats(hyperopt_stats, hyperopt_dir_name):\n hyperopt_stats_fn = os.path.join(hyperopt_dir_name, HYPEROPT_STATISTICS_FILE_NAME)\n save_json(hyperopt_stats_fn, hyperopt_stats)\n\n\ndef load_json_value(v):\n try:\n return json.loads(v)\n except Exception as e:\n logger.warning(f\"While loading json, encountered exception: {e}\")\n return v\n\n\n# define set containing names to return for TrialResults\nTRIAL_RESULTS_NAMES_SET = {f.name for f in dataclasses.fields(TrialResults)}\n\n\ndef load_json_values(d):\n # ensure metric_score is a string for the json load to eliminate extraneous exception message\n d[\"metric_score\"] = str(d[\"metric_score\"])\n\n # load only data required for TrialResults\n return {k: load_json_value(v) for k, v in d.items() if k in TRIAL_RESULTS_NAMES_SET}\n\n\ndef should_tune_preprocessing(config):\n parameters = config[HYPEROPT][PARAMETERS]\n for param_name in parameters.keys():\n if f\"{PREPROCESSING}.\" in param_name:\n return True\n return False\n\n\ndef parameter_to_dict(name, value):\n if name == \".\":\n # Parameter name \".\", means top-level config\n return value\n\n parameter_dict = {}\n curr_dict = parameter_dict\n name_list = name.split(\".\")\n for i, name_elem in enumerate(name_list):\n if i == len(name_list) - 1:\n curr_dict[name_elem] = value\n else:\n name_dict = curr_dict.get(name_elem, {})\n curr_dict[name_elem] = name_dict\n curr_dict = name_dict\n return parameter_dict\n\n\ndef feature_list_to_dict(config: ModelConfigDict) -> ModelConfigDict:\n input_features_dict = {}\n for feature in config[INPUT_FEATURES]:\n input_features_dict[feature[NAME]] = feature\n\n output_features_dict = {}\n for feature in config[OUTPUT_FEATURES]:\n output_features_dict[feature[NAME]] = feature\n\n config = copy.copy(config)\n config[INPUT_FEATURES] = input_features_dict\n config[OUTPUT_FEATURES] = output_features_dict\n return config\n\n\ndef feature_dict_to_list(config: ModelConfigDict) -> ModelConfigDict:\n # This works because Python dicts are order-preserving, so we do not need to\n # do anything special to map from a key in the dict to an index in a list\n input_features_list = []\n for feature in config[INPUT_FEATURES].values():\n input_features_list.append(feature)\n\n output_features_list = []\n for feature in config[OUTPUT_FEATURES].values():\n output_features_list.append(feature)\n\n config = copy.copy(config)\n config[INPUT_FEATURES] = input_features_list\n config[OUTPUT_FEATURES] = output_features_list\n return config\n\n\ndef substitute_parameters(\n config: ModelConfigDict,\n parameters: dict[str, Any],\n):\n \"\"\"Update Ludwig config with parameters sampled from the Hyperopt sampler.\"\"\"\n\n # Collect the sets of names for each feature grouping so we can map feature names to\n # groups\n input_feature_names = {feature[NAME] for feature in config[INPUT_FEATURES]}\n output_feature_names = {feature[NAME] for feature in config[OUTPUT_FEATURES]}\n\n # Features in the user config are provided as a list, but in hyperopt we reference\n # features by name, so convert temporarily to a dict to simplify the mergep process.\n config = feature_list_to_dict(config)\n\n # Merge parameters into the user configuration in order. As such, if there are conflicting\n # params, the later params will take precedence.\n for name, value in parameters.items():\n # User params are provided as ., but we group input / output features\n # together during the merge to make it easier and unambiguous to convert back and forth\n # TODO(travis): we should revisit the user format here, as it silently breaks situations\n # where the user has a feature named \"trainer\", \"combiner\", etc.\n prefix = name.split(\".\")[0]\n if prefix in input_feature_names:\n name = f\"{INPUT_FEATURES}.{name}\"\n elif prefix in output_feature_names:\n name = f\"{OUTPUT_FEATURES}.{name}\"\n\n param_dict = parameter_to_dict(name, value)\n config = merge_dict(config, param_dict)\n\n # Now that all features have been merged, convert back to the original list format.\n config = feature_dict_to_list(config)\n\n return config\n\n\n@DeveloperAPI\ndef get_num_duplicate_trials(hyperopt_config: HyperoptConfigDict) -> int:\n \"\"\"Returns the number of duplicate trials that will be created.\n\n Duplicate trials are only created when there are grid type parameters and num_samples > 1.\n \"\"\"\n num_samples = hyperopt_config[EXECUTOR].get(NUM_SAMPLES, 1)\n if num_samples == 1:\n return 0\n\n total_grid_search_trials = 1\n for _, param_info in hyperopt_config[PARAMETERS].items():\n if param_info.get(SPACE, None) == GRID_SEARCH:\n total_grid_search_trials *= len(param_info.get(\"values\", []))\n\n num_duplicate_trials = (total_grid_search_trials * num_samples) - total_grid_search_trials\n return num_duplicate_trials\n\n\ndef log_warning_if_all_grid_type_parameters(hyperopt_config: HyperoptConfigDict) -> None:\n \"\"\"Logs warning if all parameters have a grid type search space and num_samples > 1 since this will result in\n duplicate trials being created.\"\"\"\n num_duplicate_trials = get_num_duplicate_trials(hyperopt_config)\n if num_duplicate_trials == 0:\n return\n\n num_samples = hyperopt_config[EXECUTOR].get(NUM_SAMPLES, 1)\n warnings.warn(\n \"All hyperopt parameters in Ludwig config are using grid_search space, but number of samples \"\n f\"({num_samples}) is greater than 1. This will result in {num_duplicate_trials} duplicate trials being \"\n \"created. Consider setting `num_samples` to 1 in the hyperopt executor to prevent trial duplication.\",\n RuntimeWarning,\n )\n\n\ndef update_hyperopt_params_with_defaults(hyperopt_params: HyperoptConfigDict) -> None:\n \"\"\"Updates user's Ludwig config with default hyperopt parameters.\"\"\"\n from ludwig.hyperopt.execution import executor_registry\n\n set_default_value(hyperopt_params, EXECUTOR, {})\n set_default_value(hyperopt_params, SPLIT, VALIDATION)\n set_default_value(hyperopt_params, \"output_feature\", COMBINED)\n set_default_value(hyperopt_params, METRIC, LOSS)\n set_default_value(hyperopt_params, GOAL, MINIMIZE)\n\n set_default_values(\n hyperopt_params[EXECUTOR],\n {TYPE: RAY, NUM_SAMPLES: 1, MAX_CONCURRENT_TRIALS: AUTO},\n )\n\n if hyperopt_params[EXECUTOR].get(\"trial_driver_resources\") is None:\n hyperopt_params[EXECUTOR][\"trial_driver_resources\"] = {\"CPU\": 1, \"GPU\": 0}\n\n executor = get_from_registry(hyperopt_params[EXECUTOR][TYPE], executor_registry)\n executor_defaults = {k: v for k, v in executor.__dict__.items() if k in get_class_attributes(executor)}\n set_default_values(\n hyperopt_params[EXECUTOR],\n executor_defaults,\n )\n"
},
{
"path": "ludwig/hyperopt_cli.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport argparse\nimport logging\nimport sys\n\nfrom ludwig.backend import ALL_BACKENDS, Backend, initialize_backend\nfrom ludwig.callbacks import Callback\nfrom ludwig.contrib import add_contrib_callback_args\nfrom ludwig.globals import LUDWIG_VERSION\nfrom ludwig.hyperopt.run import hyperopt\nfrom ludwig.utils.data_utils import load_config_from_str, load_yaml\nfrom ludwig.utils.defaults import default_random_seed\nfrom ludwig.utils.print_utils import get_logging_level_registry, print_ludwig\n\nlogger = logging.getLogger(__name__)\n\n\ndef hyperopt_cli(\n config: str | dict,\n dataset: str = None,\n training_set: str = None,\n validation_set: str = None,\n test_set: str = None,\n training_set_metadata: str = None,\n data_format: str = None,\n experiment_name: str = \"experiment\",\n model_name: str = \"run\",\n # model_load_path=None,\n # model_resume_path=None,\n skip_save_training_description: bool = False,\n skip_save_training_statistics: bool = False,\n skip_save_model: bool = False,\n skip_save_progress: bool = False,\n skip_save_log: bool = False,\n skip_save_processed_input: bool = False,\n skip_save_unprocessed_output: bool = False,\n skip_save_predictions: bool = False,\n skip_save_eval_stats: bool = False,\n skip_save_hyperopt_statistics: bool = False,\n output_directory: str = \"results\",\n gpus: str | int | list[int] = None,\n gpu_memory_limit: float | None = None,\n allow_parallel_threads: bool = True,\n callbacks: list[Callback] = None,\n backend: Backend | str = None,\n random_seed: int = default_random_seed,\n hyperopt_log_verbosity: int = 3,\n **kwargs,\n):\n \"\"\"Searches for optimal hyperparameters.\n\n # Inputs\n\n :param config: (Union[str, dict]) in-memory representation of\n config or string path to a YAML config file.\n :param dataset: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing the entire dataset to be used for training.\n If it has a split column, it will be used for splitting (0 for train,\n 1 for validation, 2 for test), otherwise the dataset will be\n randomly split.\n :param training_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing training data.\n :param validation_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing validation data.\n :param test_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing test data.\n :param training_set_metadata: (Union[str, dict], default: `None`)\n metadata JSON file or loaded metadata. Intermediate preprocessed\n structure containing the mappings of the input\n dataset created the first time an input file is used in the same\n directory with the same name and a '.meta.json' extension.\n :param data_format: (str, default: `None`) format to interpret data\n sources. Will be inferred automatically if not specified. Valid\n formats are `'auto'`, `'csv'`, `'excel'`, `'feather'`,\n `'fwf'`, `'hdf5'` (cache file produced during previous training),\n `'html'` (file containing a single HTML `
`), `'json'`, `'jsonl'`,\n `'parquet'`, `'pickle'` (pickled Pandas DataFrame), `'sas'`, `'spss'`,\n `'stata'`, `'tsv'`.\n :param experiment_name: (str, default: `'experiment'`) name for\n the experiment.\n :param model_name: (str, default: `'run'`) name of the model that is\n being used.\n :param skip_save_training_description: (bool, default: `False`) disables\n saving the description JSON file.\n :param skip_save_training_statistics: (bool, default: `False`) disables\n saving training statistics JSON file.\n :param skip_save_model: (bool, default: `False`) disables\n saving model weights and hyperparameters each time the model\n improves. By default Ludwig saves model weights after each epoch\n the validation metric improves, but if the model is really big\n that can be time consuming. If you do not want to keep\n the weights and just find out what performance a model can get\n with a set of hyperparameters, use this parameter to skip it,\n but the model will not be loadable later on and the returned model\n will have the weights obtained at the end of training, instead of\n the weights of the epoch with the best validation performance.\n :param skip_save_progress: (bool, default: `False`) disables saving\n progress each epoch. By default Ludwig saves weights and stats\n after each epoch for enabling resuming of training, but if\n the model is really big that can be time consuming and will uses\n twice as much space, use this parameter to skip it, but training\n cannot be resumed later on.\n :param skip_save_log: (bool, default: `False`) disables saving\n TensorBoard logs. By default Ludwig saves logs for the TensorBoard,\n but if it is not needed turning it off can slightly increase the\n overall speed.\n :param skip_save_processed_input: (bool, default: `False`) if input\n dataset is provided it is preprocessed and cached by saving an HDF5\n and JSON files to avoid running the preprocessing again. If this\n parameter is `False`, the HDF5 and JSON file are not saved.\n :param skip_save_unprocessed_output: (bool, default: `False`) by default\n predictions and their probabilities are saved in both raw\n unprocessed numpy files containing tensors and as postprocessed\n CSV files (one for each output feature). If this parameter is True,\n only the CSV ones are saved and the numpy ones are skipped.\n :param skip_save_predictions: (bool, default: `False`) skips saving test\n predictions CSV files\n :param skip_save_eval_stats: (bool, default: `False`) skips saving test\n statistics JSON file\n :param skip_save_hyperopt_statistics: (bool, default: `False`) skips saving\n hyperopt stats file.\n :param output_directory: (str, default: `'results'`) the directory that\n will contain the training statistics, TensorBoard logs, the saved\n model and the training progress files.\n :param gpus: (list, default: `None`) list of GPUs that are available\n for training.\n :param gpu_memory_limit: (float: default: `None`) maximum memory fraction\n [0, 1] allowed to allocate per GPU device.\n :param allow_parallel_threads: (bool, default: `True`) allow PyTorch\n to use multithreading parallelism to improve performance at\n the cost of determinism.\n :param callbacks: (list, default: `None`) a list of\n `ludwig.callbacks.Callback` objects that provide hooks into the\n Ludwig pipeline.\n :param backend: (Union[Backend, str]) `Backend` or string name\n of backend to use to execute preprocessing / training steps.\n :param random_seed: (int: default: 42) random seed used for weights\n initialization, splits and any other random function.\n :param hyperopt_log_verbosity: (int: default: 3) Controls verbosity of ray tune log messages. Valid values:\n 0 = silent, 1 = only status updates, 2 = status and brief trial\n results, 3 = status and detailed trial results.\n\n # Return\n :return\" (`None`)\n \"\"\"\n return hyperopt(\n config=config,\n dataset=dataset,\n training_set=training_set,\n validation_set=validation_set,\n test_set=test_set,\n training_set_metadata=training_set_metadata,\n data_format=data_format,\n experiment_name=experiment_name,\n model_name=model_name,\n # model_load_path=model_load_path,\n # model_resume_path=model_resume_path,\n skip_save_training_description=skip_save_training_description,\n skip_save_training_statistics=skip_save_training_statistics,\n skip_save_model=skip_save_model,\n skip_save_progress=skip_save_progress,\n skip_save_log=skip_save_log,\n skip_save_processed_input=skip_save_processed_input,\n skip_save_unprocessed_output=skip_save_unprocessed_output,\n skip_save_predictions=skip_save_predictions,\n skip_save_eval_stats=skip_save_eval_stats,\n skip_save_hyperopt_statistics=skip_save_hyperopt_statistics,\n output_directory=output_directory,\n gpus=gpus,\n gpu_memory_limit=gpu_memory_limit,\n allow_parallel_threads=allow_parallel_threads,\n callbacks=callbacks,\n backend=backend,\n random_seed=random_seed,\n hyperopt_log_verbosity=hyperopt_log_verbosity,\n **kwargs,\n )\n\n\ndef cli(sys_argv):\n parser = argparse.ArgumentParser(\n description=\"This script searches for optimal Hyperparameters\",\n prog=\"ludwig hyperopt\",\n usage=\"%(prog)s [options]\",\n )\n\n # -------------------\n # Hyperopt parameters\n # -------------------\n parser.add_argument(\n \"-sshs\",\n \"--skip_save_hyperopt_statistics\",\n help=\"skips saving hyperopt statistics file\",\n action=\"store_true\",\n default=False,\n )\n\n # ----------------------------\n # Experiment naming parameters\n # ----------------------------\n parser.add_argument(\n \"--output_directory\",\n type=str,\n default=\"results\",\n help=\"directory that contains the results\",\n )\n parser.add_argument(\"--experiment_name\", type=str, default=\"hyperopt\", help=\"experiment name\")\n parser.add_argument(\"--model_name\", type=str, default=\"run\", help=\"name for the model\")\n\n # ---------------\n # Data parameters\n # ---------------\n parser.add_argument(\n \"--dataset\",\n help=\"input data file path. \"\n \"If it has a split column, it will be used for splitting \"\n \"(0: train, 1: validation, 2: test), \"\n \"otherwise the dataset will be randomly split\",\n )\n parser.add_argument(\"--training_set\", help=\"input train data file path\")\n parser.add_argument(\"--validation_set\", help=\"input validation data file path\")\n parser.add_argument(\"--test_set\", help=\"input test data file path\")\n\n parser.add_argument(\n \"--training_set_metadata\",\n help=\"input metadata JSON file path. An intermediate preprocessed file \"\n \"containing the mappings of the input file created \"\n \"the first time a file is used, in the same directory \"\n \"with the same name and a .json extension\",\n )\n\n parser.add_argument(\n \"--data_format\",\n help=\"format of the input data\",\n default=\"auto\",\n choices=[\n \"auto\",\n \"csv\",\n \"excel\",\n \"feather\",\n \"fwf\",\n \"hdf5\",\n \"html\" \"tables\",\n \"json\",\n \"jsonl\",\n \"parquet\",\n \"pickle\",\n \"sas\",\n \"spss\",\n \"stata\",\n \"tsv\",\n ],\n )\n\n parser.add_argument(\n \"-sspi\",\n \"--skip_save_processed_input\",\n help=\"skips saving intermediate HDF5 and JSON files\",\n action=\"store_true\",\n default=False,\n )\n\n # ----------------\n # Model parameters\n # ----------------\n config = parser.add_mutually_exclusive_group(required=True)\n config.add_argument(\n \"-c\",\n \"--config\",\n type=load_yaml,\n help=\"Path to the YAML file containing the model configuration\",\n )\n config.add_argument(\n \"-cs\",\n \"--config_str\",\n dest=\"config\",\n type=load_config_from_str,\n help=\"JSON or YAML serialized string of the model configuration\",\n )\n\n parser.add_argument(\n \"-mlp\",\n \"--model_load_path\",\n help=\"path of a pretrained model to load as initialization\",\n )\n parser.add_argument(\n \"-mrp\",\n \"--model_resume_path\",\n help=\"path of the model directory to resume training of\",\n )\n parser.add_argument(\n \"-sstd\",\n \"--skip_save_training_description\",\n action=\"store_true\",\n default=False,\n help=\"disables saving the description JSON file\",\n )\n parser.add_argument(\n \"-ssts\",\n \"--skip_save_training_statistics\",\n action=\"store_true\",\n default=False,\n help=\"disables saving training statistics JSON file\",\n )\n parser.add_argument(\n \"-ssm\",\n \"--skip_save_model\",\n action=\"store_true\",\n default=False,\n help=\"disables saving weights each time the model improves. \"\n \"By default Ludwig saves weights after each epoch \"\n \"the validation metric (improves, but if the model is really big \"\n \"that can be time consuming. If you do not want to keep \"\n \"the weights and just find out what performance a model can get \"\n \"with a set of hyperparameters, use this parameter to skip it\",\n )\n parser.add_argument(\n \"-ssp\",\n \"--skip_save_progress\",\n action=\"store_true\",\n default=False,\n help=\"disables saving weights after each epoch. By default ludwig saves \"\n \"weights after each epoch for enabling resuming of training, but \"\n \"if the model is really big that can be time consuming and will \"\n \"save twice as much space, use this parameter to skip it\",\n )\n parser.add_argument(\n \"-ssl\",\n \"--skip_save_log\",\n action=\"store_true\",\n default=False,\n help=\"disables saving TensorBoard logs. By default Ludwig saves \"\n \"logs for the TensorBoard, but if it is not needed turning it off \"\n \"can slightly increase the overall speed\",\n )\n\n # ------------------\n # Runtime parameters\n # ------------------\n parser.add_argument(\n \"-rs\",\n \"--random_seed\",\n type=int,\n default=42,\n help=\"a random seed that is going to be used anywhere there is a call \"\n \"to a random number generator: data splitting, parameter \"\n \"initialization and training set shuffling\",\n )\n parser.add_argument(\n \"-hlv\",\n \"--hyperopt_log_verbosity\",\n type=int,\n default=3,\n choices=[0, 1, 2, 3],\n help=\"Controls verbosity of ray tune log messages. Valid values: \"\n \"0 = silent, 1 = only status updates, 2 = status and brief trial \"\n \"results, 3 = status and detailed trial results.\",\n )\n parser.add_argument(\"-g\", \"--gpus\", nargs=\"+\", type=int, default=None, help=\"list of gpus to use\")\n parser.add_argument(\n \"-gml\",\n \"--gpu_memory_limit\",\n type=float,\n default=None,\n help=\"maximum memory fraction [0, 1] allowed to allocate per GPU device\",\n )\n parser.add_argument(\n \"-b\",\n \"--backend\",\n help=\"specifies backend to use for parallel / distributed execution, \" \"defaults to local execution\",\n choices=ALL_BACKENDS,\n )\n parser.add_argument(\n \"-l\",\n \"--logging_level\",\n default=\"info\",\n help=\"the level of logging to use\",\n choices=[\"critical\", \"error\", \"warning\", \"info\", \"debug\", \"notset\"],\n )\n\n add_contrib_callback_args(parser)\n args = parser.parse_args(sys_argv)\n\n args.callbacks = args.callbacks or []\n for callback in args.callbacks:\n callback.on_cmdline(\"hyperopt\", *sys_argv)\n\n args.logging_level = get_logging_level_registry()[args.logging_level]\n logging.getLogger(\"ludwig\").setLevel(args.logging_level)\n global logger\n logger = logging.getLogger(\"ludwig.hyperopt\")\n\n args.backend = initialize_backend(args.backend or args.config.get(\"backend\"))\n if args.backend.is_coordinator():\n print_ludwig(\"Hyperopt\", LUDWIG_VERSION)\n\n hyperopt_cli(**vars(args))\n\n\nif __name__ == \"__main__\":\n cli(sys.argv[1:])\n"
},
{
"path": "ludwig/model_export/base_model_exporter.py",
"content": "from abc import ABC, abstractmethod\n\nimport torch\n\n\nclass LudwigTorchWrapper(torch.nn.Module):\n \"\"\"Base class that establishes the contract for exporting to different file formats.\"\"\"\n\n def __init__(self, model):\n super().__init__()\n self.model = model\n\n def forward(self, x):\n return self.model({\"image_path\": x})\n\n\nclass BaseModelExporter(ABC):\n @abstractmethod\n def export(self, model_path, export_path, export_args_override):\n pass\n\n @abstractmethod\n def check_model_export(self, path):\n pass\n"
},
{
"path": "ludwig/model_export/onnx_exporter.py",
"content": "import os\n\nimport torch\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.model_export.base_model_exporter import BaseModelExporter, LudwigTorchWrapper\n\n\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nclass OnnxExporter(BaseModelExporter):\n \"\"\"Class that abstracts the convertion of torch to onnx.\"\"\"\n\n def export(self, model_path, export_path, output_model_name):\n ludwig_model = LudwigModel.load(model_path)\n model = LudwigTorchWrapper(ludwig_model.model) # Wrap the model\n model.eval() # inference mode, is this needed.. I think onnx export does this for us\n\n width = ludwig_model.config[\"input_features\"][0][\"preprocessing\"][\"width\"]\n height = ludwig_model.config[\"input_features\"][0][\"preprocessing\"][\"height\"]\n example_input = torch.randn(1, 3, width, height, requires_grad=True)\n\n torch.onnx.export(\n model,\n example_input,\n os.path.join(export_path, output_model_name),\n opset_version=18,\n export_params=True,\n do_constant_folding=True,\n input_names=[\"input\"],\n output_names=[\"combiner_hidden_1\", \"output\", \"combiner_hidden_2\"],\n )\n\n def check_model_export(self, path):\n import onnx\n\n onnx_model = onnx.load(path)\n onnx.checker.check_model(onnx_model)\n"
},
{
"path": "ludwig/models/__init__.py",
"content": ""
},
{
"path": "ludwig/models/base.py",
"content": "import contextlib\nimport logging\nfrom abc import ABCMeta, abstractmethod\nfrom collections import OrderedDict\nfrom typing import Any\n\nimport numpy as np\nimport torch\nimport torchmetrics\n\nfrom ludwig.combiners.combiners import Combiner\nfrom ludwig.constants import COMBINED, LOSS, NAME\nfrom ludwig.encoders.base import Encoder\nfrom ludwig.features.base_feature import create_passthrough_input_feature, InputFeature, ModuleWrapper, OutputFeature\nfrom ludwig.features.feature_registries import get_input_type_registry, get_output_type_registry\nfrom ludwig.features.feature_utils import LudwigFeatureDict\nfrom ludwig.modules.metric_modules import LudwigMetric\nfrom ludwig.modules.training_hooks import TrainingHook\nfrom ludwig.schema.features.base import BaseInputFeatureConfig, BaseOutputFeatureConfig, FeatureCollection\nfrom ludwig.utils.algorithms_utils import topological_sort_feature_dependencies\nfrom ludwig.utils.metric_utils import get_scalar_from_ludwig_metric\nfrom ludwig.utils.misc_utils import get_from_registry\nfrom ludwig.utils.torch_utils import DEVICE, LudwigModule, reg_loss\nfrom ludwig.utils.types import TorchDevice\n\nlogger = logging.getLogger(__name__)\n\n\nclass BaseModel(LudwigModule, metaclass=ABCMeta):\n \"\"\"Base model for use in LudwigModule.\n\n Implementations of this class should implement the following methods:\n - type()\n - forward()\n \"\"\"\n\n @staticmethod\n @abstractmethod\n def type() -> str:\n \"\"\"Returns the model type.\"\"\"\n\n def __init__(self, random_seed: int = None):\n self._random_seed = random_seed\n\n # TODO: with change to misc_utils.set_random_seed() this may be redundant\n # seems to be required for test_api.py::test_api_training_determinism\n if random_seed is not None:\n torch.random.manual_seed(random_seed)\n\n super().__init__()\n\n self.input_features = self.create_feature_dict()\n self.output_features = self.create_feature_dict()\n\n # ================ Combined loss metric ================\n self._eval_loss_metric = ModuleWrapper(torchmetrics.MeanMetric())\n self._eval_additional_losses_metrics = ModuleWrapper(torchmetrics.MeanMetric())\n\n # ================ Training Hook Handles ================\n self._forward_hook_handles: list[TrainingHook] = []\n\n def create_feature_dict(self) -> LudwigFeatureDict:\n \"\"\"Creates and returns a LudwigFeatureDict.\"\"\"\n return LudwigFeatureDict()\n\n def to_device(self, device):\n return self.to(device)\n\n def metrics_to_device(self, device: str):\n self._eval_loss_metric.module = self._eval_loss_metric.module.to(device)\n self._eval_additional_losses_metrics.module = self._eval_additional_losses_metrics.module.to(device)\n for feature in self.output_features.values():\n feature._eval_loss_metric.module = feature._eval_loss_metric.module.to(device)\n\n @classmethod\n def build_inputs(cls, input_feature_configs: FeatureCollection[BaseInputFeatureConfig]) -> dict[str, InputFeature]:\n \"\"\"Builds and returns input features in topological order.\"\"\"\n input_features = OrderedDict()\n input_features_def = topological_sort_feature_dependencies(input_feature_configs.to_list())\n for input_feature_def in input_features_def:\n input_features[input_feature_def[NAME]] = cls.build_single_input(\n getattr(input_feature_configs, input_feature_def[NAME]), input_features\n )\n return input_features\n\n @staticmethod\n def build_single_input(\n feature_config: BaseInputFeatureConfig, other_input_features: dict[str, InputFeature] | None\n ) -> InputFeature:\n \"\"\"Builds a single input feature from the input feature definition.\"\"\"\n logger.debug(f\"Input {feature_config.type} feature {feature_config.name}\")\n\n encoder_obj = None\n if feature_config.tied is not None:\n tied_input_feature_name = feature_config.tied\n if tied_input_feature_name in other_input_features:\n encoder_obj = other_input_features[tied_input_feature_name].encoder_obj\n\n return create_input_feature(feature_config, encoder_obj)\n\n @classmethod\n def build_outputs(\n cls, output_feature_configs: FeatureCollection[BaseOutputFeatureConfig], combiner: Combiner\n ) -> dict[str, OutputFeature]:\n \"\"\"Builds and returns output features in topological order.\"\"\"\n output_features_def = topological_sort_feature_dependencies(output_feature_configs.to_list())\n output_features = {}\n\n for output_feature_def in output_features_def:\n # TODO(Justin): Check that the semantics of input_size align with what the combiner's output shape returns\n # for seq2seq.\n setattr(getattr(output_feature_configs, output_feature_def[NAME]), \"input_size\", combiner.output_shape[-1])\n output_features[output_feature_def[NAME]] = cls.build_single_output(\n getattr(output_feature_configs, output_feature_def[NAME]), output_features\n )\n return output_features\n\n @staticmethod\n def build_single_output(\n feature_config: BaseOutputFeatureConfig, output_features: dict[str, OutputFeature] | None\n ) -> OutputFeature:\n \"\"\"Builds a single output feature from the output feature definition.\"\"\"\n logger.debug(f\"Output {feature_config.type} feature {feature_config.name}\")\n output_feature_class = get_from_registry(feature_config.type, get_output_type_registry())\n output_feature_obj = output_feature_class(feature_config, output_features=output_features)\n return output_feature_obj\n\n def get_model_inputs(self):\n \"\"\"Returns a dict of feature name -> sample model input.\"\"\"\n device = next(self.parameters()).device\n inputs = {\n input_feature_name: input_feature.create_sample_input().to(device)\n for input_feature_name, input_feature in self.input_features.items()\n }\n return inputs\n\n def get_model_size(self) -> int:\n \"\"\"Returns total number of parameters in model.\"\"\"\n model_tensors = self.collect_weights()\n total_size = 0\n for tnsr in model_tensors:\n total_size += tnsr[1].detach().cpu().numpy().size\n return total_size\n\n def to_torchscript(self, device: TorchDevice | None = None):\n \"\"\"Converts the ECD model as a TorchScript model.\"\"\"\n if device is None:\n device = DEVICE\n\n self.eval()\n model_inputs = self.get_model_inputs()\n\n model_to_script = self.to(device)\n model_inputs_to_script = {k: v.to(device) for k, v in model_inputs.items()}\n # We set strict=False to enable dict inputs and outputs.\n return torch.jit.trace(model_to_script, model_inputs_to_script, strict=False)\n\n def save_torchscript(self, save_path, device: TorchDevice | None = None):\n \"\"\"Saves the ECD model as a TorchScript model.\"\"\"\n if device is None:\n device = DEVICE\n\n traced = self.to_torchscript(device)\n traced.save(save_path)\n\n @property\n def input_shape(self):\n \"\"\"Returns the shape of the model's input.\"\"\"\n # TODO(justin): Remove dummy implementation. Make input_shape and output_shape functions.\n return torch.Size([1, 1])\n\n @abstractmethod\n def forward(\n self,\n inputs: (\n dict[str, torch.Tensor] | dict[str, np.ndarray] | tuple[dict[str, torch.Tensor], dict[str, torch.Tensor]]\n ),\n mask=None,\n ) -> dict[str, torch.Tensor]:\n \"\"\"Forward pass of the model.\n\n Args:\n inputs: Inputs to the model. Can be a dictionary of input names to\n input tensors or a tuple of (inputs, targets) where inputs is\n a dictionary of input names to input tensors and targets is a\n dictionary of target names to target tensors.\n mask: A mask for the inputs.\n\n Returns:\n A dictionary of output {feature name}::{tensor_name} -> output tensor.\n \"\"\"\n\n def predictions(self, inputs):\n \"\"\"Returns the model's predictions for the given inputs.\"\"\"\n outputs = self(inputs)\n return self.outputs_to_predictions(outputs)\n\n def outputs_to_predictions(self, outputs: dict[str, torch.Tensor]) -> dict[str, dict[str, torch.Tensor]]:\n \"\"\"Returns the model's predictions given the raw model outputs.\"\"\"\n predictions = {}\n for of_name in self.output_features:\n predictions[of_name] = self.output_features.get(of_name).predictions(outputs, of_name)\n return predictions\n\n def evaluation_step(self, inputs, targets):\n \"\"\"Predict the inputs and update evaluation metrics.\"\"\"\n predictions = self.predictions(inputs)\n self.update_metrics(targets, predictions)\n return predictions\n\n def predict_step(self, inputs):\n \"\"\"Predict the inputs.\"\"\"\n return self.predictions(inputs)\n\n def train_loss(\n self,\n targets,\n predictions,\n regularization_type: str | None = None,\n regularization_lambda: float | None = None,\n ) -> tuple[torch.Tensor, dict[str, torch.Tensor]]:\n \"\"\"Computes the training loss for the model.\n\n Args:\n targets: A dictionary of target names to target tensors.\n predictions: A dictionary of output names to output tensors.\n regularization_type: One of 'l1', 'l2', 'l1_l2', or None.\n regularization_lambda: The regularization lambda.\n\n Returns:\n A tuple of the loss tensor and a dictionary of loss for every\n output feature.\n \"\"\"\n train_loss = 0\n of_train_losses = {}\n for of_name, of_obj in self.output_features.items():\n of_train_loss = of_obj.train_loss(targets[of_name], predictions, of_name)\n train_loss += of_obj.loss.weight * of_train_loss\n of_train_losses[of_name] = of_train_loss\n\n additional_losses = self.losses()\n if additional_losses:\n train_loss += torch.sum(torch.stack(additional_losses)) # other losses\n\n # Add regularization loss\n if regularization_type is not None and regularization_lambda != 0:\n train_loss += reg_loss(self, regularization_type, l1=regularization_lambda, l2=regularization_lambda)\n\n return train_loss, of_train_losses\n\n def eval_loss(self, targets, predictions):\n \"\"\"Computes all evaluation losses for the model given targets and predictions.\n\n Args:\n targets: A dictionary of target names to target tensors.\n predictions: A dictionary of output names to output tensors.\n\n Returns:\n A tuple of loss values for eval losses and additional losses.\n \"\"\"\n eval_loss = 0\n for of_name, of_obj in self.output_features.items():\n of_eval_loss = of_obj.eval_loss(targets[of_name], predictions[of_name])\n eval_loss += of_obj.loss.weight * of_eval_loss\n\n additional_loss = 0\n additional_losses = self.losses()\n if additional_losses:\n additional_loss = torch.sum(torch.stack(additional_losses)) # other losses\n\n return eval_loss, additional_loss\n\n def update_metrics(self, targets, predictions):\n \"\"\"Updates the model's metrics given targets and predictions.\"\"\"\n for of_name, of_obj in self.output_features.items():\n of_obj.update_metrics(targets[of_name], predictions[of_name])\n\n eval_loss, additional_losses = self.eval_loss(targets, predictions)\n self.eval_loss_metric.update(eval_loss)\n self.eval_additional_losses_metrics.update(additional_losses)\n\n @property\n def eval_loss_metric(self) -> LudwigMetric:\n return self._eval_loss_metric.module\n\n @eval_loss_metric.setter\n def eval_loss_metric(self, value: LudwigMetric) -> None:\n self._eval_loss_metric.module = value\n\n @property\n def eval_additional_losses_metrics(self) -> LudwigMetric:\n return self._eval_additional_losses_metrics.module\n\n def get_metrics(self) -> dict[str, dict[str, float]]:\n \"\"\"Returns a dictionary of metrics for each output feature of the model.\"\"\"\n all_of_metrics = {}\n for of_name, of_obj in self.output_features.items():\n all_of_metrics[of_name] = of_obj.get_metrics()\n all_of_metrics[COMBINED] = {\n LOSS: get_scalar_from_ludwig_metric(self.eval_loss_metric)\n + get_scalar_from_ludwig_metric(self.eval_additional_losses_metrics)\n }\n return all_of_metrics\n\n def reset_metrics(self):\n \"\"\"Resets the model's metrics.\"\"\"\n for of_obj in self.output_features.values():\n of_obj.reset_metrics()\n self.eval_loss_metric.reset()\n\n def collect_weights(self, tensor_names=None, **kwargs):\n \"\"\"Returns named parameters filtered against `tensor_names` if not None.\"\"\"\n if not tensor_names:\n return self.named_parameters()\n\n # Check for bad tensor names.\n weight_names = {name for name, _ in self.named_parameters()}\n for name in tensor_names:\n if name not in weight_names:\n raise ValueError(f'Requested tensor name filter \"{name}\" not present in the model graph') # noqa: E713\n\n # Apply filter.\n tensor_set = set(tensor_names)\n return [named_param for named_param in self.named_parameters() if named_param[0] in tensor_set]\n\n def unskip(self):\n \"\"\"Converts all skipped features into their fully encoded versions.\"\"\"\n\n @abstractmethod\n def save(self, save_path: str):\n \"\"\"Saves the model to the given path.\"\"\"\n\n @abstractmethod\n def load(self, save_path: str):\n \"\"\"Loads the model from the given path.\"\"\"\n\n @abstractmethod\n def get_args(self):\n \"\"\"Returns init arguments for constructing this model.\"\"\"\n\n @contextlib.contextmanager\n def use_generation_config(self, generation_config: dict[str, Any]):\n if generation_config is not None:\n raise NotImplementedError(f\"{self.__class__.__name__} does not support generation_config. \")\n yield\n\n def _activate_forward_hooks(self):\n \"\"\"Activates/registers forward hooks for the model.\"\"\"\n\n def _deactivate_forward_hooks(self) -> None:\n \"\"\"Deactivates/de-registers forward hooks for the model (if needed).\"\"\"\n for handle in self._forward_hook_handles:\n handle.deactivate_hook()\n\n\ndef create_input_feature(feature_config: BaseInputFeatureConfig, encoder_obj: Encoder | None) -> InputFeature:\n input_feature_cls = get_from_registry(feature_config.type, get_input_type_registry())\n input_feature = input_feature_cls(feature_config, encoder_obj=encoder_obj)\n if not feature_config.encoder.skip:\n return input_feature\n return create_passthrough_input_feature(input_feature, feature_config)\n"
},
{
"path": "ludwig/models/calibrator.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nimport numpy as np\n\nfrom ludwig.backend import Backend\nfrom ludwig.models.ecd import ECD\n\n\nclass Calibrator:\n \"\"\"Calibrator calibrates the output probabilities of a model.\"\"\"\n\n def __init__(self, model: ECD, backend: Backend, batch_size: int = 128):\n self.model = model\n self.backend = backend\n self.batch_size = batch_size\n\n def calibration_enabled(self):\n \"\"\"Calibration is enabled if the config requests calibration for any output feature.\n\n If no output features have calibration enabled, the calibration phase should be skipped.\n \"\"\"\n return any(o.calibration_module is not None for o in self.model.output_features.values())\n\n def train_calibration(self, dataset, dataset_name: str):\n \"\"\"Calibrates model output probabilities on validation set after training.\n\n This works well for most datasets, though it may fail for some difficult or extremely imbalanced datasets.\n \"\"\"\n if not self.calibration_enabled():\n # Early out if no output features have calibration enabled.\n return\n with self.backend.create_predictor(self.model, batch_size=self.batch_size) as predictor:\n metrics, predictions = predictor.batch_evaluation(\n dataset, collect_predictions=True, collect_logits=True, dataset_name=dataset_name\n )\n\n dataset_df = dataset.to_df()\n for output_feature in self.model.output_features.values():\n if output_feature.calibration_module is not None:\n feature_logits_key = f\"{output_feature.feature_name}_logits\"\n if feature_logits_key in predictions:\n feature_logits = self.backend.df_engine.compute(predictions[feature_logits_key])\n feature_labels = self.backend.df_engine.compute(dataset_df[output_feature.proc_column])\n output_feature.calibration_module.train_calibration(\n np.stack(feature_logits.values, axis=0), np.stack(feature_labels.values, axis=0)\n )\n"
},
{
"path": "ludwig/models/ecd.py",
"content": "import logging\nimport os\n\nimport numpy as np\nimport torch\n\nfrom ludwig.accounting.used_tokens import get_used_tokens_for_ecd\nfrom ludwig.combiners.combiners import create_combiner\nfrom ludwig.constants import MODEL_ECD, MODEL_LLM, USED_TOKENS\nfrom ludwig.globals import MODEL_WEIGHTS_FILE_NAME\nfrom ludwig.models.base import BaseModel\nfrom ludwig.schema.model_types.ecd import ECDModelConfig\nfrom ludwig.utils import output_feature_utils\nfrom ludwig.utils.augmentation_utils import AugmentationPipelines\nfrom ludwig.utils.data_utils import clear_data_cache\nfrom ludwig.utils.fs_utils import open_file\nfrom ludwig.utils.state_dict_backward_compatibility import update_state_dict\nfrom ludwig.utils.torch_utils import get_torch_device\n\nlogger = logging.getLogger(__name__)\n\n\nclass ECD(BaseModel):\n @staticmethod\n def type() -> str:\n return MODEL_ECD\n\n def __init__(\n self,\n config_obj: ECDModelConfig,\n random_seed=None,\n **_kwargs,\n ):\n self.config_obj = config_obj\n self._random_seed = random_seed\n\n super().__init__(random_seed=self._random_seed)\n\n # ================ Inputs ================\n try:\n self.input_features.update(self.build_inputs(input_feature_configs=self.config_obj.input_features))\n except KeyError as e:\n raise KeyError(\n f\"An input feature has a name that conflicts with a class attribute of torch's ModuleDict: {e}\"\n ) from e\n\n # ================ Combiner ================\n logger.debug(f\"Combiner {self.config_obj.combiner.type}\")\n self.combiner = create_combiner(self.config_obj.combiner, input_features=self.input_features)\n\n # ================ Outputs ================\n self.output_features.update(\n self.build_outputs(output_feature_configs=self.config_obj.output_features, combiner=self.combiner)\n )\n\n # After constructing all layers, clear the cache to free up memory\n clear_data_cache()\n\n def prepare_for_training(self):\n # 1/10/23: For parity with how the LLM model type sets up adapters and quantization, LLM encoders should call\n # `prepare_for_training` at training time rather than at initialization. This loop searches for input features\n # using the LLM encoder and calls `prepare_for_training` on those encoders only. No other changes should be\n # made to the ECD model itself or any other encoders.\n for feature in self.config_obj.input_features:\n encoder_type = feature.encoder.type\n if encoder_type == MODEL_LLM:\n feature_name = feature.name\n encoder = self.input_features.get(feature_name)\n encoder.prepare_for_training()\n\n def encode(\n self,\n inputs: (\n dict[str, torch.Tensor] | dict[str, np.ndarray] | tuple[dict[str, torch.Tensor], dict[str, torch.Tensor]]\n ),\n ):\n # Convert inputs to tensors.\n for input_feature_name, input_values in inputs.items():\n if not isinstance(input_values, torch.Tensor):\n inputs[input_feature_name] = torch.from_numpy(input_values)\n else:\n inputs[input_feature_name] = input_values\n\n encoder_outputs = {}\n for input_feature_name, input_values in inputs.items():\n encoder = self.input_features.get(input_feature_name)\n encoder_output = encoder(input_values)\n encoder_outputs[input_feature_name] = encoder_output\n\n return encoder_outputs\n\n def combine(self, encoder_outputs):\n return self.combiner(encoder_outputs)\n\n def decode(self, combiner_outputs, targets, mask):\n # Invoke output features.\n output_logits = {}\n output_last_hidden = {}\n for output_feature_name, output_feature in self.output_features.items():\n # Use the presence or absence of targets to signal training or prediction.\n target = targets[output_feature_name] if targets is not None else None\n decoder_outputs = output_feature(combiner_outputs, output_last_hidden, mask=mask, target=target)\n\n # Add decoder outputs to overall output dictionary.\n for decoder_output_name, tensor in decoder_outputs.items():\n output_feature_utils.set_output_feature_tensor(\n output_logits, output_feature_name, decoder_output_name, tensor\n )\n\n # Save the hidden state of the output feature (for feature dependencies).\n output_last_hidden[output_feature_name] = decoder_outputs[\"last_hidden\"]\n return output_logits\n\n def forward(\n self,\n inputs: (\n dict[str, torch.Tensor] | dict[str, np.ndarray] | tuple[dict[str, torch.Tensor], dict[str, torch.Tensor]]\n ),\n mask=None,\n ) -> dict[str, torch.Tensor]:\n \"\"\"Forward pass of the model.\n\n Args:\n inputs: Inputs to the model. Can be a dictionary of input names to\n input tensors or a tuple of (inputs, targets) where inputs is\n a dictionary of input names to input tensors and targets is a\n dictionary of target names to target tensors.\n mask: A mask for the inputs.\n\n Returns:\n A dictionary of output {feature name}::{tensor_name} -> output tensor.\n \"\"\"\n\n if isinstance(inputs, tuple):\n inputs, targets = inputs\n # Convert targets to tensors.\n for target_feature_name, target_value in targets.items():\n if not isinstance(target_value, torch.Tensor):\n targets[target_feature_name] = torch.from_numpy(target_value)\n else:\n targets[target_feature_name] = target_value\n else:\n targets = None\n\n assert list(inputs.keys()) == self.input_features.keys()\n\n encoder_outputs = self.encode(inputs)\n combiner_outputs = self.combine(encoder_outputs)\n decoder_outputs = self.decode(combiner_outputs, targets, mask)\n\n # Compute the number of used tokens.\n decoder_outputs[USED_TOKENS] = get_used_tokens_for_ecd(inputs, targets)\n return decoder_outputs\n\n def unskip(self):\n for k in self.input_features.keys():\n self.input_features.set(k, self.input_features.get(k).unskip())\n\n def save(self, save_path):\n \"\"\"Saves the model to the given path.\"\"\"\n weights_save_path = os.path.join(save_path, MODEL_WEIGHTS_FILE_NAME)\n torch.save(self.state_dict(), weights_save_path)\n # Ensure the file is fully flushed to disk before any other process reads it\n with open(weights_save_path, \"rb\") as f:\n os.fsync(f.fileno())\n\n def load(self, save_path):\n \"\"\"Loads the model from the given path.\"\"\"\n weights_save_path = os.path.join(save_path, MODEL_WEIGHTS_FILE_NAME)\n device = torch.device(get_torch_device())\n with open_file(weights_save_path, \"rb\") as f:\n state_dict = torch.load(f, map_location=device)\n self.load_state_dict(update_state_dict(state_dict))\n\n def get_args(self):\n \"\"\"Returns init arguments for constructing this model.\"\"\"\n return (\n self.config_obj.input_features.to_list(),\n self.config_obj.combiner.to_dict(),\n self.config_obj.output_features.to_list(),\n self._random_seed,\n )\n\n def get_augmentation_pipelines(self) -> AugmentationPipelines:\n \"\"\"Returns the augmentation pipeline for this model.\"\"\"\n # dictionary to hold any augmentation pipeline\n augmentation_pipelines = {}\n\n # loop through all input features and add their augmentation pipeline to the dictionary\n for input_feature in self.config_obj.input_features:\n # if augmentation was specified for this input feature, add AugmentationPipeline to dictionary\n if input_feature.has_augmentation():\n # use input feature proc_column as key because that is what is used in the Batcher\n augmentation_pipelines[input_feature.proc_column] = self.input_features.get(\n input_feature.name\n ).get_augmentation_pipeline()\n\n return AugmentationPipelines(augmentation_pipelines)\n"
},
{
"path": "ludwig/models/embedder.py",
"content": "from collections.abc import Callable\n\nimport numpy as np\nimport pandas as pd\nimport torch\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import ENCODER_OUTPUT, MODEL_ECD, NAME, PROC_COLUMN, TYPE\nfrom ludwig.features.feature_registries import get_input_type_registry\nfrom ludwig.features.feature_utils import LudwigFeatureDict\nfrom ludwig.models.base import BaseModel\nfrom ludwig.schema.features.base import BaseInputFeatureConfig, FeatureCollection\nfrom ludwig.schema.features.utils import get_input_feature_cls\nfrom ludwig.types import FeatureConfigDict, TrainingSetMetadataDict\nfrom ludwig.utils.batch_size_tuner import BatchSizeEvaluator\nfrom ludwig.utils.dataframe_utils import from_numpy_dataset\nfrom ludwig.utils.misc_utils import get_from_registry\nfrom ludwig.utils.torch_utils import get_torch_device, LudwigModule\n\n\n@DeveloperAPI\nclass Embedder(LudwigModule):\n def __init__(self, feature_configs: list[FeatureConfigDict], metadata: TrainingSetMetadataDict):\n super().__init__()\n\n self.input_features = LudwigFeatureDict()\n\n input_feature_configs = []\n for feature in feature_configs:\n feature_cls = get_from_registry(feature[TYPE], get_input_type_registry())\n\n # TODO(travis): this assumes ECD is the selected model type. The best solution is to change the\n # input params from FeatureConfigDict types to BaseInputFeatureConfig types, which will require a\n # refactor of preprocessing to use the schema, not the dict types.\n feature_obj = get_input_feature_cls(MODEL_ECD, feature[TYPE]).from_dict(feature)\n feature_cls.update_config_with_metadata(feature_obj, metadata[feature[NAME]])\n\n # When running prediction or eval, we need the preprocessing to use the original pretrained\n # weights, which requires unsetting this field. In the future, we could avoid this by plumbing\n # through the saved weights and loading them dynamically after building the model.\n feature_obj.encoder.saved_weights_in_checkpoint = False\n\n input_feature_configs.append(feature_obj)\n\n feature_collection = FeatureCollection[BaseInputFeatureConfig](input_feature_configs)\n try:\n self.input_features.update(BaseModel.build_inputs(input_feature_configs=feature_collection))\n except KeyError as e:\n raise KeyError(\n f\"An input feature has a name that conflicts with a class attribute of torch's ModuleDict: {e}\"\n )\n\n def forward(self, inputs: dict[str, torch.Tensor]):\n encoder_outputs = {}\n for input_feature_name, input_values in inputs.items():\n encoder = self.input_features.get(input_feature_name)\n encoder_output = encoder(input_values)\n encoder_outputs[input_feature_name] = encoder_output[ENCODER_OUTPUT]\n return encoder_outputs\n\n\n@DeveloperAPI\ndef create_embed_batch_size_evaluator(\n features_to_encode: list[FeatureConfigDict], metadata: TrainingSetMetadataDict\n) -> BatchSizeEvaluator:\n class _EmbedBatchSizeEvaluator(BatchSizeEvaluator):\n def __init__(self):\n embedder = Embedder(features_to_encode, metadata)\n self.device = get_torch_device()\n self.embedder = embedder.to(self.device)\n self.embedder.eval()\n\n def step(self, batch_size: int, global_max_sequence_length: int | None = None):\n inputs = {\n input_feature_name: input_feature.create_sample_input(batch_size=batch_size).to(self.device)\n for input_feature_name, input_feature in self.embedder.input_features.items()\n }\n with torch.no_grad():\n self.embedder(inputs)\n\n return _EmbedBatchSizeEvaluator\n\n\n@DeveloperAPI\ndef create_embed_transform_fn(\n features_to_encode: list[FeatureConfigDict], metadata: TrainingSetMetadataDict\n) -> Callable:\n class EmbedTransformFn:\n def __init__(self):\n embedder = Embedder(features_to_encode, metadata)\n self.device = get_torch_device()\n self.embedder = embedder.to(self.device)\n self.embedder.eval()\n\n def __call__(self, df: pd.DataFrame) -> pd.DataFrame:\n batch = _prepare_batch(df, features_to_encode, metadata)\n name_to_proc = {i_feat.feature_name: i_feat.proc_column for i_feat in self.embedder.input_features.values()}\n inputs = {\n i_feat.feature_name: torch.from_numpy(np.array(batch[i_feat.proc_column], copy=True)).to(self.device)\n for i_feat in self.embedder.input_features.values()\n }\n with torch.no_grad():\n encoder_outputs = self.embedder(inputs)\n\n encoded = {name_to_proc[k]: v.detach().cpu().float().numpy() for k, v in encoder_outputs.items()}\n output_df = from_numpy_dataset(encoded)\n\n for c in output_df.columns:\n df[c] = output_df[c]\n\n return df\n\n return EmbedTransformFn\n\n\n# TODO(travis): consolidate with implementation in data/ray.py\ndef _prepare_batch(\n df: pd.DataFrame, features: list[FeatureConfigDict], metadata: TrainingSetMetadataDict\n) -> dict[str, np.ndarray]:\n batch = {}\n for feature in features:\n c = feature[PROC_COLUMN]\n if df[c].values.dtype == \"object\":\n # Ensure columns stacked instead of turned into np.array([np.array, ...], dtype=object) objects\n batch[c] = np.stack(df[c].values)\n else:\n batch[c] = df[c].to_numpy()\n\n for feature in features:\n c = feature[PROC_COLUMN]\n reshape = metadata.get(feature[NAME], {}).get(\"reshape\")\n if reshape is not None:\n batch[c] = batch[c].reshape((-1, *reshape))\n\n return batch\n"
},
{
"path": "ludwig/models/inference.py",
"content": "import logging\nimport os\nfrom typing import Any, TYPE_CHECKING\n\nimport pandas as pd\nimport torch\nfrom torch import nn\n\nfrom ludwig.constants import NAME, POSTPROCESSOR, PREDICTOR, PREPROCESSOR, TYPE\nfrom ludwig.data.postprocessing import convert_dict_to_df\nfrom ludwig.data.preprocessing import load_metadata\nfrom ludwig.features.feature_registries import get_input_type_registry\nfrom ludwig.features.feature_utils import get_module_dict_key_from_name, get_name_from_module_dict_key\nfrom ludwig.globals import MODEL_HYPERPARAMETERS_FILE_NAME, TRAIN_SET_METADATA_FILE_NAME\nfrom ludwig.types import ModelConfigDict, TrainingSetMetadataDict\nfrom ludwig.utils import output_feature_utils\nfrom ludwig.utils.data_utils import load_json, save_json\nfrom ludwig.utils.inference_utils import get_filename_from_stage, to_inference_module_input_from_dataframe\nfrom ludwig.utils.misc_utils import get_from_registry\nfrom ludwig.utils.output_feature_utils import get_feature_name_from_concat_name, get_tensor_name_from_concat_name\nfrom ludwig.utils.torch_utils import DEVICE\nfrom ludwig.utils.types import TorchDevice, TorchscriptPreprocessingInput\n\n# Prevents circular import errors from typing.\nif TYPE_CHECKING:\n from ludwig.models.base import BaseModel\n\nlogger = logging.getLogger(__name__)\n\n\nclass InferenceModule(nn.Module):\n \"\"\"A nn.Module subclass that wraps the inference preprocessor, predictor, and postprocessor.\"\"\"\n\n def __init__(\n self,\n preprocessor: torch.jit.ScriptModule,\n predictor: torch.jit.ScriptModule,\n postprocessor: torch.jit.ScriptModule,\n config: ModelConfigDict | None = None,\n training_set_metadata: TrainingSetMetadataDict | None = None,\n ):\n super().__init__()\n self.preprocessor = preprocessor\n self.predictor = predictor\n self.postprocessor = postprocessor\n self.config = config\n # Do not remove – used by Predibase app\n self.training_set_metadata = training_set_metadata\n\n def preprocessor_forward(self, inputs: dict[str, TorchscriptPreprocessingInput]) -> dict[str, torch.Tensor]:\n \"\"\"Forward pass through the preprocessor.\"\"\"\n return self.preprocessor(inputs)\n\n def predictor_forward(self, preproc_inputs: dict[str, torch.Tensor]) -> dict[str, torch.Tensor]:\n \"\"\"Forward pass through the predictor.\n\n Ensures that the inputs are on the correct device. The outputs are on the same device as self.predictor.\n \"\"\"\n for k, v in preproc_inputs.items():\n preproc_inputs[k] = v.to(self.predictor.device)\n\n with torch.no_grad(): # Ensure model params do not compute gradients\n predictions_flattened = self.predictor(preproc_inputs)\n return predictions_flattened\n\n def postprocessor_forward(self, predictions_flattened: dict[str, torch.Tensor]) -> dict[str, dict[str, Any]]:\n \"\"\"Forward pass through the postprocessor.\"\"\"\n postproc_outputs_flattened: dict[str, Any] = self.postprocessor(predictions_flattened)\n # Turn flat inputs into nested predictions per feature name\n postproc_outputs: dict[str, dict[str, Any]] = _unflatten_dict_by_feature_name(postproc_outputs_flattened)\n return postproc_outputs\n\n def forward(self, inputs: dict[str, TorchscriptPreprocessingInput]) -> dict[str, dict[str, Any]]:\n preproc_inputs: dict[str, torch.Tensor] = self.preprocessor_forward(inputs)\n predictions_flattened: dict[str, torch.Tensor] = self.predictor_forward(preproc_inputs)\n postproc_outputs: dict[str, dict[str, Any]] = self.postprocessor_forward(predictions_flattened)\n return postproc_outputs\n\n @torch.jit.unused\n def predict(self, dataset: pd.DataFrame, return_type: dict | pd.DataFrame = pd.DataFrame) -> pd.DataFrame | dict:\n \"\"\"Predict on a batch of data with an interface similar to LudwigModel.predict.\"\"\"\n inputs = to_inference_module_input_from_dataframe(dataset, self.config, load_paths=True)\n\n preds = self(inputs)\n\n if return_type == pd.DataFrame:\n preds = convert_dict_to_df(preds)\n return preds, None # Second return value is for compatibility with LudwigModel.predict\n\n @torch.jit.unused\n @classmethod\n def from_ludwig_model(\n cls: \"InferenceModule\",\n model: \"BaseModel\",\n config: ModelConfigDict,\n training_set_metadata: TrainingSetMetadataDict,\n device: TorchDevice | None = None,\n ):\n \"\"\"Create an InferenceModule from a trained LudwigModel.\"\"\"\n if device is None:\n logger.info(f'No device specified. Loading using device \"{DEVICE}\".')\n device = DEVICE\n\n stage_to_module = _init_inference_stages_from_ludwig_model(\n model, config, training_set_metadata, device=device, scripted=True\n )\n\n return cls(\n stage_to_module[PREPROCESSOR],\n stage_to_module[PREDICTOR],\n stage_to_module[POSTPROCESSOR],\n config=config,\n training_set_metadata=training_set_metadata,\n )\n\n @torch.jit.unused\n @classmethod\n def from_directory(\n cls: \"InferenceModule\",\n directory: str,\n device: TorchDevice | None = None,\n ):\n \"\"\"Create an InferenceModule from a directory containing a model, config, and training set metadata.\"\"\"\n if device is None:\n logger.info(f'No device specified. Loading using device \"{DEVICE}\".')\n device = DEVICE\n\n stage_to_module = _init_inference_stages_from_directory(directory, device=device)\n\n config_path = os.path.join(directory, MODEL_HYPERPARAMETERS_FILE_NAME)\n config = load_json(config_path) if os.path.exists(config_path) else None\n\n metadata_path = os.path.join(directory, TRAIN_SET_METADATA_FILE_NAME)\n training_set_metadata = load_metadata(metadata_path) if os.path.exists(metadata_path) else None\n\n return cls(\n stage_to_module[PREPROCESSOR],\n stage_to_module[PREDICTOR],\n stage_to_module[POSTPROCESSOR],\n config=config,\n training_set_metadata=training_set_metadata,\n )\n\n\nclass _InferencePreprocessor(nn.Module):\n \"\"\"Wraps preprocessing modules into a single nn.Module.\n\n TODO(geoffrey): Implement torchscript-compatible feature_utils.LudwigFeatureDict to replace\n get_module_dict_key_from_name and get_name_from_module_dict_key usage.\n \"\"\"\n\n def __init__(self, config: ModelConfigDict, training_set_metadata: TrainingSetMetadataDict):\n super().__init__()\n self.preproc_modules = nn.ModuleDict()\n for feature_config in config[\"input_features\"]:\n feature_name = feature_config[NAME]\n feature = get_from_registry(feature_config[TYPE], get_input_type_registry())\n # prevents collisions with reserved keywords\n module_dict_key = get_module_dict_key_from_name(feature_name)\n self.preproc_modules[module_dict_key] = feature.create_preproc_module(training_set_metadata[feature_name])\n\n def forward(self, inputs: dict[str, TorchscriptPreprocessingInput]) -> dict[str, torch.Tensor]:\n preproc_inputs = {}\n for module_dict_key, preproc in self.preproc_modules.items():\n feature_name = get_name_from_module_dict_key(module_dict_key)\n preproc_inputs[feature_name] = preproc(inputs[feature_name])\n return preproc_inputs\n\n\nclass _InferencePredictor(nn.Module):\n \"\"\"Wraps model forward pass + predictions into a single nn.Module.\n\n The forward call of this module returns a flattened dictionary in order to support Triton input/output.\n\n TODO(geoffrey): Implement torchscript-compatible feature_utils.LudwigFeatureDict to replace\n get_module_dict_key_from_name and get_name_from_module_dict_key usage.\n \"\"\"\n\n def __init__(self, model: \"BaseModel\", device: TorchDevice):\n super().__init__()\n self.device = torch.device(device)\n self.model = model.to_torchscript(self.device)\n self.predict_modules = nn.ModuleDict()\n for feature_name, feature in model.output_features.items():\n # prevents collisions with reserved keywords\n module_dict_key = get_module_dict_key_from_name(feature_name)\n self.predict_modules[module_dict_key] = feature.prediction_module.to(device=self.device)\n\n def forward(self, preproc_inputs: dict[str, torch.Tensor]) -> dict[str, torch.Tensor]:\n model_outputs = self.model(preproc_inputs)\n predictions_flattened: dict[str, torch.Tensor] = {}\n for module_dict_key, predict in self.predict_modules.items():\n feature_name = get_name_from_module_dict_key(module_dict_key)\n feature_predictions = predict(model_outputs, feature_name)\n # Flatten out the predictions to support Triton input/output\n for predict_key, tensor_values in feature_predictions.items():\n predict_concat_key = output_feature_utils.get_feature_concat_name(feature_name, predict_key)\n predictions_flattened[predict_concat_key] = tensor_values\n return predictions_flattened\n\n\nclass _InferencePostprocessor(nn.Module):\n \"\"\"Wraps postprocessing modules into a single nn.Module.\n\n The forward call of this module returns a flattened dictionary in order to support Triton input/output.\n\n TODO(geoffrey): Implement torchscript-compatible feature_utils.LudwigFeatureDict to replace\n get_module_dict_key_from_name and get_name_from_module_dict_key usage.\n \"\"\"\n\n def __init__(self, model: \"BaseModel\", training_set_metadata: TrainingSetMetadataDict):\n super().__init__()\n self.postproc_modules = nn.ModuleDict()\n for feature_name, feature in model.output_features.items():\n # prevents collisions with reserved keywords\n module_dict_key = get_module_dict_key_from_name(feature_name)\n self.postproc_modules[module_dict_key] = feature.create_postproc_module(training_set_metadata[feature_name])\n\n def forward(self, predictions_flattened: dict[str, torch.Tensor]) -> dict[str, Any]:\n postproc_outputs_flattened: dict[str, Any] = {}\n for module_dict_key, postproc in self.postproc_modules.items():\n feature_name = get_name_from_module_dict_key(module_dict_key)\n feature_postproc_outputs = postproc(predictions_flattened, feature_name)\n # Flatten out the predictions to support Triton input/output\n for postproc_key, tensor_values in feature_postproc_outputs.items():\n postproc_concat_key = output_feature_utils.get_feature_concat_name(feature_name, postproc_key)\n postproc_outputs_flattened[postproc_concat_key] = tensor_values\n return postproc_outputs_flattened\n\n\ndef save_ludwig_model_for_inference(\n save_path: str,\n model: \"BaseModel\",\n config: ModelConfigDict,\n training_set_metadata: TrainingSetMetadataDict,\n device: TorchDevice | None = None,\n model_only: bool = False,\n) -> None:\n \"\"\"Saves a LudwigModel (a BaseModel model, config, and training_set_metadata) for inference.\"\"\"\n if device is None:\n logger.info(f'No device specified. Saving using device \"{DEVICE}\".')\n device = DEVICE\n\n stage_to_filenames = {\n stage: get_filename_from_stage(stage, device) for stage in [PREPROCESSOR, PREDICTOR, POSTPROCESSOR]\n }\n\n stage_to_module = _init_inference_stages_from_ludwig_model(\n model, config, training_set_metadata, device, scripted=True\n )\n if model_only:\n stage_to_module[PREDICTOR].save(os.path.join(save_path, stage_to_filenames[PREDICTOR]))\n else:\n config_path = os.path.join(save_path, MODEL_HYPERPARAMETERS_FILE_NAME)\n if not os.path.exists(config_path):\n save_json(config_path, config)\n logger.info(f\"Saved model config to {config_path}\")\n\n training_set_metadata_path = os.path.join(save_path, TRAIN_SET_METADATA_FILE_NAME)\n if not os.path.exists(training_set_metadata_path):\n save_json(training_set_metadata_path, training_set_metadata)\n logger.info(f\"Saved training set metadata to {training_set_metadata_path}\")\n\n for stage, module in stage_to_module.items():\n module.save(os.path.join(save_path, stage_to_filenames[stage]))\n logger.info(f\"Saved torchscript module for {stage} to {stage_to_filenames[stage]}.\")\n\n\ndef _init_inference_stages_from_directory(\n directory: str,\n device: TorchDevice,\n) -> dict[str, torch.nn.Module]:\n \"\"\"Initializes inference stage modules from directory.\"\"\"\n stage_to_filenames = {\n stage: get_filename_from_stage(stage, device) for stage in [PREPROCESSOR, PREDICTOR, POSTPROCESSOR]\n }\n\n stage_to_module = {}\n for stage in [PREPROCESSOR, PREDICTOR, POSTPROCESSOR]:\n stage_to_module[stage] = torch.jit.load(os.path.join(directory, stage_to_filenames[stage]))\n print(f\"Loaded torchscript module for {stage} from {stage_to_filenames[stage]}.\")\n return stage_to_module\n\n\ndef _init_inference_stages_from_ludwig_model(\n model: \"BaseModel\",\n config: ModelConfigDict,\n training_set_metadata: TrainingSetMetadataDict,\n device: TorchDevice,\n scripted: bool = True,\n) -> dict[str, torch.nn.Module]:\n \"\"\"Initializes inference stage modules from a LudwigModel (a BaseModel model, config, and\n training_set_metadata).\"\"\"\n preprocessor = _InferencePreprocessor(config, training_set_metadata)\n predictor = _InferencePredictor(model, device=device)\n postprocessor = _InferencePostprocessor(model, training_set_metadata)\n\n stage_to_module = {\n PREPROCESSOR: preprocessor,\n PREDICTOR: predictor,\n POSTPROCESSOR: postprocessor,\n }\n if scripted:\n stage_to_module = {stage: torch.jit.script(module) for stage, module in stage_to_module.items()}\n return stage_to_module\n\n\ndef _unflatten_dict_by_feature_name(flattened_dict: dict[str, Any]) -> dict[str, dict[str, Any]]:\n \"\"\"Convert a flattened dictionary of objects to a nested dictionary of outputs per feature name.\"\"\"\n outputs: dict[str, dict[str, Any]] = {}\n for concat_key, tensor_values in flattened_dict.items():\n feature_name = get_feature_name_from_concat_name(concat_key)\n tensor_name = get_tensor_name_from_concat_name(concat_key)\n feature_outputs: dict[str, Any] = {}\n if feature_name not in outputs:\n outputs[feature_name] = feature_outputs\n else:\n feature_outputs = outputs[feature_name]\n feature_outputs[tensor_name] = tensor_values\n return outputs\n"
},
{
"path": "ludwig/models/llm.py",
"content": "import contextlib\nimport logging\nimport os\nfrom typing import Any\n\nimport numpy as np\nimport torch\nfrom transformers import AutoConfig, GenerationConfig\n\nfrom ludwig.accounting.used_tokens import get_used_tokens_for_llm\nfrom ludwig.constants import IGNORE_INDEX_TOKEN_ID, LOGITS, MODEL_LLM, PREDICTIONS, TEXT, USED_TOKENS\nfrom ludwig.features.base_feature import ModuleWrapper, OutputFeature\nfrom ludwig.features.feature_utils import LudwigFeatureDict\nfrom ludwig.features.text_feature import TextOutputFeature\nfrom ludwig.globals import MODEL_WEIGHTS_FILE_NAME\nfrom ludwig.models.base import BaseModel\nfrom ludwig.modules.training_hooks import NEFTuneHook\nfrom ludwig.schema.features.base import BaseOutputFeatureConfig, FeatureCollection\nfrom ludwig.schema.model_types.llm import LLMModelConfig\nfrom ludwig.utils.augmentation_utils import AugmentationPipelines\nfrom ludwig.utils.data_utils import clear_data_cache\nfrom ludwig.utils.llm_quantization_utils import convert_quantized_linear_to_linear\nfrom ludwig.utils.llm_utils import (\n add_left_padding,\n generate_merged_ids,\n get_context_len,\n get_realigned_target_and_prediction_tensors_for_inference,\n initialize_adapter,\n load_pretrained_from_config,\n pad_target_tensor_for_fine_tuning,\n remove_left_padding,\n to_device,\n)\nfrom ludwig.utils.logging_utils import log_once\nfrom ludwig.utils.output_feature_utils import set_output_feature_tensor\nfrom ludwig.utils.tokenizers import HFTokenizer\nfrom ludwig.utils.torch_utils import reg_loss\n\nlogger = logging.getLogger(__name__)\n\n\nclass DictWrapper:\n \"\"\"Wrapper for a LudwigFeatureDict module that allows for iteration over keys.\n\n The purpose of this class is to avoid exposing input and output features as modules of the LLM. This is because we\n only wish to train the underlying model, and having these additional modules can confuse systems like DeepSpeed.\n \"\"\"\n\n def __init__(self, obj: LudwigFeatureDict):\n self.obj = obj\n\n def get(self, key) -> torch.nn.Module:\n return self.obj.get(key)\n\n def set(self, key: str, module: torch.nn.Module) -> None:\n self.obj.set(key, module)\n\n def __len__(self) -> int:\n return len(self.obj)\n\n def __next__(self) -> None:\n return next(iter(self.obj))\n\n def __iter__(self) -> None:\n return iter(self.obj.keys())\n\n def keys(self) -> list[str]:\n return self.obj.keys()\n\n def values(self) -> list[torch.nn.Module]:\n return self.obj.values()\n\n def items(self) -> list[tuple[str, torch.nn.Module]]:\n return self.obj.items()\n\n def update(self, modules: dict[str, torch.nn.Module]) -> None:\n self.obj.update(modules)\n\n\nclass LLM(BaseModel):\n @staticmethod\n def type() -> str:\n return MODEL_LLM\n\n def __init__(\n self,\n config_obj: LLMModelConfig,\n random_seed=None,\n _device=None,\n **_kwargs,\n ):\n super().__init__(random_seed=random_seed)\n\n self.config_obj = config_obj\n self._random_seed = random_seed\n\n self.model_name = self.config_obj.base_model\n self.model_config = AutoConfig.from_pretrained(\n self.config_obj.base_model,\n trust_remote_code=self.config_obj.trust_remote_code,\n )\n\n self.model = load_pretrained_from_config(self.config_obj, model_config=self.model_config)\n self.curr_device = next(self.model.parameters()).device\n logger.info(\"Done.\")\n\n self.context_len = get_context_len(self.model_config)\n\n # TODO(Arnav): This needs be more flexible to account for RoPE Scaling\n # When merging input IDs and target IDs for LLM fine-tuning, we want to make sure that the merged tensor is\n # not longer than the global maximum sequence length. This is provided in the preprocessing config. We never\n # want to exceed the maximum possible context length so we also check for that.\n if self.config_obj.preprocessing.global_max_sequence_length:\n global_max_sequence_length = self.config_obj.preprocessing.global_max_sequence_length\n self.global_max_sequence_length = (\n global_max_sequence_length if global_max_sequence_length <= self.context_len else self.context_len\n )\n else:\n self.global_max_sequence_length = self.context_len\n\n # Initialize tokenizer\n self.tokenizer = HFTokenizer(\n self.config_obj.base_model,\n trust_remote_code=self.config_obj.trust_remote_code,\n ).tokenizer\n\n self._set_generation_config(self.config_obj.generation.to_dict())\n\n # ================ Inputs ================\n try:\n self.input_features.update(self.build_inputs(input_feature_configs=self.config_obj.input_features))\n except KeyError as e:\n raise KeyError(\n f\"An input feature has a name that conflicts with a class attribute of torch's ModuleDict: {e}\"\n ) from e\n\n # This is used to store the model inputs during the forward pass when fine-tuning LLMs. This allows us to have\n # access to the joint model inputs (input_ids and target_ids) when computing metrics. In particular, the target\n # ids are needed to correctly compute next token softmax cross entropy loss.\n self.model_inputs = None\n\n # ================ Outputs ================\n self.output_feature_type = self.config_obj.output_features[0].type\n\n self.output_features.update(\n self.build_outputs(\n output_feature_configs=self.config_obj.output_features,\n # Set the input size to the model vocab size instead of the tokenizer vocab size\n # because the model has additional \"head\" layers that are used to predict the next\n # token in the sequence. These head layers can add additional dimensions to the\n # logits tensor, beyond the vocab_size dimension.\n input_size=self.input_shape[-1] if self.output_feature_type == TEXT else self.model_config.vocab_size,\n )\n )\n\n # Extract the decoder object for the forward pass\n self._output_feature_decoder = ModuleWrapper(self.output_features.items()[0][1])\n\n self.attention_masks = None\n\n clear_data_cache()\n\n def create_feature_dict(self) -> DictWrapper:\n return DictWrapper(LudwigFeatureDict())\n\n @contextlib.contextmanager\n def use_generation_config(self, generation_config_dict: dict[str, Any] | None = None):\n \"\"\"Sets the generation config for the model.\"\"\"\n # Save the original generation config so that we can reset it if/when we change it when self.generation gets is\n # dynamically mutated during 1-off predict calls after fine-tuning.\n original_generation_config_dict = self.generation.to_dict()\n try:\n # no-op if generation_config is None\n if generation_config_dict is not None:\n # unwrap the original generation config, update it with the new generation config\n new_generation_config_dict = {**original_generation_config_dict, **generation_config_dict}\n self._set_generation_config(new_generation_config_dict)\n yield\n finally:\n self._set_generation_config(original_generation_config_dict)\n\n def _set_generation_config(self, new_generation_config_dict: dict[str, Any]):\n self.generation = GenerationConfig(**new_generation_config_dict)\n # We need to manually set the pad_token_id to the tokenizer's pad_token_id for certain models like GPT and\n # CodeLlama to avoid getting an error. This workaround can be found here:\n # (https://github.com/huggingface/transformers/issues/25353#issuecomment-1669339754)\n self.generation.pad_token_id = self.tokenizer.pad_token_id\n self.max_new_tokens = self.generation.max_new_tokens\n # max input length value copied from FastChat\n # https://github.com/lm-sys/FastChat/blob/0e958b852a14f4bef5f0e9d7a5e7373477329cf2/fastchat/serve/inference.py#L183 # noqa E501\n self.max_input_length = self.context_len - self.max_new_tokens - 8\n\n @property\n def output_feature_decoder(self) -> OutputFeature:\n return self._output_feature_decoder.module\n\n def initialize_adapter(self):\n \"\"\"If an adapter config is provided, we want to wrap the model with a PEFT model for fine-tuning.\"\"\"\n if self.config_obj.adapter:\n if self.config_obj.trainer.type != \"finetune\" and not self.config_obj.adapter.pretrained_adapter_weights:\n raise ValueError(\n \"Adapter config was provided, but trainer type is not set to `finetune`. Either set the trainer to \"\n \"`finetune` or remove the adapter config.\"\n )\n\n self.model = initialize_adapter(self.model, self.config_obj)\n\n logger.info(\"==================================================\")\n logger.info(\"Trainable Parameter Summary For Fine-Tuning\")\n logger.info(f\"Fine-tuning with adapter: {self.config_obj.adapter.type}\")\n self.model.print_trainable_parameters()\n logger.info(\"==================================================\")\n\n def prepare_for_training(self):\n # TODO: this implementation will not work if resuming from a previous checkpoint. Need to fix this.\n if self.config_obj.quantization:\n self.prepare_for_quantized_training()\n self.initialize_adapter()\n\n def prepare_for_quantized_training(self):\n from peft import prepare_model_for_kbit_training\n\n self.model = prepare_model_for_kbit_training(self.model, use_gradient_checkpointing=False)\n\n def to_device(self, device):\n # Always refresh curr_device from actual parameter location, since\n # nn.Module.to() can move parameters without updating curr_device.\n self.curr_device = next(self.model.parameters()).device\n self.model, device = to_device(self.model, device, self.config_obj, self.curr_device)\n self.curr_device = device\n return self\n\n @classmethod\n def build_outputs(\n cls, output_feature_configs: FeatureCollection[BaseOutputFeatureConfig], input_size: int\n ) -> dict[str, OutputFeature]:\n \"\"\"Builds and returns output feature.\"\"\"\n # TODO: only single task currently\n if len(output_feature_configs) > 1:\n raise ValueError(\"The LLM model type only supports a single output feature.\")\n\n output_feature_config = output_feature_configs[0]\n output_feature_config.input_size = input_size\n\n output_features = {}\n output_feature = cls.build_single_output(output_feature_config, output_features)\n output_features[output_feature_config.name] = output_feature\n\n return output_features\n\n def forward(\n self,\n inputs: (\n dict[str, torch.Tensor] | dict[str, np.ndarray] | tuple[dict[str, torch.Tensor], dict[str, torch.Tensor]]\n ),\n mask=None,\n ) -> dict[str, torch.Tensor]:\n \"\"\"Produces logits tensor for finetuning the model.\n\n Args:\n inputs: Inputs to the model. Can be a dictionary of input names to\n input tensors or a tuple of (inputs, targets) where inputs is\n a dictionary of input names to input tensors and targets is a\n dictionary of target names to target tensors.\n mask: A mask for the inputs.\n\n Returns:\n A dictionary of output {feature name}::{tensor_name} -> output tensor.\n \"\"\"\n input_ids, target_ids = self._unpack_inputs(inputs)\n\n # Generate merged input_id, target_id pairs for the model, and create corresponding attention masks\n # We save them as class variables so that we can use them when realigning target and prediction tensors\n self.model_inputs, self.attention_masks = generate_merged_ids(\n input_ids, target_ids, self.tokenizer, self.global_max_sequence_length\n )\n\n # TODO (jeffkinnison): Determine why the 8-bit `SCB` and `CB` matrices are deleted in the forward pass\n model_outputs = self.model(input_ids=self.model_inputs, attention_mask=self.attention_masks).get(LOGITS)\n\n if self.output_feature_type != TEXT:\n # Pass generated tokens through decoder after averaging the token probabilities\n # This is required for the classification head for the classifier decoder\n model_outputs = torch.mean(model_outputs, dim=1)\n\n if self.output_feature_type == TEXT:\n decoder_outputs = model_outputs\n else:\n decoder_outputs = self.output_feature_decoder.decoder_obj(model_outputs)\n\n # Set the output feature tensor to the decoder outputs (logits)\n outputs = {}\n of_name = self.config_obj.output_features[0].name\n set_output_feature_tensor(outputs, of_name, LOGITS, decoder_outputs)\n\n # Get predictions, probabilities and logits tensor from the output feature's predictions function\n outputs = self.output_features.get(of_name).predictions(outputs, of_name)\n\n # Cast to float32 for metric computation incase we're using deespeed with\n # reduced precision such as bfloat16.\n for prediction_key, prediction_tensor in outputs.items():\n if prediction_key != PREDICTIONS:\n # Skipping casting it to float32 since the predictions are tokens and they should be int64\n # (which is already the case)\n outputs[prediction_key] = prediction_tensor.type(torch.float32)\n\n # Add token usage.\n outputs[USED_TOKENS] = get_used_tokens_for_llm(self.model_inputs, self.tokenizer)\n return outputs\n\n def generate(\n self,\n inputs: (\n dict[str, torch.Tensor] | dict[str, np.ndarray] | tuple[dict[str, torch.Tensor], dict[str, torch.Tensor]]\n ),\n mask=None,\n ) -> dict[str, torch.Tensor]:\n \"\"\"Generates tokens using the model.\"\"\"\n log_once(f\"For generating text, using: {self.generation}\")\n input_ids, _ = self._unpack_inputs(inputs)\n\n with torch.no_grad():\n input_lengths = []\n sequences_list = []\n for input_ids_sample in input_ids:\n input_ids_sample_no_padding = remove_left_padding(input_ids_sample, self.tokenizer)\n\n if input_ids_sample_no_padding.shape[1] > self.max_input_length:\n logger.warning(\n f\"Input length {input_ids_sample_no_padding.shape[1]} is \"\n f\"greater than max input length {self.max_input_length}. Truncating.\"\n )\n input_ids_sample_no_padding = input_ids_sample_no_padding[:, -self.max_input_length :] # noqa E203\n\n input_lengths.append(input_ids_sample_no_padding.shape[1])\n\n # Ensure input_ids are on the same device as the model\n model_device = next(self.model.parameters()).device\n input_ids_sample_no_padding = input_ids_sample_no_padding.to(model_device)\n\n # Generate text using the model\n model_outputs = self.model.generate(\n input_ids=input_ids_sample_no_padding,\n attention_mask=mask,\n generation_config=self.generation,\n return_dict_in_generate=True,\n output_scores=True,\n )\n\n sequences_list.append(model_outputs.sequences[0])\n\n # Extract the predictions, probabilities and logits from the model outputs\n # through the forward pass of the output feature\n outputs = self.output_feature_decoder.decoder_obj.forward(\n sequences_list,\n input_lengths,\n self.max_new_tokens,\n )\n\n return outputs\n\n def is_merge_and_unload_set(self) -> bool:\n \"\"\"Check if the \"adapter\" configuration section exists and, if affirmative, that it contains the\n \"postprocessor\" subsection and the \"merge_adapter_into_base_model\" and \"progressbar\" directives.\n\n # Return\n\n :return (bool): whether merge_and_unload should be done.\n \"\"\"\n return (\n self.config_obj.adapter is not None\n and self.config_obj.adapter.postprocessor is not None\n and self.config_obj.adapter.postprocessor.merge_adapter_into_base_model\n )\n\n def merge_and_unload(self, progressbar: bool = False) -> None:\n \"\"\"This method merges the LoRa layers into the base model. This is needed if someone wants to use the base\n model as a standalone model. The implementation calls merge_and_unload() of the underlying LoraModel class\n (in peft).\n\n Args:\n progressbar (bool): whether to show a progressbar indicating the unload and merge process\n \"\"\"\n from peft import LoraModel\n\n if isinstance(self.model.base_model, LoraModel):\n self.model.base_model.merge_and_unload(progressbar=progressbar)\n else:\n raise ValueError(\"This operation requires an LLM model trained with a LoRA adapter.\")\n\n def _unpack_inputs(\n self,\n inputs: (\n dict[str, torch.Tensor] | dict[str, np.ndarray] | tuple[dict[str, torch.Tensor], dict[str, torch.Tensor]]\n ),\n ) -> tuple[torch.Tensor, torch.Tensor | None]:\n \"\"\"Converts input tensors to input ids.\"\"\"\n if isinstance(inputs, tuple):\n inputs, targets = inputs\n # Convert targets to tensors.\n for target_feature_name, target_value in targets.items():\n if not isinstance(target_value, torch.Tensor):\n targets[target_feature_name] = torch.from_numpy(target_value)\n else:\n targets[target_feature_name] = target_value\n else:\n targets = None\n\n assert list(inputs.keys()) == self.input_features.keys()\n\n input_ids = self.get_input_ids(inputs)\n target_ids = self.get_target_ids(targets) if targets else None\n\n return input_ids, target_ids\n\n def get_input_ids(\n self,\n inputs: (\n dict[str, torch.Tensor] | dict[str, np.ndarray] | tuple[dict[str, torch.Tensor], dict[str, torch.Tensor]]\n ),\n ) -> torch.Tensor:\n \"\"\"Returns the input ids for the text feature input.\"\"\"\n return inputs[self.config_obj.input_features[0].name].type(torch.int32)\n\n def get_target_ids(self, outputs: dict[str, torch.Tensor]) -> torch.Tensor:\n \"\"\"Returns the output ids for the text feature output.\"\"\"\n return outputs[self.config_obj.output_features[0].name].type(torch.int32)\n\n def update_metrics(self, targets, predictions):\n \"\"\"Updates the model's metrics given targets and predictions for zero-shot/few-shot.\"\"\"\n for of_name, of_obj in self.output_features.items():\n if isinstance(of_obj, TextOutputFeature):\n # Align the target length with the predictions length to enable text metric evaluation.\n _targets, _predictions = get_realigned_target_and_prediction_tensors_for_inference(\n targets, predictions, of_name, self.tokenizer\n )\n of_obj.update_metrics(_targets[of_name], _predictions[of_name], self.tokenizer)\n else:\n of_obj.update_metrics(targets[of_name], predictions[of_name])\n\n # HACK (Tim): get the device of the targets to transfer self.eval_loss_metric to the same device\n target_device = list(targets.values())[0].device\n\n eval_loss, additional_losses = self.eval_loss(targets, predictions)\n self.eval_loss_metric = self.eval_loss_metric.to(target_device)\n self.eval_loss_metric.update(eval_loss)\n self.eval_additional_losses_metrics.update(additional_losses)\n\n def update_metrics_finetune_llm(self, targets, predictions):\n \"\"\"Updates the model's metrics given targets and predictions for fine-tuning.\"\"\"\n _targets, _predictions = targets, predictions\n for of_name, of_obj in self.output_features.items():\n if isinstance(of_obj, TextOutputFeature):\n # Update the target tensor to enable text metric evaluation. This pads the target tensor with -100s\n # to match the prediction length and depends on how much of the target tensor was included in the\n # forward pass.\n _targets = self._update_target_tensor_for_finetuning(_targets, _predictions, of_name)\n if isinstance(of_obj, TextOutputFeature):\n of_obj.update_metrics(_targets[of_name], _predictions[of_name], self.tokenizer)\n else:\n of_obj.update_metrics(_targets[of_name], _predictions[of_name])\n continue\n\n of_obj.update_metrics(_targets[of_name], _predictions[of_name])\n\n eval_loss, additional_losses = self.eval_loss(_targets, _predictions)\n self.eval_loss_metric.update(eval_loss)\n self.eval_additional_losses_metrics.update(additional_losses)\n\n def train_loss(\n self,\n targets,\n predictions,\n regularization_type: str | None = None,\n regularization_lambda: float | None = None,\n ) -> tuple[torch.Tensor, dict[str, torch.Tensor]]:\n \"\"\"Computes the training loss for the model.\n\n Args:\n targets: A dictionary of target names to target tensors.\n predictions: A dictionary of output names to output tensors.\n regularization_type: One of 'l1', 'l2', 'l1_l2', or None.\n regularization_lambda: The regularization lambda.\n\n Returns:\n A tuple of the loss tensor and a dictionary of loss for every\n output feature.\n \"\"\"\n train_loss = 0\n of_train_losses = {}\n for of_name, of_obj in self.output_features.items():\n _targets, _predictions = targets, predictions\n if isinstance(of_obj, TextOutputFeature):\n _predictions = {of_name: _predictions}\n\n # Update the target tensor to enable text metric evaluation. This pads the target tensor with -100s\n # to match the prediction length and depends on how much of the target tensor was included in the\n # forward pass.\n _targets = self._update_target_tensor_for_finetuning(_targets, _predictions, of_name)\n\n # TODO(Arnav): Seems like doing this again and going between these format types in unnecessary, but\n # refactor so that we don't have to do this at a later point.\n predictions = {}\n for key, _ in _predictions[of_name].items():\n set_output_feature_tensor(predictions, of_name, key, _predictions[of_name][key])\n _predictions = predictions\n\n of_train_loss = of_obj.train_loss(_targets[of_name], _predictions, of_name)\n train_loss += of_obj.loss.weight * of_train_loss\n of_train_losses[of_name] = of_train_loss\n\n additional_losses = self.losses()\n if additional_losses:\n train_loss += torch.sum(torch.stack(additional_losses)) # other losses\n\n # Add regularization loss\n if regularization_type is not None and regularization_lambda != 0:\n train_loss += reg_loss(self, regularization_type, l1=regularization_lambda, l2=regularization_lambda)\n\n return train_loss, of_train_losses\n\n def eval_loss(self, targets, predictions):\n \"\"\"Computes all evaluation losses for the model given targets and predictions.\n\n Args:\n targets: A dictionary of target names to target tensors.\n predictions: A dictionary of output names to output tensors.\n\n Returns:\n A tuple of loss values for eval losses and additional losses.\n \"\"\"\n eval_loss = 0\n for of_name, of_obj in self.output_features.items():\n if isinstance(of_obj, TextOutputFeature):\n # Align the target length with the predictions length to enable text metric evaluation.\n _targets, _predictions = get_realigned_target_and_prediction_tensors_for_inference(\n targets, predictions, of_name, self.tokenizer\n )\n of_eval_loss = of_obj.eval_loss(_targets[of_name], _predictions[of_name])\n else:\n # HACK(geoffrey): we need a non-empty loss, so we just fill it with zeros\n of_eval_loss = torch.tensor(0.0).to(predictions[of_name][LOGITS].device)\n\n eval_loss += of_obj.loss.weight * of_eval_loss\n\n additional_loss = 0\n additional_losses = self.losses()\n if additional_losses:\n additional_loss = torch.sum(torch.stack(additional_losses)) # other losses\n\n return eval_loss, additional_loss\n\n def outputs_to_predictions(self, outputs: dict[str, torch.Tensor]) -> dict[str, dict[str, torch.Tensor]]:\n \"\"\"Returns the model's predictions for each output feature.\"\"\"\n predictions = {}\n for of_name in self.output_features:\n # TODO(travis): this will need to change when we support multiple output features\n predictions[of_name] = outputs\n return predictions\n\n def save(self, save_path):\n \"\"\"Saves the model to the given path.\"\"\"\n # TODO(travis): use the implementation of trainer itself to decide whether to save the model, to\n # avoid this hack\n if self.config_obj.trainer.type != \"none\":\n weights_save_path = os.path.join(save_path, MODEL_WEIGHTS_FILE_NAME)\n # We initialize the model's generation configuration; otherwise, we get a validation error.\n self.model.generation_config = self.generation\n self.model.save_pretrained(weights_save_path)\n else:\n logger.info(\"Skipped saving LLM without weight adjustments.\")\n\n def save_base_model(self, save_path):\n \"\"\"Saves the base LLM model to the given path.\"\"\"\n # TODO: see the \"TODO\" statement from \"LLM.save()\" in this module.\n if self.config_obj.trainer.type != \"none\":\n weights_save_path = os.path.join(save_path, MODEL_WEIGHTS_FILE_NAME)\n self.model.base_model.save_pretrained(weights_save_path)\n # While this class initializes the tokenizer (from the base_model) automatically, and hence does not\n # need to be saved if inference is to be done using LudwigModel.predict(), the rationale for saving the\n # tokenizer to HuggingFace Hub is to provide access to models fine-tuned and persisted to HuggingFace Hub\n # using Ludwig at a later time, with the ability to perform inference, independently of Ludwig itself.\n self.tokenizer.save_pretrained(weights_save_path)\n else:\n logger.info(\"Skipped saving LLM without weight adjustments.\")\n\n def save_dequantized_base_model(self, save_path: str) -> None:\n \"\"\"Upscales quantized weights of a model to fp16 and saves the result in a folder specified by save_path.\n\n Args:\n save_path (str): The path to the folder where the upscaled model weights will be saved.\n\n Returns:\n None\n \"\"\"\n from peft import PeftModel\n\n if isinstance(self.model, PeftModel):\n # Get the base model back by removing all the adapter modules without merging.\n logger.warning(\n \"LLM model is currently wrapped in a PeftModel. Removing the adapter layers and saving the base model.\"\n \"Reload the model via LudwigModel.load() to use your trained adapter layers for inference.\"\n )\n self.model = self.model.unload()\n\n # Dequantize the model weights and cast them to fp16 - replace quantized layers with appropriate\n # linear layers in-place.\n logger.info(\"Upscaling quantized weights to fp16...\")\n convert_quantized_linear_to_linear(self.model)\n logger.info(\"Done.\")\n\n # Remove the quantization configuration from the model\n # The reason we can't delete the quantization config is because it is a property of the model and\n # HF does some weird serialization of the config that causes an error when trying to access `self.model.config`\n # after you try and delete a key from the config: TypeError: Object of type dtype is not JSON serializable.\n self.model.config.quantization_config = {}\n\n # Override properties of the model to indicate that it is no longer quantized.\n # This is also necessary to ensure that the model can be saved, otherwise it will raise an error like\n # \"You are calling `save_pretrained` on a 4-bit converted model. This is currently not supported\"\n # See: https://github.com/huggingface/transformers/blob/0ad4e7e6dad670a7151aaceb1af3c272a3bf73a8/src/transformers/modeling_utils.py#L2054 # noqa\n self.model.is_loaded_in_4bit = False\n self.model.is_loaded_in_8bit = False\n\n # Save the model\n logger.info(f\"Saving upscaled model to {save_path}\")\n self.model.save_pretrained(save_path)\n logger.info(\"Done.\")\n\n # Save the tokenizer\n logger.info(f\"Saving tokenizer to {save_path}\")\n self.tokenizer.save_pretrained(save_path)\n logger.info(\"Done.\")\n\n def load(self, save_path):\n \"\"\"Loads the model from the given path.\"\"\"\n weights_save_path = os.path.join(save_path, MODEL_WEIGHTS_FILE_NAME)\n if self.config_obj.adapter:\n # Check if the saved weights are merged (no adapter_config.json) or adapter-only\n adapter_config_path = os.path.join(weights_save_path, \"adapter_config.json\")\n if os.path.exists(adapter_config_path):\n from peft import PeftModel # noqa\n\n if isinstance(self.model, PeftModel):\n # Unwrap and reload PeftModel\n self.model = self.model.base_model\n\n self.model = PeftModel.from_pretrained(self.model, weights_save_path)\n else:\n # Weights were already merged (merge_and_unload was done before save),\n # so load as a regular pretrained model.\n logger.info(\"Loading merged LoRA weights (no adapter_config.json found).\")\n self.model = load_pretrained_from_config(\n self.config_obj, model_config=self.model_config, weights_save_path=weights_save_path\n )\n elif self.config_obj.trainer.type != \"none\":\n self.model = load_pretrained_from_config(\n self.config_obj, model_config=self.model_config, weights_save_path=weights_save_path\n )\n else:\n logger.info(\"Skipped loading LLM without weight adjustments.\")\n\n def get_args(self):\n \"\"\"Returns init arguments for constructing this model.\"\"\"\n return (\n self.config_obj.input_features.to_list(),\n self.config_obj.output_features.to_list(),\n self._random_seed,\n )\n\n def _update_target_tensor_for_finetuning(\n self, targets: dict[str, torch.Tensor], predictions: dict[str, torch.Tensor], of_name: str\n ) -> dict[str, torch.Tensor]:\n \"\"\"Update target tensor for fine-tuning.\n\n This method removes left padding from target tensors, adds a eos token to the end of the target tensors,\n and pads the target tensors with -100 to ensure equal length for loss computation. It then realigns the\n target tensors with the prediction tensors.\n\n Args:\n targets (Dict[str, torch.Tensor]): A dictionary containing the target tensors.\n predictions (Dict[str, torch.Tensor]): A dictionary containing the predicted tensors.\n of_name (str): The name of the target tensor.\n\n Returns:\n Dict[str, torch.Tensor]: A dictionary containing the updated target tensors aligned with predictions.\n \"\"\"\n # Remove left padding from target tensors since we also do this for the model's forward pass when we\n # concatenate the input_ids with the target_ids. We also need to add the pad token to the end of the\n # target tensors.\n targets_without_padding = []\n lengths = []\n\n eos_token_tensor = torch.tensor([self.tokenizer.eos_token_id])\n for target in targets[of_name]:\n target = remove_left_padding(target, self.tokenizer)[0]\n target = torch.cat([target, eos_token_tensor.to(device=target.device)], dim=-1).unsqueeze(0)\n targets_without_padding.append(target)\n lengths.append(target.shape[1])\n\n # We need all target tensors to have the same length for the loss computation. We pad the target\n # tensors with -100 since we want to negate all tokens that are not target_ids during the softmax\n # cross entropy loss computation. This ensures that the loss is computed only for the target tokens.\n max_length = max(lengths)\n for i, target in enumerate(targets_without_padding):\n targets_without_padding[i] = add_left_padding(\n targets_without_padding[i][0],\n max_length,\n IGNORE_INDEX_TOKEN_ID,\n )\n\n targets[of_name] = torch.stack(targets_without_padding, dim=0).to(\n dtype=targets[of_name].dtype,\n device=targets[of_name].device,\n )\n\n # Re-align target tensors without padding to have equal length before realigning with the prediction\n # tensors. Padding left with -100 to match the length of the target tensor masks the input ids during\n # softmax cross entropy loss computation. This ensures that the loss is computed only for the target\n # token IDs. Examples:\n # BERTLMHead: https://github.com/huggingface/transformers/blob/v4.29.1/src/transformers/models/bert/modeling_bert.py#L1216-L1219 # noqa\n # GPTNeoForCausalLM: https://github.com/huggingface/transformers/blob/v4.29.1/src/transformers/models/gpt_neo/modeling_gpt_neo.py#L736 # noqa\n _targets = pad_target_tensor_for_fine_tuning(targets, predictions, self.model_inputs, of_name)\n\n return _targets\n\n def _activate_forward_hooks(self):\n \"\"\"Activates/registers forward hooks for the model.\"\"\"\n if not self.config_obj.model_parameters:\n return\n\n # Initialize forward hook handles\n if self.config_obj.model_parameters.neftune_noise_alpha:\n self._forward_hook_handles.append(\n NEFTuneHook(neftune_noise_alpha=self.config_obj.model_parameters.neftune_noise_alpha)\n )\n\n # Activate forward hooks iteratively\n for hook in self._forward_hook_handles:\n # Update the model with the forward hooks in place\n self.model = hook.activate_hook(self.model)\n\n @staticmethod\n def get_augmentation_pipelines() -> AugmentationPipelines:\n \"\"\"Returns the augmentation pipeline for this model.\"\"\"\n return AugmentationPipelines({})\n"
},
{
"path": "ludwig/models/predictor.py",
"content": "import logging\nimport os\nimport sys\nfrom abc import ABC, abstractmethod\nfrom collections import defaultdict, OrderedDict\nfrom pprint import pformat\n\nimport numpy as np\nimport pandas as pd\nimport psutil\nimport torch\nfrom torch import nn\n\nfrom ludwig.constants import COMBINED, LAST_HIDDEN, LOGITS, MODEL_ECD, MODEL_LLM\nfrom ludwig.data.dataset.base import Dataset\nfrom ludwig.data.utils import convert_to_dict\nfrom ludwig.distributed.base import DistributedStrategy, LocalStrategy\nfrom ludwig.globals import is_progressbar_disabled, PREDICTIONS_PARQUET_FILE_NAME, TEST_STATISTICS_FILE_NAME\nfrom ludwig.models.base import BaseModel\nfrom ludwig.progress_bar import LudwigProgressBar\nfrom ludwig.utils.data_utils import save_csv, save_json\nfrom ludwig.utils.dataframe_utils import from_numpy_dataset\nfrom ludwig.utils.print_utils import repr_ordered_dict\nfrom ludwig.utils.registry import Registry\nfrom ludwig.utils.strings_utils import make_safe_filename\nfrom ludwig.utils.torch_utils import get_torch_device\n\nEXCLUDE_PRED_SET = {LOGITS, LAST_HIDDEN}\nSKIP_EVAL_METRICS = {\"confusion_matrix\", \"roc_curve\"}\nSTATS_SAMPLE_SIZE = 10000\n\nlogger = logging.getLogger(__name__)\n\n\nclass BasePredictor(ABC):\n @abstractmethod\n def batch_predict(self, dataset, dataset_name=None):\n raise NotImplementedError()\n\n @abstractmethod\n def predict_single(self, batch):\n raise NotImplementedError()\n\n @abstractmethod\n def batch_evaluation(self, dataset, collect_predictions=False, collect_logits=False, dataset_name=None):\n raise NotImplementedError()\n\n @abstractmethod\n def batch_collect_activations(self, layer_names, dataset, bucketing_field=None):\n raise NotImplementedError()\n\n # Remote implementations may override this\n def shutdown(self):\n pass\n\n # Functions needed to treat Trainer as a context manager\n def __enter__(self):\n return self\n\n def __exit__(self, exc_type, exc_val, exc_tb):\n self.shutdown()\n\n\n_predictor_registry = Registry[BasePredictor]()\n\n\ndef register_predictor(model_types: list[str]):\n def wrap(cls):\n for model_type in model_types:\n _predictor_registry[model_type] = cls\n return cls\n\n return wrap\n\n\ndef get_predictor_cls(model_type: str) -> type[BasePredictor]:\n return _predictor_registry[model_type]\n\n\n@register_predictor([MODEL_ECD])\nclass Predictor(BasePredictor):\n \"\"\"Predictor is a class that uses a model to predict and evaluate.\"\"\"\n\n def __init__(\n self,\n dist_model: nn.Module,\n batch_size: int = 128,\n distributed: DistributedStrategy = None,\n report_tqdm_to_ray: bool = False,\n model: BaseModel | None = None,\n remote: bool = False,\n **kwargs,\n ):\n \"\"\"\n :param dist_model: model to use for prediction, post-wrap for distributed training\n :param batch_size: batch size to use for prediction\n :param distributed: distributed strategy to use for prediction\n :param report_tqdm_to_ray: whether to report tqdm progress to Ray\n :param model: Ludwig BaseModel before being wrapped for distributed training.\n Used to call Ludwig helper functions.\n \"\"\"\n model = model or dist_model\n assert isinstance(model, BaseModel)\n\n self._batch_size = batch_size\n self._distributed = distributed if distributed is not None else LocalStrategy()\n self.report_tqdm_to_ray = report_tqdm_to_ray\n\n device = get_torch_device()\n self.device = device\n self.dist_model = dist_model\n self.model = model\n self.model.metrics_to_device(device)\n\n if remote:\n # Only return results from rank 0 to reduce network overhead\n self.batch_predict = self._distributed.return_first(self.batch_predict)\n self.batch_evaluation = self._distributed.return_first(self.batch_evaluation)\n\n def batch_predict(self, dataset: Dataset, dataset_name: str = None, collect_logits: bool = False):\n self.dist_model = self._distributed.to_device(self.dist_model)\n prev_model_training_mode = self.dist_model.training # store previous model training mode\n self.dist_model.eval() # set model to eval mode\n\n with torch.no_grad():\n with dataset.initialize_batcher(self._batch_size, should_shuffle=False) as batcher:\n progress_bar_config = {\n \"desc\": \"Prediction\" if dataset_name is None else f\"Prediction {dataset_name: <5.5}\",\n \"total\": batcher.steps_per_epoch,\n \"file\": sys.stdout,\n \"disable\": is_progressbar_disabled(),\n }\n progress_bar = LudwigProgressBar(self.report_tqdm_to_ray, progress_bar_config, self.is_coordinator())\n predictions = defaultdict(list)\n while not batcher.last_batch():\n batch = batcher.next_batch()\n preds = self._predict(batch)\n self._accumulate_preds(\n preds, predictions, exclude_pred_set={LAST_HIDDEN} if collect_logits else EXCLUDE_PRED_SET\n )\n progress_bar.update(1)\n\n progress_bar.close()\n\n # consolidate predictions from each batch to a single tensor\n self._concat_preds(predictions)\n\n self.dist_model.train(prev_model_training_mode)\n\n return from_numpy_dataset(predictions)\n\n def predict_single(self, batch, collect_logits: bool = False):\n prev_model_training_mode = self.dist_model.training # store previous model training mode\n self.dist_model.eval() # set model to eval mode\n\n with torch.no_grad():\n predictions = defaultdict(list)\n preds = self._predict(batch)\n self._accumulate_preds(\n preds, predictions, exclude_pred_set={LAST_HIDDEN} if collect_logits else EXCLUDE_PRED_SET\n )\n self._concat_preds(predictions)\n\n # reset model to its original training mode\n self.dist_model.train(prev_model_training_mode)\n return from_numpy_dataset(predictions)\n\n def _predict(self, batch: dict[str, np.ndarray]) -> dict[str, np.ndarray]:\n \"\"\"Predict a batch of data.\n\n Params:\n model: BaseModel model\n batch: batch of data\n\n Returns:\n predictions: dictionary of predictions\n \"\"\"\n inputs = {\n i_feat.feature_name: torch.from_numpy(np.array(batch[i_feat.proc_column], copy=True)).to(self.device)\n for i_feat in self.model.input_features.values()\n }\n\n outputs = self._predict_on_inputs(inputs)\n return self.model.outputs_to_predictions(outputs)\n\n def _accumulate_preds(self, preds, predictions, exclude_pred_set=EXCLUDE_PRED_SET):\n # accumulate predictions from batch for each output feature\n for of_name, of_preds in preds.items():\n for pred_name, pred_values in of_preds.items():\n if pred_name not in exclude_pred_set:\n key = f\"{of_name}_{pred_name}\"\n predictions[key].append(pred_values.detach().cpu())\n\n def _concat_preds(self, predictions):\n for key, pred_value_list in predictions.items():\n # Without detaching, a runtime error is raised since pred_value_list\n # is a tensor that requires grad.\n predictions[key] = torch.cat(pred_value_list, dim=0).numpy()\n\n def batch_evaluation(self, dataset, collect_predictions=False, collect_logits=False, dataset_name=None):\n \"\"\"Batch evaluate model on dataset.\n\n Params:\n dataset (Union[str, dict, pandas.DataFrame]): source containing the entire dataset to be evaluated.\n collect_predictions: Return model predictions.\n collect_logits: Return model logits and final layer activations.\n\n Returns:\n Tuple of dictionaries of (metrics, predictions). The keys of metrics are determined by the metrics in the\n model config. The keys of the predictions dictionary depend on which values are requested by the caller:\n collect_predictions, collect_logits.\n \"\"\"\n self.dist_model = self._distributed.to_device(self.dist_model)\n prev_model_training_mode = self.dist_model.training # store previous model training mode\n self.dist_model.eval() # set model to eval mode\n\n with torch.no_grad():\n with dataset.initialize_batcher(\n self._batch_size, should_shuffle=False, distributed=self._distributed\n ) as batcher:\n progress_bar_config = {\n \"desc\": \"Evaluation\" if dataset_name is None else f\"Evaluation {dataset_name: <5.5}\",\n \"total\": batcher.steps_per_epoch,\n \"file\": sys.stdout,\n \"disable\": is_progressbar_disabled(),\n \"position\": 0, # Necessary to disable extra new line artifacts in training logs.\n }\n progress_bar = LudwigProgressBar(self.report_tqdm_to_ray, progress_bar_config, self.is_coordinator())\n\n predictions = defaultdict(list)\n eval_steps = (\n self.dist_model.config_obj.trainer.eval_steps\n if hasattr(self.dist_model, \"config_obj\")\n and hasattr(self.dist_model.config_obj.trainer, \"eval_steps\")\n else None\n )\n eval_steps_counter = 0\n while not batcher.last_batch():\n if eval_steps and eval_steps_counter >= eval_steps:\n logger.info(f\"Reached evaluation step {eval_steps}. Ending evaluation.\")\n break\n batch = batcher.next_batch()\n logger.debug(\n f\"evaluation for {dataset_name}: obtained next batch \"\n f\"memory used: {psutil.Process(os.getpid()).memory_info()[0] / 1e6:0.2f}MB\"\n )\n inputs = {\n i_feat.feature_name: torch.from_numpy(np.array(batch[i_feat.proc_column], copy=True)).to(\n self.device\n )\n for i_feat in self.model.input_features.values()\n }\n targets = {\n o_feat.feature_name: torch.from_numpy(np.array(batch[o_feat.proc_column], copy=True)).to(\n self.device\n )\n for o_feat in self.model.output_features.values()\n }\n\n outputs = self._predict_on_inputs(inputs)\n preds = self.model.outputs_to_predictions(outputs)\n self.model.update_metrics(targets, preds)\n\n # accumulate predictions from batch for each output feature\n if collect_predictions:\n self._accumulate_preds(\n preds, predictions, exclude_pred_set={LAST_HIDDEN} if collect_logits else EXCLUDE_PRED_SET\n )\n\n progress_bar.update(1)\n eval_steps_counter += 1\n if self.is_coordinator():\n logger.debug(\n f\"evaluation for {dataset_name}: completed batch {progress_bar.total_steps} \"\n f\"memory used: {psutil.Process(os.getpid()).memory_info()[0] / 1e6:0.2f}MB\"\n )\n progress_bar.close()\n\n # consolidate predictions from each batch to a single tensor\n if collect_predictions:\n self._concat_preds(predictions)\n\n metrics = self.model.get_metrics()\n self.model.reset_metrics()\n\n self.dist_model.train(prev_model_training_mode) # Restores previous model training mode.\n\n return metrics, from_numpy_dataset(predictions)\n\n def batch_collect_activations(self, layer_names, dataset, bucketing_field=None):\n if bucketing_field:\n raise ValueError(\"BucketedBatcher is not supported yet\")\n\n self.dist_model = self._distributed.to_device(self.dist_model)\n prev_model_training_mode = self.dist_model.training # store previous model training mode\n self.dist_model.eval() # set model to eval mode\n\n with torch.no_grad():\n with dataset.initialize_batcher(\n self._batch_size, should_shuffle=False, distributed=self._distributed\n ) as batcher:\n progress_bar_config = {\n \"desc\": \"Collecting Tensors\",\n \"total\": batcher.steps_per_epoch,\n \"file\": sys.stdout,\n \"disable\": is_progressbar_disabled(),\n }\n progress_bar = LudwigProgressBar(self.report_tqdm_to_ray, progress_bar_config, self.is_coordinator())\n\n collected_tensors = []\n while not batcher.last_batch():\n batch = batcher.next_batch()\n\n inputs = {\n i_feat.feature_name: torch.from_numpy(np.array(batch[i_feat.proc_column], copy=True)).to(\n self.device\n )\n for i_feat in self.model.input_features.values()\n }\n outputs = self._predict_on_inputs(inputs)\n collected_tensors = [(concat_name, tensor) for concat_name, tensor in outputs.items()]\n progress_bar.update(1)\n\n progress_bar.close()\n\n self.dist_model.train(prev_model_training_mode) # Restores previous model training mode.\n\n return collected_tensors\n\n def _predict_on_inputs(self, inputs: dict) -> dict:\n return self.dist_model(inputs)\n\n def is_coordinator(self):\n return self._distributed.rank() == 0\n\n\n@register_predictor([MODEL_LLM])\nclass LlmPredictor(Predictor):\n def _predict_on_inputs(self, inputs: dict) -> dict:\n return self.dist_model.generate(inputs)\n\n\nclass LlmFineTunePredictor(Predictor):\n def batch_evaluation(self, dataset, collect_predictions=False, collect_logits=False, dataset_name=None):\n \"\"\"Batch evaluate model on dataset.\n\n Params:\n dataset (Union[str, dict, pandas.DataFrame]): source containing the entire dataset to be evaluated.\n collect_predictions: Return model predictions.\n collect_logits: Return model logits and final layer activations.\n\n Returns:\n Tuple of dictionaries of (metrics, predictions, input/target/output dictionary). The keys of metrics are\n determined by the metrics in the model config. The keys of the predictions dictionary depend on which values\n are requested by the caller: collect_predictions, collect_logits. The keys of the input/target/output\n dictionary are \"inputs\", \"targets\", and \"outputs\". The values of each of these keys are dictionaries of\n feature names to lists of tensors. The tensors are the inputs, targets, and outputs for each batch.\n \"\"\"\n prev_model_training_mode = self.dist_model.training # store previous model training mode\n self.dist_model.eval() # set model to eval mode\n example_inputs = defaultdict(list)\n example_targets = defaultdict(list)\n example_outputs = defaultdict(list)\n with torch.no_grad():\n with dataset.initialize_batcher(\n self._batch_size, should_shuffle=False, distributed=self._distributed\n ) as batcher:\n progress_bar_config = {\n \"desc\": \"Evaluation\" if dataset_name is None else f\"Evaluation {dataset_name: <5.5}\",\n \"total\": batcher.steps_per_epoch,\n \"file\": sys.stdout,\n \"disable\": is_progressbar_disabled(),\n \"position\": 0, # Necessary to disable extra new line artifacts in training logs.\n }\n progress_bar = LudwigProgressBar(self.report_tqdm_to_ray, progress_bar_config, self.is_coordinator())\n\n predictions = defaultdict(list)\n eval_steps = (\n self.dist_model.config_obj.trainer.eval_steps\n if hasattr(self.dist_model, \"config_obj\")\n and hasattr(self.dist_model.config_obj.trainer, \"eval_steps\")\n else None\n )\n eval_steps_counter = 0\n while not batcher.last_batch():\n if eval_steps and eval_steps_counter >= eval_steps:\n logger.info(f\"Reached evaluation step {eval_steps}. Ending evaluation.\")\n break\n batch = batcher.next_batch()\n logger.debug(\n f\"evaluation for {dataset_name}: obtained next batch \"\n f\"memory used: {psutil.Process(os.getpid()).memory_info()[0] / 1e6:0.2f}MB\"\n )\n inputs = {\n i_feat.feature_name: torch.from_numpy(np.array(batch[i_feat.proc_column], copy=True)).to(\n self.device\n )\n for i_feat in self.model.input_features.values()\n }\n targets = {\n o_feat.feature_name: torch.from_numpy(np.array(batch[o_feat.proc_column], copy=True)).to(\n self.device\n )\n for o_feat in self.model.output_features.values()\n }\n\n outputs = self._predict_on_inputs((inputs, targets))\n preds = self.model.outputs_to_predictions(outputs)\n\n for key in inputs:\n example_inputs[key].extend(inputs[key])\n for key in targets:\n example_targets[key].extend(targets[key])\n for key in preds:\n example_outputs[key].extend(preds[key][\"predictions\"])\n\n # Need to pass through a custom fine-tune metric function because we need to transform\n # the targets into the right format for loss calculation (requires padding with -100s to the left)\n # and other tensor alignment.\n self.model.update_metrics_finetune_llm(targets, preds)\n\n # accumulate predictions from batch for each output feature\n if collect_predictions:\n self._accumulate_preds(\n preds, predictions, exclude_pred_set={LAST_HIDDEN} if collect_logits else EXCLUDE_PRED_SET\n )\n\n progress_bar.update(1)\n eval_steps_counter += 1\n if self.is_coordinator():\n logger.debug(\n f\"evaluation for {dataset_name}: completed batch {progress_bar.total_steps} \"\n f\"memory used: {psutil.Process(os.getpid()).memory_info()[0] / 1e6:0.2f}MB\"\n )\n\n progress_bar.close()\n\n # consolidate predictions from each batch to a single tensor\n if collect_predictions:\n for key, pred_value_list in predictions.items():\n predictions[key] = torch.cat(pred_value_list, dim=0).detach().cpu().numpy()\n\n metrics = self.model.get_metrics()\n self.model.reset_metrics()\n\n input_target_output_dict = {\n \"inputs\": example_inputs,\n \"targets\": example_targets,\n \"outputs\": example_outputs,\n }\n\n self.dist_model.train(prev_model_training_mode) # Restores previous model training mode.\n return metrics, from_numpy_dataset(predictions), input_target_output_dict\n\n\ndef calculate_overall_stats(output_features, predictions, dataset, training_set_metadata):\n overall_stats = {}\n for of_name, output_feature in output_features.items():\n feature_metadata = training_set_metadata[output_feature.feature_name]\n feature_metadata.update(training_set_metadata[output_feature.feature_name])\n\n feature_df = predictions.loc[:, predictions.columns.str.startswith(of_name)]\n feature_df = feature_df.rename(columns=lambda c: c[len(of_name) + 1 :])\n\n target = dataset.loc[:, output_feature.proc_column]\n\n if not isinstance(feature_df, pd.DataFrame):\n logger.warning(\n \"Full computation of stats only supported for pandas dataframes. \"\n \"Sampling the first 10000 rows of the feature and target dataframes for computing overall stats.\"\n )\n feature_df = feature_df.head(n=STATS_SAMPLE_SIZE, npartitions=-1, compute=True)\n target = target.head(n=STATS_SAMPLE_SIZE, npartitions=-1, compute=True)\n\n overall_stats[of_name] = output_feature.calculate_overall_stats(\n feature_df, # predictions\n target,\n feature_metadata, # output feature metadata\n )\n return overall_stats\n\n\ndef save_prediction_outputs(\n postprocessed_output,\n output_features,\n output_directory,\n backend,\n):\n backend.df_engine.write_predictions(\n postprocessed_output, os.path.join(output_directory, PREDICTIONS_PARQUET_FILE_NAME)\n )\n if not backend.df_engine.partitioned:\n # csv can only be written out for unpartitioned df format (i.e., pandas)\n postprocessed_dict = convert_to_dict(postprocessed_output, output_features)\n csv_filename = os.path.join(output_directory, \"{}_{}.csv\")\n for output_field, outputs in postprocessed_dict.items():\n for output_name, values in outputs.items():\n save_csv(csv_filename.format(output_field, make_safe_filename(output_name)), values)\n\n\ndef save_evaluation_stats(test_stats, output_directory):\n test_stats_fn = os.path.join(output_directory, TEST_STATISTICS_FILE_NAME)\n save_json(test_stats_fn, test_stats)\n\n\ndef print_evaluation_stats(test_stats):\n for output_field, result in test_stats.items():\n if output_field != COMBINED or (output_field == COMBINED and len(test_stats) > 2):\n logger.info(f\"\\n===== {output_field} =====\")\n for metric in sorted(list(result)):\n if metric not in SKIP_EVAL_METRICS:\n value = result[metric]\n if isinstance(value, OrderedDict):\n value_repr = repr_ordered_dict(value)\n else:\n value_repr = pformat(result[metric], indent=2)\n logger.info(f\"{metric}: {value_repr}\")\n\n\ndef get_output_columns(output_features, include_logits: bool = False):\n output_columns = []\n for of_name, feature in output_features.items():\n for pred in feature.get_prediction_set():\n if pred not in EXCLUDE_PRED_SET or (pred == LOGITS and include_logits):\n output_columns.append(f\"{of_name}_{pred}\")\n return output_columns\n"
},
{
"path": "ludwig/models/registry.py",
"content": "import logging\n\nfrom ludwig.constants import MODEL_ECD, MODEL_LLM\nfrom ludwig.models.ecd import ECD\nfrom ludwig.models.llm import LLM\n\nlogger = logging.getLogger(__name__)\n\n\nmodel_type_registry = {\n MODEL_ECD: ECD,\n MODEL_LLM: LLM,\n}\n"
},
{
"path": "ludwig/models/retrieval.py",
"content": "import hashlib\nimport json\nimport os\nfrom abc import ABC, abstractmethod\nfrom collections.abc import Callable\nfrom typing import Any, TYPE_CHECKING\n\nimport numpy as np\nimport pandas as pd\nfrom tqdm import tqdm\n\nfrom ludwig.vector_index import FAISS, get_vector_index_cls\nfrom ludwig.vector_index.base import VectorIndex\n\nif TYPE_CHECKING:\n from sentence_transformers import SentenceTransformer\n from ludwig.backend.base import Backend\n\nfrom ludwig.utils.batch_size_tuner import BatchSizeEvaluator\nfrom ludwig.utils.torch_utils import get_torch_device\n\n\ndef df_checksum(df: pd.DataFrame) -> str:\n return hashlib.sha1(pd.util.hash_pandas_object(df).values).hexdigest()\n\n\ndef df_to_row_strs(df: pd.DataFrame) -> list[str]:\n rows = df.to_dict(orient=\"records\")\n row_strs = [json.dumps(r) for r in rows]\n return row_strs\n\n\nclass RetrievalModel(ABC):\n @abstractmethod\n def create_dataset_index(self, df: pd.DataFrame, backend: \"Backend\", columns_to_index: list[str] | None = None):\n \"\"\"Creates an index for the dataset.\n\n If `columns_to_index` is None, all columns are indexed. Otherwise, only the columns in `columns_to_index` are\n used for indexing, but all columns in `df` are returned in the search results.\n \"\"\"\n\n @abstractmethod\n def search(\n self, df, backend: \"Backend\", k: int = 10, return_data: bool = False\n ) -> list[int] | list[dict[str, Any]]:\n \"\"\"Retrieve the top k results for the given query.\n\n If `return_data` is True, returns the data associated with the indices. Otherwise, returns the indices.\n \"\"\"\n\n @abstractmethod\n def save_index(self, name: str, cache_directory: str):\n \"\"\"Saves the index to the cache directory.\"\"\"\n\n @abstractmethod\n def load_index(self, name: str, cache_directory: str):\n \"\"\"Loads the index from the cache directory.\"\"\"\n\n\nclass RandomRetrieval(RetrievalModel):\n \"\"\"Random retrieval model.\n\n Gets k random indices from the dataset regardless of the query.\n \"\"\"\n\n def __init__(self, **kwargs):\n self.index = None\n self.index_data = None\n\n def create_dataset_index(self, df: pd.DataFrame, backend: \"Backend\", columns_to_index: list[str] | None = None):\n self.index = np.array(range(len(df)))\n self.index_data = df\n\n def search(\n self, df, backend: \"Backend\", k: int = 10, return_data: bool = False\n ) -> list[int] | list[dict[str, Any]]:\n results = []\n for _ in tqdm(range(len(df))):\n indices = np.random.choice(self.index, k, replace=False)\n\n if return_data:\n result = self.index_data.iloc[indices].to_dict(orient=\"records\")\n else:\n result = indices\n results.append(result)\n return results\n\n def save_index(self, name: str, cache_directory: str):\n index_file_path = os.path.join(cache_directory, name + \".index\")\n # open file to prevent using the .npy extension\n # https://numpy.org/doc/stable/reference/generated/numpy.save.html\n with open(index_file_path, \"wb\") as f:\n np.save(f, self.index)\n\n index_data_file_path = os.path.join(cache_directory, name + \"_data.csv\")\n self.index_data.to_csv(index_data_file_path, index=False)\n\n def load_index(self, name: str, cache_directory: str):\n index_file_path = os.path.join(cache_directory, name + \".index\")\n self.index = np.load(index_file_path)\n\n index_data_file_path = os.path.join(cache_directory, name + \"_data.csv\")\n self.index_data = pd.read_csv(index_data_file_path)\n\n\nclass SemanticRetrieval(RetrievalModel):\n \"\"\"Semantic retrieval model.\n\n Uses a sentence transformer model to encode the dataset and retrieve the top k most similar results to the query.\n \"\"\"\n\n def __init__(self, model_name, **kwargs):\n self.model_name = model_name\n self.model = get_semantic_retrieval_model(self.model_name)\n self.index: VectorIndex = None\n self.index_data: pd.DataFrame = None\n\n # best batch size computed during the encoding step\n self.best_batch_size = None\n\n def create_dataset_index(self, df: pd.DataFrame, backend: \"Backend\", columns_to_index: list[str] | None = None):\n if columns_to_index is None:\n columns_to_index = df.columns\n df_to_index = df[columns_to_index]\n row_strs = df_to_row_strs(df_to_index)\n\n embeddings = self._encode(row_strs, backend)\n self.index = get_vector_index_cls(FAISS).from_embeddings(embeddings)\n # Save the entire df so we can return the full row when searching\n self.index_data = df\n\n def _encode(self, row_strs: list[str], backend: \"Backend\") -> np.ndarray:\n # only do this step once\n if self.best_batch_size is None:\n self.best_batch_size = backend.tune_batch_size(\n create_semantic_retrieval_model_evaluator(self.model, row_strs), len(row_strs)\n )\n\n transform_fn = create_semantic_retrieval_model_fn(self.model, self.best_batch_size)\n df = backend.df_engine.from_pandas(pd.DataFrame({\"data\": row_strs}))\n df = backend.batch_transform(df, self.best_batch_size, transform_fn)\n df = backend.df_engine.compute(df)\n embeddings = np.stack(df[\"data\"].values).astype(np.float32)\n return embeddings\n\n def search(\n self, df: pd.DataFrame, backend: \"Backend\", k: int = 10, return_data: bool = False\n ) -> list[int] | list[dict[str, Any]]:\n row_strs = df_to_row_strs(df)\n\n query_vectors = self._encode(row_strs, backend)\n results = []\n # TODO(geoffrey): figure out why self.index.search segfaults with larger batch sizes\n for query_vector in tqdm(query_vectors, total=query_vectors.shape[0]):\n indices = self.index.search(query_vector.reshape(1, -1), k)\n if return_data:\n result = self.index_data.iloc[indices].to_dict(orient=\"records\")\n else:\n result = indices\n results.append(result)\n return results\n\n def save_index(self, name: str, cache_directory: str):\n index_file_path = os.path.join(cache_directory, name + \".index\")\n self.index.save(index_file_path)\n\n index_data_file_path = os.path.join(cache_directory, name + \"_data.csv\")\n self.index_data.to_csv(index_data_file_path, index=False)\n\n def load_index(self, name: str, cache_directory: str):\n index_file_path = os.path.join(cache_directory, name + \".index\")\n self.index = get_vector_index_cls(FAISS).from_path(index_file_path)\n\n index_data_file_path = os.path.join(cache_directory, name + \"_data.csv\")\n self.index_data = pd.read_csv(index_data_file_path)\n\n\ndef create_semantic_retrieval_model_evaluator(\n model: \"SentenceTransformer\", samples: list[str]\n) -> type[BatchSizeEvaluator]:\n class _RetrievalModelEvaluator(BatchSizeEvaluator):\n def __init__(self):\n self.model = model.to(get_torch_device())\n self.samples = samples\n\n def step(self, batch_size: int, global_max_sequence_length: int | None = None):\n self.model.encode(self.samples[:batch_size], batch_size=batch_size, show_progress_bar=False)\n\n return _RetrievalModelEvaluator\n\n\ndef create_semantic_retrieval_model_fn(\n model: \"SentenceTransformer\", batch_size: int\n) -> Callable[[pd.DataFrame], np.ndarray]:\n class _RetrievalModelFn:\n def __init__(self):\n self.model = model.to(get_torch_device())\n self.batch_size = batch_size\n\n def __call__(self, df: pd.DataFrame) -> np.ndarray:\n row_strs = df[\"data\"].tolist()\n result = self.model.encode(row_strs, batch_size=self.batch_size, show_progress_bar=False)\n df[\"data\"] = result.tolist()\n return df\n\n return _RetrievalModelFn\n\n\ndef get_semantic_retrieval_model(model_name: str) -> \"SentenceTransformer\":\n from sentence_transformers import SentenceTransformer\n\n return SentenceTransformer(model_name, device=get_torch_device())\n\n\ndef get_retrieval_model(type: str, **kwargs) -> RetrievalModel:\n if type == \"random\":\n return RandomRetrieval(**kwargs)\n elif type == \"semantic\":\n return SemanticRetrieval(**kwargs)\n else:\n raise ValueError(f\"Unsupported retrieval model type: {type}\")\n"
},
{
"path": "ludwig/modules/__init__.py",
"content": ""
},
{
"path": "ludwig/modules/attention_modules.py",
"content": "# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\n\nimport torch\nfrom torch import nn\nfrom torch.nn import functional as F\n\nfrom ludwig.utils.torch_utils import get_activation, LudwigModule\n\nlogger = logging.getLogger(__name__)\n\n\nclass FeedForwardAttentionReducer(LudwigModule):\n def __init__(self, input_size, hidden_size=256, activation=\"tanh\"):\n super().__init__()\n self.fc_layer1 = nn.Linear(input_size, hidden_size)\n self.fc_layer1_activation = get_activation(activation)\n self.fc_layer2 = nn.Linear(hidden_size, 1, bias=False)\n self.input_shape_var = None\n self.output_shape_var = None\n\n def forward(self, inputs, mask=None):\n # current_inputs shape [b, s, h]\n self.input_shape_var = inputs.size()[1:]\n hidden = self.fc_layer1(inputs) # [b, s, h']\n hidden = self.fc_layer1_activation(hidden)\n hidden = self.fc_layer2(hidden) # [b, s, 1]\n attention = F.softmax(hidden, dim=1)\n gated_inputs = torch.sum(attention * inputs, dim=1)\n self.output_shape_var = gated_inputs.size()[1:]\n return gated_inputs # [b, h]\n\n @property\n def input_shape(self) -> torch.Size:\n return self.input_shape_var\n\n @property\n def output_shape(self) -> torch.Size:\n return self.output_shape_var\n\n\nclass MultiHeadSelfAttention(LudwigModule):\n def __init__(self, input_size, hidden_size, num_heads=8):\n super().__init__()\n self.embedding_size = hidden_size\n self.num_heads = num_heads\n if hidden_size % num_heads != 0:\n raise ValueError(\n f\"When using multi-head attention, `hidden_size` ({hidden_size}), should be divisible by \"\n f\"`num_heads` ({num_heads}). Please update the `transformer` section of the model config.\"\n )\n self.projection_dim = hidden_size // num_heads\n self.query_dense = nn.Linear(input_size, hidden_size)\n self.key_dense = nn.Linear(input_size, hidden_size)\n self.value_dense = nn.Linear(input_size, hidden_size)\n self.combine_heads = nn.Linear(hidden_size, hidden_size)\n\n def separate_heads(self, inputs, batch_size):\n inputs = torch.reshape(inputs, (batch_size, -1, self.num_heads, self.projection_dim))\n return torch.permute(inputs, (0, 2, 1, 3))\n\n def forward(self, inputs: torch.Tensor, mask=None):\n # inputs.shape = [batch_size, seq_len, embedding_dim]\n batch_size = inputs.shape[0]\n query = self.query_dense(inputs) # (batch_size, seq_len, h)\n key = self.key_dense(inputs) # (batch_size, seq_len, h)\n value = self.value_dense(inputs) # (batch_size, seq_len, h)\n query = self.separate_heads(query, batch_size) # (batch_size, num_heads, seq_len, projection_dim)\n key = self.separate_heads(key, batch_size) # (batch_size, num_heads, seq_len, projection_dim)\n value = self.separate_heads(value, batch_size) # (batch_size, num_heads, seq_len, projection_dim)\n attn_mask = mask if mask is not None else None\n outputs = F.scaled_dot_product_attention(query, key, value, attn_mask=attn_mask)\n outputs = torch.permute(outputs, (0, 2, 1, 3)) # (batch_size, seq_len, num_heads, projection_dim)\n concat_outputs = torch.reshape(outputs, (batch_size, -1, self.embedding_size)) # (batch_size, seq_len, h)\n projected_outputs = self.combine_heads(concat_outputs) # (batch_size, seq_len, h)\n return projected_outputs\n\n @property\n def output_shape(self):\n return torch.Size([self.embedding_size])\n\n\nclass TransformerBlock(LudwigModule):\n def __init__(\n self,\n input_size: int,\n max_sequence_length: int,\n hidden_size: int,\n num_heads: int,\n output_size: int,\n dropout: float = 0.1,\n ):\n super().__init__()\n self.input_size = input_size\n self.max_sequence_length = max_sequence_length\n self.hidden_size = hidden_size\n\n self.self_attention = MultiHeadSelfAttention(input_size, hidden_size, num_heads=num_heads)\n self.dropout1 = nn.Dropout(dropout)\n self.layernorm1 = nn.LayerNorm(hidden_size, eps=1e-6)\n self.fully_connected = nn.Sequential(\n nn.Linear(input_size, output_size), get_activation(\"relu\"), nn.Linear(output_size, hidden_size)\n )\n self.dropout2 = nn.Dropout(dropout)\n self.layernorm2 = nn.LayerNorm(hidden_size, eps=1e-6)\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length, self.input_size])\n\n def forward(self, inputs, mask=None):\n # inputs [b, s, h]\n attn_output = self.self_attention(inputs) # [b, s, h]\n attn_output = self.dropout1(attn_output) # [b, s, h]\n ln1_output = self.layernorm1(inputs + attn_output) # [b, s, h]\n fc_output = self.fully_connected(ln1_output) # [b, s, h]\n fc_output = self.dropout2(fc_output) # [b, s, h]\n return self.layernorm2(ln1_output + fc_output) # [b, s, h]\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length, self.hidden_size])\n\n\nclass TransformerStack(LudwigModule):\n def __init__(\n self,\n input_size: int,\n max_sequence_length: int,\n hidden_size: int = 256,\n num_heads: int = 8,\n output_size: int = 256,\n num_layers: int = 1,\n dropout: float = 0.1,\n **kwargs,\n ):\n super().__init__()\n self.supports_masking = True\n self.max_sequence_length = max_sequence_length\n self.input_size = input_size\n self.hidden_size = hidden_size\n\n self.layers = nn.ModuleList()\n\n prior_input_size = input_size\n for i in range(num_layers):\n layer = TransformerBlock(\n input_size=prior_input_size,\n max_sequence_length=max_sequence_length,\n hidden_size=hidden_size,\n num_heads=num_heads,\n output_size=output_size,\n dropout=dropout,\n )\n self.layers.append(layer)\n prior_input_size = self.layers[i].output_shape[-1]\n\n for layer in self.layers:\n logger.debug(f\" {layer._get_name()}\")\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length, self.input_size])\n\n def forward(self, inputs, mask=None):\n hidden = inputs\n for layer in self.layers:\n hidden = layer(hidden, mask=mask)\n return hidden\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length, self.hidden_size])\n"
},
{
"path": "ludwig/modules/convolutional_modules.py",
"content": "# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nfrom functools import partial\nfrom typing import Any\n\nimport torch\nimport torch.nn as nn\n\nfrom ludwig.utils.image_utils import get_img_output_shape\nfrom ludwig.utils.torch_utils import get_activation, LudwigModule\n\nlogger = logging.getLogger(__name__)\n\n\nclass Conv1DLayer(LudwigModule):\n def __init__(\n self,\n in_channels=1,\n out_channels=256,\n max_sequence_length=None,\n kernel_size=3,\n strides=1,\n padding=\"same\",\n dilation=1,\n groups=1,\n use_bias=True,\n weights_initializer=\"xavier_uniform\",\n bias_initializer=\"zeros\",\n norm=None,\n norm_params=None,\n activation=\"relu\",\n dropout=0,\n pool_function=\"max\",\n pool_size=2,\n pool_strides=None,\n pool_padding=\"valid\",\n ):\n super().__init__()\n\n self.in_channels = in_channels\n self.out_channels = out_channels\n self.max_sequence_length = max_sequence_length\n self.kernel_size = kernel_size\n self.stride = strides\n self.padding = padding\n self.dilation = dilation\n self.groups = groups\n self.pool_size = pool_size\n if pool_strides is None:\n self.pool_strides = pool_size\n else:\n self.pool_strides = pool_strides\n if pool_padding == \"same\" and pool_size is not None:\n self.pool_padding = (self.pool_size - 1) // 2\n else:\n self.pool_padding = 0\n\n self.layers = nn.ModuleList()\n\n self.layers.append(\n nn.Conv1d(\n in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=(kernel_size,),\n stride=(strides,),\n padding=padding,\n dilation=(dilation,),\n )\n )\n\n if norm and norm_params is None:\n norm_params = {}\n if norm == \"batch\":\n self.layers.append(nn.BatchNorm1d(num_features=out_channels, **norm_params))\n elif norm == \"layer\":\n self.layers.append(nn.LayerNorm(normalized_shape=[out_channels, self.max_sequence_length], **norm_params))\n\n self.layers.append(get_activation(activation))\n\n if dropout > 0:\n self.layers.append(nn.Dropout(dropout))\n\n if pool_size is not None:\n pool = nn.MaxPool1d\n if pool_function in {\"average\", \"avg\", \"mean\"}:\n pool = nn.AvgPool1d\n self.layers.append(pool(kernel_size=self.pool_size, stride=self.pool_strides, padding=self.pool_padding))\n\n for layer in self.layers:\n logger.debug(f\" {layer._get_name()}\")\n\n @property\n def input_shape(self):\n \"\"\"Returns the size of the input tensor without the batch dimension.\"\"\"\n return torch.Size([self.max_sequence_length, self.in_channels])\n\n def forward(self, inputs, training=None, mask=None):\n # inputs: [batch_size, seq_size, in_channels]\n # in Torch nomenclature (N, L, C)\n hidden = inputs\n\n # put in torch compatible form [batch_size, in_channels, seq_size]\n hidden = hidden.transpose(1, 2)\n\n for layer in self.layers:\n hidden = layer(hidden)\n\n # revert back to normal form [batch_size, seq_size, out_channels]\n hidden = hidden.transpose(1, 2)\n\n return hidden # (batch_size, seq_size, out_channels)\n\n\nclass Conv1DStack(LudwigModule):\n def __init__(\n self,\n in_channels=1,\n max_sequence_length=None,\n layers=None,\n num_layers=None,\n default_num_filters=256,\n default_filter_size=3,\n default_strides=1,\n default_padding=\"same\",\n default_dilation_rate=1,\n default_use_bias=True,\n default_weights_initializer=\"xavier_uniform\",\n default_bias_initializer=\"zeros\",\n default_norm=None,\n default_norm_params=None,\n default_activation=\"relu\",\n default_dropout=0,\n default_pool_function=\"max\",\n default_pool_size=2,\n default_pool_strides=None,\n default_pool_padding=\"same\",\n **kwargs,\n ):\n super().__init__()\n\n self.max_sequence_length = max_sequence_length\n self.in_channels = in_channels\n\n if layers is None:\n if num_layers is None:\n self.layers = [\n {\"filter_size\": 7, \"pool_size\": 3},\n {\"filter_size\": 7, \"pool_size\": 3},\n {\"filter_size\": 3, \"pool_size\": None},\n {\"filter_size\": 3, \"pool_size\": None},\n {\"filter_size\": 3, \"pool_size\": None},\n {\"filter_size\": 3, \"pool_size\": 3},\n ]\n else:\n self.layers = []\n for i in range(num_layers):\n self.layers.append(\n {\n \"filter_size\": default_filter_size,\n \"num_filters\": default_num_filters,\n \"pool_size\": default_pool_size,\n \"pool_strides\": default_pool_strides,\n }\n )\n else:\n self.layers = layers\n\n for layer in self.layers:\n if \"num_filters\" not in layer:\n layer[\"num_filters\"] = default_num_filters\n if \"filter_size\" not in layer:\n layer[\"filter_size\"] = default_filter_size\n if \"strides\" not in layer:\n layer[\"strides\"] = default_strides\n if \"padding\" not in layer:\n layer[\"padding\"] = default_padding\n if \"dilation_rate\" not in layer:\n layer[\"dilation_rate\"] = default_dilation_rate\n if \"use_bias\" not in layer:\n layer[\"use_bias\"] = default_use_bias\n if \"weights_initializer\" not in layer:\n layer[\"weights_initializer\"] = default_weights_initializer\n if \"bias_initializer\" not in layer:\n layer[\"bias_initializer\"] = default_bias_initializer\n if \"norm\" not in layer:\n layer[\"norm\"] = default_norm\n if \"norm_params\" not in layer:\n layer[\"norm_params\"] = default_norm_params\n if \"activation\" not in layer:\n layer[\"activation\"] = default_activation\n if \"dropout\" not in layer:\n layer[\"dropout\"] = default_dropout\n if \"pool_function\" not in layer:\n layer[\"pool_function\"] = default_pool_function\n if \"pool_size\" not in layer:\n layer[\"pool_size\"] = default_pool_size\n if \"pool_strides\" not in layer:\n layer[\"pool_strides\"] = default_pool_strides\n if \"pool_padding\" not in layer:\n layer[\"pool_padding\"] = default_pool_padding\n\n self.stack = nn.ModuleList()\n\n prior_layer_channels = in_channels\n l_in = self.max_sequence_length # torch L_in\n for i, layer in enumerate(self.layers):\n logger.debug(f\" stack layer {i}\")\n self.stack.append(\n Conv1DLayer(\n in_channels=prior_layer_channels,\n out_channels=layer[\"num_filters\"],\n max_sequence_length=l_in,\n kernel_size=layer[\"filter_size\"],\n strides=layer[\"strides\"],\n padding=layer[\"padding\"],\n dilation=layer[\"dilation_rate\"],\n use_bias=layer[\"use_bias\"],\n weights_initializer=layer[\"weights_initializer\"],\n bias_initializer=layer[\"bias_initializer\"],\n norm=layer[\"norm\"],\n norm_params=layer[\"norm_params\"],\n activation=layer[\"activation\"],\n dropout=layer[\"dropout\"],\n pool_function=layer[\"pool_function\"],\n pool_size=layer[\"pool_size\"],\n pool_strides=layer[\"pool_strides\"],\n pool_padding=layer[\"pool_padding\"],\n )\n )\n\n # retrieve number of channels from prior layer\n input_shape = self.stack[i].input_shape\n output_shape = self.stack[i].output_shape\n\n logger.debug(f\"{self.__class__.__name__}: \" f\"input_shape {input_shape}, output shape {output_shape}\")\n\n # pass along shape for the input to the next layer\n l_in, prior_layer_channels = output_shape\n\n @property\n def input_shape(self):\n \"\"\"Returns the size of the input tensor without the batch dimension.\"\"\"\n return torch.Size([self.max_sequence_length, self.in_channels])\n\n def forward(self, inputs, mask=None):\n hidden = inputs\n\n # todo: enumerate for debugging, remove after testing\n for i, layer in enumerate(self.stack):\n hidden = layer(hidden)\n\n if hidden.shape[1] == 0:\n raise ValueError(\n \"The output of the conv stack has the second dimension \"\n \"(length of the sequence) equal to 0. \"\n \"This means that the combination of filter_size, padding, \"\n \"stride, pool_size, pool_padding and pool_stride reduces \"\n \"the sequence length more than is possible. \"\n 'Try using \"same\" padding and reducing or eliminating stride '\n \"and pool.\"\n )\n\n return hidden\n\n\nclass ParallelConv1D(LudwigModule):\n def __init__(\n self,\n in_channels=1,\n max_sequence_length=None,\n layers=None,\n default_num_filters=256,\n default_filter_size=3,\n default_strides=1,\n default_padding=\"same\",\n default_dilation_rate=1,\n default_use_bias=True,\n default_weights_initializer=\"xavier_uniform\",\n default_bias_initializer=\"zeros\",\n default_norm=None,\n default_norm_params=None,\n default_activation=\"relu\",\n default_dropout=0,\n default_pool_function=\"max\",\n default_pool_size=None,\n default_pool_strides=None,\n default_pool_padding=\"valid\",\n **kwargs,\n ):\n super().__init__()\n\n self.in_channels = in_channels\n self.max_sequence_length = max_sequence_length\n\n if layers is None:\n self.layers = [{\"filter_size\": 2}, {\"filter_size\": 3}, {\"filter_size\": 4}, {\"filter_size\": 5}]\n else:\n self.layers = layers\n\n for layer in self.layers:\n if \"num_filters\" not in layer:\n layer[\"num_filters\"] = default_num_filters\n if \"filter_size\" not in layer:\n layer[\"filter_size\"] = default_filter_size\n if \"strides\" not in layer:\n layer[\"strides\"] = default_strides\n if \"padding\" not in layer:\n layer[\"padding\"] = default_padding\n if \"dilation_rate\" not in layer:\n layer[\"dilation_rate\"] = default_dilation_rate\n if \"use_bias\" not in layer:\n layer[\"use_bias\"] = default_use_bias\n if \"weights_initializer\" not in layer:\n layer[\"weights_initializer\"] = default_weights_initializer\n if \"bias_initializer\" not in layer:\n layer[\"bias_initializer\"] = default_bias_initializer\n if \"norm\" not in layer:\n layer[\"norm\"] = default_norm\n if \"norm_params\" not in layer:\n layer[\"norm_params\"] = default_norm_params\n if \"activation\" not in layer:\n layer[\"activation\"] = default_activation\n if \"dropout\" not in layer:\n layer[\"dropout\"] = default_dropout\n if \"pool_function\" not in layer:\n layer[\"pool_function\"] = default_pool_function\n if \"pool_size\" not in layer:\n layer[\"pool_size\"] = default_pool_size\n if \"pool_strides\" not in layer:\n layer[\"pool_strides\"] = default_pool_strides\n if \"pool_padding\" not in layer:\n layer[\"pool_padding\"] = default_pool_padding\n\n self.parallel_layers = nn.ModuleList()\n\n for i, layer in enumerate(self.layers):\n logger.debug(f\" parallel layer {i}\")\n self.parallel_layers.append(\n Conv1DLayer(\n in_channels=self.in_channels,\n out_channels=layer[\"num_filters\"],\n max_sequence_length=self.max_sequence_length,\n kernel_size=layer[\"filter_size\"],\n strides=layer[\"strides\"],\n padding=layer[\"padding\"],\n dilation=layer[\"dilation_rate\"],\n use_bias=layer[\"use_bias\"],\n weights_initializer=layer[\"weights_initializer\"],\n bias_initializer=layer[\"bias_initializer\"],\n norm=layer[\"norm\"],\n norm_params=layer[\"norm_params\"],\n activation=layer[\"activation\"],\n dropout=layer[\"dropout\"],\n pool_function=layer[\"pool_function\"],\n pool_size=layer[\"pool_size\"],\n pool_strides=layer[\"pool_strides\"],\n pool_padding=layer[\"pool_padding\"],\n )\n )\n\n logger.debug(\n f\"{self.__class__.__name__} layer {i}, input shape \"\n f\"{self.parallel_layers[i].input_shape}, output shape \"\n f\"{self.parallel_layers[i].output_shape}\"\n )\n\n @property\n def input_shape(self) -> torch.Size:\n \"\"\"Returns the size of the input tensor without the batch dimension.\"\"\"\n return torch.Size([self.max_sequence_length, self.in_channels])\n\n def forward(self, inputs, mask=None):\n # inputs: [batch_size, seq_size, in_channels)\n\n hidden = inputs\n hiddens = []\n\n for layer in self.parallel_layers:\n hiddens.append(layer(hidden))\n hidden = torch.cat(hiddens, 2)\n\n if hidden.shape[1] == 0:\n raise ValueError(\n \"The output of the conv stack has the second dimension \"\n \"(length of the sequence) equal to 0. \"\n \"This means that the combination of filter_size, padding, \"\n \"stride, pool_size, pool_padding and pool_stride reduces \"\n \"the sequence length more than is possible. \"\n 'Try using \"same\" padding and reducing or eliminating stride '\n \"and pool.\"\n )\n\n # (batch_size, seq_size, len(parallel_layers) * out_channels)\n return hidden\n\n\nclass ParallelConv1DStack(LudwigModule):\n def __init__(\n self,\n in_channels=None,\n stacked_layers=None,\n max_sequence_length=None,\n default_num_filters=64,\n default_filter_size=3,\n default_strides=1,\n default_padding=\"same\",\n default_dilation_rate=1,\n default_use_bias=True,\n default_weights_initializer=\"xavier_uniform\",\n default_bias_initializer=\"zeros\",\n default_norm=None,\n default_norm_params=None,\n default_activation=\"relu\",\n default_dropout=0,\n default_pool_function=\"max\",\n default_pool_size=None,\n default_pool_strides=None,\n default_pool_padding=\"valid\",\n **kwargs,\n ):\n super().__init__()\n\n self.max_sequence_length = max_sequence_length\n self.in_channels = in_channels\n\n if stacked_layers is None:\n self.stacked_parallel_layers = [\n [{\"filter_size\": 2}, {\"filter_size\": 3}, {\"filter_size\": 4}, {\"filter_size\": 5}],\n [{\"filter_size\": 2}, {\"filter_size\": 3}, {\"filter_size\": 4}, {\"filter_size\": 5}],\n [{\"filter_size\": 2}, {\"filter_size\": 3}, {\"filter_size\": 4}, {\"filter_size\": 5}],\n ]\n\n else:\n self.stacked_parallel_layers = stacked_layers\n\n for i, parallel_layers in enumerate(self.stacked_parallel_layers):\n for j in range(len(parallel_layers)):\n layer = parallel_layers[j]\n if \"num_filters\" not in layer:\n layer[\"num_filters\"] = default_num_filters\n if \"filter_size\" not in layer:\n layer[\"filter_size\"] = default_filter_size\n if \"strides\" not in layer:\n layer[\"strides\"] = default_strides\n if \"padding\" not in layer:\n layer[\"padding\"] = default_padding\n if \"dilation_rate\" not in layer:\n layer[\"dilation_rate\"] = default_dilation_rate\n if \"use_bias\" not in layer:\n layer[\"use_bias\"] = default_use_bias\n if \"weights_initializer\" not in layer:\n layer[\"weights_initializer\"] = default_weights_initializer\n if \"bias_initializer\" not in layer:\n layer[\"bias_initializer\"] = default_bias_initializer\n if \"norm\" not in layer:\n layer[\"norm\"] = default_norm\n if \"norm_params\" not in layer:\n layer[\"norm_params\"] = default_norm_params\n if \"activation\" not in layer:\n layer[\"activation\"] = default_activation\n if \"dropout\" not in layer:\n layer[\"dropout\"] = default_dropout\n if \"pool_function\" not in layer:\n layer[\"pool_function\"] = default_pool_function\n if \"pool_size\" not in layer:\n if i == len(self.stacked_parallel_layers) - 1:\n layer[\"pool_size\"] = default_pool_size\n else:\n layer[\"pool_size\"] = None\n if \"pool_strides\" not in layer:\n layer[\"pool_strides\"] = default_pool_strides\n if \"pool_padding\" not in layer:\n layer[\"pool_padding\"] = default_pool_padding\n\n self.stack = nn.ModuleList()\n num_channels = self.in_channels\n sequence_length = self.max_sequence_length\n for i, parallel_layers in enumerate(self.stacked_parallel_layers):\n logger.debug(f\" stack layer {i}\")\n self.stack.append(ParallelConv1D(num_channels, sequence_length, layers=parallel_layers))\n\n logger.debug(\n f\"{self.__class__.__name__} layer {i}, input shape \"\n f\"{self.stack[i].input_shape}, output shape \"\n f\"{self.stack[i].output_shape}\"\n )\n\n # set input specification for the layer\n num_channels = self.stack[i].output_shape[1]\n sequence_length = self.stack[i].output_shape[0]\n\n @property\n def input_shape(self):\n \"\"\"Returns the size of the input tensor without the batch dimension.\"\"\"\n return torch.Size([self.max_sequence_length, self.in_channels])\n\n def forward(self, inputs, mask=None):\n hidden = inputs\n\n for layer in self.stack:\n hidden = layer(hidden)\n\n if hidden.shape[2] == 0:\n raise ValueError(\n \"The output of the conv stack has the second dimension \"\n \"(length of the sequence) equal to 0. \"\n \"This means that the combination of filter_size, padding, \"\n \"stride, pool_size, pool_padding and pool_stride is reduces \"\n \"the sequence length more than is possible. \"\n 'Try using \"same\" padding and reducing or eliminating stride '\n \"and pool.\"\n )\n\n return hidden\n\n\nclass Conv2DLayer(LudwigModule):\n def __init__(\n self,\n img_height: int,\n img_width: int,\n in_channels: int,\n out_channels: int = 256,\n kernel_size: int | tuple[int] = 3,\n stride: int | tuple[int] = 1,\n padding: int | tuple[int] | str = \"valid\",\n dilation: int | tuple[int] = 1,\n groups: int = 1,\n use_bias: bool = True,\n padding_mode: str = \"zeros\",\n norm: str | None = None,\n norm_params: dict[str, Any] | None = None,\n activation: str = \"relu\",\n dropout: float = 0,\n pool_function: int = \"max\",\n pool_kernel_size: int | tuple[int] = None,\n pool_stride: int | None = None,\n pool_padding: int | tuple[int] = 0,\n pool_dilation: int | tuple[int] = 1,\n ):\n super().__init__()\n\n self.layers = torch.nn.ModuleList()\n\n self._input_shape = (in_channels, img_height, img_width)\n pool_stride = pool_stride or pool_kernel_size\n\n self.layers.append(\n nn.Conv2d(\n in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=kernel_size,\n stride=stride,\n padding=padding,\n dilation=dilation,\n groups=groups,\n bias=use_bias,\n padding_mode=padding_mode,\n )\n )\n out_height, out_width = get_img_output_shape(img_height, img_width, kernel_size, stride, padding, dilation)\n\n if norm and norm_params is None:\n norm_params = {}\n if norm == \"batch\":\n # Batch norm over channels\n self.layers.append(nn.BatchNorm2d(num_features=out_channels, **norm_params))\n elif norm == \"layer\":\n # Layer norm over image height and width\n self.layers.append(nn.LayerNorm(normalized_shape=(out_height, out_width), **norm_params))\n\n self.layers.append(get_activation(activation))\n\n if dropout > 0:\n self.layers.append(nn.Dropout(dropout))\n\n if pool_kernel_size is not None:\n pool = partial(nn.MaxPool2d, dilation=pool_dilation)\n if pool_function in {\"average\", \"avg\", \"mean\"}:\n pool = nn.AvgPool2d\n self.layers.append(pool(kernel_size=pool_kernel_size, stride=pool_stride, padding=pool_padding))\n out_height, out_width = get_img_output_shape(\n img_height=out_height,\n img_width=out_width,\n kernel_size=pool_kernel_size,\n stride=pool_stride,\n padding=pool_padding,\n dilation=pool_dilation,\n )\n\n for layer in self.layers:\n logger.debug(f\" {layer._get_name()}\")\n\n self._output_shape = (out_channels, out_height, out_width)\n\n def forward(self, inputs):\n hidden = inputs\n\n for layer in self.layers:\n hidden = layer(hidden)\n\n return hidden\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size(self._output_shape)\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size(self._input_shape)\n\n\nclass Conv2DStack(LudwigModule):\n def __init__(\n self,\n img_height: int,\n img_width: int,\n layers: list[dict] | None = None,\n num_layers: int | None = None,\n first_in_channels: int | None = None,\n default_out_channels: int = 256,\n default_kernel_size: int | tuple[int] = 3,\n default_stride: int | tuple[int] = 1,\n default_padding: int | tuple[int] | str = \"valid\",\n default_dilation: int | tuple[int] = 1,\n default_groups: int = 1,\n default_use_bias: bool = True,\n default_padding_mode: str = \"zeros\",\n default_norm: str | None = None,\n default_norm_params: dict[str, Any] | None = None,\n default_activation: str = \"relu\",\n default_dropout: int = 0,\n default_pool_function: int = \"max\",\n default_pool_kernel_size: int | tuple[int] = 2,\n default_pool_stride: int | tuple[int] = None,\n default_pool_padding: int | tuple[int] = 0,\n default_pool_dilation: int | tuple[int] = 1,\n ):\n super().__init__()\n\n # Confirm that all inputs are consistent\n first_in_channels = self._check_in_channels(first_in_channels, layers)\n default_pool_stride = default_pool_stride or default_pool_kernel_size\n if layers is not None and num_layers is not None:\n raise Warning(\"Both layers and num_layers are not None.\" \"Default to using layers.\")\n if (\n first_in_channels is not None\n and layers is not None\n and len(layers) > 0\n and \"in_channels\" in layers[0]\n and layers[0][\"in_channels\"] != first_in_channels\n ):\n raise Warning(\n \"Input channels is set via layers[0]['in_channels'] and first_in_channels.\"\n \"Default to using first_in_channels.\"\n )\n\n self._input_shape = (first_in_channels, img_height, img_width)\n\n if layers is None:\n if num_layers is None:\n self.layers = [\n {\"out_channels\": 32},\n {\"out_channels\": 64},\n ]\n else:\n self.layers = []\n for i in range(num_layers):\n self.layers.append(\n {\n \"kernel_size\": default_kernel_size,\n \"out_channels\": default_out_channels,\n \"pool_kernel_size\": default_pool_kernel_size,\n }\n )\n else:\n self.layers = layers\n\n for layer in self.layers:\n if \"out_channels\" not in layer:\n layer[\"out_channels\"] = default_out_channels\n if \"kernel_size\" not in layer:\n layer[\"kernel_size\"] = default_kernel_size\n if \"stride\" not in layer:\n layer[\"stride\"] = default_stride\n if \"padding\" not in layer:\n layer[\"padding\"] = default_padding\n if \"dilation\" not in layer:\n layer[\"dilation\"] = default_dilation\n if \"groups\" not in layer:\n layer[\"groups\"] = default_groups\n if \"use_bias\" not in layer:\n layer[\"use_bias\"] = default_use_bias\n if \"padding_mode\" not in layer:\n layer[\"padding_mode\"] = default_padding_mode\n if \"norm\" not in layer:\n layer[\"norm\"] = default_norm\n if \"norm_params\" not in layer:\n layer[\"norm_params\"] = default_norm_params\n if \"activation\" not in layer:\n layer[\"activation\"] = default_activation\n if \"dropout\" not in layer:\n layer[\"dropout\"] = default_dropout\n if \"pool_function\" not in layer:\n layer[\"pool_function\"] = default_pool_function\n if \"pool_kernel_size\" not in layer:\n layer[\"pool_kernel_size\"] = default_pool_kernel_size\n if \"pool_stride\" not in layer:\n layer[\"pool_stride\"] = default_pool_stride\n if \"pool_padding\" not in layer:\n layer[\"pool_padding\"] = default_pool_padding\n if \"pool_dilation\" not in layer:\n layer[\"pool_dilation\"] = default_pool_dilation\n\n self.stack = torch.nn.ModuleList()\n\n in_channels = first_in_channels\n for i, layer in enumerate(self.layers):\n logger.debug(f\" stack layer {i}\")\n self.stack.append(\n Conv2DLayer(\n img_height=img_height,\n img_width=img_width,\n in_channels=in_channels,\n out_channels=layer[\"out_channels\"],\n kernel_size=layer[\"kernel_size\"],\n stride=layer[\"stride\"],\n padding=layer[\"padding\"],\n dilation=layer[\"dilation\"],\n groups=layer[\"groups\"],\n use_bias=layer[\"use_bias\"],\n padding_mode=layer[\"padding_mode\"],\n norm=layer[\"norm\"],\n norm_params=layer[\"norm_params\"],\n activation=layer[\"activation\"],\n dropout=layer[\"dropout\"],\n pool_function=layer[\"pool_function\"],\n pool_kernel_size=layer[\"pool_kernel_size\"],\n pool_stride=layer[\"pool_stride\"],\n pool_padding=layer[\"pool_padding\"],\n pool_dilation=layer[\"pool_dilation\"],\n )\n )\n in_channels, img_height, img_width = self.stack[-1].output_shape\n\n self._output_shape = (in_channels, img_height, img_width)\n\n def forward(self, inputs):\n hidden = inputs\n\n for layer in self.stack:\n hidden = layer(hidden)\n\n return hidden\n\n def _check_in_channels(self, first_in_channels: int | None, layers: list[dict] | None) -> None:\n \"\"\"Confirms that in_channels for first layer of the stack exists.\"\"\"\n\n if first_in_channels is not None:\n return first_in_channels\n elif layers is not None and len(layers) > 0 and \"in_channels\" in layers[0]:\n return layers[0][\"in_channels\"]\n raise ValueError(\n \"In_channels for first layer should be specified either via \" \"`first_in_channels` or `layers` arguments.\"\n )\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size(self._output_shape)\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.size(self._input_shape)\n\n\nclass Conv2DLayerFixedPadding(LudwigModule):\n def __init__(\n self,\n img_height: int,\n img_width: int,\n in_channels: int,\n out_channels=256,\n kernel_size=3,\n stride=1,\n dilation=1,\n groups=1,\n use_bias=False,\n ):\n super().__init__()\n\n self.layers = torch.nn.ModuleList()\n self._input_shape = (in_channels, img_height, img_width)\n\n padding = \"same\"\n if stride > 1:\n padding = (kernel_size - 1) // 2\n\n self.layers.append(\n nn.Conv2d(\n in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=kernel_size,\n stride=stride,\n padding=padding,\n dilation=dilation,\n groups=groups,\n bias=use_bias,\n )\n )\n img_height, img_width = get_img_output_shape(\n img_height=img_height,\n img_width=img_width,\n kernel_size=kernel_size,\n stride=stride,\n padding=padding,\n dilation=dilation,\n )\n\n for layer in self.layers:\n logger.debug(f\" {layer._get_name()}\")\n\n self._output_shape = (out_channels, img_height, img_width)\n\n def forward(self, inputs):\n hidden = inputs\n\n for layer in self.layers:\n hidden = layer(hidden)\n\n return hidden\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size(self._input_shape)\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size(self._output_shape)\n\n\nclass ResNetBlock(LudwigModule):\n def __init__(\n self,\n img_height: int,\n img_width: int,\n first_in_channels: int,\n out_channels: int,\n stride: int = 1,\n batch_norm_momentum: float = 0.1,\n batch_norm_epsilon: float = 0.001,\n projection_shortcut: LudwigModule | None = None,\n ):\n \"\"\"Resnet blocks used for ResNet34 and smaller.\n\n stride: A single int specifying the stride of the first convolution.\n The last convolution will have stride of 1.\n \"\"\"\n super().__init__()\n self._input_shape = (first_in_channels, img_height, img_width)\n\n self.conv1 = Conv2DLayerFixedPadding(\n img_height=img_height,\n img_width=img_width,\n in_channels=first_in_channels,\n out_channels=out_channels,\n kernel_size=3,\n stride=stride,\n )\n in_channels, img_height, img_width = self.conv1.output_shape\n self.norm1 = nn.BatchNorm2d(num_features=in_channels, eps=batch_norm_epsilon, momentum=batch_norm_momentum)\n self.relu1 = get_activation(\"relu\")\n\n self.conv2 = Conv2DLayerFixedPadding(\n img_height=img_height,\n img_width=img_width,\n in_channels=out_channels,\n out_channels=out_channels,\n kernel_size=3,\n stride=1,\n )\n self.norm2 = nn.BatchNorm2d(num_features=out_channels, eps=batch_norm_epsilon, momentum=batch_norm_momentum)\n self.relu2 = get_activation(\"relu\")\n\n for layer in [self.conv1, self.norm1, self.relu1, self.conv2, self.norm2, self.relu2]:\n logger.debug(f\" {layer._get_name()}\")\n\n self._output_shape = self.conv2.output_shape\n\n self.projection_shortcut = projection_shortcut\n if self.projection_shortcut is not None and self.projection_shortcut.output_shape != self._output_shape:\n raise ValueError(\n f\"Output shapes of ResnetBlock and projection_shortcut should \"\n f\"match but are {self._output_shape} and \"\n f\"{self.projection_shortcut.output_shape} respectively.\"\n )\n if self.projection_shortcut is None and self._input_shape != self._output_shape:\n self.projection_shortcut = Conv2DLayer(\n img_height=self._input_shape[1],\n img_width=self._input_shape[2],\n in_channels=first_in_channels,\n out_channels=out_channels,\n kernel_size=1,\n stride=stride,\n )\n\n def forward(self, inputs):\n shortcut = inputs\n\n if self.projection_shortcut is not None:\n shortcut = self.projection_shortcut(shortcut)\n\n hidden = self.conv1(inputs)\n hidden = self.norm1(hidden)\n hidden = self.relu1(hidden)\n hidden = self.conv2(hidden)\n hidden = self.norm2(hidden)\n\n return self.relu2(hidden + shortcut)\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size(self._input_shape)\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size(self._output_shape)\n\n\n# TODO(shreya): Combine with ResNetBlock by adding a flag.\nclass ResNetBottleneckBlock(LudwigModule):\n def __init__(\n self,\n img_height: int,\n img_width: int,\n first_in_channels: int,\n out_channels: int,\n stride: int = 1,\n batch_norm_momentum: float = 0.1,\n batch_norm_epsilon: float = 0.001,\n projection_shortcut: LudwigModule | None = None,\n ):\n \"\"\"Resnet bottleneck blocks used for ResNet50 and larger.\n\n stride: A single int specifying the stride of the middle convolution.\n The first and last convolution will have stride of 1.\n \"\"\"\n super().__init__()\n\n self._input_shape = (first_in_channels, img_height, img_width)\n\n self.conv1 = Conv2DLayerFixedPadding(\n img_height=img_height,\n img_width=img_width,\n in_channels=first_in_channels,\n out_channels=out_channels,\n kernel_size=1,\n stride=1,\n )\n in_channels, img_height, img_width = self.conv1.output_shape\n self.norm1 = nn.BatchNorm2d(num_features=in_channels, eps=batch_norm_epsilon, momentum=batch_norm_momentum)\n self.relu1 = get_activation(\"relu\")\n\n self.conv2 = Conv2DLayerFixedPadding(\n img_height=img_height,\n img_width=img_width,\n in_channels=in_channels,\n out_channels=out_channels,\n kernel_size=3,\n stride=stride,\n )\n in_channels, img_height, img_width = self.conv2.output_shape\n self.norm2 = nn.BatchNorm2d(num_features=in_channels, eps=batch_norm_epsilon, momentum=batch_norm_momentum)\n self.relu2 = get_activation(\"relu\")\n\n self.conv3 = Conv2DLayerFixedPadding(\n img_height=img_height,\n img_width=img_width,\n in_channels=in_channels,\n out_channels=4 * out_channels,\n kernel_size=1,\n stride=1,\n )\n self.norm3 = nn.BatchNorm2d(num_features=4 * out_channels, eps=batch_norm_epsilon, momentum=batch_norm_momentum)\n self.relu3 = get_activation(\"relu\")\n\n for layer in [\n self.conv1,\n self.norm1,\n self.relu1,\n self.conv2,\n self.norm2,\n self.relu2,\n self.conv3,\n self.norm3,\n self.relu3,\n ]:\n logger.debug(f\" {layer._get_name()}\")\n\n self._output_shape = self.conv3.output_shape\n\n self.projection_shortcut = projection_shortcut\n if self.projection_shortcut is not None and self.projection_shortcut.output_shape != self._output_shape:\n raise ValueError(\n f\"Output shapes of ResnetBlock and projection_shortcut should \"\n f\"match but are {self._output_shape} and \"\n f\"{self.projection_shortcut.output_shape} respectively.\"\n )\n if self.projection_shortcut is None and self._input_shape != self._output_shape:\n self.projection_shortcut = Conv2DLayer(\n img_height=self._input_shape[1],\n img_width=self._input_shape[2],\n in_channels=first_in_channels,\n out_channels=4 * out_channels,\n kernel_size=1,\n stride=stride,\n )\n\n def forward(self, inputs):\n shortcut = inputs\n\n if self.projection_shortcut is not None:\n shortcut = self.projection_shortcut(shortcut)\n\n hidden = self.conv1(inputs)\n hidden = self.norm1(hidden)\n hidden = self.relu1(hidden)\n hidden = self.conv2(hidden)\n hidden = self.norm2(hidden)\n hidden = self.relu2(hidden)\n hidden = self.conv3(hidden)\n hidden = self.norm3(hidden)\n\n return self.relu3(hidden + shortcut)\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size(self._output_shape)\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size(self._input_shape)\n\n\nclass ResNetBlockLayer(LudwigModule):\n def __init__(\n self,\n img_height: int,\n img_width: int,\n first_in_channels: int,\n out_channels: int,\n is_bottleneck: bool,\n block_fn: ResNetBlock | ResNetBottleneckBlock,\n num_blocks: int,\n stride: int | tuple[int] = 1,\n batch_norm_momentum: float = 0.1,\n batch_norm_epsilon: float = 0.001,\n ):\n super().__init__()\n\n self._input_shape = (first_in_channels, img_height, img_width)\n\n # Bottleneck blocks end with 4x the number of channels as they start with\n projection_out_channels = out_channels * 4 if is_bottleneck else out_channels\n projection_shortcut = Conv2DLayerFixedPadding(\n img_height=img_height,\n img_width=img_width,\n in_channels=first_in_channels,\n out_channels=projection_out_channels,\n kernel_size=1,\n stride=stride,\n )\n\n self.layers = torch.nn.ModuleList(\n [\n block_fn(\n img_height,\n img_width,\n first_in_channels,\n out_channels,\n stride,\n batch_norm_momentum,\n batch_norm_epsilon,\n projection_shortcut,\n )\n ]\n )\n in_channels, img_height, img_width = self.layers[-1].output_shape\n\n for _ in range(1, num_blocks):\n self.layers.append(\n block_fn(\n img_height=img_height,\n img_width=img_width,\n first_in_channels=in_channels,\n out_channels=out_channels,\n stride=1,\n batch_norm_momentum=batch_norm_momentum,\n batch_norm_epsilon=batch_norm_epsilon,\n )\n )\n in_channels, img_height, img_width = self.layers[-1].output_shape\n\n for layer in self.layers:\n logger.debug(f\" {layer._get_name()}\")\n\n self._output_shape = (in_channels, img_height, img_width)\n\n def forward(self, inputs):\n hidden = inputs\n for layer in self.layers:\n hidden = layer(hidden)\n return hidden\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size(self._output_shape)\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size(self._input_shape)\n\n\nclass ResNet(LudwigModule):\n def __init__(\n self,\n img_height: int,\n img_width: int,\n first_in_channels: int,\n out_channels: int,\n resnet_size: int = 34,\n kernel_size: int | tuple[int] = 7,\n conv_stride: int | tuple[int] = 2,\n first_pool_kernel_size: int | tuple[int] = 3,\n first_pool_stride: int | tuple[int] = 2,\n block_sizes: list[int] = None,\n block_strides: list[int | tuple[int]] = None,\n batch_norm_momentum: float = 0.1,\n batch_norm_epsilon: float = 0.001,\n ):\n \"\"\"Creates a model obtaining an image representation.\n\n Implements ResNet v2:\n Identity Mappings in Deep Residual Networks\n https://arxiv.org/pdf/1603.05027.pdf\n by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, Jul 2016.\n\n Args:\n resnet_size: A single integer for the size of the ResNet model.\n is_bottleneck: Use regular blocks or bottleneck blocks.\n out_channels: The number of filters to use for the first block layer\n of the model. This number is then doubled for each subsequent block\n layer.\n kernel_size: The kernel size to use for convolution.\n conv_stride: stride size for the initial convolutional layer\n first_pool_kernel_size: Pool size to be used for the first pooling layer.\n If none, the first pooling layer is skipped.\n first_pool_stride: stride size for the first pooling layer. Not used\n if first_pool_kernel_size is None.\n block_sizes: A list containing n values, where n is the number of sets of\n block layers desired. Each value should be the number of blocks in the\n i-th set.\n block_strides: List of integers representing the desired stride size for\n each of the sets of block layers. Should be same length as block_sizes.\n Raises:\n ValueError: if invalid version is selected.\n \"\"\"\n super().__init__()\n\n self._input_shape = (first_in_channels, img_height, img_width)\n\n is_bottleneck = self.get_is_bottleneck(resnet_size, block_sizes)\n block_class = self.get_block_fn(is_bottleneck)\n block_sizes, block_strides = self.get_blocks(resnet_size, block_sizes, block_strides)\n\n self.layers = torch.nn.ModuleList()\n self.layers.append(\n Conv2DLayerFixedPadding(\n img_height=img_height,\n img_width=img_width,\n in_channels=first_in_channels,\n out_channels=out_channels,\n kernel_size=kernel_size,\n stride=conv_stride,\n )\n )\n in_channels, img_height, img_width = self.layers[-1].output_shape\n self.layers.append(\n nn.BatchNorm2d(num_features=out_channels, eps=batch_norm_epsilon, momentum=batch_norm_momentum)\n )\n self.layers.append(get_activation(\"relu\"))\n\n if first_pool_kernel_size:\n self.layers.append(nn.MaxPool2d(kernel_size=first_pool_kernel_size, stride=first_pool_stride, padding=1))\n img_height, img_width = get_img_output_shape(\n img_height=img_height,\n img_width=img_width,\n kernel_size=first_pool_kernel_size,\n stride=first_pool_stride,\n padding=1,\n dilation=1,\n )\n\n for i, num_blocks in enumerate(block_sizes):\n self.layers.append(\n ResNetBlockLayer(\n img_height=img_height,\n img_width=img_width,\n first_in_channels=in_channels,\n out_channels=out_channels,\n is_bottleneck=is_bottleneck,\n block_fn=block_class,\n num_blocks=num_blocks,\n stride=block_strides[i],\n batch_norm_momentum=batch_norm_momentum,\n batch_norm_epsilon=batch_norm_epsilon,\n )\n )\n out_channels *= 2\n in_channels, img_height, img_width = self.layers[-1].output_shape\n\n for layer in self.layers:\n logger.debug(f\" {layer._get_name()}\")\n\n self._output_shape = (in_channels, img_height, img_width)\n\n def get_is_bottleneck(self, resnet_size: int, block_sizes: list[int]) -> bool:\n if (resnet_size is not None and resnet_size >= 50) or (block_sizes is not None and sum(block_sizes) >= 16):\n return True\n return False\n\n def get_block_fn(self, is_bottleneck: bool) -> ResNetBlock | ResNetBottleneckBlock:\n if is_bottleneck:\n return ResNetBottleneckBlock\n return ResNetBlock\n\n def get_blocks(self, resnet_size: int, block_sizes: list[int], block_strides: list[int]) -> tuple[list[int]]:\n if block_sizes is None:\n block_sizes = get_resnet_block_sizes(resnet_size)\n if block_strides is None:\n block_strides = [1] + [2 for _ in range(len(block_sizes) - 1)]\n return block_sizes, block_strides\n\n def forward(self, inputs: torch.Tensor) -> torch.Tensor:\n hidden = inputs\n for layer in self.layers:\n hidden = layer(hidden)\n\n return hidden\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size(self._output_shape)\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size(self._input_shape)\n\n\n################################################################################\n# The following code for ResNet is adapted from the TensorFlow implementation\n# https://github.com/tensorflow/models/blob/master/official/resnet/resnet_model.py\n################################################################################\n\n################################################################################\n# Convenience functions for building the ResNet model.\n################################################################################\nresnet_choices = {\n 8: [1, 2, 2],\n 14: [1, 2, 2],\n 18: [2, 2, 2, 2],\n 34: [3, 4, 6, 3],\n 50: [3, 4, 6, 3],\n 101: [3, 4, 23, 3],\n 152: [3, 8, 36, 3],\n 200: [3, 24, 36, 3],\n}\n\n\ndef get_resnet_block_sizes(resnet_size):\n \"\"\"Retrieve the size of each block_layer in the ResNet model.\n\n The number of block layers used for the Resnet model varies according\n to the size of the model. This helper grabs the layer set we want, throwing\n an error if a non-standard size has been selected.\n Args:\n resnet_size: The number of convolutional layers needed in the model.\n Returns:\n A list of block sizes to use in building the model.\n Raises:\n KeyError: if invalid resnet_size is received.\n \"\"\"\n try:\n return resnet_choices[resnet_size]\n except KeyError:\n err = \"Could not find layers for selected Resnet size.\\n\" \"Size received: {}; sizes allowed: {}.\".format(\n resnet_size, resnet_choices.keys()\n )\n raise ValueError(err)\n\n\nclass UNetDoubleConvLayer(LudwigModule):\n def __init__(\n self,\n img_height: int,\n img_width: int,\n in_channels: int,\n out_channels: int,\n norm: str = None,\n ):\n \"\"\"Two Conv2d layers, each followed by a ReLU, used for U-Net.\n\n Args:\n img_height: the input image height\n img_width: the input image width\n in_channels: the number of input channels\n out_channels: the number of output channels\n norm: the normalization to be applied\n \"\"\"\n super().__init__()\n\n self.layers = nn.ModuleList()\n\n self.layers.append(nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1))\n if norm == \"batch\":\n self.layers.append(nn.BatchNorm2d(out_channels))\n self.layers.append(nn.ReLU())\n\n self.layers.append(nn.Conv2d(out_channels, out_channels, kernel_size=3, padding=1))\n if norm == \"batch\":\n self.layers.append(nn.BatchNorm2d(out_channels))\n self.layers.append(nn.ReLU())\n\n self._input_shape = (in_channels, img_height, img_width)\n self._output_shape = (out_channels, img_height, img_width)\n\n def forward(self, inputs):\n hidden = inputs\n\n for layer in self.layers:\n hidden = layer(hidden)\n\n return hidden\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size(self._output_shape)\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size(self._input_shape)\n\n\nclass UNetDownStack(LudwigModule):\n def __init__(\n self,\n img_height: int,\n img_width: int,\n in_channels: int,\n norm: str = None,\n stack_depth: int = 4,\n ):\n \"\"\"Creates the contracting downsampling path of a U-Net stack.\n\n Implements\n U-Net: Convolutional Networks for Biomedical Image Segmentation\n https://arxiv.org/abs/1505.04597\n by Olaf Ronneberger, Philipp Fischer, Thomas Brox, May 2015.\n\n Args:\n img_height: the input image height\n img_width: the input image width\n in_channels: the number of input channels\n norm: the normalization to be applied\n stack_depth: the depth of the unet stack\n \"\"\"\n super().__init__()\n\n self.conv_layers = nn.ModuleList()\n self.down_layers = nn.ModuleList()\n\n height = img_height\n width = img_width\n in_c = in_channels\n out_c = 64\n\n self._input_shape = (in_c, height, width)\n\n for i in range(stack_depth):\n self.conv_layers.append(UNetDoubleConvLayer(height, width, in_c, out_c, norm))\n in_c = out_c\n out_c = out_c * 2\n\n self.down_layers.append(nn.MaxPool2d(kernel_size=2, stride=2))\n height = height // 2\n width = width // 2\n\n self.bottleneck = UNetDoubleConvLayer(height, width, in_c, out_c, norm)\n\n self._output_shape = (out_c, height, width)\n\n def forward(self, inputs):\n skips = [] # skip connections\n hidden = inputs\n\n for conv_layer, down_layer in zip(self.conv_layers, self.down_layers):\n hidden = conv_layer(hidden)\n skips.append(hidden)\n hidden = down_layer(hidden)\n\n hidden = self.bottleneck(hidden)\n return hidden, skips\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size(self._output_shape)\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size(self._input_shape)\n\n\nclass UNetUpStack(LudwigModule):\n def __init__(\n self,\n img_height: int,\n img_width: int,\n out_channels: int,\n norm: str = None,\n stack_depth: int = 4,\n ):\n \"\"\"Creates the expansive upsampling path of a U-Net stack.\n\n Implements\n U-Net: Convolutional Networks for Biomedical Image Segmentation\n https://arxiv.org/abs/1505.04597\n by Olaf Ronneberger, Philipp Fischer, Thomas Brox, May 2015.\n\n Args:\n img_height: the output image height\n img_width: the output image width\n out_channels: the number of output classes\n norm: the normalization to be applied\n stack_depth: the depth of the unet stack\n \"\"\"\n super().__init__()\n\n self.conv_layers = nn.ModuleList()\n self.up_layers = nn.ModuleList()\n\n height = img_height >> stack_depth\n width = img_width >> stack_depth\n in_c = 64 << stack_depth\n out_c = in_c // 2\n\n self._input_shape = (in_c, height, width)\n\n for i in range(stack_depth):\n self.up_layers.append(nn.ConvTranspose2d(in_c, out_c, kernel_size=2, stride=2))\n height = height * 2\n width = width * 2\n\n self.conv_layers.append(UNetDoubleConvLayer(height, width, out_c * 2, out_c, norm))\n in_c = out_c\n out_c = out_c // 2\n\n self.last_conv = nn.Conv2d(in_c, out_channels, kernel_size=1, padding=0)\n\n self._output_shape = (out_channels, img_height, img_width)\n\n def forward(self, inputs, skips):\n hidden = inputs\n\n for conv_layer, up_layer in zip(self.conv_layers, self.up_layers):\n hidden = up_layer(hidden)\n skip = skips.pop()\n hidden = torch.cat([hidden, skip], axis=1)\n hidden = conv_layer(hidden)\n\n hidden = self.last_conv(hidden)\n return hidden\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size(self._output_shape)\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size(self._input_shape)\n"
},
{
"path": "ludwig/modules/embedding_modules.py",
"content": "# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\n\nimport torch\nfrom torch import nn\n\nfrom ludwig.constants import TYPE\nfrom ludwig.modules.initializer_modules import get_initializer\nfrom ludwig.utils.data_utils import load_pretrained_embeddings\nfrom ludwig.utils.torch_utils import get_torch_device, LudwigModule\n\nlogger = logging.getLogger(__name__)\n\nDEVICE = get_torch_device()\n\n\ndef embedding_matrix(\n vocab: list[str],\n embedding_size: int,\n representation: str = \"dense\",\n embeddings_trainable: bool = True,\n pretrained_embeddings: str | None = None,\n force_embedding_size: bool = False,\n embedding_initializer: str | dict | None = None,\n) -> tuple[nn.Module, int]:\n \"\"\"Returns initialized torch.nn.Embedding module and embedding size.\"\"\"\n\n vocab_size = len(vocab)\n if representation == \"dense\":\n if pretrained_embeddings:\n embeddings_matrix = load_pretrained_embeddings(pretrained_embeddings, vocab)\n if embeddings_matrix.shape[-1] != embedding_size:\n if not force_embedding_size:\n embedding_size = embeddings_matrix.shape[-1]\n logger.info(f\"Setting embedding size to be equal to {embeddings_matrix.shape[-1]}.\")\n else:\n raise ValueError(\n f\"The size of the pretrained embeddings is \"\n f\"{embeddings_matrix.shape[-1]}, but the specified \"\n f\"embedding_size is {embedding_size}. Please change \"\n f\"the embedding_size accordingly.\"\n )\n embedding_initializer_obj = torch.tensor(embeddings_matrix, dtype=torch.float32)\n\n else:\n if vocab_size < embedding_size and not force_embedding_size:\n logger.info(\n f\" embedding_size ({embedding_size}) is greater than \"\n f\"vocab_size ({vocab_size}). Setting embedding size to be \"\n f\"equal to vocab_size.\"\n )\n embedding_size = vocab_size\n\n if embedding_initializer is not None:\n embedding_initializer_obj_ref = get_initializer(embedding_initializer)\n else:\n embedding_initializer_obj_ref = get_initializer({TYPE: \"uniform\", \"a\": -1.0, \"b\": 1.0})\n embedding_initializer_obj = embedding_initializer_obj_ref([vocab_size, embedding_size])\n\n embeddings = embedding_initializer_obj\n\n elif representation == \"sparse\":\n embedding_size = vocab_size\n embeddings = get_initializer(\"identity\")([vocab_size, embedding_size])\n embeddings.requires_grad = False\n else:\n raise Exception(f\"Embedding representation {representation} not supported.\")\n\n embeddings = nn.Embedding.from_pretrained(embeddings, freeze=not embeddings_trainable)\n return embeddings, embedding_size\n\n\ndef embedding_matrix_on_device(\n vocab: list[str],\n embedding_size: int,\n representation: str = \"dense\",\n embeddings_trainable: bool = True,\n pretrained_embeddings: str | None = None,\n force_embedding_size: bool = False,\n embeddings_on_cpu: bool = False,\n embedding_initializer: str | None = None,\n) -> tuple[nn.Module, int]:\n embeddings, embedding_size = embedding_matrix(\n vocab,\n embedding_size,\n representation=representation,\n embeddings_trainable=embeddings_trainable,\n pretrained_embeddings=pretrained_embeddings,\n force_embedding_size=force_embedding_size,\n embedding_initializer=embedding_initializer,\n )\n if embeddings_on_cpu:\n embeddings.to(\"cpu\")\n elif not embeddings_on_cpu and torch.cuda.is_available():\n embeddings.to(device=\"cuda\")\n\n return embeddings, embedding_size\n\n\nclass Embed(LudwigModule):\n \"\"\"Module to embed Category, Date, and H3 data types.\"\"\"\n\n def __init__(\n self,\n vocab: list[str],\n embedding_size: int,\n representation: str = \"dense\",\n embeddings_trainable: bool = True,\n pretrained_embeddings: str | None = None,\n force_embedding_size: bool = False,\n embeddings_on_cpu: bool = False,\n dropout: float = 0.0,\n embedding_initializer: str | dict | None = None,\n ):\n super().__init__()\n self.supports_masking = True\n\n self.vocab_size = len(vocab)\n self.embeddings, self.embedding_size = embedding_matrix_on_device(\n vocab,\n embedding_size,\n representation=representation,\n embeddings_trainable=embeddings_trainable,\n pretrained_embeddings=pretrained_embeddings,\n force_embedding_size=force_embedding_size,\n embeddings_on_cpu=embeddings_on_cpu,\n embedding_initializer=embedding_initializer,\n )\n\n if dropout > 0:\n self.dropout = torch.nn.Dropout(p=dropout)\n else:\n self.dropout = None\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> torch.Tensor:\n if inputs.ndim != 2 or inputs.shape[1] != 1:\n raise RuntimeError(\n f\"Embed only takes inputs of shape [batch x 1]. Received inputs with size: {inputs.size()}\"\n )\n embedded = self.embeddings(inputs.long())\n embedded = torch.squeeze(embedded, dim=1)\n if self.dropout:\n embedded = self.dropout(embedded)\n return embedded\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([1])\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.embedding_size])\n\n\nclass EmbedSet(LudwigModule):\n \"\"\"Module to embed Set data types, works on multi-hot encoded input.\"\"\"\n\n def __init__(\n self,\n vocab: list[str],\n embedding_size: int,\n representation: str = \"dense\",\n embeddings_trainable: bool = True,\n pretrained_embeddings: str | None = None,\n force_embedding_size: bool = False,\n embeddings_on_cpu: bool = False,\n dropout: float = 0.0,\n embedding_initializer: str | dict | None = None,\n aggregation_function: str = \"sum\",\n ):\n super().__init__()\n self.supports_masking = True\n\n self.vocab_size = len(vocab)\n self.embeddings, self.embedding_size = embedding_matrix_on_device(\n vocab,\n embedding_size,\n representation=representation,\n embeddings_trainable=embeddings_trainable,\n pretrained_embeddings=pretrained_embeddings,\n force_embedding_size=force_embedding_size,\n embeddings_on_cpu=embeddings_on_cpu,\n embedding_initializer=embedding_initializer,\n )\n\n if dropout > 0:\n self.dropout = torch.nn.Dropout(p=dropout)\n else:\n self.dropout = None\n\n if aggregation_function == \"sum\":\n self.aggregation_function = torch.sum\n elif aggregation_function == \"avg\":\n self.aggregation_function = torch.mean\n else:\n raise ValueError(f\"Unsupported aggregation function {aggregation_function}\")\n\n self.register_buffer(\"vocab_indices\", torch.arange(self.vocab_size))\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> torch.Tensor:\n \"\"\"\n Params:\n inputs: Boolean multi-hot tensor of size [batch x vocab_size], where\n inputs[b, i] indicates that token i is present in sample b.\n \"\"\"\n # Convert multi-hot input to input of indices\n inputs = inputs.int() * self.vocab_indices\n embedded = self.embeddings(inputs.long())\n # Mask out the 0th embedding\n mask = torch.unsqueeze(inputs, -1)\n embedded = embedded * mask\n # Sum over all positive tokens\n embedded = self.aggregation_function(embedded, dim=1)\n if self.dropout:\n embedded = self.dropout(embedded)\n return embedded\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.vocab_size])\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.embedding_size])\n\n @property\n def input_dtype(self):\n return torch.bool\n\n\nclass EmbedWeighted(LudwigModule):\n \"\"\"Module to embed Bag data type, works on input of token frequencies.\"\"\"\n\n def __init__(\n self,\n vocab: list[str],\n embedding_size: int,\n representation: str = \"dense\",\n embeddings_trainable: bool = True,\n pretrained_embeddings: str | None = None,\n force_embedding_size: bool = False,\n embeddings_on_cpu: bool = False,\n dropout: float = 0.0,\n embedding_initializer: str | None = None,\n ):\n super().__init__()\n\n self.embeddings, self.embedding_size = embedding_matrix_on_device(\n vocab,\n embedding_size,\n representation=representation,\n embeddings_trainable=embeddings_trainable,\n pretrained_embeddings=pretrained_embeddings,\n force_embedding_size=force_embedding_size,\n embeddings_on_cpu=embeddings_on_cpu,\n embedding_initializer=embedding_initializer,\n )\n self.vocab_size = len(vocab)\n\n if dropout > 0:\n self.dropout = nn.Dropout(dropout)\n else:\n self.dropout = None\n\n self.register_buffer(\"vocab_indices\", torch.arange(self.vocab_size, dtype=torch.int32))\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None) -> torch.Tensor:\n \"\"\"\n Params:\n inputs: Tensor of frequencies, where inputs[b, i] represents\n frequency of token i in sample b of batch.\n \"\"\"\n # Convert to multi-hot input\n signed_input = (inputs != 0).type(torch.int32)\n multiple_hot_indexes = signed_input * self.vocab_indices\n embedded = self.embeddings(multiple_hot_indexes)\n # Mask out the 0th embedding\n mask = torch.unsqueeze(inputs, -1)\n weighted_embedded = embedded * mask\n # Sum over the all the positive indices\n embedded_reduced = torch.sum(weighted_embedded, dim=1)\n if self.dropout:\n embedded_reduced = self.dropout(embedded_reduced)\n return embedded_reduced\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.vocab_size])\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.embedding_size])\n\n\nclass EmbedSequence(LudwigModule):\n def __init__(\n self,\n vocab: list[str],\n embedding_size: int,\n max_sequence_length: int,\n representation: str = \"dense\",\n embeddings_trainable: bool = True,\n pretrained_embeddings: str | None = None,\n force_embedding_size: bool = False,\n embeddings_on_cpu: bool = False,\n dropout: float = 0.0,\n embedding_initializer: str | None = None,\n ):\n super().__init__()\n self.supports_masking = True\n\n self.vocab_size = len(vocab)\n self.max_sequence_length = max_sequence_length\n self.embeddings, self.embedding_size = embedding_matrix_on_device(\n vocab,\n embedding_size,\n representation=representation,\n embeddings_trainable=embeddings_trainable,\n pretrained_embeddings=pretrained_embeddings,\n force_embedding_size=force_embedding_size,\n embeddings_on_cpu=embeddings_on_cpu,\n embedding_initializer=embedding_initializer,\n )\n\n if dropout > 0:\n self.dropout = nn.Dropout(dropout)\n else:\n self.dropout = None\n\n def forward(self, inputs: torch.Tensor, mask: torch.Tensor | None = None):\n if inputs.dtype not in [torch.int, torch.long]:\n raise RuntimeError(\n f\"Expected tensor of type torch.int or torch.long as input.\" f\"Received {inputs.dtype} instead.\"\n )\n\n embedded = self.embeddings(inputs)\n if self.dropout:\n embedded = self.dropout(embedded)\n return embedded\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length, self.embedding_size])\n\n\nclass TokenAndPositionEmbedding(LudwigModule):\n def __init__(\n self,\n max_sequence_length,\n vocab,\n embedding_size,\n representation=\"dense\",\n embeddings_trainable=True,\n pretrained_embeddings=None,\n force_embedding_size=False,\n embeddings_on_cpu=False,\n dropout=0.0,\n embedding_initializer=None,\n ):\n super().__init__()\n self.max_sequence_length = max_sequence_length\n self.embedding_size = embedding_size\n self.token_embed = EmbedSequence(\n vocab=vocab,\n embedding_size=embedding_size,\n max_sequence_length=max_sequence_length,\n representation=representation,\n embeddings_trainable=embeddings_trainable,\n pretrained_embeddings=pretrained_embeddings,\n force_embedding_size=force_embedding_size,\n embeddings_on_cpu=embeddings_on_cpu,\n dropout=dropout,\n embedding_initializer=embedding_initializer,\n )\n self.position_embed = nn.Embedding(\n num_embeddings=max_sequence_length, embedding_dim=self.token_embed.embedding_size\n )\n self.register_buffer(\"positions\", torch.arange(0, max_sequence_length))\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.max_sequence_length])\n\n @property\n def output_shape(self) -> torch.Size:\n return self.token_embed.output_shape\n\n def forward(self, inputs, mask: torch.Tensor | None = None):\n positions_hidden = self.position_embed(self.positions)\n token_hidden = self.token_embed(inputs)\n return token_hidden + positions_hidden\n"
},
{
"path": "ludwig/modules/fully_connected_modules.py",
"content": "# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nfrom copy import deepcopy\n\nimport torch\nfrom torch.nn import Dropout, Linear, ModuleList\n\nfrom ludwig.modules.normalization_modules import create_norm_layer\nfrom ludwig.utils.torch_utils import activations, initializer_registry, LudwigModule\n\nlogger = logging.getLogger(__name__)\n\n\nclass FCLayer(LudwigModule):\n \"\"\"A torch.nn.Linear wrapper that declares input and output shapes, and enables the customization of:\n\n 1. how weights and biases are initialized\n 2. normalization (layer and batch)\n 3. activations\n 4. dropout\n \"\"\"\n\n def __init__(\n self,\n input_size: int,\n input_rank: int = 2,\n output_size: int = 256,\n use_bias: bool = True,\n weights_initializer: str = \"xavier_uniform\",\n bias_initializer: str = \"zeros\",\n norm: str | None = None,\n norm_params: dict | None = None,\n activation: str = \"relu\",\n dropout: float = 0,\n ):\n super().__init__()\n\n self.layers = ModuleList()\n self.input_size = input_size\n self.output_size = output_size\n\n fc = Linear(in_features=input_size, out_features=output_size, bias=use_bias)\n self.layers.append(fc)\n\n weights_initializer = initializer_registry[weights_initializer]\n weights_initializer(fc.weight)\n\n if use_bias:\n bias_initializer = initializer_registry[bias_initializer]\n bias_initializer(fc.bias)\n\n if norm is not None:\n norm_params = norm_params or {}\n self.layers.append(create_norm_layer(norm, input_rank, output_size, **norm_params))\n\n # Dict for activation objects in pytorch?\n self.layers.append(activations[activation]())\n\n if dropout > 0:\n self.layers.append(Dropout(dropout))\n\n def forward(self, inputs, mask=None):\n hidden = inputs\n for layer in self.layers:\n hidden = layer(hidden)\n\n return hidden\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.input_size])\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.output_size])\n\n\nclass FCStack(LudwigModule):\n \"\"\"A stack of FCLayers.\n\n The specification of each FCLayer is specified by the `layers` dictionary parameter, whose keys correspond with an\n FCLayer's constructor arguments, i.e.\n\n [\n {\"input_size\": 2, \"output_size\": 4},\n {\"output_size\": 4, \"use_bias\": False},\n ]\n\n `default_*` parameters dictate default values to use for each FCLayer, if not specified by `layers`. If `layers` is\n `None`, then a stack of size `num_layers` of `FCLayer`s configured with all of the `default_*` parameters is used.\n\n If `layers` is None and `num_layers` is 0, then there are no fully connected layers and this module serves as a\n trivial passthrough.\n \"\"\"\n\n def __init__(\n self,\n first_layer_input_size: int,\n layers: list[dict] | None = None,\n num_layers: int = 1,\n default_input_rank: int = 2,\n default_output_size: int = 256,\n default_use_bias: bool = True,\n default_weights_initializer: str = \"xavier_uniform\",\n default_bias_initializer: str = \"zeros\",\n default_norm: str | None = None,\n default_norm_params: dict | None = None,\n default_activation: str = \"relu\",\n default_dropout: float = 0,\n residual: bool = False,\n **kwargs,\n ):\n super().__init__()\n self.input_size = first_layer_input_size\n\n self.norm_layer = None\n if default_norm is not None:\n norm_params = default_norm_params or {}\n self.norm_layer = create_norm_layer(default_norm, default_input_rank, self.input_size, **norm_params)\n\n self.dropout = None\n if default_dropout > 0:\n self.dropout = torch.nn.Dropout(default_dropout)\n\n if layers is None:\n self.layers = []\n for i in range(num_layers):\n self.layers.append({})\n else:\n # deep copy the layer definitions so that we don't modify the original\n self.layers = deepcopy(layers)\n\n if len(self.layers) > 0 and \"input_size\" not in self.layers[0]:\n self.layers[0][\"input_size\"] = first_layer_input_size\n for i, layer in enumerate(self.layers):\n if i != 0:\n layer[\"input_size\"] = self.layers[i - 1][\"output_size\"]\n if \"input_rank\" not in layer:\n layer[\"input_rank\"] = default_input_rank\n if \"output_size\" not in layer:\n layer[\"output_size\"] = default_output_size\n if \"use_bias\" not in layer:\n layer[\"use_bias\"] = default_use_bias\n if \"weights_initializer\" not in layer:\n layer[\"weights_initializer\"] = default_weights_initializer\n if \"bias_initializer\" not in layer:\n layer[\"bias_initializer\"] = default_bias_initializer\n if \"norm\" not in layer:\n layer[\"norm\"] = default_norm\n if \"norm_params\" not in layer:\n layer[\"norm_params\"] = default_norm_params\n if \"activation\" not in layer:\n layer[\"activation\"] = default_activation\n if \"dropout\" not in layer:\n layer[\"dropout\"] = default_dropout\n\n self.stack = ModuleList()\n\n for i, layer in enumerate(self.layers):\n self.stack.append(\n FCLayer(\n input_size=layer[\"input_size\"],\n input_rank=layer[\"input_rank\"],\n output_size=layer[\"output_size\"],\n use_bias=layer[\"use_bias\"],\n weights_initializer=layer[\"weights_initializer\"],\n bias_initializer=layer[\"bias_initializer\"],\n norm=layer[\"norm\"],\n norm_params=layer[\"norm_params\"],\n activation=layer[\"activation\"],\n dropout=layer[\"dropout\"],\n )\n )\n self.residual = residual\n\n def forward(self, inputs, mask=None):\n hidden = inputs\n\n if self.norm_layer is not None:\n hidden = self.norm_layer(hidden)\n\n if self.dropout is not None:\n hidden = self.dropout(hidden)\n\n prev_fc_layer_size = self.input_size\n for layer in self.stack:\n out = layer(hidden)\n if self.residual and layer.output_size == prev_fc_layer_size:\n hidden = hidden + out\n else:\n hidden = out\n prev_fc_layer_size = layer.layers[0].out_features\n return hidden\n\n @property\n def num_layers(self) -> int:\n return len(self.layers)\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.input_size])\n\n @property\n def output_shape(self) -> torch.Size:\n if len(self.stack) > 0:\n return self.stack[-1].output_shape\n return torch.Size([self.input_size])\n"
},
{
"path": "ludwig/modules/initializer_modules.py",
"content": "# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nimport torch\n\nfrom ludwig.constants import TYPE\nfrom ludwig.utils.misc_utils import get_from_registry\nfrom ludwig.utils.torch_utils import initializer_registry\n\n\ndef _create_and_init(init_fn, init_kwargs, *args, **kwargs):\n t = torch.empty(*args, **kwargs)\n init_fn(t, **init_kwargs)\n return t\n\n\ndef get_initializer(parameters):\n if parameters is None:\n return lambda *args, **kwargs: _create_and_init(initializer_registry[parameters], {}, *args, **kwargs)\n elif isinstance(parameters, str):\n initializer_fun = get_from_registry(parameters, initializer_registry)\n return lambda *args, **kwargs: _create_and_init(initializer_fun, {}, *args, **kwargs)\n elif isinstance(parameters, dict):\n initializer_fun = get_from_registry(parameters[TYPE], initializer_registry)\n init_kwargs = parameters.copy()\n del init_kwargs[TYPE]\n return lambda *args, **kwargs: _create_and_init(initializer_fun, init_kwargs, *args, **kwargs)\n else:\n raise ValueError(\n f\"Initializers parameters should be either strings or dictionaries, \"\n f\"but the provided parameters are a {type(parameters)}. \"\n f\"Parameters values: {parameters}\"\n )\n"
},
{
"path": "ludwig/modules/loss_implementations/__init__.py",
"content": ""
},
{
"path": "ludwig/modules/loss_implementations/corn.py",
"content": "# Source: https://github.com/Raschka-research-group/coral-pytorch/blob/main/coral_pytorch/losses.py\n# Sebastian Raschka 2020-2021\n# coral_pytorch\n# Author: Sebastian Raschka \n#\n# License: MIT\n\nimport torch\nimport torch.nn.functional as F\n\n\ndef corn_loss(logits, y_train, num_classes):\n \"\"\"Computes the CORN loss described in our forthcoming 'Deep Neural Networks for Rank Consistent Ordinal\n Regression based on Conditional Probabilities' manuscript.\n\n Parameters\n ----------\n logits : torch.tensor, shape=(num_examples, num_classes-1)\n Outputs of the CORN layer.\n\n y_train : torch.tensor, shape=(num_examples)\n Torch tensor containing the class labels.\n\n num_classes : int\n Number of unique class labels (class labels should start at 0).\n\n Returns\n ----------\n loss : torch.tensor\n A torch.tensor containing a single loss value.\n\n Examples\n ----------\n >>> # Consider 8 training examples\n >>> _ = torch.manual_seed(123)\n >>> X_train = torch.rand(8, 99)\n >>> y_train = torch.tensor([0, 1, 2, 2, 2, 3, 4, 4])\n >>> NUM_CLASSES = 5\n >>> #\n >>> #\n >>> # def __init__(self):\n >>> corn_net = torch.nn.Linear(99, NUM_CLASSES-1)\n >>> #\n >>> #\n >>> # def forward(self, X_train):\n >>> logits = corn_net(X_train)\n >>> logits.shape\n torch.Size([8, 4])\n >>> corn_loss(logits, y_train, NUM_CLASSES)\n tensor(0.7127, grad_fn=)\n \"\"\"\n sets = []\n for i in range(num_classes - 1):\n label_mask = y_train > i - 1\n label_tensor = (y_train[label_mask] > i).to(torch.int64)\n sets.append((label_mask, label_tensor))\n\n num_examples = 0\n losses = 0.0\n for task_index, s in enumerate(sets):\n train_examples = s[0]\n train_labels = s[1]\n\n if len(train_labels) < 1:\n continue\n\n num_examples += len(train_labels)\n pred = logits[train_examples, task_index]\n\n loss = -torch.sum(F.logsigmoid(pred) * train_labels + (F.logsigmoid(pred) - pred) * (1 - train_labels))\n losses += loss\n\n return losses / num_examples\n"
},
{
"path": "ludwig/modules/loss_modules.py",
"content": "# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nimport torch\nfrom torch import nn, Tensor\nfrom torch.nn import HuberLoss as _HuberLoss\nfrom torch.nn import L1Loss\nfrom torch.nn import MSELoss as _MSELoss\nfrom torchmetrics.functional import mean_absolute_percentage_error\n\nimport ludwig.utils.loss_utils as utils\nfrom ludwig.constants import LOGITS\nfrom ludwig.modules.loss_implementations.corn import corn_loss\nfrom ludwig.schema.features.loss.loss import (\n BaseLossConfig,\n BWCEWLossConfig,\n CORNLossConfig,\n HuberLossConfig,\n MAELossConfig,\n MAPELossConfig,\n MSELossConfig,\n NextTokenSoftmaxCrossEntropyLossConfig,\n RMSELossConfig,\n RMSPELossConfig,\n SequenceSoftmaxCrossEntropyLossConfig,\n SigmoidCrossEntropyLossConfig,\n SoftmaxCrossEntropyLossConfig,\n)\nfrom ludwig.utils import strings_utils\nfrom ludwig.utils.registry import Registry\n\n# used for Laplace smoothing for candidate samplers\nEPSILON = 1.0e-10\n\nloss_impl_registry = Registry[type[nn.Module]]()\n\n\ndef register_loss(config_cls: type[BaseLossConfig]):\n def wrap(cls: type[nn.Module]):\n loss_impl_registry[config_cls] = cls\n return cls\n\n return wrap\n\n\ndef create_loss(config: BaseLossConfig) -> nn.Module:\n return loss_impl_registry[type(config)](config)\n\n\nclass LogitsInputsMixin:\n @classmethod\n def get_loss_inputs(cls):\n \"\"\"Maps loss to the desired predicted input type.\"\"\"\n return LOGITS\n\n\n@register_loss(MSELossConfig)\nclass MSELoss(_MSELoss, LogitsInputsMixin):\n \"\"\"Mean squared error.\"\"\"\n\n def __init__(self, config: MSELossConfig):\n super().__init__()\n\n\n@register_loss(MAELossConfig)\nclass MAELoss(L1Loss, LogitsInputsMixin):\n \"\"\"Mean absolute error.\"\"\"\n\n def __init__(self, config: MAELossConfig):\n super().__init__()\n\n\n@register_loss(MAPELossConfig)\nclass MAPELoss(nn.Module, LogitsInputsMixin):\n \"\"\"Mean absolute error.\"\"\"\n\n def __init__(self, config: MAPELossConfig):\n super().__init__()\n\n def forward(self, preds: Tensor, target: Tensor) -> Tensor:\n return mean_absolute_percentage_error(preds, target)\n\n\n@register_loss(RMSELossConfig)\nclass RMSELoss(nn.Module, LogitsInputsMixin):\n \"\"\"Root mean square error.\"\"\"\n\n def __init__(self, config: RMSELossConfig):\n super().__init__()\n self.mse = nn.MSELoss()\n\n def forward(self, preds: Tensor, target: Tensor) -> Tensor:\n return torch.sqrt(self.mse(preds, target))\n\n\n@register_loss(RMSPELossConfig)\nclass RMSPELoss(nn.Module, LogitsInputsMixin):\n \"\"\"Root mean square percentage error.\"\"\"\n\n def __init__(self, config: RMSPELossConfig):\n super().__init__()\n\n def forward(self, preds: Tensor, target: Tensor) -> Tensor:\n loss = utils.rmspe_loss(target, preds)\n return loss\n\n\n@register_loss(BWCEWLossConfig)\nclass BWCEWLoss(nn.Module, LogitsInputsMixin):\n \"\"\"Binary weighted cross entropy loss.\"\"\"\n\n def __init__(self, config: BWCEWLossConfig):\n super().__init__()\n if config.positive_class_weight:\n self.loss_fn = nn.BCEWithLogitsLoss(pos_weight=torch.Tensor([config.positive_class_weight]))\n else:\n self.loss_fn = nn.BCEWithLogitsLoss(pos_weight=config.positive_class_weight)\n self.robust_lambda = config.robust_lambda\n self.confidence_penalty = config.confidence_penalty\n\n def forward(self, preds: torch.Tensor, target: torch.Tensor):\n train_loss = self.loss_fn(preds, target.float())\n # robust lambda\n if self.robust_lambda > 0:\n train_loss = (1 - self.robust_lambda) * train_loss + self.robust_lambda / 2\n\n train_mean_loss = torch.mean(train_loss)\n\n # confidence penalty\n if self.confidence_penalty > 0:\n probabilities = torch.sigmoid(preds)\n mean_penalty = utils.mean_confidence_penalty(probabilities, 2)\n train_mean_loss += self.confidence_penalty * mean_penalty\n\n return train_mean_loss\n\n\n@register_loss(SoftmaxCrossEntropyLossConfig)\nclass SoftmaxCrossEntropyLoss(nn.Module, LogitsInputsMixin):\n def __init__(self, config: SoftmaxCrossEntropyLossConfig):\n \"\"\"\n Params:\n class_weights: List or 1D tensor of length equal to number of classes.\n \"\"\"\n super().__init__()\n if config.class_weights:\n self.loss_fn = nn.CrossEntropyLoss(weight=torch.Tensor(config.class_weights))\n else:\n self.loss_fn = nn.CrossEntropyLoss()\n\n def forward(self, preds: Tensor, target: Tensor) -> Tensor:\n \"\"\"\n Params:\n preds: Tensor of shape [batch x num_classes]\n or shape [batch x num_classes x H x W]\n target: Tensor of shape [batch], where each element is integral\n between 0 and num_classes.\n or shape [batch x H x W], where each element is integral\n between 0 and num_classes.\n \"\"\"\n if len(target.shape) == 1 or len(target.shape) == 3:\n # Assumes we are providing the target as a single class, rather than a distribution\n # The target shape can be a 3D tensor [batch x H x W], for image segmentation\n target = target.long()\n return self.loss_fn(preds, target)\n\n\n@register_loss(SequenceSoftmaxCrossEntropyLossConfig)\nclass SequenceSoftmaxCrossEntropyLoss(nn.Module, LogitsInputsMixin):\n def __init__(self, config: SequenceSoftmaxCrossEntropyLossConfig):\n \"\"\"\n Params:\n class_weights: List or 1D tensor of length equal to number of classes.\n \"\"\"\n super().__init__()\n if config.class_weights:\n self.loss_fn = nn.CrossEntropyLoss(\n weight=torch.Tensor(config.class_weights), ignore_index=strings_utils.SpecialSymbol.PADDING.value\n )\n else:\n self.loss_fn = nn.CrossEntropyLoss(ignore_index=strings_utils.SpecialSymbol.PADDING.value)\n\n def forward(self, preds: Tensor, target: Tensor) -> Tensor:\n \"\"\"\n Params:\n preds: Tensor of shape [batch x sequence_length x vocab_size]\n target: Tensor of shape [batch x sequence_length], where each element is integral between 0 and vocab_size.\n \"\"\"\n target = target.long()\n return self.loss_fn(preds[1:].view(-1, preds.size(-1)), target[1:].view(-1))\n\n\n@register_loss(NextTokenSoftmaxCrossEntropyLossConfig)\nclass NextTokenSoftmaxCrossEntropyLoss(nn.Module, LogitsInputsMixin):\n def __init__(self, config: NextTokenSoftmaxCrossEntropyLossConfig):\n super().__init__()\n self.loss_fn = nn.CrossEntropyLoss()\n\n def forward(self, preds: Tensor, target: Tensor) -> Tensor:\n \"\"\"\n Params:\n preds: Tensor of shape [batch x sequence_length x vocab_size]\n target: Tensor of shape [batch x sequence_length], where each element is integral between 0 and vocab_size.\n\n Reference implementation:\n https://github.com/huggingface/transformers/blob/v4.29.1/src/transformers/models/bert/modeling_bert.py#LL1253C1-L1260C1 # noqa\n \"\"\"\n target = target.long()\n _, _, vocab_size = preds.shape\n # logits for all tensors except n+1 since each logit tensor at position i represents the log probabilities for\n # the next token i+1 if we were to do argmax on the logits ensor at position i.\n shifted_predictions = preds[:, :-1, :]\n # Shift by 1 since the logits at position 0 in predictions represent the log likelihood of target token 1\n shifted_targets = target[:, 1:]\n return self.loss_fn(shifted_predictions.reshape(-1, vocab_size), shifted_targets.reshape(-1))\n\n\n@register_loss(SigmoidCrossEntropyLossConfig)\nclass SigmoidCrossEntropyLoss(nn.Module, LogitsInputsMixin):\n def __init__(self, config: SigmoidCrossEntropyLossConfig):\n \"\"\"\n Params:\n class_weights: List or 1D tensor of length equal to number of classes.\n \"\"\"\n super().__init__()\n if config.class_weights:\n self.loss_fn = nn.BCEWithLogitsLoss(pos_weight=torch.Tensor(config.class_weights))\n else:\n self.loss_fn = nn.BCEWithLogitsLoss()\n\n def forward(self, preds: Tensor, target: Tensor) -> Tensor:\n if preds.ndim != 2:\n raise RuntimeError(\"SigmoidCrossEntropyLoss currently only supported for 2D tensors.\")\n\n return self.loss_fn(preds.type(torch.float32), target.type(torch.float32))\n\n\n@register_loss(HuberLossConfig)\nclass HuberLoss(_HuberLoss, LogitsInputsMixin):\n \"\"\"Huber loss.\"\"\"\n\n def __init__(self, config: HuberLossConfig):\n super().__init__(delta=config.delta)\n\n\n@register_loss(CORNLossConfig)\nclass CORNLoss(nn.Module, LogitsInputsMixin):\n \"\"\"CORN loss.\"\"\"\n\n def __init__(self, config: CORNLossConfig):\n super().__init__()\n\n def forward(self, preds: Tensor, target: Tensor) -> Tensor:\n num_classes = preds.shape[1]\n return corn_loss(preds, target, num_classes=num_classes)\n"
},
{
"path": "ludwig/modules/lr_scheduler.py",
"content": "import logging\nimport math\nfrom collections.abc import Callable\nfrom typing import Any\n\nfrom torch.optim import Optimizer\nfrom torch.optim.lr_scheduler import CosineAnnealingWarmRestarts, LambdaLR, ReduceLROnPlateau, SequentialLR\n\nfrom ludwig.constants import MINIMIZE, TRAINING, VALIDATION\nfrom ludwig.modules.metric_registry import get_metric_objective\nfrom ludwig.schema.lr_scheduler import LRSchedulerConfig\nfrom ludwig.utils.metric_utils import TrainerMetric\nfrom ludwig.utils.trainer_utils import ProgressTracker\n\nlogger = logging.getLogger(__name__)\n\n\nclass ReduceLROnPLateauCappedDecreases(ReduceLROnPlateau):\n def __init__(self, optimizer: Optimizer, mode: str, reduce_limit: int, factor: float, patience: int):\n super().__init__(optimizer, mode=mode, factor=factor, patience=patience)\n self.reduce_limit = reduce_limit\n self._num_reduce_lr = 0\n\n def step(self, metrics):\n if self._num_reduce_lr >= self.reduce_limit:\n # Already reduced the LR as many times as we will allow\n return\n\n return super().step(metrics)\n\n @property\n def num_reduce_lr(self) -> int:\n return self._num_reduce_lr\n\n def _reduce_lr(self, epoch=None):\n \"\"\"Overrides the base ReduceLROnPlateau implementation.\"\"\"\n self._num_reduce_lr += 1\n self.apply_lr()\n\n def apply_lr(self):\n if self._num_reduce_lr == 0:\n return\n\n for i, param_group in enumerate(self.optimizer.param_groups):\n old_lr = float(param_group[\"lr\"])\n new_lr = max(old_lr * math.pow(self.factor, self._num_reduce_lr), self.min_lrs[i])\n if old_lr - new_lr > self.eps:\n param_group[\"lr\"] = new_lr\n logger.info(f\"From ReduceLROnPLateauCappedDecreases, reducing learning rate to {new_lr}\")\n\n\nclass LRScheduler:\n def __init__(\n self,\n config: LRSchedulerConfig,\n optimizer: Optimizer,\n steps_per_checkpoint: int,\n total_steps: int,\n ):\n self.config = config\n self.optimizer = optimizer\n\n # Scheduler updated each training step\n self.step_info = StepInfo(steps_per_checkpoint, total_steps, self.config)\n self._train_scheduler = get_schedule_with_warmup_and_decay(self.config, self.optimizer, self.step_info)\n\n # Scheduler updated each eval step\n self._eval_scheduler = None\n if self.config.reduce_on_plateau > 0:\n mode = \"min\" if get_metric_objective(self.config.reduce_eval_metric) == MINIMIZE else \"max\"\n self._eval_scheduler = ReduceLROnPLateauCappedDecreases(\n optimizer=self.optimizer,\n mode=mode,\n reduce_limit=self.config.reduce_on_plateau,\n factor=self.config.reduce_on_plateau_rate,\n patience=self.config.reduce_on_plateau_patience,\n )\n\n def step(self):\n \"\"\"Called every step of training.\"\"\"\n self._train_scheduler.step()\n\n if self._eval_scheduler is not None:\n # We apply this scheduler every eval step, not train step, so we don't want to call step() here.\n # However, we need to re-apply the LR reduction to the LR from the train scheduler, as the first scheduler\n # resets the LR back to the base LR.\n self._eval_scheduler.apply_lr()\n\n def eval_step(self, progress_tracker: ProgressTracker, validation_field: str):\n \"\"\"Called every checkpoint evaluation step.\"\"\"\n if self._eval_scheduler is None:\n # No reduce on plateau\n return\n\n if self.config.reduce_eval_split == TRAINING:\n split_metrics = progress_tracker.train_metrics\n elif self.config.reduce_eval_split == VALIDATION:\n split_metrics = progress_tracker.validation_metrics\n else: # if self.config.reduce_eval_split == TEST:\n split_metrics = progress_tracker.test_metrics\n\n validation_metric = self.config.reduce_eval_metric\n last_metric: TrainerMetric = split_metrics[validation_field][validation_metric][-1]\n last_metric_value = last_metric[-1]\n\n prev_num_reductions = self._eval_scheduler.num_reduce_lr\n self._eval_scheduler.step(last_metric_value)\n\n num_reductions = self._eval_scheduler.num_reduce_lr\n if num_reductions > prev_num_reductions:\n # LR reduction -> update progress tracker\n progress_tracker.last_learning_rate_reduction_steps = progress_tracker.steps\n progress_tracker.last_learning_rate_reduction = 0\n progress_tracker.num_reductions_learning_rate += 1\n else:\n progress_tracker.last_learning_rate_reduction = (\n progress_tracker.steps - progress_tracker.last_learning_rate_reduction_steps\n )\n\n def state_dict(self) -> dict[str, Any]:\n return {\n \"train_scheduler_state\": self._train_scheduler.state_dict(),\n \"eval_scheduler_state\": self._eval_scheduler.state_dict() if self._eval_scheduler is not None else {},\n }\n\n def load_state_dict(self, d: dict[str, Any]):\n self._train_scheduler.load_state_dict(d[\"train_scheduler_state\"])\n if self._eval_scheduler is not None:\n self._eval_scheduler.load_state_dict(d[\"eval_scheduler_state\"])\n\n\nclass StepInfo:\n \"\"\"Stores the steps_per_checkpoint and total_steps used during the current training run.\n\n This class is needed by LambdaLR to allow us to update the steps on training init without resetting the entire\n LRScheduler from scratch (which would result in resetting the optimizer learning rate).\n \"\"\"\n\n def __init__(self, steps_per_checkpoint: int, total_steps: int, config: LRSchedulerConfig):\n self.config = config\n self.steps_per_checkpoint = steps_per_checkpoint\n self.num_training_steps = total_steps\n\n if self.config.warmup_fraction > 0 and self.config.warmup_evaluations > 0:\n logger.info(\n \"Both `learning_rate_scheduler.warmup_fraction` and `learning_rate_scheduler.warmup_evaluations` \"\n \"provided. The larger of the two (as a function of the total training steps) will be used.\"\n )\n\n num_warmup_steps = 0\n if self.config.warmup_fraction > 0:\n num_warmup_steps = max(self.config.warmup_fraction * self.num_training_steps, num_warmup_steps)\n if self.config.warmup_evaluations > 0:\n num_warmup_steps = max(self.config.warmup_evaluations * self.steps_per_checkpoint, num_warmup_steps)\n self.num_warmup_steps = num_warmup_steps\n\n\ndef get_schedule_with_warmup_and_decay(\n config: LRSchedulerConfig,\n optimizer: Optimizer,\n step_info: StepInfo,\n) -> LambdaLR:\n \"\"\"Creates a learning rate scheduler that updates each training step.\"\"\"\n schedulers = []\n\n # Warmup scheduler.\n if step_info.num_warmup_steps > 0:\n warmup_scheduler = LambdaLR(\n optimizer,\n lambda current_step: float(current_step) / float(max(1, step_info.num_warmup_steps)),\n )\n schedulers.append(warmup_scheduler)\n\n # Decay scheduler.\n decay = config.decay\n decay_scheduler = decay_registry[decay](config, optimizer, step_info)\n schedulers.append(decay_scheduler)\n\n if len(schedulers) == 1:\n # Only one scheduler, so no need to wrap in a SequentialLR.\n return schedulers[0]\n\n # Return a SequentialLR that applies the warmup and decay schedulers in order\n # with the warmup scheduler only applied for the first num_warmup_steps steps.\n return SequentialLR(optimizer, schedulers=schedulers, milestones=[step_info.num_warmup_steps])\n\n\ndef no_decay(current_step: int, num_training_steps: int, num_warmup_steps: int, config: LRSchedulerConfig):\n return 1.0\n\n\ndef linear_decay(current_step: int, num_training_steps: int, num_warmup_steps: int, config: LRSchedulerConfig):\n return max(\n 0.0,\n float(num_training_steps - num_warmup_steps - current_step)\n / float(max(1, num_training_steps - num_warmup_steps)),\n )\n\n\ndef exponential_decay(current_step: int, num_training_steps: int, num_warmup_steps: int, config: LRSchedulerConfig):\n decay_rate = float(config.decay_rate)\n decay_steps = float(config.decay_steps)\n step = float(current_step)\n exponent = 1 + step / decay_steps\n if config.staircase:\n exponent = math.ceil(exponent)\n return math.pow(decay_rate, exponent)\n\n\ndef wrap_decay_fn(decay_fn: Callable) -> Callable:\n def init_fn(config: LRSchedulerConfig, optimizer: Optimizer, step_info: StepInfo) -> LambdaLR:\n return LambdaLR(\n optimizer,\n lambda current_step: decay_fn(\n current_step, step_info.num_training_steps, step_info.num_warmup_steps, config\n ),\n )\n\n return init_fn\n\n\ndef init_cosine_decay(\n config: LRSchedulerConfig,\n optimizer: Optimizer,\n step_info: StepInfo,\n) -> CosineAnnealingWarmRestarts:\n t_0 = config.t_0\n if not t_0:\n t_0 = step_info.steps_per_checkpoint\n if not t_0:\n # A scheduler may be initialized with dummy values like at the start of training.\n # Ensure that t_0 != 0, as this causes an error to be raised.\n t_0 = 1\n\n return CosineAnnealingWarmRestarts(\n optimizer,\n T_0=t_0,\n T_mult=config.t_mult or 1,\n eta_min=config.eta_min or 0,\n )\n\n\ndecay_registry = {\n None: wrap_decay_fn(no_decay),\n \"linear\": wrap_decay_fn(linear_decay),\n \"exponential\": wrap_decay_fn(exponential_decay),\n \"cosine\": init_cosine_decay,\n}\n"
},
{
"path": "ludwig/modules/metric_modules.py",
"content": "# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nimport sys\nfrom abc import ABC, abstractmethod\nfrom collections.abc import Callable, Generator\nfrom contextlib import contextmanager\nfrom typing import Any\n\nimport torch\nfrom torch import Tensor, tensor\nfrom torchmetrics import MeanAbsoluteError, MeanAbsolutePercentageError\nfrom torchmetrics import MeanMetric as _MeanMetric\nfrom torchmetrics import MeanSquaredError, Metric\nfrom torchmetrics.classification import (\n BinaryAccuracy,\n BinaryAUROC,\n BinaryPrecision,\n BinaryRecall,\n BinarySpecificity,\n MulticlassAccuracy,\n MulticlassAUROC,\n)\nfrom torchmetrics.functional.regression.r2 import _r2_score_compute, _r2_score_update\nfrom torchmetrics.metric import jit_distributed_available\nfrom torchmetrics.text import BLEUScore, CharErrorRate, WordErrorRate\nfrom torchmetrics.text.perplexity import Perplexity\nfrom torchmetrics.text.rouge import ROUGEScore\n\nfrom ludwig.constants import ( # RESPONSE,\n ACCURACY,\n ACCURACY_MICRO,\n BINARY,\n BINARY_WEIGHTED_CROSS_ENTROPY,\n CATEGORY,\n CATEGORY_DISTRIBUTION,\n CORN,\n HITS_AT_K,\n HUBER,\n IGNORE_INDEX_TOKEN_ID,\n IMAGE,\n JACCARD,\n LOGITS,\n LOSS,\n MAXIMIZE,\n MEAN_ABSOLUTE_ERROR,\n MEAN_ABSOLUTE_PERCENTAGE_ERROR,\n MEAN_SQUARED_ERROR,\n MINIMIZE,\n NEXT_TOKEN_PERPLEXITY,\n NUMBER,\n PERPLEXITY,\n PRECISION,\n PREDICTIONS,\n PROBABILITIES,\n R2,\n RECALL,\n ROC_AUC,\n ROOT_MEAN_SQUARED_ERROR,\n ROOT_MEAN_SQUARED_PERCENTAGE_ERROR,\n SEQUENCE,\n SEQUENCE_ACCURACY,\n SET,\n SPECIFICITY,\n TEXT,\n TIMESERIES,\n TOKEN_ACCURACY,\n VECTOR,\n)\nfrom ludwig.distributed import get_current_dist_strategy\nfrom ludwig.modules.loss_modules import (\n BWCEWLoss,\n CORNLoss,\n HuberLoss,\n NextTokenSoftmaxCrossEntropyLoss,\n SequenceSoftmaxCrossEntropyLoss,\n SigmoidCrossEntropyLoss,\n SoftmaxCrossEntropyLoss,\n)\nfrom ludwig.modules.metric_registry import get_metric_objective, get_metric_registry, register_metric\nfrom ludwig.schema.features.loss.loss import (\n BWCEWLossConfig,\n CORNLossConfig,\n HuberLossConfig,\n SequenceSoftmaxCrossEntropyLossConfig,\n SigmoidCrossEntropyLossConfig,\n SoftmaxCrossEntropyLossConfig,\n)\nfrom ludwig.utils.loss_utils import rmspe_loss\nfrom ludwig.utils.metric_utils import masked_correct_predictions\nfrom ludwig.utils.torch_utils import sequence_length_2D\n\nlogger = logging.getLogger(__name__)\n\n\nclass LudwigMetric(Metric, ABC):\n @classmethod\n def can_report(cls, feature: \"OutputFeature\") -> bool: # noqa: F821\n return True\n\n @contextmanager\n def sync_context(\n self,\n dist_sync_fn: Callable | None = None,\n process_group: Any | None = None,\n should_sync: bool = True,\n should_unsync: bool = True,\n distributed_available: Callable | None = jit_distributed_available,\n ) -> Generator:\n \"\"\"Override the behavior of this in the base class to support custom distributed strategies.\"\"\"\n dist_strategy = get_current_dist_strategy()\n self.sync(\n dist_sync_fn=dist_strategy.gather_all_tensors_fn(),\n process_group=process_group,\n should_sync=should_sync,\n distributed_available=dist_strategy.is_available,\n )\n\n yield\n\n self.unsync(should_unsync=self._is_synced and should_unsync)\n\n\n@register_metric(ROOT_MEAN_SQUARED_ERROR, [NUMBER], MINIMIZE, PREDICTIONS)\nclass RMSEMetric(MeanSquaredError, LudwigMetric):\n \"\"\"Root mean squared error metric.\"\"\"\n\n def __init__(self, **kwargs):\n super().__init__(squared=False)\n\n\n@register_metric(PRECISION, [BINARY], MAXIMIZE, PROBABILITIES)\nclass PrecisionMetric(BinaryPrecision, LudwigMetric):\n \"\"\"Precision metric.\"\"\"\n\n def __init__(self, **kwargs):\n super().__init__()\n\n\n@register_metric(RECALL, [BINARY], MAXIMIZE, PROBABILITIES)\nclass RecallMetric(BinaryRecall, LudwigMetric):\n \"\"\"Recall metric.\"\"\"\n\n def __init__(self, **kwargs):\n super().__init__()\n\n\n@register_metric(ROC_AUC, [BINARY], MAXIMIZE, PROBABILITIES)\nclass BinaryAUROCMetric(BinaryAUROC, LudwigMetric):\n \"\"\"Area under the receiver operating curve.\"\"\"\n\n def __init__(self, **kwargs):\n super().__init__()\n\n def update(self, preds: Tensor, target: Tensor) -> None:\n super().update(preds, target.type(torch.int8))\n\n\n@register_metric(ROC_AUC, [CATEGORY, CATEGORY_DISTRIBUTION], MAXIMIZE, PROBABILITIES)\nclass CategoryAUROCMetric(MulticlassAUROC, LudwigMetric):\n \"\"\"Area under the receiver operating curve.\"\"\"\n\n def __init__(self, num_classes: int, **kwargs):\n super().__init__(num_classes=num_classes)\n\n def update(self, preds: Tensor, target: Tensor) -> None:\n if len(target.shape) > 1:\n target = torch.argmax(target, dim=1)\n super().update(preds, target)\n\n\n@register_metric(SPECIFICITY, [BINARY], MAXIMIZE, PROBABILITIES)\nclass SpecificityMetric(BinarySpecificity, LudwigMetric):\n \"\"\"Specificity metric.\"\"\"\n\n def __init__(self, **kwargs):\n super().__init__()\n\n\nclass MeanMetric(LudwigMetric):\n \"\"\"Abstract class for computing mean of metrics.\"\"\"\n\n def __init__(self, **kwargs):\n super().__init__()\n self.avg = _MeanMetric()\n\n def update(self, preds: Tensor, target: Tensor) -> None:\n self.avg.update(self.get_current_value(preds, target))\n\n def compute(self) -> Tensor:\n return self.avg.compute()\n\n def reset(self):\n super().reset()\n self.avg.reset()\n\n @abstractmethod\n def get_current_value(self, preds: Tensor, target: Tensor) -> Tensor:\n raise NotImplementedError()\n\n\n@register_metric(ROOT_MEAN_SQUARED_PERCENTAGE_ERROR, [NUMBER], MINIMIZE, PREDICTIONS)\nclass RMSPEMetric(MeanMetric):\n def __init__(self, **kwargs):\n super().__init__()\n\n \"\"\" Root mean squared percentage error metric. \"\"\"\n\n def get_current_value(self, preds: Tensor, target: Tensor) -> Tensor:\n return rmspe_loss(target, preds)\n\n\n@register_metric(R2, [NUMBER, VECTOR, TIMESERIES], MAXIMIZE, PREDICTIONS)\nclass R2Score(LudwigMetric):\n \"\"\"Custom R-squared metric implementation that modifies torchmetrics R-squared implementation to return Nan\n when there is only sample. This is because R-squared is only defined for two or more samples.\n\n Custom implementation uses code from torchmetrics v0.9.2's implementation of R2: https://github.com/Lightning-\n AI/metrics/blob/master/src/torchmetrics/regression/r2.py\n \"\"\"\n\n def __init__(\n self, num_outputs: int = 1, adjusted: int = 0, multioutput: str = \"uniform_average\", **kwargs: Any\n ) -> None:\n super().__init__(**kwargs)\n\n self.num_outputs = num_outputs\n\n if adjusted < 0 or not isinstance(adjusted, int):\n raise ValueError(\"`adjusted` parameter should be an integer larger or equal to 0.\")\n self.adjusted = adjusted\n\n allowed_multioutput = (\"raw_values\", \"uniform_average\", \"variance_weighted\")\n if multioutput not in allowed_multioutput:\n raise ValueError(\n f\"Invalid input to argument `multioutput`. Choose one of the following: {allowed_multioutput}\"\n )\n self.multioutput = multioutput\n\n self.add_state(\"sum_squared_error\", default=torch.zeros(self.num_outputs), dist_reduce_fx=\"sum\")\n self.add_state(\"sum_error\", default=torch.zeros(self.num_outputs), dist_reduce_fx=\"sum\")\n self.add_state(\"residual\", default=torch.zeros(self.num_outputs), dist_reduce_fx=\"sum\")\n self.add_state(\"total\", default=tensor(0), dist_reduce_fx=\"sum\")\n\n def update(self, preds: Tensor, target: Tensor) -> None:\n \"\"\"Update state with predictions and targets.\n\n Args:\n preds: Predictions from model\n target: Ground truth values\n \"\"\"\n sum_squared_error, sum_error, residual, n_obs = _r2_score_update(preds, target)\n\n self.sum_squared_error += sum_squared_error\n self.sum_error += sum_error\n self.residual += residual\n self.total += n_obs\n\n def compute(self) -> Tensor:\n \"\"\"Computes r2 score over the metric states.\"\"\"\n\n # self.total maps to the number of observations in preds/target computed during update()\n if self.total <= 1:\n logger.warning(\n \"\"\"R-squared (r2) is not defined for one sample. It needs at least two samples. Returning NaN.\"\"\"\n )\n return torch.tensor(float(\"nan\"))\n\n return _r2_score_compute(\n self.sum_squared_error, self.sum_error, self.residual, self.total, self.adjusted, self.multioutput\n )\n\n\n@register_metric(LOSS, [], MINIMIZE, LOGITS)\nclass LossMetric(MeanMetric, ABC):\n def __init__(self):\n super().__init__()\n\n @abstractmethod\n def get_current_value(self, preds: Tensor, target: Tensor) -> Tensor:\n raise NotImplementedError()\n\n @classmethod\n def can_report(cls, feature: \"OutputFeature\") -> bool: # noqa: F821\n return False\n\n\n@register_metric(BINARY_WEIGHTED_CROSS_ENTROPY, [BINARY], MINIMIZE, LOGITS)\nclass BWCEWLMetric(LossMetric):\n \"\"\"Binary Weighted Cross Entropy Weighted Logits Score Metric.\"\"\"\n\n def __init__(self, config: BWCEWLossConfig, **kwargs):\n super().__init__()\n self.loss_function = BWCEWLoss(config)\n\n def get_current_value(self, preds: Tensor, target: Tensor) -> Tensor:\n return self.loss_function(preds, target)\n\n\n@register_metric(\"softmax_cross_entropy\", [CATEGORY, CATEGORY_DISTRIBUTION, IMAGE], MINIMIZE, LOGITS)\nclass SoftmaxCrossEntropyMetric(LossMetric):\n def __init__(self, config: SoftmaxCrossEntropyLossConfig, **kwargs):\n super().__init__()\n self.softmax_cross_entropy_function = SoftmaxCrossEntropyLoss(config)\n\n def get_current_value(self, preds: Tensor, target: Tensor):\n return self.softmax_cross_entropy_function(preds, target)\n\n\n@register_metric(\"sequence_softmax_cross_entropy\", [SEQUENCE, TEXT], MINIMIZE, LOGITS)\nclass SequenceSoftmaxCrossEntropyMetric(LossMetric):\n def __init__(self, config: SequenceSoftmaxCrossEntropyLossConfig, **kwargs):\n super().__init__()\n self.sequence_softmax_cross_entropy_function = SequenceSoftmaxCrossEntropyLoss(config)\n\n def get_current_value(self, preds: Tensor, target: Tensor):\n return self.sequence_softmax_cross_entropy_function(preds, target)\n\n\n@register_metric(\"next_token_softmax_cross_entropy\", [SEQUENCE, TEXT], MINIMIZE, LOGITS)\nclass NextTokenSoftmaxCrossEntropyMetric(LossMetric):\n def __init__(self, config: SequenceSoftmaxCrossEntropyLossConfig, **kwargs):\n super().__init__()\n self.next_token_softmax_cross_entropy_function = NextTokenSoftmaxCrossEntropyLoss(config)\n\n def get_current_value(self, preds: Tensor, target: Tensor):\n return self.next_token_softmax_cross_entropy_function(preds, target)\n\n\n@register_metric(\"sigmoid_cross_entropy\", [SET], MINIMIZE, LOGITS)\nclass SigmoidCrossEntropyMetric(LossMetric):\n def __init__(self, config: SigmoidCrossEntropyLossConfig, **kwargs):\n super().__init__()\n self.sigmoid_cross_entropy_function = SigmoidCrossEntropyLoss(config)\n\n def get_current_value(self, preds: Tensor, target: Tensor) -> Tensor:\n return self.sigmoid_cross_entropy_function(preds, target)\n\n\n@register_metric(TOKEN_ACCURACY, [SEQUENCE, TEXT], MAXIMIZE, PREDICTIONS)\nclass TokenAccuracyMetric(MeanMetric):\n def __init__(self, **kwargs):\n super().__init__()\n\n def get_current_value(self, preds: Tensor, target: Tensor) -> Tensor:\n target = target.type(preds.dtype)\n target_sequence_length = sequence_length_2D(target)\n masked_correct_preds = masked_correct_predictions(target, preds, target_sequence_length)\n return torch.mean(masked_correct_preds)\n\n\n@register_metric(SEQUENCE_ACCURACY, [SEQUENCE, TEXT], MAXIMIZE, PREDICTIONS)\nclass SequenceAccuracyMetric(MeanMetric):\n def __init__(self, **kwargs):\n super().__init__()\n\n def get_current_value(self, preds: Tensor, target: Tensor) -> Tensor:\n return torch.sum(torch.all(preds == target, dim=1)) / target.size()[0]\n\n\n@register_metric(PERPLEXITY, [SEQUENCE, TEXT], MINIMIZE, PROBABILITIES)\nclass PerplexityMetric(Perplexity, LudwigMetric):\n def __init__(self, **kwargs):\n super().__init__(ignore_index=IGNORE_INDEX_TOKEN_ID)\n\n def update(self, preds: Tensor, target: Tensor) -> None:\n super().update(preds, target.type(torch.int64))\n\n\n@register_metric(NEXT_TOKEN_PERPLEXITY, [SEQUENCE, TEXT], MINIMIZE, PROBABILITIES)\nclass NextTokenPerplexityMetric(MeanMetric):\n def __init__(self, **kwargs):\n super().__init__()\n self.next_token_softmax_cross_entropy_function = NextTokenSoftmaxCrossEntropyLoss({})\n\n def get_current_value(self, preds: Tensor, target: Tensor):\n # Perplexity can be represented as the exponential of the cross-entropy loss.\n # https://towardsdatascience.com/perplexity-in-language-models-87a196019a94\n # We can't use torchmetrics perplexity because it calculates normal cross-entropy\n # loss as opposed to shifted cross entropy loss.\n shifted_loss = self.next_token_softmax_cross_entropy_function(preds, target)\n return torch.exp(shifted_loss)\n\n\n# @register_metric(\"bleu\", [TEXT], MAXIMIZE, RESPONSE)\n# https://github.com/ludwig-ai/ludwig/issues/3953\nclass BLEUScoreMetric(BLEUScore, LudwigMetric):\n def __init__(self, **kwargs):\n super().__init__()\n\n\n# @register_metric(\"rouge\", [TEXT], MAXIMIZE, RESPONSE)\n# https://github.com/ludwig-ai/ludwig/issues/3953\nclass ROUGEScoreMetric(ROUGEScore, LudwigMetric):\n def __init__(self, **kwargs):\n super().__init__()\n\n\n# @register_metric(\"word_error_rate\", [TEXT], MINIMIZE, RESPONSE)\n# https://github.com/ludwig-ai/ludwig/issues/3953\nclass WordErrorRateMetric(WordErrorRate, LudwigMetric):\n def __init__(self, **kwargs):\n super().__init__()\n\n\n# @register_metric(\"char_error_rate\", [TEXT], MINIMIZE, RESPONSE)\n# https://github.com/ludwig-ai/ludwig/issues/3953\nclass CharErrorRateMetric(CharErrorRate, LudwigMetric):\n def __init__(self, **kwargs):\n super().__init__()\n\n\n@register_metric(ACCURACY, [BINARY], MAXIMIZE, PREDICTIONS)\nclass Accuracy(BinaryAccuracy, LudwigMetric):\n \"\"\"R-squared metric.\"\"\"\n\n def __init__(self, **kwargs):\n super().__init__()\n\n\n@register_metric(ACCURACY, [CATEGORY, CATEGORY_DISTRIBUTION], MAXIMIZE, PREDICTIONS)\nclass CategoryAccuracy(MulticlassAccuracy, LudwigMetric):\n def __init__(self, num_classes: int, **kwargs):\n super().__init__(num_classes=num_classes)\n\n def update(self, preds: Tensor, target: Tensor) -> None:\n if len(target.shape) > 1:\n target = torch.argmax(target, dim=1)\n super().update(preds, target.type(torch.long))\n\n\n@register_metric(ACCURACY_MICRO, [CATEGORY, CATEGORY_DISTRIBUTION], MAXIMIZE, PREDICTIONS)\nclass CategoryAccuracyMicro(MulticlassAccuracy, LudwigMetric):\n def __init__(self, num_classes: int, **kwargs):\n super().__init__(num_classes=num_classes, average=\"micro\")\n\n def update(self, preds: Tensor, target: Tensor) -> None:\n if len(target.shape) > 1:\n target = torch.argmax(target, dim=1)\n super().update(preds, target.type(torch.long))\n\n\n@register_metric(HITS_AT_K, [CATEGORY, CATEGORY_DISTRIBUTION], MAXIMIZE, LOGITS)\nclass HitsAtKMetric(MulticlassAccuracy, LudwigMetric):\n def __init__(self, num_classes: int, top_k: int, **kwargs):\n super().__init__(num_classes=num_classes, top_k=top_k, **kwargs)\n\n def update(self, preds: Tensor, target: Tensor) -> None:\n if len(target.shape) > 1:\n target = torch.argmax(target, dim=1)\n super().update(preds, target.type(torch.long))\n\n @classmethod\n def can_report(cls, feature: \"OutputFeature\") -> bool: # noqa: F821\n return feature.num_classes > feature.top_k\n\n\n@register_metric(MEAN_ABSOLUTE_ERROR, [NUMBER, VECTOR, TIMESERIES], MINIMIZE, PREDICTIONS)\nclass MAEMetric(MeanAbsoluteError, LudwigMetric):\n def __init__(self, **kwargs):\n super().__init__()\n\n def update(self, preds: Tensor, target: Tensor) -> None:\n super().update(preds.detach(), target)\n\n\n@register_metric(MEAN_SQUARED_ERROR, [NUMBER, VECTOR, TIMESERIES], MINIMIZE, PREDICTIONS)\nclass MSEMetric(MeanSquaredError, LudwigMetric):\n def __init__(self, **kwargs):\n super().__init__()\n\n def update(self, preds: Tensor, target: Tensor) -> None:\n super().update(preds, target)\n\n\n@register_metric(MEAN_ABSOLUTE_PERCENTAGE_ERROR, [NUMBER, VECTOR, TIMESERIES], MINIMIZE, PREDICTIONS)\nclass MAPEMetric(MeanAbsolutePercentageError, LudwigMetric):\n def __init__(self, **kwargs):\n super().__init__()\n\n def update(self, preds: Tensor, target: Tensor) -> None:\n super().update(preds, target)\n\n\n@register_metric(JACCARD, [SET], MAXIMIZE, PROBABILITIES)\nclass JaccardMetric(MeanMetric):\n def __init__(self, threshold: float = 0.5, **kwargs):\n super().__init__()\n self.threshold = threshold\n\n def get_current_value(self, preds: Tensor, target: Tensor) -> Tensor:\n # notation: b is batch size and nc is number of unique elements in the set\n # preds: shape [b, nc] probabilities for each class\n # target: shape [b, nc] bit-mapped set representation\n preds = torch.greater_equal(preds, self.threshold) # now bit-mapped set\n target = target.type(torch.bool)\n\n intersection = torch.sum(torch.logical_and(target, preds).type(torch.float32), dim=-1)\n union = torch.sum(torch.logical_or(target, preds).type(torch.float32), dim=-1)\n\n return intersection / union # shape [b]\n\n\n@register_metric(HUBER, [NUMBER, VECTOR, TIMESERIES], MINIMIZE, PREDICTIONS)\nclass HuberMetric(LossMetric):\n def __init__(\n self,\n config: HuberLossConfig,\n **kwargs,\n ):\n super().__init__()\n self.loss_function = HuberLoss(config=config)\n\n def get_current_value(self, preds: Tensor, target: Tensor) -> Tensor:\n return self.loss_function(preds, target)\n\n\n@register_metric(CORN, [CATEGORY], MINIMIZE, PREDICTIONS)\nclass CORNMetric(LossMetric):\n def __init__(\n self,\n config: CORNLossConfig,\n **kwargs,\n ):\n super().__init__()\n self.loss_function = CORNLoss(config=config)\n\n def get_current_value(self, preds: Tensor, target: Tensor) -> Tensor:\n return self.loss_function(preds, target)\n\n\ndef get_metric_cls(metric_name: str) -> type[LudwigMetric]:\n return get_metric_registry()[metric_name]\n\n\ndef get_improved_fn(metric: str) -> Callable:\n if get_metric_objective(metric) == MINIMIZE:\n return lambda x, y: x < y\n else:\n return lambda x, y: x > y\n\n\ndef get_initial_validation_value(metric: str) -> float:\n # Use finite floats instead of inf/-inf so that training_progress.json\n # is valid JSON (RFC 8259). sys.float_info.max (~1.8e308) is larger than\n # any real metric value, so comparison semantics are identical.\n if get_metric_objective(metric) == MINIMIZE:\n return sys.float_info.max\n else:\n return -sys.float_info.max\n\n\ndef get_best_function(metric: str) -> Callable:\n if get_metric_objective(metric) == MINIMIZE:\n return min\n else:\n return max\n"
},
{
"path": "ludwig/modules/metric_registry.py",
"content": "from typing import Literal, TYPE_CHECKING\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import LOGITS, MAXIMIZE, MINIMIZE, PREDICTIONS, PROBABILITIES, RESPONSE\nfrom ludwig.utils.registry import Registry\n\nif TYPE_CHECKING:\n from ludwig.modules.metric_modules import LudwigMetric\n\n\nmetric_feature_type_registry = Registry()\nmetric_registry = Registry()\nmetric_objective_registry = Registry()\nmetric_tensor_input_registry = Registry()\n\n\ndef register_metric(\n name: str,\n feature_types: str | list[str],\n objective: Literal[MINIMIZE, MAXIMIZE],\n output_feature_tensor_name: Literal[PREDICTIONS, PROBABILITIES, LOGITS],\n):\n \"\"\"Registers a metric class.\n\n Args:\n name: The name of the metric. Used in metric reporting and in the config.\n feature_types: The feature types that this metric can be used with.\n objective: The objective of the metric. Either MINIMIZE or MAXIMIZE.\n output_feature_tensor_name: Name of the tensor from output_feature::predictions() that should be used as input.\n For example: PREDICTIONS would be used for accuracy metrics while LOGITS would be used for loss metrics.\n \"\"\"\n if isinstance(feature_types, str):\n feature_types = [feature_types]\n\n def wrap(cls):\n for feature_type in feature_types:\n feature_registry = metric_feature_type_registry.get(feature_type, {})\n feature_registry[name] = cls\n metric_feature_type_registry[feature_type] = feature_registry\n metric_registry[name] = cls\n metric_objective_registry[name] = objective\n metric_tensor_input_registry[name] = output_feature_tensor_name\n return cls\n\n return wrap\n\n\ndef get_metric_classes(feature_type: str) -> dict[str, \"LudwigMetric\"]:\n return metric_feature_type_registry[feature_type]\n\n\ndef get_metric_cls(feature_type: str, name: str) -> \"LudwigMetric\":\n return metric_feature_type_registry[feature_type][name]\n\n\n@DeveloperAPI\ndef get_metric_feature_type_registry() -> Registry:\n return metric_feature_type_registry\n\n\n@DeveloperAPI\ndef get_metric_registry() -> Registry:\n return metric_registry\n\n\n@DeveloperAPI\ndef get_metric(metric_name: str) -> \"LudwigMetric\": # noqa\n return get_metric_registry()[metric_name]\n\n\n@DeveloperAPI\ndef get_metrics_for_type(feature_type: str) -> dict[str, \"LudwigMetric\"]: # noqa\n return get_metric_feature_type_registry()[feature_type]\n\n\n@DeveloperAPI\ndef get_metric_names_for_type(feature_type: str) -> list[str]:\n return sorted(list(get_metric_feature_type_registry()[feature_type].keys()))\n\n\n@DeveloperAPI\ndef get_metric_objective(metric_name: str) -> Literal[MINIMIZE, MAXIMIZE]:\n return metric_objective_registry[metric_name]\n\n\n@DeveloperAPI\ndef get_metric_tensor_input(metric_name: str) -> Literal[PREDICTIONS, PROBABILITIES, LOGITS, RESPONSE]:\n return metric_tensor_input_registry[metric_name]\n"
},
{
"path": "ludwig/modules/mlp_mixer_modules.py",
"content": "# Copyright (c) 2021 Linux Foundation\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nimport torch\nimport torch.nn as nn\n\nfrom ludwig.utils.torch_utils import LudwigModule\n\n\nclass MLP(LudwigModule):\n def __init__(\n self,\n in_features: int | tuple[int],\n hidden_size: int,\n out_features: int | tuple[int] = None,\n dropout: float = 0.0,\n ):\n super().__init__()\n\n out_features = out_features or in_features\n\n self._input_shape = in_features\n self._output_shape = out_features\n\n self.linear1 = nn.Linear(in_features=in_features, out_features=hidden_size)\n self.linear2 = nn.Linear(in_features=hidden_size, out_features=out_features)\n self.dropout1 = nn.Dropout(dropout)\n self.dropout2 = nn.Dropout(dropout)\n\n def forward(self, inputs, **kwargs):\n hidden = self.dropout1(nn.functional.gelu(self.linear1(inputs)))\n return self.dropout2(self.linear2(hidden))\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self._input_shape])\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self._output_shape])\n\n\nclass MixerBlock(LudwigModule):\n def __init__(self, embed_size: int, n_patches: int, token_dim: int, channel_dim: int, dropout: float = 0.0):\n super().__init__()\n self._input_shape = (n_patches, embed_size)\n self._output_shape = (n_patches, embed_size)\n\n self.mlp1 = MLP(in_features=n_patches, hidden_size=token_dim, dropout=dropout)\n\n self.mlp2 = MLP(in_features=embed_size, hidden_size=channel_dim, dropout=dropout)\n\n self.layernorm1 = nn.LayerNorm(normalized_shape=embed_size)\n self.layernorm2 = nn.LayerNorm(normalized_shape=embed_size)\n\n def forward(self, inputs: torch.Tensor, **kwargs):\n assert inputs.shape[1:] == self.input_shape\n\n hidden = inputs\n hidden = self.layernorm1(hidden).transpose(1, 2)\n hidden = self.mlp1(hidden).transpose(1, 2)\n\n mid = hidden + inputs\n\n hidden = self.layernorm2(mid)\n hidden = self.mlp2(hidden)\n\n output = hidden + mid\n assert output.shape[1:] == self.output_shape\n return output\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size(self._input_shape)\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size(self._output_shape)\n\n\nclass MLPMixer(LudwigModule):\n \"\"\"MLPMixer.\n\n Implements\n MLP-Mixer: An all-MLP Architecture for Vision\n https://arxiv.org/abs/2105.01601\n \"\"\"\n\n def __init__(\n self,\n img_height: int,\n img_width: int,\n in_channels: int,\n patch_size: int = 16,\n embed_size: int = 512,\n token_size: int = 2048,\n channel_dim: int = 256,\n num_layers: int = 8,\n dropout: float = 0.0,\n avg_pool: bool = True,\n ):\n super().__init__()\n assert (img_height % patch_size == 0) and (img_width % patch_size == 0)\n\n self._input_shape = (in_channels, img_height, img_width)\n n_patches = int(img_height * img_width / (patch_size**2))\n\n self.patch_conv = nn.Conv2d(\n in_channels=in_channels, out_channels=embed_size, kernel_size=patch_size, stride=patch_size\n )\n\n self.mixer_blocks = nn.ModuleList(\n [\n MixerBlock(\n embed_size=embed_size,\n n_patches=n_patches,\n token_dim=token_size,\n channel_dim=channel_dim,\n dropout=dropout,\n )\n for _ in range(num_layers)\n ]\n )\n\n self.layer_norm = nn.LayerNorm(normalized_shape=embed_size)\n\n self.avg_pool = avg_pool\n if self.avg_pool:\n self._output_shape = torch.Size((embed_size,))\n else:\n self._output_shape = torch.Size((n_patches, embed_size))\n\n def forward(self, inputs: torch.Tensor) -> torch.Tensor:\n assert inputs.shape[1:] == self.input_shape\n hidden = self.patch_conv(inputs)\n hidden = hidden.flatten(2).transpose(1, 2)\n\n for mixer_block in self.mixer_blocks:\n hidden = mixer_block(hidden)\n hidden = self.layer_norm(hidden)\n\n if self.avg_pool:\n hidden = torch.mean(hidden, dim=1)\n\n assert hidden.shape[1:] == self.output_shape\n\n return hidden\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size(self._input_shape)\n\n @property\n def output_shape(self) -> torch.Size:\n return self._output_shape\n"
},
{
"path": "ludwig/modules/normalization_modules.py",
"content": "import logging\n\nimport numpy as np\nimport torch\nfrom torch.nn import BatchNorm1d, BatchNorm2d, LayerNorm, Module\n\nfrom ludwig.utils.torch_utils import LudwigModule\n\nlogger = logging.getLogger(__name__)\n\n\n# implementation adapted from https://github.com/dreamquark-ai/tabnet\nclass GhostBatchNormalization(LudwigModule):\n def __init__(\n self, num_features: int, momentum: float = 0.05, epsilon: float = 1e-3, virtual_batch_size: int | None = 128\n ):\n super().__init__()\n self.num_features = num_features\n self.virtual_batch_size = virtual_batch_size\n self.bn = torch.nn.BatchNorm1d(num_features, momentum=momentum, eps=epsilon)\n\n def forward(self, inputs):\n batch_size = inputs.shape[0]\n\n if self.training and self.virtual_batch_size:\n splits = inputs.chunk(int(np.ceil(batch_size / self.virtual_batch_size)), 0)\n\n if batch_size % self.virtual_batch_size == 1:\n # Skip batch normalization for the last chunk if it is size 1.\n logger.warning(\n f\"Virtual batch size `{self.virtual_batch_size}` is not a factor of the batch size `{batch_size}`, \"\n \"resulting in a chunk of size 1. Skipping batch normalization for the last chunk of size 1.\"\n )\n\n if batch_size == 1:\n logger.warning(\n \"Batch size is 1, but batch normalization requires batch size >= 2. Skipping batch normalization.\"\n \"Make sure to set `batch_size` to a value greater than 1.\"\n )\n # We temporarily set the batch_norm module to eval mode as we can't compute the running statistics\n # when the batch size is 1.\n self.bn.eval()\n splits_with_bn = [self.bn(x) if x.shape[0] >= 1 else x for x in splits]\n self.bn.train()\n else:\n splits_with_bn = [self.bn(x) if x.shape[0] > 1 else x for x in splits]\n\n return torch.cat(splits_with_bn, 0)\n\n if batch_size != 1 or not self.training:\n return self.bn(inputs)\n return inputs\n\n @property\n def moving_mean(self) -> torch.Tensor:\n return self.bn.running_mean\n\n @property\n def moving_variance(self) -> torch.Tensor:\n return self.bn.running_var\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.num_features])\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.num_features])\n\n\nclass BatchNorm1dOrIdentity(BatchNorm1d):\n \"\"\"BatchNorm1d or Identity layer if the batch_size is 1.\n\n Workaround for: https://github.com/pytorch/pytorch/issues/4534\n \"\"\"\n\n def forward(self, input: torch.Tensor) -> torch.Tensor:\n if input.shape[0] == 1:\n logger.warning(\n \"Batch size is 1, but batch normalization requires batch size >= 2. Skipping batch normalization.\"\n \"Make sure to set `batch_size` to a value greater than 1.\"\n )\n return input\n return super().forward(input)\n\n\nclass BatchNorm2dOrIdentity(BatchNorm2d):\n \"\"\"BatchNorm2d or Identity layer if the batch_size is 1.\n\n Workaround for: https://github.com/pytorch/pytorch/issues/4534\n \"\"\"\n\n def forward(self, input: torch.Tensor) -> torch.Tensor:\n if input.shape[0] == 1:\n logger.warning(\n \"Batch size is 1, but batch normalization requires batch size >= 2. Skipping batch normalization.\"\n \"Make sure to set `batch_size` to a value greater than 1.\"\n )\n return input\n return super().forward(input)\n\n\nnorm_registry = {\n \"batch_1d\": BatchNorm1dOrIdentity,\n \"batch_2d\": BatchNorm2dOrIdentity,\n \"layer\": LayerNorm,\n \"ghost\": GhostBatchNormalization,\n}\n\n\ndef create_norm_layer(norm: str, input_rank: int, num_features: int, **norm_params) -> Module:\n if norm == \"batch\":\n # We use a different batch norm depending on the input_rank.\n # TODO(travis): consider moving this behind a general BatchNorm interface to avoid this kludge.\n if input_rank not in {2, 3}:\n ValueError(f\"`input_rank` parameter expected to be either 2 or 3, but found {input_rank}.\")\n norm = f\"{norm}_{input_rank - 1}d\"\n\n norm_cls = norm_registry.get(norm)\n if norm_cls is None:\n raise ValueError(\n f\"Unsupported value for `norm` param: {norm}. Supported values are: {list(norm_registry.keys())}\"\n )\n\n return norm_cls(num_features, **norm_params)\n"
},
{
"path": "ludwig/modules/optimization_modules.py",
"content": "# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport dataclasses\nfrom typing import Optional, TYPE_CHECKING\n\nimport torch\n\nfrom ludwig.utils.misc_utils import get_from_registry\nfrom ludwig.utils.torch_utils import LudwigModule\n\nif TYPE_CHECKING:\n from ludwig.schema.optimizers import BaseOptimizerConfig, GradientClippingConfig\n\n\ndef create_clipper(gradient_clipping_config: Optional[\"GradientClippingConfig\"]):\n from ludwig.schema.optimizers import GradientClippingConfig\n\n \"\"\"Utility function that will convert a None-type gradient clipping config to the correct form.\"\"\"\n if isinstance(gradient_clipping_config, GradientClippingConfig):\n return gradient_clipping_config\n # Return default config if provided value is None:\n return GradientClippingConfig()\n\n\ndef get_optimizer_class_and_kwargs(\n optimizer_config: \"BaseOptimizerConfig\", learning_rate: float\n) -> tuple[type[torch.optim.Optimizer], dict]:\n \"\"\"Returns the optimizer class and kwargs for the optimizer.\n\n :return: Tuple of optimizer class and kwargs for the optimizer.\n \"\"\"\n from ludwig.schema.optimizers import optimizer_registry\n\n # Get the corresponding torch optimizer class for the given config:\n optimizer_cls = get_from_registry(optimizer_config.type.lower(), optimizer_registry)[0]\n\n # Create a dict of parameters to be passed to torch (i.e. everything except `type`):\n if dataclasses.is_dataclass(optimizer_config):\n config_dict = dataclasses.asdict(optimizer_config)\n elif hasattr(optimizer_config, \"to_dict\"):\n config_dict = optimizer_config.to_dict()\n else:\n config_dict = vars(optimizer_config)\n cls_kwargs = {field: value for field, value in config_dict.items() if field != \"type\"}\n cls_kwargs[\"lr\"] = learning_rate\n\n return optimizer_cls, cls_kwargs\n\n\ndef create_optimizer(\n model: LudwigModule,\n optimizer_config: \"BaseOptimizerConfig\",\n learning_rate: float,\n) -> torch.optim.Optimizer:\n \"\"\"Returns a ready-to-use torch optimizer instance based on the given optimizer config.\n\n :param model: Underlying Ludwig model\n :param learning_rate: Initial learning rate for the optimizer\n :param optimizer_config: Instance of `ludwig.modules.optimization_modules.BaseOptimizerConfig`.\n :return: Initialized instance of a torch optimizer.\n \"\"\"\n # Make sure the optimizer is compatible with the available resources:\n if (optimizer_config.is_paged or optimizer_config.is_8bit) and (\n not torch.cuda.is_available() or torch.cuda.device_count() == 0\n ):\n raise ValueError(\n \"Cannot use a paged or 8-bit optimizer on a non-GPU machine. \"\n \"Please use a different optimizer or run on a machine with a GPU.\"\n )\n\n optimizer_cls, optimizer_kwargs = get_optimizer_class_and_kwargs(optimizer_config, learning_rate)\n return optimizer_cls(model.parameters(), **optimizer_kwargs)\n"
},
{
"path": "ludwig/modules/recurrent_modules.py",
"content": "# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the 'License');\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\n\nimport torch\nfrom torch.nn import GRU, LSTM, RNN\n\nfrom ludwig.utils.misc_utils import get_from_registry\nfrom ludwig.utils.torch_utils import LudwigModule\n\nlogger = logging.getLogger(__name__)\n\nrnn_layers_registry = {\n \"rnn\": RNN,\n \"gru\": GRU,\n \"lstm\": LSTM,\n}\n\n\nclass RecurrentStack(LudwigModule):\n def __init__(\n self,\n input_size: int = None,\n hidden_size: int = 256,\n cell_type: str = \"rnn\",\n max_sequence_length: int | None = None,\n num_layers: int = 1,\n bidirectional: bool = False,\n use_bias: bool = True,\n dropout: float = 0.0,\n **kwargs,\n ):\n super().__init__()\n self.supports_masking = True\n self.input_size = input_size # api doc: H_in\n self.hidden_size = hidden_size # api doc: H_out\n self.max_sequence_length = max_sequence_length # api doc: L (sequence length)\n\n rnn_layer_class = get_from_registry(cell_type, rnn_layers_registry)\n\n rnn_params = {\"num_layers\": num_layers, \"bias\": use_bias, \"dropout\": dropout, \"bidirectional\": bidirectional}\n\n # Delegate recurrent params to PyTorch's RNN/GRU/LSTM implementations.\n self.layers = rnn_layer_class(input_size, hidden_size, batch_first=True, **rnn_params)\n\n @property\n def input_shape(self) -> torch.Size:\n if self.max_sequence_length:\n return torch.Size([self.max_sequence_length, self.input_size])\n return torch.Size([self.input_size])\n\n @property\n def output_shape(self) -> torch.Size:\n hidden_size = self.hidden_size * (2 if self.layers.bidirectional else 1)\n if self.max_sequence_length:\n return torch.Size([self.max_sequence_length, hidden_size])\n return torch.Size([hidden_size])\n\n def forward(self, inputs: torch.Tensor, mask=None):\n hidden, final_state = self.layers(inputs)\n\n if isinstance(final_state, tuple):\n # lstm cell type\n final_state = final_state[0][-1], final_state[1][-1]\n else:\n # rnn or gru cell type\n final_state = final_state[-1]\n\n return hidden, final_state\n"
},
{
"path": "ludwig/modules/reduction_modules.py",
"content": "# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\n\nimport torch\n\nfrom ludwig.modules.attention_modules import FeedForwardAttentionReducer\nfrom ludwig.utils.misc_utils import get_from_registry\nfrom ludwig.utils.torch_utils import LudwigModule, sequence_length_3D\n\nlogger = logging.getLogger(__name__)\n\n\nclass SequenceReducer(LudwigModule):\n \"\"\"Reduces the sequence dimension of an input tensor according to the specified reduce_mode. Any additional\n kwargs are passed on to the reduce mode's constructor. If using reduce_mode==\"attention\", the input_size kwarg\n must also be specified.\n\n A sequence is a tensor of 2 or more dimensions, where the shape is [batch size x sequence length x ...].\n\n :param reduce_mode: The reduction mode, one of {\"last\", \"sum\", \"mean\", \"max\", \"concat\", \"attention\", \"none\"}\n :param max_sequence_length The maximum sequence length. Only used for computation of shapes - inputs passed\n at runtime may have a smaller sequence length.\n :param encoding_size The size of each sequence element/embedding vector, or None if input is a sequence of scalars.\n \"\"\"\n\n def __init__(self, reduce_mode: str = None, max_sequence_length: int = 256, encoding_size: int = None, **kwargs):\n super().__init__()\n # save as private variable for debugging\n self._reduce_mode = reduce_mode\n self._max_sequence_length = max_sequence_length\n self._encoding_size = encoding_size\n # If embedding size specified and mode is attention, use embedding size as attention module input size\n # unless the input_size kwarg is provided.\n if reduce_mode == \"attention\" and encoding_size and \"input_size\" not in kwargs:\n kwargs[\"input_size\"] = encoding_size\n # use registry to find required reduction function\n self._reduce_obj = get_from_registry(reduce_mode, reduce_mode_registry)(**kwargs)\n\n def forward(self, inputs, mask=None):\n \"\"\"Forward pass of reducer.\n\n :param inputs: A tensor of 2 or more dimensions, where the shape is [batch size x sequence length x ...].\n :param mask: A mask tensor of 2 dimensions [batch size x sequence length]. Not yet implemented.\n\n :return: The input after applying the reduction operation to sequence dimension.\n \"\"\"\n return self._reduce_obj(inputs, mask=mask)\n\n @property\n def input_shape(self) -> torch.Size:\n \"\"\"Returns size of the input tensor without the batch dimension.\"\"\"\n if self._encoding_size is None:\n return torch.Size([self._max_sequence_length])\n else:\n return torch.Size([self._max_sequence_length, self._encoding_size])\n\n @property\n def output_shape(self) -> torch.Size:\n \"\"\"Returns size of the output tensor without the batch dimension.\"\"\"\n input_shape = self.input_shape\n if self._reduce_mode in {None, \"none\", \"None\"}:\n return input_shape\n elif self._reduce_mode == \"concat\":\n if len(input_shape) > 1:\n return input_shape[:-2] + (input_shape[-1] * input_shape[-2],)\n return input_shape\n else:\n return input_shape[1:] # Reduce sequence dimension.\n\n\nclass ReduceLast(torch.nn.Module):\n def forward(self, inputs, mask=None):\n # inputs: [batch_size, seq_size, hidden_size]\n batch_size = inputs.shape[0]\n # gather the correct outputs from the the RNN outputs (the outputs after sequence_length are all 0s)\n # todo: review for generality\n sequence_length = sequence_length_3D(inputs) - 1\n sequence_length[sequence_length < 0] = 0\n gathered = inputs[torch.arange(batch_size), sequence_length.type(torch.int64)]\n return gathered\n\n\nclass ReduceSum(torch.nn.Module):\n def forward(self, inputs, mask=None):\n return torch.sum(inputs, dim=1)\n\n\nclass ReduceMean(torch.nn.Module):\n def forward(self, inputs, mask=None):\n return torch.mean(inputs, dim=1)\n\n\nclass ReduceMax(torch.nn.Module):\n def forward(self, inputs, mask=None):\n return torch.amax(inputs, dim=1)\n\n\nclass ReduceConcat(torch.nn.Module):\n def forward(self, inputs, mask=None):\n if inputs.dim() > 2:\n return inputs.reshape(-1, inputs.shape[-1] * inputs.shape[-2])\n return inputs\n\n\nclass ReduceNone(torch.nn.Module):\n def forward(self, inputs, mask=None):\n return inputs\n\n\nreduce_mode_registry = {\n \"last\": ReduceLast,\n \"sum\": ReduceSum,\n \"mean\": ReduceMean,\n \"avg\": ReduceMean,\n \"max\": ReduceMax,\n \"concat\": ReduceConcat,\n \"attention\": FeedForwardAttentionReducer,\n # TODO: Simplify this.\n \"none\": ReduceNone,\n \"None\": ReduceNone,\n None: ReduceNone,\n}\n"
},
{
"path": "ludwig/modules/tabnet_modules.py",
"content": "import torch\nimport torch.nn as nn\n\nfrom ludwig.modules.normalization_modules import GhostBatchNormalization\nfrom ludwig.utils.entmax import Entmax15, EntmaxBisect, Sparsemax\nfrom ludwig.utils.torch_utils import LudwigModule\n\n\nclass TabNet(LudwigModule):\n def __init__(\n self,\n input_size: int,\n size: int,\n output_size: int,\n num_steps: int = 1,\n num_total_blocks: int = 4,\n num_shared_blocks: int = 2,\n relaxation_factor: float = 1.5,\n bn_momentum: float = 0.3,\n bn_epsilon: float = 1e-3,\n bn_virtual_bs: int | None = None,\n sparsity: float = 1e-5,\n entmax_mode: str = \"sparsemax\",\n entmax_alpha: float = 1.5,\n ):\n \"\"\"TabNet Will output a vector of size output_dim.\n\n Args:\n input_size: concatenated size of input feature encoder outputs\n size: Embedding feature dimension\n output_size: Output dimension for TabNet\n num_steps: Total number of steps.\n num_total_blocks: Total number of feature transformer blocks.\n num_shared_blocks: Number of shared feature transformer blocks.\n relaxation_factor: >1 will allow features to be used more than once.\n bn_momentum: Batch normalization, momentum.\n bn_epsilon: Batch normalization, epsilon.\n bn_virtual_bs: Virtual batch ize for ghost batch norm.\n entmax_mode: Entmax is a sparse family of probability mapping which generalizes softmax and sparsemax.\n entmax_mode controls the sparsity. One of {\"sparsemax\", \"entmax15\", \"constant\", \"adaptive\"}.\n entmax_alpha: Must be a number between 1.0 and 2.0. If entmax_mode is \"adaptive\", entmax_alpha is used\n as the initial value for the learnable parameter.\n \"\"\"\n super().__init__()\n self.input_size = input_size\n self.size = size\n self.output_size = output_size\n self.num_steps = num_steps\n self.bn_virtual_bs = bn_virtual_bs\n self.relaxation_factor = relaxation_factor\n self.sparsity = torch.tensor(sparsity)\n self.batch_norm = nn.BatchNorm1d(input_size, momentum=bn_momentum, eps=bn_epsilon)\n\n kargs = {\n \"num_total_blocks\": num_total_blocks,\n \"num_shared_blocks\": num_shared_blocks,\n \"bn_momentum\": bn_momentum,\n \"bn_epsilon\": bn_epsilon,\n \"bn_virtual_bs\": bn_virtual_bs,\n }\n\n # first feature transformer block is built first\n # to get the shared blocks\n self.feature_transforms = nn.ModuleList([FeatureTransformer(input_size, size + output_size, **kargs)])\n self.attentive_transforms = nn.ModuleList([None])\n for i in range(num_steps):\n self.feature_transforms.append(\n FeatureTransformer(\n input_size,\n size + output_size,\n **kargs,\n shared_fc_layers=self.feature_transforms[0].shared_fc_layers,\n )\n )\n # attentive transformers are initialized in build\n # because their outputs size depends on the number\n # of features that we determine by looking at the\n # last dimension of the input tensor\n self.attentive_transforms.append(\n AttentiveTransformer(\n size, input_size, bn_momentum, bn_epsilon, bn_virtual_bs, entmax_mode, entmax_alpha\n )\n )\n self.final_projection = nn.Linear(output_size, output_size)\n\n # Register tensors to be used in forward pass. This is needed in order to move these tensors\n # to the correct device (GPU/CPU) during the forward pass.\n self.register_buffer(\"out_accumulator\", torch.zeros(output_size))\n self.register_buffer(\"aggregated_mask\", torch.zeros(input_size))\n self.register_buffer(\"prior_scales\", torch.ones(input_size))\n\n def forward(self, features: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, list[torch.Tensor]]:\n if features.dim() != 2:\n raise ValueError(f\"Expecting incoming tensor to be dim 2, \" f\"instead dim={features.dim()}\")\n\n # shape notation\n # i_s: input_size\n # s: size\n # o_s: output_size\n # b_s: batch_size\n batch_size = features.shape[0] # b_s\n # Tile out_accumulator, aggregated_mask, and prior_scales to add batch dimension.\n out_accumulator = torch.tile(self.out_accumulator, (batch_size, 1))\n aggregated_mask = torch.tile(self.aggregated_mask, (batch_size, 1))\n prior_scales = torch.tile(self.prior_scales, (batch_size, 1))\n masks = []\n total_entropy = 0.0\n\n if batch_size != 1 or not self.training:\n # Skip batch normalization training if the batch size is 1.\n features = self.batch_norm(features) # [b_s, i_s]\n elif batch_size == 1:\n # We temporarily set the batch_norm module to eval mode as we can't compute the running statistics\n # when the batch size is 1.\n self.batch_norm.eval()\n features = self.batch_norm(features) # [b_s, i_s]\n self.batch_norm.train()\n masked_features = features\n\n x = self.feature_transforms[0](masked_features) # [b_s, s + o_s]\n\n for step_i in range(1, self.num_steps + 1):\n #########################\n # Attentive Transformer #\n #########################\n # x in following is shape [b_s, s]\n mask_values = self.attentive_transforms[step_i](x[:, self.output_size :], prior_scales) # [b_s, i_s]\n\n # relaxation factor 1 forces the feature to be only used once\n prior_scales = prior_scales * (self.relaxation_factor - mask_values) # [b_s, i_s]\n\n # entropy is used to penalize the amount of sparsity\n # in feature selection\n if self.sparsity.item() != 0.0:\n total_entropy += (\n torch.mean(torch.sum(-mask_values * torch.log(mask_values + 0.00001), dim=1)) / self.num_steps\n )\n\n masks.append(torch.unsqueeze(torch.unsqueeze(mask_values, 0), 3)) # [1, b_s, i_s, 1]\n\n #######################\n # Feature Transformer #\n #######################\n masked_features = torch.multiply(mask_values, features)\n\n x = self.feature_transforms[step_i](masked_features) # [b_s, s + o_s]\n\n # x in following is shape [b_s, o_s]\n out = nn.functional.relu(x[:, : self.output_size]) # [b_s, o_s]\n out_accumulator += out\n\n # Aggregated masks are used for visualization of the\n # feature importance attributes.\n scale = torch.sum(out, dim=1, keepdim=True) / self.num_steps\n aggregated_mask += mask_values * scale # [b_s, i_s]\n\n final_output = self.final_projection(out_accumulator) # [b_s, o_s]\n\n sparsity_loss = torch.multiply(self.sparsity, total_entropy)\n self.update_loss(\"sparsity_loss\", sparsity_loss)\n\n return final_output, aggregated_mask, masks\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.input_size])\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.output_size])\n\n\nclass FeatureBlock(LudwigModule):\n def __init__(\n self,\n input_size: int,\n size: int,\n apply_glu: bool = True,\n bn_momentum: float = 0.1,\n bn_epsilon: float = 1e-3,\n bn_virtual_bs: int = None,\n shared_fc_layer: LudwigModule = None,\n ):\n super().__init__()\n self.input_size = input_size\n self.apply_glu = apply_glu\n self.size = size\n units = size * 2 if apply_glu else size\n\n # Initialize fc_layer before assigning to shared layer for torchscript compatibilty\n self.fc_layer = nn.Linear(input_size, units, bias=False)\n if shared_fc_layer is not None:\n assert shared_fc_layer.weight.shape == self.fc_layer.weight.shape\n self.fc_layer = shared_fc_layer\n\n self.batch_norm = GhostBatchNormalization(\n units, virtual_batch_size=bn_virtual_bs, momentum=bn_momentum, epsilon=bn_epsilon\n )\n\n def forward(self, inputs):\n # shape notation\n # i_s: input_size\n # s: size\n # u: units\n # b_s: batch_size\n\n # inputs shape [b_s, i_s]\n hidden = self.fc_layer(inputs) # [b_s, u]\n hidden = self.batch_norm(hidden) # [b_s, u]\n if self.apply_glu:\n hidden = nn.functional.glu(hidden, dim=-1) # [bs, s]\n return hidden # [b_s, 2*s] if apply_glu else [b_s, s]\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.input_size])\n\n\nclass AttentiveTransformer(LudwigModule):\n def __init__(\n self,\n input_size: int,\n size: int,\n bn_momentum: float = 0.1,\n bn_epsilon: float = 1e-3,\n bn_virtual_bs: int = None,\n entmax_mode: str = \"sparsemax\",\n entmax_alpha: float = 1.5,\n ):\n super().__init__()\n self.input_size = input_size\n self.size = size\n self.entmax_mode = entmax_mode\n if entmax_mode == \"adaptive\":\n self.register_buffer(\"trainable_alpha\", torch.tensor(entmax_alpha, requires_grad=True))\n else:\n self.trainable_alpha = entmax_alpha\n\n if self.entmax_mode == \"sparsemax\":\n self.entmax_module = Sparsemax()\n elif self.entmax_mode == \"entmax15\":\n self.entmax_module = Entmax15()\n else:\n self.entmax_module = EntmaxBisect(alpha=self.trainable_alpha)\n\n self.feature_block = FeatureBlock(\n input_size,\n size,\n bn_momentum=bn_momentum,\n bn_epsilon=bn_epsilon,\n bn_virtual_bs=bn_virtual_bs,\n apply_glu=False,\n )\n\n def forward(self, inputs, prior_scales):\n # shape notation\n # i_s: input_size\n # s: size\n # b_s: batch_size\n\n # inputs shape [b_s, i_s], prior_scales shape [b_s, s]\n hidden = self.feature_block(inputs) # [b_s, s]\n hidden = hidden * prior_scales # [b_s, s]\n\n # removing the mean to try to avoid numerical instability\n # https://github.com/tensorflow/addons/issues/2314\n # https://github.com/tensorflow/tensorflow/pull/21183/files\n # In (Arik and Pfister, 2019), they call the logits z.\n # The mean(logits) can be substracted from logits to make the algorithm\n # more numerically stable. the instability in this algorithm comes mostly\n # from the z_cumsum. Substacting the mean will cause z_cumsum to be close\n # to zero.\n # hidden = hidden - tf.math.reduce_mean(hidden, axis=1)[:, tf.newaxis]\n return self.entmax_module(hidden)\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.input_size])\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.size])\n\n\n# adapted and modified from:\n# https://github.com/ostamand/tensorflow-tabnet/blob/master/tabnet/models/transformers.py\nclass FeatureTransformer(LudwigModule):\n def __init__(\n self,\n input_size: int,\n size: int,\n shared_fc_layers: list | None = None,\n num_total_blocks: int = 4,\n num_shared_blocks: int = 2,\n bn_momentum: float = 0.1,\n bn_epsilon: float = 1e-3,\n bn_virtual_bs: int = None,\n ):\n super().__init__()\n if shared_fc_layers is None:\n shared_fc_layers = []\n self.input_size = input_size\n self.num_total_blocks = num_total_blocks\n self.num_shared_blocks = num_shared_blocks\n self.size = size\n\n kwargs = {\n \"bn_momentum\": bn_momentum,\n \"bn_epsilon\": bn_epsilon,\n \"bn_virtual_bs\": bn_virtual_bs,\n }\n\n # build blocks\n self.blocks = nn.ModuleList()\n for n in range(num_total_blocks):\n # Ensure the sizes fed into FeatureBlock are correct regardless of presence of shared_fc_layer\n if n == 0:\n in_features = input_size\n else:\n in_features = size\n\n if shared_fc_layers and n < len(shared_fc_layers):\n self.blocks.append(FeatureBlock(in_features, size, **kwargs, shared_fc_layer=shared_fc_layers[n]))\n else:\n self.blocks.append(FeatureBlock(in_features, size, **kwargs))\n\n def forward(self, inputs: torch.Tensor) -> torch.Tensor:\n # shape notation\n # i_s: input_size\n # s: size\n # b_s: batch_size\n\n # inputs shape [b_s, i_s]\n hidden = self.blocks[0](inputs) # [b_s, s]\n for n in range(1, self.num_total_blocks):\n hidden = (self.blocks[n](hidden) + hidden) * (0.5**0.5) # [b_s, s]\n return hidden # [b_s, s]\n\n @property\n def shared_fc_layers(self):\n return [self.blocks[i].fc_layer for i in range(self.num_shared_blocks)]\n\n @property\n def input_shape(self) -> torch.Size:\n return torch.Size([self.input_size])\n\n @property\n def output_shape(self) -> torch.Size:\n return torch.Size([self.size])\n"
},
{
"path": "ludwig/modules/training_hooks.py",
"content": "import logging\nfrom abc import ABC, abstractmethod\n\nimport torch\n\nlogger = logging.getLogger(__name__)\n\n\nclass TrainingHook(ABC):\n \"\"\"A base class for training hooks in PyTorch.\n\n This class provides a template for implementing custom training hooks\n that can be activated, deactivated, and maintain a handle to the hook.\n\n Attributes:\n _hook_handle (Optional[torch.utils.hooks.RemovableHandle]): A handle to the\n registered forward hook, initially set to None.\n \"\"\"\n\n def __init__(self, *args, **kwargs) -> None:\n self._hook_handle = None\n\n @abstractmethod\n def hook_fn(self, module: torch.nn.Module, inputs: torch.tensor, outputs: torch.Tensor) -> torch.Tensor:\n \"\"\"Abstract method to be implemented by subclasses. This is the method that defines the custom behavior of\n the training hook during a forward pass for the specified module.\n\n Args:\n module (nn.Module): The PyTorch module for which the hook is activated.\n inputs (torch.Tensor): The input to the module during the forward pass.\n outputs (torch.Tensor): The output from the module during the forward pass.\n\n Returns:\n torch.Tensor: The output tensor from the module.\n\n Raises:\n NotImplementedError: If the method is not implemented in a subclass.\n \"\"\"\n\n def activate_hook(self, module: torch.nn.Module) -> torch.nn.Module:\n \"\"\"Activates the training hook for a given module.\n\n Args:\n module (nn.Module): The PyTorch module for which the hook is activated.\n\n Returns:\n nn.Module: The input module with the training hook activated.\n \"\"\"\n self._hook_handle = module.register_forward_hook(self.hook_fn)\n return module\n\n def deactivate_hook(self):\n \"\"\"Deactivates and removes the training hook.\"\"\"\n if self._hook_handle is not None:\n self._hook_handle.remove()\n self._hook_handle = None\n\n\nclass NEFTuneHook(TrainingHook):\n def __init__(self, *args, **kwargs) -> None:\n super().__init__(*args, **kwargs)\n self.neftune_noise_alpha = kwargs.get(\"neftune_noise_alpha\")\n\n def hook_fn(self, module: torch.nn.Module, input: torch.Tensor, output: torch.Tensor) -> torch.Tensor:\n \"\"\"Implements the NEFTune forward pass for the model using forward hooks. Note this works only for\n torch.nn. Embedding layers. This method is slightly adapted from the original source code that can be found\n here: https://github.com/neelsjain/NEFTune.\n\n The input tensor is ignored since the noise is added to the output of the embedding layer.\n\n Returns:\n torch.Tensor: The output tensor from the module.\n \"\"\"\n if module.training:\n dims = torch.tensor(output.size(1) * output.size(2))\n mag_norm = module.neftune_noise_alpha / torch.sqrt(dims)\n output = output + torch.zeros_like(output).uniform_(-mag_norm, mag_norm)\n return output\n\n def activate_hook(self, module: torch.nn.Module) -> torch.nn.Module:\n \"\"\"Activates the neftune as presented in this code and paper:\n\n Code: https://github.com/neelsjain/NEFTune\n Paper: https://arxiv.org/abs/2310.05914\n\n Args:\n module (nn.Module): The PyTorch module for which the hook is activated.\n\n Returns:\n nn.Module: The input module with the training hook activated.\n \"\"\"\n from peft import PeftModel\n\n if isinstance(module, PeftModel):\n embeddings = module.base_model.model.get_input_embeddings()\n else:\n embeddings = module.get_input_embeddings()\n\n embeddings.neftune_noise_alpha = self.neftune_noise_alpha\n self._hook_handle = embeddings.register_forward_hook(self.hook_fn)\n\n return module\n"
},
{
"path": "ludwig/predict.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport argparse\nimport logging\nimport sys\nfrom ast import literal_eval\n\nimport pandas as pd\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.backend import ALL_BACKENDS, Backend, initialize_backend\nfrom ludwig.callbacks import Callback\nfrom ludwig.constants import FULL, TEST, TRAINING, VALIDATION\nfrom ludwig.contrib import add_contrib_callback_args\nfrom ludwig.globals import LUDWIG_VERSION\nfrom ludwig.utils.print_utils import get_logging_level_registry, print_ludwig\n\nlogger = logging.getLogger(__name__)\n\n\ndef predict_cli(\n model_path: str,\n dataset: str | dict | pd.DataFrame = None,\n data_format: str = None,\n split: str = FULL,\n batch_size: int = 128,\n generation_config: str | None = None,\n skip_save_unprocessed_output: bool = False,\n skip_save_predictions: bool = False,\n output_directory: str = \"results\",\n gpus: str | int | list[int] = None,\n gpu_memory_limit: float | None = None,\n allow_parallel_threads: bool = True,\n callbacks: list[Callback] = None,\n backend: Backend | str = None,\n logging_level: int = logging.INFO,\n **kwargs,\n) -> None:\n \"\"\"Loads pre-trained model to make predictions on the provided data set.\n\n # Inputs\n\n :param model_path: (str) filepath to pre-trained model.\n :param dataset: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing the entire dataset to be used in the prediction.\n :param data_format: (str, default: `None`) format to interpret data\n sources. Will be inferred automatically if not specified. Valid\n formats are `'auto'`, `'csv'`, `'excel'`, `'feather'`,\n `'fwf'`, `'hdf5'` (cache file produced during previous training),\n `'html'` (file containing a single HTML `
`), `'json'`, `'jsonl'`,\n `'parquet'`, `'pickle'` (pickled Pandas DataFrame), `'sas'`, `'spss'`,\n `'stata'`, `'tsv'`.\n :param split: (str, default: `full`) split on which\n to perform predictions. Valid values are `'training'`, `'validation'`,\n `'test'` and `'full'`.\n :param batch_size: (int, default `128`) size of batches for processing.\n :param generation_config: (str, default: `None`) a string representing\n the parameters for generation required to perform predictions with\n an LLM. The string must be a JSON formatted dictionary with keys from\n https://huggingface.co/docs/transformers/main_classes/text_generation#transformers.GenerationConfig\n These will be merged with the generation parameters from the original\n model config.\n :param skip_save_unprocessed_output: (bool, default: `False`) by default\n predictions and their probabilities are saved in both raw\n unprocessed numpy files containing tensors and as postprocessed\n CSV files (one for each output feature). If this parameter is True,\n only the CSV ones are saved and the numpy ones are skipped.\n :param skip_save_predictions: (bool, default: `False`) skips saving test\n predictions CSV files\n :param output_directory: (str, default: `'results'`) the directory that\n will contain the training statistics, TensorBoard logs, the saved\n model and the training progress files.\n :param gpus: (list, default: `None`) list of GPUs that are available\n for training.\n :param gpu_memory_limit: (float: default: `None`) maximum memory fraction\n [0, 1] allowed to allocate per GPU device.\n :param allow_parallel_threads: (bool, default: `True`) allow PyTorch\n to use multithreading parallelism to improve performance at\n the cost of determinism.\n :param callbacks: (list, default: `None`) a list of\n `ludwig.callbacks.Callback` objects that provide hooks into the\n Ludwig pipeline.\n :param backend: (Union[Backend, str]) `Backend` or string name\n of backend to use to execute preprocessing / training steps.\n :param logging_level: (int) Log level that will be sent to stderr.\n\n # Returns\n\n :return: ('None')\n \"\"\"\n model = LudwigModel.load(\n model_path,\n logging_level=logging_level,\n backend=backend,\n gpus=gpus,\n gpu_memory_limit=gpu_memory_limit,\n allow_parallel_threads=allow_parallel_threads,\n callbacks=callbacks,\n )\n model.predict(\n dataset=dataset,\n data_format=data_format,\n split=split,\n batch_size=batch_size,\n generation_config=literal_eval(generation_config) if generation_config else None,\n skip_save_unprocessed_output=skip_save_unprocessed_output,\n skip_save_predictions=skip_save_predictions,\n output_directory=output_directory,\n return_type=\"dict\",\n )\n\n\ndef cli(sys_argv):\n parser = argparse.ArgumentParser(\n description=\"This script loads a pretrained model \" \"and uses it to predict\",\n prog=\"ludwig predict\",\n usage=\"%(prog)s [options]\",\n )\n\n # ---------------\n # Data parameters\n # ---------------\n parser.add_argument(\"--dataset\", help=\"input data file path\", required=True)\n parser.add_argument(\n \"--data_format\",\n help=\"format of the input data\",\n default=\"auto\",\n choices=[\n \"auto\",\n \"csv\",\n \"excel\",\n \"feather\",\n \"fwf\",\n \"hdf5\",\n \"html\",\n \"tables\",\n \"json\",\n \"jsonl\",\n \"parquet\",\n \"pickle\",\n \"sas\",\n \"spss\",\n \"stata\",\n \"tsv\",\n ],\n )\n parser.add_argument(\n \"-s\", \"--split\", default=FULL, choices=[TRAINING, VALIDATION, TEST, FULL], help=\"the split to test the model on\"\n )\n\n # ----------------\n # Model parameters\n # ----------------\n parser.add_argument(\"-m\", \"--model_path\", help=\"model to load\", required=True)\n parser.add_argument(\"-gc\", \"--generation_config\", help=\"generation config (LLMs only)\", default=None)\n\n # -------------------------\n # Output results parameters\n # -------------------------\n parser.add_argument(\n \"-od\", \"--output_directory\", type=str, default=\"results\", help=\"directory that contains the results\"\n )\n parser.add_argument(\n \"-ssuo\",\n \"--skip_save_unprocessed_output\",\n help=\"skips saving intermediate NPY output files\",\n action=\"store_true\",\n default=False,\n )\n parser.add_argument(\n \"-sstp\",\n \"--skip_save_predictions\",\n help=\"skips saving predictions CSV files\",\n action=\"store_true\",\n default=False,\n )\n\n # ------------------\n # Generic parameters\n # ------------------\n parser.add_argument(\"-bs\", \"--batch_size\", type=int, default=128, help=\"size of batches\")\n\n # ------------------\n # Runtime parameters\n # ------------------\n parser.add_argument(\"-g\", \"--gpus\", type=int, default=0, help=\"list of gpu to use\")\n parser.add_argument(\n \"-gml\",\n \"--gpu_memory_limit\",\n type=float,\n default=None,\n help=\"maximum memory fraction [0, 1] allowed to allocate per GPU device\",\n )\n parser.add_argument(\n \"-dpt\",\n \"--disable_parallel_threads\",\n action=\"store_false\",\n dest=\"allow_parallel_threads\",\n help=\"disable PyTorch from using multithreading for reproducibility\",\n )\n parser.add_argument(\n \"-b\",\n \"--backend\",\n help=\"specifies backend to use for parallel / distributed execution, \" \"defaults to local execution\",\n choices=ALL_BACKENDS,\n )\n parser.add_argument(\n \"-l\",\n \"--logging_level\",\n default=\"info\",\n help=\"the level of logging to use\",\n choices=[\"critical\", \"error\", \"warning\", \"info\", \"debug\", \"notset\"],\n )\n\n add_contrib_callback_args(parser)\n args = parser.parse_args(sys_argv)\n\n args.callbacks = args.callbacks or []\n for callback in args.callbacks:\n callback.on_cmdline(\"predict\", *sys_argv)\n\n args.logging_level = get_logging_level_registry()[args.logging_level]\n logging.getLogger(\"ludwig\").setLevel(args.logging_level)\n global logger\n logger = logging.getLogger(\"ludwig.predict\")\n\n args.backend = initialize_backend(args.backend)\n if args.backend.is_coordinator():\n print_ludwig(\"Predict\", LUDWIG_VERSION)\n logger.info(f\"Dataset path: {args.dataset}\")\n logger.info(f\"Model path: {args.model_path}\")\n logger.info(\"\")\n\n predict_cli(**vars(args))\n\n\nif __name__ == \"__main__\":\n cli(sys.argv[1:])\n"
},
{
"path": "ludwig/preprocess.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport argparse\nimport logging\nimport sys\n\nimport pandas as pd\nimport yaml\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.backend import ALL_BACKENDS, Backend, initialize_backend\nfrom ludwig.callbacks import Callback\nfrom ludwig.contrib import add_contrib_callback_args\nfrom ludwig.globals import LUDWIG_VERSION\nfrom ludwig.utils.data_utils import load_yaml\nfrom ludwig.utils.defaults import default_random_seed\nfrom ludwig.utils.print_utils import get_logging_level_registry, print_ludwig\n\nlogger = logging.getLogger(__name__)\n\n\ndef preprocess_cli(\n preprocessing_config: str | dict = None,\n dataset: str | dict | pd.DataFrame = None,\n training_set: str | dict | pd.DataFrame = None,\n validation_set: str | dict | pd.DataFrame = None,\n test_set: str | dict | pd.DataFrame = None,\n training_set_metadata: str | dict = None,\n data_format: str = None,\n random_seed: int = default_random_seed,\n logging_level: int = logging.INFO,\n callbacks: list[Callback] = None,\n backend: Backend | str = None,\n **kwargs\n) -> None:\n \"\"\"*train* defines the entire training procedure used by Ludwig's internals. Requires most of the parameters\n that are taken into the model. Builds a full ludwig model and performs the training.\n\n :param preprocessing_config: (Union[str, dict]) in-memory representation of\n config or string path to a YAML config file.\n :param dataset: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing the entire dataset to be used for training.\n If it has a split column, it will be used for splitting (0 for train,\n 1 for validation, 2 for test), otherwise the dataset will be\n randomly split.\n :param training_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing training data.\n :param validation_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing validation data.\n :param test_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing test data.\n :param training_set_metadata: (Union[str, dict], default: `None`)\n metadata JSON file or loaded metadata. Intermediate preprocessed\n structure containing the mappings of the input\n dataset created the first time an input file is used in the same\n directory with the same name and a '.meta.json' extension.\n :param data_format: (str, default: `None`) format to interpret data\n sources. Will be inferred automatically if not specified. Valid\n formats are `'auto'`, `'csv'`, `'excel'`, `'feather'`,\n `'fwf'`, `'hdf5'` (cache file produced during previous training),\n `'html'` (file containing a single HTML `
`), `'json'`, `'jsonl'`,\n `'parquet'`, `'pickle'` (pickled Pandas DataFrame), `'sas'`, `'spss'`,\n `'stata'`, `'tsv'`.\n :param experiment_name: (str, default: `'experiment'`) name for\n the experiment.\n :param model_name: (str, default: `'run'`) name of the model that is\n being used.\n :param model_load_path: (str, default: `None`) if this is specified the\n loaded model will be used as initialization\n (useful for transfer learning).\n :param model_resume_path: (str, default: `None`) resumes training of\n the model from the path specified. The config is restored.\n In addition to config, training statistics, loss for each\n epoch and the state of the optimizer are restored such that\n training can be effectively continued from a previously interrupted\n training process.\n :param skip_save_training_description: (bool, default: `False`) disables\n saving the description JSON file.\n :param skip_save_training_statistics: (bool, default: `False`) disables\n saving training statistics JSON file.\n :param skip_save_model: (bool, default: `False`) disables\n saving model weights and hyperparameters each time the model\n improves. By default Ludwig saves model weights after each epoch\n the validation metric improves, but if the model is really big\n that can be time consuming. If you do not want to keep\n the weights and just find out what performance a model can get\n with a set of hyperparameters, use this parameter to skip it,\n but the model will not be loadable later on and the returned model\n will have the weights obtained at the end of training, instead of\n the weights of the epoch with the best validation performance.\n :param skip_save_progress: (bool, default: `False`) disables saving\n progress each epoch. By default Ludwig saves weights and stats\n after each epoch for enabling resuming of training, but if\n the model is really big that can be time consuming and will uses\n twice as much space, use this parameter to skip it, but training\n cannot be resumed later on.\n :param skip_save_log: (bool, default: `False`) disables saving\n TensorBoard logs. By default Ludwig saves logs for the TensorBoard,\n but if it is not needed turning it off can slightly increase the\n overall speed.\n :param skip_save_processed_input: (bool, default: `False`) if input\n dataset is provided it is preprocessed and cached by saving an HDF5\n and JSON files to avoid running the preprocessing again. If this\n parameter is `False`, the HDF5 and JSON file are not saved.\n :param output_directory: (str, default: `'results'`) the directory that\n will contain the training statistics, TensorBoard logs, the saved\n model and the training progress files.\n :param gpus: (list, default: `None`) list of GPUs that are available\n for training.\n :param gpu_memory_limit: (float: default: `None`) maximum memory fraction\n [0, 1] allowed to allocate per GPU device.\n :param allow_parallel_threads: (bool, default: `True`) allow PyTorch\n to use multithreading parallelism to improve performance at\n the cost of determinism.\n :param callbacks: (list, default: `None`) a list of\n `ludwig.callbacks.Callback` objects that provide hooks into the\n Ludwig pipeline.\n :param backend: (Union[Backend, str]) `Backend` or string name\n of backend to use to execute preprocessing / training steps.\n :param random_seed: (int: default: 42) random seed used for weights\n initialization, splits and any other random function.\n :param logging_level: (int) Log level that will be sent to stderr.\n\n # Return\n\n :return: (`None`)\n \"\"\"\n model = LudwigModel(\n config=preprocessing_config,\n logging_level=logging_level,\n callbacks=callbacks,\n backend=backend,\n )\n model.preprocess(\n dataset=dataset,\n training_set=training_set,\n validation_set=validation_set,\n test_set=test_set,\n training_set_metadata=training_set_metadata,\n data_format=data_format,\n skip_save_processed_input=False,\n random_seed=random_seed,\n )\n\n\ndef cli(sys_argv):\n parser = argparse.ArgumentParser(\n description=\"This script preprocess a dataset\", prog=\"ludwig preprocess\", usage=\"%(prog)s [options]\"\n )\n\n # ---------------\n # Data parameters\n # ---------------\n parser.add_argument(\n \"--dataset\",\n help=\"input data file path. \"\n \"If it has a split column, it will be used for splitting \"\n \"(0: train, 1: validation, 2: test), \"\n \"otherwise the dataset will be randomly split\",\n )\n parser.add_argument(\"--training_set\", help=\"input train data file path\")\n parser.add_argument(\"--validation_set\", help=\"input validation data file path\")\n parser.add_argument(\"--test_set\", help=\"input test data file path\")\n\n parser.add_argument(\n \"--training_set_metadata\",\n help=\"input metadata JSON file path. An intermediate preprocessed file \"\n \"containing the mappings of the input file created \"\n \"the first time a file is used, in the same directory \"\n \"with the same name and a .json extension\",\n )\n\n parser.add_argument(\n \"--data_format\",\n help=\"format of the input data\",\n default=\"auto\",\n choices=[\n \"auto\",\n \"csv\",\n \"excel\",\n \"feather\",\n \"fwf\",\n \"hdf5\",\n \"html\" \"tables\",\n \"json\",\n \"jsonl\",\n \"parquet\",\n \"pickle\",\n \"sas\",\n \"spss\",\n \"stata\",\n \"tsv\",\n ],\n )\n\n # ----------------\n # Model parameters\n # ----------------\n preprocessing_def = parser.add_mutually_exclusive_group(required=True)\n preprocessing_def.add_argument(\n \"-pc\",\n \"--preprocessing_config\",\n dest=\"preprocessing_config\",\n type=load_yaml,\n help=\"YAML file describing the preprocessing. \"\n \"Ignores --preprocessing_config.\"\n \"Uses the same format of config, \"\n \"but ignores encoder specific parameters, \"\n \"decoder specific parameters, combiner and training parameters\",\n )\n preprocessing_def.add_argument(\n \"-pcs\",\n \"--preprocessing_config_str\",\n type=yaml.safe_load,\n help=\"preproceesing config. \"\n \"Uses the same format of config, \"\n \"but ignores encoder specific parameters, \"\n \"decoder specific parameters, combiner and training parameters\",\n )\n\n # ------------------\n # Runtime parameters\n # ------------------\n parser.add_argument(\n \"-rs\",\n \"--random_seed\",\n type=int,\n default=42,\n help=\"a random seed that is going to be used anywhere there is a call \"\n \"to a random number generator: data splitting, parameter \"\n \"initialization and training set shuffling\",\n )\n parser.add_argument(\n \"-b\",\n \"--backend\",\n help=\"specifies backend to use for parallel / distributed execution, \" \"defaults to local execution\",\n choices=ALL_BACKENDS,\n )\n parser.add_argument(\n \"-l\",\n \"--logging_level\",\n default=\"info\",\n help=\"the level of logging to use\",\n choices=[\"critical\", \"error\", \"warning\", \"info\", \"debug\", \"notset\"],\n )\n\n add_contrib_callback_args(parser)\n args = parser.parse_args(sys_argv)\n\n args.callbacks = args.callbacks or []\n for callback in args.callbacks:\n callback.on_cmdline(\"preprocess\", *sys_argv)\n\n args.logging_level = get_logging_level_registry()[args.logging_level]\n logging.getLogger(\"ludwig\").setLevel(args.logging_level)\n global logger\n logger = logging.getLogger(\"ludwig.preprocess\")\n\n args.backend = initialize_backend(args.backend)\n if args.backend.is_coordinator():\n print_ludwig(\"Preprocess\", LUDWIG_VERSION)\n\n preprocess_cli(**vars(args))\n\n\nif __name__ == \"__main__\":\n cli(sys.argv[1:])\n"
},
{
"path": "ludwig/progress_bar.py",
"content": "import uuid\n\nimport tqdm\n\ntry:\n import ray.train as rt\nexcept ImportError:\n rt = None\n\n\nclass LudwigProgressBarActions:\n CREATE = \"create\"\n UPDATE = \"update\"\n CLOSE = \"close\"\n\n\nclass LudwigProgressBar:\n \"\"\"Class for progress bars that supports distributed progress bars in ray.\n\n # Inputs\n\n :param report_to_ray: (bool) use the ray.train.report method\n to report progress to the ray driver. If false then this behaves as a normal tqdm\n progress bar\n :param config: (dict) the tqdm configs used for the progress bar. See https://github.com/tqdm/tqdm#parameters\n for list of parameters\n :param is_coordinator: (bool) whether the calling process is the coordinator process.\n\n # Example usage:\n\n ```python\n from ludwig.progress_bar import LudwigProgressBar\n\n config = {\"total\": 20, \"desc\": \"Sample progress bar\"}\n pbar = LudwigProgressBar(report_to_ray=False, config=config, is_coordinator=True)\n for i in range(20):\n pbar.update(1)\n pbar.close()\n ```\n \"\"\"\n\n def __init__(\n self,\n report_to_ray: bool,\n config: dict,\n is_coordinator: bool,\n ) -> None:\n \"\"\"Constructor for the LudwigProgressBar class.\n\n # Inputs\n\n :param report_to_ray: (bool) use the ray.train.report method\n to report progress to the ray driver. If false then this behaves as a normal tqdm\n progress bar\n :param config: (dict) the tqdm configs used for the progress bar. See https://github.com/tqdm/tqdm#parameters\n for list of parameters\n :param is_coordinator: (bool) whether the calling process is the coordinator process.\n\n # Return\n\n :return: (None) `None`\n \"\"\"\n if report_to_ray and rt is None:\n raise ValueError(\"Set report_to_ray=True but ray is not installed. Run `pip install ray`\")\n\n self.id = str(uuid.uuid4())[-8:]\n\n self.report_to_ray = report_to_ray\n self.is_coordinator = is_coordinator\n self.config = config\n\n self.total_steps = 0\n self.progress_bar = None\n\n if not self.report_to_ray:\n if self.is_coordinator:\n self.progress_bar = tqdm.tqdm(**config)\n else:\n if \"file\" in self.config:\n self.config.pop(\"file\")\n # All processes need to call ray.train.report since ray has a lock that blocks\n # a process when calling report if there are processes that haven't called it. Similar\n # to a distributed checkpoint. Therefore we pass the flag to the driver.\n # In Ray 2.x, rt.report() only accepts metrics and checkpoint kwargs,\n # so we pass progress_bar data inside the metrics dict.\n rt.report(\n metrics={\n \"progress_bar\": {\n \"id\": self.id,\n \"config\": self.config,\n \"action\": LudwigProgressBarActions.CREATE,\n \"is_coordinator\": self.is_coordinator,\n }\n }\n )\n\n def set_postfix(self, ordered_dict: dict = None, **kwargs) -> None:\n \"\"\"Sets the postfix (additional stats) for the progress bar.\"\"\"\n if self.progress_bar:\n self.progress_bar.set_postfix(ordered_dict, **kwargs)\n\n def update(self, steps: int) -> None:\n \"\"\"Updates the progress bar.\n\n # Inputs\n\n :param steps: (int) number of steps to update the progress bar by\n\n # Return\n\n :return: (None) `None`\n \"\"\"\n self.total_steps += steps\n if self.progress_bar:\n self.progress_bar.update(steps)\n elif self.report_to_ray:\n rt.report(\n metrics={\n \"progress_bar\": {\n \"id\": self.id,\n \"update_by\": steps,\n \"is_coordinator\": self.is_coordinator,\n \"action\": LudwigProgressBarActions.UPDATE,\n }\n }\n )\n\n def close(self) -> None:\n \"\"\"Closes the progress bar.\n\n # Return\n\n :return: (None) `None`\n \"\"\"\n if self.progress_bar:\n self.progress_bar.close()\n elif self.report_to_ray:\n rt.report(\n metrics={\n \"progress_bar\": {\n \"id\": self.id,\n \"is_coordinator\": self.is_coordinator,\n \"action\": LudwigProgressBarActions.CLOSE,\n }\n }\n )\n"
},
{
"path": "ludwig/schema/__init__.py",
"content": "# TODO(travis): figure out why we need these imports to avoid circular import error\nfrom ludwig.schema.combiners.utils import get_combiner_jsonschema # noqa\nfrom ludwig.schema.features.utils import get_input_feature_jsonschema, get_output_feature_jsonschema # noqa\nfrom ludwig.schema.hyperopt import get_hyperopt_jsonschema # noqa\nfrom ludwig.schema.trainer import get_model_type_jsonschema, get_trainer_jsonschema # noqa\n"
},
{
"path": "ludwig/schema/combiners/__init__.py",
"content": "import ludwig.schema.combiners.comparator # noqa: F401\nimport ludwig.schema.combiners.concat # noqa: F401\nimport ludwig.schema.combiners.project_aggregate # noqa: F401\nimport ludwig.schema.combiners.sequence # noqa: F401\nimport ludwig.schema.combiners.sequence_concat # noqa: F401\nimport ludwig.schema.combiners.tab_transformer # noqa: F401\nimport ludwig.schema.combiners.tabnet # noqa: F401\nimport ludwig.schema.combiners.transformer # noqa: F401\n"
},
{
"path": "ludwig/schema/combiners/base.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass BaseCombinerConfig(schema_utils.BaseMarshmallowConfig):\n \"\"\"Base combiner config class.\"\"\"\n\n type: str\n"
},
{
"path": "ludwig/schema/combiners/common_transformer_options.py",
"content": "from typing import Any\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.schema import common_fields\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.metadata import COMBINER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass CommonTransformerConfig:\n \"\"\"Common transformer parameter values.\"\"\"\n\n dropout: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"Dropout rate for the transformer block.\",\n parameter_metadata=COMBINER_METADATA[\"transformer\"][\"dropout\"],\n )\n\n transformer_output_size: int = schema_utils.NonNegativeInteger(\n default=256,\n description=\"Size of the fully connected layer after self attention in the transformer block. This is usually \"\n \"the same as `hidden_size` and `embedding_size`.\",\n parameter_metadata=COMBINER_METADATA[\"transformer\"][\"transformer_output_size\"],\n )\n\n hidden_size: int = schema_utils.NonNegativeInteger(\n default=256,\n description=\"The number of hidden units of the TransformerStack as well as the dimension that each incoming \"\n \"input feature is projected to before feeding to the TransformerStack.\",\n parameter_metadata=COMBINER_METADATA[\"transformer\"][\"hidden_size\"],\n )\n\n num_layers: int = schema_utils.PositiveInteger(\n default=1,\n description=\"The number of transformer layers.\",\n parameter_metadata=COMBINER_METADATA[\"transformer\"][\"num_layers\"],\n )\n\n num_heads: int = schema_utils.NonNegativeInteger(\n default=8,\n description=\"Number of heads of the self attention in the transformer block.\",\n parameter_metadata=COMBINER_METADATA[\"transformer\"][\"num_heads\"],\n )\n\n use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether the layer uses a bias vector.\",\n parameter_metadata=COMBINER_METADATA[\"transformer\"][\"use_bias\"],\n )\n\n bias_initializer: str | dict = common_fields.BiasInitializerField()\n\n weights_initializer: str | dict = common_fields.WeightsInitializerField()\n\n # TODO(#1673): Add conditional logic for fields like this one:\n num_fc_layers: int = schema_utils.NonNegativeInteger(\n default=0,\n description=\"The number of stacked fully connected layers (only applies if `reduce_output` is not null).\",\n parameter_metadata=COMBINER_METADATA[\"transformer\"][\"num_fc_layers\"],\n )\n\n output_size: int = schema_utils.PositiveInteger(\n default=256,\n description=\"Output size of a fully connected layer.\",\n parameter_metadata=COMBINER_METADATA[\"transformer\"][\"output_size\"],\n )\n\n norm: str | None = common_fields.NormField()\n\n norm_params: dict | None = common_fields.NormParamsField()\n\n fc_layers: list[dict[str, Any]] | None = common_fields.FCLayersField()\n\n fc_dropout: float = common_fields.DropoutField()\n\n fc_activation: str = schema_utils.ActivationOptions(\n default=\"relu\",\n parameter_metadata=COMBINER_METADATA[\"transformer\"][\"fc_activation\"],\n )\n\n fc_residual: bool = common_fields.ResidualField()\n"
},
{
"path": "ludwig/schema/combiners/comparator.py",
"content": "from typing import Any\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.schema import common_fields\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.combiners.base import BaseCombinerConfig\nfrom ludwig.schema.combiners.utils import register_combiner_config\nfrom ludwig.schema.metadata import COMBINER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@register_combiner_config(\"comparator\")\n@ludwig_dataclass\nclass ComparatorCombinerConfig(BaseCombinerConfig):\n \"\"\"Parameters for comparator combiner.\"\"\"\n\n def __post_init__(self):\n if self.num_fc_layers == 0 and self.fc_layers is None:\n raise ConfigValidationError(\n \"`combiner.type=comparator` requires at least one fully connected layer. \"\n \"Set `num_fc_layers > 0` or `fc_layers`.\"\n )\n\n if not self.entity_1:\n raise ConfigValidationError(\n \"`combiner.entity_1` is required and must contain as least one input feature name.\"\n )\n\n if not self.entity_2:\n raise ConfigValidationError(\n \"`combiner.entity_2` is required and must contain as least one input feature name.\"\n )\n\n type: str = schema_utils.ProtectedString(\n \"comparator\",\n description=COMBINER_METADATA[\"comparator\"][\"type\"].long_description,\n )\n\n entity_1: list[str] = schema_utils.List(\n default=None,\n description=(\n \"The list of input feature names `[feature_1, feature_2, ...]` constituting the first entity to compare. \"\n \"*Required*.\"\n ),\n parameter_metadata=COMBINER_METADATA[\"comparator\"][\"entity_1\"],\n )\n\n entity_2: list[str] = schema_utils.List(\n default=None,\n description=(\n \"The list of input feature names `[feature_1, feature_2, ...]` constituting the second entity to compare. \"\n \"*Required*.\"\n ),\n parameter_metadata=COMBINER_METADATA[\"comparator\"][\"entity_2\"],\n )\n\n dropout: float = common_fields.DropoutField()\n\n activation: str = schema_utils.ActivationOptions(default=\"relu\")\n\n use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether the layer uses a bias vector.\",\n parameter_metadata=COMBINER_METADATA[\"comparator\"][\"use_bias\"],\n )\n\n bias_initializer: str | dict = common_fields.BiasInitializerField()\n\n weights_initializer: str | dict = common_fields.WeightsInitializerField()\n\n num_fc_layers: int = common_fields.NumFCLayersField(default=1)\n\n output_size: int = schema_utils.PositiveInteger(\n default=256,\n description=\"Output size of a fully connected layer.\",\n parameter_metadata=COMBINER_METADATA[\"comparator\"][\"output_size\"],\n )\n\n norm: str | None = common_fields.NormField()\n\n norm_params: dict | None = common_fields.NormParamsField()\n\n fc_layers: list[dict[str, Any]] | None = common_fields.FCLayersField()\n"
},
{
"path": "ludwig/schema/combiners/concat.py",
"content": "from typing import Any\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.schema import common_fields\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.combiners.base import BaseCombinerConfig\nfrom ludwig.schema.combiners.utils import register_combiner_config\nfrom ludwig.schema.metadata import COMBINER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@register_combiner_config(\"concat\")\n@ludwig_dataclass\nclass ConcatCombinerConfig(BaseCombinerConfig):\n \"\"\"Parameters for concat combiner.\"\"\"\n\n type: str = schema_utils.ProtectedString(\n \"concat\",\n description=COMBINER_METADATA[\"concat\"][\"type\"].long_description,\n )\n\n dropout: float = common_fields.DropoutField()\n\n activation: str = schema_utils.ActivationOptions(default=\"relu\")\n\n flatten_inputs: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to flatten input tensors to a vector.\",\n parameter_metadata=COMBINER_METADATA[\"concat\"][\"flatten_inputs\"],\n )\n\n residual: bool = common_fields.ResidualField()\n\n use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether the layer uses a bias vector.\",\n parameter_metadata=COMBINER_METADATA[\"concat\"][\"use_bias\"],\n )\n\n bias_initializer: str | dict = common_fields.BiasInitializerField()\n\n weights_initializer: str | dict = common_fields.WeightsInitializerField()\n\n num_fc_layers: int = common_fields.NumFCLayersField()\n\n output_size: int = schema_utils.PositiveInteger(\n default=256,\n description=\"Output size of a fully connected layer.\",\n parameter_metadata=COMBINER_METADATA[\"concat\"][\"output_size\"],\n )\n\n norm: str | None = common_fields.NormField()\n\n norm_params: dict | None = common_fields.NormParamsField()\n\n fc_layers: list[dict[str, Any]] | None = common_fields.FCLayersField()\n"
},
{
"path": "ludwig/schema/combiners/project_aggregate.py",
"content": "from typing import Any\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.combiners.base import BaseCombinerConfig\nfrom ludwig.schema.combiners.utils import register_combiner_config\nfrom ludwig.schema.metadata import COMBINER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@register_combiner_config(\"project_aggregate\")\n@ludwig_dataclass\nclass ProjectAggregateCombinerConfig(BaseCombinerConfig):\n type: str = schema_utils.ProtectedString(\n \"project_aggregate\",\n description=COMBINER_METADATA[\"project_aggregate\"][\"type\"].long_description,\n )\n\n projection_size: int = schema_utils.PositiveInteger(\n default=128,\n description=\"All combiner inputs are projected to this size before being aggregated.\",\n parameter_metadata=COMBINER_METADATA[\"project_aggregate\"][\"projection_size\"],\n )\n\n residual: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to add residual skip connection between the fully connected layers in the stack.\",\n parameter_metadata=COMBINER_METADATA[\"project_aggregate\"][\"residual\"],\n )\n\n dropout: float = schema_utils.FloatRange(\n default=0.0,\n min=0,\n max=1,\n description=\"Dropout rate to apply to each fully connected layer.\",\n parameter_metadata=COMBINER_METADATA[\"project_aggregate\"][\"dropout\"],\n )\n\n activation: str = schema_utils.ActivationOptions(\n default=\"relu\",\n description=\"Activation to apply to each fully connected layer.\",\n parameter_metadata=COMBINER_METADATA[\"project_aggregate\"][\"activation\"],\n )\n\n num_fc_layers: int = schema_utils.NonNegativeInteger(\n default=2,\n description=\"Number of fully connected layers after aggregation.\",\n parameter_metadata=COMBINER_METADATA[\"project_aggregate\"][\"num_fc_layers\"],\n )\n\n output_size: int = schema_utils.PositiveInteger(\n default=128,\n description=\"Output size of each layer of the stack of fully connected layers.\",\n parameter_metadata=COMBINER_METADATA[\"project_aggregate\"][\"output_size\"],\n )\n\n norm: str | None = schema_utils.StringOptions(\n [\"batch\", \"layer\"],\n default=\"layer\",\n description=\"Normalization to apply to each projection and fully connected layer.\",\n parameter_metadata=COMBINER_METADATA[\"project_aggregate\"][\"norm\"],\n )\n\n norm_params: dict | None = schema_utils.Dict(\n description=\"Parameters of the normalization to apply to each projection and fully connected layer.\",\n parameter_metadata=COMBINER_METADATA[\"project_aggregate\"][\"norm_params\"],\n )\n\n fc_layers: list[dict[str, Any]] | None = schema_utils.DictList(\n description=\"Full specification of the fully connected layers after the aggregation. It should be a list of \"\n \"dict, each dict representing one layer of the fully connected layer stack. \",\n parameter_metadata=COMBINER_METADATA[\"project_aggregate\"][\"fc_layers\"],\n )\n\n use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether the layers use a bias vector.\",\n parameter_metadata=COMBINER_METADATA[\"project_aggregate\"][\"use_bias\"],\n )\n\n bias_initializer: str | dict = schema_utils.InitializerOrDict(\n default=\"zeros\",\n description=\"Initializer to use for the bias of the projection and for the fully connected layers.\",\n parameter_metadata=COMBINER_METADATA[\"project_aggregate\"][\"bias_initializer\"],\n )\n\n weights_initializer: str | dict = schema_utils.InitializerOrDict(\n default=\"xavier_uniform\",\n description=\"Initializer to use for the weights of the projection and for the fully connected layers.\",\n parameter_metadata=COMBINER_METADATA[\"project_aggregate\"][\"weights_initializer\"],\n )\n"
},
{
"path": "ludwig/schema/combiners/sequence.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import MODEL_ECD, SEQUENCE\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.combiners.base import BaseCombinerConfig\nfrom ludwig.schema.combiners.sequence_concat import MAIN_SEQUENCE_FEATURE_DESCRIPTION\nfrom ludwig.schema.combiners.utils import register_combiner_config\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import EncoderDataclassField\nfrom ludwig.schema.metadata import COMBINER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\"\"\"\nSEQUENCE encoders that always return 2D [batch_size, hidden_size] tensors, regardless of how they are parameterized.\nThese should never be used with modules that expect 3D tensors, such as the SequenceCombiner.\n\"\"\"\n_2D_SEQUENCE_ENCODERS = [\"embed\"]\n\n\n@DeveloperAPI\n@register_combiner_config(\"sequence\")\n@ludwig_dataclass\nclass SequenceCombinerConfig(BaseCombinerConfig):\n \"\"\"Parameters for sequence combiner.\"\"\"\n\n type: str = schema_utils.ProtectedString(\n \"sequence\",\n description=COMBINER_METADATA[\"sequence\"][\"type\"].long_description,\n )\n\n main_sequence_feature: str | None = schema_utils.String(\n default=None,\n allow_none=True,\n description=MAIN_SEQUENCE_FEATURE_DESCRIPTION,\n parameter_metadata=COMBINER_METADATA[\"sequence\"][\"main_sequence_feature\"],\n )\n\n encoder: BaseEncoderConfig = EncoderDataclassField(\n MODEL_ECD,\n feature_type=SEQUENCE,\n default=\"parallel_cnn\",\n description=\"Encoder to apply to `main_sequence_feature`. The encoder must produce\"\n \" a tensor of size [batch_size, sequence_length, hidden_size]\",\n blocklist=_2D_SEQUENCE_ENCODERS,\n )\n\n reduce_output: str | None = schema_utils.ReductionOptions(\n default=None,\n description=\"Strategy to use to aggregate the embeddings of the items of the set.\",\n parameter_metadata=COMBINER_METADATA[\"sequence\"][\"reduce_output\"],\n )\n"
},
{
"path": "ludwig/schema/combiners/sequence_concat.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.combiners.base import BaseCombinerConfig\nfrom ludwig.schema.combiners.utils import register_combiner_config\nfrom ludwig.schema.metadata import COMBINER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\nMAIN_SEQUENCE_FEATURE_DESCRIPTION = \"\"\"\nName of a sequence, text, or time series feature to concatenate the outputs\nof the other features to. If no `main_sequence_feature` is specified, the combiner will look through all the features in\nthe order they are defined in the configuration and will look for a feature with a rank 3 tensor output (sequence, text\nor time series). If it cannot find one it will raise an exception, otherwise the output of that feature will be used for\nconcatenating the other features along the sequence `s` dimension. If there are other input features with a rank 3\noutput tensor, the combiner will concatenate them alongside the `s` dimension. All sequence-like input features must\nhave identical `s` dimension, otherwise an error will be thrown.\n\"\"\"\n\n\n@DeveloperAPI\n@register_combiner_config(\"sequence_concat\")\n@ludwig_dataclass\nclass SequenceConcatCombinerConfig(BaseCombinerConfig):\n \"\"\"Parameters for sequence concat combiner.\"\"\"\n\n @staticmethod\n def module_name():\n return \"sequence_concat\"\n\n type: str = schema_utils.ProtectedString(\n \"sequence_concat\",\n description=COMBINER_METADATA[\"sequence_concat\"][\"type\"].long_description,\n )\n\n main_sequence_feature: str | None = schema_utils.String(\n default=None,\n allow_none=True,\n description=MAIN_SEQUENCE_FEATURE_DESCRIPTION,\n parameter_metadata=COMBINER_METADATA[\"sequence_concat\"][\"main_sequence_feature\"],\n )\n\n reduce_output: str | None = schema_utils.ReductionOptions(\n default=None,\n description=\"Strategy to use to aggregate the embeddings of the items of the set.\",\n parameter_metadata=COMBINER_METADATA[\"sequence_concat\"][\"reduce_output\"],\n )\n"
},
{
"path": "ludwig/schema/combiners/tab_transformer.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.combiners.base import BaseCombinerConfig\nfrom ludwig.schema.combiners.common_transformer_options import CommonTransformerConfig\nfrom ludwig.schema.combiners.utils import register_combiner_config\nfrom ludwig.schema.metadata import COMBINER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@register_combiner_config(\"tabtransformer\")\n@ludwig_dataclass\nclass TabTransformerCombinerConfig(BaseCombinerConfig, CommonTransformerConfig):\n \"\"\"Parameters for tab transformer combiner.\"\"\"\n\n type: str = schema_utils.ProtectedString(\n \"tabtransformer\",\n description=COMBINER_METADATA[\"tabtransformer\"][\"type\"].long_description,\n )\n\n embed_input_feature_name: str | int | None = schema_utils.Embed(\n description=\"This value controls the size of the embeddings. Valid values are `add` which uses the \"\n \"`hidden_size` value or an integer that is set to a specific value. In the case of an integer \"\n \"value, it must be smaller than hidden_size.\",\n parameter_metadata=COMBINER_METADATA[\"tabtransformer\"][\"embed_input_feature_name\"],\n )\n\n reduce_output: str = schema_utils.ReductionOptions(\n default=\"concat\",\n description=\"Strategy to use to aggregate the output of the transformer.\",\n parameter_metadata=COMBINER_METADATA[\"tabtransformer\"][\"reduce_output\"],\n )\n"
},
{
"path": "ludwig/schema/combiners/tabnet.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.combiners.base import BaseCombinerConfig\nfrom ludwig.schema.combiners.utils import register_combiner_config\nfrom ludwig.schema.metadata import COMBINER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@register_combiner_config(\"tabnet\")\n@ludwig_dataclass\nclass TabNetCombinerConfig(BaseCombinerConfig):\n \"\"\"Parameters for tabnet combiner.\"\"\"\n\n type: str = schema_utils.ProtectedString(\n \"tabnet\",\n description=COMBINER_METADATA[\"tabnet\"][\"type\"].long_description,\n )\n\n size: int = schema_utils.PositiveInteger(\n default=32,\n description=\"Size of the hidden layers. `N_a` in (Arik and Pfister, 2019).\",\n parameter_metadata=COMBINER_METADATA[\"tabnet\"][\"size\"],\n )\n\n dropout: float = schema_utils.FloatRange(\n default=0.05,\n min=0,\n max=1,\n description=\"Dropout rate for the transformer block.\",\n parameter_metadata=COMBINER_METADATA[\"tabnet\"][\"dropout\"],\n )\n\n output_size: int = schema_utils.PositiveInteger(\n default=128,\n description=\"Output size of a fully connected layer. `N_d` in (Arik and Pfister, 2019).\",\n parameter_metadata=COMBINER_METADATA[\"tabnet\"][\"output_size\"],\n )\n\n num_steps: int = schema_utils.NonNegativeInteger(\n default=3,\n description=\"Number of steps / repetitions of the the attentive transformer and feature transformer \"\n \"computations. `N_steps` in (Arik and Pfister, 2019).\",\n parameter_metadata=COMBINER_METADATA[\"tabnet\"][\"num_steps\"],\n )\n\n num_total_blocks: int = schema_utils.NonNegativeInteger(\n default=4,\n description=\"Total number of feature transformer blocks at each step.\",\n parameter_metadata=COMBINER_METADATA[\"tabnet\"][\"num_total_blocks\"],\n )\n\n num_shared_blocks: int = schema_utils.NonNegativeInteger(\n default=2,\n description=\"Number of shared feature transformer blocks across the steps.\",\n parameter_metadata=COMBINER_METADATA[\"tabnet\"][\"num_shared_blocks\"],\n )\n\n relaxation_factor: float = schema_utils.FloatRange(\n default=1.5,\n description=\"Factor that influences how many times a feature should be used across the steps of computation. \"\n \"a value of 1 implies it each feature should be use once, a higher value allows for multiple \"\n \"usages. `gamma` in (Arik and Pfister, 2019).\",\n parameter_metadata=COMBINER_METADATA[\"tabnet\"][\"relaxation_factor\"],\n )\n\n bn_epsilon: float = schema_utils.FloatRange(\n default=1e-3,\n description=\"Epsilon to be added to the batch norm denominator.\",\n parameter_metadata=COMBINER_METADATA[\"tabnet\"][\"bn_epsilon\"],\n )\n\n bn_momentum: float = schema_utils.FloatRange(\n default=0.05,\n description=\"Momentum of the batch norm. 1 - `m_B` from the TabNet paper.\",\n parameter_metadata=COMBINER_METADATA[\"tabnet\"][\"bn_momentum\"],\n )\n\n bn_virtual_bs: int | None = schema_utils.PositiveInteger(\n default=1024,\n allow_none=True,\n description=\"Size of the virtual batch size used by ghost batch norm. If null, regular batch norm is used \"\n \"instead. `B_v` from the TabNet paper.\",\n parameter_metadata=COMBINER_METADATA[\"tabnet\"][\"bn_virtual_bs\"],\n )\n\n sparsity: float = schema_utils.FloatRange(\n default=1e-4,\n description=\"Multiplier of the sparsity inducing loss. `lambda_sparse` in (Arik and Pfister, 2019).\",\n parameter_metadata=COMBINER_METADATA[\"tabnet\"][\"sparsity\"],\n )\n\n entmax_mode: str = schema_utils.StringOptions(\n [\"entmax15\", \"sparsemax\", \"constant\", \"adaptive\"],\n default=\"sparsemax\",\n description=(\n \"Entmax is a sparse family of probability mapping which generalizes softmax and sparsemax. \"\n \"`entmax_mode` controls the sparsity\"\n ),\n parameter_metadata=COMBINER_METADATA[\"tabnet\"][\"entmax_mode\"],\n )\n\n entmax_alpha: float = schema_utils.FloatRange(\n default=1.5,\n min=1,\n max=2,\n description=(\n \"Must be a number between 1.0 and 2.0. If entmax_mode is `adaptive`, \"\n \"`entmax_alpha` is used as the initial value for the learnable parameter. \"\n \"1 corresponds to softmax, 2 is sparsemax.\"\n ),\n parameter_metadata=COMBINER_METADATA[\"tabnet\"][\"entmax_alpha\"],\n )\n"
},
{
"path": "ludwig/schema/combiners/transformer.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.combiners.base import BaseCombinerConfig\nfrom ludwig.schema.combiners.common_transformer_options import CommonTransformerConfig\nfrom ludwig.schema.combiners.utils import register_combiner_config\nfrom ludwig.schema.metadata import COMBINER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@register_combiner_config(\"transformer\")\n@ludwig_dataclass\nclass TransformerCombinerConfig(BaseCombinerConfig, CommonTransformerConfig):\n \"\"\"Parameters for transformer combiner.\"\"\"\n\n type: str = schema_utils.ProtectedString(\n \"transformer\",\n description=COMBINER_METADATA[\"transformer\"][\"type\"].long_description,\n )\n\n reduce_output: str | None = schema_utils.ReductionOptions(\n default=\"mean\",\n description=\"Strategy to use to aggregate the output of the transformer.\",\n parameter_metadata=COMBINER_METADATA[\"transformer\"][\"reduce_output\"],\n )\n"
},
{
"path": "ludwig/schema/combiners/utils.py",
"content": "from typing import Any\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import TYPE\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.combiners.base import BaseCombinerConfig\nfrom ludwig.schema.metadata import COMBINER_METADATA\nfrom ludwig.schema.metadata.parameter_metadata import convert_metadata_to_json, ParameterMetadata\nfrom ludwig.utils.registry import Registry\n\nDEFAULT_VALUE = \"concat\"\nDESCRIPTION = \"Select the combiner type.\"\n\ncombiner_config_registry = Registry[type[BaseCombinerConfig]]()\n\n\n@DeveloperAPI\ndef register_combiner_config(name: str):\n def wrap(cls: type[BaseCombinerConfig]):\n combiner_config_registry[name] = cls\n return cls\n\n return wrap\n\n\n@DeveloperAPI\ndef get_combiner_registry():\n return combiner_config_registry\n\n\n@DeveloperAPI\ndef get_combiner_jsonschema():\n \"\"\"Returns a JSON schema structured to only require a `type` key and then conditionally apply a corresponding\n combiner's field constraints.\"\"\"\n combiner_types = sorted(list(combiner_config_registry.keys()))\n parameter_metadata = convert_metadata_to_json(\n ParameterMetadata.from_dict(\n {\n \"commonly_used\": True,\n \"expected_impact\": 3,\n \"ui_display_name\": \"Combiner Type\",\n }\n )\n )\n return {\n \"type\": \"object\",\n \"properties\": {\n \"type\": {\n \"type\": \"string\",\n \"enum\": combiner_types,\n \"enumDescriptions\": get_combiner_descriptions(),\n \"default\": DEFAULT_VALUE,\n \"title\": \"combiner_options\",\n \"description\": DESCRIPTION,\n \"parameter_metadata\": parameter_metadata,\n },\n },\n \"allOf\": get_combiner_conds(),\n \"required\": [\"type\"],\n }\n\n\n@DeveloperAPI\ndef get_combiner_descriptions():\n \"\"\"This function returns a dictionary of combiner descriptions available at the type selection.\n\n The process works as follows - 1) Get a dictionary of valid combiners from the combiner config registry,\n but inverse the key/value pairs since we need to index `valid_combiners` later with an altered version\n of the combiner config class name. 2) Loop through Combiner Metadata entries, if a metadata entry has a\n combiner name that matches a valid combiner, add the description metadata to the output dictionary.\n\n Returns:\n dict: A dictionary of combiner descriptions.\n \"\"\"\n return {k: convert_metadata_to_json(v[TYPE]) for k, v in COMBINER_METADATA.items() if k in combiner_config_registry}\n\n\n@DeveloperAPI\ndef get_combiner_conds() -> list[dict[str, Any]]:\n \"\"\"Returns a list of if-then JSON clauses for each combiner type in `combiner_registry` and its properties'\n constraints.\"\"\"\n combiner_types = sorted(list(combiner_config_registry.keys()))\n conds = []\n for combiner_type in combiner_types:\n combiner_cls = combiner_config_registry[combiner_type]\n schema_cls = combiner_cls\n combiner_schema = schema_utils.unload_jsonschema_from_marshmallow_class(schema_cls)\n combiner_props = combiner_schema[\"properties\"]\n schema_utils.remove_duplicate_fields(combiner_props)\n combiner_cond = schema_utils.create_cond({\"type\": combiner_type}, combiner_props)\n conds.append(combiner_cond)\n return conds\n\n\nclass CombinerSelection(schema_utils.TypeSelection):\n def __init__(self):\n # For registration of all combiners\n import ludwig.combiners.combiners # noqa\n\n super().__init__(registry=combiner_config_registry, default_value=DEFAULT_VALUE, description=DESCRIPTION)\n\n def get_schema_from_registry(self, key: str) -> type[schema_utils.BaseMarshmallowConfig]:\n return self.registry[key]\n\n def _jsonschema_type_mapping(self):\n return get_combiner_jsonschema()\n"
},
{
"path": "ludwig/schema/common_fields.py",
"content": "from dataclasses import Field\n\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.metadata import COMMON_METADATA\nfrom ludwig.schema.metadata.parameter_metadata import ParameterMetadata\nfrom ludwig.utils.torch_utils import initializer_registry\n\n\ndef DropoutField(default: float = 0.0, description: str = None, parameter_metadata: ParameterMetadata = None) -> Field:\n description = description or \"Default dropout rate applied to fully connected layers.\"\n full_description = description + (\n \" Increasing dropout is a common form of regularization to combat overfitting. \"\n \"The dropout is expressed as the probability of an element to be zeroed out (0.0 means no dropout).\"\n )\n parameter_metadata = parameter_metadata or COMMON_METADATA[\"dropout\"]\n return schema_utils.FloatRange(\n default=default,\n min=0,\n max=1,\n description=full_description,\n parameter_metadata=parameter_metadata,\n )\n\n\ndef ResidualField(\n default: bool = False, description: str = None, parameter_metadata: ParameterMetadata = None\n) -> Field:\n description = description or (\n \"Whether to add a residual connection to each fully connected layer block. \"\n \"Requires all fully connected layers to have the same `output_size`.\"\n )\n parameter_metadata = parameter_metadata or COMMON_METADATA[\"residual\"]\n return schema_utils.Boolean(\n default=False,\n description=description,\n parameter_metadata=parameter_metadata,\n )\n\n\ndef NumFCLayersField(\n default: int = 0, description: str = None, parameter_metadata: ParameterMetadata = None, non_zero=False\n) -> Field:\n assert (not non_zero) or (default > 0 and non_zero)\n\n description = description or \"Number of stacked fully connected layers to apply.\"\n full_description = description + (\n \" Increasing layers adds capacity to the model, enabling it to learn more complex feature interactions.\"\n )\n parameter_metadata = parameter_metadata or COMMON_METADATA[\"num_fc_layers\"]\n\n # When using a dense encoder, the number of fully connected layers must be strictly greater than 0.\n if non_zero:\n return schema_utils.PositiveInteger(\n default=default, allow_none=False, description=full_description, parameter_metadata=parameter_metadata\n )\n return schema_utils.NonNegativeInteger(\n default=default,\n allow_none=False,\n description=full_description,\n parameter_metadata=parameter_metadata,\n )\n\n\ndef NormField(\n default: str | None = None, description: str = None, parameter_metadata: ParameterMetadata = None\n) -> Field:\n description = description or \"Default normalization applied at the beginnging of fully connected layers.\"\n parameter_metadata = parameter_metadata or COMMON_METADATA[\"norm\"]\n return schema_utils.StringOptions(\n [\"batch\", \"layer\", \"ghost\"],\n default=default,\n allow_none=True,\n description=description,\n parameter_metadata=parameter_metadata,\n )\n\n\ndef NormParamsField(description: str = None, parameter_metadata: ParameterMetadata = None) -> Field:\n description = description or \"Default parameters passed to the `norm` module.\"\n parameter_metadata = parameter_metadata or COMMON_METADATA[\"norm_params\"]\n return schema_utils.Dict(\n description=description,\n parameter_metadata=parameter_metadata,\n )\n\n\ndef FCLayersField(description: str = None, parameter_metadata: ParameterMetadata = None) -> Field:\n description = description or (\n \"List of dictionaries containing the parameters of all the fully connected layers. \"\n \"The length of the list determines the number of stacked fully connected layers \"\n \"and the content of each dictionary determines the parameters for a specific layer. \"\n \"The available parameters for each layer are: `activation`, `dropout`, `norm`, `norm_params`, \"\n \"`output_size`, `use_bias`, `bias_initializer` and `weights_initializer`. If any of those values \"\n \"is missing from the dictionary, the default one provided as a standalone parameter will be used instead.\"\n )\n parameter_metadata = parameter_metadata or COMMON_METADATA[\"fc_layers\"]\n return schema_utils.DictList(\n description=description,\n parameter_metadata=parameter_metadata,\n )\n\n\nINITIALIZER_SUFFIX = \"\"\"\nAlternatively it is possible to specify a dictionary with a key `type` that identifies the type of initializer and\nother keys for its parameters, e.g. `{type: normal, mean: 0, stddev: 0}`. For a description of the parameters of each\ninitializer, see [torch.nn.init](https://pytorch.org/docs/stable/nn.init.html).\n\"\"\"\n\n\ndef BiasInitializerField(\n default: str = \"zeros\", description: str = None, parameter_metadata: ParameterMetadata = None\n) -> Field:\n initializers_str = \", \".join([f\"`{i}`\" for i in initializer_registry.keys()])\n description = description or \"Initializer for the bias vector.\"\n full_description = f\"{description} Options: {initializers_str}. {INITIALIZER_SUFFIX}\"\n parameter_metadata = parameter_metadata or COMMON_METADATA[\"bias_initializer\"]\n return schema_utils.InitializerOrDict(\n default=default,\n description=full_description,\n parameter_metadata=parameter_metadata,\n )\n\n\ndef WeightsInitializerField(\n default: str = \"xavier_uniform\", description: str = None, parameter_metadata: ParameterMetadata = None\n) -> Field:\n initializers_str = \", \".join([f\"`{i}`\" for i in initializer_registry.keys()])\n description = description or \"Initializer for the weight matrix.\"\n full_description = f\"{description} Options: {initializers_str}. {INITIALIZER_SUFFIX}\"\n parameter_metadata = parameter_metadata or COMMON_METADATA[\"weights_initializer\"]\n return schema_utils.InitializerOrDict(\n default=default,\n description=full_description,\n parameter_metadata=parameter_metadata,\n )\n\n\ndef EmbeddingInitializerField(\n default: str | None = None, description: str = None, parameter_metadata: ParameterMetadata = None\n) -> Field:\n description = description or \"Initializer for the embedding matrix.\"\n parameter_metadata = parameter_metadata or COMMON_METADATA[\"embedding_initializer\"]\n return schema_utils.StringOptions(\n list(initializer_registry.keys()),\n default=default,\n allow_none=True,\n description=description,\n parameter_metadata=parameter_metadata,\n )\n\n\ndef EmbeddingSizeField(\n default: int = 256, description: str = None, parameter_metadata: ParameterMetadata = None\n) -> Field:\n description = description or (\n \"The maximum embedding size. The actual size will be `min(vocabulary_size, embedding_size)` for \"\n \"`dense` representations and exactly `vocabulary_size` for the `sparse` encoding, where `vocabulary_size` \"\n \"is the number of unique strings appearing in the training set input column plus the number of \"\n \"special tokens (``, ``, ``, ``).\"\n )\n parameter_metadata = parameter_metadata or COMMON_METADATA[\"embedding_size\"]\n return schema_utils.PositiveInteger(\n default=default,\n description=description,\n parameter_metadata=parameter_metadata,\n )\n\n\ndef EmbeddingsOnCPUField(\n default: bool = False, description: str = None, parameter_metadata: ParameterMetadata = None\n) -> Field:\n description = description or (\n \"Whether to force the placement of the embedding matrix in regular memory and have the CPU resolve them. \"\n \"By default embedding matrices are stored on GPU memory if a GPU is used, as it allows for faster access, \"\n \"but in some cases the embedding matrix may be too large. This parameter forces the placement of the \"\n \"embedding matrix in regular memory and the CPU is used for embedding lookup, slightly slowing down the \"\n \"process as a result of data transfer between CPU and GPU memory.\"\n )\n parameter_metadata = parameter_metadata or COMMON_METADATA[\"embeddings_on_cpu\"]\n return schema_utils.Boolean(\n default=default,\n description=description,\n parameter_metadata=parameter_metadata,\n )\n\n\ndef EmbeddingsTrainableField(\n default: bool = True, description: str = None, parameter_metadata: ParameterMetadata = None\n) -> Field:\n description = description or (\n \"If `true` embeddings are trained during the training process, if `false` embeddings are fixed. \"\n \"It may be useful when loading pretrained embeddings for avoiding finetuning them. This parameter \"\n \"has effect only when `representation` is `dense`; `sparse` one-hot encodings are not trainable.\"\n )\n parameter_metadata = parameter_metadata or COMMON_METADATA[\"embeddings_trainable\"]\n return schema_utils.Boolean(\n default=default,\n description=description,\n parameter_metadata=parameter_metadata,\n )\n\n\ndef PretrainedEmbeddingsField(\n default: str | None = None, description: str = None, parameter_metadata: ParameterMetadata = None\n) -> Field:\n description = description or (\n \"Path to a file containing pretrained embeddings. By default `dense` embeddings are initialized \"\n \"randomly, but this parameter allows to specify a path to a file containing embeddings in the \"\n \"[GloVe format](https://nlp.stanford.edu/projects/glove/). When the file containing the embeddings is \"\n \"loaded, only the embeddings with labels present in the vocabulary are kept, the others are discarded. \"\n \"If the vocabulary contains strings that have no match in the embeddings file, their embeddings are \"\n \"initialized with the average of all other embedding plus some random noise to make them different \"\n \"from each other. This parameter has effect only if `representation` is `dense`.\"\n )\n parameter_metadata = parameter_metadata or COMMON_METADATA[\"pretrained_embeddings\"]\n return schema_utils.String(\n default=default,\n allow_none=True,\n description=description,\n parameter_metadata=parameter_metadata,\n )\n\n\ndef MaxSequenceLengthField(\n default: int | None = None, description: str = None, parameter_metadata: ParameterMetadata = None\n) -> Field:\n description = description or \"[internal] Maximum sequence length from preprocessing.\"\n parameter_metadata = parameter_metadata or COMMON_METADATA[\"max_sequence_length\"]\n return schema_utils.PositiveInteger(\n default=default,\n allow_none=True,\n description=description,\n parameter_metadata=parameter_metadata,\n )\n\n\ndef VocabField(\n default: list | None = None, description: str = None, parameter_metadata: ParameterMetadata = None\n) -> Field:\n description = description or \"[internal] Vocabulary for the encoder from preprocessing.\"\n parameter_metadata = parameter_metadata or COMMON_METADATA[\"vocab\"]\n return schema_utils.List(\n default=default,\n description=description,\n parameter_metadata=parameter_metadata,\n )\n\n\ndef VocabSizeField(\n default: list | None = None, description: str = None, parameter_metadata: ParameterMetadata = None\n) -> Field:\n description = description or \"[internal] Size of the vocabulary from preprocessing.\"\n parameter_metadata = parameter_metadata or COMMON_METADATA[\"vocab_size\"]\n return schema_utils.PositiveInteger(\n default=default,\n allow_none=True,\n description=description,\n parameter_metadata=parameter_metadata,\n )\n\n\ndef RepresentationField(\n default: str = \"dense\", description: str = None, parameter_metadata: ParameterMetadata = None\n) -> Field:\n description = description or (\n \"Representation of the embedding. `dense` means the embeddings are initialized randomly, \"\n \"`sparse` means they are initialized to be one-hot encodings.\"\n )\n parameter_metadata = parameter_metadata or COMMON_METADATA[\"representation\"]\n return schema_utils.StringOptions(\n [\"dense\", \"sparse\"],\n default=default,\n description=description,\n parameter_metadata=parameter_metadata,\n )\n\n\ndef ReduceOutputField(\n default: str | None = \"sum\", description: str = None, parameter_metadata: ParameterMetadata = None\n) -> Field:\n description = description or (\n \"How to reduce the output tensor along the `s` sequence length dimension if the rank of the \"\n \"tensor is greater than 2.\"\n )\n parameter_metadata = parameter_metadata or COMMON_METADATA[\"reduce_output\"]\n return schema_utils.ReductionOptions(\n default=default,\n description=description,\n parameter_metadata=parameter_metadata,\n )\n"
},
{
"path": "ludwig/schema/decoders/__init__.py",
"content": "# Register all decoders\nimport ludwig.schema.decoders.base\nimport ludwig.schema.decoders.image_decoders # noqa\nimport ludwig.schema.decoders.llm_decoders # noqa\nimport ludwig.schema.decoders.sequence_decoders # noqa\n"
},
{
"path": "ludwig/schema/decoders/base.py",
"content": "from abc import ABC\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import BINARY, CATEGORY, MODEL_ECD, MODEL_LLM, NUMBER, SET, TIMESERIES, VECTOR\nfrom ludwig.schema import common_fields\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.decoders.utils import register_decoder_config\nfrom ludwig.schema.metadata import DECODER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass BaseDecoderConfig(schema_utils.BaseMarshmallowConfig, ABC):\n \"\"\"Base class for decoders.\"\"\"\n\n type: str = schema_utils.StringOptions(\n [\"regressor\", \"classifier\", \"projector\", \"generator\", \"tagger\"],\n default=None,\n allow_none=True,\n description=\"The type of decoder to use.\",\n parameter_metadata=DECODER_METADATA[\"BaseDecoder\"][\"type\"],\n )\n\n fc_layers: list[dict] = common_fields.FCLayersField()\n\n num_fc_layers: int = common_fields.NumFCLayersField(\n description=\"Number of fully-connected layers if `fc_layers` not specified.\"\n )\n\n fc_output_size: int = schema_utils.PositiveInteger(\n default=256,\n description=\"Output size of fully connected stack.\",\n parameter_metadata=DECODER_METADATA[\"BaseDecoder\"][\"fc_output_size\"],\n )\n\n fc_use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether the layer uses a bias vector in the fc_stack.\",\n parameter_metadata=DECODER_METADATA[\"BaseDecoder\"][\"fc_use_bias\"],\n )\n\n fc_weights_initializer: str | dict = schema_utils.OneOfOptionsField(\n default=\"xavier_uniform\",\n allow_none=True,\n description=\"The weights initializer to use for the layers in the fc_stack\",\n field_options=[\n schema_utils.InitializerOptions(\n description=\"Preconfigured initializer to use for the layers in the fc_stack.\",\n parameter_metadata=DECODER_METADATA[\"BaseDecoder\"][\"fc_weights_initializer\"],\n ),\n schema_utils.Dict(\n description=\"Custom initializer to use for the layers in the fc_stack.\",\n parameter_metadata=DECODER_METADATA[\"BaseDecoder\"][\"fc_weights_initializer\"],\n ),\n ],\n parameter_metadata=DECODER_METADATA[\"BaseDecoder\"][\"fc_weights_initializer\"],\n )\n\n fc_bias_initializer: str | dict = schema_utils.OneOfOptionsField(\n default=\"zeros\",\n allow_none=True,\n description=\"The bias initializer to use for the layers in the fc_stack\",\n field_options=[\n schema_utils.InitializerOptions(\n description=\"Preconfigured bias initializer to use for the layers in the fc_stack.\",\n parameter_metadata=DECODER_METADATA[\"BaseDecoder\"][\"fc_bias_initializer\"],\n ),\n schema_utils.Dict(\n description=\"Custom bias initializer to use for the layers in the fc_stack.\",\n parameter_metadata=DECODER_METADATA[\"BaseDecoder\"][\"fc_bias_initializer\"],\n ),\n ],\n parameter_metadata=DECODER_METADATA[\"BaseDecoder\"][\"fc_bias_initializer\"],\n )\n\n fc_norm: str = common_fields.NormField()\n\n fc_norm_params: dict = common_fields.NormParamsField()\n\n fc_activation: str = schema_utils.ActivationOptions(default=\"relu\")\n\n fc_dropout: float = common_fields.DropoutField()\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass PassthroughDecoderConfig(BaseDecoderConfig):\n \"\"\"PassthroughDecoderConfig is a dataclass that configures the parameters used for a passthrough decoder.\"\"\"\n\n @classmethod\n def module_name(cls):\n return \"PassthroughDecoder\"\n\n type: str = schema_utils.ProtectedString(\n \"passthrough\",\n description=\"The passthrough decoder simply returns the raw numerical values coming from the combiner as \"\n \"outputs\",\n parameter_metadata=DECODER_METADATA[\"PassthroughDecoder\"][\"type\"],\n )\n\n input_size: int = schema_utils.PositiveInteger(\n default=1,\n description=\"Size of the input to the decoder.\",\n parameter_metadata=DECODER_METADATA[\"PassthroughDecoder\"][\"input_size\"],\n )\n\n\n@DeveloperAPI\n@register_decoder_config(\"regressor\", [BINARY, NUMBER], model_types=[MODEL_ECD])\n@ludwig_dataclass\nclass RegressorConfig(BaseDecoderConfig):\n \"\"\"RegressorConfig is a dataclass that configures the parameters used for a regressor decoder.\"\"\"\n\n @classmethod\n def module_name(cls):\n return \"Regressor\"\n\n type: str = schema_utils.ProtectedString(\n \"regressor\",\n description=DECODER_METADATA[\"Regressor\"][\"type\"].long_description,\n )\n\n input_size: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Size of the input to the decoder.\",\n parameter_metadata=DECODER_METADATA[\"Regressor\"][\"input_size\"],\n )\n\n use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether the layer uses a bias vector.\",\n parameter_metadata=DECODER_METADATA[\"Regressor\"][\"use_bias\"],\n )\n\n weights_initializer: str = schema_utils.InitializerOptions(\n description=\"Initializer for the weight matrix.\",\n parameter_metadata=DECODER_METADATA[\"Regressor\"][\"weights_initializer\"],\n )\n\n bias_initializer: str = schema_utils.InitializerOptions(\n default=\"zeros\",\n description=\"Initializer for the bias vector.\",\n parameter_metadata=DECODER_METADATA[\"Regressor\"][\"bias_initializer\"],\n )\n\n\n@DeveloperAPI\n@register_decoder_config(\"projector\", [VECTOR, TIMESERIES], model_types=[MODEL_ECD])\n@ludwig_dataclass\nclass ProjectorConfig(BaseDecoderConfig):\n \"\"\"ProjectorConfig is a dataclass that configures the parameters used for a projector decoder.\"\"\"\n\n @classmethod\n def module_name(cls):\n return \"Projector\"\n\n type: str = schema_utils.ProtectedString(\n \"projector\",\n description=DECODER_METADATA[\"Projector\"][\"type\"].long_description,\n )\n\n input_size: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Size of the input to the decoder.\",\n parameter_metadata=DECODER_METADATA[\"Projector\"][\"input_size\"],\n )\n\n output_size: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Size of the output of the decoder.\",\n parameter_metadata=DECODER_METADATA[\"Projector\"][\"output_size\"],\n )\n\n use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether the layer uses a bias vector.\",\n parameter_metadata=DECODER_METADATA[\"Projector\"][\"use_bias\"],\n )\n\n weights_initializer: str = schema_utils.InitializerOptions(\n description=\"Initializer for the weight matrix.\",\n parameter_metadata=DECODER_METADATA[\"Projector\"][\"weights_initializer\"],\n )\n\n bias_initializer: str = schema_utils.InitializerOptions(\n default=\"zeros\",\n description=\"Initializer for the bias vector.\",\n parameter_metadata=DECODER_METADATA[\"Projector\"][\"bias_initializer\"],\n )\n\n activation: str = schema_utils.ActivationOptions(\n default=None,\n description=\" Indicates the activation function applied to the output.\",\n parameter_metadata=DECODER_METADATA[\"Projector\"][\"activation\"],\n )\n\n multiplier: float = schema_utils.FloatRange(\n default=1.0,\n min=0,\n min_inclusive=False,\n description=(\n \"Multiplier to scale the activated outputs by. Useful when setting `activation` to something \"\n \"that outputs a value between [-1, 1] like tanh to re-scale values back to order of magnitude of \"\n \"the data you're trying to predict. A good rule of thumb in such cases is to pick a value like \"\n \"`x * (max - min)` where x is a scalar in the range [1, 2]. For example, if you're trying to predict \"\n \"something like temperature, it might make sense to pick a multiplier on the order of `100`.\"\n ),\n )\n\n clip: list[int] | tuple[int] = schema_utils.FloatRangeTupleDataclassField(\n n=2,\n default=None,\n allow_none=True,\n min=0,\n max=999999999,\n description=\"Clip the output of the decoder to be within the given range.\",\n parameter_metadata=DECODER_METADATA[\"Projector\"][\"clip\"],\n )\n\n\n@DeveloperAPI\n@register_decoder_config(\"classifier\", [CATEGORY, SET], model_types=[MODEL_ECD, MODEL_LLM])\n@ludwig_dataclass\nclass ClassifierConfig(BaseDecoderConfig):\n @classmethod\n def module_name(cls):\n return \"Classifier\"\n\n type: str = schema_utils.ProtectedString(\n \"classifier\",\n description=DECODER_METADATA[\"Classifier\"][\"type\"].long_description,\n )\n\n input_size: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Size of the input to the decoder.\",\n parameter_metadata=DECODER_METADATA[\"Classifier\"][\"input_size\"],\n )\n\n num_classes: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Number of classes to predict.\",\n parameter_metadata=DECODER_METADATA[\"Classifier\"][\"num_classes\"],\n )\n\n use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether the layer uses a bias vector.\",\n parameter_metadata=DECODER_METADATA[\"Classifier\"][\"use_bias\"],\n )\n\n weights_initializer: str = schema_utils.InitializerOptions(\n description=\"Initializer for the weight matrix.\",\n parameter_metadata=DECODER_METADATA[\"Classifier\"][\"weights_initializer\"],\n )\n\n bias_initializer: str = schema_utils.InitializerOptions(\n default=\"zeros\",\n description=\"Initializer for the bias vector.\",\n parameter_metadata=DECODER_METADATA[\"Classifier\"][\"bias_initializer\"],\n )\n"
},
{
"path": "ludwig/schema/decoders/image_decoders.py",
"content": "from typing import TYPE_CHECKING\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import IMAGE, MODEL_ECD\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.decoders.base import BaseDecoderConfig\nfrom ludwig.schema.decoders.utils import register_decoder_config\nfrom ludwig.schema.metadata import DECODER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\nif TYPE_CHECKING:\n from ludwig.schema.features.preprocessing.image import ImagePreprocessingConfig\n\n\nclass ImageDecoderConfig(BaseDecoderConfig):\n def set_fixed_preprocessing_params(self, model_type: str, preprocessing: \"ImagePreprocessingConfig\"):\n preprocessing.requires_equal_dimensions = False\n preprocessing.height = None\n preprocessing.width = None\n\n\n@DeveloperAPI\n@register_decoder_config(\"unet\", [IMAGE], model_types=[MODEL_ECD])\n@ludwig_dataclass\nclass UNetDecoderConfig(ImageDecoderConfig):\n @staticmethod\n def module_name():\n return \"UNetDecoder\"\n\n type: str = schema_utils.ProtectedString(\n \"unet\",\n description=DECODER_METADATA[\"UNetDecoder\"][\"type\"].long_description,\n )\n\n input_size: int = schema_utils.PositiveInteger(\n default=1024,\n description=\"Size of the input to the decoder.\",\n parameter_metadata=DECODER_METADATA[\"UNetDecoder\"][\"input_size\"],\n )\n\n height: int = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"Height of the output image.\",\n parameter_metadata=DECODER_METADATA[\"UNetDecoder\"][\"height\"],\n )\n\n width: int = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"Width of the output image.\",\n parameter_metadata=DECODER_METADATA[\"UNetDecoder\"][\"width\"],\n )\n\n num_channels: int | None = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"Number of channels in the output image. \",\n parameter_metadata=DECODER_METADATA[\"UNetDecoder\"][\"num_channels\"],\n )\n\n conv_norm: str | None = schema_utils.StringOptions(\n [\"batch\"],\n default=\"batch\",\n allow_none=True,\n description=\"This is the default norm that will be used for each double conv layer.\" \"It can be null or batch.\",\n parameter_metadata=DECODER_METADATA[\"UNetDecoder\"][\"conv_norm\"],\n )\n\n num_classes: int | None = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"Number of classes to predict in the output. \",\n parameter_metadata=DECODER_METADATA[\"UNetDecoder\"][\"num_classes\"],\n )\n"
},
{
"path": "ludwig/schema/decoders/llm_decoders.py",
"content": "from typing import Any\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import CATEGORY, MODEL_LLM, TEXT\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.decoders.base import BaseDecoderConfig\nfrom ludwig.schema.decoders.utils import register_decoder_config\nfrom ludwig.schema.utils import BaseMarshmallowConfig, ludwig_dataclass\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass BaseExtractorDecoderConfig(BaseMarshmallowConfig):\n tokenizer: str = \"hf_tokenizer\"\n\n input_size: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Size of the input to the decoder.\",\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"\",\n allow_none=True,\n description=\"Path to the pretrained model or model identifier from huggingface.co/models.\",\n )\n\n vocab_file: str = schema_utils.String(\n default=\"\",\n allow_none=True,\n description=\"Path to the vocabulary file.\",\n )\n\n max_new_tokens: int = schema_utils.Integer(\n default=None,\n allow_none=True,\n description=\"Maximum number of new tokens that will be generated.\",\n )\n\n\n@DeveloperAPI\n@register_decoder_config(\"text_extractor\", [TEXT], model_types=[MODEL_LLM])\n@ludwig_dataclass\nclass TextExtractorDecoderConfig(BaseExtractorDecoderConfig, BaseDecoderConfig):\n @classmethod\n def module_name(cls):\n return \"TextExtractorDecoder\"\n\n type: str = schema_utils.ProtectedString(\"text_extractor\")\n\n\n@DeveloperAPI\n@register_decoder_config(\"category_extractor\", [CATEGORY], model_types=[MODEL_LLM])\n@ludwig_dataclass\nclass CategoryExtractorDecoderConfig(BaseExtractorDecoderConfig, BaseDecoderConfig):\n @classmethod\n def module_name(cls):\n return \"CategoryExtractorDecoder\"\n\n type: str = schema_utils.ProtectedString(\"category_extractor\")\n\n # Match is a dict of label class\n match: dict[str, dict[str, Any]] = schema_utils.Dict(\n default=None,\n allow_none=False,\n description=\"A dictionary of label classes and their corresponding \"\n \"match patterns definitions that will be used to parse the output \"\n \"of the LLM.\",\n )\n\n str2idx: dict[str, int] = schema_utils.Dict(\n default=None,\n allow_none=True,\n description=\"A dictionary of label classes and their corresponding \"\n \"indices that will be used to parse the output of the LLM.\",\n )\n\n fallback_label: str = schema_utils.String(\n default=\"\",\n allow_none=True,\n description=\"The label to use if the parser fails to parse the input.\",\n )\n"
},
{
"path": "ludwig/schema/decoders/sequence_decoders.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import MODEL_ECD, SEQUENCE, TEXT\nfrom ludwig.schema import common_fields\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.decoders.base import BaseDecoderConfig\nfrom ludwig.schema.decoders.utils import register_decoder_config\nfrom ludwig.schema.metadata import DECODER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@register_decoder_config(\"generator\", [SEQUENCE, TEXT], model_types=[MODEL_ECD])\n@ludwig_dataclass\nclass SequenceGeneratorDecoderConfig(BaseDecoderConfig):\n @staticmethod\n def module_name():\n return \"SequenceGeneratorDecoder\"\n\n type: str = schema_utils.ProtectedString(\n \"generator\",\n description=DECODER_METADATA[\"SequenceGeneratorDecoder\"][\"type\"].long_description,\n )\n\n vocab_size: int = common_fields.VocabSizeField()\n\n max_sequence_length: int = common_fields.MaxSequenceLengthField()\n\n cell_type: str = schema_utils.StringOptions(\n [\"rnn\", \"lstm\", \"gru\"],\n default=\"gru\",\n description=\"Type of recurrent cell to use.\",\n parameter_metadata=DECODER_METADATA[\"SequenceGeneratorDecoder\"][\"cell_type\"],\n )\n\n input_size: int = schema_utils.PositiveInteger(\n default=256,\n description=\"Size of the input to the decoder.\",\n parameter_metadata=DECODER_METADATA[\"SequenceGeneratorDecoder\"][\"input_size\"],\n )\n\n reduce_input: str = schema_utils.StringOptions(\n [\"sum\", \"mean\", \"avg\", \"max\", \"concat\", \"last\"],\n default=\"sum\",\n description=\"How to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first \"\n \"dimension (second if you count the batch dimension)\",\n parameter_metadata=DECODER_METADATA[\"SequenceGeneratorDecoder\"][\"reduce_input\"],\n )\n\n num_layers: int = schema_utils.PositiveInteger(\n default=1,\n description=\"The number of stacked recurrent layers.\",\n parameter_metadata=DECODER_METADATA[\"SequenceGeneratorDecoder\"][\"num_layers\"],\n )\n\n\n@DeveloperAPI\n@register_decoder_config(\"tagger\", [SEQUENCE, TEXT], model_types=[MODEL_ECD])\n@ludwig_dataclass\nclass SequenceTaggerDecoderConfig(BaseDecoderConfig):\n @classmethod\n def module_name(cls):\n return \"SequenceTaggerDecoder\"\n\n type: str = schema_utils.ProtectedString(\n \"tagger\",\n description=DECODER_METADATA[\"SequenceTaggerDecoder\"][\"type\"].long_description,\n )\n\n input_size: int = schema_utils.PositiveInteger(\n default=256,\n description=\"Size of the input to the decoder.\",\n parameter_metadata=DECODER_METADATA[\"SequenceTaggerDecoder\"][\"input_size\"],\n )\n\n vocab_size: int = common_fields.VocabSizeField()\n\n max_sequence_length: int = common_fields.MaxSequenceLengthField()\n\n use_attention: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to apply a multi-head self attention layer before prediction.\",\n parameter_metadata=DECODER_METADATA[\"SequenceTaggerDecoder\"][\"use_attention\"],\n )\n\n use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether the layer uses a bias vector.\",\n parameter_metadata=DECODER_METADATA[\"SequenceTaggerDecoder\"][\"use_bias\"],\n )\n\n attention_embedding_size: int = schema_utils.PositiveInteger(\n default=256,\n description=\"The embedding size of the multi-head self attention layer.\",\n parameter_metadata=DECODER_METADATA[\"SequenceTaggerDecoder\"][\"attention_embedding_size\"],\n )\n\n attention_num_heads: int = schema_utils.PositiveInteger(\n default=8,\n description=\"The number of attention heads in the multi-head self attention layer.\",\n parameter_metadata=DECODER_METADATA[\"SequenceTaggerDecoder\"][\"attention_num_heads\"],\n )\n"
},
{
"path": "ludwig/schema/decoders/utils.py",
"content": "from dataclasses import Field\nfrom typing import Any, TYPE_CHECKING\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import MODEL_ECD, TYPE\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.metadata import DECODER_METADATA\nfrom ludwig.schema.metadata.parameter_metadata import convert_metadata_to_json\nfrom ludwig.utils.registry import Registry\n\nif TYPE_CHECKING:\n from ludwig.schema.decoders.base import BaseDecoderConfig\n\n\ndecoder_config_registry = Registry()\n\n\n@DeveloperAPI\ndef register_decoder_config(name: str, features: str | list[str], model_types: list[str] | None = None):\n if model_types is None:\n model_types = [MODEL_ECD]\n\n if isinstance(features, str):\n features = [features]\n\n def wrap(cls):\n for model_type in model_types:\n for feature in features:\n key = (model_type, feature)\n feature_registry = decoder_config_registry.get(key, {})\n feature_registry[name] = cls\n decoder_config_registry[key] = feature_registry\n return cls\n\n return wrap\n\n\n@DeveloperAPI\ndef get_decoder_cls(model_type: str, feature: str, name: str):\n return decoder_config_registry[(model_type, feature)][name]\n\n\n@DeveloperAPI\ndef get_decoder_classes(model_type: str, feature: str) -> dict[str, type[\"BaseDecoderConfig\"]]:\n return decoder_config_registry[(model_type, feature)]\n\n\n@DeveloperAPI\ndef get_decoder_descriptions(model_type: str, feature_type: str):\n \"\"\"This function returns a dictionary of decoder descriptions available at the type selection.\n\n The process works as follows - 1) Get a dictionary of valid decoders from the decoder config registry,\n but inverse the key/value pairs since we need to index `valid_decoders` later with an altered version\n of the decoder config class name. 2) Loop through Decoder Metadata entries, if a metadata entry has a\n decoder name that matches a valid decoder, add the description metadata to the output dictionary.\n\n Args:\n model_type (str): The model type to get decoder descriptions for\n feature_type (str): The feature type to get decoder descriptions for\n Returns:\n dict: A dictionary of decoder descriptions\n \"\"\"\n output = {}\n valid_decoders = {\n cls.module_name() if hasattr(cls, \"module_name\") else None: registered_name\n for registered_name, cls in get_decoder_classes(model_type, feature_type).items()\n }\n\n for k, v in DECODER_METADATA.items():\n if k in valid_decoders.keys():\n output[valid_decoders[k]] = convert_metadata_to_json(v[TYPE])\n\n return output\n\n\n@DeveloperAPI\ndef get_decoder_conds(decoder_classes: dict[str, type[\"BaseDecoderConfig\"]]) -> list[dict[str, Any]]:\n \"\"\"Returns a JSON schema of conditionals to validate against decoder types for specific feature types.\"\"\"\n conds = []\n for decoder_type, decoder_cls in decoder_classes.items():\n other_props = schema_utils.unload_jsonschema_from_marshmallow_class(decoder_cls)[\"properties\"]\n schema_utils.remove_duplicate_fields(other_props)\n decoder_cond = schema_utils.create_cond(\n {\"type\": decoder_type},\n other_props,\n )\n conds.append(decoder_cond)\n return conds\n\n\n@DeveloperAPI\ndef DecoderDataclassField(model_type: str, feature_type: str, default: str) -> Field:\n \"\"\"Custom dataclass field that when used inside a dataclass will allow the user to specify a decoder config.\n\n Returns: Initialized dataclass field that converts an untyped dict with params to a decoder config.\n \"\"\"\n decoder_registry = get_decoder_classes(model_type, feature_type)\n\n class DecoderSelection(schema_utils.TypeSelection):\n def __init__(self):\n super().__init__(registry=decoder_registry, default_value=default, allow_str_value=True)\n\n def get_schema_from_registry(self, key: str) -> type[schema_utils.BaseMarshmallowConfig]:\n return decoder_registry[key]\n\n def _jsonschema_type_mapping(self):\n return {\n \"type\": \"object\",\n \"properties\": {\n \"type\": {\n \"type\": \"string\",\n \"enum\": list(decoder_registry.keys()),\n \"enumDescriptions\": get_decoder_descriptions(model_type, feature_type),\n \"default\": default,\n },\n },\n \"title\": \"decoder_options\",\n \"allOf\": get_decoder_conds(decoder_registry),\n }\n\n return DecoderSelection().get_default_field()\n"
},
{
"path": "ludwig/schema/defaults/__init__.py",
"content": ""
},
{
"path": "ludwig/schema/defaults/base.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass BaseDefaultsConfig(schema_utils.BaseMarshmallowConfig):\n \"\"\"Base defaults config class.\"\"\"\n"
},
{
"path": "ludwig/schema/defaults/ecd.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import (\n AUDIO,\n BAG,\n BINARY,\n CATEGORY,\n DATE,\n H3,\n IMAGE,\n NUMBER,\n SEQUENCE,\n SET,\n TEXT,\n TIMESERIES,\n VECTOR,\n)\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.defaults.base import BaseDefaultsConfig\nfrom ludwig.schema.defaults.utils import DefaultsDataclassField\nfrom ludwig.schema.features.base import BaseFeatureConfig\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass ECDDefaultsConfig(BaseDefaultsConfig):\n audio: BaseFeatureConfig = DefaultsDataclassField(feature_type=AUDIO)\n\n bag: BaseFeatureConfig = DefaultsDataclassField(feature_type=BAG)\n\n binary: BaseFeatureConfig = DefaultsDataclassField(feature_type=BINARY)\n\n category: BaseFeatureConfig = DefaultsDataclassField(feature_type=CATEGORY)\n\n date: BaseFeatureConfig = DefaultsDataclassField(feature_type=DATE)\n\n h3: BaseFeatureConfig = DefaultsDataclassField(feature_type=H3)\n\n image: BaseFeatureConfig = DefaultsDataclassField(feature_type=IMAGE)\n\n number: BaseFeatureConfig = DefaultsDataclassField(feature_type=NUMBER)\n\n sequence: BaseFeatureConfig = DefaultsDataclassField(feature_type=SEQUENCE)\n\n set: BaseFeatureConfig = DefaultsDataclassField(feature_type=SET)\n\n text: BaseFeatureConfig = DefaultsDataclassField(feature_type=TEXT)\n\n timeseries: BaseFeatureConfig = DefaultsDataclassField(feature_type=TIMESERIES)\n\n vector: BaseFeatureConfig = DefaultsDataclassField(feature_type=VECTOR)\n\n\n@DeveloperAPI\nclass ECDDefaultsField(schema_utils.DictMarshmallowField):\n def __init__(self):\n super().__init__(ECDDefaultsConfig)\n"
},
{
"path": "ludwig/schema/defaults/llm.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import TEXT\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.defaults.base import BaseDefaultsConfig\nfrom ludwig.schema.defaults.utils import DefaultsDataclassField\nfrom ludwig.schema.features.base import BaseFeatureConfig\nfrom ludwig.schema.features.utils import llm_defaults_config_registry\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass LLMDefaultsConfig(BaseDefaultsConfig):\n text: BaseFeatureConfig = DefaultsDataclassField(feature_type=TEXT, defaults_registry=llm_defaults_config_registry)\n\n\n@DeveloperAPI\nclass LLMDefaultsField(schema_utils.DictMarshmallowField):\n def __init__(self):\n super().__init__(LLMDefaultsConfig)\n"
},
{
"path": "ludwig/schema/defaults/utils.py",
"content": "from dataclasses import field\n\nimport ludwig.schema.utils as schema_utils\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.schema.features.utils import ecd_defaults_config_registry\nfrom ludwig.utils.registry import Registry\n\n\n@DeveloperAPI\ndef DefaultsDataclassField(feature_type: str, defaults_registry: Registry = ecd_defaults_config_registry):\n \"\"\"Custom dataclass field that when used inside a dataclass will allow the user to specify a nested default\n config for a specific feature type.\n\n Returns: Initialized dataclass field that converts an untyped dict with params to a defaults config.\n \"\"\"\n\n class DefaultMarshmallowField(schema_utils.LudwigSchemaField):\n \"\"\"Custom field that deserializes a dict for a valid defaults config from the feature_registry and creates\n a corresponding JSON schema for external usage.\"\"\"\n\n def _deserialize(self, value, attr, data, **kwargs):\n if value is None:\n return None\n if isinstance(value, dict):\n defaults_class = defaults_registry[feature_type]\n try:\n return defaults_class.Schema().load(value)\n except (TypeError, ConfigValidationError) as error:\n raise ConfigValidationError(f\"Invalid params: {value}, see `{attr}` definition. Error: {error}\")\n raise ConfigValidationError(f\"Invalid params: {value}\")\n\n def _jsonschema_type_mapping(self):\n defaults_cls = defaults_registry[feature_type]\n props = schema_utils.unload_jsonschema_from_marshmallow_class(defaults_cls)[\"properties\"]\n return {\n \"type\": \"object\",\n \"properties\": props,\n \"additionalProperties\": False,\n \"title\": \"defaults_options\",\n }\n\n try:\n defaults_cls = defaults_registry[feature_type]\n dump_default = defaults_cls.Schema().dump({})\n load_default = lambda: defaults_cls.Schema().load({})\n\n return field(\n metadata={\n \"marshmallow_field\": DefaultMarshmallowField(\n allow_none=False,\n dump_default=dump_default,\n load_default=load_default,\n )\n },\n default_factory=load_default,\n )\n except Exception as e:\n raise ConfigValidationError(\n f\"Unsupported feature type: {feature_type}. Allowed: {defaults_registry.keys()}. \" f\"Details: {e}\"\n )\n"
},
{
"path": "ludwig/schema/encoders/__init__.py",
"content": "# Register all encoder schemas\nimport ludwig.schema.encoders.bag_encoders\nimport ludwig.schema.encoders.category_encoders\nimport ludwig.schema.encoders.date_encoders\nimport ludwig.schema.encoders.h3_encoders\nimport ludwig.schema.encoders.image\nimport ludwig.schema.encoders.sequence_encoders\nimport ludwig.schema.encoders.set_encoders\nimport ludwig.schema.encoders.text_encoders # noqa\n"
},
{
"path": "ludwig/schema/encoders/bag_encoders.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import BAG\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import register_encoder_config\nfrom ludwig.schema.metadata import ENCODER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@register_encoder_config(\"embed\", BAG)\n@ludwig_dataclass\nclass BagEmbedWeightedConfig(BaseEncoderConfig):\n @staticmethod\n def module_name():\n return \"BagEmbedWeighted\"\n\n type: str = schema_utils.ProtectedString(\n \"embed\",\n description=ENCODER_METADATA[\"BagEmbedWeighted\"][\"type\"].long_description,\n )\n\n dropout: float = schema_utils.FloatRange(\n default=0.0,\n min=0,\n max=1,\n description=\"Dropout probability for the embedding.\",\n parameter_metadata=ENCODER_METADATA[\"BagEmbedWeighted\"][\"dropout\"],\n )\n\n activation: str = schema_utils.ActivationOptions(\n description=\"The default activation function that will be used for each layer.\",\n parameter_metadata=ENCODER_METADATA[\"BagEmbedWeighted\"][\"activation\"],\n )\n\n vocab: list[str] = schema_utils.List(\n default=None,\n description=\"Vocabulary of the encoder\",\n parameter_metadata=ENCODER_METADATA[\"BagEmbedWeighted\"][\"vocab\"],\n )\n\n representation: str = schema_utils.StringOptions(\n [\"dense\", \"sparse\"],\n default=\"dense\",\n description=\"The representation of the embedding. Either dense or sparse.\",\n parameter_metadata=ENCODER_METADATA[\"BagEmbedWeighted\"][\"representation\"],\n )\n\n embedding_size: int = schema_utils.PositiveInteger(\n default=50,\n description=\"The maximum embedding size, the actual size will be min(vocabulary_size, embedding_size) for \"\n \"dense representations and exactly vocabulary_size for the sparse encoding, where vocabulary_size \"\n \"is the number of different strings appearing in the training set in the input column (plus 1 for \"\n \"the unknown token placeholder ).\",\n parameter_metadata=ENCODER_METADATA[\"BagEmbedWeighted\"][\"embedding_size\"],\n )\n\n force_embedding_size: bool = schema_utils.Boolean(\n default=False,\n description=\"Force the embedding size to be equal to the vocabulary size. This parameter has effect only if \"\n \"representation is dense.\",\n parameter_metadata=ENCODER_METADATA[\"BagEmbedWeighted\"][\"force_embedding_size\"],\n )\n\n embeddings_on_cpu: bool = schema_utils.Boolean(\n default=False,\n description=\"By default embedding matrices are stored on GPU memory if a GPU is used, as it allows for faster \"\n \"access, but in some cases the embedding matrix may be too large. This parameter forces the \"\n \"placement of the embedding matrix in regular memory and the CPU is used for embedding lookup, \"\n \"slightly slowing down the process as a result of data transfer between CPU and GPU memory.\",\n parameter_metadata=ENCODER_METADATA[\"BagEmbedWeighted\"][\"embeddings_on_cpu\"],\n )\n\n embeddings_trainable: bool = schema_utils.Boolean(\n default=True,\n description=\"If true embeddings are trained during the training process, if false embeddings are fixed. It \"\n \"may be useful when loading pretrained embeddings for avoiding fine tuning them. This parameter \"\n \"has effect only when representation is dense as sparse one-hot encodings are not trainable.\",\n parameter_metadata=ENCODER_METADATA[\"BagEmbedWeighted\"][\"embeddings_trainable\"],\n )\n\n pretrained_embeddings: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"By default dense embeddings are initialized randomly, but this parameter allows to specify a \"\n \"path to a file containing embeddings in the GloVe format. When the file containing the \"\n \"embeddings is loaded, only the embeddings with labels present in the vocabulary are kept, \"\n \"the others are discarded. If the vocabulary contains strings that have no match in the \"\n \"embeddings file, their embeddings are initialized with the average of all other embedding plus \"\n \"some random noise to make them different from each other. This parameter has effect only if \"\n \"representation is dense.\",\n parameter_metadata=ENCODER_METADATA[\"BagEmbedWeighted\"][\"pretrained_embeddings\"],\n )\n\n use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether the layer uses a bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"BagEmbedWeighted\"][\"use_bias\"],\n )\n\n bias_initializer: str = schema_utils.InitializerOptions(\n default=\"zeros\",\n description=\"Initializer to use for the bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"BagEmbedWeighted\"][\"bias_initializer\"],\n )\n\n weights_initializer: str = schema_utils.InitializerOptions(\n description=\"Initializer to use for the weights matrix.\",\n parameter_metadata=ENCODER_METADATA[\"BagEmbedWeighted\"][\"weights_initializer\"],\n )\n\n output_size: int = schema_utils.PositiveInteger(\n default=10,\n description=\"If output_size is not already specified in fc_layers this is the default output_size that will \"\n \"be used for each layer. It indicates the size of the output of a fully connected layer.\",\n parameter_metadata=ENCODER_METADATA[\"BagEmbedWeighted\"][\"output_size\"],\n )\n\n norm: str = schema_utils.StringOptions(\n [\"batch\", \"layer\"],\n default=None,\n allow_none=True,\n description=\"The default norm that will be used for each layer.\",\n parameter_metadata=ENCODER_METADATA[\"BagEmbedWeighted\"][\"norm\"],\n )\n\n norm_params: dict = schema_utils.Dict(\n default=None,\n description=\"Parameters used if norm is either `batch` or `layer`.\",\n parameter_metadata=ENCODER_METADATA[\"BagEmbedWeighted\"][\"norm_params\"],\n )\n\n num_fc_layers: int = schema_utils.NonNegativeInteger(\n default=0,\n description=\"This is the number of stacked fully connected layers that the input to the feature passes \"\n \"through. Their output is projected in the feature's output space.\",\n parameter_metadata=ENCODER_METADATA[\"BagEmbedWeighted\"][\"num_fc_layers\"],\n )\n\n fc_layers: list[dict] = schema_utils.DictList( # TODO (Connor): Add nesting logic for fc_layers\n default=None,\n description=\"List of dictionaries containing the parameters for each fully connected layer.\",\n parameter_metadata=ENCODER_METADATA[\"BagEmbedWeighted\"][\"fc_layers\"],\n )\n"
},
{
"path": "ludwig/schema/encoders/base.py",
"content": "from abc import ABC\nfrom typing import TYPE_CHECKING\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import BINARY, MODEL_ECD, MODEL_LLM, NUMBER, TEXT, TIMESERIES, VECTOR\nfrom ludwig.schema import common_fields\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.encoders.utils import register_encoder_config\nfrom ludwig.schema.metadata import ENCODER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\nif TYPE_CHECKING:\n from ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass BaseEncoderConfig(schema_utils.BaseMarshmallowConfig, ABC):\n \"\"\"Base class for encoders.\"\"\"\n\n type: str\n\n skip: bool = schema_utils.Boolean(\n False,\n \"[internal] Whether to skip encoder and use input as output.\",\n parameter_metadata=ENCODER_METADATA[\"BaseEncoder\"][\"skip\"],\n )\n\n def set_fixed_preprocessing_params(self, model_type: str, preprocessing: \"BasePreprocessingConfig\"):\n pass\n\n def is_pretrained(self) -> bool:\n return False\n\n def can_cache_embeddings(self) -> bool:\n return False\n\n\n@DeveloperAPI\n@register_encoder_config(\"passthrough\", [TEXT], model_types=[MODEL_LLM])\n@register_encoder_config(\"passthrough\", [BINARY, NUMBER, VECTOR], model_types=[MODEL_ECD])\n@ludwig_dataclass\nclass PassthroughEncoderConfig(BaseEncoderConfig):\n \"\"\"PassthroughEncoderConfig is a dataclass that configures the parameters used for a passthrough encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"PassthroughEncoder\"\n\n type: str = schema_utils.ProtectedString(\n \"passthrough\",\n description=ENCODER_METADATA[\"PassthroughEncoder\"][\"type\"].long_description,\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"dense\", [BINARY, NUMBER, VECTOR, TIMESERIES])\n@ludwig_dataclass\nclass DenseEncoderConfig(BaseEncoderConfig):\n \"\"\"DenseEncoderConfig is a dataclass that configures the parameters used for a dense encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"DenseEncoder\"\n\n type: str = schema_utils.ProtectedString(\n \"dense\",\n description=ENCODER_METADATA[\"DenseEncoder\"][\"type\"].long_description,\n )\n\n dropout: float = common_fields.DropoutField()\n\n activation: str = schema_utils.ActivationOptions()\n\n input_size: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Size of the input to the dense encoder.\",\n parameter_metadata=ENCODER_METADATA[\"DenseEncoder\"][\"input_size\"],\n )\n\n output_size: int = schema_utils.PositiveInteger(\n default=256,\n description=\"Size of the output of the feature.\",\n parameter_metadata=ENCODER_METADATA[\"DenseEncoder\"][\"output_size\"],\n )\n\n use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether the layer uses a bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"DenseEncoder\"][\"use_bias\"],\n )\n\n bias_initializer: str | dict = common_fields.BiasInitializerField()\n\n weights_initializer: str | dict = common_fields.WeightsInitializerField()\n\n norm: str = common_fields.NormField()\n\n norm_params: dict = common_fields.NormParamsField()\n\n num_layers: int = common_fields.NumFCLayersField(default=1, non_zero=True)\n\n fc_layers: list[dict] = common_fields.FCLayersField()\n"
},
{
"path": "ludwig/schema/encoders/category_encoders.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import CATEGORY, MODEL_ECD\nfrom ludwig.schema import common_fields\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import register_encoder_config\nfrom ludwig.schema.metadata import ENCODER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@register_encoder_config(\"passthrough\", CATEGORY, model_types=[MODEL_ECD])\n@ludwig_dataclass\nclass CategoricalPassthroughEncoderConfig(BaseEncoderConfig):\n \"\"\"CategoricalPassthroughEncoderConfig is a dataclass that configures the parameters used for a categorical\n passthrough encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"CategoricalPassthroughEncoder\"\n\n type: str = schema_utils.ProtectedString(\n \"passthrough\",\n description=ENCODER_METADATA[\"PassthroughEncoder\"][\"type\"].long_description,\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"dense\", CATEGORY)\n@ludwig_dataclass\nclass CategoricalEmbedConfig(BaseEncoderConfig):\n @staticmethod\n def module_name():\n return \"CategoricalEmbed\"\n\n type: str = schema_utils.ProtectedString(\n \"dense\",\n description=ENCODER_METADATA[\"CategoricalEmbed\"][\"type\"].long_description,\n )\n\n dropout: float = common_fields.DropoutField()\n\n vocab: list[str] = common_fields.VocabField()\n\n embedding_initializer: str = common_fields.EmbeddingInitializerField()\n\n embedding_size: int = common_fields.EmbeddingSizeField(\n default=50,\n description=(\n \"The maximum embedding size, the actual size will be min(vocabulary_size, embedding_size) for \"\n \"dense representations and exactly vocabulary_size for the sparse encoding, where vocabulary_size \"\n \"is the number of different strings appearing in the training set in the column the feature is \"\n \"named after (plus 1 for ).\"\n ),\n )\n\n embeddings_on_cpu: bool = common_fields.EmbeddingsOnCPUField()\n\n embeddings_trainable: bool = common_fields.EmbeddingsTrainableField()\n\n pretrained_embeddings: str = common_fields.PretrainedEmbeddingsField()\n\n\n@DeveloperAPI\n@register_encoder_config(\"sparse\", CATEGORY)\n@ludwig_dataclass\nclass CategoricalSparseConfig(BaseEncoderConfig):\n @staticmethod\n def module_name():\n return \"CategorySparse\"\n\n type: str = schema_utils.ProtectedString(\n \"sparse\",\n description=ENCODER_METADATA[\"CategoricalSparse\"][\"type\"].long_description,\n )\n\n dropout: float = common_fields.DropoutField()\n\n vocab: list[str] = common_fields.VocabField()\n\n embedding_initializer: str = common_fields.EmbeddingInitializerField()\n\n embeddings_on_cpu: bool = common_fields.EmbeddingsOnCPUField()\n\n # TODO(travis): seems like this is not really a valid user option. We should probably just remove these\n # params entirely and update the encoder implementation.\n embeddings_trainable: bool = common_fields.EmbeddingsTrainableField(default=False)\n\n pretrained_embeddings: str = common_fields.PretrainedEmbeddingsField()\n\n\n@DeveloperAPI\n@register_encoder_config(\"onehot\", CATEGORY, model_types=[MODEL_ECD])\n@ludwig_dataclass\nclass CategoricalOneHotEncoderConfig(BaseEncoderConfig):\n \"\"\"CategoricalOneHotEncoderConfig is a dataclass that configures the parameters used for a categorical onehot\n encoder.\"\"\"\n\n type: str = schema_utils.ProtectedString(\n \"onehot\",\n description=\"Type of encoder.\",\n )\n\n vocab: list[str] = common_fields.VocabField()\n\n def can_cache_embeddings(self) -> bool:\n return True\n"
},
{
"path": "ludwig/schema/encoders/date_encoders.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import DATE\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import register_encoder_config\nfrom ludwig.schema.metadata import ENCODER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@register_encoder_config(\"embed\", DATE)\n@ludwig_dataclass\nclass DateEmbedConfig(BaseEncoderConfig):\n @staticmethod\n def module_name():\n return \"DateEmbed\"\n\n type: str = schema_utils.ProtectedString(\n \"embed\",\n description=ENCODER_METADATA[\"DateEmbed\"][\"type\"].long_description,\n )\n\n dropout: float = schema_utils.FloatRange(\n default=0.0,\n min=0,\n max=1,\n description=\"Dropout probability for the embedding.\",\n parameter_metadata=ENCODER_METADATA[\"DateEmbed\"][\"dropout\"],\n )\n\n activation: str = schema_utils.ActivationOptions(\n description=\"The default activation function that will be used for each layer.\",\n parameter_metadata=ENCODER_METADATA[\"DateEmbed\"][\"activation\"],\n )\n\n use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether the layer uses a bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"DateEmbed\"][\"use_bias\"],\n )\n\n bias_initializer: str = schema_utils.InitializerOptions(\n default=\"zeros\",\n description=\"Initializer to use for the bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"DateEmbed\"][\"bias_initializer\"],\n )\n\n weights_initializer: str = schema_utils.InitializerOptions(\n description=\"Initializer to use for the weights matrix.\",\n parameter_metadata=ENCODER_METADATA[\"DateEmbed\"][\"weights_initializer\"],\n )\n\n embedding_size: int = schema_utils.PositiveInteger(\n default=10,\n description=\"The maximum embedding size adopted.\",\n parameter_metadata=ENCODER_METADATA[\"DateEmbed\"][\"embedding_size\"],\n )\n\n embeddings_on_cpu: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to force the placement of the embedding matrix in regular memory and have the CPU \"\n \"resolve them.\",\n parameter_metadata=ENCODER_METADATA[\"DateEmbed\"][\"embeddings_on_cpu\"],\n )\n\n output_size: int = schema_utils.PositiveInteger(\n default=10,\n description=\"If an output_size is not already specified in fc_layers this is the default output_size that \"\n \"will be used for each layer. It indicates the size of the output of a fully connected layer.\",\n parameter_metadata=ENCODER_METADATA[\"DateEmbed\"][\"output_size\"],\n )\n\n norm: str = schema_utils.StringOptions(\n [\"batch\", \"layer\"],\n default=None,\n allow_none=True,\n description=\"The default norm that will be used for each layer.\",\n parameter_metadata=ENCODER_METADATA[\"DateEmbed\"][\"norm\"],\n )\n\n norm_params: dict = schema_utils.Dict(\n default=None,\n description=\"Parameters used if norm is either `batch` or `layer`.\",\n parameter_metadata=ENCODER_METADATA[\"DateEmbed\"][\"norm_params\"],\n )\n\n num_fc_layers: int = schema_utils.NonNegativeInteger(\n default=0,\n description=\"The number of stacked fully connected layers.\",\n parameter_metadata=ENCODER_METADATA[\"DateEmbed\"][\"num_fc_layers\"],\n )\n\n # TODO (Connor): Add nesting logic for fc_layers, see fully_connected_module.py\n fc_layers: list[dict] = schema_utils.DictList(\n default=None,\n description=\"List of dictionaries containing the parameters for each fully connected layer.\",\n parameter_metadata=ENCODER_METADATA[\"DateEmbed\"][\"fc_layers\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"wave\", DATE)\n@ludwig_dataclass\nclass DateWaveConfig(BaseEncoderConfig):\n @staticmethod\n def module_name():\n return \"DateWave\"\n\n type: str = schema_utils.ProtectedString(\n \"wave\",\n description=ENCODER_METADATA[\"DateWave\"][\"type\"].long_description,\n )\n\n dropout: float = schema_utils.FloatRange(\n default=0.0,\n min=0,\n max=1,\n description=\"Dropout probability for the embedding.\",\n parameter_metadata=ENCODER_METADATA[\"DateWave\"][\"dropout\"],\n )\n\n activation: str = schema_utils.ActivationOptions(\n description=\"The default activation function that will be used for each layer.\",\n parameter_metadata=ENCODER_METADATA[\"DateWave\"][\"activation\"],\n )\n\n use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether the layer uses a bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"DateWave\"][\"use_bias\"],\n )\n\n bias_initializer: str = schema_utils.InitializerOptions(\n default=\"zeros\",\n description=\"Initializer to use for the bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"DateWave\"][\"bias_initializer\"],\n )\n\n weights_initializer: str = schema_utils.InitializerOptions(\n description=\"Initializer to use for the weights matrix.\",\n parameter_metadata=ENCODER_METADATA[\"DateWave\"][\"weights_initializer\"],\n )\n\n output_size: int = schema_utils.PositiveInteger(\n default=10,\n description=\"If an output_size is not already specified in fc_layers this is the default output_size that \"\n \"will be used for each layer. It indicates the size of the output of a fully connected layer.\",\n parameter_metadata=ENCODER_METADATA[\"DateWave\"][\"output_size\"],\n )\n\n norm: str = schema_utils.StringOptions(\n [\"batch\", \"layer\"],\n default=None,\n allow_none=True,\n description=\"The default norm that will be used for each layer.\",\n parameter_metadata=ENCODER_METADATA[\"DateWave\"][\"norm\"],\n )\n\n norm_params: dict = schema_utils.Dict(\n default=None,\n description=\"Parameters used if norm is either `batch` or `layer`.\",\n parameter_metadata=ENCODER_METADATA[\"DateWave\"][\"norm_params\"],\n )\n\n num_fc_layers: int = schema_utils.PositiveInteger(\n default=1,\n description=\"The number of stacked fully connected layers.\",\n parameter_metadata=ENCODER_METADATA[\"DateWave\"][\"num_fc_layers\"],\n )\n\n # TODO (Connor): Add nesting logic for fc_layers, see fully_connected_module.py\n fc_layers: list[dict] = schema_utils.DictList(\n default=None,\n description=\"List of dictionaries containing the parameters for each fully connected layer.\",\n parameter_metadata=ENCODER_METADATA[\"DateWave\"][\"fc_layers\"],\n )\n"
},
{
"path": "ludwig/schema/encoders/h3_encoders.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import H3\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import register_encoder_config\nfrom ludwig.schema.metadata import ENCODER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@register_encoder_config(\"embed\", H3)\n@ludwig_dataclass\nclass H3EmbedConfig(BaseEncoderConfig):\n @staticmethod\n def module_name():\n return \"H3Embed\"\n\n type: str = schema_utils.ProtectedString(\n \"embed\",\n description=ENCODER_METADATA[\"H3Embed\"][\"type\"].long_description,\n )\n\n dropout: float = schema_utils.FloatRange(\n default=0.0,\n min=0,\n max=1,\n description=\"Dropout probability for the embedding.\",\n parameter_metadata=ENCODER_METADATA[\"H3Embed\"][\"dropout\"],\n )\n\n activation: str = schema_utils.ActivationOptions(\n description=\"The default activation function that will be used for each layer.\",\n parameter_metadata=ENCODER_METADATA[\"H3Embed\"][\"activation\"],\n )\n\n use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether the layer uses a bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"H3Embed\"][\"use_bias\"],\n )\n\n bias_initializer: str = schema_utils.InitializerOptions(\n default=\"zeros\",\n description=\"Initializer to use for the bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"H3Embed\"][\"bias_initializer\"],\n )\n\n weights_initializer: str = schema_utils.InitializerOptions(\n description=\"Initializer to use for the weights matrix.\",\n parameter_metadata=ENCODER_METADATA[\"H3Embed\"][\"weights_initializer\"],\n )\n\n embedding_size: int = schema_utils.PositiveInteger(\n default=10,\n description=\"The maximum embedding size adopted.\",\n parameter_metadata=ENCODER_METADATA[\"H3Embed\"][\"embedding_size\"],\n )\n\n embeddings_on_cpu: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to force the placement of the embedding matrix in regular memory and have the CPU \"\n \"resolve them.\",\n parameter_metadata=ENCODER_METADATA[\"H3Embed\"][\"embeddings_on_cpu\"],\n )\n\n reduce_output: str = schema_utils.ReductionOptions(\n default=\"sum\",\n description=\"How to reduce the output tensor along the `s` sequence length dimension if the rank of the \"\n \"tensor is greater than 2.\",\n parameter_metadata=ENCODER_METADATA[\"H3Embed\"][\"reduce_output\"],\n )\n\n output_size: int = schema_utils.PositiveInteger(\n default=10,\n description=\"If an output_size is not already specified in fc_layers this is the default output_size that \"\n \"will be used for each layer. It indicates the size of the output of a fully connected layer.\",\n parameter_metadata=ENCODER_METADATA[\"H3Embed\"][\"output_size\"],\n )\n\n norm: str = schema_utils.StringOptions(\n [\"batch\", \"layer\"],\n default=None,\n allow_none=True,\n description=\"The default norm that will be used for each layer.\",\n parameter_metadata=ENCODER_METADATA[\"H3Embed\"][\"norm\"],\n )\n\n norm_params: dict = schema_utils.Dict(\n default=None,\n description=\"Parameters used if norm is either `batch` or `layer`.\",\n parameter_metadata=ENCODER_METADATA[\"H3Embed\"][\"norm_params\"],\n )\n\n num_fc_layers: int = schema_utils.NonNegativeInteger(\n default=0,\n description=\"The number of stacked fully connected layers.\",\n parameter_metadata=ENCODER_METADATA[\"H3Embed\"][\"num_fc_layers\"],\n )\n\n fc_layers: list[dict] = schema_utils.DictList( # TODO (Connor): Add nesting logic for fc_layers\n default=None,\n description=\"List of dictionaries containing the parameters for each fully connected layer.\",\n parameter_metadata=ENCODER_METADATA[\"H3Embed\"][\"fc_layers\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"weighted_sum\", H3)\n@ludwig_dataclass\nclass H3WeightedSumConfig(BaseEncoderConfig):\n @staticmethod\n def module_name():\n return \"H3WeightedSum\"\n\n type: str = schema_utils.ProtectedString(\n \"weighted_sum\",\n description=ENCODER_METADATA[\"H3WeightedSum\"][\"type\"].long_description,\n )\n\n dropout: float = schema_utils.FloatRange(\n default=0.0,\n min=0,\n max=1,\n description=\"Dropout probability for the embedding.\",\n parameter_metadata=ENCODER_METADATA[\"H3WeightedSum\"][\"dropout\"],\n )\n\n activation: str = schema_utils.ActivationOptions(\n description=\"The default activation function that will be used for each layer.\",\n parameter_metadata=ENCODER_METADATA[\"H3WeightedSum\"][\"activation\"],\n )\n\n use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether the layer uses a bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"H3WeightedSum\"][\"use_bias\"],\n )\n\n bias_initializer: str = schema_utils.InitializerOptions(\n default=\"zeros\",\n description=\"Initializer to use for the bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"H3WeightedSum\"][\"bias_initializer\"],\n )\n\n weights_initializer: str = schema_utils.InitializerOptions(\n description=\"Initializer to use for the weights matrix.\",\n parameter_metadata=ENCODER_METADATA[\"H3WeightedSum\"][\"weights_initializer\"],\n )\n\n embedding_size: int = schema_utils.PositiveInteger(\n default=10,\n description=\"The maximum embedding size adopted.\",\n parameter_metadata=ENCODER_METADATA[\"H3WeightedSum\"][\"embedding_size\"],\n )\n\n embeddings_on_cpu: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to force the placement of the embedding matrix in regular memory and have the CPU \"\n \"resolve them.\",\n parameter_metadata=ENCODER_METADATA[\"H3WeightedSum\"][\"embeddings_on_cpu\"],\n )\n\n should_softmax: bool = schema_utils.Boolean(\n default=False,\n description=\"Determines if the weights of the weighted sum should be passed though a softmax layer before \"\n \"being used.\",\n parameter_metadata=ENCODER_METADATA[\"H3WeightedSum\"][\"should_softmax\"],\n )\n\n output_size: int = schema_utils.PositiveInteger(\n default=10,\n description=\"If an output_size is not already specified in fc_layers this is the default output_size that \"\n \"will be used for each layer. It indicates the size of the output of a fully connected layer.\",\n parameter_metadata=ENCODER_METADATA[\"H3WeightedSum\"][\"output_size\"],\n )\n\n norm: str = schema_utils.StringOptions(\n [\"batch\", \"layer\"],\n default=None,\n allow_none=True,\n description=\"The default norm that will be used for each layer.\",\n parameter_metadata=ENCODER_METADATA[\"H3WeightedSum\"][\"norm\"],\n )\n\n norm_params: dict = schema_utils.Dict(\n default=None,\n description=\"Parameters used if norm is either `batch` or `layer`.\",\n parameter_metadata=ENCODER_METADATA[\"H3WeightedSum\"][\"norm_params\"],\n )\n\n num_fc_layers: int = schema_utils.NonNegativeInteger(\n default=0,\n description=\"The number of stacked fully connected layers.\",\n parameter_metadata=ENCODER_METADATA[\"H3WeightedSum\"][\"num_fc_layers\"],\n )\n\n fc_layers: list[dict] = schema_utils.DictList( # TODO (Connor): Add nesting logic for fc_layers\n default=None,\n description=\"List of dictionaries containing the parameters for each fully connected layer.\",\n parameter_metadata=ENCODER_METADATA[\"H3WeightedSum\"][\"fc_layers\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"rnn\", H3)\n@ludwig_dataclass\nclass H3RNNConfig(BaseEncoderConfig):\n @staticmethod\n def module_name():\n return \"H3RNN\"\n\n type: str = schema_utils.ProtectedString(\n \"rnn\",\n description=ENCODER_METADATA[\"H3RNN\"][\"type\"].long_description,\n )\n\n dropout: float = schema_utils.FloatRange(\n default=0.0,\n min=0,\n max=1,\n description=\"The dropout rate\",\n parameter_metadata=ENCODER_METADATA[\"H3RNN\"][\"dropout\"],\n )\n\n recurrent_dropout: float = schema_utils.FloatRange(\n default=0.0,\n min=0,\n max=1,\n description=\"The dropout rate for the recurrent state\",\n parameter_metadata=ENCODER_METADATA[\"H3RNN\"][\"recurrent_dropout\"],\n )\n\n activation: str = schema_utils.ActivationOptions(\n default=\"tanh\",\n description=\"The activation function to use\",\n parameter_metadata=ENCODER_METADATA[\"H3RNN\"][\"activation\"],\n )\n\n recurrent_activation: str = schema_utils.ActivationOptions(\n default=\"sigmoid\",\n description=\"The activation function to use in the recurrent step\",\n parameter_metadata=ENCODER_METADATA[\"H3RNN\"][\"recurrent_activation\"],\n )\n\n cell_type: str = schema_utils.StringOptions(\n [\"rnn\", \"lstm\", \"lstm_block\", \"ln\", \"lstm_cudnn\", \"gru\", \"gru_block\", \"gru_cudnn\"],\n default=\"rnn\",\n description=\"The type of recurrent cell to use. Available values are: `rnn`, `lstm`, `lstm_block`, `lstm`, \"\n \"`ln`, `lstm_cudnn`, `gru`, `gru_block`, `gru_cudnn`. For reference about the differences between \"\n \"the cells please refer to PyTorch's documentation. We suggest to use the `block` variants on \"\n \"CPU and the `cudnn` variants on GPU because of their increased speed. \",\n parameter_metadata=ENCODER_METADATA[\"H3RNN\"][\"cell_type\"],\n )\n\n num_layers: int = schema_utils.PositiveInteger(\n default=1,\n description=\"The number of stacked recurrent layers.\",\n parameter_metadata=ENCODER_METADATA[\"H3RNN\"][\"num_layers\"],\n )\n\n hidden_size: int = schema_utils.PositiveInteger(\n default=10,\n description=\"The size of the hidden representation within the transformer block. It is usually the same as \"\n \"the embedding_size, but if the two values are different, a projection layer will be added before \"\n \"the first transformer block.\",\n parameter_metadata=ENCODER_METADATA[\"H3RNN\"][\"hidden_size\"],\n )\n\n use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use a bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"H3RNN\"][\"use_bias\"],\n )\n\n unit_forget_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"If true, add 1 to the bias of the forget gate at initialization\",\n parameter_metadata=ENCODER_METADATA[\"H3RNN\"][\"unit_forget_bias\"],\n )\n\n bias_initializer: str = schema_utils.InitializerOptions(\n default=\"zeros\",\n description=\"Initializer to use for the bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"H3RNN\"][\"bias_initializer\"],\n )\n\n weights_initializer: str = schema_utils.InitializerOptions(\n description=\"Initializer to use for the weights matrix.\",\n parameter_metadata=ENCODER_METADATA[\"H3RNN\"][\"weights_initializer\"],\n )\n\n recurrent_initializer: str = schema_utils.InitializerOptions(\n default=\"orthogonal\",\n description=\"The initializer for recurrent matrix weights\",\n parameter_metadata=ENCODER_METADATA[\"H3RNN\"][\"recurrent_initializer\"],\n )\n\n reduce_output: str = schema_utils.ReductionOptions(\n default=\"last\",\n description=\"How to reduce the output tensor along the `s` sequence length dimension if the rank of the \"\n \"tensor is greater than 2.\",\n parameter_metadata=ENCODER_METADATA[\"H3RNN\"][\"reduce_output\"],\n )\n\n embedding_size: int = schema_utils.PositiveInteger(\n default=10,\n description=\"The maximum embedding size adopted.\",\n parameter_metadata=ENCODER_METADATA[\"H3RNN\"][\"embedding_size\"],\n )\n\n embeddings_on_cpu: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to force the placement of the embedding matrix in regular memory and have the CPU \"\n \"resolve them.\",\n parameter_metadata=ENCODER_METADATA[\"H3RNN\"][\"embeddings_on_cpu\"],\n )\n\n bidirectional: bool = schema_utils.Boolean(\n default=False,\n description=\"If true, two recurrent networks will perform encoding in the forward and backward direction and \"\n \"their outputs will be concatenated.\",\n parameter_metadata=ENCODER_METADATA[\"H3RNN\"][\"bidirectional\"],\n )\n"
},
{
"path": "ludwig/schema/encoders/image/__init__.py",
"content": "import ludwig.schema.encoders.image.base\nimport ludwig.schema.encoders.image.timm # noqa\nimport ludwig.schema.encoders.image.torchvision # noqa\n"
},
{
"path": "ludwig/schema/encoders/image/base.py",
"content": "from typing import Any, TYPE_CHECKING\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import IMAGE\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import register_encoder_config\nfrom ludwig.schema.metadata import ENCODER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\nfrom ludwig.utils.torch_utils import initializer_registry\n\nif TYPE_CHECKING:\n from ludwig.schema.features.preprocessing.image import ImagePreprocessingConfig\n\n\nclass ImageEncoderConfig(BaseEncoderConfig):\n def set_fixed_preprocessing_params(self, model_type: str, preprocessing: \"ImagePreprocessingConfig\"):\n preprocessing.requires_equal_dimensions = False\n preprocessing.height = None\n preprocessing.width = None\n\n\n@DeveloperAPI\n@register_encoder_config(\"stacked_cnn\", IMAGE)\n@ludwig_dataclass\nclass Stacked2DCNNConfig(ImageEncoderConfig):\n @staticmethod\n def module_name():\n return \"Stacked2DCNN\"\n\n type: str = schema_utils.ProtectedString(\n \"stacked_cnn\",\n description=ENCODER_METADATA[\"Stacked2DCNN\"][\"type\"].long_description,\n )\n\n conv_dropout: int | None = schema_utils.FloatRange(\n default=0.0,\n min=0,\n max=1,\n description=\"Dropout rate\",\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"conv_dropout\"],\n )\n\n conv_activation: str = schema_utils.ActivationOptions(\n description=\"If an activation is not already specified in conv_layers this is the default activation that \"\n \"will be used for each layer. It indicates the activation function applied to the output.\",\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"conv_activation\"],\n )\n\n height: int = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"Height of the input image.\",\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"height\"],\n )\n\n width: int = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"Width of the input image.\",\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"width\"],\n )\n\n num_channels: int | None = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"Number of channels to use in the encoder. \",\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"num_channels\"],\n )\n\n out_channels: int | None = schema_utils.NonNegativeInteger(\n default=32,\n description=\"Indicates the number of filters, and by consequence the output channels of the 2d convolution. \"\n \"If out_channels is not already specified in conv_layers this is the default out_channels that \"\n \"will be used for each layer. \",\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"out_channels\"],\n )\n\n kernel_size: int | tuple[int] | None = schema_utils.OneOfOptionsField(\n default=3,\n description=\"An integer or pair of integers specifying the kernel size. A single integer specifies a square \"\n \"kernel, while a pair of integers specifies the height and width of the kernel in that order (h, \"\n \"w). If a kernel_size is not specified in conv_layers this kernel_size that will be used for \"\n \"each layer.\",\n field_options=[\n schema_utils.PositiveInteger(allow_none=False, description=\"\", default=3),\n schema_utils.List(list_type=int, allow_none=False),\n ],\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"kernel_size\"],\n )\n\n stride: int | tuple[int] | None = schema_utils.OneOfOptionsField(\n default=1,\n description=\"An integer or pair of integers specifying the stride of the convolution along the height and \"\n \"width. If a stride is not already specified in conv_layers, specifies the default stride of the \"\n \"2D convolutional kernel that will be used for each layer.\",\n field_options=[\n schema_utils.PositiveInteger(allow_none=False, description=\"\", default=1),\n schema_utils.List(list_type=int, allow_none=False),\n ],\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"stride\"],\n )\n\n padding_mode: str | None = schema_utils.StringOptions(\n options=[\"zeros\", \"reflect\", \"replicate\", \"circular\"],\n default=\"zeros\",\n description=\"If padding_mode is not already specified in conv_layers, specifies the default padding_mode of \"\n \"the 2D convolutional kernel that will be used for each layer.\",\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"padding_mode\"],\n )\n\n padding: int | tuple[int] | str | None = schema_utils.OneOfOptionsField(\n default=\"valid\",\n allow_none=True,\n description=\"An int, pair of ints (h, w), or one of ['valid', 'same'] specifying the padding used for\"\n \"convolution kernels.\",\n field_options=[\n schema_utils.NonNegativeInteger(allow_none=True, description=\"\", default=None),\n schema_utils.List(list_type=int, allow_none=False),\n schema_utils.StringOptions(options=[\"valid\", \"same\"], default=\"valid\", allow_none=False),\n ],\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"padding\"],\n )\n\n dilation: int | tuple[int] | None = schema_utils.OneOfOptionsField(\n default=1,\n allow_none=True,\n description=\"An int or pair of ints specifying the dilation rate to use for dilated convolution. If dilation \"\n \"is not already specified in conv_layers, specifies the default dilation of the 2D convolutional \"\n \"kernel that will be used for each layer.\",\n field_options=[\n schema_utils.PositiveInteger(allow_none=True, description=\"\", default=None),\n schema_utils.List(list_type=int, allow_none=False),\n ],\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"dilation\"],\n )\n\n groups: int | None = schema_utils.PositiveInteger(\n default=1,\n description=\"Groups controls the connectivity between convolution inputs and outputs. When groups = 1, each \"\n \"output channel depends on every input channel. When groups > 1, input and output channels are \"\n \"divided into groups separate groups, where each output channel depends only on the inputs in its \"\n \"respective input channel group. in_channels and out_channels must both be divisible by groups.\",\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"groups\"],\n )\n\n pool_function: str | None = schema_utils.StringOptions(\n [\"max\", \"average\", \"avg\", \"mean\"],\n default=\"max\",\n description=\"Pooling function to use.\",\n parameter_metadata=ENCODER_METADATA[\"conv_params\"][\"pool_function\"],\n )\n\n pool_kernel_size: int | tuple[int] | None = schema_utils.OneOfOptionsField(\n default=2,\n allow_none=True,\n description=\"An integer or pair of integers specifying the pooling size. If pool_kernel_size is not specified \"\n \"in conv_layers this is the default value that will be used for each layer.\",\n field_options=[\n schema_utils.PositiveInteger(allow_none=True, description=\"\", default=None),\n schema_utils.List(list_type=int, allow_none=False),\n ],\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"pool_kernel_size\"],\n )\n\n pool_stride: int | tuple[int] | None = schema_utils.OneOfOptionsField(\n default=None,\n allow_none=True,\n description=\"An integer or pair of integers specifying the pooling stride, which is the factor by which the \"\n \"pooling layer downsamples the feature map. Defaults to pool_kernel_size.\",\n field_options=[\n schema_utils.PositiveInteger(allow_none=True, description=\"\", default=None),\n schema_utils.List(list_type=int, allow_none=False),\n ],\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"pool_stride\"],\n )\n\n pool_padding: int | tuple[int] | None = schema_utils.OneOfOptionsField(\n default=0,\n allow_none=True,\n description=\"An integer or pair of ints specifying pooling padding (h, w).\",\n field_options=[\n schema_utils.NonNegativeInteger(allow_none=True, description=\"\", default=None),\n schema_utils.List(list_type=int, allow_none=False),\n ],\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"pool_padding\"],\n )\n\n pool_dilation: int | tuple[int] | None = schema_utils.OneOfOptionsField(\n default=1,\n allow_none=True,\n description=\"An integer or pair of ints specifying pooling dilation rate (h, w).\",\n field_options=[\n schema_utils.PositiveInteger(default=None, allow_none=True, description=\"\"),\n schema_utils.List(list_type=int, allow_none=False),\n ],\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"pool_dilation\"],\n )\n\n output_size: int | None = schema_utils.PositiveInteger(\n default=128,\n description=\"If output_size is not already specified in fc_layers this is the default output_size that will \"\n \"be used for each layer. It indicates the size of the output of a fully connected layer. \",\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"output_size\"],\n )\n\n conv_use_bias: bool | None = schema_utils.Boolean(\n default=True,\n description=\"If bias not already specified in conv_layers, specifies if the 2D convolutional kernel should \"\n \"have a bias term.\",\n )\n\n conv_norm: str | None = schema_utils.StringOptions(\n [\"batch\", \"layer\"],\n default=None,\n allow_none=True,\n description=\"If a norm is not already specified in conv_layers this is the default norm that will be used for \"\n \"each layer. It indicates the normalization applied to the activations and can be null, \"\n \"batch or layer.\",\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"conv_norm\"],\n )\n\n conv_norm_params: dict[str, Any] | None = schema_utils.Dict(\n default=None,\n description=\"Parameters used if conv_norm is either batch or layer. \",\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"conv_norm_params\"],\n )\n\n num_conv_layers: int | None = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"Number of convolutional layers to use in the encoder. \",\n parameter_metadata=ENCODER_METADATA[\"conv_params\"][\"num_conv_layers\"],\n )\n\n conv_layers: list[dict] | None = schema_utils.DictList(\n default=None,\n description=\"List of convolutional layers to use in the encoder. \",\n parameter_metadata=ENCODER_METADATA[\"conv_params\"][\"conv_layers\"],\n )\n\n fc_dropout: float | None = schema_utils.FloatRange(\n default=0.0,\n min=0,\n max=1,\n description=\"Dropout rate\",\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"fc_dropout\"],\n )\n\n fc_activation: str | None = schema_utils.ActivationOptions(\n description=\"If an activation is not already specified in fc_layers this is the default activation that will \"\n \"be used for each layer. It indicates the activation function applied to the output.\",\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"fc_activation\"],\n )\n\n fc_use_bias: bool | None = schema_utils.Boolean(\n default=True,\n description=\"Whether the layer uses a bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"fc_use_bias\"],\n )\n\n fc_bias_initializer: str | None = schema_utils.StringOptions(\n sorted(list(initializer_registry.keys())),\n default=\"zeros\",\n description=\"Initializer for the bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"fc_bias_initializer\"],\n )\n\n fc_weights_initializer: str | None = schema_utils.StringOptions(\n sorted(list(initializer_registry.keys())),\n default=\"xavier_uniform\",\n description=\"Initializer for the weights matrix.\",\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"fc_weights_initializer\"],\n )\n\n fc_norm: str | None = schema_utils.StringOptions(\n [\"batch\", \"layer\"],\n default=None,\n allow_none=True,\n description=\"If a norm is not already specified in fc_layers this is the default norm that will be used for \"\n \"each layer. It indicates the norm of the output and can be null, batch or layer.\",\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"fc_norm\"],\n )\n\n fc_norm_params: dict[str, Any] | None = schema_utils.Dict(\n default=None,\n description=\"Parameters used if norm is either batch or layer. For information on parameters used with batch \"\n \"see Torch's documentation on batch normalization or for layer see Torch's documentation on layer \"\n \"normalization.\",\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"fc_norm_params\"],\n )\n\n num_fc_layers: int | None | None = schema_utils.PositiveInteger(\n default=1,\n description=\"The number of stacked fully connected layers.\",\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"num_fc_layers\"],\n )\n\n fc_layers: list[dict] | None | None = schema_utils.DictList(\n default=None,\n description=\"A list of dictionaries containing the parameters of all the fully connected layers. The length \"\n \"of the list determines the number of stacked fully connected layers and the content of each \"\n \"dictionary determines the parameters for a specific layer. The available parameters for each \"\n \"layer are: activation, dropout, norm, norm_params, output_size, use_bias, bias_initializer and \"\n \"weights_initializer. If any of those values is missing from the dictionary, the default one \"\n \"specified as a parameter of the encoder will be used instead. \",\n parameter_metadata=ENCODER_METADATA[\"Stacked2DCNN\"][\"fc_layers\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"_resnet_legacy\", IMAGE)\n@ludwig_dataclass\nclass ResNetConfig(ImageEncoderConfig):\n @staticmethod\n def module_name():\n return \"ResNet\"\n\n type: str = schema_utils.ProtectedString(\n \"_resnet_legacy\",\n description=ENCODER_METADATA[\"ResNet\"][\"type\"].long_description,\n )\n\n dropout: float | None = schema_utils.FloatRange(\n default=0.0,\n min=0,\n max=1,\n description=\"Dropout rate\",\n parameter_metadata=ENCODER_METADATA[\"ResNet\"][\"dropout\"],\n )\n\n activation: str | None = schema_utils.ActivationOptions(\n description=\"if an activation is not already specified in fc_layers this is the default activation that will \"\n \"be used for each layer. It indicates the activation function applied to the output.\",\n parameter_metadata=ENCODER_METADATA[\"ResNet\"][\"activation\"],\n )\n\n height: int = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"Height of the input image.\",\n parameter_metadata=ENCODER_METADATA[\"ResNet\"][\"height\"],\n )\n\n width: int = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"Width of the input image.\",\n parameter_metadata=ENCODER_METADATA[\"ResNet\"][\"width\"],\n )\n\n resnet_size: int | None = schema_utils.PositiveInteger(\n default=50,\n description=\"The size of the ResNet model to use.\",\n parameter_metadata=ENCODER_METADATA[\"ResNet\"][\"resnet_size\"],\n )\n\n num_channels: int | None = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"Number of channels to use in the encoder. \",\n parameter_metadata=ENCODER_METADATA[\"ResNet\"][\"num_channels\"],\n )\n\n out_channels: int | None = schema_utils.NonNegativeInteger(\n default=32,\n description=\"Indicates the number of filters, and by consequence the output channels of the 2d convolution. \"\n \"If out_channels is not already specified in conv_layers this is the default out_channels that \"\n \"will be used for each layer. \",\n parameter_metadata=ENCODER_METADATA[\"ResNet\"][\"out_channels\"],\n )\n\n kernel_size: int | tuple[int] | None = schema_utils.OneOfOptionsField(\n default=3,\n allow_none=True,\n description=\"An integer or pair of integers specifying the kernel size. A single integer specifies a square \"\n \"kernel, while a pair of integers specifies the height and width of the kernel in that order (h, \"\n \"w). If a kernel_size is not specified in conv_layers this kernel_size that will be used for \"\n \"each layer.\",\n field_options=[\n schema_utils.PositiveInteger(allow_none=True, description=\"\", default=None),\n schema_utils.List(list_type=int, allow_none=False),\n ],\n parameter_metadata=ENCODER_METADATA[\"ResNet\"][\"kernel_size\"],\n )\n\n conv_stride: int | tuple[int] = schema_utils.OneOfOptionsField(\n default=1,\n allow_none=True,\n description=\"An integer or pair of integers specifying the stride of the initial convolutional layer.\",\n field_options=[\n schema_utils.PositiveInteger(allow_none=True, description=\"\", default=None),\n schema_utils.List(list_type=int, allow_none=False),\n ],\n parameter_metadata=ENCODER_METADATA[\"ResNet\"][\"conv_stride\"],\n )\n\n first_pool_kernel_size: int | tuple[int] = schema_utils.OneOfOptionsField(\n default=None,\n allow_none=True,\n description=\"Pool size to be used for the first pooling layer. If none, the first pooling layer is skipped.\",\n field_options=[\n schema_utils.PositiveInteger(allow_none=True, description=\"\", default=None),\n schema_utils.List(list_type=int, allow_none=False),\n ],\n parameter_metadata=ENCODER_METADATA[\"ResNet\"][\"first_pool_kernel_size\"],\n )\n\n first_pool_stride: int | tuple[int] = schema_utils.OneOfOptionsField(\n default=None,\n allow_none=True,\n description=\"Stride for first pooling layer. If null, defaults to first_pool_kernel_size.\",\n field_options=[\n schema_utils.PositiveInteger(allow_none=True, description=\"\", default=None),\n schema_utils.List(list_type=int, allow_none=False),\n ],\n parameter_metadata=ENCODER_METADATA[\"ResNet\"][\"first_pool_stride\"],\n )\n\n batch_norm_momentum: float = schema_utils.NonNegativeFloat(\n default=0.9,\n description=\"Momentum of the batch norm running statistics.\",\n parameter_metadata=ENCODER_METADATA[\"ResNet\"][\"batch_norm_momentum\"],\n )\n\n batch_norm_epsilon: float = schema_utils.NonNegativeFloat(\n default=0.001,\n description=\"Epsilon of the batch norm.\",\n parameter_metadata=ENCODER_METADATA[\"ResNet\"][\"batch_norm_epsilon\"],\n )\n\n use_bias: bool | None = schema_utils.Boolean(\n default=True,\n description=\"Whether the layer uses a bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"ResNet\"][\"use_bias\"],\n )\n\n bias_initializer: str | None = schema_utils.StringOptions(\n sorted(list(initializer_registry.keys())),\n default=\"zeros\",\n description=\"initializer for the bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"ResNet\"][\"bias_initializer\"],\n )\n\n weights_initializer: str | None = schema_utils.StringOptions(\n sorted(list(initializer_registry.keys())),\n default=\"xavier_uniform\",\n description=\"Initializer for the weights matrix.\",\n parameter_metadata=ENCODER_METADATA[\"ResNet\"][\"weights_initializer\"],\n )\n\n output_size: int | None = schema_utils.PositiveInteger(\n default=128,\n description=\"if output_size is not already specified in fc_layers this is the default output_size that will \"\n \"be used for each layer. It indicates the size of the output of a fully connected layer. \",\n parameter_metadata=ENCODER_METADATA[\"ResNet\"][\"output_size\"],\n )\n\n norm: str | None = schema_utils.StringOptions(\n [\"batch\", \"layer\"],\n default=None,\n allow_none=True,\n description=\"if a norm is not already specified in fc_layers this is the default norm that will be used for \"\n \"each layer. It indicates the norm of the output and can be null, batch or layer.\",\n parameter_metadata=ENCODER_METADATA[\"ResNet\"][\"norm\"],\n )\n\n norm_params: dict[str, Any] | None = schema_utils.Dict(\n default=None,\n description=\"parameters used if norm is either batch or layer. For information on parameters used with batch \"\n \"see Torch's documentation on batch normalization or for layer see Torch's documentation on layer \"\n \"normalization.\",\n parameter_metadata=ENCODER_METADATA[\"ResNet\"][\"norm_params\"],\n )\n\n num_fc_layers: int | None | None = schema_utils.PositiveInteger(\n default=1,\n description=\"The number of stacked fully connected layers.\",\n parameter_metadata=ENCODER_METADATA[\"ResNet\"][\"num_fc_layers\"],\n )\n\n fc_layers: list[dict] | None | None = schema_utils.DictList(\n default=None,\n description=\"A list of dictionaries containing the parameters of all the fully connected layers. The length \"\n \"of the list determines the number of stacked fully connected layers and the content of each \"\n \"dictionary determines the parameters for a specific layer. The available parameters for each \"\n \"layer are: activation, dropout, norm, norm_params, output_size, use_bias, bias_initializer and \"\n \"weights_initializer. If any of those values is missing from the dictionary, the default one \"\n \"specified as a parameter of the encoder will be used instead. \",\n parameter_metadata=ENCODER_METADATA[\"ResNet\"][\"fc_layers\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"mlp_mixer\", IMAGE)\n@ludwig_dataclass\nclass MLPMixerConfig(ImageEncoderConfig):\n @staticmethod\n def module_name():\n return \"MLPMixer\"\n\n type: str = schema_utils.ProtectedString(\n \"mlp_mixer\",\n description=ENCODER_METADATA[\"MLPMixer\"][\"type\"].long_description,\n )\n\n dropout: float = schema_utils.FloatRange(\n default=0.0,\n min=0,\n max=1,\n description=\"Dropout rate.\",\n parameter_metadata=ENCODER_METADATA[\"MLPMixer\"][\"dropout\"],\n )\n\n height: int = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"Height of the input image.\",\n parameter_metadata=ENCODER_METADATA[\"MLPMixer\"][\"height\"],\n )\n\n width: int = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"Width of the input image.\",\n parameter_metadata=ENCODER_METADATA[\"MLPMixer\"][\"width\"],\n )\n\n num_channels: int = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"Number of channels to use in the encoder. \",\n parameter_metadata=ENCODER_METADATA[\"MLPMixer\"][\"num_channels\"],\n )\n\n patch_size: int = schema_utils.PositiveInteger(\n default=16,\n description=\"The image patch size. Each patch is patch_size² pixels. Must evenly divide the image width and \"\n \"height.\",\n parameter_metadata=ENCODER_METADATA[\"MLPMixer\"][\"patch_size\"],\n )\n\n embed_size: int = schema_utils.PositiveInteger(\n default=512,\n description=\"The patch embedding size, the output size of the mixer if avg_pool is true.\",\n parameter_metadata=ENCODER_METADATA[\"MLPMixer\"][\"embed_size\"],\n )\n\n token_size: int = schema_utils.PositiveInteger(\n default=2048,\n description=\"The per-patch embedding size.\",\n parameter_metadata=ENCODER_METADATA[\"MLPMixer\"][\"token_size\"],\n )\n\n channel_dim: int = schema_utils.PositiveInteger(\n default=256,\n description=\"Number of channels in hidden layer.\",\n parameter_metadata=ENCODER_METADATA[\"MLPMixer\"][\"channel_dim\"],\n )\n\n num_layers: int = schema_utils.PositiveInteger(\n default=8,\n description=\"The depth of the network (the number of Mixer blocks).\",\n parameter_metadata=ENCODER_METADATA[\"MLPMixer\"][\"num_layers\"],\n )\n\n avg_pool: bool = schema_utils.Boolean(\n default=True,\n description=\"If true, pools output over patch dimension, outputs a vector of shape (embed_size). If false, \"\n \"the output tensor is of shape (n_patches, embed_size), where n_patches is img_height x img_width \"\n \"/ patch_size².\",\n parameter_metadata=ENCODER_METADATA[\"MLPMixer\"][\"avg_pool\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"_vit_legacy\", IMAGE)\n@ludwig_dataclass\nclass ViTConfig(ImageEncoderConfig):\n @staticmethod\n def module_name():\n return \"ViT\"\n\n type: str = schema_utils.ProtectedString(\n \"_vit_legacy\",\n description=ENCODER_METADATA[\"ViT\"][\"type\"].long_description,\n )\n\n height: int = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"Height of the input image.\",\n parameter_metadata=ENCODER_METADATA[\"ViT\"][\"height\"],\n )\n\n width: int = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"Width of the input image.\",\n parameter_metadata=ENCODER_METADATA[\"ViT\"][\"width\"],\n )\n\n num_hidden_layers: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"ViT\"][\"num_hidden_layers\"],\n )\n\n hidden_size: int = schema_utils.PositiveInteger(\n default=768,\n description=\"Dimensionality of the encoder layers and the pooling layer.\",\n parameter_metadata=ENCODER_METADATA[\"ViT\"][\"hidden_size\"],\n )\n\n hidden_act: str = schema_utils.StringOptions(\n [\"relu\", \"gelu\", \"selu\", \"gelu_new\"],\n default=\"gelu\",\n description=\"Hidden layer activation, one of gelu, relu, selu or gelu_new.\",\n parameter_metadata=ENCODER_METADATA[\"ViT\"][\"hidden_act\"],\n )\n\n hidden_dropout_prob: float = schema_utils.NonNegativeFloat(\n default=0.1,\n description=\"The dropout rate for all fully connected layers in the embeddings, encoder, and pooling.\",\n parameter_metadata=ENCODER_METADATA[\"ViT\"][\"hidden_dropout_prob\"],\n )\n\n num_attention_heads: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of attention heads in each attention layer.\",\n parameter_metadata=ENCODER_METADATA[\"ViT\"][\"num_attention_heads\"],\n )\n\n attention_probs_dropout_prob: float = schema_utils.NonNegativeFloat(\n default=0.1,\n description=\"The dropout rate for the attention probabilities.\",\n parameter_metadata=ENCODER_METADATA[\"ViT\"][\"attention_probs_dropout_prob\"],\n )\n\n intermediate_size: int = schema_utils.PositiveInteger(\n default=3072,\n description=\"Dimensionality of the intermediate (i.e., feed-forward) layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"ViT\"][\"intermediate_size\"],\n )\n\n initializer_range: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\",\n parameter_metadata=ENCODER_METADATA[\"ViT\"][\"initializer_range\"],\n )\n\n layer_norm_eps: float = schema_utils.NonNegativeFloat(\n default=1e-12,\n description=\"The epsilon used by the layer normalization layers.\",\n parameter_metadata=ENCODER_METADATA[\"ViT\"][\"layer_norm_eps\"],\n )\n\n gradient_checkpointing: bool = schema_utils.Boolean(\n default=False,\n description=\"\",\n parameter_metadata=ENCODER_METADATA[\"ViT\"][\"gradient_checkpointing\"],\n )\n\n patch_size: int = schema_utils.PositiveInteger(\n default=16,\n description=\"The image patch size. Each patch is patch_size² pixels. Must evenly divide the image width and \"\n \"height.\",\n parameter_metadata=ENCODER_METADATA[\"ViT\"][\"patch_size\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"ViT\"][\"saved_weights_in_checkpoint\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=True,\n description=\"Is the encoder trainable.\",\n parameter_metadata=ENCODER_METADATA[\"ViT\"][\"trainable\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Use pre-trained model weights from Hugging Face.\",\n parameter_metadata=ENCODER_METADATA[\"ViT\"][\"use_pretrained\"],\n )\n\n pretrained_model: str = schema_utils.String(\n default=\"google/vit-base-patch16-224\",\n description=\"The name of the pre-trained model to use.\",\n parameter_metadata=ENCODER_METADATA[\"ViT\"][\"pretrained_model\"],\n )\n\n def set_fixed_preprocessing_params(self, model_type: str, preprocessing: \"ImagePreprocessingConfig\"):\n \"\"\"If the encoder is not in trainable mode, override the image width and height to be compatible with the\n pretrained encoder image dimension requirements.\"\"\"\n if self.requires_equal_dimensions() and self.required_width() != self.required_height():\n raise ValueError(\"Invalid definition. `required_width` and `required_height` are not equal\")\n\n preprocessing.requires_equal_dimensions = self.requires_equal_dimensions()\n if not self.trainable or self.use_pretrained:\n preprocessing.height = self.required_height()\n preprocessing.width = self.required_width()\n\n @classmethod\n def requires_equal_dimensions(cls) -> bool:\n return True\n\n @classmethod\n def required_width(cls) -> int | None:\n return 224\n\n @classmethod\n def required_height(cls) -> int | None:\n return 224\n\n def is_pretrained(self) -> bool:\n return self.use_pretrained\n\n\n@DeveloperAPI\n@register_encoder_config(\"unet\", IMAGE)\n@ludwig_dataclass\nclass UNetEncoderConfig(ImageEncoderConfig):\n @staticmethod\n def module_name():\n return \"UNetEncoder\"\n\n type: str = schema_utils.ProtectedString(\n \"unet\",\n description=ENCODER_METADATA[\"UNetEncoder\"][\"type\"].long_description,\n )\n\n height: int = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"Height of the input image.\",\n parameter_metadata=ENCODER_METADATA[\"UNetEncoder\"][\"height\"],\n )\n\n width: int = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"Width of the input image.\",\n parameter_metadata=ENCODER_METADATA[\"UNetEncoder\"][\"width\"],\n )\n\n num_channels: int | None = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"Number of channels in the input image. \",\n parameter_metadata=ENCODER_METADATA[\"UNetEncoder\"][\"num_channels\"],\n )\n\n conv_norm: str | None = schema_utils.StringOptions(\n [\"batch\"],\n default=\"batch\",\n allow_none=True,\n description=\"This is the default norm that will be used for each double conv layer.\" \"It can be null or batch.\",\n parameter_metadata=ENCODER_METADATA[\"UNetEncoder\"][\"conv_norm\"],\n )\n"
},
{
"path": "ludwig/schema/encoders/image/timm.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import IMAGE\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import register_encoder_config\nfrom ludwig.schema.metadata import ENCODER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@ludwig_dataclass\nclass TimmBaseConfig(BaseEncoderConfig):\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Download model weights from pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"TimmEncoder\"][\"use_pretrained\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to use weights saved in the Ludwig checkpoint instead of pretrained weights.\",\n parameter_metadata=ENCODER_METADATA[\"TimmEncoder\"][\"saved_weights_in_checkpoint\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether the encoder parameters are trainable.\",\n parameter_metadata=ENCODER_METADATA[\"TimmEncoder\"][\"trainable\"],\n )\n\n def is_pretrained(self) -> bool:\n return self.use_pretrained\n\n\n@DeveloperAPI\n@register_encoder_config(\"timm\", IMAGE)\n@ludwig_dataclass\nclass TimmEncoderConfig(TimmBaseConfig):\n type: str = schema_utils.ProtectedString(\"timm\", description=\"Type of encoder.\")\n\n model_name: str = schema_utils.String(\n default=\"caformer_s18\",\n description=(\n \"Name of the timm model to use. Any model from the timm library is supported. \"\n \"See https://huggingface.co/docs/timm for available models.\"\n ),\n parameter_metadata=ENCODER_METADATA[\"TimmEncoder\"][\"model_name\"],\n )\n\n\n# Convenience aliases for MetaFormer variants with curated model_name options\n\nCAFORMER_MODELS = [\n \"caformer_s18\",\n \"caformer_s36\",\n \"caformer_m36\",\n \"caformer_b36\",\n \"caformer_s18.sail_in22k_ft_in1k\",\n \"caformer_s18.sail_in22k_ft_in1k_384\",\n \"caformer_s36.sail_in22k_ft_in1k\",\n \"caformer_s36.sail_in22k_ft_in1k_384\",\n \"caformer_m36.sail_in22k_ft_in1k\",\n \"caformer_m36.sail_in22k_ft_in1k_384\",\n \"caformer_b36.sail_in22k_ft_in1k\",\n \"caformer_b36.sail_in22k_ft_in1k_384\",\n]\n\nCONVFORMER_MODELS = [\n \"convformer_s18\",\n \"convformer_s36\",\n \"convformer_m36\",\n \"convformer_b36\",\n \"convformer_s18.sail_in22k_ft_in1k\",\n \"convformer_s18.sail_in22k_ft_in1k_384\",\n \"convformer_s36.sail_in22k_ft_in1k\",\n \"convformer_s36.sail_in22k_ft_in1k_384\",\n \"convformer_m36.sail_in22k_ft_in1k\",\n \"convformer_m36.sail_in22k_ft_in1k_384\",\n \"convformer_b36.sail_in22k_ft_in1k\",\n \"convformer_b36.sail_in22k_ft_in1k_384\",\n]\n\nPOOLFORMER_MODELS = [\n \"poolformerv2_s12\",\n \"poolformerv2_s24\",\n \"poolformerv2_s36\",\n \"poolformerv2_m36\",\n \"poolformerv2_m48\",\n \"poolformer_s12\",\n \"poolformer_s24\",\n \"poolformer_s36\",\n \"poolformer_m36\",\n \"poolformer_m48\",\n]\n\n\n@DeveloperAPI\n@register_encoder_config(\"caformer\", IMAGE)\n@ludwig_dataclass\nclass TimmCAFormerEncoderConfig(TimmBaseConfig):\n type: str = schema_utils.ProtectedString(\"caformer\", description=\"Type of encoder.\")\n\n model_name: str = schema_utils.StringOptions(\n CAFORMER_MODELS,\n default=\"caformer_s18\",\n allow_none=False,\n description=(\n \"CAFormer model variant. Hybrid Conv+Attention MetaFormer achieving SOTA accuracy. \"\n \"Variants with '.sail_in22k_ft_in1k' are pretrained on ImageNet-21K and finetuned on ImageNet-1K. \"\n \"Variants with '_384' use 384x384 input resolution.\"\n ),\n parameter_metadata=ENCODER_METADATA[\"TimmCAFormerEncoder\"][\"model_name\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"convformer\", IMAGE)\n@ludwig_dataclass\nclass TimmConvFormerEncoderConfig(TimmBaseConfig):\n type: str = schema_utils.ProtectedString(\"convformer\", description=\"Type of encoder.\")\n\n model_name: str = schema_utils.StringOptions(\n CONVFORMER_MODELS,\n default=\"convformer_s18\",\n allow_none=False,\n description=(\n \"ConvFormer model variant. Pure CNN MetaFormer that outperforms ConvNeXt. \"\n \"Variants with '.sail_in22k_ft_in1k' are pretrained on ImageNet-21K and finetuned on ImageNet-1K.\"\n ),\n parameter_metadata=ENCODER_METADATA[\"TimmConvFormerEncoder\"][\"model_name\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"poolformer\", IMAGE)\n@ludwig_dataclass\nclass TimmPoolFormerEncoderConfig(TimmBaseConfig):\n type: str = schema_utils.ProtectedString(\"poolformer\", description=\"Type of encoder.\")\n\n model_name: str = schema_utils.StringOptions(\n POOLFORMER_MODELS,\n default=\"poolformerv2_s12\",\n allow_none=False,\n description=(\n \"PoolFormer model variant. MetaFormer using simple average pooling as token mixer. \"\n \"V2 variants use StarReLU activation and improved training recipe.\"\n ),\n parameter_metadata=ENCODER_METADATA[\"TimmPoolFormerEncoder\"][\"model_name\"],\n )\n"
},
{
"path": "ludwig/schema/encoders/image/torchvision.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import IMAGE\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import register_encoder_config\nfrom ludwig.schema.metadata import ENCODER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@ludwig_dataclass\nclass TVBaseEncoderConfig(BaseEncoderConfig):\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Download model weights from pre-trained model.\",\n parameter_metadata=ENCODER_METADATA[\"TVBaseEncoder\"][\"use_pretrained\"],\n )\n\n model_cache_dir: str | None = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"Directory path to cache pretrained model weights.\",\n parameter_metadata=ENCODER_METADATA[\"TVBaseEncoder\"][\"model_cache_dir\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to save the weights in the checkpoint.\",\n parameter_metadata=ENCODER_METADATA[\"TVBaseEncoder\"][\"saved_weights_in_checkpoint\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=True,\n description=\"Is the encoder trainable.\",\n parameter_metadata=ENCODER_METADATA[\"TVBaseEncoder\"][\"trainable\"],\n )\n\n def is_pretrained(self) -> bool:\n return self.use_pretrained\n\n\n@DeveloperAPI\n@register_encoder_config(\"alexnet\", IMAGE)\n@ludwig_dataclass\nclass TVAlexNetEncoderConfig(TVBaseEncoderConfig):\n type: str = schema_utils.ProtectedString(\"alexnet\", description=\"Type of encoder.\")\n\n model_variant: str = schema_utils.StringOptions(\n [\"base\"],\n default=\"base\",\n allow_none=False,\n description=\"Pretrained model variant to use.\",\n parameter_metadata=ENCODER_METADATA[\"TVAlexNetEncoder\"][\"model_variant\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"convnext\", IMAGE)\n@ludwig_dataclass\nclass TVConvNeXtEncoderConfig(TVBaseEncoderConfig):\n type: str = schema_utils.ProtectedString(\"convnext\", description=\"Type of encoder.\")\n\n model_variant: str = schema_utils.StringOptions(\n [\"tiny\", \"small\", \"base\", \"large\"],\n default=\"base\",\n allow_none=False,\n description=\"Pretrained model variant to use.\",\n parameter_metadata=ENCODER_METADATA[\"TVConvNeXtEncoder\"][\"model_variant\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"densenet\", IMAGE)\n@ludwig_dataclass\nclass TVDenseNetEncoderConfig(TVBaseEncoderConfig):\n type: str = schema_utils.ProtectedString(\"densenet\", description=\"Type of encoder.\")\n\n model_variant: int = schema_utils.IntegerOptions(\n [121, 161, 169, 201],\n default=121,\n allow_none=False,\n description=\"Pretrained model variant to use.\",\n parameter_metadata=ENCODER_METADATA[\"TVDenseNetEncoder\"][\"model_variant\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"efficientnet\", IMAGE)\n@ludwig_dataclass\nclass TVEfficientNetEncoderConfig(TVBaseEncoderConfig):\n type: str = schema_utils.ProtectedString(\"efficientnet\", description=\"Type of encoder.\")\n\n model_variant: str = schema_utils.StringOptions(\n [\n \"b0\",\n \"b1\",\n \"b2\",\n \"b3\",\n \"b4\",\n \"b5\",\n \"b6\",\n \"b7\",\n \"v2_s\",\n \"v2_m\",\n \"v2_l\",\n ],\n default=\"b0\",\n allow_none=False,\n description=\"Pretrained model variant to use.\",\n parameter_metadata=ENCODER_METADATA[\"TVEfficientNetEncoder\"][\"model_variant\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"googlenet\", IMAGE)\n@ludwig_dataclass\nclass TVGoogLeNetEncoderConfig(TVBaseEncoderConfig):\n type: str = schema_utils.ProtectedString(\"googlenet\", description=\"Type of encoder.\")\n\n model_variant: str = schema_utils.StringOptions(\n [\"base\"],\n default=\"base\",\n allow_none=False,\n description=\"Pretrained model variant to use.\",\n parameter_metadata=ENCODER_METADATA[\"TVGoogLeNetEncoder\"][\"model_variant\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"inceptionv3\", IMAGE)\n@ludwig_dataclass\nclass TVInceptionV3EncoderConfig(TVBaseEncoderConfig):\n type: str = schema_utils.ProtectedString(\"inceptionv3\", description=\"Type of encoder.\")\n\n model_variant: str = schema_utils.StringOptions(\n [\"base\"],\n default=\"base\",\n allow_none=False,\n description=\"Pretrained model variant to use.\",\n parameter_metadata=ENCODER_METADATA[\"TVGoogLeNetEncoder\"][\"model_variant\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"maxvit\", IMAGE)\n@ludwig_dataclass\nclass TVMaxVitEncoderConfig(TVBaseEncoderConfig):\n type: str = schema_utils.ProtectedString(\"maxvit\", description=\"Type of encoder.\")\n\n model_variant: str = schema_utils.StringOptions(\n [\"t\"],\n default=\"t\",\n allow_none=False,\n description=\"Pretrained model variant to use.\",\n parameter_metadata=ENCODER_METADATA[\"TVMNASNetEncoder\"][\"model_variant\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"mnasnet\", IMAGE)\n@ludwig_dataclass\nclass TVMNASNetEncoderConfig(TVBaseEncoderConfig):\n type: str = schema_utils.ProtectedString(\"mnasnet\", description=\"Type of encoder.\")\n\n model_variant: str = schema_utils.StringOptions(\n [\"0_5\", \"0_75\", \"1_0\", \"1_3\"],\n default=\"0_5\",\n allow_none=False,\n description=\"Pretrained model variant to use.\",\n parameter_metadata=ENCODER_METADATA[\"TVMNASNetEncoder\"][\"model_variant\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"mobilenetv2\", IMAGE)\n@ludwig_dataclass\nclass TVMobileNetV2EncoderConfig(TVBaseEncoderConfig):\n type: str = schema_utils.ProtectedString(\"mobilenetv2\", description=\"Type of encoder.\")\n\n model_variant: str = schema_utils.StringOptions(\n [\"base\"],\n default=\"base\",\n allow_none=False,\n description=\"Pretrained model variant to use.\",\n parameter_metadata=ENCODER_METADATA[\"TVMobileNetV2Encoder\"][\"model_variant\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"mobilenetv3\", IMAGE)\n@ludwig_dataclass\nclass TVMobileNetV3EncoderConfig(TVBaseEncoderConfig):\n type: str = schema_utils.ProtectedString(\"mobilenetv3\", description=\"Type of encoder.\")\n\n model_variant: str = schema_utils.StringOptions(\n [\n \"small\",\n \"large\",\n ],\n default=\"small\",\n allow_none=False,\n description=\"Pretrained model variant to use.\",\n parameter_metadata=ENCODER_METADATA[\"TVMobileNetV3Encoder\"][\"model_variant\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"regnet\", IMAGE)\n@ludwig_dataclass\nclass TVRegNetEncoderConfig(TVBaseEncoderConfig):\n type: str = schema_utils.ProtectedString(\"regnet\", description=\"Type of encoder.\")\n\n model_variant: str = schema_utils.StringOptions(\n [\n \"x_1_6gf\",\n \"x_16gf\",\n \"x_32gf\",\n \"x_3_2gf\",\n \"x_400mf\",\n \"x_800mf\",\n \"x_8gf\",\n \"y_128gf\",\n \"y_16gf\",\n \"y_1_6gf\",\n \"y_32gf\",\n \"y_3_2gf\",\n \"y_400mf\",\n \"y_800mf\",\n \"y_8gf\",\n ],\n default=\"x_1_6gf\",\n allow_none=False,\n description=\"Pretrained model variant to use.\",\n parameter_metadata=ENCODER_METADATA[\"TVRegNetEncoder\"][\"model_variant\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"resnet\", IMAGE)\n@ludwig_dataclass\nclass TVResNetEncoderConfig(TVBaseEncoderConfig):\n type: str = schema_utils.ProtectedString(\"resnet\", description=\"Type of encoder.\")\n\n model_variant: int = schema_utils.IntegerOptions(\n [18, 34, 50, 101, 152],\n default=50,\n allow_none=False,\n description=\"Pretrained model variant to use.\",\n parameter_metadata=ENCODER_METADATA[\"TVResNetEncoder\"][\"model_variant\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"resnext\", IMAGE)\n@ludwig_dataclass\nclass TVResNeXtEncoderConfig(TVBaseEncoderConfig):\n type: str = schema_utils.ProtectedString(\"resnext\", description=\"Type of encoder.\")\n\n model_variant: str = schema_utils.StringOptions(\n [\"50_32x4d\", \"101_32x8d\", \"101_64x4d\"],\n default=\"50_32x4d\",\n allow_none=False,\n description=\"Pretrained model variant to use.\",\n parameter_metadata=ENCODER_METADATA[\"TVResNeXtEncoder\"][\"model_variant\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"shufflenet_v2\", IMAGE)\n@ludwig_dataclass\nclass TVShuffleNetV2EncoderConfig(TVBaseEncoderConfig):\n type: str = schema_utils.ProtectedString(\"shufflenet_v2\", description=\"Type of encoder.\")\n\n model_variant: str = schema_utils.StringOptions(\n [\n \"x0_5\",\n \"x1_0\",\n \"x1_5\",\n \"x2_0\",\n ],\n default=\"x0_5\",\n allow_none=False,\n description=\"Pretrained model variant to use.\",\n parameter_metadata=ENCODER_METADATA[\"TVShuffleNetV2Encoder\"][\"model_variant\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"squeezenet\", IMAGE)\n@ludwig_dataclass\nclass TVSqueezeNetEncoderConfig(TVBaseEncoderConfig):\n type: str = schema_utils.ProtectedString(\"squeezenet\", description=\"Type of encoder.\")\n\n model_variant: str = schema_utils.StringOptions(\n [\n \"1_0\",\n \"1_1\",\n ],\n default=\"1_0\",\n allow_none=False,\n description=\"Pretrained model variant to use.\",\n parameter_metadata=ENCODER_METADATA[\"TVSqueezeNetEncoder\"][\"model_variant\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"swin_transformer\", IMAGE)\n@ludwig_dataclass\nclass TVSwinTransformerEncoderConfig(TVBaseEncoderConfig):\n type: str = schema_utils.ProtectedString(\"swin_transformer\", description=\"Type of encoder.\")\n\n model_variant: str = schema_utils.StringOptions(\n [\n \"t\",\n \"s\",\n \"b\",\n ],\n default=\"t\",\n allow_none=False,\n description=\"Pretrained model variant to use.\",\n parameter_metadata=ENCODER_METADATA[\"TVSwinTransformerEncoder\"][\"model_variant\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"vit\", IMAGE)\n@ludwig_dataclass\nclass TVViTEncoderConfig(TVBaseEncoderConfig):\n type: str = schema_utils.ProtectedString(\"vit\", description=\"Type of encoder.\")\n\n model_variant: str = schema_utils.StringOptions(\n [\n \"b_16\",\n \"b_32\",\n \"l_16\",\n \"l_32\",\n \"h_14\",\n ],\n default=\"b_16\",\n allow_none=False,\n description=\"Pretrained model variant to use.\",\n parameter_metadata=ENCODER_METADATA[\"TVViTEncoder\"][\"model_variant\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"vgg\", IMAGE)\n@ludwig_dataclass\nclass TVVGGEncoderConfig(TVBaseEncoderConfig):\n type: str = schema_utils.ProtectedString(\"vgg\", description=\"Type of encoder.\")\n\n model_variant: int | str = schema_utils.OneOfOptionsField(\n default=11,\n description=\"Pretrained model variant to use.\",\n field_options=[\n schema_utils.IntegerOptions(\n [\n 11,\n 13,\n 16,\n 19,\n ],\n default=11,\n allow_none=False,\n ),\n schema_utils.StringOptions(\n [\n \"11_bn\",\n \"13_bn\",\n \"16_bn\",\n \"19_bn\",\n ],\n default=\"11_bn\",\n allow_none=False,\n ),\n ],\n allow_none=False,\n parameter_metadata=ENCODER_METADATA[\"TVVGGEncoder\"][\"model_variant\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"wide_resnet\", IMAGE)\n@ludwig_dataclass\nclass TVWideResNetEncoderConfig(TVBaseEncoderConfig):\n type: str = schema_utils.ProtectedString(\"wide_resnet\", description=\"Type of encoder.\")\n\n model_variant: str = schema_utils.StringOptions(\n [\n \"50_2\",\n \"101_2\",\n ],\n default=\"50_2\",\n allow_none=False,\n description=\"Pretrained model variant to use.\",\n parameter_metadata=ENCODER_METADATA[\"TVViTEncoder\"][\"model_variant\"],\n )\n"
},
{
"path": "ludwig/schema/encoders/sequence_encoders.py",
"content": "from dataclasses import Field\nfrom typing import TYPE_CHECKING\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import AUDIO, SEQUENCE, TEXT, TIMESERIES\nfrom ludwig.schema import common_fields\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import register_encoder_config\nfrom ludwig.schema.metadata import ENCODER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\nif TYPE_CHECKING:\n from ludwig.schema.features.preprocessing.sequence import SequencePreprocessingConfig\n\nCONV_LAYERS_DESCRIPTION = \"\"\"\nA list of dictionaries containing the parameters of all the convolutional layers.\nThe length of the list determines the number of stacked convolutional layers and the content of each dictionary\ndetermines the parameters for a specific layer. The available parameters for each layer are: `activation`, `dropout`,\n`norm`, `norm_params`, `num_filters`, `filter_size`, `strides`, `padding`, `dilation_rate`, `use_bias`, `pool_function`,\n`pool_padding`, `pool_size`, `pool_strides`, `bias_initializer`, `weights_initializer`. If any of those values is\nmissing from the dictionary, the default one specified as a parameter of the encoder will be used instead. If both\n`conv_layers` and `num_conv_layers` are `null`, a default list will be assigned to `conv_layers` with the value\n`[{filter_size: 7, pool_size: 3}, {filter_size: 7, pool_size: 3}, {filter_size: 3, pool_size: null},\n{filter_size: 3, pool_size: null}, {filter_size: 3, pool_size: null}, {filter_size: 3, pool_size: 3}]`.\n\"\"\"\n\nNUM_CONV_LAYERS_DESCRIPTION = \"The number of stacked convolutional layers when `conv_layers` is `null`.\"\n\n\ndef NumFiltersField(default: int = 256) -> Field:\n return schema_utils.PositiveInteger(\n default=default,\n description=\"Number of filters, and by consequence number of output channels of the 1d convolution.\",\n parameter_metadata=ENCODER_METADATA[\"conv_params\"][\"num_filters\"],\n )\n\n\ndef FilterSizeField(default: int = 3) -> Field:\n return schema_utils.PositiveInteger(\n default=default,\n description=\"Size of the 1d convolutional filter. It indicates how wide the 1d convolutional filter is.\",\n parameter_metadata=ENCODER_METADATA[\"conv_params\"][\"filter_size\"],\n )\n\n\ndef PoolFunctionField(default: str = \"max\") -> Field:\n return schema_utils.ReductionOptions(\n default=default,\n description=(\n \"Pooling function to use. `max` will select the maximum value. Any of `average`, `avg`, or \"\n \"`mean` will compute the mean value\"\n ),\n parameter_metadata=ENCODER_METADATA[\"conv_params\"][\"pool_function\"],\n )\n\n\ndef PoolSizeField(default: int | None = None) -> Field:\n return schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=(\n \"The default pool_size that will be used for each layer. If a pool_size is not already specified \"\n \"in conv_layers this is the default pool_size that will be used for each layer. It indicates the size of \"\n \"the max pooling that will be performed along the `s` sequence dimension after the convolution operation.\"\n ),\n parameter_metadata=ENCODER_METADATA[\"conv_params\"][\"pool_size\"],\n )\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass SequenceEncoderConfig(BaseEncoderConfig):\n \"\"\"Base class for sequence encoders.\"\"\"\n\n def set_fixed_preprocessing_params(self, model_type: str, preprocessing: \"SequencePreprocessingConfig\"):\n if isinstance(preprocessing, dict):\n preprocessing[\"cache_encoder_embeddings\"] = False\n else:\n preprocessing.cache_encoder_embeddings = False\n\n\n@DeveloperAPI\n@register_encoder_config(\"passthrough\", [TIMESERIES])\n@ludwig_dataclass\nclass SequencePassthroughConfig(SequenceEncoderConfig):\n @staticmethod\n def module_name():\n return \"SequencePassthrough\"\n\n type: str = schema_utils.ProtectedString(\n \"passthrough\",\n description=ENCODER_METADATA[\"SequencePassthrough\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = common_fields.MaxSequenceLengthField()\n\n encoding_size: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"The size of the encoding vector, or None if sequence elements are scalars.\",\n parameter_metadata=ENCODER_METADATA[\"SequencePassthrough\"][\"encoding_size\"],\n )\n\n reduce_output: str = common_fields.ReduceOutputField(default=None)\n\n\n@DeveloperAPI\n@register_encoder_config(\"embed\", [SEQUENCE, TEXT])\n@ludwig_dataclass\nclass SequenceEmbedConfig(SequenceEncoderConfig):\n @staticmethod\n def module_name():\n return \"SequenceEmbed\"\n\n type: str = schema_utils.ProtectedString(\n \"embed\",\n description=ENCODER_METADATA[\"SequenceEmbed\"][\"type\"].long_description,\n )\n\n dropout: float = common_fields.DropoutField(description=\"Dropout rate applied to the embedding.\")\n\n max_sequence_length: int = common_fields.MaxSequenceLengthField()\n\n representation: str = common_fields.RepresentationField()\n\n vocab: list = common_fields.VocabField()\n\n weights_initializer: str = common_fields.WeightsInitializerField(default=\"uniform\")\n\n reduce_output: str = common_fields.ReduceOutputField()\n\n embedding_size: int = common_fields.EmbeddingSizeField()\n\n embeddings_on_cpu: bool = common_fields.EmbeddingsOnCPUField()\n\n embeddings_trainable: bool = common_fields.EmbeddingsTrainableField()\n\n pretrained_embeddings: str = common_fields.PretrainedEmbeddingsField()\n\n\n@DeveloperAPI\n@register_encoder_config(\"parallel_cnn\", [AUDIO, SEQUENCE, TEXT, TIMESERIES])\n@ludwig_dataclass\nclass ParallelCNNConfig(SequenceEncoderConfig):\n @staticmethod\n def module_name():\n return \"ParallelCNN\"\n\n type: str = schema_utils.ProtectedString(\n \"parallel_cnn\",\n description=ENCODER_METADATA[\"ParallelCNN\"][\"type\"].long_description,\n )\n\n dropout: float = common_fields.DropoutField(description=\"Dropout rate applied to the embedding.\")\n\n activation: str = schema_utils.ActivationOptions(\n description=\"The default activation function that will be used for each layer.\"\n )\n\n max_sequence_length: int = common_fields.MaxSequenceLengthField()\n\n representation: str = common_fields.RepresentationField()\n\n vocab: list = common_fields.VocabField()\n\n use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use a bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"ParallelCNN\"][\"use_bias\"],\n )\n\n bias_initializer: str = common_fields.BiasInitializerField()\n\n weights_initializer: str = common_fields.WeightsInitializerField()\n\n should_embed: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to embed the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"ParallelCNN\"][\"should_embed\"],\n )\n\n embedding_size: int = common_fields.EmbeddingSizeField()\n\n embeddings_on_cpu: bool = common_fields.EmbeddingsOnCPUField()\n\n embeddings_trainable: bool = common_fields.EmbeddingsTrainableField()\n\n pretrained_embeddings: str = common_fields.PretrainedEmbeddingsField()\n\n reduce_output: str = common_fields.ReduceOutputField()\n\n num_conv_layers: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=NUM_CONV_LAYERS_DESCRIPTION,\n parameter_metadata=ENCODER_METADATA[\"conv_params\"][\"num_conv_layers\"],\n )\n\n conv_layers: list[dict] = schema_utils.DictList( # TODO (Connor): Add nesting logic for conv_layers\n default=None,\n description=CONV_LAYERS_DESCRIPTION,\n parameter_metadata=ENCODER_METADATA[\"conv_params\"][\"conv_layers\"],\n )\n\n num_filters: int = NumFiltersField()\n\n filter_size: int = FilterSizeField()\n\n pool_function: str = PoolFunctionField()\n\n pool_size: int = PoolSizeField()\n\n output_size: int = schema_utils.PositiveInteger(\n default=256,\n description=\"The default output_size that will be used for each layer.\",\n parameter_metadata=ENCODER_METADATA[\"ParallelCNN\"][\"output_size\"],\n )\n\n norm: str = schema_utils.StringOptions(\n [\"batch\", \"layer\"],\n default=None,\n allow_none=True,\n description=\"The default norm that will be used for each layer.\",\n parameter_metadata=ENCODER_METADATA[\"ParallelCNN\"][\"norm\"],\n )\n\n norm_params: dict = schema_utils.Dict(\n default=None,\n description=\"Parameters used if norm is either `batch` or `layer`.\",\n parameter_metadata=ENCODER_METADATA[\"ParallelCNN\"][\"norm_params\"],\n )\n\n num_fc_layers: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Number of parallel fully connected layers to use.\",\n parameter_metadata=ENCODER_METADATA[\"ParallelCNN\"][\"num_fc_layers\"],\n )\n\n fc_layers: list[dict] = schema_utils.DictList( # TODO (Connor): Add nesting logic for fc_layers\n default=None,\n description=\"List of dictionaries containing the parameters for each fully connected layer.\",\n parameter_metadata=ENCODER_METADATA[\"ParallelCNN\"][\"fc_layers\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"stacked_cnn\", [AUDIO, SEQUENCE, TEXT, TIMESERIES])\n@ludwig_dataclass\nclass StackedCNNConfig(SequenceEncoderConfig):\n @staticmethod\n def module_name():\n return \"StackedCNN\"\n\n type: str = schema_utils.ProtectedString(\n \"stacked_cnn\",\n description=ENCODER_METADATA[\"StackedCNN\"][\"type\"].long_description,\n )\n\n dropout: float = common_fields.DropoutField(description=\"Dropout rate applied to the embedding.\")\n\n activation: str = schema_utils.ActivationOptions(\n description=\"The default activation function that will be used for each layer.\"\n )\n\n max_sequence_length: int = common_fields.MaxSequenceLengthField()\n\n representation: str = common_fields.RepresentationField()\n\n vocab: list = common_fields.VocabField()\n\n num_conv_layers: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=NUM_CONV_LAYERS_DESCRIPTION,\n parameter_metadata=ENCODER_METADATA[\"conv_params\"][\"num_conv_layers\"],\n )\n\n conv_layers: list[dict] = schema_utils.DictList( # TODO (Connor): Add nesting logic for conv_layers\n default=None,\n description=CONV_LAYERS_DESCRIPTION,\n parameter_metadata=ENCODER_METADATA[\"conv_params\"][\"conv_layers\"],\n )\n\n num_filters: int = NumFiltersField()\n\n filter_size: int = FilterSizeField()\n\n pool_function: str = PoolFunctionField()\n\n pool_size: int = PoolSizeField()\n\n strides: int = schema_utils.PositiveInteger(\n default=1,\n description=\"Stride length of the convolution.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNN\"][\"strides\"],\n )\n\n padding: str = schema_utils.StringOptions(\n [\"valid\", \"same\"],\n default=\"same\",\n description=\"Padding to use.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNN\"][\"padding\"],\n )\n\n dilation_rate: int = schema_utils.PositiveInteger(\n default=1,\n description=\"Dilation rate to use for dilated convolution.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNN\"][\"dilation_rate\"],\n )\n\n pool_strides: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Factor to scale down.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNN\"][\"pool_strides\"],\n )\n\n pool_padding: str = schema_utils.StringOptions(\n [\"valid\", \"same\"],\n default=\"same\",\n description=\"Padding to use.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNN\"][\"pool_padding\"],\n )\n\n use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use a bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNN\"][\"use_bias\"],\n )\n\n bias_initializer: str = common_fields.BiasInitializerField()\n\n weights_initializer: str = common_fields.WeightsInitializerField()\n\n should_embed: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to embed the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNN\"][\"should_embed\"],\n )\n\n embedding_size: int = common_fields.EmbeddingSizeField()\n\n embeddings_on_cpu: bool = common_fields.EmbeddingsOnCPUField()\n\n embeddings_trainable: bool = common_fields.EmbeddingsTrainableField()\n\n pretrained_embeddings: str = common_fields.PretrainedEmbeddingsField()\n\n reduce_output: str = common_fields.ReduceOutputField()\n\n output_size: int = schema_utils.PositiveInteger(\n default=256,\n description=\"The default output_size that will be used for each layer.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNN\"][\"output_size\"],\n )\n\n norm: str = schema_utils.StringOptions(\n [\"batch\", \"layer\"],\n default=None,\n allow_none=True,\n description=\"The default norm that will be used for each layer.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNN\"][\"norm\"],\n )\n\n norm_params: dict = schema_utils.Dict(\n default=None,\n description=\"Parameters used if norm is either `batch` or `layer`.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNN\"][\"norm_params\"],\n )\n\n num_fc_layers: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Number of parallel fully connected layers to use.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNN\"][\"num_fc_layers\"],\n )\n\n fc_layers: list[dict] = schema_utils.DictList( # TODO (Connor): Add nesting logic for fc_layers\n default=None,\n description=\"List of dictionaries containing the parameters for each fully connected layer.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNN\"][\"fc_layers\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"stacked_parallel_cnn\", [AUDIO, SEQUENCE, TEXT, TIMESERIES])\n@ludwig_dataclass\nclass StackedParallelCNNConfig(SequenceEncoderConfig):\n @staticmethod\n def module_name():\n return \"StackedParallelCNN\"\n\n type: str = schema_utils.ProtectedString(\n \"stacked_parallel_cnn\",\n description=ENCODER_METADATA[\"StackedParallelCNN\"][\"type\"].long_description,\n )\n\n dropout: float = common_fields.DropoutField(description=\"Dropout rate applied to the embedding.\")\n\n activation: str = schema_utils.ActivationOptions(\n description=\"The default activation function that will be used for each layer.\"\n )\n\n max_sequence_length: int = common_fields.MaxSequenceLengthField()\n\n representation: str = common_fields.RepresentationField()\n\n vocab: list = common_fields.VocabField()\n\n num_stacked_layers: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"If stacked_layers is null, this is the number of elements in the stack of parallel convolutional \"\n \"layers. \",\n parameter_metadata=ENCODER_METADATA[\"StackedParallelCNN\"][\"num_stacked_layers\"],\n )\n\n stacked_layers: list[dict] = schema_utils.DictList(\n default=None,\n description=\"a nested list of lists of dictionaries containing the parameters of the stack of parallel \"\n \"convolutional layers. The length of the list determines the number of stacked parallel \"\n \"convolutional layers, length of the sub-lists determines the number of parallel conv layers and \"\n \"the content of each dictionary determines the parameters for a specific layer. \",\n parameter_metadata=ENCODER_METADATA[\"StackedParallelCNN\"][\"stacked_layers\"],\n )\n\n num_filters: int = NumFiltersField()\n\n filter_size: int = FilterSizeField()\n\n pool_function: str = PoolFunctionField()\n\n pool_size: int = PoolSizeField()\n\n use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use a bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"StackedParallelCNN\"][\"use_bias\"],\n )\n\n bias_initializer: str = common_fields.BiasInitializerField()\n\n weights_initializer: str = common_fields.WeightsInitializerField()\n\n should_embed: bool = schema_utils.Boolean(\n default=True,\n description=\"If True the input sequence is expected to be made of integers and will be mapped into embeddings\",\n parameter_metadata=ENCODER_METADATA[\"StackedParallelCNN\"][\"should_embed\"],\n )\n\n embedding_size: int = common_fields.EmbeddingSizeField()\n\n embeddings_on_cpu: bool = common_fields.EmbeddingsOnCPUField()\n\n embeddings_trainable: bool = common_fields.EmbeddingsTrainableField()\n\n pretrained_embeddings: str = common_fields.PretrainedEmbeddingsField()\n\n reduce_output: str = common_fields.ReduceOutputField()\n\n output_size: int = schema_utils.PositiveInteger(\n default=256,\n description=\"The default output_size that will be used for each layer.\",\n parameter_metadata=ENCODER_METADATA[\"StackedParallelCNN\"][\"output_size\"],\n )\n\n norm: str = schema_utils.StringOptions(\n [\"batch\", \"layer\"],\n default=None,\n allow_none=True,\n description=\"The default norm that will be used for each layer.\",\n parameter_metadata=ENCODER_METADATA[\"StackedParallelCNN\"][\"norm\"],\n )\n\n norm_params: dict = schema_utils.Dict(\n default=None,\n description=\"Parameters used if norm is either `batch` or `layer`.\",\n parameter_metadata=ENCODER_METADATA[\"StackedParallelCNN\"][\"norm_params\"],\n )\n\n num_fc_layers: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Number of parallel fully connected layers to use.\",\n parameter_metadata=ENCODER_METADATA[\"StackedParallelCNN\"][\"num_fc_layers\"],\n )\n\n fc_layers: list[dict] = schema_utils.DictList( # TODO (Connor): Add nesting logic for fc_layers\n default=None,\n description=\"List of dictionaries containing the parameters for each fully connected layer.\",\n parameter_metadata=ENCODER_METADATA[\"StackedParallelCNN\"][\"fc_layers\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"rnn\", [AUDIO, SEQUENCE, TEXT, TIMESERIES])\n@ludwig_dataclass\nclass StackedRNNConfig(SequenceEncoderConfig):\n @staticmethod\n def module_name():\n return \"StackedRNN\"\n\n type: str = schema_utils.ProtectedString(\n \"rnn\",\n description=ENCODER_METADATA[\"StackedRNN\"][\"type\"].long_description,\n )\n\n dropout: float = common_fields.DropoutField(description=\"Dropout rate.\")\n\n recurrent_dropout: float = schema_utils.FloatRange(\n default=0.0,\n min=0,\n max=1,\n description=\"The dropout rate for the recurrent state\",\n parameter_metadata=ENCODER_METADATA[\"StackedRNN\"][\"recurrent_dropout\"],\n )\n\n activation: str = schema_utils.ActivationOptions(default=\"tanh\", description=\"The default activation function.\")\n\n recurrent_activation: str = schema_utils.ActivationOptions(\n default=\"sigmoid\",\n description=\"The activation function to use in the recurrent step\",\n parameter_metadata=ENCODER_METADATA[\"StackedRNN\"][\"recurrent_activation\"],\n )\n\n max_sequence_length: int = common_fields.MaxSequenceLengthField()\n\n representation: str = common_fields.RepresentationField()\n\n vocab: list = common_fields.VocabField()\n\n cell_type: str = schema_utils.StringOptions(\n [\"rnn\", \"lstm\", \"gru\"],\n default=\"rnn\",\n description=\"The type of recurrent cell to use. Available values are: `rnn`, `lstm`, `gru`. For reference \"\n \"about the differences between the cells please refer to \"\n \"[torch.nn Recurrent Layers](https://pytorch.org/docs/stable/nn.html#recurrent-layers).\",\n parameter_metadata=ENCODER_METADATA[\"StackedRNN\"][\"cell_type\"],\n )\n\n num_layers: int = schema_utils.PositiveInteger(\n default=1,\n description=\"The number of stacked recurrent layers.\",\n parameter_metadata=ENCODER_METADATA[\"StackedRNN\"][\"num_layers\"],\n )\n\n state_size: int = schema_utils.PositiveInteger(\n default=256,\n description=\"The size of the state of the rnn.\",\n parameter_metadata=ENCODER_METADATA[\"StackedRNN\"][\"state_size\"],\n )\n\n bidirectional: bool = schema_utils.Boolean(\n default=False,\n description=\"If true, two recurrent networks will perform encoding in the forward and backward direction and \"\n \"their outputs will be concatenated.\",\n parameter_metadata=ENCODER_METADATA[\"StackedRNN\"][\"bidirectional\"],\n )\n\n unit_forget_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"If true, add 1 to the bias of the forget gate at initialization\",\n parameter_metadata=ENCODER_METADATA[\"StackedRNN\"][\"unit_forget_bias\"],\n )\n\n recurrent_initializer: str = schema_utils.InitializerOptions(\n default=\"orthogonal\",\n description=\"The initializer for recurrent matrix weights\",\n parameter_metadata=ENCODER_METADATA[\"StackedRNN\"][\"recurrent_initializer\"],\n )\n\n use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use a bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"StackedRNN\"][\"use_bias\"],\n )\n\n bias_initializer: str = common_fields.BiasInitializerField()\n\n weights_initializer: str = common_fields.WeightsInitializerField()\n\n should_embed: bool = schema_utils.Boolean(\n default=True,\n description=\"If True the input sequence is expected to be made of integers and will be mapped into embeddings\",\n parameter_metadata=ENCODER_METADATA[\"StackedRNN\"][\"should_embed\"],\n )\n\n embedding_size: int = common_fields.EmbeddingSizeField()\n\n embeddings_on_cpu: bool = common_fields.EmbeddingsOnCPUField()\n\n embeddings_trainable: bool = common_fields.EmbeddingsTrainableField()\n\n pretrained_embeddings: str = common_fields.PretrainedEmbeddingsField()\n\n reduce_output: str = common_fields.ReduceOutputField(default=\"last\")\n\n output_size: int = schema_utils.PositiveInteger(\n default=256,\n description=\"The default output_size that will be used for each layer.\",\n parameter_metadata=ENCODER_METADATA[\"StackedRNN\"][\"output_size\"],\n )\n\n norm: str = common_fields.NormField(description=\"The default norm that will be used for each layer.\")\n\n norm_params: dict = common_fields.NormParamsField()\n\n num_fc_layers: int = common_fields.NumFCLayersField(description=\"Number of parallel fully connected layers to use.\")\n\n fc_activation: str = schema_utils.ActivationOptions()\n\n fc_dropout: float = common_fields.DropoutField()\n\n fc_layers: list[dict] = common_fields.FCLayersField()\n\n\n@DeveloperAPI\n@register_encoder_config(\"cnnrnn\", [AUDIO, SEQUENCE, TEXT, TIMESERIES])\n@ludwig_dataclass\nclass StackedCNNRNNConfig(SequenceEncoderConfig):\n @staticmethod\n def module_name():\n return \"StackedCNNRNN\"\n\n type: str = schema_utils.ProtectedString(\n \"cnnrnn\",\n description=ENCODER_METADATA[\"StackedCNNRNN\"][\"type\"].long_description,\n )\n\n dropout: float = common_fields.DropoutField(description=\"Dropout rate.\")\n\n recurrent_dropout: float = schema_utils.FloatRange(\n default=0.0,\n min=0,\n max=1,\n description=\"The dropout rate for the recurrent state\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNNRNN\"][\"recurrent_dropout\"],\n )\n\n conv_dropout: float = schema_utils.FloatRange(\n default=0.0,\n min=0,\n max=1,\n description=\"The dropout rate for the convolutional layers\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNNRNN\"][\"conv_dropout\"],\n )\n\n activation: str = schema_utils.ActivationOptions(\n default=\"tanh\", description=\"The default activation function to use.\"\n )\n\n recurrent_activation: str = schema_utils.ActivationOptions(\n default=\"sigmoid\",\n description=\"The activation function to use in the recurrent step\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNNRNN\"][\"recurrent_activation\"],\n )\n\n conv_activation: str = schema_utils.ActivationOptions(\n description=\"The default activation function that will be used for each convolutional layer.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNNRNN\"][\"conv_activation\"],\n )\n\n max_sequence_length: int = common_fields.MaxSequenceLengthField()\n\n representation: str = common_fields.RepresentationField()\n\n vocab: list = common_fields.VocabField()\n\n cell_type: str = schema_utils.StringOptions(\n [\"rnn\", \"lstm\", \"gru\"],\n default=\"rnn\",\n description=\"The type of recurrent cell to use. Available values are: `rnn`, `lstm`, `gru`. For reference \"\n \"about the differences between the cells please refer to \"\n \"[torch.nn Recurrent Layers](https://pytorch.org/docs/stable/nn.html#recurrent-layers).\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNNRNN\"][\"cell_type\"],\n )\n\n num_conv_layers: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=NUM_CONV_LAYERS_DESCRIPTION,\n parameter_metadata=ENCODER_METADATA[\"conv_params\"][\"num_conv_layers\"],\n )\n\n conv_layers: list[dict] = schema_utils.DictList( # TODO (Connor): Add nesting logic for conv_layers\n default=None,\n description=CONV_LAYERS_DESCRIPTION,\n parameter_metadata=ENCODER_METADATA[\"conv_params\"][\"conv_layers\"],\n )\n\n num_filters: int = NumFiltersField()\n\n filter_size: int = FilterSizeField(default=5)\n\n pool_function: str = PoolFunctionField()\n\n pool_size: int = PoolSizeField(default=2)\n\n strides: int = schema_utils.PositiveInteger(\n default=1,\n description=\"Stride length of the convolution.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNNRNN\"][\"strides\"],\n )\n\n padding: str = schema_utils.StringOptions(\n [\"valid\", \"same\"],\n default=\"same\",\n description=\"Padding to use.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNNRNN\"][\"padding\"],\n )\n\n dilation_rate: int = schema_utils.PositiveInteger(\n default=1,\n description=\"Dilation rate to use for dilated convolution.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNNRNN\"][\"dilation_rate\"],\n )\n\n pool_strides: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Factor to scale down.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNNRNN\"][\"pool_strides\"],\n )\n\n pool_padding: str = schema_utils.StringOptions(\n [\"valid\", \"same\"],\n default=\"same\",\n description=\"Padding to use.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNNRNN\"][\"pool_padding\"],\n )\n\n num_rec_layers: int = schema_utils.PositiveInteger(\n default=1,\n description=\"The number of stacked recurrent layers.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNNRNN\"][\"num_rec_layers\"],\n )\n\n state_size: int = schema_utils.PositiveInteger(\n default=256,\n description=\"The size of the state of the rnn.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNNRNN\"][\"state_size\"],\n )\n\n bidirectional: bool = schema_utils.Boolean(\n default=False,\n description=\"If true, two recurrent networks will perform encoding in the forward and backward direction and \"\n \"their outputs will be concatenated.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNNRNN\"][\"bidirectional\"],\n )\n\n unit_forget_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"If true, add 1 to the bias of the forget gate at initialization\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNNRNN\"][\"unit_forget_bias\"],\n )\n\n recurrent_initializer: str = schema_utils.InitializerOptions(\n default=\"orthogonal\",\n description=\"The initializer for recurrent matrix weights\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNNRNN\"][\"recurrent_initializer\"],\n )\n\n use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use a bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNNRNN\"][\"use_bias\"],\n )\n\n bias_initializer: str = common_fields.BiasInitializerField()\n\n weights_initializer: str = common_fields.WeightsInitializerField()\n\n should_embed: bool = schema_utils.Boolean(\n default=True,\n description=\"If True the input sequence is expected to be made of integers and will be mapped into embeddings\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNNRNN\"][\"should_embed\"],\n )\n\n embedding_size: int = common_fields.EmbeddingSizeField()\n\n embeddings_on_cpu: bool = common_fields.EmbeddingsOnCPUField()\n\n embeddings_trainable: bool = common_fields.EmbeddingsTrainableField()\n\n pretrained_embeddings: str = common_fields.PretrainedEmbeddingsField()\n\n reduce_output: str = common_fields.ReduceOutputField(default=\"last\")\n\n output_size: int = schema_utils.PositiveInteger(\n default=256,\n description=\"The default output_size that will be used for each layer.\",\n parameter_metadata=ENCODER_METADATA[\"StackedCNNRNN\"][\"output_size\"],\n )\n\n norm: str = common_fields.NormField(description=\"The default norm that will be used for each layer.\")\n\n norm_params: dict = common_fields.NormParamsField()\n\n num_fc_layers: int = common_fields.NumFCLayersField(description=\"Number of parallel fully connected layers to use.\")\n\n fc_activation: str = schema_utils.ActivationOptions()\n\n fc_dropout: float = common_fields.DropoutField()\n\n fc_layers: list[dict] = common_fields.FCLayersField()\n\n\n@DeveloperAPI\n@register_encoder_config(\"transformer\", [SEQUENCE, TEXT, TIMESERIES])\n@ludwig_dataclass\nclass StackedTransformerConfig(SequenceEncoderConfig):\n @staticmethod\n def module_name():\n return \"StackedTransformer\"\n\n type: str = schema_utils.ProtectedString(\n \"transformer\",\n description=ENCODER_METADATA[\"StackedTransformer\"][\"type\"].long_description,\n )\n\n dropout: float = common_fields.DropoutField(default=0.1, description=\"The dropout rate for the transformer block.\")\n\n max_sequence_length: int = common_fields.MaxSequenceLengthField()\n\n representation: str = common_fields.RepresentationField()\n\n vocab: list = common_fields.VocabField()\n\n num_layers: int = schema_utils.PositiveInteger(\n default=1,\n description=\"The number of transformer layers.\",\n parameter_metadata=ENCODER_METADATA[\"StackedTransformer\"][\"num_layers\"],\n )\n\n hidden_size: int = schema_utils.PositiveInteger(\n default=256,\n description=\"The size of the hidden representation within the transformer block. It is usually the same as \"\n \"the embedding_size, but if the two values are different, a projection layer will be added before \"\n \"the first transformer block.\",\n parameter_metadata=ENCODER_METADATA[\"StackedTransformer\"][\"hidden_size\"],\n )\n\n num_heads: int = schema_utils.PositiveInteger(\n default=8,\n description=\"Number of attention heads in each transformer block.\",\n parameter_metadata=ENCODER_METADATA[\"StackedTransformer\"][\"num_heads\"],\n )\n\n transformer_output_size: int = schema_utils.PositiveInteger(\n default=256,\n description=\"Size of the fully connected layer after self attention in the transformer block. This is usually \"\n \"the same as hidden_size and embedding_size.\",\n parameter_metadata=ENCODER_METADATA[\"StackedTransformer\"][\"transformer_output_size\"],\n )\n\n use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use a bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"StackedTransformer\"][\"use_bias\"],\n )\n\n bias_initializer: str = common_fields.BiasInitializerField()\n\n weights_initializer: str = common_fields.WeightsInitializerField()\n\n should_embed: bool = schema_utils.Boolean(\n default=True,\n description=\"If True the input sequence is expected to be made of integers and will be mapped into embeddings\",\n parameter_metadata=ENCODER_METADATA[\"StackedTransformer\"][\"should_embed\"],\n )\n\n embedding_size: int = common_fields.EmbeddingSizeField()\n\n embeddings_on_cpu: bool = common_fields.EmbeddingsOnCPUField()\n\n embeddings_trainable: bool = common_fields.EmbeddingsTrainableField()\n\n pretrained_embeddings: str = common_fields.PretrainedEmbeddingsField()\n\n reduce_output: str = common_fields.ReduceOutputField(default=\"last\")\n\n output_size: int = schema_utils.PositiveInteger(\n default=256,\n description=\"The default output_size that will be used for each layer.\",\n parameter_metadata=ENCODER_METADATA[\"StackedTransformer\"][\"output_size\"],\n )\n\n norm: str = common_fields.NormField(description=\"The default norm that will be used for each layer.\")\n\n norm_params: dict = common_fields.NormParamsField()\n\n num_fc_layers: int = common_fields.NumFCLayersField(description=\"Number of parallel fully connected layers to use.\")\n\n fc_activation: str = schema_utils.ActivationOptions()\n\n fc_dropout: float = common_fields.DropoutField()\n\n fc_layers: list[dict] = common_fields.FCLayersField()\n"
},
{
"path": "ludwig/schema/encoders/set_encoders.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import SET\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import register_encoder_config\nfrom ludwig.schema.metadata import ENCODER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@register_encoder_config(\"embed\", SET)\n@ludwig_dataclass\nclass SetSparseEncoderConfig(BaseEncoderConfig):\n @staticmethod\n def module_name():\n return \"SetSparseEncoder\"\n\n type: str = schema_utils.ProtectedString(\n \"embed\",\n description=ENCODER_METADATA[\"SetSparseEncoder\"][\"type\"].long_description,\n )\n\n dropout: float = schema_utils.FloatRange(\n default=0.0,\n min=0,\n max=1,\n description=\"Dropout probability for the embedding.\",\n parameter_metadata=ENCODER_METADATA[\"SetSparseEncoder\"][\"dropout\"],\n )\n\n activation: str = schema_utils.ActivationOptions(\n description=\"The default activation function that will be used for each layer.\",\n parameter_metadata=ENCODER_METADATA[\"SetSparseEncoder\"][\"activation\"],\n )\n\n representation: str = schema_utils.StringOptions(\n [\"dense\", \"sparse\"],\n default=\"dense\",\n description=\"The representation of the embedding. Either dense or sparse.\",\n parameter_metadata=ENCODER_METADATA[\"SetSparseEncoder\"][\"representation\"],\n )\n\n vocab: list[str] = schema_utils.List(\n default=None,\n description=\"Vocabulary of the encoder\",\n parameter_metadata=ENCODER_METADATA[\"SetSparseEncoder\"][\"vocab\"],\n )\n\n use_bias: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether the layer uses a bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"SetSparseEncoder\"][\"use_bias\"],\n )\n\n bias_initializer: str = schema_utils.InitializerOptions(\n default=\"zeros\",\n description=\"Initializer to use for the bias vector.\",\n parameter_metadata=ENCODER_METADATA[\"SetSparseEncoder\"][\"bias_initializer\"],\n )\n\n weights_initializer: str = schema_utils.InitializerOptions(\n description=\"Initializer to use for the weights matrix.\",\n parameter_metadata=ENCODER_METADATA[\"SetSparseEncoder\"][\"weights_initializer\"],\n )\n\n embedding_size: int = schema_utils.PositiveInteger(\n default=50,\n description=\"The maximum embedding size, the actual size will be min(vocabulary_size, embedding_size) for \"\n \"dense representations and exactly vocabulary_size for the sparse encoding, where vocabulary_size \"\n \"is the number of different strings appearing in the training set in the input column (plus 1 for \"\n \"the unknown token placeholder ).\",\n parameter_metadata=ENCODER_METADATA[\"SetSparseEncoder\"][\"embedding_size\"],\n )\n\n embeddings_on_cpu: bool = schema_utils.Boolean(\n default=False,\n description=\"By default embedding matrices are stored on GPU memory if a GPU is used, as it allows for faster \"\n \"access, but in some cases the embedding matrix may be too large. This parameter forces the \"\n \"placement of the embedding matrix in regular memory and the CPU is used for embedding lookup, \"\n \"slightly slowing down the process as a result of data transfer between CPU and GPU memory.\",\n parameter_metadata=ENCODER_METADATA[\"SetSparseEncoder\"][\"embeddings_on_cpu\"],\n )\n\n embeddings_trainable: bool = schema_utils.Boolean(\n default=True,\n description=\"If true embeddings are trained during the training process, if false embeddings are fixed. It \"\n \"may be useful when loading pretrained embeddings for avoiding finetuning them. This parameter \"\n \"has effect only when representation is dense as sparse one-hot encodings are not trainable.\",\n parameter_metadata=ENCODER_METADATA[\"SetSparseEncoder\"][\"embeddings_trainable\"],\n )\n\n pretrained_embeddings: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"By default dense embeddings are initialized randomly, but this parameter allows to specify a \"\n \"path to a file containing embeddings in the GloVe format. When the file containing the \"\n \"embeddings is loaded, only the embeddings with labels present in the vocabulary are kept, \"\n \"the others are discarded. If the vocabulary contains strings that have no match in the \"\n \"embeddings file, their embeddings are initialized with the average of all other embedding plus \"\n \"some random noise to make them different from each other. This parameter has effect only if \"\n \"representation is dense.\",\n parameter_metadata=ENCODER_METADATA[\"SetSparseEncoder\"][\"pretrained_embeddings\"],\n )\n\n output_size: int = schema_utils.PositiveInteger(\n default=10,\n description=\"If output_size is not already specified in fc_layers this is the default output_size that will \"\n \"be used for each layer. It indicates the size of the output of a fully connected layer.\",\n parameter_metadata=ENCODER_METADATA[\"SetSparseEncoder\"][\"output_size\"],\n )\n\n norm: str = schema_utils.StringOptions(\n [\"batch\", \"layer\"],\n default=None,\n allow_none=True,\n description=\"The default norm that will be used for each layer.\",\n parameter_metadata=ENCODER_METADATA[\"SetSparseEncoder\"][\"norm\"],\n )\n\n norm_params: dict = schema_utils.Dict(\n default=None,\n description=\"Parameters used if norm is either `batch` or `layer`.\",\n parameter_metadata=ENCODER_METADATA[\"SetSparseEncoder\"][\"norm_params\"],\n )\n\n num_fc_layers: int = schema_utils.NonNegativeInteger(\n default=0,\n description=\"This is the number of stacked fully connected layers that the input to the feature passes \"\n \"through. Their output is projected in the feature's output space.\",\n parameter_metadata=ENCODER_METADATA[\"SetSparseEncoder\"][\"num_fc_layers\"],\n )\n\n fc_layers: list[dict] = schema_utils.DictList( # TODO (Connor): Add nesting logic for fc_layers\n default=None,\n description=\"List of dictionaries containing the parameters for each fully connected layer.\",\n parameter_metadata=ENCODER_METADATA[\"SetSparseEncoder\"][\"fc_layers\"],\n )\n"
},
{
"path": "ludwig/schema/encoders/text/__init__.py",
"content": ""
},
{
"path": "ludwig/schema/encoders/text/encoders.py",
"content": "from collections.abc import Callable\nfrom typing import TYPE_CHECKING\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import MODEL_ECD, TEXT\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.encoders.sequence_encoders import SequenceEncoderConfig\nfrom ludwig.schema.encoders.text.hf_model_params import DebertaModelParams\nfrom ludwig.schema.encoders.utils import register_encoder_config\nfrom ludwig.schema.llms.base_model import BaseModelDataclassField\nfrom ludwig.schema.llms.model_parameters import ModelParametersConfig, ModelParametersConfigField\nfrom ludwig.schema.llms.peft import AdapterDataclassField, BaseAdapterConfig\nfrom ludwig.schema.llms.quantization import QuantizationConfig, QuantizationConfigField\nfrom ludwig.schema.metadata import ENCODER_METADATA\nfrom ludwig.schema.metadata.parameter_metadata import INTERNAL_ONLY, ParameterMetadata\nfrom ludwig.schema.utils import ludwig_dataclass\n\nif TYPE_CHECKING:\n from ludwig.schema.features.preprocessing.text import TextPreprocessingConfig\n\n\nclass HFEncoderConfig(SequenceEncoderConfig):\n trainable: bool\n use_pretrained: bool\n pretrained_model_name_or_path: str\n reduce_output: str\n\n def set_fixed_preprocessing_params(self, model_type: str, preprocessing: \"TextPreprocessingConfig\"):\n model_name = self.pretrained_model_name_or_path\n if model_name is None and self.use_pretrained:\n # no default model name, so model name is required by the subclass\n raise ValueError(\n f\"Missing required parameter for `{self.type}` encoder: `pretrained_model_name_or_path` when \"\n \"`use_pretrained` is True.\"\n )\n preprocessing.tokenizer = \"hf_tokenizer\"\n preprocessing.pretrained_model_name_or_path = model_name\n if not self.can_cache_embeddings():\n preprocessing.cache_encoder_embeddings = False\n\n def is_pretrained(self) -> bool:\n return self.use_pretrained\n\n def can_cache_embeddings(self) -> bool:\n \"\"\"Returns true if the encoder's output embeddings will not change during training.\"\"\"\n return not self.trainable and self.reduce_output != \"attention\"\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass HFEncoderImplConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the base HF encoder implmenetation.\"\"\"\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"HFEncoder\"][\"use_pretrained\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"HFEncoder\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n )\n\n # Internal params set based on preprocessing metadata\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"\",\n parameter_metadata=INTERNAL_ONLY,\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=None,\n description=\"\",\n parameter_metadata=INTERNAL_ONLY,\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=(\n \"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\"\n ),\n parameter_metadata=INTERNAL_ONLY,\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"albert\", TEXT)\n@ludwig_dataclass\nclass ALBERTConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an ALBERT encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"ALBERT\"\n\n type: str = schema_utils.ProtectedString(\n \"albert\",\n description=ENCODER_METADATA[\"ALBERT\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"albert-base-v2\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"saved_weights_in_checkpoint\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n reduce_output: str = schema_utils.String(\n default=\"cls_pooled\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"reduce_output\"],\n )\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=30000,\n description=\"Vocabulary size of the ALBERT model. Defines the number of different tokens that can be \"\n \"represented by the inputs_ids passed.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"vocab_size\"],\n )\n\n embedding_size: int = schema_utils.PositiveInteger(\n default=128,\n description=\"Dimensionality of vocabulary embeddings.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"embedding_size\"],\n )\n\n hidden_size: int = schema_utils.PositiveInteger(\n default=768,\n description=\"Dimensionality of the encoder layers and the pooler layer.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"hidden_size\"],\n )\n\n num_hidden_layers: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"num_hidden_layers\"],\n )\n\n num_hidden_groups: int = schema_utils.PositiveInteger(\n default=1,\n description=\"Number of groups for the hidden layers, parameters in the same group are shared.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"num_hidden_groups\"],\n )\n\n num_attention_heads: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"num_attention_heads\"],\n )\n\n intermediate_size: int = schema_utils.PositiveInteger(\n default=3072,\n description=\"The dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer \"\n \"encoder.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"intermediate_size\"],\n )\n\n inner_group_num: int = schema_utils.PositiveInteger(\n default=1,\n description=\"The number of inner repetition of attention and ffn.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"inner_group_num\"],\n )\n\n hidden_act: str = schema_utils.StringOptions(\n [\"gelu\", \"relu\", \"silu\", \"gelu_new\"],\n default=\"gelu_new\",\n description=\"The non-linear activation function (function or string) in the encoder and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"hidden_act\"],\n )\n\n hidden_dropout_prob: float = schema_utils.FloatRange(\n default=0.0,\n min=0,\n max=1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"hidden_dropout_prob\"],\n )\n\n attention_probs_dropout_prob: float = schema_utils.FloatRange(\n default=0.0,\n min=0,\n max=1,\n description=\"The dropout ratio for the attention probabilities.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"attention_probs_dropout_prob\"],\n )\n\n max_position_embeddings: int = schema_utils.PositiveInteger(\n default=512,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"max_position_embeddings\"],\n )\n\n type_vocab_size: int = schema_utils.PositiveInteger(\n default=2,\n description=\"The vocabulary size of the token_type_ids passed when calling AlbertModel or TFAlbertModel.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"type_vocab_size\"],\n )\n\n initializer_range: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"initializer_range\"],\n )\n\n layer_norm_eps: float = schema_utils.NonNegativeFloat(\n default=1e-12,\n description=\"The epsilon used by the layer normalization layers.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"layer_norm_eps\"],\n )\n\n classifier_dropout_prob: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout ratio for attached classifiers.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"classifier_dropout_prob\"],\n )\n\n position_embedding_type: str = schema_utils.StringOptions(\n [\"absolute\", \"relative_key\", \"relative_key_query\"],\n default=\"absolute\",\n description=\"\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"position_embedding_type\"],\n )\n\n pad_token_id: int = schema_utils.Integer(\n default=0,\n description=\"The ID of the token to use as padding.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"pad_token_id\"],\n )\n\n bos_token_id: int = schema_utils.Integer(\n default=2,\n description=\"The beginning of sequence token ID.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"bos_token_id\"],\n )\n\n eos_token_id: int = schema_utils.Integer(\n default=3,\n description=\"The end of sequence token ID.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"eos_token_id\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"pretrained_kwargs\"],\n )\n\n\n# TODO: uncomment when sentencepiece doesn't cause segfaults: https://github.com/ludwig-ai/ludwig/issues/2983\n@DeveloperAPI\n# @register_encoder_config(\"mt5\", TEXT)\n@ludwig_dataclass\nclass MT5Config(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an MT5 encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"MT5\"\n\n type: str = schema_utils.ProtectedString(\n \"mt5\",\n description=ENCODER_METADATA[\"MT5\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"google/mt5-base\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"saved_weights_in_checkpoint\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"reduce_output\"],\n )\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=250112,\n description=\"Vocabulary size of the T5 model. Defines the number of different tokens that can be represented \"\n \"by the inputs_ids passed when calling T5Model or TFT5Model.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"vocab_size\"],\n )\n\n d_model: int = schema_utils.PositiveInteger(\n default=512,\n description=\"Size of the encoder layers and the pooler layer.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"d_model\"],\n )\n\n d_kv: int = schema_utils.PositiveInteger(\n default=64,\n description=\"Size of the key, query, value projections per attention head. d_kv has to be equal to d_model // \"\n \"num_heads.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"d_kv\"],\n )\n\n d_ff: int = schema_utils.PositiveInteger(\n default=1024,\n description=\"Size of the intermediate feed forward layer in each T5Block.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"d_ff\"],\n )\n\n num_layers: int = schema_utils.PositiveInteger(\n default=8,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"num_layers\"],\n )\n\n num_decoder_layers: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Number of hidden layers in the Transformer decoder. Will use the same value as num_layers if not \"\n \"set.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"num_decoder_layers\"],\n )\n\n num_heads: int = schema_utils.PositiveInteger(\n default=6,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"num_heads\"],\n )\n\n relative_attention_num_buckets: int = schema_utils.PositiveInteger(\n default=32,\n description=\"The number of buckets to use for each attention layer.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"relative_attention_num_buckets\"],\n )\n\n dropout_rate: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The ratio for all dropout layers.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"dropout_rate\"],\n )\n\n layer_norm_epsilon: float = schema_utils.NonNegativeFloat(\n default=1e-06,\n description=\"The epsilon used by the layer normalization layers.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"layer_norm_epsilon\"],\n )\n\n initializer_factor: float = schema_utils.NonNegativeFloat(\n default=1.0,\n description=\"A factor for initializing all weight matrices (should be kept to 1, used internally for \"\n \"initialization testing)\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"initializer_factor\"],\n )\n\n feed_forward_proj: str = schema_utils.StringOptions(\n [\"relu\", \"gated-gelu\"],\n default=\"gated-gelu\",\n description=\"Type of feed forward layer to be used. \",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"feed_forward_proj\"],\n )\n\n is_encoder_decoder: bool = schema_utils.Boolean(\n default=True,\n description=\"\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"is_encoder_decoder\"],\n )\n\n use_cache: bool = schema_utils.Boolean(\n default=True,\n description=\"\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"use_cache\"],\n )\n\n tokenizer_class: str = schema_utils.String(\n default=\"T5Tokenizer\",\n description=\"\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"tokenizer_class\"],\n )\n\n tie_word_embeddings: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether the model's input and output word embeddings should be tied.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"tie_word_embeddings\"],\n )\n\n pad_token_id: int = schema_utils.Integer(\n default=0,\n description=\"The ID of the token to use as padding.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"pad_token_id\"],\n )\n\n eos_token_id: int = schema_utils.Integer(\n default=1,\n description=\"The end of sequence token ID.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"eos_token_id\"],\n )\n\n decoder_start_token_id: int = schema_utils.Integer(\n default=0,\n description=\"If an encoder-decoder model starts decoding with a different token than _bos_, the id of that \"\n \"token.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"decoder_start_token_id\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"xlmroberta\", TEXT)\n@ludwig_dataclass\nclass XLMRoBERTaConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an XLMRoBERTa encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"XLMRoBERTa\"\n\n type: str = schema_utils.ProtectedString(\n \"xlmroberta\",\n description=ENCODER_METADATA[\"XLMRoBERTa\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"xlm-roberta-base\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"saved_weights_in_checkpoint\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"cls_pooled\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Vocabulary size of the XLMRoBERTa model.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"vocab_size\"],\n )\n\n pad_token_id: int = schema_utils.Integer(\n default=1,\n description=\"The ID of the token to use as padding.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"pad_token_id\"],\n )\n\n bos_token_id: int = schema_utils.Integer(\n default=0,\n description=\"The beginning of sequence token ID.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"bos_token_id\"],\n )\n\n eos_token_id: int = schema_utils.Integer(\n default=2,\n description=\"The end of sequence token ID.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"eos_token_id\"],\n )\n\n max_position_embeddings: int = schema_utils.PositiveInteger(\n default=514,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large just in case (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"max_position_embeddings\"],\n )\n\n type_vocab_size: int = schema_utils.PositiveInteger(\n default=1,\n description=\"The vocabulary size of the token_type_ids passed in.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"type_vocab_size\"],\n )\n\n add_pooling_layer: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to add a pooling layer to the encoder.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"add_pooling_layer\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"bert\", TEXT)\n@ludwig_dataclass\nclass BERTConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an BERT encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"BERT\"\n\n type: str = schema_utils.ProtectedString(\n \"bert\",\n description=ENCODER_METADATA[\"BERT\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"bert-base-uncased\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"saved_weights_in_checkpoint\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n reduce_output: str = schema_utils.String(\n default=\"cls_pooled\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"reduce_output\"],\n )\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=30522,\n description=\"Vocabulary size of the BERT model. Defines the number of different tokens that can be \"\n \"represented by the inputs_ids passed when calling BertModel or TFBertModel.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"vocab_size\"],\n )\n\n hidden_size: int = schema_utils.PositiveInteger(\n default=768,\n description=\"Dimensionality of the encoder layers and the pooler layer.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"hidden_size\"],\n )\n\n num_hidden_layers: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"num_hidden_layers\"],\n )\n\n num_attention_heads: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"num_attention_heads\"],\n )\n\n intermediate_size: int = schema_utils.PositiveInteger(\n default=3072,\n description=\"Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"intermediate_size\"],\n )\n\n hidden_act: str | Callable = schema_utils.StringOptions( # TODO: add support for callable\n [\"gelu\", \"relu\", \"silu\", \"gelu_new\"],\n default=\"gelu\",\n description=\"The non-linear activation function (function or string) in the encoder and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"hidden_act\"],\n )\n\n hidden_dropout_prob: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"hidden_dropout_prob\"],\n )\n\n attention_probs_dropout_prob: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout ratio for the attention probabilities.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"attention_probs_dropout_prob\"],\n )\n\n max_position_embeddings: int = schema_utils.PositiveInteger(\n default=512,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large just in case (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"max_position_embeddings\"],\n )\n\n type_vocab_size: int = schema_utils.PositiveInteger(\n default=2,\n description=\"The vocabulary size of the token_type_ids passed when calling BertModel or TFBertModel.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"type_vocab_size\"],\n )\n\n initializer_range: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"initializer_range\"],\n )\n\n layer_norm_eps: float = schema_utils.NonNegativeFloat(\n default=1e-12,\n description=\"The epsilon used by the layer normalization layers.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"layer_norm_eps\"],\n )\n\n pad_token_id: int = schema_utils.Integer(\n default=0,\n description=\"The ID of the token to use as padding.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"pad_token_id\"],\n )\n\n gradient_checkpointing: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to use gradient checkpointing.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"gradient_checkpointing\"],\n )\n\n position_embedding_type: str = schema_utils.StringOptions(\n [\"absolute\", \"relative_key\", \"relative_key_query\"],\n default=\"absolute\",\n description=\"Type of position embedding.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"position_embedding_type\"],\n )\n\n classifier_dropout: float = schema_utils.FloatRange(\n default=None,\n allow_none=True,\n min=0,\n max=1,\n description=\"The dropout ratio for the classification head.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"classifier_dropout\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"deberta\", TEXT)\n@ludwig_dataclass\nclass DebertaV2Config(HFEncoderImplConfig, DebertaModelParams):\n \"\"\"This dataclass configures the schema used for a DeBERTa-v2 / v3 encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"DeBERTa\"\n\n type: str = schema_utils.ProtectedString(\n \"deberta\",\n description=ENCODER_METADATA[\"DeBERTa\"][\"type\"].long_description,\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"sileod/deberta-v3-base-tasksource-nli\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"DeBERTa\"][\"pretrained_model_name_or_path\"],\n )\n\n reduce_output: str = schema_utils.StringOptions(\n [\"cls_pooled\", \"last\", \"sum\", \"mean\", \"max\", \"concat\", \"attention\"],\n default=\"sum\",\n allow_none=True,\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n )\n\n\n# TODO: uncomment once we figure out host memory issue: https://github.com/ludwig-ai/ludwig/issues/3107\n@DeveloperAPI\n# @register_encoder_config(\"xlm\", TEXT)\n@ludwig_dataclass\nclass XLMConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an XLM encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"XLM\"\n\n type: str = schema_utils.ProtectedString(\n \"xlm\",\n description=ENCODER_METADATA[\"XLM\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"xlm-mlm-en-2048\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"saved_weights_in_checkpoint\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"reduce_output\"],\n )\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=30145,\n description=\"Vocabulary size of the BERT model. Defines the number of different tokens that can be \"\n \"represented by the inputs_ids passed when calling XLMModel or TFXLMModel.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"vocab_size\"],\n )\n\n emb_dim: int = schema_utils.PositiveInteger(\n default=2048,\n description=\"Dimensionality of the encoder layers and the pooler layer.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"emb_dim\"],\n )\n\n n_layers: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"n_layers\"],\n )\n\n n_heads: int = schema_utils.PositiveInteger(\n default=16,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"n_heads\"],\n )\n\n dropout: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"dropout\"],\n )\n\n attention_dropout: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout probability for the attention mechanism.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"attention_dropout\"],\n )\n\n gelu_activation: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether or not to use gelu for the activations instead of relu.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"gelu_activation\"],\n )\n\n sinusoidal_embeddings: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether or not to use sinusoidal positional embeddings instead of absolute positional embeddings.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"sinusoidal_embeddings\"],\n )\n\n causal: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether or not the model should behave in a causal manner. Causal models use a triangular \"\n \"attention mask in order to only attend to the left-side context instead if a bidirectional \"\n \"context.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"causal\"],\n )\n\n asm: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether or not to use an adaptive log softmax projection layer instead of a linear layer for the \"\n \"prediction layer.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"asm\"],\n )\n\n n_langs: int = schema_utils.PositiveInteger(\n default=1,\n description=\"The number of languages the model handles. Set to 1 for monolingual models.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"n_langs\"],\n )\n\n use_lang_emb: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use language embeddings. Some models use additional language embeddings, \"\n \"see the multilingual models page for information on how to use them.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"use_lang_emb\"],\n )\n\n max_position_embeddings: int = schema_utils.PositiveInteger(\n default=512,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large just in case (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"max_position_embeddings\"],\n )\n\n embed_init_std: float = schema_utils.NonNegativeFloat(\n default=2048**-0.5,\n description=\"The standard deviation of the truncated_normal_initializer for initializing the embedding \"\n \"matrices.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"embed_init_std\"],\n )\n\n layer_norm_eps: float = schema_utils.NonNegativeFloat(\n default=1e-12,\n description=\"The epsilon used by the layer normalization layers.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"layer_norm_eps\"],\n )\n\n init_std: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices \"\n \"except the embedding matrices.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"init_std\"],\n )\n\n bos_index: int = schema_utils.NonNegativeInteger(\n default=0,\n description=\"The index of the beginning of sentence token in the vocabulary.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"bos_index\"],\n )\n\n eos_index: int = schema_utils.NonNegativeInteger(\n default=1,\n description=\"The index of the end of sentence token in the vocabulary.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"eos_index\"],\n )\n\n pad_index: int = schema_utils.NonNegativeInteger(\n default=2,\n description=\"The index of the padding token in the vocabulary.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"pad_index\"],\n )\n\n unk_index: int = schema_utils.NonNegativeInteger(\n default=3,\n description=\"The index of the unknown token in the vocabulary.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"unk_index\"],\n )\n\n mask_index: int = schema_utils.NonNegativeInteger(\n default=5,\n description=\"The index of the masking token in the vocabulary.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"mask_index\"],\n )\n\n is_encoder: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether or not the initialized model should be a transformer encoder or decoder as seen in \"\n \"Vaswani et al.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"is_encoder\"],\n )\n\n start_n_top: int = schema_utils.PositiveInteger(\n default=5,\n description=\"Used in the SQuAD evaluation script.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"start_n_top\"],\n )\n\n end_n_top: int = schema_utils.PositiveInteger(\n default=5,\n description=\"Used in the SQuAD evaluation script.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"end_n_top\"],\n )\n\n mask_token_id: int = schema_utils.Integer(\n default=0,\n description=\"Model agnostic parameter to identify masked tokens when generating text in an MLM context.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"mask_token_id\"],\n )\n\n lang_id: int = schema_utils.Integer(\n default=0,\n description=\"The ID of the language used by the model. This parameter is used when generating text in a given \"\n \"language.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"lang_id\"],\n )\n\n pad_token_id: int = schema_utils.Integer(\n default=2,\n description=\"The ID of the token to use as padding.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"pad_token_id\"],\n )\n\n bos_token_id: int = schema_utils.Integer(\n default=0,\n description=\"The beginning of sequence token ID.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"bos_token_id\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"gpt\", TEXT)\n@ludwig_dataclass\nclass GPTConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an GPT encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"GPT\"\n\n type: str = schema_utils.ProtectedString(\n \"gpt\",\n description=ENCODER_METADATA[\"GPT\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"max_sequence_length\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"reduce_output\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"openai-gpt\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"saved_weights_in_checkpoint\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=30522,\n description=\"Vocabulary size of the GPT model. Defines the number of different tokens that can be \"\n \"represented by the inputs_ids passed when calling OpenAIGPTModel or TFOpenAIGPTModel.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"vocab_size\"],\n )\n\n n_positions: int = schema_utils.PositiveInteger(\n default=40478,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large just in case (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"n_positions\"],\n )\n\n n_ctx: int = schema_utils.PositiveInteger(\n default=512,\n description=\"Dimensionality of the causal mask (usually same as n_positions)\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"n_ctx\"],\n )\n\n n_embd: int = schema_utils.PositiveInteger(\n default=768,\n description=\"Dimensionality of the embeddings and hidden states.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"n_embd\"],\n )\n\n n_layer: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"n_layer\"],\n )\n\n n_head: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"n_head\"],\n )\n\n afn: str = schema_utils.StringOptions(\n [\"gelu\", \"relu\", \"silu\"], # gelu_new results in a KeyError.\n default=\"gelu\",\n description=\"The non-linear activation function (function or string) in the encoder and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"afn\"],\n )\n\n resid_pdrop: float = schema_utils.FloatRange(\n default=0.1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"resid_pdrop\"],\n )\n\n embd_pdrop: float = schema_utils.FloatRange(\n default=0.1,\n description=\"The dropout ratio for the embeddings.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"embd_pdrop\"],\n )\n\n attn_pdrop: float = schema_utils.FloatRange(\n default=0.1,\n description=\"The dropout ratio for the attention.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"attn_pdrop\"],\n )\n\n layer_norm_epsilon: float = schema_utils.NonNegativeFloat(\n default=1e-5,\n description=\"The epsilon to use in the layer normalization layers\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"layer_norm_epsilon\"],\n )\n\n initializer_range: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"initializer_range\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"gpt2\", TEXT)\n@ludwig_dataclass\nclass GPT2Config(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an GPT2 encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"GPT2\"\n\n type: str = schema_utils.ProtectedString(\n \"gpt2\",\n description=ENCODER_METADATA[\"GPT2\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"gpt2\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"pretrained_model_name_or_path\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=50257,\n description=\"Vocabulary size of the GPT-2 model. Defines the number of different tokens that can be \"\n \"represented by the inputs_ids passed when calling GPT2Model or TFGPT2Model.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"vocab_size\"],\n )\n\n n_positions: int = schema_utils.PositiveInteger(\n default=1024,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large just in case (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"n_positions\"],\n )\n\n n_ctx: int = schema_utils.PositiveInteger(\n default=1024,\n description=\"Dimensionality of the causal mask (usually same as n_positions)\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"n_ctx\"],\n )\n\n n_embd: int = schema_utils.PositiveInteger(\n default=768,\n description=\"Dimensionality of the embeddings and hidden states.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"n_embd\"],\n )\n\n n_layer: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"n_layer\"],\n )\n\n n_head: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"n_head\"],\n )\n\n n_inner: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Dimensionality of the inner feed-forward layers. None will set it to 4 times n_embd\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"n_inner\"],\n )\n\n activation_function: str = schema_utils.StringOptions(\n [\"relu\", \"silu\", \"gelu\", \"tanh\", \"gelu_new\"],\n default=\"gelu_new\",\n description=\"Activation function, to be selected in the list ['relu', 'silu', 'gelu', 'tanh', 'gelu_new'].\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"activation_function\"],\n )\n\n resid_pdrop: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"resid_pdrop\"],\n )\n\n embd_pdrop: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout ratio for the embeddings.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"embd_pdrop\"],\n )\n\n attn_pdrop: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout ratio for the attention.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"attn_pdrop\"],\n )\n\n layer_norm_epsilon: float = schema_utils.NonNegativeFloat(\n default=1e-5,\n description=\"The epsilon to use in the layer normalization layers.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"layer_norm_epsilon\"],\n )\n\n initializer_range: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"initializer_range\"],\n )\n\n scale_attn_weights: bool = schema_utils.Boolean(\n default=True,\n description=\"Scale attention weights by dividing by sqrt(hidden_size).\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"scale_attn_weights\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"roberta\", TEXT)\n@ludwig_dataclass\nclass RoBERTaConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an RoBERTa encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"RoBERTa\"\n\n type: str = schema_utils.ProtectedString(\n \"roberta\",\n description=ENCODER_METADATA[\"RoBERTa\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"roberta-base\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"saved_weights_in_checkpoint\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"cls_pooled\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Vocabulary size of the RoBERTa model.\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"vocab_size\"],\n )\n\n pad_token_id: int = schema_utils.Integer(\n default=1,\n description=\"The ID of the token to use as padding.\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"pad_token_id\"],\n )\n\n bos_token_id: int = schema_utils.Integer(\n default=0,\n description=\"The beginning of sequence token ID.\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"bos_token_id\"],\n )\n\n eos_token_id: int = schema_utils.Integer(\n default=2,\n description=\"The end of sequence token ID.\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"eos_token_id\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"transformer_xl\", TEXT)\n@ludwig_dataclass\nclass TransformerXLConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an TransformerXL encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"TransformerXL\"\n\n type: str = schema_utils.ProtectedString(\n \"transformer_xl\",\n description=ENCODER_METADATA[\"TransformerXL\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"transfo-xl-wt103\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"saved_weights_in_checkpoint\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=267735,\n description=\"Vocabulary size of the TransfoXL model. Defines the number of different tokens that can be \"\n \"represented by the inputs_ids passed when calling TransfoXLModel or TFTransfoXLModel.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"vocab_size\"],\n )\n\n cutoffs: list[int] = schema_utils.List(\n int,\n default=[20000, 40000, 200000],\n description=\"Cutoffs for the adaptive softmax.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"cutoffs\"],\n )\n\n d_model: int = schema_utils.PositiveInteger(\n default=1024,\n description=\"Dimensionality of the model’s hidden states.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"d_model\"],\n )\n\n d_embed: int = schema_utils.PositiveInteger(\n default=1024,\n description=\"Dimensionality of the embeddings\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"d_embed\"],\n )\n\n n_head: int = schema_utils.PositiveInteger(\n default=16,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"n_head\"],\n )\n\n d_head: int = schema_utils.PositiveInteger(\n default=64,\n description=\"Dimensionality of the model’s heads.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"d_head\"],\n )\n\n d_inner: int = schema_utils.PositiveInteger(\n default=4096,\n description=\" Inner dimension in FF\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"d_inner\"],\n )\n\n div_val: int = schema_utils.PositiveInteger(\n default=4,\n description=\"Divident value for adapative input and softmax.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"div_val\"],\n )\n\n pre_lnorm: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether or not to apply LayerNorm to the input instead of the output in the blocks.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"pre_lnorm\"],\n )\n\n n_layer: int = schema_utils.PositiveInteger(\n default=18,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"n_layer\"],\n )\n\n mem_len: int = schema_utils.PositiveInteger(\n default=1600,\n description=\"Length of the retained previous heads.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"mem_len\"],\n )\n\n clamp_len: int = schema_utils.PositiveInteger(\n default=1000,\n description=\"Use the same pos embeddings after clamp_len.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"clamp_len\"],\n )\n\n same_length: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether or not to use the same attn length for all tokens\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"same_length\"],\n )\n\n proj_share_all_but_first: bool = schema_utils.Boolean(\n default=True,\n description=\"True to share all but first projs, False not to share.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"proj_share_all_but_first\"],\n )\n\n attn_type: int = schema_utils.IntegerRange(\n default=0,\n min=0,\n max=3,\n description=\"Attention type. 0 for Transformer-XL, 1 for Shaw et al, 2 for Vaswani et al, 3 for Al Rfou et al.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"attn_type\"],\n )\n\n sample_softmax: int = schema_utils.Integer(\n default=-1,\n description=\"Number of samples in the sampled softmax.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"sample_softmax\"],\n )\n\n adaptive: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether or not to use adaptive softmax.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"adaptive\"],\n )\n\n dropout: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"dropout\"],\n )\n\n dropatt: float = schema_utils.NonNegativeFloat(\n default=0.0,\n description=\"The dropout ratio for the attention probabilities.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"dropatt\"],\n )\n\n untie_r: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether ot not to untie relative position biases.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"untie_r\"],\n )\n\n init: str = schema_utils.String(\n default=\"normal\",\n description=\"Parameter initializer to use.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"init\"],\n )\n\n init_range: float = schema_utils.NonNegativeFloat(\n default=0.01,\n description=\"Parameters initialized by U(-init_range, init_range).\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"init_range\"],\n )\n\n proj_init_std: float = schema_utils.NonNegativeFloat(\n default=0.01,\n description=\"Parameters initialized by N(0, init_std)\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"proj_init_std\"],\n )\n\n init_std: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"Parameters initialized by N(0, init_std)\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"init_std\"],\n )\n\n layer_norm_epsilon: float = schema_utils.NonNegativeFloat(\n default=1e-5,\n description=\"The epsilon to use in the layer normalization layers\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"layer_norm_epsilon\"],\n )\n\n eos_token_id: int = schema_utils.Integer(\n default=0,\n description=\"The end of sequence token ID.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"eos_token_id\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"xlnet\", TEXT)\n@ludwig_dataclass\nclass XLNetConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an XLNet encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"XLNet\"\n\n type: str = schema_utils.ProtectedString(\n \"xlnet\",\n description=ENCODER_METADATA[\"XLNet\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"xlnet-base-cased\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"saved_weights_in_checkpoint\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=32000,\n description=\"Vocabulary size of the XLNet model. Defines the number of different tokens that can be \"\n \"represented by the inputs_ids passed when calling XLNetModel or TFXLNetModel.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"vocab_size\"],\n )\n\n d_model: int = schema_utils.PositiveInteger(\n default=768,\n description=\"Dimensionality of the encoder layers and the pooler layer.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"d_model\"],\n )\n\n n_layer: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"n_layer\"],\n )\n\n n_head: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"n_head\"],\n )\n\n d_inner: int = schema_utils.PositiveInteger(\n default=3072,\n description=\"Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"d_inner\"],\n )\n\n ff_activation: str = schema_utils.StringOptions(\n [\"gelu\", \"relu\", \"silu\", \"gelu_new\"],\n default=\"gelu\",\n description=\"The non-linear activation function (function or string) in the encoder and pooler. If string, \"\n \"'gelu', 'relu', 'silu' and 'gelu_new' are supported.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"ff_activation\"],\n )\n\n untie_r: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether or not to untie relative position biases\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"untie_r\"],\n )\n\n attn_type: str = schema_utils.StringOptions(\n [\"bi\"],\n default=\"bi\",\n description=\"The attention type used by the model. Currently only 'bi' is supported.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"attn_type\"],\n )\n\n initializer_range: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"initializer_range\"],\n )\n\n layer_norm_eps: float = schema_utils.NonNegativeFloat(\n default=1e-12,\n description=\"The epsilon used by the layer normalization layers.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"layer_norm_eps\"],\n )\n\n dropout: float = schema_utils.FloatRange(\n default=0.1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"dropout\"],\n )\n\n mem_len: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"The number of tokens to cache. The key/value pairs that have already been pre-computed in a \"\n \"previous forward pass won’t be re-computed. \",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"mem_len\"],\n )\n\n reuse_len: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"The number of tokens in the current batch to be cached and reused in the future.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"reuse_len\"],\n )\n\n use_mems_eval: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether or not the model should make use of the recurrent memory mechanism in evaluation mode.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"use_mems_eval\"],\n )\n\n use_mems_train: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether or not the model should make use of the recurrent memory mechanism in train mode.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"use_mems_train\"],\n )\n\n bi_data: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether or not to use bidirectional input pipeline. Usually set to True during pretraining and \"\n \"False during finetuning.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"bi_data\"],\n )\n\n clamp_len: int = schema_utils.Integer(\n default=-1,\n description=\"Clamp all relative distances larger than clamp_len. Setting this attribute to -1 means no \"\n \"clamping.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"clamp_len\"],\n )\n\n same_length: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether or not to use the same attention length for each token.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"same_length\"],\n )\n\n summary_type: str = schema_utils.StringOptions(\n [\"last\", \"first\", \"mean\", \"cls_index\", \"attn\"],\n default=\"last\",\n description=\"Argument used when doing sequence summary. Used in the sequence classification and multiple \"\n \"choice models.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"summary_type\"],\n )\n\n summary_use_proj: bool = schema_utils.Boolean(\n default=True,\n description=\"\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"summary_use_proj\"],\n )\n\n summary_activation: str = schema_utils.String(\n default=\"tanh\",\n description=\"Argument used when doing sequence summary. Used in the sequence classification and multiple \"\n \"choice models.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"summary_activation\"],\n )\n\n summary_last_dropout: float = schema_utils.FloatRange(\n default=0.1,\n description=\"Used in the sequence classification and multiple choice models.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"summary_last_dropout\"],\n )\n\n start_n_top: int = schema_utils.PositiveInteger(\n default=5,\n description=\"Used in the SQuAD evaluation script.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"start_n_top\"],\n )\n\n end_n_top: int = schema_utils.PositiveInteger(\n default=5,\n description=\" Used in the SQuAD evaluation script.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"end_n_top\"],\n )\n\n pad_token_id: int = schema_utils.Integer(\n default=5,\n description=\"The ID of the token to use as padding.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"pad_token_id\"],\n )\n\n bos_token_id: int = schema_utils.Integer(\n default=1,\n description=\"The beginning of sequence token ID.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"bos_token_id\"],\n )\n\n eos_token_id: int = schema_utils.Integer(\n default=2,\n description=\"The end of sequence token ID.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"eos_token_id\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"distilbert\", TEXT)\n@ludwig_dataclass\nclass DistilBERTConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an DistilBERT encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"DistilBERT\"\n\n type: str = schema_utils.ProtectedString(\n \"distilbert\",\n description=ENCODER_METADATA[\"DistilBERT\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"distilbert-base-uncased\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"saved_weights_in_checkpoint\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=30522,\n description=\"Vocabulary size of the DistilBERT model. Defines the number of different tokens that can be \"\n \"represented by the inputs_ids passed when calling DistilBertModel or TFDistilBertModel.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"vocab_size\"],\n )\n\n dropout: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"dropout\"],\n )\n\n max_position_embeddings: int = schema_utils.PositiveInteger(\n default=512,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large just in case (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"max_position_embeddings\"],\n )\n\n sinusoidal_pos_embds: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to use sinusoidal positional embeddings.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"sinusoidal_pos_embds\"],\n )\n\n n_layers: int = schema_utils.PositiveInteger(\n default=6,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"n_layers\"],\n )\n\n n_heads: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"n_heads\"],\n )\n\n dim: int = schema_utils.PositiveInteger(\n default=768,\n description=\" Dimensionality of the encoder layers and the pooler layer.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"dim\"],\n )\n\n hidden_dim: int = schema_utils.PositiveInteger(\n default=3072,\n description=\"The size of the “intermediate” (often named feed-forward) layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"hidden_dim\"],\n )\n\n attention_dropout: float = schema_utils.NonNegativeFloat(\n default=0.1,\n description=\"The dropout ratio for the attention probabilities.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"attention_dropout\"],\n )\n\n activation: str | Callable = schema_utils.StringOptions( # TODO: Add support for callable\n [\"gelu\", \"relu\", \"silu\", \"gelu_new\"],\n default=\"gelu\",\n description=\"The non-linear activation function (function or string) in the encoder and pooler. If string, \"\n \"'gelu', 'relu', 'silu' and 'gelu_new' are supported.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"activation\"],\n )\n\n initializer_range: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"initializer_range\"],\n )\n\n qa_dropout: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout probabilities used in the question answering model DistilBertForQuestionAnswering.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"qa_dropout\"],\n )\n\n seq_classif_dropout: float = schema_utils.FloatRange(\n default=0.2,\n min=0,\n max=1,\n description=\"The dropout probabilities used in the sequence classification and the multiple choice model \"\n \"DistilBertForSequenceClassification.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"seq_classif_dropout\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"pretrained_kwargs\"],\n )\n\n\n# TODO: uncomment when CTRL bug (https://github.com/ludwig-ai/ludwig/issues/2977) has been fixed to add back in\n@DeveloperAPI\n# @register_encoder_config(\"ctrl\", TEXT)\n@ludwig_dataclass\nclass CTRLConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an CTRL encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"CTRL\"\n\n type: str = schema_utils.ProtectedString(\n \"ctrl\",\n description=ENCODER_METADATA[\"CTRL\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"ctrl\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"saved_weights_in_checkpoint\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=246534,\n description=\"Vocabulary size of the CTRL model. Defines the number of different tokens that can be \"\n \"represented by the inputs_ids passed when calling CTRLModel or TFCTRLModel.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"vocab_size\"],\n )\n\n n_positions: int = schema_utils.PositiveInteger(\n default=256,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large just in case (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"n_positions\"],\n )\n\n n_ctx: int = schema_utils.PositiveInteger(\n default=256,\n description=\"Dimensionality of the causal mask (usually same as n_positions)\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"n_ctx\"],\n )\n\n n_embd: int = schema_utils.PositiveInteger(\n default=1280,\n description=\"Dimensionality of the embeddings and hidden states.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"n_embd\"],\n )\n\n dff: int = schema_utils.PositiveInteger(\n default=8192,\n description=\"Dimensionality of the inner dimension of the feed forward networks (FFN).\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"dff\"],\n )\n\n n_layer: int = schema_utils.PositiveInteger(\n default=48,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"n_layer\"],\n )\n\n n_head: int = schema_utils.PositiveInteger(\n default=16,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"n_head\"],\n )\n\n resid_pdrop: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\" The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"resid_pdrop\"],\n )\n\n embd_pdrop: float = schema_utils.NonNegativeFloat(\n default=0.1,\n description=\"The dropout ratio for the embeddings.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"embd_pdrop\"],\n )\n\n attn_pdrop: float = schema_utils.NonNegativeFloat(\n default=0.1,\n description=\"The dropout ratio for the attention.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"attn_pdrop\"],\n )\n\n layer_norm_epsilon: float = schema_utils.NonNegativeFloat(\n default=1e-6,\n description=\"The epsilon to use in the layer normalization layers\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"layer_norm_epsilon\"],\n )\n\n initializer_range: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"initializer_range\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"camembert\", TEXT)\n@ludwig_dataclass\nclass CamemBERTConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an CamemBERT encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"CamemBERT\"\n\n type: str = schema_utils.ProtectedString(\n \"camembert\",\n description=ENCODER_METADATA[\"CamemBERT\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"use_pretrained\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"saved_weights_in_checkpoint\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"camembert-base\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"pretrained_model_name_or_path\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=32005,\n description=\"Vocabulary size of the CamemBERT model.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"vocab_size\"],\n )\n\n hidden_size: int = schema_utils.PositiveInteger(\n default=768,\n description=\"Dimensionality of the encoder layers and the pooler layer.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"hidden_size\"],\n )\n\n num_hidden_layers: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"num_hidden_layers\"],\n )\n\n num_attention_heads: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"num_attention_heads\"],\n )\n\n intermediate_size: int = schema_utils.PositiveInteger(\n default=3072,\n description=\"Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"intermediate_size\"],\n )\n\n hidden_act: str | Callable = schema_utils.StringOptions( # TODO: add support for callable\n [\"gelu\", \"relu\", \"silu\", \"gelu_new\"],\n default=\"gelu\",\n description=\"The non-linear activation function (function or string) in the encoder and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"hidden_act\"],\n )\n\n hidden_dropout_prob: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"hidden_dropout_prob\"],\n )\n\n attention_probs_dropout_prob: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout ratio for the attention probabilities.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"attention_probs_dropout_prob\"],\n )\n\n max_position_embeddings: int = schema_utils.PositiveInteger(\n default=514,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large just in case (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"max_position_embeddings\"],\n )\n\n type_vocab_size: int = schema_utils.PositiveInteger(\n default=1,\n description=\"The vocabulary size of the token_type_ids passed when calling BertModel or TFBertModel.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"type_vocab_size\"],\n )\n\n initializer_range: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"initializer_range\"],\n )\n\n layer_norm_eps: float = schema_utils.NonNegativeFloat(\n default=1e-05,\n description=\"The epsilon used by the layer normalization layers.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"layer_norm_eps\"],\n )\n\n pad_token_id: int = schema_utils.Integer(\n default=1,\n description=\"The ID of the token to use as padding.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"pad_token_id\"],\n )\n\n gradient_checkpointing: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to use gradient checkpointing.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"gradient_checkpointing\"],\n )\n\n position_embedding_type: str = schema_utils.StringOptions(\n [\"absolute\", \"relative_key\", \"relative_key_query\"],\n default=\"absolute\",\n description=\"Type of position embedding.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"position_embedding_type\"],\n )\n\n classifier_dropout: float = schema_utils.FloatRange(\n default=None,\n allow_none=True,\n min=0,\n max=1,\n description=\"The dropout ratio for the classification head.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"classifier_dropout\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"t5\", TEXT)\n@ludwig_dataclass\nclass T5Config(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an T5 encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"T5\"\n\n type: str = schema_utils.ProtectedString(\n \"t5\",\n description=ENCODER_METADATA[\"T5\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"t5-small\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"saved_weights_in_checkpoint\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=32128,\n description=\"Vocabulary size of the T5 model. Defines the number of different tokens that can be represented \"\n \"by the inputs_ids passed when calling T5Model or TFT5Model.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"vocab_size\"],\n )\n\n d_model: int = schema_utils.PositiveInteger(\n default=512,\n description=\"Size of the encoder layers and the pooler layer.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"d_model\"],\n )\n\n d_kv: int = schema_utils.PositiveInteger(\n default=64,\n description=\"Size of the key, query, value projections per attention head. d_kv has to be equal to d_model // \"\n \"num_heads.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"d_kv\"],\n )\n\n d_ff: int = schema_utils.PositiveInteger(\n default=2048,\n description=\"Size of the intermediate feed forward layer in each T5Block.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"d_ff\"],\n )\n\n num_layers: int = schema_utils.PositiveInteger(\n default=6,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"num_layers\"],\n )\n\n num_decoder_layers: int = schema_utils.PositiveInteger(\n default=6,\n description=\"Number of hidden layers in the Transformer decoder. Will use the same value as num_layers if not \"\n \"set.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"num_decoder_layers\"],\n )\n\n num_heads: int = schema_utils.PositiveInteger(\n default=8,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"num_heads\"],\n )\n\n relative_attention_num_buckets: int = schema_utils.PositiveInteger(\n default=32,\n description=\"The number of buckets to use for each attention layer.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"relative_attention_num_buckets\"],\n )\n\n dropout_rate: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The ratio for all dropout layers.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"dropout_rate\"],\n )\n\n layer_norm_eps: float = schema_utils.NonNegativeFloat(\n default=1e-6,\n description=\"The epsilon used by the layer normalization layers.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"layer_norm_eps\"],\n )\n\n initializer_factor: float = schema_utils.NonNegativeFloat(\n default=1,\n description=\"A factor for initializing all weight matrices (should be kept to 1, used internally for \"\n \"initialization testing).\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"initializer_factor\"],\n )\n\n feed_forward_proj: str = schema_utils.StringOptions(\n [\"relu\", \"gated-gelu\"],\n default=\"relu\",\n description=\"Type of feed forward layer to be used. Should be one of 'relu' or 'gated-gelu'. T5v1.1 uses the \"\n \"'gated-gelu' feed forward projection. Original T5 uses 'relu'.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"feed_forward_proj\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"flaubert\", TEXT)\n@ludwig_dataclass\nclass FlauBERTConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an FlauBERT encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"FlauBERT\"\n\n type: str = schema_utils.ProtectedString(\n \"flaubert\",\n description=ENCODER_METADATA[\"FlauBERT\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"flaubert/flaubert_small_cased\",\n description=\"Name of path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"saved_weights_in_checkpoint\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=30145,\n description=\"Vocabulary size of the FlauBERT model. Defines the number of different tokens that can be \"\n \"represented by the inputs_ids passed when calling FlaubertModel or TFFlaubertModel.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"vocab_size\"],\n )\n\n pre_norm: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to apply the layer normalization before or after the feed forward layer following the \"\n \"attention in each layer (Vaswani et al., Tensor2Tensor for Neural Machine Translation. 2018)\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"pre_norm\"],\n )\n\n layerdrop: float = schema_utils.FloatRange(\n default=0.2,\n min=0,\n max=1,\n description=\"Probability to drop layers during training (Fan et al., Reducing Transformer Depth on Demand \"\n \"with Structured Dropout. ICLR 2020)\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"layerdrop\"],\n )\n\n emb_dim: int = schema_utils.PositiveInteger(\n default=512,\n description=\"Dimensionality of the encoder layers and the pooler layer.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"emb_dim\"],\n )\n\n n_layers: int = schema_utils.PositiveInteger(\n default=6,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"n_layers\"],\n )\n\n n_heads: int = schema_utils.PositiveInteger(\n default=8,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"n_heads\"],\n )\n\n dropout: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"dropout\"],\n )\n\n attention_dropout: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout probability for the attention mechanism\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"attention_dropout\"],\n )\n\n gelu_activation: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether or not to use a gelu activation instead of relu.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"gelu_activation\"],\n )\n\n sinusoidal_embeddings: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether or not to use sinusoidal positional embeddings instead of absolute positional embeddings.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"sinusoidal_embeddings\"],\n )\n\n causal: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether or not the model should behave in a causal manner. Causal models use a triangular \"\n \"attention mask in order to only attend to the left-side context instead if a bidirectional \"\n \"context.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"causal\"],\n )\n\n asm: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether or not to use an adaptive log softmax projection layer instead of a linear layer for the \"\n \"prediction layer.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"asm\"],\n )\n\n n_langs: int = schema_utils.PositiveInteger(\n default=1,\n description=\"The number of languages the model handles. Set to 1 for monolingual models.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"n_langs\"],\n )\n\n use_lang_emb: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use language embeddings. Some models use additional language embeddings, \"\n \"see the multilingual models page for information on how to use them.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"use_lang_emb\"],\n )\n\n max_position_embeddings: int = schema_utils.PositiveInteger(\n default=512,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large just in case (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"max_position_embeddings\"],\n )\n\n embed_init_std: float = schema_utils.NonNegativeFloat(\n default=2048**-0.5,\n description=\"The standard deviation of the truncated_normal_initializer for initializing the embedding \"\n \"matrices.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"embed_init_std\"],\n )\n\n init_std: int = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices \"\n \"except the embedding matrices.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"init_std\"],\n )\n\n layer_norm_eps: float = schema_utils.NonNegativeFloat(\n default=1e-06,\n description=\"The epsilon used by the layer normalization layers.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"layer_norm_eps\"],\n )\n\n bos_index: int = schema_utils.NonNegativeInteger(\n default=0,\n description=\"The index of the beginning of sentence token in the vocabulary.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"bos_index\"],\n )\n\n eos_index: int = schema_utils.NonNegativeInteger(\n default=1,\n description=\"The index of the end of sentence token in the vocabulary.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"eos_index\"],\n )\n\n pad_index: int = schema_utils.NonNegativeInteger(\n default=2,\n description=\"The index of the padding token in the vocabulary.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"pad_index\"],\n )\n\n unk_index: int = schema_utils.NonNegativeInteger(\n default=3,\n description=\"The index of the unknown token in the vocabulary.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"unk_index\"],\n )\n\n mask_index: int = schema_utils.NonNegativeInteger(\n default=5,\n description=\"The index of the masking token in the vocabulary.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"mask_index\"],\n )\n\n is_encoder: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether or not the initialized model should be a transformer encoder or decoder as seen in \"\n \"Vaswani et al.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"is_encoder\"],\n )\n\n mask_token_id: int = schema_utils.Integer(\n default=0,\n description=\"Model agnostic parameter to identify masked tokens when generating text in an MLM context.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"mask_token_id\"],\n )\n\n lang_id: int = schema_utils.Integer(\n default=0,\n description=\"The ID of the language used by the model. This parameter is used when generating text in a given \"\n \"language.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"lang_id\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"electra\", TEXT)\n@ludwig_dataclass\nclass ELECTRAConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an ELECTRA encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"ELECTRA\"\n\n type: str = schema_utils.ProtectedString(\n \"electra\",\n description=ENCODER_METADATA[\"ELECTRA\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"google/electra-small-discriminator\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"saved_weights_in_checkpoint\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=30522,\n description=\"Vocabulary size of the ELECTRA model. Defines the number of different tokens that can be \"\n \"represented by the inputs_ids passed when calling ElectraModel or TFElectraModel.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"vocab_size\"],\n )\n\n embedding_size: int = schema_utils.PositiveInteger(\n default=128,\n description=\"Dimensionality of the encoder layers and the pooler layer.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"embedding_size\"],\n )\n\n hidden_size: int = schema_utils.PositiveInteger(\n default=256,\n description=\"Dimensionality of the encoder layers and the pooler layer.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"hidden_size\"],\n )\n\n num_hidden_layers: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"num_hidden_layers\"],\n )\n\n num_attention_heads: int = schema_utils.PositiveInteger(\n default=4,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"num_attention_heads\"],\n )\n\n intermediate_size: int = schema_utils.PositiveInteger(\n default=1024,\n description=\"Dimensionality of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"intermediate_size\"],\n )\n\n hidden_act: str | Callable = schema_utils.StringOptions( # TODO: add support for callable\n [\"gelu\", \"relu\", \"silu\", \"gelu_new\"],\n default=\"gelu\",\n description=\"The non-linear activation function (function or string) in the encoder and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"hidden_act\"],\n )\n\n hidden_dropout_prob: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"hidden_dropout_prob\"],\n )\n\n attention_probs_dropout_prob: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout ratio for the attention probabilities.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"attention_probs_dropout_prob\"],\n )\n\n max_position_embeddings: int = schema_utils.PositiveInteger(\n default=512,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large just in case (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"max_position_embeddings\"],\n )\n\n type_vocab_size: int = schema_utils.PositiveInteger(\n default=2,\n description=\"The vocabulary size of the token_type_ids passed when calling ElectraModel or TFElectraModel.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"type_vocab_size\"],\n )\n\n initializer_range: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"initializer_range\"],\n )\n\n layer_norm_eps: float = schema_utils.NonNegativeFloat(\n default=1e-12,\n description=\"The epsilon used by the layer normalization layers.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"layer_norm_eps\"],\n )\n\n position_embedding_type: str = schema_utils.StringOptions(\n [\"absolute\", \"relative_key\", \"relative_key_query\"],\n default=\"absolute\",\n description=\"Type of position embedding.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"position_embedding_type\"],\n )\n\n classifier_dropout: float = schema_utils.FloatRange(\n default=None,\n allow_none=True,\n min=0,\n max=1,\n description=\"The dropout ratio for the classification head.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"classifier_dropout\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"longformer\", TEXT)\n@ludwig_dataclass\nclass LongformerConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for a Longformer encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"Longformer\"\n\n type: str = schema_utils.ProtectedString(\n \"longformer\",\n description=ENCODER_METADATA[\"Longformer\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"use_pretrained\"],\n )\n\n attention_window: list[int] | int = schema_utils.OneOfOptionsField(\n default=512,\n allow_none=False,\n description=\"Size of an attention window around each token. If an int, use the same size for all layers. To \"\n \"specify a different window size for each layer, use a List[int] where len(attention_window) == \"\n \"num_hidden_layers.\",\n field_options=[\n schema_utils.PositiveInteger(allow_none=False, description=\"\", default=512),\n schema_utils.List(list_type=int, allow_none=False),\n ],\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"attention_window\"],\n )\n\n sep_token_id: int = schema_utils.Integer(\n default=2,\n description=\"ID of the separator token, which is used when building a sequence from multiple sequences\",\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"sep_token_id\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"allenai/longformer-base-4096\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ParameterMetadata(internal_only=True),\n )\n\n reduce_output: str = schema_utils.String(\n default=\"cls_pooled\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=50265,\n description=\"Vocabulary size of the Longformer model.\",\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"vocab_size\"],\n )\n\n max_position_embeddings: int = schema_utils.PositiveInteger(\n default=4098,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large just in case (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"max_position_embeddings\"],\n )\n\n type_vocab_size: int = schema_utils.PositiveInteger(\n default=1,\n description=\"The vocabulary size of the token_type_ids passed when calling LongformerEncoder\",\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"type_vocab_size\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"auto_transformer\", TEXT)\n@ludwig_dataclass\nclass AutoTransformerConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an AutoTransformer encoder.\"\"\"\n\n def __post_init__(self):\n if self.pretrained_model_name_or_path is None:\n raise ConfigValidationError(\n \"`pretrained_model_name_or_path` must be specified for encoder: `auto_transformer`.\"\n )\n\n @staticmethod\n def module_name():\n return \"AutoTransformer\"\n\n @property\n def use_pretrained(self) -> bool:\n # Always set this to True since we always want to use the pretrained weights\n # We don't currently support training from scratch for AutoTransformers\n return True\n\n type: str = schema_utils.ProtectedString(\n \"auto_transformer\",\n description=ENCODER_METADATA[\"AutoTransformer\"][\"type\"].long_description,\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"AutoTransformer\"][\"pretrained_model_name_or_path\"],\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"AutoTransformer\"][\"max_sequence_length\"],\n )\n\n reduce_output: str = schema_utils.ReductionOptions(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"AutoTransformer\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"AutoTransformer\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"AutoTransformer\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=(\n \"Vocabulary size of the AutoTransformer model. If None, the vocab size will be inferred \"\n \"from the given pretrained model\"\n ),\n parameter_metadata=ENCODER_METADATA[\"AutoTransformer\"][\"vocab_size\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"AutoTransformer\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"tf_idf\", TEXT, model_types=[MODEL_ECD])\n@ludwig_dataclass\nclass TfIdfEncoderConfig(SequenceEncoderConfig):\n type: str = schema_utils.ProtectedString(\"tf_idf\")\n\n max_sequence_length: int = schema_utils.Integer(default=None, allow_none=True, parameter_metadata=INTERNAL_ONLY)\n\n str2idf: dict[str, int] = schema_utils.Dict(parameter_metadata=INTERNAL_ONLY)\n\n vocab: list = schema_utils.List(default=None, parameter_metadata=INTERNAL_ONLY)\n\n vocab_size: int = schema_utils.Integer(default=None, allow_none=True, parameter_metadata=INTERNAL_ONLY)\n\n def set_fixed_preprocessing_params(self, model_type: str, preprocessing: \"TextPreprocessingConfig\"):\n preprocessing.compute_idf = True\n\n def can_cache_embeddings(self) -> bool:\n return True\n\n\n@DeveloperAPI\n@register_encoder_config(\"llm\", TEXT, model_types=[MODEL_ECD])\n@ludwig_dataclass\nclass LLMEncoderConfig(SequenceEncoderConfig):\n type: str = schema_utils.ProtectedString(\"llm\")\n base_model: str = BaseModelDataclassField()\n max_sequence_length: int = schema_utils.Integer(default=None, allow_none=True, parameter_metadata=INTERNAL_ONLY)\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n quantization: QuantizationConfig | None = QuantizationConfigField().get_default_field()\n model_parameters: ModelParametersConfig | None = ModelParametersConfigField().get_default_field()\n"
},
{
"path": "ludwig/schema/encoders/text/hf_model_params.py",
"content": "from ludwig.schema import utils as schema_utils\nfrom ludwig.schema.metadata.parameter_metadata import INTERNAL_ONLY\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\"\"\"\nNOTE TO DEVELOPERS: the implementation of the schema classes below must match the parameters of the HF PretrainedConfig\nclass exactly. This is because we convert this object into the matching HF PretrainedConfig object before passing it to\nthe model. Additionally, for loading and saving pretrained models, we take the config from the existing model and load\nit into this config before saving. As such, if any params needed by the pretrained model are missing, we will not be\nable to load checkpoints correctly.\n\nA common mistake is to look at the PretrainedConfig __init__ method params and ignore any additional **kwargs. In some\ncases, these kwargs are used to set additional params on the config object. For example, the DebertaConfig class has\n`position_buckets` as a kwarg param, but it nonetheless requires this to construct the model architecture.\n\nTo debug issues with missing parameters, try printing out the `model.config` of the pretrained transformer and check\nfor any params it includes that are not present in your schema config.\n\"\"\"\n\n\n@ludwig_dataclass\nclass DebertaModelParams(schema_utils.BaseMarshmallowConfig):\n @classmethod\n def get_hf_config_param_names(cls) -> set[str]:\n return DebertaModelParams.get_valid_field_names()\n\n # Model architecture params for training from scratch\n # TODO(travis): conditionally disable setting these when `use_pretrained=True`.\n vocab_size: int = schema_utils.PositiveInteger(\n default=None,\n description=\"\",\n parameter_metadata=INTERNAL_ONLY,\n )\n\n hidden_size: int = schema_utils.PositiveInteger(\n default=1536,\n description=\"Dimensionality of the encoder layers and the pooler layer.\",\n )\n\n num_hidden_layers: int = schema_utils.PositiveInteger(\n default=24,\n description=\"Number of hidden layers in the Transformer encoder.\",\n )\n\n num_attention_heads: int = schema_utils.PositiveInteger(\n default=24,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n )\n\n intermediate_size: int = schema_utils.PositiveInteger(\n default=6144,\n description=\"Dimensionality of the 'intermediate' (often named feed-forward) layer in the Transformer encoder.\",\n )\n\n hidden_act: str = schema_utils.StringOptions(\n options=[\"gelu\", \"relu\", \"silu\", \"tanh\", \"gelu_fast\", \"mish\", \"linear\", \"sigmoid\", \"gelu_new\"],\n default=\"gelu\",\n description=\"The non-linear activation function (function or string) in the encoder and pooler.\",\n )\n\n hidden_dropout_prob: float = schema_utils.NonNegativeFloat(\n default=0.1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n )\n\n attention_probs_dropout_prob: float = schema_utils.NonNegativeFloat(\n default=0.1,\n description=\"The dropout ratio for the attention probabilities.\",\n )\n\n max_position_embeddings: int = schema_utils.PositiveInteger(\n default=512,\n description=(\n \"The maximum sequence length that this model might ever be used with. Typically set this to something \"\n \"large just in case (e.g., 512 or 1024 or 2048).\"\n ),\n )\n\n type_vocab_size: int = schema_utils.NonNegativeInteger(\n default=0,\n description=(\"The vocabulary size of the `token_type_ids`.\"),\n )\n\n initializer_range: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=(\n \"The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\"\n ),\n )\n\n layer_norm_eps: float = schema_utils.NonNegativeFloat(\n default=1e-7,\n description=\"The epsilon used by the layer normalization layers.\",\n )\n\n relative_attention: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether use relative position encoding.\",\n )\n\n max_relative_positions: int = schema_utils.Integer(\n default=-1,\n description=(\n \"The range of relative positions `[-max_position_embeddings, max_position_embeddings]`. Use the same \"\n \"value as `max_position_embeddings`.\"\n ),\n )\n\n pad_token_id: int = schema_utils.Integer(\n default=0,\n description=\"The value used to pad input_ids.\",\n )\n\n position_biased_input: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether add absolute position embedding to content embedding.\",\n )\n\n pos_att_type: list[str] = schema_utils.List(\n default=[\"p2c\", \"c2p\"],\n description=(\n \"The type of relative position attention, it can be a combination of `['p2c', 'c2p']`, e.g. `['p2c']`, \"\n \"`['p2c', 'c2p']`, `['p2c', 'c2p']`.\"\n ),\n )\n\n layer_norm_eps: float = schema_utils.NonNegativeFloat(\n default=1e-12,\n description=\"The epsilon used by the layer normalization layers.\",\n )\n\n pooler_hidden_size: int = schema_utils.PositiveInteger(\n default=1536,\n description=\"The hidden size of the pooler layers.\",\n )\n\n pooler_dropout: float = schema_utils.NonNegativeFloat(\n default=0,\n description=\"The dropout ratio for the pooler layers.\",\n )\n\n pooler_hidden_act: str = schema_utils.StringOptions(\n options=[\"gelu\", \"relu\", \"silu\", \"tanh\", \"gelu_fast\", \"mish\", \"linear\", \"sigmoid\", \"gelu_new\"],\n default=\"gelu\",\n description=\"The activation function (function or string) in the pooler.\",\n )\n\n position_buckets: int = schema_utils.PositiveInteger(\n default=256,\n description=\"The number of buckets to use for each attention layer.\",\n )\n\n share_att_key: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to share attention key across layers.\",\n )\n\n norm_rel_ebd: str = schema_utils.StringOptions(\n options=[\"layer_norm\", \"none\"],\n default=\"layer_norm\",\n description=\"The normalization method for relative embeddings.\",\n )\n"
},
{
"path": "ludwig/schema/encoders/text_encoders.py",
"content": "from collections.abc import Callable\nfrom typing import TYPE_CHECKING\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import MODEL_ECD, TEXT\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.encoders.sequence_encoders import SequenceEncoderConfig\nfrom ludwig.schema.encoders.text.hf_model_params import DebertaModelParams\nfrom ludwig.schema.encoders.utils import register_encoder_config\nfrom ludwig.schema.llms.base_model import BaseModelDataclassField\nfrom ludwig.schema.llms.model_parameters import ModelParametersConfig, ModelParametersConfigField\nfrom ludwig.schema.llms.peft import AdapterDataclassField, BaseAdapterConfig\nfrom ludwig.schema.llms.quantization import QuantizationConfig, QuantizationConfigField\nfrom ludwig.schema.metadata import ENCODER_METADATA\nfrom ludwig.schema.metadata.parameter_metadata import INTERNAL_ONLY, ParameterMetadata\nfrom ludwig.schema.utils import ludwig_dataclass\n\nif TYPE_CHECKING:\n from ludwig.schema.features.preprocessing.text import TextPreprocessingConfig\n\n\nclass HFEncoderConfig(SequenceEncoderConfig):\n trainable: bool\n use_pretrained: bool\n pretrained_model_name_or_path: str\n reduce_output: str\n\n def set_fixed_preprocessing_params(self, model_type: str, preprocessing: \"TextPreprocessingConfig\"):\n model_name = self.pretrained_model_name_or_path\n if model_name is None and self.use_pretrained:\n # no default model name, so model name is required by the subclass\n raise ValueError(\n f\"Missing required parameter for `{self.type}` encoder: `pretrained_model_name_or_path` when \"\n \"`use_pretrained` is True.\"\n )\n preprocessing.tokenizer = \"hf_tokenizer\"\n preprocessing.pretrained_model_name_or_path = model_name\n if not self.can_cache_embeddings():\n preprocessing.cache_encoder_embeddings = False\n\n def is_pretrained(self) -> bool:\n return self.use_pretrained\n\n def can_cache_embeddings(self) -> bool:\n \"\"\"Returns true if the encoder's output embeddings will not change during training.\"\"\"\n return not self.trainable and self.reduce_output != \"attention\"\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass HFEncoderImplConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the base HF encoder implmenetation.\"\"\"\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"HFEncoder\"][\"use_pretrained\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"HFEncoder\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n )\n\n # Internal params set based on preprocessing metadata\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"\",\n parameter_metadata=INTERNAL_ONLY,\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=None,\n description=\"\",\n parameter_metadata=INTERNAL_ONLY,\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=(\n \"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\"\n ),\n parameter_metadata=INTERNAL_ONLY,\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"albert\", TEXT)\n@ludwig_dataclass\nclass ALBERTConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an ALBERT encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"ALBERT\"\n\n type: str = schema_utils.ProtectedString(\n \"albert\",\n description=ENCODER_METADATA[\"ALBERT\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"albert-base-v2\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"saved_weights_in_checkpoint\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n reduce_output: str = schema_utils.String(\n default=\"cls_pooled\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"reduce_output\"],\n )\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=30000,\n description=\"Vocabulary size of the ALBERT model. Defines the number of different tokens that can be \"\n \"represented by the inputs_ids passed.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"vocab_size\"],\n )\n\n embedding_size: int = schema_utils.PositiveInteger(\n default=128,\n description=\"Dimensionality of vocabulary embeddings.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"embedding_size\"],\n )\n\n hidden_size: int = schema_utils.PositiveInteger(\n default=768,\n description=\"Dimensionality of the encoder layers and the pooler layer.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"hidden_size\"],\n )\n\n num_hidden_layers: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"num_hidden_layers\"],\n )\n\n num_hidden_groups: int = schema_utils.PositiveInteger(\n default=1,\n description=\"Number of groups for the hidden layers, parameters in the same group are shared.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"num_hidden_groups\"],\n )\n\n num_attention_heads: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"num_attention_heads\"],\n )\n\n intermediate_size: int = schema_utils.PositiveInteger(\n default=3072,\n description=\"The dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer \"\n \"encoder.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"intermediate_size\"],\n )\n\n inner_group_num: int = schema_utils.PositiveInteger(\n default=1,\n description=\"The number of inner repetition of attention and ffn.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"inner_group_num\"],\n )\n\n hidden_act: str = schema_utils.StringOptions(\n [\"gelu\", \"relu\", \"silu\", \"gelu_new\"],\n default=\"gelu_new\",\n description=\"The non-linear activation function (function or string) in the encoder and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"hidden_act\"],\n )\n\n hidden_dropout_prob: float = schema_utils.FloatRange(\n default=0.0,\n min=0,\n max=1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"hidden_dropout_prob\"],\n )\n\n attention_probs_dropout_prob: float = schema_utils.FloatRange(\n default=0.0,\n min=0,\n max=1,\n description=\"The dropout ratio for the attention probabilities.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"attention_probs_dropout_prob\"],\n )\n\n max_position_embeddings: int = schema_utils.PositiveInteger(\n default=512,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"max_position_embeddings\"],\n )\n\n type_vocab_size: int = schema_utils.PositiveInteger(\n default=2,\n description=\"The vocabulary size of the token_type_ids passed when calling AlbertModel or TFAlbertModel.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"type_vocab_size\"],\n )\n\n initializer_range: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"initializer_range\"],\n )\n\n layer_norm_eps: float = schema_utils.NonNegativeFloat(\n default=1e-12,\n description=\"The epsilon used by the layer normalization layers.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"layer_norm_eps\"],\n )\n\n classifier_dropout_prob: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout ratio for attached classifiers.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"classifier_dropout_prob\"],\n )\n\n position_embedding_type: str = schema_utils.StringOptions(\n [\"absolute\", \"relative_key\", \"relative_key_query\"],\n default=\"absolute\",\n description=\"\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"position_embedding_type\"],\n )\n\n pad_token_id: int = schema_utils.Integer(\n default=0,\n description=\"The ID of the token to use as padding.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"pad_token_id\"],\n )\n\n bos_token_id: int = schema_utils.Integer(\n default=2,\n description=\"The beginning of sequence token ID.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"bos_token_id\"],\n )\n\n eos_token_id: int = schema_utils.Integer(\n default=3,\n description=\"The end of sequence token ID.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"eos_token_id\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"ALBERT\"][\"pretrained_kwargs\"],\n )\n\n\n# TODO: uncomment when sentencepiece doesn't cause segfaults: https://github.com/ludwig-ai/ludwig/issues/2983\n@DeveloperAPI\n# @register_encoder_config(\"mt5\", TEXT)\n@ludwig_dataclass\nclass MT5Config(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an MT5 encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"MT5\"\n\n type: str = schema_utils.ProtectedString(\n \"mt5\",\n description=ENCODER_METADATA[\"MT5\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"google/mt5-base\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"saved_weights_in_checkpoint\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"reduce_output\"],\n )\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=250112,\n description=\"Vocabulary size of the T5 model. Defines the number of different tokens that can be represented \"\n \"by the inputs_ids passed when calling T5Model or TFT5Model.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"vocab_size\"],\n )\n\n d_model: int = schema_utils.PositiveInteger(\n default=512,\n description=\"Size of the encoder layers and the pooler layer.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"d_model\"],\n )\n\n d_kv: int = schema_utils.PositiveInteger(\n default=64,\n description=\"Size of the key, query, value projections per attention head. d_kv has to be equal to d_model // \"\n \"num_heads.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"d_kv\"],\n )\n\n d_ff: int = schema_utils.PositiveInteger(\n default=1024,\n description=\"Size of the intermediate feed forward layer in each T5Block.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"d_ff\"],\n )\n\n num_layers: int = schema_utils.PositiveInteger(\n default=8,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"num_layers\"],\n )\n\n num_decoder_layers: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Number of hidden layers in the Transformer decoder. Will use the same value as num_layers if not \"\n \"set.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"num_decoder_layers\"],\n )\n\n num_heads: int = schema_utils.PositiveInteger(\n default=6,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"num_heads\"],\n )\n\n relative_attention_num_buckets: int = schema_utils.PositiveInteger(\n default=32,\n description=\"The number of buckets to use for each attention layer.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"relative_attention_num_buckets\"],\n )\n\n dropout_rate: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The ratio for all dropout layers.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"dropout_rate\"],\n )\n\n layer_norm_epsilon: float = schema_utils.NonNegativeFloat(\n default=1e-06,\n description=\"The epsilon used by the layer normalization layers.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"layer_norm_epsilon\"],\n )\n\n initializer_factor: float = schema_utils.NonNegativeFloat(\n default=1.0,\n description=\"A factor for initializing all weight matrices (should be kept to 1, used internally for \"\n \"initialization testing)\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"initializer_factor\"],\n )\n\n feed_forward_proj: str = schema_utils.StringOptions(\n [\"relu\", \"gated-gelu\"],\n default=\"gated-gelu\",\n description=\"Type of feed forward layer to be used. \",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"feed_forward_proj\"],\n )\n\n is_encoder_decoder: bool = schema_utils.Boolean(\n default=True,\n description=\"\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"is_encoder_decoder\"],\n )\n\n use_cache: bool = schema_utils.Boolean(\n default=True,\n description=\"\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"use_cache\"],\n )\n\n tokenizer_class: str = schema_utils.String(\n default=\"T5Tokenizer\",\n description=\"\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"tokenizer_class\"],\n )\n\n tie_word_embeddings: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether the model's input and output word embeddings should be tied.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"tie_word_embeddings\"],\n )\n\n pad_token_id: int = schema_utils.Integer(\n default=0,\n description=\"The ID of the token to use as padding.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"pad_token_id\"],\n )\n\n eos_token_id: int = schema_utils.Integer(\n default=1,\n description=\"The end of sequence token ID.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"eos_token_id\"],\n )\n\n decoder_start_token_id: int = schema_utils.Integer(\n default=0,\n description=\"If an encoder-decoder model starts decoding with a different token than _bos_, the id of that \"\n \"token.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"decoder_start_token_id\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"MT5\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"xlmroberta\", TEXT)\n@ludwig_dataclass\nclass XLMRoBERTaConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an XLMRoBERTa encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"XLMRoBERTa\"\n\n type: str = schema_utils.ProtectedString(\n \"xlmroberta\",\n description=ENCODER_METADATA[\"XLMRoBERTa\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"xlm-roberta-base\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"saved_weights_in_checkpoint\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"cls_pooled\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Vocabulary size of the XLMRoBERTa model.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"vocab_size\"],\n )\n\n pad_token_id: int = schema_utils.Integer(\n default=1,\n description=\"The ID of the token to use as padding.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"pad_token_id\"],\n )\n\n bos_token_id: int = schema_utils.Integer(\n default=0,\n description=\"The beginning of sequence token ID.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"bos_token_id\"],\n )\n\n eos_token_id: int = schema_utils.Integer(\n default=2,\n description=\"The end of sequence token ID.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"eos_token_id\"],\n )\n\n max_position_embeddings: int = schema_utils.PositiveInteger(\n default=514,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large just in case (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"max_position_embeddings\"],\n )\n\n type_vocab_size: int = schema_utils.PositiveInteger(\n default=1,\n description=\"The vocabulary size of the token_type_ids passed in.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"type_vocab_size\"],\n )\n\n add_pooling_layer: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to add a pooling layer to the encoder.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"add_pooling_layer\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"XLMRoBERTa\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"bert\", TEXT)\n@ludwig_dataclass\nclass BERTConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an BERT encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"BERT\"\n\n type: str = schema_utils.ProtectedString(\n \"bert\",\n description=ENCODER_METADATA[\"BERT\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"bert-base-uncased\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"saved_weights_in_checkpoint\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n reduce_output: str = schema_utils.String(\n default=\"cls_pooled\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"reduce_output\"],\n )\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=30522,\n description=\"Vocabulary size of the BERT model. Defines the number of different tokens that can be \"\n \"represented by the inputs_ids passed when calling BertModel or TFBertModel.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"vocab_size\"],\n )\n\n hidden_size: int = schema_utils.PositiveInteger(\n default=768,\n description=\"Dimensionality of the encoder layers and the pooler layer.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"hidden_size\"],\n )\n\n num_hidden_layers: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"num_hidden_layers\"],\n )\n\n num_attention_heads: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"num_attention_heads\"],\n )\n\n intermediate_size: int = schema_utils.PositiveInteger(\n default=3072,\n description=\"Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"intermediate_size\"],\n )\n\n hidden_act: str | Callable = schema_utils.StringOptions( # TODO: add support for callable\n [\"gelu\", \"relu\", \"silu\", \"gelu_new\"],\n default=\"gelu\",\n description=\"The non-linear activation function (function or string) in the encoder and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"hidden_act\"],\n )\n\n hidden_dropout_prob: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"hidden_dropout_prob\"],\n )\n\n attention_probs_dropout_prob: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout ratio for the attention probabilities.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"attention_probs_dropout_prob\"],\n )\n\n max_position_embeddings: int = schema_utils.PositiveInteger(\n default=512,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large just in case (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"max_position_embeddings\"],\n )\n\n type_vocab_size: int = schema_utils.PositiveInteger(\n default=2,\n description=\"The vocabulary size of the token_type_ids passed when calling BertModel or TFBertModel.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"type_vocab_size\"],\n )\n\n initializer_range: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"initializer_range\"],\n )\n\n layer_norm_eps: float = schema_utils.NonNegativeFloat(\n default=1e-12,\n description=\"The epsilon used by the layer normalization layers.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"layer_norm_eps\"],\n )\n\n pad_token_id: int = schema_utils.Integer(\n default=0,\n description=\"The ID of the token to use as padding.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"pad_token_id\"],\n )\n\n gradient_checkpointing: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to use gradient checkpointing.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"gradient_checkpointing\"],\n )\n\n position_embedding_type: str = schema_utils.StringOptions(\n [\"absolute\", \"relative_key\", \"relative_key_query\"],\n default=\"absolute\",\n description=\"Type of position embedding.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"position_embedding_type\"],\n )\n\n classifier_dropout: float = schema_utils.FloatRange(\n default=None,\n allow_none=True,\n min=0,\n max=1,\n description=\"The dropout ratio for the classification head.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"classifier_dropout\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"BERT\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"deberta\", TEXT)\n@ludwig_dataclass\nclass DebertaV2Config(HFEncoderImplConfig, DebertaModelParams):\n \"\"\"This dataclass configures the schema used for a DeBERTa-v2 / v3 encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"DeBERTa\"\n\n type: str = schema_utils.ProtectedString(\n \"deberta\",\n description=ENCODER_METADATA[\"DeBERTa\"][\"type\"].long_description,\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"tasksource/deberta-base-long-nli\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"DeBERTa\"][\"pretrained_model_name_or_path\"],\n )\n\n reduce_output: str = schema_utils.StringOptions(\n [\"cls_pooled\", \"last\", \"sum\", \"mean\", \"max\", \"concat\", \"attention\"],\n default=\"sum\",\n allow_none=True,\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n )\n\n\n# TODO: uncomment once we figure out host memory issue: https://github.com/ludwig-ai/ludwig/issues/3107\n@DeveloperAPI\n# @register_encoder_config(\"xlm\", TEXT)\n@ludwig_dataclass\nclass XLMConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an XLM encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"XLM\"\n\n type: str = schema_utils.ProtectedString(\n \"xlm\",\n description=ENCODER_METADATA[\"XLM\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"xlm-mlm-en-2048\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"saved_weights_in_checkpoint\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"reduce_output\"],\n )\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=30145,\n description=\"Vocabulary size of the BERT model. Defines the number of different tokens that can be \"\n \"represented by the inputs_ids passed when calling XLMModel or TFXLMModel.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"vocab_size\"],\n )\n\n emb_dim: int = schema_utils.PositiveInteger(\n default=2048,\n description=\"Dimensionality of the encoder layers and the pooler layer.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"emb_dim\"],\n )\n\n n_layers: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"n_layers\"],\n )\n\n n_heads: int = schema_utils.PositiveInteger(\n default=16,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"n_heads\"],\n )\n\n dropout: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"dropout\"],\n )\n\n attention_dropout: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout probability for the attention mechanism.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"attention_dropout\"],\n )\n\n gelu_activation: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether or not to use gelu for the activations instead of relu.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"gelu_activation\"],\n )\n\n sinusoidal_embeddings: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether or not to use sinusoidal positional embeddings instead of absolute positional embeddings.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"sinusoidal_embeddings\"],\n )\n\n causal: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether or not the model should behave in a causal manner. Causal models use a triangular \"\n \"attention mask in order to only attend to the left-side context instead if a bidirectional \"\n \"context.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"causal\"],\n )\n\n asm: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether or not to use an adaptive log softmax projection layer instead of a linear layer for the \"\n \"prediction layer.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"asm\"],\n )\n\n n_langs: int = schema_utils.PositiveInteger(\n default=1,\n description=\"The number of languages the model handles. Set to 1 for monolingual models.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"n_langs\"],\n )\n\n use_lang_emb: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use language embeddings. Some models use additional language embeddings, \"\n \"see the multilingual models page for information on how to use them.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"use_lang_emb\"],\n )\n\n max_position_embeddings: int = schema_utils.PositiveInteger(\n default=512,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large just in case (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"max_position_embeddings\"],\n )\n\n embed_init_std: float = schema_utils.NonNegativeFloat(\n default=2048**-0.5,\n description=\"The standard deviation of the truncated_normal_initializer for initializing the embedding \"\n \"matrices.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"embed_init_std\"],\n )\n\n layer_norm_eps: float = schema_utils.NonNegativeFloat(\n default=1e-12,\n description=\"The epsilon used by the layer normalization layers.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"layer_norm_eps\"],\n )\n\n init_std: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices \"\n \"except the embedding matrices.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"init_std\"],\n )\n\n bos_index: int = schema_utils.NonNegativeInteger(\n default=0,\n description=\"The index of the beginning of sentence token in the vocabulary.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"bos_index\"],\n )\n\n eos_index: int = schema_utils.NonNegativeInteger(\n default=1,\n description=\"The index of the end of sentence token in the vocabulary.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"eos_index\"],\n )\n\n pad_index: int = schema_utils.NonNegativeInteger(\n default=2,\n description=\"The index of the padding token in the vocabulary.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"pad_index\"],\n )\n\n unk_index: int = schema_utils.NonNegativeInteger(\n default=3,\n description=\"The index of the unknown token in the vocabulary.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"unk_index\"],\n )\n\n mask_index: int = schema_utils.NonNegativeInteger(\n default=5,\n description=\"The index of the masking token in the vocabulary.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"mask_index\"],\n )\n\n is_encoder: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether or not the initialized model should be a transformer encoder or decoder as seen in \"\n \"Vaswani et al.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"is_encoder\"],\n )\n\n start_n_top: int = schema_utils.PositiveInteger(\n default=5,\n description=\"Used in the SQuAD evaluation script.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"start_n_top\"],\n )\n\n end_n_top: int = schema_utils.PositiveInteger(\n default=5,\n description=\"Used in the SQuAD evaluation script.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"end_n_top\"],\n )\n\n mask_token_id: int = schema_utils.Integer(\n default=0,\n description=\"Model agnostic parameter to identify masked tokens when generating text in an MLM context.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"mask_token_id\"],\n )\n\n lang_id: int = schema_utils.Integer(\n default=0,\n description=\"The ID of the language used by the model. This parameter is used when generating text in a given \"\n \"language.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"lang_id\"],\n )\n\n pad_token_id: int = schema_utils.Integer(\n default=2,\n description=\"The ID of the token to use as padding.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"pad_token_id\"],\n )\n\n bos_token_id: int = schema_utils.Integer(\n default=0,\n description=\"The beginning of sequence token ID.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"bos_token_id\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"XLM\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"gpt\", TEXT)\n@ludwig_dataclass\nclass GPTConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an GPT encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"GPT\"\n\n type: str = schema_utils.ProtectedString(\n \"gpt\",\n description=ENCODER_METADATA[\"GPT\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"max_sequence_length\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"reduce_output\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"openai-gpt\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"saved_weights_in_checkpoint\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=30522,\n description=\"Vocabulary size of the GPT model. Defines the number of different tokens that can be \"\n \"represented by the inputs_ids passed when calling OpenAIGPTModel or TFOpenAIGPTModel.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"vocab_size\"],\n )\n\n n_positions: int = schema_utils.PositiveInteger(\n default=40478,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large just in case (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"n_positions\"],\n )\n\n n_ctx: int = schema_utils.PositiveInteger(\n default=512,\n description=\"Dimensionality of the causal mask (usually same as n_positions)\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"n_ctx\"],\n )\n\n n_embd: int = schema_utils.PositiveInteger(\n default=768,\n description=\"Dimensionality of the embeddings and hidden states.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"n_embd\"],\n )\n\n n_layer: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"n_layer\"],\n )\n\n n_head: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"n_head\"],\n )\n\n afn: str = schema_utils.StringOptions(\n [\"gelu\", \"relu\", \"silu\"], # gelu_new results in a KeyError.\n default=\"gelu\",\n description=\"The non-linear activation function (function or string) in the encoder and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"afn\"],\n )\n\n resid_pdrop: float = schema_utils.FloatRange(\n default=0.1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"resid_pdrop\"],\n )\n\n embd_pdrop: float = schema_utils.FloatRange(\n default=0.1,\n description=\"The dropout ratio for the embeddings.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"embd_pdrop\"],\n )\n\n attn_pdrop: float = schema_utils.FloatRange(\n default=0.1,\n description=\"The dropout ratio for the attention.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"attn_pdrop\"],\n )\n\n layer_norm_epsilon: float = schema_utils.NonNegativeFloat(\n default=1e-5,\n description=\"The epsilon to use in the layer normalization layers\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"layer_norm_epsilon\"],\n )\n\n initializer_range: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"initializer_range\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"GPT\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"gpt2\", TEXT)\n@ludwig_dataclass\nclass GPT2Config(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an GPT2 encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"GPT2\"\n\n type: str = schema_utils.ProtectedString(\n \"gpt2\",\n description=ENCODER_METADATA[\"GPT2\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"gpt2\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"pretrained_model_name_or_path\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=50257,\n description=\"Vocabulary size of the GPT-2 model. Defines the number of different tokens that can be \"\n \"represented by the inputs_ids passed when calling GPT2Model or TFGPT2Model.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"vocab_size\"],\n )\n\n n_positions: int = schema_utils.PositiveInteger(\n default=1024,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large just in case (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"n_positions\"],\n )\n\n n_ctx: int = schema_utils.PositiveInteger(\n default=1024,\n description=\"Dimensionality of the causal mask (usually same as n_positions)\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"n_ctx\"],\n )\n\n n_embd: int = schema_utils.PositiveInteger(\n default=768,\n description=\"Dimensionality of the embeddings and hidden states.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"n_embd\"],\n )\n\n n_layer: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"n_layer\"],\n )\n\n n_head: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"n_head\"],\n )\n\n n_inner: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Dimensionality of the inner feed-forward layers. None will set it to 4 times n_embd\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"n_inner\"],\n )\n\n activation_function: str = schema_utils.StringOptions(\n [\"relu\", \"silu\", \"gelu\", \"tanh\", \"gelu_new\"],\n default=\"gelu_new\",\n description=\"Activation function, to be selected in the list ['relu', 'silu', 'gelu', 'tanh', 'gelu_new'].\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"activation_function\"],\n )\n\n resid_pdrop: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"resid_pdrop\"],\n )\n\n embd_pdrop: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout ratio for the embeddings.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"embd_pdrop\"],\n )\n\n attn_pdrop: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout ratio for the attention.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"attn_pdrop\"],\n )\n\n layer_norm_epsilon: float = schema_utils.NonNegativeFloat(\n default=1e-5,\n description=\"The epsilon to use in the layer normalization layers.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"layer_norm_epsilon\"],\n )\n\n initializer_range: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"initializer_range\"],\n )\n\n scale_attn_weights: bool = schema_utils.Boolean(\n default=True,\n description=\"Scale attention weights by dividing by sqrt(hidden_size).\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"scale_attn_weights\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"GPT2\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"roberta\", TEXT)\n@ludwig_dataclass\nclass RoBERTaConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an RoBERTa encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"RoBERTa\"\n\n type: str = schema_utils.ProtectedString(\n \"roberta\",\n description=ENCODER_METADATA[\"RoBERTa\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"roberta-base\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"saved_weights_in_checkpoint\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"cls_pooled\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Vocabulary size of the RoBERTa model.\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"vocab_size\"],\n )\n\n pad_token_id: int = schema_utils.Integer(\n default=1,\n description=\"The ID of the token to use as padding.\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"pad_token_id\"],\n )\n\n bos_token_id: int = schema_utils.Integer(\n default=0,\n description=\"The beginning of sequence token ID.\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"bos_token_id\"],\n )\n\n eos_token_id: int = schema_utils.Integer(\n default=2,\n description=\"The end of sequence token ID.\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"eos_token_id\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"RoBERTa\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"transformer_xl\", TEXT)\n@ludwig_dataclass\nclass TransformerXLConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an TransformerXL encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"TransformerXL\"\n\n type: str = schema_utils.ProtectedString(\n \"transformer_xl\",\n description=ENCODER_METADATA[\"TransformerXL\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"transfo-xl-wt103\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"saved_weights_in_checkpoint\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=267735,\n description=\"Vocabulary size of the TransfoXL model. Defines the number of different tokens that can be \"\n \"represented by the inputs_ids passed when calling TransfoXLModel or TFTransfoXLModel.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"vocab_size\"],\n )\n\n cutoffs: list[int] = schema_utils.List(\n int,\n default=[20000, 40000, 200000],\n description=\"Cutoffs for the adaptive softmax.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"cutoffs\"],\n )\n\n d_model: int = schema_utils.PositiveInteger(\n default=1024,\n description=\"Dimensionality of the model’s hidden states.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"d_model\"],\n )\n\n d_embed: int = schema_utils.PositiveInteger(\n default=1024,\n description=\"Dimensionality of the embeddings\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"d_embed\"],\n )\n\n n_head: int = schema_utils.PositiveInteger(\n default=16,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"n_head\"],\n )\n\n d_head: int = schema_utils.PositiveInteger(\n default=64,\n description=\"Dimensionality of the model’s heads.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"d_head\"],\n )\n\n d_inner: int = schema_utils.PositiveInteger(\n default=4096,\n description=\" Inner dimension in FF\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"d_inner\"],\n )\n\n div_val: int = schema_utils.PositiveInteger(\n default=4,\n description=\"Divident value for adapative input and softmax.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"div_val\"],\n )\n\n pre_lnorm: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether or not to apply LayerNorm to the input instead of the output in the blocks.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"pre_lnorm\"],\n )\n\n n_layer: int = schema_utils.PositiveInteger(\n default=18,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"n_layer\"],\n )\n\n mem_len: int = schema_utils.PositiveInteger(\n default=1600,\n description=\"Length of the retained previous heads.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"mem_len\"],\n )\n\n clamp_len: int = schema_utils.PositiveInteger(\n default=1000,\n description=\"Use the same pos embeddings after clamp_len.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"clamp_len\"],\n )\n\n same_length: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether or not to use the same attn length for all tokens\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"same_length\"],\n )\n\n proj_share_all_but_first: bool = schema_utils.Boolean(\n default=True,\n description=\"True to share all but first projs, False not to share.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"proj_share_all_but_first\"],\n )\n\n attn_type: int = schema_utils.IntegerRange(\n default=0,\n min=0,\n max=3,\n description=\"Attention type. 0 for Transformer-XL, 1 for Shaw et al, 2 for Vaswani et al, 3 for Al Rfou et al.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"attn_type\"],\n )\n\n sample_softmax: int = schema_utils.Integer(\n default=-1,\n description=\"Number of samples in the sampled softmax.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"sample_softmax\"],\n )\n\n adaptive: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether or not to use adaptive softmax.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"adaptive\"],\n )\n\n dropout: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"dropout\"],\n )\n\n dropatt: float = schema_utils.NonNegativeFloat(\n default=0.0,\n description=\"The dropout ratio for the attention probabilities.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"dropatt\"],\n )\n\n untie_r: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether ot not to untie relative position biases.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"untie_r\"],\n )\n\n init: str = schema_utils.String(\n default=\"normal\",\n description=\"Parameter initializer to use.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"init\"],\n )\n\n init_range: float = schema_utils.NonNegativeFloat(\n default=0.01,\n description=\"Parameters initialized by U(-init_range, init_range).\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"init_range\"],\n )\n\n proj_init_std: float = schema_utils.NonNegativeFloat(\n default=0.01,\n description=\"Parameters initialized by N(0, init_std)\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"proj_init_std\"],\n )\n\n init_std: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"Parameters initialized by N(0, init_std)\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"init_std\"],\n )\n\n layer_norm_epsilon: float = schema_utils.NonNegativeFloat(\n default=1e-5,\n description=\"The epsilon to use in the layer normalization layers\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"layer_norm_epsilon\"],\n )\n\n eos_token_id: int = schema_utils.Integer(\n default=0,\n description=\"The end of sequence token ID.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"eos_token_id\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"TransformerXL\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"xlnet\", TEXT)\n@ludwig_dataclass\nclass XLNetConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an XLNet encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"XLNet\"\n\n type: str = schema_utils.ProtectedString(\n \"xlnet\",\n description=ENCODER_METADATA[\"XLNet\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"xlnet-base-cased\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"saved_weights_in_checkpoint\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=32000,\n description=\"Vocabulary size of the XLNet model. Defines the number of different tokens that can be \"\n \"represented by the inputs_ids passed when calling XLNetModel or TFXLNetModel.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"vocab_size\"],\n )\n\n d_model: int = schema_utils.PositiveInteger(\n default=768,\n description=\"Dimensionality of the encoder layers and the pooler layer.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"d_model\"],\n )\n\n n_layer: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"n_layer\"],\n )\n\n n_head: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"n_head\"],\n )\n\n d_inner: int = schema_utils.PositiveInteger(\n default=3072,\n description=\"Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"d_inner\"],\n )\n\n ff_activation: str = schema_utils.StringOptions(\n [\"gelu\", \"relu\", \"silu\", \"gelu_new\"],\n default=\"gelu\",\n description=\"The non-linear activation function (function or string) in the encoder and pooler. If string, \"\n \"'gelu', 'relu', 'silu' and 'gelu_new' are supported.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"ff_activation\"],\n )\n\n untie_r: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether or not to untie relative position biases\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"untie_r\"],\n )\n\n attn_type: str = schema_utils.StringOptions(\n [\"bi\"],\n default=\"bi\",\n description=\"The attention type used by the model. Currently only 'bi' is supported.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"attn_type\"],\n )\n\n initializer_range: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"initializer_range\"],\n )\n\n layer_norm_eps: float = schema_utils.NonNegativeFloat(\n default=1e-12,\n description=\"The epsilon used by the layer normalization layers.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"layer_norm_eps\"],\n )\n\n dropout: float = schema_utils.FloatRange(\n default=0.1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"dropout\"],\n )\n\n mem_len: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"The number of tokens to cache. The key/value pairs that have already been pre-computed in a \"\n \"previous forward pass won’t be re-computed. \",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"mem_len\"],\n )\n\n reuse_len: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"The number of tokens in the current batch to be cached and reused in the future.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"reuse_len\"],\n )\n\n use_mems_eval: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether or not the model should make use of the recurrent memory mechanism in evaluation mode.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"use_mems_eval\"],\n )\n\n use_mems_train: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether or not the model should make use of the recurrent memory mechanism in train mode.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"use_mems_train\"],\n )\n\n bi_data: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether or not to use bidirectional input pipeline. Usually set to True during pretraining and \"\n \"False during finetuning.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"bi_data\"],\n )\n\n clamp_len: int = schema_utils.Integer(\n default=-1,\n description=\"Clamp all relative distances larger than clamp_len. Setting this attribute to -1 means no \"\n \"clamping.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"clamp_len\"],\n )\n\n same_length: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether or not to use the same attention length for each token.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"same_length\"],\n )\n\n summary_type: str = schema_utils.StringOptions(\n [\"last\", \"first\", \"mean\", \"cls_index\", \"attn\"],\n default=\"last\",\n description=\"Argument used when doing sequence summary. Used in the sequence classification and multiple \"\n \"choice models.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"summary_type\"],\n )\n\n summary_use_proj: bool = schema_utils.Boolean(\n default=True,\n description=\"\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"summary_use_proj\"],\n )\n\n summary_activation: str = schema_utils.String(\n default=\"tanh\",\n description=\"Argument used when doing sequence summary. Used in the sequence classification and multiple \"\n \"choice models.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"summary_activation\"],\n )\n\n summary_last_dropout: float = schema_utils.FloatRange(\n default=0.1,\n description=\"Used in the sequence classification and multiple choice models.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"summary_last_dropout\"],\n )\n\n start_n_top: int = schema_utils.PositiveInteger(\n default=5,\n description=\"Used in the SQuAD evaluation script.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"start_n_top\"],\n )\n\n end_n_top: int = schema_utils.PositiveInteger(\n default=5,\n description=\" Used in the SQuAD evaluation script.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"end_n_top\"],\n )\n\n pad_token_id: int = schema_utils.Integer(\n default=5,\n description=\"The ID of the token to use as padding.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"pad_token_id\"],\n )\n\n bos_token_id: int = schema_utils.Integer(\n default=1,\n description=\"The beginning of sequence token ID.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"bos_token_id\"],\n )\n\n eos_token_id: int = schema_utils.Integer(\n default=2,\n description=\"The end of sequence token ID.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"eos_token_id\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"XLNet\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"distilbert\", TEXT)\n@ludwig_dataclass\nclass DistilBERTConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an DistilBERT encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"DistilBERT\"\n\n type: str = schema_utils.ProtectedString(\n \"distilbert\",\n description=ENCODER_METADATA[\"DistilBERT\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"distilbert-base-uncased\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"saved_weights_in_checkpoint\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=30522,\n description=\"Vocabulary size of the DistilBERT model. Defines the number of different tokens that can be \"\n \"represented by the inputs_ids passed when calling DistilBertModel or TFDistilBertModel.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"vocab_size\"],\n )\n\n dropout: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"dropout\"],\n )\n\n max_position_embeddings: int = schema_utils.PositiveInteger(\n default=512,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large just in case (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"max_position_embeddings\"],\n )\n\n sinusoidal_pos_embds: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to use sinusoidal positional embeddings.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"sinusoidal_pos_embds\"],\n )\n\n n_layers: int = schema_utils.PositiveInteger(\n default=6,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"n_layers\"],\n )\n\n n_heads: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"n_heads\"],\n )\n\n dim: int = schema_utils.PositiveInteger(\n default=768,\n description=\" Dimensionality of the encoder layers and the pooler layer.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"dim\"],\n )\n\n hidden_dim: int = schema_utils.PositiveInteger(\n default=3072,\n description=\"The size of the “intermediate” (often named feed-forward) layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"hidden_dim\"],\n )\n\n attention_dropout: float = schema_utils.NonNegativeFloat(\n default=0.1,\n description=\"The dropout ratio for the attention probabilities.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"attention_dropout\"],\n )\n\n activation: str | Callable = schema_utils.StringOptions( # TODO: Add support for callable\n [\"gelu\", \"relu\", \"silu\", \"gelu_new\"],\n default=\"gelu\",\n description=\"The non-linear activation function (function or string) in the encoder and pooler. If string, \"\n \"'gelu', 'relu', 'silu' and 'gelu_new' are supported.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"activation\"],\n )\n\n initializer_range: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"initializer_range\"],\n )\n\n qa_dropout: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout probabilities used in the question answering model DistilBertForQuestionAnswering.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"qa_dropout\"],\n )\n\n seq_classif_dropout: float = schema_utils.FloatRange(\n default=0.2,\n min=0,\n max=1,\n description=\"The dropout probabilities used in the sequence classification and the multiple choice model \"\n \"DistilBertForSequenceClassification.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"seq_classif_dropout\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"DistilBERT\"][\"pretrained_kwargs\"],\n )\n\n\n# TODO: uncomment when CTRL bug (https://github.com/ludwig-ai/ludwig/issues/2977) has been fixed to add back in\n@DeveloperAPI\n# @register_encoder_config(\"ctrl\", TEXT)\n@ludwig_dataclass\nclass CTRLConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an CTRL encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"CTRL\"\n\n type: str = schema_utils.ProtectedString(\n \"ctrl\",\n description=ENCODER_METADATA[\"CTRL\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"ctrl\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"saved_weights_in_checkpoint\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=246534,\n description=\"Vocabulary size of the CTRL model. Defines the number of different tokens that can be \"\n \"represented by the inputs_ids passed when calling CTRLModel or TFCTRLModel.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"vocab_size\"],\n )\n\n n_positions: int = schema_utils.PositiveInteger(\n default=256,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large just in case (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"n_positions\"],\n )\n\n n_ctx: int = schema_utils.PositiveInteger(\n default=256,\n description=\"Dimensionality of the causal mask (usually same as n_positions)\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"n_ctx\"],\n )\n\n n_embd: int = schema_utils.PositiveInteger(\n default=1280,\n description=\"Dimensionality of the embeddings and hidden states.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"n_embd\"],\n )\n\n dff: int = schema_utils.PositiveInteger(\n default=8192,\n description=\"Dimensionality of the inner dimension of the feed forward networks (FFN).\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"dff\"],\n )\n\n n_layer: int = schema_utils.PositiveInteger(\n default=48,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"n_layer\"],\n )\n\n n_head: int = schema_utils.PositiveInteger(\n default=16,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"n_head\"],\n )\n\n resid_pdrop: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\" The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"resid_pdrop\"],\n )\n\n embd_pdrop: float = schema_utils.NonNegativeFloat(\n default=0.1,\n description=\"The dropout ratio for the embeddings.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"embd_pdrop\"],\n )\n\n attn_pdrop: float = schema_utils.NonNegativeFloat(\n default=0.1,\n description=\"The dropout ratio for the attention.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"attn_pdrop\"],\n )\n\n layer_norm_epsilon: float = schema_utils.NonNegativeFloat(\n default=1e-6,\n description=\"The epsilon to use in the layer normalization layers\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"layer_norm_epsilon\"],\n )\n\n initializer_range: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"initializer_range\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"CTRL\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"camembert\", TEXT)\n@ludwig_dataclass\nclass CamemBERTConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an CamemBERT encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"CamemBERT\"\n\n type: str = schema_utils.ProtectedString(\n \"camembert\",\n description=ENCODER_METADATA[\"CamemBERT\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"use_pretrained\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"saved_weights_in_checkpoint\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"camembert-base\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"pretrained_model_name_or_path\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=32005,\n description=\"Vocabulary size of the CamemBERT model.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"vocab_size\"],\n )\n\n hidden_size: int = schema_utils.PositiveInteger(\n default=768,\n description=\"Dimensionality of the encoder layers and the pooler layer.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"hidden_size\"],\n )\n\n num_hidden_layers: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"num_hidden_layers\"],\n )\n\n num_attention_heads: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"num_attention_heads\"],\n )\n\n intermediate_size: int = schema_utils.PositiveInteger(\n default=3072,\n description=\"Dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"intermediate_size\"],\n )\n\n hidden_act: str | Callable = schema_utils.StringOptions( # TODO: add support for callable\n [\"gelu\", \"relu\", \"silu\", \"gelu_new\"],\n default=\"gelu\",\n description=\"The non-linear activation function (function or string) in the encoder and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"hidden_act\"],\n )\n\n hidden_dropout_prob: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"hidden_dropout_prob\"],\n )\n\n attention_probs_dropout_prob: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout ratio for the attention probabilities.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"attention_probs_dropout_prob\"],\n )\n\n max_position_embeddings: int = schema_utils.PositiveInteger(\n default=514,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large just in case (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"max_position_embeddings\"],\n )\n\n type_vocab_size: int = schema_utils.PositiveInteger(\n default=1,\n description=\"The vocabulary size of the token_type_ids passed when calling BertModel or TFBertModel.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"type_vocab_size\"],\n )\n\n initializer_range: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"initializer_range\"],\n )\n\n layer_norm_eps: float = schema_utils.NonNegativeFloat(\n default=1e-05,\n description=\"The epsilon used by the layer normalization layers.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"layer_norm_eps\"],\n )\n\n pad_token_id: int = schema_utils.Integer(\n default=1,\n description=\"The ID of the token to use as padding.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"pad_token_id\"],\n )\n\n gradient_checkpointing: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to use gradient checkpointing.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"gradient_checkpointing\"],\n )\n\n position_embedding_type: str = schema_utils.StringOptions(\n [\"absolute\", \"relative_key\", \"relative_key_query\"],\n default=\"absolute\",\n description=\"Type of position embedding.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"position_embedding_type\"],\n )\n\n classifier_dropout: float = schema_utils.FloatRange(\n default=None,\n allow_none=True,\n min=0,\n max=1,\n description=\"The dropout ratio for the classification head.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"classifier_dropout\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"CamemBERT\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"t5\", TEXT)\n@ludwig_dataclass\nclass T5Config(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an T5 encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"T5\"\n\n type: str = schema_utils.ProtectedString(\n \"t5\",\n description=ENCODER_METADATA[\"T5\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"t5-small\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"saved_weights_in_checkpoint\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=32128,\n description=\"Vocabulary size of the T5 model. Defines the number of different tokens that can be represented \"\n \"by the inputs_ids passed when calling T5Model or TFT5Model.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"vocab_size\"],\n )\n\n d_model: int = schema_utils.PositiveInteger(\n default=512,\n description=\"Size of the encoder layers and the pooler layer.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"d_model\"],\n )\n\n d_kv: int = schema_utils.PositiveInteger(\n default=64,\n description=\"Size of the key, query, value projections per attention head. d_kv has to be equal to d_model // \"\n \"num_heads.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"d_kv\"],\n )\n\n d_ff: int = schema_utils.PositiveInteger(\n default=2048,\n description=\"Size of the intermediate feed forward layer in each T5Block.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"d_ff\"],\n )\n\n num_layers: int = schema_utils.PositiveInteger(\n default=6,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"num_layers\"],\n )\n\n num_decoder_layers: int = schema_utils.PositiveInteger(\n default=6,\n description=\"Number of hidden layers in the Transformer decoder. Will use the same value as num_layers if not \"\n \"set.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"num_decoder_layers\"],\n )\n\n num_heads: int = schema_utils.PositiveInteger(\n default=8,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"num_heads\"],\n )\n\n relative_attention_num_buckets: int = schema_utils.PositiveInteger(\n default=32,\n description=\"The number of buckets to use for each attention layer.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"relative_attention_num_buckets\"],\n )\n\n dropout_rate: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The ratio for all dropout layers.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"dropout_rate\"],\n )\n\n layer_norm_eps: float = schema_utils.NonNegativeFloat(\n default=1e-6,\n description=\"The epsilon used by the layer normalization layers.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"layer_norm_eps\"],\n )\n\n initializer_factor: float = schema_utils.NonNegativeFloat(\n default=1,\n description=\"A factor for initializing all weight matrices (should be kept to 1, used internally for \"\n \"initialization testing).\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"initializer_factor\"],\n )\n\n feed_forward_proj: str = schema_utils.StringOptions(\n [\"relu\", \"gated-gelu\"],\n default=\"relu\",\n description=\"Type of feed forward layer to be used. Should be one of 'relu' or 'gated-gelu'. T5v1.1 uses the \"\n \"'gated-gelu' feed forward projection. Original T5 uses 'relu'.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"feed_forward_proj\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"T5\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"flaubert\", TEXT)\n@ludwig_dataclass\nclass FlauBERTConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an FlauBERT encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"FlauBERT\"\n\n type: str = schema_utils.ProtectedString(\n \"flaubert\",\n description=ENCODER_METADATA[\"FlauBERT\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"flaubert/flaubert_small_cased\",\n description=\"Name of path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"saved_weights_in_checkpoint\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=30145,\n description=\"Vocabulary size of the FlauBERT model. Defines the number of different tokens that can be \"\n \"represented by the inputs_ids passed when calling FlaubertModel or TFFlaubertModel.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"vocab_size\"],\n )\n\n pre_norm: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to apply the layer normalization before or after the feed forward layer following the \"\n \"attention in each layer (Vaswani et al., Tensor2Tensor for Neural Machine Translation. 2018)\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"pre_norm\"],\n )\n\n layerdrop: float = schema_utils.FloatRange(\n default=0.2,\n min=0,\n max=1,\n description=\"Probability to drop layers during training (Fan et al., Reducing Transformer Depth on Demand \"\n \"with Structured Dropout. ICLR 2020)\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"layerdrop\"],\n )\n\n emb_dim: int = schema_utils.PositiveInteger(\n default=512,\n description=\"Dimensionality of the encoder layers and the pooler layer.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"emb_dim\"],\n )\n\n n_layers: int = schema_utils.PositiveInteger(\n default=6,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"n_layers\"],\n )\n\n n_heads: int = schema_utils.PositiveInteger(\n default=8,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"n_heads\"],\n )\n\n dropout: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"dropout\"],\n )\n\n attention_dropout: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout probability for the attention mechanism\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"attention_dropout\"],\n )\n\n gelu_activation: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether or not to use a gelu activation instead of relu.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"gelu_activation\"],\n )\n\n sinusoidal_embeddings: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether or not to use sinusoidal positional embeddings instead of absolute positional embeddings.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"sinusoidal_embeddings\"],\n )\n\n causal: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether or not the model should behave in a causal manner. Causal models use a triangular \"\n \"attention mask in order to only attend to the left-side context instead if a bidirectional \"\n \"context.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"causal\"],\n )\n\n asm: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether or not to use an adaptive log softmax projection layer instead of a linear layer for the \"\n \"prediction layer.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"asm\"],\n )\n\n n_langs: int = schema_utils.PositiveInteger(\n default=1,\n description=\"The number of languages the model handles. Set to 1 for monolingual models.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"n_langs\"],\n )\n\n use_lang_emb: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use language embeddings. Some models use additional language embeddings, \"\n \"see the multilingual models page for information on how to use them.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"use_lang_emb\"],\n )\n\n max_position_embeddings: int = schema_utils.PositiveInteger(\n default=512,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large just in case (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"max_position_embeddings\"],\n )\n\n embed_init_std: float = schema_utils.NonNegativeFloat(\n default=2048**-0.5,\n description=\"The standard deviation of the truncated_normal_initializer for initializing the embedding \"\n \"matrices.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"embed_init_std\"],\n )\n\n init_std: int = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices \"\n \"except the embedding matrices.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"init_std\"],\n )\n\n layer_norm_eps: float = schema_utils.NonNegativeFloat(\n default=1e-06,\n description=\"The epsilon used by the layer normalization layers.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"layer_norm_eps\"],\n )\n\n bos_index: int = schema_utils.NonNegativeInteger(\n default=0,\n description=\"The index of the beginning of sentence token in the vocabulary.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"bos_index\"],\n )\n\n eos_index: int = schema_utils.NonNegativeInteger(\n default=1,\n description=\"The index of the end of sentence token in the vocabulary.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"eos_index\"],\n )\n\n pad_index: int = schema_utils.NonNegativeInteger(\n default=2,\n description=\"The index of the padding token in the vocabulary.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"pad_index\"],\n )\n\n unk_index: int = schema_utils.NonNegativeInteger(\n default=3,\n description=\"The index of the unknown token in the vocabulary.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"unk_index\"],\n )\n\n mask_index: int = schema_utils.NonNegativeInteger(\n default=5,\n description=\"The index of the masking token in the vocabulary.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"mask_index\"],\n )\n\n is_encoder: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether or not the initialized model should be a transformer encoder or decoder as seen in \"\n \"Vaswani et al.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"is_encoder\"],\n )\n\n mask_token_id: int = schema_utils.Integer(\n default=0,\n description=\"Model agnostic parameter to identify masked tokens when generating text in an MLM context.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"mask_token_id\"],\n )\n\n lang_id: int = schema_utils.Integer(\n default=0,\n description=\"The ID of the language used by the model. This parameter is used when generating text in a given \"\n \"language.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"lang_id\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"FlauBERT\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"electra\", TEXT)\n@ludwig_dataclass\nclass ELECTRAConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an ELECTRA encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"ELECTRA\"\n\n type: str = schema_utils.ProtectedString(\n \"electra\",\n description=ENCODER_METADATA[\"ELECTRA\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"use_pretrained\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"google/electra-small-discriminator\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"saved_weights_in_checkpoint\"],\n )\n\n reduce_output: str = schema_utils.String(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=30522,\n description=\"Vocabulary size of the ELECTRA model. Defines the number of different tokens that can be \"\n \"represented by the inputs_ids passed when calling ElectraModel or TFElectraModel.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"vocab_size\"],\n )\n\n embedding_size: int = schema_utils.PositiveInteger(\n default=128,\n description=\"Dimensionality of the encoder layers and the pooler layer.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"embedding_size\"],\n )\n\n hidden_size: int = schema_utils.PositiveInteger(\n default=256,\n description=\"Dimensionality of the encoder layers and the pooler layer.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"hidden_size\"],\n )\n\n num_hidden_layers: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Number of hidden layers in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"num_hidden_layers\"],\n )\n\n num_attention_heads: int = schema_utils.PositiveInteger(\n default=4,\n description=\"Number of attention heads for each attention layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"num_attention_heads\"],\n )\n\n intermediate_size: int = schema_utils.PositiveInteger(\n default=1024,\n description=\"Dimensionality of the “intermediate” (i.e., feed-forward) layer in the Transformer encoder.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"intermediate_size\"],\n )\n\n hidden_act: str | Callable = schema_utils.StringOptions( # TODO: add support for callable\n [\"gelu\", \"relu\", \"silu\", \"gelu_new\"],\n default=\"gelu\",\n description=\"The non-linear activation function (function or string) in the encoder and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"hidden_act\"],\n )\n\n hidden_dropout_prob: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"hidden_dropout_prob\"],\n )\n\n attention_probs_dropout_prob: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"The dropout ratio for the attention probabilities.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"attention_probs_dropout_prob\"],\n )\n\n max_position_embeddings: int = schema_utils.PositiveInteger(\n default=512,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large just in case (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"max_position_embeddings\"],\n )\n\n type_vocab_size: int = schema_utils.PositiveInteger(\n default=2,\n description=\"The vocabulary size of the token_type_ids passed when calling ElectraModel or TFElectraModel.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"type_vocab_size\"],\n )\n\n initializer_range: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"The standard deviation of the truncated_normal_initializer for initializing all weight matrices.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"initializer_range\"],\n )\n\n layer_norm_eps: float = schema_utils.NonNegativeFloat(\n default=1e-12,\n description=\"The epsilon used by the layer normalization layers.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"layer_norm_eps\"],\n )\n\n position_embedding_type: str = schema_utils.StringOptions(\n [\"absolute\", \"relative_key\", \"relative_key_query\"],\n default=\"absolute\",\n description=\"Type of position embedding.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"position_embedding_type\"],\n )\n\n classifier_dropout: float = schema_utils.FloatRange(\n default=None,\n allow_none=True,\n min=0,\n max=1,\n description=\"The dropout ratio for the classification head.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"classifier_dropout\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"ELECTRA\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"longformer\", TEXT)\n@ludwig_dataclass\nclass LongformerConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for a Longformer encoder.\"\"\"\n\n @staticmethod\n def module_name():\n return \"Longformer\"\n\n type: str = schema_utils.ProtectedString(\n \"longformer\",\n description=ENCODER_METADATA[\"Longformer\"][\"type\"].long_description,\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"max_sequence_length\"],\n )\n\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. If false, the model will train from \"\n \"scratch which is very computationally expensive.\",\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"use_pretrained\"],\n )\n\n attention_window: list[int] | int = schema_utils.OneOfOptionsField(\n default=512,\n allow_none=False,\n description=\"Size of an attention window around each token. If an int, use the same size for all layers. To \"\n \"specify a different window size for each layer, use a List[int] where len(attention_window) == \"\n \"num_hidden_layers.\",\n field_options=[\n schema_utils.PositiveInteger(allow_none=False, description=\"\", default=512),\n schema_utils.List(list_type=int, allow_none=False),\n ],\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"attention_window\"],\n )\n\n sep_token_id: int = schema_utils.Integer(\n default=2,\n description=\"ID of the separator token, which is used when building a sequence from multiple sequences\",\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"sep_token_id\"],\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=\"allenai/longformer-base-4096\",\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"pretrained_model_name_or_path\"],\n )\n\n saved_weights_in_checkpoint: bool = schema_utils.Boolean(\n default=False,\n description=\"Are the pretrained encoder weights saved in this model's checkpoint? Automatically set to\"\n \"True for trained models to prevent loading pretrained encoder weights from model hub.\",\n parameter_metadata=ParameterMetadata(internal_only=True),\n )\n\n reduce_output: str = schema_utils.String(\n default=\"cls_pooled\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=50265,\n description=\"Vocabulary size of the Longformer model.\",\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"vocab_size\"],\n )\n\n max_position_embeddings: int = schema_utils.PositiveInteger(\n default=4098,\n description=\"The maximum sequence length that this model might ever be used with. Typically set this to \"\n \"something large just in case (e.g., 512 or 1024 or 2048).\",\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"max_position_embeddings\"],\n )\n\n type_vocab_size: int = schema_utils.PositiveInteger(\n default=1,\n description=\"The vocabulary size of the token_type_ids passed when calling LongformerEncoder\",\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"type_vocab_size\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"Longformer\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"auto_transformer\", TEXT)\n@ludwig_dataclass\nclass AutoTransformerConfig(HFEncoderConfig):\n \"\"\"This dataclass configures the schema used for an AutoTransformer encoder.\"\"\"\n\n def __post_init__(self):\n # Always force use_pretrained=True — we don't support training from scratch for AutoTransformers\n self.use_pretrained = True\n if self.pretrained_model_name_or_path is None:\n raise ConfigValidationError(\n \"`pretrained_model_name_or_path` must be specified for encoder: `auto_transformer`.\"\n )\n\n @staticmethod\n def module_name():\n return \"AutoTransformer\"\n\n # Always True — we don't support training from scratch for AutoTransformers\n use_pretrained: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the pretrained weights for the model. Always True for AutoTransformers.\",\n )\n\n type: str = schema_utils.ProtectedString(\n \"auto_transformer\",\n description=ENCODER_METADATA[\"AutoTransformer\"][\"type\"].long_description,\n )\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"Name or path of the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"AutoTransformer\"][\"pretrained_model_name_or_path\"],\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Maximum length of the input sequence.\",\n parameter_metadata=ENCODER_METADATA[\"AutoTransformer\"][\"max_sequence_length\"],\n )\n\n reduce_output: str = schema_utils.ReductionOptions(\n default=\"sum\",\n description=\"The method used to reduce a sequence of tensors down to a single tensor.\",\n parameter_metadata=ENCODER_METADATA[\"AutoTransformer\"][\"reduce_output\"],\n )\n\n trainable: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to finetune the model on your dataset.\",\n parameter_metadata=ENCODER_METADATA[\"AutoTransformer\"][\"trainable\"],\n )\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n\n vocab: list = schema_utils.List(\n default=None,\n description=\"Vocabulary for the encoder\",\n parameter_metadata=ENCODER_METADATA[\"AutoTransformer\"][\"vocab\"],\n )\n\n vocab_size: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=(\n \"Vocabulary size of the AutoTransformer model. If None, the vocab size will be inferred \"\n \"from the given pretrained model\"\n ),\n parameter_metadata=ENCODER_METADATA[\"AutoTransformer\"][\"vocab_size\"],\n )\n\n pretrained_kwargs: dict = schema_utils.Dict(\n default=None,\n description=\"Additional kwargs to pass to the pretrained model.\",\n parameter_metadata=ENCODER_METADATA[\"AutoTransformer\"][\"pretrained_kwargs\"],\n )\n\n\n@DeveloperAPI\n@register_encoder_config(\"tf_idf\", TEXT, model_types=[MODEL_ECD])\n@ludwig_dataclass\nclass TfIdfEncoderConfig(SequenceEncoderConfig):\n type: str = schema_utils.ProtectedString(\"tf_idf\")\n\n max_sequence_length: int = schema_utils.Integer(default=None, allow_none=True, parameter_metadata=INTERNAL_ONLY)\n\n str2idf: dict[str, int] = schema_utils.Dict(parameter_metadata=INTERNAL_ONLY)\n\n vocab: list = schema_utils.List(default=None, parameter_metadata=INTERNAL_ONLY)\n\n vocab_size: int = schema_utils.Integer(default=None, allow_none=True, parameter_metadata=INTERNAL_ONLY)\n\n def set_fixed_preprocessing_params(self, model_type: str, preprocessing: \"TextPreprocessingConfig\"):\n preprocessing.compute_idf = True\n\n def can_cache_embeddings(self) -> bool:\n return True\n\n\n@DeveloperAPI\n@register_encoder_config(\"llm\", TEXT, model_types=[MODEL_ECD])\n@ludwig_dataclass\nclass LLMEncoderConfig(SequenceEncoderConfig):\n type: str = schema_utils.ProtectedString(\"llm\")\n base_model: str = BaseModelDataclassField()\n max_sequence_length: int = schema_utils.Integer(default=None, allow_none=True, parameter_metadata=INTERNAL_ONLY)\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n quantization: QuantizationConfig | None = QuantizationConfigField().get_default_field()\n model_parameters: ModelParametersConfig | None = ModelParametersConfigField().get_default_field()\n"
},
{
"path": "ludwig/schema/encoders/utils.py",
"content": "from dataclasses import Field\nfrom typing import Any, TYPE_CHECKING\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import MODEL_ECD, TYPE\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.metadata import ENCODER_METADATA\nfrom ludwig.schema.metadata.parameter_metadata import convert_metadata_to_json\nfrom ludwig.utils.registry import Registry\n\nif TYPE_CHECKING:\n from ludwig.schema.encoders.base import BaseEncoderConfig\n\n\nencoder_config_registry = Registry()\n\n\n@DeveloperAPI\ndef register_encoder_config(name: str, features: str | list[str], model_types: list[str] | None = None):\n if model_types is None:\n model_types = [MODEL_ECD]\n\n if isinstance(features, str):\n features = [features]\n\n def wrap(cls):\n for model_type in model_types:\n for feature in features:\n key = (model_type, feature)\n feature_registry = encoder_config_registry.get(key, {})\n feature_registry[name] = cls\n encoder_config_registry[key] = feature_registry\n return cls\n\n return wrap\n\n\n@DeveloperAPI\ndef get_encoder_cls(model_type: str, feature: str, name: str):\n return encoder_config_registry[(model_type, feature)][name]\n\n\n@DeveloperAPI\ndef get_encoder_classes(model_type: str, feature: str) -> dict[str, type[\"BaseEncoderConfig\"]]:\n return encoder_config_registry[(model_type, feature)]\n\n\n@DeveloperAPI\ndef get_encoder_descriptions(model_type: str, feature_type: str) -> dict[str, Any]:\n \"\"\"This function returns a dictionary of encoder descriptions available at the type selection.\n\n The process works as follows - 1) Get a dictionary of valid encoders from the encoder config registry,\n but inverse the key/value pairs since we need to index `valid_encoders` later with an altered version\n of the encoder config class name. 2) Loop through Encoder Metadata entries, if a metadata entry has an\n encoder name that matches a valid encoder, add the description metadata to the output dictionary.\n\n Args:\n model_type (str): The model type to get encoder descriptions for\n feature_type (str): The feature type to get encoder descriptions for\n Returns:\n dict: A dictionary mapping encoder registered names to their respective description metadata.\n \"\"\"\n output = {}\n valid_encoders = {\n cls.module_name() if hasattr(cls, \"module_name\") else None: registered_name\n for registered_name, cls in get_encoder_classes(model_type, feature_type).items()\n }\n\n for k, v in ENCODER_METADATA.items():\n if k in valid_encoders.keys():\n output[valid_encoders[k]] = convert_metadata_to_json(v[TYPE])\n\n return output\n\n\n@DeveloperAPI\ndef get_encoder_conds(encoder_classes: dict[str, type[\"BaseEncoderConfig\"]]) -> list[dict[str, Any]]:\n \"\"\"Returns a JSON schema of conditionals to validate against encoder types for specific feature types.\"\"\"\n conds = []\n for encoder_type, encoder_cls in encoder_classes.items():\n other_props = schema_utils.unload_jsonschema_from_marshmallow_class(encoder_cls)[\"properties\"]\n schema_utils.remove_duplicate_fields(other_props)\n encoder_cond = schema_utils.create_cond(\n {\"type\": encoder_type},\n other_props,\n )\n conds.append(encoder_cond)\n return conds\n\n\n@DeveloperAPI\ndef EncoderDataclassField(\n model_type: str, feature_type: str, default: str, description: str = \"\", blocklist: list[str] = []\n) -> Field:\n \"\"\"Custom dataclass field that when used inside a dataclass will allow the user to specify an encoder config.\n\n Returns: Initialized dataclass field that converts an untyped dict with params to an encoder config.\n \"\"\"\n encoder_registry = get_encoder_classes(model_type, feature_type)\n\n class EncoderSelection(schema_utils.TypeSelection):\n def __init__(self):\n super().__init__(\n registry=encoder_registry, default_value=default, description=description, allow_str_value=True\n )\n\n def get_schema_from_registry(self, key: str) -> type[schema_utils.BaseMarshmallowConfig]:\n return encoder_registry[key]\n\n def _jsonschema_type_mapping(self):\n # NOTE: Edit carefully if necessary! We want these enums to remain in a consistent order, so do not use sets\n # or other unordered data structures to chaperone the registry keys around.\n #\n # Also, note the placement inside this function - since this is a list, it will not update with any late\n # additions to the registry (e.g. in our tests)!\n enum = [e for e in encoder_registry.keys() if e not in blocklist]\n\n return {\n \"type\": \"object\",\n \"properties\": {\n \"type\": {\n \"type\": \"string\",\n \"enum\": enum,\n \"enumDescriptions\": get_encoder_descriptions(model_type, feature_type),\n \"default\": default,\n },\n },\n \"title\": \"encoder_options\",\n \"allOf\": get_encoder_conds(encoder_registry),\n }\n\n return EncoderSelection().get_default_field()\n"
},
{
"path": "ludwig/schema/export_schema.py",
"content": "\"\"\"Export Ludwig config JSON schema.\n\nUsage:\n python -m ludwig.schema.export_schema [--model-type ecd|llm|combined] [--output FILE]\n ludwig export_schema [--model-type ecd|llm|combined] [--output FILE]\n\nGenerates a JSON Schema (Draft 7) for Ludwig config validation.\n\"\"\"\n\nimport argparse\nimport json\n\nfrom ludwig.config_validation.validation import get_schema\nfrom ludwig.constants import MODEL_ECD, MODEL_LLM\nfrom ludwig.globals import LUDWIG_VERSION\n\nSCHEMA_BASE_URL = \"https://ludwig-ai.github.io/schema\"\n\n\ndef _strip_parameter_metadata(obj):\n \"\"\"Recursively remove ``parameter_metadata`` keys from a schema dict.\n\n The Ludwig schema generator attaches ``parameter_metadata`` objects to\n every field (UI display hints, suggested values, etc.). These are useful\n internally but add significant bloat to the published JSON Schema and are\n not relevant for validation or IDE auto-complete.\n \"\"\"\n if isinstance(obj, dict):\n return {k: _strip_parameter_metadata(v) for k, v in obj.items() if k != \"parameter_metadata\"}\n if isinstance(obj, list):\n return [_strip_parameter_metadata(item) for item in obj]\n return obj\n\n\ndef export_schema(model_type: str = MODEL_ECD, *, strip_metadata: bool = True) -> dict:\n \"\"\"Export the full Ludwig config JSON schema for a given model type.\"\"\"\n schema = get_schema(model_type)\n schema[\"$schema\"] = \"http://json-schema.org/draft-07/schema#\"\n schema[\"$id\"] = f\"{SCHEMA_BASE_URL}/ludwig-config-{model_type}.json\"\n schema[\"title\"] = f\"Ludwig {model_type.upper()} Configuration\"\n schema[\"description\"] = f\"Configuration schema for Ludwig {model_type.upper()} models (v{LUDWIG_VERSION})\"\n if strip_metadata:\n schema = _strip_parameter_metadata(schema)\n return schema\n\n\ndef export_combined_schema(*, strip_metadata: bool = True) -> dict:\n \"\"\"Export a combined schema that covers both ECD and LLM model types.\"\"\"\n ecd_schema = get_schema(MODEL_ECD)\n llm_schema = get_schema(MODEL_LLM)\n\n # Merge properties from both schemas\n all_properties = {}\n all_properties.update(ecd_schema.get(\"properties\", {}))\n all_properties.update(llm_schema.get(\"properties\", {}))\n\n combined = {\n \"$schema\": \"http://json-schema.org/draft-07/schema#\",\n \"$id\": f\"{SCHEMA_BASE_URL}/ludwig-config.json\",\n \"title\": \"Ludwig Configuration\",\n \"description\": f\"Configuration schema for Ludwig models (v{LUDWIG_VERSION})\",\n \"type\": \"object\",\n \"properties\": all_properties,\n \"required\": [\"input_features\", \"output_features\"],\n \"additionalProperties\": True,\n }\n if strip_metadata:\n combined = _strip_parameter_metadata(combined)\n return combined\n\n\ndef main(sys_argv=None):\n parser = argparse.ArgumentParser(description=\"Export Ludwig config JSON schema\")\n parser.add_argument(\n \"--model-type\",\n choices=[MODEL_ECD, MODEL_LLM, \"combined\"],\n default=\"combined\",\n help=\"Model type to export schema for (default: combined)\",\n )\n parser.add_argument(\"--output\", \"-o\", type=str, default=None, help=\"Output file (default: stdout)\")\n parser.add_argument(\n \"--full\",\n action=\"store_true\",\n help=\"Include parameter_metadata in the output (default: stripped)\",\n )\n args = parser.parse_args(sys_argv)\n\n strip_metadata = not args.full\n\n if args.model_type == \"combined\":\n schema = export_combined_schema(strip_metadata=strip_metadata)\n else:\n schema = export_schema(args.model_type, strip_metadata=strip_metadata)\n\n output = json.dumps(schema, indent=2, sort_keys=False)\n\n if args.output:\n with open(args.output, \"w\") as f:\n f.write(output)\n f.write(\"\\n\")\n else:\n print(output)\n\n\nif __name__ == \"__main__\":\n main()\n"
},
{
"path": "ludwig/schema/features/__init__.py",
"content": "import ludwig.schema.features.audio_feature\nimport ludwig.schema.features.bag_feature\nimport ludwig.schema.features.binary_feature\nimport ludwig.schema.features.category_feature\nimport ludwig.schema.features.date_feature\nimport ludwig.schema.features.h3_feature\nimport ludwig.schema.features.image_feature\nimport ludwig.schema.features.number_feature\nimport ludwig.schema.features.sequence_feature\nimport ludwig.schema.features.set_feature\nimport ludwig.schema.features.text_feature\nimport ludwig.schema.features.timeseries_feature\nimport ludwig.schema.features.vector_feature # noqa\n"
},
{
"path": "ludwig/schema/features/audio_feature.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import AUDIO, MODEL_ECD\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import EncoderDataclassField\nfrom ludwig.schema.features.base import BaseInputFeatureConfig\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import PreprocessingDataclassField\nfrom ludwig.schema.features.utils import ecd_defaults_config_registry, ecd_input_config_registry, input_mixin_registry\nfrom ludwig.schema.utils import BaseMarshmallowConfig, ludwig_dataclass\n\n\n@DeveloperAPI\n@ecd_defaults_config_registry.register(AUDIO)\n@input_mixin_registry.register(AUDIO)\n@ludwig_dataclass\nclass AudioInputFeatureConfigMixin(BaseMarshmallowConfig):\n \"\"\"AudioInputFeatureConfigMixin is a dataclass that configures the parameters used in both the audio input\n feature and the audio global defaults section of the Ludwig Config.\"\"\"\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=AUDIO)\n\n encoder: BaseEncoderConfig = EncoderDataclassField(\n MODEL_ECD,\n feature_type=AUDIO,\n default=\"parallel_cnn\",\n )\n\n\n@DeveloperAPI\n@ecd_input_config_registry.register(AUDIO)\n@ludwig_dataclass\nclass AudioInputFeatureConfig(AudioInputFeatureConfigMixin, BaseInputFeatureConfig):\n \"\"\"AudioInputFeatureConfig is a dataclass that configures the parameters used for an audio input feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(AUDIO)\n"
},
{
"path": "ludwig/schema/features/augmentation/__init__.py",
"content": "# Register all augmentation schemas\nimport ludwig.schema.features.augmentation.image # noqa: F401\n"
},
{
"path": "ludwig/schema/features/augmentation/base.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass BaseAugmentationConfig(schema_utils.BaseMarshmallowConfig):\n \"\"\"Base class for augmentation.\"\"\"\n\n type: str\n"
},
{
"path": "ludwig/schema/features/augmentation/image.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import AUGMENTATION, IMAGE, TYPE\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.features.augmentation.base import BaseAugmentationConfig\nfrom ludwig.schema.features.augmentation.utils import register_augmentation_config\nfrom ludwig.schema.metadata import FEATURE_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@register_augmentation_config(name=\"auto_augmentation\", features=IMAGE)\n@ludwig_dataclass\nclass AutoAugmentationConfig(BaseAugmentationConfig):\n \"\"\"Automatic augmentation operation.\"\"\"\n\n type: str = schema_utils.ProtectedString(\n \"auto_augmentation\",\n parameter_metadata=FEATURE_METADATA[IMAGE][AUGMENTATION][TYPE],\n )\n method: str = schema_utils.String(\n default=\"trivial_augment\",\n description=\"Specifies the method for applying automatic data augmentation\",\n parameter_metadata=FEATURE_METADATA[IMAGE][AUGMENTATION][\"auto_augmentation_method\"],\n )\n\n\n@DeveloperAPI\n@register_augmentation_config(name=\"random_horizontal_flip\", features=IMAGE)\n@ludwig_dataclass\nclass RandomHorizontalFlipConfig(BaseAugmentationConfig):\n \"\"\"Random horizontal flip augmentation operation.\"\"\"\n\n type: str = schema_utils.ProtectedString(\n \"random_horizontal_flip\",\n parameter_metadata=FEATURE_METADATA[IMAGE][AUGMENTATION][TYPE],\n )\n\n\n@DeveloperAPI\n@register_augmentation_config(name=\"random_vertical_flip\", features=IMAGE)\n@ludwig_dataclass\nclass RandomVerticalFlipConfig(BaseAugmentationConfig):\n \"\"\"Random vertical flip augmentation operation.\"\"\"\n\n type: str = schema_utils.ProtectedString(\n \"random_vertical_flip\",\n parameter_metadata=FEATURE_METADATA[IMAGE][AUGMENTATION][TYPE],\n )\n\n\n@DeveloperAPI\n@register_augmentation_config(name=\"random_rotate\", features=IMAGE)\n@ludwig_dataclass\nclass RandomRotateConfig(BaseAugmentationConfig):\n \"\"\"Random rotation augmentation operation.\"\"\"\n\n type: str = schema_utils.ProtectedString(\n \"random_rotate\",\n parameter_metadata=FEATURE_METADATA[IMAGE][AUGMENTATION][\"type\"],\n )\n degree: int = schema_utils.Integer(\n default=15,\n description=\"Range of angle for random rotation, i.e., [-degree, +degree].\",\n parameter_metadata=FEATURE_METADATA[IMAGE][AUGMENTATION][\"rotation_degree\"],\n )\n\n\n@DeveloperAPI\n@register_augmentation_config(name=\"random_blur\", features=IMAGE)\n@ludwig_dataclass\nclass RandomBlurConfig(BaseAugmentationConfig):\n \"\"\"Random blur augmentation operation.\"\"\"\n\n type: str = schema_utils.ProtectedString(\n \"random_blur\",\n parameter_metadata=FEATURE_METADATA[IMAGE][AUGMENTATION][TYPE],\n )\n kernel_size: int = schema_utils.Integer(\n default=3,\n description=\"Kernel size for random blur.\",\n parameter_metadata=FEATURE_METADATA[IMAGE][AUGMENTATION][\"kernel_size\"],\n )\n\n\n@DeveloperAPI\n@register_augmentation_config(name=\"random_brightness\", features=IMAGE)\n@ludwig_dataclass\nclass RandomBrightnessConfig(BaseAugmentationConfig):\n \"\"\"Random brightness augmentation operation.\"\"\"\n\n type: str = schema_utils.ProtectedString(\n \"random_brightness\",\n parameter_metadata=FEATURE_METADATA[IMAGE][AUGMENTATION][TYPE],\n )\n\n min: float = schema_utils.FloatRange(\n default=0.5,\n description=\"Minimum factor for random brightness.\",\n parameter_metadata=FEATURE_METADATA[IMAGE][AUGMENTATION][\"min_brightness\"],\n )\n\n max: float = schema_utils.FloatRange(\n default=2.0,\n description=\"Maximum factor for random brightness.\",\n parameter_metadata=FEATURE_METADATA[IMAGE][AUGMENTATION][\"max_brightness\"],\n )\n\n\n@DeveloperAPI\n@register_augmentation_config(name=\"random_contrast\", features=IMAGE)\n@ludwig_dataclass\nclass RandomContrastConfig(BaseAugmentationConfig):\n \"\"\"Random Contrast augmentation operation.\"\"\"\n\n type: str = schema_utils.ProtectedString(\n \"random_contrast\",\n parameter_metadata=FEATURE_METADATA[IMAGE][AUGMENTATION][TYPE],\n )\n\n min: float = schema_utils.FloatRange(\n default=0.5,\n description=\"Minimum factor for random contrast.\",\n parameter_metadata=FEATURE_METADATA[IMAGE][AUGMENTATION][\"min_contrast\"],\n )\n\n max: float = schema_utils.FloatRange(\n default=2.0,\n description=\"Maximum factor for random contrast.\",\n parameter_metadata=FEATURE_METADATA[IMAGE][AUGMENTATION][\"max_contrast\"],\n )\n"
},
{
"path": "ludwig/schema/features/augmentation/utils.py",
"content": "import copy\nfrom dataclasses import field\nfrom typing import Any\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import TYPE\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.features.augmentation.base import BaseAugmentationConfig\nfrom ludwig.utils.registry import Registry\n\n_augmentation_config_registry = Registry()\n\n\n@DeveloperAPI\ndef get_augmentation_config_registry() -> Registry:\n return _augmentation_config_registry\n\n\n@DeveloperAPI\ndef register_augmentation_config(name: str, features: str | list[str]):\n if isinstance(features, str):\n features = [features]\n\n def wrap(cls):\n for feature in features:\n augmentation_registry = get_augmentation_config_registry().get(feature, {})\n augmentation_registry[name] = cls\n get_augmentation_config_registry()[feature] = augmentation_registry\n return cls\n\n return wrap\n\n\n@DeveloperAPI\ndef get_augmentation_cls(feature: str, name: str):\n return get_augmentation_config_registry()[feature][name]\n\n\n@DeveloperAPI\ndef get_augmentation_classes(feature: str):\n return get_augmentation_config_registry()[feature]\n\n\n@DeveloperAPI\ndef AugmentationDataclassField(\n feature_type: str,\n default: str | BaseAugmentationConfig = False,\n default_augmentations: list[BaseAugmentationConfig] | None = None,\n description: str = \"\",\n):\n \"\"\"Custom dataclass field that when used inside a dataclass will allow the user to specify an augmentation\n config.\n\n Args:\n default: The default augmentation config to use.\n default_augmentations: The default list of augmentations to use when param value is set to `True`.\n description: The description of the augmentation config.\n\n Returns: Initialized dataclass field that converts a list with params to an augmentation config.\n \"\"\"\n\n default_augmentations = default_augmentations or []\n default_augmentations = [a.to_dict() for a in default_augmentations]\n\n if isinstance(default, bool):\n default = default_augmentations if default else []\n\n class AugmentationContainerMarshmallowField(schema_utils.LudwigSchemaField):\n \"\"\"Custom field that deserializes a list for a valid augmentation config from the augmentation_registry and\n creates a corresponding JSON schema for external usage.\"\"\"\n\n def _deserialize(self, value, attr, data, **kwargs):\n if isinstance(value, bool):\n value = default_augmentations if value else []\n\n if not isinstance(value, list):\n raise ConfigValidationError(f\"Augmentation config must be a list, found: {type(value)}\")\n\n augmentation_classes = get_augmentation_classes(feature_type)\n augmentation_list = []\n for augmentation in value:\n augmentation_op = augmentation[TYPE]\n if augmentation_op in augmentation_classes:\n augmentation_cls = augmentation_classes[augmentation_op]\n pre = augmentation_cls()\n try:\n augmentation_list.append(pre.Schema().load(augmentation))\n except (TypeError, ConfigValidationError) as error:\n raise ConfigValidationError(\n f\"Invalid augmentation params: {value}, see `{pre}` definition. Error: {error}\"\n )\n else:\n raise ConfigValidationError(\n f\"Invalid augmentation type: '{augmentation_op}', \"\n f\"expected one of: {list(augmentation_classes.keys())}\"\n )\n return augmentation_list\n\n def _jsonschema_type_mapping(self):\n return get_augmentation_list_jsonschema(feature_type, default)\n\n try:\n assert isinstance(default, list), \"Augmentation config must be a list.\"\n load_augmentation_list = []\n dump_augmentation_list = []\n for augmentation in default:\n augmentation_op = augmentation[TYPE]\n augmentation_cls = get_augmentation_cls(feature_type, augmentation_op)\n pre = augmentation_cls()\n try:\n load_augmentation_list.append(pre.Schema().load(augmentation))\n dump_augmentation_list.append(pre.Schema().dump(augmentation))\n except (TypeError, ConfigValidationError) as error:\n raise ConfigValidationError(\n f\"Invalid augmentation params: {default}, see `{pre}` definition. Error: {error}\"\n )\n\n load_default = lambda: copy.deepcopy(load_augmentation_list)\n dump_default = dump_augmentation_list\n\n return field(\n metadata={\n \"marshmallow_field\": AugmentationContainerMarshmallowField(\n allow_none=False,\n dump_default=dump_default,\n load_default=load_default,\n )\n },\n default_factory=load_default,\n )\n except Exception as e:\n raise ConfigValidationError(f\"Unsupported augmentation type. See augmentation_registry. \" f\"Details: {e}\")\n\n\n@DeveloperAPI\ndef get_augmentation_list_jsonschema(feature_type: str, default: list[dict[str, Any]]):\n \"\"\"This function returns a JSON augmentation schema.\n\n Returns: JSON Schema\n \"\"\"\n augmentation_types = sorted(list(get_augmentation_config_registry()[feature_type].keys()))\n schema = {\n \"oneOf\": [\n {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"type\": {\n \"type\": \"string\",\n \"enum\": augmentation_types,\n \"title\": \"type\",\n \"description\": \"Type of augmentation to apply.\",\n },\n },\n \"additionalProperties\": True,\n \"allOf\": get_augmentation_list_conds(feature_type),\n \"required\": [\"type\"],\n },\n \"title\": \"array_option\",\n },\n {\"type\": \"boolean\", \"description\": \"Apply standard augmentation pipeline.\", \"title\": \"boolean_option\"},\n ],\n \"title\": \"augmentation\",\n }\n\n return schema\n\n\n@DeveloperAPI\ndef get_augmentation_list_conds(feature_type: str):\n \"\"\"This function returns a list of if-then JSON clauses for each augmentation type along with their properties\n and constraints.\n\n Returns: List of JSON clauses\n \"\"\"\n conds = []\n for augmentation_op in get_augmentation_classes(feature_type):\n schema_cls = get_augmentation_cls(feature_type, augmentation_op)\n augmentation_schema = schema_utils.unload_jsonschema_from_marshmallow_class(schema_cls)\n augmentation_props = augmentation_schema[\"properties\"]\n schema_utils.remove_duplicate_fields(augmentation_props)\n augmentation_cond = schema_utils.create_cond({\"type\": augmentation_op}, augmentation_props)\n conds.append(augmentation_cond)\n return conds\n"
},
{
"path": "ludwig/schema/features/bag_feature.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import BAG, MODEL_ECD\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import EncoderDataclassField\nfrom ludwig.schema.features.base import BaseInputFeatureConfig\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import PreprocessingDataclassField\nfrom ludwig.schema.features.utils import ecd_defaults_config_registry, ecd_input_config_registry, input_mixin_registry\nfrom ludwig.schema.utils import BaseMarshmallowConfig, ludwig_dataclass\n\n\n@DeveloperAPI\n@ecd_defaults_config_registry.register(BAG)\n@input_mixin_registry.register(BAG)\n@ludwig_dataclass\nclass BagInputFeatureConfigMixin(BaseMarshmallowConfig):\n \"\"\"BagInputFeatureConfigMixin is a dataclass that configures the parameters used in both the bag input feature\n and the bag global defaults section of the Ludwig Config.\"\"\"\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=BAG)\n\n encoder: BaseEncoderConfig = EncoderDataclassField(\n MODEL_ECD,\n feature_type=BAG,\n default=\"embed\",\n )\n\n\n@DeveloperAPI\n@ecd_input_config_registry.register(BAG)\n@ludwig_dataclass\nclass BagInputFeatureConfig(BagInputFeatureConfigMixin, BaseInputFeatureConfig):\n \"\"\"BagInputFeatureConfig is a dataclass that configures the parameters used for a bag input feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(BAG)\n"
},
{
"path": "ludwig/schema/features/base.py",
"content": "import logging\nfrom collections.abc import Iterable\nfrom dataclasses import field\nfrom typing import Any, Generic, TypeVar\n\nfrom rich.console import Console\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import (\n AUDIO,\n BAG,\n BINARY,\n CATEGORY,\n DATE,\n H3,\n IMAGE,\n MODEL_ECD,\n MODEL_LLM,\n NUMBER,\n SEQUENCE,\n SET,\n TEXT,\n TIMESERIES,\n VECTOR,\n)\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.features.utils import (\n ecd_input_config_registry,\n ecd_output_config_registry,\n get_input_feature_jsonschema,\n get_output_feature_jsonschema,\n llm_input_config_registry,\n llm_output_config_registry,\n)\nfrom ludwig.schema.metadata.parameter_metadata import INTERNAL_ONLY, ParameterMetadata\nfrom ludwig.schema.utils import ludwig_dataclass\n\nlogger = logging.getLogger(__name__)\n_error_console = Console(stderr=True, style=\"bold red\")\n_info_console = Console(stderr=True, style=\"bold green\")\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass BaseFeatureConfig(schema_utils.BaseMarshmallowConfig):\n \"\"\"Base class for feature configs.\"\"\"\n\n def __post_init__(self):\n # TODO(travis): this should be done through marshmallow dataclass' `required` field param,\n # but requires a refactor`\n if self.name is None:\n raise ConfigValidationError(\"All features must have a name.\")\n if self.type is None:\n raise ConfigValidationError(f\"Feature {self.name} must have a type.\")\n\n active: bool = True\n\n name: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"Name of the feature.\",\n )\n\n type: str = schema_utils.StringOptions(\n default=None,\n allow_none=True,\n options=[AUDIO, BAG, BINARY, CATEGORY, DATE, H3, IMAGE, NUMBER, SEQUENCE, SET, TEXT, TIMESERIES, VECTOR],\n description=\"Type of the feature.\",\n )\n\n column: str = schema_utils.String(\n allow_none=True,\n default=None,\n description=\"The column name of this feature. Defaults to name if not specified.\",\n )\n\n proc_column: str = schema_utils.String(\n allow_none=True,\n default=None,\n description=\"The name of the preprocessed column name of this feature. Internal only.\",\n parameter_metadata=ParameterMetadata(internal_only=True),\n )\n\n def enable(self):\n \"\"\"This function allows the user to specify which features from a dataset should be included during model\n training. This is the equivalent to toggling on a feature in the model creation UI.\n\n Returns:\n None\n \"\"\"\n if self.active:\n _error_console.print(\"This feature is already enabled!\")\n else:\n self.active = True\n _info_console.print(f\"{self.name} feature enabled!\\n\")\n logger.info(self.__repr__())\n\n def disable(self):\n \"\"\"This function allows the user to specify which features from a dataset should not be included during\n model training. This is the equivalent to toggling off a feature in the model creation UI.\n\n Returns:\n None\n \"\"\"\n if not self.active:\n _error_console.print(\"This feature is already disabled!\")\n else:\n self.active = False\n _info_console.print(f\"{self.name} feature disabled!\\n\")\n logger.info(self.__repr__())\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass BaseInputFeatureConfig(BaseFeatureConfig):\n \"\"\"Base input feature config class.\"\"\"\n\n tied: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"Name of input feature to tie the weights of the encoder with. It needs to be the name of a \"\n \"feature of the same type and with the same encoder parameters. If text or sequence features are tied, \"\n \"consider setting the `sequence_length` parameter in `preprocessing` to ensure that the tied features have \"\n \"equal sized outputs. This is necessary when using the `sequence` combiner.\",\n )\n\n def has_augmentation(self) -> bool:\n return False\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass ECDInputFeatureConfig(BaseFeatureConfig):\n pass\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass BaseOutputFeatureConfig(BaseFeatureConfig):\n \"\"\"Base output feature config class.\"\"\"\n\n reduce_input: str = schema_utils.ReductionOptions(\n default=\"sum\",\n description=\"How to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first \"\n \"dimension (second if you count the batch dimension)\",\n )\n\n default_validation_metric: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"Internal only use parameter: default validation metric for output feature.\",\n parameter_metadata=INTERNAL_ONLY,\n )\n\n dependencies: list[str] = schema_utils.List(\n default=[],\n description=\"List of input features that this feature depends on.\",\n )\n\n reduce_dependencies: str = schema_utils.ReductionOptions(\n default=\"sum\",\n description=\"How to reduce the dependencies of the output feature.\",\n )\n\n input_size: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Size of the input to the decoder.\",\n parameter_metadata=ParameterMetadata(internal_only=True),\n )\n\n num_classes: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Size of the input to the decoder.\",\n parameter_metadata=ParameterMetadata(internal_only=True),\n )\n\n\nT = TypeVar(\"T\", bound=BaseFeatureConfig)\n\n\nclass FeatureCollection(Generic[T], schema_utils.ListSerializable):\n def __init__(self, features: list[T]):\n self._features = features\n self._name_to_feature = {f.name: f for f in features}\n for k, v in self._name_to_feature.items():\n setattr(self, k, v)\n\n def to_list(self) -> list[dict[str, Any]]:\n out_list = []\n for feature in self._features:\n out_list.append(feature.to_dict())\n return out_list\n\n def items(self) -> Iterable[tuple[str, T]]:\n return self._name_to_feature.items()\n\n def __iter__(self):\n return iter(self._features)\n\n def __len__(self):\n return len(self._features)\n\n def __getitem__(self, i) -> T:\n if isinstance(i, str):\n return self._name_to_feature[i]\n else:\n return self._features[i]\n\n\nclass FeatureList(schema_utils.LudwigSchemaField):\n \"\"\"A schema field that deserializes a list of dicts into a FeatureCollection.\n\n Each item is resolved via the inner TypeSelection's resolve() method.\n \"\"\"\n\n def __init__(\n self,\n inner: schema_utils.TypeSelection,\n min_length: int | None = None,\n max_length: int | None = None,\n equal: int | None = None,\n metadata: dict | None = None,\n ):\n self.inner = inner\n self.min_length = min_length\n self.max_length = max_length\n self.equal = equal\n self.metadata = metadata or {}\n\n def _deserialize(self, value, attr, data, **kwargs) -> FeatureCollection:\n if not isinstance(value, list):\n raise ConfigValidationError(f\"Expected a list of features for '{attr}', got {type(value).__name__}\")\n\n # Validate length constraints\n n = len(value)\n if self.equal is not None and n != self.equal:\n raise ConfigValidationError(f\"Expected exactly {self.equal} feature(s) for '{attr}', got {n}\")\n if self.min_length is not None and n < self.min_length:\n raise ConfigValidationError(f\"Expected at least {self.min_length} feature(s) for '{attr}', got {n}\")\n if self.max_length is not None and n > self.max_length:\n raise ConfigValidationError(f\"Expected at most {self.max_length} feature(s) for '{attr}', got {n}\")\n\n feature_list = [self.inner.resolve(item) for item in value]\n return FeatureCollection(feature_list)\n\n def _jsonschema_type_mapping(self):\n inner_schema = self.inner._jsonschema_type_mapping() or {}\n result = {\"type\": \"array\", \"items\": inner_schema}\n if self.min_length is not None:\n result[\"minItems\"] = self.min_length\n if self.max_length is not None:\n result[\"maxItems\"] = self.max_length\n if self.equal is not None:\n result[\"minItems\"] = self.equal\n result[\"maxItems\"] = self.equal\n return result\n\n\nclass FeaturesTypeSelection(schema_utils.TypeSelection):\n def __init__(\n self,\n *args,\n min_length: int | None = 1,\n max_length: int | None = None,\n supplementary_metadata=None,\n **kwargs,\n ):\n super().__init__(*args, **kwargs)\n self.min_length = min_length\n self.max_length = max_length\n self.supplementary_metadata = {} if supplementary_metadata is None else supplementary_metadata\n\n def get_list_field(self):\n min_length = self.min_length\n max_length = self.max_length\n equal = None\n if min_length == max_length:\n min_length = None\n max_length = None\n equal = self.max_length\n\n return field(\n metadata={\n \"marshmallow_field\": FeatureList(\n self,\n min_length=min_length,\n max_length=max_length,\n equal=equal,\n metadata=self.supplementary_metadata,\n )\n },\n )\n\n\nclass ECDInputFeatureSelection(FeaturesTypeSelection):\n def __init__(self):\n super().__init__(\n registry=ecd_input_config_registry,\n description=\"Type of the input feature\",\n supplementary_metadata={\"uniqueItemProperties\": [\"name\"]},\n )\n\n def _jsonschema_type_mapping(self):\n return get_input_feature_jsonschema(MODEL_ECD)\n\n\nclass LLMInputFeatureSelection(FeaturesTypeSelection):\n def __init__(self):\n super().__init__(registry=llm_input_config_registry, description=\"Type of the input feature\")\n\n def _jsonschema_type_mapping(self):\n return get_input_feature_jsonschema(MODEL_LLM)\n\n\nclass ECDOutputFeatureSelection(FeaturesTypeSelection):\n def __init__(self):\n super().__init__(registry=ecd_output_config_registry, description=\"Type of the output feature\")\n\n def _jsonschema_type_mapping(self):\n return get_output_feature_jsonschema(MODEL_ECD)\n\n\nclass LLMOutputFeatureSelection(FeaturesTypeSelection):\n def __init__(self):\n # TODO(Arnav): Remove the hard check on max_length once we support multiple output features.\n super().__init__(max_length=1, registry=llm_output_config_registry, description=\"Type of the output feature\")\n\n def _jsonschema_type_mapping(self):\n return get_output_feature_jsonschema(MODEL_LLM)\n"
},
{
"path": "ludwig/schema/features/binary_feature.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import BINARY, BINARY_WEIGHTED_CROSS_ENTROPY, MODEL_ECD, ROC_AUC\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.decoders.base import BaseDecoderConfig\nfrom ludwig.schema.decoders.utils import DecoderDataclassField\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import EncoderDataclassField\nfrom ludwig.schema.features.base import BaseInputFeatureConfig, BaseOutputFeatureConfig\nfrom ludwig.schema.features.loss.loss import BaseLossConfig\nfrom ludwig.schema.features.loss.utils import LossDataclassField\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import PreprocessingDataclassField\nfrom ludwig.schema.features.utils import (\n ecd_defaults_config_registry,\n ecd_input_config_registry,\n ecd_output_config_registry,\n input_mixin_registry,\n output_mixin_registry,\n)\nfrom ludwig.schema.metadata import FEATURE_METADATA\nfrom ludwig.schema.metadata.parameter_metadata import INTERNAL_ONLY\nfrom ludwig.schema.utils import BaseMarshmallowConfig, ludwig_dataclass\n\n\n@DeveloperAPI\n@input_mixin_registry.register(BINARY)\n@ludwig_dataclass\nclass BinaryInputFeatureConfigMixin(BaseMarshmallowConfig):\n \"\"\"BinaryInputFeatureConfigMixin is a dataclass that configures the parameters used in both the binary input\n feature and the binary global defaults section of the Ludwig Config.\"\"\"\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=BINARY)\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass BinaryInputFeatureConfig(BinaryInputFeatureConfigMixin, BaseInputFeatureConfig):\n \"\"\"BinaryInputFeatureConfig is a dataclass that configures the parameters used for a binary input feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(BINARY)\n\n encoder: BaseEncoderConfig = None\n\n\n@DeveloperAPI\n@ecd_input_config_registry.register(BINARY)\n@ludwig_dataclass\nclass ECDBinaryInputFeatureConfig(BinaryInputFeatureConfig):\n encoder: BaseEncoderConfig = EncoderDataclassField(\n MODEL_ECD,\n feature_type=BINARY,\n default=\"passthrough\",\n )\n\n\n@DeveloperAPI\n@output_mixin_registry.register(BINARY)\n@ludwig_dataclass\nclass BinaryOutputFeatureConfigMixin(BaseMarshmallowConfig):\n \"\"\"BinaryOutputFeatureConfigMixin is a dataclass that configures the parameters used in both the binary output\n feature and the binary global defaults section of the Ludwig Config.\"\"\"\n\n decoder: BaseDecoderConfig = None\n\n loss: BaseLossConfig = LossDataclassField(\n feature_type=BINARY,\n default=BINARY_WEIGHTED_CROSS_ENTROPY,\n )\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass BinaryOutputFeatureConfig(BinaryOutputFeatureConfigMixin, BaseOutputFeatureConfig):\n \"\"\"BinaryOutputFeatureConfig is a dataclass that configures the parameters used for a binary output feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(BINARY)\n\n calibration: bool = schema_utils.Boolean(\n default=False,\n description=\"Calibrate the model's output probabilities using temperature scaling.\",\n parameter_metadata=FEATURE_METADATA[BINARY][\"calibration\"],\n )\n\n default_validation_metric: str = schema_utils.StringOptions(\n [ROC_AUC],\n default=ROC_AUC,\n description=\"Internal only use parameter: default validation metric for binary output feature.\",\n parameter_metadata=INTERNAL_ONLY,\n )\n\n dependencies: list = schema_utils.List(\n default=[],\n description=\"List of input features that this feature depends on.\",\n parameter_metadata=FEATURE_METADATA[BINARY][\"dependencies\"],\n )\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=\"binary_output\")\n\n reduce_dependencies: str = schema_utils.ReductionOptions(\n default=\"sum\",\n description=\"How to reduce the dependencies of the output feature.\",\n parameter_metadata=FEATURE_METADATA[BINARY][\"reduce_dependencies\"],\n )\n\n reduce_input: str = schema_utils.ReductionOptions(\n default=\"sum\",\n description=\"How to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first \"\n \"dimension (second if you count the batch dimension)\",\n parameter_metadata=FEATURE_METADATA[BINARY][\"reduce_input\"],\n )\n\n threshold: float = schema_utils.FloatRange(\n default=0.5,\n min=0,\n max=1,\n description=\"The threshold used to convert output probabilities to predictions. Predicted probabilities greater\"\n \"than or equal to threshold are mapped to True.\",\n parameter_metadata=FEATURE_METADATA[BINARY][\"threshold\"],\n )\n\n\n@DeveloperAPI\n@ecd_output_config_registry.register(BINARY)\n@ludwig_dataclass\nclass ECDBinaryOutputFeatureConfig(BinaryOutputFeatureConfig):\n decoder: BaseDecoderConfig = DecoderDataclassField(\n MODEL_ECD,\n feature_type=BINARY,\n default=\"regressor\",\n )\n\n\n@DeveloperAPI\n@ecd_defaults_config_registry.register(BINARY)\n@ludwig_dataclass\nclass BinaryDefaultsConfig(BinaryInputFeatureConfigMixin, BinaryOutputFeatureConfigMixin):\n # NOTE(travis): defaults use ECD input feature as it contains all the encoders\n encoder: BaseEncoderConfig = EncoderDataclassField(\n MODEL_ECD,\n feature_type=BINARY,\n default=\"passthrough\",\n )\n\n decoder: BaseDecoderConfig = DecoderDataclassField(\n MODEL_ECD,\n feature_type=BINARY,\n default=\"regressor\",\n )\n"
},
{
"path": "ludwig/schema/features/category_feature.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import ACCURACY, CATEGORY, CATEGORY_DISTRIBUTION, MODEL_ECD, MODEL_LLM, SOFTMAX_CROSS_ENTROPY\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.decoders.base import BaseDecoderConfig\nfrom ludwig.schema.decoders.utils import DecoderDataclassField\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import EncoderDataclassField\nfrom ludwig.schema.features.base import BaseInputFeatureConfig, BaseOutputFeatureConfig\nfrom ludwig.schema.features.loss.loss import BaseLossConfig\nfrom ludwig.schema.features.loss.utils import LossDataclassField\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import PreprocessingDataclassField\nfrom ludwig.schema.features.utils import (\n ecd_defaults_config_registry,\n ecd_input_config_registry,\n ecd_output_config_registry,\n input_mixin_registry,\n llm_output_config_registry,\n output_mixin_registry,\n)\nfrom ludwig.schema.metadata import FEATURE_METADATA\nfrom ludwig.schema.metadata.parameter_metadata import INTERNAL_ONLY\nfrom ludwig.schema.utils import BaseMarshmallowConfig, ludwig_dataclass\n\n\n@DeveloperAPI\n@input_mixin_registry.register(CATEGORY)\n@ludwig_dataclass\nclass CategoryInputFeatureConfigMixin(BaseMarshmallowConfig):\n \"\"\"CategoryInputFeatureConfigMixin is a dataclass that configures the parameters used in both the category\n input feature and the category global defaults section of the Ludwig Config.\"\"\"\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=CATEGORY)\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass CategoryInputFeatureConfig(CategoryInputFeatureConfigMixin, BaseInputFeatureConfig):\n \"\"\"CategoryInputFeatureConfig is a dataclass that configures the parameters used for a category input\n feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(CATEGORY)\n\n encoder: BaseEncoderConfig = None\n\n\n@DeveloperAPI\n@ecd_input_config_registry.register(CATEGORY)\n@ludwig_dataclass\nclass ECDCategoryInputFeatureConfig(CategoryInputFeatureConfig):\n encoder: BaseEncoderConfig = EncoderDataclassField(\n MODEL_ECD,\n feature_type=CATEGORY,\n default=\"dense\",\n )\n\n\n@DeveloperAPI\n@output_mixin_registry.register(CATEGORY)\n@ludwig_dataclass\nclass CategoryOutputFeatureConfigMixin(BaseMarshmallowConfig):\n \"\"\"CategoryOutputFeatureConfigMixin is a dataclass that configures the parameters used in both the category\n output feature and the category global defaults section of the Ludwig Config.\"\"\"\n\n decoder: BaseDecoderConfig = None\n\n loss: BaseLossConfig = LossDataclassField(\n feature_type=CATEGORY,\n default=SOFTMAX_CROSS_ENTROPY,\n )\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass CategoryOutputFeatureConfig(CategoryOutputFeatureConfigMixin, BaseOutputFeatureConfig):\n \"\"\"CategoryOutputFeatureConfig is a dataclass that configures the parameters used for a category output\n feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(CATEGORY)\n\n calibration: bool = schema_utils.Boolean(\n default=False,\n description=\"Calibrate the model's output probabilities using temperature scaling.\",\n parameter_metadata=FEATURE_METADATA[CATEGORY][\"calibration\"],\n )\n\n default_validation_metric: str = schema_utils.StringOptions(\n [ACCURACY],\n default=ACCURACY,\n description=\"Internal only use parameter: default validation metric for category output feature.\",\n parameter_metadata=INTERNAL_ONLY,\n )\n\n dependencies: list = schema_utils.List(\n default=[],\n description=\"List of input features that this feature depends on.\",\n parameter_metadata=FEATURE_METADATA[CATEGORY][\"dependencies\"],\n )\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=\"category_output\")\n\n reduce_dependencies: str = schema_utils.ReductionOptions(\n default=\"sum\",\n description=\"How to reduce the dependencies of the output feature.\",\n parameter_metadata=FEATURE_METADATA[CATEGORY][\"reduce_dependencies\"],\n )\n\n reduce_input: str = schema_utils.ReductionOptions(\n default=\"sum\",\n description=\"How to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first \"\n \"dimension (second if you count the batch dimension)\",\n parameter_metadata=FEATURE_METADATA[CATEGORY][\"reduce_input\"],\n )\n\n top_k: int = schema_utils.PositiveInteger(\n default=3,\n description=\"Determines the parameter k, the number of categories to consider when computing the top_k \"\n \"measure. It computes accuracy but considering as a match if the true category appears in the \"\n \"first k predicted categories ranked by decoder's confidence.\",\n parameter_metadata=FEATURE_METADATA[CATEGORY][\"top_k\"],\n )\n\n\n@DeveloperAPI\n@ecd_output_config_registry.register(CATEGORY)\n@ludwig_dataclass\nclass ECDCategoryOutputFeatureConfig(CategoryOutputFeatureConfig):\n decoder: BaseDecoderConfig = DecoderDataclassField(\n MODEL_ECD,\n feature_type=CATEGORY,\n default=\"classifier\",\n )\n\n\n@DeveloperAPI\n@ecd_output_config_registry.register(CATEGORY_DISTRIBUTION)\n@ludwig_dataclass\nclass CategoryDistributionOutputFeatureConfig(CategoryOutputFeatureConfig):\n \"\"\"CategoryDistributionOutputFeatureConfig is a dataclass that configures the parameters used for a\n category_distribution output feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(CATEGORY_DISTRIBUTION)\n\n decoder: BaseDecoderConfig = DecoderDataclassField(\n MODEL_ECD,\n feature_type=CATEGORY,\n default=\"classifier\",\n )\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=\"category_distribution_output\")\n\n\n@DeveloperAPI\n@ecd_defaults_config_registry.register(CATEGORY)\n@ludwig_dataclass\nclass CategoryDefaultsConfig(CategoryInputFeatureConfigMixin, CategoryOutputFeatureConfigMixin):\n encoder: BaseEncoderConfig = EncoderDataclassField(\n MODEL_ECD,\n feature_type=CATEGORY,\n default=\"dense\",\n )\n\n decoder: BaseDecoderConfig = DecoderDataclassField(\n MODEL_ECD,\n feature_type=CATEGORY,\n default=\"classifier\",\n )\n\n\n@DeveloperAPI\n@ecd_defaults_config_registry.register(CATEGORY_DISTRIBUTION)\n@ludwig_dataclass\nclass CategoryDistributionDefaultsConfig(CategoryOutputFeatureConfigMixin):\n pass\n\n\n@DeveloperAPI\n@llm_output_config_registry.register(CATEGORY)\n@ludwig_dataclass\nclass LLMCategoryOutputFeatureConfig(CategoryOutputFeatureConfig):\n \"\"\"LLMCategoryOutputFeatureConfig is a dataclass that configures the parameters used for a category output\n feature when using the Ludwig Light Model.\"\"\"\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=\"category_llm\")\n\n decoder: BaseDecoderConfig = DecoderDataclassField(\n MODEL_LLM,\n feature_type=CATEGORY,\n default=\"category_extractor\",\n )\n"
},
{
"path": "ludwig/schema/features/date_feature.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import DATE, MODEL_ECD\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import EncoderDataclassField\nfrom ludwig.schema.features.base import BaseInputFeatureConfig\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import PreprocessingDataclassField\nfrom ludwig.schema.features.utils import ecd_defaults_config_registry, ecd_input_config_registry, input_mixin_registry\nfrom ludwig.schema.utils import BaseMarshmallowConfig, ludwig_dataclass\n\n\n@DeveloperAPI\n@ecd_defaults_config_registry.register(DATE)\n@input_mixin_registry.register(DATE)\n@ludwig_dataclass\nclass DateInputFeatureConfigMixin(BaseMarshmallowConfig):\n \"\"\"DateInputFeatureConfigMixin is a dataclass that configures the parameters used in both the date input\n feature and the date global defaults section of the Ludwig Config.\"\"\"\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=DATE)\n\n encoder: BaseEncoderConfig = EncoderDataclassField(\n MODEL_ECD,\n feature_type=DATE,\n default=\"embed\",\n )\n\n\n@DeveloperAPI\n@ecd_input_config_registry.register(DATE)\n@ludwig_dataclass\nclass DateInputFeatureConfig(DateInputFeatureConfigMixin, BaseInputFeatureConfig):\n \"\"\"DateInputFeature is a dataclass that configures the parameters used for a date input feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(DATE)\n"
},
{
"path": "ludwig/schema/features/h3_feature.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import H3, MODEL_ECD\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import EncoderDataclassField\nfrom ludwig.schema.features.base import BaseInputFeatureConfig\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import PreprocessingDataclassField\nfrom ludwig.schema.features.utils import ecd_defaults_config_registry, ecd_input_config_registry, input_mixin_registry\nfrom ludwig.schema.utils import BaseMarshmallowConfig, ludwig_dataclass\n\n\n@DeveloperAPI\n@ecd_defaults_config_registry.register(H3)\n@input_mixin_registry.register(H3)\n@ludwig_dataclass\nclass H3InputFeatureConfigMixin(BaseMarshmallowConfig):\n \"\"\"H3InputFeatureConfigMixin is a dataclass that configures the parameters used in both the h3 input feature\n and the h3 global defaults section of the Ludwig Config.\"\"\"\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=H3)\n\n encoder: BaseEncoderConfig = EncoderDataclassField(\n MODEL_ECD,\n feature_type=H3,\n default=\"embed\",\n )\n\n\n@DeveloperAPI\n@ecd_input_config_registry.register(H3)\n@ludwig_dataclass\nclass H3InputFeatureConfig(H3InputFeatureConfigMixin, BaseInputFeatureConfig):\n \"\"\"H3InputFeatureConfig is a dataclass that configures the parameters used for an h3 input feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(H3)\n"
},
{
"path": "ludwig/schema/features/image_feature.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import IMAGE, LOSS, MODEL_ECD, SOFTMAX_CROSS_ENTROPY\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.decoders.base import BaseDecoderConfig\nfrom ludwig.schema.decoders.utils import DecoderDataclassField\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import EncoderDataclassField\nfrom ludwig.schema.features.augmentation.base import BaseAugmentationConfig\nfrom ludwig.schema.features.augmentation.image import RandomHorizontalFlipConfig, RandomRotateConfig\nfrom ludwig.schema.features.augmentation.utils import AugmentationDataclassField\nfrom ludwig.schema.features.base import BaseInputFeatureConfig, BaseOutputFeatureConfig\nfrom ludwig.schema.features.loss.loss import BaseLossConfig\nfrom ludwig.schema.features.loss.utils import LossDataclassField\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import PreprocessingDataclassField\nfrom ludwig.schema.features.utils import (\n ecd_defaults_config_registry,\n ecd_input_config_registry,\n ecd_output_config_registry,\n input_mixin_registry,\n output_mixin_registry,\n)\nfrom ludwig.schema.metadata import FEATURE_METADATA\nfrom ludwig.schema.metadata.parameter_metadata import INTERNAL_ONLY\nfrom ludwig.schema.utils import BaseMarshmallowConfig, ludwig_dataclass\n\n# Augmentation operations when augmentation is set to True\nAUGMENTATION_DEFAULT_OPERATIONS = [\n RandomHorizontalFlipConfig(),\n RandomRotateConfig(),\n]\n\n\n@DeveloperAPI\n@ecd_defaults_config_registry.register(IMAGE)\n@input_mixin_registry.register(IMAGE)\n@ludwig_dataclass\nclass ImageInputFeatureConfigMixin(BaseMarshmallowConfig):\n \"\"\"ImageInputFeatureConfigMixin is a dataclass that configures the parameters used in both the image input\n feature and the image global defaults section of the Ludwig Config.\"\"\"\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=IMAGE)\n\n encoder: BaseEncoderConfig = EncoderDataclassField(\n MODEL_ECD,\n feature_type=IMAGE,\n default=\"stacked_cnn\",\n )\n\n augmentation: list[BaseAugmentationConfig] = AugmentationDataclassField(\n feature_type=IMAGE,\n default=False,\n default_augmentations=AUGMENTATION_DEFAULT_OPERATIONS,\n description=\"Augmentation operation configuration.\",\n )\n\n def has_augmentation(self) -> bool:\n # Check for None, False, and []\n return bool(self.augmentation)\n\n\n@DeveloperAPI\n@ecd_input_config_registry.register(IMAGE)\n@ludwig_dataclass\nclass ImageInputFeatureConfig(ImageInputFeatureConfigMixin, BaseInputFeatureConfig):\n \"\"\"ImageInputFeatureConfig is a dataclass that configures the parameters used for an image input feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(IMAGE)\n\n\n@DeveloperAPI\n@output_mixin_registry.register(IMAGE)\n@ludwig_dataclass\nclass ImageOutputFeatureConfigMixin(BaseMarshmallowConfig):\n \"\"\"ImageOutputFeatureConfigMixin is a dataclass that configures the parameters used in both the image output\n feature and the image global defaults section of the Ludwig Config.\"\"\"\n\n decoder: BaseDecoderConfig = DecoderDataclassField(\n MODEL_ECD,\n feature_type=IMAGE,\n default=\"unet\",\n )\n\n loss: BaseLossConfig = LossDataclassField(\n feature_type=IMAGE,\n default=SOFTMAX_CROSS_ENTROPY,\n )\n\n\n@DeveloperAPI\n@ecd_output_config_registry.register(IMAGE)\n@ludwig_dataclass\nclass ImageOutputFeatureConfig(ImageOutputFeatureConfigMixin, BaseOutputFeatureConfig):\n \"\"\"ImageOutputFeatureConfig is a dataclass that configures the parameters used for an image output feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(IMAGE)\n\n dependencies: list = schema_utils.List(\n default=[],\n description=\"List of input features that this feature depends on.\",\n parameter_metadata=FEATURE_METADATA[IMAGE][\"dependencies\"],\n )\n\n default_validation_metric: str = schema_utils.StringOptions(\n [LOSS],\n default=LOSS,\n description=\"Internal only use parameter: default validation metric for image output feature.\",\n parameter_metadata=INTERNAL_ONLY,\n )\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=\"image_output\")\n\n reduce_dependencies: str = schema_utils.ReductionOptions(\n default=None,\n description=\"How to reduce the dependencies of the output feature.\",\n parameter_metadata=FEATURE_METADATA[IMAGE][\"reduce_dependencies\"],\n )\n\n reduce_input: str = schema_utils.ReductionOptions(\n default=None,\n description=\"How to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first \"\n \"dimension (second if you count the batch dimension)\",\n parameter_metadata=FEATURE_METADATA[IMAGE][\"reduce_input\"],\n )\n\n\n@DeveloperAPI\n@ecd_defaults_config_registry.register(IMAGE)\n@ludwig_dataclass\nclass ImageDefaultsConfig(ImageInputFeatureConfigMixin, ImageOutputFeatureConfigMixin):\n pass\n"
},
{
"path": "ludwig/schema/features/loss/__init__.py",
"content": "from ludwig.schema.features.loss.loss import get_loss_classes, get_loss_cls, get_loss_schema_registry # noqa\n"
},
{
"path": "ludwig/schema/features/loss/loss.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import (\n BINARY,\n BINARY_WEIGHTED_CROSS_ENTROPY,\n CATEGORY,\n CORN,\n HUBER,\n IMAGE,\n MEAN_ABSOLUTE_ERROR,\n MEAN_ABSOLUTE_PERCENTAGE_ERROR,\n MEAN_SQUARED_ERROR,\n NEXT_TOKEN_SOFTMAX_CROSS_ENTROPY,\n NUMBER,\n ROOT_MEAN_SQUARED_ERROR,\n ROOT_MEAN_SQUARED_PERCENTAGE_ERROR,\n SEQUENCE,\n SEQUENCE_SOFTMAX_CROSS_ENTROPY,\n SET,\n SIGMOID_CROSS_ENTROPY,\n SOFTMAX_CROSS_ENTROPY,\n TEXT,\n TIMESERIES,\n VECTOR,\n)\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.metadata import LOSS_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\nfrom ludwig.utils.registry import Registry\n\nROBUST_LAMBDA_DESCRIPTION = (\n \"Replaces the loss with `(1 - robust_lambda) * loss + robust_lambda / c` where `c` is the number of \"\n \"classes. Useful in case of noisy labels.\"\n)\n\nCONFIDENCE_PENALTY_DESCRIPTION = (\n \"Penalizes overconfident predictions (low entropy) by adding an additional term \"\n \"that penalizes too confident predictions by adding a `a * (max_entropy - entropy) / max_entropy` \"\n \"term to the loss, where a is the value of this parameter. Useful in case of noisy labels.\"\n)\n\nCLASS_WEIGHTS_DESCRIPTION = (\n \"Weights to apply to each class in the loss. If not specified, all classes are weighted equally. \"\n \"The value can be a vector of weights, one for each class, that is multiplied to the \"\n \"loss of the datapoints that have that class as ground truth. It is an alternative to oversampling in \"\n \"case of unbalanced class distribution. The ordering of the vector follows the category to integer ID \"\n \"mapping in the JSON metadata file (the `` class needs to be included too). Alternatively, the value \"\n \"can be a dictionary with class strings as keys and weights as values, like \"\n \"`{class_a: 0.5, class_b: 0.7, ...}`.\"\n)\n\nCLASS_SIMILARITIES_DESCRIPTION = (\n \"If not `null` it is a `c x c` matrix in the form of a list of lists that contains the mutual similarity of \"\n \"classes. It is used if `class_similarities_temperature` is greater than 0. The ordering of the vector follows \"\n \"the category to integer ID mapping in the JSON metadata file (the `` class needs to be included too).\"\n)\n\nCLASS_SIMILARITIES_TEMPERATURE_DESCRIPTION = (\n \"The temperature parameter of the softmax that is performed on each row of `class_similarities`. The output of \"\n \"that softmax is used to determine the supervision vector to provide instead of the one hot vector that would be \"\n \"provided otherwise for each datapoint. The intuition behind it is that errors between similar classes are more \"\n \"tolerable than errors between really different classes.\"\n)\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass BaseLossConfig(schema_utils.BaseMarshmallowConfig):\n \"\"\"Base class for feature configs.\"\"\"\n\n type: str\n\n weight: float = 1.0\n\n @classmethod\n def name(cls) -> str:\n return \"[undefined]\"\n\n\n_loss_registry = Registry[type[BaseLossConfig]]()\n_loss_feature_registry = Registry[dict[str, type[BaseLossConfig]]]()\n\n\n@DeveloperAPI\ndef get_loss_schema_registry() -> Registry[type[BaseLossConfig]]:\n return _loss_registry\n\n\n@DeveloperAPI\ndef get_loss_cls(feature: str, name: str) -> type[BaseLossConfig]:\n return _loss_feature_registry[feature][name]\n\n\n@DeveloperAPI\ndef get_loss_classes(feature: str) -> dict[str, type[BaseLossConfig]]:\n return _loss_feature_registry[feature]\n\n\ndef register_loss(features: str | list[str]):\n if isinstance(features, str):\n features = [features]\n\n def wrap(cls: type[BaseLossConfig]):\n _loss_registry[cls.type] = cls\n for feature in features:\n feature_registry = _loss_feature_registry.get(feature, {})\n feature_registry[cls.type] = cls\n _loss_feature_registry[feature] = feature_registry\n return cls\n\n return wrap\n\n\n@DeveloperAPI\n@register_loss([NUMBER, TIMESERIES, VECTOR])\n@ludwig_dataclass\nclass MSELossConfig(BaseLossConfig):\n type: str = schema_utils.ProtectedString(\n MEAN_SQUARED_ERROR,\n description=\"Type of loss.\",\n )\n\n weight: float = schema_utils.NonNegativeFloat(\n default=1.0,\n description=\"Weight of the loss.\",\n parameter_metadata=LOSS_METADATA[\"MSELoss\"][\"weight\"],\n )\n\n @classmethod\n def name(self) -> str:\n return \"Mean Squared Error (MSE)\"\n\n\n@DeveloperAPI\n@register_loss([NUMBER, TIMESERIES, VECTOR])\n@ludwig_dataclass\nclass MAELossConfig(BaseLossConfig):\n type: str = schema_utils.ProtectedString(\n MEAN_ABSOLUTE_ERROR,\n description=\"Type of loss.\",\n )\n\n weight: float = schema_utils.NonNegativeFloat(\n default=1.0,\n description=\"Weight of the loss.\",\n parameter_metadata=LOSS_METADATA[\"MAELoss\"][\"weight\"],\n )\n\n @classmethod\n def name(self) -> str:\n return \"Mean Absolute Error (MAE)\"\n\n\n@DeveloperAPI\n@register_loss([NUMBER, TIMESERIES, VECTOR])\n@ludwig_dataclass\nclass MAPELossConfig(BaseLossConfig):\n type: str = schema_utils.ProtectedString(\n MEAN_ABSOLUTE_PERCENTAGE_ERROR,\n description=\"Type of loss.\",\n )\n\n weight: float = schema_utils.NonNegativeFloat(\n default=1.0,\n description=\"Weight of the loss.\",\n parameter_metadata=LOSS_METADATA[\"MAELoss\"][\"weight\"],\n )\n\n @classmethod\n def name(self) -> str:\n return \"Mean Absolute Percentage Error (MAPE)\"\n\n\n@DeveloperAPI\n@register_loss([NUMBER])\n@ludwig_dataclass\nclass RMSELossConfig(BaseLossConfig):\n type: str = schema_utils.ProtectedString(\n ROOT_MEAN_SQUARED_ERROR,\n description=\"Type of loss.\",\n )\n\n weight: float = schema_utils.NonNegativeFloat(\n default=1.0,\n description=\"Weight of the loss.\",\n parameter_metadata=LOSS_METADATA[\"RMSELoss\"][\"weight\"],\n )\n\n @classmethod\n def name(self) -> str:\n return \"Root Mean Squared Error (RMSE)\"\n\n\n@DeveloperAPI\n@register_loss([NUMBER])\n@ludwig_dataclass\nclass RMSPELossConfig(BaseLossConfig):\n type: str = schema_utils.ProtectedString(\n ROOT_MEAN_SQUARED_PERCENTAGE_ERROR,\n description=\"Type of loss.\",\n )\n\n weight: float = schema_utils.NonNegativeFloat(\n default=1.0,\n description=\"Weight of the loss.\",\n parameter_metadata=LOSS_METADATA[\"RMSPELoss\"][\"weight\"],\n )\n\n @classmethod\n def name(self) -> str:\n return \"Root Mean Squared Percentage Error (RMSPE)\"\n\n\n@DeveloperAPI\n@register_loss([BINARY])\n@ludwig_dataclass\nclass BWCEWLossConfig(BaseLossConfig):\n type: str = schema_utils.ProtectedString(\n BINARY_WEIGHTED_CROSS_ENTROPY,\n description=\"Type of loss.\",\n )\n\n positive_class_weight: float = schema_utils.NonNegativeFloat(\n default=None,\n allow_none=True,\n description=\"Weight of the positive class.\",\n parameter_metadata=LOSS_METADATA[\"BWCEWLoss\"][\"positive_class_weight\"],\n )\n\n robust_lambda: int = schema_utils.NonNegativeInteger(\n default=0,\n description=ROBUST_LAMBDA_DESCRIPTION,\n parameter_metadata=LOSS_METADATA[\"BWCEWLoss\"][\"robust_lambda\"],\n )\n\n confidence_penalty: float = schema_utils.NonNegativeFloat(\n default=0,\n description=CONFIDENCE_PENALTY_DESCRIPTION,\n parameter_metadata=LOSS_METADATA[\"BWCEWLoss\"][\"confidence_penalty\"],\n )\n\n weight: float = schema_utils.NonNegativeFloat(\n default=1.0,\n description=\"Weight of the loss.\",\n parameter_metadata=LOSS_METADATA[\"BWCEWLoss\"][\"weight\"],\n )\n\n @classmethod\n def name(self) -> str:\n return \"Binary Weighted Cross Entropy (BWCE)\"\n\n\n@DeveloperAPI\n@register_loss([CATEGORY, VECTOR, IMAGE])\n@ludwig_dataclass\nclass SoftmaxCrossEntropyLossConfig(BaseLossConfig):\n type: str = schema_utils.ProtectedString(\n SOFTMAX_CROSS_ENTROPY,\n description=\"Type of loss.\",\n )\n\n class_weights: list[float] | dict | None = schema_utils.OneOfOptionsField(\n default=None,\n description=CLASS_WEIGHTS_DESCRIPTION,\n field_options=[\n schema_utils.Dict(default=None, allow_none=True),\n schema_utils.List(list_type=float, allow_none=False),\n ],\n parameter_metadata=LOSS_METADATA[\"SoftmaxCrossEntropyLoss\"][\"class_weights\"],\n )\n\n robust_lambda: int = schema_utils.NonNegativeInteger(\n default=0,\n description=ROBUST_LAMBDA_DESCRIPTION,\n parameter_metadata=LOSS_METADATA[\"SoftmaxCrossEntropyLoss\"][\"robust_lambda\"],\n )\n\n confidence_penalty: float = schema_utils.NonNegativeFloat(\n default=0,\n description=CONFIDENCE_PENALTY_DESCRIPTION,\n parameter_metadata=LOSS_METADATA[\"SoftmaxCrossEntropyLoss\"][\"confidence_penalty\"],\n )\n\n class_similarities: list = schema_utils.List(\n list,\n default=None,\n description=CLASS_SIMILARITIES_DESCRIPTION,\n parameter_metadata=LOSS_METADATA[\"SoftmaxCrossEntropyLoss\"][\"class_similarities\"],\n )\n\n class_similarities_temperature: int = schema_utils.NonNegativeInteger(\n default=0,\n description=CLASS_SIMILARITIES_TEMPERATURE_DESCRIPTION,\n parameter_metadata=LOSS_METADATA[\"SoftmaxCrossEntropyLoss\"][\"class_similarities_temperature\"],\n )\n\n weight: float = schema_utils.NonNegativeFloat(\n default=1.0,\n description=\"Weight of the loss.\",\n parameter_metadata=LOSS_METADATA[\"SoftmaxCrossEntropyLoss\"][\"weight\"],\n )\n\n @classmethod\n def name(self) -> str:\n return \"Softmax Cross Entropy\"\n\n\n@DeveloperAPI\n@register_loss([SEQUENCE, TEXT])\n@ludwig_dataclass\nclass SequenceSoftmaxCrossEntropyLossConfig(BaseLossConfig):\n type: str = schema_utils.ProtectedString(\n SEQUENCE_SOFTMAX_CROSS_ENTROPY,\n description=\"Type of loss.\",\n )\n\n class_weights: list[float] | dict | None = schema_utils.OneOfOptionsField(\n default=None,\n description=CLASS_WEIGHTS_DESCRIPTION,\n field_options=[\n schema_utils.Dict(default=None, allow_none=True),\n schema_utils.List(list_type=float, allow_none=False),\n ],\n parameter_metadata=LOSS_METADATA[\"SequenceSoftmaxCrossEntropyLoss\"][\"class_weights\"],\n )\n\n robust_lambda: int = schema_utils.NonNegativeInteger(\n default=0,\n description=ROBUST_LAMBDA_DESCRIPTION,\n parameter_metadata=LOSS_METADATA[\"SequenceSoftmaxCrossEntropyLoss\"][\"robust_lambda\"],\n )\n\n confidence_penalty: float = schema_utils.NonNegativeFloat(\n default=0,\n description=CONFIDENCE_PENALTY_DESCRIPTION,\n parameter_metadata=LOSS_METADATA[\"SequenceSoftmaxCrossEntropyLoss\"][\"confidence_penalty\"],\n )\n\n class_similarities: list = schema_utils.List(\n list,\n default=None,\n description=CLASS_SIMILARITIES_DESCRIPTION,\n parameter_metadata=LOSS_METADATA[\"SequenceSoftmaxCrossEntropyLoss\"][\"class_similarities\"],\n )\n\n class_similarities_temperature: int = schema_utils.NonNegativeInteger(\n default=0,\n description=CLASS_SIMILARITIES_TEMPERATURE_DESCRIPTION,\n parameter_metadata=LOSS_METADATA[\"SequenceSoftmaxCrossEntropyLoss\"][\"class_similarities_temperature\"],\n )\n\n weight: float = schema_utils.NonNegativeFloat(\n default=1.0,\n description=\"Weight of the loss.\",\n parameter_metadata=LOSS_METADATA[\"SequenceSoftmaxCrossEntropyLoss\"][\"weight\"],\n )\n\n unique: bool = schema_utils.Boolean(\n default=False,\n description=\"If true, the loss is only computed for unique elements in the sequence.\",\n parameter_metadata=LOSS_METADATA[\"SequenceSoftmaxCrossEntropyLoss\"][\"unique\"],\n )\n\n @classmethod\n def name(self) -> str:\n return \"Sequence Softmax Cross Entropy\"\n\n\n@DeveloperAPI\n@register_loss([SEQUENCE, TEXT])\n@ludwig_dataclass\nclass NextTokenSoftmaxCrossEntropyLossConfig(SequenceSoftmaxCrossEntropyLossConfig):\n type: str = schema_utils.ProtectedString(\n NEXT_TOKEN_SOFTMAX_CROSS_ENTROPY,\n description=\"Type of loss.\",\n )\n\n @classmethod\n def name(self) -> str:\n return \"Next Token Softmax Cross Entropy\"\n\n\n@DeveloperAPI\n@register_loss([SET])\n@ludwig_dataclass\nclass SigmoidCrossEntropyLossConfig(BaseLossConfig):\n type: str = schema_utils.ProtectedString(\n SIGMOID_CROSS_ENTROPY,\n description=\"Type of loss.\",\n )\n\n class_weights: list[float] | dict | None = schema_utils.OneOfOptionsField(\n default=None,\n description=CLASS_WEIGHTS_DESCRIPTION,\n field_options=[\n schema_utils.Dict(default=None, allow_none=True),\n schema_utils.List(list_type=float, allow_none=False),\n ],\n parameter_metadata=LOSS_METADATA[\"SigmoidCrossEntropyLoss\"][\"class_weights\"],\n )\n\n weight: float = schema_utils.NonNegativeFloat(\n default=1.0,\n description=\"Weight of the loss.\",\n parameter_metadata=LOSS_METADATA[\"SigmoidCrossEntropyLoss\"][\"weight\"],\n )\n\n @classmethod\n def name(self) -> str:\n return \"Sigmoid Cross Entropy\"\n\n\n@DeveloperAPI\n@register_loss([NUMBER, TIMESERIES, VECTOR])\n@ludwig_dataclass\nclass HuberLossConfig(BaseLossConfig):\n type: str = schema_utils.ProtectedString(\n HUBER,\n description=(\n \"Loss that combines advantages of both `mean_absolute_error` (MAE) and `mean_squared_error` (MSE). The \"\n \"delta-scaled L1 region makes the loss less sensitive to outliers than MSE, while the L2 region provides \"\n \"smoothness over MAE near 0. See [Huber loss](https://en.wikipedia.org/wiki/Huber_loss) for more details.\"\n ),\n )\n\n delta: float = schema_utils.FloatRange(\n default=1.0,\n min=0,\n min_inclusive=False,\n description=\"Threshold at which to change between delta-scaled L1 and L2 loss.\",\n )\n\n weight: float = schema_utils.NonNegativeFloat(\n default=1.0,\n description=\"Weight of the loss.\",\n parameter_metadata=LOSS_METADATA[\"MSELoss\"][\"weight\"],\n )\n\n @classmethod\n def name(self) -> str:\n return \"Huber Loss\"\n\n\n@DeveloperAPI\n@register_loss([CATEGORY])\n@ludwig_dataclass\nclass CORNLossConfig(BaseLossConfig):\n \"\"\"Conditional Ordinal Regression for Neural networks, used for ordered cateogry values.\n\n Source:\n Xintong Shi, Wenzhi Cao, and Sebastian Raschka (2021).\n Deep Neural Networks for Rank-Consistent Ordinal Regression Based On Conditional Probabilities.\n Arxiv preprint; https://arxiv.org/abs/2111.08851\n \"\"\"\n\n type: str = schema_utils.ProtectedString(\n CORN,\n description=\"Type of loss.\",\n )\n\n weight: float = schema_utils.NonNegativeFloat(\n default=1.0,\n description=\"Weight of the loss.\",\n parameter_metadata=LOSS_METADATA[\"MSELoss\"][\"weight\"],\n )\n\n @classmethod\n def name(self) -> str:\n return \"Conditional Ordinal Regression (CORN)\"\n\n @property\n def class_weights(self) -> int:\n return 1.0\n\n @property\n def class_similarities_temperature(self) -> int:\n return 0\n"
},
{
"path": "ludwig/schema/features/loss/utils.py",
"content": "from dataclasses import Field\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.features.loss import get_loss_classes, get_loss_cls\n\n\n@DeveloperAPI\ndef get_loss_conds(feature_type: str):\n \"\"\"Returns a JSON schema of conditionals to validate against loss types for specific feature types.\"\"\"\n conds = []\n for loss in get_loss_classes(feature_type):\n loss_cls = get_loss_cls(feature_type, loss)\n other_props = schema_utils.unload_jsonschema_from_marshmallow_class(loss_cls)[\"properties\"]\n schema_utils.remove_duplicate_fields(other_props)\n loss_cond = schema_utils.create_cond(\n {\"type\": loss},\n other_props,\n )\n conds.append(loss_cond)\n return conds\n\n\n@DeveloperAPI\ndef LossDataclassField(feature_type: str, default: str) -> Field:\n loss_registry = get_loss_classes(feature_type)\n\n class LossSelection(schema_utils.TypeSelection):\n def __init__(self):\n super().__init__(registry=loss_registry, default_value=default)\n\n def get_schema_from_registry(self, key: str) -> type[schema_utils.BaseMarshmallowConfig]:\n return get_loss_cls(feature_type, key)\n\n def _jsonschema_type_mapping(self):\n return {\n \"type\": \"object\",\n \"properties\": {\n \"type\": {\"type\": \"string\", \"enum\": list(loss_registry.keys()), \"default\": default},\n },\n \"title\": \"loss_options\",\n \"allOf\": get_loss_conds(feature_type),\n }\n\n return LossSelection().get_default_field()\n"
},
{
"path": "ludwig/schema/features/number_feature.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import MEAN_SQUARED_ERROR, MODEL_ECD, NUMBER\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.decoders.base import BaseDecoderConfig\nfrom ludwig.schema.decoders.utils import DecoderDataclassField\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import EncoderDataclassField\nfrom ludwig.schema.features.base import BaseInputFeatureConfig, BaseOutputFeatureConfig\nfrom ludwig.schema.features.loss.loss import BaseLossConfig\nfrom ludwig.schema.features.loss.utils import LossDataclassField\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import PreprocessingDataclassField\nfrom ludwig.schema.features.utils import (\n ecd_defaults_config_registry,\n ecd_input_config_registry,\n ecd_output_config_registry,\n input_mixin_registry,\n output_mixin_registry,\n)\nfrom ludwig.schema.metadata import FEATURE_METADATA\nfrom ludwig.schema.metadata.parameter_metadata import INTERNAL_ONLY\nfrom ludwig.schema.utils import BaseMarshmallowConfig, ludwig_dataclass\n\n\n@DeveloperAPI\n@input_mixin_registry.register(NUMBER)\n@ludwig_dataclass\nclass NumberInputFeatureConfigMixin(BaseMarshmallowConfig):\n \"\"\"NumberInputFeatureConfigMixin is a dataclass that configures the parameters used in both the number input\n feature and the number global defaults section of the Ludwig Config.\"\"\"\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=NUMBER)\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass NumberInputFeatureConfig(NumberInputFeatureConfigMixin, BaseInputFeatureConfig):\n \"\"\"NumberInputFeatureConfig is a dataclass that configures the parameters used for a number input feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(NUMBER)\n\n encoder: BaseEncoderConfig = None\n\n\n@DeveloperAPI\n@ecd_input_config_registry.register(NUMBER)\n@ludwig_dataclass\nclass ECDNumberInputFeatureConfig(NumberInputFeatureConfig):\n encoder: BaseEncoderConfig = EncoderDataclassField(\n MODEL_ECD,\n feature_type=NUMBER,\n default=\"passthrough\",\n )\n\n\n@DeveloperAPI\n@output_mixin_registry.register(NUMBER)\n@ludwig_dataclass\nclass NumberOutputFeatureConfigMixin(BaseMarshmallowConfig):\n \"\"\"NumberOutputFeatureConfigMixin is a dataclass that configures the parameters used in both the number output\n feature and the number global defaults section of the Ludwig Config.\"\"\"\n\n decoder: BaseDecoderConfig = None\n\n loss: BaseLossConfig = LossDataclassField(\n feature_type=NUMBER,\n default=MEAN_SQUARED_ERROR,\n )\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass NumberOutputFeatureConfig(NumberOutputFeatureConfigMixin, BaseOutputFeatureConfig):\n \"\"\"NumberOutputFeatureConfig is a dataclass that configures the parameters used for a category output\n feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(NUMBER)\n\n clip: list[int] | tuple[int] = schema_utils.FloatRangeTupleDataclassField(\n n=2,\n default=None,\n allow_none=True,\n min=0,\n max=999999999,\n description=\"Clip the predicted output to the specified range.\",\n parameter_metadata=FEATURE_METADATA[NUMBER][\"clip\"],\n )\n\n default_validation_metric: str = schema_utils.StringOptions(\n [MEAN_SQUARED_ERROR],\n default=MEAN_SQUARED_ERROR,\n description=\"Internal only use parameter: default validation metric for number output feature.\",\n parameter_metadata=INTERNAL_ONLY,\n )\n\n dependencies: list = schema_utils.List(\n default=[],\n description=\"List of input features that this feature depends on.\",\n parameter_metadata=FEATURE_METADATA[NUMBER][\"dependencies\"],\n )\n\n reduce_dependencies: str = schema_utils.ReductionOptions(\n default=\"sum\",\n description=\"How to reduce the dependencies of the output feature.\",\n parameter_metadata=FEATURE_METADATA[NUMBER][\"reduce_dependencies\"],\n )\n\n reduce_input: str = schema_utils.ReductionOptions(\n default=\"sum\",\n description=\"How to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first \"\n \"dimension (second if you count the batch dimension)\",\n parameter_metadata=FEATURE_METADATA[NUMBER][\"reduce_input\"],\n )\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=\"number_output\")\n\n\n@DeveloperAPI\n@ecd_output_config_registry.register(NUMBER)\n@ludwig_dataclass\nclass ECDNumberOutputFeatureConfig(NumberOutputFeatureConfig):\n decoder: BaseDecoderConfig = DecoderDataclassField(\n MODEL_ECD,\n feature_type=NUMBER,\n default=\"regressor\",\n )\n\n\n@DeveloperAPI\n@ecd_defaults_config_registry.register(NUMBER)\n@ludwig_dataclass\nclass NumberDefaultsConfig(NumberInputFeatureConfigMixin, NumberOutputFeatureConfigMixin):\n encoder: BaseEncoderConfig = EncoderDataclassField(\n MODEL_ECD,\n feature_type=NUMBER,\n default=\"passthrough\",\n )\n\n decoder: BaseDecoderConfig = DecoderDataclassField(\n MODEL_ECD,\n feature_type=NUMBER,\n default=\"regressor\",\n )\n"
},
{
"path": "ludwig/schema/features/preprocessing/__init__.py",
"content": "# Register all preprocessors\nfrom ludwig.schema.features.preprocessing import ( # noqa\n audio,\n bag,\n binary,\n category,\n date,\n h3,\n image,\n number,\n sequence,\n set,\n text,\n timeseries,\n vector,\n)\n"
},
{
"path": "ludwig/schema/features/preprocessing/audio.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import AUDIO, BFILL, MISSING_VALUE_STRATEGY_OPTIONS, PREPROCESSING\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import register_preprocessor\nfrom ludwig.schema.metadata import FEATURE_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@register_preprocessor(AUDIO)\n@ludwig_dataclass\nclass AudioPreprocessingConfig(BasePreprocessingConfig):\n audio_file_length_limit_in_s: int = schema_utils.NonNegativeFloat(\n default=7.5,\n allow_none=False,\n description=\"Float value that defines the maximum limit of the audio file in seconds. All files longer than \"\n \"this limit are cut off. All files shorter than this limit are padded with padding_value\",\n parameter_metadata=FEATURE_METADATA[AUDIO][PREPROCESSING][\"audio_file_length_limit_in_s\"],\n )\n\n missing_value_strategy: str = schema_utils.StringOptions(\n MISSING_VALUE_STRATEGY_OPTIONS,\n default=BFILL,\n allow_none=False,\n description=\"What strategy to follow when there's a missing value in an audio column\",\n parameter_metadata=FEATURE_METADATA[AUDIO][PREPROCESSING][\"missing_value_strategy\"],\n )\n\n fill_value: float = schema_utils.NonNegativeFloat(\n default=None,\n allow_none=True,\n description=\"The value to replace missing values with in case the missing_value_strategy is fill_with_const\",\n parameter_metadata=FEATURE_METADATA[AUDIO][PREPROCESSING][\"fill_value\"],\n )\n\n computed_fill_value: float = schema_utils.NonNegativeFloat(\n default=None,\n allow_none=True,\n description=\"The internally computed fill value to replace missing values with in case the \"\n \"missing_value_strategy is fill_with_mode or fill_with_mean\",\n parameter_metadata=FEATURE_METADATA[AUDIO][PREPROCESSING][\"computed_fill_value\"],\n )\n\n in_memory: bool = schema_utils.Boolean(\n default=True,\n description=\"Defines whether the audio dataset will reside in memory during the training process or will be \"\n \"dynamically fetched from disk (useful for large datasets). In the latter case a training batch \"\n \"of input audio will be fetched from disk each training iteration.\",\n parameter_metadata=FEATURE_METADATA[AUDIO][PREPROCESSING][\"in_memory\"],\n )\n\n padding_value: float = schema_utils.NonNegativeFloat(\n default=0.0,\n allow_none=False,\n description=\"Float value that is used for padding.\",\n parameter_metadata=FEATURE_METADATA[AUDIO][PREPROCESSING][\"padding_value\"],\n )\n\n norm: str = schema_utils.StringOptions(\n [\"per_file\"],\n default=None,\n allow_none=True,\n description=\"Normalization strategy for the audio files. If None, no normalization is performed. If \"\n \"per_file, z-norm is applied on a 'per file' level\",\n parameter_metadata=FEATURE_METADATA[AUDIO][PREPROCESSING][\"norm\"],\n )\n\n type: str = schema_utils.StringOptions(\n [\"fbank\", \"group_delay\", \"raw\", \"stft\", \"stft_phase\"],\n default=\"fbank\",\n description=\"Defines the type of audio feature to be used.\",\n parameter_metadata=FEATURE_METADATA[AUDIO][PREPROCESSING][\"type\"],\n )\n\n window_length_in_s: float = schema_utils.NonNegativeFloat(\n default=0.04,\n description=\"Defines the window length used for the short time Fourier transformation. This is only needed if \"\n \"the audio_feature_type is 'raw'.\",\n parameter_metadata=FEATURE_METADATA[AUDIO][PREPROCESSING][\"window_length_in_s\"],\n )\n\n window_shift_in_s: float = schema_utils.NonNegativeFloat(\n default=0.02,\n description=\"Defines the window shift used for the short time Fourier transformation (also called \"\n \"hop_length). This is only needed if the audio_feature_type is 'raw'. \",\n parameter_metadata=FEATURE_METADATA[AUDIO][PREPROCESSING][\"window_shift_in_s\"],\n )\n\n num_fft_points: float = schema_utils.NonNegativeFloat(\n default=None,\n allow_none=True,\n description=\"Defines the number of fft points used for the short time Fourier transformation\",\n parameter_metadata=FEATURE_METADATA[AUDIO][PREPROCESSING][\"num_fft_points\"],\n )\n\n window_type: str = schema_utils.StringOptions(\n [\"bartlett\", \"blackman\", \"hamming\", \"hann\"],\n default=\"hamming\",\n description=\"Defines the type window the signal is weighted before the short time Fourier transformation.\",\n parameter_metadata=FEATURE_METADATA[AUDIO][PREPROCESSING][\"window_type\"],\n )\n\n num_filter_bands: int = schema_utils.PositiveInteger(\n default=80,\n description=\"Defines the number of filters used in the filterbank. Only needed if audio_feature_type \"\n \"is 'fbank'\",\n parameter_metadata=FEATURE_METADATA[AUDIO][PREPROCESSING][\"num_filter_bands\"],\n )\n"
},
{
"path": "ludwig/schema/features/preprocessing/bag.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import BAG, FILL_WITH_CONST, MISSING_VALUE_STRATEGY_OPTIONS, PREPROCESSING\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import register_preprocessor\nfrom ludwig.schema.metadata import FEATURE_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\nfrom ludwig.utils import strings_utils\nfrom ludwig.utils.tokenizers import tokenizer_registry\n\n\n@DeveloperAPI\n@register_preprocessor(BAG)\n@ludwig_dataclass\nclass BagPreprocessingConfig(BasePreprocessingConfig):\n tokenizer: str = schema_utils.StringOptions(\n tokenizer_registry.keys(),\n default=\"space\",\n allow_none=False,\n description=\"Defines how to transform the raw text content of the dataset column to a set of elements. The \"\n \"default value space splits the string on spaces. Common options include: underscore (splits on \"\n \"underscore), comma (splits on comma), json (decodes the string into a set or a list through a \"\n \"JSON parser).\",\n parameter_metadata=FEATURE_METADATA[BAG][PREPROCESSING][\"tokenizer\"],\n )\n\n missing_value_strategy: str = schema_utils.StringOptions(\n MISSING_VALUE_STRATEGY_OPTIONS,\n default=FILL_WITH_CONST,\n allow_none=False,\n description=\"What strategy to follow when there's a missing value in a set column\",\n parameter_metadata=FEATURE_METADATA[BAG][PREPROCESSING][\"missing_value_strategy\"],\n )\n\n fill_value: str = schema_utils.String(\n default=strings_utils.UNKNOWN_SYMBOL,\n allow_none=False,\n description=\"The value to replace missing values with in case the missing_value_strategy is fill_with_const\",\n parameter_metadata=FEATURE_METADATA[BAG][PREPROCESSING][\"fill_value\"],\n )\n\n computed_fill_value: str = schema_utils.String(\n default=strings_utils.UNKNOWN_SYMBOL,\n allow_none=False,\n description=\"The internally computed fill value to replace missing values with in case the \"\n \"missing_value_strategy is fill_with_mode or fill_with_mean\",\n parameter_metadata=FEATURE_METADATA[BAG][PREPROCESSING][\"computed_fill_value\"],\n )\n\n lowercase: bool = schema_utils.Boolean(\n default=False,\n description=\"If true, converts the string to lowercase before tokenizing.\",\n parameter_metadata=FEATURE_METADATA[BAG][PREPROCESSING][\"lowercase\"],\n )\n\n most_common: int = schema_utils.PositiveInteger(\n default=10000,\n allow_none=True,\n description=\"The maximum number of most common tokens to be considered. If the data contains more than this \"\n \"amount, the most infrequent tokens will be treated as unknown.\",\n parameter_metadata=FEATURE_METADATA[BAG][PREPROCESSING][\"most_common\"],\n )\n"
},
{
"path": "ludwig/schema/features/preprocessing/base.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.schema import utils as schema_utils\n\n\n@DeveloperAPI\nclass BasePreprocessingConfig(schema_utils.BaseMarshmallowConfig):\n \"\"\"Base class for input feature preprocessing. Not meant to be used directly.\n\n The dataclass format prevents arbitrary properties from being set. Consequently, in child classes, all properties\n from the corresponding input feature class are copied over: check each class to check which attributes are different\n from the preprocessing of each feature.\n \"\"\"\n"
},
{
"path": "ludwig/schema/features/preprocessing/binary.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import (\n BFILL,\n BINARY,\n DROP_ROW,\n FFILL,\n FILL_WITH_FALSE,\n FILL_WITH_MODE,\n FILL_WITH_TRUE,\n PREPROCESSING,\n)\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import register_preprocessor\nfrom ludwig.schema.metadata import FEATURE_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\nfrom ludwig.utils import strings_utils\n\n\n@DeveloperAPI\n@register_preprocessor(BINARY)\n@ludwig_dataclass\nclass BinaryPreprocessingConfig(BasePreprocessingConfig):\n \"\"\"BinaryPreprocessingConfig is a dataclass that configures the parameters used for a binary input feature.\"\"\"\n\n missing_value_strategy: str = schema_utils.StringOptions(\n [FILL_WITH_MODE, BFILL, FFILL, DROP_ROW, FILL_WITH_FALSE, FILL_WITH_TRUE],\n default=FILL_WITH_FALSE,\n allow_none=False,\n description=\"What strategy to follow when there's a missing value in a binary column\",\n parameter_metadata=FEATURE_METADATA[BINARY][PREPROCESSING][\"missing_value_strategy\"],\n )\n\n fallback_true_label: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"The label to interpret as 1 (True) when the binary feature doesn't have a \"\n \"conventional boolean value\",\n parameter_metadata=FEATURE_METADATA[BINARY][PREPROCESSING][\"fallback_true_label\"],\n )\n\n fill_value: int | float | str = schema_utils.OneOfOptionsField(\n default=None,\n allow_none=True,\n field_options=[\n schema_utils.FloatRange(default=None, allow_none=True, min=0, max=1, description=\"\"),\n schema_utils.StringOptions(options=strings_utils.all_bool_strs(), default=\"Y\", allow_none=False),\n schema_utils.Boolean(default=True, description=\"\"),\n ],\n description=\"The value to replace missing values with in case the missing_value_strategy is fill_with_const\",\n parameter_metadata=FEATURE_METADATA[BINARY][PREPROCESSING][\"fill_value\"],\n )\n\n computed_fill_value: int | float | str = schema_utils.OneOfOptionsField(\n default=None,\n allow_none=True,\n field_options=[\n schema_utils.FloatRange(default=1.0, allow_none=False, min=0, max=1, description=\"\"),\n schema_utils.StringOptions(options=strings_utils.all_bool_strs(), default=\"Y\", allow_none=False),\n schema_utils.Boolean(default=True, description=\"\"),\n ],\n description=\"The internally computed fill value to replace missing values with in case the \"\n \"missing_value_strategy is fill_with_mode or fill_with_mean\",\n parameter_metadata=FEATURE_METADATA[BINARY][PREPROCESSING][\"computed_fill_value\"],\n )\n\n\n@DeveloperAPI\n@register_preprocessor(\"binary_output\")\n@ludwig_dataclass\nclass BinaryOutputPreprocessingConfig(BinaryPreprocessingConfig):\n missing_value_strategy: str = schema_utils.StringOptions(\n [FILL_WITH_MODE, BFILL, FFILL, DROP_ROW, FILL_WITH_FALSE, FILL_WITH_TRUE],\n default=DROP_ROW,\n allow_none=False,\n description=\"What strategy to follow when there's a missing value in a binary output feature\",\n parameter_metadata=FEATURE_METADATA[BINARY][PREPROCESSING][\"missing_value_strategy\"],\n )\n\n fallback_true_label: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"The label to interpret as 1 (True) when the binary feature doesn't have a \"\n \"conventional boolean value\",\n parameter_metadata=FEATURE_METADATA[BINARY][PREPROCESSING][\"fallback_true_label\"],\n )\n"
},
{
"path": "ludwig/schema/features/preprocessing/category.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import CATEGORY, DROP_ROW, FILL_WITH_CONST, MISSING_VALUE_STRATEGY_OPTIONS, PREPROCESSING\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import register_preprocessor\nfrom ludwig.schema.metadata import FEATURE_METADATA, PREPROCESSING_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\nfrom ludwig.utils import strings_utils\n\n\n@DeveloperAPI\n@register_preprocessor(CATEGORY)\n@ludwig_dataclass\nclass CategoryPreprocessingConfig(BasePreprocessingConfig):\n \"\"\"CategoryPreprocessingConfig is a dataclass that configures the parameters used for a category input\n feature.\"\"\"\n\n missing_value_strategy: str = schema_utils.StringOptions(\n MISSING_VALUE_STRATEGY_OPTIONS,\n default=FILL_WITH_CONST,\n allow_none=False,\n description=\"What strategy to follow when there's a missing value in a category column\",\n parameter_metadata=FEATURE_METADATA[CATEGORY][PREPROCESSING][\"missing_value_strategy\"],\n )\n\n fill_value: str = schema_utils.String(\n default=strings_utils.UNKNOWN_SYMBOL,\n allow_none=False,\n description=(\n \"The value to replace missing values with in case the `missing_value_strategy` is `fill_with_const`\"\n ),\n parameter_metadata=FEATURE_METADATA[CATEGORY][PREPROCESSING][\"fill_value\"],\n )\n\n computed_fill_value: str = schema_utils.String(\n default=strings_utils.UNKNOWN_SYMBOL,\n allow_none=False,\n description=\"The internally computed fill value to replace missing values with in case the \"\n \"missing_value_strategy is fill_with_mode or fill_with_mean\",\n parameter_metadata=FEATURE_METADATA[CATEGORY][PREPROCESSING][\"computed_fill_value\"],\n )\n\n lowercase: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether the string has to be lowercased before being handled by the tokenizer.\",\n parameter_metadata=FEATURE_METADATA[CATEGORY][PREPROCESSING][\"lowercase\"],\n )\n\n most_common: int = schema_utils.PositiveInteger(\n default=10000,\n allow_none=True,\n description=\"The maximum number of most common tokens to be considered. if the data contains more than this \"\n \"amount, the most infrequent tokens will be treated as unknown.\",\n parameter_metadata=FEATURE_METADATA[CATEGORY][PREPROCESSING][\"most_common\"],\n )\n\n cache_encoder_embeddings: bool = schema_utils.Boolean(\n default=False,\n description=(\n \"For fixed encoders, compute encoder embeddings in preprocessing to avoid this step at train time. \"\n \"Can speed up the time taken per step during training, but will invalidate the preprocessed data \"\n \"if the encoder type is changed.\"\n ),\n parameter_metadata=PREPROCESSING_METADATA[\"cache_encoder_embeddings\"],\n )\n\n\n@DeveloperAPI\n@register_preprocessor(\"category_output\")\n@ludwig_dataclass\nclass CategoryOutputPreprocessingConfig(CategoryPreprocessingConfig):\n missing_value_strategy: str = schema_utils.StringOptions(\n MISSING_VALUE_STRATEGY_OPTIONS,\n default=DROP_ROW,\n allow_none=False,\n description=\"What strategy to follow when there's a missing value in a category output feature\",\n parameter_metadata=FEATURE_METADATA[CATEGORY][PREPROCESSING][\"missing_value_strategy\"],\n )\n\n lowercase: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether the string has to be lowercased before being handled by the tokenizer.\",\n parameter_metadata=FEATURE_METADATA[CATEGORY][PREPROCESSING][\"lowercase\"],\n )\n\n most_common: int = schema_utils.PositiveInteger(\n default=10000,\n allow_none=True,\n description=\"The maximum number of most common tokens to be considered. if the data contains more than this \"\n \"amount, the most infrequent tokens will be treated as unknown.\",\n parameter_metadata=FEATURE_METADATA[CATEGORY][PREPROCESSING][\"most_common\"],\n )\n\n\n@DeveloperAPI\n@register_preprocessor(\"category_distribution_output\")\n@ludwig_dataclass\nclass CategoryDistributionOutputPreprocessingConfig(BasePreprocessingConfig):\n def __post_init__(self):\n if self.vocab is None:\n raise ConfigValidationError(\"`vocab` must be specified for `category_distribution` output feature.\")\n\n missing_value_strategy: str = schema_utils.StringOptions(\n MISSING_VALUE_STRATEGY_OPTIONS,\n default=DROP_ROW,\n allow_none=False,\n description=\"What strategy to follow when there's a missing value in a category output feature\",\n parameter_metadata=FEATURE_METADATA[CATEGORY][PREPROCESSING][\"missing_value_strategy\"],\n )\n\n vocab: list[str] = schema_utils.List(default=None)\n\n\n@DeveloperAPI\n@register_preprocessor(\"category_llm\")\n@ludwig_dataclass\nclass LLMCategoryOutputPreprocessingConfig(CategoryOutputPreprocessingConfig):\n def __post_init__(self):\n if self.vocab is None:\n raise ConfigValidationError(\"`vocab` must be specified for `category_llm` output feature.\")\n if self.fallback_label is None:\n raise ConfigValidationError(\"`fallback_label` must be specified for `category_llm` output feature.\")\n\n vocab: list[str] = schema_utils.List(\n default=None,\n allow_none=False,\n description=\"The list of labels that the model can predict.\",\n )\n\n fallback_label: str = schema_utils.String(\n default=\"\",\n allow_none=False,\n description=\"The label to use when the model doesn't match any of the labels in the `labels` list.\",\n )\n"
},
{
"path": "ludwig/schema/features/preprocessing/date.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import BFILL, DATE, DROP_ROW, FFILL, FILL_WITH_CONST, PREPROCESSING\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import register_preprocessor\nfrom ludwig.schema.metadata import FEATURE_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@register_preprocessor(DATE)\n@ludwig_dataclass\nclass DatePreprocessingConfig(BasePreprocessingConfig):\n missing_value_strategy: str = schema_utils.StringOptions(\n [FILL_WITH_CONST, BFILL, FFILL, DROP_ROW],\n default=FILL_WITH_CONST,\n allow_none=False,\n description=\"What strategy to follow when there's a missing value in a date column\",\n parameter_metadata=FEATURE_METADATA[DATE][PREPROCESSING][\"missing_value_strategy\"],\n )\n\n fill_value: str = schema_utils.String(\n default=\"\",\n allow_none=False,\n description=\"The value to replace missing values with in case the missing_value_strategy is fill_with_const\",\n parameter_metadata=FEATURE_METADATA[DATE][PREPROCESSING][\"fill_value\"],\n )\n\n computed_fill_value: str = schema_utils.String(\n default=\"\",\n allow_none=False,\n description=\"The internally computed fill value to replace missing values with in case the \"\n \"missing_value_strategy is fill_with_mode or fill_with_mean\",\n parameter_metadata=FEATURE_METADATA[DATE][PREPROCESSING][\"computed_fill_value\"],\n )\n\n datetime_format: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"This parameter can either be a datetime format string, or null, in which case the datetime \"\n \"format will be inferred automatically.\",\n parameter_metadata=FEATURE_METADATA[DATE][PREPROCESSING][\"datetime_format\"],\n )\n"
},
{
"path": "ludwig/schema/features/preprocessing/h3.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import FILL_WITH_CONST, H3, MISSING_VALUE_STRATEGY_OPTIONS, PREPROCESSING\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import register_preprocessor\nfrom ludwig.schema.metadata import FEATURE_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@register_preprocessor(H3)\n@ludwig_dataclass\nclass H3PreprocessingConfig(BasePreprocessingConfig):\n missing_value_strategy: str = schema_utils.StringOptions(\n MISSING_VALUE_STRATEGY_OPTIONS,\n default=FILL_WITH_CONST,\n allow_none=False,\n description=\"What strategy to follow when there's a missing value in an h3 column\",\n parameter_metadata=FEATURE_METADATA[H3][PREPROCESSING][\"missing_value_strategy\"],\n )\n\n fill_value: int = schema_utils.PositiveInteger(\n default=576495936675512319,\n allow_none=False,\n description=\"The value to replace missing values with in case the missing_value_strategy is fill_with_const\",\n parameter_metadata=FEATURE_METADATA[H3][PREPROCESSING][\"fill_value\"],\n )\n\n computed_fill_value: int = schema_utils.PositiveInteger(\n default=576495936675512319,\n allow_none=False,\n description=\"The internally computed fill value to replace missing values with in case the \"\n \"missing_value_strategy is fill_with_mode or fill_with_mean\",\n parameter_metadata=FEATURE_METADATA[H3][PREPROCESSING][\"computed_fill_value\"],\n )\n"
},
{
"path": "ludwig/schema/features/preprocessing/image.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import BFILL, DROP_ROW, IMAGE, IMAGENET1K, MISSING_VALUE_STRATEGY_OPTIONS, PREPROCESSING\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import register_preprocessor\nfrom ludwig.schema.metadata import FEATURE_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@register_preprocessor(IMAGE)\n@ludwig_dataclass\nclass ImagePreprocessingConfig(BasePreprocessingConfig):\n missing_value_strategy: str = schema_utils.StringOptions(\n MISSING_VALUE_STRATEGY_OPTIONS,\n default=BFILL,\n allow_none=False,\n description=\"What strategy to follow when there's a missing value in an image column\",\n parameter_metadata=FEATURE_METADATA[IMAGE][PREPROCESSING][\"missing_value_strategy\"],\n )\n\n fill_value: float = schema_utils.NonNegativeFloat(\n default=None,\n allow_none=True,\n description=\"The maximum number of most common tokens to be considered. If the data contains more than this \"\n \"amount, the most infrequent tokens will be treated as unknown.\",\n parameter_metadata=FEATURE_METADATA[IMAGE][PREPROCESSING][\"fill_value\"],\n )\n\n computed_fill_value: float = schema_utils.NonNegativeFloat(\n default=None,\n allow_none=True,\n description=\"The internally computed fill value to replace missing values with in case the \"\n \"missing_value_strategy is fill_with_mode or fill_with_mean\",\n parameter_metadata=FEATURE_METADATA[IMAGE][PREPROCESSING][\"computed_fill_value\"],\n )\n\n height: int | None = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"The image height in pixels. If this parameter is set, images will be resized to the specified \"\n \"height using the resize_method parameter. If None, images will be resized to the size of the \"\n \"first image in the dataset.\",\n parameter_metadata=FEATURE_METADATA[IMAGE][PREPROCESSING][\"height\"],\n )\n\n width: int | None = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"The image width in pixels. If this parameter is set, images will be resized to the specified \"\n \"width using the resize_method parameter. If None, images will be resized to the size of the \"\n \"first image in the dataset.\",\n parameter_metadata=FEATURE_METADATA[IMAGE][PREPROCESSING][\"width\"],\n )\n\n num_channels: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Number of channels in the images. If specified, images will be read in the mode specified by the \"\n \"number of channels. If not specified, the number of channels will be inferred from the image \"\n \"format of the first valid image in the dataset.\",\n parameter_metadata=FEATURE_METADATA[IMAGE][PREPROCESSING][\"num_channels\"],\n )\n\n resize_method: str = schema_utils.StringOptions(\n [\"crop_or_pad\", \"interpolate\"],\n default=\"interpolate\",\n allow_none=False,\n description=\"The method to use for resizing images.\",\n parameter_metadata=FEATURE_METADATA[IMAGE][PREPROCESSING][\"resize_method\"],\n )\n\n infer_image_num_channels: bool = schema_utils.Boolean(\n default=True,\n description=\"If true, then the number of channels in the dataset is inferred from a sample of the first image \"\n \"in the dataset.\",\n parameter_metadata=FEATURE_METADATA[IMAGE][PREPROCESSING][\"infer_image_num_channels\"],\n )\n\n infer_image_dimensions: bool = schema_utils.Boolean(\n default=True,\n description=\"If true, then the height and width of images in the dataset will be inferred from a sample of \"\n \"the first image in the dataset. Each image that doesn't conform to these dimensions will be \"\n \"resized according to resize_method. If set to false, then the height and width of images in the \"\n \"dataset will be specified by the user.\",\n parameter_metadata=FEATURE_METADATA[IMAGE][PREPROCESSING][\"infer_image_dimensions\"],\n )\n\n infer_image_max_height: int = schema_utils.PositiveInteger(\n default=256,\n allow_none=False,\n description=\"If infer_image_dimensions is set, this is used as the maximum height of the images in \"\n \"the dataset.\",\n parameter_metadata=FEATURE_METADATA[IMAGE][PREPROCESSING][\"infer_image_max_height\"],\n )\n\n infer_image_max_width: int = schema_utils.PositiveInteger(\n default=256,\n allow_none=False,\n description=\"If infer_image_dimensions is set, this is used as the maximum width of the images in \"\n \"the dataset.\",\n parameter_metadata=FEATURE_METADATA[IMAGE][PREPROCESSING][\"infer_image_max_width\"],\n )\n\n infer_image_sample_size: int = schema_utils.PositiveInteger(\n default=100,\n allow_none=False,\n description=\"The sample size used for inferring dimensions of images in infer_image_dimensions.\",\n parameter_metadata=FEATURE_METADATA[IMAGE][PREPROCESSING][\"infer_image_sample_size\"],\n )\n\n standardize_image: str | None = schema_utils.StringOptions(\n [IMAGENET1K],\n default=None,\n allow_none=True,\n description=\"Standardize image by per channel mean centering and standard deviation scaling .\",\n parameter_metadata=FEATURE_METADATA[IMAGE][PREPROCESSING][\"standardize_image\"],\n )\n\n in_memory: bool = schema_utils.Boolean(\n default=True,\n description=\"Defines whether image dataset will reside in memory during the training process or will be \"\n \"dynamically fetched from disk (useful for large datasets). In the latter case a training batch \"\n \"of input images will be fetched from disk each training iteration.\",\n parameter_metadata=FEATURE_METADATA[IMAGE][PREPROCESSING][\"in_memory\"],\n )\n\n num_processes: int = schema_utils.PositiveInteger(\n default=1,\n allow_none=False,\n description=\"Specifies the number of processes to run for preprocessing images.\",\n parameter_metadata=FEATURE_METADATA[IMAGE][PREPROCESSING][\"num_processes\"],\n )\n\n requires_equal_dimensions: bool = schema_utils.Boolean(\n default=False,\n description=\"If true, then width and height must be equal.\",\n parameter_metadata=FEATURE_METADATA[IMAGE][PREPROCESSING][\"requires_equal_dimensions\"],\n )\n\n num_classes: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Number of channel classes in the images. If specified, this value will be validated \"\n \"against the inferred number of classes. Use 2 to convert grayscale images to binary images.\",\n parameter_metadata=FEATURE_METADATA[IMAGE][PREPROCESSING][\"num_classes\"],\n )\n\n infer_image_num_classes: bool = schema_utils.Boolean(\n default=False,\n description=\"If true, then the number of channel classes in the dataset will be inferred from a sample of \"\n \"the first image in the dataset. Each unique channel value will be mapped to a class and preprocessing will \"\n \"create a masked image based on the channel classes. \",\n parameter_metadata=FEATURE_METADATA[IMAGE][PREPROCESSING][\"infer_image_num_classes\"],\n )\n\n\n@DeveloperAPI\n@register_preprocessor(\"image_output\")\n@ludwig_dataclass\nclass ImageOutputPreprocessingConfig(ImagePreprocessingConfig):\n missing_value_strategy: str = schema_utils.StringOptions(\n MISSING_VALUE_STRATEGY_OPTIONS,\n default=DROP_ROW,\n allow_none=False,\n description=\"What strategy to follow when there's a missing value in an image column\",\n parameter_metadata=FEATURE_METADATA[IMAGE][PREPROCESSING][\"missing_value_strategy\"],\n )\n"
},
{
"path": "ludwig/schema/features/preprocessing/number.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import (\n DROP_ROW,\n FILL_WITH_CONST,\n FILL_WITH_MEAN,\n MISSING_VALUE_STRATEGY_OPTIONS,\n NUMBER,\n PREPROCESSING,\n)\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import register_preprocessor\nfrom ludwig.schema.metadata import FEATURE_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@register_preprocessor(NUMBER)\n@ludwig_dataclass\nclass NumberPreprocessingConfig(BasePreprocessingConfig):\n \"\"\"NumberPreprocessingConfig is a dataclass that configures the parameters used for a number input feature.\"\"\"\n\n missing_value_strategy: str = schema_utils.StringOptions(\n MISSING_VALUE_STRATEGY_OPTIONS + [FILL_WITH_MEAN],\n default=FILL_WITH_CONST,\n allow_none=False,\n description=\"What strategy to follow when there's a missing value in a number column\",\n parameter_metadata=FEATURE_METADATA[NUMBER][PREPROCESSING][\"missing_value_strategy\"],\n )\n\n fill_value: float = schema_utils.FloatRange(\n default=0.0,\n allow_none=False,\n description=\"The value to replace missing values with in case the missing_value_strategy is fill_with_const\",\n parameter_metadata=FEATURE_METADATA[NUMBER][PREPROCESSING][\"fill_value\"],\n )\n\n computed_fill_value: float = schema_utils.FloatRange(\n default=0.0,\n allow_none=False,\n description=\"The internally computed fill value to replace missing values with in case the \"\n \"missing_value_strategy is fill_with_mode or fill_with_mean\",\n parameter_metadata=FEATURE_METADATA[NUMBER][PREPROCESSING][\"computed_fill_value\"],\n )\n\n normalization: str = schema_utils.StringOptions(\n [\"zscore\", \"minmax\", \"log1p\", \"iq\"],\n default=\"zscore\",\n allow_none=True,\n description=(\n \"Normalization strategy to use for this number feature. If the value is `null` no normalization is \"\n \"performed.\"\n ),\n parameter_metadata=FEATURE_METADATA[NUMBER][PREPROCESSING][\"normalization\"],\n )\n\n outlier_strategy: str | None = schema_utils.StringOptions(\n MISSING_VALUE_STRATEGY_OPTIONS + [FILL_WITH_MEAN, None],\n default=None,\n allow_none=True,\n description=(\n \"Determines how outliers will be handled in the dataset. In most cases, replacing outliers with the \"\n \"column mean (`fill_with_mean`) will be sufficient, but in others the outliers may be damaging enough \"\n \"to merit dropping the entire row of data (`drop_row`). In some cases, the best way to handle outliers \"\n \"is to leave them in the data, which is the behavior when this parameter is left as `null`.\"\n ),\n parameter_metadata=FEATURE_METADATA[NUMBER][PREPROCESSING][\"outlier_strategy\"],\n )\n\n outlier_threshold: float | None = schema_utils.FloatRange(\n default=3.0,\n allow_none=False,\n min=0.0,\n description=(\n \"Standard deviations from the mean past which a value is considered an outlier. The 3-sigma \"\n \"rule in statistics tells us that when data is normally distributed, 95% of the data will lie within 2 \"\n \"standard deviations of the mean, and greater than 99% of the data will lie within 3 standard deviations \"\n \"of the mean (see: [68–95–99.7 rule](https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule)). \"\n \"As such anything farther away than that is highly likely to be an outlier, and may distort the learning \"\n \"process by disproportionately affecting the model.\"\n ),\n parameter_metadata=FEATURE_METADATA[NUMBER][PREPROCESSING][\"outlier_threshold\"],\n )\n\n computed_outlier_fill_value: float = schema_utils.FloatRange(\n default=0.0,\n allow_none=False,\n description=\"The internally computed fill value to replace outliers with in case the \"\n \"outlier_strategy is fill_with_mode or fill_with_mean\",\n parameter_metadata=FEATURE_METADATA[NUMBER][PREPROCESSING][\"computed_outlier_fill_value\"],\n )\n\n\n@DeveloperAPI\n@register_preprocessor(\"number_output\")\n@ludwig_dataclass\nclass NumberOutputPreprocessingConfig(NumberPreprocessingConfig):\n missing_value_strategy: str = schema_utils.StringOptions(\n MISSING_VALUE_STRATEGY_OPTIONS + [FILL_WITH_MEAN],\n default=DROP_ROW,\n allow_none=False,\n description=\"What strategy to follow when there's a missing value in a number output feature\",\n parameter_metadata=FEATURE_METADATA[NUMBER][PREPROCESSING][\"missing_value_strategy\"],\n )\n\n normalization: str = schema_utils.StringOptions(\n [\"zscore\", \"minmax\", \"log1p\", \"iq\"],\n default=None,\n allow_none=True,\n description=\"Normalization strategy to use for this number feature.\",\n parameter_metadata=FEATURE_METADATA[NUMBER][PREPROCESSING][\"normalization\"],\n )\n"
},
{
"path": "ludwig/schema/features/preprocessing/sequence.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import DROP_ROW, FILL_WITH_CONST, MISSING_VALUE_STRATEGY_OPTIONS, PREPROCESSING, SEQUENCE\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import register_preprocessor\nfrom ludwig.schema.metadata import FEATURE_METADATA, PREPROCESSING_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\nfrom ludwig.utils import strings_utils\n\n\n@DeveloperAPI\n@register_preprocessor(SEQUENCE)\n@ludwig_dataclass\nclass SequencePreprocessingConfig(BasePreprocessingConfig):\n tokenizer: str = schema_utils.String(\n default=\"space\",\n allow_none=False,\n description=\"Defines how to map from the raw string content of the dataset column to a sequence of elements.\",\n parameter_metadata=FEATURE_METADATA[SEQUENCE][PREPROCESSING][\"tokenizer\"],\n )\n\n vocab_file: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"Filepath string to a UTF-8 encoded file containing the sequence's vocabulary. On each line the \"\n \"first string until \\t or \\n is considered a word.\",\n parameter_metadata=FEATURE_METADATA[SEQUENCE][PREPROCESSING][\"vocab_file\"],\n )\n\n sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"The desired length (number of tokens) of the sequence. Sequences that are longer than this value \"\n \"will be truncated and sequences shorter than this value will be padded. If None, sequence length will be \"\n \"inferred from the training dataset.\",\n parameter_metadata=FEATURE_METADATA[SEQUENCE][PREPROCESSING][\"sequence_length\"],\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=256,\n allow_none=True,\n description=\"The maximum length (number of tokens) of the sequence. Sequences that are longer than this value \"\n \"will be truncated. Useful as a stopgap measure if `sequence_length` is set to `None`. If `None`, max sequence \"\n \"length will be inferred from the training dataset.\",\n parameter_metadata=FEATURE_METADATA[SEQUENCE][PREPROCESSING][\"max_sequence_length\"],\n )\n\n most_common: int = schema_utils.PositiveInteger(\n default=20000,\n allow_none=False,\n description=\"The maximum number of most common tokens in the vocabulary. If the data contains more than this \"\n \"amount, the most infrequent symbols will be treated as unknown.\",\n parameter_metadata=FEATURE_METADATA[SEQUENCE][PREPROCESSING][\"most_common\"],\n )\n\n padding_symbol: str = schema_utils.String(\n default=strings_utils.PADDING_SYMBOL,\n allow_none=False,\n description=\"The string used as a padding symbol. This special token is mapped to the integer ID 0 in the \"\n \"vocabulary.\",\n parameter_metadata=FEATURE_METADATA[SEQUENCE][PREPROCESSING][\"padding_symbol\"],\n )\n\n unknown_symbol: str = schema_utils.String(\n default=strings_utils.UNKNOWN_SYMBOL,\n allow_none=False,\n description=\"The string used as an unknown placeholder. This special token is mapped to the integer ID 1 in \"\n \"the vocabulary.\",\n parameter_metadata=FEATURE_METADATA[SEQUENCE][PREPROCESSING][\"unknown_symbol\"],\n )\n\n padding: str = schema_utils.StringOptions(\n [\"left\", \"right\"],\n default=\"right\",\n allow_none=False,\n description=\"The direction of the padding.\",\n parameter_metadata=FEATURE_METADATA[SEQUENCE][PREPROCESSING][\"padding\"],\n )\n\n lowercase: bool = schema_utils.Boolean(\n default=False,\n description=\"If true, converts the string to lowercase before tokenizing.\",\n parameter_metadata=FEATURE_METADATA[SEQUENCE][PREPROCESSING][\"lowercase\"],\n )\n\n missing_value_strategy: str = schema_utils.StringOptions(\n MISSING_VALUE_STRATEGY_OPTIONS,\n default=FILL_WITH_CONST,\n allow_none=False,\n description=\"What strategy to follow when there's a missing value in a text column\",\n parameter_metadata=FEATURE_METADATA[SEQUENCE][PREPROCESSING][\"missing_value_strategy\"],\n )\n\n fill_value: str = schema_utils.String(\n default=strings_utils.UNKNOWN_SYMBOL,\n allow_none=False,\n description=\"The value to replace missing values with in case the missing_value_strategy is fill_with_const\",\n parameter_metadata=FEATURE_METADATA[SEQUENCE][PREPROCESSING][\"fill_value\"],\n )\n\n computed_fill_value: str = schema_utils.String(\n default=strings_utils.UNKNOWN_SYMBOL,\n allow_none=False,\n description=\"The internally computed fill value to replace missing values with in case the \"\n \"missing_value_strategy is fill_with_mode or fill_with_mean\",\n parameter_metadata=FEATURE_METADATA[SEQUENCE][PREPROCESSING][\"computed_fill_value\"],\n )\n\n ngram_size: int = schema_utils.PositiveInteger(\n default=2,\n allow_none=False,\n description=\"The size of the ngram when using the `ngram` tokenizer (e.g, 2 = bigram, 3 = trigram, etc.).\",\n parameter_metadata=FEATURE_METADATA[SEQUENCE][PREPROCESSING][\"ngram_size\"],\n )\n\n cache_encoder_embeddings: bool = schema_utils.Boolean(\n default=False,\n description=\"Compute encoder embeddings in preprocessing, speeding up training time considerably.\",\n parameter_metadata=PREPROCESSING_METADATA[\"cache_encoder_embeddings\"],\n )\n\n\n@DeveloperAPI\n@register_preprocessor(\"sequence_output\")\n@ludwig_dataclass\nclass SequenceOutputPreprocessingConfig(SequencePreprocessingConfig):\n missing_value_strategy: str = schema_utils.StringOptions(\n MISSING_VALUE_STRATEGY_OPTIONS,\n default=DROP_ROW,\n allow_none=False,\n description=\"What strategy to follow when there's a missing value in a sequence output feature\",\n parameter_metadata=FEATURE_METADATA[SEQUENCE][PREPROCESSING][\"missing_value_strategy\"],\n )\n\n sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"The desired length (number of tokens) of the sequence. Sequences that are longer than this value \"\n \"will be truncated and sequences shorter than this value will be padded. If None, sequence length will be \"\n \"inferred from the training dataset.\",\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=256,\n allow_none=True,\n description=\"The maximum length (number of tokens) of the sequence. Sequences that are longer than this value \"\n \"will be truncated. Useful as a stopgap measure if `sequence_length` is set to `None`. If `None`, max sequence \"\n \"length will be inferred from the training dataset.\",\n parameter_metadata=FEATURE_METADATA[SEQUENCE][PREPROCESSING][\"max_sequence_length\"],\n )\n\n tokenizer: str = schema_utils.String(\n default=\"space\",\n allow_none=False,\n description=\"Defines how to map from the raw string content of the dataset column to a sequence of elements.\",\n parameter_metadata=FEATURE_METADATA[SEQUENCE][PREPROCESSING][\"tokenizer\"],\n )\n\n lowercase: bool = schema_utils.Boolean(\n default=False,\n description=\"If true, converts the string to lowercase before tokenizing.\",\n parameter_metadata=FEATURE_METADATA[SEQUENCE][PREPROCESSING][\"lowercase\"],\n )\n\n most_common: int = schema_utils.PositiveInteger(\n default=20000,\n allow_none=False,\n description=\"The maximum number of most common tokens in the vocabulary. If the data contains more than this \"\n \"amount, the most infrequent symbols will be treated as unknown.\",\n parameter_metadata=FEATURE_METADATA[SEQUENCE][PREPROCESSING][\"most_common\"],\n )\n\n ngram_size: int = schema_utils.PositiveInteger(\n default=2,\n allow_none=False,\n description=\"The size of the ngram when using the `ngram` tokenizer (e.g, 2 = bigram, 3 = trigram, etc.).\",\n parameter_metadata=FEATURE_METADATA[SEQUENCE][PREPROCESSING][\"ngram_size\"],\n )\n"
},
{
"path": "ludwig/schema/features/preprocessing/set.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import DROP_ROW, FILL_WITH_CONST, MISSING_VALUE_STRATEGY_OPTIONS, PREPROCESSING, SET\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import register_preprocessor\nfrom ludwig.schema.metadata import FEATURE_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\nfrom ludwig.utils import strings_utils\n\n\n@DeveloperAPI\n@register_preprocessor(SET)\n@ludwig_dataclass\nclass SetPreprocessingConfig(BasePreprocessingConfig):\n tokenizer: str = schema_utils.String(\n default=\"space\",\n allow_none=False,\n description=\"Defines how to transform the raw text content of the dataset column to a set of elements. The \"\n \"default value space splits the string on spaces. Common options include: underscore (splits on \"\n \"underscore), comma (splits on comma), json (decodes the string into a set or a list through a \"\n \"JSON parser).\",\n parameter_metadata=FEATURE_METADATA[SET][PREPROCESSING][\"tokenizer\"],\n )\n\n missing_value_strategy: str = schema_utils.StringOptions(\n MISSING_VALUE_STRATEGY_OPTIONS,\n default=FILL_WITH_CONST,\n allow_none=False,\n description=\"What strategy to follow when there's a missing value in a set column\",\n parameter_metadata=FEATURE_METADATA[SET][PREPROCESSING][\"missing_value_strategy\"],\n )\n\n fill_value: str = schema_utils.String(\n default=strings_utils.UNKNOWN_SYMBOL,\n allow_none=False,\n description=\"The value to replace missing values with in case the missing_value_strategy is fill_with_const\",\n parameter_metadata=FEATURE_METADATA[SET][PREPROCESSING][\"fill_value\"],\n )\n\n computed_fill_value: str = schema_utils.String(\n default=strings_utils.UNKNOWN_SYMBOL,\n allow_none=False,\n description=\"The internally computed fill value to replace missing values with in case the \"\n \"missing_value_strategy is fill_with_mode or fill_with_mean\",\n parameter_metadata=FEATURE_METADATA[SET][PREPROCESSING][\"computed_fill_value\"],\n )\n\n lowercase: bool = schema_utils.Boolean(\n default=False,\n description=\"If true, converts the string to lowercase before tokenizing.\",\n parameter_metadata=FEATURE_METADATA[SET][PREPROCESSING][\"lowercase\"],\n )\n\n most_common: int = schema_utils.PositiveInteger(\n default=10000,\n allow_none=True,\n description=\"The maximum number of most common tokens to be considered. If the data contains more than this \"\n \"amount, the most infrequent tokens will be treated as unknown.\",\n parameter_metadata=FEATURE_METADATA[SET][PREPROCESSING][\"most_common\"],\n )\n\n\n@DeveloperAPI\n@register_preprocessor(\"set_output\")\n@ludwig_dataclass\nclass SetOutputPreprocessingConfig(SetPreprocessingConfig):\n missing_value_strategy: str = schema_utils.StringOptions(\n MISSING_VALUE_STRATEGY_OPTIONS,\n default=DROP_ROW,\n allow_none=False,\n description=\"What strategy to follow when there's a missing value in a set output feature\",\n parameter_metadata=FEATURE_METADATA[SET][PREPROCESSING][\"missing_value_strategy\"],\n )\n\n tokenizer: str = schema_utils.String(\n default=\"space\",\n allow_none=False,\n description=\"Defines how to transform the raw text content of the dataset column to a set of elements. The \"\n \"default value space splits the string on spaces. Common options include: underscore (splits on \"\n \"underscore), comma (splits on comma), json (decodes the string into a set or a list through a \"\n \"JSON parser).\",\n parameter_metadata=FEATURE_METADATA[SET][PREPROCESSING][\"tokenizer\"],\n )\n\n lowercase: bool = schema_utils.Boolean(\n default=False,\n description=\"If true, converts the string to lowercase before tokenizing.\",\n parameter_metadata=FEATURE_METADATA[SET][PREPROCESSING][\"lowercase\"],\n )\n\n most_common: int = schema_utils.PositiveInteger(\n default=10000,\n allow_none=True,\n description=\"The maximum number of most common tokens to be considered. If the data contains more than this \"\n \"amount, the most infrequent tokens will be treated as unknown.\",\n parameter_metadata=FEATURE_METADATA[SET][PREPROCESSING][\"most_common\"],\n )\n"
},
{
"path": "ludwig/schema/features/preprocessing/text.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import DROP_ROW, FILL_WITH_CONST, MISSING_VALUE_STRATEGY_OPTIONS, PREPROCESSING, TEXT\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import register_preprocessor\nfrom ludwig.schema.llms.prompt import PromptConfig, PromptConfigField\nfrom ludwig.schema.metadata import FEATURE_METADATA, PREPROCESSING_METADATA\nfrom ludwig.schema.metadata.parameter_metadata import INTERNAL_ONLY\nfrom ludwig.schema.utils import ludwig_dataclass\nfrom ludwig.utils import strings_utils\nfrom ludwig.utils.tokenizers import tokenizer_registry\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass BaseTextPreprocessingConfig(BasePreprocessingConfig):\n \"\"\"TextPreprocessingConfig is a dataclass that configures the parameters used for a text input feature.\"\"\"\n\n pretrained_model_name_or_path: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"This can be either the name of a pretrained HuggingFace model or a path where it was downloaded.\",\n parameter_metadata=FEATURE_METADATA[TEXT][PREPROCESSING][\"pretrained_model_name_or_path\"],\n )\n\n tokenizer: str = schema_utils.StringOptions(\n tokenizer_registry.keys(),\n default=\"space_punct\",\n allow_none=False,\n description=\"Defines how to map from the raw string content of the dataset column to a sequence of elements.\",\n parameter_metadata=FEATURE_METADATA[TEXT][PREPROCESSING][\"tokenizer\"],\n )\n\n vocab_file: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"Filepath string to a UTF-8 encoded file containing the sequence's vocabulary. On each line the \"\n \"first string until `\\\\t` or `\\\\n` is considered a word.\",\n parameter_metadata=FEATURE_METADATA[TEXT][PREPROCESSING][\"vocab_file\"],\n )\n\n sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"The desired length (number of tokens) of the sequence. Sequences that are longer than this value \"\n \"will be truncated and sequences shorter than this value will be padded. If None, sequence length will be \"\n \"inferred from the training dataset.\",\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=256,\n allow_none=True,\n description=\"The maximum length (number of tokens) of the sequence. Sequences that are longer than this value \"\n \"will be truncated. Useful as a stopgap measure if `sequence_length` is set to `None`. If `None`, max sequence \"\n \"length will be inferred from the training dataset.\",\n parameter_metadata=FEATURE_METADATA[TEXT][PREPROCESSING][\"max_sequence_length\"],\n )\n\n most_common: int = schema_utils.PositiveInteger(\n default=20000,\n allow_none=False,\n description=\"The maximum number of most common tokens in the vocabulary. If the data contains more than this \"\n \"amount, the most infrequent symbols will be treated as unknown.\",\n parameter_metadata=FEATURE_METADATA[TEXT][PREPROCESSING][\"most_common\"],\n )\n\n padding_symbol: str = schema_utils.String(\n default=strings_utils.PADDING_SYMBOL,\n allow_none=False,\n description=\"The string used as the padding symbol for sequence features. Ignored for features using \"\n \"huggingface encoders, which have their own vocabulary.\",\n parameter_metadata=FEATURE_METADATA[TEXT][PREPROCESSING][\"padding_symbol\"],\n )\n\n unknown_symbol: str = schema_utils.String(\n default=strings_utils.UNKNOWN_SYMBOL,\n allow_none=False,\n description=\"The string used as the unknown symbol for sequence features. Ignored for features using \"\n \"huggingface encoders, which have their own vocabulary.\",\n parameter_metadata=FEATURE_METADATA[TEXT][PREPROCESSING][\"unknown_symbol\"],\n )\n\n padding: str = schema_utils.StringOptions(\n [\"left\", \"right\"],\n default=\"right\",\n allow_none=False,\n description=\"The direction of the padding.\",\n parameter_metadata=FEATURE_METADATA[TEXT][PREPROCESSING][\"padding\"],\n )\n\n lowercase: bool = schema_utils.Boolean(\n default=False,\n description=\"If true, converts the string to lowercase before tokenizing.\",\n parameter_metadata=FEATURE_METADATA[TEXT][PREPROCESSING][\"lowercase\"],\n )\n\n missing_value_strategy: str = schema_utils.StringOptions(\n MISSING_VALUE_STRATEGY_OPTIONS,\n default=FILL_WITH_CONST,\n allow_none=False,\n description=\"What strategy to follow when there's a missing value in a text column.\",\n parameter_metadata=FEATURE_METADATA[TEXT][PREPROCESSING][\"missing_value_strategy\"],\n )\n\n fill_value: str = schema_utils.String(\n default=strings_utils.UNKNOWN_SYMBOL,\n allow_none=False,\n description=(\n \"The value to replace missing values with in case the `missing_value_strategy` is `fill_with_const`.\"\n ),\n parameter_metadata=FEATURE_METADATA[TEXT][PREPROCESSING][\"fill_value\"],\n )\n\n computed_fill_value: str = schema_utils.String(\n default=strings_utils.UNKNOWN_SYMBOL,\n allow_none=False,\n description=\"The internally computed fill value to replace missing values with in case the \"\n \"`missing_value_strategy` is `fill_with_mode` or `fill_with_mean`.\",\n parameter_metadata=FEATURE_METADATA[TEXT][PREPROCESSING][\"computed_fill_value\"],\n )\n\n ngram_size: int = schema_utils.PositiveInteger(\n default=2,\n allow_none=False,\n description=\"The size of the ngram when using the `ngram` tokenizer (e.g, 2 = bigram, 3 = trigram, etc.).\",\n parameter_metadata=FEATURE_METADATA[TEXT][PREPROCESSING][\"ngram_size\"],\n )\n\n cache_encoder_embeddings: bool = schema_utils.Boolean(\n default=False,\n description=(\n \"For pretrained encoders, compute encoder embeddings in preprocessing, \"\n \"speeding up training time considerably. Only supported when `encoder.trainable=false`.\"\n ),\n parameter_metadata=PREPROCESSING_METADATA[\"cache_encoder_embeddings\"],\n )\n\n compute_idf: bool = schema_utils.Boolean(\n default=False,\n parameter_metadata=INTERNAL_ONLY,\n )\n\n\n@DeveloperAPI\n@register_preprocessor(TEXT)\n@ludwig_dataclass\nclass TextPreprocessingConfig(BaseTextPreprocessingConfig):\n \"\"\"TextPreprocessingConfig is a dataclass that configures the parameters used for a text input feature.\"\"\"\n\n prompt: PromptConfig = PromptConfigField().get_default_field()\n\n\n@DeveloperAPI\n@register_preprocessor(\"text_llm_input\")\n@ludwig_dataclass\nclass LLMTextInputPreprocessingConfig(BaseTextPreprocessingConfig):\n \"\"\"LLMs require the prompt to be provided at the top-level, not preprocessing.\"\"\"\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"The maximum length (number of tokens) of the sequence. Sequences that are longer than this value \"\n \"will be truncated. Useful as a stopgap measure if `sequence_length` is set to `None`. If `None`, max sequence \"\n \"length will be inferred from the training dataset.\",\n parameter_metadata=FEATURE_METADATA[TEXT][PREPROCESSING][\"max_sequence_length\"],\n )\n\n\n@DeveloperAPI\n@register_preprocessor(\"text_output\")\n@ludwig_dataclass\nclass TextOutputPreprocessingConfig(BaseTextPreprocessingConfig):\n missing_value_strategy: str = schema_utils.StringOptions(\n MISSING_VALUE_STRATEGY_OPTIONS,\n default=DROP_ROW,\n allow_none=False,\n description=\"What strategy to follow when there's a missing value in a text output feature.\",\n parameter_metadata=FEATURE_METADATA[TEXT][PREPROCESSING][\"missing_value_strategy\"],\n )\n\n sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"The desired length (number of tokens) of the sequence. Sequences that are longer than this value \"\n \"will be truncated and sequences shorter than this value will be padded. If None, sequence length will be \"\n \"inferred from the training dataset.\",\n parameter_metadata=FEATURE_METADATA[TEXT][PREPROCESSING][\"sequence_length\"],\n )\n\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=256,\n allow_none=True,\n description=\"The maximum length (number of tokens) of the sequence. Sequences that are longer than this value \"\n \"will be truncated. Useful as a stopgap measure if `sequence_length` is set to `None`. If `None`, max sequence \"\n \"length will be inferred from the training dataset.\",\n parameter_metadata=FEATURE_METADATA[TEXT][PREPROCESSING][\"max_sequence_length\"],\n )\n\n tokenizer: str = schema_utils.StringOptions(\n tokenizer_registry.keys(),\n default=\"space_punct\",\n allow_none=False,\n description=\"Defines how to map from the raw string content of the dataset column to a sequence of elements.\",\n parameter_metadata=FEATURE_METADATA[TEXT][PREPROCESSING][\"tokenizer\"],\n )\n\n lowercase: bool = schema_utils.Boolean(\n default=False,\n description=\"If true, converts the string to lowercase before tokenizing.\",\n parameter_metadata=FEATURE_METADATA[TEXT][PREPROCESSING][\"lowercase\"],\n )\n\n most_common: int = schema_utils.PositiveInteger(\n default=20000,\n allow_none=False,\n description=\"The maximum number of most common tokens in the vocabulary. If the data contains more than this \"\n \"amount, the most infrequent symbols will be treated as unknown.\",\n parameter_metadata=FEATURE_METADATA[TEXT][PREPROCESSING][\"most_common\"],\n )\n\n ngram_size: int = schema_utils.PositiveInteger(\n default=2,\n allow_none=False,\n description=\"The size of the ngram when using the `ngram` tokenizer (e.g, 2 = bigram, 3 = trigram, etc.).\",\n parameter_metadata=FEATURE_METADATA[TEXT][PREPROCESSING][\"ngram_size\"],\n )\n\n\n@DeveloperAPI\n@register_preprocessor(\"text_llm_output\")\n@ludwig_dataclass\nclass LLMTextOutputPreprocessingConfig(TextOutputPreprocessingConfig):\n max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"The maximum length (number of tokens) of the sequence. Sequences that are longer than this value \"\n \"will be truncated. Useful as a stopgap measure if `sequence_length` is set to `None`. If `None`, max sequence \"\n \"length will be inferred from the training dataset.\",\n parameter_metadata=FEATURE_METADATA[TEXT][PREPROCESSING][\"max_sequence_length\"],\n )\n"
},
{
"path": "ludwig/schema/features/preprocessing/timeseries.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import DROP_ROW, FILL_WITH_CONST, MISSING_VALUE_STRATEGY_OPTIONS, PREPROCESSING, TIMESERIES\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import register_preprocessor\nfrom ludwig.schema.metadata import FEATURE_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\nfrom ludwig.utils.tokenizers import tokenizer_registry\n\n\n@ludwig_dataclass\nclass BaseTimeseriesPreprocessingConfig(BasePreprocessingConfig):\n tokenizer: str = schema_utils.StringOptions(\n tokenizer_registry.keys(),\n default=\"space\",\n allow_none=False,\n description=\"Defines how to map from the raw string content of the dataset column to a sequence of elements.\",\n parameter_metadata=FEATURE_METADATA[TIMESERIES][PREPROCESSING][\"tokenizer\"],\n )\n\n timeseries_length_limit: int = schema_utils.PositiveInteger(\n default=256,\n allow_none=False,\n description=\"Defines the maximum length of the timeseries. All timeseries longer than this limit are cut off.\",\n parameter_metadata=FEATURE_METADATA[TIMESERIES][PREPROCESSING][\"timeseries_length_limit\"],\n )\n\n padding_value: float = schema_utils.NonNegativeFloat(\n default=0.0,\n allow_none=False,\n description=\"Float value that is used for padding and replacing missing values within a row.\",\n parameter_metadata=FEATURE_METADATA[TIMESERIES][PREPROCESSING][\"padding_value\"],\n )\n\n padding: str = schema_utils.StringOptions(\n [\"left\", \"right\"],\n default=\"right\",\n allow_none=False,\n description=\"The direction of the padding.\",\n parameter_metadata=FEATURE_METADATA[TIMESERIES][PREPROCESSING][\"padding\"],\n )\n\n missing_value_strategy: str = schema_utils.StringOptions(\n MISSING_VALUE_STRATEGY_OPTIONS,\n default=FILL_WITH_CONST,\n allow_none=False,\n description=(\n \"What strategy to follow when there's a missing value in a column. Currently applies only to a row missing \"\n \"in its entirety, not invididual elements within the row. For now, `NaN` values within a row are filled \"\n \"using the `padding_value`.\"\n ),\n parameter_metadata=FEATURE_METADATA[TIMESERIES][PREPROCESSING][\"missing_value_strategy\"],\n )\n\n fill_value: str = schema_utils.String(\n default=\"\",\n allow_none=False,\n description=(\n \"The value to replace missing values with in case the `missing_value_strategy` is `fill_with_const`.\"\n ),\n parameter_metadata=FEATURE_METADATA[TIMESERIES][PREPROCESSING][\"fill_value\"],\n )\n\n computed_fill_value: str = schema_utils.String(\n default=\"\",\n allow_none=False,\n description=(\n \"The internally computed fill value to replace missing values with in case the \"\n \"`missing_value_strategy` is `fill_with_mode` or `fill_with_mean`.\"\n ),\n parameter_metadata=FEATURE_METADATA[TIMESERIES][PREPROCESSING][\"computed_fill_value\"],\n )\n\n\n@DeveloperAPI\n@register_preprocessor(TIMESERIES)\n@ludwig_dataclass\nclass TimeseriesPreprocessingConfig(BaseTimeseriesPreprocessingConfig):\n window_size: int = schema_utils.NonNegativeInteger(\n default=0,\n allow_none=False,\n description=(\n \"Optional lookback window size used to convert a column-major dataset (one observation per row) \"\n \"into a row-major dataset (each row has a timeseries window of observations). Starting from a given \"\n \"observation, a sliding window is taken going `window_size - 1` rows back to form the timeseries input \"\n \"feature. If this value is left as 0, then it is assumed that the dataset has been provided in row-major \"\n \"format (i.e., it has already been preprocessed such that each row is a timeseries window).\"\n ),\n )\n\n missing_value_strategy: str = schema_utils.StringOptions(\n MISSING_VALUE_STRATEGY_OPTIONS,\n default=FILL_WITH_CONST,\n allow_none=False,\n description=\"What strategy to follow when a row of data is missing.\",\n parameter_metadata=FEATURE_METADATA[TIMESERIES][PREPROCESSING][\"missing_value_strategy\"],\n )\n\n\n@DeveloperAPI\n@register_preprocessor(\"timeseries_output\")\n@ludwig_dataclass\nclass TimeseriesOutputPreprocessingConfig(BaseTimeseriesPreprocessingConfig):\n horizon: int = schema_utils.NonNegativeInteger(\n default=0,\n allow_none=False,\n description=(\n \"Optional forecasting horizon used to convert a column-major dataset (one observation per row) \"\n \"into a row-major dataset (each row has a timeseries window of observations). Starting from a given \"\n \"observation, a sliding window is token going `horizon` rows forward in time, excluding the observation \"\n \"in the current row. If this value is left as 0, then it is assumed that the dataset has been provided in \"\n \"row-major format (i.e., it has already been preprocessed such that each row is a timeseries window).\"\n ),\n )\n\n missing_value_strategy: str = schema_utils.StringOptions(\n MISSING_VALUE_STRATEGY_OPTIONS,\n default=DROP_ROW,\n allow_none=False,\n description=\"What strategy to follow when a row of data is missing.\",\n parameter_metadata=FEATURE_METADATA[TIMESERIES][PREPROCESSING][\"missing_value_strategy\"],\n )\n"
},
{
"path": "ludwig/schema/features/preprocessing/utils.py",
"content": "from dataclasses import field\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.utils.registry import Registry\n\npreprocessing_registry = Registry()\n\n\n@DeveloperAPI\ndef register_preprocessor(name: str):\n def wrap(preprocessing_config: BasePreprocessingConfig):\n preprocessing_registry[name] = preprocessing_config\n return preprocessing_config\n\n return wrap\n\n\n@DeveloperAPI\ndef PreprocessingDataclassField(feature_type: str):\n \"\"\"Custom dataclass field that when used inside a dataclass will allow the user to specify a preprocessing\n config.\n\n Returns: Initialized dataclass field that converts an untyped dict with params to a preprocessing config.\n \"\"\"\n\n class PreprocessingMarshmallowField(schema_utils.LudwigSchemaField):\n \"\"\"Custom field that deserializes a dict for a valid preprocessing config from the preprocessing_registry\n and creates a corresponding JSON schema for external usage.\"\"\"\n\n def _deserialize(self, value, attr, data, **kwargs):\n if value is None:\n return None\n if isinstance(value, dict):\n if feature_type in preprocessing_registry:\n pre = preprocessing_registry[feature_type]\n try:\n return pre.Schema().load(value)\n except (TypeError, ConfigValidationError) as error:\n raise ConfigValidationError(\n f\"Invalid preprocessing params: {value}, see `{pre}` definition. Error: {error}\"\n )\n raise ConfigValidationError(\n f\"Invalid params for preprocessor: {value}, expect dict with at least a valid `type` attribute.\"\n )\n raise ConfigValidationError(\"Field should be None or dict\")\n\n def _jsonschema_type_mapping(self):\n preprocessor_cls = preprocessing_registry[feature_type]\n props = schema_utils.unload_jsonschema_from_marshmallow_class(preprocessor_cls)[\"properties\"]\n return {\n \"type\": \"object\",\n \"properties\": props,\n \"title\": \"preprocessing_options\",\n \"additionalProperties\": True,\n }\n\n try:\n preprocessor = preprocessing_registry[feature_type]\n load_default = lambda: preprocessor.Schema().load({})\n dump_default = preprocessor.Schema().dump({})\n\n return field(\n metadata={\n \"marshmallow_field\": PreprocessingMarshmallowField(\n allow_none=False,\n dump_default=dump_default,\n load_default=load_default,\n )\n },\n default_factory=load_default,\n )\n except Exception as e:\n raise ConfigValidationError(\n f\"Unsupported preprocessing type: {feature_type}. See preprocessing_registry. \" f\"Details: {e}\"\n )\n"
},
{
"path": "ludwig/schema/features/preprocessing/vector.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import DROP_ROW, FILL_WITH_CONST, MISSING_VALUE_STRATEGY_OPTIONS, PREPROCESSING, VECTOR\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import register_preprocessor\nfrom ludwig.schema.metadata import FEATURE_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@register_preprocessor(VECTOR)\n@ludwig_dataclass\nclass VectorPreprocessingConfig(BasePreprocessingConfig):\n vector_size: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"The size of the vector. If None, the vector size will be inferred from the data.\",\n parameter_metadata=FEATURE_METADATA[VECTOR][PREPROCESSING][\"vector_size\"],\n )\n\n missing_value_strategy: str = schema_utils.StringOptions(\n MISSING_VALUE_STRATEGY_OPTIONS,\n default=FILL_WITH_CONST,\n allow_none=False,\n description=\"What strategy to follow when there's a missing value in a vector column\",\n parameter_metadata=FEATURE_METADATA[VECTOR][PREPROCESSING][\"missing_value_strategy\"],\n )\n\n fill_value: str = schema_utils.String(\n default=\"\",\n allow_none=False,\n pattern=r\"^([0-9]+(\\.[0-9]*)?\\s*)*$\",\n description=\"The value to replace missing values with in case the missing_value_strategy is fill_with_const\",\n parameter_metadata=FEATURE_METADATA[VECTOR][PREPROCESSING][\"fill_value\"],\n )\n\n computed_fill_value: str = schema_utils.String(\n default=\"\",\n allow_none=False,\n pattern=r\"^([0-9]+(\\.[0-9]*)?\\s*)*$\",\n description=\"The internally computed fill value to replace missing values with in case the \"\n \"missing_value_strategy is fill_with_mode or fill_with_mean\",\n parameter_metadata=FEATURE_METADATA[VECTOR][PREPROCESSING][\"computed_fill_value\"],\n )\n\n\n@DeveloperAPI\n@register_preprocessor(\"vector_output\")\n@ludwig_dataclass\nclass VectorOutputPreprocessingConfig(VectorPreprocessingConfig):\n missing_value_strategy: str = schema_utils.StringOptions(\n MISSING_VALUE_STRATEGY_OPTIONS,\n default=DROP_ROW,\n allow_none=False,\n description=\"What strategy to follow when there's a missing value in a vector output feature\",\n parameter_metadata=FEATURE_METADATA[VECTOR][PREPROCESSING][\"missing_value_strategy\"],\n )\n\n vector_size: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"The size of the vector. If None, the vector size will be inferred from the data.\",\n parameter_metadata=FEATURE_METADATA[VECTOR][PREPROCESSING][\"vector_size\"],\n )\n"
},
{
"path": "ludwig/schema/features/sequence_feature.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import LOSS, MODEL_ECD, SEQUENCE, SEQUENCE_SOFTMAX_CROSS_ENTROPY\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.decoders.base import BaseDecoderConfig\nfrom ludwig.schema.decoders.utils import DecoderDataclassField\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import EncoderDataclassField\nfrom ludwig.schema.features.base import BaseInputFeatureConfig, BaseOutputFeatureConfig\nfrom ludwig.schema.features.loss.loss import BaseLossConfig\nfrom ludwig.schema.features.loss.utils import LossDataclassField\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import PreprocessingDataclassField\nfrom ludwig.schema.features.utils import (\n ecd_defaults_config_registry,\n ecd_input_config_registry,\n ecd_output_config_registry,\n input_mixin_registry,\n output_mixin_registry,\n)\nfrom ludwig.schema.metadata import FEATURE_METADATA\nfrom ludwig.schema.metadata.parameter_metadata import INTERNAL_ONLY\nfrom ludwig.schema.utils import BaseMarshmallowConfig, ludwig_dataclass\n\n\n@DeveloperAPI\n@input_mixin_registry.register(SEQUENCE)\n@ludwig_dataclass\nclass SequenceInputFeatureConfigMixin(BaseMarshmallowConfig):\n \"\"\"SequenceInputFeatureConfigMixin is a dataclass that configures the parameters used in both the sequence\n input feature and the sequence global defaults section of the Ludwig Config.\"\"\"\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=SEQUENCE)\n\n encoder: BaseEncoderConfig = EncoderDataclassField(\n MODEL_ECD,\n feature_type=SEQUENCE,\n default=\"embed\",\n )\n\n\n@DeveloperAPI\n@ecd_input_config_registry.register(SEQUENCE)\n@ludwig_dataclass\nclass SequenceInputFeatureConfig(SequenceInputFeatureConfigMixin, BaseInputFeatureConfig):\n \"\"\"SequenceInputFeatureConfig is a dataclass that configures the parameters used for a sequence input\n feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(SEQUENCE)\n\n\n@DeveloperAPI\n@output_mixin_registry.register(SEQUENCE)\n@ludwig_dataclass\nclass SequenceOutputFeatureConfigMixin(BaseMarshmallowConfig):\n \"\"\"SequenceOutputFeatureConfigMixin is a dataclass that configures the parameters used in both the sequence\n output feature and the sequence global defaults section of the Ludwig Config.\"\"\"\n\n decoder: BaseDecoderConfig = DecoderDataclassField(\n MODEL_ECD,\n feature_type=SEQUENCE,\n default=\"generator\",\n )\n\n loss: BaseLossConfig = LossDataclassField(\n feature_type=SEQUENCE,\n default=SEQUENCE_SOFTMAX_CROSS_ENTROPY,\n )\n\n\n@DeveloperAPI\n@ecd_output_config_registry.register(SEQUENCE)\n@ludwig_dataclass\nclass SequenceOutputFeatureConfig(SequenceOutputFeatureConfigMixin, BaseOutputFeatureConfig):\n \"\"\"SequenceOutputFeatureConfig is a dataclass that configures the parameters used for a sequence output\n feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(SEQUENCE)\n\n default_validation_metric: str = schema_utils.StringOptions(\n [LOSS],\n default=LOSS,\n description=\"Internal only use parameter: default validation metric for sequence output feature.\",\n parameter_metadata=INTERNAL_ONLY,\n )\n\n dependencies: list = schema_utils.List(\n default=[],\n description=\"List of input features that this feature depends on.\",\n parameter_metadata=FEATURE_METADATA[SEQUENCE][\"dependencies\"],\n )\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=\"sequence_output\")\n\n reduce_dependencies: str = schema_utils.ReductionOptions(\n default=\"sum\",\n description=\"How to reduce the dependencies of the output feature.\",\n parameter_metadata=FEATURE_METADATA[SEQUENCE][\"reduce_dependencies\"],\n )\n\n reduce_input: str = schema_utils.ReductionOptions(\n default=\"sum\",\n description=\"How to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first \"\n \"dimension (second if you count the batch dimension)\",\n parameter_metadata=FEATURE_METADATA[SEQUENCE][\"reduce_input\"],\n )\n\n\n@DeveloperAPI\n@ecd_defaults_config_registry.register(SEQUENCE)\n@ludwig_dataclass\nclass SequenceDefaultsConfig(SequenceInputFeatureConfigMixin, SequenceOutputFeatureConfigMixin):\n pass\n"
},
{
"path": "ludwig/schema/features/set_feature.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import JACCARD, MODEL_ECD, SET, SIGMOID_CROSS_ENTROPY\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.decoders.base import BaseDecoderConfig\nfrom ludwig.schema.decoders.utils import DecoderDataclassField\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import EncoderDataclassField\nfrom ludwig.schema.features.base import BaseInputFeatureConfig, BaseOutputFeatureConfig\nfrom ludwig.schema.features.loss.loss import BaseLossConfig\nfrom ludwig.schema.features.loss.utils import LossDataclassField\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import PreprocessingDataclassField\nfrom ludwig.schema.features.utils import (\n ecd_defaults_config_registry,\n ecd_input_config_registry,\n ecd_output_config_registry,\n input_mixin_registry,\n output_mixin_registry,\n)\nfrom ludwig.schema.metadata import FEATURE_METADATA\nfrom ludwig.schema.metadata.parameter_metadata import INTERNAL_ONLY\nfrom ludwig.schema.utils import BaseMarshmallowConfig, ludwig_dataclass\n\n\n@DeveloperAPI\n@input_mixin_registry.register(SET)\n@ludwig_dataclass\nclass SetInputFeatureConfigMixin(BaseMarshmallowConfig):\n \"\"\"SetInputFeatureConfigMixin is a dataclass that configures the parameters used in both the set input feature\n and the set global defaults section of the Ludwig Config.\"\"\"\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=SET)\n\n encoder: BaseEncoderConfig = EncoderDataclassField(\n MODEL_ECD,\n feature_type=SET,\n default=\"embed\",\n )\n\n\n@DeveloperAPI\n@ecd_input_config_registry.register(SET)\n@ludwig_dataclass\nclass SetInputFeatureConfig(SetInputFeatureConfigMixin, BaseInputFeatureConfig):\n \"\"\"SetInputFeatureConfig is a dataclass that configures the parameters used for a set input feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(SET)\n\n\n@DeveloperAPI\n@output_mixin_registry.register(SET)\n@ludwig_dataclass\nclass SetOutputFeatureConfigMixin(BaseMarshmallowConfig):\n \"\"\"SetOutputFeatureConfigMixin is a dataclass that configures the parameters used in both the set output\n feature and the set global defaults section of the Ludwig Config.\"\"\"\n\n decoder: BaseDecoderConfig = DecoderDataclassField(\n MODEL_ECD,\n feature_type=SET,\n default=\"classifier\",\n )\n\n loss: BaseLossConfig = LossDataclassField(\n feature_type=SET,\n default=SIGMOID_CROSS_ENTROPY,\n )\n\n\n@DeveloperAPI\n@ecd_output_config_registry.register(SET)\n@ludwig_dataclass\nclass SetOutputFeatureConfig(SetOutputFeatureConfigMixin, BaseOutputFeatureConfig):\n \"\"\"SetOutputFeatureConfig is a dataclass that configures the parameters used for a set output feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(SET)\n\n default_validation_metric: str = schema_utils.StringOptions(\n [JACCARD],\n default=JACCARD,\n description=\"Internal only use parameter: default validation metric for set output feature.\",\n parameter_metadata=INTERNAL_ONLY,\n )\n\n dependencies: list = schema_utils.List(\n default=[],\n description=\"List of input features that this feature depends on.\",\n parameter_metadata=FEATURE_METADATA[SET][\"dependencies\"],\n )\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=\"set_output\")\n\n reduce_dependencies: str = schema_utils.ReductionOptions(\n default=\"sum\",\n description=\"How to reduce the dependencies of the output feature.\",\n parameter_metadata=FEATURE_METADATA[SET][\"reduce_dependencies\"],\n )\n\n reduce_input: str = schema_utils.ReductionOptions(\n default=\"sum\",\n description=\"How to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first \"\n \"dimension (second if you count the batch dimension)\",\n parameter_metadata=FEATURE_METADATA[SET][\"reduce_input\"],\n )\n\n threshold: float = schema_utils.FloatRange(\n default=0.5,\n min=0,\n max=1,\n description=\"The threshold used to convert output probabilities to predictions. Tokens with predicted\"\n \"probabilities greater than or equal to threshold are predicted to be in the output set (True).\",\n parameter_metadata=FEATURE_METADATA[SET][\"threshold\"],\n )\n\n\n@DeveloperAPI\n@ecd_defaults_config_registry.register(SET)\n@ludwig_dataclass\nclass SetDefaultsConfig(SetInputFeatureConfigMixin, SetOutputFeatureConfigMixin):\n pass\n"
},
{
"path": "ludwig/schema/features/text_feature.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import (\n LOSS,\n MODEL_ECD,\n MODEL_LLM,\n NEXT_TOKEN_SOFTMAX_CROSS_ENTROPY,\n SEQUENCE_SOFTMAX_CROSS_ENTROPY,\n TEXT,\n)\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.decoders.base import BaseDecoderConfig\nfrom ludwig.schema.decoders.utils import DecoderDataclassField\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import EncoderDataclassField\nfrom ludwig.schema.features.base import BaseInputFeatureConfig, BaseOutputFeatureConfig\nfrom ludwig.schema.features.loss.loss import BaseLossConfig\nfrom ludwig.schema.features.loss.utils import LossDataclassField\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import PreprocessingDataclassField\nfrom ludwig.schema.features.utils import (\n ecd_defaults_config_registry,\n ecd_input_config_registry,\n ecd_output_config_registry,\n input_mixin_registry,\n llm_defaults_config_registry,\n llm_input_config_registry,\n llm_output_config_registry,\n output_mixin_registry,\n)\nfrom ludwig.schema.metadata import FEATURE_METADATA\nfrom ludwig.schema.metadata.parameter_metadata import INTERNAL_ONLY\nfrom ludwig.schema.utils import BaseMarshmallowConfig, ludwig_dataclass\n\n\n@DeveloperAPI\n@input_mixin_registry.register(TEXT)\n@ludwig_dataclass\nclass TextInputFeatureConfigMixin(BaseMarshmallowConfig):\n \"\"\"TextInputFeatureConfigMixin is a dataclass that configures the parameters used in both the text input\n feature and the text global defaults section of the Ludwig Config.\"\"\"\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=TEXT)\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass TextInputFeatureConfig(TextInputFeatureConfigMixin, BaseInputFeatureConfig):\n \"\"\"TextInputFeatureConfig is a dataclass that configures the parameters used for a text input feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(TEXT)\n\n encoder: BaseEncoderConfig = None\n\n\n@DeveloperAPI\n@ecd_input_config_registry.register(TEXT)\n@ludwig_dataclass\nclass ECDTextInputFeatureConfig(TextInputFeatureConfig):\n encoder: BaseEncoderConfig = EncoderDataclassField(\n MODEL_ECD,\n feature_type=TEXT,\n default=\"parallel_cnn\",\n )\n\n\n@DeveloperAPI\n@llm_input_config_registry.register(TEXT)\n@ludwig_dataclass\nclass LLMTextInputFeatureConfig(TextInputFeatureConfig):\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=\"text_llm_input\")\n\n encoder: BaseEncoderConfig = EncoderDataclassField(\n MODEL_LLM,\n feature_type=TEXT,\n default=\"passthrough\",\n )\n\n\n@DeveloperAPI\n@output_mixin_registry.register(TEXT)\n@ludwig_dataclass\nclass TextOutputFeatureConfigMixin(BaseMarshmallowConfig):\n \"\"\"TextOutputFeatureConfigMixin is a dataclass that configures the parameters used in both the text output\n feature and the text global defaults section of the Ludwig Config.\"\"\"\n\n decoder: BaseDecoderConfig = None\n\n loss: BaseLossConfig = LossDataclassField(\n feature_type=TEXT,\n default=SEQUENCE_SOFTMAX_CROSS_ENTROPY,\n )\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass TextOutputFeatureConfig(TextOutputFeatureConfigMixin, BaseOutputFeatureConfig):\n \"\"\"TextOutputFeatureConfig is a dataclass that configures the parameters used for a text output feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(TEXT)\n\n class_similarities: list = schema_utils.List(\n list,\n default=None,\n description=\"If not null this parameter is a c x c matrix in the form of a list of lists that contains the \"\n \"mutual similarity of classes. It is used if `class_similarities_temperature` is greater than 0. \",\n parameter_metadata=FEATURE_METADATA[TEXT][\"class_similarities\"],\n )\n\n default_validation_metric: str = schema_utils.StringOptions(\n [LOSS],\n default=LOSS,\n description=\"Internal only use parameter: default validation metric for binary output feature.\",\n parameter_metadata=INTERNAL_ONLY,\n )\n\n dependencies: list = schema_utils.List(\n default=[],\n description=\"List of input features that this feature depends on.\",\n parameter_metadata=FEATURE_METADATA[TEXT][\"dependencies\"],\n )\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=\"text_output\")\n\n reduce_dependencies: str = schema_utils.ReductionOptions(\n default=\"sum\",\n description=\"How to reduce the dependencies of the output feature.\",\n parameter_metadata=FEATURE_METADATA[TEXT][\"reduce_dependencies\"],\n )\n\n reduce_input: str = schema_utils.ReductionOptions(\n default=\"sum\",\n description=\"How to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first \"\n \"dimension (second if you count the batch dimension)\",\n parameter_metadata=FEATURE_METADATA[TEXT][\"reduce_input\"],\n )\n\n\n@DeveloperAPI\n@ecd_output_config_registry.register(TEXT)\n@ludwig_dataclass\nclass ECDTextOutputFeatureConfig(TextOutputFeatureConfig):\n decoder: BaseDecoderConfig = DecoderDataclassField(\n MODEL_ECD,\n feature_type=TEXT,\n default=\"generator\",\n )\n\n\n@DeveloperAPI\n@llm_output_config_registry.register(TEXT)\n@ludwig_dataclass\nclass LLMTextOutputFeatureConfig(TextOutputFeatureConfig):\n default_validation_metric: str = schema_utils.StringOptions(\n [LOSS],\n default=LOSS,\n description=\"Internal only use parameter: default validation metric for text output feature for LLMs.\",\n parameter_metadata=INTERNAL_ONLY,\n )\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=\"text_llm_output\")\n\n decoder: BaseDecoderConfig = DecoderDataclassField(\n MODEL_LLM,\n feature_type=TEXT,\n default=\"text_extractor\",\n )\n\n loss: BaseLossConfig = LossDataclassField(\n feature_type=TEXT,\n default=NEXT_TOKEN_SOFTMAX_CROSS_ENTROPY,\n )\n\n\n@DeveloperAPI\n@ecd_defaults_config_registry.register(TEXT)\n@ludwig_dataclass\nclass ECDTextDefaultsConfig(TextInputFeatureConfigMixin, TextOutputFeatureConfigMixin):\n encoder: BaseEncoderConfig = EncoderDataclassField(\n MODEL_ECD,\n feature_type=TEXT,\n default=\"parallel_cnn\",\n )\n\n decoder: BaseDecoderConfig = DecoderDataclassField(\n MODEL_ECD,\n feature_type=TEXT,\n default=\"generator\",\n )\n\n loss: BaseLossConfig = LossDataclassField(\n feature_type=TEXT,\n default=SEQUENCE_SOFTMAX_CROSS_ENTROPY,\n )\n\n\n@DeveloperAPI\n@llm_defaults_config_registry.register(TEXT)\n@ludwig_dataclass\nclass LLMTextDefaultsConfig(TextInputFeatureConfigMixin, TextOutputFeatureConfigMixin):\n encoder: BaseEncoderConfig = EncoderDataclassField(\n MODEL_LLM,\n feature_type=TEXT,\n default=\"passthrough\",\n )\n\n decoder: BaseDecoderConfig = DecoderDataclassField(\n MODEL_LLM,\n feature_type=TEXT,\n default=\"text_extractor\",\n )\n\n # TODO(Arnav): Refactor LossDataclassField to only accept loss types that are valid for the model\n loss: BaseLossConfig = LossDataclassField(\n feature_type=TEXT,\n default=NEXT_TOKEN_SOFTMAX_CROSS_ENTROPY,\n )\n"
},
{
"path": "ludwig/schema/features/timeseries_feature.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import HUBER, MEAN_SQUARED_ERROR, MODEL_ECD, TIMESERIES, VECTOR\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.decoders.base import BaseDecoderConfig\nfrom ludwig.schema.decoders.utils import DecoderDataclassField\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import EncoderDataclassField\nfrom ludwig.schema.features.base import BaseInputFeatureConfig, BaseOutputFeatureConfig\nfrom ludwig.schema.features.loss.loss import BaseLossConfig\nfrom ludwig.schema.features.loss.utils import LossDataclassField\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import PreprocessingDataclassField\nfrom ludwig.schema.features.utils import (\n ecd_defaults_config_registry,\n ecd_input_config_registry,\n ecd_output_config_registry,\n input_mixin_registry,\n output_mixin_registry,\n)\nfrom ludwig.schema.metadata import FEATURE_METADATA\nfrom ludwig.schema.metadata.parameter_metadata import INTERNAL_ONLY\nfrom ludwig.schema.utils import BaseMarshmallowConfig, ludwig_dataclass\n\n\n@DeveloperAPI\n@input_mixin_registry.register(TIMESERIES)\n@ludwig_dataclass\nclass TimeseriesInputFeatureConfigMixin(BaseMarshmallowConfig):\n \"\"\"TimeseriesInputFeatureConfigMixin is a dataclass that configures the parameters used in both the timeseries\n input feature and the timeseries global defaults section of the Ludwig Config.\"\"\"\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=TIMESERIES)\n\n encoder: BaseEncoderConfig = EncoderDataclassField(\n MODEL_ECD,\n feature_type=TIMESERIES,\n default=\"parallel_cnn\",\n )\n\n\n@DeveloperAPI\n@ecd_input_config_registry.register(TIMESERIES)\n@ludwig_dataclass\nclass TimeseriesInputFeatureConfig(TimeseriesInputFeatureConfigMixin, BaseInputFeatureConfig):\n \"\"\"TimeseriesInputFeatureConfig is a dataclass that configures the parameters used for a timeseries input\n feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(TIMESERIES)\n\n\n@DeveloperAPI\n@output_mixin_registry.register(TIMESERIES)\n@ludwig_dataclass\nclass TimeseriesOutputFeatureConfigMixin(BaseMarshmallowConfig):\n \"\"\"TimeseriesOutputFeatureConfigMixin configures the parameters used in both the timeseries output feature and\n the timeseries global defaults section of the Ludwig Config.\"\"\"\n\n decoder: BaseDecoderConfig = DecoderDataclassField(\n MODEL_ECD,\n feature_type=TIMESERIES,\n default=\"projector\",\n )\n\n loss: BaseLossConfig = LossDataclassField(\n feature_type=TIMESERIES,\n default=HUBER,\n )\n\n\n@DeveloperAPI\n@ecd_output_config_registry.register(TIMESERIES)\n@ludwig_dataclass\nclass TimeseriesOutputFeatureConfig(BaseOutputFeatureConfig, TimeseriesOutputFeatureConfigMixin):\n \"\"\"TimeseriesOutputFeatureConfig configures the parameters used for a timeseries output feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(TIMESERIES)\n\n dependencies: list = schema_utils.List(\n default=[],\n description=\"List of input features that this feature depends on.\",\n parameter_metadata=FEATURE_METADATA[VECTOR][\"dependencies\"],\n )\n\n default_validation_metric: str = schema_utils.StringOptions(\n [MEAN_SQUARED_ERROR],\n default=MEAN_SQUARED_ERROR,\n description=\"Internal parameter.\",\n parameter_metadata=INTERNAL_ONLY,\n )\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=\"timeseries_output\")\n\n reduce_dependencies: str = schema_utils.ReductionOptions(\n default=None,\n description=\"How to reduce the dependencies of the output feature.\",\n parameter_metadata=FEATURE_METADATA[VECTOR][\"reduce_dependencies\"],\n )\n\n reduce_input: str = schema_utils.ReductionOptions(\n default=None,\n description=\"How to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first \"\n \"dimension (second if you count the batch dimension)\",\n parameter_metadata=FEATURE_METADATA[VECTOR][\"reduce_input\"],\n )\n\n horizon: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Internal parameter. Obtained from preprocessing\",\n parameter_metadata=INTERNAL_ONLY,\n )\n\n\n@DeveloperAPI\n@ecd_defaults_config_registry.register(TIMESERIES)\n@ludwig_dataclass\nclass TimeseriesDefaultsConfig(TimeseriesInputFeatureConfigMixin, TimeseriesOutputFeatureConfigMixin):\n pass\n"
},
{
"path": "ludwig/schema/features/utils.py",
"content": "from collections import defaultdict\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import MODEL_ECD, MODEL_LLM\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.utils.registry import Registry\n\ninput_config_registries = defaultdict(Registry)\noutput_config_registries = defaultdict(Registry)\n\necd_input_config_registry = input_config_registries[MODEL_ECD]\nllm_input_config_registry = input_config_registries[MODEL_LLM]\n\necd_output_config_registry = output_config_registries[MODEL_ECD]\nllm_output_config_registry = output_config_registries[MODEL_LLM]\n\ninput_mixin_registry = Registry()\noutput_mixin_registry = Registry()\n\"\"\"ECD models support the full range of feature parameters available in Ludwig, so any feature schema can be\nregistered into it.\n\nSee `BinaryDefaultsConfig` for an example.\n\"\"\"\necd_defaults_config_registry = Registry()\n\nllm_defaults_config_registry = Registry()\n\n\ndef input_config_registry(model_type: str) -> Registry:\n return input_config_registries[model_type]\n\n\ndef output_config_registry(model_type: str) -> Registry:\n return output_config_registries[model_type]\n\n\n@DeveloperAPI\ndef get_input_feature_cls(model_type: str, name: str):\n # TODO(travis): not needed once we remove existing model config implementation\n return input_config_registries[model_type][name]\n\n\n@DeveloperAPI\ndef get_output_feature_cls(model_type: str, name: str):\n # TODO(ksbrar): What is this?\n return output_config_registries[model_type][name]\n\n\n@DeveloperAPI\ndef get_input_feature_jsonschema(model_type: str):\n \"\"\"This function returns a JSON schema structured to only requires a `type` key and then conditionally applies\n a corresponding input feature's field constraints.\n\n Returns: JSON Schema\n \"\"\"\n input_feature_types = sorted(list(input_config_registry(model_type).keys()))\n schema = {\n \"type\": \"object\",\n \"properties\": {\n \"name\": {\"type\": \"string\", \"title\": \"name\", \"description\": \"Name of the input feature.\"},\n \"type\": {\n \"type\": \"string\",\n \"enum\": input_feature_types,\n \"title\": \"type\",\n \"description\": \"Type of the input feature\",\n },\n \"column\": {\"type\": \"string\", \"title\": \"column\", \"description\": \"Name of the column.\"},\n },\n \"additionalProperties\": True,\n \"allOf\": get_input_feature_conds(model_type),\n \"required\": [\"name\", \"type\"],\n \"title\": \"input_feature\",\n }\n\n return schema\n\n\n@DeveloperAPI\ndef get_input_feature_conds(model_type: str):\n \"\"\"This function returns a list of if-then JSON clauses for each input feature type along with their properties\n and constraints.\n\n Returns: List of JSON clauses\n \"\"\"\n input_feature_types = sorted(list(input_config_registry(model_type).keys()))\n conds = []\n for feature_type in input_feature_types:\n schema_cls = get_input_feature_cls(model_type, feature_type)\n feature_schema = schema_utils.unload_jsonschema_from_marshmallow_class(schema_cls)\n feature_props = feature_schema[\"properties\"]\n schema_utils.remove_duplicate_fields(feature_props)\n\n feature_cond = schema_utils.create_cond({\"type\": feature_type}, feature_props)\n conds.append(feature_cond)\n return conds\n\n\n@DeveloperAPI\ndef get_output_feature_jsonschema(model_type: str):\n \"\"\"This function returns a JSON schema structured to only requires a `type` key and then conditionally applies\n a corresponding output feature's field constraints.\n\n Returns: JSON Schema\n \"\"\"\n output_feature_types = sorted(list(output_config_registry(model_type).keys()))\n schema = {\n \"type\": \"object\",\n \"properties\": {\n \"name\": {\"type\": \"string\", \"title\": \"name\", \"description\": \"Name of the output feature.\"},\n \"type\": {\n \"type\": \"string\",\n \"enum\": output_feature_types,\n \"title\": \"type\",\n \"description\": \"Type of the output feature\",\n },\n \"column\": {\"type\": \"string\", \"title\": \"column\", \"description\": \"Name of the column.\"},\n },\n \"additionalProperties\": True,\n \"allOf\": get_output_feature_conds(model_type),\n \"required\": [\"name\", \"type\"],\n \"title\": \"output_feature\",\n }\n\n return schema\n\n\n@DeveloperAPI\ndef get_output_feature_conds(model_type: str):\n \"\"\"This function returns a list of if-then JSON clauses for each output feature type along with their\n properties and constraints.\n\n Returns: List of JSON clauses\n \"\"\"\n output_feature_types = sorted(list(output_config_registry(model_type).keys()))\n conds = []\n for feature_type in output_feature_types:\n schema_cls = get_output_feature_cls(model_type, feature_type)\n feature_schema = schema_utils.unload_jsonschema_from_marshmallow_class(schema_cls)\n feature_props = feature_schema[\"properties\"]\n schema_utils.remove_duplicate_fields(feature_props)\n feature_cond = schema_utils.create_cond({\"type\": feature_type}, feature_props)\n conds.append(feature_cond)\n return conds\n"
},
{
"path": "ludwig/schema/features/vector_feature.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import MEAN_SQUARED_ERROR, MODEL_ECD, VECTOR\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.decoders.base import BaseDecoderConfig\nfrom ludwig.schema.decoders.utils import DecoderDataclassField\nfrom ludwig.schema.encoders.base import BaseEncoderConfig\nfrom ludwig.schema.encoders.utils import EncoderDataclassField\nfrom ludwig.schema.features.base import BaseInputFeatureConfig, BaseOutputFeatureConfig\nfrom ludwig.schema.features.loss.loss import BaseLossConfig\nfrom ludwig.schema.features.loss.utils import LossDataclassField\nfrom ludwig.schema.features.preprocessing.base import BasePreprocessingConfig\nfrom ludwig.schema.features.preprocessing.utils import PreprocessingDataclassField\nfrom ludwig.schema.features.utils import (\n ecd_defaults_config_registry,\n ecd_input_config_registry,\n ecd_output_config_registry,\n input_mixin_registry,\n output_mixin_registry,\n)\nfrom ludwig.schema.metadata import FEATURE_METADATA\nfrom ludwig.schema.metadata.parameter_metadata import INTERNAL_ONLY\nfrom ludwig.schema.utils import BaseMarshmallowConfig, ludwig_dataclass\n\n\n@DeveloperAPI\n@input_mixin_registry.register(VECTOR)\n@ludwig_dataclass\nclass VectorInputFeatureConfigMixin(BaseMarshmallowConfig):\n \"\"\"VectorInputFeatureConfigMixin is a dataclass that configures the parameters used in both the vector input\n feature and the vector global defaults section of the Ludwig Config.\"\"\"\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=VECTOR)\n\n encoder: BaseEncoderConfig = EncoderDataclassField(\n MODEL_ECD,\n feature_type=VECTOR,\n default=\"dense\",\n )\n\n\n@DeveloperAPI\n@ecd_input_config_registry.register(VECTOR)\n@ludwig_dataclass\nclass VectorInputFeatureConfig(VectorInputFeatureConfigMixin, BaseInputFeatureConfig):\n \"\"\"VectorInputFeatureConfig is a dataclass that configures the parameters used for a vector input feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(VECTOR)\n\n\n@DeveloperAPI\n@output_mixin_registry.register(VECTOR)\n@ludwig_dataclass\nclass VectorOutputFeatureConfigMixin(BaseMarshmallowConfig):\n \"\"\"VectorOutputFeatureConfigMixin is a dataclass that configures the parameters used in both the vector output\n feature and the vector global defaults section of the Ludwig Config.\"\"\"\n\n decoder: BaseDecoderConfig = DecoderDataclassField(\n MODEL_ECD,\n feature_type=VECTOR,\n default=\"projector\",\n )\n\n loss: BaseLossConfig = LossDataclassField(\n feature_type=VECTOR,\n default=MEAN_SQUARED_ERROR,\n )\n\n\n@DeveloperAPI\n@ecd_output_config_registry.register(VECTOR)\n@ludwig_dataclass\nclass VectorOutputFeatureConfig(VectorOutputFeatureConfigMixin, BaseOutputFeatureConfig):\n \"\"\"VectorOutputFeatureConfig is a dataclass that configures the parameters used for a vector output feature.\"\"\"\n\n type: str = schema_utils.ProtectedString(VECTOR)\n\n dependencies: list = schema_utils.List(\n default=[],\n description=\"List of input features that this feature depends on.\",\n parameter_metadata=FEATURE_METADATA[VECTOR][\"dependencies\"],\n )\n\n default_validation_metric: str = schema_utils.StringOptions(\n [MEAN_SQUARED_ERROR],\n default=MEAN_SQUARED_ERROR,\n description=\"Internal only use parameter: default validation metric for binary output feature.\",\n parameter_metadata=INTERNAL_ONLY,\n )\n\n preprocessing: BasePreprocessingConfig = PreprocessingDataclassField(feature_type=\"vector_output\")\n\n reduce_dependencies: str = schema_utils.ReductionOptions(\n default=None,\n description=\"How to reduce the dependencies of the output feature.\",\n parameter_metadata=FEATURE_METADATA[VECTOR][\"reduce_dependencies\"],\n )\n\n reduce_input: str = schema_utils.ReductionOptions(\n default=None,\n description=\"How to reduce an input that is not a vector, but a matrix or a higher order tensor, on the first \"\n \"dimension (second if you count the batch dimension)\",\n parameter_metadata=FEATURE_METADATA[VECTOR][\"reduce_input\"],\n )\n\n softmax: bool = schema_utils.Boolean(\n default=False,\n description=\"Determines whether to apply a softmax at the end of the decoder. This is useful for predicting a \"\n \"vector of values that sum up to 1 and can be interpreted as probabilities.\",\n parameter_metadata=FEATURE_METADATA[VECTOR][\"softmax\"],\n )\n\n vector_size: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"The size of the vector. If None, the vector size will be inferred from the data.\",\n parameter_metadata=FEATURE_METADATA[VECTOR][\"vector_size\"],\n )\n\n\n@DeveloperAPI\n@ecd_defaults_config_registry.register(VECTOR)\n@ludwig_dataclass\nclass VectorDefaultsConfig(VectorInputFeatureConfigMixin, VectorOutputFeatureConfigMixin):\n pass\n"
},
{
"path": "ludwig/schema/hyperopt/__init__.py",
"content": "from abc import ABC\n\nimport ludwig.schema.hyperopt.parameter # noqa: F401\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import LOSS, TEST, TRAIN, VALIDATION\nfrom ludwig.modules import metric_modules # noqa: Needed to ensure that the metric registry is populated.\nfrom ludwig.modules.metric_registry import get_metric_registry\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.hyperopt.executor import ExecutorConfig, ExecutorDataclassField\nfrom ludwig.schema.hyperopt.search_algorithm import BaseSearchAlgorithmConfig, SearchAlgorithmDataclassField\nfrom ludwig.schema.utils import ludwig_dataclass as dataclass\n\n\n@DeveloperAPI\n@dataclass\nclass HyperoptConfig(schema_utils.BaseMarshmallowConfig, ABC):\n \"\"\"Basic hyperopt settings.\"\"\"\n\n output_feature: str = schema_utils.String( # TODO: make more restrictive\n default=\"combined\",\n description=(\n \"The name of the output feature that we want to optimize the metric or loss of. Available values \"\n \"are `combined` or the name of any output feature provided in the configuration. `combined` is a special \"\n \"output feature that allows to optimize for the aggregated loss and metrics of all output features.\"\n ),\n )\n\n goal: str = schema_utils.StringOptions(\n options=[\"minimize\", \"maximize\"],\n default=\"minimize\",\n allow_none=False,\n description=(\n \"Indicates if to minimize or maximize a metric or a loss of any of the output features on any of the \"\n \"dataset splits. Available values are: minimize (default) or maximize.\"\n ),\n )\n\n metric: str = schema_utils.StringOptions(\n options=get_metric_registry().keys(),\n default=LOSS,\n allow_none=False,\n description=(\n \"The metric that we want to optimize for. The default one is loss, but depending on the type of the \"\n \"feature defined in output_feature, different metrics and losses are available. Check the metrics section \"\n \"of the specific output feature type to figure out what metrics are available to use.\"\n ),\n )\n\n split: str = schema_utils.StringOptions(\n options=[TRAIN, VALIDATION, TEST],\n default=VALIDATION,\n allow_none=False,\n description=(\n \"The split of data that we want to compute our metric on. By default it is the validation split, but \"\n \"you have the flexibility to specify also train or test splits.\"\n ),\n )\n\n eval_split: str = schema_utils.StringOptions(\n options=[TRAIN, VALIDATION, TEST],\n default=VALIDATION,\n allow_none=False,\n description=(\n \"The split of data that we want to run evaluation on. By default it is the validation split, but \"\n \"you have the flexibility to specify also train or test splits.\"\n ),\n )\n\n search_alg: BaseSearchAlgorithmConfig = SearchAlgorithmDataclassField(\n description=(\n \"Specifies the algorithm to sample the defined parameters space. Candidate algorithms are those \"\n \"found in [Ray Tune's Search Algorithms](https://docs.ray.io/en/latest/tune/api/suggestion.html).\"\n )\n )\n\n executor: ExecutorConfig = ExecutorDataclassField(\n description=(\n \"specifies how to execute the hyperparameter optimization. The execution could happen locally in a serial \"\n \"manner or in parallel across multiple workers and with GPUs as well if available. The executor section \"\n \"includes specification for work scheduling and the number of samples to generate.\"\n )\n )\n\n parameters: dict = schema_utils.Dict(\n allow_none=False,\n description=(\n \"This section consists of a set of hyperparameters to optimize. They are provided as keys (the names of \"\n \"the parameters) and values associated with them (that define the search space). The values vary depending \"\n \"on the type of the hyperparameter. Syntax for this section is based on [Ray Tune's Search Space \"\n \"parameters](https://docs.ray.io/en/latest/tune/api/search_space.html).\"\n ),\n )\n\n\n@DeveloperAPI\ndef get_hyperopt_jsonschema():\n props = schema_utils.unload_jsonschema_from_marshmallow_class(HyperoptConfig)[\"properties\"]\n\n return {\n \"type\": [\"object\", \"null\"],\n \"properties\": props,\n \"title\": \"hyperopt_options\",\n \"description\": \"Settings for hyperopt\",\n }\n\n\n@DeveloperAPI\nclass HyperoptField(schema_utils.DictMarshmallowField):\n def __init__(self):\n super().__init__(HyperoptConfig, default_missing=True)\n\n def _jsonschema_type_mapping(self):\n return get_hyperopt_jsonschema()\n"
},
{
"path": "ludwig/schema/hyperopt/executor.py",
"content": "from dataclasses import field\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import RAY\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.hyperopt.scheduler import BaseSchedulerConfig, SchedulerDataclassField\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass ExecutorConfig(schema_utils.BaseMarshmallowConfig):\n \"\"\"Basic executor settings.\"\"\"\n\n type: str = schema_utils.ProtectedString(RAY)\n\n num_samples: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=(\n \"This parameter, along with the `space` specifications in the `parameters` section, controls how many \"\n \"trials are generated.\"\n ),\n )\n\n time_budget_s: int = schema_utils.PositiveInteger(\n default=3600, allow_none=True, description=\"The number of seconds for the entire hyperopt run.\"\n )\n\n trial_driver_resources: dict[str, float] = schema_utils.Dict(\n default=None,\n description=(\n \"The resources reserved by each trial driver. This differs from cpu_resources_per_trial and \"\n \"gpu_resources_per_trial because these resources are reserved for the driver, not its subsequent \"\n \"workers. Only used when the trials themselves are on the Ray backend. Defaults to 1 CPU.\"\n ),\n )\n\n cpu_resources_per_trial: int = schema_utils.PositiveInteger(\n default=1, description=\"The number of CPU cores allocated to each trial\"\n )\n\n gpu_resources_per_trial: int = schema_utils.NonNegativeInteger(\n default=0, description=\"The number of GPU devices allocated to each trial\"\n )\n\n kubernetes_namespace: str | None = schema_utils.String(\n default=None,\n allow_none=True,\n description=(\n \"When running on Kubernetes, provide the namespace of the Ray cluster to sync results between \"\n \"pods. See the Ray docs for more info.\"\n ),\n )\n\n max_concurrent_trials: str | int | None = schema_utils.OneOfOptionsField(\n default=\"auto\",\n allow_none=True,\n description=(\"The maximum number of trials to train concurrently. Defaults to auto if not specified.\"),\n field_options=[\n schema_utils.PositiveInteger(\n default=1, allow_none=False, description=\"Manually set a number of concurrent trials.\"\n ),\n schema_utils.StringOptions(\n options=[\"auto\"],\n default=\"auto\",\n allow_none=False,\n description=\"Automatically set number of concurrent trials.\",\n ),\n ],\n )\n\n scheduler: BaseSchedulerConfig = SchedulerDataclassField(description=\"\")\n\n\n@DeveloperAPI\ndef ExecutorDataclassField(description: str, default: dict = {}):\n class ExecutorMarshmallowField(schema_utils.LudwigSchemaField):\n def _deserialize(self, value, attr, data, **kwargs):\n if isinstance(value, dict):\n try:\n return ExecutorConfig.Schema().load(value)\n except (TypeError, ConfigValidationError):\n raise ConfigValidationError(f\"Invalid params for executor: {value}, see ExecutorConfig class.\")\n raise ConfigValidationError(\"Field should be dict\")\n\n def _jsonschema_type_mapping(self):\n return {\n **schema_utils.unload_jsonschema_from_marshmallow_class(ExecutorConfig),\n \"title\": \"executor\",\n \"description\": description,\n }\n\n if not isinstance(default, dict):\n raise ConfigValidationError(f\"Invalid default: `{default}`\")\n\n load_default = lambda: ExecutorConfig.Schema().load(default)\n dump_default = ExecutorConfig.Schema().dump(default)\n\n return field(\n metadata={\n \"marshmallow_field\": ExecutorMarshmallowField(\n allow_none=False,\n load_default=load_default,\n dump_default=dump_default,\n metadata={\"description\": description, \"parameter_metadata\": None},\n )\n },\n default_factory=load_default,\n )\n"
},
{
"path": "ludwig/schema/hyperopt/parameter.py",
"content": "from pydantic.fields import FieldInfo\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.hyperopt.utils import register_parameter_config\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\ndef quantization_number_field(dtype: type[float] | type[int] = float, default=None) -> FieldInfo:\n description = (\n \"Quantization number. Output values will be rounded to the nearest increment of `q` in range.\"\n \"Quantization makes the upper bound inclusive.\"\n )\n if dtype is int:\n field = schema_utils.Integer(default=default, allow_none=True, description=description)\n else:\n field = schema_utils.FloatRange(default=default, allow_none=True, description=description)\n\n return field\n\n\ndef log_base_field(default: float = 10) -> FieldInfo:\n return schema_utils.FloatRange(default=default, description=\"Logarithmic base.\")\n\n\n@DeveloperAPI\n@register_parameter_config(\"choice\")\n@ludwig_dataclass\nclass ChoiceParameterConfig(schema_utils.BaseMarshmallowConfig):\n \"\"\"Config for a randomly sampled categorical search space.\"\"\"\n\n space: str = schema_utils.ProtectedString(\"choice\")\n\n categories: list = schema_utils.OneOfOptionsField(\n default=None,\n allow_none=True,\n description=(\n \"The list of values to use in creating the categorical space. The type of each value of the list is \"\n \"general, i.e., they could be strings, integers, floats and anything else, even entire dictionaries.\"\n ),\n field_options=[\n schema_utils.List(list_type=float, allow_none=False, description=\"The list of floats to randomly sample.\"),\n schema_utils.List(list_type=int, allow_none=False, description=\"The list of integers to randomly sample.\"),\n schema_utils.List(list_type=str, allow_none=False, description=\"The list of strings to randomly sample.\"),\n schema_utils.List(\n list_type=list,\n inner_type=dict,\n allow_none=False,\n description=\"The list of lists of configs to randomly sample.\",\n ),\n schema_utils.DictList(allow_none=False, description=\"A list of nested config parameters to sample.\"),\n ],\n )\n\n\n@DeveloperAPI\n@register_parameter_config(\"grid_search\")\n@ludwig_dataclass\nclass GridSearchParameterConfig(schema_utils.BaseMarshmallowConfig):\n \"\"\"Config for a grid search space.\"\"\"\n\n space: str = schema_utils.ProtectedString(\"grid_search\")\n\n values: list = schema_utils.OneOfOptionsField(\n default=None,\n allow_none=True,\n description=(\n \"The list of values to use in creating the grid search space. The type of each value of the list is \"\n \"general, i.e., they could be strings, integers, floats and anything else, even entire dictionaries.\"\n ),\n field_options=[\n schema_utils.List(list_type=float, allow_none=False, description=\"The list of floats to randomly sample.\"),\n schema_utils.List(list_type=int, allow_none=False, description=\"The list of integers to randomly sample.\"),\n schema_utils.List(list_type=str, allow_none=False, description=\"The list of strings to randomly sample.\"),\n ],\n )\n\n\n@DeveloperAPI\n@register_parameter_config(\"uniform\")\n@ludwig_dataclass\nclass UniformParameterConfig(schema_utils.BaseMarshmallowConfig):\n \"\"\"Config for a real-valued uniform search space.\"\"\"\n\n space: str = schema_utils.ProtectedString(\"uniform\")\n\n lower: float = schema_utils.FloatRange(default=None, description=\"The minimum value the parameter can have.\")\n\n upper: float = schema_utils.FloatRange(default=None, description=\"The maximum value the parameter can have.\")\n\n\n@DeveloperAPI\n@register_parameter_config(\"quniform\")\n@ludwig_dataclass\nclass QUniformParameterConfig(UniformParameterConfig):\n \"\"\"Config for a real-valued uniform search space with quantization.\"\"\"\n\n space: str = schema_utils.ProtectedString(\"quniform\")\n\n q: float = quantization_number_field()\n\n\n@DeveloperAPI\n@register_parameter_config(\"loguniform\")\n@ludwig_dataclass\nclass LogUniformParameterConfig(UniformParameterConfig):\n \"\"\"Config for a log-scaled real-valued uniform numeric search space.\"\"\"\n\n space: str = schema_utils.ProtectedString(\"loguniform\")\n\n base: float = log_base_field()\n\n\n@DeveloperAPI\n@register_parameter_config(\"qloguniform\")\n@ludwig_dataclass\nclass QLogUniformParameterConfig(UniformParameterConfig):\n \"\"\"Config for a log-scaled real-valued uniform search space with quantization.\"\"\"\n\n space: str = schema_utils.ProtectedString(\"qloguniform\")\n\n q: float = quantization_number_field()\n\n base: float = log_base_field()\n\n\n@DeveloperAPI\n@register_parameter_config(\"randn\")\n@ludwig_dataclass\nclass RandnParameterConfig(schema_utils.BaseMarshmallowConfig):\n \"\"\"Config for a Gaussian search space.\"\"\"\n\n space: str = schema_utils.ProtectedString(\"randn\")\n\n mean: float = schema_utils.FloatRange(default=0.0, description=\"Mean of the normal distribution.\")\n\n sd: float = schema_utils.FloatRange(default=1.0, description=\"Standard deviation of the normal distribution.\")\n\n\n@DeveloperAPI\n@register_parameter_config(\"qrandn\")\n@ludwig_dataclass\nclass QRandnParameterConfig(RandnParameterConfig):\n \"\"\"Config for a Gaussian search space with quantization.\"\"\"\n\n space: str = schema_utils.ProtectedString(\"qrandn\")\n\n q: float = quantization_number_field()\n\n\n@DeveloperAPI\n@register_parameter_config(\"randint\")\n@ludwig_dataclass\nclass RandintParameterConfig(schema_utils.BaseMarshmallowConfig):\n \"\"\"Config for an integer-valued uniform search space.\"\"\"\n\n space: str = schema_utils.ProtectedString(\"randint\")\n\n lower: int = schema_utils.Integer(default=None, description=\"The minimum value the parameter can have.\")\n\n upper: int = schema_utils.Integer(default=None, description=\"The maximum value the parameter can have.\")\n\n\n@DeveloperAPI\n@register_parameter_config(\"qrandint\")\n@ludwig_dataclass\nclass QRandintParameterConfig(RandintParameterConfig):\n \"\"\"Config for an integer-valued uniform search space with quantization.\"\"\"\n\n space: str = schema_utils.ProtectedString(\"qrandint\")\n\n q: int = quantization_number_field(dtype=int)\n\n\n@DeveloperAPI\n@register_parameter_config(\"lograndint\")\n@ludwig_dataclass\nclass LogRandintParameterConfig(RandintParameterConfig):\n \"\"\"Config for an log-scaled integer-valued search space.\"\"\"\n\n space: str = schema_utils.ProtectedString(\"lograndint\")\n\n base: float = log_base_field()\n\n\n@DeveloperAPI\n@register_parameter_config(\"qlograndint\")\n@ludwig_dataclass\nclass QLogRandintParameterConfig(RandintParameterConfig):\n \"\"\"Config for an log-scaled integer-valued search space with quantization.\"\"\"\n\n space: str = schema_utils.ProtectedString(\"qlograndint\")\n\n q: int = quantization_number_field(dtype=int)\n\n base: float = log_base_field()\n"
},
{
"path": "ludwig/schema/hyperopt/scheduler.py",
"content": "from abc import ABC\nfrom collections.abc import Callable\nfrom dataclasses import field\nfrom importlib import import_module\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.hyperopt import utils as hyperopt_utils\nfrom ludwig.schema.utils import ludwig_dataclass\n\n# ----------------------------------------------------------------------------------------------------------------------\n# To prevent direct dependency on ray import, the following static key stores are duplicated:\n\n# from ray.tune.schedulers import SCHEDULER_IMPORT\n# https://github.com/ray-project/ray/blob/137a1b12c3b31a3622fa5f721a05a64e9b559b05/python/ray/tune/schedulers/__init__.py#L28\n\n# from ray.tune.result import DEFAULT_RESULT_KEYS\n# Taken from https://github.com/ray-project/ray/blob/137a1b12c3b31a3622fa5f721a05a64e9b559b05/python/ray/tune/result.py\nTRAINING_ITERATION = \"training_iteration\"\nTIME_TOTAL_S = \"time_total_s\"\nTIMESTEPS_TOTAL = \"timesteps_total\"\nMEAN_ACCURACY = \"mean_accuracy\"\nMEAN_LOSS = \"mean_loss\"\nDEFAULT_RESULT_KEYS = (TRAINING_ITERATION, TIME_TOTAL_S, TIMESTEPS_TOTAL, MEAN_ACCURACY, MEAN_LOSS)\n\n# from ray.tune.result import DEFAULT_METRIC\nRAY_TUNE_DESULT_DEFAULT_METRIC = \"_metric\"\n# ----------------------------------------------------------------------------------------------------------------------\n\n\n# Field aliases to cut down on code reuse:\n@DeveloperAPI\ndef metric_alias(default=None):\n return schema_utils.StringOptions(\n options=list(DEFAULT_RESULT_KEYS) + [RAY_TUNE_DESULT_DEFAULT_METRIC],\n default=default,\n allow_none=default is None,\n description=(\n \"The training result objective value attribute. Stopping procedures will use this attribute. If None but a \"\n \"mode was passed, the ray.tune.result.DEFAULT_METRIC will be used per default.\"\n ),\n )\n\n\n@DeveloperAPI\ndef time_attr_alias(default=TRAINING_ITERATION):\n return schema_utils.StringOptions(\n options=list(DEFAULT_RESULT_KEYS),\n default=default,\n allow_none=False,\n description=(\n \"A training result attr to use for comparing time. Note that you can pass in something non-temporal such as\"\n \" training_iteration as a measure of progress, the only requirement is that the attribute should increase \"\n \"monotonically.\"\n ),\n )\n\n\n@DeveloperAPI\ndef max_t_alias(default=100):\n return schema_utils.PositiveInteger(\n default=default,\n description=(\n \"max time units per trial. Trials will be stopped after max_t time units (determined by time_attr) have \"\n \"passed.\"\n ),\n )\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass BaseSchedulerConfig(schema_utils.BaseMarshmallowConfig, ABC):\n \"\"\"Base class for schedulers.\n\n Not meant to be used directly.\n \"\"\"\n\n type: str\n \"\"\"Name corresponding to a scheduler in `ludwig.schema.hyperopt.scheduler.scheduler_registry`.\n\n Technically mutable, but attempting to load a derived scheduler with `type` set to a mismatched value will result in\n a `ValidationError`.\n \"\"\"\n\n time_attr: str = time_attr_alias()\n\n metric: str | None = metric_alias()\n\n mode: str | None = schema_utils.StringOptions(\n options=[\"min\", \"max\"],\n default=None,\n allow_none=True,\n description=(\n \"One of {min, max}. Determines whether objective is minimizing or maximizing the metric attribute.\"\n ),\n )\n\n def dependencies_installed(self):\n \"\"\"Some search algorithms require additional packages to be installed, check that they are available.\"\"\"\n missing_packages = []\n missing_installs = []\n for package_name, install_name in hyperopt_utils.get_scheduler_dependencies(self.type):\n try:\n import_module(package_name)\n except ImportError:\n missing_packages.append(package_name)\n missing_installs.append(install_name)\n\n if missing_packages:\n missing_packages = \", \".join(missing_packages)\n missing_installs = \" \".join(missing_installs)\n raise ImportError(\n f\"Some packages needed to use hyperopt scheduler {self.type} are not installed: \"\n f\"{missing_packages}. To add these dependencies, run `pip install {missing_installs}`. For more \"\n \"details, please refer to Ray Tune documentation for this scheduler.\"\n )\n return True\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass BaseHyperbandSchedulerConfig(BaseSchedulerConfig):\n max_t: int = max_t_alias()\n\n\n@DeveloperAPI\n@hyperopt_utils.register_scheduler_config(\"async_hyperband\")\n@hyperopt_utils.register_scheduler_config(\"asynchyperband\")\n@hyperopt_utils.register_scheduler_config(\"asha\")\n@ludwig_dataclass\nclass AsyncHyperbandSchedulerConfig(BaseHyperbandSchedulerConfig):\n \"\"\"Asynchronous hyperband (ASHA) scheduler settings.\"\"\"\n\n type: str = schema_utils.ProtectedString(\"async_hyperband\")\n\n max_t: int = max_t_alias()\n\n grace_period: int = schema_utils.PositiveInteger(\n default=1,\n description=(\n \"Only stop trials at least this old in time. The units are the same as the attribute named by `time_attr`.\"\n ),\n )\n\n reduction_factor: int = schema_utils.NonNegativeFloat(\n default=4, description=(\"Used to set halving rate and amount. This is simply a unit-less scalar.\")\n )\n\n brackets: int = schema_utils.PositiveInteger(\n default=1,\n description=(\n \"Number of brackets. Each bracket has a different halving rate, specified by the reduction factor.\"\n ),\n )\n\n stop_last_trials: bool = schema_utils.Boolean(\n default=True, description=\"Whether to terminate the trials after reaching `max_t`.\"\n )\n\n\n@DeveloperAPI\n@hyperopt_utils.register_scheduler_config(\"hyperband\")\n@ludwig_dataclass\nclass HyperbandSchedulerConfig(BaseHyperbandSchedulerConfig):\n \"\"\"Standard hyperband scheduler settings.\"\"\"\n\n type: str = schema_utils.ProtectedString(\"hyperband\")\n\n max_t: int = max_t_alias(default=81)\n\n reduction_factor: int = schema_utils.NonNegativeFloat(\n default=3, description=(\"Used to set halving rate and amount. This is simply a unit-less scalar.\")\n )\n\n stop_last_trials: bool = schema_utils.Boolean(\n default=True, description=(\"Whether to terminate the trials after reaching max_t. Defaults to True.\")\n )\n\n\n@DeveloperAPI\n@hyperopt_utils.register_scheduler_config(\"median_stopping_rule\")\n@hyperopt_utils.register_scheduler_config(\"medianstoppingrule\")\n@ludwig_dataclass\nclass MedianStoppingRuleSchedulerConfig(BaseSchedulerConfig):\n \"\"\"Median Stopping Rule scheduler settings.\"\"\"\n\n type: str = schema_utils.ProtectedString(\"median_stopping_rule\")\n\n time_attr: str = time_attr_alias(TIME_TOTAL_S)\n\n grace_period: float = schema_utils.NonNegativeFloat(\n default=60.0,\n description=(\n \"Only stop trials at least this old in time. The mean will only be computed from this time onwards. The \"\n \"units are the same as the attribute named by `time_attr`.\"\n ),\n )\n\n min_samples_required: int = schema_utils.PositiveInteger(\n default=3, description=(\"Minimum number of trials to compute median over.\")\n )\n\n min_time_slice: int = schema_utils.NonNegativeInteger(\n default=0,\n description=(\n \"Each trial runs at least this long before yielding (assuming it isn't stopped). Note: trials ONLY yield \"\n \"if there are not enough samples to evaluate performance for the current result AND there are other \"\n \"trials waiting to run. The units are the same as the attribute named by `time_attr`.\"\n ),\n )\n\n hard_stop: bool = schema_utils.Boolean(\n default=True,\n description=(\n \"If False, pauses trials instead of stopping them. When all other trials are complete, paused trials will \"\n \"be resumed and allowed to run FIFO.\"\n ),\n )\n\n\n@DeveloperAPI\n@hyperopt_utils.register_scheduler_config(\"pbt\")\n@ludwig_dataclass\nclass PopulationBasedTrainingSchedulerConfig(BaseSchedulerConfig):\n \"\"\"Population Based Training scheduler settings.\"\"\"\n\n type: str = schema_utils.ProtectedString(\"pbt\")\n\n time_attr: str = time_attr_alias(TIME_TOTAL_S)\n\n perturbation_interval: float = schema_utils.NonNegativeFloat(\n default=60.0,\n description=(\n \"Models will be considered for perturbation at this interval of `time_attr`. Note that perturbation incurs \"\n \"checkpoint overhead, so you shouldn't set this to be too frequent.\"\n ),\n )\n\n burn_in_period: float = schema_utils.NonNegativeFloat(\n default=60.0,\n description=(\n \"Models will not be considered for perturbation before this interval of time_attr has passed. This \"\n \"guarantees that models are trained for at least a certain amount of time or timesteps before being \"\n \"perturbed.\"\n ),\n )\n\n hyperparam_mutations: dict | None = schema_utils.Dict(\n default=None,\n description=(\n \"Hyperparams to mutate. The format is as follows: for each key, either a list, function, or a tune search \"\n \"space object (`tune.loguniform`, tune.uniform, etc.) can be provided. A list specifies an allowed set of \"\n \"categorical values. A function or tune search space object specifies the distribution of a continuous \"\n \"parameter. You must use `tune.choice`, `tune.uniform`, `tune.loguniform`, etc.. Arbitrary \"\n \"`tune.sample_from` objects are not supported. A key can also hold a dict for nested hyperparameters. You \"\n \"must specify at least one of `hyperparam_mutations` or `custom_explore_fn`. Tune will sample the search \"\n \"space provided by `hyperparam_mutations` for the initial hyperparameter values if the corresponding \"\n \"hyperparameters are not present in a trial's initial config.\"\n ),\n )\n\n quantile_fraction: float = schema_utils.FloatRange(\n default=0.25,\n allow_none=False,\n min=0,\n max=0.5,\n description=(\n \"Parameters are transferred from the top `quantile_fraction` fraction of trials to the bottom \"\n \"`quantile_fraction` fraction. Needs to be between 0 and 0.5. Setting it to 0 essentially implies doing \"\n \"no exploitation at all.\"\n ),\n )\n\n resample_probability: float = schema_utils.NonNegativeFloat(\n default=0.25,\n description=(\n \"The probability of resampling from the original distribution when applying `hyperparam_mutations`. If \"\n \"not resampled, the value will be perturbed by a factor chosen from `perturbation_factors` if continuous, \"\n \"or changed to an adjacent value if discrete.\"\n ),\n )\n\n perturbation_factors: tuple[float, float] = schema_utils.FloatRangeTupleDataclassField(\n default=(1.2, 0.8),\n allow_none=False,\n max=None,\n description=(\"Scaling factors to choose between when mutating a continuous hyperparameter.\"),\n )\n\n # TODO: Add schema support for Callable\n custom_explore_fn: str | Callable = schema_utils.String(\n default=None,\n allow_none=True,\n description=(\n \"You can also specify a custom exploration function. This function is invoked as `f(config)` after \"\n \"built-in perturbations from `hyperparam_mutations` are applied, and should return config updated as \"\n \"needed. You must specify at least one of `hyperparam_mutations` or `custom_explore_fn`.\"\n ),\n )\n\n log_config: bool = schema_utils.Boolean(\n default=True,\n description=(\n \"Whether to log the ray config of each model to `local_dir` at each exploit. Allows config schedule to be \"\n \"reconstructed.\"\n ),\n )\n\n require_attrs: bool = schema_utils.Boolean(\n default=True,\n description=(\n \"Whether to require `time_attr` and metric to appear in result for every iteration. If True, error will \"\n \"be raised if these values are not present in trial result.\"\n ),\n )\n\n synch: bool = schema_utils.Boolean(\n default=False,\n description=(\n \"If False, will use asynchronous implementation of PBT. Trial perturbations occur every \"\n \"`perturbation_interval` for each trial independently. If True, will use synchronous implementation of \"\n \"PBT. Perturbations will occur only after all trials are synced at the same `time_attr` every \"\n \"`perturbation_interval`. Defaults to False. See Appendix A.1 here https://arxiv.org/pdf/1711.09846.pdf.\"\n ),\n )\n\n\n@DeveloperAPI\n@hyperopt_utils.register_scheduler_config(\"pbt_replay\")\n@ludwig_dataclass\nclass PopulationBasedTrainingReplaySchedulerConfig(BaseSchedulerConfig):\n \"\"\"Population Based Training Replay scheduler settings.\"\"\"\n\n type: str = schema_utils.ProtectedString(\"pbt_replay\")\n\n # TODO: This should technically be a required paremeter. Do we need to add support for required params?\n policy_file: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=(\n \"The PBT policy file. Usually this is stored in `~/ray_results/experiment_name/pbt_policy_xxx.txt` where \"\n \"`xxx` is the trial ID.\"\n ),\n )\n\n\n@DeveloperAPI\n@hyperopt_utils.register_scheduler_config(\"pb2\", dependencies=[(\"sklearn\", \"scikit-learn\"), (\"GPy\", \"GPy\")])\n@ludwig_dataclass\nclass PopulationBasedBanditsSchedulerConfig(BaseSchedulerConfig):\n \"\"\"Population Based Bandits (PB2) scheduler settings.\"\"\"\n\n type: str = schema_utils.ProtectedString(\"pb2\")\n\n time_attr: str = time_attr_alias(TIME_TOTAL_S)\n\n perturbation_interval: float = schema_utils.NonNegativeFloat(\n default=60.0,\n description=(\n \"Models will be considered for perturbation at this interval of `time_attr`. Note that perturbation \"\n \"incurs checkpoint overhead, so you shouldn't set this to be too frequent.\"\n ),\n )\n\n hyperparam_bounds: dict | None = schema_utils.Dict(\n default=None,\n description=(\n \"Hyperparameters to mutate. The format is as follows: for each key, enter a list of the form [min, max] \"\n \"representing the minimum and maximum possible hyperparameter values.\"\n ),\n )\n\n quantile_fraction: float = schema_utils.FloatRange(\n default=0.25,\n allow_none=False,\n min=0,\n max=0.5,\n description=(\n \"Parameters are transferred from the top `quantile_fraction` fraction of trials to the bottom \"\n \"`quantile_fraction` fraction. Needs to be between 0 and 0.5. Setting it to 0 essentially implies doing \"\n \"no exploitation at all.\"\n ),\n )\n\n log_config: bool = schema_utils.Boolean(\n default=True,\n description=(\n \"Whether to log the ray config of each model to `local_dir` at each exploit. Allows config schedule to be \"\n \"reconstructed.\"\n ),\n )\n\n require_attrs: bool = schema_utils.Boolean(\n default=True,\n description=(\n \"Whether to require `time_attr` and metric to appear in result for every iteration. If True, error will \"\n \"be raised if these values are not present in trial result.\"\n ),\n )\n\n synch: bool = schema_utils.Boolean(\n default=False,\n description=(\n \"If False, will use asynchronous implementation of PBT. Trial perturbations occur every \"\n \"`perturbation_interval` for each trial independently. If True, will use synchronous implementation of \"\n \"PBT. Perturbations will occur only after all trials are synced at the same `time_attr` every \"\n \"`perturbation_interval`. Defaults to False. See Appendix A.1 here https://arxiv.org/pdf/1711.09846.pdf.\"\n ),\n )\n\n\n@DeveloperAPI\n@hyperopt_utils.register_scheduler_config(\"hb_bohb\")\n@ludwig_dataclass\nclass BOHBSchedulerConfig(BaseHyperbandSchedulerConfig):\n \"\"\"Hyperband for BOHB (hb_bohb) scheduler settings.\"\"\"\n\n type: str = schema_utils.ProtectedString(\"hb_bohb\")\n\n max_t: int = max_t_alias(default=81)\n\n reduction_factor: int = schema_utils.NonNegativeFloat(\n default=3, description=(\"Used to set halving rate and amount. This is simply a unit-less scalar.\")\n )\n\n stop_last_trials: bool = schema_utils.Boolean(\n default=True, description=(\"Whether to terminate the trials after reaching `max_t`. Defaults to True.\")\n )\n\n\n# TODO: Double-check support for this\n@DeveloperAPI\n@hyperopt_utils.register_scheduler_config(\"fifo\")\n@ludwig_dataclass\nclass FIFOSchedulerConfig(BaseSchedulerConfig):\n \"\"\"FIFO trial scheduler settings.\"\"\"\n\n type: str = schema_utils.ProtectedString(\"fifo\")\n\n\n# TODO: Double-check support for this as well as whether Callable args work properly\n@DeveloperAPI\n@hyperopt_utils.register_scheduler_config(\"resource_changing\")\n@ludwig_dataclass\nclass ResourceChangingSchedulerConfig(BaseSchedulerConfig):\n \"\"\"Resource changing scheduler settings.\"\"\"\n\n type: str = schema_utils.ProtectedString(\"resource_changing\")\n\n base_scheduler: str | None | Callable = schema_utils.String(\n default=None,\n allow_none=True,\n description=(\"The scheduler to provide decisions about trials. If None, a default FIFOScheduler will be used.\"),\n )\n\n resources_allocation_function: str | Callable = schema_utils.String(\n default=None,\n allow_none=True,\n description=(\n \"The callable used to change live trial resource requiements during tuning. This callable will be called on\"\n \" each trial as it finishes one step of training. The callable must take four arguments: `TrialRunner`, \"\n \"current `Trial`, current result `dict` and the `ResourceChangingScheduler` calling it. The callable must \"\n \"return a `PlacementGroupFactory`, `Resources`, `dict` or None (signifying no need for an update). If \"\n \"`resources_allocation_function` is None, no resource requirements will be changed at any time. By \"\n \" default, `DistributeResources` will be used, distributing available CPUs and GPUs over all running \"\n \"trials in a robust way, without any prioritization.\"\n ),\n )\n\n\n@DeveloperAPI\ndef get_scheduler_conds():\n \"\"\"Returns a JSON schema of conditionals to validate against scheduler types defined in\n `ludwig.schema.hyperopt.scheduler_registry`.\"\"\"\n conds = []\n for scheduler_config in hyperopt_utils.scheduler_config_registry:\n scheduler_cls = hyperopt_utils.scheduler_config_registry[scheduler_config]\n other_props = schema_utils.unload_jsonschema_from_marshmallow_class(scheduler_cls)[\"properties\"]\n schema_utils.remove_duplicate_fields(other_props)\n preproc_cond = schema_utils.create_cond(\n {\"type\": scheduler_config},\n other_props,\n )\n conds.append(preproc_cond)\n return conds\n\n\n@DeveloperAPI\ndef SchedulerDataclassField(default={\"type\": \"fifo\"}, description=\"Hyperopt scheduler settings.\"):\n \"\"\"Custom dataclass field that when used inside of a dataclass will allow any scheduler in\n `ludwig.schema.hyperopt.scheduler.scheduler_registry`. Sets default scheduler to 'fifo'.\n\n :param default: Dict specifying a scheduler with a `type` field and its associated parameters. Will attempt to use\n `type` to load scheduler from registry with given params. (default: {\"type\": \"fifo\"}).\n :return: Initialized dataclass field that converts untyped dicts with params to scheduler dataclass instances.\n \"\"\"\n\n class SchedulerMarshmallowField(schema_utils.LudwigSchemaField):\n \"\"\"Custom field that deserializes a dict to a valid scheduler from\n `ludwig.schema.hyperopt.scheduler_registry` and creates a corresponding `oneOf` JSON schema for external\n usage.\"\"\"\n\n def _deserialize(self, value, attr, data, **kwargs):\n if value is None:\n return None\n if isinstance(value, dict):\n if \"type\" in value and value[\"type\"] in hyperopt_utils.scheduler_config_registry:\n scheduler_config_cls = hyperopt_utils.scheduler_config_registry[value[\"type\"].lower()]\n try:\n return scheduler_config_cls.Schema().load(value)\n except (TypeError, ConfigValidationError) as e:\n raise ConfigValidationError(\n f\"Invalid params for scheduler: {value}, see `{opt}` definition. Error: {e}\"\n )\n raise ConfigValidationError(\n f\"Invalid params for scheduler: {value}, expect dict with at least a valid `type` attribute.\"\n )\n raise ConfigValidationError(\"Field should be None or dict\")\n\n def _jsonschema_type_mapping(self):\n # Note that this uses the same conditional pattern as combiners:\n return {\n \"type\": \"object\",\n \"properties\": {\n \"type\": {\n \"type\": \"string\",\n \"enum\": list(hyperopt_utils.scheduler_config_registry.keys()),\n \"default\": default[\"type\"],\n \"description\": \"The type of scheduler to use during hyperopt\",\n },\n },\n \"title\": \"scheduler_options\",\n \"allOf\": get_scheduler_conds(),\n \"required\": [\"type\"],\n \"description\": description,\n }\n\n if (\n not isinstance(default, dict)\n or \"type\" not in default\n or default[\"type\"] not in hyperopt_utils.scheduler_config_registry\n ):\n raise ConfigValidationError(f\"Invalid default: `{default}`\")\n try:\n opt = hyperopt_utils.scheduler_config_registry[default[\"type\"].lower()]\n load_default = lambda: opt.Schema().load(default)\n dump_default = opt.Schema().dump(default)\n\n return field(\n metadata={\n \"marshmallow_field\": SchedulerMarshmallowField(\n allow_none=False,\n dump_default=dump_default,\n load_default=load_default,\n metadata={\"description\": description},\n )\n },\n default_factory=load_default,\n )\n except Exception as e:\n raise ConfigValidationError(\n f\"Unsupported scheduler type: {default['type']}. See scheduler_config_registry. Details: {e}\"\n )\n"
},
{
"path": "ludwig/schema/hyperopt/search_algorithm.py",
"content": "from dataclasses import field\nfrom importlib.util import find_spec\nfrom typing import Any\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.hyperopt import utils as hyperopt_utils\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\ndef points_to_evaluate_field(description: str | None = None):\n return schema_utils.DictList(\n description=description\n or (\n \"Initial parameter suggestions to be run first. This is for when you already have some good parameters \"\n \"you want to run first to help the algorithm make better suggestions for future parameters. Needs to be \"\n \"a list of dicts containing the configurations.\"\n ),\n )\n\n\ndef evaluated_rewards_field(description: str | None = None):\n return schema_utils.List(\n description=description\n or (\n \"If you have previously evaluated the parameters passed in as points_to_evaluate you can avoid re-running \"\n \"those trials by passing in the reward attributes as a list so the optimiser can be told the results \"\n \"without needing to re-compute the trial. Must be the same length as `points_to_evaluate`.\"\n )\n )\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass BaseSearchAlgorithmConfig(schema_utils.BaseMarshmallowConfig):\n \"\"\"Basic search algorithm settings.\"\"\"\n\n type: str = schema_utils.String(default=\"variant_generator\", description=\"The search algorithm to use.\")\n\n def set_random_state(self, ludwig_random_state: int) -> None:\n \"\"\"Overwrite the config random state.\n\n Search algorithms refer to random state by different names, however we want to overwrite unset random states\n with the Ludwig random state. This method uses a registry of random state field names to provide a single\n interface across all search algorithms.\n \"\"\"\n rs_field = hyperopt_utils.get_search_algorithm_random_state_field(self.type)\n if rs_field is not None and self.__getattribute__(rs_field) is None:\n self.__setattr__(rs_field, ludwig_random_state)\n\n def dependencies_installed(self) -> bool:\n \"\"\"Some search algorithms require additional packages to be installed, check that they are available.\"\"\"\n missing_packages = []\n missing_installs = []\n for package_name, install_name in hyperopt_utils.get_search_algorithm_dependencies(self.type):\n if find_spec(package_name) is None:\n missing_packages.append(package_name)\n missing_installs.append(install_name)\n\n if missing_packages:\n missing_packages = \", \".join(missing_packages)\n missing_installs = \" \".join(missing_installs)\n raise ImportError(\n f\"Some packages needed to use hyperopt search algorithm {self.type} are not installed: \"\n f\"{missing_packages}. To add these dependencies, run `pip install {missing_installs}`. For more \"\n \"details, please refer to Ray Tune documentation for this search algorithm.\"\n )\n return True\n\n\n@DeveloperAPI\ndef SearchAlgorithmDataclassField(description: str = \"\", default: dict = {\"type\": \"variant_generator\"}):\n class SearchAlgorithmMarshmallowField(schema_utils.LudwigSchemaField):\n def _deserialize(self, value, attr, data, **kwargs):\n if isinstance(value, dict):\n try:\n return BaseSearchAlgorithmConfig.Schema().load(value)\n except (TypeError, ConfigValidationError):\n raise ConfigValidationError(\n f\"Invalid params for scheduler: {value}, see SearchAlgorithmConfig class.\"\n )\n raise ConfigValidationError(\"Field should be dict\")\n\n def _jsonschema_type_mapping(self):\n return {\n # **schema_utils.unload_jsonschema_from_marshmallow_class(BaseSearchAlgorithmConfig),\n \"type\": \"object\",\n \"properties\": {\n \"type\": {\n \"type\": \"string\",\n \"enum\": list(hyperopt_utils.search_algorithm_config_registry.keys()),\n \"default\": default[\"type\"],\n \"description\": \"The type of scheduler to use during hyperopt\",\n },\n },\n \"title\": \"search_algorithm_options\",\n \"required\": [\"type\"],\n \"description\": description,\n }\n\n if not isinstance(default, dict):\n raise ConfigValidationError(f\"Invalid default: `{default}`\")\n\n load_default = lambda: BaseSearchAlgorithmConfig.Schema().load(default)\n dump_default = BaseSearchAlgorithmConfig.Schema().dump(default)\n\n return field(\n metadata={\n \"marshmallow_field\": SearchAlgorithmMarshmallowField(\n allow_none=False,\n load_default=load_default,\n dump_default=dump_default,\n metadata={\"description\": description, \"parameter_metadata\": None},\n )\n },\n default_factory=load_default,\n )\n\n\n@DeveloperAPI\n@hyperopt_utils.register_search_algorithm_config(\"random\", random_state_field=\"random_state\")\n@hyperopt_utils.register_search_algorithm_config(\"variant_generator\", random_state_field=\"random_state\")\n@ludwig_dataclass\nclass BasicVariantSAConfig(BaseSearchAlgorithmConfig):\n type: str = schema_utils.StringOptions(options=[\"random\", \"variant_generator\"], default=\"random\", allow_none=False)\n\n points_to_evaluate: list[dict] | None = schema_utils.DictList(\n description=(\n \"Initial parameter suggestions to be run first. This is for when you already have some good parameters \"\n \"you want to run first to help the algorithm make better suggestions for future parameters. Needs to be \"\n \"a list of dicts containing the configurations.\"\n )\n )\n\n max_concurrent: int = schema_utils.NonNegativeInteger(\n default=0, description=\"Maximum number of concurrently running trials. If 0 (default), no maximum is enforced.\"\n )\n\n constant_grid_search: bool = schema_utils.Boolean(\n default=False,\n description=(\n \"If this is set to True, Ray Tune will first try to sample random values and keep them constant over grid \"\n \"search parameters. If this is set to False (default), Ray Tune will sample new random parameters in each \"\n \"grid search condition.\"\n ),\n )\n\n random_state: int = schema_utils.Integer(\n default=None,\n allow_none=True,\n description=(\n \"Seed or numpy random generator to use for reproducible results. If None (default), will use the global \"\n \"numpy random generator (np.random). Please note that full reproducibility cannot be guaranteed in a \"\n \"distributed environment.\"\n ),\n )\n\n\n@DeveloperAPI\n@hyperopt_utils.register_search_algorithm_config(\n \"ax\", dependencies=[(\"ax\", \"ax-platform\"), (\"sqlalchemy\", \"sqlalchemy\")]\n)\n@ludwig_dataclass\nclass AxSAConfig(BaseSearchAlgorithmConfig):\n type: str = schema_utils.ProtectedString(\"ax\")\n\n space: list[dict] | None = schema_utils.DictList(\n description=(\n r\"Parameters in the experiment search space. Required elements in the dictionaries are: \\“name\\” (name of \"\n r\"this parameter, string), \\“type\\” (type of the parameter: \\“range\\”, \\“fixed\\”, or \\“choice\\”, string), \"\n r\"\\“bounds\\” for range parameters (list of two values, lower bound first), \\“values\\” for choice \"\n r\"parameters (list of values), and \\“value\\” for fixed parameters (single value).\"\n )\n )\n\n points_to_evaluate: list[dict] | None = points_to_evaluate_field()\n\n parameter_constraints: list | None = schema_utils.List(\n description=r\"Parameter constraints, such as \\“x3 >= x4\\” or \\“x3 + x4 >= 2\\”.\"\n )\n\n outcome_constraints: list | None = schema_utils.List(\n description=r\"Outcome constraints of form \\“metric_name >= bound\\”, like \\“m1 <= 3.\\”\"\n )\n\n\n@DeveloperAPI\n@hyperopt_utils.register_search_algorithm_config(\n \"bayesopt\", random_state_field=\"random_state\", dependencies=[(\"bayes_opt\", \"bayesian-optimization\")]\n)\n@ludwig_dataclass\nclass BayesOptSAConfig(BaseSearchAlgorithmConfig):\n type: str = schema_utils.ProtectedString(\"bayesopt\")\n\n space: dict | None = schema_utils.Dict(\n description=(\n \"Continuous search space. Parameters will be sampled from this space which will be used to run trials\"\n )\n )\n\n points_to_evaluate: list[dict] | None = points_to_evaluate_field()\n\n utility_kwargs: dict | None = schema_utils.Dict(\n description=(\n \"Parameters to define the utility function. The default value is a dictionary with three keys: \"\n \"- kind: ucb (Upper Confidence Bound) - kappa: 2.576 - xi: 0.0\"\n )\n )\n\n random_state: int = schema_utils.Integer(default=None, allow_none=True, description=\"Used to initialize BayesOpt.\")\n\n random_search_steps: int = schema_utils.Integer(\n default=10,\n description=(\n \"Number of initial random searches. This is necessary to avoid initial local overfitting of \"\n \"the Bayesian process.\"\n ),\n )\n\n verbose: int = schema_utils.IntegerOptions(\n options=[0, 1, 2], default=0, description=\"The level of verbosity. `0` is least verbose, `2` is most verbose.\"\n )\n\n patience: int = schema_utils.NonNegativeInteger(\n default=5, description=\"Number of epochs to wait for a change in the top models.\"\n )\n\n skip_duplicate: bool = schema_utils.Boolean(\n default=True,\n description=(\n \"If False, the optimizer will allow duplicate points to be registered. This behavior may be desired in \"\n \"high noise situations where repeatedly probing the same point will give different answers. In other \"\n \"situations, the acquisition may occasionaly generate a duplicate point.\"\n ),\n )\n\n\n@DeveloperAPI\n@hyperopt_utils.register_search_algorithm_config(\"blendsearch\", dependencies=[(\"flaml\", \"flaml[blendsearch]\")])\n@ludwig_dataclass\nclass BlendsearchSAConfig(BaseSearchAlgorithmConfig):\n type: str = schema_utils.ProtectedString(\"blendsearch\")\n\n\n@DeveloperAPI\n@hyperopt_utils.register_search_algorithm_config(\n \"bohb\", random_state_field=\"seed\", dependencies=[(\"hpbandster\", \"hpbandster\"), (\"ConfigSpace\", \"ConfigSpace\")]\n)\n@ludwig_dataclass\nclass BOHBSAConfig(BaseSearchAlgorithmConfig):\n type: str = schema_utils.ProtectedString(\"bohb\")\n\n space: dict | None = schema_utils.Dict(\n description=(\n \"Continuous ConfigSpace search space. Parameters will be sampled from this space which will be used \"\n \"to run trials.\"\n )\n )\n\n bohb_config: dict | None = schema_utils.Dict(description=\"configuration for HpBandSter BOHB algorithm\")\n\n points_to_evaluate: list[dict] | None = points_to_evaluate_field()\n\n seed: int | None = schema_utils.Integer(\n default=None,\n allow_none=True,\n description=(\n \"Optional random seed to initialize the random number generator. Setting this should lead to identical \"\n \"initial configurations at each run.\"\n ),\n )\n\n max_concurrent: int = schema_utils.Integer(\n default=0,\n description=(\n \"Number of maximum concurrent trials. If this Searcher is used in a `ConcurrencyLimiter`, the \"\n \"`max_concurrent` value passed to it will override the value passed here. Set to <= 0 for no limit on \"\n \"concurrency.\"\n ),\n )\n\n\n@DeveloperAPI\n@hyperopt_utils.register_search_algorithm_config(\"cfo\", dependencies=[(\"flaml\", \"flaml\")])\n@ludwig_dataclass\nclass CFOSAConfig(BaseSearchAlgorithmConfig):\n type: str = schema_utils.ProtectedString(\"cfo\")\n\n\n@DeveloperAPI\n@hyperopt_utils.register_search_algorithm_config(\n \"dragonfly\", random_state_field=\"random_state_seed\", dependencies=[(\"dragonfly\", \"dragonfly-opt\")]\n)\n@ludwig_dataclass\nclass DragonflySAConfig(BaseSearchAlgorithmConfig):\n type: str = schema_utils.ProtectedString(\"dragonfly\")\n\n optimizer: str | None = schema_utils.StringOptions(\n options=[\"random\", \"bandit\", \"genetic\"],\n default=None,\n allow_none=True,\n description=(\n \"Optimizer provided from dragonfly. Choose an optimiser that extends `BlackboxOptimiser`. If this is a \"\n \"string, `domain` must be set and `optimizer` must be one of [random, bandit, genetic].\"\n ),\n )\n\n domain: str | None = schema_utils.StringOptions(\n options=[\"cartesian\", \"euclidean\"],\n default=None,\n allow_none=True,\n description=(\n \"Optional domain. Should only be set if you don't pass an optimizer as the `optimizer` argument. If set, \"\n \"has to be one of `[cartesian, euclidean]`.\"\n ),\n )\n\n space: list[dict] | None = schema_utils.DictList(\n description=(\n \"Search space. Should only be set if you don't pass an optimizer as the `optimizer` argument. Defines the \"\n \"search space and requires a `domain` to be set. Can be automatically converted from the `param_space` \"\n \"dict passed to `tune.Tuner()`.\"\n )\n )\n\n points_to_evaluate: list[dict] | None = points_to_evaluate_field()\n\n evaluated_rewards: list | None = evaluated_rewards_field()\n\n random_state_seed: int | None = schema_utils.Integer(\n default=None,\n allow_none=True,\n description=(\n \"Seed for reproducible results. Defaults to None. Please note that setting this to a value will change \"\n \"global random state for `numpy` on initalization and loading from checkpoint.\"\n ),\n )\n\n\n@DeveloperAPI\n@hyperopt_utils.register_search_algorithm_config(\n \"hebo\", random_state_field=\"random_state_seed\", dependencies=[(\"hebo\", \"HEBO\")]\n)\n@ludwig_dataclass\nclass HEBOSAConfig(BaseSearchAlgorithmConfig):\n type: str = schema_utils.ProtectedString(\"hebo\")\n\n space: list[dict] | None = schema_utils.DictList(\n description=\"A dict mapping parameter names to Tune search spaces or a HEBO DesignSpace object.\"\n )\n\n points_to_evaluate: list[dict] | None = points_to_evaluate_field()\n\n evaluated_rewards: list | None = evaluated_rewards_field()\n\n random_state_seed: int | None = schema_utils.Integer(\n default=None,\n allow_none=True,\n description=(\n \"Seed for reproducible results. Defaults to None. Please note that setting this to a value will change \"\n \"global random state for `numpy` on initalization and loading from checkpoint.\"\n ),\n )\n\n max_concurrent: int = schema_utils.NonNegativeInteger(\n default=8,\n description=(\n \"Number of maximum concurrent trials. If this Searcher is used in a `ConcurrencyLimiter`, the \"\n \"`max_concurrent` value passed to it will override the value passed here.\"\n ),\n )\n\n\n@DeveloperAPI\n@hyperopt_utils.register_search_algorithm_config(\n \"hyperopt\", random_state_field=\"random_state_seed\", dependencies=[(\"hyperopt\", \"hyperopt\")]\n)\n@ludwig_dataclass\nclass HyperoptSAConfig(BaseSearchAlgorithmConfig):\n type: str = schema_utils.ProtectedString(\"hyperopt\")\n\n space: list[dict] | None = schema_utils.DictList(\n description=(\n \"HyperOpt configuration. Parameters will be sampled from this configuration and will be used to override \"\n \"parameters generated in the variant generation process.\"\n )\n )\n\n points_to_evaluate: list[dict] | None = points_to_evaluate_field()\n\n n_initial_points: int = schema_utils.PositiveInteger(\n default=20,\n description=(\n \"The number of random evaluations of the objective function before starting to approximate it with tree \"\n \"parzen estimators. Defaults to 20.\"\n ),\n )\n\n random_state_seed: int | None = schema_utils.Integer(\n default=None,\n allow_none=True,\n description=(\"Seed for reproducible results. Defaults to None.\"),\n )\n\n gamma: float = schema_utils.FloatRange(\n min=0.0,\n max=1.0,\n default=0.25,\n description=(\n \"The split to use in TPE. TPE models two splits of the evaluated hyperparameters: the top performing \"\n \"`gamma` percent, and the remaining examples. For more details, see [Making a Science of Model Search: \"\n \"Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures.]\"\n \"(http://proceedings.mlr.press/v28/bergstra13.pdf).\"\n ),\n )\n\n\n@DeveloperAPI\n@hyperopt_utils.register_search_algorithm_config(\"nevergrad\", dependencies=[(\"nevergrad\", \"nevergrad\")])\n@ludwig_dataclass\nclass NevergradSAConfig(BaseSearchAlgorithmConfig):\n type: str = schema_utils.ProtectedString(\"nevergrad\")\n\n # TODO: Add a registry mapping string names to nevergrad optimizers\n # optimizer: Optional[str] = None\n\n # TODO: Add schemas for nevergrad optimizer kwargs\n optimizer_kwargs: dict | None = schema_utils.Dict(description=\"Kwargs passed in when instantiating the optimizer.\")\n\n space: list[dict] | None = schema_utils.DictList(\n description=(\n \"Nevergrad parametrization to be passed to optimizer on instantiation, or list of parameter names if you \"\n \"passed an optimizer object.\"\n )\n )\n\n points_to_evaluate: list[dict] | None = points_to_evaluate_field()\n\n\n@DeveloperAPI\n@hyperopt_utils.register_search_algorithm_config(\n \"optuna\", random_state_field=\"seed\", dependencies=[(\"optuna\", \"optuna\")]\n)\n@ludwig_dataclass\nclass OptunaSAConfig(BaseSearchAlgorithmConfig):\n type: str = schema_utils.ProtectedString(\"optuna\")\n\n space: dict | None = schema_utils.Dict(\n description=(\n \"Hyperparameter search space definition for Optuna's sampler. This can be either a dict with parameter \"\n \"names as keys and optuna.distributions as values, or a Callable - in which case, it should be a \"\n \"define-by-run function using optuna.trial to obtain the hyperparameter values. The function should \"\n \"return either a dict of constant values with names as keys, or None. For more information, see \"\n \"[the Optuna docs]\"\n \"(https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/002_configurations.html).\"\n )\n )\n\n points_to_evaluate: list[dict] | None = points_to_evaluate_field()\n\n # TODO: Add a registry of Optuna samplers schemas\n # sampler = None\n\n seed: int | None = schema_utils.Integer(\n default=None,\n allow_none=True,\n description=(\n \"Seed to initialize sampler with. This parameter is only used when `sampler=None`. In all other cases, \"\n \"the sampler you pass should be initialized with the seed already.\"\n ),\n )\n\n evaluated_rewards: list | None = evaluated_rewards_field()\n\n\n@DeveloperAPI\n@hyperopt_utils.register_search_algorithm_config(\"skopt\", dependencies=[(\"skopt\", \"scikit-optimize\")])\nclass SkoptSAConfig(BaseSearchAlgorithmConfig):\n type: str = schema_utils.ProtectedString(\"skopt\")\n\n optimizer: Any | None = None\n\n space: dict | None = schema_utils.Dict(\n description=(\n \"A dict mapping parameter names to valid parameters, i.e. tuples for numerical parameters and lists \"\n \"for categorical parameters. If you passed an optimizer instance as the optimizer argument, this should \"\n \"be a list of parameter names instead.\"\n )\n )\n\n points_to_evaluate: list[dict] | None = points_to_evaluate_field()\n\n evaluated_rewards: list | None = evaluated_rewards_field(\n description=(\n \"If you have previously evaluated the parameters passed in as points_to_evaluate you can avoid \"\n \"re-running those trials by passing in the reward attributes as a list so the optimiser can be told the \"\n \"results without needing to re-compute the trial. Must be the same length as points_to_evaluate. (See \"\n \"tune/examples/skopt_example.py)\"\n )\n )\n\n convert_to_python: bool = schema_utils.Boolean(\n default=True,\n description=\"SkOpt outputs numpy primitives (e.g. `np.int64`) instead of Python types. If this setting is set \"\n \"to `True`, the values will be converted to Python primitives.\",\n )\n\n\n@DeveloperAPI\n@hyperopt_utils.register_search_algorithm_config(\"zoopt\", dependencies=[(\"zoopt\", \"zoopt\")])\n@ludwig_dataclass\nclass ZooptSAConfig(BaseSearchAlgorithmConfig):\n type: str = schema_utils.ProtectedString(\"zoopt\")\n\n algo: str = schema_utils.ProtectedString(\n pstring=\"asracos\",\n description=\"To specify an algorithm in zoopt you want to use. Only support ASRacos currently.\",\n )\n\n budget: int | None = schema_utils.PositiveInteger(\n default=None, allow_none=True, description=\"Optional. Number of samples.\"\n )\n\n dim_dict: dict | None = schema_utils.Dict(\n description=(\n \"Dimension dictionary. For continuous dimensions: (continuous, search_range, precision); For discrete \"\n \"dimensions: (discrete, search_range, has_order); For grid dimensions: (grid, grid_list). More details \"\n \"can be found in zoopt package.\"\n )\n )\n\n points_to_evaluate: list[dict] | None = points_to_evaluate_field()\n\n parallel_num: int = schema_utils.PositiveInteger(\n default=1,\n description=(\n \"How many workers to parallel. Note that initial phase may start less workers than this number. More \"\n \"details can be found in zoopt package.\"\n ),\n )\n"
},
{
"path": "ludwig/schema/hyperopt/utils.py",
"content": "from collections.abc import Callable\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.utils.registry import Registry\n\nparameter_config_registry = Registry()\nscheduler_config_registry = Registry()\nscheduler_dependencies_registry = Registry()\nsearch_algorithm_config_registry = Registry()\nsearch_algorithm_dependencies_registry = Registry()\nsearch_algorithm_random_state_field_registry = Registry()\n\n\n@DeveloperAPI\ndef get_parameter_cls(name: str) -> type[\"BaseParameterConfig\"]: # noqa: F821\n \"\"\"Get a registered hyperopt parameter config class by name.\n\n Args:\n name: the name of a parameter config class registered in `ludwig.schema.hyperopt.parameter`\n\n Returns:\n A parameter config class from `ludwig.schema.hyperopt.parameter`\n \"\"\"\n return parameter_config_registry[name]\n\n\n@DeveloperAPI\ndef get_scheduler_cls(name: str) -> type[\"BaseSchedulerConfig\"]: # noqa: F821\n \"\"\"Get a registered hyperopt scheduler config class by name.\n\n Args:\n name: the name of a scheduler config class registered in `ludwig.schema.hyperopt.scheduler`\n\n Returns:\n A scheduler config class from `ludwig.schema.hyperopt.scheduler`\n \"\"\"\n return search_algorithm_config_registry[name]\n\n\n@DeveloperAPI\ndef get_scheduler_dependencies(name: str) -> list[str]:\n \"\"\"Get the list of dependencies for a registered hyperopt scheduler.\n\n Args:\n name: the name of a scheduler config class registered in `ludwig.schema.hyperopt.scheduler`\n\n Returns:\n The list of imports needed to use the scheduler\n \"\"\"\n return scheduler_dependencies_registry[name]\n\n\n@DeveloperAPI\ndef get_search_algorithm_cls(name: str) -> type[\"BaseSearchAlgorithmConfig\"]: # noqa: F821\n \"\"\"Get a registered hyperopt search algorithm config class by name.\n\n Args:\n name: the name of a search algorithm config class registered in `ludwig.schema.hyperopt.search_algorithm`\n\n Returns:\n A scheduler config class from `ludwig.schema.hyperopt.search_algorithm`\n \"\"\"\n return search_algorithm_config_registry[name]\n\n\n@DeveloperAPI\ndef get_search_algorithm_dependencies(name: str) -> list[str]:\n \"\"\"Get the list of dependencies for a registered hyperopt search algorithm.\n\n Args:\n name: the name of a search algorithm config class registered in `ludwig.schema.hyperopt.search_algorithm`\n\n Returns:\n The list of imports needed to use the search algorithm\n \"\"\"\n return search_algorithm_dependencies_registry[name]\n\n\n@DeveloperAPI\ndef get_search_algorithm_random_state_field(name: str):\n \"\"\"Get the field name of the random state for a registered hyperopt search algorithm.\n\n Args:\n name: the name of a search algorithm config class registered in `ludwig.schema.hyperopt.search_algorithm`\n\n Returns:\n The name of the random state field in the config\n \"\"\"\n return search_algorithm_random_state_field_registry[name]\n\n\n@DeveloperAPI\ndef register_parameter_config(name: str) -> Callable:\n \"\"\"Register a parameter config class by name.\n\n Args:\n name: the name to register the parameter class under, does not need to correspond to the value of `space`\n\n Returns:\n Wrapper function to decorate a `BaseParameterConfig` subclass\n \"\"\"\n\n def wrap(cls: type[\"BaseParameterConfig\"]) -> type[\"BaseParameterConfig\"]: # noqa: F821\n \"\"\"Add a parameter config class to the registry.\n\n Args:\n cls: a subclass of `BaseParameterConfig`\n\n Returns:\n `cls` unaltered\n \"\"\"\n parameter_config_registry[name] = cls\n return cls\n\n return wrap\n\n\n@DeveloperAPI\ndef register_scheduler_config(name: str, dependencies: list[tuple[str]] | None = None):\n \"\"\"Register a scheduler config class by name.\n\n Args:\n name: the name to scheduler the parameter class under, does not need to correspond to the value of `type`\n dependencies: the list of scheduler dependency package name/install name pairs, e.g.\n `(\"sklearn\", \"scikit-learn\")`\n\n Returns:\n Wrapper function to decorate a `BaseSchedulerConfig` subclass\n \"\"\"\n\n def wrap(scheduler_config: type[\"BaseSchedulerConfig\"]) -> type[\"BaseSchedulerConfig\"]: # noqa: F821\n \"\"\"Add a parameter config class to the registry.\n\n Args:\n cls: a subclass of `BaseParameterConfig`\n\n Returns:\n `cls` unaltered\n \"\"\"\n scheduler_config_registry[name] = scheduler_config\n scheduler_dependencies_registry[name] = dependencies if dependencies is not None else []\n return scheduler_config\n\n return wrap\n\n\n# TODO: create a search alg metadata class to register in place of individual metadata args\n@DeveloperAPI\ndef register_search_algorithm_config(\n name: str, random_state_field: str | None = None, dependencies: list[tuple[str, str]] | None = None\n) -> Callable:\n \"\"\"Register a search algorithm config class by name.\n\n Args:\n name: the name to register the search algorithm class under, does not need to correspond to the value of `type`\n random_state_field: the name of the random state in this search algorithm\n dependencies: the list of search algorithm dependency package name/install name pairs, e.g.\n `(\"sklearn\", \"scikit-learn\")`\n\n Returns:\n Wrapper function to decorate a `BaseSearchAlgorithmConfig` subclass\n \"\"\"\n\n def wrap(cls: type[\"BaseSearchAlgorithmConfig\"]) -> type[\"BaseSearchAlgorithmConfig\"]: # noqa: F821\n search_algorithm_config_registry[name] = cls\n search_algorithm_dependencies_registry[name] = dependencies if dependencies is not None else []\n search_algorithm_random_state_field_registry[name] = random_state_field\n return cls\n\n return wrap\n"
},
{
"path": "ludwig/schema/jsonschema.py",
"content": "\"\"\"JSON Schema generation for Ludwig config classes.\n\nUses pydantic's model_json_schema() under the hood, replacing the previous marshmallow-based converter.\n\"\"\"\n\n\ndef marshmallow_schema_to_jsonschema_dict(schema_instance):\n \"\"\"Backward-compatible JSON schema generation.\n\n Previously converted marshmallow schemas. Now uses pydantic's model_json_schema().\n The schema_instance can be either:\n - A pydantic model class (BaseMarshmallowConfig subclass)\n - A _SchemaAdapter instance\n - Legacy: called with a marshmallow Schema instance (raises helpful error)\n \"\"\"\n from ludwig.schema.utils import _SchemaAdapter, BaseMarshmallowConfig\n\n # Handle _SchemaAdapter\n if isinstance(schema_instance, _SchemaAdapter):\n cls = schema_instance._cls\n elif isinstance(schema_instance, type) and issubclass(schema_instance, BaseMarshmallowConfig):\n cls = schema_instance\n elif isinstance(schema_instance, BaseMarshmallowConfig):\n cls = type(schema_instance)\n else:\n raise TypeError(\n f\"Expected a Ludwig config class or schema adapter, got {type(schema_instance)}. \"\n \"Marshmallow schemas are no longer supported. Use pydantic BaseModel subclasses.\"\n )\n\n schema_dict = cls.model_json_schema()\n name = cls.__name__\n\n # Wrap in definitions format for backward compat\n return {\n \"$schema\": \"http://json-schema.org/draft-07/schema#\",\n \"definitions\": {name: schema_dict},\n \"$ref\": f\"#/definitions/{name}\",\n }\n"
},
{
"path": "ludwig/schema/llms/__init__.py",
"content": ""
},
{
"path": "ludwig/schema/llms/base_model.py",
"content": "import logging\nimport os\nfrom dataclasses import field\n\nfrom transformers import AutoConfig\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import BASE_MODEL\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.metadata import LLM_METADATA\nfrom ludwig.schema.metadata.parameter_metadata import convert_metadata_to_json\n\nlogger = logging.getLogger(__name__)\n\n# Maps a preset LLM name to the full slash-delimited HF path. If the user chooses a preset LLM, the preset LLM name is\n# replaced with the full slash-delimited HF path using this map, after JSON validation but before config object\n# initialization.\nMODEL_PRESETS = {\n # Bloom\n \"bloomz-3b\": \"bigscience/bloomz-3b\",\n \"bloomz-7b1\": \"bigscience/bloomz-7b1\",\n # CodeLlama\n \"codellama-7b\": \"codellama/CodeLlama-7b-hf\",\n \"codellama-13b\": \"codellama/CodeLlama-13b-hf\",\n \"codellama-34b\": \"codellama/CodeLlama-34b-hf\",\n \"codellama-7b-instruct\": \"codellama/CodeLlama-7b-instruct-hf\",\n \"codellama-13b-instruct\": \"codellama/CodeLlama-13b-instruct-hf\",\n \"codellama-34b-instruct\": \"codellama/CodeLlama-34b-instruct-hf\",\n # GPT Neo and GPT J\n \"gpt-neo-2.7B\": \"EleutherAI/gpt-neo-2.7B\",\n \"gpt-j-6b\": \"EleutherAI/gpt-j-6b\",\n # LLama-2\n \"llama-2-7b\": \"meta-llama/Llama-2-7b-hf\",\n \"llama-2-13b\": \"meta-llama/Llama-2-13b-hf\",\n \"llama-2-70b\": \"meta-llama/Llama-2-70b-hf\",\n \"llama-2-7b-chat\": \"meta-llama/Llama-2-7b-chat-hf\",\n \"llama-2-13b-chat\": \"meta-llama/Llama-2-13b-chat-hf\",\n \"llama-2-70b-chat\": \"meta-llama/Llama-2-70b-chat-hf\",\n # Mistral\n \"mistral-7b\": \"mistralai/Mistral-7B-v0.1\",\n \"mistral-7b-instruct\": \"mistralai/Mistral-7B-Instruct-v0.1\",\n # Mixtral\n \"mixtral-8x7b\": \"mistralai/Mixtral-8x7B-v0.1\",\n \"mixtral-8x7b-instruct\": \"mistralai/Mixtral-8x7B-Instruct-v0.1\",\n # OPT\n \"opt-350m\": \"facebook/opt-350m\",\n \"opt-1.3b\": \"facebook/opt-1.3b\",\n \"opt-6.7b\": \"facebook/opt-6.7b\",\n # Pythia\n \"pythia-2.8b\": \"EleutherAI/pythia-2.8b\",\n \"pythia-12b\": \"EleutherAI/pythia-12b\",\n # Vicuna\n \"vicuna-7b\": \"lmsys/vicuna-7b-v1.3\",\n \"vicuna-13b\": \"lmsys/vicuna-13b-v1.3\",\n # Zephyr\n \"zephyr-7b-alpha\": \"HuggingFaceH4/zephyr-7b-alpha\",\n \"zephyr-7b-beta\": \"HuggingFaceH4/zephyr-7b-beta\",\n # Phi\n \"phi-1\": \"microsoft/phi-1\",\n \"phi-1_5\": \"microsoft/phi-1_5\",\n \"phi-2\": \"microsoft/phi-2\",\n}\n\n\n@DeveloperAPI\ndef BaseModelDataclassField():\n description = (\n \"Base pretrained model to use. This can be one of the presets defined by Ludwig, a fully qualified \"\n \"name of a pretrained model from the HuggingFace Hub, or a path to a directory containing a \"\n \"pretrained model.\"\n )\n\n def validate(model_name: str):\n \"\"\"Validates and upgrades the given model name to its full path, if applicable.\n\n If the name exists in `MODEL_PRESETS`, returns the corresponding value from the dict; otherwise checks if the\n given name (which should be a full path) exists locally or in the transformers library.\n \"\"\"\n if isinstance(model_name, str):\n if model_name in MODEL_PRESETS:\n return MODEL_PRESETS[model_name]\n if os.path.isdir(model_name):\n return model_name\n try:\n AutoConfig.from_pretrained(model_name, trust_remote_code=True)\n return model_name\n except OSError:\n raise ConfigValidationError(\n f\"Specified base model `{model_name}` could not be loaded. If this is a private repository, make \"\n f\"sure to set HUGGING_FACE_HUB_TOKEN in your environment. Check that {model_name} is a valid \"\n \"pretrained CausalLM listed on huggingface or a valid local directory containing the weights for a \"\n \"pretrained CausalLM from huggingface. See: \"\n \"https://huggingface.co/models?pipeline_tag=text-generation&sort=downloads for a full list.\"\n )\n raise ConfigValidationError(\n f\"`base_model` should be a string, instead given: {model_name}. This can be a preset or any pretrained \"\n \"CausalLM on huggingface. See: https://huggingface.co/models?pipeline_tag=text-generation&sort=downloads\"\n )\n\n class BaseModelField(schema_utils.LudwigSchemaField):\n def _serialize(self, value, attr, obj, **kwargs):\n if isinstance(value, str):\n return value\n raise ConfigValidationError(f\"Value to serialize is not a string: {value}\")\n\n def _deserialize(self, value, attr, obj, **kwargs):\n return validate(value)\n\n def _jsonschema_type_mapping(self):\n return {\n \"anyOf\": [\n {\n \"type\": \"string\",\n \"enum\": list(MODEL_PRESETS.keys()),\n \"description\": (\n \"Pick from a set of popular LLMs of different sizes across a variety of architecture types.\"\n ),\n \"title\": \"preset\",\n \"parameter_metadata\": convert_metadata_to_json(LLM_METADATA[BASE_MODEL][\"_anyOf\"][\"preset\"]),\n },\n {\n \"type\": \"string\",\n \"description\": \"Enter the full path to a huggingface LLM.\",\n \"title\": \"custom\",\n \"parameter_metadata\": convert_metadata_to_json(LLM_METADATA[BASE_MODEL][\"_anyOf\"][\"custom\"]),\n },\n ],\n \"description\": description,\n \"title\": \"base_model_options\",\n \"parameter_metadata\": convert_metadata_to_json(LLM_METADATA[BASE_MODEL][\"_meta\"]),\n }\n\n return field(\n metadata={\n \"marshmallow_field\": BaseModelField(\n required=True,\n allow_none=False,\n validate=validate,\n metadata={ # TODO: extra metadata dict probably unnecessary, but currently a widespread pattern\n \"description\": description,\n \"parameter_metadata\": convert_metadata_to_json(LLM_METADATA[BASE_MODEL][\"_meta\"]),\n },\n ),\n },\n # TODO: This is an unfortunate side-effect of dataclass init order - you cannot have non-default fields follow\n # default fields, so we have to give `base_model` a fake default of `None`.\n default=None,\n )\n"
},
{
"path": "ludwig/schema/llms/generation.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.metadata import LLM_METADATA\n\n\n@DeveloperAPI\n@schema_utils.ludwig_dataclass\nclass LLMGenerationConfig(schema_utils.BaseMarshmallowConfig):\n \"\"\"Parameters for LLM Generation Config.\n\n Should match the parameters in\n https://huggingface.co/docs/transformers/v4.28.0/en/main_classes/text_generation#transformers.GenerationConfig\n \"\"\"\n\n # Parameters that control the length of the output\n\n max_new_tokens: int | None = schema_utils.PositiveInteger(\n default=32,\n allow_none=True,\n description=\"The maximum number of new tokens to generate, ignoring the number of tokens in the input prompt. \"\n \"If not set, this is dynamically determined by Ludwig based on either the `max_sequence_length` of the ouput \"\n \"feature, the global_max_sequence_length specified in preprocessing (if specified), or the \"\n \"maximum context length supported by the model (in the order specified).\",\n parameter_metadata=LLM_METADATA[\"generation\"][\"max_new_tokens\"],\n )\n\n min_new_tokens: int | None = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"The minimum number of new tokens to generate, ignoring the number of tokens in the input prompt.\",\n parameter_metadata=LLM_METADATA[\"generation\"][\"min_new_tokens\"],\n )\n\n max_length: int = schema_utils.PositiveInteger(\n default=32,\n allow_none=True,\n description=\"The maximum length the generated tokens can have. Corresponds to the length of the input prompt \"\n \"+ max_new_tokens. Its effect is overridden by max_new_tokens, if also set.\",\n parameter_metadata=LLM_METADATA[\"generation\"][\"max_length\"],\n )\n\n min_length: int = schema_utils.NonNegativeInteger(\n default=0,\n allow_none=True,\n description=\"The minimum length of the sequence to be generated. Corresponds to the length of the \"\n \"input prompt + min_new_tokens. Its effect is overridden by min_new_tokens, if also set.\",\n parameter_metadata=LLM_METADATA[\"generation\"][\"min_length\"],\n )\n\n early_stopping: bool | str | None = schema_utils.Boolean(\n default=False,\n description=\"Controls the stopping condition for beam-based methods, like beam-search. It accepts the following\"\n \" values: True, where the generation stops as soon as there are num_beams complete candidates; False, where an \"\n \"heuristic is applied and the generation stops when is it very unlikely to find better candidates; `never`, \"\n \"where the beam search procedure only stops when there cannot be better candidates (canonical beam search \"\n \"algorithm)\",\n )\n\n max_time: float | None = schema_utils.FloatRange(\n default=None,\n min=None,\n max=None,\n allow_none=True,\n description=\"The maximum amount of time you allow the computation to run for in seconds. generation will still\"\n \" finish the current pass after allocated time has been passed. \",\n )\n\n # Parameters that control the generation strategy used\n\n do_sample: bool | None = schema_utils.Boolean(\n default=True,\n description=\"Whether or not to use sampling ; use greedy decoding otherwise.\",\n parameter_metadata=LLM_METADATA[\"generation\"][\"do_sample\"],\n )\n\n num_beams: int | None = schema_utils.PositiveInteger(\n default=1,\n allow_none=True,\n description=\"Number of beams for beam search. 1 means no beam search and is the default value.\"\n \" The beam search strategy generates the translation word by word from left-to-right while keeping a fixed\"\n \" number (beam) of active candidates at each time step during token generation. By increasing the beam size,\"\n \" the translation performance can increase at the expense of significantly reducing the decoder speed.\",\n parameter_metadata=LLM_METADATA[\"generation\"][\"num_beams\"],\n )\n\n num_beam_groups: int | None = schema_utils.PositiveInteger(\n default=1,\n allow_none=True,\n description=\"Number of groups to divide num_beams into in order to ensure diversity among different groups of \"\n \"beams. 1 means no group beam search.\",\n )\n\n penalty_alpha: float | None = schema_utils.NonNegativeFloat(\n default=None,\n allow_none=True,\n description=\"The values balance the model confidence and the degeneration penalty in contrastive \"\n \" search decoding.\",\n )\n\n use_cache: bool | None = schema_utils.Boolean(\n default=True,\n description=\"Whether or not the model should use the past last key/values attentions (if applicable to the \"\n \"model) to speed up decoding.\",\n parameter_metadata=LLM_METADATA[\"generation\"][\"use_cache\"],\n )\n\n prompt_lookup_num_tokens: int | None = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"The number of tokens to consider as a candidate from the prompt for prompt lookup decoding, \"\n \" an alternate way of performing assisted generation. If set to 0, the prompt lookup decoding is not used.\",\n parameter_metadata=LLM_METADATA[\"generation\"][\"prompt_lookup_num_tokens\"],\n )\n\n # Parameters for manipulation of the model output logits\n\n temperature: float | None = schema_utils.NonNegativeFloat(\n default=0.1,\n allow_none=True,\n description=\"Temperature is used to control the randomness of predictions.\"\n \" A high temperature value (closer to 1) makes the output more diverse and random, while a lower temperature\"\n \" (closer to 0) makes the model's responses more deterministic and focused on the most likely outcome.\"\n \" In other words, temperature adjusts the probability distribution from which the model picks the next token.\",\n parameter_metadata=LLM_METADATA[\"generation\"][\"temperature\"],\n )\n\n top_k: int | None = schema_utils.PositiveInteger(\n default=50,\n allow_none=True,\n description=\"The number of highest probability vocabulary tokens to keep for top-k-filtering.\",\n parameter_metadata=LLM_METADATA[\"generation\"][\"top_k\"],\n )\n\n top_p: float | None = schema_utils.FloatRange(\n default=1.0,\n min=0.0,\n max=1.0,\n allow_none=True,\n description=\"If set to float < 1, only the most probable tokens with probabilities that add up to \"\n \"top_p or higher are kept for generation.\",\n parameter_metadata=LLM_METADATA[\"generation\"][\"top_p\"],\n )\n\n typical_p: float | None = schema_utils.FloatRange(\n default=1.0,\n min=0.0,\n max=1.0,\n allow_none=True,\n description=\"Local typicality measures how similar the conditional probability of predicting a target token \"\n \"next is to the expected conditional probability of predicting a random token next, given the partial text \"\n \"already generated. If set to float < 1, the smallest set of the most locally typical tokens with \"\n \"probabilities that add up to typical_p or higher are kept for generation.\",\n )\n\n epsilon_cutoff: float | None = schema_utils.FloatRange(\n default=0.0,\n min=0.0,\n max=1.0,\n allow_none=True,\n description=\"If set to float strictly between 0 and 1, only tokens with a conditional probability greater \"\n \"than epsilon_cutoff will be sampled. In the paper, suggested values range from 3e-4 to 9e-4, depending on the\"\n \" size of the model.\",\n )\n\n eta_cutoff: float | None = schema_utils.FloatRange(\n default=0.0,\n min=0.0,\n max=1.0,\n allow_none=True,\n description=\"Eta sampling is a hybrid of locally typical sampling and epsilon sampling. If set to float \"\n \"strictly between 0 and 1, a token is only considered if it is greater than either eta_cutoff or \"\n \"sqrt(eta_cutoff) * exp(-entropy(softmax(next_token_logits))). The latter term is intuitively the expected next\"\n \" token probability, scaled by sqrt(eta_cutoff). In the paper, suggested values range from 3e-4 to 2e-3, \"\n \"depending on the size of the model.\",\n )\n\n diversity_penalty: float | None = schema_utils.NonNegativeFloat(\n default=0.0,\n allow_none=True,\n description=\"The value used to control the diversity of the generated text. The higher the value, the more \"\n \"diverse the text will be. If set to 0, no diversity is enforced.\"\n \"This value is subtracted from a beam(s) score if it generates a token same as any beam from other group at a\"\n \"particular time. Note that diversity_penalty is only effective if group beam search is enabled.\",\n )\n\n repetition_penalty: float | None = schema_utils.NonNegativeFloat(\n default=1.0,\n allow_none=True,\n description=\"The parameter for repetition penalty. 1.0 means no penalty. \"\n \"See [this paper](https://arxiv.org/pdf/1909.05858.pdf) for more details.\",\n )\n\n encoder_repetition_penalty: float | None = schema_utils.NonNegativeFloat(\n default=1.0,\n allow_none=True,\n description=\"The paramater for encoder_repetition_penalty. An exponential penalty on sequences that are not\"\n \" in the original input. 1.0 means no penalty.\",\n )\n\n length_penalty: float | None = schema_utils.Float(\n default=1.0,\n allow_none=True,\n description=\"Exponential penalty to the length that is used with beam-based generation. It is applied as an \"\n \"exponent to the sequence length, which in turn is used to divide the score of the sequence. Since the score is\"\n \" the log likelihood of the sequence (i.e. negative), length_penalty > 0.0 promotes longer sequences, while \"\n \"length_penalty < 0.0 encourages shorter sequences.\",\n )\n\n no_repeat_ngram_size: int | None = schema_utils.NonNegativeInteger(\n default=0,\n allow_none=True,\n description=\"If set to int > 0, all ngrams of that size can only occur once.\",\n )\n\n bad_words_ids: list[list[int]] | None = schema_utils.List(\n default=None,\n allow_none=True,\n description=\"List of token ids that are not allowed to be generated. In order to get the tokens of the words \"\n \"that should not appear in the generated text, use tokenizer(bad_word, add_prefix_space=True).input_ids.\",\n )\n\n force_words_ids: list[list[int]] | None = schema_utils.List(\n default=None,\n allow_none=True,\n description=\"List of token ids that are forced to be generated by the model. In order to get the tokens of the\"\n \" words that should appear in the generated text, use tokenizer(force_word, add_prefix_space=True).input_ids.\",\n )\n\n renormalize_logits: bool | None = schema_utils.Boolean(\n default=False,\n description=\"Whether to renormalize the logits after temperature and top_k/top_p filtering.\",\n )\n\n # TODO(This needs to be defined based on the Constraint class)\n # constraints:\n\n forced_bos_token_id: int | None = schema_utils.Integer(\n default=None,\n allow_none=True,\n description=\"The id of the token to force as the first generated token after the decoder_start_token_id.\"\n \"Useful for multilingual models like mBART where the first generated token needs to be the target language\"\n \"token.\",\n )\n\n forced_eos_token_id: int | list[int] | None = schema_utils.Integer(\n default=None,\n allow_none=True,\n description=\"The id of the token to force as the last generated token when max_length is reached. Optionally, \"\n \"use a list to set multiple end-of-sequence tokens.\",\n )\n\n remove_invalid_values: bool | None = schema_utils.Boolean(\n default=False,\n description=\"Whether to remove possible nan and inf outputs of the model to prevent the generation method to \"\n \"crash. Note that using remove_invalid_values can slow down generation.\",\n )\n\n exponential_decay_length_penalty: tuple[int, float] | None = schema_utils.FloatRange(\n default=None,\n min=0.0,\n max=1.0,\n allow_none=True,\n description=\"This Tuple adds an exponentially increasing length penalty, after a certain amount of tokens have \"\n \"been generated. The tuple shall consist of: (start_index, decay_factor) where start_index indicates where \"\n \"penalty starts and decay_factor represents the factor of exponential decay\",\n )\n\n suppress_tokens: list[int] | None = schema_utils.List(\n list_type=int,\n default=None,\n allow_none=True,\n description=\"A list of tokens that will be suppressed at generation. The SupressTokens logit processor will set\"\n \" their log probs to -inf so that they are not sampled.\",\n )\n\n begin_suppress_tokens: list[int] | None = schema_utils.List(\n list_type=int,\n default=None,\n allow_none=True,\n description=\"A list of tokens that will be suppressed at the beginning of the generation. The \"\n \"SupressBeginTokens logit processor will set their log probs to -inf so that they are not sampled.\",\n )\n\n forced_decoder_ids: list[list[int]] | None = schema_utils.List(\n default=None,\n allow_none=True,\n description=\"A list of forced decoder ids. The ForcedDecoderIds logit processor will set the log probs of all \"\n \"tokens that are not in the list to -inf so that they are not sampled.\",\n )\n\n sequence_bias: dict[tuple[int], float] | None = schema_utils.Dict(\n default=None,\n allow_none=True,\n description=\"A dictionary of token ids to bias the generation towards. The SequenceBias logit processor will \"\n \"add the bias to the log probs of the tokens in the dictionary. Positive biases increase the odds of the \"\n \"sequence being selected, while negative biases do the opposite. \",\n )\n\n guidance_scale: float | None = schema_utils.FloatRange(\n default=None,\n min=0.0,\n allow_none=True,\n description=\"The guidance scale for classifier free guidance (CFG). CFG is enabled by setting guidance_scale >\"\n \" 1. Higher guidance scale encourages the model to generate samples that are more closely linked to the input\"\n \" prompt, usually at the expense of poorer quality.\",\n )\n\n # Special tokens that can be used at generation time\n\n pad_token_id: int | None = schema_utils.Integer(\n default=None,\n allow_none=True,\n description=\"The id of the padding token. If not set, the padding token id of the tokenizer is used.\",\n )\n\n bos_token_id: int | None = schema_utils.Integer(\n default=None,\n allow_none=True,\n description=\"The id of the beginning of sentence token. If not set, the bos token id of the tokenizer is used.\",\n )\n\n eos_token_id: int | list[int] | None = schema_utils.Integer(\n default=None,\n allow_none=True,\n description=\"The id of the end of sentence token. If not set, the eos token id of the tokenizer is used.\",\n )\n\n\n@DeveloperAPI\nclass LLMGenerationConfigField(schema_utils.DictMarshmallowField):\n def __init__(self):\n super().__init__(LLMGenerationConfig)\n\n def _jsonschema_type_mapping(self):\n return schema_utils.unload_jsonschema_from_marshmallow_class(LLMGenerationConfig)\n"
},
{
"path": "ludwig/schema/llms/model_parameters.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass RoPEScalingConfig(schema_utils.BaseMarshmallowConfig):\n \"\"\"Dynamic RoPE-scaling (rotary position embeddings) to extend the context length of LLM like LLaMA, GPT-NeoX,\n or Falcon.\n\n This parameter is a dictionary containing the scaling configuration for the RoPE embeddings. Currently supports\n three scaling strategies: linear and dynamic. Their scaling factor must be an float greater than 1. The expected\n format is {'rope_type': strategy name, 'factor': scaling factor}\n \"\"\"\n\n def __post_init__(self):\n # Both parameters must be set, or none.\n if not self.rope_type:\n raise ConfigValidationError(\n f\"`rope_scaling`'s `rope_type` field must be one of ['linear', 'dynamic'], got {self.rope_type}\"\n )\n\n if not self.factor:\n raise ConfigValidationError(\n f\"When using `rope_scaling`, `factor` must be specified and be > 1. Got {self.factor}.\"\n )\n\n rope_type: str | None = schema_utils.StringOptions(\n options=[\"linear\", \"dynamic\"],\n default=None,\n allow_none=True,\n description=\"Currently supports two strategies: linear and dynamic scaling.\",\n )\n\n factor: float | None = schema_utils.FloatRange(\n default=None,\n allow_none=True,\n min=1.0,\n min_inclusive=False,\n description=\"The scaling factor for RoPE embeddings.\",\n )\n\n\n@DeveloperAPI\nclass RoPEScalingConfigField(schema_utils.DictMarshmallowField):\n def __init__(self):\n super().__init__(RoPEScalingConfig, default_missing=True)\n\n def _jsonschema_type_mapping(self):\n return schema_utils.unload_jsonschema_from_marshmallow_class(RoPEScalingConfig, title=\"rope_scaling\")\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass ModelParametersConfig(schema_utils.BaseMarshmallowConfig):\n rope_scaling: RoPEScalingConfig = RoPEScalingConfigField().get_default_field()\n\n neftune_noise_alpha: int | None = schema_utils.IntegerRange(\n default=0,\n min=0,\n allow_none=True,\n description=\"The alpha parameter for the embedding noise, which controls the amount of noise added to the \"\n \"embeddings. The higher the value, the more noise is added. This is based on the paper NEFTune: Noisy \"\n \"Embeddings Improve Instruction Finetuning. Paper: https://arxiv.org/pdf/2310.05914.pdf. Default: 0.\"\n \"Suggested values: 5, 10\",\n )\n\n def to_dict(self):\n config = {}\n if self.rope_scaling:\n config[\"rope_scaling\"] = self.rope_scaling.to_dict()\n if self.neftune_noise_alpha:\n config[\"neftune_noise_alpha\"] = self.neftune_noise_alpha\n return config\n\n\n@DeveloperAPI\nclass ModelParametersConfigField(schema_utils.DictMarshmallowField):\n def __init__(self):\n super().__init__(ModelParametersConfig, default_missing=True)\n\n def _jsonschema_type_mapping(self):\n return {\n \"oneOf\": [\n {\"type\": \"null\", \"title\": \"disabled\", \"description\": \"Skip configurable model parameters.\"},\n {\n **schema_utils.unload_jsonschema_from_marshmallow_class(ModelParametersConfig),\n \"title\": \"enabled\",\n \"description\": \"Set model parameters options.\",\n },\n ],\n \"title\": \"Model Parameters\",\n \"description\": \"Configurable model parameters for LLMs.\",\n }\n"
},
{
"path": "ludwig/schema/llms/peft.py",
"content": "from abc import ABC, abstractmethod\nfrom typing import TYPE_CHECKING\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.metadata import LLM_METADATA\nfrom ludwig.schema.metadata.parameter_metadata import convert_metadata_to_json\nfrom ludwig.schema.utils import ludwig_dataclass\nfrom ludwig.utils.registry import Registry\n\nif TYPE_CHECKING:\n from peft import PeftConfig\n\n\nadapter_registry = Registry()\n\n\n@DeveloperAPI\ndef register_adapter(name: str):\n def wrap(config: BaseAdapterConfig):\n adapter_registry[name] = config\n return config\n\n return wrap\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass LoraPostprocessorConfig(schema_utils.BaseMarshmallowConfig):\n \"\"\"This Dataclass is a schema for the nested postprocessing config under adapter of type \"lora\".\"\"\"\n\n merge_adapter_into_base_model: bool = schema_utils.Boolean(\n default=False,\n description=\"\"\"Instructs whether or not the fine-tuned LoRA weights are to be merged into the base LLM model so\nthat the complete fine-tuned model is available to be used and/or persisted, and then reused upon loading as a single\nmodel (rather than having to load base and fine-tuned models separately).\"\"\",\n )\n progressbar: bool = schema_utils.Boolean(\n default=False,\n description=\"Instructs whether or not to show a progress bar indicating the unload and merge process.\",\n )\n\n\n@DeveloperAPI\nclass LoraPostprocessorConfigField(schema_utils.DictMarshmallowField):\n def __init__(self):\n super().__init__(LoraPostprocessorConfig)\n\n def _jsonschema_type_mapping(self):\n return schema_utils.unload_jsonschema_from_marshmallow_class(LoraPostprocessorConfig, title=\"LoraPostprocessor\")\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass BaseAdapterConfig(schema_utils.BaseMarshmallowConfig, ABC):\n type: str\n\n pretrained_adapter_weights: str | None = schema_utils.String(\n default=None, description=\"Path to pretrained weights.\", allow_none=True\n )\n\n postprocessor: LoraPostprocessorConfig = LoraPostprocessorConfigField().get_default_field()\n\n @abstractmethod\n def to_config(self, **kwargs) -> \"PeftConfig\":\n pass\n\n\n@DeveloperAPI\n@register_adapter(name=\"lora\")\n@ludwig_dataclass\nclass LoraConfig(BaseAdapterConfig):\n def __post_init__(self):\n if self.alpha is None:\n self.alpha = self.r * 2\n\n type: str = schema_utils.ProtectedString(\n \"lora\",\n description=LLM_METADATA[\"adapter\"][\"lora\"][\"type\"].long_description,\n )\n\n r: int = schema_utils.PositiveInteger(\n default=8,\n description=\"Lora attention dimension.\",\n parameter_metadata=LLM_METADATA[\"adapter\"][\"lora\"][\"r\"],\n )\n\n alpha: int | None = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"The alpha parameter for Lora scaling. Defaults to `2 * r`.\",\n parameter_metadata=LLM_METADATA[\"adapter\"][\"lora\"][\"alpha\"],\n )\n\n dropout: float = schema_utils.NonNegativeFloat(\n default=0.05,\n description=\"The dropout probability for Lora layers.\",\n parameter_metadata=LLM_METADATA[\"adapter\"][\"lora\"][\"dropout\"],\n )\n\n # TODO(travis): figure out why calling this `bias` doesn't work\n bias_type: str = schema_utils.StringOptions(\n options=[\"none\", \"all\", \"lora_only\"],\n default=\"none\",\n description=\"Bias type for Lora.\",\n )\n\n target_modules: list[str] | None = schema_utils.List(\n default=None,\n allow_none=True,\n description=(\n \"List of module names or regex expression of the module names to replace with LoRA. \"\n \"For example, ['q', 'v'] or '.*decoder.*(SelfAttention|EncDecAttention).*(q|v)$'. \"\n \"Defaults to targeting the query and value matrices of all self-attention and encoder-decoder attention \"\n \"layers.\"\n ),\n parameter_metadata=LLM_METADATA[\"adapter\"][\"lora\"][\"target_modules\"],\n )\n\n use_rslora: bool = schema_utils.Boolean(\n default=False,\n description=(\n \"When set to True, uses Rank-Stabilized LoRA which sets the adapter scaling factor to \"\n \"lora_alpha/math.sqrt(r), since it was proven to work better. Otherwise, it will use the original \"\n \"default value of lora_alpha/r. Paper: https://arxiv.org/abs/2312.03732.\"\n ),\n parameter_metadata=LLM_METADATA[\"adapter\"][\"lora\"][\"use_rslora\"],\n )\n\n use_dora: bool = schema_utils.Boolean(\n default=False,\n description=(\n \"Enable 'Weight-Decomposed Low-Rank Adaptation' (DoRA). This technique decomposes the updates of the \"\n \"weights into two parts, magnitude and direction. Direction is handled by normal LoRA, whereas the \"\n \"magnitude is handled by a separate learnable parameter. This can improve the performance of LoRA, \"\n \"especially at low ranks. Right now, DoRA only supports non-quantized linear layers. DoRA introduces a \"\n \"bigger overhead than pure LoRA, so it is recommended to merge weights for inference. For more \"\n \"information, see https://arxiv.org/abs/2402.09353\"\n ),\n parameter_metadata=LLM_METADATA[\"adapter\"][\"lora\"][\"use_dora\"],\n )\n\n def to_config(self, task_type: str = None, **kwargs) -> \"PeftConfig\":\n from peft import LoraConfig as _LoraConfig\n\n return _LoraConfig(\n r=self.r,\n lora_alpha=self.alpha,\n lora_dropout=self.dropout,\n bias=self.bias_type,\n target_modules=self.target_modules,\n task_type=task_type,\n use_rslora=self.use_rslora,\n use_dora=self.use_dora,\n )\n\n @classmethod\n def name(cls) -> str:\n return \"LoRA\"\n\n @classmethod\n def description(cls) -> str:\n return LLM_METADATA[\"adapter\"][\"lora\"][\"type\"].long_description\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass BasePromptLearningConfig(BaseAdapterConfig):\n \"\"\"Config for prompt learning adapters. Not meant to be used directly.\n\n Adapted from https://github.com/huggingface/peft/blob/main/src/peft/utils/config.py (PromptLearningConfig)\n \"\"\"\n\n num_virtual_tokens: int = schema_utils.PositiveInteger(\n default=8,\n description=\"Number of virtual tokens to add to the prompt. Virtual tokens are used to control the behavior of \"\n \" the model during inference. \",\n parameter_metadata=LLM_METADATA[\"adapter\"][\"prompt_learning\"][\"num_virtual_tokens\"],\n )\n\n token_dim: int | None = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"The hidden embedding dimension of the base transformer model.\",\n )\n\n num_transformer_submodules: int | None = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"The number of transformer submodules in the base transformer model.\",\n )\n\n num_attention_heads: int | None = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"The number of attention heads in the base transformer model.\",\n )\n\n num_layers: int | None = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"The number of layers in the base transformer model.\",\n )\n\n\n# TODO(travis): fix text generation when using prompt tuning:\n# RuntimeError: shape '[-1, 17]' is invalid for input of size 9\n# @DeveloperAPI\n# @register_adapter(\"prompt_tuning\")\n# @ludwig_dataclass\n# class PromptTuningConfig(BasePromptLearningConfig):\n# \"\"\"Adapted from https://github.com/huggingface/peft/blob/main/src/peft/tuners/prompt_tuning.py.\"\"\"\n\n# def __post_init__(self):\n# if self.prompt_tuning_init == \"TEXT\" and not self.prompt_tuning_init_text:\n# raise ConfigValidationError(\n# \"Must provide `prompt_tuning_init_text` when `prompt_tuning_init` is set to `TEXT`.\"\n# )\n\n\"\"\"# type: str = schema_utils.ProtectedString(\"prompt_tuning\")\"\"\" # Quotes allow mypy to run without syntax errors.\n\n# prompt_tuning_init: str = schema_utils.StringOptions(\n# [\"RANDOM\", \"TEXT\"],\n# default=\"RANDOM\",\n# description=\"The type of initialization to use for the prompt embedding. \",\n# parameter_metadata=LLM_METADATA[\"adapter\"][\"prompt_tuning\"][\"prompt_tuning_init\"],\n# )\n\n# prompt_tuning_init_text: str = schema_utils.String(\n# default=\"\",\n# description=\"The text to use to initialize the prompt embedding.\",\n# parameter_metadata=LLM_METADATA[\"adapter\"][\"prompt_tuning\"][\"prompt_tuning_init_text\"],\n# )\n\n# def to_config(self, **kwargs) -> \"PeftConfig\":\n# from peft import PromptTuningConfig as _PromptTuningConfig\n\n# return _PromptTuningConfig(\n# num_virtual_tokens=self.num_virtual_tokens,\n# token_dim=self.token_dim,\n# num_transformer_submodules=self.num_transformer_submodules,\n# num_attention_heads=self.num_attention_heads,\n# num_layers=self.num_layers,\n# prompt_tuning_init=self.prompt_tuning_init,\n# prompt_tuning_init_text=self.prompt_tuning_init_text,\n# **kwargs\n# )\n\n\n# TODO(travis): fix prefix tuning and p-tuning to work with DDP\n# @DeveloperAPI\n# @register_adapter(\"prefix_tuning\")\n# @schema_utils.ludwig_dataclass\n# class PrefixTuningConfig(BasePromptLearningConfig):\n# \"\"\"Adapted from https://github.com/huggingface/peft/blob/main/src/peft/tuners/prefix_tuning.py.\"\"\"\n\n\"\"\"# type: str = schema_utils.ProtectedString(\"prefix_tuning\")\"\"\" # Quotes allow mypy to run without syntax errors.\n\n# encoder_hidden_size: Optional[int] = schema_utils.Integer(\n# default=None,\n# allow_none=True,\n# description=\"The hidden embedding dimension of the prompt encoder.\",\n# )\n\n# prefix_projection: bool = schema_utils.Boolean(\n# default=False,\n# description=\"Whether to use a projection layer in the prompt encoder to project the prefix tokens\",\n# )\n\n# def to_config(self, task_type: str = None, **kwargs) -> \"PeftConfig\":\n# from peft import PrefixTuningConfig as _PrefixTuningConfig\n\n# return _PrefixTuningConfig(\n# num_virtual_tokens=self.num_virtual_tokens,\n# token_dim=self.token_dim,\n# num_transformer_submodules=self.num_transformer_submodules,\n# num_attention_heads=self.num_attention_heads,\n# num_layers=self.num_layers,\n# encoder_hidden_size=self.encoder_hidden_size,\n# prefix_projection=self.prefix_projection,\n# task_type=task_type,\n# )\n\n\n# @DeveloperAPI\n# @register_adapter(\"p_tuning\")\n# @ludwig_dataclass\n# class PTuningConfig(BasePromptLearningConfig):\n\"\"\"# type: str = schema_utils.ProtectedString(\"p_tuning\")\"\"\" # Quotes allow mypy to run without syntax errors.\n\n# encoder_reparameterization_type: str = schema_utils.StringOptions(\n# [\"MLP\", \"LSTM\"],\n# default=\"MLP\",\n# allow_none=False,\n# description=\"The type of reparameterization to use for the prompt encoder.\",\n# )\n\n# encoder_hidden_size: Optional[int] = schema_utils.PositiveInteger(\n# default=None,\n# allow_none=True,\n# description=\"The hidden embedding dimension of the prompt encoder.\",\n# )\n\n# encoder_num_layers: Optional[int] = schema_utils.PositiveInteger(\n# default=2,\n# allow_none=True,\n# description=\"The number of layers in the prompt encoder.\",\n# )\n\n# encoder_dropout: Optional[float] = schema_utils.FloatRange(\n# default=0.0,\n# min=0.0,\n# max=1.0,\n# description=\"The dropout probability for the prompt encoder.\",\n# )\n\n# def to_config(self, task_type: str = None, **kwargs) -> \"PeftConfig\":\n# from peft import PromptEncoderConfig as _PromptEncoderConfig\n\n# return _PromptEncoderConfig(\n# num_virtual_tokens=self.num_virtual_tokens,\n# token_dim=self.token_dim,\n# num_transformer_submodules=self.num_transformer_submodules,\n# num_attention_heads=self.num_attention_heads,\n# num_layers=self.num_layers,\n# encoder_reparameterization_type=self.encoder_reparameterization_type,\n# encoder_hidden_size=self.encoder_hidden_size,\n# encoder_num_layers=self.encoder_num_layers,\n# encoder_dropout=self.encoder_dropout,\n# task_type=task_type,\n# )\n\n\n@DeveloperAPI\n@register_adapter(\"adalora\")\n@ludwig_dataclass\nclass AdaloraConfig(LoraConfig):\n \"\"\"Adapted from https://github.com/huggingface/peft/blob/main/src/peft/tuners/adalora.py.\"\"\"\n\n type: str = schema_utils.ProtectedString(\n \"adalora\",\n description=LLM_METADATA[\"adapter\"][\"adalora\"][\"type\"].long_description,\n )\n\n target_r: int = schema_utils.PositiveInteger(\n default=8,\n description=\"Target Lora Matrix Dimension. The target average rank of incremental matrix.\",\n )\n\n init_r: int = schema_utils.PositiveInteger(\n default=12,\n description=\"Initial Lora Matrix Dimension. The initial rank for each incremental matrix.\",\n )\n\n tinit: int = schema_utils.NonNegativeInteger(\n default=0,\n description=\"The steps of initial fine-tuning warmup.\",\n )\n\n tfinal: int = schema_utils.NonNegativeInteger(\n default=0,\n description=\"The steps of final fine-tuning warmup.\",\n )\n\n delta_t: int = schema_utils.NonNegativeInteger(\n default=1,\n description=\"The time internval between two budget allocations. The step interval of rank allocation.\",\n )\n\n beta1: float = schema_utils.FloatRange(\n default=0.85,\n min=0.0,\n max=1.0,\n description=\"The hyperparameter of EMA for sensitivity smoothing.\",\n )\n\n beta2: float = schema_utils.FloatRange(\n default=0.85,\n min=0.0,\n max=1.0,\n description=\" The hyperparameter of EMA for undertainty quantification.\",\n )\n\n orth_reg_weight: float = schema_utils.FloatRange(\n default=0.5,\n min=0.0,\n max=1.0,\n description=\"The coefficient of orthogonality regularization.\",\n )\n\n total_step: int = schema_utils.PositiveInteger(\n default=10000,\n allow_none=False,\n description=\"The total training steps for AdaLoRA rank allocation scheduling. \"\n \"Must be a positive integer (required by peft >= 0.14).\",\n )\n\n rank_pattern: dict | None = schema_utils.Dict(\n default=None,\n allow_none=True,\n description=\"The allocated rank for each weight matrix by RankAllocator.\",\n )\n\n def to_config(self, **kwargs) -> \"PeftConfig\":\n from peft import AdaLoraConfig as _AdaLoraConfig\n\n return _AdaLoraConfig(\n r=self.r,\n lora_alpha=self.alpha,\n lora_dropout=self.dropout,\n bias=self.bias_type,\n target_r=self.target_r,\n init_r=self.init_r,\n tinit=self.tinit,\n tfinal=self.tfinal,\n deltaT=self.delta_t,\n beta1=self.beta1,\n beta2=self.beta2,\n orth_reg_weight=self.orth_reg_weight,\n total_step=self.total_step,\n rank_pattern=self.rank_pattern,\n )\n\n @classmethod\n def name(cls) -> str:\n return \"AdaLoRA\"\n\n @classmethod\n def description(cls) -> str:\n return LLM_METADATA[\"adapter\"][\"adalora\"][\"type\"].long_description\n\n\n@DeveloperAPI\n# TODO: 02/21/2024: Disabling AdaptionPrompt (waiting for PEFT release to fix\n# \"TypeError: LlamaRotaryEmbedding.forward() missing 1 required positional argument: 'position_ids')\"\n# (this is reflected in https://github.com/ludwig-ai/ludwig/issues/3938).\n# \n# @register_adapter(\"adaption_prompt\")\n@ludwig_dataclass\nclass AdaptionPromptConfig(BaseAdapterConfig):\n \"\"\"Adapted from https://github.com/huggingface/peft/blob/main/src/peft/tuners/adaption_prompt/config.py.\"\"\"\n\n def __post_init__(self):\n if not self.adapter_len:\n raise ConfigValidationError(\n \"`adapter_len` must be set to a value greater than 0 when finetuning is enabled and the adapter\"\n \"type is `adaption_prompt`. This is the length of the adaption prompt to insert.\"\n )\n\n if not self.adapter_layers:\n raise ConfigValidationError(\n \"`adapter_layers` must be set to a value greater than 0 when finetuning is enabled and the adapter\"\n \"type is `adaption_prompt`. This is the number of adapter layers to insert.\"\n )\n\n type: str = schema_utils.ProtectedString(\n \"adaption_prompt\",\n description=LLM_METADATA[\"adapter\"][\"adaption_prompt\"][\"type\"].long_description,\n )\n\n adapter_len: int = schema_utils.PositiveInteger(\n default=4,\n description=\"Number of adapter tokens to insert.\",\n parameter_metadata=LLM_METADATA[\"adapter\"][\"adaption_prompt\"][\"adapter_len\"],\n )\n\n adapter_layers: int = schema_utils.PositiveInteger(\n default=1,\n allow_none=False,\n description=\"Number of adapter layers to insert (from the top).\",\n parameter_metadata=LLM_METADATA[\"adapter\"][\"adaption_prompt\"][\"adapter_layers\"],\n )\n\n def to_config(self, task_type: str = None, **kwargs) -> \"PeftConfig\":\n from peft import AdaptionPromptConfig as _AdaptionPromptConfig\n\n return _AdaptionPromptConfig(\n adapter_len=self.adapter_len,\n adapter_layers=self.adapter_layers,\n task_type=task_type,\n )\n\n @classmethod\n def name(cls) -> str:\n return \"Adaption Prompt\"\n\n @classmethod\n def description(cls) -> str:\n return LLM_METADATA[\"adapter\"][\"adaption_prompt\"][\"type\"].long_description\n\n\n@DeveloperAPI\n@register_adapter(\"ia3\")\n@ludwig_dataclass\nclass IA3Config(BaseAdapterConfig):\n type: str = schema_utils.ProtectedString(\n \"ia3\",\n description=LLM_METADATA[\"adapter\"][\"ia3\"][\"type\"].long_description,\n )\n\n target_modules: list[str] | None = schema_utils.List(\n default=None,\n allow_none=True,\n description=\"The names of the modules to apply (IA)^3 to.\",\n parameter_metadata=LLM_METADATA[\"adapter\"][\"ia3\"][\"target_modules\"],\n )\n\n feedforward_modules: list[str] | None = schema_utils.List(\n default=None,\n allow_none=True,\n description=(\n \"The names of the modules to be treated as feedforward modules, as in the original paper. These modules \"\n \"will have (IA)^3 vectors multiplied to the input, instead of the output. feedforward_modules must be a \"\n \"name or a subset of names present in target_modules.\"\n ),\n parameter_metadata=LLM_METADATA[\"adapter\"][\"ia3\"][\"feedforward_modules\"],\n )\n\n fan_in_fan_out: bool = schema_utils.Boolean(\n default=False,\n description=(\n \"Set this to True if the layer to replace stores weight like (fan_in, fan_out). For example, gpt-2 uses \"\n \"Conv1D which stores weights like (fan_in, fan_out) and hence this should be set to True. \"\n ),\n parameter_metadata=LLM_METADATA[\"adapter\"][\"ia3\"][\"fan_in_fan_out\"],\n )\n\n modules_to_save: list[str] | None = schema_utils.List(\n list_type=str,\n default=None,\n allow_none=True,\n description=(\n \"List of modules apart from (IA)^3 layers to be set as trainable and saved in the final checkpoint.\"\n ),\n parameter_metadata=LLM_METADATA[\"adapter\"][\"ia3\"][\"modules_to_save\"],\n )\n\n init_ia3_weights: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to initialize the vectors in the (IA)^3 layers, defaults to True.\",\n parameter_metadata=LLM_METADATA[\"adapter\"][\"ia3\"][\"init_ia3_weights\"],\n )\n\n def to_config(self, task_type: str = None, **kwargs) -> \"PeftConfig\":\n from peft import IA3Config as _IA3Config\n\n return _IA3Config(\n target_modules=self.target_modules,\n feedforward_modules=self.feedforward_modules,\n fan_in_fan_out=self.fan_in_fan_out,\n modules_to_save=self.modules_to_save,\n init_ia3_weights=self.init_ia3_weights,\n task_type=task_type,\n )\n\n @classmethod\n def name(cls) -> str:\n return \"IA3\"\n\n @classmethod\n def description(cls) -> str:\n return LLM_METADATA[\"adapter\"][\"ia3\"][\"type\"].long_description\n\n\n@DeveloperAPI\ndef get_adapter_conds():\n conds = []\n for adapter_type, adapter_cls in adapter_registry.items():\n other_props = schema_utils.unload_jsonschema_from_marshmallow_class(adapter_cls)[\"properties\"]\n schema_utils.remove_duplicate_fields(other_props)\n preproc_cond = schema_utils.create_cond(\n {\"type\": adapter_type},\n other_props,\n )\n conds.append(preproc_cond)\n return conds\n\n\n@DeveloperAPI\ndef AdapterDataclassField(default: str | None = None):\n description = \"Whether to use parameter-efficient fine-tuning\"\n\n class AdapterSelection(schema_utils.TypeSelection):\n def __init__(self):\n super().__init__(\n registry=adapter_registry,\n default_value=default,\n description=description,\n parameter_metadata=None,\n allow_str_value=True,\n allow_none=True,\n )\n\n def get_schema_from_registry(self, key: str) -> type[schema_utils.BaseMarshmallowConfig]:\n return adapter_registry[key]\n\n @staticmethod\n def _jsonschema_type_mapping():\n return {\n \"oneOf\": [\n {\n \"type\": \"object\",\n \"properties\": {\n \"type\": {\n \"type\": \"string\",\n \"enum\": list(adapter_registry.keys()),\n \"description\": \"The type of PEFT adapter to use during fine-tuning\",\n },\n },\n \"title\": \"Perform parameter efficient fine-tuning\",\n \"allOf\": get_adapter_conds(),\n \"required\": [\"type\"],\n \"description\": \"The type of PEFT adapter to use during fine-tuning\",\n \"parameter_metadata\": convert_metadata_to_json(LLM_METADATA[\"adapter\"][\"_oneOf\"][\"allOf\"]),\n },\n {\n \"type\": \"null\",\n \"title\": \"adapter_null_option\",\n \"description\": \"Disable the adapter.\",\n \"parameter_metadata\": convert_metadata_to_json(LLM_METADATA[\"adapter\"][\"_oneOf\"][\"none\"]),\n },\n ],\n \"title\": \"adapter_options\",\n \"description\": \"Whether to use parameter-efficient fine-tuning\",\n \"parameter_metadata\": convert_metadata_to_json(LLM_METADATA[\"adapter\"][\"_meta\"]),\n \"default\": default,\n }\n\n return AdapterSelection().get_default_field()\n"
},
{
"path": "ludwig/schema/llms/prompt.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import SEMANTIC\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.metadata import LLM_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass RetrievalConfig(schema_utils.BaseMarshmallowConfig):\n \"\"\"This Dataclass is a schema for the nested retrieval config under prompt.\"\"\"\n\n def __post_init__(self):\n # TODO: have a dynamically loaded schema based on the selection of the type param\n # https://github.com/ludwig-ai/ludwig/pull/3351#discussion_r1181910954\n # Ensure k is non-zero if we're using a retrieval strategy\n if self.type is not None and self.k == 0:\n self.k = 1\n\n if self.type is None and self.k != 0:\n raise ConfigValidationError(\"k must be 0 if retrieval type is None.\")\n elif self.type is not None and self.k <= 0:\n raise ConfigValidationError(\"k must be greater than 0 if retrieval type is not None.\")\n\n if self.type is None and self.model_name is not None:\n raise ConfigValidationError(\"model_name must be None if retrieval type is None.\")\n elif self.type == SEMANTIC and self.model_name is None:\n raise ConfigValidationError(f\"model_name must not be None if retrieval type is '{SEMANTIC}'.\")\n\n type: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=(\n \"The type of retrieval to use for the prompt. If `None`, then no retrieval is used, and the task \"\n \"is framed as a zero-shot learning problem. If not `None` (e.g. either 'random' or 'semantic'), then \"\n \"samples are retrieved from an index of the training set and used to augment the input to the model \"\n \"in a few-shot learning setting.\"\n ),\n parameter_metadata=LLM_METADATA[\"prompt\"][\"retrieval\"][\"type\"],\n )\n\n index_name: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"The name of the index to use for the prompt. Indices are stored in the ludwig cache by default.\",\n parameter_metadata=LLM_METADATA[\"prompt\"][\"retrieval\"][\"index_name\"],\n )\n\n model_name: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"The model used to generate the embeddings used to retrieve samples to inject in the prompt.\",\n parameter_metadata=LLM_METADATA[\"prompt\"][\"retrieval\"][\"model_name\"],\n )\n\n k: int = schema_utils.NonNegativeInteger(\n default=0,\n description=\"The number of samples to retrieve.\",\n parameter_metadata=LLM_METADATA[\"prompt\"][\"retrieval\"][\"k\"],\n )\n\n\n@DeveloperAPI\nclass RetrievalConfigField(schema_utils.DictMarshmallowField):\n def __init__(self):\n super().__init__(RetrievalConfig)\n\n def _jsonschema_type_mapping(self):\n return schema_utils.unload_jsonschema_from_marshmallow_class(RetrievalConfig, title=\"Retrieval\")\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass PromptConfig(schema_utils.BaseMarshmallowConfig):\n \"\"\"This Dataclass is a schema for the nested prompt config under preprocessing.\"\"\"\n\n template: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=(\n \"The template to use for the prompt. Must contain at least one of the columns from the input dataset \"\n \"or `__sample__` as a variable surrounded in curly brackets {} to indicate where to insert the \"\n \"current feature. Multiple columns can be inserted, e.g.: `The {color} {animal} jumped over \"\n \"the {size} {object}`, where every term in curly brackets is a column in the dataset. If a `task` \"\n \"is specified, then the template must also contain the `__task__` variable. If `retrieval` is specified, \"\n \"then the template must also contain the `__context__` variable. If no template is provided, then a \"\n \"default will be used based on the retrieval settings, and a task must be set in the config.\"\n ),\n parameter_metadata=LLM_METADATA[\"prompt\"][\"template\"],\n )\n\n task: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"The task to use for the prompt. Required if `template` is not set.\",\n parameter_metadata=LLM_METADATA[\"prompt\"][\"task\"],\n )\n\n retrieval: RetrievalConfig = RetrievalConfigField().get_default_field()\n\n\n@DeveloperAPI\nclass PromptConfigField(schema_utils.DictMarshmallowField):\n def __init__(self):\n super().__init__(PromptConfig)\n\n def _jsonschema_type_mapping(self):\n return schema_utils.unload_jsonschema_from_marshmallow_class(PromptConfig)\n"
},
{
"path": "ludwig/schema/llms/quantization.py",
"content": "import warnings\n\nfrom transformers import BitsAndBytesConfig\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.metadata import LLM_METADATA\nfrom ludwig.schema.metadata.parameter_metadata import convert_metadata_to_json\nfrom ludwig.schema.utils import ludwig_dataclass\n\nwarnings.filterwarnings(\n action=\"ignore\",\n category=UserWarning,\n module=\"bitsandbytes.cuda_setup.main\",\n)\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass QuantizationConfig(schema_utils.BaseMarshmallowConfig):\n bits: int = schema_utils.IntegerOptions(\n options=[4, 8],\n default=4,\n description=\"The quantization level to apply to weights on load.\",\n parameter_metadata=LLM_METADATA[\"quantization\"][\"bits\"],\n )\n\n llm_int8_threshold: float = schema_utils.NonNegativeFloat(\n default=6.0,\n description=(\n \"This corresponds to the outlier threshold for outlier detection as described in `LLM.int8() : 8-bit \"\n \"Matrix Multiplication for Transformers at Scale` paper: https://arxiv.org/abs/2208.07339. Any hidden \"\n \"states value that is above this threshold will be considered an outlier and the operation on those \"\n \"values will be done in fp16. Values are usually normally distributed, that is, most values are in the \"\n \"range [-3.5, 3.5], but there are some exceptional systematic outliers that are very differently \"\n \"distributed for large models. These outliers are often in the interval [-60, -6] or [6, 60]. Int8 \"\n \"quantization works well for values of magnitude ~5, but beyond that, there is a significant performance \"\n \"penalty. A good default threshold is 6, but a lower threshold might be needed for more unstable models \"\n \"(small models, fine-tuning).\"\n ),\n )\n\n llm_int8_has_fp16_weight: bool = schema_utils.Boolean(\n default=False,\n description=(\n \"This flag runs LLM.int8() with 16-bit main weights. This is useful for fine-tuning as the weights do \"\n \"not have to be converted back and forth for the backward pass.\"\n ),\n )\n\n bnb_4bit_compute_dtype: str = schema_utils.StringOptions(\n options=[\"float32\", \"float16\", \"bfloat16\"],\n default=\"float16\",\n description=(\n \"This sets the computational type which might be different than the input type. For example, inputs \"\n \"might be fp32, but computation can be set to bf16 for speedups.\"\n ),\n )\n\n bnb_4bit_use_double_quant: bool = schema_utils.Boolean(\n default=True,\n description=(\n \"This flag is used for nested quantization where the quantization constants from the first quantization \"\n \"are quantized again.\"\n ),\n )\n\n bnb_4bit_quant_type: str = schema_utils.StringOptions(\n options=[\"fp4\", \"nf4\"],\n default=\"nf4\",\n description=\"This sets the quantization data type in the bnb.nn.Linear4Bit layers.\",\n )\n\n def to_bitsandbytes(self) -> BitsAndBytesConfig:\n return BitsAndBytesConfig(\n load_in_4bit=self.bits == 4,\n load_in_8bit=self.bits == 8,\n llm_int8_threshold=self.llm_int8_threshold,\n llm_int8_has_fp16_weight=self.llm_int8_has_fp16_weight,\n bnb_4bit_compute_dtype=self.bnb_4bit_compute_dtype,\n bnb_4bit_use_double_quant=self.bnb_4bit_use_double_quant,\n bnb_4bit_quant_type=self.bnb_4bit_quant_type,\n )\n\n\n@DeveloperAPI\nclass QuantizationConfigField(schema_utils.DictMarshmallowField):\n def __init__(self):\n super().__init__(QuantizationConfig, default_missing=True)\n\n def _jsonschema_type_mapping(self):\n return {\n \"oneOf\": [\n {\n \"type\": \"null\",\n \"title\": \"disabled\",\n \"description\": \"Disable quantization.\",\n \"parameter_metadata\": convert_metadata_to_json(LLM_METADATA[\"quantization\"][\"_oneOf\"][\"none\"]),\n },\n {\n **schema_utils.unload_jsonschema_from_marshmallow_class(QuantizationConfig),\n \"title\": \"enabled\",\n \"description\": \"Set quantization options.\",\n \"parameter_metadata\": convert_metadata_to_json(LLM_METADATA[\"quantization\"][\"_oneOf\"][\"object\"]),\n },\n ],\n \"title\": \"quantization\",\n \"description\": \"Set quantization options.\",\n \"parameter_metadata\": convert_metadata_to_json(LLM_METADATA[\"quantization\"][\"_meta\"]),\n }\n"
},
{
"path": "ludwig/schema/lr_scheduler.py",
"content": "from abc import ABC\nfrom dataclasses import field\n\nimport ludwig.schema.utils as schema_utils\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import LOSS, MODEL_ECD, TRAINING\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.schema.metadata import TRAINER_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass LRSchedulerConfig(schema_utils.BaseMarshmallowConfig, ABC):\n \"\"\"Configuration for learning rate scheduler parameters.\"\"\"\n\n decay: str = schema_utils.StringOptions(\n options=[\"linear\", \"exponential\", \"cosine\"],\n default=None,\n allow_none=True,\n description=\"Turn on decay of the learning rate.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"learning_rate_scheduler\"][\"decay\"],\n )\n\n decay_rate: float = schema_utils.FloatRange(\n default=0.96,\n min=0,\n max=1,\n description=\"Decay per epoch (%): Factor to decrease the Learning rate.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"learning_rate_scheduler\"][\"decay_rate\"],\n )\n\n decay_steps: int = schema_utils.PositiveInteger(\n default=10000,\n description=\"The number of steps to take in the exponential learning rate decay.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"learning_rate_scheduler\"][\"decay_steps\"],\n )\n\n staircase: bool = schema_utils.Boolean(\n default=False,\n description=\"Decays the learning rate at discrete intervals.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"learning_rate_scheduler\"][\"staircase\"],\n )\n\n reduce_on_plateau: int = schema_utils.NonNegativeInteger(\n default=0,\n description=(\n \"How many times to reduce the learning rate when the algorithm hits a plateau (i.e. the performance on the \"\n \"training set does not improve)\"\n ),\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"learning_rate_scheduler\"][\"reduce_on_plateau\"],\n )\n\n reduce_on_plateau_patience: int = schema_utils.NonNegativeInteger(\n default=10,\n description=(\n \"How many evaluation steps have to pass before the learning rate reduces \" \"when `reduce_on_plateau > 0`.\"\n ),\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"learning_rate_scheduler\"][\"reduce_on_plateau_patience\"],\n )\n\n reduce_on_plateau_rate: float = schema_utils.FloatRange(\n default=0.1,\n min=0,\n max=1,\n description=\"Rate at which we reduce the learning rate when `reduce_on_plateau > 0`.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"learning_rate_scheduler\"][\"reduce_on_plateau_rate\"],\n )\n\n warmup_evaluations: int = schema_utils.NonNegativeFloat(\n default=0,\n description=\"Number of evaluation steps to warmup the learning rate for.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"learning_rate_scheduler\"][\"warmup_evaluations\"],\n )\n\n warmup_fraction: float = schema_utils.NonNegativeFloat(\n default=0.0,\n description=\"Fraction of total training steps to warmup the learning rate for.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"learning_rate_scheduler\"][\"warmup_fraction\"],\n )\n\n reduce_eval_metric: str = schema_utils.String(\n default=LOSS,\n allow_none=False,\n description=(\n \"Metric plateau used to trigger when we reduce the learning rate \" \"when `reduce_on_plateau > 0`.\"\n ),\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"learning_rate_scheduler\"][\"reduce_eval_metric\"],\n )\n\n reduce_eval_split: str = schema_utils.String(\n default=TRAINING,\n allow_none=False,\n description=(\n \"Which dataset split to listen on for reducing the learning rate \" \"when `reduce_on_plateau > 0`.\"\n ),\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"learning_rate_scheduler\"][\"reduce_eval_split\"],\n )\n\n # Parameters for CosineAnnealingWarmRestarts scheduler\n\n t_0: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Number of steps before the first restart for cosine annealing decay. If not specified, it\"\n \" will be set to `steps_per_checkpoint`.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"learning_rate_scheduler\"][\"t_0\"],\n )\n\n t_mult: int = schema_utils.PositiveInteger(\n default=1,\n description=\"Period multiplier after each restart for cosine annealing decay. Defaults to 1, i.e.,\"\n \" restart every `t_0` steps. If set to a larger value, the period between restarts increases by that\"\n \" multiplier. For e.g., if t_mult is 2, then the periods would be: t_0, 2*t_0, 2^2*t_0, 2^3*t_0, etc.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"learning_rate_scheduler\"][\"t_mult\"],\n )\n\n eta_min: float = schema_utils.FloatRange(\n default=0,\n min=0,\n max=1,\n description=\"Minimum learning rate allowed for cosine annealing decay. Default: 0.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"learning_rate_scheduler\"][\"eta_min\"],\n )\n\n\n# TODO(travis): too much boilerplate here, we should find a way to abstract all this and only require specifying the\n# minimal amount needed for the new config object.\n@DeveloperAPI\ndef LRSchedulerDataclassField(description: str, default: dict = None):\n \"\"\"Returns custom dataclass field for `LRSchedulerConfig`. Allows `None` by default.\n\n Args:\n description: Description of the dataclass field\n default: dict that specifies param values that will be loaded by its schema class (default: None).\n \"\"\"\n allow_none = True\n default = default or {}\n\n class LRSchedulerMarshmallowField(schema_utils.LudwigSchemaField):\n \"\"\"Custom field class for learning rate scheduler.\n\n Deserializes a dict to a valid instance of `LRSchedulerConfig` and creates a corresponding JSON schema for\n external usage.\n \"\"\"\n\n def _deserialize(self, value, attr, data, **kwargs):\n if value is None:\n return value\n if isinstance(value, dict):\n try:\n return LRSchedulerConfig.Schema().load(value)\n except (TypeError, ConfigValidationError) as e:\n raise ConfigValidationError(\n f\"Invalid params for learning rate scheduler: {value}, see LRSchedulerConfig class. Error: {e}\"\n )\n raise ConfigValidationError(\"Field should be None or dict\")\n\n def _jsonschema_type_mapping(self):\n return {\n **schema_utils.unload_jsonschema_from_marshmallow_class(LRSchedulerConfig),\n \"title\": \"learning_rate_scheduler_options\",\n \"description\": description,\n }\n\n if not isinstance(default, dict):\n raise ConfigValidationError(f\"Invalid default: `{default}`\")\n\n load_default = lambda: LRSchedulerConfig.Schema().load(default)\n dump_default = LRSchedulerConfig.Schema().dump(default)\n\n return field(\n metadata={\n \"marshmallow_field\": LRSchedulerMarshmallowField(\n allow_none=allow_none,\n load_default=load_default,\n dump_default=dump_default,\n metadata={\n \"description\": description,\n },\n )\n },\n default_factory=load_default,\n )\n"
},
{
"path": "ludwig/schema/metadata/__init__.py",
"content": "import os\nfrom typing import Any\n\nimport yaml\n\nfrom ludwig.schema.metadata.parameter_metadata import ParameterMetadata\n\n_PATH_HERE = os.path.abspath(os.path.dirname(__file__))\n_CONFIG_DIR = os.path.join(_PATH_HERE, \"configs\")\n\n\ndef _to_metadata(d: dict[str, Any]) -> ParameterMetadata | dict[str, Any]:\n is_nested = False\n for k, v in list(d.items()):\n if isinstance(v, dict):\n d[k] = _to_metadata(v)\n is_nested = True\n\n if is_nested:\n return d\n\n return ParameterMetadata.from_dict(d)\n\n\ndef _load(fname: str) -> dict[str, Any]:\n with open(os.path.join(_CONFIG_DIR, fname)) as f:\n return _to_metadata(yaml.safe_load(f))\n\n\nCOMMON_METADATA = _load(\"common.yaml\")\nCOMBINER_METADATA = _load(\"combiners.yaml\")\nDECODER_METADATA = _load(\"decoders.yaml\")\nENCODER_METADATA = _load(\"encoders.yaml\")\nFEATURE_METADATA = _load(\"features.yaml\")\nPREPROCESSING_METADATA = _load(\"preprocessing.yaml\")\nTRAINER_METADATA = _load(\"trainer.yaml\")\nOPTIMIZER_METADATA = _load(\"optimizers.yaml\")\nLOSS_METADATA = _load(\"loss.yaml\")\nLLM_METADATA = _load(\"llm.yaml\")\n"
},
{
"path": "ludwig/schema/metadata/configs/combiners.yaml",
"content": "comparator:\n type:\n short_description: Used for recommendation problems, features associated with distinct entities, output depends on entity-level comparison.\n long_description:\n The comparator combiner compares the hidden representation of two entities defined by lists of\n features. It assumes all outputs from encoders are tensors of size `b x h` where `b` is the batch\n size and `h` is the hidden dimension, which can be different for each input. If the input tensors\n have a different shape, it automatically flattens them. It then concatenates the representations\n of each entity and projects them both to vectors of size `output_size`. Finally, it compares the\n two entity representations by dot product, element-wise multiplication, absolute difference and\n bilinear product. It returns the final `b x h` tensor where `h` is the size of the concatenation\n of the four comparisons.\n activation:\n default_value_reasoning:\n The Rectified Linear Units (ReLU) function is the\n standard activation function used for adding non-linearity. It is simple,\n fast, and empirically works well (https://arxiv.org/abs/1803.08375).\n description_implications:\n Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n expected_impact: 2\n suggested_values:\n The default value will work well in the majority of the\n cases\n ui_display_name: Activation\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n entity_1:\n literature_references:\n - https://ludwig.ai/0.6/configuration/combiner/#comparator-combiner\n ui_display_name: Entity 1\n expected_impact: 3\n entity_2:\n literature_references:\n - https://ludwig.ai/0.6/configuration/combiner/#comparator-combiner\n ui_display_name: Entity 2\n expected_impact: 3\n fc_layers:\n default_value_reasoning:\n By default the stack is built by using num_fc_layers,\n output_size, use_bias, weights_initializer, bias_initializer, norm, norm_params,\n activation, dropout. When a list of dictionaries is provided, the stack\n is built following the parameters of each dict for building each layer.\n description_implications:\n The more layers that are specified the deeper and\n higher capacity the model will be. This makes it possible to potentially\n achieve better performance when a big anough amount of data is provided,\n but also makes the model more computationally expensive and potentially\n more prone to overfitting.\n example_value:\n - dropout: 0.1\n output_size: 128\n - norm: layer\n output_size: 64\n expected_impact: 1\n related_parameters:\n - output_size\n - use_bias\n - weights_initializer\n - bias_initializer\n - norm\n - norm_params\n - activation\n - dropout\n suggested_values_reasoning:\n It is easier to define a stack of fully connected\n layers by just specifying num_fc_layers, output_size and the other individual\n parameters. It will create a stack of layers with identical properties.\n Use this parameter only if you need a fine grained level of control of\n each individual layer in the stack.\n ui_display_name: Fully Connected Layers\n norm:\n default_value_reasoning:\n While batch normalization and layer normalization\n usually lead to improvements, it can be useful to start with fewer bells\n and whistles.\n description_implications:\n Normalization helps stabilize the learning process\n and can have a regularizing effect that can help with generalization.\n It's often suggested that with normalization, you can use a higher learning\n rate.\n example_value:\n - batch\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/\n related_parameters:\n - norm_params\n suggested_values: '\"batch\" or \"layer\"'\n suggested_values_reasoning:\n Normalization tries to solve \"internal covariate\n shift\" that comes from the changing distributions of the inputs to layers\n deep in the network when weights are updated. For example, batch normalization\n standardizes the inputs to a layer for each mini-batch. Try out different\n normalizations to see if that helps with training stability\n ui_display_name: Normalization Type\n norm_params:\n ui_display_name: null\n expected_impact: 1\n num_fc_layers:\n default_value_reasoning:\n The encoder already has learnable parameters.Sometimes\n the default is 1 for modules where the FC stack is used for shape management,\n or the only source of learnable parameters.\n description_implications:\n Increasing num_fc_layers will increase the capacity\n of the model. The model will be slower to train, and there's a higher\n risk of overfitting.\n example_value:\n - 1\n expected_impact: 3\n other_information:\n Not all modules that have fc_layers also have an accompanying\n num_fc_layers parameter. Where both are present, fc_layers takes precedent\n over num_fc_layers. Specifying num_fc_layers alone uses fully connected\n layers that are configured by the defaults in FCStack.\n related_parameters:\n - fc_layers\n suggested_values: 0-1\n suggested_values_reasoning:\n The full model likely contains many learnable\n parameters. Consider starting with very few, or without any additional\n fully connected layers and add them if you observe evidence of limited\n model capacity. Sometimes the default is 1 for modules where the FC stack\n is used for shape management, or the only source of learnable parameters.\n ui_display_name: Number of Fully Connected Layers\n output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 15 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\n use_bias:\n default_value_reasoning:\n \"Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to use bias terms.\n\n\n Batch Normalization, however, adds a trainable shift parameter which is\n added to the activation. When Batch Normalization is used in a layer,\n bias terms are redundant and may be removed.\"\n description_implications:\n Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to leave this parameter set to True.\n example_value:\n - true\n expected_impact: 1\n other_information:\n If fc_layers is not specified, or use_bias is not specified\n for individual layers, the value of use_bias will be used as the default\n for all layers.\n related_parameters:\n - bias_initializer, fc_layers\n suggested_values: \"TRUE\"\n ui_display_name: Use Bias\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\"\n - \"Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\nconcat:\n type:\n short_description: Concatenates outputs of all encoders and passes concatenated representation through stack of fully connected layers.\n long_description:\n The concat combiner assumes all outputs from encoders are tensors of size `b x h` where `b` is\n the batch size and `h` is the hidden dimension, which can differ for each input. It\n concatenates along the `h` dimension, and then (optionally) passes the concatenated tensor\n through a stack of fully connected layers. It returns the final `b x h` tensor where `h` is the\n size of the last fully connected layer or the sum of the sizes of the `h` of all inputs in the\n case there are no fully connected layers. If there is only a single input feature and no fully\n connected layers, the output of the input feature encoder is passed through the combiner\n unchanged.\n activation:\n default_value_reasoning:\n The Rectified Linear Units (ReLU) function is the\n standard activation function used for adding non-linearity. It is simple,\n fast, and empirically works well (https://arxiv.org/abs/1803.08375).\n description_implications:\n Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n expected_impact: 2\n suggested_values:\n The default value will work well in the majority of the\n cases\n ui_display_name: Activation\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n fc_layers:\n default_value_reasoning:\n By default the stack is built by using num_fc_layers,\n output_size, use_bias, weights_initializer, bias_initializer, norm, norm_params,\n activation, dropout. When a list of dictionaries is provided, the stack\n is built following the parameters of each dict for building each layer.\n description_implications:\n The more layers that are specified the deeper and\n higher capacity the model will be. This makes it possible to potentially\n achieve better performance when a big anough amount of data is provided,\n but also makes the model more computationally expensive and potentially\n more prone to overfitting.\n example_value:\n - dropout: 0.1\n output_size: 128\n - norm: layer\n output_size: 64\n expected_impact: 1\n related_parameters:\n - output_size\n - use_bias\n - weights_initializer\n - bias_initializer\n - norm\n - norm_params\n - activation\n - dropout\n suggested_values_reasoning:\n It is easier to define a stack of fully connected\n layers by just specifying num_fc_layers, output_size and the other individual\n parameters. It will create a stack of layers with identical properties.\n Use this parameter only if you need a fine grained level of control of\n each individual layer in the stack.\n ui_display_name: Fully Connected Layers\n flatten_inputs:\n ui_display_name: null\n expected_impact: 1\n norm:\n default_value_reasoning:\n While batch normalization and layer normalization\n usually lead to improvements, it can be useful to start with fewer bells\n and whistles.\n description_implications:\n Normalization helps stabilize the learning process\n and can have a regularizing effect that can help with generalization.\n It's often suggested that with normalization, you can use a higher learning\n rate.\n example_value:\n - batch\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/\n related_parameters:\n - norm_params\n suggested_values: '\"batch\" or \"layer\"'\n suggested_values_reasoning:\n Normalization tries to solve \"internal covariate\n shift\" that comes from the changing distributions of the inputs to layers\n deep in the network when weights are updated. For example, batch normalization\n standardizes the inputs to a layer for each mini-batch. Try out different\n normalizations to see if that helps with training stability\n ui_display_name: Normalization Type\n norm_params:\n ui_display_name: null\n expected_impact: 1\n num_fc_layers:\n default_value_reasoning:\n The encoder already has learnable parameters.Sometimes\n the default is 1 for modules where the FC stack is used for shape management,\n or the only source of learnable parameters.\n description_implications:\n Increasing num_fc_layers will increase the capacity\n of the model. The model will be slower to train, and there's a higher\n risk of overfitting.\n example_value:\n - 1\n expected_impact: 3\n other_information:\n Not all modules that have fc_layers also have an accompanying\n num_fc_layers parameter. Where both are present, fc_layers takes precedent\n over num_fc_layers. Specifying num_fc_layers alone uses fully connected\n layers that are configured by the defaults in FCStack.\n related_parameters:\n - fc_layers\n suggested_values: 0-1\n suggested_values_reasoning:\n The full model likely contains many learnable\n parameters. Consider starting with very few, or without any additional\n fully connected layers and add them if you observe evidence of limited\n model capacity. Sometimes the default is 1 for modules where the FC stack\n is used for shape management, or the only source of learnable parameters.\n ui_display_name: Number of Fully Connected Layers\n output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 16 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\n residual:\n ui_display_name: null\n expected_impact: 1\n use_bias:\n default_value_reasoning:\n \"Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to use bias terms.\n\n\n Batch Normalization, however, adds a trainable shift parameter which is\n added to the activation. When Batch Normalization is used in a layer,\n bias terms are redundant and may be removed.\"\n description_implications:\n Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to leave this parameter set to True.\n example_value:\n - true\n expected_impact: 1\n other_information:\n If fc_layers is not specified, or use_bias is not specified\n for individual layers, the value of use_bias will be used as the default\n for all layers.\n related_parameters:\n - bias_initializer, fc_layers\n suggested_values: \"TRUE\"\n ui_display_name: Use Bias\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\"\n - \"Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\nproject_aggregate:\n type:\n short_description: Projects the encoder outputs to a common size then takes the average.\n long_description:\n The project aggregate combiner projects the input vectors to a common size\n and then aggregates them by taking the average across all the vectors.\n activation:\n default_value_reasoning:\n The Rectified Linear Units (ReLU) function is the\n standard activation function used for adding non-linearity. It is simple,\n fast, and empirically works well (https://arxiv.org/abs/1803.08375).\n description_implications:\n Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n expected_impact: 2\n suggested_values:\n The default value will work well in the majority of the\n cases\n ui_display_name: Activation\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n fc_layers:\n default_value_reasoning:\n By default the stack is built by using num_fc_layers,\n output_size, use_bias, weights_initializer, bias_initializer, norm, norm_params,\n activation, dropout. When a list of dictionaries is provided, the stack\n is built following the parameters of each dict for building each layer.\n description_implications:\n The more layers that are specified the deeper and\n higher capacity the model will be. This makes it possible to potentially\n achieve better performance when a big anough amount of data is provided,\n but also makes the model more computationally expensive and potentially\n more prone to overfitting.\n example_value:\n - dropout: 0.1\n output_size: 128\n - norm: layer\n output_size: 64\n expected_impact: 1\n related_parameters:\n - output_size\n - use_bias\n - weights_initializer\n - bias_initializer\n - norm\n - norm_params\n - activation\n - dropout\n suggested_values_reasoning:\n It is easier to define a stack of fully connected\n layers by just specifying num_fc_layers, output_size and the other individual\n parameters. It will create a stack of layers with identical properties.\n Use this parameter only if you need a fine grained level of control of\n each individual layer in the stack.\n ui_display_name: Fully Connected Layers\n norm:\n default_value_reasoning:\n While batch normalization and layer normalization\n usually lead to improvements, it can be useful to start with fewer bells\n and whistles.\n description_implications:\n Normalization helps stabilize the learning process\n and can have a regularizing effect that can help with generalization.\n It's often suggested that with normalization, you can use a higher learning\n rate.\n example_value:\n - batch\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/\n related_parameters:\n - norm_params\n suggested_values: '\"batch\" or \"layer\"'\n suggested_values_reasoning:\n Normalization tries to solve \"internal covariate\n shift\" that comes from the changing distributions of the inputs to layers\n deep in the network when weights are updated. For example, batch normalization\n standardizes the inputs to a layer for each mini-batch. Try out different\n normalizations to see if that helps with training stability\n ui_display_name: Normalization Type\n norm_params:\n ui_display_name: null\n expected_impact: 1\n num_fc_layers:\n default_value_reasoning:\n The encoder already has learnable parameters.Sometimes\n the default is 1 for modules where the FC stack is used for shape management,\n or the only source of learnable parameters.\n description_implications:\n Increasing num_fc_layers will increase the capacity\n of the model. The model will be slower to train, and there's a higher\n risk of overfitting.\n example_value:\n - 1\n expected_impact: 3\n other_information:\n Not all modules that have fc_layers also have an accompanying\n num_fc_layers parameter. Where both are present, fc_layers takes precedent\n over num_fc_layers. Specifying num_fc_layers alone uses fully connected\n layers that are configured by the defaults in FCStack.\n related_parameters:\n - fc_layers\n suggested_values: 0-1\n suggested_values_reasoning:\n The full model likely contains many learnable\n parameters. Consider starting with very few, or without any additional\n fully connected layers and add them if you observe evidence of limited\n model capacity. Sometimes the default is 1 for modules where the FC stack\n is used for shape management, or the only source of learnable parameters.\n ui_display_name: Number of Fully Connected Layers\n output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 17 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\n projection_size:\n ui_display_name: null\n expected_impact: 1\n residual:\n ui_display_name: null\n expected_impact: 1\n use_bias:\n default_value_reasoning:\n \"Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to use bias terms.\n\n\n Batch Normalization, however, adds a trainable shift parameter which is\n added to the activation. When Batch Normalization is used in a layer,\n bias terms are redundant and may be removed.\"\n description_implications:\n Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to leave this parameter set to True.\n example_value:\n - true\n expected_impact: 1\n other_information:\n If fc_layers is not specified, or use_bias is not specified\n for individual layers, the value of use_bias will be used as the default\n for all layers.\n related_parameters:\n - bias_initializer, fc_layers\n suggested_values: \"TRUE\"\n ui_display_name: Use Bias\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\"\n - \"Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\nsequence:\n type:\n short_description: Stacks a sequence concat combiner with a sequence encoder.\n long_description:\n The sequence combiner stacks a sequence concat combiner with a sequence encoder. All the\n considerations about input tensor ranks described for the sequence concat combiner apply also in\n this case, but the main difference is that this combiner uses the `b x s x h` output of the\n sequence concat combiner, where `b` is the batch size, `s` is the sequence length and `h` is the\n sum of the hidden dimensions of all input features, as input for any of the sequence encoders\n described in the sequence features encoders section. All considerations on the shape of\n the outputs for the sequence encoders also apply to the sequence combiner.\n encoder:\n ui_display_name: null\n expected_impact: 3\n main_sequence_feature:\n ui_display_name: null\n expected_impact: 3\n reduce_output:\n ui_display_name: null\n expected_impact: 1\nsequence_concat:\n type:\n short_description: Concatenates the outputs of multiple sequence features.\n long_description:\n The sequence_concat combiner assumes at least one output from the encoders is a tensor of size\n `b x s x h` where `b` is the batch size, `s` is the length of the sequence and `h` is the hidden\n dimension. A sequence-like (sequence, text or time series) input feature can be specified with\n the `main_sequence_feature` parameter which takes the name of sequence-like input feature as its\n value. If no `main_sequence_feature` is specified, the combiner will look through all the\n features in the order they are defined in the configuration and will look for a feature with a\n rank 3 tensor output (sequence, text or time series). If it cannot find one it will raise an\n exception, otherwise the output of that feature will be used for concatenating the other features\n along the sequence `s` dimension.\n\n If there are other input features with a rank 3 output tensor, the combiner will concatenate\n them alongside the s dimension, which means that all of them must have identical s dimension,\n otherwise a dimension mismatch error will be returned thrown during training when a datapoint\n with two sequential features of different lengths are provided.\n\n Other features that have a b x h rank 2 tensor output will be replicated s times and\n concatenated to the s dimension. The final output is a b x s x h' tensor where h' is the size of\n the concatenation of the h dimensions of all input features.\n main_sequence_feature:\n ui_display_name: null\n expected_impact: 3\n reduce_output:\n ui_display_name: null\n expected_impact: 1\ntabnet:\n type:\n short_description: Tabnet is specifically tailored for high performance on tabular data.\n long_description:\n The tabnet combiner implements the TabNet model, which uses attention and sparsity to achieve\n high performance on tabular data. It assumes all outputs from encoders are tensors of size b x h\n where b is the batch size and h is the hidden dimension, which can be different for each input.\n If the input tensors have a different shape, it automatically flattens them. It returns the\n final b x h' tensor where h' is the user-specified output size.\n literature_references:\n - https://arxiv.org/abs/1908.07442\n compute_tier: 1\n bn_epsilon:\n default_value_reasoning:\n Default value found in popular ML packages like Keras\n and Tensorflow.\n description_implications:\n An epsilon is added to the denominator of the batch\n normalization operation so that the function converges. Setting the epsilon\n to 0 is inadvisable.\n example_value:\n - 1.0e-05\n expected_impact: 1\n literature_references:\n - \"[Keras example](https://keras.io/api/layers/normalization_layers/batch_normalization/)\"\n suggested_values: 1e-3-1e-9\n suggested_values_reasoning: Common epsilon choices\n ui_display_name: Batch Normalization Epsilon\n bn_momentum:\n description_implications:\n \"Higher values result in faster updates, but more\n sensitivity to noise in the dataset. Lower values result in slower updates.\n\n\n If momentum is set to 0, moving statistics will not be updated during\n training. This is likely to cause variance between train and test performance,\n and is not recommended.\"\n example_value:\n - 0.05\n literature_references:\n - \"TabNet Paper: https://arxiv.org/abs/1908.07442\"\n - \"Torch Batch Norm: https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm1d.html\"\n other_information:\n \"`bn_momentum` is only used if `norm`: `batch`. For other\n values of `norm` it has no effect.\n\n\n `bn_momentum` is different from optimizer momentum. Batch norm moving\n estimate statistics are updated according to the rule:\n\n x_hat = (1 - momentum) * x_hat + momentum * x_t,\n\n where x_hat is the estimated statistic and x_t is the new observed value.\"\n suggested_values: 0.01-0.2\n ui_display_name: Batch Norm Momentum\n expected_impact: 1\n bn_virtual_bs:\n default_value_reasoning: Paper default.\n description_implications:\n Virtual Batch Normalization is a normalization method\n that extends batch normalization. Regular batch normalization causes the\n output of a neural network for an input example to be highly dependent\n on several other inputs in the same minibatch. To avoid this problem\n in virtual batch normalization (VBN), each example is normalized based\n on the statistics collected on a reference batch of examples that are\n chosen once and fixed at the start of training, and on itself. The reference\n batch is normalized using only its own statistics. VBN is computationally\n expensive because it requires running forward propagation on two minibatches\n of data, so the authors use it only in the generator network. A higher\n virtual batch size could improve normalization, but it also causes training\n to run slower since each batch will be sampled multiple times.\n expected_impact: 1\n literature_references:\n - https://paperswithcode.com/method/virtual-batch-normalization\n ui_display_name: \"Ghost Normalization: Virtual batch size\"\n dropout:\n default_value_reasoning: Taken from published literature (https://arxiv.org/abs/1908.07442).\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n entmax_alpha:\n ui_display_name: null\n expected_impact: 1\n entmax_mode:\n ui_display_name: null\n expected_impact: 1\n num_shared_blocks:\n ui_display_name: null\n expected_impact: 1\n num_steps:\n ui_display_name: null\n expected_impact: 1\n num_total_blocks:\n ui_display_name: null\n expected_impact: 1\n output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 18 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\n relaxation_factor:\n ui_display_name: null\n expected_impact: 1\n size:\n ui_display_name: null\n expected_impact: 3\n sparsity:\n ui_display_name: null\n expected_impact: 1\ntabtransformer:\n type:\n short_description: Projects and concatenates features, then passes them through a transformer.\n long_description:\n The tabtransformer combiner combines input features in the following sequence of operations.\n Except for binary and number features, the combiner projects features to an embedding size.\n These features are concatenated as if they were a sequence and passed through a transformer.\n After the transformer, the number and binary features are concatenated (which are of size 1) and\n then concatenated with the output of the transformer and is passed to a stack of fully connected\n layers (from TabTransformer Tabular Data Modeling Using Contextual Embeddings). It assumes all\n outputs from encoders are tensors of size `b x h` where `b` is the batch size and `h` is the\n hidden dimension, which can be different for each input. If the input tensors have a different\n shape, it automatically flattens them. It then projects each input tensor to the same hidden /\n embedding size and encodes them with a stack of Transformer layers. Finally, the transformer\n combiner applies a reduction to the outputs of the Transformer stack, followed by the above\n concatenation and optional fully connected layers. The output is a `b x h` tensor where `h` is the\n size of the last fully connected layer or the hidden / embedding size, or a `b x n x h` where `n`\n is the number of input features and `h` is the hidden / embedding size if no reduction is applied.\n literature_references:\n - https://arxiv.org/abs/2012.06678\n compute_tier: 2\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n dropout:\n default_value_reasoning: Taken from published literature (https://arxiv.org/abs/1706.03762).\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n embed_input_feature_name:\n default_value_reasoning:\n Though the ideal embedding size depends on the task\n and dataset, setting the feature embedding size equal to the hidden size\n and adding feature embeddings to hidden representations ('add') is a good\n starting point.\n description_implications:\n Input feature name embeddings have been shown to\n improve performance of deep learning methods on tabular data. Feature\n name embeddings play a similar role to positional embeddings in a language\n model, allowing the network to learn conditional dependencies between\n input features.\n example_value:\n - 64\n literature_references:\n - \"TabTransformer: Tabular Data Modeling Using Contextual Embeddings\"\n other_information:\n Must be an integer, 'add', or null. If an integer, specifies\n the embedding size for input feature names. Input feature name embeddings\n will be concatenated to hidden representations. Must be less than or equal\n to hidden_size. If 'add', input feature names use embeddings the same\n size as hidden_size, and are added (element-wise) to the hidden representations.\n If null, input feature embeddings are not used.\n related_parameters:\n - hidden_size\n ui_display_name: Embed Input Feature Name\n expected_impact: 3\n fc_activation:\n default_value_reasoning:\n The Rectified Linear Units (ReLU) function is the\n standard activation function used for adding non-linearity. It is simple,\n fast, and empirically works well (https://arxiv.org/abs/1803.08375).\n description_implications:\n Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n example_value:\n - relu\n expected_impact: 1\n literature_references:\n - https://ml-cheatsheet.readthedocs.io/en/latest/activation_functions.html\n related_parameters:\n - activation, activation_function, conv_activation, recurrent_activation\n suggested_values: relu, alternatively leakyRelu or elu\n suggested_values_reasoning:\n The default value will work well in the majority\n of the cases\n ui_display_name: FC Activation\n fc_dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 1\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: FC Dropout\n fc_layers:\n default_value_reasoning:\n By default the stack is built by using num_fc_layers,\n output_size, use_bias, weights_initializer, bias_initializer, norm, norm_params,\n activation, dropout. When a list of dictionaries is provided, the stack\n is built following the parameters of each dict for building each layer.\n description_implications:\n The more layers that are specified the deeper and\n higher capacity the model will be. This makes it possible to potentially\n achieve better performance when a big anough amount of data is provided,\n but also makes the model more computationally expensive and potentially\n more prone to overfitting.\n example_value:\n - dropout: 0.1\n output_size: 128\n - norm: layer\n output_size: 64\n expected_impact: 1\n suggested_values_reasoning:\n It is easier to define a stack of fully connected\n layers by just specifying num_fc_layers, output_size and the other individual\n parameters. It will create a stack of layers with identical properties.\n Use this parameter only if you need a fine grained level of control of\n each individual layer in the stack.\n ui_display_name: Fully Connected Layers\n fc_residual:\n ui_display_name: null\n expected_impact: 1\n hidden_size:\n default_value_reasoning: Not too big, not too small.\n description_implications:\n Increasing the hidden size makes the model larger\n and slower to train, increases the model's capacity to capture more complexity.\n It also increases the chance of overfitting.\n expected_impact: 2\n suggested_values: 10 - 2048\n suggested_values_reasoning:\n Increasing the hidden size makes sense if the\n model is underfitting. It's useful to train both smaller and larger models\n to see how model capacity affects performance. This should only be explored\n after the architecture of the model has been settled.\n ui_display_name: Hidden Size\n norm:\n default_value_reasoning:\n While batch normalization and layer normalization\n usually lead to improvements, it can be useful to start with fewer bells\n and whistles.\n description_implications:\n Normalization helps stabilize the learning process\n and can have a regularizing effect that can help with generalization.\n It's often suggested that with normalization, you can use a higher learning\n rate.\n example_value:\n - batch\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/\n related_parameters:\n - norm_params\n suggested_values: '\"batch\" or \"layer\"'\n suggested_values_reasoning:\n Normalization tries to solve \"internal covariate\n shift\" that comes from the changing distributions of the inputs to layers\n deep in the network when weights are updated. For example, batch normalization\n standardizes the inputs to a layer for each mini-batch. Try out different\n normalizations to see if that helps with training stability\n ui_display_name: Normalization Type\n norm_params:\n ui_display_name: null\n expected_impact: 1\n num_fc_layers:\n default_value_reasoning:\n The encoder already has learnable parameters.Sometimes\n the default is 1 for modules where the FC stack is used for shape management,\n or the only source of learnable parameters.\n description_implications:\n Increasing num_fc_layers will increase the capacity\n of the model. The model will be slower to train, and there's a higher\n risk of overfitting.\n example_value:\n - 1\n expected_impact: 3\n other_information:\n Not all modules that have fc_layers also have an accompanying\n num_fc_layers parameter. Where both are present, fc_layers takes precedent\n over num_fc_layers. Specifying num_fc_layers alone uses fully connected\n layers that are configured by the defaults in FCStack.\n related_parameters:\n - fc_layers\n suggested_values: 0-1\n suggested_values_reasoning:\n The full model likely contains many learnable\n parameters. Consider starting with very few, or without any additional\n fully connected layers and add them if you observe evidence of limited\n model capacity. Sometimes the default is 1 for modules where the FC stack\n is used for shape management, or the only source of learnable parameters.\n ui_display_name: Number of Fully Connected Layers\n num_heads:\n default_value_reasoning:\n \"The middle value explored in the original TabTransformer\n paper. Source: https://arxiv.org/pdf/2012.06678.pdf\"\n description_implications:\n Increasing the number of attention heads can increase\n model performance at the cost of additional compute and memory.\n example_value:\n - 8\n expected_impact: 1\n literature_references:\n - https://arxiv.org/pdf/2012.06678.pdf\n suggested_values: 16\n suggested_values_reasoning:\n If your model is underperforming, increasing the\n number of attention heads can improve its ability to correlate items in\n a sequence.\n ui_display_name: Number of attention heads\n num_layers:\n default_value_reasoning:\n The ideal number of layers depends on the data. For\n many data types, one layer is sufficient.\n description_implications:\n \"The ideal number of transformer layers depends\n on the length and complexity of input sequences, as well as the task.\n\n\n For more complex tasks, and higher number of transformer layers may be\n useful. However, too many layers will increase memory and slow training\n while providing diminishing returns of model performance.\"\n example_value:\n - 1\n expected_impact: 1\n suggested_values: 1 - 12\n suggested_values_reasoning:\n Increasing the number of layers may improve encoder\n performance. However, more layers will increase training time and may\n cause overfitting. Small numbers of layers usually work best.\n ui_display_name: Number of Transformer Layers\n output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 19 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n transformer_output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 2\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 20 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Transformer Output Size\n use_bias:\n default_value_reasoning:\n \"Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to use bias terms.\n\n\n Batch Normalization, however, adds a trainable shift parameter which is\n added to the activation. When Batch Normalization is used in a layer,\n bias terms are redundant and may be removed.\"\n description_implications:\n Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to leave this parameter set to True.\n example_value:\n - true\n expected_impact: 1\n other_information:\n If fc_layers is not specified, or use_bias is not specified\n for individual layers, the value of use_bias will be used as the default\n for all layers.\n related_parameters:\n - bias_initializer, fc_layers\n suggested_values: \"TRUE\"\n ui_display_name: Use Bias\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\"\n - \"Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\ntransformer:\n type:\n short_description: The transformer combiner combines input features using a stack of Transformer blocks.\n long_description:\n The transformer combiner combines input features using a stack of Transformer blocks (from\n Attention Is All You Need). It assumes all outputs from encoders are tensors of size `b x h`\n where `b` is the batch size and `h` is the hidden dimension, which can be different for each\n input. If the input tensors have a different shape, it automatically flattens them. It then\n projects each input tensor to the same hidden / embedding size and encodes them with a stack of\n Transformer layers. Finally, the transformer combiner applies a reduction to the outputs of the\n Transformer stack, followed by optional fully connected layers. The output is a `b x h` tensor\n where `h` is the size of the last fully connected layer or the hidden / embedding size, or a\n `b x n x h` where `n` is the number of input features and `h` is the hidden / embedding size if\n no reduction is applied.\n literature_references:\n - https://arxiv.org/abs/1706.03762\n compute_tier: 2\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n dropout:\n default_value_reasoning: Taken from published literature (https://arxiv.org/abs/1706.03762).\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n fc_activation:\n default_value_reasoning:\n The Rectified Linear Units (ReLU) function is the\n standard activation function used for adding non-linearity. It is simple,\n fast, and empirically works well (https://arxiv.org/abs/1803.08375).\n description_implications:\n Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n example_value:\n - relu\n expected_impact: 1\n literature_references:\n - https://ml-cheatsheet.readthedocs.io/en/latest/activation_functions.html\n related_parameters:\n - activation, activation_function, conv_activation, recurrent_activation\n suggested_values: relu, alternatively leakyRelu or elu\n suggested_values_reasoning:\n The default value will work well in the majority\n of the cases\n ui_display_name: FC Activation\n fc_dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 1\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: FC Dropout\n fc_layers:\n default_value_reasoning:\n By default the stack is built by using num_fc_layers,\n output_size, use_bias, weights_initializer, bias_initializer, norm, norm_params,\n activation, dropout. When a list of dictionaries is provided, the stack\n is built following the parameters of each dict for building each layer.\n description_implications:\n The more layers that are specified the deeper and\n higher capacity the model will be. This makes it possible to potentially\n achieve better performance when a big anough amount of data is provided,\n but also makes the model more computationally expensive and potentially\n more prone to overfitting.\n example_value:\n - dropout: 0.1\n output_size: 128\n - norm: layer\n output_size: 64\n expected_impact: 1\n suggested_values_reasoning:\n It is easier to define a stack of fully connected\n layers by just specifying num_fc_layers, output_size and the other individual\n parameters. It will create a stack of layers with identical properties.\n Use this parameter only if you need a fine grained level of control of\n each individual layer in the stack.\n ui_display_name: Fully Connected Layers\n fc_residual:\n ui_display_name: null\n hidden_size:\n default_value_reasoning: Not too big, not too small.\n description_implications:\n Increasing the hidden size makes the model larger\n and slower to train, increases the model's capacity to capture more complexity.\n It also increases the chance of overfitting.\n expected_impact: 2\n suggested_values: 10 - 2048\n suggested_values_reasoning:\n Increasing the hidden size makes sense if the\n model is underfitting. It's useful to train both smaller and larger models\n to see how model capacity affects performance. This should only be explored\n after the architecture of the model has been settled.\n ui_display_name: Hidden Size\n norm:\n default_value_reasoning:\n While batch normalization and layer normalization\n usually lead to improvements, it can be useful to start with fewer bells\n and whistles.\n description_implications:\n Normalization helps stabilize the learning process\n and can have a regularizing effect that can help with generalization.\n It's often suggested that with normalization, you can use a higher learning\n rate.\n example_value:\n - batch\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/\n related_parameters:\n - norm_params\n suggested_values: '\"batch\" or \"layer\"'\n suggested_values_reasoning:\n Normalization tries to solve \"internal covariate\n shift\" that comes from the changing distributions of the inputs to layers\n deep in the network when weights are updated. For example, batch normalization\n standardizes the inputs to a layer for each mini-batch. Try out different\n normalizations to see if that helps with training stability\n ui_display_name: Normalization Type\n norm_params:\n ui_display_name: null\n expected_impact: 1\n num_fc_layers:\n default_value_reasoning:\n The encoder already has learnable parameters.Sometimes\n the default is 1 for modules where the FC stack is used for shape management,\n or the only source of learnable parameters.\n description_implications:\n Increasing num_fc_layers will increase the capacity\n of the model. The model will be slower to train, and there's a higher\n risk of overfitting.\n example_value:\n - 1\n expected_impact: 3\n other_information:\n Not all modules that have fc_layers also have an accompanying\n num_fc_layers parameter. Where both are present, fc_layers takes precedent\n over num_fc_layers. Specifying num_fc_layers alone uses fully connected\n layers that are configured by the defaults in FCStack.\n related_parameters:\n - fc_layers\n suggested_values: 0-1\n suggested_values_reasoning:\n The full model likely contains many learnable\n parameters. Consider starting with very few, or without any additional\n fully connected layers and add them if you observe evidence of limited\n model capacity. Sometimes the default is 1 for modules where the FC stack\n is used for shape management, or the only source of learnable parameters.\n ui_display_name: Number of Fully Connected Layers\n num_heads:\n ui_display_name: null\n expected_impact: 1\n num_layers:\n default_value_reasoning:\n The ideal number of layers depends on the data. For\n many data types, one layer is sufficient.\n description_implications:\n \"The ideal number of transformer layers depends\n on the length and complexity of input sequences, as well as the task.\n\n\n For more complex tasks, and higher number of transformer layers may be\n useful. However, too many layers will increase memory and slow training\n while providing diminishing returns of model performance.\"\n example_value:\n - 1\n expected_impact: 1\n suggested_values: 1 - 12\n suggested_values_reasoning:\n Increasing the number of layers may improve encoder\n performance. However, more layers will increase training time and may\n cause overfitting. Small numbers of layers usually work best.\n ui_display_name: Number of Transformer Layers\n output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 21 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n transformer_output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 2\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 22 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Transformer Output Size\n use_bias:\n default_value_reasoning:\n \"Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to use bias terms.\n\n\n Batch Normalization, however, adds a trainable shift parameter which is\n added to the activation. When Batch Normalization is used in a layer,\n bias terms are redundant and may be removed.\"\n description_implications:\n Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to leave this parameter set to True.\n example_value:\n - true\n expected_impact: 1\n other_information:\n If fc_layers is not specified, or use_bias is not specified\n for individual layers, the value of use_bias will be used as the default\n for all layers.\n related_parameters:\n - bias_initializer, fc_layers\n suggested_values: \"TRUE\"\n ui_display_name: Use Bias\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\"\n - \"Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\n"
},
{
"path": "ludwig/schema/metadata/configs/common.yaml",
"content": "activation:\n default_value_reasoning: The Rectified Linear Units (ReLU) function is the\n standard activation function used for adding non-linearity. It is simple,\n fast, and empirically works well (https://arxiv.org/abs/1803.08375).\n description_implications: Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n expected_impact: 2\n suggested_values: relu\n suggested_values_reasoning: ReLU will work well in the majority of the cases\n ui_display_name: Activation\nbias_initializer:\n default_value_reasoning: It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications: It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning: It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\ndropout:\n default_value_reasoning: Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications: \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning: Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\nfc_layers:\n default_value_reasoning: By default the stack is built by using num_fc_layers,\n output_size, use_bias, weights_initializer, bias_initializer, norm, norm_params,\n activation, dropout. When a list of dictionaries is provided, the stack\n is built following the parameters of each dict for building each layer.\n description_implications: The more layers that are specified the deeper and\n higher capacity the model will be. This makes it possible to potentially\n achieve better performance when a big anough amount of data is provided,\n but also makes the model more computationally expensive and potentially\n more prone to overfitting.\n example_value:\n - dropout: 0.1\n output_size: 128\n - norm: layer\n output_size: 64\n expected_impact: 1\n related_parameters:\n - output_size\n - use_bias\n - weights_initializer\n - bias_initializer\n - norm\n - norm_params\n - activation\n - dropout\n suggested_values_reasoning: It is easier to define a stack of fully connected\n layers by just specifying num_fc_layers, output_size and the other individual\n parameters. It will create a stack of layers with identical properties.\n Use this parameter only if you need a fine grained level of control of\n each individual layer in the stack.\n ui_display_name: Fully Connected Layers\nflatten_inputs:\n ui_display_name: null\n expected_impact: 1\nnorm:\n default_value_reasoning: While batch normalization and layer normalization\n usually lead to improvements, it can be useful to start with fewer bells\n and whistles.\n description_implications: Normalization helps stabilize the learning process\n and can have a regularizing effect that can help with generalization.\n It's often suggested that with normalization, you can use a higher learning\n rate.\n example_value:\n - batch\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/\n related_parameters:\n - norm_params\n suggested_values_reasoning: Normalization tries to solve \"internal covariate\n shift\" that comes from the changing distributions of the inputs to layers\n deep in the network when weights are updated. For example, batch normalization\n standardizes the inputs to a layer for each mini-batch. Try out different\n normalizations to see if that helps with training stability\n ui_display_name: Normalization Type\nnorm_params:\n ui_display_name: null\n expected_impact: 1\nnum_fc_layers:\n default_value_reasoning:\n The encoder already has learnable parameters.Sometimes\n the default is 1 for modules where the FC stack is used for shape management,\n or the only source of learnable parameters.\n description_implications: Increasing num_fc_layers will increase the capacity\n of the model. The model will be slower to train, and there's a higher\n risk of overfitting.\n example_value:\n - 1\n expected_impact: 3\n other_information:\n Not all modules that have fc_layers also have an accompanying\n num_fc_layers parameter. Where both are present, fc_layers takes precedent\n over num_fc_layers. Specifying num_fc_layers alone uses fully connected\n layers that are configured by the defaults in FCStack.\n related_parameters:\n - fc_layers\n suggested_values: 0-1\n suggested_values_reasoning: The full model likely contains many learnable\n parameters. Consider starting with very few, or without any additional\n fully connected layers and add them if you observe evidence of limited\n model capacity. Sometimes the default is 1 for modules where the FC stack\n is used for shape management, or the only source of learnable parameters.\n ui_display_name: Number of Fully Connected Layers\noutput_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications: If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information: If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 16 - 1024\n suggested_values_reasoning: Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\nresidual:\n ui_display_name: null\n expected_impact: 1\nuse_bias:\n default_value_reasoning: \"Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to use bias terms.\n\n\n Batch Normalization, however, adds a trainable shift parameter which is\n added to the activation. When Batch Normalization is used in a layer,\n bias terms are redundant and may be removed.\"\n description_implications: Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to leave this parameter set to True.\n example_value:\n - true\n expected_impact: 1\n other_information: If fc_layers is not specified, or use_bias is not specified\n for individual layers, the value of use_bias will be used as the default\n for all layers.\n related_parameters:\n - bias_initializer, fc_layers\n suggested_values: \"TRUE\"\n ui_display_name: Use Bias\nweights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications: The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\"\n - \"Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning: Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\nembedding_initializer:\n default_value_reasoning: According to https://arxiv.org/abs/1711.09160, choice\n of embedding initialization is not important as long as the variance is\n kept reasonably low.\n description_implications:\n According to https://arxiv.org/abs/1711.09160, choice\n of embedding initialization is not important as long as the variance is\n kept reasonably low.\n example_value:\n - kaiming\n expected_impact: 1\n literature_references:\n - https://arxiv.org/abs/1711.09160\n suggested_values: kaiming\n suggested_values_reasoning: https://discuss.huggingface.co/t/state-of-the-art-technique-for-initializing-embedding-matrix/326\n ui_display_name: Embedding Initialization\nembedding_size:\n default_value_reasoning: Not too big, not too small.\n description_implications: 'An embedding is a relatively low-dimensional space\n that is used to translate high-dimensional vectors like words, which can\n have a large vocbulary size. Ideally, after an embedding is trained, it\n captures some of the semantics of the input by placing semantically similar\n inputs close together in the embedding space.\n\n\n In most cases, the embedding size is chosen empirically, by trial and\n error. From https://www.amazon.com/dp/1098115783, \"one rule of thumb is\n to use the fourth root of the total number of unique categorical elements\n while another is that the embedding dimension should be approximately\n 1.6 times the square root of the number of unique elements in the category,\n and no less than 600.\"\n\n\n Increasing the embedding size may cause the model to train more slowly,\n but the higher dimensionality can also improve overall quality.'\n expected_impact: 3\n literature_references:\n - https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture\n suggested_values: 1.6 * sqrt(vocab_size)\n suggested_values_reasoning:\n Rule of thumb suggested by a deep learning textbook.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Embedding Size\nembeddings_on_cpu:\n default_value_reasoning: By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access.\n description_implications: By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access. However, in some\n cases when the vocabulary size is very large, the full embedding matrix\n may be really big and unwieldy to have in GPU memory. This parameter forces\n the placement of the embedding matrix in regular memory and the CPU is\n used to access them. This may slow down training due to additional data\n transfer between CPU and GPU memory, but can lead to healthier GPU memory\n resource usage.\n expected_impact: 1\n suggested_values:\n - false\n suggested_values_reasoning:\n If GPU memory is not a constraint, having embeddings\n stored and accessed within the GPU is faster.\n ui_display_name: Embeddings on CPU\nembeddings_trainable:\n default_value_reasoning:\n If trained from scratch, embedding vectors are typically\n learned alongside the rest of the model.\n description_implications:\n Typically this value is only set to False if pre-trained\n embeddings are uploaded. Even then, it is reasonable to leave it as True\n in order to fine-tune the embeddings.\n expected_impact: 1\n related_parameters:\n - embedding_size, representation, pretrained_embeddings\n ui_display_name: (under Embeddings header) Trainable?\npretrained_embeddings:\n default_value_reasoning: Embeddings are commonly trained from scratch, or\n incorporated as part of a pre-trained model package.\n description_implications: If pretrained embeddings are specified, then the\n model may have a head start in its representation of various input entities.\n example_value:\n - ~/Downloads/glove.6B.100d.txt\n expected_impact: 0\n related_parameters:\n - embedding_size, embeddings_trainable\n ui_display_name: Pretrained embeddings path\nmax_sequence_length:\n default_value_reasoning: Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\nvocab:\n default_value_reasoning: Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\nvocab_size:\n internal_only: true\n ui_display_name: Not displayed\nrepresentation:\n default_value_reasoning: Trainable, randomly initialized embedding vectors\n often lead to more subtle representations of input entities than one-hot\n vectors.\n description_implications: If set to sparse, the representations for input\n entities are fixed as one-hot vectors. This leads to less flexible representations\n for input entities, but could lead to faster training since there are\n less learnable parameters.\n expected_impact: 1\n other_information: \"\"\n related_parameters:\n - embedding_size, embeddings_trainable, pretrained_embeddings\n ui_display_name: Representation approach\nreduce_output:\n default_value_reasoning: Sums the tensors along the sequence dimension.\n description_implications: \"\\\"last\\\", \\\"sum\\\", \\\"mean\\\", and \\\"max\\\" are the\\\n \\ fastest and most memory-efficient operations\\u2013 they result in tensors\\\n \\ that are the same-size as a single item in the input sequence. However,\\\n \\ these are simple aggregation operations, therefore some information\\\n \\ may be lost. \\n\\n\\\"concat\\\" concatenates each tensor together, creating\\\n \\ a `(sequence length)*(tensor size)`-element tensor. \\\"concat\\\" preserves\\\n \\ this information, but can be very memory-intensive and should only be\\\n \\ applied if the sequence length and/or tensor size is small. \\n\\n\\\"attention\\\"\\\n \\ takes a weighted sum of the items in the sequence, where the weights\\\n \\ for each item in the sequence are determined by the model on-the-fly\\\n \\ based on the features of the item itself. This is both slower and and\\\n \\ more memory-intensive than the other operations; however, it can also\\\n \\ provide a richer \\\"global\\\" representation of the sequence.\"\n expected_impact: 1\n related_parameters:\n - max_sequence_length\n suggested_values: '\"attention\". This and the default covers 95% of use cases.'\n suggested_values_reasoning: If you would like better performance and are not\n compute/memory-constrained, attention-based reduction can potentially\n provide a richer global representation than the default, but note that attention\n reduction does not work with `cache_encoder_embeddings=true`.\n ui_display_name: Sequence Reducer\n"
},
{
"path": "ludwig/schema/metadata/configs/decoders.yaml",
"content": "BaseDecoder:\n type:\n expected_impact: 1\n fc_layers:\n expected_impact: 1\n num_fc_layers:\n expected_impact: 3\n fc_output_size:\n expected_impact: 3\n fc_use_bias:\n expected_impact: 1\n fc_weights_initializer:\n expected_impact: 1\n fc_bias_initializer:\n expected_impact: 1\n fc_norm:\n expected_impact: 2\n fc_norm_params:\n expected_impact: 1\n fc_activation:\n expected_impact: 2\n fc_dropout:\n expected_impact: 3\nClassifier:\n type:\n short_description:\n Projects combiner output to a vector the size of the number of available classes.\n long_description:\n The classifier decoder is a (potentially empty) stack of fully connected layers, followed by a\n projection into a vector of size of the number of available classes, followed by a sigmoid.\n expected_impact: 0\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n input_size:\n other_information: Internal Only\n internal_only: true\n related_parameters:\n - \"No\"\n ui_display_name: Not Displayed\n num_classes:\n other_information: Internal Only\n internal_only: true\n ui_display_name: Not Displayed\n use_bias:\n default_value_reasoning:\n \"Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to use bias terms.\n\n\n Batch Normalization, however, adds a trainable shift parameter which is\n added to the activation. When Batch Normalization is used in a layer,\n bias terms are redundant and may be removed.\"\n description_implications:\n Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to leave this parameter set to True.\n example_value:\n - true\n expected_impact: 1\n other_information:\n If fc_layers is not specified, or use_bias is not specified\n for individual layers, the value of use_bias will be used as the default\n for all layers.\n related_parameters:\n - bias_initializer, fc_layers\n suggested_values: true\n ui_display_name: Use Bias\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\"\n - \"Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\nProjector:\n type:\n short_description:\n Projects combiner output into an output vector.\n long_description:\n The Projector decoder is a (potentially empty) stack of fully connected layers, followed by a\n projection into a tensor of the vector size (optionally followed by a softmax in the case of\n multi-class classification).\n expected_impact: 0\n activation:\n description_implications:\n Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n expected_impact: 2\n suggested_values:\n The default value will work well in the majority of the\n cases\n ui_display_name: Activation\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n clip:\n ui_display_name: null\n expected_impact: 1\n input_size:\n other_information: Internal Only\n internal_only: true\n related_parameters:\n - \"No\"\n ui_display_name: Not Displayed\n output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 10 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\n use_bias:\n default_value_reasoning:\n \"Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to use bias terms.\n\n\n Batch Normalization, however, adds a trainable shift parameter which is\n added to the activation. When Batch Normalization is used in a layer,\n bias terms are redundant and may be removed.\"\n description_implications:\n Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to leave this parameter set to True.\n example_value:\n - true\n expected_impact: 1\n other_information:\n If fc_layers is not specified, or use_bias is not specified\n for individual layers, the value of use_bias will be used as the default\n for all layers.\n related_parameters:\n - bias_initializer, fc_layers\n suggested_values: true\n ui_display_name: Use Bias\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\"\n - \"Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\nRegressor:\n type:\n short_description:\n Projects combiner output to a single number.\n long_description:\n The regressor decoder is a (potentially empty) stack of fully connected layers, followed by a\n projection to a single number.\n expected_impact: 0\n activation:\n ui_display_name: null\n expected_impact: 2\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n input_size:\n other_information: Internal Only\n internal_only: true\n related_parameters:\n - \"No\"\n ui_display_name: Not Displayed\n use_bias:\n default_value_reasoning:\n \"Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to use bias terms.\n\n\n Batch Normalization, however, adds a trainable shift parameter which is\n added to the activation. When Batch Normalization is used in a layer,\n bias terms are redundant and may be removed.\"\n description_implications:\n Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to leave this parameter set to True.\n example_value:\n - true\n expected_impact: 1\n other_information:\n If fc_layers is not specified, or use_bias is not specified\n for individual layers, the value of use_bias will be used as the default\n for all layers.\n related_parameters:\n - bias_initializer, fc_layers\n suggested_values: true\n ui_display_name: Use Bias\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\"\n - \"Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\nPassthroughDecoder:\n type:\n short_description:\n Provides the raw input from the combiner.\n long_description:\n The passthrough decoder simply returns the raw output coming from the combiner.\n expected_impact: 0\n input_size:\n other_information: Internal Only\n internal_only: true\n related_parameters:\n - \"No\"\n ui_display_name: Not Displayed\nSequenceGeneratorDecoder:\n type:\n short_description:\n Generates a sequence by sampling from the model.\n long_description:\n The generator decoder is a (potentially empty) stack of fully connected layers, followed by an\n RNN that generates outputs feeding on its own previous predictions and generates a tensor of\n size `b x s' x c`, where `b` is the batch size, `s'` is the length of the generated sequence and\n `c` is the number of classes, followed by a softmax_cross_entropy. During training teacher\n forcing is adopted, meaning the list of targets is provided as both inputs and outputs (shifted\n by 1), while at evaluation time greedy decoding (generating one token at a time and feeding it\n as input for the next step) is performed by beam search, using a beam of 1 by default. In\n general a generator expects a `b x h` shaped input tensor, where `h` is a hidden dimension. The\n `h` vectors are (after an optional stack of fully connected layers) fed into the rnn generator.\n One exception is when the generator uses attention, as in that case the expected size of the\n input tensor is `b x s x h`, which is the output of a sequence, text or time series input\n feature without reduced outputs or the output of a sequence-based combiner. If a `b x h` input\n is provided to a generator decoder using an RNN with attention instead, an error will be raised\n during model building.\n expected_impact: 0\n cell_type:\n ui_display_name: null\n expected_impact: 3\n input_size:\n other_information: Internal Only\n internal_only: true\n related_parameters:\n - \"No\"\n ui_display_name: Not Displayed\n max_sequence_length:\n expected_impact: 3\n ui_display_name: null\n num_layers:\n default_value_reasoning:\n The ideal number of layers depends on the data and\n task. For many data types, one layer is sufficient.\n description_implications:\n Increasing the number of layers may improve model\n performance for longer sequences or more complex tasks.\n example_value:\n - 1\n expected_impact: 3\n suggested_values: 1-3\n suggested_values_reasoning:\n Increasing the number of layers may improve encoder\n performance. However, more layers will increase training time and may\n cause overfitting. Small numbers of layers usually work best.\n ui_display_name: Number of Recurrent Layers\n reduce_input:\n description_implications:\n \"\\u201Clast\\u201D: Reduces tensor by taking the\\\n \\ last non-zero element per sequence in the sequence dimension.\\n\\u201C\\\n sum\\u201D: Reduces tensor by summing across the sequence dimension.\\n\\u201C\\\n mean\\u201D: Reduces tensor by taking the mean of the sequence dimension.\\n\\\n \\u201Cavg\\u201D: synonym for \\u201Cmean\\u201D.\\n\\u201Cmax\\u201D: Reduces\\\n \\ tensor by taking the maximum value of the last dimension across the\\\n \\ sequence dimension.\\n\\u201Cconcat\\u201D: Reduces tensor by concatenating\\\n \\ the second and last dimension.\\n\\u201Cattention\\u201D: Reduces tensor\\\n \\ by summing across the sequence dimension after applying feedforward\\\n \\ attention.\\n\\u201Cnone\\u201D: no reduction.\"\n expected_impact: 2\n ui_display_name: Combiner Reduce Mode\n vocab_size:\n ui_display_name: Not displayed\nSequenceTaggerDecoder:\n type:\n short_description:\n Used for classifying each element of an input sequence.\n long_description:\n The tagger decoder is a (potentially empty) stack of fully connected layers,\n followed by a projection into a tensor of size `b x s x c`, where `b` is the batch size,\n `s` is the length of the sequence and `c` is the number of classes, followed by a\n `softmax_cross_entropy`.\n\n This decoder requires its input to be shaped as `b x s x h`, where `h` is\n a hidden dimension, which is the output of a sequence, text or time series input feature without\n reduced outputs or the output of a sequence-based combiner. This can be done by ensuring that\n at least one of the sequence, text or time series input feature's encoders has `reduce_output` set to\n `None`. This will prevent a `b x h` input from being provided to this decoder and an error\n from being raised during model building.\n\n The tagger decoder also requires the `reduce_input` parameter of the output feature to be set to `None`.\n If this is not set, Ludwig will automatically override the value by setting it to None and log a warning.\n expected_impact: 0\n attention_embedding_size:\n default_value_reasoning: Not too big, not too small.\n description_implications:\n Increasing the embedding size may cause the model\n to train more slowly, but the higher dimensionality can also improve overall\n quality.\n expected_impact: 2\n suggested_values: 128 - 2048\n suggested_values_reasoning:\n Try models with smaller or larger embedding sizes\n to observe relative impact.\n ui_display_name: Attention Embedding Size\n attention_num_heads:\n ui_display_name: null\n expected_impact: 1\n input_size:\n other_information: Internal Only\n internal_only: true\n related_parameters:\n - \"No\"\n ui_display_name: Not Displayed\n max_sequence_length:\n expected_impact: 3\n ui_display_name: null\n use_attention:\n ui_display_name: null\n expected_impact: 1\n use_bias:\n default_value_reasoning:\n \"Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to use bias terms.\n\n\n Batch Normalization, however, adds a trainable shift parameter which is\n added to the activation. When Batch Normalization is used in a layer,\n bias terms are redundant and may be removed.\"\n description_implications:\n Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to leave this parameter set to True.\n example_value:\n - true\n expected_impact: 1\n other_information:\n If fc_layers is not specified, or use_bias is not specified\n for individual layers, the value of use_bias will be used as the default\n for all layers.\n related_parameters:\n - bias_initializer, fc_layers\n suggested_values:\n - true\n ui_display_name: Use Bias\n vocab_size:\n ui_display_name: Not displayed\n internal_only: true\nUNetDecoder:\n type:\n short_description: The UNet decoder convolutional and up-conv layers\n long_description:\n Stacks of two 2D convolutional layers with optional normalization\n and relu activation, preceeded by an up-conv layer in all but the\n final level of the decoder.\n compute_tier: 1\n conv_norm:\n expected_impact: 2\n ui_display_name: Convolutional Normalization\n height:\n default_value_reasoning:\n Computed internally, automatically, based on image\n data preprocessing.\n internal_only: true\n ui_display_name: NOT DISPLAYED\n input_size:\n other_information: Internal Only\n internal_only: true\n related_parameters:\n - \"No\"\n ui_display_name: Not Displayed\n num_channels:\n default_value_reasoning:\n Computed internally, automatically, based on image\n data preprocessing.\n internal_only: true\n ui_display_name: NOT DISPLAYED\n num_classes:\n default_value_reasoning:\n Computed internally, automatically, based on image\n data preprocessing.\n internal_only: true\n ui_display_name: NOT DISPLAYED\n width:\n default_value_reasoning:\n Computed internally, automatically, based on image\n data preprocessing.\n internal_only: true\n ui_display_name: NOT DISPLAYED\n"
},
{
"path": "ludwig/schema/metadata/configs/encoders.yaml",
"content": "BaseEncoder:\n skip:\n internal_only: true\n other_information: Internal Only\n ui_display_name: Not Displayed\nHFEncoder:\n trainable:\n default_value_reasoning:\n Trainable is disabled by default to make the model useful for generating fast baselines, which can be\n further sped up by setting `preprocessing.cache_encoder_embeddings`. In many cases strong performance\n can be achieved without adjusting the weights of the pretrained model, but for best performance we\n recommend setting this to true.\n description_implications:\n \"Ludwig currently supports two variations on fine-tuning, configured via the trainable encoder parameter:\n (1) modifying the weights of the pretrained encoder to adapt them to the downstream task (trainable=true),\n or (2) keeping the pretrained encoder weights fixed and training a stack of dense layers that sit\n downstream as the combiner and decoder modules (trainable=false, default). This is sometimes distinguished\n as transfer learning. Allowing the weights to be modified by setting trainable=true can significantly\n improve performance on the downstream task, but will take significantly longer to train (due to the\n additional backward passes over the pretrained model parameters). Additionally, more care needs to be\n taken when selecting hyperparameters when trainable=true to prevent\n [catastrophic forgettng](https://en.wikipedia.org/wiki/Catastrophic_interference), whereby the\n model forgets all of the valuable information it learned during pretraining.\"\n expected_impact: 3\n literature_references:\n - http://d2l.ai/chapter_computer-vision/fine-tuning.html\"\n related_parameters:\n - use_pretrained, pretrained_model, saved_weights_in_checkpoint\n suggested_values:\n - false\n suggested_values_reasoning:\n Freezing the weights (i.e. `trainable = False`)\n is only worth trying if you are loading in pretrained weights. In that\n case, check to see if your model is overfitting. If so, freezing the weights\n (and therefore reducing model complexity) may be beneficial.\n ui_display_name: Trainable\n use_pretrained:\n default_value_reasoning:\n By default, the model is initialized as a pretrained\n model.\n description_implications:\n Pretrained models have typically already learned\n features that are difficult to learn from scratch. They are particularly\n beneficial when training on small amounts of data.\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/transfer-learning-for-deep-learning/\n related_parameters:\n - trainable, pretrained_model_name, pretrained_model_name_or_path, pretrained_kwargs\n suggested_values:\n - false\n suggested_values_reasoning:\n If you have a large amount of data and/or you\n have data that differs from the typical distribution, then it might be\n worth training the model from scratch.\n ui_display_name: Use Pretrained\n reduce_output:\n default_value_reasoning: Sums the tensors along the sequence dimension.\n description_implications:\n \"\\\"last\\\", \\\"sum\\\", \\\"mean\\\", and \\\"max\\\" are the\\\n \\ fastest and most memory-efficient operations\\u2013 they result in tensors\\\n \\ that are the same-size as a single item in the input sequence. However,\\\n \\ these are simple aggregation operations, therefore some information\\\n \\ may be lost. \\n\\n\\\"concat\\\" concatenates each tensor together, creating\\\n \\ a `(sequence length)*(tensor size)`-element tensor. \\\"concat\\\" preserves\\\n \\ this information, but can be very memory-intensive and should only be\\\n \\ applied if the sequence length and/or tensor size is small. \\n\\n\\\"attention\\\"\\\n \\ takes a weighted sum of the items in the sequence, where the weights\\\n \\ for each item in the sequence are determined by the model on-the-fly\\\n \\ based on the features of the item itself. This is both slower and and\\\n \\ more memory-intensive than the other operations; however, it can also\\\n \\ provide a richer \\\"global\\\" representation of the sequence.\"\n expected_impact: 1\n related_parameters:\n - max_sequence_length\n suggested_values: '\"attention\". This and the default covers 95% of use cases.'\n suggested_values_reasoning:\n If you would like better performance and are not\n compute/memory-constrained, attention-based reduction can potentially\n provide a richer global representation than the default.\n ui_display_name: Sequence Reducer\nALBERT:\n type:\n short_description: Similar to BERT with lower memory footprint and faster training.\n long_description:\n The `albert` encoder loads a pretrained [ALBERT](https://arxiv.org/abs/1909.11942) (default `albert-base-v2`) model\n using the Hugging Face transformers package. Albert is similar to BERT, with significantly lower memory usage and\n somewhat faster training time:.\n compute_tier: 2\n attention_probs_dropout_prob:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 1\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - hidden_dropout_prob, classifier_dropout_prob\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: attention_probs_dropout_prob\n bos_token_id:\n default_value_reasoning: Default value used in pre-trained HF encoder.\n ui_display_name: Beginning-of-Sentence Token Id\n expected_impact: 1\n classifier_dropout_prob:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 1\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - hidden_dropout_prob, attention_probs_dropout_prob\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: classifier_dropout_prob\n embedding_size:\n default_value_reasoning: Not too big, not too small.\n description_implications:\n 'An embedding is a relatively low-dimensional space\n that is used to translate high-dimensional vectors like words, which can\n have a large vocbulary size. Ideally, after an embedding is trained, it\n captures some of the semantics of the input by placing semantically similar\n inputs close together in the embedding space.\n\n\n In most cases, the embedding size is chosen empirically, by trial and\n error. From https://www.amazon.com/dp/1098115783, \"one rule of thumb is\n to use the fourth root of the total number of unique categorical elements\n while another is that the embedding dimension should be approximately\n 1.6 times the square root of the number of unique elements in the category,\n and no less than 600.\"\n\n\n Increasing the embedding size may cause the model to train more slowly,\n but the higher dimensionality can also improve overall quality.'\n expected_impact: 1\n literature_references:\n - https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture\n suggested_values: 1.6 * sqrt(vocab_size)\n suggested_values_reasoning:\n Rule of thumb suggested by a deep learning textbook.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Embedding Size\n eos_token_id:\n default_value_reasoning: Default value used in pre-trained HF encoder.\n ui_display_name: End-of-Sentence Token Id\n expected_impact: 1\n hidden_act:\n default_value_reasoning: Taken from huggingface.\n description_implications:\n Changing this activation function will only affect\n the feed-forward layers of the transformer.\n example_value:\n - relu\n expected_impact: 1\n literature_references:\n - \"[Hugging face docs for ALBERT config](https://huggingface.co/docs/transformers/model_doc/albert#transformers.AlbertConfig.hidden_act)\\n\\\n \\r\\n[Relevant StackOverflow discussion](https://ai.stackexchange.com/questions/30341/why-does-a-transformer-not-use-an-activation-function-following-the-multi-head-a)\"\n suggested_values: gelu\n suggested_values_reasoning: Taken from huggingface defaults.\n ui_display_name: Hidden Layer Activation\n hidden_dropout_prob:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 1\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - \"attention_probs_dropout_prob,\n\n classifier_dropout_prob\"\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: hidden_dropout_prob\n hidden_size:\n default_value_reasoning: Huggingface default.\n description_implications:\n Increasing the hidden size makes the model larger\n and slower to train, increases the model's capacity to capture more complexity.\n It also increases the chance of overfitting.\n expected_impact: 1\n suggested_values: 10 - 2048\n suggested_values_reasoning:\n Increasing the hidden size makes sense if the\n model is underfitting. It's useful to train both smaller and larger models\n to see how model capacity affects performance. This should only be explored\n after the architecture of the model has been settled.\n ui_display_name: Hidden Size\n initializer_range:\n description_implications:\n There is an ideal value for this variable that doesn't\n lead to the outputs of these matrices to vanish or explode\n example_value:\n - 0.02\n expected_impact: 1\n other_information: Must be greater than 0\n related_parameters:\n - weights_initializer\n suggested_values: 0.01-0.05\n suggested_values_reasoning:\n Large values will likely lead to very large outputs.\n Small values will lead to vanishing outputs.\n ui_display_name: null\n inner_group_num:\n ui_display_name: null\n expected_impact: 1\n intermediate_size:\n ui_display_name: null\n expected_impact: 1\n layer_norm_eps:\n ui_display_name: null\n expected_impact: 1\n max_position_embeddings:\n default_value_reasoning: Taken from huggingface.\n description_implications:\n The size of the position embeddings table. This typically coincides with the\n maximum sequence length this model might ever be used with. Typically set this\n to something large just in case (e.g. 512, 1024, 2048).\n expected_impact: 1\n suggested_values: 512\n suggested_values_reasoning:\n Out of the box value based on published literature.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Max Position Embeddings\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n num_attention_heads:\n ui_display_name: null\n expected_impact: 1\n num_hidden_groups:\n ui_display_name: null\n expected_impact: 1\n num_hidden_layers:\n ui_display_name: null\n expected_impact: 1\n pad_token_id:\n ui_display_name: null\n expected_impact: 1\n position_embedding_type:\n ui_display_name: null\n expected_impact: 1\n pretrained_kwargs:\n default_value_reasoning: These arguments typically don't need to be specified.\n expected_impact: 1\n related_parameters:\n - pretrained_model_name_or_path\n suggested_values: Default\n ui_display_name: null\n pretrained_model_name_or_path:\n default_value_reasoning:\n The default model is the canonical model for this\n model architecture, and is therefore a good starting point for most use\n cases.\n description_implications:\n \"There are two factors to consider when choosing\\\n \\ a pre-trained model: (1) size, and (2) task similarity. \\n\\nThe larger\\\n \\ the model, the more subtle its comprehension of inputs can become. However,\\\n \\ larger models are also more compute and memory-intensive to train.\\n\\\n \\nModels pretrained on highly-related source tasks are more likely to\\\n \\ be successful on the target task. Consider searching the HuggingFace\\\n \\ model repository for models trained on similar tasks.\"\n expected_impact: 2\n literature_references:\n - https://arxiv.org/abs/1909.11942\n related_parameters:\n - use_pretrained, trainable, pretrained_kwargs\n suggested_values: albert-large-v2, albert-base-chinese\n suggested_values_reasoning:\n \"If you would like better performance and are\n not compute/memory-constrained, increasing model capacity can potentially\n provide a richer representation than the default. The suggested value\n upsizes the model while maintaining the same model architecture.\n\n\n Language models trained on general corpora typically generalize well.\n Consider deviating from the default only if the text in the dataset originates\n from another domain (e.g. languages other than English).\"\n ui_display_name: Pretrained model\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n saved_weights_in_checkpoint:\n default_value_reasoning:\n The weights of the encoder are not necessarily saved\n in the checkpoint. The user has to save them first.\n description_implications:\n The memory footprint for some of these encoders\n can be large.\n internal_only: true\n related_parameters:\n - skip_save_model\n suggested_values:\n - false\n suggested_values_reasoning:\n Some of these encoders are large, so it might\n be better to load them as needed, especially if 1. they're not used frequently\n 2. the user doesn't have a lot of storage.\n ui_display_name: null\n trainable:\n expected_impact: 3\n ui_display_name: null\n type_vocab_size:\n ui_display_name: null\n expected_impact: 1\n use_pretrained:\n default_value_reasoning:\n By default, the model is initialized as a pretrained\n model.\n description_implications:\n Pretrained models have typically already learned\n features that are difficult to learn from scratch. They are particularly\n beneficial when training on small amounts of data.\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/transfer-learning-for-deep-learning/\n related_parameters:\n - trainable, pretrained_model_name, pretrained_model_name_or_path, pretrained_kwargs\n suggested_values:\n - false\n suggested_values_reasoning:\n If you have a large amount of data and/or you\n have data that differs from the typical distribution, then it might be\n worth training the model from scratch.\n ui_display_name: Use Pretrained\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n vocab_size:\n internal_only: true\n ui_display_name: Not displayed\nAutoTransformer:\n type:\n short_description: Automatically retrieves the architecture from the provided model name/path.\n long_description:\n The `auto_transformer` encoder automatically instantiates the model architecture for the specified\n `pretrained_model_name_or_path`. Unlike the other HF encoders, `auto_transformer` does not provide a default value for\n `pretrained_model_name_or_path`, this is its only mandatory parameter. See the Hugging Face\n [AutoModels documentation](https://huggingface.co/docs/transformers/model_doc/auto) for more details.\n literature_references:\n - https://huggingface.co/docs/transformers/model_doc/auto\n compute_tier: 2\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n pretrained_kwargs:\n ui_display_name: null\n expected_impact: 1\n pretrained_model_name_or_path:\n ui_display_name: null\n expected_impact: 3\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n trainable:\n expected_impact: 3\n ui_display_name: null\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n vocab_size:\n internal_only: true\n ui_display_name: Not displayed\nBERT:\n type:\n short_description: Bidirectional transformer great for language modeling.\n long_description:\n The bert encoder loads a pretrained BERT (default bert-base-uncased) model using the Hugging\n Face transformers package. BERT is a bidirectional transformer pretrained using a combination of\n masked language modeling objective and next sentence prediction on a large corpus comprising the\n Toronto Book Corpus and Wikipedia.\n literature_references:\n - https://arxiv.org/abs/1810.04805\n compute_tier: 2\n attention_probs_dropout_prob:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 2\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - hidden_dropout_prob, classifier_dropout\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: attention_probs_dropout_prob\n classifier_dropout:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 2\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - hidden_dropout_prob, attention_probs_dropout_prob\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: classifier_dropout\n gradient_checkpointing:\n ui_display_name: null\n expected_impact: 1\n hidden_act:\n default_value_reasoning: Taken from huggingface.\n description_implications:\n Changing this activation function will only affect\n the feed-forward layers of the transformer.\n example_value:\n - relu\n expected_impact: 1\n literature_references:\n - \"[Huggingface docs for BERT config](https://huggingface.co/docs/transformers/model_doc/bert#transformers.BertConfig.hidden_act)\\n\\\n \\r\\n[Relevant StackOverflow discussion](https://ai.stackexchange.com/questions/30341/why-does-a-transformer-not-use-an-activation-function-following-the-multi-head-a)\"\n suggested_values: gelu\n suggested_values_reasoning: Taken from huggingface defaults.\n ui_display_name: Hidden Layer Activation\n hidden_dropout_prob:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 2\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - attention_probs_dropout_prob, classifier_dropout\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: hidden_dropout_prob\n hidden_size:\n default_value_reasoning: Huggingface default.\n description_implications:\n Increasing the hidden size makes the model larger\n and slower to train, increases the model's capacity to capture more complexity.\n It also increases the chance of overfitting.\n expected_impact: 1\n suggested_values: 10 - 2048\n suggested_values_reasoning:\n Increasing the hidden size makes sense if the\n model is underfitting. It's useful to train both smaller and larger models\n to see how model capacity affects performance. This should only be explored\n after the architecture of the model has been settled.\n ui_display_name: Hidden Size\n initializer_range:\n description_implications:\n There is an ideal value for this variable that doesn't\n lead to the outputs of these matrices to vanish or explode\n example_value:\n - 0.02\n expected_impact: 1\n other_information: Must be greater than 0\n related_parameters:\n - weights_initializer\n suggested_values: 0.01-0.05\n suggested_values_reasoning:\n Large values will likely lead to very large outputs.\n Small values will lead to vanishing outputs.\n ui_display_name: null\n intermediate_size:\n ui_display_name: null\n expected_impact: 1\n layer_norm_eps:\n ui_display_name: null\n expected_impact: 1\n max_position_embeddings:\n default_value_reasoning: Taken from huggingface.\n description_implications:\n The size of the position embeddings table. This typically coincides with the\n maximum sequence length this model might ever be used with. Typically set this\n to something large just in case (e.g. 512, 1024, 2048).\n expected_impact: 2\n suggested_values: 512\n suggested_values_reasoning:\n Out of the box value based on published literature.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Max Position Embeddings\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n num_attention_heads:\n ui_display_name: null\n expected_impact: 1\n num_hidden_layers:\n ui_display_name: null\n expected_impact: 1\n pad_token_id:\n ui_display_name: null\n expected_impact: 1\n position_embedding_type:\n ui_display_name: null\n expected_impact: 1\n pretrained_kwargs:\n ui_display_name: null\n expected_impact: 1\n pretrained_model_name_or_path:\n ui_display_name: null\n expected_impact: 2\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n saved_weights_in_checkpoint:\n default_value_reasoning:\n The weights of the encoder are not necessarily saved\n in the checkpoint. The user has to save them first.\n description_implications:\n The memory footprint for some of these encoders\n can be large.\n internal_only: true\n related_parameters:\n - skip_save_model\n suggested_values:\n - false\n suggested_values_reasoning:\n Some of these encoders are large, so it might\n be better to load them as needed, especially if 1. they're not used frequently\n 2. the user doesn't have a lot of storage.\n ui_display_name: null\n trainable:\n expected_impact: 3\n ui_display_name: null\n type_vocab_size:\n ui_display_name: null\n expected_impact: 1\n use_pretrained:\n default_value_reasoning:\n By default, the model is initialized as a pretrained\n model.\n description_implications:\n Pretrained models have typically already learned\n features that are difficult to learn from scratch. They are particularly\n beneficial when training on small amounts of data.\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/transfer-learning-for-deep-learning/\n related_parameters:\n - trainable, pretrained_model_name, pretrained_model_name_or_path, pretrained_kwargs\n suggested_values:\n - false\n suggested_values_reasoning:\n If you have a large amount of data and/or you\n have data that differs from the typical distribution, then it might be\n worth training the model from scratch.\n ui_display_name: Use Pretrained\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n vocab_size:\n internal_only: true\n ui_display_name: Not displayed\nBagEmbedWeighted:\n type:\n short_description: Transforms feature to vector, maps to sparse or dense embeddings, then aggregates.\n long_description:\n The embed weighted encoder first transforms the element frequency vector to sparse integer\n lists, which are then mapped to either dense or sparse embeddings (one-hot encodings). Lastly,\n embeddings are aggregated as a weighted sum where each embedding is multiplied by its respective\n element's frequency. Inputs are of size b while outputs are of size b x h where b is the batch\n size and h is the dimensionality of the embeddings.\n activation:\n default_value_reasoning:\n The Rectified Linear Units (ReLU) function is the\n standard activation function used for adding non-linearity. It is simple,\n fast, and empirically works well (https://arxiv.org/abs/1803.08375).\n description_implications:\n Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n expected_impact: 2\n suggested_values:\n The default value will work well in the majority of the\n cases\n ui_display_name: Activation\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n embedding_size:\n default_value_reasoning: Not too big, not too small.\n description_implications:\n 'An embedding is a relatively low-dimensional space\n that is used to translate high-dimensional vectors like words, which can\n have a large vocbulary size. Ideally, after an embedding is trained, it\n captures some of the semantics of the input by placing semantically similar\n inputs close together in the embedding space.\n\n\n In most cases, the embedding size is chosen empirically, by trial and\n error. From https://www.amazon.com/dp/1098115783, \"one rule of thumb is\n to use the fourth root of the total number of unique categorical elements\n while another is that the embedding dimension should be approximately\n 1.6 times the square root of the number of unique elements in the category,\n and no less than 600.\"\n\n\n Increasing the embedding size may cause the model to train more slowly,\n but the higher dimensionality can also improve overall quality.'\n expected_impact: 3\n literature_references:\n - https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture\n suggested_values: 1.6 * sqrt(vocab_size)\n suggested_values_reasoning:\n Rule of thumb suggested by a deep learning textbook.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Embedding Size\n embeddings_on_cpu:\n default_value_reasoning:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access.\n description_implications:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access. However, in some\n cases when the vocabulary size is very large, the full embedding matrix\n may be really big and unwieldy to have in GPU memory. This parameter forces\n the placement of the embedding matrix in regular memory and the CPU is\n used to access them. This may slow down training due to additional data\n transfer between CPU and GPU memory, but can lead to healthier GPU memory\n resource usage.\n expected_impact: 1\n suggested_values:\n - false\n suggested_values_reasoning:\n If GPU memory is not a constraint, having embeddings\n stored and accessed within the GPU is faster.\n ui_display_name: Embeddings on CPU\n embeddings_trainable:\n default_value_reasoning:\n If trained from scratch, embedding vectors are typically\n learned alongside the rest of the model.\n description_implications:\n Typically this value is only set to False if pre-trained\n embeddings are uploaded. Even then, it is reasonable to leave it as True\n in order to fine-tune the embeddings.\n expected_impact: 1\n related_parameters:\n - embedding_size, representation, pretrained_embeddings\n ui_display_name: (under Embeddings header) Trainable?\n fc_layers:\n default_value_reasoning:\n By default the stack is built by using num_fc_layers,\n output_size, use_bias, weights_initializer, bias_initializer, norm, norm_params,\n activation, dropout. When a list of dictionaries is provided, the stack\n is built following the parameters of each dict for building each layer.\n description_implications:\n The more layers that are specified the deeper and\n higher capacity the model will be. This makes it possible to potentially\n achieve better performance when a big anough amount of data is provided,\n but also makes the model more computationally expensive and potentially\n more prone to overfitting.\n example_value:\n - dropout: 0.1\n output_size: 128\n - norm: layer\n output_size: 64\n expected_impact: 1\n related_parameters:\n - output_size\n - use_bias\n - weights_initializer\n - bias_initializer\n - norm\n - norm_params\n - activation\n - dropout\n suggested_values_reasoning:\n It is easier to define a stack of fully connected\n layers by just specifying num_fc_layers, output_size and the other individual\n parameters. It will create a stack of layers with identical properties.\n Use this parameter only if you need a fine grained level of control of\n each individual layer in the stack.\n ui_display_name: Fully Connected Layers\n force_embedding_size:\n default_value_reasoning:\n It is not often the case that the user has a strict\n need for using an embedding size that should be larger than the vocabulary\n size.\n description_implications:\n Should only be True if the user has a strict need\n for using an embedding size that should be larger than the vocabulary\n size. For example, there may be size requirements across multiple features\n imposed by downstream modules like the ComparatorCombiner.\n expected_impact: 1\n related_parameters:\n - embedding_size\n suggested_values:\n - false\n suggested_values_reasoning: True for advanced usage only.\n ui_display_name: Force Embedding Size\n norm:\n default_value_reasoning:\n While batch normalization and layer normalization\n usually lead to improvements, it can be useful to start with fewer bells\n and whistles.\n description_implications:\n Normalization helps stabilize the learning process\n and can have a regularizing effect that can help with generalization.\n It's often suggested that with normalization, you can use a higher learning\n rate.\n example_value:\n - batch\n expected_impact: 2\n literature_references:\n - https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/\n related_parameters:\n - norm_params\n suggested_values: '\"batch\" or \"layer\"'\n suggested_values_reasoning:\n Normalization tries to solve \"internal covariate\n shift\" that comes from the changing distributions of the inputs to layers\n deep in the network when weights are updated. For example, batch normalization\n standardizes the inputs to a layer for each mini-batch. Try out different\n normalizations to see if that helps with training stability\n ui_display_name: Normalization Type\n norm_params:\n default_value_reasoning:\n The default parameters that come with Torch's implementation\n of these normalization types are a trusted starting point.\n description_implications:\n There are a variety of ways a certain set of parameters\n specificed could influence performance here. Broadly speaking the different\n values passed in here allow for different levels of smoothness to be observed\n in the learning curves. Since setting this parameters depends on the type\n of `norm` set, see [BatchNorm2d](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html)\n for more information on the parameters to set for batch normalization,\n and see [LayerNorm](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html)\n for more information on the parameters to set for layer normalization.\n example_value:\n - affine: false\n momentum: 0.2\n num_features: 100\n expected_impact: 1\n literature_references:\n - \"For BatchNorm2d: https://arxiv.org/abs/1502.03167\n\n\n For LayerNorm: https://arxiv.org/abs/1607.06450\"\n related_parameters:\n - \"`norm`\"\n suggested_values: Depends on the type of `norm` set.\n suggested_values_reasoning: \"NO\"\n ui_display_name: Normalization Parameters\n num_fc_layers:\n default_value_reasoning:\n The encoder already has learnable parameters.Sometimes\n the default is 1 for modules where the FC stack is used for shape management,\n or the only source of learnable parameters.\n description_implications:\n Increasing num_fc_layers will increase the capacity\n of the model. The model will be slower to train, and there's a higher\n risk of overfitting.\n example_value:\n - 1\n expected_impact: 1\n other_information:\n Not all modules that have fc_layers also have an accompanying\n num_fc_layers parameter. Where both are present, fc_layers takes precedent\n over num_fc_layers. Specifying num_fc_layers alone uses fully connected\n layers that are configured by the defaults in FCStack.\n related_parameters:\n - fc_layers\n suggested_values: 0-1\n suggested_values_reasoning:\n The full model likely contains many learnable\n parameters. Consider starting with very few, or without any additional\n fully connected layers and add them if you observe evidence of limited\n model capacity. Sometimes the default is 1 for modules where the FC stack\n is used for shape management, or the only source of learnable parameters.\n ui_display_name: Number of Fully Connected Layers\n output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 10 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\n pretrained_embeddings:\n default_value_reasoning:\n Embeddings are commonly trained from scratch, or\n incorporated as part of a pre-trained model package.\n description_implications:\n If pretrained embeddings are specified, then the\n model may have a head start in its representation of various input entities.\n example_value:\n - ~/Downloads/glove.6B.100d.txt\n expected_impact: 0\n related_parameters:\n - embedding_size, embeddings_trainable\n ui_display_name: Pretrained embeddings path\n representation:\n default_value_reasoning:\n Trainable, randomly initialized embedding vectors\n often lead to more subtle representations of input entities than one-hot\n vectors.\n description_implications:\n If set to sparse, the representations for input\n entities are fixed as one-hot vectors. This leads to less flexible representations\n for input entities, but could lead to faster training since there are\n less learnable parameters.\n expected_impact: 1\n other_information: \"\"\n related_parameters:\n - embedding_size, embeddings_trainable, pretrained_embeddings\n ui_display_name: Representation approach\n use_bias:\n default_value_reasoning:\n \"Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to use bias terms.\n\n\n Batch Normalization, however, adds a trainable shift parameter which is\n added to the activation. When Batch Normalization is used in a layer,\n bias terms are redundant and may be removed.\"\n description_implications:\n Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to leave this parameter set to True.\n example_value:\n - true\n expected_impact: 1\n other_information:\n If fc_layers is not specified, or use_bias is not specified\n for individual layers, the value of use_bias will be used as the default\n for all layers.\n related_parameters:\n - bias_initializer, fc_layers\n suggested_values:\n - true\n ui_display_name: Use Bias\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n weights_initializer:\n default_value_reasoning: Taken from published [literature](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\n\n\n Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\nCTRL:\n type:\n short_description: Language model trained to condition on control codes that govern style, content and task-specific behavior.\n long_description:\n The `ctrl` encoder loads a pretrained [CTRL](https://arxiv.org/abs/1909.05858) (default `ctrl`) model using the Hugging\n Face transformers package. CTRL is a conditional transformer language model trained to condition on control codes that\n govern style, content, and task-specific behavior.\n literature_references:\n - https://arxiv.org/abs/1909.05858\n compute_tier: 2\n attn_pdrop:\n ui_display_name: null\n dff:\n ui_display_name: null\n embd_pdrop:\n ui_display_name: null\n initializer_range:\n description_implications:\n There is an ideal value for this variable that doesn't\n lead to the outputs of these matrices to vanish or explode\n example_value:\n - 0.02\n expected_impact: 1\n other_information: Must be greater than 0\n related_parameters:\n - weights_initializer\n suggested_values: 0.01-0.05\n suggested_values_reasoning:\n Large values will likely lead to very large outputs.\n Small values will lead to vanishing outputs.\n ui_display_name: null\n layer_norm_epsilon:\n ui_display_name: null\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n n_ctx:\n ui_display_name: null\n n_embd:\n ui_display_name: null\n n_head:\n ui_display_name: null\n n_layer:\n ui_display_name: null\n n_positions:\n ui_display_name: null\n pretrained_kwargs:\n ui_display_name: null\n pretrained_model_name_or_path:\n ui_display_name: null\n expected_impact: 2\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n resid_pdrop:\n ui_display_name: null\n saved_weights_in_checkpoint:\n default_value_reasoning:\n The weights of the encoder are not necessarily saved\n in the checkpoint. The user has to save them first.\n description_implications:\n The memory footprint for some of these encoders\n can be large.\n internal_only: true\n related_parameters:\n - skip_save_model\n suggested_values:\n - false\n suggested_values_reasoning:\n Some of these encoders are large, so it might\n be better to load them as needed, especially if 1. they're not used frequently\n 2. the user doesn't have a lot of storage.\n ui_display_name: null\n trainable:\n expected_impact: 3\n ui_display_name: null\n use_pretrained:\n default_value_reasoning:\n By default, the model is initialized as a pretrained\n model.\n description_implications:\n Pretrained models have typically already learned\n features that are difficult to learn from scratch. They are particularly\n beneficial when training on small amounts of data.\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/transfer-learning-for-deep-learning/\n related_parameters:\n - trainable, pretrained_model_name, pretrained_model_name_or_path, pretrained_kwargs\n suggested_values:\n - false\n suggested_values_reasoning:\n If you have a large amount of data and/or you\n have data that differs from the typical distribution, then it might be\n worth training the model from scratch.\n ui_display_name: Use Pretrained\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n vocab_size:\n internal_only: true\n ui_display_name: Not displayed\nCamemBERT:\n type:\n short_description: Language model trained on large French text corpus.\n long_description:\n The `camembert` encoder loads a pretrained [CamemBERT](https://arxiv.org/abs/1911.03894)\n (default `jplu/tf-camembert-base`) model using the Hugging Face transformers package. CamemBERT is pre-trained on a\n large French language web-crawled text corpus.\n literature_references:\n - https://arxiv.org/abs/1911.03894\n compute_tier: 2\n attention_probs_dropout_prob:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 2\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - classifier_dropout, hidden_dropout_prob\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: attention_probs_dropout_prob\n classifier_dropout:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 2\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - attention_probs_dropout_prob, hidden_dropout_prob\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: classifier_dropout\n gradient_checkpointing:\n ui_display_name: null\n hidden_act:\n default_value_reasoning: Taken from huggingface.\n description_implications:\n Changing this activation function will only affect\n the feed-forward layers of the transformer.\n example_value:\n - relu\n expected_impact: 1\n literature_references:\n - \"[Relevant StackOverflow discussion](https://ai.stackexchange.com/questions/30341/why-does-a-transformer-not-use-an-activation-function-following-the-multi-head-a)\"\n suggested_values: gelu\n suggested_values_reasoning: Taken from huggingface defaults.\n ui_display_name: Hidden Layer Activation\n hidden_dropout_prob:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 2\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - \"attention_probs_dropout_prob, \\nclassifier_dropout\"\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: hidden_dropout_prob\n hidden_size:\n default_value_reasoning: Huggingface default.\n description_implications:\n Increasing the hidden size makes the model larger\n and slower to train, increases the model's capacity to capture more complexity.\n It also increases the chance of overfitting.\n expected_impact: 1\n suggested_values: 10 - 2048\n suggested_values_reasoning:\n Increasing the hidden size makes sense if the\n model is underfitting. It's useful to train both smaller and larger models\n to see how model capacity affects performance. This should only be explored\n after the architecture of the model has been settled.\n ui_display_name: Hidden Size\n initializer_range:\n description_implications:\n There is an ideal value for this variable that doesn't\n lead to the outputs of these matrices to vanish or explode\n example_value:\n - 0.02\n expected_impact: 1\n other_information: Must be greater than 0\n related_parameters:\n - weights_initializer\n suggested_values: 0.01-0.05\n suggested_values_reasoning:\n Large values will likely lead to very large outputs.\n Small values will lead to vanishing outputs.\n ui_display_name: null\n intermediate_size:\n ui_display_name: null\n layer_norm_eps:\n ui_display_name: null\n max_position_embeddings:\n default_value_reasoning: Taken from huggingface.\n description_implications:\n The size of the position embeddings table. This typically coincides with the\n maximum sequence length this model might ever be used with. Typically set this\n to something large just in case (e.g. 512, 1024, 2048).\n expected_impact: 2\n suggested_values: 512\n suggested_values_reasoning:\n Out of the box value based on published literature.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Max Position Embeddings\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n num_attention_heads:\n ui_display_name: null\n num_hidden_layers:\n ui_display_name: null\n pad_token_id:\n ui_display_name: null\n position_embedding_type:\n ui_display_name: null\n pretrained_kwargs:\n ui_display_name: null\n pretrained_model_name_or_path:\n ui_display_name: null\n expected_impact: 2\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n saved_weights_in_checkpoint:\n default_value_reasoning:\n The weights of the encoder are not necessarily saved\n in the checkpoint. The user has to save them first.\n description_implications:\n The memory footprint for some of these encoders\n can be large.\n internal_only: true\n related_parameters:\n - skip_save_model\n suggested_values:\n - false\n suggested_values_reasoning:\n Some of these encoders are large, so it might\n be better to load them as needed, especially if 1. they're not used frequently\n 2. the user doesn't have a lot of storage.\n ui_display_name: null\n trainable:\n expected_impact: 3\n ui_display_name: null\n type_vocab_size:\n ui_display_name: null\n use_pretrained:\n default_value_reasoning:\n By default, the model is initialized as a pretrained\n model.\n description_implications:\n Pretrained models have typically already learned\n features that are difficult to learn from scratch. They are particularly\n beneficial when training on small amounts of data.\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/transfer-learning-for-deep-learning/\n related_parameters:\n - trainable, pretrained_model_name, pretrained_model_name_or_path, pretrained_kwargs\n suggested_values:\n - false\n suggested_values_reasoning:\n If you have a large amount of data and/or you\n have data that differs from the typical distribution, then it might be\n worth training the model from scratch.\n ui_display_name: Use Pretrained\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n vocab_size:\n internal_only: true\n ui_display_name: Not displayed\nCategoricalEmbed:\n type:\n short_description: Maps the categorical feature to a dense embedding.\n long_description:\n The dense encoder maps to a dense embedding and is returned as outputs of size `b x h`,\n where `b` is the batch size and `h` is the dimensionality of the embeddings.\n dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n embedding_initializer:\n default_value_reasoning:\n According to https://arxiv.org/abs/1711.09160, choice\n of embedding initialization is not important as long as the variance is\n kept reasonably low.\n description_implications:\n According to https://arxiv.org/abs/1711.09160, choice\n of embedding initialization is not important as long as the variance is\n kept reasonably low.\n example_value:\n - kaiming\n expected_impact: 1\n literature_references:\n - https://arxiv.org/abs/1711.09160\n suggested_values: kaiming\n suggested_values_reasoning: https://discuss.huggingface.co/t/state-of-the-art-technique-for-initializing-embedding-matrix/326\n ui_display_name: Embedding Initialization\n embedding_size:\n default_value_reasoning: Not too big, not too small.\n description_implications:\n 'An embedding is a relatively low-dimensional space\n that is used to translate high-dimensional vectors like words, which can\n have a large vocbulary size. Ideally, after an embedding is trained, it\n captures some of the semantics of the input by placing semantically similar\n inputs close together in the embedding space.\n\n\n In most cases, the embedding size is chosen empirically, by trial and\n error. From https://www.amazon.com/dp/1098115783, \"one rule of thumb is\n to use the fourth root of the total number of unique categorical elements\n while another is that the embedding dimension should be approximately\n 1.6 times the square root of the number of unique elements in the category,\n and no less than 600.\"\n\n\n Increasing the embedding size may cause the model to train more slowly,\n but the higher dimensionality can also improve overall quality.'\n expected_impact: 3\n literature_references:\n - https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture\n suggested_values: 1.6 * sqrt(vocab_size)\n suggested_values_reasoning:\n Rule of thumb suggested by a deep learning textbook.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Embedding Size\n embeddings_on_cpu:\n default_value_reasoning:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access.\n description_implications:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access. However, in some\n cases when the vocabulary size is very large, the full embedding matrix\n may be really big and unwieldy to have in GPU memory. This parameter forces\n the placement of the embedding matrix in regular memory and the CPU is\n used to access them. This may slow down training due to additional data\n transfer between CPU and GPU memory, but can lead to healthier GPU memory\n resource usage.\n expected_impact: 1\n suggested_values:\n - false\n suggested_values_reasoning:\n If GPU memory is not a constraint, having embeddings\n stored and accessed within the GPU is faster.\n ui_display_name: Embeddings on CPU\n embeddings_trainable:\n ui_display_name: null\n expected_impact: 1\n pretrained_embeddings:\n ui_display_name: null\n expected_impact: 0\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\nCategoricalSparse:\n type:\n short_description: Maps the categorical feature to a sparse embedding.\n long_description:\n The sparse encoder maps to a sparse embedding (one-hot encodings) and is returned as outputs of\n size `b x h`, where `b` is the batch size and `h` is the dimensionality of the embeddings.\n dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n embedding_initializer:\n default_value_reasoning:\n According to https://arxiv.org/abs/1711.09160, choice\n of embedding initialization is not important as long as the variance is\n kept reasonably low.\n description_implications:\n According to https://arxiv.org/abs/1711.09160, choice\n of embedding initialization is not important as long as the variance is\n kept reasonably low.\n example_value:\n - kaiming\n expected_impact: 1\n literature_references:\n - https://arxiv.org/abs/1711.09161\n suggested_values: kaiming\n suggested_values_reasoning: https://discuss.huggingface.co/t/state-of-the-art-technique-for-initializing-embedding-matrix/327\n ui_display_name: Embedding Initialization\n embedding_size:\n default_value_reasoning: Not too big, not too small.\n description_implications:\n 'An embedding is a relatively low-dimensional space\n that is used to translate high-dimensional vectors like words, which can\n have a large vocbulary size. Ideally, after an embedding is trained, it\n captures some of the semantics of the input by placing semantically similar\n inputs close together in the embedding space.\n\n\n In most cases, the embedding size is chosen empirically, by trial and\n error. From https://www.amazon.com/dp/1098115783, \"one rule of thumb is\n to use the fourth root of the total number of unique categorical elements\n while another is that the embedding dimension should be approximately\n 1.6 times the square root of the number of unique elements in the category,\n and no less than 600.\"\n\n\n Increasing the embedding size may cause the model to train more slowly,\n but the higher dimensionality can also improve overall quality.'\n expected_impact: 3\n literature_references:\n - https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture\n suggested_values: 1.6 * sqrt(vocab_size)\n suggested_values_reasoning:\n Rule of thumb suggested by a deep learning textbook.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Embedding Size\n embeddings_on_cpu:\n default_value_reasoning:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access.\n description_implications:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access. However, in some\n cases when the vocabulary size is very large, the full embedding matrix\n may be really big and unwieldy to have in GPU memory. This parameter forces\n the placement of the embedding matrix in regular memory and the CPU is\n used to access them. This may slow down training due to additional data\n transfer between CPU and GPU memory, but can lead to healthier GPU memory\n resource usage.\n expected_impact: 1\n suggested_values:\n - false\n suggested_values_reasoning:\n If GPU memory is not a constraint, having embeddings\n stored and accessed within the GPU is faster.\n ui_display_name: Embeddings on CPU\n embeddings_trainable:\n ui_display_name: null\n expected_impact: 1\n pretrained_embeddings:\n ui_display_name: null\n expected_impact: 0\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\nDateEmbed:\n type:\n short_description: Embeds the date elements passes them through fully connected layers.\n long_description:\n The Embed encoder passes the year through a fully connected layer of one neuron and embeds all\n other elements for the date, concatenates them and passes the concatenated representation\n through fully connected layers.\n activation:\n default_value_reasoning:\n The Rectified Linear Units (ReLU) function is the\n standard activation function used for adding non-linearity. It is simple,\n fast, and empirically works well (https://arxiv.org/abs/1803.08375).\n description_implications:\n Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n expected_impact: 2\n suggested_values:\n The default value will work well in the majority of the\n cases\n ui_display_name: Activation\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n embedding_size:\n default_value_reasoning: Not too big, not too small.\n description_implications:\n 'An embedding is a relatively low-dimensional space\n that is used to translate high-dimensional vectors like words, which can\n have a large vocbulary size. Ideally, after an embedding is trained, it\n captures some of the semantics of the input by placing semantically similar\n inputs close together in the embedding space.\n\n\n In most cases, the embedding size is chosen empirically, by trial and\n error. From https://www.amazon.com/dp/1098115783, \"one rule of thumb is\n to use the fourth root of the total number of unique categorical elements\n while another is that the embedding dimension should be approximately\n 1.6 times the square root of the number of unique elements in the category,\n and no less than 600.\"\n\n\n Increasing the embedding size may cause the model to train more slowly,\n but the higher dimensionality can also improve overall quality.'\n expected_impact: 3\n literature_references:\n - https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture\n suggested_values: 1.6 * sqrt(vocab_size)\n suggested_values_reasoning:\n Rule of thumb suggested by a deep learning textbook.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Embedding Size\n embeddings_on_cpu:\n default_value_reasoning:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access.\n description_implications:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access. However, in some\n cases when the vocabulary size is very large, the full embedding matrix\n may be really big and unwieldy to have in GPU memory. This parameter forces\n the placement of the embedding matrix in regular memory and the CPU is\n used to access them. This may slow down training due to additional data\n transfer between CPU and GPU memory, but can lead to healthier GPU memory\n resource usage.\n expected_impact: 1\n suggested_values:\n - false\n suggested_values_reasoning:\n If GPU memory is not a constraint, having embeddings\n stored and accessed within the GPU is faster.\n ui_display_name: Embeddings on CPU\n fc_layers:\n default_value_reasoning:\n By default the stack is built by using num_fc_layers,\n output_size, use_bias, weights_initializer, bias_initializer, norm, norm_params,\n activation, dropout. When a list of dictionaries is provided, the stack\n is built following the parameters of each dict for building each layer.\n description_implications:\n The more layers that are specified the deeper and\n higher capacity the model will be. This makes it possible to potentially\n achieve better performance when a big anough amount of data is provided,\n but also makes the model more computationally expensive and potentially\n more prone to overfitting.\n example_value:\n - dropout: 0.1\n output_size: 128\n - norm: layer\n output_size: 64\n expected_impact: 1\n related_parameters:\n - output_size\n - use_bias\n - weights_initializer\n - bias_initializer\n - norm\n - norm_params\n - activation\n - dropout\n suggested_values_reasoning:\n It is easier to define a stack of fully connected\n layers by just specifying num_fc_layers, output_size and the other individual\n parameters. It will create a stack of layers with identical properties.\n Use this parameter only if you need a fine grained level of control of\n each individual layer in the stack.\n ui_display_name: Fully Connected Layers\n norm:\n default_value_reasoning:\n While batch normalization and layer normalization\n usually lead to improvements, it can be useful to start with fewer bells\n and whistles.\n description_implications:\n Normalization helps stabilize the learning process\n and can have a regularizing effect that can help with generalization.\n It's often suggested that with normalization, you can use a higher learning\n rate.\n example_value:\n - batch\n expected_impact: 2\n literature_references:\n - https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/\n related_parameters:\n - norm_params\n suggested_values: '\"batch\" or \"layer\"'\n suggested_values_reasoning:\n Normalization tries to solve \"internal covariate\n shift\" that comes from the changing distributions of the inputs to layers\n deep in the network when weights are updated. For example, batch normalization\n standardizes the inputs to a layer for each mini-batch. Try out different\n normalizations to see if that helps with training stability\n ui_display_name: Normalization Type\n norm_params:\n default_value_reasoning:\n The default parameters that come with Torch's implementation\n of these normalization types are a trusted starting point.\n description_implications:\n There are a variety of ways a certain set of parameters\n specificed could influence performance here. Broadly speaking the different\n values passed in here allow for different levels of smoothness to be observed\n in the learning curves. Since setting this parameters depends on the type\n of `norm` set, see [BatchNorm2d](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html)\n for more information on the parameters to set for batch normalization,\n and see [LayerNorm](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html)\n for more information on the parameters to set for layer normalization.\n example_value:\n - affine: false\n momentum: 0.2\n num_features: 100\n expected_impact: 1\n literature_references:\n - \"For BatchNorm2d: https://arxiv.org/abs/1502.03167\n\n\n For LayerNorm: https://arxiv.org/abs/1607.06450\"\n related_parameters:\n - \"`norm`\"\n suggested_values: Depends on the type of `norm` set.\n suggested_values_reasoning: \"NO\"\n ui_display_name: Normalization Parameters\n num_fc_layers:\n default_value_reasoning:\n The encoder already has learnable parameters.Sometimes\n the default is 1 for modules where the FC stack is used for shape management,\n or the only source of learnable parameters.\n description_implications:\n Increasing num_fc_layers will increase the capacity\n of the model. The model will be slower to train, and there's a higher\n risk of overfitting.\n example_value:\n - 1\n expected_impact: 1\n other_information:\n Not all modules that have fc_layers also have an accompanying\n num_fc_layers parameter. Where both are present, fc_layers takes precedent\n over num_fc_layers. Specifying num_fc_layers alone uses fully connected\n layers that are configured by the defaults in FCStack.\n related_parameters:\n - fc_layers\n suggested_values: 0-1\n suggested_values_reasoning:\n The full model likely contains many learnable\n parameters. Consider starting with very few, or without any additional\n fully connected layers and add them if you observe evidence of limited\n model capacity. Sometimes the default is 1 for modules where the FC stack\n is used for shape management, or the only source of learnable parameters.\n ui_display_name: Number of Fully Connected Layers\n output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 10 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\n use_bias:\n default_value_reasoning:\n \"Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to use bias terms.\n\n\n Batch Normalization, however, adds a trainable shift parameter which is\n added to the activation. When Batch Normalization is used in a layer,\n bias terms are redundant and may be removed.\"\n description_implications:\n Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to leave this parameter set to True.\n example_value:\n - true\n expected_impact: 1\n other_information:\n If fc_layers is not specified, or use_bias is not specified\n for individual layers, the value of use_bias will be used as the default\n for all layers.\n related_parameters:\n - bias_initializer, fc_layers\n suggested_values:\n - true\n ui_display_name: Use Bias\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\n\n\n Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\nDateWave:\n type:\n short_description: Embeds the date elements by taking the cosine of their value before passing through fully connected layers.\n long_description:\n The Wave encoder passes the year through a fully connected layer of one neuron and represents\n all other elements for the date by taking the cosine of their value with a different period (12\n for months, 31 for days, etc.), concatenates them and passes the concatenated representation\n through fully connected layers.\n activation:\n default_value_reasoning:\n The Rectified Linear Units (ReLU) function is the\n standard activation function used for adding non-linearity. It is simple,\n fast, and empirically works well (https://arxiv.org/abs/1803.08375).\n description_implications:\n Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n expected_impact: 2\n suggested_values:\n The default value will work well in the majority of the\n cases\n ui_display_name: Activation\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n fc_layers:\n default_value_reasoning:\n By default the stack is built by using num_fc_layers,\n output_size, use_bias, weights_initializer, bias_initializer, norm, norm_params,\n activation, dropout. When a list of dictionaries is provided, the stack\n is built following the parameters of each dict for building each layer.\n description_implications:\n The more layers that are specified the deeper and\n higher capacity the model will be. This makes it possible to potentially\n achieve better performance when a big anough amount of data is provided,\n but also makes the model more computationally expensive and potentially\n more prone to overfitting.\n example_value:\n - dropout: 0.1\n output_size: 128\n - norm: layer\n output_size: 64\n expected_impact: 1\n related_parameters:\n - output_size\n - use_bias\n - weights_initializer\n - bias_initializer\n - norm\n - norm_params\n - activation\n - dropout\n suggested_values_reasoning:\n It is easier to define a stack of fully connected\n layers by just specifying num_fc_layers, output_size and the other individual\n parameters. It will create a stack of layers with identical properties.\n Use this parameter only if you need a fine grained level of control of\n each individual layer in the stack.\n ui_display_name: Fully Connected Layers\n norm:\n default_value_reasoning:\n While batch normalization and layer normalization\n usually lead to improvements, it can be useful to start with fewer bells\n and whistles.\n description_implications:\n Normalization helps stabilize the learning process\n and can have a regularizing effect that can help with generalization.\n It's often suggested that with normalization, you can use a higher learning\n rate.\n example_value:\n - batch\n expected_impact: 2\n literature_references:\n - https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/\n related_parameters:\n - norm_params\n suggested_values: '\"batch\" or \"layer\"'\n suggested_values_reasoning:\n Normalization tries to solve \"internal covariate\n shift\" that comes from the changing distributions of the inputs to layers\n deep in the network when weights are updated. For example, batch normalization\n standardizes the inputs to a layer for each mini-batch. Try out different\n normalizations to see if that helps with training stability\n ui_display_name: Normalization Type\n norm_params:\n default_value_reasoning:\n The default parameters that come with Torch's implementation\n of these normalization types are a trusted starting point.\n description_implications:\n There are a variety of ways a certain set of parameters\n specificed could influence performance here. Broadly speaking the different\n values passed in here allow for different levels of smoothness to be observed\n in the learning curves. Since setting this parameters depends on the type\n of `norm` set, see [BatchNorm2d](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html)\n for more information on the parameters to set for batch normalization,\n and see [LayerNorm](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html)\n for more information on the parameters to set for layer normalization.\n example_value:\n - affine: false\n momentum: 0.2\n num_features: 100\n expected_impact: 1\n literature_references:\n - \"For BatchNorm2d: https://arxiv.org/abs/1502.03167\n\n\n For LayerNorm: https://arxiv.org/abs/1607.06450\"\n related_parameters:\n - \"`norm`\"\n suggested_values: Depends on the type of `norm` set.\n suggested_values_reasoning: \"NO\"\n ui_display_name: Normalization Parameters\n num_fc_layers:\n default_value_reasoning:\n The encoder already has learnable parameters.Sometimes\n the default is 1 for modules where the FC stack is used for shape management,\n or the only source of learnable parameters.\n description_implications:\n Increasing num_fc_layers will increase the capacity\n of the model. The model will be slower to train, and there's a higher\n risk of overfitting.\n example_value:\n - 1\n expected_impact: 1\n other_information:\n Not all modules that have fc_layers also have an accompanying\n num_fc_layers parameter. Where both are present, fc_layers takes precedent\n over num_fc_layers. Specifying num_fc_layers alone uses fully connected\n layers that are configured by the defaults in FCStack.\n related_parameters:\n - fc_layers\n suggested_values: 0-1\n suggested_values_reasoning:\n The full model likely contains many learnable\n parameters. Consider starting with very few, or without any additional\n fully connected layers and add them if you observe evidence of limited\n model capacity. Sometimes the default is 1 for modules where the FC stack\n is used for shape management, or the only source of learnable parameters.\n ui_display_name: Number of Fully Connected Layers\n output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 10 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\n use_bias:\n default_value_reasoning:\n \"Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to use bias terms.\n\n\n Batch Normalization, however, adds a trainable shift parameter which is\n added to the activation. When Batch Normalization is used in a layer,\n bias terms are redundant and may be removed.\"\n description_implications:\n Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to leave this parameter set to True.\n example_value:\n - true\n expected_impact: 1\n other_information:\n If fc_layers is not specified, or use_bias is not specified\n for individual layers, the value of use_bias will be used as the default\n for all layers.\n related_parameters:\n - bias_initializer, fc_layers\n suggested_values:\n - true\n ui_display_name: Use Bias\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\n\n\n Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\nDenseEncoder:\n type:\n short_description: Passes the raw numerical values through fully connected layers.\n long_description:\n The dense encoder passes the raw numerical values through fully connected layers. In this case\n the inputs of size `b` are transformed to size `b x h`.\n activation:\n default_value_reasoning:\n The Rectified Linear Units (ReLU) function is the\n standard activation function used for adding non-linearity. It is simple,\n fast, and empirically works well (https://arxiv.org/abs/1803.08375).\n description_implications:\n Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n expected_impact: 2\n suggested_values:\n The default value will work well in the majority of the\n cases\n ui_display_name: Activation\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n input_size:\n internal_only: true\n other_information: Internal Only\n related_parameters:\n - \"No\"\n ui_display_name: Not Displayed\n fc_layers:\n ui_display_name: null\n expected_impact: 1\n norm:\n default_value_reasoning:\n While batch normalization and layer normalization\n usually lead to improvements, it can be useful to start with fewer bells\n and whistles.\n description_implications:\n Normalization helps stabilize the learning process\n and can have a regularizing effect that can help with generalization.\n It's often suggested that with normalization, you can use a higher learning\n rate.\n example_value:\n - batch\n expected_impact: 2\n literature_references:\n - https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/\n related_parameters:\n - norm_params\n suggested_values: '\"batch\" or \"layer\"'\n suggested_values_reasoning:\n Normalization tries to solve \"internal covariate\n shift\" that comes from the changing distributions of the inputs to layers\n deep in the network when weights are updated. For example, batch normalization\n standardizes the inputs to a layer for each mini-batch. Try out different\n normalizations to see if that helps with training stability\n ui_display_name: Normalization Type\n norm_params:\n default_value_reasoning:\n The default parameters that come with Torch's implementation\n of these normalization types are a trusted starting point.\n description_implications:\n There are a variety of ways a certain set of parameters\n specificed could influence performance here. Broadly speaking the different\n values passed in here allow for different levels of smoothness to be observed\n in the learning curves. Since setting this parameters depends on the type\n of `norm` set, see [BatchNorm2d](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html)\n for more information on the parameters to set for batch normalization,\n and see [LayerNorm](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html)\n for more information on the parameters to set for layer normalization.\n example_value:\n - affine: false\n momentum: 0.2\n num_features: 100\n expected_impact: 1\n literature_references:\n - \"For BatchNorm2d: https://arxiv.org/abs/1502.03167\n\n\n For LayerNorm: https://arxiv.org/abs/1607.06450\"\n related_parameters:\n - \"`norm`\"\n suggested_values: Depends on the type of `norm` set.\n suggested_values_reasoning: \"NO\"\n ui_display_name: Normalization Parameters\n num_layers:\n default_value_reasoning:\n The ideal number of layers depends on the data. For\n many data types, one layer is sufficient.\n description_implications:\n \"Increasing the number of layers may improve model\n performance by allowing the model to synthesize learned features derived\n from the original input. If the input is simple, ex. a category with a\n few options, increasing the number of layers has no benefit. For more\n complex inputs, additional layers add more 'processing power' to extract\n useful information from the input.\n\n\n However, more layers will increase training time and may reduce accuracy\n due to overfitting.\"\n example_value:\n - 1\n expected_impact: 3\n other_information:\n If you have multiple input features, varying the number\n of layers in the combiner or output feature decoder will have more impact.\n related_parameters:\n - layers\n suggested_values: 1-3\n suggested_values_reasoning:\n Increasing the number of layers may improve encoder\n performance. However, more layers will increase training time and may\n cause overfitting. Small numbers of layers usually work best.\n ui_display_name: Number of Layers\n output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 10 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\n use_bias:\n ui_display_name: null\n expected_impact: 1\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\n\n\n Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\nDistilBERT:\n type:\n short_description: A distilled version of BERT base that is 40% smaller and 60% faster with 95% of performance preserved.\n long_description:\n The `distilbert` encoder loads a pretrained [DistilBERT](https://medium.com/huggingface/distilbert-8cf3380435b5)\n (default `distilbert-base-uncased`) model using the Hugging Face transformers package. DistilBERT is a small, fast, cheap and light\n Transformer model trained by distilling BERT base. It has 40% less parameters than\n bert-base-uncased, runs 60% faster while preserving over 95% of BERT’s performances as measured\n on the GLUE language understanding benchmark.\n compute_tier: 2\n activation:\n default_value_reasoning:\n This is the default activation function used in the\n Distillbert huggingface implementation\n description_implications:\n Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n expected_impact: 2\n suggested_values:\n The default value will work well in the majority of the\n cases\n ui_display_name: Activation\n attention_dropout:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 2\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - dropout, qa_dropout, seq_classif_dropout\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: attention_dropout\n dim:\n ui_display_name: null\n dropout:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 2\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - \"attention_dropout,\n\n qa_dropout,\n\n seq_classif_dropout\"\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: dropout\n hidden_dim:\n ui_display_name: null\n initializer_range:\n description_implications:\n There is an ideal value for this variable that doesn't\n lead to the outputs of these matrices to vanish or explode\n example_value:\n - 0.02\n expected_impact: 1\n other_information: Must be greater than 0\n related_parameters:\n - weights_initializer\n suggested_values: 0.01-0.05\n suggested_values_reasoning:\n Large values will likely lead to very large outputs.\n Small values will lead to vanishing outputs.\n ui_display_name: null\n max_position_embeddings:\n default_value_reasoning: Taken from huggingface.\n description_implications:\n The size of the position embeddings table. This typically coincides with the\n maximum sequence length this model might ever be used with. Typically set this\n to something large just in case (e.g. 512, 1024, 2048).\n expected_impact: 2\n suggested_values: 512\n suggested_values_reasoning:\n Out of the box value based on published literature.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Max Position Embeddings\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n n_heads:\n ui_display_name: null\n n_layers:\n ui_display_name: null\n pretrained_kwargs:\n ui_display_name: null\n pretrained_model_name_or_path:\n ui_display_name: null\n expected_impact: 2\n qa_dropout:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 1\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - dropout, attention_dropout, seq_classif_dropout\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: qa_dropout\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n saved_weights_in_checkpoint:\n default_value_reasoning:\n The weights of the encoder are not necessarily saved\n in the checkpoint. The user has to save them first.\n description_implications:\n The memory footprint for some of these encoders\n can be large.\n internal_only: true\n related_parameters:\n - skip_save_model\n suggested_values:\n - false\n suggested_values_reasoning:\n Some of these encoders are large, so it might\n be better to load them as needed, especially if 1. they're not used frequently\n 2. the user doesn't have a lot of storage.\n ui_display_name: null\n seq_classif_dropout:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 1\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - \"dropout,\n\n attention_dropout,\n\n qa_dropout\"\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: seq_classif_dropout\n sinusoidal_pos_embds:\n ui_display_name: null\n trainable:\n expected_impact: 3\n ui_display_name: null\n use_pretrained:\n default_value_reasoning:\n By default, the model is initialized as a pretrained\n model.\n description_implications:\n Pretrained models have typically already learned\n features that are difficult to learn from scratch. They are particularly\n beneficial when training on small amounts of data.\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/transfer-learning-for-deep-learning/\n related_parameters:\n - trainable, pretrained_model_name, pretrained_model_name_or_path, pretrained_kwargs\n suggested_values:\n - false\n suggested_values_reasoning:\n If you have a large amount of data and/or you\n have data that differs from the typical distribution, then it might be\n worth training the model from scratch.\n ui_display_name: Use Pretrained\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n vocab_size:\n internal_only: true\n ui_display_name: Not displayed\nELECTRA:\n type:\n short_description: Transformer encoder that can be used to encode a sequence of tokens with little compute\n long_description:\n The `electra`` encoder loads a pretrained [ELECTRA](https://openreview.net/pdf?id=r1xMH1BtvB) model using the Hugging Face transformers package.\n ELECTRA is a new pretraining approach which trains two transformer models the generator and the\n discriminator. The generator’s role is to replace tokens in a sequence, and is therefore trained\n as a masked language model. The discriminator, which is the model we’re interested in, tries to\n identify which tokens were replaced by the generator in the sequence.\n literature_references:\n - https://openreview.net/pdf?id=r1xMH1BtvB\n compute_tier: 2\n attention_probs_dropout_prob:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 2\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - hidden_dropout_prob, classifier_dropout\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: attention_probs_dropout_prob\n classifier_dropout:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 2\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - hidden_dropout_prob, attention_probs_dropout_prob\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: classifier_dropout\n embedding_size:\n default_value_reasoning: Not too big, not too small.\n description_implications:\n 'An embedding is a relatively low-dimensional space\n that is used to translate high-dimensional vectors like words, which can\n have a large vocbulary size. Ideally, after an embedding is trained, it\n captures some of the semantics of the input by placing semantically similar\n inputs close together in the embedding space.\n\n\n In most cases, the embedding size is chosen empirically, by trial and\n error. From https://www.amazon.com/dp/1098115783, \"one rule of thumb is\n to use the fourth root of the total number of unique categorical elements\n while another is that the embedding dimension should be approximately\n 1.6 times the square root of the number of unique elements in the category,\n and no less than 600.\"\n\n\n Increasing the embedding size may cause the model to train more slowly,\n but the higher dimensionality can also improve overall quality.'\n expected_impact: 1\n literature_references:\n - https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture\n suggested_values: 1.6 * sqrt(vocab_size)\n suggested_values_reasoning:\n Rule of thumb suggested by a deep learning textbook.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Embedding Size\n hidden_act:\n default_value_reasoning: Taken from huggingface.\n description_implications:\n Changing this activation function will only affect\n the feed-forward layers of the transformer.\n example_value:\n - relu\n expected_impact: 1\n literature_references:\n - \"[Huggingface docs for ELECTRA config](https://huggingface.co/docs/transformers/model_doc/electra#transformers.ElectraConfig.hidden_act)\n\n\n [Relevant StackOverflow discussion](https://ai.stackexchange.com/questions/30341/why-does-a-transformer-not-use-an-activation-function-following-the-multi-head-a)\"\n suggested_values: gelu\n suggested_values_reasoning: Taken from huggingface defaults.\n ui_display_name: Hidden Layer Activation\n hidden_dropout_prob:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 2\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - \"attention_probs_dropout_prob,\n\n classifier_dropout\"\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: hidden_dropout_prob\n hidden_size:\n default_value_reasoning: Huggingface default.\n description_implications:\n Increasing the hidden size makes the model larger\n and slower to train, increases the model's capacity to capture more complexity.\n It also increases the chance of overfitting.\n expected_impact: 1\n suggested_values: 10 - 2048\n suggested_values_reasoning:\n Increasing the hidden size makes sense if the\n model is underfitting. It's useful to train both smaller and larger models\n to see how model capacity affects performance. This should only be explored\n after the architecture of the model has been settled.\n ui_display_name: Hidden Size\n initializer_range:\n description_implications:\n There is an ideal value for this variable that doesn't\n lead to the outputs of these matrices to vanish or explode\n example_value:\n - 0.02\n expected_impact: 1\n other_information: Must be greater than 0\n related_parameters:\n - weights_initializer\n suggested_values: 0.01-0.05\n suggested_values_reasoning:\n Large values will likely lead to very large outputs.\n Small values will lead to vanishing outputs.\n ui_display_name: null\n intermediate_size:\n ui_display_name: null\n layer_norm_eps:\n ui_display_name: null\n max_position_embeddings:\n default_value_reasoning: Taken from huggingface.\n description_implications:\n The size of the position embeddings table. This typically coincides with the\n maximum sequence length this model might ever be used with. Typically set this\n to something large just in case (e.g. 512, 1024, 2048).\n expected_impact: 2\n suggested_values: 512\n suggested_values_reasoning:\n Out of the box value based on published literature.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Max Position Embeddings\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n num_attention_heads:\n ui_display_name: null\n num_hidden_layers:\n ui_display_name: null\n position_embedding_type:\n ui_display_name: null\n pretrained_kwargs:\n ui_display_name: null\n pretrained_model_name_or_path:\n ui_display_name: null\n expected_impact: 2\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n saved_weights_in_checkpoint:\n default_value_reasoning:\n The weights of the encoder are not necessarily saved\n in the checkpoint. The user has to save them first.\n description_implications:\n The memory footprint for some of these encoders\n can be large.\n internal_only: true\n related_parameters:\n - skip_save_model\n suggested_values:\n - false\n suggested_values_reasoning:\n Some of these encoders are large, so it might\n be better to load them as needed, especially if 1. they're not used frequently\n 2. the user doesn't have a lot of storage.\n ui_display_name: null\n trainable:\n expected_impact: 3\n ui_display_name: null\n type_vocab_size:\n ui_display_name: null\n use_pretrained:\n default_value_reasoning:\n By default, the model is initialized as a pretrained\n model.\n description_implications:\n Pretrained models have typically already learned\n features that are difficult to learn from scratch. They are particularly\n beneficial when training on small amounts of data.\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/transfer-learning-for-deep-learning/\n related_parameters:\n - trainable, pretrained_model_name, pretrained_model_name_or_path, pretrained_kwargs\n suggested_values:\n - false\n suggested_values_reasoning:\n If you have a large amount of data and/or you\n have data that differs from the typical distribution, then it might be\n worth training the model from scratch.\n ui_display_name: Use Pretrained\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n vocab_size:\n internal_only: true\n ui_display_name: Not displayed\nFlauBERT:\n type:\n short_description: Language model with BERT related architecture trained on large French text corpus.\n long_description:\n The `flaubert`` encoder loads a pretrained [FlauBERT](https://arxiv.org/abs/1912.05372) (default `jplu/tf-flaubert-base-uncased``) model\n using the Hugging Face transformers package. FlauBERT has an architecture similar to BERT and is\n pre-trained on a large French language corpus.\n literature_references:\n - https://arxiv.org/abs/1912.05372\n compute_tier: 2\n asm:\n ui_display_name: null\n expected_impact: 1\n attention_dropout:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 1\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - dropout\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: attention_dropout\n bos_index:\n ui_display_name: null\n expected_impact: 1\n causal:\n ui_display_name: null\n expected_impact: 1\n dropout:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 2\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - attention_dropout\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: dropout\n emb_dim:\n ui_display_name: null\n expected_impact: 1\n embed_init_std:\n ui_display_name: null\n expected_impact: 1\n eos_index:\n ui_display_name: null\n expected_impact: 1\n gelu_activation:\n ui_display_name: null\n expected_impact: 1\n init_std:\n ui_display_name: null\n expected_impact: 1\n is_encoder:\n ui_display_name: null\n expected_impact: 1\n lang_id:\n ui_display_name: null\n expected_impact: 1\n layer_norm_eps:\n ui_display_name: null\n expected_impact: 1\n layerdrop:\n ui_display_name: null\n expected_impact: 1\n mask_index:\n ui_display_name: null\n expected_impact: 1\n mask_token_id:\n default_value_reasoning: Default value used in pre-trained HF encoder.\n ui_display_name: Mask Token ID\n expected_impact: 1\n max_position_embeddings:\n default_value_reasoning: Taken from huggingface.\n description_implications:\n The size of the position embeddings table. This typically coincides with the\n maximum sequence length this model might ever be used with. Typically set this\n to something large just in case (e.g. 512, 1024, 2048).\n expected_impact: 1\n suggested_values: 512\n suggested_values_reasoning:\n Out of the box value based on published literature.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Max Position Embeddings\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n n_heads:\n ui_display_name: null\n expected_impact: 1\n n_langs:\n default_value_reasoning: Default value used in pre-trained HF encoder.\n expected_impact: 1\n ui_display_name: Number of Languages\n n_layers:\n ui_display_name: null\n expected_impact: 1\n pad_index:\n ui_display_name: null\n expected_impact: 1\n pre_norm:\n ui_display_name: null\n expected_impact: 1\n pretrained_kwargs:\n ui_display_name: null\n expected_impact: 1\n pretrained_model_name_or_path:\n ui_display_name: null\n expected_impact: 2\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n saved_weights_in_checkpoint:\n default_value_reasoning:\n The weights of the encoder are not necessarily saved\n in the checkpoint. The user has to save them first.\n description_implications:\n The memory footprint for some of these encoders\n can be large.\n internal_only: true\n related_parameters:\n - skip_save_model\n suggested_values:\n - false\n suggested_values_reasoning:\n Some of these encoders are large, so it might\n be better to load them as needed, especially if 1. they're not used frequently\n 2. the user doesn't have a lot of storage.\n ui_display_name: null\n sinusoidal_embeddings:\n ui_display_name: null\n expected_impact: 1\n trainable:\n ui_display_name: null\n expected_impact: 3\n unk_index:\n ui_display_name: null\n expected_impact: 1\n use_lang_emb:\n ui_display_name: null\n expected_impact: 1\n use_pretrained:\n default_value_reasoning:\n By default, the model is initialized as a pretrained\n model.\n description_implications:\n Pretrained models have typically already learned\n features that are difficult to learn from scratch. They are particularly\n beneficial when training on small amounts of data.\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/transfer-learning-for-deep-learning/\n related_parameters:\n - trainable, pretrained_model_name, pretrained_model_name_or_path, pretrained_kwargs\n suggested_values:\n - false\n suggested_values_reasoning:\n If you have a large amount of data and/or you\n have data that differs from the typical distribution, then it might be\n worth training the model from scratch.\n ui_display_name: Use Pretrained\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n vocab_size:\n internal_only: true\n ui_display_name: Not displayed\nGPT2:\n type:\n short_description: GPT-2 is a pre-trained language model used for NLP tasks like generation, summarization, and translation.\n long_description: The `gpt2` encoder loads a pretrained\n [GPT-2](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)\n (default `gpt2`) model using the Hugging Face transformers package. GPT-2 is a causal (unidirectional) transformer pretrained using language\n modeling on a very large corpus of ~40 GB of text data.\n literature_references:\n - https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf\n compute_tier: 3\n activation_function:\n ui_display_name: null\n attn_pdrop:\n ui_display_name: null\n embd_pdrop:\n ui_display_name: null\n initializer_range:\n description_implications:\n There is an ideal value for this variable that doesn't\n lead to the outputs of these matrices to vanish or explode\n example_value:\n - 0.02\n expected_impact: 1\n other_information: Must be greater than 0\n related_parameters:\n - weights_initializer\n suggested_values: 0.01-0.05\n suggested_values_reasoning:\n Large values will likely lead to very large outputs.\n Small values will lead to vanishing outputs.\n ui_display_name: null\n layer_norm_epsilon:\n ui_display_name: null\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n n_ctx:\n ui_display_name: null\n n_embd:\n ui_display_name: null\n n_head:\n ui_display_name: null\n n_inner:\n ui_display_name: null\n n_layer:\n ui_display_name: null\n n_positions:\n ui_display_name: null\n pretrained_kwargs:\n ui_display_name: null\n pretrained_model_name_or_path:\n ui_display_name: null\n expected_impact: 2\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n resid_pdrop:\n ui_display_name: null\n scale_attn_weights:\n ui_display_name: null\n trainable:\n expected_impact: 3\n ui_display_name: null\n use_pretrained:\n default_value_reasoning:\n By default, the model is initialized as a pretrained\n model.\n description_implications:\n Pretrained models have typically already learned\n features that are difficult to learn from scratch. They are particularly\n beneficial when training on small amounts of data.\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/transfer-learning-for-deep-learning/\n related_parameters:\n - trainable, pretrained_model_name, pretrained_model_name_or_path, pretrained_kwargs\n suggested_values:\n - false\n suggested_values_reasoning:\n If you have a large amount of data and/or you\n have data that differs from the typical distribution, then it might be\n worth training the model from scratch.\n ui_display_name: Use Pretrained\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n vocab_size:\n internal_only: true\n ui_display_name: Not displayed\nGPT:\n type:\n short_description: GPT is a pre-trained language model used for NLP tasks like generation, summarization, and translation.\n long_description: The `gpt` encoder loads a pretrained\n [GPT](https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf)\n (default `openai-gpt`) model using the Hugging Face transformers package.\n GPT is a causal (unidirectional) transformer pre-trained using language modeling on a large corpus with long range dependencies, the Toronto Book Corpus.\n literature_references:\n - https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf\n compute_tier: 2\n afn:\n ui_display_name: null\n attn_pdrop:\n ui_display_name: null\n embd_pdrop:\n ui_display_name: null\n initializer_range:\n description_implications:\n There is an ideal value for this variable that doesn't\n lead to the outputs of these matrices to vanish or explode\n example_value:\n - 0.02\n expected_impact: 1\n other_information: Must be greater than 0\n related_parameters:\n - weights_initializer\n suggested_values: 0.01-0.05\n suggested_values_reasoning:\n Large values will likely lead to very large outputs.\n Small values will lead to vanishing outputs.\n ui_display_name: null\n layer_norm_epsilon:\n ui_display_name: null\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n n_ctx:\n ui_display_name: null\n n_embd:\n ui_display_name: null\n n_head:\n ui_display_name: null\n n_layer:\n ui_display_name: null\n n_positions:\n ui_display_name: null\n pretrained_kwargs:\n ui_display_name: null\n pretrained_model_name_or_path:\n ui_display_name: null\n expected_impact: 2\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n resid_pdrop:\n ui_display_name: null\n saved_weights_in_checkpoint:\n default_value_reasoning:\n The weights of the encoder are not necessarily saved\n in the checkpoint. The user has to save them first.\n description_implications:\n The memory footprint for some of these encoders\n can be large.\n internal_only: true\n related_parameters:\n - skip_save_model\n suggested_values:\n - false\n suggested_values_reasoning:\n Some of these encoders are large, so it might\n be better to load them as needed, especially if 1. they're not used frequently\n 2. the user doesn't have a lot of storage.\n ui_display_name: null\n trainable:\n expected_impact: 3\n ui_display_name: null\n use_pretrained:\n default_value_reasoning:\n By default, the model is initialized as a pretrained\n model.\n description_implications:\n Pretrained models have typically already learned\n features that are difficult to learn from scratch. They are particularly\n beneficial when training on small amounts of data.\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/transfer-learning-for-deep-learning/\n related_parameters:\n - trainable, pretrained_model_name, pretrained_model_name_or_path, pretrained_kwargs\n suggested_values:\n - false\n suggested_values_reasoning:\n If you have a large amount of data and/or you\n have data that differs from the typical distribution, then it might be\n worth training the model from scratch.\n ui_display_name: Use Pretrained\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n vocab_size:\n internal_only: true\n ui_display_name: Not displayed\nH3Embed:\n type:\n short_description: Encodes each H3 component with embeddings then takes a sum and passes them through fully connected layers.\n long_description:\n The Embed encoder encodes each component of the H3 representation (mode, edge, resolution,\n base cell and children cells) with embeddings. Children cells with value 0 will be masked out.\n After the embedding, all embeddings are summed and optionally passed through a stack of fully\n connected layers.\n activation:\n default_value_reasoning:\n The Rectified Linear Units (ReLU) function is the\n standard activation function used for adding non-linearity. It is simple,\n fast, and empirically works well (https://arxiv.org/abs/1803.08375).\n description_implications:\n Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n expected_impact: 2\n suggested_values:\n The default value will work well in the majority of the\n cases\n ui_display_name: Activation\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n embedding_size:\n default_value_reasoning: Not too big, not too small.\n description_implications:\n 'An embedding is a relatively low-dimensional space\n that is used to translate high-dimensional vectors like words, which can\n have a large vocbulary size. Ideally, after an embedding is trained, it\n captures some of the semantics of the input by placing semantically similar\n inputs close together in the embedding space.\n\n\n In most cases, the embedding size is chosen empirically, by trial and\n error. From https://www.amazon.com/dp/1098115783, \"one rule of thumb is\n to use the fourth root of the total number of unique categorical elements\n while another is that the embedding dimension should be approximately\n 1.6 times the square root of the number of unique elements in the category,\n and no less than 600.\"\n\n\n Increasing the embedding size may cause the model to train more slowly,\n but the higher dimensionality can also improve overall quality.'\n expected_impact: 3\n literature_references:\n - https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture\n suggested_values: 1.6 * sqrt(vocab_size)\n suggested_values_reasoning:\n Rule of thumb suggested by a deep learning textbook.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Embedding Size\n embeddings_on_cpu:\n default_value_reasoning:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access.\n description_implications:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access. However, in some\n cases when the vocabulary size is very large, the full embedding matrix\n may be really big and unwieldy to have in GPU memory. This parameter forces\n the placement of the embedding matrix in regular memory and the CPU is\n used to access them. This may slow down training due to additional data\n transfer between CPU and GPU memory, but can lead to healthier GPU memory\n resource usage.\n expected_impact: 1\n suggested_values:\n - false\n suggested_values_reasoning:\n If GPU memory is not a constraint, having embeddings\n stored and accessed within the GPU is faster.\n ui_display_name: Embeddings on CPU\n fc_layers:\n default_value_reasoning:\n By default the stack is built by using num_fc_layers,\n output_size, use_bias, weights_initializer, bias_initializer, norm, norm_params,\n activation, dropout. When a list of dictionaries is provided, the stack\n is built following the parameters of each dict for building each layer.\n description_implications:\n The more layers that are specified the deeper and\n higher capacity the model will be. This makes it possible to potentially\n achieve better performance when a big anough amount of data is provided,\n but also makes the model more computationally expensive and potentially\n more prone to overfitting.\n example_value:\n - dropout: 0.1\n output_size: 128\n - norm: layer\n output_size: 64\n expected_impact: 1\n related_parameters:\n - output_size\n - use_bias\n - weights_initializer\n - bias_initializer\n - norm\n - norm_params\n - activation\n - dropout\n suggested_values_reasoning:\n It is easier to define a stack of fully connected\n layers by just specifying num_fc_layers, output_size and the other individual\n parameters. It will create a stack of layers with identical properties.\n Use this parameter only if you need a fine grained level of control of\n each individual layer in the stack.\n ui_display_name: Fully Connected Layers\n norm:\n default_value_reasoning:\n While batch normalization and layer normalization\n usually lead to improvements, it can be useful to start with fewer bells\n and whistles.\n description_implications:\n Normalization helps stabilize the learning process\n and can have a regularizing effect that can help with generalization.\n It's often suggested that with normalization, you can use a higher learning\n rate.\n example_value:\n - batch\n expected_impact: 2\n literature_references:\n - https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/\n related_parameters:\n - norm_params\n suggested_values: '\"batch\" or \"layer\"'\n suggested_values_reasoning:\n Normalization tries to solve \"internal covariate\n shift\" that comes from the changing distributions of the inputs to layers\n deep in the network when weights are updated. For example, batch normalization\n standardizes the inputs to a layer for each mini-batch. Try out different\n normalizations to see if that helps with training stability\n ui_display_name: Normalization Type\n norm_params:\n default_value_reasoning:\n The default parameters that come with Torch's implementation\n of these normalization types are a trusted starting point.\n description_implications:\n There are a variety of ways a certain set of parameters\n specificed could influence performance here. Broadly speaking the different\n values passed in here allow for different levels of smoothness to be observed\n in the learning curves. Since setting this parameters depends on the type\n of `norm` set, see [BatchNorm2d](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html)\n for more information on the parameters to set for batch normalization,\n and see [LayerNorm](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html)\n for more information on the parameters to set for layer normalization.\n example_value:\n - affine: false\n momentum: 0.2\n num_features: 100\n expected_impact: 1\n literature_references:\n - \"For BatchNorm2d: https://arxiv.org/abs/1502.03167\n\n\n For LayerNorm: https://arxiv.org/abs/1607.06450\"\n related_parameters:\n - \"`norm`\"\n suggested_values: Depends on the type of `norm` set.\n suggested_values_reasoning: \"NO\"\n ui_display_name: Normalization Parameters\n num_fc_layers:\n default_value_reasoning:\n The encoder already has learnable parameters.Sometimes\n the default is 1 for modules where the FC stack is used for shape management,\n or the only source of learnable parameters.\n description_implications:\n Increasing num_fc_layers will increase the capacity\n of the model. The model will be slower to train, and there's a higher\n risk of overfitting.\n example_value:\n - 1\n expected_impact: 1\n other_information:\n Not all modules that have fc_layers also have an accompanying\n num_fc_layers parameter. Where both are present, fc_layers takes precedent\n over num_fc_layers. Specifying num_fc_layers alone uses fully connected\n layers that are configured by the defaults in FCStack.\n related_parameters:\n - fc_layers\n suggested_values: 0-1\n suggested_values_reasoning:\n The full model likely contains many learnable\n parameters. Consider starting with very few, or without any additional\n fully connected layers and add them if you observe evidence of limited\n model capacity. Sometimes the default is 1 for modules where the FC stack\n is used for shape management, or the only source of learnable parameters.\n ui_display_name: Number of Fully Connected Layers\n output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 10 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\n reduce_output:\n default_value_reasoning: Sums the tensors along the sequence dimension.\n description_implications:\n \"\\\"last\\\", \\\"sum\\\", \\\"mean\\\", and \\\"max\\\" are the\\\n \\ fastest and most memory-efficient operations\\u2013 they result in tensors\\\n \\ that are the same-size as a single item in the input sequence. However,\\\n \\ these are simple aggregation operations, therefore some information\\\n \\ may be lost. \\n\\n\\\"concat\\\" concatenates each tensor together, creating\\\n \\ a `(sequence length)*(tensor size)`-element tensor. \\\"concat\\\" preserves\\\n \\ this information, but can be very memory-intensive and should only be\\\n \\ applied if the sequence length and/or tensor size is small. \\n\\n\\\"attention\\\"\\\n \\ takes a weighted sum of the items in the sequence, where the weights\\\n \\ for each item in the sequence are determined by the model on-the-fly\\\n \\ based on the features of the item itself. This is both slower and and\\\n \\ more memory-intensive than the other operations; however, it can also\\\n \\ provide a richer \\\"global\\\" representation of the sequence.\"\n expected_impact: 1\n related_parameters:\n - max_sequence_length\n suggested_values: '\"attention\". This and the default covers 95% of use cases.'\n suggested_values_reasoning:\n If you would like better performance and are not\n compute/memory-constrained, attention-based reduction can potentially\n provide a richer global representation than the default.\n ui_display_name: Sequence Reducer\n use_bias:\n ui_display_name: null\n expected_impact: 1\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\n\n\n Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\nH3RNN:\n type:\n short_description: Encodes each H3 component with embeddings then passes them through an RNN encoder.\n long_description:\n The RNN encoder encodes each component of the H3 representation (mode, edge, resolution,\n base cell and children cells) with embeddings. Children cells with value 0 will be masked out.\n After the embedding, all embeddings are passed through an RNN encoder. The intuition behind this\n is that, starting from the base cell, the sequence of children cells can be seen as a sequence\n encoding the path in the tree of all H3 hexes.\n activation:\n ui_display_name: null\n expected_impact: 1\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 2\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n bidirectional:\n default_value_reasoning:\n For short sequences, it is reasonable to use a vanilla\n RNN.\n description_implications:\n Setting bidirectional to True may increase the compute\n and memory requirements of the model, but may also increase model performance\n on long sequences.\n expected_impact: 0\n literature_references:\n - https://devopedia.org/bidirectional-rnn#:~:text=RNN%20has%20the%20limitation%20that,forward%20and%20reverse%20time%20order.\n related_parameters:\n - cell_type, activation, recurrent_activation, use_bias\n suggested_values:\n - true\n suggested_values_reasoning:\n \"RNNs can sometimes suffer from catastrophic forgetting\n (source: https://en.wikipedia.org/wiki/Catastrophic_interference ) on\n long sequences. Allowing the RNN to read from both the beginning and end\n of the sequence can improve its representation at each timestep.\"\n ui_display_name: Bidirectional\n cell_type:\n default_value_reasoning:\n The LSTM cell has proven to be the most performant\n of the three cells.\n description_implications:\n \"There are two reasons to consider other cell types:\n (1) compute costs and (2) catastrophic forgetting (source: https://en.wikipedia.org/wiki/Catastrophic_interference\n ). RNNs have marginally less compute costs, but are prone to catastrophic\n forgetting.\"\n expected_impact: 3\n related_parameters:\n - \"bidirectional\n\n activation\n\n recurrent_activation\n\n use_bias\"\n ui_display_name: Cell Type\n dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - recurrent_dropout\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n embedding_size:\n default_value_reasoning: Not too big, not too small.\n description_implications:\n 'An embedding is a relatively low-dimensional space\n that is used to translate high-dimensional vectors like words, which can\n have a large vocbulary size. Ideally, after an embedding is trained, it\n captures some of the semantics of the input by placing semantically similar\n inputs close together in the embedding space.\n\n\n In most cases, the embedding size is chosen empirically, by trial and\n error. From https://www.amazon.com/dp/1098115783, \"one rule of thumb is\n to use the fourth root of the total number of unique categorical elements\n while another is that the embedding dimension should be approximately\n 1.6 times the square root of the number of unique elements in the category,\n and no less than 600.\"\n\n\n Increasing the embedding size may cause the model to train more slowly,\n but the higher dimensionality can also improve overall quality.'\n expected_impact: 3\n literature_references:\n - https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture\n suggested_values: 1.6 * sqrt(vocab_size)\n suggested_values_reasoning:\n Rule of thumb suggested by a deep learning textbook.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Embedding Size\n embeddings_on_cpu:\n default_value_reasoning:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access.\n description_implications:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access. However, in some\n cases when the vocabulary size is very large, the full embedding matrix\n may be really big and unwieldy to have in GPU memory. This parameter forces\n the placement of the embedding matrix in regular memory and the CPU is\n used to access them. This may slow down training due to additional data\n transfer between CPU and GPU memory, but can lead to healthier GPU memory\n resource usage.\n expected_impact: 1\n suggested_values:\n - false\n suggested_values_reasoning:\n If GPU memory is not a constraint, having embeddings\n stored and accessed within the GPU is faster.\n ui_display_name: Embeddings on CPU\n hidden_size:\n default_value_reasoning:\n H3 values numbers, so a small RNN dimensionality\n is likely sufficient.\n description_implications:\n Increasing the hidden size makes the model larger\n and slower to train, increases the model's capacity to capture more complexity.\n It also increases the chance of overfitting.\n expected_impact: 2\n suggested_values: 10 - 2048\n suggested_values_reasoning:\n Increasing the hidden size makes sense if the\n model is underfitting. It's useful to train both smaller and larger models\n to see how model capacity affects performance. This should only be explored\n after the architecture of the model has been settled.\n ui_display_name: Hidden Size\n num_layers:\n default_value_reasoning:\n The ideal number of layers depends on the data. For\n many data types, one layer is sufficient.\n description_implications:\n Increasing the number of layers may improve model\n performance for longer sequences or more complex tasks.\n example_value:\n - 1\n expected_impact: 3\n other_information:\n If you have multiple input features, varying the number\n of layers in the combiner or output feature decoder will have more impact.\n related_parameters:\n - layers\n suggested_values: 1-3\n suggested_values_reasoning:\n Increasing the number of layers may improve encoder\n performance. However, more layers will increase training time and may\n cause overfitting. Small numbers of layers usually work best.\n ui_display_name: Number of Recurrent Layers\n recurrent_activation:\n default_value_reasoning: sigmoid' is commonly used\n expected_impact: 1\n other_information:\n I don't think that this parameter is used anywhere in the\n code base. It's being passed down but not used in the actual RNN forwarding\n functions.\n suggested_values: sigmoid, ReLu, tanh\n ui_display_name: null\n recurrent_dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 2\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - dropout\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Recurrent Dropout\n recurrent_initializer:\n ui_display_name: null\n expected_impact: 1\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n unit_forget_bias:\n ui_display_name: null\n expected_impact: 1\n use_bias:\n ui_display_name: null\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\n\n\n Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\nH3WeightedSum:\n type:\n short_description: Encodes each H3 component with embeddings then takes a weighted sum.\n long_description:\n The Weighted Sum encoder encodes each component of the H3 representation (mode, edge,\n resolution, base cell and children cells) with embeddings. Children cells with value 0 will be\n masked out. After the embedding, all embeddings are summed with a weighted sum (with learned\n weights) and optionally passed through a stack of fully connected layers.\n activation:\n default_value_reasoning:\n The Rectified Linear Units (ReLU) function is the\n standard activation function used for adding non-linearity. It is simple,\n fast, and empirically works well (https://arxiv.org/abs/1803.08375).\n description_implications:\n Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n expected_impact: 2\n suggested_values:\n The default value will work well in the majority of the\n cases\n ui_display_name: Activation\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n embedding_size:\n default_value_reasoning: Not too big, not too small.\n description_implications:\n 'An embedding is a relatively low-dimensional space\n that is used to translate high-dimensional vectors like words, which can\n have a large vocbulary size. Ideally, after an embedding is trained, it\n captures some of the semantics of the input by placing semantically similar\n inputs close together in the embedding space.\n\n\n In most cases, the embedding size is chosen empirically, by trial and\n error. From https://www.amazon.com/dp/1098115783, \"one rule of thumb is\n to use the fourth root of the total number of unique categorical elements\n while another is that the embedding dimension should be approximately\n 1.6 times the square root of the number of unique elements in the category,\n and no less than 600.\"\n\n\n Increasing the embedding size may cause the model to train more slowly,\n but the higher dimensionality can also improve overall quality.'\n expected_impact: 3\n literature_references:\n - https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture\n suggested_values: 1.6 * sqrt(vocab_size)\n suggested_values_reasoning:\n Rule of thumb suggested by a deep learning textbook.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Embedding Size\n embeddings_on_cpu:\n default_value_reasoning:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access.\n description_implications:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access. However, in some\n cases when the vocabulary size is very large, the full embedding matrix\n may be really big and unwieldy to have in GPU memory. This parameter forces\n the placement of the embedding matrix in regular memory and the CPU is\n used to access them. This may slow down training due to additional data\n transfer between CPU and GPU memory, but can lead to healthier GPU memory\n resource usage.\n expected_impact: 1\n suggested_values:\n - false\n suggested_values_reasoning:\n If GPU memory is not a constraint, having embeddings\n stored and accessed within the GPU is faster.\n ui_display_name: Embeddings on CPU\n fc_layers:\n default_value_reasoning:\n By default the stack is built by using num_fc_layers,\n output_size, use_bias, weights_initializer, bias_initializer, norm, norm_params,\n activation, dropout. When a list of dictionaries is provided, the stack\n is built following the parameters of each dict for building each layer.\n description_implications:\n The more layers that are specified the deeper and\n higher capacity the model will be. This makes it possible to potentially\n achieve better performance when a big anough amount of data is provided,\n but also makes the model more computationally expensive and potentially\n more prone to overfitting.\n example_value:\n - dropout: 0.1\n output_size: 128\n - norm: layer\n output_size: 64\n expected_impact: 1\n related_parameters:\n - output_size\n - use_bias\n - weights_initializer\n - bias_initializer\n - norm\n - norm_params\n - activation\n - dropout\n suggested_values_reasoning:\n It is easier to define a stack of fully connected\n layers by just specifying num_fc_layers, output_size and the other individual\n parameters. It will create a stack of layers with identical properties.\n Use this parameter only if you need a fine grained level of control of\n each individual layer in the stack.\n ui_display_name: Fully Connected Layers\n norm:\n default_value_reasoning:\n While batch normalization and layer normalization\n usually lead to improvements, it can be useful to start with fewer bells\n and whistles.\n description_implications:\n Normalization helps stabilize the learning process\n and can have a regularizing effect that can help with generalization.\n It's often suggested that with normalization, you can use a higher learning\n rate.\n example_value:\n - batch\n expected_impact: 2\n literature_references:\n - https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/\n related_parameters:\n - norm_params\n suggested_values: '\"batch\" or \"layer\"'\n suggested_values_reasoning:\n Normalization tries to solve \"internal covariate\n shift\" that comes from the changing distributions of the inputs to layers\n deep in the network when weights are updated. For example, batch normalization\n standardizes the inputs to a layer for each mini-batch. Try out different\n normalizations to see if that helps with training stability\n ui_display_name: Normalization Type\n norm_params:\n default_value_reasoning:\n The default parameters that come with Torch's implementation\n of these normalization types are a trusted starting point.\n description_implications:\n There are a variety of ways a certain set of parameters\n specificed could influence performance here. Broadly speaking the different\n values passed in here allow for different levels of smoothness to be observed\n in the learning curves. Since setting this parameters depends on the type\n of `norm` set, see [BatchNorm2d](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html)\n for more information on the parameters to set for batch normalization,\n and see [LayerNorm](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html)\n for more information on the parameters to set for layer normalization.\n example_value:\n - affine: false\n momentum: 0.2\n num_features: 100\n expected_impact: 1\n literature_references:\n - \"For BatchNorm2d: https://arxiv.org/abs/1502.03167\n\n\n For LayerNorm: https://arxiv.org/abs/1607.06450\"\n related_parameters:\n - \"`norm`\"\n suggested_values: Depends on the type of `norm` set.\n suggested_values_reasoning: \"NO\"\n ui_display_name: Normalization Parameters\n num_fc_layers:\n default_value_reasoning:\n The encoder already has learnable parameters.Sometimes\n the default is 1 for modules where the FC stack is used for shape management,\n or the only source of learnable parameters.\n description_implications:\n Increasing num_fc_layers will increase the capacity\n of the model. The model will be slower to train, and there's a higher\n risk of overfitting.\n example_value:\n - 1\n expected_impact: 1\n other_information:\n Not all modules that have fc_layers also have an accompanying\n num_fc_layers parameter. Where both are present, fc_layers takes precedent\n over num_fc_layers. Specifying num_fc_layers alone uses fully connected\n layers that are configured by the defaults in FCStack.\n related_parameters:\n - fc_layers\n suggested_values: 0-1\n suggested_values_reasoning:\n The full model likely contains many learnable\n parameters. Consider starting with very few, or without any additional\n fully connected layers and add them if you observe evidence of limited\n model capacity. Sometimes the default is 1 for modules where the FC stack\n is used for shape management, or the only source of learnable parameters.\n ui_display_name: Number of Fully Connected Layers\n output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 10 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\n should_softmax:\n ui_display_name: null\n expected_impact: 1\n use_bias:\n ui_display_name: null\n expected_impact: 1\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\n\n\n Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\nLongformer:\n type:\n short_description: Transformer optimized for longer text inputs.\n long_description:\n The `longformer` encoder loads a pretrained [Longformer](https://arxiv.org/pdf/2004.05150.pdf)\n (default `allenai/longformer-base-4096`) model using the Hugging Face transformers package. Longformer is a good choice\n for longer text, as it supports sequences up to 4096 tokens long.\n literature_references:\n - https://arxiv.org/pdf/2004.05150.pdf\n compute_tier: 2\n attention_window:\n ui_display_name: null\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n num_tokens:\n ui_display_name: null\n max_position_embeddings:\n default_value_reasoning: Taken from huggingface.\n description_implications:\n \"An embedding is a relatively low-dimensional space\n that is used to translate high-dimensional vectors like words or positions,\n which can have a large vocbulary size. Ideally, after an embedding is\n trained, it captures some of the semantics of the input by placing semantically\n similar inputs close together in the embedding space.\n\n Increasing the embedding size may cause the model to train more slowly,\n but the higher dimensionality can also improve overall quality.\"\n expected_impact: 2\n suggested_values: 512\n suggested_values_reasoning:\n Out of the box value based on published literature.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Max Position Embeddings\n type_vocab_size:\n ui_display_name: null\n pretrained_kwargs:\n ui_display_name: null\n pretrained_model_name_or_path:\n ui_display_name: null\n expected_impact: 2\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n saved_weights_in_checkpoint:\n default_value_reasoning:\n The weights of the encoder are not necessarily saved\n in the checkpoint. The user has to save them first.\n description_implications:\n The memory footprint for some of these encoders\n can be large.\n internal_only: true\n related_parameters:\n - skip_save_model\n suggested_values:\n - false\n suggested_values_reasoning:\n Some of these encoders are large, so it might\n be better to load them as needed, especially if 1. they're not used frequently\n 2. the user doesn't have a lot of storage.\n ui_display_name: null\n sep_token_id:\n ui_display_name: null\n trainable:\n expected_impact: 3\n ui_display_name: null\n use_pretrained:\n default_value_reasoning:\n By default, the model is initialized as a pretrained\n model.\n description_implications:\n Pretrained models have typically already learned\n features that are difficult to learn from scratch. They are particularly\n beneficial when training on small amounts of data.\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/transfer-learning-for-deep-learning/\n related_parameters:\n - trainable, pretrained_model_name, pretrained_model_name_or_path, pretrained_kwargs\n suggested_values:\n - false\n suggested_values_reasoning:\n If you have a large amount of data and/or you\n have data that differs from the typical distribution, then it might be\n worth training the model from scratch.\n ui_display_name: Use Pretrained\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n vocab_size:\n internal_only: true\n ui_display_name: Not displayed\nMLPMixer:\n type:\n short_description: Image encoder which applies fully connected layers to different patches of the image.\n long_description:\n MLP-Mixer divides the image into equal-sized patches, applying fully connected layers to each\n patch to compute per-patch representations (tokens) and combining the representations with\n fully-connected mixer layers.\n compute_tier: 1\n avg_pool:\n ui_display_name: null\n channel_dim:\n ui_display_name: null\n dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n embed_size:\n ui_display_name: null\n height:\n internal_only: true\n ui_display_name: null\n num_channels:\n ui_display_name: null\n num_layers:\n default_value_reasoning:\n The ideal number of layers depends on the size and\n complexity of the input images. The default value is used in the paper\n and tested on several image datasets.\n description_implications:\n Increasing the number of layers may improve model\n performance for larger images or more complex image tasks.\n example_value:\n - 8\n expected_impact: 3\n literature_references:\n - \"MLP-Mixer: An all-MLP Architecture for Vision\n\n https://arxiv.org/abs/2105.01601\"\n suggested_values: 4 - 32\n suggested_values_reasoning:\n Values from 8 - 32 are tested in the paper. It\n is possible that fewer layers will be sufficient for some tasks.\n ui_display_name: Number of Layers\n patch_size:\n default_value_reasoning: Taken from MLP-Mixer paper.\n description_implications:\n \"The implications of the image patch size for this\\\n \\ layer depend on other factors, such as the true resolution of the incoming\\\n \\ image dataset. If the patch size is kept consistent but a higher resolution\\\n \\ image is used as input, then the resulting chunked sequence of tokens\\\n \\ will be longer than it would have been if the input resolution was lower.\\\n \\ \\n\\nThe original MLP-Mixer paper also notes that there is a tradeoff\\\n \\ with respect to the projection units learned by a model. In their findings,\\\n \\ a 32x32 patch size model learned very structured low frequency projection\\\n \\ units, while the equivalent 16x16 model learned high frequencies and\\\n \\ showed no clear structure.\"\n expected_impact: 2\n literature_references:\n - \"[MLP Mixer paper](https://arxiv.org/pdf/2105.01601.pdf)\"\n suggested_values:\n - 16\n - 32\n suggested_values_reasoning:\n 16 and 32 are the values used in the original\n MLP Mixer paper\n ui_display_name: Patch Size\n token_size:\n ui_display_name: null\n width:\n internal_only: true\n ui_display_name: null\nMT5:\n type:\n short_description: MT5 is a multilingual variant of T5 useful for multilingual NLP use cases.\n long_description:\n The `mt5` encoder loads a pretrained [MT5](https://arxiv.org/abs/2010.11934) (default `google/mt5-base`) model using the\n Hugging Face transformers package. MT5 is a multilingual variant of T5 trained on a dataset of 101 languages.\n compute_tier: 2\n d_ff:\n default_value_reasoning: Default value matches the pre-trained encoder.\n description_implications:\n If using a pre-trained encoder, this parameter will\n be automatically derived from the pre-trained model.\n expected_impact: 1\n ui_display_name: Dimensionality of Feed-Forward Layer\n d_kv:\n ui_display_name: null\n d_model:\n ui_display_name: null\n decoder_start_token_id:\n ui_display_name: null\n dropout_rate:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: dropout_rate\n eos_token_id:\n default_value_reasoning: Default value used in pre-trained HF encoder.\n ui_display_name: End-of-Sentence Token Id\n feed_forward_proj:\n ui_display_name: null\n initializer_factor:\n ui_display_name: null\n is_encoder_decoder:\n ui_display_name: null\n layer_norm_epsilon:\n ui_display_name: null\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n num_decoder_layers:\n ui_display_name: null\n num_heads:\n ui_display_name: null\n num_layers:\n default_value_reasoning:\n The default value matches the number of layers in\n the default pretrained encoder.\n description_implications:\n \"The ideal number of transformer layers depends\n on the length and complexity of input sequences, as well as the task.\n\n\n If using a pre-trained encoder, this parameter will be automatically derived\n from the pre-trained model.\"\n example_value:\n - 8\n expected_impact: 3\n related_parameters:\n - pretrained_model_or_path\n suggested_values: 1 - 12\n suggested_values_reasoning:\n Increasing the number of layers may improve encoder\n performance. However, more layers will increase training time and may\n cause overfitting. Small numbers of layers usually work best.\n ui_display_name: Number of Transformer Layers\n pad_token_id:\n ui_display_name: null\n pretrained_kwargs:\n ui_display_name: null\n pretrained_model_name_or_path:\n ui_display_name: null\n expected_impact: 2\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n relative_attention_num_buckets:\n ui_display_name: null\n saved_weights_in_checkpoint:\n default_value_reasoning:\n The weights of the encoder are not necessarily saved\n in the checkpoint. The user has to save them first.\n description_implications:\n The memory footprint for some of these encoders\n can be large.\n internal_only: true\n related_parameters:\n - skip_save_model\n suggested_values:\n - false\n suggested_values_reasoning:\n Some of these encoders are large, so it might\n be better to load them as needed, especially if 1. they're not used frequently\n 2. the user doesn't have a lot of storage.\n ui_display_name: null\n tie_word_embeddings:\n default_value_reasoning:\n Keeping the word embeddings separate ensures maximum\n modeling flexibility.\n description_implications:\n The main tradeoff between True and False values\n is in compute costs and model flexibility. If set to False, the model\n will require more memory, but may be more flexible. If set to True, the\n opposite is true.\n example_value:\n - true\n expected_impact: 2\n suggested_values:\n - true\n suggested_values_reasoning:\n \"If set to True, then the word embeddings will\n be shared between the encoder and decoder. There are two main reasons\n to set this value to True: (1) saving compute resources. Word embedding\n tables can be very large and using a single table between the encoder\n and decoder can cut one's memory usage in half. (2) If the domain of\n the generated text is highly similar to the input text. For example, if\n training a Question and Answering (QA) text model, where both the questions\n and answers are in the same language, the word embeddings used by the\n encoder are likely usable by the decoder and vice-versa. On the other\n hand, if training a translation model between two languages, the word\n embeddings are not likely to be shareable by both model components.\"\n ui_display_name: null\n tokenizer_class:\n ui_display_name: null\n trainable:\n expected_impact: 3\n ui_display_name: null\n use_cache:\n ui_display_name: null\n use_pretrained:\n default_value_reasoning:\n By default, the model is initialized as a pretrained\n model.\n description_implications:\n Pretrained models have typically already learned\n features that are difficult to learn from scratch. They are particularly\n beneficial when training on small amounts of data.\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/transfer-learning-for-deep-learning/\n related_parameters:\n - trainable, pretrained_model_name, pretrained_model_name_or_path, pretrained_kwargs\n suggested_values:\n - false\n suggested_values_reasoning:\n If you have a large amount of data and/or you\n have data that differs from the typical distribution, then it might be\n worth training the model from scratch.\n ui_display_name: Use Pretrained\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n vocab_size:\n internal_only: true\n ui_display_name: Not displayed\nParallelCNN:\n type:\n short_description: Default option for processing sequence, audio, and text data types.\n long_description:\n The Parallel CNN works by first mapping the input integer sequence b x s (where b is the batch\n size and s is the length of the sequence) into a sequence of embeddings, then it passes the\n embedding through a number of parallel 1d convolutional layers with different filter size (by\n default 4 layers with filter size 2, 3, 4 and 5), followed by max pooling and concatenation.\n This single vector concatenating the outputs of the parallel convolutional layers is then passed\n through a stack of fully connected layers and returned as a b x h tensor where h is the output\n size of the last fully connected layer.\n compute_tier: 1\n activation:\n default_value_reasoning:\n The Rectified Linear Units (ReLU) function is the\n standard activation function used for adding non-linearity. It is simple,\n fast, and empirically works well (https://arxiv.org/abs/1803.08375).\n description_implications:\n Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n expected_impact: 2\n suggested_values:\n The default value will work well in the majority of the\n cases\n ui_display_name: Activation\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n embedding_size:\n default_value_reasoning: Not too big, not too small.\n description_implications:\n 'An embedding is a relatively low-dimensional space\n that is used to translate high-dimensional vectors like words, which can\n have a large vocbulary size. Ideally, after an embedding is trained, it\n captures some of the semantics of the input by placing semantically similar\n inputs close together in the embedding space.\n\n\n In most cases, the embedding size is chosen empirically, by trial and\n error. From https://www.amazon.com/dp/1098115783, \"one rule of thumb is\n to use the fourth root of the total number of unique categorical elements\n while another is that the embedding dimension should be approximately\n 1.6 times the square root of the number of unique elements in the category,\n and no less than 600.\"\n\n\n Increasing the embedding size may cause the model to train more slowly,\n but the higher dimensionality can also improve overall quality.'\n expected_impact: 3\n literature_references:\n - https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture\n suggested_values: 1.6 * sqrt(vocab_size)\n suggested_values_reasoning:\n Rule of thumb suggested by a deep learning textbook.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Embedding Size\n embeddings_on_cpu:\n default_value_reasoning:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access.\n description_implications:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access. However, in some\n cases when the vocabulary size is very large, the full embedding matrix\n may be really big and unwieldy to have in GPU memory. This parameter forces\n the placement of the embedding matrix in regular memory and the CPU is\n used to access them. This may slow down training due to additional data\n transfer between CPU and GPU memory, but can lead to healthier GPU memory\n resource usage.\n expected_impact: 1\n suggested_values:\n - false\n suggested_values_reasoning:\n If GPU memory is not a constraint, having embeddings\n stored and accessed within the GPU is faster.\n ui_display_name: Embeddings on CPU\n embeddings_trainable:\n ui_display_name: null\n expected_impact: 1\n fc_layers:\n default_value_reasoning:\n By default the stack is built by using num_fc_layers,\n output_size, use_bias, weights_initializer, bias_initializer, norm, norm_params,\n activation, dropout. When a list of dictionaries is provided, the stack\n is built following the parameters of each dict for building each layer.\n description_implications:\n The more layers that are specified the deeper and\n higher capacity the model will be. This makes it possible to potentially\n achieve better performance when a big anough amount of data is provided,\n but also makes the model more computationally expensive and potentially\n more prone to overfitting.\n example_value:\n - dropout: 0.1\n output_size: 128\n - norm: layer\n output_size: 64\n expected_impact: 1\n related_parameters:\n - output_size\n - use_bias\n - weights_initializer\n - bias_initializer\n - norm\n - norm_params\n - activation\n - dropout\n suggested_values_reasoning:\n It is easier to define a stack of fully connected\n layers by just specifying num_fc_layers, output_size and the other individual\n parameters. It will create a stack of layers with identical properties.\n Use this parameter only if you need a fine grained level of control of\n each individual layer in the stack.\n ui_display_name: Fully Connected Layers\n filter_size:\n ui_display_name: null\n expected_impact: 2\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n norm:\n default_value_reasoning:\n While batch normalization and layer normalization\n usually lead to improvements, it can be useful to start with fewer bells\n and whistles.\n description_implications:\n Normalization helps stabilize the learning process\n and can have a regularizing effect that can help with generalization.\n It's often suggested that with normalization, you can use a higher learning\n rate.\n example_value:\n - batch\n expected_impact: 2\n literature_references:\n - https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/\n related_parameters:\n - norm_params\n suggested_values: '\"batch\" or \"layer\"'\n suggested_values_reasoning:\n Normalization tries to solve \"internal covariate\n shift\" that comes from the changing distributions of the inputs to layers\n deep in the network when weights are updated. For example, batch normalization\n standardizes the inputs to a layer for each mini-batch. Try out different\n normalizations to see if that helps with training stability\n ui_display_name: Normalization Type\n norm_params:\n default_value_reasoning:\n The default parameters that come with Torch's implementation\n of these normalization types are a trusted starting point.\n description_implications:\n There are a variety of ways a certain set of parameters\n specificed could influence performance here. Broadly speaking the different\n values passed in here allow for different levels of smoothness to be observed\n in the learning curves. Since setting this parameters depends on the type\n of `norm` set, see [BatchNorm2d](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html)\n for more information on the parameters to set for batch normalization,\n and see [LayerNorm](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html)\n for more information on the parameters to set for layer normalization.\n example_value:\n - affine: false\n momentum: 0.2\n num_features: 100\n expected_impact: 1\n literature_references:\n - \"For BatchNorm2d: https://arxiv.org/abs/1502.03167\n\n\n For LayerNorm: https://arxiv.org/abs/1607.06450\"\n related_parameters:\n - \"`norm`\"\n suggested_values: Depends on the type of `norm` set.\n suggested_values_reasoning: \"NO\"\n ui_display_name: Normalization Parameters\n num_fc_layers:\n default_value_reasoning:\n The encoder already has learnable parameters.Sometimes\n the default is 1 for modules where the FC stack is used for shape management,\n or the only source of learnable parameters.\n description_implications:\n Increasing num_fc_layers will increase the capacity\n of the model. The model will be slower to train, and there's a higher\n risk of overfitting.\n example_value:\n - 1\n expected_impact: 1\n other_information:\n Not all modules that have fc_layers also have an accompanying\n num_fc_layers parameter. Where both are present, fc_layers takes precedent\n over num_fc_layers. Specifying num_fc_layers alone uses fully connected\n layers that are configured by the defaults in FCStack.\n related_parameters:\n - fc_layers\n suggested_values: 0-1\n suggested_values_reasoning:\n The full model likely contains many learnable\n parameters. Consider starting with very few, or without any additional\n fully connected layers and add them if you observe evidence of limited\n model capacity. Sometimes the default is 1 for modules where the FC stack\n is used for shape management, or the only source of learnable parameters.\n ui_display_name: Number of Fully Connected Layers\n output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 10 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\n pool_function:\n ui_display_name: Pooling function\n expected_impact: 1\n pool_size:\n ui_display_name: null\n expected_impact: 1\n pretrained_embeddings:\n ui_display_name: null\n expected_impact: 0\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n representation:\n ui_display_name: null\n expected_impact: 1\n should_embed:\n internal_only: true\n ui_display_name: Not displayed\n use_bias:\n default_value_reasoning:\n \"Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to use bias terms.\n\n\n Batch Normalization, however, adds a trainable shift parameter which is\n added to the activation. When Batch Normalization is used in a layer,\n bias terms are redundant and may be removed.\"\n description_implications:\n Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to leave this parameter set to True.\n example_value:\n - true\n expected_impact: 1\n other_information:\n If fc_layers is not specified, or use_bias is not specified\n for individual layers, the value of use_bias will be used as the default\n for all layers.\n related_parameters:\n - bias_initializer, fc_layers\n suggested_values:\n - true\n ui_display_name: Use Bias\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\n\n\n Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\nPassthroughEncoder:\n type:\n short_description: Passes the raw input through to the combiner.\n long_description:\n The passthrough encoder simply returns the raw numerical values coming from the input\n placeholders as outputs. Inputs are of size `b` while outputs are of size `b x 1` where `b` is\n the batch size.\n input_size:\n internal_only: true\n other_information: Internal Only\n related_parameters:\n - \"No\"\n ui_display_name: Not Displayed\nBinaryPassthroughEncoder:\n type:\n short_description: Passes the raw input through to the combiner.\n long_description:\n The passthrough encoder simply returns the raw numerical values coming from the input\n placeholders as outputs. Inputs are of size `b` while outputs are of size `b x 1` where `b` is\n the batch size.\n input_size:\n internal_only: true\n other_information: Internal Only\n related_parameters:\n - \"No\"\n ui_display_name: Not Displayed\nCategoricalPassthroughEncoder:\n type:\n short_description: Passes the raw input through to the combiner.\n long_description:\n The passthrough encoder simply returns the raw numerical values coming from the input\n placeholders as outputs. Inputs are of size `b` while outputs are of size `b x 1` where `b` is\n the batch size.\n input_size:\n internal_only: true\n other_information: Internal Only\n related_parameters:\n - \"No\"\n ui_display_name: Not Displayed\nResNet:\n type:\n short_description: Residual network achieving very high performance on computer vision tasks.\n long_description:\n ResNet - short for residual network is part of a family of extremely deep architectures showing\n compelling accuracy and nice convergence behaviors for computer vision applications. It is a type\n of CNN architecture designed to support hundreds or thousands of convolutional layers.\n compute_tier: 2\n activation:\n default_value_reasoning:\n The Rectified Linear Units (ReLU) function is the\n standard activation function used for adding non-linearity. It is simple,\n fast, and empirically works well (https://arxiv.org/abs/1803.08375).\n description_implications:\n Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n expected_impact: 2\n suggested_values:\n The default value will work well in the majority of the\n cases\n ui_display_name: Activation\n batch_norm_epsilon:\n ui_display_name: null\n batch_norm_momentum:\n ui_display_name: null\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n conv_stride:\n ui_display_name: null\n dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n fc_layers:\n default_value_reasoning:\n By default the stack is built by using num_fc_layers,\n output_size, use_bias, weights_initializer, bias_initializer, norm, norm_params,\n activation, dropout. When a list of dictionaries is provided, the stack\n is built following the parameters of each dict for building each layer.\n description_implications:\n The more layers that are specified the deeper and\n higher capacity the model will be. This makes it possible to potentially\n achieve better performance when a big anough amount of data is provided,\n but also makes the model more computationally expensive and potentially\n more prone to overfitting.\n example_value:\n - dropout: 0.1\n output_size: 128\n - norm: layer\n output_size: 64\n expected_impact: 1\n related_parameters:\n - output_size\n - use_bias\n - weights_initializer\n - bias_initializer\n - norm\n - norm_params\n - activation\n - dropout\n suggested_values_reasoning:\n It is easier to define a stack of fully connected\n layers by just specifying num_fc_layers, output_size and the other individual\n parameters. It will create a stack of layers with identical properties.\n Use this parameter only if you need a fine grained level of control of\n each individual layer in the stack.\n ui_display_name: Fully Connected Layers\n first_pool_kernel_size:\n ui_display_name: null\n expected_impact: 1\n first_pool_stride:\n ui_display_name: null\n expected_impact: 1\n height:\n internal_only: true\n ui_display_name: null\n kernel_size:\n ui_display_name: null\n norm:\n default_value_reasoning:\n While batch normalization and layer normalization\n usually lead to improvements, it can be useful to start with fewer bells\n and whistles.\n description_implications:\n Normalization helps stabilize the learning process\n and can have a regularizing effect that can help with generalization.\n It's often suggested that with normalization, you can use a higher learning\n rate.\n example_value:\n - batch\n expected_impact: 2\n literature_references:\n - https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/\n related_parameters:\n - norm_params\n suggested_values: '\"batch\" or \"layer\"'\n suggested_values_reasoning:\n Normalization tries to solve \"internal covariate\n shift\" that comes from the changing distributions of the inputs to layers\n deep in the network when weights are updated. For example, batch normalization\n standardizes the inputs to a layer for each mini-batch. Try out different\n normalizations to see if that helps with training stability\n ui_display_name: Normalization Type\n norm_params:\n default_value_reasoning:\n The default parameters that come with Torch's implementation\n of these normalization types are a trusted starting point.\n description_implications:\n There are a variety of ways a certain set of parameters\n specificed could influence performance here. Broadly speaking the different\n values passed in here allow for different levels of smoothness to be observed\n in the learning curves. Since setting this parameters depends on the type\n of `norm` set, see [BatchNorm2d](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html)\n for more information on the parameters to set for batch normalization,\n and see [LayerNorm](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html)\n for more information on the parameters to set for layer normalization.\n example_value:\n - affine: false\n momentum: 0.2\n num_features: 100\n expected_impact: 1\n literature_references:\n - \"For BatchNorm2d: https://arxiv.org/abs/1502.03167\n\n\n For LayerNorm: https://arxiv.org/abs/1607.06450\"\n related_parameters:\n - \"`norm`\"\n suggested_values: Depends on the type of `norm` set.\n suggested_values_reasoning: \"NO\"\n ui_display_name: Normalization Parameters\n num_channels:\n ui_display_name: null\n num_fc_layers:\n default_value_reasoning:\n The encoder already has learnable parameters.Sometimes\n the default is 1 for modules where the FC stack is used for shape management,\n or the only source of learnable parameters.\n description_implications:\n Increasing num_fc_layers will increase the capacity\n of the model. The model will be slower to train, and there's a higher\n risk of overfitting.\n example_value:\n - 1\n expected_impact: 1\n other_information:\n Not all modules that have fc_layers also have an accompanying\n num_fc_layers parameter. Where both are present, fc_layers takes precedent\n over num_fc_layers. Specifying num_fc_layers alone uses fully connected\n layers that are configured by the defaults in FCStack.\n related_parameters:\n - fc_layers\n suggested_values: 0-1\n suggested_values_reasoning:\n The full model likely contains many learnable\n parameters. Consider starting with very few, or without any additional\n fully connected layers and add them if you observe evidence of limited\n model capacity. Sometimes the default is 1 for modules where the FC stack\n is used for shape management, or the only source of learnable parameters.\n ui_display_name: Number of Fully Connected Layers\n out_channels:\n ui_display_name: null\n output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 10 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\n resnet_size:\n ui_display_name: null\n use_bias:\n ui_display_name: null\n expected_impact: 1\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\n\n\n Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\n width:\n internal_only: true\n ui_display_name: null\nDeBERTa:\n type:\n short_description: Improved version of BERT and RoBERTa, achieving good baseline performance on many tasks.\n long_description:\n The [DeBERTa](https://arxiv.org/abs/2006.03654) encoder improves the BERT and RoBERTa models using\n disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out performs RoBERTa\n on a majority of NLU tasks with 80GB training data.\n\n In [DeBERTa V3](https://arxiv.org/abs/2111.09543), the authors further improved the efficiency of DeBERTa\n using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa,\n the V3 version significantly improves the model performance on downstream tasks.\n compute_tier: 2\n literature_references:\n - https://arxiv.org/abs/2006.03654\n - https://arxiv.org/abs/2111.09543\n pretrained_model_name_or_path:\n default_value_reasoning:\n The default model was selected based on the benchmarking work done by IBM's\n [model recycling](https://ibm.github.io/model-recycling/microsoft_deberta-v3-base_table.html) project.\n In that study, the selected model ranked first among all variants of the `microsoft/deberta-v3-base`\n architecture on an evaluation across 36 different datasets.\n description_implications:\n Considerations when selecting a pretrained model version include number of parameters (how long the model\n will take to fine-tuning / perform inference), general model performance on various benchmarks, and\n specific model performance on the task you wish to fine-tune it on.\n expected_impact: 2\n related_parameters:\n - use_pretrained, trainable, pretrained_kwargs\n ui_display_name: Pretrained model\nRoBERTa:\n type:\n short_description: BERT based model that has higher accuracy and is easier parallelize due to larger mini-batches.\n long_description:\n The `roberta` encoder loads a pretrained [RoBERTa](https://arxiv.org/abs/1907.11692) (default `roberta-base`) model\n using the Hugging Face transformers package. Replication of BERT pretraining which may match or exceed the performance\n of BERT. RoBERTa builds on BERT and modifies key hyperparameters, removing the\n next-sentence pretraining objective and training with much larger mini-batches and learning\n rates.\n literature_references:\n - https://arxiv.org/abs/1907.11692\n compute_tier: 2\n bos_token_id:\n default_value_reasoning: Default value used in pre-trained HF encoder.\n ui_display_name: Beginning-of-Sentence Token Id\n eos_token_id:\n default_value_reasoning: \n example_value:\n - Default value used in pre-trained HF encoder.\n expected_impact: 1\n ui_display_name: null\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n pad_token_id:\n ui_display_name: null\n pretrained_kwargs:\n ui_display_name: null\n pretrained_model_name_or_path:\n ui_display_name: null\n expected_impact: 2\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n saved_weights_in_checkpoint:\n default_value_reasoning:\n The weights of the encoder are not necessarily saved\n in the checkpoint. The user has to save them first.\n description_implications:\n The memory footprint for some of these encoders\n can be large.\n internal_only: true\n related_parameters:\n - skip_save_model\n suggested_values:\n - false\n suggested_values_reasoning:\n Some of these encoders are large, so it might\n be better to load them as needed, especially if 1. they're not used frequently\n 2. the user doesn't have a lot of storage.\n ui_display_name: null\n trainable:\n expected_impact: 3\n ui_display_name: null\n use_pretrained:\n default_value_reasoning:\n By default, the model is initialized as a pretrained\n model.\n description_implications:\n Pretrained models have typically already learned\n features that are difficult to learn from scratch. They are particularly\n beneficial when training on small amounts of data.\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/transfer-learning-for-deep-learning/\n related_parameters:\n - trainable, pretrained_model_name, pretrained_model_name_or_path, pretrained_kwargs\n suggested_values:\n - false\n suggested_values_reasoning:\n If you have a large amount of data and/or you\n have data that differs from the typical distribution, then it might be\n worth training the model from scratch.\n ui_display_name: Use Pretrained\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n vocab_size:\n internal_only: true\n ui_display_name: Not displayed\nSequenceEmbed:\n type:\n short_description: Maps each element of the sequence to an embedding.\n long_description:\n The embed encoder simply maps each integer in the sequence to an embedding, creating a `b x s x h`\n tensor where `b` is the batch size, `s` is the length of the sequence and `h` is the embedding\n size. The tensor is reduced along the `s` dimension to obtain a single vector of size `h` for each\n element of the batch.\n dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n embedding_size:\n default_value_reasoning: Not too big, not too small.\n description_implications:\n 'An embedding is a relatively low-dimensional space\n that is used to translate high-dimensional vectors like words, which can\n have a large vocbulary size. Ideally, after an embedding is trained, it\n captures some of the semantics of the input by placing semantically similar\n inputs close together in the embedding space.\n\n\n In most cases, the embedding size is chosen empirically, by trial and\n error. From https://www.amazon.com/dp/1098115783, \"one rule of thumb is\n to use the fourth root of the total number of unique categorical elements\n while another is that the embedding dimension should be approximately\n 1.6 times the square root of the number of unique elements in the category,\n and no less than 600.\"\n\n\n Increasing the embedding size may cause the model to train more slowly,\n but the higher dimensionality can also improve overall quality.'\n expected_impact: 3\n literature_references:\n - https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture\n suggested_values: 1.6 * sqrt(vocab_size)\n suggested_values_reasoning:\n Rule of thumb suggested by a deep learning textbook.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Embedding Size\n embeddings_on_cpu:\n default_value_reasoning:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access.\n description_implications:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access. However, in some\n cases when the vocabulary size is very large, the full embedding matrix\n may be really big and unwieldy to have in GPU memory. This parameter forces\n the placement of the embedding matrix in regular memory and the CPU is\n used to access them. This may slow down training due to additional data\n transfer between CPU and GPU memory, but can lead to healthier GPU memory\n resource usage.\n expected_impact: 1\n suggested_values:\n - false\n suggested_values_reasoning:\n If GPU memory is not a constraint, having embeddings\n stored and accessed within the GPU is faster.\n ui_display_name: Embeddings on CPU\n embeddings_trainable:\n ui_display_name: null\n expected_impact: 1\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n pretrained_embeddings:\n ui_display_name: null\n expected_impact: 0\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n representation:\n ui_display_name: null\n expected_impact: 1\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\n\n\n Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\nSequencePassthrough:\n type:\n short_description: Transforms sequence values to a floats then reduces to obtain a vector for each element.\n long_description:\n The passthrough encoder simply transforms each input value into a float value and adds a\n dimension to the input tensor, creating a b x s x 1 tensor where b is the batch size and s is\n the length of the sequence. The tensor is reduced along the s dimension to obtain a single\n vector of size h for each element of the batch.\n encoding_size:\n default_value_reasoning:\n The default `reduce_output` method does not use this\n parameter, so by default this parameter is not set.\n description_implications:\n This parameter must be equal to the size of the\n input. Otherwise, an error will occur.\n example_value:\n - 128\n expected_impact: 1\n related_parameters:\n - reduce_output\n suggested_values_reasoning: NONE\n ui_display_name: null\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n reduce_output:\n ui_display_name: null\n expected_impact: 1\nSetSparseEncoder:\n type:\n short_description: Maps raw values to sparse integer lists, then maps to dense/sparse embeddings, then reduces to final vector.\n long_description:\n The Embed encoder takes the raw binary values coming from the input placeholders and transforms\n them to sparse integer lists, then they are mapped to either dense or sparse embeddings (one-hot\n encodings), finally they are reduced on the sequence dimension and returned as an aggregated\n embedding vector. Inputs are of size b while outputs are of size b x h where b is the batch size\n and h is the dimensionality of the embeddings.\n activation:\n default_value_reasoning:\n The Rectified Linear Units (ReLU) function is the\n standard activation function used for adding non-linearity. It is simple,\n fast, and empirically works well (https://arxiv.org/abs/1803.08375).\n description_implications:\n Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n expected_impact: 2\n suggested_values:\n The default value will work well in the majority of the\n cases\n ui_display_name: Activation\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n embedding_size:\n default_value_reasoning: Not too big, not too small.\n description_implications:\n 'An embedding is a relatively low-dimensional space\n that is used to translate high-dimensional vectors like words, which can\n have a large vocbulary size. Ideally, after an embedding is trained, it\n captures some of the semantics of the input by placing semantically similar\n inputs close together in the embedding space.\n\n\n In most cases, the embedding size is chosen empirically, by trial and\n error. From https://www.amazon.com/dp/1098115783, \"one rule of thumb is\n to use the fourth root of the total number of unique categorical elements\n while another is that the embedding dimension should be approximately\n 1.6 times the square root of the number of unique elements in the category,\n and no less than 600.\"\n\n\n Increasing the embedding size may cause the model to train more slowly,\n but the higher dimensionality can also improve overall quality.'\n expected_impact: 3\n literature_references:\n - https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture\n suggested_values: 1.6 * sqrt(vocab_size)\n suggested_values_reasoning:\n Rule of thumb suggested by a deep learning textbook.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Embedding Size\n embeddings_on_cpu:\n default_value_reasoning:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access.\n description_implications:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access. However, in some\n cases when the vocabulary size is very large, the full embedding matrix\n may be really big and unwieldy to have in GPU memory. This parameter forces\n the placement of the embedding matrix in regular memory and the CPU is\n used to access them. This may slow down training due to additional data\n transfer between CPU and GPU memory, but can lead to healthier GPU memory\n resource usage.\n expected_impact: 1\n suggested_values:\n - false\n suggested_values_reasoning:\n If GPU memory is not a constraint, having embeddings\n stored and accessed within the GPU is faster.\n ui_display_name: Embeddings on CPU\n embeddings_trainable:\n ui_display_name: null\n expected_impact: 1\n fc_layers:\n default_value_reasoning:\n By default the stack is built by using num_fc_layers,\n output_size, use_bias, weights_initializer, bias_initializer, norm, norm_params,\n activation, dropout. When a list of dictionaries is provided, the stack\n is built following the parameters of each dict for building each layer.\n description_implications:\n The more layers that are specified the deeper and\n higher capacity the model will be. This makes it possible to potentially\n achieve better performance when a big anough amount of data is provided,\n but also makes the model more computationally expensive and potentially\n more prone to overfitting.\n example_value:\n - dropout: 0.1\n output_size: 128\n - norm: layer\n output_size: 64\n expected_impact: 1\n related_parameters:\n - output_size\n - use_bias\n - weights_initializer\n - bias_initializer\n - norm\n - norm_params\n - activation\n - dropout\n suggested_values_reasoning:\n It is easier to define a stack of fully connected\n layers by just specifying num_fc_layers, output_size and the other individual\n parameters. It will create a stack of layers with identical properties.\n Use this parameter only if you need a fine grained level of control of\n each individual layer in the stack.\n ui_display_name: Fully Connected Layers\n norm:\n default_value_reasoning:\n While batch normalization and layer normalization\n usually lead to improvements, it can be useful to start with fewer bells\n and whistles.\n description_implications:\n Normalization helps stabilize the learning process\n and can have a regularizing effect that can help with generalization.\n It's often suggested that with normalization, you can use a higher learning\n rate.\n example_value:\n - batch\n expected_impact: 2\n literature_references:\n - https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/\n related_parameters:\n - norm_params\n suggested_values: '\"batch\" or \"layer\"'\n suggested_values_reasoning:\n Normalization tries to solve \"internal covariate\n shift\" that comes from the changing distributions of the inputs to layers\n deep in the network when weights are updated. For example, batch normalization\n standardizes the inputs to a layer for each mini-batch. Try out different\n normalizations to see if that helps with training stability\n ui_display_name: Normalization Type\n norm_params:\n default_value_reasoning:\n The default parameters that come with Torch's implementation\n of these normalization types are a trusted starting point.\n description_implications:\n There are a variety of ways a certain set of parameters\n specificed could influence performance here. Broadly speaking the different\n values passed in here allow for different levels of smoothness to be observed\n in the learning curves. Since setting this parameters depends on the type\n of `norm` set, see [BatchNorm2d](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html)\n for more information on the parameters to set for batch normalization,\n and see [LayerNorm](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html)\n for more information on the parameters to set for layer normalization.\n example_value:\n - affine: false\n momentum: 0.2\n num_features: 100\n expected_impact: 1\n literature_references:\n - \"For BatchNorm2d: https://arxiv.org/abs/1502.03167\n\n\n For LayerNorm: https://arxiv.org/abs/1607.06450\"\n related_parameters:\n - \"`norm`\"\n suggested_values: Depends on the type of `norm` set.\n suggested_values_reasoning: \"NO\"\n ui_display_name: Normalization Parameters\n num_fc_layers:\n default_value_reasoning:\n The encoder already has learnable parameters.Sometimes\n the default is 1 for modules where the FC stack is used for shape management,\n or the only source of learnable parameters.\n description_implications:\n Increasing num_fc_layers will increase the capacity\n of the model. The model will be slower to train, and there's a higher\n risk of overfitting.\n example_value:\n - 1\n expected_impact: 1\n other_information:\n Not all modules that have fc_layers also have an accompanying\n num_fc_layers parameter. Where both are present, fc_layers takes precedent\n over num_fc_layers. Specifying num_fc_layers alone uses fully connected\n layers that are configured by the defaults in FCStack.\n related_parameters:\n - fc_layers\n suggested_values: 0-1\n suggested_values_reasoning:\n The full model likely contains many learnable\n parameters. Consider starting with very few, or without any additional\n fully connected layers and add them if you observe evidence of limited\n model capacity. Sometimes the default is 1 for modules where the FC stack\n is used for shape management, or the only source of learnable parameters.\n ui_display_name: Number of Fully Connected Layers\n output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 10 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\n pretrained_embeddings:\n ui_display_name: null\n expected_impact: 0\n representation:\n ui_display_name: null\n expected_impact: 1\n use_bias:\n default_value_reasoning:\n \"Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to use bias terms.\n\n\n Batch Normalization, however, adds a trainable shift parameter which is\n added to the activation. When Batch Normalization is used in a layer,\n bias terms are redundant and may be removed.\"\n description_implications:\n Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to leave this parameter set to True.\n example_value:\n - true\n expected_impact: 1\n other_information:\n If fc_layers is not specified, or use_bias is not specified\n for individual layers, the value of use_bias will be used as the default\n for all layers.\n related_parameters:\n - bias_initializer, fc_layers\n suggested_values:\n - true\n ui_display_name: Use Bias\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\n\n\n Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\nStacked2DCNN:\n type:\n short_description: Stack of 2D convolutional layers followed by an optional stack of fully connected layers.\n long_description:\n Stack of 2D convolutional layers with optional normalization, dropout, and down-sampling\n pooling layers, followed by an optional stack of fully connected layers.\n compute_tier: 1\n conv_activation:\n expected_impact: 1\n ui_display_name: Convolutional Activation\n conv_bias:\n expected_impact: 1\n ui_display_name: Convolutional Bias\n conv_dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - \"conv_dropout,\n\n fc_dropout\"\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Convolutional Dropout\n conv_norm:\n expected_impact: 2\n ui_display_name: Convolutional Normalization\n conv_norm_params:\n expected_impact: 1\n ui_display_name: Convolutional Normalization Parameters\n dilation:\n expected_impact: 1\n ui_display_name: Dilation\n fc_activation:\n default_value_reasoning:\n The Rectified Linear Units (ReLU) function is the\n standard activation function used for adding non-linearity. It is simple,\n fast, and empirically works well (https://arxiv.org/abs/1803.08375).\n description_implications:\n Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n example_value:\n - relu\n expected_impact: 1\n literature_references:\n - https://ml-cheatsheet.readthedocs.io/en/latest/activation_functions.html\n related_parameters:\n - activation, activation_function, conv_activation, recurrent_activation\n suggested_values: relu, alternatively leakyRelu or elu\n suggested_values_reasoning:\n The default value will work well in the majority\n of the cases\n ui_display_name: FC Activation\n fc_bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n fc_dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 1\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - \"conv_dropout,\n\n fc_dropout\"\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: FC Dropout\n fc_layers:\n default_value_reasoning:\n By default the stack is built by using num_fc_layers,\n output_size, use_bias, weights_initializer, bias_initializer, norm, norm_params,\n activation, dropout. When a list of dictionaries is provided, the stack\n is built following the parameters of each dict for building each layer.\n description_implications:\n The more layers that are specified the deeper and\n higher capacity the model will be. This makes it possible to potentially\n achieve better performance when a big anough amount of data is provided,\n but also makes the model more computationally expensive and potentially\n more prone to overfitting.\n example_value:\n - dropout: 0.1\n output_size: 128\n - norm: layer\n output_size: 64\n expected_impact: 1\n related_parameters:\n - output_size\n - use_bias\n - weights_initializer\n - bias_initializer\n - norm\n - norm_params\n - activation\n - dropout\n suggested_values_reasoning:\n It is easier to define a stack of fully connected\n layers by just specifying num_fc_layers, output_size and the other individual\n parameters. It will create a stack of layers with identical properties.\n Use this parameter only if you need a fine grained level of control of\n each individual layer in the stack.\n ui_display_name: Fully Connected Layers\n fc_norm:\n description_implications:\n Normalization helps stabilize the learning process\n and can have a regularizing effect that can help with generalization.\n It's often suggested that with normalization, you can use a higher learning\n rate. See Torch's documentation on batch normalization or for layer see\n Torch's documentation on layer normalization.\n expected_impact: 2\n related_parameters:\n - fc_norm_params\n suggested_values: batch\n ui_display_name: Fully Connected Normalization\n fc_norm_params:\n description_implications:\n There are a variety of ways a certain set of parameters\n specificed could influence performance here. Broadly speaking the different\n values passed in here allow for different levels of smoothness to be observed\n in the learning curves. Since setting this parameters depends on the type\n of `norm` set, see [BatchNorm2d](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html)\n for more information on the parameters to set for batch normalization,\n and see [LayerNorm](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html)\n for more information on the parameters to set for layer normalization.\n expected_impact: 2\n related_parameters:\n - fc_norm\n suggested_values: Depends on the type of `norm` set.\n ui_display_name: Fully Connected Normalization Parameters\n fc_use_bias:\n expected_impact: 1\n ui_display_name: FC Use Bias\n fc_weights_initializer:\n expected_impact: 1\n ui_display_name: FC Weights Initializer\n groups:\n expected_impact: 1\n ui_display_name: Groups\n height:\n default_value_reasoning:\n Computed internally, automatically, based on image\n data preprocessing.\n internal_only: true\n ui_display_name: NOT DISPLAYED\n kernel_size:\n expected_impact: 1\n ui_display_name: Kernel Size\n num_channels:\n default_value_reasoning:\n Computed internally, automatically, based on image\n data preprocessing.\n ui_display_name: NOT DISPLAYED\n num_fc_layers:\n default_value_reasoning:\n The encoder already has learnable parameters.Sometimes\n the default is 1 for modules where the FC stack is used for shape management,\n or the only source of learnable parameters.\n description_implications:\n Increasing num_fc_layers will increase the capacity\n of the model. The model will be slower to train, and there's a higher\n risk of overfitting.\n example_value:\n - 1\n expected_impact: 1\n other_information:\n Not all modules that have fc_layers also have an accompanying\n num_fc_layers parameter. Where both are present, fc_layers takes precedent\n over num_fc_layers. Specifying num_fc_layers alone uses fully connected\n layers that are configured by the defaults in FCStack.\n related_parameters:\n - fc_layers\n suggested_values: 0-1\n suggested_values_reasoning:\n The full model likely contains many learnable\n parameters. Consider starting with very few, or without any additional\n fully connected layers and add them if you observe evidence of limited\n model capacity. Sometimes the default is 1 for modules where the FC stack\n is used for shape management, or the only source of learnable parameters.\n ui_display_name: Number of Fully Connected Layers\n out_channels:\n expected_impact: 2\n ui_display_name: Number of Output Channels\n output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 10 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\n padding:\n default_value_reasoning:\n When padding is set to 'valid' like in the default\n case, no padding is added. As a default value putting in the raw image\n is the goal here.\n description_implications:\n By increasing the amount of padding, you can increase\n the accuracy of the image analysis for certain circumstances.\n example_value:\n - \"'same'\"\n expected_impact: 1\n literature_references:\n - https://www.geeksforgeeks.org/cnn-introduction-to-padding/\n related_parameters:\n - \"padding_mode,\n\n resize method\"\n suggested_values:\n \"Same' padding if images are of different dimensions. \\n\\\n Specific [h, w] entries can be valuable on a per dataset basis.\"\n suggested_values_reasoning:\n If your images already have padding, there is\n no need to add padding, so the default is fine. If your images come in\n different dimensions, then 'same' padding can help pad the images to standardized\n dimensions. For certain images, adding padding to the edges can help the\n CNN process the images better which can improve model performance. This\n depends on the images however.\n ui_display_name: Padding\n padding_mode:\n expected_impact: 1\n ui_display_name: Padding Mode\n pool_dilation:\n expected_impact: 1\n ui_display_name: Pool Dilation\n pool_kernel_size:\n expected_impact: 1\n ui_display_name: Pool Kernel Size\n pool_padding:\n expected_impact: 1\n ui_display_name: Pool Padding\n pool_stride:\n expected_impact: 1\n ui_display_name: Pool Stride\n stride:\n expected_impact: 1\n ui_display_name: Stride\n width:\n default_value_reasoning:\n Computed internally, automatically, based on image\n data preprocessing.\n internal_only: true\n ui_display_name: NOT DISPLAYED\nStackedCNN:\n type:\n short_description: Maps inputs to embeddings then passes them through a stack of 1d convolutional layers.\n long_description:\n The Stacked CNN works by first mapping the input integer sequence b x s (where b is the batch\n size and s is the length of the sequence) into a sequence of embeddings, then it passes the\n embedding through a stack of 1d convolutional layers with different filter size (by default 6\n layers with filter size 7, 7, 3, 3, 3 and 3), followed by an optional final pool and by a\n flatten operation. This single flatten vector is then passed through a stack of fully connected\n layers and returned as a b x h tensor where h is the output size of the last fully connected\n layer.\n compute_tier: 1\n activation:\n default_value_reasoning:\n The Rectified Linear Units (ReLU) function is the\n standard activation function used for adding non-linearity. It is simple,\n fast, and empirically works well (https://arxiv.org/abs/1803.08375).\n description_implications:\n Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n expected_impact: 2\n suggested_values:\n The default value will work well in the majority of the\n cases\n ui_display_name: Activation\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n dilation_rate:\n default_value_reasoning:\n The standard discrete convolution is the same as\n a 1-dilated convolution.\n description_implications:\n Higher dilation rates increase the effective size\n of the convolutional filter. Dilated convolution may improve performance\n if the data is very correlated locally and also contains long-term dependencies.\n example_value:\n - 2\n expected_impact: 1\n other_information: Dilated convolution is also known as atrous convolution.\n related_parameters:\n - filter_size\n suggested_values: 1-3\n suggested_values_reasoning:\n The dilation rate is a factor which increases\n the spacing between elements of the convolutional filter\n ui_display_name: Dilation Rate\n dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n embedding_size:\n default_value_reasoning: Not too big, not too small.\n description_implications:\n 'An embedding is a relatively low-dimensional space\n that is used to translate high-dimensional vectors like words, which can\n have a large vocbulary size. Ideally, after an embedding is trained, it\n captures some of the semantics of the input by placing semantically similar\n inputs close together in the embedding space.\n\n\n In most cases, the embedding size is chosen empirically, by trial and\n error. From https://www.amazon.com/dp/1098115783, \"one rule of thumb is\n to use the fourth root of the total number of unique categorical elements\n while another is that the embedding dimension should be approximately\n 1.6 times the square root of the number of unique elements in the category,\n and no less than 600.\"\n\n\n Increasing the embedding size may cause the model to train more slowly,\n but the higher dimensionality can also improve overall quality.'\n expected_impact: 3\n literature_references:\n - https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture\n suggested_values: 1.6 * sqrt(vocab_size)\n suggested_values_reasoning:\n Rule of thumb suggested by a deep learning textbook.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Embedding Size\n embeddings_on_cpu:\n default_value_reasoning:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access.\n description_implications:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access. However, in some\n cases when the vocabulary size is very large, the full embedding matrix\n may be really big and unwieldy to have in GPU memory. This parameter forces\n the placement of the embedding matrix in regular memory and the CPU is\n used to access them. This may slow down training due to additional data\n transfer between CPU and GPU memory, but can lead to healthier GPU memory\n resource usage.\n expected_impact: 1\n suggested_values:\n - false\n suggested_values_reasoning:\n If GPU memory is not a constraint, having embeddings\n stored and accessed within the GPU is faster.\n ui_display_name: Embeddings on CPU\n embeddings_trainable:\n ui_display_name: null\n expected_impact: 1\n fc_layers:\n default_value_reasoning:\n By default the stack is built by using num_fc_layers,\n output_size, use_bias, weights_initializer, bias_initializer, norm, norm_params,\n activation, dropout. When a list of dictionaries is provided, the stack\n is built following the parameters of each dict for building each layer.\n description_implications:\n The more layers that are specified the deeper and\n higher capacity the model will be. This makes it possible to potentially\n achieve better performance when a big anough amount of data is provided,\n but also makes the model more computationally expensive and potentially\n more prone to overfitting.\n example_value:\n - dropout: 0.1\n output_size: 128\n - norm: layer\n output_size: 64\n expected_impact: 1\n related_parameters:\n - output_size\n - use_bias\n - weights_initializer\n - bias_initializer\n - norm\n - norm_params\n - activation\n - dropout\n suggested_values_reasoning:\n It is easier to define a stack of fully connected\n layers by just specifying num_fc_layers, output_size and the other individual\n parameters. It will create a stack of layers with identical properties.\n Use this parameter only if you need a fine grained level of control of\n each individual layer in the stack.\n ui_display_name: Fully Connected Layers\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n norm:\n default_value_reasoning:\n While batch normalization and layer normalization\n usually lead to improvements, it can be useful to start with fewer bells\n and whistles.\n description_implications:\n Normalization helps stabilize the learning process\n and can have a regularizing effect that can help with generalization.\n It's often suggested that with normalization, you can use a higher learning\n rate.\n example_value:\n - batch\n expected_impact: 2\n literature_references:\n - https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/\n related_parameters:\n - norm_params\n suggested_values: '\"batch\" or \"layer\"'\n suggested_values_reasoning:\n Normalization tries to solve \"internal covariate\n shift\" that comes from the changing distributions of the inputs to layers\n deep in the network when weights are updated. For example, batch normalization\n standardizes the inputs to a layer for each mini-batch. Try out different\n normalizations to see if that helps with training stability\n ui_display_name: Normalization Type\n norm_params:\n default_value_reasoning:\n The default parameters that come with Torch's implementation\n of these normalization types are a trusted starting point.\n description_implications:\n There are a variety of ways a certain set of parameters\n specificed could influence performance here. Broadly speaking the different\n values passed in here allow for different levels of smoothness to be observed\n in the learning curves. Since setting this parameters depends on the type\n of `norm` set, see [BatchNorm2d](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html)\n for more information on the parameters to set for batch normalization,\n and see [LayerNorm](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html)\n for more information on the parameters to set for layer normalization.\n example_value:\n - affine: false\n momentum: 0.2\n num_features: 100\n expected_impact: 1\n literature_references:\n - \"For BatchNorm2d: https://arxiv.org/abs/1502.03167\n\n\n For LayerNorm: https://arxiv.org/abs/1607.06450\"\n related_parameters:\n - \"`norm`\"\n suggested_values: Depends on the type of `norm` set.\n suggested_values_reasoning: \"NO\"\n ui_display_name: Normalization Parameters\n num_fc_layers:\n default_value_reasoning:\n The encoder already has learnable parameters.Sometimes\n the default is 1 for modules where the FC stack is used for shape management,\n or the only source of learnable parameters.\n description_implications:\n Increasing num_fc_layers will increase the capacity\n of the model. The model will be slower to train, and there's a higher\n risk of overfitting.\n example_value:\n - 1\n expected_impact: 1\n other_information:\n Not all modules that have fc_layers also have an accompanying\n num_fc_layers parameter. Where both are present, fc_layers takes precedent\n over num_fc_layers. Specifying num_fc_layers alone uses fully connected\n layers that are configured by the defaults in FCStack.\n related_parameters:\n - fc_layers\n suggested_values: 0-1\n suggested_values_reasoning:\n The full model likely contains many learnable\n parameters. Consider starting with very few, or without any additional\n fully connected layers and add them if you observe evidence of limited\n model capacity. Sometimes the default is 1 for modules where the FC stack\n is used for shape management, or the only source of learnable parameters.\n ui_display_name: Number of Fully Connected Layers\n num_filters:\n ui_display_name: null\n output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 10 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\n padding:\n ui_display_name: null\n pool_function:\n ui_display_name: null\n expected_impact: 1\n pool_padding:\n ui_display_name: null\n expected_impact: 1\n pool_size:\n ui_display_name: null\n expected_impact: 1\n pool_strides:\n ui_display_name: null\n expected_impact: 1\n pretrained_embeddings:\n ui_display_name: null\n expected_impact: 0\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n representation:\n ui_display_name: null\n expected_impact: 1\n should_embed:\n internal_only: true\n ui_display_name: Not displayed\n strides:\n default_value_reasoning:\n In general, it makes sense to have a smaller stride\n that fits the input. Imagining the simple 2D image as our input, two pixels\n next to eachother are strongly correlated while pixels that are further\n apart will have a comparatively weaker correlation. Consequently, a higher\n stride may cause significant information loss.\n description_implications:\n Changing the stride of a convolutional layer is\n one form of downsampling (another being pooling). In the case of a large\n stride, significant amounts of information is thrown away as the filter\n convolves over its input. This should be usually avoided but may be desirable\n in cases in which the user has some deep knowledge of the filter or of\n the rest of the model architecture that makes it comfortable to allow\n a higher level compression in the output feature map of this layer.\n example_value:\n - 1\n expected_impact: 2\n literature_references:\n - \"[d2l.ai blog post](http://d2l.ai/chapter_convolutional-neural-networks/padding-and-strides.html)\n\n\n [machinelearningmastery blogpost](https://machinelearningmastery.com/padding-and-stride-for-convolutional-neural-networks/)\n\n\n [crossvalidated discussion](https://stats.stackexchange.com/questions/296027/choosing-filter-size-strides-etc-in-a-cnn)\"\n related_parameters:\n - pool_strides, default_strides, default_pool_strides, block_strides\n suggested_values: 1-2\n suggested_values_reasoning:\n In general, points that are closer to eachother\n in the input feature space will be more strongly correlated to eachother,\n so it is a good idea to select a stride that captures these neighboring\n relationships.\n ui_display_name: Stride\n use_bias:\n ui_display_name: null\n expected_impact: 1\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\n\n\n Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\nStackedCNNRNN:\n type:\n short_description: Maps inputs to embeddings, passes them through convolutional layer stack, then recurrent layer stack.\n long_description:\n The cnnrnn encoder works by first mapping the input integer sequence b x s (where b is the batch\n size and s is the length of the sequence) into a sequence of embeddings, then it passes the\n embedding through a stack of convolutional layers (by default 2), that is followed by a stack of\n recurrent layers (by default 1), followed by a reduce operation that by default only returns the\n last output, but can perform other reduce functions.\n compute_tier: 1\n activation:\n ui_display_name: null\n expected_impact: 2\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n bidirectional:\n ui_display_name: null\n expected_impact: 0\n cell_type:\n ui_display_name: null\n expected_impact: 3\n conv_activation:\n ui_display_name: null\n expected_impact: 1\n conv_dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - \"conv_dropout,\n\n dropout,\n\n recurrent_dropout,\n\n fc_dropout\"\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Convolutional Dropout\n dilation_rate:\n default_value_reasoning:\n The standard discrete convolution is the same as\n a 1-dilated convolution.\n description_implications:\n Higher dilation rates increase the effective size\n of the convolutional filter. Dilated convolution may improve performance\n if the data is very correlated locally and also contains long-term dependencies.\n example_value:\n - 2\n expected_impact: 1\n other_information: Dilated convolution is also known as atrous convolution.\n related_parameters:\n - filter_size\n suggested_values: 1-3\n suggested_values_reasoning:\n The dilation rate is a factor which increases\n the spacing between elements of the convolutional filter\n ui_display_name: Dilation Rate\n dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - \"conv_dropout,\n\n dropout,\n\n recurrent_dropout,\n\n fc_dropout\"\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n embedding_size:\n default_value_reasoning: Not too big, not too small.\n description_implications:\n 'An embedding is a relatively low-dimensional space\n that is used to translate high-dimensional vectors like words, which can\n have a large vocbulary size. Ideally, after an embedding is trained, it\n captures some of the semantics of the input by placing semantically similar\n inputs close together in the embedding space.\n\n\n In most cases, the embedding size is chosen empirically, by trial and\n error. From https://www.amazon.com/dp/1098115783, \"one rule of thumb is\n to use the fourth root of the total number of unique categorical elements\n while another is that the embedding dimension should be approximately\n 1.6 times the square root of the number of unique elements in the category,\n and no less than 600.\"\n\n\n Increasing the embedding size may cause the model to train more slowly,\n but the higher dimensionality can also improve overall quality.'\n expected_impact: 3\n literature_references:\n - https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture\n suggested_values: 1.6 * sqrt(vocab_size)\n suggested_values_reasoning:\n Rule of thumb suggested by a deep learning textbook.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Embedding Size\n embeddings_on_cpu:\n default_value_reasoning:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access.\n description_implications:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access. However, in some\n cases when the vocabulary size is very large, the full embedding matrix\n may be really big and unwieldy to have in GPU memory. This parameter forces\n the placement of the embedding matrix in regular memory and the CPU is\n used to access them. This may slow down training due to additional data\n transfer between CPU and GPU memory, but can lead to healthier GPU memory\n resource usage.\n expected_impact: 1\n suggested_values:\n - false\n suggested_values_reasoning:\n If GPU memory is not a constraint, having embeddings\n stored and accessed within the GPU is faster.\n ui_display_name: Embeddings on CPU\n embeddings_trainable:\n ui_display_name: null\n expected_impact: 1\n fc_activation:\n default_value_reasoning:\n The Rectified Linear Units (ReLU) function is the\n standard activation function used for adding non-linearity. It is simple,\n fast, and empirically works well (https://arxiv.org/abs/1803.08375).\n description_implications:\n Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n example_value:\n - relu\n expected_impact: 1\n literature_references:\n - https://ml-cheatsheet.readthedocs.io/en/latest/activation_functions.html\n related_parameters:\n - activation, activation_function, conv_activation, recurrent_activation\n suggested_values: relu, alternatively leakyRelu or elu\n suggested_values_reasoning:\n The default value will work well in the majority\n of the cases\n ui_display_name: FC Activation\n fc_dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 1\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - \"conv_dropout,\n\n dropout,\n\n recurrent_dropout,\n\n fc_dropout\"\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: FC Dropout\n fc_layers:\n default_value_reasoning:\n By default the stack is built by using num_fc_layers,\n output_size, use_bias, weights_initializer, bias_initializer, norm, norm_params,\n activation, dropout. When a list of dictionaries is provided, the stack\n is built following the parameters of each dict for building each layer.\n description_implications:\n The more layers that are specified the deeper and\n higher capacity the model will be. This makes it possible to potentially\n achieve better performance when a big anough amount of data is provided,\n but also makes the model more computationally expensive and potentially\n more prone to overfitting.\n example_value:\n - dropout: 0.1\n output_size: 128\n - norm: layer\n output_size: 64\n expected_impact: 1\n related_parameters:\n - output_size\n - use_bias\n - weights_initializer\n - bias_initializer\n - norm\n - norm_params\n - activation\n - dropout\n suggested_values_reasoning:\n It is easier to define a stack of fully connected\n layers by just specifying num_fc_layers, output_size and the other individual\n parameters. It will create a stack of layers with identical properties.\n Use this parameter only if you need a fine grained level of control of\n each individual layer in the stack.\n ui_display_name: Fully Connected Layers\n filter_size:\n ui_display_name: null\n expected_impact: 2\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n norm:\n default_value_reasoning:\n While batch normalization and layer normalization\n usually lead to improvements, it can be useful to start with fewer bells\n and whistles.\n description_implications:\n Normalization helps stabilize the learning process\n and can have a regularizing effect that can help with generalization.\n It's often suggested that with normalization, you can use a higher learning\n rate.\n example_value:\n - batch\n expected_impact: 2\n literature_references:\n - https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/\n related_parameters:\n - norm_params\n suggested_values: '\"batch\" or \"layer\"'\n suggested_values_reasoning:\n Normalization tries to solve \"internal covariate\n shift\" that comes from the changing distributions of the inputs to layers\n deep in the network when weights are updated. For example, batch normalization\n standardizes the inputs to a layer for each mini-batch. Try out different\n normalizations to see if that helps with training stability\n ui_display_name: Normalization Type\n norm_params:\n default_value_reasoning:\n The default parameters that come with Torch's implementation\n of these normalization types are a trusted starting point.\n description_implications:\n There are a variety of ways a certain set of parameters\n specificed could influence performance here. Broadly speaking the different\n values passed in here allow for different levels of smoothness to be observed\n in the learning curves. Since setting this parameters depends on the type\n of `norm` set, see [BatchNorm2d](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html)\n for more information on the parameters to set for batch normalization,\n and see [LayerNorm](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html)\n for more information on the parameters to set for layer normalization.\n example_value:\n - affine: false\n momentum: 0.2\n num_features: 100\n expected_impact: 1\n literature_references:\n - \"For BatchNorm2d: https://arxiv.org/abs/1502.03167\n\n\n For LayerNorm: https://arxiv.org/abs/1607.06450\"\n related_parameters:\n - \"`norm`\"\n suggested_values: Depends on the type of `norm` set.\n suggested_values_reasoning: \"NO\"\n ui_display_name: Normalization Parameters\n num_fc_layers:\n default_value_reasoning:\n The encoder already has learnable parameters.Sometimes\n the default is 1 for modules where the FC stack is used for shape management,\n or the only source of learnable parameters.\n description_implications:\n Increasing num_fc_layers will increase the capacity\n of the model. The model will be slower to train, and there's a higher\n risk of overfitting.\n example_value:\n - 1\n expected_impact: 1\n other_information:\n Not all modules that have fc_layers also have an accompanying\n num_fc_layers parameter. Where both are present, fc_layers takes precedent\n over num_fc_layers. Specifying num_fc_layers alone uses fully connected\n layers that are configured by the defaults in FCStack.\n related_parameters:\n - fc_layers\n suggested_values: 0-1\n suggested_values_reasoning:\n The full model likely contains many learnable\n parameters. Consider starting with very few, or without any additional\n fully connected layers and add them if you observe evidence of limited\n model capacity. Sometimes the default is 1 for modules where the FC stack\n is used for shape management, or the only source of learnable parameters.\n ui_display_name: Number of Fully Connected Layers\n num_filters:\n ui_display_name: null\n num_rec_layers:\n ui_display_name: null\n output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 10 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\n padding:\n ui_display_name: null\n pool_function:\n ui_display_name: null\n expected_impact: 1\n pool_padding:\n ui_display_name: null\n expected_impact: 1\n pool_size:\n ui_display_name: null\n expected_impact: 1\n pool_strides:\n ui_display_name: null\n expected_impact: 1\n pretrained_embeddings:\n ui_display_name: null\n expected_impact: 0\n recurrent_activation:\n default_value_reasoning: sigmoid' is commonly used\n expected_impact: 1\n other_information:\n I don't think that this parameter is used anywhere in the\n code base. It's being passed down but not used in the actual RNN forwarding\n functions.\n suggested_values: sigmoid, ReLu, tanh\n ui_display_name: null\n recurrent_dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 2\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - \"conv_dropout,\n\n dropout,\n\n recurrent_dropout,\n\n fc_dropout\"\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Recurrent Dropout\n recurrent_initializer:\n ui_display_name: null\n expected_impact: 1\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n representation:\n ui_display_name: null\n expected_impact: 1\n should_embed:\n internal_only: true\n ui_display_name: Not displayed\n state_size:\n ui_display_name: null\n expected_impact: 3\n strides:\n default_value_reasoning:\n In general, it makes sense to have a smaller stride\n that fits the input. Imagining the simple 2D image as our input, two pixels\n next to eachother are strongly correlated while pixels that are further\n apart will have a comparatively weaker correlation. Consequently, a higher\n stride may cause significant information loss.\n description_implications:\n Changing the stride of a convolutional layer is\n one form of downsampling (another being pooling). In the case of a large\n stride, significant amounts of information is thrown away as the filter\n convolves over its input. This should be usually avoided but may be desirable\n in cases in which the user has some deep knowledge of the filter or of\n the rest of the model architecture that makes it comfortable to allow\n a higher level compression in the output feature map of this layer.\n example_value:\n - 1\n expected_impact: 2\n literature_references:\n - \"[d2l.ai blog post](http://d2l.ai/chapter_convolutional-neural-networks/padding-and-strides.html)\n\n\n [machinelearningmastery blogpost](https://machinelearningmastery.com/padding-and-stride-for-convolutional-neural-networks/)\n\n\n [crossvalidated discussion](https://stats.stackexchange.com/questions/296027/choosing-filter-size-strides-etc-in-a-cnn)\"\n related_parameters:\n - pool_strides, default_strides, default_pool_strides, block_strides\n suggested_values: 1-2\n suggested_values_reasoning:\n In general, points that are closer to eachother\n in the input feature space will be more strongly correlated to eachother,\n so it is a good idea to select a stride that captures these neighboring\n relationships.\n ui_display_name: Stride\n unit_forget_bias:\n ui_display_name: null\n expected_impact: 1\n use_bias:\n default_value_reasoning:\n \"Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to use bias terms.\n\n\n Batch Normalization, however, adds a trainable shift parameter which is\n added to the activation. When Batch Normalization is used in a layer,\n bias terms are redundant and may be removed.\"\n description_implications:\n Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to leave this parameter set to True.\n example_value:\n - true\n expected_impact: 1\n other_information:\n If fc_layers is not specified, or use_bias is not specified\n for individual layers, the value of use_bias will be used as the default\n for all layers.\n related_parameters:\n - bias_initializer, fc_layers\n suggested_values:\n - true\n ui_display_name: Use Bias\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\n\n\n Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\nStackedParallelCNN:\n type:\n short_description: Combination of Parallel CNN and Stacked CNN encoders utilizing a stack of parallel convolutional layers.\n long_description:\n The stacked parallel cnn encoder is a combination of the Parallel CNN and the Stacked CNN\n encoders where each layer of the stack is composed of parallel convolutional layers. It works by\n first mapping the input integer sequence b x s (where b is the batch size and s is the length of\n the sequence) into a sequence of embeddings, then it passes the embedding through a stack of\n several parallel 1d convolutional layers with different filter size, followed by an optional\n final pool and by a flatten operation. This single flattened vector is then passed through a\n stack of fully connected layers and returned as a b x h tensor where h is the output size of the\n last fully connected layer.\n compute_tier: 1\n activation:\n default_value_reasoning:\n The Rectified Linear Units (ReLU) function is the\n standard activation function used for adding non-linearity. It is simple,\n fast, and empirically works well (https://arxiv.org/abs/1803.08375).\n description_implications:\n Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n expected_impact: 2\n suggested_values:\n The default value will work well in the majority of the\n cases\n ui_display_name: Activation\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n embedding_size:\n default_value_reasoning: Not too big, not too small.\n description_implications:\n 'An embedding is a relatively low-dimensional space\n that is used to translate high-dimensional vectors like words, which can\n have a large vocbulary size. Ideally, after an embedding is trained, it\n captures some of the semantics of the input by placing semantically similar\n inputs close together in the embedding space.\n\n\n In most cases, the embedding size is chosen empirically, by trial and\n error. From https://www.amazon.com/dp/1098115783, \"one rule of thumb is\n to use the fourth root of the total number of unique categorical elements\n while another is that the embedding dimension should be approximately\n 1.6 times the square root of the number of unique elements in the category,\n and no less than 600.\"\n\n\n Increasing the embedding size may cause the model to train more slowly,\n but the higher dimensionality can also improve overall quality.'\n expected_impact: 3\n literature_references:\n - https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture\n suggested_values: 1.6 * sqrt(vocab_size)\n suggested_values_reasoning:\n Rule of thumb suggested by a deep learning textbook.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Embedding Size\n embeddings_on_cpu:\n default_value_reasoning:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access.\n description_implications:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access. However, in some\n cases when the vocabulary size is very large, the full embedding matrix\n may be really big and unwieldy to have in GPU memory. This parameter forces\n the placement of the embedding matrix in regular memory and the CPU is\n used to access them. This may slow down training due to additional data\n transfer between CPU and GPU memory, but can lead to healthier GPU memory\n resource usage.\n expected_impact: 1\n suggested_values:\n - false\n suggested_values_reasoning:\n If GPU memory is not a constraint, having embeddings\n stored and accessed within the GPU is faster.\n ui_display_name: Embeddings on CPU\n embeddings_trainable:\n ui_display_name: null\n expected_impact: 1\n fc_layers:\n default_value_reasoning:\n By default the stack is built by using num_fc_layers,\n output_size, use_bias, weights_initializer, bias_initializer, norm, norm_params,\n activation, dropout. When a list of dictionaries is provided, the stack\n is built following the parameters of each dict for building each layer.\n description_implications:\n The more layers that are specified the deeper and\n higher capacity the model will be. This makes it possible to potentially\n achieve better performance when a big anough amount of data is provided,\n but also makes the model more computationally expensive and potentially\n more prone to overfitting.\n example_value:\n - dropout: 0.1\n output_size: 128\n - norm: layer\n output_size: 64\n expected_impact: 1\n related_parameters:\n - output_size\n - use_bias\n - weights_initializer\n - bias_initializer\n - norm\n - norm_params\n - activation\n - dropout\n suggested_values_reasoning:\n It is easier to define a stack of fully connected\n layers by just specifying num_fc_layers, output_size and the other individual\n parameters. It will create a stack of layers with identical properties.\n Use this parameter only if you need a fine grained level of control of\n each individual layer in the stack.\n ui_display_name: Fully Connected Layers\n filter_size:\n ui_display_name: null\n expected_impact: 2\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n norm:\n default_value_reasoning:\n While batch normalization and layer normalization\n usually lead to improvements, it can be useful to start with fewer bells\n and whistles.\n description_implications:\n Normalization helps stabilize the learning process\n and can have a regularizing effect that can help with generalization.\n It's often suggested that with normalization, you can use a higher learning\n rate.\n example_value:\n - batch\n expected_impact: 2\n literature_references:\n - https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/\n related_parameters:\n - norm_params\n suggested_values: '\"batch\" or \"layer\"'\n suggested_values_reasoning:\n Normalization tries to solve \"internal covariate\n shift\" that comes from the changing distributions of the inputs to layers\n deep in the network when weights are updated. For example, batch normalization\n standardizes the inputs to a layer for each mini-batch. Try out different\n normalizations to see if that helps with training stability\n ui_display_name: Normalization Type\n norm_params:\n default_value_reasoning:\n The default parameters that come with Torch's implementation\n of these normalization types are a trusted starting point.\n description_implications:\n There are a variety of ways a certain set of parameters\n specificed could influence performance here. Broadly speaking the different\n values passed in here allow for different levels of smoothness to be observed\n in the learning curves. Since setting this parameters depends on the type\n of `norm` set, see [BatchNorm2d](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html)\n for more information on the parameters to set for batch normalization,\n and see [LayerNorm](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html)\n for more information on the parameters to set for layer normalization.\n example_value:\n - affine: false\n momentum: 0.2\n num_features: 100\n expected_impact: 1\n literature_references:\n - \"For BatchNorm2d: https://arxiv.org/abs/1502.03167\n\n\n For LayerNorm: https://arxiv.org/abs/1607.06450\"\n related_parameters:\n - \"`norm`\"\n suggested_values: Depends on the type of `norm` set.\n suggested_values_reasoning: \"NO\"\n ui_display_name: Normalization Parameters\n num_fc_layers:\n default_value_reasoning:\n The encoder already has learnable parameters.Sometimes\n the default is 1 for modules where the FC stack is used for shape management,\n or the only source of learnable parameters.\n description_implications:\n Increasing num_fc_layers will increase the capacity\n of the model. The model will be slower to train, and there's a higher\n risk of overfitting.\n example_value:\n - 1\n expected_impact: 1\n other_information:\n Not all modules that have fc_layers also have an accompanying\n num_fc_layers parameter. Where both are present, fc_layers takes precedent\n over num_fc_layers. Specifying num_fc_layers alone uses fully connected\n layers that are configured by the defaults in FCStack.\n related_parameters:\n - fc_layers\n suggested_values: 0-1\n suggested_values_reasoning:\n The full model likely contains many learnable\n parameters. Consider starting with very few, or without any additional\n fully connected layers and add them if you observe evidence of limited\n model capacity. Sometimes the default is 1 for modules where the FC stack\n is used for shape management, or the only source of learnable parameters.\n ui_display_name: Number of Fully Connected Layers\n num_filters:\n ui_display_name: null\n num_stacked_layers:\n description_implications:\n While superceded by `stacked_layers`, this can directly\n change the depth of the current stack of parallel convolutional layers.\n example_value:\n - 1\n expected_impact: 1\n related_parameters:\n - stacked_layers\n ui_display_name: Number of Stacked Layers\n output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 10 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\n pool_function:\n ui_display_name: null\n expected_impact: 1\n pretrained_embeddings:\n ui_display_name: null\n expected_impact: 0\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n representation:\n ui_display_name: null\n expected_impact: 1\n should_embed:\n internal_only: true\n ui_display_name: Not displayed\n stacked_layers:\n ui_display_name: null\n use_bias:\n default_value_reasoning:\n \"Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to use bias terms.\n\n\n Batch Normalization, however, adds a trainable shift parameter which is\n added to the activation. When Batch Normalization is used in a layer,\n bias terms are redundant and may be removed.\"\n description_implications:\n Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to leave this parameter set to True.\n example_value:\n - true\n expected_impact: 1\n other_information:\n If fc_layers is not specified, or use_bias is not specified\n for individual layers, the value of use_bias will be used as the default\n for all layers.\n related_parameters:\n - bias_initializer, fc_layers\n suggested_values:\n - true\n ui_display_name: Use Bias\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\n\n\n Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\nStackedRNN:\n type:\n short_description: Utilizes a stack of recurrent layers followed by a reduce operation.\n long_description:\n The rnn encoder works by first mapping the input integer sequence b x s (where b is the batch\n size and s is the length of the sequence) into a sequence of embeddings, then it passes the\n embedding through a stack of recurrent layers (by default 1 layer), followed by a reduce\n operation that by default only returns the last output, but can perform other reduce functions.\n compute_tier: 1\n activation:\n ui_display_name: null\n expected_impact: 2\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n bidirectional:\n ui_display_name: null\n expected_impact: 0\n cell_type:\n ui_display_name: null\n expected_impact: 3\n dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - \"dropout,\n\n recurrent_dropout,\n\n fc_dropout\"\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n embedding_size:\n default_value_reasoning: Not too big, not too small.\n description_implications:\n 'An embedding is a relatively low-dimensional space\n that is used to translate high-dimensional vectors like words, which can\n have a large vocbulary size. Ideally, after an embedding is trained, it\n captures some of the semantics of the input by placing semantically similar\n inputs close together in the embedding space.\n\n\n In most cases, the embedding size is chosen empirically, by trial and\n error. From https://www.amazon.com/dp/1098115783, \"one rule of thumb is\n to use the fourth root of the total number of unique categorical elements\n while another is that the embedding dimension should be approximately\n 1.6 times the square root of the number of unique elements in the category,\n and no less than 600.\"\n\n\n Increasing the embedding size may cause the model to train more slowly,\n but the higher dimensionality can also improve overall quality.'\n expected_impact: 3\n literature_references:\n - https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture\n suggested_values: 1.6 * sqrt(vocab_size)\n suggested_values_reasoning:\n Rule of thumb suggested by a deep learning textbook.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Embedding Size\n embeddings_on_cpu:\n default_value_reasoning:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access.\n description_implications:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access. However, in some\n cases when the vocabulary size is very large, the full embedding matrix\n may be really big and unwieldy to have in GPU memory. This parameter forces\n the placement of the embedding matrix in regular memory and the CPU is\n used to access them. This may slow down training due to additional data\n transfer between CPU and GPU memory, but can lead to healthier GPU memory\n resource usage.\n expected_impact: 1\n suggested_values:\n - false\n suggested_values_reasoning:\n If GPU memory is not a constraint, having embeddings\n stored and accessed within the GPU is faster.\n ui_display_name: Embeddings on CPU\n embeddings_trainable:\n ui_display_name: null\n expected_impact: 1\n fc_activation:\n default_value_reasoning:\n The Rectified Linear Units (ReLU) function is the\n standard activation function used for adding non-linearity. It is simple,\n fast, and empirically works well (https://arxiv.org/abs/1803.08375).\n description_implications:\n Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n example_value:\n - relu\n expected_impact: 1\n literature_references:\n - https://ml-cheatsheet.readthedocs.io/en/latest/activation_functions.html\n related_parameters:\n - activation, activation_function, conv_activation, recurrent_activation\n suggested_values: relu, alternatively leakyRelu or elu\n suggested_values_reasoning:\n The default value will work well in the majority\n of the cases\n ui_display_name: FC Activation\n fc_dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 1\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - dropout, recurrent_dropout\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: FC Dropout\n fc_layers:\n default_value_reasoning:\n By default the stack is built by using num_fc_layers,\n output_size, use_bias, weights_initializer, bias_initializer, norm, norm_params,\n activation, dropout. When a list of dictionaries is provided, the stack\n is built following the parameters of each dict for building each layer.\n description_implications:\n The more layers that are specified the deeper and\n higher capacity the model will be. This makes it possible to potentially\n achieve better performance when a big anough amount of data is provided,\n but also makes the model more computationally expensive and potentially\n more prone to overfitting.\n example_value:\n - dropout: 0.1\n output_size: 128\n - norm: layer\n output_size: 64\n expected_impact: 1\n related_parameters:\n - output_size\n - use_bias\n - weights_initializer\n - bias_initializer\n - norm\n - norm_params\n - activation\n - dropout\n suggested_values_reasoning:\n It is easier to define a stack of fully connected\n layers by just specifying num_fc_layers, output_size and the other individual\n parameters. It will create a stack of layers with identical properties.\n Use this parameter only if you need a fine grained level of control of\n each individual layer in the stack.\n ui_display_name: Fully Connected Layers\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n norm:\n default_value_reasoning:\n While batch normalization and layer normalization\n usually lead to improvements, it can be useful to start with fewer bells\n and whistles.\n description_implications:\n Normalization helps stabilize the learning process\n and can have a regularizing effect that can help with generalization.\n It's often suggested that with normalization, you can use a higher learning\n rate.\n example_value:\n - batch\n expected_impact: 2\n literature_references:\n - https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/\n related_parameters:\n - norm_params\n suggested_values: '\"batch\" or \"layer\"'\n suggested_values_reasoning:\n Normalization tries to solve \"internal covariate\n shift\" that comes from the changing distributions of the inputs to layers\n deep in the network when weights are updated. For example, batch normalization\n standardizes the inputs to a layer for each mini-batch. Try out different\n normalizations to see if that helps with training stability\n ui_display_name: Normalization Type\n norm_params:\n default_value_reasoning:\n The default parameters that come with Torch's implementation\n of these normalization types are a trusted starting point.\n description_implications:\n There are a variety of ways a certain set of parameters\n specificed could influence performance here. Broadly speaking the different\n values passed in here allow for different levels of smoothness to be observed\n in the learning curves. Since setting this parameters depends on the type\n of `norm` set, see [BatchNorm2d](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html)\n for more information on the parameters to set for batch normalization,\n and see [LayerNorm](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html)\n for more information on the parameters to set for layer normalization.\n example_value:\n - affine: false\n momentum: 0.2\n num_features: 100\n expected_impact: 1\n literature_references:\n - \"For BatchNorm2d: https://arxiv.org/abs/1502.03167\n\n\n For LayerNorm: https://arxiv.org/abs/1607.06450\"\n related_parameters:\n - \"`norm`\"\n suggested_values: Depends on the type of `norm` set.\n suggested_values_reasoning: \"NO\"\n ui_display_name: Normalization Parameters\n num_fc_layers:\n default_value_reasoning:\n The encoder already has learnable parameters.Sometimes\n the default is 1 for modules where the FC stack is used for shape management,\n or the only source of learnable parameters.\n description_implications:\n Increasing num_fc_layers will increase the capacity\n of the model. The model will be slower to train, and there's a higher\n risk of overfitting.\n example_value:\n - 1\n expected_impact: 1\n other_information:\n Not all modules that have fc_layers also have an accompanying\n num_fc_layers parameter. Where both are present, fc_layers takes precedent\n over num_fc_layers. Specifying num_fc_layers alone uses fully connected\n layers that are configured by the defaults in FCStack.\n related_parameters:\n - fc_layers\n suggested_values: 0-1\n suggested_values_reasoning:\n The full model likely contains many learnable\n parameters. Consider starting with very few, or without any additional\n fully connected layers and add them if you observe evidence of limited\n model capacity. Sometimes the default is 1 for modules where the FC stack\n is used for shape management, or the only source of learnable parameters.\n ui_display_name: Number of Fully Connected Layers\n num_layers:\n default_value_reasoning:\n The ideal number of layers depends on the data. For\n many data types, one layer is sufficient.\n description_implications:\n Increasing the number of layers may improve model\n performance for longer sequences or more complex tasks.\n example_value:\n - 1\n expected_impact: 3\n suggested_values: 1-3\n suggested_values_reasoning:\n Increasing the number of layers may improve encoder\n performance. However, more layers will increase training time and may\n cause overfitting. Small numbers of layers usually work best.\n ui_display_name: Number of Recurrent Layers\n output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 10 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\n pretrained_embeddings:\n ui_display_name: null\n expected_impact: 0\n recurrent_activation:\n default_value_reasoning: sigmoid' is commonly used\n expected_impact: 1\n other_information:\n I don't think that this parameter is used anywhere in the\n code base. It's being passed down but not used in the actual RNN forwarding\n functions.\n suggested_values: sigmoid, ReLu, tanh\n ui_display_name: null\n recurrent_dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 2\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - \"dropout,\n\n recurrent_dropout,\n\n fc_dropout\"\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Recurrent Dropout\n recurrent_initializer:\n ui_display_name: null\n expected_impact: 1\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n representation:\n ui_display_name: null\n expected_impact: 1\n should_embed:\n internal_only: true\n ui_display_name: Not displayed\n state_size:\n ui_display_name: null\n expected_impact: 3\n unit_forget_bias:\n ui_display_name: null\n expected_impact: 1\n use_bias:\n default_value_reasoning:\n \"Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to use bias terms.\n\n\n Batch Normalization, however, adds a trainable shift parameter which is\n added to the activation. When Batch Normalization is used in a layer,\n bias terms are redundant and may be removed.\"\n description_implications:\n Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to leave this parameter set to True.\n example_value:\n - true\n expected_impact: 1\n other_information:\n If fc_layers is not specified, or use_bias is not specified\n for individual layers, the value of use_bias will be used as the default\n for all layers.\n related_parameters:\n - bias_initializer, fc_layers\n suggested_values:\n - true\n ui_display_name: Use Bias\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\n\n\n Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\nStackedTransformer:\n type:\n short_description: Stack of transformer blocks with optional stack of fully connected layers.\n long_description:\n The transformer encoder implements a stack of transformer blocks, replicating the architecture\n introduced in the Attention is all you need paper, and adds am optional stack of fully connected\n layers at the end.\n literature_references:\n - https://arxiv.org/abs/1706.03762\n compute_tier: 2\n bias_initializer:\n default_value_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights.\n description_implications:\n It's rare to see any performance gains from choosing\n a different bias initialization. Some practitioners like to use a small\n constant value such as 0.01 for all biases to ensure that all ReLU units\n are activated in the beginning and have some effect on the gradient. However,\n it's still an open question as to whether this provides consistent improvement.\n expected_impact: 1\n literature_references:\n - https://cs231n.github.io/neural-networks-2/\n related_parameters:\n - weights_initializer\n suggested_values: zeros\n suggested_values_reasoning:\n It is possible and common to initialize the biases\n to be zero, since the asymmetry breaking is provided by the small random\n numbers in the weights. For ReLU non-linearities, some people like to\n use small constant value such as 0.01 for all biases because this ensures\n that all ReLU units fire in the beginning and therefore obtain and propagate\n some gradient. However, it is not clear if this provides a consistent\n improvement (in fact some results seem to indicate that this performs\n worse) and it is more common to simply use 0 bias initialization.\n ui_display_name: Bias Initializer\n dropout:\n default_value_reasoning: Taken from published literature (https://arxiv.org/abs/1908.07442).\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - fc_dropout\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Dropout\n embedding_size:\n default_value_reasoning: Not too big, not too small.\n description_implications:\n 'An embedding is a relatively low-dimensional space\n that is used to translate high-dimensional vectors like words, which can\n have a large vocbulary size. Ideally, after an embedding is trained, it\n captures some of the semantics of the input by placing semantically similar\n inputs close together in the embedding space.\n\n\n In most cases, the embedding size is chosen empirically, by trial and\n error. From https://www.amazon.com/dp/1098115783, \"one rule of thumb is\n to use the fourth root of the total number of unique categorical elements\n while another is that the embedding dimension should be approximately\n 1.6 times the square root of the number of unique elements in the category,\n and no less than 600.\"\n\n\n Increasing the embedding size may cause the model to train more slowly,\n but the higher dimensionality can also improve overall quality.'\n expected_impact: 3\n literature_references:\n - https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture\n suggested_values: 1.6 * sqrt(vocab_size)\n suggested_values_reasoning:\n Rule of thumb suggested by a deep learning textbook.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Embedding Size\n embeddings_on_cpu:\n default_value_reasoning:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access.\n description_implications:\n By default embeddings matrices are stored on GPU\n memory if a GPU is used, as it allows for faster access. However, in some\n cases when the vocabulary size is very large, the full embedding matrix\n may be really big and unwieldy to have in GPU memory. This parameter forces\n the placement of the embedding matrix in regular memory and the CPU is\n used to access them. This may slow down training due to additional data\n transfer between CPU and GPU memory, but can lead to healthier GPU memory\n resource usage.\n expected_impact: 1\n suggested_values:\n - false\n suggested_values_reasoning:\n If GPU memory is not a constraint, having embeddings\n stored and accessed within the GPU is faster.\n ui_display_name: Embeddings on CPU\n embeddings_trainable:\n ui_display_name: null\n expected_impact: 1\n fc_activation:\n default_value_reasoning:\n The Rectified Linear Units (ReLU) function is the\n standard activation function used for adding non-linearity. It is simple,\n fast, and empirically works well (https://arxiv.org/abs/1803.08375).\n description_implications:\n Changing the activation functions has an impact\n on the computational load of the model and might require further hypterparameter\n tuning\n example_value:\n - relu\n expected_impact: 1\n literature_references:\n - https://ml-cheatsheet.readthedocs.io/en/latest/activation_functions.html\n related_parameters:\n - activation, activation_function, conv_activation, recurrent_activation\n suggested_values: relu, alternatively leakyRelu or elu\n suggested_values_reasoning:\n The default value will work well in the majority\n of the cases\n ui_display_name: FC Activation\n fc_dropout:\n default_value_reasoning:\n Dropout can cause training to become less stable.\n Consider start with a dropout-free baseline, and add dropout gradually\n in subsequent experiments.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 1\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - dropout\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: FC Dropout\n fc_layers:\n default_value_reasoning:\n By default the stack is built by using num_fc_layers,\n output_size, use_bias, weights_initializer, bias_initializer, norm, norm_params,\n activation, dropout. When a list of dictionaries is provided, the stack\n is built following the parameters of each dict for building each layer.\n description_implications:\n The more layers that are specified the deeper and\n higher capacity the model will be. This makes it possible to potentially\n achieve better performance when a big anough amount of data is provided,\n but also makes the model more computationally expensive and potentially\n more prone to overfitting.\n example_value:\n - dropout: 0.1\n output_size: 128\n - norm: layer\n output_size: 64\n expected_impact: 1\n related_parameters:\n - output_size\n - use_bias\n - weights_initializer\n - bias_initializer\n - norm\n - norm_params\n - activation\n - dropout\n suggested_values_reasoning:\n It is easier to define a stack of fully connected\n layers by just specifying num_fc_layers, output_size and the other individual\n parameters. It will create a stack of layers with identical properties.\n Use this parameter only if you need a fine grained level of control of\n each individual layer in the stack.\n ui_display_name: Fully Connected Layers\n hidden_size:\n default_value_reasoning: Taken from literature (https://arxiv.org/abs/1706.03762)\n description_implications:\n Increasing the hidden size makes the model larger\n and slower to train, increases the model's capacity to capture more complexity.\n It also increases the chance of overfitting.\n expected_impact: 2\n suggested_values: 10 - 2048\n suggested_values_reasoning:\n Increasing the hidden size makes sense if the\n model is underfitting. It's useful to train both smaller and larger models\n to see how model capacity affects performance. This should only be explored\n after the architecture of the model has been settled.\n ui_display_name: Hidden Size\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes and the positional embedding matrix are\n computed accurately.\n internal_only: true\n ui_display_name: null\n norm:\n default_value_reasoning:\n While batch normalization and layer normalization\n usually lead to improvements, it can be useful to start with fewer bells\n and whistles.\n description_implications:\n Normalization helps stabilize the learning process\n and can have a regularizing effect that can help with generalization.\n It's often suggested that with normalization, you can use a higher learning\n rate.\n example_value:\n - batch\n expected_impact: 2\n literature_references:\n - https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/\n related_parameters:\n - norm_params\n suggested_values: '\"batch\" or \"layer\"'\n suggested_values_reasoning:\n Normalization tries to solve \"internal covariate\n shift\" that comes from the changing distributions of the inputs to layers\n deep in the network when weights are updated. For example, batch normalization\n standardizes the inputs to a layer for each mini-batch. Try out different\n normalizations to see if that helps with training stability\n ui_display_name: Normalization Type\n norm_params:\n default_value_reasoning:\n The default parameters that come with Torch's implementation\n of these normalization types are a trusted starting point.\n description_implications:\n There are a variety of ways a certain set of parameters\n specificed could influence performance here. Broadly speaking the different\n values passed in here allow for different levels of smoothness to be observed\n in the learning curves. Since setting this parameters depends on the type\n of `norm` set, see [BatchNorm2d](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html)\n for more information on the parameters to set for batch normalization,\n and see [LayerNorm](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html)\n for more information on the parameters to set for layer normalization.\n example_value:\n - affine: false\n momentum: 0.2\n num_features: 100\n expected_impact: 1\n literature_references:\n - \"For BatchNorm2d: https://arxiv.org/abs/1502.03167\n For LayerNorm: https://arxiv.org/abs/1607.06450\"\n related_parameters:\n - \"`norm`\"\n suggested_values: Depends on the type of `norm` set.\n suggested_values_reasoning: \"NO\"\n ui_display_name: Normalization Parameters\n num_fc_layers:\n default_value_reasoning:\n The encoder already has learnable parameters.Sometimes\n the default is 1 for modules where the FC stack is used for shape management,\n or the only source of learnable parameters.\n description_implications:\n Increasing num_fc_layers will increase the capacity\n of the model. The model will be slower to train, and there's a higher\n risk of overfitting.\n example_value:\n - 1\n expected_impact: 1\n other_information:\n Not all modules that have fc_layers also have an accompanying\n num_fc_layers parameter. Where both are present, fc_layers takes precedent\n over num_fc_layers. Specifying num_fc_layers alone uses fully connected\n layers that are configured by the defaults in FCStack.\n related_parameters:\n - fc_layers\n suggested_values: 0-1\n suggested_values_reasoning:\n The full model likely contains many learnable\n parameters. Consider starting with very few, or without any additional\n fully connected layers and add them if you observe evidence of limited\n model capacity. Sometimes the default is 1 for modules where the FC stack\n is used for shape management, or the only source of learnable parameters.\n ui_display_name: Number of Fully Connected Layers\n num_heads:\n ui_display_name: null\n num_layers:\n default_value_reasoning:\n The ideal number of layers depends on the data. For\n many data types, one layer is sufficient.\n description_implications:\n \"The ideal number of transformer layers depends\n on the length and complexity of input sequences, as well as the task.\n\n\n For more complex tasks, and higher number of transformer layers may be\n useful. However, too many layers will increase memory and slow training\n while providing diminishing returns of model performance.\"\n example_value:\n - 1\n expected_impact: 3\n suggested_values: 1 - 12\n suggested_values_reasoning:\n Increasing the number of layers may improve encoder\n performance. However, more layers will increase training time and may\n cause overfitting. Small numbers of layers usually work best.\n ui_display_name: Number of Transformer Layers\n output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 3\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 10 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Output Size\n pretrained_embeddings:\n ui_display_name: null\n expected_impact: 0\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n representation:\n ui_display_name: null\n expected_impact: 1\n should_embed:\n internal_only: true\n ui_display_name: Not displayed\n transformer_output_size:\n default_value_reasoning: A modest value, not too small, not too large.\n description_implications:\n If there are fully connected layers in this module,\n increasing the output size of each fully connected layer will increase\n the capacity of the model. However, the model may be slower to train,\n and there's a higher risk of overfitting. If it seems like the model could\n use even more capacity, consider increasing the number of fully connected\n layers, or explore other architectures.\n expected_impact: 2\n other_information:\n If num_fc_layers=0 and fc_layers=None, and there are no\n fully connected layers defined on the module, then this parameter may\n have no effect on the module's final output shape.\n related_parameters:\n - num_fc_layers, fc_layers\n suggested_values: 10 - 1024\n suggested_values_reasoning:\n Increasing the output size increases the capacity\n of the model. If this seems to have a positive effect, then it could be\n worth increasing the number of layers, or trying a different architecture\n with a larger capacity.\n ui_display_name: Transformer Output Size\n use_bias:\n default_value_reasoning:\n \"Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to use bias terms.\n\n\n Batch Normalization, however, adds a trainable shift parameter which is\n added to the activation. When Batch Normalization is used in a layer,\n bias terms are redundant and may be removed.\"\n description_implications:\n Bias terms may improve model accuracy, and don't\n have much impact in terms of memory or training speed. For most models\n it is reasonable to leave this parameter set to True.\n example_value:\n - true\n expected_impact: 1\n other_information:\n If fc_layers is not specified, or use_bias is not specified\n for individual layers, the value of use_bias will be used as the default\n for all layers.\n related_parameters:\n - bias_initializer, fc_layers\n suggested_values:\n - true\n ui_display_name: Use Bias\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n weights_initializer:\n default_value_reasoning: Taken from [this paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).\n description_implications:\n The method you choose to initialize layer weights\n during training can have a big impact on performance as well as the reproducibility\n of your final model between runs. As an example, if you were to randomly\n initialize weights you would risk non-reproducibility (and possibly general\n training performance), but sticking with constant values for initialization\n might significantly increase the time needed for model convergence. Generally,\n choosing one of the probabilistic approaches strikes a balance between\n the two extremes, and the literature kicked off by the landmark [*Xavier\n et al.* paper](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)\n provides a few good options. See this nice discussion from [Weights and\n Biases](https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.)\n for more information.\n expected_impact: 1\n literature_references:\n - \"Weights and Biases blog post: https://wandb.ai/site/articles/the-effects-of-weight-initialization-on-neural-nets#:~:text=Studies%20have%20shown%20that%20initializing,net%20train%20better%20and%20faster.\n\n\n Xavier et al. paper: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\"\n suggested_values: xavier_uniform\n suggested_values_reasoning:\n Changing the weights initialization scheme is\n something to consider if a model is having trouble with convergence, or\n otherwise it is something to experiment with after other factors are considered.\n The default choice (`xavier_uniform`) is a suitable starting point for\n most tasks.\n ui_display_name: Layer Weights Initializer\nT5:\n type:\n short_description: Text-to-text approach transformer with good transfer performance on multiple tasks.\n long_description:\n The `t5` encoder loads a pretrained [T5](https://arxiv.org/pdf/1910.10683.pdf) (default `t5-small`) model using the\n Hugging Face transformers package. T5 (Text-to-Text Transfer Transformer) is pre-trained on a huge text dataset crawled\n from the web and shows good transfer performance on multiple tasks.\n compute_tier: 2\n d_ff:\n default_value_reasoning: Default value matches the pre-trained encoder.\n description_implications:\n If using a pre-trained encoder, this parameter will\n be automatically derived from the pre-trained model.\n expected_impact: 1\n ui_display_name: Dimensionality of Feed-Forward Layer\n d_kv:\n ui_display_name: null\n d_model:\n ui_display_name: null\n dropout_rate:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 2\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: dropout_rate\n feed_forward_proj:\n ui_display_name: null\n initializer_factor:\n ui_display_name: null\n layer_norm_eps:\n ui_display_name: null\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n num_decoder_layers:\n ui_display_name: null\n num_heads:\n ui_display_name: null\n num_layers:\n default_value_reasoning:\n The default value matches the number of layers in\n the default pretrained encoder.\n description_implications:\n \"The ideal number of transformer layers depends\n on the length and complexity of input sequences, as well as the task.\n\n\n If using a pre-trained model, this parameter will be automatically derived\n from the pre-trained model.\"\n example_value:\n - 6\n expected_impact: 2\n related_parameters:\n - pretrained_model_or_path\n suggested_values: 1 - 12\n suggested_values_reasoning:\n Increasing the number of layers may improve encoder\n performance. However, more layers will increase training time and may\n cause overfitting. Small numbers of layers usually work best.\n ui_display_name: Number of Transformer Layers\n pretrained_kwargs:\n ui_display_name: null\n pretrained_model_name_or_path:\n ui_display_name: null\n expected_impact: 2\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n relative_attention_num_buckets:\n ui_display_name: null\n saved_weights_in_checkpoint:\n default_value_reasoning:\n The weights of the encoder are not necessarily saved\n in the checkpoint. The user has to save them first.\n description_implications:\n The memory footprint for some of these encoders\n can be large.\n internal_only: true\n related_parameters:\n - skip_save_model\n suggested_values:\n - false\n suggested_values_reasoning:\n Some of these encoders are large, so it might\n be better to load them as needed, especially if 1. they're not used frequently\n 2. the user doesn't have a lot of storage.\n ui_display_name: null\n trainable:\n expected_impact: 3\n ui_display_name: null\n use_pretrained:\n default_value_reasoning:\n By default, the model is initialized as a pretrained\n model.\n description_implications:\n Pretrained models have typically already learned\n features that are difficult to learn from scratch. They are particularly\n beneficial when training on small amounts of data.\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/transfer-learning-for-deep-learning/\n related_parameters:\n - trainable, pretrained_model_name, pretrained_model_name_or_path, pretrained_kwargs\n suggested_values:\n - false\n suggested_values_reasoning:\n If you have a large amount of data and/or you\n have data that differs from the typical distribution, then it might be\n worth training the model from scratch.\n ui_display_name: Use Pretrained\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n vocab_size:\n internal_only: true\n ui_display_name: Not displayed\nTransformerXL:\n type:\n short_description: Transformer architecture that introduces the notion of recurrence to the deep self-attention network.\n long_description:\n The `transformer_xl` encoder loads a pretrained [Transformer-XL](https://arxiv.org/abs/1901.02860)\n (default `transfo-xl-wt103`) model using the Hugging Face transformers package. Adds novel positional encoding scheme\n which improves understanding and generation of long-form text up to thousands of tokens. Transformer-XL is a causal (uni-directional)\n transformer with relative positioning (sinusoïdal) embeddings which can reuse previously\n computed hidden-states to attend to longer context (memory). This model also uses adaptive\n softmax inputs and outputs (tied).\n compute_tier: 2\n adaptive:\n default_value_reasoning: Huggingface default.\n description_implications:\n Adaptive softmax is a speedup technique for computing\n probability distributions over words. For text with large vocabulary,\n adaptive softmax improves both training speed.\n expected_impact: 1\n related_parameters:\n - vocab_size\n ui_display_name: Adaptive Softmax\n attn_type:\n ui_display_name: null\n clamp_len:\n ui_display_name: null\n cutoffs:\n ui_display_name: null\n d_embed:\n ui_display_name: null\n d_head:\n ui_display_name: null\n d_inner:\n ui_display_name: null\n d_model:\n ui_display_name: null\n div_val:\n ui_display_name: null\n dropatt:\n ui_display_name: null\n dropout:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 2\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: dropout\n eos_token_id:\n default_value_reasoning: Default value used in pre-trained HF encoder.\n ui_display_name: End-of-Sequence Token Id\n init:\n ui_display_name: null\n init_range:\n ui_display_name: null\n init_std:\n ui_display_name: null\n layer_norm_epsilon:\n ui_display_name: null\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n mem_len:\n ui_display_name: null\n n_head:\n ui_display_name: null\n n_layer:\n ui_display_name: null\n pre_lnorm:\n ui_display_name: null\n pretrained_kwargs:\n ui_display_name: null\n pretrained_model_name_or_path:\n ui_display_name: null\n expected_impact: 2\n proj_init_std:\n ui_display_name: null\n proj_share_all_but_first:\n ui_display_name: null\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n same_length:\n ui_display_name: null\n sample_softmax:\n ui_display_name: null\n saved_weights_in_checkpoint:\n default_value_reasoning:\n The weights of the encoder are not necessarily saved\n in the checkpoint. The user has to save them first.\n description_implications:\n The memory footprint for some of these encoders\n can be large.\n internal_only: true\n related_parameters:\n - skip_save_model\n suggested_values:\n - false\n suggested_values_reasoning:\n Some of these encoders are large, so it might\n be better to load them as needed, especially if 1. they're not used frequently\n 2. the user doesn't have a lot of storage.\n ui_display_name: null\n trainable:\n expected_impact: 3\n ui_display_name: null\n untie_r:\n ui_display_name: null\n use_pretrained:\n default_value_reasoning:\n By default, the model is initialized as a pretrained\n model.\n description_implications:\n Pretrained models have typically already learned\n features that are difficult to learn from scratch. They are particularly\n beneficial when training on small amounts of data.\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/transfer-learning-for-deep-learning/\n related_parameters:\n - trainable, pretrained_model_name, pretrained_model_name_or_path, pretrained_kwargs\n suggested_values:\n - false\n suggested_values_reasoning:\n If you have a large amount of data and/or you\n have data that differs from the typical distribution, then it might be\n worth training the model from scratch.\n ui_display_name: Use Pretrained\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n vocab_size:\n internal_only: true\n ui_display_name: Not displayed\nTVAlexNetEncoder:\n model_variant:\n ui_display_name: Model Variant\n type:\n ui_display_name: Type\nTVBaseEncoder:\n model_cache_dir:\n ui_display_name: Model Cache Directory\n saved_weights_in_checkpoint:\n default_value_reasoning:\n The weights of the encoder are not necessarily saved\n in the checkpoint. The user has to save them first.\n description_implications:\n The memory footprint for some of these encoders\n can be large.\n internal_only: true\n related_parameters:\n - skip_save_model\n suggested_values:\n - false\n suggested_values_reasoning:\n Some of these encoders are large, so it might\n be better to load them as needed, especially if 1. they're not used frequently\n 2. the user doesn't have a lot of storage.\n ui_display_name: Saved Weights in Checkpoint\n trainable:\n default_value_reasoning: By default, model components are trainable.\n description_implications:\n The tradeoff when using `trainable` is between speed\n and flexibility. If False, less weights are subject to change and the\n model will therefore train faster. However, the representations output\n by this component are fixed for each input.\n expected_impact: 3\n literature_references:\n - \"https://www.ibm.com/cloud/learn/overfitting\n\n\n http://d2l.ai/chapter_computer-vision/fine-tuning.html\"\n related_parameters:\n - use_pretrained, pretrained_model, saved_weights_in_checkpoint\n suggested_values:\n - false\n suggested_values_reasoning:\n Freezing the weights (i.e. `trainable = False`)\n is only worth trying if you are loading in pretrained weights. In that\n case, check to see if your model is overfitting. If so, freezing the weights\n (and therefore reducing model complexity) may be beneficial.\n ui_display_name: Trainable\n use_pretrained:\n default_value_reasoning:\n By default, the model is initialized as a pretrained\n model.\n description_implications:\n Pretrained models have typically already learned\n features that are difficult to learn from scratch. They are particularly\n beneficial when training on small amounts of data.\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/transfer-learning-for-deep-learning/\n related_parameters:\n - trainable, pretrained_model_name, pretrained_model_name_or_path, pretrained_kwargs\n suggested_values:\n - false\n suggested_values_reasoning:\n If you have a large amount of data and/or you\n have data that differs from the typical distribution, then it might be\n worth training the model from scratch.\n ui_display_name: Use Pretrained\nTVConvNeXtEncoder:\n model_variant:\n ui_display_name: Model Variant\n type:\n ui_display_name: Type\nTVDenseNetEncoder:\n model_variant:\n ui_display_name: Model Variant\n type:\n ui_display_name: Type\nTVEfficientNetEncoder:\n model_variant:\n ui_display_name: Model Variant\n type:\n ui_display_name: Type\nTVGoogLeNetEncoder:\n model_variant:\n ui_display_name: Model Variant\n type:\n ui_display_name: Type\nTVInceptionV3Encoder:\n model_variant:\n ui_display_name: Model Variant\n type:\n ui_display_name: Type\nTVMaxVitEncoder:\n model_variant:\n ui_display_name: Model Variant\n type:\n ui_display_name: Type\nTVMNASNetEncoder:\n model_variant:\n ui_display_name: Model Variant\n type:\n ui_display_name: Type\nTVMobileNetV2Encoder:\n model_variant:\n ui_display_name: Model Variant\n type:\n ui_display_name: Type\nTVMobileNetV3Encoder:\n model_variant:\n ui_display_name: Model Variant\n type:\n ui_display_name: Type\nTVRegNetEncoder:\n model_variant:\n ui_display_name: Model Variant\n type:\n ui_display_name: Type\nTVResNetEncoder:\n model_variant:\n ui_display_name: Model Variant\n type:\n ui_display_name: Type\nTVResNeXtEncoder:\n model_variant:\n ui_display_name: Model Variant\n type:\n ui_display_name: Type\nTVShuffleNetV2Encoder:\n model_variant:\n ui_display_name: Model Variant\n type:\n ui_display_name: Type\nTVSqueezeNetEncoder:\n model_variant:\n ui_display_name: Model Variant\n type:\n ui_display_name: Type\nTVSwinTransformerEncoder:\n model_variant:\n ui_display_name: Model Variant\n type:\n ui_display_name: Type\nTVViTEncoder:\n model_variant:\n ui_display_name: Model Variant\n type:\n ui_display_name: Type\nTVVGGEncoder:\n model_variant:\n ui_display_name: Model Variant\n type:\n ui_display_name: Type\nTVWideResNetEncoder:\n model_variant:\n ui_display_name: Model Variant\n type:\n ui_display_name: Type\nViT:\n type:\n short_description: ViT encoder divides images into patches, performs a linear transformation, and then applies a transformer.\n long_description:\n ViT, short for Vision Transformer, divides the image into equal-sized patches, uses a linear\n transformation to encode each flattened patch, then applies a deep transformer architecture to\n the sequence of encoded patches.\n compute_tier: 2\n attention_probs_dropout_prob:\n default_value_reasoning: Taken from literature (https://arxiv.org/abs/2010.11929).\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - \"hidden_dropout_prob,\n\n attention_probs_dropout_prob\"\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Attention Dropout\n gradient_checkpointing:\n ui_display_name: null\n height:\n internal_only: true\n ui_display_name: null\n hidden_act:\n default_value_reasoning: Taken from huggingface.\n description_implications:\n Changing this activation function will only affect\n the feed-forward layers of the transformer.\n example_value:\n - relu\n expected_impact: 2\n literature_references:\n - \"[Huggingface docs for ViT config](https://huggingface.co/docs/transformers/model_doc/vit#transformers.ViTConfig.hidden_act)\n\n\n [Relevant StackOverflow discussion](https://ai.stackexchange.com/questions/30341/why-does-a-transformer-not-use-an-activation-function-following-the-multi-head-a)\"\n suggested_values: gelu\n suggested_values_reasoning: Taken from huggingface defaults.\n ui_display_name: Hidden Layer Activation\n hidden_dropout_prob:\n default_value_reasoning: Taken from literature (https://arxiv.org/abs/2010.11929).\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 3\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - \"hidden_dropout_prob,\n\n attention_probs_dropout_prob\"\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: Hidden Dropout\n hidden_size:\n default_value_reasoning: Huggingface default.\n description_implications:\n Increasing the hidden size makes the model larger\n and slower to train, increases the model's capacity to capture more complexity.\n It also increases the chance of overfitting.\n expected_impact: 2\n suggested_values: 10 - 2048\n suggested_values_reasoning:\n Increasing the hidden size makes sense if the\n model is underfitting. It's useful to train both smaller and larger models\n to see how model capacity affects performance. This should only be explored\n after the architecture of the model has been settled.\n ui_display_name: Hidden Size\n initializer_range:\n description_implications:\n There is an ideal value for this variable that doesn't\n lead to the outputs of these matrices to vanish or explode\n example_value:\n - 0.02\n expected_impact: 1\n other_information: Must be greater than 0\n related_parameters:\n - weights_initializer\n suggested_values: 0.01-0.05\n suggested_values_reasoning:\n Large values will likely lead to very large outputs.\n Small values will lead to vanishing outputs.\n ui_display_name: null\n intermediate_size:\n ui_display_name: null\n layer_norm_eps:\n ui_display_name: null\n num_attention_heads:\n ui_display_name: null\n num_channels:\n ui_display_name: null\n num_hidden_layers:\n ui_display_name: null\n patch_size:\n default_value_reasoning: Taken from ViT paper.\n description_implications:\n \"The implications of the image patch size for this\\\n \\ layer depend on other factors, such as the true resolution of the incoming\\\n \\ image dataset. If the patch size is kept consistent but a higher resolution\\\n \\ image is used as input, then the resulting chunked sequence of tokens\\\n \\ will be longer than it would have been if the input resolution was lower.\\\n \\ \\n\\nThe ViT paper notes that decreasing the patch size in this way led\\\n \\ to robust improvements without introducing other parameters.\"\n expected_impact: 2\n literature_references:\n - \"[Huggingface docs](https://huggingface.co/docs/transformers/model_doc/vit)\n\n\n [ViT paper](https://arxiv.org/abs/2010.11929)\"\n suggested_values:\n - 16\n - 32\n suggested_values_reasoning:\n 16 and 32 are the values used in the original\n ViT paper.\n ui_display_name: Patch Size\n pretrained_model:\n default_value_reasoning:\n The default model is the canonical model for this\n model architecture, and is therefore a good starting point for most use\n cases.\n description_implications:\n \"There are two factors to consider when choosing\\\n \\ a pre-trained model: (1) size, and (2) task similarity. \\n\\nThe larger\\\n \\ the model, the more subtle its comprehension of inputs can become. However,\\\n \\ larger models are also more compute and memory-intensive to train.\\n\\\n \\nModels pretrained on highly-related source tasks are more likely to\\\n \\ be successful on the target task. Consider searching the HuggingFace\\\n \\ model repository for models trained on similar tasks.\"\n expected_impact: 3\n literature_references:\n - https://arxiv.org/abs/2010.11929\n related_parameters:\n - use_pretrained, trainable, pretrained_kwargs\n suggested_values: google/vit-large-patch16-224\n suggested_values_reasoning:\n \"If you would like better performance and are\n not compute/memory-constrained, increasing model capacity can potentially\n provide a richer representation than the default. The suggested value\n upsizes the model while maintaining the same model architecture.\n\n\n Model trained on internet-scale datasets typically generalize well. Consider\n deviating from the default only if the images in the dataset originate\n from another domain (e.g. medical images, geospatial data).\"\n ui_display_name: Pretrained model name\n saved_weights_in_checkpoint:\n default_value_reasoning:\n The weights of the encoder are not necessarily saved\n in the checkpoint. The user has to save them first.\n description_implications:\n The memory footprint for some of these encoders\n can be large.\n internal_only: true\n related_parameters:\n - skip_save_model\n suggested_values:\n - false\n suggested_values_reasoning:\n Some of these encoders are large, so it might\n be better to load them as needed, especially if 1. they're not used frequently\n 2. the user doesn't have a lot of storage.\n ui_display_name: null\n trainable:\n default_value_reasoning: By default, model components are trainable.\n description_implications:\n The tradeoff when using `trainable` is between speed\n and flexibility. If False, less weights are subject to change and the\n model will therefore train faster. However, the representations output\n by this component are fixed for each input.\n expected_impact: 3\n literature_references:\n - \"https://www.ibm.com/cloud/learn/overfitting\n\n\n http://d2l.ai/chapter_computer-vision/fine-tuning.html\"\n related_parameters:\n - use_pretrained, pretrained_model, saved_weights_in_checkpoint\n suggested_values:\n - false\n suggested_values_reasoning:\n Freezing the weights (i.e. `trainable = False`)\n is only worth trying if you are loading in pretrained weights. In that\n case, check to see if your model is overfitting. If so, freezing the weights\n (and therefore reducing model complexity) may be beneficial.\n ui_display_name: Trainable\n use_pretrained:\n default_value_reasoning:\n By default, the model is initialized as a pretrained\n model.\n description_implications:\n Pretrained models have typically already learned\n features that are difficult to learn from scratch. They are particularly\n beneficial when training on small amounts of data.\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/transfer-learning-for-deep-learning/\n related_parameters:\n - trainable, pretrained_model_name, pretrained_model_name_or_path, pretrained_kwargs\n suggested_values:\n - false\n suggested_values_reasoning:\n If you have a large amount of data and/or you\n have data that differs from the typical distribution, then it might be\n worth training the model from scratch.\n ui_display_name: Use Pretrained\n width:\n internal_only: true\n ui_display_name: null\nXLM:\n type:\n short_description: XLM is pre-trained by cross-language modeling.\n long_description:\n The `xlm` encoder loads a pretrained [XLM](https://arxiv.org/abs/1901.07291) (default `xlm-mlm-en-2048`) model using the\n Hugging Face transformers package. Pre-trained by cross-language modeling.\n compute_tier: 2\n asm:\n ui_display_name: null\n attention_dropout:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 2\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - dropout\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: attention_dropout\n bos_index:\n ui_display_name: null\n bos_token_id:\n default_value_reasoning: Default value used in pre-trained HF encoder.\n ui_display_name: Beginning-of-Sentence Token Id\n causal:\n ui_display_name: null\n dropout:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 2\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - attention_dropout\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: dropout\n emb_dim:\n ui_display_name: null\n embed_init_std:\n ui_display_name: null\n end_n_top:\n ui_display_name: null\n eos_index:\n ui_display_name: null\n gelu_activation:\n ui_display_name: null\n expected_impact: 1\n init_std:\n ui_display_name: null\n is_encoder:\n ui_display_name: null\n lang_id:\n ui_display_name: null\n layer_norm_eps:\n ui_display_name: null\n mask_index:\n ui_display_name: null\n mask_token_id:\n default_value_reasoning: Default value used in pre-trained HF encoder.\n ui_display_name: Mask Token ID\n max_position_embeddings:\n default_value_reasoning: Taken from huggingface.\n description_implications:\n The size of the position embeddings table. This typically coincides with the\n maximum sequence length this model might ever be used with. Typically set this\n to something large just in case (e.g. 512, 1024, 2048).\n expected_impact: 2\n suggested_values: 512\n suggested_values_reasoning:\n Out of the box value based on published literature.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Max Position Embeddings\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n n_heads:\n ui_display_name: null\n n_langs:\n default_value_reasoning: Default value used in pre-trained HF encoder.\n expected_impact: 1\n ui_display_name: Number of Languages\n n_layers:\n ui_display_name: null\n pad_index:\n ui_display_name: null\n pad_token_id:\n ui_display_name: null\n pretrained_kwargs:\n ui_display_name: null\n pretrained_model_name_or_path:\n ui_display_name: null\n expected_impact: 2\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n saved_weights_in_checkpoint:\n default_value_reasoning:\n The weights of the encoder are not necessarily saved\n in the checkpoint. The user has to save them first.\n description_implications:\n The memory footprint for some of these encoders\n can be large.\n internal_only: true\n related_parameters:\n - skip_save_model\n suggested_values:\n - false\n suggested_values_reasoning:\n Some of these encoders are large, so it might\n be better to load them as needed, especially if 1. they're not used frequently\n 2. the user doesn't have a lot of storage.\n ui_display_name: null\n sinusoidal_embeddings:\n ui_display_name: null\n start_n_top:\n ui_display_name: null\n trainable:\n expected_impact: 3\n ui_display_name: null\n unk_index:\n ui_display_name: null\n use_lang_emb:\n ui_display_name: null\n use_pretrained:\n default_value_reasoning:\n By default, the model is initialized as a pretrained\n model.\n description_implications:\n Pretrained models have typically already learned\n features that are difficult to learn from scratch. They are particularly\n beneficial when training on small amounts of data.\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/transfer-learning-for-deep-learning/\n related_parameters:\n - trainable, pretrained_model_name, pretrained_model_name_or_path, pretrained_kwargs\n suggested_values:\n - false\n suggested_values_reasoning:\n If you have a large amount of data and/or you\n have data that differs from the typical distribution, then it might be\n worth training the model from scratch.\n ui_display_name: Use Pretrained\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n vocab_size:\n internal_only: true\n ui_display_name: Not displayed\nXLMRoBERTa:\n type:\n short_description: XLM-RoBERTa a large multi-lingual language model trained on 2.5TB of filtered CommonCrawl data.\n long_description:\n The `xlmroberta` encoder loads a pretrained [XLM-RoBERTa](https://arxiv.org/abs/1911.02116)\n (default `jplu/tf-xlm-reoberta-base`) model using the Hugging Face transformers package. XLM-RoBERTa is a multi-language\n model similar to BERT, trained on 100 languages. XLM-RoBERTa is based on Facebook’s RoBERTa model\n released in 2019. It is a large multi-lingual language model, trained on 2.5TB of filtered\n CommonCrawl data.\n compute_tier: 2\n add_pooling_layer:\n ui_display_name: null\n bos_token_id:\n default_value_reasoning: Default value used in pre-trained HF encoder.\n ui_display_name: Beginning-of-Sentence Token Id\n eos_token_id:\n default_value_reasoning: Default value used in pre-trained HF encoder.\n ui_display_name: End-of-Sentence Token Id\n max_position_embeddings:\n default_value_reasoning: Taken from huggingface.\n description_implications:\n The size of the position embeddings table. This typically coincides with the\n maximum sequence length this model might ever be used with. Typically set this\n to something large just in case (e.g. 512, 1024, 2048).\n expected_impact: 1\n suggested_values: 512\n suggested_values_reasoning:\n Out of the box value based on published literature.\n Try models with smaller or larger embedding sizes to observe relative\n impact.\n ui_display_name: Max Position Embeddings\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n pad_token_id:\n ui_display_name: null\n pretrained_kwargs:\n ui_display_name: null\n pretrained_model_name_or_path:\n ui_display_name: null\n expected_impact: 2\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n saved_weights_in_checkpoint:\n default_value_reasoning:\n The weights of the encoder are not necessarily saved\n in the checkpoint. The user has to save them first.\n description_implications:\n The memory footprint for some of these encoders\n can be large.\n internal_only: true\n related_parameters:\n - skip_save_model\n suggested_values:\n - false\n suggested_values_reasoning:\n Some of these encoders are large, so it might\n be better to load them as needed, especially if 1. they're not used frequently\n 2. the user doesn't have a lot of storage.\n ui_display_name: null\n trainable:\n expected_impact: 3\n ui_display_name: null\n type_vocab_size:\n ui_display_name: null\n expected_impact: 1\n use_pretrained:\n default_value_reasoning:\n By default, the model is initialized as a pretrained\n model.\n description_implications:\n Pretrained models have typically already learned\n features that are difficult to learn from scratch. They are particularly\n beneficial when training on small amounts of data.\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/transfer-learning-for-deep-learning/\n related_parameters:\n - trainable, pretrained_model_name, pretrained_model_name_or_path, pretrained_kwargs\n suggested_values:\n - false\n suggested_values_reasoning:\n If you have a large amount of data and/or you\n have data that differs from the typical distribution, then it might be\n worth training the model from scratch.\n ui_display_name: Use Pretrained\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n vocab_size:\n internal_only: true\n ui_display_name: Not displayed\nXLNet:\n type:\n short_description: XLNet is a transformer that outperforms BERT on a variety of benchmarks.\n long_description:\n The `xlnet` encoder loads a pretrained [XLNet](https://arxiv.org/abs/1906.08237) (default `xlnet-base-cased`) model\n using the Hugging Face transformers package. XLnet is an extension of the Transformer-XL model pre-trained using\n an autoregressive method to learn bidirectional contexts by maximizing the expected likelihood\n over all permutations of the input sequence factorization order. XLNet outperforms BERT on a\n variety of benchmarks.\n compute_tier: 2\n attn_type:\n ui_display_name: null\n bi_data:\n ui_display_name: null\n bos_token_id:\n default_value_reasoning: Default value used in pre-trained HF encoder.\n ui_display_name: Beginning-of-Sentence Token Id\n clamp_len:\n ui_display_name: null\n d_inner:\n ui_display_name: null\n d_model:\n ui_display_name: null\n dropout:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 2\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - summary_last_dropout\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: dropout\n end_n_top:\n ui_display_name: null\n eos_token_id:\n default_value_reasoning: Default value used in pre-trained HF encoder.\n ui_display_name: End-of-Sequence Token Id\n ff_activation:\n ui_display_name: null\n expected_impact: 1\n initializer_range:\n description_implications:\n There is an ideal value for this variable that doesn't\n lead to the outputs of these matrices to vanish or explode\n example_value:\n - 0.02\n expected_impact: 1\n other_information: Must be greater than 0\n related_parameters:\n - weights_initializer\n suggested_values: 0.01-0.05\n suggested_values_reasoning:\n Large values will likely lead to very large outputs.\n Small values will lead to vanishing outputs.\n ui_display_name: null\n layer_norm_eps:\n ui_display_name: null\n max_sequence_length:\n default_value_reasoning:\n Sets the maximum sequence length of the expected\n inputs, so input/output shapes are computed accurately.\n internal_only: true\n ui_display_name: null\n mem_len:\n ui_display_name: null\n n_head:\n ui_display_name: null\n n_layer:\n ui_display_name: null\n pad_token_id:\n ui_display_name: null\n pretrained_kwargs:\n ui_display_name: null\n pretrained_model_name_or_path:\n ui_display_name: null\n expected_impact: 2\n reduce_output:\n ui_display_name: null\n expected_impact: 1\n reuse_len:\n ui_display_name: null\n same_length:\n ui_display_name: null\n saved_weights_in_checkpoint:\n default_value_reasoning:\n The weights of the encoder are not necessarily saved\n in the checkpoint. The user has to save them first.\n description_implications:\n The memory footprint for some of these encoders\n can be large.\n internal_only: true\n related_parameters:\n - skip_save_model\n suggested_values:\n - false\n suggested_values_reasoning:\n Some of these encoders are large, so it might\n be better to load them as needed, especially if 1. they're not used frequently\n 2. the user doesn't have a lot of storage.\n ui_display_name: null\n start_n_top:\n ui_display_name: null\n summary_activation:\n default_value_reasoning: Default value used in pre-trained HF encoder.\n ui_display_name: Summary Activation Function\n expected_impact: 1\n summary_last_dropout:\n default_value_reasoning: Huggingface default.\n description_implications:\n \"Dropout is a computationally cheap regularization\\\n \\ method where during training, some neurons are randomly ignored or \\u201C\\\n dropped out\\u201D. Increasing dropout has the effect of making the training\\\n \\ process more noisy and lowering overall network capacity, but it can\\\n \\ be an effective regularization method to reduce overfitting and improve\\\n \\ generalization.\"\n example_value:\n - 0.2\n expected_impact: 1\n literature_references:\n - https://pytorch.org/docs/stable/generated/torch.nn.Dropout.html\n related_parameters:\n - dropout\n suggested_values: 0.05 - 0.8\n suggested_values_reasoning:\n Tuning dropout is really something to be done\n when all of the big choices about architecture have been settled. Consider\n starting with 0.5 and adjusting the dropout depending on observed model\n performance.\n ui_display_name: summary_last_dropout\n summary_type:\n ui_display_name: null\n summary_use_proj:\n ui_display_name: null\n trainable:\n expected_impact: 3\n ui_display_name: null\n untie_r:\n ui_display_name: null\n use_mems_eval:\n ui_display_name: null\n use_mems_train:\n ui_display_name: null\n use_pretrained:\n default_value_reasoning:\n By default, the model is initialized as a pretrained\n model.\n description_implications:\n Pretrained models have typically already learned\n features that are difficult to learn from scratch. They are particularly\n beneficial when training on small amounts of data.\n expected_impact: 3\n literature_references:\n - https://machinelearningmastery.com/transfer-learning-for-deep-learning/\n related_parameters:\n - trainable, pretrained_model_name, pretrained_model_name_or_path, pretrained_kwargs\n suggested_values:\n - false\n suggested_values_reasoning:\n If you have a large amount of data and/or you\n have data that differs from the typical distribution, then it might be\n worth training the model from scratch.\n ui_display_name: Use Pretrained\n vocab:\n default_value_reasoning:\n Computed and passed along internally according to\n preprocessing settings.\n example_value:\n - a\n - b\n - c\n internal_only: true\n ui_display_name: Not Displayed\n vocab_size:\n internal_only: true\n ui_display_name: Not displayed\nconv_params:\n num_conv_layers:\n description_implications:\n The more layers that are specified the deeper and\n higher capacity the model will be. This makes it possible to potentially\n achieve better performance when a large amount of data is provided, but\n also makes the model more computationally expensive and potentially more\n prone to overfitting.\n expected_impact: 3\n related_parameters:\n - conv_layers\n ui_display_name: Number of Convolutional Layers\n conv_layers:\n description_implications:\n The more layers that are specified the deeper and\n higher capacity the model will be. This makes it possible to potentially\n achieve better performance when a large amount of data is provided, but\n also makes the model more computationally expensive and potentially more\n prone to overfitting.\n expected_impact: 1\n related_parameters:\n - num_conv_layers\n ui_display_name: Convolutional Layers\n pool_function:\n default_value_reasoning:\n \"Within a given sliding window (e.g. a \\\"patch\\\"\\\n \\ of a 3-channel image), the maximum value for each channel is kept. All\\\n \\ other values in the patch are discarded. Repeat this step for every\\\n \\ patch and you have a more compact representation of the image. \\n\\n\\\n Intuitively, each patch encodes the features from a particular part of\\\n \\ an image, and it is more informative to look at the most prominent features\\\n \\ of an image than the average of all of them.\"\n description_implications:\n Both average and max pooling can achieve strong\n performance.\n expected_impact: 1\n literature_references:\n - \"https://pytorch.org/docs/stable/generated/torch.nn.MaxPool2d.html\n\n\n https://machinelearningmastery.com/pooling-layers-for-convolutional-neural-networks/\"\n suggested_values: Default\n suggested_values_reasoning: \"No\"\n ui_display_name: Pooling function\n pool_size:\n ui_display_name: null\n expected_impact: 1\n num_filters:\n ui_display_name: null\n filter_size:\n ui_display_name: null\n expected_impact: 2\nUNetEncoder:\n type:\n short_description: The UNet encoder convolutional and max pool layers\n long_description:\n Stacks of two 2D convolutional layers with optional normalization\n and relu activation, followed by a max pool layer in all but the\n final level of the encoder.\n compute_tier: 1\n conv_norm:\n expected_impact: 2\n ui_display_name: Convolutional Normalization\n height:\n default_value_reasoning:\n Computed internally, automatically, based on image\n data preprocessing.\n internal_only: true\n ui_display_name: NOT DISPLAYED\n num_channels:\n default_value_reasoning:\n Computed internally, automatically, based on image\n data preprocessing.\n internal_only: true\n ui_display_name: NOT DISPLAYED\n width:\n default_value_reasoning:\n Computed internally, automatically, based on image\n data preprocessing.\n internal_only: true\n ui_display_name: NOT DISPLAYED\nTimmEncoder:\n model_name:\n ui_display_name: Model Name\n use_pretrained:\n ui_display_name: Use Pretrained\n saved_weights_in_checkpoint:\n internal_only: true\n ui_display_name: Saved Weights in Checkpoint\n trainable:\n ui_display_name: Trainable\nTimmCAFormerEncoder:\n model_name:\n ui_display_name: Model Name\nTimmConvFormerEncoder:\n model_name:\n ui_display_name: Model Name\nTimmPoolFormerEncoder:\n model_name:\n ui_display_name: Model Name\n"
},
{
"path": "ludwig/schema/metadata/configs/features.yaml",
"content": "audio:\n preprocessing:\n audio_file_length_limit_in_s:\n ui_display_name: null\n expected_impact: 2\n computed_fill_value:\n internal_only: true\n ui_display_name: null\n fill_value:\n ui_display_name: Fill Value\n expected_impact: 2\n in_memory:\n ui_display_name: null\n expected_impact: 1\n missing_value_strategy:\n default_value_reasoning:\n The default `fill_with_const` replaces missing\n values with the value specified by `fill_value`.\n description_implications:\n Determines how missing values will be handled\n in the dataset. Not all strategies are valid for all datatypes. For\n example, `fill_with_mean` is applicable to continuous numerical data.\n Note that choosing to drop rows with missing values could result in\n losing information, especially if there is a high proportion of missing\n values in the dataset.\n expected_impact: 3\n related_parameters:\n - fill_value\n ui_display_name: Missing Value Strategy\n norm:\n default_value_reasoning:\n While batch normalization and layer normalization\n usually lead to improvements, it can be useful to start with fewer\n bells and whistles.\n description_implications:\n Normalization helps stabilize the learning process\n and can have a regularizing effect that can help with generalization.\n It's often suggested that with normalization, you can use a higher\n learning rate.\n example_value:\n - batch\n expected_impact: 2\n literature_references:\n - https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/\n related_parameters:\n - norm_params\n suggested_values: '\"batch\" or \"layer\"'\n suggested_values_reasoning:\n Normalization tries to solve \"internal covariate\n shift\" that comes from the changing distributions of the inputs to\n layers deep in the network when weights are updated. For example,\n batch normalization standardizes the inputs to a layer for each mini-batch.\n Try out different normalizations to see if that helps with training\n stability\n ui_display_name: Normalization Type\n num_fft_points:\n ui_display_name: null\n expected_impact: 1\n num_filter_bands:\n literature_references:\n - \"https://medium.com/analytics-vidhya/simplifying-audio-data-fft-stft-mfcc-for-machine-learning-and-deep-learning-443a2f962e0e \"\n related_parameters:\n - window_length_in_s\n - type\n - window_shift_in_s\n ui_display_name: Type\n expected_impact: 1\n padding_value:\n ui_display_name: null\n expected_impact: 1\n type:\n default_value_reasoning:\n The default type fbank is set based on values\n that we have tested and determined to be a good starting point for\n audio feature preprocessing. This is not to say that it is the best\n way to process every audio feature, it is just a good starting place\n that performs well in general.\n description_implications:\n The different type of audio you select hear\n will determine how your audio feature is preprocessed and transformed\n into trainable data for the model.\n example_value:\n - stft\n expected_impact: 3\n literature_references:\n - \"https://medium.com/analytics-vidhya/simplifying-audio-data-fft-stft-mfcc-for-machine-learning-and-deep-learning-443a2f962e0e \"\n other_information:\n Audio feature preprocessing depends heavily on the\n type of audio data you are dealing with. The type of audio preprocessing\n you will want to use will be dictated by the audio data you are dealing\n with.\n related_parameters:\n - audio_file_length_limit_in_s\n - norm\n - padding_value\n - in_memory\n ui_display_name: Type\n window_length_in_s:\n literature_references:\n - \"https://medium.com/analytics-vidhya/simplifying-audio-data-fft-stft-mfcc-for-machine-learning-and-deep-learning-443a2f962e0e \"\n related_parameters:\n - window_shift_in_s\n - type\n - num_filter_bands\n ui_display_name: Window Length in Seconds\n expected_impact: 2\n window_shift_in_s:\n literature_references:\n - \"https://medium.com/analytics-vidhya/simplifying-audio-data-fft-stft-mfcc-for-machine-learning-and-deep-learning-443a2f962e0e \"\n related_parameters:\n - window_length_in_s\n - type\n - num_filter_bands\n ui_display_name: Window Shift in Seconds\n expected_impact: 2\n window_type:\n ui_display_name: null\n expected_impact: 2\nbag:\n preprocessing:\n computed_fill_value:\n internal_only: true\n ui_display_name: null\n fill_value:\n ui_display_name: Fill Value\n expected_impact: 2\n lowercase:\n ui_display_name: null\n expected_impact: 2\n missing_value_strategy:\n default_value_reasoning:\n The default `fill_with_const` replaces missing\n values with the value specified by `fill_value`.\n description_implications:\n Determines how missing values will be handled\n in the dataset. Not all strategies are valid for all datatypes. For\n example, `fill_with_mean` is applicable to continuous numerical data.\n Note that choosing to drop rows with missing values could result in\n losing information, especially if there is a high proportion of missing\n values in the dataset.\n expected_impact: 3\n related_parameters:\n - fill_value\n ui_display_name: Missing Value Strategy\n most_common:\n default_value_reasoning:\n If there are more than 10000 unique categories\n in the data, it is likely that they will follow a long-tailed distribution\n and the least common ones may not provide a lot of information\n description_implications:\n A smaller number will reduce the vocabulary,\n making the embedding matrix smaller and reduce the memory footprint,\n but will also collapse more tokens into the rare one, so the model\n may perform worse when rare tokens appear in the data\n example_value:\n - 10000\n expected_impact: 2\n other_information: Specifying a vocab_file overrides this parameter\n related_parameters:\n - vocab_file, pretrained_embeddings\n suggested_values:\n A value that covers at least 95% of the tokens in the\n data\n suggested_values_reasoning:\n Depending on the data distribution and how\n important rare tokens are, 90%, 95% or 99% of the number of tokens\n will leave out only very rare tokens that should not influence performance\n substantially\n ui_display_name: Most common (vocabulary size)\n tokenizer:\n ui_display_name: null\n expected_impact: 3\nbinary:\n preprocessing:\n computed_fill_value:\n internal_only: true\n ui_display_name: null\n fallback_true_label:\n description_implications:\n Modeling performance should not be affected,\n but the semantics of some binary metrics may change like for \"false\n positives\", \"false negatives\", etc. if the true label is pinned to\n the other value.\n expected_impact: 2\n ui_display_name: Fallback True Label\n fill_value:\n expected_impact: 2\n ui_display_name: Fill Value\n missing_value_strategy:\n default_value_reasoning:\n The default `fill_with_const` replaces missing\n values with the value specified by `fill_value`.\n description_implications:\n Determines how missing values will be handled\n in the dataset. Not all strategies are valid for all datatypes. For\n example, `fill_with_mean` is applicable to continuous numerical data.\n Note that choosing to drop rows with missing values could result in\n losing information, especially if there is a high proportion of missing\n values in the dataset.\n related_parameters:\n - fill_value\n ui_display_name: Missing Value Strategy\n expected_impact: 3\n calibration:\n expected_impact: 3\n dependencies:\n expected_impact: 1\n reduce_dependencies:\n expected_impact: 1\n reduce_input:\n expected_impact: 1\n threshold:\n expected_impact: 3\ncategory:\n preprocessing:\n computed_fill_value:\n internal_only: true\n ui_display_name: null\n fill_value:\n expected_impact: 2\n ui_display_name: Fill Value\n lowercase:\n ui_display_name: null\n expected_impact: 2\n missing_value_strategy:\n default_value_reasoning:\n The default `fill_with_const` replaces missing\n values with the value specified by `fill_value`.\n description_implications:\n Determines how missing values will be handled\n in the dataset. Not all strategies are valid for all datatypes. For\n example, `fill_with_mean` is applicable to continuous numerical data.\n Note that choosing to drop rows with missing values could result in\n losing information, especially if there is a high proportion of missing\n values in the dataset.\n related_parameters:\n - fill_value\n ui_display_name: Missing Value Strategy\n expected_impact: 3\n most_common:\n default_value_reasoning:\n If there are more than 10000 unique categories\n in the data, it is likely that they will follow a long-tailed distribution\n and the least common ones may not provide a lot of information\n description_implications:\n A smaller number will reduce the vocabulary,\n making the embedding matrix smaller and reduce the memory footprint,\n but will also collapse more tokens into the rare one, so the model\n may perform worse when rare tokens appear in the data\n example_value:\n - 10000\n expected_impact: 2\n other_information: Specifying a vocab_file overrides this parameter\n related_parameters:\n - vocab_file, pretrained_embeddings\n suggested_values:\n A value that covers at least 95% of the tokens in the\n data\n suggested_values_reasoning:\n Depending on the data distribution and how\n important rare tokens are, 90%, 95% or 99% of the number of tokens\n will leave out only very rare tokens that should not influence performance\n substantially\n ui_display_name: Most common (vocabulary size)\n calibration:\n expected_impact: 3\n dependencies:\n expected_impact: 1\n reduce_dependencies:\n expected_impact: 1\n reduce_input:\n expected_impact: 1\n top_k:\n expected_impact: 3\ndate:\n preprocessing:\n computed_fill_value:\n internal_only: true\n ui_display_name: null\n datetime_format:\n default_value_reasoning:\n Ludwig will try to infer the date format automatically,\n but a specific format can be provided. The date string spec is the\n same as the one described in python's datetime.\n description_implications:\n If Ludwig has trouble parsing dates, it could\n be useful to specify an explicit format that Ludwig should parse date\n feature values as. This could also serve as a form of normalization,\n for example, if not all datetimes have the same granularity (some\n have days, some have times), then the common format (i.e. %d %m %Y)\n serves as a truncator.\n example_value:\n - \"%d %b %Y\"\n expected_impact: 2\n suggested_values_reasoning: Have Ludwig figure out the date format automatically.\n ui_display_name: Datetime format\n fill_value:\n expected_impact: 2\n ui_display_name: Fill Value\n missing_value_strategy:\n default_value_reasoning:\n The default `fill_with_const` replaces missing\n values with the value specified by `fill_value`.\n description_implications:\n Determines how missing values will be handled\n in the dataset. Not all strategies are valid for all datatypes. For\n example, `fill_with_mean` is applicable to continuous numerical data.\n Note that choosing to drop rows with missing values could result in\n losing information, especially if there is a high proportion of missing\n values in the dataset.\n related_parameters:\n - fill_value\n ui_display_name: Missing Value Strategy\n expected_impact: 3\nh3:\n preprocessing:\n computed_fill_value:\n internal_only: true\n ui_display_name: null\n fill_value:\n expected_impact: 2\n ui_display_name: Fill Value\n missing_value_strategy:\n default_value_reasoning:\n The default `fill_with_const` replaces missing\n values with the value specified by `fill_value`.\n description_implications:\n Determines how missing values will be handled\n in the dataset. Not all strategies are valid for all datatypes. For\n example, `fill_with_mean` is applicable to continuous numerical data.\n Note that choosing to drop rows with missing values could result in\n losing information, especially if there is a high proportion of missing\n values in the dataset.\n related_parameters:\n - fill_value\n ui_display_name: Missing Value Strategy\n expected_impact: 3\nimage:\n # TODO: review metadata generated by Copilot\n augmentation:\n auto_augmentation_method:\n default_value_reasoning: Trivial augment is computationally more efficient than the other methods.\n description_implications:\n Type of auto-augmentation method to apply to batch of images to improve model generalization\n example_value:\n \"trivial_augment\"\n expected_impact: 1\n ui_display_name: Auto Augmentation Method\n max_brightness:\n default_value_reasoning: The default value of 3.0.\n description_implications:\n The maximum factor by which the brightness of\n the image will be randomly changed.\n example_value:\n - 3.9\n expected_impact: 1\n ui_display_name: Maximum Brightness\n min_brightness:\n default_value_reasoning: The default value of 0.1.\n description_implications:\n The minimum brightness factor to apply to the\n image.\n example_value:\n - 0.5\n expected_impact: 1\n ui_display_name: Minimum Brightness\n max_contrast:\n default_value_reasoning: The default value of 3.0\n description_implications:\n The maximum factor by which the contrast of\n the image will be randomly changed.\n example_value:\n - 3.0\n expected_impact: 1\n ui_display_name: Maximum Contrast\n min_contrast:\n default_value_reasoning: The default value of 0.1.\n description_implications:\n The minimum contrast factor to apply to the\n image.\n example_value:\n - 0.1\n expected_impact: 1\n ui_display_name: Minimum contrast\n kernel_size:\n default_value_reasoning: The default value is 3.\n description_implications: The kernel size is the size of the filter\n matrix. A larger kernel size will result in a blurrier image, while a\n smaller kernel size will result in less blurring.\n example_value:\n - 3\n expected_impact: 2\n suggested_values:\n - 3\n - 5\n - 7\n suggested_values_reasoning:\n The default value is 3, which is a common\n value for image processing\n ui_display_name: Kernel Size\n rotation_degree:\n default_value_reasoning: The default value of 15 means that the\n image will be randomly rotated between -15 to +15 degrees.\n description_implications: The degree of rotation to apply to the image.\n expected_impact: 1\n ui_display_name: Rotation Degree\n type:\n description_implications: The type of augmentation to perform on the\n image.\n expected_impact: 1\n ui_display_name: Type\n preprocessing:\n computed_fill_value:\n internal_only: true\n ui_display_name: null\n fill_value:\n expected_impact: 2\n ui_display_name: Fill Value\n height:\n ui_display_name: null\n expected_impact: 2\n in_memory:\n ui_display_name: null\n expected_impact: 1\n infer_image_dimensions:\n ui_display_name: null\n expected_impact: 1\n infer_image_max_height:\n ui_display_name: null\n expected_impact: 1\n infer_image_max_width:\n ui_display_name: null\n expected_impact: 1\n infer_image_num_channels:\n ui_display_name: null\n expected_impact: 1\n infer_image_sample_size:\n ui_display_name: null\n expected_impact: 1\n infer_image_num_classes:\n ui_display_name: null\n expected_impact: 1\n missing_value_strategy:\n default_value_reasoning:\n The default `fill_with_const` replaces missing\n values with the value specified by `fill_value`.\n description_implications:\n Determines how missing values will be handled\n in the dataset. Not all strategies are valid for all datatypes. For\n example, `fill_with_mean` is applicable to continuous numerical data.\n Note that choosing to drop rows with missing values could result in\n losing information, especially if there is a high proportion of missing\n values in the dataset.\n related_parameters:\n - fill_value\n ui_display_name: Missing Value Strategy\n expected_impact: 3\n num_channels:\n ui_display_name: null\n expected_impact: 2\n num_classes:\n ui_display_name: null\n expected_impact: 2\n num_processes:\n ui_display_name: null\n expected_impact: 2\n resize_method:\n default_value_reasoning:\n Interpolation may stretch or squish the image,\n but it does not remove content or change the statistical distribution\n of image values so it is more appropriate for most tasks.\n description_implications:\n \"interpolation will not change the content of\n the image, but it will change the aspect ratio.\n\n\n crop_or_pad will preserve the aspect ratio of the image, but may remove\n some content (in the case of cropping).\"\n expected_impact: 1\n related_parameters:\n - height, width\n ui_display_name: Resize Method\n standardize_image:\n ui_display_name: null\n expected_impact: 1\n width:\n ui_display_name: null\n expected_impact: 2\n requires_equal_dimensions:\n ui_display_name: null\n expected_impact: 1\n dependencies:\n expected_impact: 1\n reduce_dependencies:\n expected_impact: 1\n reduce_input:\n expected_impact: 1\nnumber:\n preprocessing:\n computed_fill_value:\n internal_only: true\n ui_display_name: null\n computed_outlier_fill_value:\n internal_only: true\n ui_display_name: null\n fill_value:\n expected_impact: 2\n ui_display_name: Fill Value\n missing_value_strategy:\n default_value_reasoning:\n The default `fill_with_const` replaces missing\n values with the value specified by `fill_value`.\n description_implications:\n Determines how missing values will be handled\n in the dataset. Not all strategies are valid for all datatypes. For\n example, `fill_with_mean` is applicable to continuous numerical data.\n Note that choosing to drop rows with missing values could result in\n losing information, especially if there is a high proportion of missing\n values in the dataset.\n related_parameters:\n - fill_value\n ui_display_name: Missing Value Strategy\n expected_impact: 3\n outlier_strategy:\n default_value_reasoning:\n Outlier definitions and how to handle them are very task-specific, so we leave\n this feature disabled by default and ask the user to choose the strategy that works best for them.\n description_implications:\n Determines how outliers will be handled in the dataset. In most cases replacing outliers with the\n column mean (`fill_with_mean`) will be sufficient, but in others the outliers may be damaging enough\n to merit dropping the entire row of data (`drop_row`). In some cases, the best way to handle outliers\n is to leave them in the data, which is the behavior when this parameter is left as `null`.\n related_parameters:\n - outlier_threshold\n suggested_values: fill_with_mean\n ui_display_name: Outlier Strategy\n expected_impact: 3\n outlier_threshold:\n default_value_reasoning:\n The definition of an outlier is often dataset and task dependent, but 2 or 3 standard deviations from\n the mean is a common heuristic.\n description_implications:\n \"Determines the threshold past which a number will be considered an outlier in the dataset. The 3-sigma\n rule in statistics tells us that when data is normally distributed, 95% of the data will lie within 2\n standard deviations of the mean, and greater than 99% of the data will lie within 3 standard deviations\n of the mean (see: 68–95–99.7 rule). As such anything farther away than that is highly likely to be an\n outlier, and may distort the learning process by disproportionately affecting the model.\"\n related_parameters:\n - outlier_strategy\n literature_references:\n - https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule\n suggested_values: 2 - 3\n ui_display_name: Outlier Threshold\n expected_impact: 2\n normalization:\n default_value_reasoning:\n Z-score normalization helps improve the training stability and convergence\n of neural networks by rescaling the numeric input features to have a mean\n of 0 and a standard deviation of 1, reducing the variability and distribution\n of the data. This improves neural network training.\n description_implications:\n The goal of normalization is to transform features\n to be on a similar scale. Normalization can be a form of feature smoothing\n that improves the performance and training stability of the model.\n Normalizations may result in different effects on the semantics of\n your number features. The best normalization technique is one that\n empirically works well, so try new ideas if you think they'll work\n well on your feature distribution.\n expected_impact: 3\n literature_references:\n - https://developers.google.com/machine-learning/data-prep/transform/normalization\n suggested_values: zscore\n suggested_values_reasoning:\n \"Z-score is a variation of scaling that represents\\\n \\ the number of standard deviations away from the mean. You would\\\n \\ use z-score to ensure your feature distributions have mean = 0 and\\\n \\ std = 1. It\\u2019s useful when there are a few outliers, but not\\\n \\ so extreme that you need clipping.\"\n ui_display_name: Normalization\n clip:\n expected_impact: 2\n dependencies:\n expected_impact: 1\n reduce_dependencies:\n expected_impact: 1\n reduce_input:\n expected_impact: 1\nsequence:\n preprocessing:\n computed_fill_value:\n internal_only: true\n ui_display_name: null\n fill_value:\n expected_impact: 2\n ui_display_name: Fill Value\n lowercase:\n ui_display_name: null\n expected_impact: 2\n sequence_length:\n default_value_reasoning:\n The default value is `None`. Which means that the sequence length will be inferred from the dataset,\n which may save you compute resources on datasets with short sequence samples.\n description_implications:\n A larger sequence length keeps more information\n from the data, but also makes it more computationally expensive (more\n memory and longer training time). A smaller sequence length keeps\n less information from the data, but also makes it less computationally\n expensive (less memory and shorter training time).\n expected_impact: 3\n related_parameters:\n - max_sequence_length\n suggested_values:\n If tying the weights of multiple sequence encoders together,\n this parameter may need to be set to ensure that all sequence features have the same sequence length.\n ui_display_name: Sequence Length\n max_sequence_length:\n default_value_reasoning:\n The default value is 256. Every sequence will\n be truncated to this length.\n description_implications:\n A larger sequence length keeps more information\n from the data, but also makes it more computationally expensive (more\n memory and longer training time). A smaller sequence length keeps\n less information from the data, but also makes it less computationally\n expensive (less memory and shorter training time).\n expected_impact: 3\n related_parameters:\n - vocab_size, embedding_size\n suggested_values:\n Use the lowest value that covers most of your input\n data. Only increase the value if crucial parts of the input data are\n truncated.\n ui_display_name: Maximum Sequence Length\n missing_value_strategy:\n default_value_reasoning:\n The default `fill_with_const` replaces missing\n values with the value specified by `fill_value`.\n description_implications:\n Determines how missing values will be handled\n in the dataset. Not all strategies are valid for all datatypes. For\n example, `fill_with_mean` is applicable to continuous numerical data.\n Note that choosing to drop rows with missing values could result in\n losing information, especially if there is a high proportion of missing\n values in the dataset.\n related_parameters:\n - fill_value\n ui_display_name: Missing Value Strategy\n expected_impact: 3\n most_common:\n default_value_reasoning:\n If there are more than 10000 unique categories\n in the data, it is likely that they will follow a long-tailed distribution\n and the least common ones may not provide a lot of information\n description_implications:\n A smaller number will reduce the vocabulary,\n making the embedding matrix smaller and reduce the memory footprint,\n but will also collapse more tokens into the rare one, so the model\n may perform worse when rare tokens appear in the data\n example_value:\n - 10000\n expected_impact: 2\n other_information: Specifying a vocab_file overrides this parameter\n related_parameters:\n - vocab_file, pretrained_embeddings\n suggested_values:\n A value that covers at least 95% of the tokens in the\n data\n suggested_values_reasoning:\n Depending on the data distribution and how\n important rare tokens are, 90%, 95% or 99% of the number of tokens\n will leave out only very rare tokens that should not influence performance\n substantially\n ui_display_name: Most common (vocabulary size)\n ngram_size:\n default_value_reasoning: Size of the n-gram when using the `ngram` tokenizer.\n example_value:\n - 3\n ui_display_name: n-gram size\n expected_impact: 2\n padding:\n ui_display_name: null\n expected_impact: 1\n padding_symbol:\n ui_display_name: null\n expected_impact: 1\n tokenizer:\n ui_display_name: null\n expected_impact: 3\n unknown_symbol:\n ui_display_name: null\n expected_impact: 1\n vocab_file:\n default_value_reasoning:\n The vocabulary can be parsed automatically from\n the incoming input features.\n description_implications:\n It can be useful to specify your own vocabulary\n list if the vocabulary is very large, there's no out of the box tokenizer\n that fits your data, or if there are several uncommon or infrequently\n occurring tokens that we want to guarantee to be a part of the vocabulary,\n rather than treated as an unknown.\n expected_impact: 0\n ui_display_name: Vocab File\n dependencies:\n expected_impact: 1\n reduce_dependencies:\n expected_impact: 1\n reduce_input:\n expected_impact: 1\nset:\n preprocessing:\n computed_fill_value:\n internal_only: true\n ui_display_name: null\n fill_value:\n expected_impact: 2\n ui_display_name: Fill Value\n lowercase:\n ui_display_name: null\n expected_impact: 2\n missing_value_strategy:\n default_value_reasoning:\n The default `fill_with_const` replaces missing\n values with the value specified by `fill_value`.\n description_implications:\n Determines how missing values will be handled\n in the dataset. Not all strategies are valid for all datatypes. For\n example, `fill_with_mean` is applicable to continuous numerical data.\n Note that choosing to drop rows with missing values could result in\n losing information, especially if there is a high proportion of missing\n values in the dataset.\n expected_impact: 3\n related_parameters:\n - fill_value\n ui_display_name: Missing Value Strategy\n most_common:\n default_value_reasoning:\n If there are more than 10000 unique categories\n in the data, it is likely that they will follow a long-tailed distribution\n and the least common ones may not provide a lot of information\n description_implications:\n A smaller number will reduce the vocabulary,\n making the embedding matrix smaller and reduce the memory footprint,\n but will also collapse more tokens into the rare one, so the model\n may perform worse when rare tokens appear in the data\n example_value:\n - 10000\n expected_impact: 2\n other_information: Specifying a vocab_file overrides this parameter\n related_parameters:\n - vocab_file, pretrained_embeddings\n suggested_values:\n A value that covers at least 95% of the tokens in the\n data\n suggested_values_reasoning:\n Depending on the data distribution and how\n important rare tokens are, 90%, 95% or 99% of the number of tokens\n will leave out only very rare tokens that should not influence performance\n substantially\n ui_display_name: Most common (vocabulary size)\n tokenizer:\n ui_display_name: null\n expected_impact: 3\n dependencies:\n expected_impact: 1\n reduce_dependencies:\n expected_impact: 1\n reduce_input:\n expected_impact: 1\n threshold:\n expected_impact: 3\ntext:\n preprocessing:\n computed_fill_value:\n example_value:\n - Depends on dtype\n internal_only: true\n related_parameters:\n - missing_value_strategy, fill_value\n ui_display_name: DOCSTRING ONLY\n fill_value:\n expected_impact: 2\n ui_display_name: Fill Value\n lowercase:\n default_value_reasoning:\n Reading the text in lowercase enables the model\n to treat capitalized and lowercase words as the same, effectively\n increasing the number of data points per word.\n description_implications:\n If you set lowercase to False, then capitalized\n words are seen as completely separate entities than lowercase words.\n example_value:\n - true\n expected_impact: 2\n related_parameters:\n - vocab_size\n suggested_values: \"TRUE\"\n suggested_values_reasoning:\n If there is a strong reason to treat capitalized\n words and lowercased words differently, then set this to False. Otherwise,\n it is preferable to bucket the words and make the model case-insensitive.\n ui_display_name: Convert to lowercase\n sequence_length:\n default_value_reasoning:\n The default value is `None`. Which means that the sequence length will be inferred from the dataset,\n which may save you compute resources on datasets with short text samples.\n description_implications:\n A larger sequence length keeps more information\n from the data, but also makes it more computationally expensive (more\n memory and longer training time). A smaller sequence length keeps\n less information from the data, but also makes it less computationally\n expensive (less memory and shorter training time).\n expected_impact: 3\n related_parameters:\n - max_sequence_length\n suggested_values:\n If tying the weights of multiple text encoders together,\n this parameter may need to be set to ensure that all text features have the same sequence length.\n ui_display_name: Sequence Length\n max_sequence_length:\n default_value_reasoning:\n The default value is 256. Every sequence will\n be truncated to this length.\n description_implications:\n A larger sequence length keeps more information\n from the data, but also makes it more computationally expensive (more\n memory and longer training time). A smaller sequence length keeps\n less information from the data, but also makes it less computationally\n expensive (less memory and shorter training time).\n expected_impact: 3\n related_parameters:\n - vocab_size, embedding_size\n suggested_values:\n Use the lowest value that covers most of your input\n data. Only increase the value if crucial parts of the input data are\n truncated.\n ui_display_name: Maximum Sequence Length\n missing_value_strategy:\n default_value_reasoning:\n The default `fill_with_const` replaces missing\n values with the value specified by `fill_value`.\n description_implications:\n Determines how missing values will be handled\n in the dataset. Not all strategies are valid for all datatypes. For\n example, `fill_with_mean` is applicable to continuous numerical data.\n Note that choosing to drop rows with missing values could result in\n losing information, especially if there is a high proportion of missing\n values in the dataset.\n related_parameters:\n - fill_value\n ui_display_name: Missing Value Strategy\n expected_impact: 3\n most_common:\n default_value_reasoning:\n If there are more than 10000 unique categories\n in the data, it is likely that they will follow a long-tailed distribution\n and the least common ones may not provide a lot of information\n description_implications:\n A smaller number will reduce the vocabulary,\n making the embedding matrix smaller and reduce the memory footprint,\n but will also collapse more tokens into the rare one, so the model\n may perform worse when rare tokens appear in the data\n example_value:\n - 10000\n expected_impact: 2\n other_information: Specifying a vocab_file overrides this parameter\n related_parameters:\n - vocab_file, pretrained_embeddings\n suggested_values:\n A value that covers at least 95% of the tokens in the\n data\n suggested_values_reasoning:\n Depending on the data distribution and how\n important rare tokens are, 90%, 95% or 99% of the number of tokens\n will leave out only very rare tokens that should not influence performance\n substantially\n ui_display_name: Most common (vocabulary size)\n ngram_size:\n default_value_reasoning: Size of the n-gram when using the `ngram` tokenizer.\n example_value:\n - 3\n ui_display_name: n-gram size\n expected_impact: 2\n padding:\n default_value_reasoning:\n We usually want to add padding to the end of\n a text sequence to fill in any remaining space as opposed to the beggining\n so we set the default to right.\n description_implications:\n If you pad to the left, the encoded vector will\n have leading padding tokens as opposed to trailing padding tokens.\n This could matter based on the type of text input you are expecting.\n expected_impact: 1\n related_parameters:\n - \"padding_symbol,\n\n max_sequence_length\"\n suggested_values: \"'right'\"\n suggested_values_reasoning:\n right padding is the usual way to add padding\n to a text sequence\n ui_display_name: Padding\n padding_symbol:\n ui_display_name: null\n expected_impact: 1\n pretrained_model_name_or_path:\n internal_only: true\n ui_display_name: null\n expected_impact: 0\n tokenizer:\n default_value_reasoning:\n 'The default tokenizer is `space_punct`, an abbreviation\n of \"Space punctuation\". This tokenizer creates sub-words by dividing\n the text on whitespace and punctuation characters. For example: The\n text `''hello world!isn''t this great?''` would be transformed to\n `[''hello'', ''world'', ''!'', ''isn'', \"''\", ''t'', ''this'', ''great'',\n ''?'']`. This is the default value because it is a fast tokenizer\n that works reasonably well.'\n description_implications:\n Choosing a tokenizer can be difficult. The primary\n thing to check is that the tokenizer you have selected is compatible\n with the language(s) in your text data. This means either selecting\n a tokenizer that is language-specific (i.e. `french_tokenize` if working\n with French text) or general enough that its tokenizations are language-agnostic\n (i.e. `space_punct`).\n example_value:\n - space_punct\n expected_impact: 3\n literature_references:\n - https://huggingface.co/course/chapter2/4?fw=pt\n related_parameters:\n - vocab_file, pretrained_model_name_or_path\n suggested_values: sentencepiece\n suggested_values_reasoning:\n \"SentencePiece is a tokenizer developed by\n Google which utilizes Byte-Pair Encoding (BPE), which strikes a good\n balance between character-level and word-level tokenization (more\n info on BPE here: https://towardsdatascience.com/byte-pair-encoding-the-dark-horse-of-modern-nlp-eb36c7df4f10\n ). This tokenizer is language-agnostic and more sophisticated than\n the default.\"\n ui_display_name: Tokenizer\n unknown_symbol:\n ui_display_name: null\n expected_impact: 1\n vocab_file:\n default_value_reasoning:\n The vocabulary can be parsed automatically from\n the incoming input features.\n description_implications:\n It can be useful to specify your own vocabulary\n list if the vocabulary is very large, there's no out of the box tokenizer\n that fits your data, or if there are several uncommon or infrequently\n occurring tokens that we want to guarantee to be a part of the vocabulary,\n rather than treated as an unknown.\n expected_impact: 0\n ui_display_name: Vocab File\n class_similarities:\n expected_impact: 1\n dependencies:\n expected_impact: 1\n reduce_dependencies:\n expected_impact: 1\n reduce_input:\n expected_impact: 1\ntimeseries:\n preprocessing:\n computed_fill_value:\n internal_only: true\n ui_display_name: null\n fill_value:\n expected_impact: 2\n ui_display_name: Fill Value\n missing_value_strategy:\n default_value_reasoning:\n The default `fill_with_const` replaces missing\n values with the value specified by `fill_value`.\n description_implications:\n Determines how missing values will be handled\n in the dataset. Not all strategies are valid for all datatypes. For\n example, `fill_with_mean` is applicable to continuous numerical data.\n Note that choosing to drop rows with missing values could result in\n losing information, especially if there is a high proportion of missing\n values in the dataset.\n related_parameters:\n - fill_value\n ui_display_name: Missing Value Strategy\n expected_impact: 3\n padding:\n ui_display_name: null\n expected_impact: 1\n padding_value:\n ui_display_name: null\n expected_impact: 1\n timeseries_length_limit:\n ui_display_name: null\n expected_impact: 2\n tokenizer:\n ui_display_name: null\n expected_impact: 3\nvector:\n preprocessing:\n computed_fill_value:\n internal_only: true\n ui_display_name: null\n fill_value:\n expected_impact: 2\n ui_display_name: Fill Value\n missing_value_strategy:\n default_value_reasoning:\n The default `fill_with_const` replaces missing\n values with the value specified by `fill_value`.\n description_implications:\n Determines how missing values will be handled\n in the dataset. Not all strategies are valid for all datatypes. For\n example, `fill_with_mean` is applicable to continuous numerical data.\n Note that choosing to drop rows with missing values could result in\n losing information, especially if there is a high proportion of missing\n values in the dataset.\n expected_impact: 3\n related_parameters:\n - fill_value\n ui_display_name: Missing Value Strategy\n vector_size:\n ui_display_name: null\n expected_impact: 3\n dependencies:\n expected_impact: 1\n reduce_dependencies:\n expected_impact: 1\n reduce_input:\n expected_impact: 1\n softmax:\n expected_impact: 3\n vector_size:\n expected_impact: 3\n"
},
{
"path": "ludwig/schema/metadata/configs/llm.yaml",
"content": "base_model:\n _anyOf:\n preset:\n ui_display_name: Preset\n expected_impact: 3\n custom:\n ui_display_name: Custom\n expected_impact: 3\n _meta:\n ui_display_name: Model Name\n expected_impact: 3\n ui_component_type: radio_string_combined\n short_description: This can be one of the presets or a fully qualified name of a pretrained model from the HuggingFace Hub\ngeneration:\n temperature:\n ui_display_name: Temperature\n default_value_reasoning:\n Increasing the temperature will allow the model to generate more diverse sequences,\n but will also increase the likelihood of generating nonsense. As such, we recommend setting this value to\n something closer to 0 for classification tasks, and something closer to 1 for text generation tasks where the\n goal is to generate novel text.\n expected_impact: 3\n max_new_tokens:\n ui_display_name: Max New Tokens\n default_value_reasoning:\n Increasing the maximum number of new tokens will allow the model\n to generate longer sequences, but because inference time scales linearly with the sequence length,\n longer sequences will be much slower to generate. For classification tasks, it's generally better to\n use a smaller number of new tokens, while for text generation tasks, it's generally better to use a larger\n number of new tokens.\n expected_impact: 3\n num_beams:\n ui_display_name: Number of Beams\n default_value_reasoning:\n Increasing the number of beams will allow the model to generate more diverse sequences,\n but will also increase inference time. Some backends (like DeepSpeed) also do not support beam search.\n As such, we recommend leaving this as 1 in most cases, unless you're finding the quality of the generated\n sequences to be lacking.\n expected_impact: 2\n top_k:\n ui_display_name: Top K\n expected_impact: 2\n top_p:\n ui_display_name: Top P\n expected_impact: 2\n max_length:\n ui_display_name: Max Length\n expected_impact: 2\n min_length:\n ui_display_name: Min Length\n expected_impact: 2\n min_new_tokens:\n ui_display_name: Min New Tokens\n expected_impact: 2\n do_sample:\n ui_display_name: Do Sample\n expected_impact: 2\n use_cache:\n ui_display_name: Use Cache\n expected_impact: 2\n prompt_lookup_num_tokens:\n ui_display_name: Prompt Lookup Num Tokens\n expected_impact: 2\nprompt:\n retrieval:\n type:\n ui_display_name: Type\n expected_impact: 3\n index_name:\n ui_display_name: Index Name\n expected_impact: 2\n model_name:\n ui_display_name: Model Name\n expected_impact: 2\n k:\n ui_display_name: Top K\n expected_impact: 2\n task:\n ui_display_name: Task\n ui_component_type: textarea\n expected_impact: 3\n template:\n ui_display_name: Template\n ui_component_type: textarea\n expected_impact: 3\nadapter:\n _oneOf:\n allOf:\n ui_display_name: Perform parameter efficient fine-tuning\n expected_impact: 3\n none:\n ui_display_name: Disabled\n expected_impact: 3\n _meta:\n expected_impact: 3\n ui_component_type: radio_string_combined\n lora:\n type:\n long_description: |\n LoRA is a simple, yet effective, method for parameter-efficient fine-tuning of pretrained language models.\n It works by adding a small number of trainable parameters to the model, which are used to adapt the\n pretrained parameters to the downstream task. This allows the model to be fine-tuned with a much smaller\n number of training examples, and can even be used to fine-tune models on tasks that have no training data\n available at all.\n r:\n ui_display_name: R\n expected_impact: 3\n alpha:\n ui_display_name: Alpha\n expected_impact: 1\n dropout:\n ui_display_name: Dropout\n expected_impact: 2\n target_modules:\n ui_display_name: Target Modules\n expected_impact: 2\n use_rslora:\n ui_display_name: Enable RSLora\n expected_impact: 2\n use_dora:\n ui_display_name: Enable DoRa\n expected_impact: 2\n adalora:\n type:\n long_description: |\n AdaLoRA is an extension of LoRA that allows the model to adapt the pretrained parameters to the downstream\n task in a task-specific manner. This is done by adding a small number of trainable parameters to the model,\n which are used to adapt the pretrained parameters to the downstream task. This allows the model to be\n fine-tuned with a much smaller number of training examples, and can even be used to fine-tune models on tasks\n that have no training data available at all.\n prompt_learning:\n num_virtual_tokens:\n ui_display_name: Num Virtual Tokens\n expected_impact: 3\n prompt_tuning:\n prompt_tuning_init:\n ui_display_name: Prompt Tuning Init\n expected_impact: 2\n prompt_tuning_init_text:\n ui_display_name: Prompt Tuning Init Text\n expected_impact: 2\n adaption_prompt:\n type:\n long_description: |\n Adaption Prompt is taken from the paper\n [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://arxiv.org/pdf/2303.16199.pdf).\n It adds a set of learnable adaption prompts and prepends them to the word tokens at higher transformer layers.\n Then, a zero-initialized attention mechanism with zero gating is introduced, which adaptively injects\n new instructional cues into LLaMA, while effectively preserving its pre-trained knowledge. According to\n the paper, LLaMA-Adapter can generate high-quality responses, comparable to Alpaca with fully fine-tuned\n 7B parameters.\n adapter_len:\n ui_display_name: Adapter Length\n expected_impact: 3\n adapter_layers:\n ui_display_name: Adapter Layers\n expected_impact: 3\n ia3:\n type:\n long_description: |\n [Infused Adapter by Inhibiting and Amplifying Inner Activations](https://arxiv.org/pdf/2205.05638.pdf), or IA3,\n is a method that adds three learned vectors `l_k``, `l_v``, and `l_ff`, to rescale the keys and values of the self-attention and encoder-decoder attention layers, and the intermediate activation of the position-wise feed-forward network respectively. These learned vectors are the only trainable parameters during fine-tuning, and thus the original weights remain frozen. Dealing with learned vectors (as opposed to learned low-rank updates to a weight matrix like LoRA) keeps the number of trainable parameters much smaller.\n target_modules:\n ui_display_name: Target Modules\n expected_impact: 3\n feedforward_modules:\n ui_display_name: Feedforward Modules\n expected_impact: 3\n fan_in_fan_out:\n ui_display_name: Fan In Fan Out\n expected_impact: 3\n modules_to_save:\n ui_display_name: Modules to Save\n expected_impact: 3\n init_ia3_weights:\n ui_display_name: Init IA3 Weights\n expected_impact: 3\nquantization:\n _oneOf:\n object:\n ui_display_name: Quantization\n expected_impact: 3\n none:\n ui_display_name: No Quantization\n expected_impact: 3\n _meta:\n expected_impact: 3\n ui_component_type: radio_string_combined\n bits:\n ui_display_name: Bits per parameter\n expected_impact: 3\n"
},
{
"path": "ludwig/schema/metadata/configs/loss.yaml",
"content": "MSELoss:\n weight:\n expected_impact: 2\nMAELoss:\n weight:\n expected_impact: 2\nRMSELoss:\n weight:\n expected_impact: 2\nRMSPELoss:\n weight:\n expected_impact: 2\nBWCEWLoss:\n positive_class_weight:\n expected_impact: 3\n robust_lambda:\n expected_impact: 2\n confidence_penalty:\n expected_impact: 2\n weight:\n expected_impact: 2\nSoftmaxCrossEntropyLoss:\n class_weights:\n expected_impact: 3\n robust_lambda:\n expected_impact: 2\n confidence_penalty:\n expected_impact: 2\n class_similarities:\n expected_impact: 2\n class_similarities_temperature:\n expected_impact: 2\n weight:\n expected_impact: 2\nSequenceSoftmaxCrossEntropyLoss:\n class_weights:\n expected_impact: 3\n robust_lambda:\n expected_impact: 2\n confidence_penalty:\n expected_impact: 2\n class_similarities:\n expected_impact: 2\n class_similarities_temperature:\n expected_impact: 2\n weight:\n expected_impact: 2\n unique:\n expected_impact: 2\nSigmoidCrossEntropyLoss:\n class_weights:\n expected_impact: 3\n weight:\n expected_impact: 2\n"
},
{
"path": "ludwig/schema/metadata/configs/optimizers.yaml",
"content": "gradient_clipping:\n default_value_reasoning:\n A conservative cap on the maximum gradient size to apply\n over a single training step.\n description_implications:\n Gradient clipping is a technique to prevent exploding\n gradients in very deep networks. Increasing gradient clipping can help with\n model training loss curve stability, but it can also make training less efficient\n as weight at each training step is capped.\n expected_impact: 1\n suggested_values_reasoning:\n It's usually sensible to have some conservative notion\n of gradient clipping to make modeling robust to a particularly bad or noisy\n batch of examples.\n ui_display_name: Gradient Clipping\nmomentum:\n expected_impact: 1\nweight_decay:\n expected_impact: 1\ndampening:\n expected_impact: 1\nnesterov:\n expected_impact: 1\nmax_iter:\n expected_impact: 1\nmax_eval:\n expected_impact: 1\ntolerance_grad:\n expected_impact: 1\ntolerance_change:\n expected_impact: 1\nhistory_size:\n expected_impact: 1\nline_search_fn:\n expected_impact: 1\nbetas:\n expected_impact: 1\namsgrad:\n expected_impact: 1\nrho:\n expected_impact: 1\ninitial_accumulator_value:\n expected_impact: 1\nlr_decay:\n expected_impact: 1\nlearning_rate_power:\n expected_impact: 1\nl1_regularization_strength:\n expected_impact: 1\nl2_regularization_strength:\n expected_impact: 1\nmomentum_decay:\n expected_impact: 1\nalpha:\n expected_impact: 1\neps:\n expected_impact: 1\ncentered:\n expected_impact: 1\n"
},
{
"path": "ludwig/schema/metadata/configs/preprocessing.yaml",
"content": "force_split:\n default_value_reasoning:\n We do not expect most datasets to have an explicit \"split\"\n column in the data. Used mostly internally by ludwig datasets.\n expected_impact: 3\n related_parameters:\n - split_probabilities, stratify\n ui_display_name: Force Split\noversample_minority:\n default_value_reasoning:\n We do not want to randomly oversample by default since\n this is a strategy to deal with imbalanced datasets, but can cause issues\n if not implemented correctly.\n description_implications:\n The higher the value you choose gets to 1, the closer\n you will be to having an equal imbalance ratio (i.e. 1:1 positive to negative\n class), however this can lead to problems of overfitting when oversampling\n is used too liberally. As a rule of thumb, starting oversampling with a very\n conservative approach and increasing in small incremements is probably the\n best way to improve your model without experiencing model overfitting.\n example_value:\n - 0.5\n expected_impact: 2\n literature_references:\n - https://machinelearningmastery.com/random-oversampling-and-undersampling-for-imbalanced-classification/\n other_information:\n This parameter is one of many strategies to combat issues with\n class imbalance, though it is not a cure all. Oversampling too much can cause\n overfitting which can adversely affect your model so use with caution.\n suggested_values: Depends on imbalance ratio and dataset size\n ui_display_name: Oversample Minority\nsample_ratio:\n default_value_reasoning:\n The default value is 1.0 because we do not want to shrink\n the dataset by default. In the rare occurences when you do want to downsample\n the entire dataset, this parameter is available, however it is not enabled\n by default, hence a default value of 1.0\n description_implications:\n Decreases the amount of data you are inputting into\n the model. Could be useful if you have more data than you need and you are\n concerned with computational costs.\n example_value:\n - 0.8\n expected_impact: 2\n suggested_values: Depends on data size\n ui_display_name: Sample Ratio\nsample_size:\n default_value_reasoning:\n The default value is None because we do not want to shrink\n the dataset by default, and we do not know the size of an arbitrary dataset.\n By setting the default to None, we fall back on the sample_ratio to determine\n the size of the dataset.\n description_implications:\n Decreases the amount of data you are inputting into\n the model. Could be useful if you have more data than you need and you are\n concerned with computational costs. More useful than sample_ratio if you\n know the exact number of samples you want to train on instead of knowing the proportion.\n example_value:\n - 1000\n expected_impact: 2\n suggested_values: Depends on data size\n ui_display_name: Sample Size\ncolumn:\n expected_impact: 3\n ui_display_name: Split Column\n ui_component_type: column_selector\nsplit_probabilities:\n default_value_reasoning:\n Most of the dataset should be used for training, with\n some portion heldout for validation and testing.\n description_implications:\n \"In machine learning, data splitting is typically done\n to avoid overfitting. That is an instance where a machine learning model fits\n its training data too well and fails to reliably fit additional data.\n\n\n The training set is the portion of data used to train the model. The model\n should observe and learn from the training set, optimizing any of its parameters.\n\n\n The dev set is a data set of examples used to change learning process parameters.\n It is also called the cross-validation or model validation set. This set of\n data has the goal of ranking the model's accuracy and can help with model\n selection.\n\n\n The testing set is the portion of data that is tested in the final model and\n is compared against the previous sets of data. The testing set acts as an\n evaluation of the final mode and algorithm.\"\n expected_impact: 3\n literature_references:\n - \"https://www.techtarget.com/searchenterpriseai/definition/data-splitting#:~:text=Data%20splitting%20is%20when%20data,creating%20models%20based%20on%20data. \"\n other_information: \"Split data into train, validation, and test.\n\n\n By default, Ludwig looks for a column named split (case-sensitive) which is\n expected to consist of 3 possible values that correspond to different datasets:\n\n\n 0: train\n\n 1: validation\n\n 2: test\n\n If the data does not contain the split column, then data is randomly split\n based on splitting percentages, defined by split_probabilities.\n\n\n If force_split is true, the the split column in the dataset is ignored and\n the dataset is randomly split based on splitting percentages, defined by split_probabilities.\"\n related_parameters:\n - force_split, stratify\n suggested_values:\n - 0.8\n - 0.1\n - 0.1\n suggested_values_reasoning:\n For larger datasets, it can be beneficial to use more\n data for training, since the test and validation sets are still plenty big\n for getting a good sense of model generalization.\n ui_display_name: Split Probabilities\nstratify:\n default_value_reasoning:\n The default is set to None since we do not want to stratify\n unless specifically told to do so. There are a variety of reasons for this,\n but one example is that our data set may not even have a categorical feature\n to stratify on.\n description_implications: Depends on dataset\n example_value:\n - Category_Feature_A\n expected_impact: 3\n literature_references:\n - https://medium.com/analytics-vidhya/stratified-sampling-in-machine-learning-f5112b5b9cfe\n related_parameters:\n - force_split, split_probabilities\n suggested_values: Depends on dataset\n ui_display_name: Stratify\nundersample_majority:\n default_value_reasoning:\n We do not want to randomly undersample by default since\n this is a strategy to deal with imbalanced datasets, but can cause issues\n if not implemented correctly.\n description_implications:\n The higher the value you choose gets to 1, the closer\n you will be to having an equal imbalance ratio (i.e. 1:1 positive to negative\n class), however this can lead to problems of data loss when undersampling\n is used too liberally. As a rule of thumb, starting undersampling with a very\n conservative approach and increasing in small incremements is probably the\n best way to improve your model without experiencing catastrophic data loss\n effects.\n example_value:\n - 0.5\n expected_impact: 2\n literature_references:\n - https://machinelearningmastery.com/random-oversampling-and-undersampling-for-imbalanced-classification/\n other_information:\n This parameter is one of many strategies to combat issues with\n class imbalance, though it is not a cure all. Undersampling too much can cause\n loss of data which can adversely affect your model so use with caution.\n suggested_values: Depends on imbalance ratio and dataset size\n ui_display_name: Undersample Majority\ncache_encoder_embeddings:\n default_value_reasoning:\n Caching encoder embeddings means preprocessed data is not reusable across other model architectures, so\n it's not always the case that you would always want to enable it when possible.\n expected_impact: 1\n ui_display_name: Cache Encoder Embeddings\nglobal_max_sequence_length:\n expected_impact: 2\n ui_display_name: Global Max Sequence Length\n description_implications:\n Specifically for LLMs. This is the maximum number of tokens going into the model's forward pass during training. Sequences will be truncated to this length after merging the tokens from the input with tokens from the target. If not set, the total length of the merged input and target token sequences will be used.\n example_value:\n - 512\n"
},
{
"path": "ludwig/schema/metadata/configs/trainer.yaml",
"content": "ecd:\n effective_batch_size:\n commonly_used: true\n expected_impact: 2\n related_parameters:\n - batch_size\n suggested_values: auto\n ui_display_name: Effective Batch Size\n batch_size:\n commonly_used: true\n default_value_reasoning: Not too big, not too small.\n description_implications:\n There's conflicting evidence about what batch size to\n use. Using a higher batch size will achieve the highest throughput and training\n efficiency. However, there's also evidence that depending on other hyperparameters,\n a smaller batch size may produce a higher quality model.\n Batch size and learning rate are strongly intertwined,\n so a commonly adopted strategy to set them is to find a the largest batch size\n that allows the training process not to run out of memory,\n and then find the best learning rate that makes the training converge\n with that batch size.\n expected_impact: 3\n related_parameters:\n - eval_batch_size\n - learning_rate\n suggested_values: auto\n suggested_values_reasoning:\n Auto batch size will determine the largest batch size that allows\n the training process not to run out of memory.\n Alternatively, try at least a few different batch sizes to get a\n sense of whether and how batch size affects model performance.\n ui_display_name: Batch Size\n bucketing_field:\n expected_impact: 1\n other_information:\n When not null, when creating batches, instead of shuffling\n randomly, the length along the last dimension of the matrix of the specified\n input feature (i.e. the length of a sequence or text)\n is used for bucketing examples and then randomly shuffled examples\n from the same bin are sampled. Padding is trimmed to the longest example in\n the batch. The specified feature should be either a sequence or text feature\n and the encoder encoding it has to be rnn. When used, bucketing improves speed\n of rnn encoding up to 1.5x, depending on the length distribution of the inputs.\n ui_display_name: Bucketing Field\n checkpoints_per_epoch:\n default_value_reasoning:\n Per-epoch behavior, which scales according to the dataset\n size.\n description_implications:\n \"Epoch-based evaluation (using the default: 0) is an\n appropriate fit for small datasets that fit in memory and\n train quickly. Commonly available tabular datasets fit in this cateogry.\n However, this is a poor fit for unstructured datasets, which tend to be much\n larger, and train more slowly due to larger models.\n It's important to setup evaluation such that you do not wait several hours\n before getting a single evaluation result. In general, it is not necessary\n for models to train over the entirety of a dataset, nor evaluate over the\n entirety of a test set, to produce useful monitoring metrics and signals to\n indicate model health.\n It is also more engaging and more valuable to ensure a frequent pulse of evaluation\n metrics, even if they are partial.\"\n expected_impact: 2\n related_parameters:\n - train_steps\n - steps_per_checkpoint\n suggested_values: 2 - 10, for larger datasets\n suggested_values_reasoning:\n Running evaluation too frequently can be wasteful\n while running evaluation not frequently enough can be prohibitively uninformative.\n In many large-scale training runs, evaluation is often configured to run on\n a sub-epoch time scale, or every few thousand steps.\n ui_display_name: Checkpoints per epoch\n layers_to_freeze_regex:\n default_value_reasoning:\n By default no layers will be frozen when fine-tuning a pretrained model.\n description_implications:\n Freezing specific layers can improve a pretrained model's performance in a number\n of ways. At a basic level, freezing early layers can prevent overfitting by retaining\n more general features (beneficial for small datasets). Also can reduce computational\n resource use and lower overall training time due to less gradient calculations.\n expected_impact: 1\n early_stop:\n default_value_reasoning:\n Deep learning models are prone to overfitting. It's generally\n a good policy to set up some early stopping criteria as it's not useful to\n have a model train after it's maximized what it can learn. 5 consecutive rounds\n of evaluation where there hasn't been any improvement on the validation set\n (including chance) is a reasonable policy to start with.\n description_implications:\n Decreasing this value is a more aggressive policy. Decreasing\n early stopping makes model training less forgiving, as the model has less\n runway to demonstrate consecutive metric improvements before the training\n run is quit. This can be efficient for pruning bad models earlier, but since\n the training process is inherently non-deterministic and noisy, sometimes\n improvements happen very gradually over a long period of time.\n Extending this value leads to longer training times,\n but potentially also better final performance.\n expected_impact: 3\n related_parameters:\n - epochs\n - train_steps\n suggested_values: 5 - 10\n suggested_values_reasoning:\n There's potentially a lot of randomness in how models\n train, but so many consecutive rounds of no improvement is usually a good\n indicator that the model converged or overfitted.\n ui_display_name: Early Stop\n epochs:\n default_value_reasoning:\n A very high training length ceiling. Models will almost\n always hit early stopping criteria before hitting a 100-epoch ceiling.\n description_implications:\n Decreasing this will shorten the overall runway for\n training the model.\n expected_impact: 3\n related_parameters:\n - train_steps\n suggested_values: 100\n suggested_values_reasoning:\n Usually it's sensible to leave this very high and\n rely on a solid early stopping policy to dictate when the model should stop\n training. Some models and hyperparameter configurations require many epochs\n through the dataset to converge while others converge before a single epoch\n through the data.\n ui_display_name: Epochs\n eval_batch_size:\n default_value_reasoning: Use the same batch size used for training.\n description_implications:\n By increasing the `eval_batch_size` past the `batch_size`\n parameter set value, you allow for more parallelism in the batch evaluation\n step and speed up evaluation. For example, if you have to evaluate the model\n on a test set of size 1000, it is faster to evaluate two times with two batches\n of size 500 as opposed to ten times with ten batches of 100.\n Setting this parameter higher without getting past out memory limits\n will speed up the model training process overall.\n example_value:\n - 512\n expected_impact: 1\n other_information:\n Should only set the eval_batch_size to a level that you can fit\n in memory.\n related_parameters:\n - batch_size\n suggested_values:\n - 256\n - 512\n - 1024\n suggested_values_reasoning:\n By observing memory consumption on training jobs,\n you can get a sense of how much extra memory is available for increasing this\n value. A good rule of thumb can be experimentally doubling the eval batch\n size if you do not have insight into memory usage.\n ui_display_name: Evaluation Batch Size\n evaluate_training_set:\n default_value_reasoning:\n It could be useful to monitor evaluation metrics on the\n training set to understand convergence.\n description_implications:\n Running evaluation on the full training set, when your\n training set is large, can be a huge computational cost. Turning off training\n set evaluation will lead to significant gains in training throughput and efficiency.\n For small datasets that train and evaluate quickly, the choice is trivial.\n expected_impact: 1\n suggested_values: false\n suggested_values_reasoning:\n Running full-scale evaluation on the full training\n set doesn't usually provide any useful information over the validation dataset.\n Even with this set to False, continuous training loss metrics are still computed,\n so it will still be easy to spot signs of overfitting like when the training-validation\n loss curves diverge.\n ui_display_name: Evaluate Training Set\n gradient_clipping:\n default_value_reasoning:\n A conservative cap on the maximum gradient size to apply\n over a single training step.\n description_implications:\n Gradient clipping is a technique to prevent exploding\n gradients in very deep networks. Increasing gradient clipping can help with\n model training loss curve stability, but it can also make training slower\n as weights may not be updated as fast.\n expected_impact: 2\n suggested_values_reasoning:\n It's usually sensible to enable gradient clipping to make modeling robust\n to particularly bad or noisy batches of examples.\n ui_display_name: Gradient Clipping\n increase_batch_size_eval_metric:\n expected_impact: 1\n ui_display_name: \"Batch Size Increase: Evaluation Metric\"\n increase_batch_size_eval_split:\n expected_impact: 1\n ui_display_name: \"Batch Size Increase: Evaluation Split\"\n increase_batch_size_on_plateau:\n expected_impact: 1\n ui_display_name: Batch Size Increase On Plateau\n increase_batch_size_on_plateau_patience:\n expected_impact: 1\n ui_display_name: \"Batch Size Increase On Plateau: Patience\"\n increase_batch_size_on_plateau_rate:\n expected_impact: 1\n ui_display_name: \"Batch Size Increase On Plateau: Rate\"\n learning_rate:\n commonly_used: true\n default_value_reasoning: Middle of the road learning rate to start with.\n description_implications:\n The learning rate is a hyperparameter that controls\n how much to change the model in response to the estimated error each time\n the model weights are updated. Increasing the learning rate may decrease learning\n curve stability but also increase learning speed and efficiency, leading to\n faster model convergence. Decreasing the learning rate can help stabilize\n learning curves at the cost of slower time to convergence.\n expected_impact: 3\n suggested_values: 0.00001 - 0.1 or auto\n related_parameters:\n - decay\n suggested_values_reasoning:\n Tabular models trained from scratch typically use\n learning rates around 1e-3 while learning rates for pre-trained models should\n be much smaller, typically around 1e-5, which is important to mitigate catastrophic\n forgetting. To make the model more robust to any specific choice of learning\n rate, consider turning enabling learning rate decay.\n ui_display_name: Learning Rate\n learning_rate_scaling:\n default_value_reasoning:\n Traditionally the learning rate is scaled linearly with\n the number of workers to reflect the proportion by which the effective batch\n size is increased.\n description_implications:\n Traditionally the learning rate is scaled linearly with\n the number of workers to reflect the proportion by which the effective batch\n size is increased. For very large batch sizes, a softer square-root scale\n can sometimes lead to better model performance. If the learning rate is hand-tuned\n for a given number of workers, setting this value to constant can be used\n to disable scale-up.\n expected_impact: 1\n suggested_values: linear or sqrt\n suggested_values_reasoning:\n Traditionally the learning rate is scaled linearly\n with the number of workers to reflect the proportion by which the effective\n batch size is increased. For very large batch sizes, a softer square-root\n scale can sometimes lead to better model performance. If the learning rate\n is hand-tuned for a given number of workers, setting this value to constant\n can be used to disable scale-up.\n ui_display_name: Learning Rate Scaling\n max_batch_size:\n default_value_reasoning: Not typically required.\n description_implications:\n Value used to manually limit the batch sizes explored\n by auto batch size tuning and batch size increasing on plateau.\n example_value:\n - 1024\n expected_impact: 1\n related_parameters:\n - batch_size\n - increase_batch_size_on_plateau\n ui_display_name: Max Batch Size\n optimizer:\n default_value_reasoning:\n First try Adam because it is shown to return good\n results without an advanced fine tuning.\n description_implications:\n \"Choosing a good optimizer for your machine learning\n project can be overwhelming. Popular deep learning libraries such as PyTorch\n or TensorFLow offer a broad selection of different optimizers, each\n with its own strengths and weaknesses. However, picking the wrong optimizer\n can have a substantial negative impact on the performance of your machine\n learning model [1][2]. This makes optimizers a critical design choice in\n the process of building, testing, and deploying your machine learning model.\"\n expected_impact: 3\n literature_references:\n - https://www.youtube.com/watch?v=mdKjMPmcWjY\n suggested_values: adam, adamw\n suggested_values_reasoning:\n \"As a rule of thumb: If you have the resources to\n find a good learning rate schedule, SGD with momentum is a solid choice. If\n you are in need of quick results without extensive hyperparameter tuning,\n adaptive gradient methods like adam or adamw are good choices.\"\n ui_display_name: Optimizer\n regularization_lambda:\n default_value_reasoning:\n How to tune the overall impact of the regularization\n term by multiplying its value by a scalar known as lambda (also called the\n regularization rate).\n description_implications:\n \"When choosing a lambda value, the goal is to strike\n the right balance between simplicity and training-data fit:\n If your lambda value is too high, your model will be simple, but you run the\n risk of underfitting your data. Your model won't learn enough about the training\n data to make useful predictions.\n If your lambda value is too low, your model will be more complex, and you\n run the risk of overfitting your data. Your model will learn too much about\n the particularities of the training data, and won't be able to generalize\n to new data. The ideal value of lambda produces a model that generalizes well\n to new, previously unseen data. Unfortunately, that ideal value of lambda\n is data-dependent, so you'll need to do some tuning. We recommend trying\n a handful of values (0.001, 0.02, ... 0.4) gradually increasing the value until\n training curves get worse\"\n expected_impact: 2\n literature_references:\n - \"https://developers.google.com/machine-learning/crash-course/regularization-for-simplicity/lambda \"\n related_parameters:\n - regularization_type\n suggested_values: 0.1\n suggested_values_reasoning:\n \"The most common type of regularization is L2, also\n called weight decay, with values often on a logarithmic\n scale between 0 and 0.1, such as 0.1, 0.001, 0.0001, etc.\"\n ui_display_name: Regularization Lambda\n regularization_type:\n default_value_reasoning: L2 is a standard regularization to start with.\n description_implications:\n \"L1 regularization penalizes the sum of absolute values\n of the weights, whereas L2 regularization penalizes the sum of squares of\n the weights.\n The L1 regularization solution is sparse, meaning some weights will be zero,\n others will be large.\n The L2 regularization solution is non-sparse, most weights will be small.\n L2 regularization does not perform feature\n selection, since weights are only reduced to values near 0 instead of 0.\n L1 regularization implicitly performs feature selection. L1 regularization is more\n robust to outliers.\"\n expected_impact: 3\n literature_references:\n - \"https://neptune.ai/blog/fighting-overfitting-with-l1-or-l2-regularization#:~:text=The%20differences%20between%20L1%20and,regularization%20solution%20is%20non%2Dsparse. \"\n related_parameters:\n - regularization_lambda\n suggested_values: L2\n ui_display_name: Regularization Type\n should_shuffle:\n default_value_reasoning:\n In general, it's a good idea to mix up data on each batch\n so that the neural network gets the broadest exposure to the dataset.\n description_implications:\n Turning off mini-batch shuffling can make training faster,\n but it may lead to worse performance overall as shuffling helps mitigate overfitting.\n expected_impact: 1\n literature_references:\n - \"https://stats.stackexchange.com/questions/245502/why-should-we-shuffle-data-while-training-a-neural-network#:~:text=it%20helps%20the%20training%20converge,the%20order%20of%20the%20training \"\n suggested_values: true\n suggested_values_reasoning:\n One of the most powerful things about neural networks\n is that they can be very complex functions, allowing one to learn very complex\n relationships between your input and output data. These relationships can\n include things you would never expect, such as the order in which data is\n fed in per epoch. If the order of data within each epoch is the same, then\n the model may use this as a way of reducing the training error, which is a\n sort of overfitting.\n ui_display_name: Should Shuffle\n steps_per_checkpoint:\n default_value_reasoning:\n By default, we evaluate once per epoch, which scales\n according to the dataset size.\n description_implications:\n \"Epoch-based evaluation (using the default: 0) is an\n appropriate fit for tabular datasets, which are small, fit in memory, and\n train quickly.\n However, this is a poor fit for unstructured datasets, which tend to be much\n larger, and train more slowly due to larger models.\n It's important to setup evaluation such that you do not wait several hours\n before getting a single evaluation result. In general, it is not necessary\n for models to train over the entirety of a dataset, nor evaluate over the\n entirety of a test set, to produce useful monitoring metrics and signals to\n indicate model health.\n It is also more engaging and more valuable to ensure a frequent pulse of evaluation\n metrics, even if they are partial.\"\n expected_impact: 1\n related_parameters:\n - checkpoints_per_epoch\n suggested_values: 1000-10000 for larger datasets\n suggested_values_reasoning:\n Running evaluation too frequently can be wasteful\n while running evaluation not frequently enough can be prohibitively uninformative.\n In many large-scale training runs, evaluation is often configured to run on\n a sub-epoch time scale, or every few thousand steps.\n ui_display_name: Steps Per Checkpoint\n train_steps:\n default_value_reasoning:\n This defaults to `epochs`, which is a very high training\n length ceiling. Models will almost always hit early stopping criteria before\n reaching the absolute end of the training runway.\n description_implications:\n Decreasing this parameter will shorten the overall runway for\n training the model.\n expected_impact: 1\n related_parameters:\n - epochs\n suggested_values: Leave unset, or 1000000, 1 for debugging\n suggested_values_reasoning:\n Usually it's sensible to leave the value of this parameter very high and\n rely on a solid early stopping policy to dictate when the model should stop\n training. Some models and hyperparameter configurations require many epochs\n through the dataset to converge while others converge before a single epoch\n through the data.\n ui_display_name: Train Steps\n eval_steps:\n default_value_reasoning:\n The default value is None because we do not want to lower the number of evaluation steps\n by default, and we do not know the size of an arbitrary dataset.\n By setting the default to None, we simply evaluate on the full evaluation set.\n description_implications:\n The smaller this value of this parameter, the less time evaluation will take.\n expected_impact: 2\n suggested_values: Depends on data size and prioritization of quality vs. speed\n suggested_values_reasoning:\n Normally, evaluation should use the entire evaluation set, and this is\n recommended to achieve the highest quality evaluation. However, using\n the full evaluation set can be slow, so the value of this parameter should\n be set depending on which is more important for the task at hand -- quality\n or speed.\n ui_display_name: Evaluation Steps\n use_mixed_precision:\n default_value_reasoning:\n Speed up training by using float16 parameters where it\n makes sense.\n description_implications:\n Mixed precision training on GPU can dramatically speedup\n training, with some risks to model convergence.\n expected_impact: 3\n literature_references:\n - https://pytorch.org/blog/what-every-user-should-know-about-mixed-precision-training-in-pytorch/\n suggested_values: false\n suggested_values_reasoning:\n Suggested to enable this if training is taking too\n long on GPU.\n ui_display_name: Use Mixed Precision\n compile:\n default_value_reasoning:\n Model compilation has been shown to significantly speedup training by upwards of 20%, but does impose\n some delay to compile the model at the beginning of training. This feature is experimental for now,\n but may become the default in future versions.\n description_implications:\n Model compilation on GPU, when used in conjunction with automatic mixed precision, can speed up training\n by upwards of 20%.\n expected_impact: 3\n suggested_values: false\n suggested_values_reasoning:\n Suggested to enable this if training is taking too\n long on GPU.\n ui_display_name: Compile\n gradient_accumulation_steps:\n default_value_reasoning:\n Gradient accumulation is something that should be enabled only once it has been observed that either GPU\n utilization is low due to low bandwidth between distributed workers, or that there is too much variance\n in the training process due to very low batch sizes.\n description_implications:\n Gradient accumulation is useful to (1) reduce network bandwidth overhead in multi-node distributed training\n scenarios where bandwidth is the bottleneck, and (2) train with larger effective batch sizes when the max\n batch size the GPU can accommodate is very small. The first scenario occurs when the interconnect between\n nodes is slow, so performing gradient synchronization (allreduce) less frequently will speed up training.\n The second scenario occurs in cases where the model being trained is very large (e.g., LLM) so training with\n a larger batch size will help to smooth out the variance from training with a very small batch size.\n expected_impact: 2\n suggested_values: false\n suggested_values_reasoning:\n Suggested to enable this if training is proceeding very slowly in distributed training (and GPU\n utilization is low), or the batch size is very small and the loss curves look very spiky.\n ui_display_name: Gradient Accumulation Steps\n enable_gradient_checkpointing:\n expected_impact: 2\n ui_display_name: Enable Gradient Checkpointing\n default_value_reasoning:\n Gradient checkpointing is a technique to reduce the memory footprint of the model by\n trading compute for memory. This is useful when training very large models that run into out of memory\n errors very quickly during training. It is particularly helpful when doing non-quantization based training\n (adapter based or full fine-tuning). Gradient checkpointing works by recomputing the activations of the\n model during the backward pass, rather than storing them in memory during the forward pass.\n This is a tradeoff between compute and memory, as the activations need to be recomputed during\n the backward pass, but the memory footprint is reduced. This is set to false by default because\n it is not always beneficial to use gradient checkpointing, and it can sometimes slow down training.\n validation_field:\n default_value_reasoning:\n Concrete evaluation metrics are usually better than loss,\n the penalty for a bad prediction, which is only a proxy for prediction correctness.\n description_implications:\n This parameter affects 1) what the early stopping policy\n looks at to determine when to early stop and 2) hyperparameter optimization\n for determining the best trial.\n expected_impact: 1\n related_parameters:\n - validation_metric\n suggested_values: default behavior\n ui_display_name: Validation Field\n validation_metric:\n description_implications:\n This parameter affects 1) what the early stopping policy\n looks at to determine when to early stop and 2) hyperparameter optimization\n for determining the best trial.\n expected_impact: 1\n related_parameters:\n - validation_field\n suggested_values: default behavior\n ui_display_name: Validation Metric\n learning_rate_scheduler:\n warmup_evaluations:\n default_value_reasoning:\n \"Learning rate warmup is most commonly used when training with large batch sizes / distributed\n training to avoid taking overly large steps at the beginning of training that might result in the\n process getting stuck in a local optimum. Conventional wisdom when training with large batch sizes is\n to use a larger learning rate (see: `learning_rate_scaling`) but gradually warm up to the larger learning\n rate over a few epochs of training in the beginning.\n Even when not training with large batch sizes, the randomness of how weights are initialized can result\n in strange, noisy gradient updates during the beginning of your training run. As such, it's generally\n recommended to use a small amount of warmup (e.g., 1 epoch / evaluation) even when the batch size is\n relatively small.\"\n description_implications:\n Learning rate warmup sets a very low learning rate at the beginning of training and gradually\n (linearly) increases to the base learning rate each step (batch) during training.\n After your warmup steps you use your \"regular\" learning rate or learning rate scheduler.\n expected_impact: 2\n related_parameters:\n - warmup_fraction\n - learning_rate_scaling\n literature_references:\n - https://arxiv.org/abs/1711.00489\n - https://datascience.stackexchange.com/questions/55991/in-the-context-of-deep-learning-what-is-training-warmup-steps\n suggested_values: 0 - 5\n suggested_values_reasoning:\n You don't want to warm up for too long, as after the model is starting to hill climb, you want to use the\n full weight of the learning rate to descend into good loss minima.\n\n If you observe your loss curve converging very early into training, within the first few epochs, then\n increasing learning rate warmup may help to mitigate this effect. Pretrained models can benefit from more\n warmup to help offset the effects of catastrophic forgetting due to an overly high learning rate.\n ui_display_name: Warmup Evaluations\n warmup_fraction:\n default_value_reasoning:\n Similar to `warmup_evaluations` but expressed as a fraction of the total number of training steps, rather\n that a certain number of evaluation phases.\n description_implications: See `warmup_evaluations`.\n expected_impact: 2\n related_parameters:\n - warmup_evaluations\n - learning_rate_scaling\n suggested_values: 0.05 - 0.2\n suggested_values_reasoning:\n You don't want to warm up for too long, as after the\n model is starting to hill climb, you want to use the full weight of the learning\n rate to descend into good loss minima.\n ui_display_name: Warmup Fraction\n decay:\n description_implications:\n \"It\\u2019s almost always a good idea to use a schedule.\\\n \\ For most models, try the exponential decay schedule first.\\n\\nThe exponential\\\n \\ schedule divides the learning rate by the same factor (%) every epoch. This\\\n \\ means that the learning rate will decrease rapidly in the first few epochs,\\\n \\ and spend more epochs with a lower value, but never reach exactly zero.\\\n \\ As a rule of thumb, compared to training without a schedule, you can use\\\n \\ a slightly higher maximum learning rate. Since the learning rate changes\\\n \\ over time, the whole training is not so sensitive to the value picked.\"\n expected_impact: 3\n literature_references:\n - \"https://peltarion.com/knowledge-center/documentation/modeling-view/run-a-model/optimization-principles-(in-deep-learning)/learning-rate-schedule \"\n related_parameters:\n - decay_rate\n - decay_steps\n - learning_rate\n suggested_values: exponential\n suggested_values_reasoning:\n Starting with exponential decay is a safe place to start, as it is a \"softer\" decrease in the learning\n rate over time, as compared with linear, which is more steep after the initial drop. Linear decay is\n most useful when the risk of catastrophic forgetting is very high (e.g, for fine-tuning pretrained\n models). Cosine annealing is a type of learning rate schedule that has the effect of starting with a\n large learning rate that is relatively rapidly decreased to a minimum value before being increased\n rapidly again. The resetting of the learning rate acts like a simulated restart of the learning process.\n If you observe your loss curves shooting up (even on the training set) in later epochs, increasing the\n decay rate may help mitigate this effect.\n ui_display_name: Decay\n decay_rate:\n default_value_reasoning:\n 4-5% decay each step is an empirically useful decay\n rate to start with.\n description_implications:\n Increasing the decay rate will lower the learning rate\n faster. This could make the model more robust to a bad (too high) initial\n learning rate, but a decay rate that is too high could prohibit the model\n from learning anything at all.\n expected_impact: 2\n literature_references:\n - \"https://peltarion.com/knowledge-center/documentation/modeling-view/run-a-model/optimization-principles-(in-deep-learning)/learning-rate-schedule \"\n related_parameters:\n - decay_steps\n - learning_rate\n suggested_values: 0.9 - 0.96\n suggested_values_reasoning:\n Since this controls exponential decay, even a small\n decay rate will still be strongly impactful.\n ui_display_name: Decay Rate\n decay_steps:\n default_value_reasoning:\n This default essentially enables the `learning_rate`\n to decay by a factor of the `decay_rate` at 10000 training steps.\n description_implications:\n By increasing the value of decay steps, you are increasing\n the number of training steps it takes to decay the learning rate by a factor\n of `decay_rate`. In other words, the bigger this parameter, the slower the\n learning rate decays.\n example_value:\n - 5000\n expected_impact: 2\n related_parameters:\n - decay_rate\n - learning_rate\n suggested_values: 10000 +/- 500 at a time\n suggested_values_reasoning:\n The decay in the learning rate is calculated as the\n training step divided by the `decay_steps` plus one. Then the `decay_rate`\n is raised to the power of this exponent which is then multiplied to the current\n learning rate. All this to say that the learning rate is only decayed by a\n factor of the set `decay_rate` when the training step reaches the `decay_steps`\n and then subsequently when it reaches any multiple of `decay_steps`. You can\n think of `decay_steps` as a rate of decay for the `decay_rate`.\n ui_display_name: Decay Steps\n staircase:\n default_value_reasoning: Performs learning rate decay in stepwise discrete manner.\n description_implications:\n An excessively aggressive decay results in optimizers\n never reaching the minima, whereas a slow decay leads to chaotic updates without\n significant improvement. Discrete learning rate decay is another parameter to help\n tune a balance.\n expected_impact: 1\n literature_references:\n - https://neptune.ai/blog/how-to-choose-a-learning-rate-scheduler\n suggested_values: false\n suggested_values_reasoning:\n We have not found strong evidence that discretely\n decaying the learning rate is superior to doing so continuously in general,\n but in specific tasks it might have a positive impact.\n ui_display_name: Staircase\n reduce_on_plateau:\n expected_impact: 3\n ui_display_name: Reduce On Plateau\n reduce_on_plateau_patience:\n expected_impact: 2\n ui_display_name: Reduce On Plateau Patience\n reduce_on_plateau_rate:\n expected_impact: 2\n ui_display_name: Reduce On Plateau Rate\n reduce_eval_metric:\n expected_impact: 1\n ui_display_name: Reduce Eval Metric\n reduce_eval_split:\n expected_impact: 1\n ui_display_name: Reduce Eval Split\n t_0:\n expected_impact: 1\n ui_display_name: T_0\n t_mult:\n expected_impact: 1\n ui_display_name: T_mult\n eta_min:\n expected_impact: 1\n ui_display_name: Eta Min\ngbm:\n learning_rate:\n commonly_used: true\n default_value_reasoning: Middle of the road learning rate to start with.\n description_implications:\n The learning rate is a hyperparameter that controls\n how much to change the model in response to the estimated error each time\n the model weights are updated. Increasing the learning rate may decrease learning\n curve stability but also increase learning speed and efficiency, leading to\n faster model convergence. Decreasing the learning rate can help stabilize\n learning curves at the cost of slower time to convergence.\n expected_impact: 3\n suggested_values: 0.00001 - 0.1 or auto\n related_parameters:\n - decay\n suggested_values_reasoning:\n Tabular models trained from scratch typically use\n learning rates around 1e-3 while learning rates for pre-trained models should\n be much smaller, typically around 1e-5, which is important to mitigate catastrophic\n forgetting. To make the model more robust to any specific choice of learning\n rate, consider turning enabling learning rate decay.\n ui_display_name: Learning Rate\n early_stop:\n default_value_reasoning:\n Deep learning models are prone to overfitting. It's generally\n a good policy to set up some early stopping criteria as it's not useful to\n have a model train after it's maximized what it can learn. 5 consecutive rounds\n of evaluation where there hasn't been any improvement on the validation set\n (including chance) is a reasonable policy to start with.\n description_implications:\n Decreasing this value is a more aggressive policy. Decreasing\n early stopping makes model training less forgiving, as the model has less\n runway to demonstrate consecutive metric improvements before the training\n run is quit. This can be efficient for pruning bad models earlier, but since\n the training process is inherently non-deterministic and noisy, sometimes\n improvements happen very gradually over a long period of time.\n Extending this value leads to longer training times,\n but potentially also better final performance.\n expected_impact: 3\n related_parameters:\n - epochs\n - train_steps\n suggested_values: 5 - 10\n suggested_values_reasoning:\n There's potentially a lot of randomness in how models\n train, but so many consecutive rounds of no improvement is usually a good\n indicator that the model converged or overfitted.\n ui_display_name: Early Stop\n eval_batch_size:\n default_value_reasoning: Use the same batch size used for training.\n description_implications:\n By increasing the `eval_batch_size` past the `batch_size`\n parameter set value, you allow for more parallelism in the batch evaluation\n step and speed up evaluation. For example, if you have to evaluate the model\n on a test set of size 1000, it is faster to evaluate two times with two batches\n of size 500 as opposed to ten times with ten batches of 100.\n Setting this parameter higher without getting past out memory limits\n will speed up the model training process overall.\n example_value:\n - 512\n expected_impact: 1\n other_information:\n Should only set the eval_batch_size to a level that you can fit\n in memory.\n related_parameters:\n - batch_size\n suggested_values:\n - 256\n - 512\n - 1024\n suggested_values_reasoning:\n By observing memory consumption on training jobs,\n you can get a sense of how much extra memory is available for increasing this\n value. A good rule of thumb can be experimentally doubling the eval batch\n size if you do not have insight into memory usage.\n ui_display_name: Evaluation Batch Size\n evaluate_training_set:\n default_value_reasoning:\n It could be useful to monitor evaluation metrics on the\n training set to understand convergence.\n description_implications:\n Running evaluation on the full training set, when your\n training set is large, can be a huge computational cost. Turning off training\n set evaluation will lead to significant gains in training throughput and efficiency.\n For small datasets that train and evaluate quickly, the choice is trivial.\n expected_impact: 1\n suggested_values: false\n suggested_values_reasoning:\n Running full-scale evaluation on the full training\n set doesn't usually provide any useful information over the validation dataset.\n Even with this set to False, continuous training loss metrics are still computed,\n so it will still be easy to spot signs of overfitting like when the training-validation\n loss curves diverge.\n ui_display_name: Evaluate Training Set\n validation_field:\n default_value_reasoning:\n Concrete evaluation metrics are usually better than loss,\n the penalty for a bad prediction, which is only a proxy for prediction correctness.\n description_implications:\n This parameter affects 1) what the early stopping policy\n looks at to determine when to early stop and 2) hyperparameter optimization\n for determining the best trial.\n expected_impact: 1\n related_parameters:\n - validation_metric\n suggested_values: default behavior\n ui_display_name: Validation Field\n validation_metric:\n description_implications:\n This parameter affects 1) what the early stopping policy\n looks at to determine when to early stop and 2) hyperparameter optimization\n for determining the best trial.\n expected_impact: 1\n related_parameters:\n - validation_field\n suggested_values: default behavior\n ui_display_name: Validation Metric\n max_depth:\n expected_impact: 3\n drop_rate:\n expected_impact: 2\n tree_learner:\n expected_impact: 2\n boosting_type:\n expected_impact: 3\n boosting_rounds_per_checkpoint:\n expected_impact: 2\n num_boost_round:\n expected_impact: 2\n num_leaves:\n expected_impact: 2\n min_data_in_leaf:\n expected_impact: 2\n min_sum_hessian_in_leaf:\n expected_impact: 1\n bagging_fraction:\n expected_impact: 3\n pos_bagging_fraction:\n expected_impact: 2\n neg_bagging_fraction:\n expected_impact: 2\n bagging_freq:\n expected_impact: 2\n bagging_seed:\n expected_impact: 2\n feature_fraction:\n expected_impact: 3\n feature_fraction_bynode:\n expected_impact: 2\n feature_fraction_seed:\n expected_impact: 2\n extra_trees:\n expected_impact: 3\n extra_seed:\n expected_impact: 2\n max_delta_step:\n expected_impact: 1\n lambda_l1:\n expected_impact: 3\n lambda_l2:\n expected_impact: 3\n linear_lambda:\n expected_impact: 2\n min_gain_to_split:\n expected_impact: 1\n max_drop:\n expected_impact: 2\n skip_drop:\n expected_impact: 2\n xgboost_dart_mode:\n expected_impact: 1\n uniform_drop:\n expected_impact: 2\n drop_seed:\n expected_impact: 2\n top_rate:\n expected_impact: 1\n other_rate:\n expected_impact: 1\n min_data_per_group:\n expected_impact: 1\n max_cat_threshold:\n expected_impact: 1\n cat_l2:\n expected_impact: 1\n cat_smooth:\n expected_impact: 1\n max_cat_to_onehot:\n expected_impact: 1\n cegb_tradeoff:\n expected_impact: 1\n cegb_penalty_split:\n expected_impact: 1\n path_smooth:\n expected_impact: 1\n verbose:\n expected_impact: 1\n max_bin:\n expected_impact: 1\n feature_pre_filter:\n expected_impact: 1\nllm:\n type:\n commonly_used: true\n default_value_reasoning:\n It's useful to start with zero-shot or few-shot learning to see what the model\n can do as a baseline before fine-tuning.\n suggested_values: none or finetune\n suggested_values_reasoning:\n If you want to perform zero shot learning or few shot learning, you should set this to `none`.\n If you want to perform fine-tuning, you should set this to `finetune`.\n ui_display_name: Trainer Type\n expected_impact: 3\n"
},
{
"path": "ludwig/schema/metadata/feature_metadata.py",
"content": ""
},
{
"path": "ludwig/schema/metadata/parameter_metadata.py",
"content": "import json\nfrom dataclasses import dataclass\nfrom enum import Enum\nfrom typing import Any\n\nfrom dataclasses_json import dataclass_json\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.utils.misc_utils import memoized_method\n\n\n@DeveloperAPI\nclass ExpectedImpact(int, Enum):\n \"\"\"The expected impact of determining a \"good\" value for a specific parameter.\n\n - HIGH: this parameter should almost always be included in a hyperopt run and can make or break a good model.\n - MEDIUM: this parameter can sometimes make or break a good model.\n - LOW: this parameter usually does not have a significant impact on model performance.\n \"\"\"\n\n UNKNOWN = 0\n LOW = 1\n MEDIUM = 2\n HIGH = 3\n\n\n@DeveloperAPI\nclass ComputeTier(int, Enum):\n \"\"\"The compute tier defines the type of compute resources that a model typically needs to get good\n throughput.\"\"\"\n\n CPU = 0\n \"\"\"Model can train effectively on CPU hardware.\"\"\"\n\n GPU_LOW = 1\n \"\"\"Model can train effectively on commodity GPU hardware, or inference optimized SKUs like NVIDIA T4.\"\"\"\n\n GPU_MEDIUM = 2\n \"\"\"Model can train effectively on training-optimized GPU hardware like V100, A10G, or A5000.\"\"\"\n\n GPU_HIGH = 3\n \"\"\"Model requires high-end GPUs like A100 or H100 to achieve good throughput.\"\"\"\n\n\n@DeveloperAPI\n@dataclass_json()\n@dataclass\nclass ParameterMetadata:\n \"\"\"Contains descriptive information that pertains to a Ludwig configuration parameter.\"\"\"\n\n short_description: str = \"\"\n \"\"\"Quick description generally for UI display.\"\"\"\n\n long_description: str = \"\"\n \"\"\"In depth description generally for documentation purposes.\"\"\"\n\n ui_display_name: str | None = \"\"\n \"\"\"How this parameter can be displayed in a human-readable form.\"\"\"\n\n default_value_reasoning: str | None = None\n \"\"\"The reasoning behind the default value for this parameter.\"\"\"\n\n example_value: list[Any] | None = None\n \"\"\"Examples of other values that can be used for this parameter.\"\"\"\n\n related_parameters: list[str] | None = None\n \"\"\"List of related parameters that this parameter interacts with or depends on.\"\"\"\n\n other_information: str | None = None\n \"\"\"Other information that is relevant for this parameter.\"\"\"\n\n description_implications: str | None = None\n \"\"\"The intuition for how model performance would change if this parameter is changed.\"\"\"\n\n suggested_values: Any = None\n \"\"\"What values would a machine learning expert suggest users try to help improve their model?\n\n Should cover 95% (2-sigma) worth of use-cases.\n \"\"\"\n\n suggested_values_reasoning: str | None = None\n \"\"\"The reasoning behind the suggested values, as well as model performance indicators or other intuition that\n could help inform a user to make an educated decision about what values to experiment with for this\n parameter.\"\"\"\n\n commonly_used: bool = False\n \"\"\"True if this parameter could be frequently used, would have a high impact, and/or would be interesting for a\n machine learning practitioner.\"\"\"\n\n expected_impact: ExpectedImpact = ExpectedImpact.UNKNOWN\n \"\"\"The expected impact of determining a \"good\" value for this parameter.\"\"\"\n\n literature_references: list[str] | None = None\n \"\"\"List of links, papers, and blog posts to learn more.\"\"\"\n\n internal_only: bool = False\n \"\"\"True if this parameter is used strictly internally and should not be exposed to users.\"\"\"\n\n compute_tier: ComputeTier = ComputeTier.CPU\n \"\"\"The compute tier defines the type of compute resources that a model typically needs to get good\n throughput.\"\"\"\n\n ui_component_type: str | None = None\n \"\"\"Override for HTML component type that should be used to render this field in UIs.\"\"\"\n\n @memoized_method(maxsize=1)\n def to_json_dict(self) -> dict[str, Any]:\n return json.loads(self.to_json())\n\n\n@DeveloperAPI\ndef convert_metadata_to_json(pm: ParameterMetadata) -> dict[str, Any]:\n \"\"\"Converts a ParameterMetadata dict to a normal JSON dict.\n\n NOTE: Without the json.loads call, to_json() returns\n a string repr that is improperly parsed.\n \"\"\"\n if not pm:\n return ParameterMetadata().to_json_dict()\n return pm.to_json_dict()\n\n\n# This is a quick way to flag schema parameters as internal only via the `parameter_metadata` argument\nINTERNAL_ONLY = ParameterMetadata(internal_only=True)\n"
},
{
"path": "ludwig/schema/model_config.py",
"content": "# TODO(travis) consider removing this in the future after deprecation period\nfrom ludwig.schema.model_types.base import ModelConfig # noqa\n"
},
{
"path": "ludwig/schema/model_types/__init__.py",
"content": "import ludwig.schema.model_types.ecd # noqa\nimport ludwig.schema.model_types.llm # noqa\n"
},
{
"path": "ludwig/schema/model_types/base.py",
"content": "import copy\nfrom abc import ABC\nfrom typing import Any\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.config_validation.checks import get_config_check_registry\nfrom ludwig.config_validation.validation import check_schema\nfrom ludwig.constants import (\n BACKEND,\n COLUMN,\n DEPENDENCIES,\n ENCODER,\n INPUT_FEATURES,\n MODEL_ECD,\n NAME,\n OUTPUT_FEATURES,\n TIED,\n)\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.globals import LUDWIG_VERSION\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.defaults.base import BaseDefaultsConfig\nfrom ludwig.schema.features.base import BaseInputFeatureConfig, BaseOutputFeatureConfig, FeatureCollection\nfrom ludwig.schema.hyperopt import HyperoptConfig\nfrom ludwig.schema.model_types.utils import (\n merge_fixed_preprocessing_params,\n merge_with_defaults,\n sanitize_and_filter_combiner_entities_,\n set_derived_feature_columns_,\n set_hyperopt_defaults_,\n set_llm_parameters,\n set_preprocessing_parameters,\n set_tagger_decoder_parameters,\n set_validation_parameters,\n)\nfrom ludwig.schema.preprocessing import PreprocessingConfig\nfrom ludwig.schema.trainer import BaseTrainerConfig\nfrom ludwig.schema.utils import ludwig_dataclass\nfrom ludwig.types import ModelConfigDict\nfrom ludwig.utils.backward_compatibility import upgrade_config_dict_to_latest_version\nfrom ludwig.utils.data_utils import get_sanitized_feature_name, load_yaml\nfrom ludwig.utils.registry import Registry\n\nmodel_type_schema_registry = Registry()\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass ModelConfig(schema_utils.BaseMarshmallowConfig, ABC):\n input_features: FeatureCollection[BaseInputFeatureConfig]\n output_features: FeatureCollection[BaseOutputFeatureConfig]\n\n model_type: str\n\n trainer: BaseTrainerConfig\n preprocessing: PreprocessingConfig\n defaults: BaseDefaultsConfig\n hyperopt: HyperoptConfig | None = None\n\n backend: dict[str, Any] = schema_utils.Dict() # TODO(jeffkinnison): Add backend schema\n ludwig_version: str = schema_utils.ProtectedString(LUDWIG_VERSION)\n\n def __post_init__(self):\n merge_fixed_preprocessing_params(self)\n set_validation_parameters(self)\n set_hyperopt_defaults_(self)\n set_tagger_decoder_parameters(self)\n sanitize_and_filter_combiner_entities_(self)\n\n # Reconcile LLM parameters\n set_llm_parameters(self)\n\n # Reconcile conflicting preprocessing parameters\n set_preprocessing_parameters(self)\n\n # Derive proc_col for each feature from the feature's preprocessing parameters\n # after all preprocessing parameters have been set\n set_derived_feature_columns_(self)\n\n # Auxiliary checks.\n get_config_check_registry().check_config(self)\n\n @staticmethod\n def from_dict(config: ModelConfigDict) -> \"ModelConfig\":\n config = copy.deepcopy(config)\n config = upgrade_config_dict_to_latest_version(config)\n\n # Use sanitized feature names.\n # NOTE: This must be kept consistent with build_dataset()\n for input_feature in config[INPUT_FEATURES]:\n input_feature[NAME] = get_sanitized_feature_name(input_feature[NAME])\n if COLUMN in input_feature and input_feature[COLUMN]:\n input_feature[COLUMN] = get_sanitized_feature_name(input_feature[COLUMN])\n for output_feature in config[OUTPUT_FEATURES]:\n output_feature[NAME] = get_sanitized_feature_name(output_feature[NAME])\n if COLUMN in output_feature and output_feature[COLUMN]:\n output_feature[COLUMN] = get_sanitized_feature_name(output_feature[COLUMN])\n\n # Sanitize tied feature names.\n for input_feature in config[INPUT_FEATURES]:\n if TIED in input_feature and input_feature[TIED]:\n input_feature[TIED] = get_sanitized_feature_name(input_feature[TIED])\n\n # Sanitize dependent feature names.\n for output_feature in config[OUTPUT_FEATURES]:\n if DEPENDENCIES in output_feature and output_feature[DEPENDENCIES]:\n output_feature[DEPENDENCIES] = [\n get_sanitized_feature_name(feature_name) for feature_name in output_feature[DEPENDENCIES]\n ]\n\n config[\"model_type\"] = config.get(\"model_type\", MODEL_ECD)\n model_type = config[\"model_type\"]\n if model_type not in model_type_schema_registry:\n raise ConfigValidationError(\n f\"Invalid model type: '{model_type}', expected one of: {list(model_type_schema_registry.keys())}\"\n )\n\n config = merge_with_defaults(config)\n\n # TODO(travis): handle this with helper function\n backend = config.get(BACKEND)\n if isinstance(backend, str):\n config[BACKEND] = {\"type\": backend}\n\n # JSON schema validation. Note that this is desireable on top of `schema.load(config)` below because marshmallow\n # deserialization permits additional properties while JSON schema validation, for schema (e.g. `trainer`) that\n # have `additionalProperties=False`, does not.\n #\n # Illustrative example: test_validate_config_misc.py::test_validate_no_trainer_type\n #\n # TODO: Set `additionalProperties=False` for all Ludwig schema, and look into passing in `unknown='RAISE'` to\n # marshmallow.load(), which raises an error for unknown fields during deserialization.\n # https://marshmallow.readthedocs.io/en/stable/marshmallow.schema.html#marshmallow.schema.Schema.load\n check_schema(config)\n\n cls = model_type_schema_registry[model_type]\n schema = cls.get_class_schema()()\n try:\n config_obj: ModelConfig = schema.load(config)\n except ConfigValidationError:\n raise\n except ValueError as e:\n raise ConfigValidationError(f\"Config validation error raised during config deserialization: {e}\") from e\n except (OSError, ValueError) as e:\n raise ConfigValidationError(f\"Config validation error raised during config post-init: {e}\") from e\n\n return config_obj\n\n @staticmethod\n def from_yaml(config_path: str) -> \"ModelConfig\":\n return ModelConfig.from_dict(load_yaml(config_path))\n\n def get_feature_names(self) -> set[str]:\n \"\"\"Returns a set of all feature names.\"\"\"\n feature_names = set()\n feature_names.update([f.column for f in self.input_features])\n feature_names.update([f.column for f in self.output_features])\n return feature_names\n\n def get_feature_config(self, feature_column_name: str) -> BaseInputFeatureConfig | None:\n \"\"\"Returns the feature config for the given feature name.\"\"\"\n for feature in self.input_features:\n if feature.column == feature_column_name:\n return feature\n for feature in self.output_features:\n if feature.column == feature_column_name:\n return feature\n\n\n@DeveloperAPI\ndef register_model_type(name: str):\n def wrap(model_type_config: ModelConfig) -> ModelConfig:\n model_type_schema_registry[name] = model_type_config\n return model_type_config\n\n return wrap\n\n\ndef _merge_encoder_cache_params(preprocessing_params: dict[str, Any], encoder_params: dict[str, Any]) -> dict[str, Any]:\n if preprocessing_params.get(\"cache_encoder_embeddings\"):\n preprocessing_params[ENCODER] = encoder_params\n return preprocessing_params\n"
},
{
"path": "ludwig/schema/model_types/ecd.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.combiners.base import BaseCombinerConfig\nfrom ludwig.schema.combiners.utils import CombinerSelection\nfrom ludwig.schema.defaults.ecd import ECDDefaultsConfig, ECDDefaultsField\nfrom ludwig.schema.features.base import (\n BaseInputFeatureConfig,\n BaseOutputFeatureConfig,\n ECDInputFeatureSelection,\n ECDOutputFeatureSelection,\n FeatureCollection,\n)\nfrom ludwig.schema.hyperopt import HyperoptConfig, HyperoptField\nfrom ludwig.schema.model_types.base import ModelConfig, register_model_type\nfrom ludwig.schema.preprocessing import PreprocessingConfig, PreprocessingField\nfrom ludwig.schema.trainer import ECDTrainerConfig, ECDTrainerField\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@register_model_type(name=\"ecd\")\n@ludwig_dataclass\nclass ECDModelConfig(ModelConfig):\n \"\"\"Parameters for ECD.\"\"\"\n\n model_type: str = schema_utils.ProtectedString(\"ecd\")\n\n input_features: FeatureCollection[BaseInputFeatureConfig] = ECDInputFeatureSelection().get_list_field()\n output_features: FeatureCollection[BaseOutputFeatureConfig] = ECDOutputFeatureSelection().get_list_field()\n\n combiner: BaseCombinerConfig = CombinerSelection().get_default_field()\n\n trainer: ECDTrainerConfig = ECDTrainerField().get_default_field()\n preprocessing: PreprocessingConfig = PreprocessingField().get_default_field()\n defaults: ECDDefaultsConfig = ECDDefaultsField().get_default_field()\n hyperopt: HyperoptConfig | None = HyperoptField().get_default_field()\n"
},
{
"path": "ludwig/schema/model_types/llm.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.defaults.llm import LLMDefaultsConfig, LLMDefaultsField\nfrom ludwig.schema.features.base import (\n BaseInputFeatureConfig,\n BaseOutputFeatureConfig,\n FeatureCollection,\n LLMInputFeatureSelection,\n LLMOutputFeatureSelection,\n)\nfrom ludwig.schema.hyperopt import HyperoptConfig, HyperoptField\nfrom ludwig.schema.llms.base_model import BaseModelDataclassField\nfrom ludwig.schema.llms.generation import LLMGenerationConfig, LLMGenerationConfigField\nfrom ludwig.schema.llms.model_parameters import ModelParametersConfig, ModelParametersConfigField\nfrom ludwig.schema.llms.peft import AdapterDataclassField, BaseAdapterConfig\nfrom ludwig.schema.llms.prompt import PromptConfig, PromptConfigField\nfrom ludwig.schema.llms.quantization import QuantizationConfig, QuantizationConfigField\nfrom ludwig.schema.model_types.base import ModelConfig, register_model_type\nfrom ludwig.schema.preprocessing import PreprocessingConfig, PreprocessingField\nfrom ludwig.schema.trainer import LLMTrainerConfig, LLMTrainerDataclassField\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@register_model_type(name=\"llm\")\n@ludwig_dataclass\nclass LLMModelConfig(ModelConfig):\n \"\"\"Parameters for LLM Model Type.\"\"\"\n\n model_type: str = schema_utils.ProtectedString(\"llm\")\n\n base_model: str = BaseModelDataclassField()\n\n input_features: FeatureCollection[BaseInputFeatureConfig] = LLMInputFeatureSelection().get_list_field()\n output_features: FeatureCollection[BaseOutputFeatureConfig] = LLMOutputFeatureSelection().get_list_field()\n\n preprocessing: PreprocessingConfig = PreprocessingField().get_default_field()\n defaults: LLMDefaultsConfig | None = LLMDefaultsField().get_default_field()\n hyperopt: HyperoptConfig | None = HyperoptField().get_default_field()\n\n prompt: PromptConfig = PromptConfigField().get_default_field()\n\n # trainer: LLMTrainerConfig = LLMTrainerField().get_default_field()\n trainer: LLMTrainerConfig = LLMTrainerDataclassField(\n description=\"The trainer to use for the model\",\n )\n\n generation: LLMGenerationConfig = LLMGenerationConfigField().get_default_field()\n\n adapter: BaseAdapterConfig | None = AdapterDataclassField()\n quantization: QuantizationConfig | None = QuantizationConfigField().get_default_field()\n model_parameters: ModelParametersConfig | None = ModelParametersConfigField().get_default_field()\n\n trust_remote_code: bool = schema_utils.Boolean(\n default=False,\n description=(\n \"Whether to trust and execute remote code from the HuggingFace model repository. \"\n \"Required for some models (e.g. Phi-2, Qwen) that use custom architectures. \"\n \"Only enable this for models you trust.\"\n ),\n )\n"
},
{
"path": "ludwig/schema/model_types/utils.py",
"content": "import copy\nimport logging\nimport sys\nimport warnings\nfrom collections.abc import Mapping\nfrom typing import Any, TYPE_CHECKING\n\nfrom transformers import AutoConfig\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import (\n CATEGORY,\n COMBINED,\n DECODER,\n DEFAULTS,\n ENCODER,\n GRID_SEARCH,\n INPUT_FEATURES,\n LOSS,\n MODEL_ECD,\n MODEL_LLM,\n OUTPUT_FEATURES,\n PARAMETERS,\n PREPROCESSING,\n SEQUENCE,\n SPACE,\n TEXT,\n TYPE,\n)\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.features.feature_utils import compute_feature_hash\nfrom ludwig.schema.features.utils import output_config_registry\nfrom ludwig.schema.hyperopt.scheduler import BaseHyperbandSchedulerConfig\nfrom ludwig.schema.llms.generation import LLMGenerationConfig\nfrom ludwig.schema.trainer import ECDTrainerConfig\nfrom ludwig.types import HyperoptConfigDict, ModelConfigDict\nfrom ludwig.utils.data_utils import get_sanitized_feature_name\nfrom ludwig.utils.llm_utils import get_context_len\n\nif TYPE_CHECKING:\n from ludwig.schema.model_types.base import ModelConfig\n\n\nlogger = logging.getLogger(__name__)\n\n\n@DeveloperAPI\ndef merge_with_defaults(config_dict: ModelConfigDict) -> ModelConfigDict:\n # Recursive merge of the features, except that if we find a dictionary containing\n # an explicit \"type\" key, we ignore defaults if they don't match.\n defaults = config_dict.get(DEFAULTS)\n if not defaults:\n return config_dict\n\n config_dict = copy.deepcopy(config_dict)\n _merge_features_(config_dict.get(INPUT_FEATURES, []), defaults, {DECODER, LOSS})\n _merge_features_(config_dict.get(OUTPUT_FEATURES, []), defaults, {ENCODER, PREPROCESSING})\n return config_dict\n\n\ndef _merge_features_(features: list[dict[str, Any]], defaults: dict[str, Any], exclude_keys: set[str]):\n for feature in features:\n ftype = feature.get(TYPE)\n if not ftype:\n continue\n\n default_feature = defaults.get(ftype, {})\n merged_feature = _merge_dict_with_types(default_feature, feature, exclude_keys)\n\n # In-place replacement of the old feature with the new\n feature.clear()\n feature.update(merged_feature)\n\n\ndef _merge_dict_with_types(dct: dict[str, Any], merge_dct: dict[str, Any], exclude_keys: set[str]) -> dict[str, Any]:\n dct = copy.deepcopy(dct)\n dct = {k: v for k, v in dct.items() if k not in exclude_keys}\n\n for k, v in merge_dct.items():\n # TODO(travis): below type comparison is not perfect, as it doesn't consider the case where the default type\n # is omitted while the encoder type is explicitly set to the default type, in which case they\n # should resolve to equal, but will be considered different.\n if (\n k in dct\n and isinstance(dct[k], dict)\n and isinstance(v, Mapping)\n and dct[k].get(TYPE) == v.get(TYPE, dct[k].get(TYPE))\n ):\n dct[k] = _merge_dict_with_types(dct[k], v, exclude_keys)\n else:\n dct[k] = v\n return dct\n\n\n@DeveloperAPI\ndef merge_fixed_preprocessing_params(config: \"ModelConfig\"):\n \"\"\"Update preprocessing parameters if encoders require fixed preprocessing parameters.\"\"\"\n for feature in config.input_features:\n feature.encoder.set_fixed_preprocessing_params(config.model_type, feature.preprocessing)\n\n\ndef set_validation_parameters(config: \"ModelConfig\"):\n \"\"\"Sets validation-related parameters used for early stopping, determining the best hyperopt trial, etc.\"\"\"\n if not config.output_features:\n return\n\n # First set the validation field so we know what feature we're validating on\n if not config.trainer.validation_field:\n if config.trainer.validation_metric is None or config.trainer.validation_metric == LOSS:\n # Loss is valid for all features.\n config.trainer.validation_field = config.output_features[0].name\n else:\n # Determine the proper validation field for the user, like if the user specifies \"accuracy\" but forgets to\n # change the validation field from \"combined\" to the name of the feature that produces accuracy metrics.\n from ludwig.utils.metric_utils import get_feature_to_metric_names_map\n\n feature_to_metric_names_map = get_feature_to_metric_names_map(config.output_features.to_list())\n validation_field = None\n for feature_name, metric_names in feature_to_metric_names_map.items():\n if config.trainer.validation_metric in metric_names:\n if validation_field is None:\n validation_field = feature_name\n else:\n raise ConfigValidationError(\n f\"The validation_metric: '{config.trainer.validation_metric}' corresponds to multiple \"\n f\"possible validation_fields, '{validation_field}' and '{feature_name}'. Please explicitly \"\n \"specify the validation_field that should be used with the validation_metric \"\n f\"'{config.trainer.validation_metric}'.\"\n )\n if validation_field is None:\n raise ConfigValidationError(\n \"User-specified trainer.validation_metric is not valid for any output feature.\"\n )\n\n config.trainer.validation_field = validation_field\n\n # If the field is combined, then make sure the metric is loss and then return\n if config.trainer.validation_field == COMBINED:\n # Only loss is supported for combined\n if not config.trainer.validation_metric:\n config.trainer.validation_metric = LOSS\n elif config.trainer.validation_metric != LOSS:\n raise ConfigValidationError(\n f\"Must set validation_metric=loss when validation_field=combined, \"\n f\"found validation_metric={config.trainer.validation_metric}\"\n )\n return\n\n # Field is not combined, so use the default validation metric for the single feature\n validation_features = [f for f in config.output_features if f.name == config.trainer.validation_field]\n if len(validation_features) > 1:\n raise ConfigValidationError(\n f\"Found more than one feature matching validation field: {config.trainer.validation_field}\"\n )\n if len(validation_features) == 0:\n raise ConfigValidationError(\n f\"No output feature found matching validation field: {config.trainer.validation_field}\"\n )\n\n validation_feature = validation_features[0]\n if not config.trainer.validation_metric:\n # The user has not explicitly set any validation fields.\n # Default to using the first output feature's default validation metric.\n out_type = validation_feature.type\n config.trainer.validation_metric = output_config_registry(config.model_type)[out_type].default_validation_metric\n\n\ndef set_derived_feature_columns_(config_obj: \"ModelConfig\"):\n \"\"\"Assigns column and proc_column values to features that do not have them set.\n\n Proc_column is set to a hash of the feature's preprocessing configuration.\n \"\"\"\n for feature in config_obj.input_features:\n if feature.column is None:\n feature.column = feature.name\n if feature.proc_column is None:\n feature.proc_column = compute_feature_hash(feature.to_dict())\n\n for feature in config_obj.output_features:\n if feature.column is None:\n feature.column = feature.name\n if feature.proc_column is None:\n feature.proc_column = compute_feature_hash(feature.to_dict())\n\n\ndef sanitize_and_filter_combiner_entities_(config: \"ModelConfig\"):\n if config.model_type != MODEL_ECD or config.combiner.type != \"comparator\":\n return\n\n input_feature_names = {input_feature.name for input_feature in config.input_features}\n\n # Sanitize feature names.\n config.combiner.entity_1 = [get_sanitized_feature_name(fname) for fname in config.combiner.entity_1]\n config.combiner.entity_2 = [get_sanitized_feature_name(fname) for fname in config.combiner.entity_2]\n\n entity_1_excluded = {fname for fname in config.combiner.entity_1 if fname not in input_feature_names}\n if entity_1_excluded:\n logger.warning(\n f\"Excluding `entity_1` features {entity_1_excluded} from the comparator combiner because they are not \"\n f\"present in the `input_features`.\"\n )\n\n config.combiner.entity_1 = [fname for fname in config.combiner.entity_1 if fname not in entity_1_excluded]\n\n entity_2_excluded = {fname for fname in config.combiner.entity_2 if fname not in input_feature_names}\n if entity_2_excluded:\n logger.warning(\n f\"Excluding `entity_2` features {entity_2_excluded} from the comparator combiner because they are not \"\n f\"present in the `input_features`.\"\n )\n\n config.combiner.entity_2 = [fname for fname in config.combiner.entity_2 if fname not in entity_2_excluded]\n\n\ndef set_hyperopt_defaults_(config: \"ModelConfig\"):\n \"\"\"This function was migrated from defaults.py with the intention of setting some hyperopt defaults while the\n hyperopt section of the config object is not fully complete.\n\n Returns:\n None -> modifies trainer and hyperopt sections\n \"\"\"\n if not config.hyperopt:\n return\n\n # Set default num_samples based on search space if not set by user\n if config.hyperopt.executor.num_samples is None:\n _contains_grid_search_params = contains_grid_search_parameters(config.hyperopt.to_dict())\n if _contains_grid_search_params:\n logger.info(\n \"Setting hyperopt num_samples to 1 to prevent duplicate trials from being run. Duplicate trials are\"\n \" created when there are hyperopt parameters that use the `grid_search` search space.\",\n )\n config.hyperopt.executor.num_samples = 1\n else:\n logger.info(\"Setting hyperopt num_samples to 10.\")\n config.hyperopt.executor.num_samples = 10\n\n scheduler = config.hyperopt.executor.scheduler\n if scheduler.type == \"fifo\":\n # FIFO scheduler has no constraints\n return\n\n # Disable early stopping when using a scheduler. We achieve this by setting the parameter\n # to -1, which ensures the condition to apply early stopping is never met.\n early_stop = config.trainer.early_stop\n if early_stop is not None and early_stop != -1:\n warnings.warn(\"Can't utilize `early_stop` while using a hyperopt scheduler. Setting early stop to -1.\")\n config.trainer.early_stop = -1\n\n if isinstance(config.trainer, ECDTrainerConfig) and isinstance(scheduler, BaseHyperbandSchedulerConfig):\n # TODO(travis): explore similar constraints for other model types that may not have epochs\n max_t = scheduler.max_t\n time_attr = scheduler.time_attr\n epochs = config.trainer.epochs\n if max_t is not None:\n if time_attr == \"time_total_s\":\n if epochs is None:\n # Continue training until time limit hit\n config.trainer.epochs = sys.maxsize\n # else continue training until either time or trainer epochs limit hit\n elif epochs is not None and epochs != max_t:\n raise ValueError(\n \"Cannot set trainer `epochs` when using hyperopt scheduler w/different training_iteration `max_t`. \"\n \"Unset one of these parameters in your config or make sure their values match.\"\n )\n else:\n # Run trainer until scheduler epochs limit hit\n config.trainer.epochs = max_t\n elif epochs is not None:\n scheduler.max_t = epochs # run scheduler until trainer epochs limit hit\n\n\ndef set_preprocessing_parameters(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Reconcile conflicting preprocessing parameters in place.\"\"\"\n _set_max_sequence_length(config)\n\n\ndef _set_max_sequence_length(config: \"ModelConfig\") -> None: # noqa: F821\n \"\"\"Ensures that `max_sequence_length` is never less than `sequence_length`.\"\"\"\n\n types_with_sequence_length = [SEQUENCE, TEXT]\n for input_feature in config.input_features:\n if input_feature.type in types_with_sequence_length:\n sequence_length = input_feature.preprocessing.sequence_length\n max_sequence_length = input_feature.preprocessing.max_sequence_length\n if sequence_length is not None and sequence_length > max_sequence_length:\n warnings.warn(\n \"if `sequence_length` is not None, `max_sequence_length` must be greater than or equal \"\n \"to `sequence_length`. Setting `max_sequence_length` to `sequence_length`.\"\n )\n input_feature.preprocessing.max_sequence_length = sequence_length\n\n\ndef set_tagger_decoder_parameters(config: \"ModelConfig\") -> None:\n \"\"\"Overrides the reduce_input parameter for text and sequence output features when a tagger decoder is used.\n This is done to ensure that the decoder correctly gets a 3D tensor as input.\n\n Returns:\n None -> modifies output_features\n \"\"\"\n for output_feature in config.output_features:\n if output_feature.type in {TEXT, SEQUENCE} and output_feature.decoder.type == \"tagger\":\n if output_feature.reduce_input is not None:\n warnings.warn(\n \"reduce_input must be set to `None` when using a tagger decoder for your output feature. \"\n f\"Setting reduce_input to `None` for `{output_feature.name}`.\"\n )\n output_feature.reduce_input = None\n\n\ndef set_llm_parameters(config: \"ModelConfig\") -> None:\n if config.model_type != MODEL_LLM:\n return\n\n # Set preprocessing parameters for text features for LLM model type\n _set_llm_tokenizers(config)\n\n # Set max_new_tokens in generation config to the max sequence length of the output features\n _set_generation_max_new_tokens(config)\n\n # HACK(Arnav): Set Mixtral target modules when using LoRA\n # GitHub issue: https://github.com/ludwig-ai/ludwig/issues/3853\n # PEFT PR: https://github.com/huggingface/peft/pull/1376\n _set_mixtral_target_modules(config)\n\n # HACK(Arnav): Set Phi-2 target modules when using LoRA\n # GitHub issue: https://github.com/ludwig-ai/ludwig/issues/3910\n # PEFT PR: https://github.com/huggingface/peft/pull/1375\n _set_phi2_target_modules(config)\n\n # HACK(Arnav): Set Phi-3 target modules when using LoRA\n _set_phi3_target_modules(config)\n\n # HACK(Arnav): Set Gemma target modules when using LoRA\n # GitHub issue: https://github.com/ludwig-ai/ludwig/issues/3937\n # PEFT PR: https://github.com/huggingface/peft/pull/1499\n _set_gemma_target_modules(config)\n\n\ndef _set_llm_tokenizers(config: \"ModelConfig\") -> None:\n \"\"\"Sets the tokenizers for the LLM model to the pretrained model name or path. This ensures that they use the\n correct shared vocabulary from the tokenizer.\n\n This also ensures padding is correctly set to left padding to prevent the LLM from trying to continue to sequence\n based on the right padding tokens, which might exist based on sequence length.\n \"\"\"\n pretrained_model_name_or_path = config.base_model\n if not isinstance(pretrained_model_name_or_path, str) or pretrained_model_name_or_path is None:\n raise ValueError(\"Must set `base_model` when using the LLM model.\")\n\n for input_feature in config.input_features:\n if input_feature.type == TEXT:\n input_feature.preprocessing.tokenizer = \"hf_tokenizer\"\n input_feature.preprocessing.pretrained_model_name_or_path = pretrained_model_name_or_path\n input_feature.preprocessing.padding = \"left\"\n\n for output_feature in config.output_features:\n if output_feature.type == TEXT:\n # Add tokenizer parameters to preprocessing so it can be used during post processing\n output_feature.preprocessing.tokenizer = \"hf_tokenizer\"\n output_feature.preprocessing.pretrained_model_name_or_path = pretrained_model_name_or_path\n output_feature.preprocessing.padding = \"left\"\n\n # Add tokenizer parameters to decoder so it can be used during the forward pass\n output_feature.decoder.pretrained_model_name_or_path = pretrained_model_name_or_path\n output_feature.decoder.max_new_tokens = config.generation.max_new_tokens\n elif output_feature.type == CATEGORY:\n # Tokenizer parameters\n output_feature.decoder.tokenizer = \"hf_tokenizer\"\n output_feature.decoder.pretrained_model_name_or_path = pretrained_model_name_or_path\n # Parameters for building decoder vocabulary\n output_feature.decoder.fallback_label = output_feature.preprocessing.fallback_label\n\n\ndef _get_maximum_possible_sequence_length(config: \"ModelConfig\", default_max_sequence_length: int) -> int:\n \"\"\"Returns the maximum possible sequence length for the LLM model based on the model config.\"\"\"\n max_possible_sequence_length = default_max_sequence_length\n if config.output_features[0].preprocessing.max_sequence_length is not None:\n # Note: We don't need to check for max between feature.preprocessing.max_sequence_length and\n # defaults.text.preprocessing.max_sequence_length because the latter is only applied to input features.\n max_possible_sequence_length = max(\n default_max_sequence_length, config.output_features[0].preprocessing.max_sequence_length\n )\n elif config.preprocessing.global_max_sequence_length is not None:\n # This is not perfect since it includes tokens from both input + output features, but this at least\n # ensures that max possible of the sequence length is used. It is very likely that the model learns\n # to generate sequences than this value.\n max_possible_sequence_length = max(\n max_possible_sequence_length, config.preprocessing.global_max_sequence_length\n )\n elif max_possible_sequence_length == default_max_sequence_length:\n # It's possible that both max_sequence_length and global_max_sequence_length are not set, in which case\n # we should fall back to the window size of the pretrained model. By this point, because of schema validation\n # checks, we know that the base_model exists so we can safely grab the base model's config.\n # TODO (Arnav): Figure out how to factor in rope scaling factor into this calculation.\n model_config = AutoConfig.from_pretrained(config.base_model)\n max_possible_sequence_length = get_context_len(model_config)\n # Artifically leave a buffer of half the total model window size to trade off\n # runtime while likely covering a majority of the max sequence length.\n max_possible_sequence_length = max_possible_sequence_length // 2\n return max_possible_sequence_length\n\n\ndef _set_generation_max_new_tokens(config: \"ModelConfig\") -> None:\n \"\"\"Sets the max_new_tokens parameter in the generation config to the max sequence length of the output\n features.\n\n This ensures that the generation config is set to the correct value for the LLM model type.\n \"\"\"\n _DEFAULT_MAX_SEQUENCE_LENGTH = LLMGenerationConfig().max_new_tokens\n if config.generation.max_new_tokens != _DEFAULT_MAX_SEQUENCE_LENGTH:\n # Max new tokens is explicitly set by user, so don't override\n return\n\n if config.output_features[0].type != TEXT:\n # This is trickier to set for other output features, so don't override for now.\n # TODO: Add better support for category output features\n return\n\n max_possible_sequence_length = _get_maximum_possible_sequence_length(config, _DEFAULT_MAX_SEQUENCE_LENGTH)\n\n logger.info(\n f\"Setting generation max_new_tokens to {max_possible_sequence_length} to correspond with the max \"\n \"sequence length assigned to the output feature or the global max sequence length. This will ensure that \"\n \"the correct number of tokens are generated at inference time. To override this behavior, set \"\n \"`generation.max_new_tokens` to a different value in your Ludwig config.\"\n )\n config.generation.max_new_tokens = max_possible_sequence_length\n\n\ndef _set_mixtral_target_modules(config: \"ModelConfig\") -> None:\n \"\"\"If the base model is Mixtral 7x8, LoRA is enabled and the target modules are not set, set the target modules\n to q_proj and v_proj.\"\"\"\n if config.base_model not in {\"mistralai/Mixtral-8x7B-v0.1\", \"mistralai/Mixtral-8x7B-Instruct-v0.1\"}:\n return\n\n if not config.adapter:\n return\n\n if config.adapter.type != \"lora\" or config.adapter.target_modules:\n return\n\n target_modules = [\"q_proj\", \"v_proj\"]\n\n logger.info(f\"Setting adapter target modules to {target_modules} for Mixtral 7x8 base model with LoRA adapter.\")\n config.adapter.target_modules = target_modules\n\n\ndef _set_phi2_target_modules(config: \"ModelConfig\") -> None:\n \"\"\"If the base model is Phi-2, LoRA is enabled and the target modules are not set, set the target modules to\n maximize performance.\"\"\"\n if config.base_model not in {\n \"microsoft/phi-1\",\n \"microsoft/phi-1_5\",\n \"microsoft/phi-2\",\n }:\n return\n\n if not config.adapter:\n return\n\n if config.adapter.type != \"lora\" or config.adapter.target_modules:\n return\n\n target_modules = [\"q_proj\", \"k_proj\", \"v_proj\", \"dense\", \"fc1\", \"fc2\"]\n\n logger.info(f\"Setting adapter target modules to {target_modules} for Phi-2 base model with LoRA adapter.\")\n config.adapter.target_modules = target_modules\n\n\ndef _set_phi3_target_modules(config: \"ModelConfig\") -> None:\n if config.base_model not in {\n \"microsoft/Phi-3-mini-4k-instruct\",\n \"microsoft/Phi-3-mini-128k-instruct\",\n }:\n return\n\n if not config.adapter:\n return\n\n if config.adapter.type != \"lora\" or config.adapter.target_modules:\n return\n\n target_modules = [\"qkv_proj\", \"o_proj\", \"gate_up_proj\", \"down_proj\"]\n\n logger.info(f\"Setting adapter target modules to {target_modules} for Phi-3 base model with LoRA adapter.\")\n config.adapter.target_modules = target_modules\n\n\ndef _set_gemma_target_modules(config: \"ModelConfig\") -> None:\n \"\"\"If the base model is Gemma, LoRA is enabled and the target modules are not set, set the target modules to\n maximize performance.\"\"\"\n if config.base_model not in {\"google/gemma-2b\", \"google/gemma-2b-it\", \"google/gemma-7b\", \"google/gemma-7b-it\"}:\n return\n\n if not config.adapter:\n return\n\n if config.adapter.type != \"lora\" or config.adapter.target_modules:\n return\n\n target_modules = [\"q_proj\", \"v_proj\"]\n config.adapter.target_modules = target_modules\n\n\n@DeveloperAPI\ndef contains_grid_search_parameters(hyperopt_config: HyperoptConfigDict) -> bool:\n \"\"\"Returns True if any hyperopt parameter in the config is using the grid_search space.\"\"\"\n for _, param_info in hyperopt_config[PARAMETERS].items():\n if param_info.get(SPACE, None) == GRID_SEARCH:\n return True\n return False\n"
},
{
"path": "ludwig/schema/optimizers.py",
"content": "from abc import ABC\nfrom dataclasses import field\nfrom typing import ClassVar\n\nimport torch\n\ntry:\n import bitsandbytes as bnb\nexcept Exception:\n bnb = None\nimport ludwig.schema.utils as schema_utils\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.schema.metadata import OPTIMIZER_METADATA\nfrom ludwig.schema.metadata.parameter_metadata import convert_metadata_to_json, ParameterMetadata\nfrom ludwig.schema.utils import ludwig_dataclass\nfrom ludwig.utils.registry import Registry\n\noptimizer_registry = Registry()\n\n\n@DeveloperAPI\ndef register_optimizer(name: str):\n def wrap(optimizer_config: BaseOptimizerConfig):\n optimizer_registry[name] = (optimizer_config.optimizer_class, optimizer_config)\n return optimizer_config\n\n return wrap\n\n\n@DeveloperAPI\ndef get_optimizer_cls(name: str):\n \"\"\"Get the optimizer schema class from the optimizer schema class registry.\"\"\"\n return optimizer_registry[name][1]\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass BaseOptimizerConfig(schema_utils.BaseMarshmallowConfig, ABC):\n \"\"\"Base class for optimizers. Not meant to be used directly.\n\n The dataclass format prevents arbitrary properties from being set. Consequently, in child classes, all properties\n from the corresponding `torch.optim.Optimizer` class are copied over: check each class to check which attributes are\n different from the torch-specified defaults.\n \"\"\"\n\n optimizer_class: ClassVar[torch.optim.Optimizer | None] = None\n \"Class variable pointing to the corresponding `torch.optim.Optimizer` class.\"\n\n type: str\n \"\"\"Name corresponding to an optimizer `ludwig.modules.optimization_modules.optimizer_registry`.\n\n Technically mutable, but attempting to load a derived optimizer with `type` set to a mismatched value will result in\n a `ValidationError`.\n \"\"\"\n\n @property\n def is_paged(self) -> bool:\n \"\"\"Returns True if the optimizer is a Paged optimizer.\"\"\"\n return False\n\n @property\n def is_8bit(self) -> bool:\n \"\"\"Returns True if the optimizer is an 8-bit optimizer.\"\"\"\n return False\n\n\n@DeveloperAPI\n@register_optimizer(name=\"sgd\")\n@ludwig_dataclass\nclass SGDOptimizerConfig(BaseOptimizerConfig):\n \"\"\"Parameters for stochastic gradient descent.\"\"\"\n\n optimizer_class: ClassVar[torch.optim.Optimizer] = torch.optim.SGD\n \"\"\"Points to `torch.optim.SGD`.\"\"\"\n\n type: str = schema_utils.ProtectedString(\"sgd\")\n \"\"\"Must be 'sgd' - corresponds to name in `ludwig.modules.optimization_modules.optimizer_registry` (default:\n 'sgd')\"\"\"\n\n # Defaults taken from https://pytorch.org/docs/stable/generated/torch.optim.SGD.html#torch.optim.SGD :\n momentum: float = schema_utils.NonNegativeFloat(\n default=0.0,\n description=\"Momentum factor.\",\n parameter_metadata=OPTIMIZER_METADATA[\"momentum\"],\n )\n\n weight_decay: float = schema_utils.NonNegativeFloat(\n default=0.0,\n description=\"Weight decay ($L2$ penalty).\",\n parameter_metadata=OPTIMIZER_METADATA[\"weight_decay\"],\n )\n\n dampening: float = schema_utils.NonNegativeFloat(\n default=0.0,\n description=\"Dampening for momentum.\",\n parameter_metadata=OPTIMIZER_METADATA[\"dampening\"],\n )\n\n nesterov: bool = schema_utils.Boolean(\n default=False,\n description=\"Enables Nesterov momentum.\",\n parameter_metadata=OPTIMIZER_METADATA[\"nesterov\"],\n )\n\n\nif bnb is not None:\n\n @DeveloperAPI\n @register_optimizer(name=\"sgd_8bit\")\n @ludwig_dataclass\n class SGD8BitOptimizerConfig(SGDOptimizerConfig):\n \"\"\"Parameters for stochastic gradient descent.\"\"\"\n\n optimizer_class: ClassVar[torch.optim.Optimizer] = bnb.optim.SGD8bit\n\n type: str = schema_utils.ProtectedString(\"sgd_8bit\")\n\n block_wise: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to use block wise update.\",\n )\n\n percentile_clipping: int = schema_utils.IntegerRange(\n default=100,\n min=0,\n max=100,\n description=\"Percentile clipping.\",\n )\n\n @property\n def is_8bit(self) -> bool:\n return True\n\n\n@DeveloperAPI\n@register_optimizer(name=\"lbfgs\")\n@ludwig_dataclass\nclass LBFGSOptimizerConfig(BaseOptimizerConfig):\n \"\"\"Parameters for stochastic gradient descent.\"\"\"\n\n optimizer_class: ClassVar[torch.optim.Optimizer] = torch.optim.LBFGS\n \"\"\"Points to `torch.optim.LBFGS`.\"\"\"\n\n type: str = schema_utils.ProtectedString(\"lbfgs\")\n \"\"\"Must be 'lbfgs' - corresponds to name in `ludwig.modules.optimization_modules.optimizer_registry` (default:\n 'lbfgs')\"\"\"\n\n # Defaults taken from https://pytorch.org/docs/stable/generated/torch.optim.LBFGS.html#torch.optim.LBFGS\n max_iter: int = schema_utils.Integer(\n default=20,\n description=\"Maximum number of iterations per optimization step.\",\n parameter_metadata=OPTIMIZER_METADATA[\"max_iter\"],\n )\n\n max_eval: int = schema_utils.Integer(\n default=None,\n allow_none=True,\n description=\"Maximum number of function evaluations per optimization step. Default: `max_iter` * 1.25.\",\n parameter_metadata=OPTIMIZER_METADATA[\"max_eval\"],\n )\n\n tolerance_grad: float = schema_utils.NonNegativeFloat(\n default=1e-07,\n description=\"Termination tolerance on first order optimality.\",\n parameter_metadata=OPTIMIZER_METADATA[\"tolerance_grad\"],\n )\n\n tolerance_change: float = schema_utils.NonNegativeFloat(\n default=1e-09,\n description=\"Termination tolerance on function value/parameter changes.\",\n parameter_metadata=OPTIMIZER_METADATA[\"tolerance_change\"],\n )\n\n history_size: int = schema_utils.Integer(\n default=100, description=\"Update history size.\", parameter_metadata=OPTIMIZER_METADATA[\"history_size\"]\n )\n\n line_search_fn: str = schema_utils.StringOptions(\n [\"strong_wolfe\"],\n default=None,\n allow_none=True,\n description=\"Line search function to use.\",\n parameter_metadata=OPTIMIZER_METADATA[\"line_search_fn\"],\n )\n\n\n@DeveloperAPI\n@register_optimizer(name=\"adam\")\n@ludwig_dataclass\nclass AdamOptimizerConfig(BaseOptimizerConfig):\n \"\"\"Parameters for adam optimization.\"\"\"\n\n optimizer_class: ClassVar[torch.optim.Optimizer] = torch.optim.Adam\n \"\"\"Points to `torch.optim.Adam`.\"\"\"\n\n type: str = schema_utils.ProtectedString(\"adam\")\n \"\"\"Must be 'adam' - corresponds to name in `ludwig.modules.optimization_modules.optimizer_registry`\n (default: 'adam')\"\"\"\n\n # Defaults taken from https://pytorch.org/docs/stable/generated/torch.optim.Adam.html#torch.optim.Adam :\n betas: tuple[float, float] = schema_utils.FloatRangeTupleDataclassField(\n default=(0.9, 0.999),\n description=\"Coefficients used for computing running averages of gradient and its square.\",\n parameter_metadata=OPTIMIZER_METADATA[\"betas\"],\n )\n\n eps: float = schema_utils.NonNegativeFloat(\n default=1e-08,\n description=\"Term added to the denominator to improve numerical stability.\",\n parameter_metadata=OPTIMIZER_METADATA[\"eps\"],\n )\n\n weight_decay: float = schema_utils.NonNegativeFloat(\n default=0.0, description=\"Weight decay (L2 penalty).\", parameter_metadata=OPTIMIZER_METADATA[\"weight_decay\"]\n )\n\n amsgrad: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to use the AMSGrad variant of this algorithm from the paper 'On the Convergence of Adam \"\n \"and Beyond'.\",\n parameter_metadata=OPTIMIZER_METADATA[\"amsgrad\"],\n )\n\n\nif bnb is not None:\n\n @DeveloperAPI\n @register_optimizer(name=\"adam_8bit\")\n @ludwig_dataclass\n class Adam8BitOptimizerConfig(AdamOptimizerConfig):\n optimizer_class: ClassVar[torch.optim.Optimizer] = bnb.optim.Adam8bit\n\n type: str = schema_utils.ProtectedString(\"adam_8bit\")\n\n block_wise: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use block wise update.\",\n )\n\n percentile_clipping: int = schema_utils.IntegerRange(\n default=100,\n min=0,\n max=100,\n description=\"Percentile clipping.\",\n )\n\n @property\n def is_8bit(self) -> bool:\n return True\n\n @DeveloperAPI\n @register_optimizer(name=\"paged_adam\")\n @ludwig_dataclass\n class PagedAdamOptimizerConfig(Adam8BitOptimizerConfig):\n optimizer_class: ClassVar[torch.optim.Optimizer] = bnb.optim.PagedAdam\n\n type: str = schema_utils.ProtectedString(\"paged_adam\")\n\n @property\n def is_paged(self) -> bool:\n return True\n\n @property\n def is_8bit(self) -> bool:\n return False\n\n @DeveloperAPI\n @register_optimizer(name=\"paged_adam_8bit\")\n @ludwig_dataclass\n class PagedAdam8BitOptimizerConfig(PagedAdamOptimizerConfig):\n optimizer_class: ClassVar[torch.optim.Optimizer] = bnb.optim.PagedAdam8bit\n\n type: str = schema_utils.ProtectedString(\"paged_adam_8bit\")\n\n @property\n def is_8bit(self) -> bool:\n return True\n\n\n@DeveloperAPI\n@register_optimizer(name=\"adamw\")\n@ludwig_dataclass\nclass AdamWOptimizerConfig(BaseOptimizerConfig):\n \"\"\"Parameters for adamw optimization.\"\"\"\n\n optimizer_class: ClassVar[torch.optim.Optimizer] = torch.optim.AdamW\n \"\"\"Points to `torch.optim.AdamW`.\"\"\"\n\n type: str = schema_utils.ProtectedString(\"adamw\")\n \"\"\"Must be 'adamw' - corresponds to name in `ludwig.modules.optimization_modules.optimizer_registry`\n (default: 'adamw')\"\"\"\n\n # Defaults taken from https://pytorch.org/docs/stable/generated/torch.optim.Adam.html#torch.optim.Adam :\n betas: tuple[float, float] = schema_utils.FloatRangeTupleDataclassField(\n default=(0.9, 0.999),\n description=\"Coefficients used for computing running averages of gradient and its square.\",\n parameter_metadata=OPTIMIZER_METADATA[\"betas\"],\n )\n\n eps: float = schema_utils.NonNegativeFloat(\n default=1e-08,\n description=\"Term added to the denominator to improve numerical stability.\",\n parameter_metadata=OPTIMIZER_METADATA[\"eps\"],\n )\n\n weight_decay: float = schema_utils.NonNegativeFloat(\n default=0.0, description=\"Weight decay ($L2$ penalty).\", parameter_metadata=OPTIMIZER_METADATA[\"weight_decay\"]\n )\n\n amsgrad: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to use the AMSGrad variant of this algorithm from the paper 'On the Convergence of Adam \"\n \"and Beyond'. \",\n parameter_metadata=OPTIMIZER_METADATA[\"amsgrad\"],\n )\n\n\nif bnb is not None:\n\n @DeveloperAPI\n @register_optimizer(name=\"adamw_8bit\")\n @ludwig_dataclass\n class AdamW8BitOptimizerConfig(AdamWOptimizerConfig):\n optimizer_class: ClassVar[torch.optim.Optimizer] = bnb.optim.AdamW8bit\n\n type: str = schema_utils.ProtectedString(\"adamw_8bit\")\n\n block_wise: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use block wise update.\",\n )\n\n percentile_clipping: int = schema_utils.IntegerRange(\n default=100,\n min=0,\n max=100,\n description=\"Percentile clipping.\",\n )\n\n @property\n def is_8bit(self) -> bool:\n return True\n\n @DeveloperAPI\n @register_optimizer(name=\"paged_adamw\")\n @ludwig_dataclass\n class PagedAdamWOptimizerConfig(AdamW8BitOptimizerConfig):\n optimizer_class: ClassVar[torch.optim.Optimizer] = bnb.optim.PagedAdamW\n\n type: str = schema_utils.ProtectedString(\"paged_adamw\")\n\n @property\n def is_paged(self) -> bool:\n return True\n\n @property\n def is_8bit(self) -> bool:\n return False\n\n @DeveloperAPI\n @register_optimizer(name=\"paged_adamw_8bit\")\n @ludwig_dataclass\n class PagedAdamW8BitOptimizerConfig(PagedAdamWOptimizerConfig):\n optimizer_class: ClassVar[torch.optim.Optimizer] = bnb.optim.PagedAdamW8bit\n\n type: str = schema_utils.ProtectedString(\"paged_adamw_8bit\")\n\n @property\n def is_8bit(self) -> bool:\n return True\n\n\n@DeveloperAPI\n@register_optimizer(name=\"adadelta\")\n@ludwig_dataclass\nclass AdadeltaOptimizerConfig(BaseOptimizerConfig):\n \"\"\"Parameters for adadelta optimization.\"\"\"\n\n optimizer_class: ClassVar[torch.optim.Optimizer] = torch.optim.Adadelta\n \"\"\"Points to `torch.optim.Adadelta`.\"\"\"\n\n type: str = schema_utils.ProtectedString(\"adadelta\")\n \"\"\"Must be 'adadelta' - corresponds to name in `ludwig.modules.optimization_modules.optimizer_registry`\n (default: 'adadelta')\"\"\"\n\n # Defaults taken from https://pytorch.org/docs/stable/generated/torch.optim.Adadelta.html#torch.optim.Adadelta :\n rho: float = schema_utils.FloatRange(\n default=0.9,\n min=0,\n max=1,\n description=\"Coefficient used for computing a running average of squared gradients.\",\n parameter_metadata=OPTIMIZER_METADATA[\"rho\"],\n )\n\n eps: float = schema_utils.NonNegativeFloat(\n default=1e-06,\n description=\"Term added to the denominator to improve numerical stability.\",\n parameter_metadata=OPTIMIZER_METADATA[\"eps\"],\n )\n\n weight_decay: float = schema_utils.NonNegativeFloat(\n default=0.0, description=\"Weight decay ($L2$ penalty).\", parameter_metadata=OPTIMIZER_METADATA[\"weight_decay\"]\n )\n\n\n@DeveloperAPI\n@register_optimizer(name=\"adagrad\")\n@ludwig_dataclass\nclass AdagradOptimizerConfig(BaseOptimizerConfig):\n \"\"\"Parameters for adagrad optimization.\"\"\"\n\n # Example docstring\n optimizer_class: ClassVar[torch.optim.Optimizer] = torch.optim.Adagrad\n \"\"\"Points to `torch.optim.Adagrad`.\"\"\"\n\n type: str = schema_utils.ProtectedString(\"adagrad\")\n \"\"\"Must be 'adagrad' - corresponds to name in `ludwig.modules.optimization_modules.optimizer_registry`\n (default: 'adagrad')\"\"\"\n\n # Defaults taken from https://pytorch.org/docs/stable/generated/torch.optim.Adagrad.html#torch.optim.Adagrad :\n initial_accumulator_value: float = schema_utils.NonNegativeFloat(\n default=0, description=\"\", parameter_metadata=OPTIMIZER_METADATA[\"initial_accumulator_value\"]\n )\n\n lr_decay: float = schema_utils.FloatRange(\n default=0, description=\"Learning rate decay.\", parameter_metadata=OPTIMIZER_METADATA[\"lr_decay\"]\n )\n\n weight_decay: float = schema_utils.FloatRange(\n default=0, description=\"Weight decay ($L2$ penalty).\", parameter_metadata=OPTIMIZER_METADATA[\"weight_decay\"]\n )\n\n eps: float = schema_utils.FloatRange(\n default=1e-10,\n description=\"Term added to the denominator to improve numerical stability.\",\n parameter_metadata=OPTIMIZER_METADATA[\"eps\"],\n )\n\n\nif bnb is not None:\n\n @DeveloperAPI\n @register_optimizer(name=\"adagrad_8bit\")\n @ludwig_dataclass\n class Adagrad8BitOptimizerConfig(AdagradOptimizerConfig):\n optimizer_class: ClassVar[torch.optim.Optimizer] = bnb.optim.Adagrad8bit\n\n type: str = schema_utils.ProtectedString(\"adagrad_8bit\")\n\n block_wise: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use block wise update.\",\n )\n\n percentile_clipping: int = schema_utils.IntegerRange(\n default=100,\n min=0,\n max=100,\n description=\"Percentile clipping.\",\n )\n\n @property\n def is_8bit(self) -> bool:\n return True\n\n\n@DeveloperAPI\n@register_optimizer(name=\"adamax\")\n@ludwig_dataclass\nclass AdamaxOptimizerConfig(BaseOptimizerConfig):\n \"\"\"Parameters for adamax optimization.\"\"\"\n\n optimizer_class: ClassVar[torch.optim.Optimizer] = torch.optim.Adamax\n \"\"\"Points to `torch.optim.Adamax`.\"\"\"\n\n type: str = schema_utils.ProtectedString(\"adamax\")\n \"\"\"Must be 'adamax' - corresponds to name in `ludwig.modules.optimization_modules.optimizer_registry`\n (default: 'adamax')\"\"\"\n\n # Defaults taken from https://pytorch.org/docs/stable/generated/torch.optim.Adamax.html#torch.optim.Adamax :\n betas: tuple[float, float] = schema_utils.FloatRangeTupleDataclassField(\n default=(0.9, 0.999),\n description=\"Coefficients used for computing running averages of gradient and its square.\",\n parameter_metadata=OPTIMIZER_METADATA[\"betas\"],\n )\n\n eps: float = schema_utils.NonNegativeFloat(\n default=1e-08,\n description=\"Term added to the denominator to improve numerical stability.\",\n parameter_metadata=OPTIMIZER_METADATA[\"eps\"],\n )\n\n weight_decay: float = schema_utils.NonNegativeFloat(\n default=0.0, description=\"Weight decay ($L2$ penalty).\", parameter_metadata=OPTIMIZER_METADATA[\"weight_decay\"]\n )\n\n\n# NOTE: keep ftrl and nadam optimizers out of registry:\n# @register_optimizer(name=\"ftrl\")\n@DeveloperAPI\n@ludwig_dataclass\nclass FtrlOptimizerConfig(BaseOptimizerConfig):\n # optimizer_class: ClassVar[torch.optim.Optimizer] = torch.optim.Ftrl\n type: str = schema_utils.ProtectedString(\"ftrl\")\n\n learning_rate_power: float = schema_utils.FloatRange(\n default=-0.5, max=0, parameter_metadata=OPTIMIZER_METADATA[\"learning_rate_power\"]\n )\n\n initial_accumulator_value: float = schema_utils.NonNegativeFloat(\n default=0.1, parameter_metadata=OPTIMIZER_METADATA[\"initial_accumulator_value\"]\n )\n\n l1_regularization_strength: float = schema_utils.NonNegativeFloat(\n default=0.0, parameter_metadata=OPTIMIZER_METADATA[\"l1_regularization_strength\"]\n )\n\n l2_regularization_strength: float = schema_utils.NonNegativeFloat(\n default=0.0, parameter_metadata=OPTIMIZER_METADATA[\"l2_regularization_strength\"]\n )\n\n\n@DeveloperAPI\n@register_optimizer(name=\"nadam\")\n@ludwig_dataclass\nclass NadamOptimizerConfig(BaseOptimizerConfig):\n optimizer_class: ClassVar[torch.optim.Optimizer] = torch.optim.NAdam\n \"\"\"Points to `torch.optim.NAdam`.\"\"\"\n\n type: str = schema_utils.ProtectedString(\"nadam\")\n\n # Defaults taken from https://pytorch.org/docs/stable/generated/torch.optim.NAdam.html#torch.optim.NAdam :\n\n betas: tuple[float, float] = schema_utils.FloatRangeTupleDataclassField(\n default=(0.9, 0.999),\n description=\"Coefficients used for computing running averages of gradient and its square.\",\n parameter_metadata=OPTIMIZER_METADATA[\"betas\"],\n )\n\n eps: float = schema_utils.NonNegativeFloat(\n default=1e-08,\n description=\"Term added to the denominator to improve numerical stability.\",\n parameter_metadata=OPTIMIZER_METADATA[\"eps\"],\n )\n\n weight_decay: float = schema_utils.NonNegativeFloat(\n default=0.0, description=\"Weight decay ($L2$ penalty).\", parameter_metadata=OPTIMIZER_METADATA[\"weight_decay\"]\n )\n\n momentum_decay: float = schema_utils.NonNegativeFloat(\n default=4e-3, description=\"Momentum decay.\", parameter_metadata=OPTIMIZER_METADATA[\"momentum_decay\"]\n )\n\n\n@DeveloperAPI\n@register_optimizer(name=\"rmsprop\")\n@ludwig_dataclass\nclass RMSPropOptimizerConfig(BaseOptimizerConfig):\n \"\"\"Parameters for rmsprop optimization.\"\"\"\n\n optimizer_class: ClassVar[torch.optim.Optimizer] = torch.optim.RMSprop\n \"\"\"Points to `torch.optim.RMSprop`.\"\"\"\n\n type: str = schema_utils.ProtectedString(\"rmsprop\")\n \"\"\"Must be 'rmsprop' - corresponds to name in `ludwig.modules.optimization_modules.optimizer_registry`\n (default: 'rmsprop')\"\"\"\n\n # Defaults taken from https://pytorch.org/docs/stable/generated/torch.optim.RMSprop.html#torch.optim.RMSprop:\n momentum: float = schema_utils.NonNegativeFloat(\n default=0.0,\n description=\"Momentum factor.\",\n parameter_metadata=OPTIMIZER_METADATA[\"momentum\"],\n )\n\n alpha: float = schema_utils.NonNegativeFloat(\n default=0.99,\n description=\"Smoothing constant.\",\n parameter_metadata=OPTIMIZER_METADATA[\"alpha\"],\n )\n\n eps: float = schema_utils.NonNegativeFloat(\n default=1e-08,\n description=\"Term added to the denominator to improve numerical stability.\",\n parameter_metadata=OPTIMIZER_METADATA[\"eps\"],\n )\n\n centered: bool = schema_utils.Boolean(\n default=False,\n description=\"If True, computes the centered RMSProp, and the gradient is normalized by an estimation of its \"\n \"variance.\",\n parameter_metadata=OPTIMIZER_METADATA[\"centered\"],\n )\n\n weight_decay: float = schema_utils.NonNegativeFloat(default=0.0, description=\"Weight decay ($L2$ penalty).\")\n\n\nif bnb is not None:\n\n @DeveloperAPI\n @register_optimizer(name=\"rmsprop_8bit\")\n @ludwig_dataclass\n class RMSProp8BitOptimizerConfig(RMSPropOptimizerConfig):\n optimizer_class: ClassVar[torch.optim.Optimizer] = bnb.optim.RMSprop8bit\n\n type: str = schema_utils.ProtectedString(\"rmsprop_8bit\")\n\n block_wise: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use block wise update.\",\n )\n\n percentile_clipping: int = schema_utils.IntegerRange(\n default=100,\n min=0,\n max=100,\n description=\"Percentile clipping.\",\n )\n\n @property\n def is_8bit(self) -> bool:\n return True\n\n\nif bnb is not None:\n\n @DeveloperAPI\n @register_optimizer(name=\"lamb\")\n @ludwig_dataclass\n class LAMBOptimizerConfig(BaseOptimizerConfig):\n \"\"\"Layer-wise Adaptive Moments optimizer for Batch training.\n\n Paper: https://arxiv.org/pdf/1904.00962.pdf\n \"\"\"\n\n optimizer_class: ClassVar[torch.optim.Optimizer] = bnb.optim.LAMB\n\n type: str = schema_utils.ProtectedString(\"lamb\")\n\n bias_correction: bool = schema_utils.Boolean(\n default=True,\n )\n\n betas: tuple[float, float] = schema_utils.FloatRangeTupleDataclassField(\n default=(0.9, 0.999),\n description=\"Coefficients used for computing running averages of gradient and its square.\",\n parameter_metadata=OPTIMIZER_METADATA[\"betas\"],\n )\n\n eps: float = schema_utils.NonNegativeFloat(\n default=1e-08,\n description=\"Term added to the denominator to improve numerical stability.\",\n parameter_metadata=OPTIMIZER_METADATA[\"eps\"],\n )\n\n weight_decay: float = schema_utils.NonNegativeFloat(\n default=0.0,\n description=\"Weight decay (L2 penalty).\",\n parameter_metadata=OPTIMIZER_METADATA[\"weight_decay\"],\n )\n\n amsgrad: bool = schema_utils.Boolean(\n default=False,\n description=(\n \"Whether to use the AMSGrad variant of this algorithm from the paper \"\n \"'On the Convergence of Adam and Beyond'.\"\n ),\n parameter_metadata=OPTIMIZER_METADATA[\"amsgrad\"],\n )\n\n adam_w_mode: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use the AdamW mode of this algorithm from the paper \"\n \"'Decoupled Weight Decay Regularization'.\",\n )\n\n percentile_clipping: int = schema_utils.IntegerRange(\n default=100,\n min=0,\n max=100,\n description=\"Percentile clipping.\",\n )\n\n block_wise: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to use block wise update.\",\n )\n\n max_unorm: float = schema_utils.FloatRange(\n default=1.0,\n min=0.0,\n max=1.0,\n )\n\n @DeveloperAPI\n @register_optimizer(name=\"lamb_8bit\")\n @ludwig_dataclass\n class LAMB8BitOptimizerConfig(LAMBOptimizerConfig):\n optimizer_class: ClassVar[torch.optim.Optimizer] = bnb.optim.LAMB8bit\n\n type: str = schema_utils.ProtectedString(\"lamb_8bit\")\n\n @property\n def is_8bit(self) -> bool:\n return True\n\n\nif bnb is not None:\n\n @DeveloperAPI\n @register_optimizer(name=\"lars\")\n @ludwig_dataclass\n class LARSOptimizerConfig(BaseOptimizerConfig):\n \"\"\"Layerwise Adaptive Rate Scaling.\n\n Paper: https://arxiv.org/pdf/1708.03888.pdf\n \"\"\"\n\n optimizer_class: ClassVar[torch.optim.Optimizer] = bnb.optim.LARS\n\n type: str = schema_utils.ProtectedString(\"lars\")\n\n # 0.9 taken from the original paper - momentum requires a non zero value\n # https://arxiv.org/pdf/1708.03888v3.pdf\n momentum: float = schema_utils.FloatRange(\n default=0.9,\n min=0.0,\n max=1.0,\n min_inclusive=False,\n description=\"Momentum factor.\",\n parameter_metadata=OPTIMIZER_METADATA[\"momentum\"],\n )\n\n dampening: float = schema_utils.FloatRange(\n default=0.0,\n min=0.0,\n max=1.0,\n description=\"Dampening for momentum.\",\n parameter_metadata=OPTIMIZER_METADATA[\"dampening\"],\n )\n\n weight_decay: float = schema_utils.NonNegativeFloat(\n default=0.0,\n description=\"Weight decay (L2 penalty).\",\n parameter_metadata=OPTIMIZER_METADATA[\"weight_decay\"],\n )\n\n nesterov: bool = schema_utils.Boolean(\n default=False,\n description=\"Enables Nesterov momentum.\",\n parameter_metadata=OPTIMIZER_METADATA[\"nesterov\"],\n )\n\n percentile_clipping: int = schema_utils.IntegerRange(\n default=100,\n min=0,\n max=100,\n description=\"Percentile clipping.\",\n )\n\n max_unorm: float = schema_utils.FloatRange(\n default=1.0,\n min=0.0,\n max=1.0,\n )\n\n @DeveloperAPI\n @register_optimizer(name=\"lars_8bit\")\n @ludwig_dataclass\n class LARS8BitOptimizerConfig(LARSOptimizerConfig):\n optimizer_class: ClassVar[torch.optim.Optimizer] = bnb.optim.LARS8bit\n\n type: str = schema_utils.ProtectedString(\"lars_8bit\")\n\n @property\n def is_8bit(self) -> bool:\n return True\n\n\nif bnb is not None:\n\n @DeveloperAPI\n @register_optimizer(name=\"lion\")\n @ludwig_dataclass\n class LIONOptimizerConfig(BaseOptimizerConfig):\n \"\"\"Evolved Sign Momentum.\n\n Paper: https://arxiv.org/pdf/2302.06675.pdf\n \"\"\"\n\n optimizer_class: ClassVar[torch.optim.Optimizer] = bnb.optim.Lion\n\n type: str = schema_utils.ProtectedString(\"lion\")\n\n betas: tuple[float, float] = schema_utils.FloatRangeTupleDataclassField(\n default=(0.9, 0.999),\n description=\"Coefficients used for computing running averages of gradient and its square.\",\n parameter_metadata=OPTIMIZER_METADATA[\"betas\"],\n )\n\n weight_decay: float = schema_utils.NonNegativeFloat(\n default=0.0,\n description=\"Weight decay (L2 penalty).\",\n parameter_metadata=OPTIMIZER_METADATA[\"weight_decay\"],\n )\n\n percentile_clipping: int = schema_utils.IntegerRange(\n default=100,\n min=0,\n max=100,\n description=\"Percentile clipping.\",\n )\n\n block_wise: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to use block wise update.\",\n )\n\n @DeveloperAPI\n @register_optimizer(name=\"lion_8bit\")\n @ludwig_dataclass\n class LION8BitOptimizerConfig(LIONOptimizerConfig):\n optimizer_class: ClassVar[torch.optim.Optimizer] = bnb.optim.Lion8bit\n\n type: str = schema_utils.ProtectedString(\"lion_8bit\")\n\n @property\n def is_8bit(self) -> bool:\n return True\n\n @DeveloperAPI\n @register_optimizer(name=\"paged_lion\")\n @ludwig_dataclass\n class PagedLionOptimizerConfig(LIONOptimizerConfig):\n optimizer_class: ClassVar[torch.optim.Optimizer] = bnb.optim.PagedLion\n\n type: str = schema_utils.ProtectedString(\"paged_lion\")\n\n @property\n def is_paged(self) -> bool:\n return True\n\n @DeveloperAPI\n @register_optimizer(name=\"paged_lion_8bit\")\n @ludwig_dataclass\n class PagedLion8BitOptimizerConfig(PagedLionOptimizerConfig):\n optimizer_class: ClassVar[torch.optim.Optimizer] = bnb.optim.PagedLion8bit\n\n type: str = schema_utils.ProtectedString(\"paged_lion_8bit\")\n\n @property\n def is_8bit(self) -> bool:\n return True\n\n\n@DeveloperAPI\ndef get_optimizer_conds():\n \"\"\"Returns a JSON schema of conditionals to validate against optimizer types defined in\n `ludwig.modules.optimization_modules.optimizer_registry`.\"\"\"\n conds = []\n for optimizer in optimizer_registry:\n optimizer_cls = optimizer_registry[optimizer][1]\n other_props = schema_utils.unload_jsonschema_from_marshmallow_class(optimizer_cls)[\"properties\"]\n schema_utils.remove_duplicate_fields(other_props)\n preproc_cond = schema_utils.create_cond(\n {\"type\": optimizer},\n other_props,\n )\n conds.append(preproc_cond)\n return conds\n\n\n@DeveloperAPI\ndef OptimizerDataclassField(default=\"adam\", description=\"\", parameter_metadata: ParameterMetadata = None):\n \"\"\"Custom dataclass field that when used inside of a dataclass will allow any optimizer in\n `ludwig.modules.optimization_modules.optimizer_registry`.\n\n Sets default optimizer to 'adam'.\n\n :param default: Dict specifying an optimizer with a `type` field and its associated parameters. Will attempt to use\n `type` to load optimizer from registry with given params. (default: {\"type\": \"adam\"}).\n :return: Initialized dataclass field that converts untyped dicts with params to optimizer dataclass instances.\n \"\"\"\n\n class OptimizerSelection(schema_utils.TypeSelection):\n \"\"\"Custom marshmallow field that deserializes a dict to a valid optimizer from\n `ludwig.modules.optimization_modules.optimizer_registry` and creates a corresponding `oneOf` JSON schema\n for external usage.\"\"\"\n\n def __init__(self):\n super().__init__(\n registry=optimizer_registry,\n default_value=default,\n description=description,\n parameter_metadata=parameter_metadata,\n )\n\n def get_schema_from_registry(self, key: str) -> type[schema_utils.BaseMarshmallowConfig]:\n return get_optimizer_cls(key)\n\n def _jsonschema_type_mapping(self):\n # Note that this uses the same conditional pattern as combiners:\n return {\n \"type\": \"object\",\n \"properties\": {\n \"type\": {\n \"type\": \"string\",\n \"enum\": list(optimizer_registry.keys()),\n \"default\": default,\n \"description\": \"The type of optimizer to use during the learning process\",\n },\n },\n \"title\": \"optimizer_options\",\n \"allOf\": get_optimizer_conds(),\n \"required\": [\"type\"],\n \"description\": description,\n }\n\n return OptimizerSelection().get_default_field()\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass GradientClippingConfig(schema_utils.BaseMarshmallowConfig):\n \"\"\"Dataclass that holds gradient clipping parameters.\"\"\"\n\n clipglobalnorm: float | None = schema_utils.FloatRange(\n default=0.5,\n allow_none=True,\n description=\"Maximum allowed norm of the gradients\",\n parameter_metadata=OPTIMIZER_METADATA[\"gradient_clipping\"],\n )\n\n # TODO(travis): is this redundant with `clipglobalnorm`?\n clipnorm: float | None = schema_utils.FloatRange(\n default=None,\n allow_none=True,\n description=\"Maximum allowed norm of the gradients\",\n parameter_metadata=OPTIMIZER_METADATA[\"gradient_clipping\"],\n )\n\n clipvalue: float | None = schema_utils.FloatRange(\n default=None,\n allow_none=True,\n description=\"Maximum allowed value of the gradients\",\n parameter_metadata=OPTIMIZER_METADATA[\"gradient_clipping\"],\n )\n\n\n@DeveloperAPI\ndef GradientClippingDataclassField(description: str, default: dict = {}):\n \"\"\"Returns custom dataclass field for `ludwig.modules.optimization_modules.GradientClippingConfig`. Allows\n `None` by default.\n\n :param description: Description of the gradient dataclass field\n :param default: dict that specifies clipping param values that will be loaded by its schema class (default: {}).\n \"\"\"\n allow_none = True\n\n class GradientClippingMarshmallowField(schema_utils.LudwigSchemaField):\n \"\"\"Custom field class for gradient clipping.\n\n Deserializes a dict to a valid instance of `ludwig.modules.optimization_modules.GradientClippingConfig` and\n creates a corresponding JSON schema for external usage.\n \"\"\"\n\n def _deserialize(self, value, attr, data, **kwargs):\n if value is None:\n return value\n if isinstance(value, dict):\n try:\n return GradientClippingConfig.Schema().load(value)\n except (TypeError, ConfigValidationError):\n raise ConfigValidationError(\n f\"Invalid params for gradient clipping: {value}, see GradientClippingConfig class.\"\n )\n raise ConfigValidationError(\"Field should be None or dict\")\n\n def _jsonschema_type_mapping(self):\n return {\n \"oneOf\": [\n {\"type\": \"null\", \"title\": \"disabled\", \"description\": \"Disable gradient clipping.\"},\n {\n **schema_utils.unload_jsonschema_from_marshmallow_class(GradientClippingConfig),\n \"title\": \"enabled_options\",\n },\n ],\n \"title\": \"gradient_clipping_options\",\n \"description\": description,\n }\n\n if not isinstance(default, dict):\n raise ConfigValidationError(f\"Invalid default: `{default}`\")\n\n def load_default():\n return GradientClippingConfig.Schema().load(default)\n\n dump_default = GradientClippingConfig.Schema().dump(default)\n\n return field(\n metadata={\n \"marshmallow_field\": GradientClippingMarshmallowField(\n allow_none=allow_none,\n load_default=load_default,\n dump_default=dump_default,\n metadata={\n \"description\": description,\n \"parameter_metadata\": convert_metadata_to_json(OPTIMIZER_METADATA[\"gradient_clipping\"]),\n },\n )\n },\n default_factory=load_default,\n )\n"
},
{
"path": "ludwig/schema/preprocessing.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import RANDOM\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.metadata import PREPROCESSING_METADATA\nfrom ludwig.schema.split import BaseSplitConfig, SplitDataclassField\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass PreprocessingConfig(schema_utils.BaseMarshmallowConfig):\n \"\"\"Global preprocessing config is a dataclass that configures the parameters used for global preprocessing.\"\"\"\n\n sample_ratio: float = schema_utils.NonNegativeFloat(\n default=1.0,\n description=\"The ratio of the dataset to use. For instance, if 0.5, half of the dataset \"\n \"provided will be used.\",\n parameter_metadata=PREPROCESSING_METADATA[\"sample_ratio\"],\n )\n\n sample_size: float = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"The maximum number of samples from the dataset to use. Cannot be set if sample_ratio is set to be \"\n \"< 1.0. If sample_ratio is set to 1.0, this will override the number of samples to used.\",\n parameter_metadata=PREPROCESSING_METADATA[\"sample_size\"],\n )\n\n oversample_minority: float = schema_utils.NonNegativeFloat(\n default=None,\n allow_none=True,\n description=\"If not None, the minority class will be oversampled to reach the specified ratio respective to \"\n \"the majority class. \",\n parameter_metadata=PREPROCESSING_METADATA[\"oversample_minority\"],\n )\n\n undersample_majority: float = schema_utils.NonNegativeFloat(\n default=None,\n allow_none=True,\n description=\"If not None, the majority class will be undersampled to reach the specified ratio respective \"\n \"to the minority class. \",\n parameter_metadata=PREPROCESSING_METADATA[\"undersample_majority\"],\n )\n\n split: BaseSplitConfig = SplitDataclassField(\n default=RANDOM,\n )\n\n global_max_sequence_length: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Specifically for LLMs. This is the maximum length of the input sequence going into the model's \"\n \"forward pass during training. Sequences will be truncated to this length after merging inputs and targets. \"\n \"If not set, the total length of the merged input and target token sequences will be used.\",\n parameter_metadata=PREPROCESSING_METADATA[\"global_max_sequence_length\"],\n )\n\n\n@DeveloperAPI\nclass PreprocessingField(schema_utils.DictMarshmallowField):\n def __init__(self):\n super().__init__(PreprocessingConfig)\n"
},
{
"path": "ludwig/schema/profiler.py",
"content": "from dataclasses import field\n\nimport ludwig.schema.utils as schema_utils\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.schema.utils import ludwig_dataclass\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass ProfilerConfig(schema_utils.BaseMarshmallowConfig):\n \"\"\"Dataclass that holds profiling parameters for torch profile scheduler.\n\n The profiler will skip the first skip_first steps, then wait for wait steps, then do the warmup for the next warmup\n steps, then do the active recording for the next active steps and then repeat the cycle starting with wait steps.\n The optional number of cycles is specified with the repeat parameter, the zero value means that the cycles will\n continue until the profiling is finished.\n \"\"\"\n\n wait: int = schema_utils.IntegerRange(\n default=1,\n min=0,\n description=\"The number of steps to wait profiling.\",\n )\n\n warmup: int = schema_utils.IntegerRange(\n default=1,\n min=0,\n description=\"The number of steps for profiler warmup after waiting finishes.\",\n )\n\n active: int = schema_utils.IntegerRange(\n default=3,\n min=0,\n description=\"The number of steps that are actively recorded. Values more than 10 wil dramatically slow down \"\n \"tensorboard loading.\",\n )\n\n repeat: int = schema_utils.IntegerRange(\n default=5,\n min=0,\n description=\"The optional number of profiling cycles. Use 0 to profile the entire training run.\",\n )\n\n skip_first: int = schema_utils.IntegerRange(\n default=0,\n min=0,\n max=100,\n description=\"The number of steps to skip in the beginning of training.\",\n )\n\n\n@DeveloperAPI\ndef ProfilerDataclassField(description: str, default: dict = {}):\n \"\"\"Returns custom dataclass field for `ludwig.modules.profiler.ProfilerConfig`. Allows `None` by default.\n\n :param description: Description of the torch profiler field\n :param default: dict that specifies clipping param values that will be loaded by its schema class (default: {}).\n \"\"\"\n allow_none = True\n\n class ProfilingMarshmallowField(schema_utils.LudwigSchemaField):\n \"\"\"Custom field class for the torch profiler.\n\n Deserializes a dict to a valid instance of `ludwig.modules.optimization_modules.ProfilerConfig` and\n creates a corresponding JSON schema for external usage.\n \"\"\"\n\n def _deserialize(self, value, attr, data, **kwargs):\n if value is None:\n return value\n if isinstance(value, dict):\n try:\n return ProfilerConfig.Schema().load(value)\n except (TypeError, ConfigValidationError):\n raise ConfigValidationError(\n f\"Invalid params for profiling config: {value}, see ProfilerConfig class.\"\n )\n raise ConfigValidationError(\"Field should be None or dict\")\n\n def _jsonschema_type_mapping(self):\n return {\n **schema_utils.unload_jsonschema_from_marshmallow_class(ProfilerConfig),\n \"title\": \"profiler_options\",\n \"description\": description,\n }\n\n if not isinstance(default, dict):\n raise ConfigValidationError(f\"Invalid default: `{default}`\")\n\n def load_default():\n return ProfilerConfig.Schema().load(default)\n\n dump_default = ProfilerConfig.Schema().dump(default)\n\n return field(\n metadata={\n \"marshmallow_field\": ProfilingMarshmallowField(\n allow_none=allow_none,\n load_default=load_default,\n dump_default=dump_default,\n metadata={\n \"description\": description,\n \"parameter_metadata\": None,\n },\n )\n },\n default_factory=load_default,\n )\n"
},
{
"path": "ludwig/schema/split.py",
"content": "from dataclasses import Field\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import SPLIT, TYPE\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.metadata import PREPROCESSING_METADATA\nfrom ludwig.schema.utils import ludwig_dataclass\nfrom ludwig.utils.registry import Registry\n\nsplit_config_registry = Registry()\nDEFAULT_PROBABILITIES = [0.7, 0.1, 0.2]\n\n\n@DeveloperAPI\ndef get_split_cls(name: str):\n return split_config_registry[name]\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass BaseSplitConfig(schema_utils.BaseMarshmallowConfig):\n \"\"\"This Dataclass is a base schema for the nested split config under preprocessing.\"\"\"\n\n type: str\n \"Name corresponding to the splitting type.\"\n\n\n@DeveloperAPI\n@split_config_registry.register(\"random\")\n@ludwig_dataclass\nclass RandomSplitConfig(BaseSplitConfig):\n \"\"\"This Dataclass generates a schema for the random splitting config.\"\"\"\n\n type: str = schema_utils.ProtectedString(\n \"random\",\n description=\"Type of splitting to use during preprocessing.\",\n )\n\n probabilities: list = schema_utils.List(\n list_type=float,\n default=DEFAULT_PROBABILITIES,\n description=\"Probabilities for splitting data into train, validation, and test sets.\",\n parameter_metadata=PREPROCESSING_METADATA[\"split_probabilities\"],\n )\n\n\n@DeveloperAPI\n@split_config_registry.register(\"fixed\")\n@ludwig_dataclass\nclass FixedSplitConfig(BaseSplitConfig):\n \"\"\"This Dataclass generates a schema for the fixed splitting config.\"\"\"\n\n type: str = schema_utils.ProtectedString(\n \"fixed\",\n description=\"Type of splitting to use during preprocessing.\",\n )\n\n column: str = schema_utils.String(\n default=SPLIT,\n allow_none=False,\n description=\"The column name to use for fixed splitting.\",\n parameter_metadata=PREPROCESSING_METADATA[\"column\"],\n )\n\n\n@DeveloperAPI\n@split_config_registry.register(\"stratify\")\n@ludwig_dataclass\nclass StratifySplitConfig(BaseSplitConfig):\n \"\"\"This Dataclass generates a schema for the fixed splitting config.\"\"\"\n\n type: str = schema_utils.ProtectedString(\n \"stratify\",\n description=\"Type of splitting to use during preprocessing.\",\n )\n\n column: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"The column name to base the stratified splitting on.\",\n parameter_metadata=PREPROCESSING_METADATA[\"column\"],\n )\n\n probabilities: list = schema_utils.List(\n list_type=float,\n default=DEFAULT_PROBABILITIES,\n description=\"Probabilities for splitting data into train, validation, and test sets.\",\n parameter_metadata=PREPROCESSING_METADATA[\"split_probabilities\"],\n )\n\n\n@DeveloperAPI\n@split_config_registry.register(\"datetime\")\n@ludwig_dataclass\nclass DateTimeSplitConfig(BaseSplitConfig):\n \"\"\"This Dataclass generates a schema for the fixed splitting config.\"\"\"\n\n type: str = schema_utils.ProtectedString(\n \"datetime\",\n description=\"Type of splitting to use during preprocessing.\",\n )\n\n column: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"The column name to perform datetime splitting on.\",\n parameter_metadata=PREPROCESSING_METADATA[\"column\"],\n )\n\n probabilities: list = schema_utils.List(\n list_type=float,\n default=DEFAULT_PROBABILITIES,\n description=\"Proportion of data to split into train, validation, and test sets.\",\n parameter_metadata=PREPROCESSING_METADATA[\"split_probabilities\"],\n )\n\n\n@DeveloperAPI\n@split_config_registry.register(\"hash\")\n@ludwig_dataclass\nclass HashSplitConfig(BaseSplitConfig):\n \"\"\"This Dataclass generates a schema for the hash splitting config.\n\n This is useful for deterministically splitting on a unique ID. Even when additional rows are added to the dataset in\n the future, each ID will retain its original split assignment.\n\n This approach does not guarantee that the split proportions will be assigned exactly, but the larger the dataset,\n the more closely the assignment should match the given proportions.\n\n This approach can be used on a column with duplicates, but it will further skew the assignments of rows to splits.\n \"\"\"\n\n type: str = schema_utils.ProtectedString(\n \"hash\",\n description=\"Type of splitting to use during preprocessing.\",\n )\n\n column: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"The column name to perform hash splitting on.\",\n parameter_metadata=PREPROCESSING_METADATA[\"column\"],\n )\n\n probabilities: list = schema_utils.List(\n list_type=float,\n default=DEFAULT_PROBABILITIES,\n description=\"Proportion of data to split into train, validation, and test sets.\",\n parameter_metadata=PREPROCESSING_METADATA[\"split_probabilities\"],\n )\n\n\n@DeveloperAPI\ndef get_split_conds():\n \"\"\"Returns a JSON schema of conditionals to validate against optimizer types defined in\n `ludwig.modules.optimization_modules.optimizer_registry`.\"\"\"\n conds = []\n for splitter in split_config_registry.data:\n splitter_cls = split_config_registry.data[splitter]\n other_props = schema_utils.unload_jsonschema_from_marshmallow_class(splitter_cls)[\"properties\"]\n schema_utils.remove_duplicate_fields(other_props, [TYPE])\n splitter_cond = schema_utils.create_cond(\n {\"type\": splitter},\n other_props,\n )\n conds.append(splitter_cond)\n return conds\n\n\n@DeveloperAPI\ndef SplitDataclassField(default: str) -> Field:\n \"\"\"Custom dataclass field that when used inside a dataclass will allow the user to specify a nested split\n config.\n\n Returns: Initialized dataclass field that converts an untyped dict with params to a split config.\n \"\"\"\n\n class SplitSelection(schema_utils.TypeSelection):\n def __init__(self):\n super().__init__(registry=split_config_registry.data, default_value=default)\n\n def get_schema_from_registry(self, key: str) -> type[schema_utils.BaseMarshmallowConfig]:\n return split_config_registry.data[key]\n\n def _jsonschema_type_mapping(self):\n return {\n \"type\": \"object\",\n \"properties\": {\n \"type\": {\n \"type\": \"string\",\n \"description\": \"Type of splitting to use during preprocessing.\",\n \"enum\": list(split_config_registry.data.keys()),\n \"default\": default,\n },\n },\n \"title\": \"split_options\",\n \"allOf\": get_split_conds(),\n }\n\n return SplitSelection().get_default_field()\n"
},
{
"path": "ludwig/schema/trainer.py",
"content": "import re\nfrom abc import ABC\n\nimport torch\nfrom packaging.version import parse as parse_version\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import (\n AUTO,\n EFFECTIVE_BATCH_SIZE,\n LOSS,\n MAX_BATCH_SIZE,\n MAX_POSSIBLE_BATCH_SIZE,\n MODEL_ECD,\n MODEL_LLM,\n TRAINING,\n)\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.schema import utils as schema_utils\nfrom ludwig.schema.lr_scheduler import LRSchedulerConfig, LRSchedulerDataclassField\nfrom ludwig.schema.metadata import TRAINER_METADATA\nfrom ludwig.schema.optimizers import (\n BaseOptimizerConfig,\n GradientClippingConfig,\n GradientClippingDataclassField,\n OptimizerDataclassField,\n)\nfrom ludwig.schema.profiler import ProfilerConfig, ProfilerDataclassField\nfrom ludwig.schema.utils import ludwig_dataclass\nfrom ludwig.utils.registry import Registry\n\n_torch_200 = parse_version(torch.__version__) >= parse_version(\"2.0\")\n\n\ntrainer_schema_registry = Registry()\n_llm_trainer_schema_registry = Registry()\n\n\n@DeveloperAPI\ndef register_trainer_schema(model_type: str):\n def wrap(trainer_config: BaseTrainerConfig):\n trainer_schema_registry[model_type] = trainer_config\n return trainer_config\n\n return wrap\n\n\n@DeveloperAPI\ndef register_llm_trainer_schema(trainer_type: str):\n def wrap(trainer_config: BaseTrainerConfig):\n _llm_trainer_schema_registry[trainer_type] = trainer_config\n return trainer_config\n\n return wrap\n\n\n@DeveloperAPI\ndef get_llm_trainer_cls(trainer_type: str):\n \"\"\"Returns the adapter config class registered with the given name.\"\"\"\n return _llm_trainer_schema_registry[trainer_type]\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass BaseTrainerConfig(schema_utils.BaseMarshmallowConfig, ABC):\n \"\"\"Common trainer parameter values.\"\"\"\n\n validation_field: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"The field for which the `validation_metric` is used for validation-related mechanics like early \"\n \"stopping, parameter change plateaus, as well as what hyperparameter optimization uses to determine the best \"\n \"trial. If unset (default), the first output feature is used. If explicitly specified, neither \"\n \"`validation_field` nor `validation_metric` are overwritten.\",\n )\n\n validation_metric: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=(\n \"Metric from `validation_field` that is used. If validation_field is not explicitly specified, this is \"\n \"overwritten to be the first output feature type's `default_validation_metric`, consistent with \"\n \"validation_field. If the validation_metric is specified, then we will use the first output feature that \"\n \"produces this metric as the `validation_field`.\"\n ),\n )\n\n early_stop: int = schema_utils.IntegerRange(\n default=5,\n min=-1,\n description=(\n \"Number of consecutive rounds of evaluation without any improvement on the `validation_metric` that \"\n \"triggers training to stop. Can be set to -1, which disables early stopping entirely.\"\n ),\n )\n\n skip_all_evaluation: bool = schema_utils.Boolean(\n default=False,\n description=(\n \"Whether to skip evaluation entirely. If you are training a model with a well-known configuration on a \"\n \"well-known dataset and are confident about the expected results, you might skip all evaluation. Moreover, \"\n \"evaluating a model, especially on large validation or test sets, can be time-consuming.\"\n ),\n )\n\n enable_profiling: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to enable profiling of the training process using torch.profiler.profile.\",\n )\n\n profiler: ProfilerConfig | None = ProfilerDataclassField(\n description=\"Parameter values for profiling config.\",\n default={},\n )\n\n def can_tune_batch_size(self) -> bool:\n return True\n\n\n@DeveloperAPI\n@register_trainer_schema(MODEL_ECD)\n@ludwig_dataclass\nclass ECDTrainerConfig(BaseTrainerConfig):\n \"\"\"Dataclass that configures most of the hyperparameters used for ECD model training.\"\"\"\n\n def __post_init__(self):\n if self.compile and not _torch_200:\n raise ConfigValidationError(\n \"Trainer param `compile: true` requires PyTorch 2.0.0 or higher. Please upgrade PyTorch and try again.\"\n )\n\n if self.effective_batch_size != AUTO and self.max_batch_size < self.effective_batch_size:\n raise ConfigValidationError(\n f\"`max_batch_size` ({self.max_batch_size}) must be greater than or equal to \"\n f\"`effective_batch_size` ({self.effective_batch_size}).\"\n )\n\n if self.effective_batch_size != AUTO and self.batch_size != AUTO:\n if self.effective_batch_size < self.batch_size:\n raise ConfigValidationError(\n f\"`effective_batch_size` ({self.effective_batch_size}) \"\n f\"must be greater than or equal to `batch_size` ({self.batch_size}).\"\n )\n\n if self.effective_batch_size % self.batch_size != 0:\n raise ConfigValidationError(\n f\"`effective_batch_size` ({self.effective_batch_size}) \"\n f\"must be divisible by `batch_size` ({self.batch_size}).\"\n )\n\n if self.effective_batch_size != AUTO and self.gradient_accumulation_steps != AUTO:\n if self.effective_batch_size < self.gradient_accumulation_steps:\n raise ConfigValidationError(\n f\"`effective_batch_size` ({self.effective_batch_size}) must be greater than or equal to \"\n f\"`gradient_accumulation_steps` ({self.gradient_accumulation_steps}).\"\n )\n\n if self.effective_batch_size % self.gradient_accumulation_steps != 0:\n raise ConfigValidationError(\n f\"`effective_batch_size` ({self.effective_batch_size}) must be divisible by \"\n f\"`gradient_accumulation_steps` ({self.gradient_accumulation_steps}).\"\n )\n\n if self.layers_to_freeze_regex:\n try:\n re.compile(self.layers_to_freeze_regex)\n except re.error:\n raise ConfigValidationError(\n f\"`layers_to_freeze_regex` ({self.layers_to_freeze_regex}) must be a valid regular expression.\"\n )\n\n learning_rate: float | str = schema_utils.OneOfOptionsField(\n default=0.001,\n allow_none=False,\n description=(\n \"Controls how much to change the model in response to the estimated error each time the model weights are \"\n \"updated. If 'auto', the optimal learning rate is estimated by choosing the learning rate that produces \"\n \"the smallest non-diverging gradient update.\"\n ),\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"learning_rate\"],\n field_options=[\n schema_utils.FloatRange(default=0.001, allow_none=False, min=0, max=1),\n schema_utils.StringOptions(options=[\"auto\"], default=\"auto\", allow_none=False),\n ],\n )\n\n learning_rate_scheduler: LRSchedulerConfig = LRSchedulerDataclassField(\n description=\"Parameter values for learning rate scheduler.\",\n default=None,\n )\n\n epochs: int = schema_utils.PositiveInteger(\n default=100,\n description=\"Number of epochs the algorithm is intended to be run over. Overridden if `train_steps` is set\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"epochs\"],\n )\n\n checkpoints_per_epoch: int = schema_utils.NonNegativeInteger(\n default=0,\n description=(\n \"Number of checkpoints per epoch. For example, 2 -> checkpoints are written every half of an epoch. Note \"\n \"that it is invalid to specify both non-zero `steps_per_checkpoint` and non-zero `checkpoints_per_epoch`.\"\n ),\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"checkpoints_per_epoch\"],\n )\n\n train_steps: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=(\n \"Maximum number of training steps the algorithm is intended to be run over. Unset by default. \"\n \"If set, will override `epochs` and if left unset then `epochs` is used to determine training length.\"\n ),\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"train_steps\"],\n )\n\n eval_steps: float = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"The number of steps to use for evaluation. If None, the entire evaluation set will be used.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"eval_steps\"],\n )\n\n steps_per_checkpoint: int = schema_utils.NonNegativeInteger(\n default=0,\n description=(\n \"How often the model is checkpointed. Also dictates maximum evaluation frequency. If 0 the model is \"\n \"checkpointed after every epoch.\"\n ),\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"steps_per_checkpoint\"],\n )\n\n effective_batch_size: int | str = schema_utils.OneOfOptionsField(\n default=AUTO,\n allow_none=False,\n description=(\n \"The effective batch size is the total number of samples used to compute a single gradient update \"\n \"to the model weights. This differs from `batch_size` by taking `gradient_accumulation_steps` and number \"\n \"of training worker processes into account. In practice, \"\n \"`effective_batch_size = batch_size * gradient_accumulation_steps * num_workers`. \"\n \"If 'auto', the effective batch size is derivied implicitly from `batch_size`, but if set explicitly, then \"\n \"one of `batch_size` or `gradient_accumulation_steps` must be set to something other than 'auto', and \"\n \"consequently will be set following the formula given above.\"\n ),\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][EFFECTIVE_BATCH_SIZE],\n field_options=[\n schema_utils.PositiveInteger(default=128, description=\"\", allow_none=False),\n schema_utils.StringOptions(options=[\"auto\"], default=\"auto\", allow_none=False),\n ],\n )\n\n batch_size: int | str = schema_utils.OneOfOptionsField(\n default=AUTO,\n allow_none=False,\n description=(\n \"The number of training examples utilized in one training step of the model. If ’auto’, the \"\n \"batch size that maximized training throughput (samples / sec) will be used. For CPU training, the \"\n \"tuned batch size is capped at 128 as throughput benefits of large batch sizes are less noticeable without \"\n \"a GPU.\"\n ),\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"batch_size\"],\n field_options=[\n schema_utils.PositiveInteger(default=128, description=\"\", allow_none=False),\n schema_utils.StringOptions(options=[\"auto\"], default=\"auto\", allow_none=False),\n ],\n )\n\n max_batch_size: int = schema_utils.PositiveInteger(\n default=MAX_POSSIBLE_BATCH_SIZE,\n allow_none=True,\n description=(\n \"Auto batch size tuning and increasing batch size on plateau will be capped at this value. The default \"\n \"value is 2^40.\"\n ),\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][MAX_BATCH_SIZE],\n )\n\n gradient_accumulation_steps: int | str = schema_utils.OneOfOptionsField(\n default=AUTO,\n allow_none=False,\n description=\"Number of steps to accumulate gradients over before performing a weight update.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"gradient_accumulation_steps\"],\n field_options=[\n schema_utils.PositiveInteger(default=1, description=\"\", allow_none=False),\n schema_utils.StringOptions(options=[\"auto\"], default=\"auto\", allow_none=False),\n ],\n )\n\n early_stop: int = schema_utils.IntegerRange(\n default=5,\n min=-1,\n description=(\n \"Number of consecutive rounds of evaluation without any improvement on the `validation_metric` that \"\n \"triggers training to stop. Can be set to -1, which disables early stopping entirely.\"\n ),\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"early_stop\"],\n )\n\n eval_batch_size: None | int | str = schema_utils.OneOfOptionsField(\n default=None,\n allow_none=True,\n description=(\n \"Size of batch to pass to the model for evaluation. If it is `0` or `None`, the same value of `batch_size` \"\n \"is used. This is useful to speedup evaluation with a much bigger batch size than training, if enough \"\n \"memory is available. If ’auto’, the biggest batch size (power of 2) that can fit in memory will be used.\"\n ),\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"eval_batch_size\"],\n field_options=[\n schema_utils.PositiveInteger(default=128, description=\"\", allow_none=False),\n schema_utils.StringOptions(options=[\"auto\"], default=\"auto\", allow_none=False),\n ],\n )\n\n evaluate_training_set: bool = schema_utils.Boolean(\n default=False,\n description=(\n \"Whether to evaluate on the entire training set during evaluation. By default, training metrics will be \"\n \"computed at the end of each training step, and accumulated up to the evaluation phase. In practice, \"\n \"computing training set metrics during training is up to 30% faster than running a separate evaluation \"\n \"pass over the training set, but results in more noisy training metrics, particularly during the earlier \"\n \"epochs. It's recommended to only set this to True if you need very exact training set metrics, and are \"\n \"willing to pay a significant performance penalty for them.\"\n ),\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"evaluate_training_set\"],\n )\n\n validation_field: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"The field for which the `validation_metric` is used for validation-related mechanics like early \"\n \"stopping, parameter change plateaus, as well as what hyperparameter optimization uses to determine the best \"\n \"trial. If unset (default), the first output feature is used. If explicitly specified, neither \"\n \"`validation_field` nor `validation_metric` are overwritten.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"validation_field\"],\n )\n\n validation_metric: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=(\n \"Metric from `validation_field` that is used. If validation_field is not explicitly specified, this is \"\n \"overwritten to be the first output feature type's `default_validation_metric`, consistent with \"\n \"validation_field. If the validation_metric is specified, then we will use the first output feature that \"\n \"produces this metric as the `validation_field`.\"\n ),\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"validation_metric\"],\n )\n\n optimizer: BaseOptimizerConfig = OptimizerDataclassField(\n default=\"adam\",\n description=(\n \"Optimizer type and its parameters. The optimizer is responsble for applying the gradients computed \"\n \"from the loss during backpropagation as updates to the model weights.\"\n ),\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"optimizer\"],\n )\n\n regularization_type: str | None = schema_utils.RegularizerOptions(\n default=\"l2\",\n allow_none=True,\n description=\"Type of regularization.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"regularization_type\"],\n )\n\n regularization_lambda: float = schema_utils.FloatRange(\n default=0.0,\n min=0,\n max=1,\n description=\"Strength of the regularization.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"regularization_lambda\"],\n )\n\n should_shuffle: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to shuffle batches during training when true.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"should_shuffle\"],\n )\n\n increase_batch_size_on_plateau: int = schema_utils.NonNegativeInteger(\n default=0,\n description=\"The number of times to increase the batch size on a plateau.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"increase_batch_size_on_plateau\"],\n )\n\n increase_batch_size_on_plateau_patience: int = schema_utils.NonNegativeInteger(\n default=5,\n description=\"How many epochs to wait for before increasing the batch size.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"increase_batch_size_on_plateau_patience\"],\n )\n\n increase_batch_size_on_plateau_rate: float = schema_utils.NonNegativeFloat(\n default=2.0,\n description=\"Rate at which the batch size increases.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"increase_batch_size_on_plateau_rate\"],\n )\n\n increase_batch_size_eval_metric: str = schema_utils.String(\n default=LOSS,\n description=\"Which metric to listen on for increasing the batch size.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"increase_batch_size_eval_metric\"],\n )\n\n increase_batch_size_eval_split: str = schema_utils.String(\n default=TRAINING,\n description=\"Which dataset split to listen on for increasing the batch size.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"increase_batch_size_eval_split\"],\n )\n\n gradient_clipping: GradientClippingConfig | None = GradientClippingDataclassField(\n description=\"Parameter values for gradient clipping.\",\n default={},\n )\n\n learning_rate_scaling: str = schema_utils.StringOptions(\n [\"constant\", \"sqrt\", \"linear\"],\n default=\"linear\",\n description=\"Scale by which to increase the learning rate as the number of distributed workers increases. \"\n \"Traditionally the learning rate is scaled linearly with the number of workers to reflect the \"\n \"proportion by\"\n \" which the effective batch size is increased. For very large batch sizes, a softer square-root \"\n \"scale can \"\n \"sometimes lead to better model performance. If the learning rate is hand-tuned for a given \"\n \"number of \"\n \"workers, setting this value to constant can be used to disable scale-up.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"learning_rate_scaling\"],\n )\n\n bucketing_field: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=\"Feature to use for bucketing datapoints\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"bucketing_field\"],\n )\n\n use_mixed_precision: bool = schema_utils.Boolean(\n default=False,\n description=\"Enable automatic mixed-precision (AMP) during training.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"use_mixed_precision\"],\n )\n\n compile: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to compile the model before training.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"compile\"],\n )\n\n enable_gradient_checkpointing: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to enable gradient checkpointing, which trades compute for memory.\"\n \"This is useful for training very deep models with limited memory.\",\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"enable_gradient_checkpointing\"],\n )\n\n layers_to_freeze_regex: str = schema_utils.String(\n default=None,\n allow_none=True,\n description=(\n \"Freeze specific layers based on provided regex. Freezing specific layers can improve a \"\n \"pretrained model's performance in a number of ways. At a basic level, freezing early layers can \"\n \"prevent overfitting by retaining more general features (beneficial for small datasets). Also can \"\n \"reduce computational resource use and lower overall training time due to less gradient calculations. \"\n ),\n )\n\n def update_batch_size_grad_accum(self, num_workers: int):\n from ludwig.utils.trainer_utils import get_rendered_batch_size_grad_accum\n\n self.batch_size, self.gradient_accumulation_steps = get_rendered_batch_size_grad_accum(self, num_workers)\n\n\n@DeveloperAPI\n@ludwig_dataclass\nclass LLMTrainerConfig(BaseTrainerConfig):\n \"\"\"Base class for all LLM trainer configs.\"\"\"\n\n learning_rate: float | str = schema_utils.OneOfOptionsField(\n default=0.0002,\n allow_none=False,\n description=(\n \"Controls how much to change the model in response to the estimated error each time the model weights are \"\n \"updated. If 'auto', the optimal learning rate is estimated by choosing the learning rate that produces \"\n \"the smallest non-diverging gradient update.\"\n ),\n parameter_metadata=TRAINER_METADATA[MODEL_ECD][\"learning_rate\"],\n field_options=[\n schema_utils.FloatRange(default=0.001, allow_none=False, min=0, max=1),\n schema_utils.StringOptions(options=[\"auto\"], default=\"auto\", allow_none=False),\n ],\n )\n\n batch_size: int = schema_utils.PositiveInteger(\n default=1,\n description=\"Batch size used for training in the LLM trainer.\",\n )\n\n base_learning_rate: float = schema_utils.NonNegativeFloat(\n default=0.0,\n description=\"Base learning rate used for training in the LLM trainer.\",\n )\n\n should_shuffle: bool = schema_utils.Boolean(\n default=True,\n description=\"Whether to shuffle the training data in the LLM trainer.\",\n )\n\n epochs: int = schema_utils.PositiveInteger(\n default=3,\n description=\"Number of epochs to train in the LLM trainer.\",\n )\n\n train_steps: int = schema_utils.PositiveInteger(\n default=None,\n allow_none=True,\n description=\"Number of training steps to train in the LLM trainer.\",\n )\n\n eval_steps: float = schema_utils.NonNegativeInteger(\n default=None,\n allow_none=True,\n description=\"The number of steps to evaluate in the LLM trainer.\",\n )\n\n steps_per_checkpoint: int = schema_utils.NonNegativeInteger(\n default=0,\n description=\"Number of steps per checkpoint in the LLM trainer.\",\n )\n\n checkpoints_per_epoch: int = schema_utils.NonNegativeInteger(\n default=0,\n description=\"Number of checkpoints per epoch in the LLM trainer.\",\n )\n\n early_stop: int = schema_utils.IntegerRange(\n default=-1,\n min=-1,\n description=(\n \"Number of consecutive rounds of evaluation without any improvement on the `validation_metric` that \"\n \"triggers training to stop. Can be set to -1, which disables early stopping entirely.\"\n ),\n )\n\n eval_batch_size: int = schema_utils.PositiveInteger(\n default=2,\n description=\"Batch size used for evaluation in the LLM trainer.\",\n )\n\n evaluate_training_set: bool = schema_utils.Boolean(\n default=False,\n description=\"Whether to evaluate the training set in the LLM trainer. Note: this operation may be slow.\",\n )\n\n\n@DeveloperAPI\n@register_llm_trainer_schema(\"none\")\n@ludwig_dataclass\nclass NoneTrainerConfig(LLMTrainerConfig):\n \"\"\"Dataclass that configures most of the hyperparameters used for zero-shot / few-shot LLM model training.\"\"\"\n\n # Required for lookup during trainer initialization\n type: str = schema_utils.ProtectedString(\n \"none\",\n description=\"The type of trainer used to train the model. \",\n parameter_metadata=TRAINER_METADATA[MODEL_LLM][\"type\"],\n )\n\n def can_tune_batch_size(self) -> bool:\n return False\n\n\n@DeveloperAPI\n@register_llm_trainer_schema(\"finetune\")\n@ludwig_dataclass\nclass FineTuneTrainerConfig(ECDTrainerConfig):\n \"\"\"Dataclass that configures most of the hyperparameters used for fine-tuning LLM model training.\"\"\"\n\n # Required for lookup during trainer initialization\n type: str = schema_utils.ProtectedString(\"finetune\")\n\n base_learning_rate: float = schema_utils.NonNegativeFloat(\n default=0.0,\n description=\"Base learning rate used for training in the LLM trainer.\",\n )\n\n batch_size: int | str | None = schema_utils.OneOfOptionsField(\n default=1,\n allow_none=False,\n description=(\n \"The number of training examples utilized in one training step of the model. If `auto`, the \"\n \"batch size that maximized training throughput (samples / sec) will be used.\"\n ),\n field_options=[\n schema_utils.PositiveInteger(default=1, description=\"\", allow_none=False),\n schema_utils.StringOptions(options=[\"auto\"], default=\"auto\", allow_none=False),\n ],\n )\n\n eval_batch_size: int | str | None = schema_utils.OneOfOptionsField(\n default=2,\n allow_none=True,\n description=(\n \"Size of batch to pass to the model for evaluation. If it is `0` or `None`, the same value of `batch_size` \"\n \"is used. This is useful to speedup evaluation with a much bigger batch size than training, if enough \"\n \"memory is available. If `auto`, the biggest batch size (power of 2) that can fit in memory will be used.\"\n ),\n field_options=[\n schema_utils.PositiveInteger(default=2, description=\"\", allow_none=False),\n schema_utils.StringOptions(options=[\"auto\"], default=\"auto\", allow_none=False),\n ],\n )\n\n\n@DeveloperAPI\ndef get_model_type_jsonschema(model_type: str = MODEL_ECD):\n if model_type == MODEL_LLM:\n enum = [MODEL_LLM]\n else:\n enum = [MODEL_ECD]\n\n return {\n \"type\": \"string\",\n \"enum\": enum,\n \"default\": MODEL_ECD,\n \"title\": \"model_type\",\n \"description\": \"Select the model type.\",\n }\n\n\n@DeveloperAPI\ndef get_trainer_jsonschema(model_type: str):\n trainer_cls = trainer_schema_registry[model_type]\n props = schema_utils.unload_jsonschema_from_marshmallow_class(trainer_cls)[\"properties\"]\n\n return {\n \"type\": \"object\",\n \"properties\": props,\n \"title\": \"trainer_options\",\n \"additionalProperties\": False,\n \"description\": \"Schema for trainer determined by Model Type\",\n }\n\n\n@DeveloperAPI\nclass ECDTrainerField(schema_utils.DictMarshmallowField):\n def __init__(self):\n super().__init__(ECDTrainerConfig)\n\n def _jsonschema_type_mapping(self):\n return get_trainer_jsonschema(MODEL_ECD)\n\n\n@DeveloperAPI\ndef get_llm_trainer_conds():\n \"\"\"Returns a JSON schema of conditionals to validate against adapter types.\"\"\"\n conds = []\n for trainer in _llm_trainer_schema_registry:\n trainer_cls = _llm_trainer_schema_registry[trainer]\n other_props = schema_utils.unload_jsonschema_from_marshmallow_class(trainer_cls)[\"properties\"]\n schema_utils.remove_duplicate_fields(other_props)\n preproc_cond = schema_utils.create_cond(\n {\"type\": trainer},\n other_props,\n )\n conds.append(preproc_cond)\n return conds\n\n\n@DeveloperAPI\ndef LLMTrainerDataclassField(default=\"none\", description=\"\"):\n class LLMTrainerSelection(schema_utils.TypeSelection):\n def __init__(self):\n super().__init__(\n registry=_llm_trainer_schema_registry,\n default_value=default,\n description=description,\n )\n\n def get_schema_from_registry(self, key: str) -> type[schema_utils.BaseMarshmallowConfig]:\n return get_llm_trainer_cls(key)\n\n def _jsonschema_type_mapping(self):\n return {\n \"type\": \"object\",\n \"properties\": {\n \"type\": {\n \"type\": \"string\",\n \"enum\": list(_llm_trainer_schema_registry.keys()),\n \"default\": default,\n \"description\": \"The type of LLM trainer to use\",\n },\n },\n \"title\": \"llm_trainer_options\",\n \"allOf\": get_llm_trainer_conds(),\n \"required\": [\"type\"],\n \"description\": description,\n }\n\n return LLMTrainerSelection().get_default_field()\n"
},
{
"path": "ludwig/schema/utils.py",
"content": "\"\"\"Ludwig schema utilities - pydantic 2 based.\n\nThis module provides the foundation for Ludwig's declarative config system.\nAll config classes inherit from BaseMarshmallowConfig (a pydantic BaseModel)\nand use field factory functions (String, Integer, Float, etc.) that return\npydantic Field() objects.\n\"\"\"\n\nimport copy\nimport logging\nimport os\nimport warnings\nfrom abc import ABC, abstractmethod\nfrom functools import lru_cache\nfrom typing import Any\n\nimport yaml\nfrom pydantic import BaseModel, ConfigDict, Field, model_validator\nfrom pydantic import ValidationError as PydanticValidationError\nfrom pydantic.fields import FieldInfo\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import ACTIVE, COLUMN, LUDWIG_SCHEMA_VALIDATION_POLICY, NAME, PROC_COLUMN, TYPE\nfrom ludwig.error import ConfigValidationError\nfrom ludwig.modules.reduction_modules import reduce_mode_registry\nfrom ludwig.schema.metadata import COMMON_METADATA\nfrom ludwig.schema.metadata.parameter_metadata import convert_metadata_to_json, ParameterMetadata\nfrom ludwig.utils.misc_utils import scrub_creds\nfrom ludwig.utils.registry import Registry\nfrom ludwig.utils.torch_utils import activations, initializer_registry\n\n# ============================================================================\n# LudwigSchemaField - base class replacing marshmallow fields.Field\n# ============================================================================\n\n\nclass LudwigSchemaField:\n \"\"\"Plain Python base class for Ludwig schema fields.\n\n Replaces marshmallow fields.Field as the base for TypeSelection, DictMarshmallowField (NestedConfigField), and all\n custom field classes. The contract (get_default_field, _jsonschema_type_mapping, _deserialize) stays identical.\n \"\"\"\n\n def __init__(self, **kwargs):\n # Store all keyword arguments as attributes for backward compat\n for k, v in kwargs.items():\n setattr(self, k, v)\n\n def get_default_field(self) -> FieldInfo:\n \"\"\"Create a pydantic FieldInfo for this field.\n\n Override in subclasses.\n \"\"\"\n return Field(default=None)\n\n def _jsonschema_type_mapping(self):\n \"\"\"Return a JSON schema dict for this field.\n\n Override in subclasses.\n \"\"\"\n return None\n\n def _deserialize(self, value, attr, data, **kwargs):\n \"\"\"Deserialize a raw value.\n\n Override in subclasses.\n \"\"\"\n return value\n\n\nlogger = logging.getLogger(__name__)\n\nRECURSION_STOP_ENUM = {\"weights_initializer\", \"bias_initializer\", \"norm_params\"}\n\n\ndef ludwig_dataclass(cls):\n \"\"\"No-op decorator.\n\n Config classes now inherit directly from BaseMarshmallowConfig (pydantic BaseModel).\n \"\"\"\n return cls\n\n\n# TODO: Change to RAISE and update descriptions once we want to enforce strict schemas.\nLUDWIG_SCHEMA_VALIDATION_POLICY_VAR = os.environ.get(LUDWIG_SCHEMA_VALIDATION_POLICY, \"exclude\").lower()\n\n\nclass _SchemaAdapter:\n \"\"\"Adapts pydantic model to marshmallow-like Schema interface for backward compatibility.\n\n This allows existing code that calls cls.Schema().load(data), cls.Schema().dump(data), and cls.Schema().fields to\n continue working.\n \"\"\"\n\n def __init__(self, cls):\n self._cls = cls\n\n def __call__(self):\n \"\"\"Allow Schema()() pattern (double-call).\"\"\"\n return self\n\n def load(self, data):\n \"\"\"Validate and create a config instance from a dict.\"\"\"\n return self._cls.model_validate(data)\n\n def dump(self, data):\n \"\"\"Serialize a config instance or dict to a plain dict.\"\"\"\n if isinstance(data, BaseMarshmallowConfig):\n return data.to_dict()\n if isinstance(data, dict):\n try:\n instance = self._cls.model_validate(data)\n return instance.to_dict()\n except Exception:\n return data\n return data\n\n @property\n def fields(self):\n \"\"\"Return field info dict (pydantic model_fields).\"\"\"\n return self._cls.model_fields\n\n\n# Sentinel for TypeSelection and DictMarshmallowField metadata markers\nclass _TypeSelectionMarker:\n \"\"\"Marker stored in Field.metadata to indicate this field uses TypeSelection dispatch.\"\"\"\n\n def __init__(self, type_selection):\n self.type_selection = type_selection\n\n\nclass _NestedConfigMarker:\n \"\"\"Marker stored in Field.metadata to indicate this field uses DictMarshmallowField dispatch.\"\"\"\n\n def __init__(self, cls, allow_none=True):\n self.cls = cls\n self.allow_none = allow_none\n\n\nConfigT = Any # TypeVar(\"ConfigT\", bound=\"BaseMarshmallowConfig\")\n\n\ndef _convert_dataclass_field_to_pydantic(dc_field) -> FieldInfo:\n \"\"\"Convert a dataclasses.Field to a pydantic FieldInfo.\n\n This is the bridge that allows old marshmallow-style field definitions\n (using dataclasses.field(metadata={\"marshmallow_field\": ...})) to work\n with pydantic BaseModel classes during the migration period.\n \"\"\"\n import dataclasses as _dc\n\n metadata_list = []\n marshmallow_field = None\n\n # Extract marshmallow_field from metadata\n if dc_field.metadata:\n marshmallow_field = dc_field.metadata.get(\"marshmallow_field\")\n if marshmallow_field is not None:\n # Store as a marker so model_validator can use it for dispatch\n if isinstance(marshmallow_field, TypeSelection):\n metadata_list.append(_TypeSelectionMarker(marshmallow_field))\n elif isinstance(marshmallow_field, DictMarshmallowField):\n # Check if the subclass overrides _jsonschema_type_mapping\n has_custom_schema = (\n type(marshmallow_field)._jsonschema_type_mapping\n is not DictMarshmallowField._jsonschema_type_mapping\n )\n if has_custom_schema:\n # Store as MarshmallowFieldMarker to preserve custom JSON schema generation\n metadata_list.append(_MarshmallowFieldMarker(marshmallow_field))\n else:\n metadata_list.append(_NestedConfigMarker(marshmallow_field.cls, marshmallow_field.allow_none))\n else:\n # Generic marshmallow field - store for reference\n metadata_list.append(_MarshmallowFieldMarker(marshmallow_field))\n\n # Extract default and create FieldInfo.\n # Note: pydantic 2's Field() does not accept a `metadata` kwarg — set it on the FieldInfo after creation.\n if dc_field.default is not _dc.MISSING:\n fi = Field(default=dc_field.default)\n elif dc_field.default_factory is not _dc.MISSING:\n fi = Field(default_factory=dc_field.default_factory)\n else:\n # No default - this is a required field\n fi = Field()\n if metadata_list:\n fi.metadata = metadata_list\n return fi\n\n\nclass _MarshmallowFieldMarker:\n \"\"\"Stores a marshmallow field for backward compat during migration.\"\"\"\n\n def __init__(self, marshmallow_field):\n self.marshmallow_field = marshmallow_field\n\n\nclass _LudwigModelMeta(type(BaseModel)):\n \"\"\"Metaclass that bridges marshmallow-dataclass patterns to pydantic 2.\n\n Handles two key behaviors:\n 1. Converts dataclasses.Field objects to pydantic FieldInfo in __new__\n 2. Allows class-level access to field defaults via __getattr__\n \"\"\"\n\n def __new__(mcs, name, bases, namespace, **kwargs):\n import dataclasses as _dc\n\n annotations = namespace.get(\"__annotations__\", {})\n\n # Detect @property definitions and prevent pydantic from treating them as field defaults.\n # Properties that don't shadow inherited fields work fine as-is because pydantic\n # only processes annotated attributes. Properties that DO shadow inherited fields\n # should be converted to fields with constant defaults instead (done at the schema\n # class level, not here).\n _saved_properties: dict[str, property] = {}\n for attr_name, value in list(namespace.items()):\n if isinstance(value, property) and attr_name in annotations:\n # A property in this class's own annotations would confuse pydantic.\n # Remove it from annotations (it won't become a field).\n _saved_properties[attr_name] = value\n del namespace[attr_name]\n annotations.pop(attr_name, None)\n\n # Convert dataclass field() objects and marshmallow field descriptors to pydantic Field()\n for attr_name in list(annotations.keys()):\n if attr_name in namespace:\n value = namespace[attr_name]\n if isinstance(value, _dc.Field):\n namespace[attr_name] = _convert_dataclass_field_to_pydantic(value)\n elif isinstance(value, LudwigSchemaField) and hasattr(value, \"get_default_field\"):\n # TypeSelection and DictMarshmallowField instances need conversion\n namespace[attr_name] = value.get_default_field()\n\n # Auto-widen annotations to bridge marshmallow→pydantic gap.\n # In marshmallow, annotations were decorative. In pydantic, they're enforced.\n import types\n import typing\n\n for attr_name, ann in list(annotations.items()):\n # Skip ClassVar annotations\n origin = getattr(ann, \"__origin__\", None)\n if origin is typing.ClassVar:\n continue\n\n if attr_name not in namespace:\n continue\n\n value = namespace[attr_name]\n\n # For fields with markers (TypeSelection/DictMarshmallowField/MarshmallowField),\n # set annotation to Any since the actual validation happens in the marker\n if isinstance(value, FieldInfo):\n jse = getattr(value, \"json_schema_extra\", None)\n has_marker = False\n if isinstance(jse, dict) and \"metadata\" in jse:\n has_marker = any(\n isinstance(m, (_TypeSelectionMarker, _NestedConfigMarker, _MarshmallowFieldMarker))\n for m in jse[\"metadata\"]\n )\n for meta in getattr(value, \"metadata\", None) or []:\n if isinstance(meta, (_TypeSelectionMarker, _NestedConfigMarker, _MarshmallowFieldMarker)):\n has_marker = True\n break\n\n if has_marker:\n annotations[attr_name] = Any\n continue\n\n # Widen to include None if default is None or enum contains None\n from pydantic_core import PydanticUndefined\n\n should_widen = value.default is None and value.default is not PydanticUndefined\n if not should_widen:\n # Also widen if the enum (from allow_none=True in StringOptions etc.) contains None\n jse_enum = (jse or {}).get(\"enum\") if isinstance(jse, dict) else None\n if isinstance(jse_enum, list) and None in jse_enum:\n should_widen = True\n if not should_widen:\n # Also widen if allow_none=True was explicitly set in the field factory\n if isinstance(jse, dict) and jse.get(\"allow_none\"):\n should_widen = True\n\n if should_widen:\n is_union = origin in (types.UnionType,)\n try:\n is_union = is_union or origin is typing.Union\n except (AttributeError, TypeError):\n pass\n\n has_none = False\n if is_union:\n has_none = type(None) in getattr(ann, \"__args__\", ())\n\n if not has_none:\n try:\n annotations[attr_name] = ann | None\n except TypeError:\n pass\n\n elif value is None:\n # Plain None default\n try:\n annotations[attr_name] = ann | None\n except TypeError:\n pass\n\n namespace[\"__annotations__\"] = annotations\n import warnings\n\n with warnings.catch_warnings():\n warnings.filterwarnings(\"ignore\", message=\"Field name .* shadows an attribute in parent\")\n cls = super().__new__(mcs, name, bases, namespace, **kwargs)\n\n # Restore @property descriptors that we removed from namespace.\n if _saved_properties:\n for pname, prop in _saved_properties.items():\n setattr(cls, pname, prop)\n\n return cls\n\n def __getattr__(cls, name: str) -> Any:\n \"\"\"Allow accessing field defaults as class attributes (e.g., cls.type).\"\"\"\n for klass in cls.__mro__:\n pf = vars(klass).get(\"__pydantic_fields__\")\n if pf is not None and isinstance(pf, dict) and name in pf:\n field_info = pf[name]\n from pydantic_core import PydanticUndefined\n\n if field_info.default is not PydanticUndefined:\n return field_info.default\n break\n raise AttributeError(name)\n\n\n@DeveloperAPI\nclass BaseMarshmallowConfig(BaseModel, metaclass=_LudwigModelMeta):\n \"\"\"Base pydantic model for all Ludwig config classes.\n\n Maintains backward-compatible API (from_dict, to_dict, Schema, etc.) while using pydantic 2 internally for\n validation and serialization.\n \"\"\"\n\n model_config = ConfigDict(\n extra=\"ignore\" if LUDWIG_SCHEMA_VALIDATION_POLICY_VAR == \"exclude\" else \"forbid\",\n arbitrary_types_allowed=True,\n validate_default=False,\n revalidate_instances=\"never\",\n populate_by_name=True,\n strict=False,\n )\n\n @model_validator(mode=\"before\")\n @classmethod\n def _pre_validate(cls, data: Any) -> Any:\n \"\"\"Pre-validation: log deprecation warnings, resolve TypeSelection/nested fields.\"\"\"\n if not isinstance(data, dict):\n return data\n\n # Log deprecation warnings for unknown fields\n valid_fields = set(cls.model_fields.keys())\n for key in list(data.keys()):\n if key not in valid_fields and key != \"type\":\n warnings.warn(\n f'\"{key}\" is not a valid parameter for the \"{cls.__name__}\" schema, will be flagged '\n \"as an error in a future version\",\n DeprecationWarning,\n )\n\n # Resolve TypeSelection, DictMarshmallowField, and legacy marshmallow fields\n for fname, finfo in cls.model_fields.items():\n if fname not in data:\n continue\n value = data[fname]\n\n # Get markers from both metadata and json_schema_extra\n markers = list(finfo.metadata or [])\n jse = finfo.json_schema_extra\n if isinstance(jse, dict) and \"metadata\" in jse:\n markers.extend(jse[\"metadata\"])\n\n for meta in markers:\n if isinstance(meta, _TypeSelectionMarker):\n data[fname] = meta.type_selection.resolve(value)\n break\n elif isinstance(meta, _NestedConfigMarker):\n if isinstance(value, BaseMarshmallowConfig):\n break # Already a config instance, skip re-validation\n if isinstance(value, dict):\n try:\n data[fname] = meta.cls.model_validate(value)\n except Exception as e:\n raise ConfigValidationError(\n f\"Invalid params: {value}, see `{meta.cls}` definition. Error: {e}\"\n )\n break\n elif isinstance(meta, _MarshmallowFieldMarker):\n # Legacy marshmallow field - use its _deserialize for validation\n # Skip if value is already a config instance (avoid double-validation)\n if isinstance(value, BaseMarshmallowConfig):\n break\n mfield = meta.marshmallow_field\n if hasattr(mfield, \"_deserialize\") and value is not None:\n try:\n data[fname] = mfield._deserialize(value, fname, data)\n except Exception as e:\n # Re-raise ConfigValidationError (from __post_init__) and\n # from _deserialize rather than swallowing them\n if isinstance(e, ConfigValidationError):\n raise\n pass # Let pydantic handle other validation errors\n break\n\n return data\n\n @model_validator(mode=\"after\")\n def _validate_field_constraints(self):\n \"\"\"Post-validation: enforce enum constraints stored in json_schema_extra.\"\"\"\n for fname, finfo in type(self).model_fields.items():\n value = getattr(self, fname, None)\n extra = finfo.json_schema_extra\n if not isinstance(extra, dict):\n continue\n\n # Validate enum constraints (from StringOptions, IntegerOptions)\n if \"enum\" in extra and value is not None:\n allowed = extra[\"enum\"]\n if value not in allowed:\n raise ValueError(f\"Field '{fname}': value {value!r} not in allowed options {allowed}\")\n\n # Validate float tuple range constraints\n if \"_float_tuple_range\" in extra and value is not None:\n spec = extra[\"_float_tuple_range\"]\n if not isinstance(value, (tuple, list)) or len(value) != spec[\"n\"]:\n raise ValueError(f\"Field '{fname}': expected {spec['n']}-tuple, got {value!r}\")\n for v in value:\n if spec.get(\"min\") is not None and v < spec[\"min\"]:\n raise ValueError(f\"Field '{fname}': value {v} below minimum {spec['min']}\")\n if spec.get(\"max\") is not None and v > spec[\"max\"]:\n raise ValueError(f\"Field '{fname}': value {v} above maximum {spec['max']}\")\n\n # Validate embed field (int or str from options)\n if \"_embed_options\" in extra and value is not None:\n embed_options = extra[\"_embed_options\"]\n if isinstance(value, str) and value not in embed_options:\n raise ValueError(f\"Field '{fname}': string value {value!r} not in {embed_options}\")\n if not isinstance(value, (str, int)):\n raise ValueError(f\"Field '{fname}': expected str, int, or None, got {type(value).__name__}\")\n\n # Validate initializer_or_dict field\n if \"_initializer_options\" in extra and value is not None:\n init_options = extra[\"_initializer_options\"]\n if isinstance(value, str) and value not in init_options:\n raise ValueError(f\"Field '{fname}': initializer {value!r} not in {init_options}\")\n if isinstance(value, dict):\n if \"type\" not in value:\n raise ValueError(f\"Field '{fname}': dict must contain 'type' key\")\n if value[\"type\"] not in init_options:\n raise ValueError(f\"Field '{fname}': initializer type {value['type']!r} not in {init_options}\")\n if not isinstance(value, (str, dict)):\n raise ValueError(f\"Field '{fname}': expected str or dict, got {type(value).__name__}\")\n\n return self\n\n def __setattr__(self, name: str, value: Any) -> None:\n \"\"\"Allow setting arbitrary attributes on config instances.\n\n Ludwig code dynamically sets attributes like saved_weights_in_checkpoint, proc_column, etc. on config objects.\n Pydantic 2 normally rejects setting attributes not defined as fields, so we override to allow it.\n \"\"\"\n try:\n super().__setattr__(name, value)\n except ValueError:\n # Attribute not in model fields - allow it anyway (dataclass behavior)\n object.__setattr__(self, name, value)\n\n def model_post_init(self, __context: Any) -> None:\n \"\"\"Bridge: call __post_init__ if defined by subclass (dataclass convention).\"\"\"\n super().model_post_init(__context)\n # Check if THIS class (or a parent) defines __post_init__\n post_init = getattr(type(self), \"__post_init__\", None)\n if post_init is not None:\n post_init(self)\n\n def to_dict(self) -> dict[str, Any]:\n \"\"\"Get a dictionary representation of this config.\n\n Recursively converts nested config objects and scrubs credentials.\n \"\"\"\n return scrub_creds(convert_submodules(vars(self)))\n\n @classmethod\n def from_dict(cls, d: dict[str, Any]) -> \"BaseMarshmallowConfig\":\n \"\"\"Create a config instance from a dictionary.\"\"\"\n return cls.model_validate(d)\n\n @classmethod\n @lru_cache(maxsize=None)\n def get_valid_field_names(cls) -> set[str]:\n \"\"\"Return the set of valid field names for this config class.\"\"\"\n return set(cls.model_fields.keys())\n\n @classmethod\n @lru_cache(maxsize=None)\n def get_class_schema(cls):\n \"\"\"Return a schema adapter for backward compatibility.\n\n Returns an object with .load() and .fields methods.\n \"\"\"\n return _SchemaAdapter(cls)\n\n @classmethod\n def Schema(cls):\n \"\"\"Backward compatibility: return a schema adapter with .load(), .dump(), .fields.\"\"\"\n return _SchemaAdapter(cls)\n\n def __repr__(self):\n return yaml.dump(self.to_dict(), sort_keys=False)\n\n\n@DeveloperAPI\ndef get_marshmallow_field_class_name(field_info):\n \"\"\"Returns a human-readable string of the field class name.\n\n For backward compat, checks both pydantic metadata and marshmallow_field.\n \"\"\"\n # Check for marshmallow_field in metadata (legacy)\n if hasattr(field_info, \"metadata\"):\n for meta in field_info.metadata or []:\n if hasattr(meta, \"__class__\"):\n return meta.__class__.__name__\n # For pydantic FieldInfo, return the annotation name\n if hasattr(field_info, \"annotation\"):\n return str(field_info.annotation)\n return \"Unknown\"\n\n\n@DeveloperAPI\ndef load_config(cls: type[\"BaseMarshmallowConfig\"], **kwargs) -> \"BaseMarshmallowConfig\":\n \"\"\"Takes a config class and instantiates it with the given keyword args as parameters.\"\"\"\n assert_is_a_marshmallow_class(cls)\n return cls.model_validate(kwargs)\n\n\n@DeveloperAPI\ndef load_trainer_with_kwargs(model_type: str, kwargs: dict) -> tuple[\"BaseMarshmallowConfig\", dict[str, Any]]:\n \"\"\"Special case of `load_config_with_kwargs` for the trainer schemas.\"\"\"\n from ludwig.constants import MODEL_LLM\n from ludwig.schema.trainer import ECDTrainerConfig, LLMTrainerConfig\n\n if model_type == MODEL_LLM:\n trainer_schema = LLMTrainerConfig\n else:\n trainer_schema = ECDTrainerConfig\n\n return load_config_with_kwargs(trainer_schema, kwargs)\n\n\n@DeveloperAPI\ndef load_config_with_kwargs(\n cls: type[\"BaseMarshmallowConfig\"], kwargs_overrides\n) -> tuple[\"BaseMarshmallowConfig\", dict[str, Any]]:\n \"\"\"Instantiates a config class filtering kwargs to only valid fields.\n\n Returns a tuple of (config, remaining_kwargs).\n \"\"\"\n assert_is_a_marshmallow_class(cls)\n fields = cls.model_fields.keys()\n return load_config(cls, **{k: v for k, v in kwargs_overrides.items() if k in fields}), {\n k: v for k, v in kwargs_overrides.items() if k not in fields\n }\n\n\n@DeveloperAPI\ndef convert_submodules(config_dict: dict) -> dict[str, Any]:\n \"\"\"Helper for converting submodules to dictionaries during config serialization.\"\"\"\n output_dict = copy.deepcopy(config_dict)\n\n for k, v in output_dict.items():\n if isinstance(v, dict):\n convert_submodules(v)\n elif isinstance(v, BaseMarshmallowConfig):\n output_dict[k] = v.to_dict()\n convert_submodules(output_dict[k])\n elif isinstance(v, list):\n output_dict[k] = [x.to_dict() if isinstance(x, BaseMarshmallowConfig) else x for x in v]\n elif isinstance(v, ListSerializable):\n output_dict[k] = v.to_list()\n\n return output_dict\n\n\n@DeveloperAPI\ndef create_cond(if_pred: dict, then_pred: dict):\n \"\"\"Returns a JSONSchema conditional for the given if-then predicates.\"\"\"\n return {\n \"if\": {\"properties\": {k: {\"const\": v} for k, v in if_pred.items()}},\n \"then\": {\"properties\": then_pred},\n }\n\n\n@DeveloperAPI\ndef remove_duplicate_fields(properties: dict, fields: list[str] | None = None) -> None:\n \"\"\"Util function for removing duplicated schema elements.\"\"\"\n duplicate_fields = [NAME, TYPE, COLUMN, PROC_COLUMN, ACTIVE] if fields is None else fields\n for key in duplicate_fields:\n if key in properties:\n del properties[key]\n\n\n@DeveloperAPI\nclass ListSerializable(ABC):\n @abstractmethod\n def to_list(self) -> list:\n pass\n\n\n@DeveloperAPI\ndef assert_is_a_marshmallow_class(cls):\n \"\"\"Assert that cls is a Ludwig config class (pydantic BaseModel).\"\"\"\n assert issubclass(\n cls, BaseMarshmallowConfig\n ), f\"Expected a Ludwig config class (BaseMarshmallowConfig subclass), but `{cls}` is not.\"\n\n\ndef _default_matches_json_type(default_val, type_str) -> bool:\n \"\"\"Check if a default value is consistent with a JSON schema type string.\n\n Returns True if the default value matches the type string, False otherwise. This is used to avoid emitting 'type':\n 'integer' when the default is 7.5 (float), which was a common pattern in the marshmallow era where type enforcement\n was looser.\n \"\"\"\n if isinstance(type_str, list):\n # Union type like [\"integer\", \"null\"]\n return any(_default_matches_json_type(default_val, t) for t in type_str)\n _CHECKS = {\n \"string\": lambda v: isinstance(v, str),\n \"integer\": lambda v: isinstance(v, int) and not isinstance(v, bool),\n \"number\": lambda v: isinstance(v, (int, float)) and not isinstance(v, bool),\n \"boolean\": lambda v: isinstance(v, bool),\n \"object\": lambda v: isinstance(v, dict),\n \"array\": lambda v: isinstance(v, (list, tuple)),\n \"null\": lambda v: v is None,\n }\n check = _CHECKS.get(type_str)\n if check is None:\n return True # Unknown type, don't block\n return check(default_val)\n\n\ndef _field_info_to_jsonschema(fname: str, finfo: FieldInfo, annotation: type | None = None) -> dict:\n \"\"\"Convert a pydantic FieldInfo to a JSON schema fragment.\n\n Checks metadata markers for TypeSelection/DictMarshmallowField/legacy marshmallow fields, and falls back to type-\n based mapping for plain fields.\n \"\"\"\n # Check for markers in both metadata and json_schema_extra\n markers = list(finfo.metadata or [])\n jse = finfo.json_schema_extra\n if isinstance(jse, dict) and \"metadata\" in jse:\n markers.extend(jse[\"metadata\"])\n\n for meta in markers:\n if isinstance(meta, _TypeSelectionMarker):\n ts = meta.type_selection\n custom = ts._jsonschema_type_mapping()\n if custom is not None:\n return custom\n return {\"type\": \"object\"}\n\n if isinstance(meta, _NestedConfigMarker):\n return unload_jsonschema_from_marshmallow_class(meta.cls)\n\n if isinstance(meta, _MarshmallowFieldMarker):\n mf = meta.marshmallow_field\n if hasattr(mf, \"_jsonschema_type_mapping\"):\n custom = mf._jsonschema_type_mapping()\n if custom is not None:\n return custom\n # Handle FeatureList-style fields with inner and length constraints\n if hasattr(mf, \"inner\") and mf.inner is not None:\n inner_schema = {}\n if hasattr(mf.inner, \"_jsonschema_type_mapping\"):\n inner_schema = mf.inner._jsonschema_type_mapping() or {}\n result = {\"type\": \"array\", \"items\": inner_schema}\n if hasattr(mf, \"min_length\") and mf.min_length is not None:\n result[\"minItems\"] = mf.min_length\n if hasattr(mf, \"max_length\") and mf.max_length is not None:\n result[\"maxItems\"] = mf.max_length\n return result\n return {\"type\": \"object\"}\n\n # Handle InitializerOrDict fields\n from pydantic_core import PydanticUndefined\n\n extra = finfo.json_schema_extra\n if isinstance(extra, dict) and \"_initializer_options\" in extra:\n init_options = extra[\"_initializer_options\"]\n return {\n \"oneOf\": [\n {\"type\": \"string\", \"enum\": init_options},\n {\n \"type\": \"object\",\n \"properties\": {\"type\": {\"type\": \"string\", \"enum\": init_options}},\n \"required\": [\"type\"],\n \"additionalProperties\": True,\n },\n {\"type\": \"null\"},\n ],\n \"default\": finfo.default if finfo.default is not PydanticUndefined else \"xavier_uniform\",\n \"description\": finfo.description or \"\",\n }\n\n # Build schema from field info\n schema: dict[str, Any] = {}\n\n # Description\n desc = finfo.description or \"\"\n if desc:\n schema[\"description\"] = desc\n\n # Default value\n from pydantic_core import PydanticUndefined\n\n if finfo.default is not PydanticUndefined:\n if not callable(finfo.default) and not isinstance(finfo.default, property):\n schema[\"default\"] = finfo.default\n\n # Enum constraint from json_schema_extra\n extra = finfo.json_schema_extra\n if isinstance(extra, dict):\n if \"enum\" in extra:\n schema[\"enum\"] = extra[\"enum\"]\n if \"parameter_metadata\" in extra:\n schema[\"parameter_metadata\"] = copy.deepcopy(extra[\"parameter_metadata\"])\n\n # Always include parameter_metadata (default if not explicitly provided)\n if \"parameter_metadata\" not in schema:\n schema[\"parameter_metadata\"] = convert_metadata_to_json(None)\n\n # Map type annotation to JSON schema type\n # Only emit type if annotation and default are consistent (avoid mismatches\n # like annotation=int but default=7.5 which was common in marshmallow era)\n if annotation is not None:\n type_str = _annotation_to_json_type(annotation)\n if type_str:\n # If the enum contains None, the JSON schema type must include \"null\"\n enum_vals = schema.get(\"enum\")\n if enum_vals is not None and None in enum_vals:\n if isinstance(type_str, list):\n if \"null\" not in type_str:\n type_str = type_str + [\"null\"]\n elif type_str != \"null\":\n type_str = [type_str, \"null\"]\n\n # Check for mismatch between annotation type and default value\n from pydantic_core import PydanticUndefined\n\n default_val = finfo.default if finfo.default is not PydanticUndefined else None\n if default_val is not None and not _default_matches_json_type(default_val, type_str):\n pass # Skip emitting type to avoid JSON schema validation failures\n else:\n schema[\"type\"] = type_str\n\n # Range constraints and pattern from pydantic Field metadata\n from annotated_types import Ge, Gt, Le, Lt\n\n for meta in finfo.metadata or []:\n if isinstance(meta, Ge):\n schema[\"minimum\"] = meta.ge\n elif isinstance(meta, Gt):\n schema[\"exclusiveMinimum\"] = meta.gt\n elif isinstance(meta, Le):\n schema[\"maximum\"] = meta.le\n elif isinstance(meta, Lt):\n schema[\"exclusiveMaximum\"] = meta.lt\n elif hasattr(meta, \"pattern\") and getattr(meta, \"pattern\", None) is not None:\n schema[\"pattern\"] = meta.pattern\n\n return schema\n\n\ndef _annotation_to_json_type(annotation) -> str | list | None:\n \"\"\"Map a Python type annotation to a JSON schema type string.\"\"\"\n import types\n\n origin = getattr(annotation, \"__origin__\", None)\n\n # Handle Python 3.10+ union types (e.g. float | None) which are instances of\n # types.UnionType directly, without __origin__\n if isinstance(annotation, types.UnionType):\n args = annotation.__args__\n has_none = type(None) in args\n non_none = [a for a in args if a is not type(None)]\n if len(non_none) == 1:\n base = _annotation_to_json_type(non_none[0])\n if has_none and base:\n return [base, \"null\"]\n return base\n return None\n\n # Also handle typing.Union\n try:\n import typing\n\n if origin is typing.Union:\n args = annotation.__args__\n has_none = type(None) in args\n non_none = [a for a in args if a is not type(None)]\n if len(non_none) == 1:\n base = _annotation_to_json_type(non_none[0])\n if has_none and base:\n return [base, \"null\"]\n return base\n return None\n except (AttributeError, TypeError):\n pass\n\n _TYPE_MAP = {\n str: \"string\",\n int: \"integer\",\n float: \"number\",\n bool: \"boolean\",\n dict: \"object\",\n list: \"array\",\n tuple: \"array\",\n }\n\n if annotation in _TYPE_MAP:\n return _TYPE_MAP[annotation]\n\n return None\n\n\n@DeveloperAPI\ndef unload_jsonschema_from_marshmallow_class(mclass, additional_properties: bool = True, title: str = None) -> dict:\n \"\"\"Get a JSON schema dict for a Ludwig config class.\n\n Iterates over pydantic model_fields and checks metadata markers for TypeSelection, DictMarshmallowField, and legacy\n marshmallow fields.\n \"\"\"\n assert_is_a_marshmallow_class(mclass)\n\n properties = {}\n annotations = {}\n\n # Gather annotations from the class and its MRO\n for klass in reversed(mclass.__mro__):\n annotations.update(getattr(klass, \"__annotations__\", {}))\n\n for fname, finfo in mclass.model_fields.items():\n ann = annotations.get(fname)\n properties[fname] = _field_info_to_jsonschema(fname, finfo, ann)\n\n schema = {\n \"type\": \"object\",\n \"properties\": properties,\n \"additionalProperties\": additional_properties,\n }\n if title is not None:\n schema[\"title\"] = title\n return schema\n\n\n# ============================================================================\n# Field Factory Functions\n# ============================================================================\n# All return pydantic Field() objects (FieldInfo) that can be used as class\n# variable defaults in BaseMarshmallowConfig subclasses.\n# ============================================================================\n\n\ndef _make_json_schema_extra(\n description: str = \"\",\n parameter_metadata: ParameterMetadata = None,\n **extra,\n) -> dict | None:\n \"\"\"Build json_schema_extra dict for Field(), returning None if empty.\"\"\"\n result = {}\n if parameter_metadata:\n result[\"parameter_metadata\"] = convert_metadata_to_json(parameter_metadata)\n result.update(extra)\n return result or None\n\n\n@DeveloperAPI\ndef InitializerOptions(default: str = \"xavier_uniform\", description=\"\", parameter_metadata: ParameterMetadata = None):\n \"\"\"Utility wrapper that returns a `StringOptions` field with keys from `initializer_registry`.\"\"\"\n return StringOptions(\n list(initializer_registry.keys()),\n default=default,\n allow_none=False,\n description=description,\n parameter_metadata=parameter_metadata,\n )\n\n\n@DeveloperAPI\ndef ActivationOptions(default: str | None = \"relu\", description=None, parameter_metadata: ParameterMetadata = None):\n \"\"\"Utility wrapper that returns a `StringOptions` field with keys from `activations` registry.\"\"\"\n description = description or \"Default activation function applied to the output of the fully connected layers.\"\n parameter_metadata = parameter_metadata or COMMON_METADATA[\"activation\"]\n return StringOptions(\n list(activations.keys()),\n default=default,\n allow_none=True,\n description=description,\n parameter_metadata=parameter_metadata,\n )\n\n\n@DeveloperAPI\ndef ReductionOptions(default: None | str = None, description=\"\", parameter_metadata: ParameterMetadata = None):\n \"\"\"Utility wrapper that returns a `StringOptions` field with keys from `reduce_mode_registry`.\"\"\"\n return StringOptions(\n list(reduce_mode_registry.keys()),\n default=default,\n allow_none=True,\n description=description,\n parameter_metadata=parameter_metadata,\n )\n\n\n@DeveloperAPI\ndef RegularizerOptions(\n default: None | str,\n allow_none: bool = False,\n description=\"\",\n parameter_metadata: ParameterMetadata = None,\n):\n \"\"\"Utility wrapper that returns a `StringOptions` field with prefilled regularizer options.\"\"\"\n return StringOptions(\n [\"l1\", \"l2\", \"l1_l2\"],\n default=default,\n allow_none=allow_none,\n description=description,\n parameter_metadata=parameter_metadata,\n )\n\n\n@DeveloperAPI\ndef String(\n description: str,\n default: None | str,\n allow_none: bool = False,\n pattern: str = None,\n parameter_metadata: ParameterMetadata = None,\n):\n \"\"\"Returns a pydantic Field for string values.\"\"\"\n if not allow_none and default is not None and not isinstance(default, str):\n raise ValueError(f\"Provided default `{default}` should be a string!\")\n\n extra_kwargs = {}\n if allow_none:\n extra_kwargs[\"allow_none\"] = True\n json_extra = _make_json_schema_extra(description=description, parameter_metadata=parameter_metadata, **extra_kwargs)\n kwargs = {}\n if pattern is not None:\n kwargs[\"pattern\"] = pattern\n\n return Field(\n default=default,\n description=description,\n json_schema_extra=json_extra,\n **kwargs,\n )\n\n\n@DeveloperAPI\ndef StringOptions(\n options: list[str],\n default: None | str,\n allow_none: bool = False,\n description: str = \"\",\n parameter_metadata: ParameterMetadata = None,\n):\n \"\"\"Returns a pydantic Field that enforces string inputs must be one of `options`.\"\"\"\n options = list(options) # ensure list, not dict_keys or other iterable\n assert len(options) > 0, \"Must provide non-empty list of options!\"\n\n if default is not None:\n assert isinstance(default, str), f\"Provided default `{default}` should be a string!\"\n\n if allow_none and None not in options:\n options = options + [None]\n if not allow_none and None in options:\n options = [o for o in options if o is not None]\n\n assert len(options) == len(\n {o for o in options if o is not None} | ({None} if None in options else set())\n ), f\"Provided options must be unique! See: {options}\"\n assert default in options, f\"Provided default `{default}` is not one of allowed options: {options}\"\n\n json_extra = _make_json_schema_extra(\n description=description,\n parameter_metadata=parameter_metadata,\n enum=options,\n )\n return Field(default=default, description=description, json_schema_extra=json_extra)\n\n\n@DeveloperAPI\ndef ProtectedString(\n pstring: str,\n description: str = \"\",\n parameter_metadata: ParameterMetadata = None,\n):\n \"\"\"Alias for a `StringOptions` field with only one option.\"\"\"\n return StringOptions(\n options=[pstring],\n default=pstring,\n allow_none=False,\n description=description,\n parameter_metadata=parameter_metadata,\n )\n\n\n@DeveloperAPI\ndef IntegerOptions(\n options: list[int],\n default: None | int,\n allow_none: bool = False,\n description: str = \"\",\n parameter_metadata: ParameterMetadata = None,\n):\n \"\"\"Returns a pydantic Field that enforces integer inputs must be one of `options`.\"\"\"\n if len(options) <= 0:\n raise ValueError(\"Must provide non-empty list of options!\")\n if default is not None and not isinstance(default, int):\n raise ValueError(f\"Provided default `{default}` should be an int!\")\n if allow_none and None not in options:\n options = list(options) + [None]\n if not allow_none and None in options:\n options = [o for o in options if o is not None]\n if default not in options:\n raise ValueError(f\"Provided default `{default}` is not one of allowed options: {options}\")\n\n json_extra = _make_json_schema_extra(\n description=description,\n parameter_metadata=parameter_metadata,\n enum=options,\n )\n return Field(default=default, description=description, json_schema_extra=json_extra)\n\n\n@DeveloperAPI\ndef Boolean(default: bool, description: str = \"\", parameter_metadata: ParameterMetadata = None):\n \"\"\"Returns a pydantic Field for boolean values.\"\"\"\n if default is not None and not isinstance(default, bool):\n raise ValueError(f\"Invalid default: `{default}`\")\n\n json_extra = _make_json_schema_extra(description=description, parameter_metadata=parameter_metadata)\n return Field(default=default, description=description, json_schema_extra=json_extra)\n\n\n@DeveloperAPI\ndef Integer(\n default: None | int,\n allow_none=False,\n description=\"\",\n parameter_metadata: ParameterMetadata = None,\n):\n \"\"\"Returns a pydantic Field strictly enforcing integer inputs.\"\"\"\n if default is not None and not isinstance(default, int):\n raise ValueError(f\"Invalid default: `{default}`\")\n\n extra_kwargs = {}\n if allow_none:\n extra_kwargs[\"allow_none\"] = True\n json_extra = _make_json_schema_extra(description=description, parameter_metadata=parameter_metadata, **extra_kwargs)\n return Field(default=default, description=description, json_schema_extra=json_extra)\n\n\n@DeveloperAPI\ndef PositiveInteger(\n description: str,\n default: None | int,\n allow_none: bool = False,\n parameter_metadata: ParameterMetadata = None,\n):\n \"\"\"Returns a pydantic Field enforcing positive integer inputs (>= 1).\"\"\"\n if default is not None:\n if not isinstance(default, int) or default < 1:\n raise ValueError(f\"Invalid default: `{default}`\")\n\n extra_kwargs = {}\n if allow_none:\n extra_kwargs[\"allow_none\"] = True\n json_extra = _make_json_schema_extra(description=description, parameter_metadata=parameter_metadata, **extra_kwargs)\n return Field(default=default, ge=1, description=description, json_schema_extra=json_extra)\n\n\n@DeveloperAPI\ndef NonNegativeInteger(\n description: str,\n default: None | int,\n allow_none: bool = False,\n parameter_metadata: ParameterMetadata = None,\n):\n \"\"\"Returns a pydantic Field enforcing nonnegative integer inputs (>= 0).\"\"\"\n if default is not None:\n if not isinstance(default, int) or default < 0:\n raise ValueError(f\"Invalid default: `{default}`\")\n\n extra_kwargs = {}\n if allow_none:\n extra_kwargs[\"allow_none\"] = True\n json_extra = _make_json_schema_extra(description=description, parameter_metadata=parameter_metadata, **extra_kwargs)\n return Field(default=default, ge=0, description=description, json_schema_extra=json_extra)\n\n\n@DeveloperAPI\ndef IntegerRange(\n description: str,\n default: None | int,\n allow_none: bool = False,\n parameter_metadata: ParameterMetadata = None,\n min: int = None,\n max: int = None,\n min_inclusive: bool = True,\n max_inclusive: bool = True,\n):\n \"\"\"Returns a pydantic Field enforcing integer inputs within a range.\"\"\"\n if default is not None:\n if not isinstance(default, int):\n raise ValueError(f\"Invalid default: `{default}`\")\n if min is not None and ((min_inclusive and default < min) or (not min_inclusive and default <= min)):\n raise ValueError(f\"Invalid default: `{default}` (below min {min})\")\n if max is not None and ((max_inclusive and default > max) or (not max_inclusive and default >= max)):\n raise ValueError(f\"Invalid default: `{default}` (above max {max})\")\n\n kwargs = {}\n if min is not None:\n kwargs[\"ge\" if min_inclusive else \"gt\"] = min\n if max is not None:\n kwargs[\"le\" if max_inclusive else \"lt\"] = max\n\n extra_kwargs = {}\n if allow_none:\n extra_kwargs[\"allow_none\"] = True\n json_extra = _make_json_schema_extra(description=description, parameter_metadata=parameter_metadata, **extra_kwargs)\n return Field(default=default, description=description, json_schema_extra=json_extra, **kwargs)\n\n\n@DeveloperAPI\ndef Float(\n default: None | float | int,\n allow_none=False,\n description=\"\",\n parameter_metadata: ParameterMetadata = None,\n):\n \"\"\"Returns a pydantic Field for float inputs.\"\"\"\n if default is not None and not isinstance(default, (float, int)):\n raise ValueError(f\"Invalid default: `{default}`\")\n\n extra_kwargs = {}\n if allow_none:\n extra_kwargs[\"allow_none\"] = True\n json_extra = _make_json_schema_extra(description=description, parameter_metadata=parameter_metadata, **extra_kwargs)\n return Field(default=default, description=description, json_schema_extra=json_extra)\n\n\n@DeveloperAPI\ndef NonNegativeFloat(\n default: None | float,\n allow_none: bool = False,\n description: str = \"\",\n max: float | None = None,\n parameter_metadata: ParameterMetadata = None,\n):\n \"\"\"Returns a pydantic Field enforcing nonnegative float inputs.\"\"\"\n if default is not None:\n if not isinstance(default, (float, int)) or default < 0:\n raise ValueError(f\"Invalid default: `{default}`\")\n if max is not None and default > max:\n raise ValueError(f\"Invalid default: `{default}` (above max {max})\")\n\n kwargs = {\"ge\": 0.0}\n if max is not None:\n kwargs[\"le\"] = max\n\n extra_kwargs = {}\n if allow_none:\n extra_kwargs[\"allow_none\"] = True\n json_extra = _make_json_schema_extra(description=description, parameter_metadata=parameter_metadata, **extra_kwargs)\n return Field(default=default, description=description, json_schema_extra=json_extra, **kwargs)\n\n\n@DeveloperAPI\ndef FloatRange(\n default: None | float,\n allow_none: bool = False,\n description: str = \"\",\n parameter_metadata: ParameterMetadata = None,\n min: int = None,\n max: int = None,\n min_inclusive: bool = True,\n max_inclusive: bool = True,\n):\n \"\"\"Returns a pydantic Field enforcing float inputs within a range.\"\"\"\n if default is not None:\n if not isinstance(default, (float, int)):\n raise ValueError(f\"Invalid default: `{default}`\")\n\n kwargs = {}\n if min is not None:\n kwargs[\"ge\" if min_inclusive else \"gt\"] = min\n if max is not None:\n kwargs[\"le\" if max_inclusive else \"lt\"] = max\n\n extra_kwargs = {}\n if allow_none:\n extra_kwargs[\"allow_none\"] = True\n json_extra = _make_json_schema_extra(description=description, parameter_metadata=parameter_metadata, **extra_kwargs)\n return Field(default=default, description=description, json_schema_extra=json_extra, **kwargs)\n\n\n@DeveloperAPI\ndef Dict(\n default: None | dict = None,\n allow_none: bool = True,\n description: str = \"\",\n parameter_metadata: ParameterMetadata = None,\n):\n \"\"\"Returns a pydantic Field for dict values.\"\"\"\n allow_none = allow_none or default is None\n\n if default is not None:\n if not isinstance(default, dict):\n raise ValueError(f\"Invalid default: `{default}`\")\n if not all(isinstance(k, str) for k in default.keys()):\n raise ValueError(f\"Invalid default: `{default}` (non-string keys)\")\n elif not allow_none:\n default = {}\n\n json_extra = _make_json_schema_extra(description=description, parameter_metadata=parameter_metadata)\n\n if default is None:\n return Field(default=None, description=description, json_schema_extra=json_extra)\n return Field(default_factory=lambda: copy.deepcopy(default), description=description, json_schema_extra=json_extra)\n\n\n@DeveloperAPI\ndef List(\n list_type: type[str] | type[int] | type[float] | type[list] = str,\n inner_type: type[str] | type[int] | type[float] | type[dict] = float,\n default: None | list[Any] = None,\n allow_none: bool = True,\n description: str = \"\",\n parameter_metadata: ParameterMetadata = None,\n):\n \"\"\"Returns a pydantic Field for list values.\"\"\"\n if default is not None:\n if not isinstance(default, list):\n raise ValueError(f\"Invalid default: `{default}`\")\n elif not allow_none:\n default = []\n\n json_extra = _make_json_schema_extra(description=description, parameter_metadata=parameter_metadata)\n\n if default is None:\n return Field(default=None, description=description, json_schema_extra=json_extra)\n return Field(default_factory=lambda: copy.deepcopy(default), description=description, json_schema_extra=json_extra)\n\n\n@DeveloperAPI\ndef DictList(\n default: None | list[dict] = None,\n allow_none: bool = True,\n description: str = \"\",\n parameter_metadata: ParameterMetadata = None,\n):\n \"\"\"Returns a pydantic Field for list-of-dicts values.\"\"\"\n if default is not None:\n if not isinstance(default, list) or not all(isinstance(d, dict) for d in default):\n raise ValueError(f\"Invalid default: `{default}`\")\n elif not allow_none:\n default = []\n\n json_extra = _make_json_schema_extra(description=description, parameter_metadata=parameter_metadata)\n\n if default is None:\n return Field(default=None, description=description, json_schema_extra=json_extra)\n return Field(default_factory=lambda: copy.deepcopy(default), description=description, json_schema_extra=json_extra)\n\n\n@DeveloperAPI\ndef Embed(description: str = \"\", parameter_metadata: ParameterMetadata = None):\n \"\"\"Returns a pydantic Field for embedding input feature names (int, str, or None).\"\"\"\n _embed_options = [\"add\"]\n json_extra = _make_json_schema_extra(\n description=description,\n parameter_metadata=parameter_metadata,\n _embed_options=_embed_options,\n )\n return Field(default=None, description=description, json_schema_extra=json_extra)\n\n\n@DeveloperAPI\ndef InitializerOrDict(\n default: str = \"xavier_uniform\", description: str = \"\", parameter_metadata: ParameterMetadata = None\n):\n \"\"\"Returns a pydantic Field allowing str or dict initializer values.\"\"\"\n initializers = list(initializer_registry.keys())\n if not isinstance(default, str) or default not in initializers:\n raise ValueError(f\"Invalid default: `{default}`\")\n\n json_extra = _make_json_schema_extra(\n description=description,\n parameter_metadata=parameter_metadata,\n _initializer_options=initializers,\n )\n return Field(default=default, description=description, json_schema_extra=json_extra)\n\n\n@DeveloperAPI\ndef FloatRangeTupleDataclassField(\n n: int = 2,\n default: tuple | None = (0.9, 0.999),\n allow_none: bool = False,\n min: int | None = 0,\n max: int | None = 1,\n description: str = \"\",\n parameter_metadata: ParameterMetadata = None,\n):\n \"\"\"Returns a pydantic Field for an N-dim tuple with values in a range.\"\"\"\n if default is not None:\n if n != len(default):\n raise ValueError(f\"Dimension of tuple '{n}' must match dimension of default val. '{default}'\")\n for v in default:\n if min is not None and v < min:\n raise ValueError(f\"Invalid default: value {v} below minimum {min}\")\n if max is not None and v > max:\n raise ValueError(f\"Invalid default: value {v} above maximum {max}\")\n if default is None and not allow_none:\n raise ValueError(\"Default value must not be None if allow_none is False\")\n\n extra_kwargs = {}\n if allow_none:\n extra_kwargs[\"allow_none\"] = True\n json_extra = _make_json_schema_extra(\n description=description,\n parameter_metadata=parameter_metadata,\n _float_tuple_range={\"n\": n, \"min\": min, \"max\": max},\n **extra_kwargs,\n )\n return Field(default=default, description=description, json_schema_extra=json_extra)\n\n\n@DeveloperAPI\ndef OneOfOptionsField(\n default: Any,\n description: str,\n field_options: list,\n allow_none: bool = False,\n parameter_metadata: ParameterMetadata = None,\n):\n \"\"\"Returns a pydantic Field that accepts values matching any of the field_options.\n\n Pydantic union validation handles the multi-type dispatch. The field_options are stored in json_schema_extra for\n JSON schema generation.\n \"\"\"\n extra_kwargs = {}\n if allow_none:\n extra_kwargs[\"allow_none\"] = True\n json_extra = _make_json_schema_extra(\n description=description,\n parameter_metadata=parameter_metadata,\n _oneof_options=True,\n **extra_kwargs,\n )\n\n if default is None or isinstance(default, (int, str, bool)):\n return Field(default=default, description=description, json_schema_extra=json_extra)\n return Field(default_factory=lambda: copy.deepcopy(default), description=description, json_schema_extra=json_extra)\n\n\n# ============================================================================\n# TypeSelection - Polymorphic config dispatch based on registry\n# ============================================================================\n\n\nclass TypeSelection(LudwigSchemaField):\n \"\"\"Resolves polymorphic config types from a registry based on a key field.\n\n Used for fields like encoder, decoder, optimizer where the config class depends on a \"type\" key in the dict value.\n \"\"\"\n\n def __init__(\n self,\n registry: Registry,\n default_value: str | None = None,\n key: str = \"type\",\n description: str = \"\",\n parameter_metadata: ParameterMetadata = None,\n allow_str_value: bool = False,\n allow_none: bool = False,\n **kwargs,\n ):\n self.registry = registry\n self.default_value = default_value\n self.key = key\n self.allow_str_value = allow_str_value\n self.allow_none = allow_none\n self.description = description\n self.parameter_metadata = parameter_metadata\n\n def _deserialize(self, value, attr, data, **kwargs):\n \"\"\"Marshmallow deserialization - delegates to resolve().\"\"\"\n return self.resolve(value)\n\n def resolve(self, value):\n \"\"\"Resolve a raw value (dict, str, None) to a config instance.\"\"\"\n if value is None:\n if self.allow_none:\n return None\n return None\n\n # Already a config instance\n if isinstance(value, BaseMarshmallowConfig):\n return value\n\n if self.allow_str_value and isinstance(value, str):\n value = self.str_value_to_object(value)\n\n if isinstance(value, dict):\n cls_type = value.get(self.key)\n cls_type = cls_type.lower() if cls_type else self.default_value\n if cls_type and cls_type in self.registry:\n cls = self.get_schema_from_registry(cls_type)\n try:\n return cls.model_validate(value)\n except (TypeError, PydanticValidationError) as e:\n raise ConfigValidationError(f\"Invalid params: {value}, see `{cls}` definition\") from e\n raise ConfigValidationError(f\"Invalid type: '{cls_type}', expected one of: {list(self.registry.keys())}\")\n\n maybe_str = \", `str`,\" if self.allow_str_value else \"\"\n raise ConfigValidationError(f\"Invalid param {value}, expected `None`{maybe_str} or `dict`\")\n\n def str_value_to_object(self, value: str) -> dict:\n \"\"\"Convert a string shorthand to a dict with the type key.\"\"\"\n return {self.key: value}\n\n def get_schema_from_registry(self, key: str) -> type[BaseMarshmallowConfig]:\n \"\"\"Look up a config class from the registry.\"\"\"\n return self.registry[key]\n\n def get_default_field(self) -> FieldInfo:\n \"\"\"Create a pydantic Field wrapping this TypeSelection.\n\n The TypeSelection instance is stored in Field.metadata so the base class's model_validator can use it for\n dispatch.\n \"\"\"\n if self.default_value is not None:\n cls = self.get_schema_from_registry(self.default_value.lower())\n key = self.key\n dv = self.default_value\n\n def default_factory(cls=cls, key=key, dv=dv):\n return cls.model_validate({key: dv})\n\n else:\n\n def default_factory():\n return None\n\n fi = Field(default_factory=default_factory)\n fi.metadata = [_TypeSelectionMarker(self)]\n return fi\n\n def _jsonschema_type_mapping(self):\n \"\"\"Override in subclass for custom JSON schema.\"\"\"\n return None\n\n\n@DeveloperAPI\nclass DictMarshmallowField(LudwigSchemaField):\n \"\"\"Validates a dict as a specific config class (non-polymorphic).\n\n Used for fields where a dict should be deserialized into a fixed config class.\n \"\"\"\n\n def __init__(\n self,\n cls: type[BaseMarshmallowConfig],\n allow_none: bool = True,\n default_missing: bool = False,\n description: str = \"\",\n **kwargs,\n ):\n self.cls = cls\n self.allow_none = allow_none\n self.default_missing = default_missing\n self.description = description\n\n def _deserialize(self, value, attr, data, **kwargs):\n \"\"\"Deserialize a dict to a config instance via pydantic model_validate.\"\"\"\n if value is None:\n return value\n if isinstance(value, dict):\n try:\n return self.cls.model_validate(value)\n except (TypeError, PydanticValidationError) as e:\n raise ConfigValidationError(f\"Invalid params: {value}, see `{self.cls}` definition\") from e\n raise ConfigValidationError(\"Field should be None or dict\")\n\n def get_default_field(self) -> FieldInfo:\n \"\"\"Create a pydantic Field wrapping this DictMarshmallowField.\"\"\"\n if not self.default_missing:\n cls = self.cls\n\n def default_factory(cls=cls):\n return cls.model_validate({})\n\n else:\n\n def default_factory():\n return None\n\n # Check if subclass overrides _jsonschema_type_mapping - if so, use\n # MarshmallowFieldMarker to preserve custom JSON schema generation\n has_custom_schema = type(self)._jsonschema_type_mapping is not DictMarshmallowField._jsonschema_type_mapping\n if has_custom_schema:\n marker = _MarshmallowFieldMarker(self)\n else:\n marker = _NestedConfigMarker(self.cls, self.allow_none)\n\n fi = Field(default_factory=default_factory)\n fi.metadata = [marker]\n return fi\n\n def _jsonschema_type_mapping(self):\n return unload_jsonschema_from_marshmallow_class(self.cls)\n\n\n# Backward compatibility aliases\nValidationError = ConfigValidationError\nNestedConfigField = DictMarshmallowField\nLudwigConfig = BaseMarshmallowConfig\nunload_jsonschema_from_config_class = unload_jsonschema_from_marshmallow_class\n"
},
{
"path": "ludwig/serve.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport argparse\nimport io\nimport json\nimport logging\nimport os\nimport sys\nimport tempfile\n\nimport pandas as pd\nimport torch\nfrom torchvision.io import decode_image\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.constants import AUDIO, COLUMN\nfrom ludwig.contrib import add_contrib_callback_args\nfrom ludwig.globals import LUDWIG_VERSION\nfrom ludwig.utils.print_utils import get_logging_level_registry, print_ludwig\nfrom ludwig.utils.server_utils import NumpyJSONResponse\n\nlogger = logging.getLogger(__name__)\n\ntry:\n import uvicorn\n from fastapi import FastAPI\n from starlette.datastructures import UploadFile\n from starlette.middleware import Middleware\n from starlette.middleware.cors import CORSMiddleware\n from starlette.requests import Request\nexcept ImportError as e:\n logger.error(e)\n logger.error(\n \" fastapi and other serving dependencies cannot be loaded\"\n \"and may have not been installed. \"\n \"In order to install all serving dependencies run \"\n \"pip install ludwig[serve]\"\n )\n sys.exit(-1)\n\nALL_FEATURES_PRESENT_ERROR = {\"error\": \"entry must contain all input features\"}\n\nCOULD_NOT_RUN_INFERENCE_ERROR = {\"error\": \"Unexpected Error: could not run inference on model\"}\n\n\ndef server(model, allowed_origins=None):\n middleware = [Middleware(CORSMiddleware, allow_origins=allowed_origins)] if allowed_origins else None\n app = FastAPI(middleware=middleware)\n\n config = model.config\n input_features = {f[COLUMN] for f in config[\"input_features\"]}\n\n @app.get(\"/\")\n def check_health():\n return NumpyJSONResponse({\"message\": \"Ludwig server is up\"})\n\n @app.post(\"/predict\")\n async def predict(request: Request):\n try:\n form = await request.form()\n entry, files = convert_input(form, model.model.input_features)\n except Exception:\n logger.exception(\"Failed to parse predict form\")\n return NumpyJSONResponse(COULD_NOT_RUN_INFERENCE_ERROR, status_code=500)\n\n try:\n if (entry.keys() & input_features) != input_features:\n missing_features = set(input_features) - set(entry.keys())\n return NumpyJSONResponse(\n {\n \"error\": \"Data received does not contain all input features. \"\n f\"Missing features: {missing_features}.\"\n },\n status_code=400,\n )\n try:\n resp, _ = model.predict(dataset=[entry], data_format=dict)\n resp = resp.to_dict(\"records\")[0]\n return NumpyJSONResponse(resp)\n except Exception as exc:\n logger.exception(f\"Failed to run predict: {exc}\")\n return NumpyJSONResponse(COULD_NOT_RUN_INFERENCE_ERROR, status_code=500)\n finally:\n for f in files:\n os.remove(f.name)\n\n @app.post(\"/batch_predict\")\n async def batch_predict(request: Request):\n try:\n form = await request.form()\n data, files = convert_batch_input(form, model.model.input_features)\n data_df = pd.DataFrame.from_records(data[\"data\"], index=data.get(\"index\"), columns=data[\"columns\"])\n except Exception:\n logger.exception(\"Failed to parse batch_predict form\")\n return NumpyJSONResponse(COULD_NOT_RUN_INFERENCE_ERROR, status_code=500)\n\n if (set(data_df.columns) & input_features) != input_features:\n missing_features = set(input_features) - set(data_df.columns)\n return NumpyJSONResponse(\n {\n \"error\": \"Data received does not contain all input features. \"\n f\"Missing features: {missing_features}.\"\n },\n status_code=400,\n )\n try:\n resp, _ = model.predict(dataset=data_df)\n resp = resp.to_dict(\"split\")\n return NumpyJSONResponse(resp)\n except Exception:\n logger.exception(\"Failed to run batch_predict: {}\")\n return NumpyJSONResponse(COULD_NOT_RUN_INFERENCE_ERROR, status_code=500)\n\n return app\n\n\ndef _write_file(v, files):\n # Convert UploadFile to a NamedTemporaryFile to ensure it's on the disk\n suffix = os.path.splitext(v.filename)[1]\n named_file = tempfile.NamedTemporaryFile(delete=False, suffix=suffix)\n files.append(named_file)\n named_file.write(v.file.read())\n named_file.close()\n return named_file.name\n\n\ndef _read_image_buffer(v):\n # read bytes sent via REST API and convert to image tensor\n # in [channels, height, width] format\n byte_string = io.BytesIO(v.file.read()).read()\n image = decode_image(torch.frombuffer(byte_string, dtype=torch.uint8))\n return image # channels, height, width\n\n\ndef convert_input(form, input_features):\n \"\"\"Returns a new input and a list of files to be cleaned up.\"\"\"\n new_input = {}\n files = []\n for k, v in form.multi_items():\n if isinstance(v, UploadFile):\n # check if audio or image file\n if input_features.get(k).type() == AUDIO:\n new_input[k] = _write_file(v, files)\n else:\n new_input[k] = _read_image_buffer(v)\n else:\n new_input[k] = v\n\n return new_input, files\n\n\ndef convert_batch_input(form, input_features):\n \"\"\"Returns a new input and a list of files to be cleaned up.\"\"\"\n file_index = {}\n files = []\n for k, v in form.multi_items():\n if isinstance(v, UploadFile):\n file_index[v.filename] = v\n\n data = json.loads(form[\"dataset\"])\n for row in data[\"data\"]:\n for i, value in enumerate(row):\n if value in file_index:\n feature_name = data[\"columns\"][i]\n if input_features.get(feature_name).type() == AUDIO:\n row[i] = _write_file(file_index[value], files)\n else:\n row[i] = _read_image_buffer(file_index[value])\n\n return data, files\n\n\ndef run_server(\n model_path: str,\n host: str,\n port: int,\n allowed_origins: list,\n) -> None:\n \"\"\"Loads a pre-trained model and serve it on an http server.\n\n # Inputs\n\n :param model_path: (str) filepath to pre-trained model.\n :param host: (str, default: `0.0.0.0`) host ip address for the server to use.\n :param port: (int, default: `8000`) port number for the server to use.\n :param allowed_origins: (list) list of origins allowed to make cross-origin requests.\n\n # Return\n\n :return: (`None`)\n \"\"\"\n # Use local backend for serving to use pandas DataFrames.\n model = LudwigModel.load(model_path, backend=\"local\")\n app = server(model, allowed_origins)\n uvicorn.run(app, host=host, port=port)\n\n\ndef cli(sys_argv):\n parser = argparse.ArgumentParser(\n description=\"This script serves a pretrained model\", prog=\"ludwig serve\", usage=\"%(prog)s [options]\"\n )\n\n # ----------------\n # Model parameters\n # ----------------\n parser.add_argument(\"-m\", \"--model_path\", help=\"model to load\", required=True)\n\n parser.add_argument(\n \"-l\",\n \"--logging_level\",\n default=\"info\",\n help=\"the level of logging to use\",\n choices=[\"critical\", \"error\", \"warning\", \"info\", \"debug\", \"notset\"],\n )\n\n # ----------------\n # Server parameters\n # ----------------\n parser.add_argument(\n \"-p\",\n \"--port\",\n help=\"port for server (default: 8000)\",\n default=8000,\n type=int,\n )\n\n parser.add_argument(\"-H\", \"--host\", help=\"host for server (default: 0.0.0.0)\", default=\"0.0.0.0\")\n\n parser.add_argument(\n \"-ao\",\n \"--allowed_origins\",\n nargs=\"*\",\n help=\"A list of origins that should be permitted to make cross-origin requests. \"\n 'Use \"*\" to allow any origin. See https://www.starlette.io/middleware/#corsmiddleware.',\n )\n\n add_contrib_callback_args(parser)\n args = parser.parse_args(sys_argv)\n\n args.callbacks = args.callbacks or []\n for callback in args.callbacks:\n callback.on_cmdline(\"serve\", *sys_argv)\n\n args.logging_level = get_logging_level_registry()[args.logging_level]\n logging.getLogger(\"ludwig\").setLevel(args.logging_level)\n global logger\n logger = logging.getLogger(\"ludwig.serve\")\n\n print_ludwig(\"Serve\", LUDWIG_VERSION)\n\n run_server(args.model_path, args.host, args.port, args.allowed_origins)\n\n\nif __name__ == \"__main__\":\n cli(sys.argv[1:])\n"
},
{
"path": "ludwig/train.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport argparse\nimport logging\nimport sys\n\nimport pandas as pd\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.backend import ALL_BACKENDS, Backend, initialize_backend\nfrom ludwig.callbacks import Callback\nfrom ludwig.constants import CONTINUE_PROMPT, HYPEROPT, HYPEROPT_WARNING\nfrom ludwig.contrib import add_contrib_callback_args\nfrom ludwig.globals import LUDWIG_VERSION\nfrom ludwig.utils.data_utils import load_config_from_str, load_yaml\nfrom ludwig.utils.defaults import default_random_seed\nfrom ludwig.utils.print_utils import get_logging_level_registry, print_ludwig, query_yes_no\n\nlogger = logging.getLogger(__name__)\n\n\ndef train_cli(\n config: str | dict = None,\n dataset: str | dict | pd.DataFrame = None,\n training_set: str | dict | pd.DataFrame = None,\n validation_set: str | dict | pd.DataFrame = None,\n test_set: str | dict | pd.DataFrame = None,\n training_set_metadata: str | dict = None,\n data_format: str = None,\n experiment_name: str = \"api_experiment\",\n model_name: str = \"run\",\n model_load_path: str = None,\n model_resume_path: str = None,\n skip_save_training_description: bool = False,\n skip_save_training_statistics: bool = False,\n skip_save_model: bool = False,\n skip_save_progress: bool = False,\n skip_save_log: bool = False,\n skip_save_processed_input: bool = False,\n output_directory: str = \"results\",\n gpus: str | int | list[int] = None,\n gpu_memory_limit: float | None = None,\n allow_parallel_threads: bool = True,\n callbacks: list[Callback] = None,\n backend: Backend | str = None,\n random_seed: int = default_random_seed,\n logging_level: int = logging.INFO,\n **kwargs\n) -> None:\n \"\"\"*train* defines the entire training procedure used by Ludwig's internals. Requires most of the parameters\n that are taken into the model. Builds a full ludwig model and performs the training.\n\n :param config: (Union[str, dict]) in-memory representation of\n config or string path to a YAML config file.\n :param dataset: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing the entire dataset to be used for training.\n If it has a split column, it will be used for splitting (0 for train,\n 1 for validation, 2 for test), otherwise the dataset will be\n randomly split.\n :param training_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing training data.\n :param validation_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing validation data.\n :param test_set: (Union[str, dict, pandas.DataFrame], default: `None`)\n source containing test data.\n :param training_set_metadata: (Union[str, dict], default: `None`)\n metadata JSON file or loaded metadata. Intermediate preprocessed\n structure containing the mappings of the input\n dataset created the first time an input file is used in the same\n directory with the same name and a '.meta.json' extension.\n :param data_format: (str, default: `None`) format to interpret data\n sources. Will be inferred automatically if not specified. Valid\n formats are `'auto'`, `'csv'`, `'excel'`, `'feather'`,\n `'fwf'`, `'hdf5'` (cache file produced during previous training),\n `'html'` (file containing a single HTML `
`), `'json'`, `'jsonl'`,\n `'parquet'`, `'pickle'` (pickled Pandas DataFrame), `'sas'`, `'spss'`,\n `'stata'`, `'tsv'`.\n :param experiment_name: (str, default: `'experiment'`) name for\n the experiment.\n :param model_name: (str, default: `'run'`) name of the model that is\n being used.\n :param model_load_path: (str, default: `None`) if this is specified the\n loaded model will be used as initialization\n (useful for transfer learning).\n :param model_resume_path: (str, default: `None`) resumes training of\n the model from the path specified. The config is restored.\n In addition to config, training statistics, loss for each\n epoch and the state of the optimizer are restored such that\n training can be effectively continued from a previously interrupted\n training process.\n :param skip_save_training_description: (bool, default: `False`) disables\n saving the description JSON file.\n :param skip_save_training_statistics: (bool, default: `False`) disables\n saving training statistics JSON file.\n :param skip_save_model: (bool, default: `False`) disables\n saving model weights and hyperparameters each time the model\n improves. By default Ludwig saves model weights after each epoch\n the validation metric improves, but if the model is really big\n that can be time consuming. If you do not want to keep\n the weights and just find out what performance a model can get\n with a set of hyperparameters, use this parameter to skip it,\n but the model will not be loadable later on and the returned model\n will have the weights obtained at the end of training, instead of\n the weights of the epoch with the best validation performance.\n :param skip_save_progress: (bool, default: `False`) disables saving\n progress each epoch. By default Ludwig saves weights and stats\n after each epoch for enabling resuming of training, but if\n the model is really big that can be time consuming and will uses\n twice as much space, use this parameter to skip it, but training\n cannot be resumed later on.\n :param skip_save_log: (bool, default: `False`) disables saving\n TensorBoard logs. By default Ludwig saves logs for the TensorBoard,\n but if it is not needed turning it off can slightly increase the\n overall speed.\n :param skip_save_processed_input: (bool, default: `False`) if input\n dataset is provided it is preprocessed and cached by saving an HDF5\n and JSON files to avoid running the preprocessing again. If this\n parameter is `False`, the HDF5 and JSON file are not saved.\n :param output_directory: (str, default: `'results'`) the directory that\n will contain the training statistics, TensorBoard logs, the saved\n model and the training progress files.\n :param gpus: (list, default: `None`) list of GPUs that are available\n for training.\n :param gpu_memory_limit: (float: default: `None`) maximum memory fraction\n [0, 1] allowed to allocate per GPU device.\n :param allow_parallel_threads: (bool, default: `True`) allow PyTorch\n to use multithreading parallelism to improve performance at\n the cost of determinism.\n :param callbacks: (list, default: `None`) a list of\n `ludwig.callbacks.Callback` objects that provide hooks into the\n Ludwig pipeline.\n :param backend: (Union[Backend, str]) `Backend` or string name\n of backend to use to execute preprocessing / training steps.\n :param random_seed: (int: default: 42) random seed used for weights\n initialization, splits and any other random function.\n :param logging_level: (int) Log level that will be sent to stderr.\n\n # Return\n\n :return: (`None`)\n \"\"\"\n if HYPEROPT in config:\n if not query_yes_no(HYPEROPT_WARNING + CONTINUE_PROMPT):\n exit(1)\n # Stop gap: remove hyperopt from the config to prevent interference with training step sizes\n # TODO: https://github.com/ludwig-ai/ludwig/issues/2633\n # Need to investigate why the presence of hyperopt in the config interferes with training step sizes\n config.pop(HYPEROPT)\n\n if model_load_path:\n model = LudwigModel.load(\n model_load_path,\n logging_level=logging_level,\n backend=backend,\n gpus=gpus,\n gpu_memory_limit=gpu_memory_limit,\n allow_parallel_threads=allow_parallel_threads,\n callbacks=callbacks,\n )\n else:\n model = LudwigModel(\n config=config,\n logging_level=logging_level,\n backend=backend,\n gpus=gpus,\n gpu_memory_limit=gpu_memory_limit,\n allow_parallel_threads=allow_parallel_threads,\n callbacks=callbacks,\n )\n model.train(\n dataset=dataset,\n training_set=training_set,\n validation_set=validation_set,\n test_set=test_set,\n training_set_metadata=training_set_metadata,\n data_format=data_format,\n experiment_name=experiment_name,\n model_name=model_name,\n model_resume_path=model_resume_path,\n skip_save_training_description=skip_save_training_description,\n skip_save_training_statistics=skip_save_training_statistics,\n skip_save_model=skip_save_model,\n skip_save_progress=skip_save_progress,\n skip_save_log=skip_save_log,\n skip_save_processed_input=skip_save_processed_input,\n output_directory=output_directory,\n random_seed=random_seed,\n )\n\n\ndef cli(sys_argv):\n parser = argparse.ArgumentParser(\n description=\"This script trains a model\", prog=\"ludwig train\", usage=\"%(prog)s [options]\"\n )\n\n # ----------------------------\n # Experiment naming parameters\n # ----------------------------\n parser.add_argument(\"--output_directory\", type=str, default=\"results\", help=\"directory that contains the results\")\n parser.add_argument(\"--experiment_name\", type=str, default=\"experiment\", help=\"experiment name\")\n parser.add_argument(\"--model_name\", type=str, default=\"run\", help=\"name for the model\")\n\n # ---------------\n # Data parameters\n # ---------------\n parser.add_argument(\n \"--dataset\",\n help=\"input data file path. \"\n \"If it has a split column, it will be used for splitting \"\n \"(0: train, 1: validation, 2: test), \"\n \"otherwise the dataset will be randomly split\",\n )\n parser.add_argument(\"--training_set\", help=\"input train data file path\")\n parser.add_argument(\"--validation_set\", help=\"input validation data file path\")\n parser.add_argument(\"--test_set\", help=\"input test data file path\")\n\n parser.add_argument(\n \"--training_set_metadata\",\n help=\"input metadata JSON file path. An intermediate preprocessed file \"\n \"containing the mappings of the input file created \"\n \"the first time a file is used, in the same directory \"\n \"with the same name and a .json extension\",\n )\n\n parser.add_argument(\n \"--data_format\",\n help=\"format of the input data\",\n default=\"auto\",\n choices=[\n \"auto\",\n \"csv\",\n \"excel\",\n \"feather\",\n \"fwf\",\n \"hdf5\",\n \"html\" \"tables\",\n \"json\",\n \"jsonl\",\n \"parquet\",\n \"pickle\",\n \"sas\",\n \"spss\",\n \"stata\",\n \"tsv\",\n ],\n )\n\n parser.add_argument(\n \"-sspi\",\n \"--skip_save_processed_input\",\n help=\"skips saving intermediate HDF5 and JSON files\",\n action=\"store_true\",\n default=False,\n )\n\n # ----------------\n # Model parameters\n # ----------------\n config = parser.add_mutually_exclusive_group(required=True)\n config.add_argument(\n \"-c\",\n \"--config\",\n type=load_yaml,\n help=\"Path to the YAML file containing the model configuration\",\n )\n config.add_argument(\n \"-cs\",\n \"--config_str\",\n dest=\"config\",\n type=load_config_from_str,\n help=\"JSON or YAML serialized string of the model configuration\",\n )\n\n parser.add_argument(\"-mlp\", \"--model_load_path\", help=\"path of a pretrained model to load as initialization\")\n parser.add_argument(\"-mrp\", \"--model_resume_path\", help=\"path of the model directory to resume training of\")\n parser.add_argument(\n \"-sstd\",\n \"--skip_save_training_description\",\n action=\"store_true\",\n default=False,\n help=\"disables saving the description JSON file\",\n )\n parser.add_argument(\n \"-ssts\",\n \"--skip_save_training_statistics\",\n action=\"store_true\",\n default=False,\n help=\"disables saving training statistics JSON file\",\n )\n parser.add_argument(\n \"-ssm\",\n \"--skip_save_model\",\n action=\"store_true\",\n default=False,\n help=\"disables saving weights each time the model improves. \"\n \"By default Ludwig saves weights after each epoch \"\n \"the validation metric (improves, but if the model is really big \"\n \"that can be time consuming. If you do not want to keep \"\n \"the weights and just find out what performance a model can get \"\n \"with a set of hyperparameters, use this parameter to skip it\",\n )\n parser.add_argument(\n \"-ssp\",\n \"--skip_save_progress\",\n action=\"store_true\",\n default=False,\n help=\"disables saving weights after each epoch. By default ludwig saves \"\n \"weights after each epoch for enabling resuming of training, but \"\n \"if the model is really big that can be time consuming and will \"\n \"save twice as much space, use this parameter to skip it\",\n )\n parser.add_argument(\n \"-ssl\",\n \"--skip_save_log\",\n action=\"store_true\",\n default=False,\n help=\"disables saving TensorBoard logs. By default Ludwig saves \"\n \"logs for the TensorBoard, but if it is not needed turning it off \"\n \"can slightly increase the overall speed\",\n )\n\n # ------------------\n # Runtime parameters\n # ------------------\n parser.add_argument(\n \"-rs\",\n \"--random_seed\",\n type=int,\n default=42,\n help=\"a random seed that is going to be used anywhere there is a call \"\n \"to a random number generator: data splitting, parameter \"\n \"initialization and training set shuffling\",\n )\n parser.add_argument(\"-g\", \"--gpus\", nargs=\"+\", type=int, default=None, help=\"list of gpus to use\")\n parser.add_argument(\n \"-gml\",\n \"--gpu_memory_limit\",\n type=float,\n default=None,\n help=\"maximum memory fraction [0, 1] allowed to allocate per GPU device\",\n )\n parser.add_argument(\n \"-dpt\",\n \"--disable_parallel_threads\",\n action=\"store_false\",\n dest=\"allow_parallel_threads\",\n help=\"disable PyTorch from using multithreading for reproducibility\",\n )\n parser.add_argument(\n \"-b\",\n \"--backend\",\n help=\"specifies backend to use for parallel / distributed execution, \" \"defaults to local execution\",\n choices=ALL_BACKENDS,\n )\n parser.add_argument(\n \"-l\",\n \"--logging_level\",\n default=\"info\",\n help=\"the level of logging to use\",\n choices=[\"critical\", \"error\", \"warning\", \"info\", \"debug\", \"notset\"],\n )\n\n add_contrib_callback_args(parser)\n args = parser.parse_args(sys_argv)\n\n args.callbacks = args.callbacks or []\n for callback in args.callbacks:\n callback.on_cmdline(\"train\", *sys_argv)\n\n args.logging_level = get_logging_level_registry()[args.logging_level]\n logging.getLogger(\"ludwig\").setLevel(args.logging_level)\n global logger\n logger = logging.getLogger(\"ludwig.train\")\n\n args.backend = initialize_backend(args.backend or args.config.get(\"backend\"))\n if args.backend.is_coordinator():\n print_ludwig(\"Train\", LUDWIG_VERSION)\n\n train_cli(**vars(args))\n\n\nif __name__ == \"__main__\":\n cli(sys.argv[1:])\n"
},
{
"path": "ludwig/trainers/__init__.py",
"content": "# register trainers\n\nimport ludwig.trainers.trainer # noqa: F401\n\ntry:\n import ludwig.trainers.trainer_llm # noqa: F401\nexcept ImportError:\n pass\n"
},
{
"path": "ludwig/trainers/base.py",
"content": "from abc import ABC, abstractmethod\n\nfrom ludwig.data.dataset.base import Dataset\nfrom ludwig.globals import MODEL_FILE_NAME\nfrom ludwig.schema.trainer import BaseTrainerConfig\nfrom ludwig.types import ModelConfigDict\nfrom ludwig.utils.defaults import default_random_seed\n\n\nclass BaseTrainer(ABC):\n @abstractmethod\n def train(self, training_set, validation_set=None, test_set=None, save_path=MODEL_FILE_NAME, **kwargs):\n raise NotImplementedError()\n\n @abstractmethod\n def train_online(\n self,\n dataset,\n ):\n raise NotImplementedError()\n\n @abstractmethod\n def tune_batch_size(\n self,\n config: ModelConfigDict,\n training_set: Dataset,\n random_seed: int = default_random_seed,\n max_trials: int = 10,\n halving_limit: int = 3,\n tune_for_training: bool = True,\n ) -> int:\n raise NotImplementedError()\n\n @property\n @abstractmethod\n def validation_field(self):\n raise NotImplementedError()\n\n @property\n @abstractmethod\n def validation_metric(self):\n raise NotImplementedError()\n\n # Remote implementations may override this\n def shutdown(self):\n pass\n\n @property\n def local_rank(self) -> int:\n return 0\n\n def barrier(self):\n pass\n\n # Functions needed to treat Trainer as a context manager\n def __enter__(self):\n return self\n\n def __exit__(self, exc_type, exc_val, exc_tb):\n self.shutdown()\n\n @staticmethod\n @abstractmethod\n def get_schema_cls() -> BaseTrainerConfig:\n raise NotImplementedError()\n"
},
{
"path": "ludwig/trainers/registry.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.utils.registry import DEFAULT_KEYS, Registry\n\n_trainers_registry = Registry()\n_ray_trainers_registry = Registry()\n\n_llm_trainers_registry = Registry()\n_llm_ray_trainers_registry = Registry()\n\n\n@DeveloperAPI\ndef get_trainers_registry() -> Registry:\n return _trainers_registry\n\n\n@DeveloperAPI\ndef get_ray_trainers_registry() -> Registry:\n return _ray_trainers_registry\n\n\n@DeveloperAPI\ndef get_llm_trainers_registry() -> Registry:\n return _llm_trainers_registry\n\n\n@DeveloperAPI\ndef get_llm_ray_trainers_registry() -> Registry:\n return _llm_ray_trainers_registry\n\n\n@DeveloperAPI\ndef register_trainer(model_type: str, default=False):\n \"\"\"Register a trainer class that supports training the given model types.\n\n Using default=True will make the trainer the default trainer for the model type.\n\n Args:\n model_type: The model_type which dictates the trainer type to use.\n default: Whether the trainer should be the default trainer for the model type.\n \"\"\"\n\n def wrap(cls):\n _trainers_registry[model_type] = cls\n if default:\n if DEFAULT_KEYS[0] in _trainers_registry:\n raise ValueError(f\"Default trainer already registered for model type {model_type}\")\n for key in DEFAULT_KEYS:\n _trainers_registry[key] = cls\n return cls\n\n return wrap\n\n\n@DeveloperAPI\ndef register_ray_trainer(model_type: str, default=False):\n \"\"\"Register a trainer class that supports training the given model types with Ray backend.\n\n Using default=True will make the trainer the default trainer for the model type.\n\n Args:\n model_type: The model_type which dictates the trainer type to use.\n default: Whether the trainer should be the default trainer for the model type.\n \"\"\"\n\n def wrap(cls):\n _ray_trainers_registry[model_type] = cls\n if default:\n if DEFAULT_KEYS[0] in _ray_trainers_registry:\n raise ValueError(f\"Default trainer already registered for model type {model_type}\")\n for key in DEFAULT_KEYS:\n _ray_trainers_registry[key] = cls\n return cls\n\n return wrap\n\n\n@DeveloperAPI\ndef register_llm_trainer(trainer_type: str, default=False):\n \"\"\"Register a trainer class that supports training the specific type of training strategy for LLM Models.\n\n Using default=True will make the trainer the default trainer for the LLM model type.\n\n Args:\n trainer_type: The trainer_type which dictates what training strategy to use.\n default: Whether the trainer should be the default trainer for LLMs.\n \"\"\"\n\n def wrap(cls):\n _llm_trainers_registry[trainer_type] = cls\n if default:\n if DEFAULT_KEYS[0] in _trainers_registry:\n raise ValueError(f\"Default trainer {trainer_type} already registered for LLM\")\n for key in DEFAULT_KEYS:\n _llm_trainers_registry[key] = cls\n return cls\n\n return wrap\n\n\n@DeveloperAPI\ndef register_llm_ray_trainer(trainer_type: str, default=False):\n \"\"\"Register a trainer class that supports training the specific type of training strategy for LLM Models with\n Ray backend.\n\n Using default=True will make the trainer the default trainer for the LLM model type.\n\n Args:\n trainer_type: The trainer_type which dictates what training strategy to use.\n default: Whether the trainer should be the default trainer for LLMs.\n \"\"\"\n\n def wrap(cls):\n _llm_ray_trainers_registry[trainer_type] = cls\n if default:\n if DEFAULT_KEYS[0] in _trainers_registry:\n raise ValueError(f\"Default ray trainer {trainer_type} already registered for LLM\")\n for key in DEFAULT_KEYS:\n _llm_ray_trainers_registry[key] = cls\n return cls\n\n return wrap\n"
},
{
"path": "ludwig/trainers/trainer.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\"\"\"This module contains the class and auxiliary methods of a model.\"\"\"\n\nimport contextlib\nimport csv\nimport logging\nimport math\nimport os\nimport os.path\nimport signal\nimport sys\nimport tempfile\nimport threading\nimport time\nfrom collections.abc import Callable\n\nimport numpy as np\nimport packaging\nimport pandas as pd\nimport psutil\nimport torch\nfrom torch.utils.tensorboard import SummaryWriter\n\nfrom ludwig.constants import (\n AUTO,\n LOSS,\n MAX_CPU_BATCH_SIZE,\n MINIMIZE,\n MODEL_ECD,\n MODEL_LLM,\n TEST,\n TRAINING,\n USED_TOKENS,\n VALIDATION,\n)\nfrom ludwig.data.dataset.base import Dataset\nfrom ludwig.distributed.base import DistributedStrategy, LocalStrategy\nfrom ludwig.globals import (\n is_progressbar_disabled,\n MODEL_FILE_NAME,\n MODEL_HYPERPARAMETERS_FILE_NAME,\n TRAINING_CHECKPOINTS_DIR_PATH,\n TRAINING_PROGRESS_TRACKER_FILE_NAME,\n)\nfrom ludwig.models.ecd import ECD\nfrom ludwig.models.llm import LLM\nfrom ludwig.models.predictor import Predictor\nfrom ludwig.modules.lr_scheduler import LRScheduler\nfrom ludwig.modules.metric_modules import get_improved_fn, get_initial_validation_value\nfrom ludwig.modules.metric_registry import get_metric_objective\nfrom ludwig.modules.optimization_modules import create_clipper\nfrom ludwig.progress_bar import LudwigProgressBar\nfrom ludwig.schema.trainer import ECDTrainerConfig\nfrom ludwig.trainers.base import BaseTrainer\nfrom ludwig.trainers.registry import register_trainer\nfrom ludwig.types import ModelConfigDict\nfrom ludwig.utils import time_utils\nfrom ludwig.utils.batch_size_tuner import BatchSizeEvaluator\nfrom ludwig.utils.checkpoint_utils import Checkpoint, CheckpointManager\nfrom ludwig.utils.config_utils import get_quantization\nfrom ludwig.utils.data_utils import load_json\nfrom ludwig.utils.defaults import default_random_seed\nfrom ludwig.utils.fs_utils import path_exists\nfrom ludwig.utils.llm_utils import update_embedding_layer\nfrom ludwig.utils.metric_utils import get_metric_names, TrainerMetric\nfrom ludwig.utils.metrics_printed_table import print_metrics_table\nfrom ludwig.utils.misc_utils import set_random_seed\nfrom ludwig.utils.model_utils import contains_nan_or_inf_tensors\nfrom ludwig.utils.torch_utils import get_torch_device\nfrom ludwig.utils.trainer_utils import (\n append_metrics,\n freeze_layers_regex,\n get_final_steps_per_checkpoint,\n get_latest_metrics_dict,\n get_new_progress_tracker,\n get_total_expected_checkpoints,\n get_total_steps,\n ProgressTracker,\n)\n\nlogger = logging.getLogger(__name__)\n\n\n_TORCH210 = packaging.version.parse(torch.__version__) >= packaging.version.parse(\"2.1.0\")\n\n\n@register_trainer(MODEL_ECD, default=True)\nclass Trainer(BaseTrainer):\n \"\"\"Trainer is a class that trains a model.\"\"\"\n\n @staticmethod\n def get_schema_cls():\n return ECDTrainerConfig\n\n def __init__(\n self,\n config: ECDTrainerConfig,\n model: ECD,\n resume: float = False,\n skip_save_model: bool = False,\n skip_save_progress: bool = False,\n skip_save_log: bool = False,\n callbacks: list = None,\n report_tqdm_to_ray=False,\n random_seed: float = default_random_seed,\n distributed: DistributedStrategy | None = None,\n device: str | None = None,\n **kwargs,\n ):\n \"\"\"Trains a model with a set of options and hyperparameters listed below. Customizable.\n\n :param model: Underlying Ludwig model\n :type model: `ludwig.models.ecd.ECD`\n :param resume: Resume training a model that was being trained. (default: False).\n :type resume: Boolean\n :param skip_save_model: Disables saving model weights and hyperparameters each time the model improves. By\n default Ludwig saves model weights after each round of evaluation the validation metric (improves, but\n if the model is really big that can be time consuming. If you do not want to keep the weights and just\n find out what performance a model can get with a set of hyperparameters, use this parameter to skip it,\n but the model will not be loadable later on. (default: False).\n :type skip_save_model: Boolean\n :param skip_save_progress: Disables saving progress each round of evaluation. By default Ludwig saves weights\n and stats after each round of evaluation for enabling resuming of training, but if the model is really\n big that can be time consuming and will uses twice as much space, use this parameter to skip it, but\n training cannot be resumed later on. (default: False).\n :type skip_save_progress: Boolean\n :param skip_save_log: Disables saving TensorBoard logs. By default Ludwig saves logs for the TensorBoard, but if\n it is not needed turning it off can slightly increase the overall speed. (default: False).\n :type skip_save_log: Boolean\n :param callbacks: List of `ludwig.callbacks.Callback` objects that provide hooks into the Ludwig pipeline.\n (default: None).\n :type callbacks: list\n :param report_tqdm_to_ray: Enables using the ray based tqdm Callback for progress bar reporting\n :param random_seed: Default initialization for the random seeds (default: 42).\n :type random_seed: Float\n :param distributed: Distributed strategy (default: None).\n :type distributed: `DistributedStrategy`\n :param device: Device to load the model on from a saved checkpoint (default: None).\n :type device: str\n :param config: `ludwig.schema.trainer.BaseTrainerConfig` instance that specifies training hyperparameters\n (default: `ludwig.schema.trainer.ECDTrainerConfig()`).\n \"\"\"\n self.distributed = distributed if distributed is not None else LocalStrategy()\n\n self.epochs = config.epochs\n self.train_steps = config.train_steps\n self.enable_profiling = config.enable_profiling\n self.steps_per_epoch = 0 # Computed during training, after batcher has been initialized.\n self.total_steps = 0 # Computed during training, after batcher has been initialized.\n self.total_expected_checkpoints = 0 # Computed during training, after batcher has been initialized.\n\n self.regularization_lambda = config.regularization_lambda\n self.regularization_type = config.regularization_type\n self.batch_size = config.batch_size\n self.effective_batch_size = config.effective_batch_size\n self.max_batch_size = config.max_batch_size\n self.eval_batch_size = config.batch_size if config.eval_batch_size is None else config.eval_batch_size\n self.should_shuffle = config.should_shuffle\n self._validation_field = config.validation_field\n self._validation_metric = config.validation_metric\n self.early_stop = config.early_stop\n self.layers_to_freeze_regex = config.layers_to_freeze_regex\n self.steps_per_checkpoint = config.steps_per_checkpoint\n self.checkpoints_per_epoch = config.checkpoints_per_epoch\n self.evaluate_training_set = config.evaluate_training_set\n self.skip_all_evaluation = config.skip_all_evaluation\n self.increase_batch_size_on_plateau = config.increase_batch_size_on_plateau\n self.increase_batch_size_on_plateau_patience = config.increase_batch_size_on_plateau_patience\n self.increase_batch_size_on_plateau_rate = config.increase_batch_size_on_plateau_rate\n self.increase_batch_size_eval_metric = config.increase_batch_size_eval_metric\n self.increase_batch_size_eval_split = config.increase_batch_size_eval_split\n self.gradient_accumulation_steps = (\n config.gradient_accumulation_steps\n if self.distributed.allow_gradient_accumulation() and config.gradient_accumulation_steps != AUTO\n else 1\n )\n self.resume = resume\n self.skip_save_model = skip_save_model\n self.skip_save_progress = skip_save_progress\n self.skip_save_log = skip_save_log\n self.random_seed = random_seed\n self.received_sigint = False\n self.report_tqdm_to_ray = report_tqdm_to_ray\n self.callbacks = callbacks or []\n self.device = device\n if self.device is None:\n self.device = get_torch_device()\n\n self.model = model\n self.model.prepare_for_training()\n self.model = self.distributed.to_device(self.model)\n self.model.metrics_to_device(self.device)\n\n self.compiled_model = self.model\n if config.compile:\n self.compiled_model = torch.compile(self.model)\n logger.info(\"Training with torchdynamo compiled model\")\n\n # ================ Optimizer tuning ================\n self.gradient_clipping_config = create_clipper(config.gradient_clipping)\n\n self.config = config\n\n self.base_learning_rate = None\n self.dist_model = None\n self.optimizer = None\n self.scheduler = None\n\n self.prepare()\n\n # Setup for automatic mixed precision (AMP)\n self.use_amp = config.use_mixed_precision and self.distributed.allow_mixed_precision()\n if self.use_amp:\n if torch.cuda.is_available():\n logger.info(\"Enabling automatic mixed precision (AMP)\")\n else:\n logger.info(\"`trainer.use_mixed_precision=True`, but no GPU device found. Setting to `False`\")\n self.use_amp = False\n self.scaler = torch.amp.GradScaler(\"cuda\") if self.use_amp else None\n\n # when training starts the sigint handler will be replaced with\n # set_steps_to_1_or_quit so this is needed to remember\n # the original sigint to restore at the end of training\n # and before set_steps_to_1_or_quit returns\n self.original_sigint_handler = None\n\n def prepare(self):\n base_learning_rate = self.config.learning_rate\n if self.distributed:\n lr_scale_fn = learning_rate_scale_fns[self.config.learning_rate_scaling]\n base_learning_rate *= lr_scale_fn(self.distributed.size())\n self.base_learning_rate = base_learning_rate\n\n # Given that regex is supplied, freeze layers\n if self.config.layers_to_freeze_regex:\n freeze_layers_regex(self.config, self.model)\n\n # We may need to replace the embedding layer when using 8-bit optimizers from bitsandbytes.\n update_embedding_layer(self.compiled_model, self.config)\n\n # Register any post forward hooks for the model\n self.compiled_model._activate_forward_hooks()\n\n # Enable gradient checkpointing if configured\n if self.config.enable_gradient_checkpointing:\n # TODO(Arnav): Add support for gradient checkpointing in the compiled model\n # when the model is an ECD model using torch.utils.checkpoint (torch.utils.checkpoint.sequential())\n if not isinstance(self.compiled_model, LLM):\n logger.warning(\"Gradient checkpointing is currently only supported for model_type: llm. Skipping...\")\n elif not hasattr(self.compiled_model, \"model\") and not hasattr(\n self.compiled_model.model, \"gradient_checkpointing_enable\"\n ):\n logger.warning(\"Gradient checkpointing is not supported by this model. Skipping...\")\n elif hasattr(self.compiled_model.model, \"gradient_checkpointing_enable\"):\n if _TORCH210:\n # https://pytorch.org/docs/stable/checkpoint.html\n # https://github.com/huggingface/transformers/blob/02f8738ef8c674300c314d004ba436cb5aaca165/src/transformers/modeling_utils.py#L2094 # noqa: E501\n self.compiled_model.model.gradient_checkpointing_enable(\n gradient_checkpointing_kwargs={\"use_reentrant\": True}\n )\n else:\n self.compiled_model.model.gradient_checkpointing_enable()\n # `use_cache=True` is incompatible with gradient checkpointing.\n self.compiled_model.model.config.use_cache = False\n self.compiled_model.model.enable_input_require_grads()\n logger.info(\"Gradient checkpointing enabled for training.\")\n else:\n raise RuntimeError(\"Error when trying to enable gradient checkpointing.\")\n\n self.dist_model, self.optimizer = self.distributed.prepare(\n self.compiled_model,\n self.config,\n self.base_learning_rate,\n )\n\n # NOTE: This is a partially configured LRScheduler. It will be updated in the first call to train_step.\n self.scheduler = LRScheduler(self.config.learning_rate_scheduler, self.optimizer, 0, 0)\n\n def train_step(\n self,\n inputs: dict[str, torch.Tensor],\n targets: dict[str, torch.Tensor],\n should_step: bool = True,\n profiler: torch.profiler.profile | None = None,\n ) -> tuple[torch.Tensor, dict[str, torch.Tensor], torch.Tensor]:\n \"\"\"Performs a single training step.\n\n Params:\n inputs: A dictionary of input data, from feature name to tensor.\n targets: A dictionary of target data, from feature name to tensor.\n should_step: Whether to perform a step of the optimizer after computing gradients.\n\n Returns:\n A tuple of:\n 1. loss tensor\n 2. dictionary of loss for every output feature.\n 3. tokens usage tensor\n \"\"\"\n if isinstance(self.optimizer, torch.optim.LBFGS):\n # NOTE: AMP is not supported for L-BFGS yet.\n # NOTE: gradient accumulation is not supported for L-BFGS yet.\n\n def closure():\n # Allows L-BFGS to reevaluate the loss function\n self.distributed.zero_grad(self.optimizer)\n model_outputs = self.dist_model((inputs, targets))\n loss, _ = self.model.train_loss(\n targets, model_outputs, self.regularization_type, self.regularization_lambda\n )\n loss.backward()\n return loss\n\n self.distributed.step(self.optimizer, closure)\n\n # Obtain model predictions and loss\n model_outputs = self.dist_model((inputs, targets))\n loss, all_losses = self.model.train_loss(\n targets, model_outputs, self.regularization_type, self.regularization_lambda\n )\n\n if not self.evaluate_training_set:\n # Update evaluation metrics with current model params:\n # noisy but fast way to get metrics on the training set\n predictions = self.model.outputs_to_predictions(model_outputs)\n self.model.update_metrics(targets, predictions)\n\n return loss, all_losses, model_outputs[USED_TOKENS]\n\n with torch.amp.autocast(\"cuda\") if self.use_amp else contextlib.nullcontext():\n with self.distributed.prepare_model_update(self.dist_model, should_step=should_step):\n # Obtain model predictions and loss\n model_outputs = self.dist_model((inputs, targets))\n loss, all_losses = self.model.train_loss(\n targets, model_outputs, self.regularization_type, self.regularization_lambda\n )\n loss = loss / self.gradient_accumulation_steps\n\n used_tokens = model_outputs[USED_TOKENS]\n\n # Begin the backward pass\n variables = self.dist_model.parameters()\n if self.use_amp:\n self.scaler.scale(loss).backward()\n else:\n self.distributed.backward(loss, self.dist_model)\n\n if not should_step:\n # Short-circuit the parameter updates if we are still accumulating gradients\n return loss, all_losses, used_tokens\n\n # Wait for gradient aggregation to complete before clipping the gradients.\n # When using AMP, we need to do this before unscaling.\n self.distributed.wait_optimizer_synced(self.optimizer)\n\n if self.use_amp:\n # In-place unscaling of all gradients before weights update\n # Do this before gradient clipping per docs:\n # https://pytorch.org/docs/master/notes/amp_examples.html#gradient-clipping\n self.scaler.unscale_(self.optimizer)\n\n if self.distributed.allow_clip_gradients():\n # Clip gradients\n self.clip_grads(variables)\n\n # Apply gradient updates\n with self.distributed.prepare_optimizer_update(self.optimizer):\n # Because we already synchronized above, we skip doing so here\n if self.use_amp:\n self.scaler.step(self.optimizer)\n else:\n self.distributed.step(self.optimizer)\n\n if self.use_amp:\n # Update scaler in case of overflow/underflow\n self.scaler.update()\n\n if not self.evaluate_training_set:\n # Update evaluation metrics with current model params:\n # noisy but fast way to get metrics on the training set\n predictions = self.model.outputs_to_predictions(model_outputs)\n self.model.update_metrics(targets, predictions)\n\n self.distributed.zero_grad(self.optimizer)\n\n if profiler:\n profiler.step()\n\n return loss, all_losses, used_tokens\n\n def clip_grads(self, variables):\n \"\"\"Applies gradient clipping.\"\"\"\n if self.gradient_clipping_config.clipglobalnorm:\n torch.nn.utils.clip_grad_norm_(variables, self.gradient_clipping_config.clipglobalnorm)\n if self.gradient_clipping_config.clipnorm:\n torch.nn.utils.clip_grad_norm_(variables, self.gradient_clipping_config.clipnorm)\n if self.gradient_clipping_config.clipvalue:\n torch.nn.utils.clip_grad_value_(variables, self.gradient_clipping_config.clipvalue)\n\n @classmethod\n def write_eval_summary(\n cls,\n summary_writer,\n metrics,\n step,\n ):\n if not summary_writer:\n return\n\n for feature_name, output_feature in metrics.items():\n for metric_name, metrics in output_feature.items():\n if metrics:\n metric_tag = f\"{feature_name}/epoch_{metric_name}\"\n metric_val = metrics[-1][-1]\n summary_writer.add_scalar(metric_tag, metric_val, global_step=step)\n summary_writer.flush()\n\n @classmethod\n def write_step_summary(\n cls, train_summary_writer, combined_loss, all_losses, step, used_tokens, total_tokens_used, learning_rate=None\n ):\n if not train_summary_writer:\n return\n\n # token information.\n train_summary_writer.add_scalar(\"tokens/tokens\", used_tokens, global_step=step)\n train_summary_writer.add_scalar(\"tokens/total_tokens_used\", total_tokens_used, global_step=step)\n\n # combined loss\n train_summary_writer.add_scalar(\"combined/step_training_loss\", combined_loss, global_step=step)\n\n # all other losses\n for feature_name, loss in all_losses.items():\n loss_tag = f\"{feature_name}/step_training_loss\"\n train_summary_writer.add_scalar(loss_tag, loss.detach().float(), global_step=step)\n\n if learning_rate:\n train_summary_writer.add_scalar(\"combined/step_learning_rate\", learning_rate, global_step=step)\n\n # Log CUDA memory stats.\n if torch.cuda.is_available():\n for i in range(torch.cuda.device_count()):\n device = torch.device(f\"cuda:{i}\")\n memory_stats = torch.cuda.memory_stats(device=device)\n gb_memory_stats = {k: v / (1000**3) for k, v in memory_stats.items()}\n # Allocated bytes.\n train_summary_writer.add_scalar(\n f\"cuda/device{i}/allocated_gb.all.current\",\n gb_memory_stats[\"allocated_bytes.all.current\"],\n global_step=step,\n )\n train_summary_writer.add_scalar(\n f\"cuda/device{i}/allocated_gb.all.peak\",\n gb_memory_stats[\"allocated_bytes.all.peak\"],\n global_step=step,\n )\n train_summary_writer.add_scalar(\n f\"cuda/device{i}/allocated_gb.all.allocated\",\n gb_memory_stats[\"allocated_bytes.all.allocated\"],\n global_step=step,\n )\n train_summary_writer.add_scalar(\n f\"cuda/device{i}/allocated_gb.all.freed\",\n gb_memory_stats[\"allocated_bytes.all.freed\"],\n global_step=step,\n )\n\n # Reserved bytes.\n train_summary_writer.add_scalar(\n f\"cuda/device{i}/reserved_gb.all.current\",\n gb_memory_stats[\"reserved_bytes.all.current\"],\n global_step=step,\n )\n train_summary_writer.add_scalar(\n f\"cuda/device{i}/reserved_gb.all.peak\", gb_memory_stats[\"reserved_bytes.all.peak\"], global_step=step\n )\n train_summary_writer.add_scalar(\n f\"cuda/device{i}/reserved_gb.all.allocated\",\n gb_memory_stats[\"reserved_bytes.all.allocated\"],\n global_step=step,\n )\n train_summary_writer.add_scalar(\n f\"cuda/device{i}/reserved_gb.all.freed\",\n gb_memory_stats[\"reserved_bytes.all.freed\"],\n global_step=step,\n )\n\n # Active bytes.\n train_summary_writer.add_scalar(\n f\"cuda/device{i}/active_gb.all.current\",\n gb_memory_stats[\"active_bytes.all.current\"],\n global_step=step,\n )\n train_summary_writer.add_scalar(\n f\"cuda/device{i}/active_gb.all.peak\", gb_memory_stats[\"active_bytes.all.peak\"], global_step=step\n )\n train_summary_writer.add_scalar(\n f\"cuda/device{i}/active_gb.all.allocated\",\n gb_memory_stats[\"active_bytes.all.allocated\"],\n global_step=step,\n )\n train_summary_writer.add_scalar(\n f\"cuda/device{i}/active_gb.all.freed\", gb_memory_stats[\"active_bytes.all.freed\"], global_step=step\n )\n\n # Global free memory.\n train_summary_writer.add_scalar(\n f\"cuda/device{i}/global_free_memory_gb\",\n torch.cuda.mem_get_info(device=device)[0] / (1000**3),\n global_step=step,\n )\n\n # Total memory occupied.\n train_summary_writer.add_scalar(\n f\"cuda/device{i}/total_memory_occupied_gb\",\n torch.cuda.mem_get_info(device=device)[1] / (1000**3),\n global_step=step,\n )\n\n # Total memory used.\n train_summary_writer.add_scalar(\n f\"cuda/device{i}/total_memory_used_gb\",\n (torch.cuda.mem_get_info(device=device)[1] - torch.cuda.mem_get_info(device=device)[0]) / (1000**3),\n global_step=step,\n )\n\n # Utilization.\n # https://pytorch.org/docs/stable/generated/torch.cuda.utilization.html#torch.cuda.utilization\n train_summary_writer.add_scalar(\n f\"cuda/device{i}/utilization\",\n torch.cuda.utilization(device=device),\n global_step=step,\n )\n train_summary_writer.flush()\n\n def is_cpu_training(self):\n return torch.device(self.device) == torch.device(\"cpu\")\n\n def tune_batch_size(\n self,\n config: ModelConfigDict,\n training_set: Dataset,\n random_seed: int = default_random_seed,\n max_trials: int = 20,\n halving_limit: int = 3,\n snapshot_weights: bool = True,\n on_best_batch_size_updated: Callable[[int, float, int], None] | None = None,\n tune_for_training: bool = True,\n global_max_sequence_length: int | None = None,\n ) -> int:\n logger.info(\"Tuning batch size...\")\n skip_save_model = self.skip_save_model\n skip_save_progress = self.skip_save_progress\n skip_save_log = self.skip_save_log\n # Set temporary values\n self.skip_save_model = True\n self.skip_save_progress = True\n self.skip_save_log = True\n\n # When training on CPU, larger batch sizes offer limited benefits due to lack of effective\n # parallelization within a batch. As such, to increase chances of stable training, we cap the maximum\n # batch size at MAX_CPU_BATCH_SIZE\n max_batch_size = (\n self.max_batch_size if torch.cuda.is_available() else min(self.max_batch_size, MAX_CPU_BATCH_SIZE)\n )\n\n if self.effective_batch_size != AUTO:\n # If an effective batch size is set, we must ensure that batch size tuning doesn't exceed it\n max_batch_size = min(self.effective_batch_size, max_batch_size)\n\n if not tune_for_training:\n # No need to save and restore model and optimizer states, as they aren't modified during predict\n snapshot_weights = False\n\n self.dist_model.train() # Sets model training mode.\n evaluator = (\n self._create_batch_size_evaluator() if tune_for_training else self._create_predict_batch_size_evaluator()\n )\n with tempfile.TemporaryDirectory() as tmpdir:\n if snapshot_weights:\n # Save a snapshot of the model and optimizer state to restore later, as they will be modified\n # when we call the train step as part of the auto-tuning. This is undesirable, particularly for\n # pretrained models.\n checkpoint = self.distributed.create_checkpoint_handle(\n dist_model=self.dist_model, model=self.model, optimizer=self.optimizer, scheduler=self.scheduler\n )\n checkpoint.save(os.path.join(tmpdir, \"latest.ckpt\"), global_step=0)\n try:\n best_batch_size = evaluator.select_best_batch_size(\n len(training_set), max_batch_size, max_trials, self.is_coordinator(), global_max_sequence_length\n )\n best_batch_size = self.distributed.broadcast_object(best_batch_size)\n\n if tune_for_training:\n # Update batch size / gradient accumulation before preparing the trainer. This is needed primarily\n # for DeepSpeed, which needs to know the batch size and gradient accumulation steps before init\n self.config.batch_size = best_batch_size\n self.config.update_batch_size_grad_accum(self.distributed.size())\n self.batch_size = self.config.batch_size\n self.gradient_accumulation_steps = self.config.gradient_accumulation_steps\n\n return best_batch_size\n finally:\n # Restore original parameters to defaults\n self.skip_save_model = skip_save_model\n self.skip_save_progress = skip_save_progress\n self.skip_save_log = skip_save_log\n\n if snapshot_weights:\n # Restore the model weights prior to batch size tuning to undo any updates made to the weights\n if self.distributed.prepare_before_load():\n # Some distributed strategies, like DeepSpeed, need to re-init before loading the model\n self.prepare()\n self.resume_weights_and_optimizer(str(tmpdir), checkpoint)\n\n def _create_batch_size_evaluator(self) -> BatchSizeEvaluator:\n trainer = self\n\n class _TrainerBatchSizeEvaluator(BatchSizeEvaluator):\n def reset(self):\n trainer.model.reset_metrics()\n trainer.optimizer.zero_grad()\n\n def step(self, batch_size: int, global_max_sequence_length: int | None = None):\n trainer.distributed.set_batch_size(trainer.dist_model, batch_size)\n inputs = {\n input_feature_name: input_feature.create_sample_input(batch_size=batch_size).to(trainer.device)\n for input_feature_name, input_feature in trainer.model.input_features.items()\n }\n targets = {\n output_feature_name: output_feature.create_sample_output(batch_size=batch_size).to(trainer.device)\n for output_feature_name, output_feature in trainer.model.output_features.items()\n }\n trainer.train_step(inputs, targets)\n\n return _TrainerBatchSizeEvaluator()\n\n def _create_predict_batch_size_evaluator(self) -> BatchSizeEvaluator:\n trainer = self\n\n class _PredictBatchSizeEvaluator(BatchSizeEvaluator):\n def reset(self):\n trainer.model.reset_metrics()\n trainer.optimizer.zero_grad()\n\n def step(self, batch_size: int, global_max_sequence_length: int | None = None):\n trainer.distributed.set_batch_size(trainer.dist_model, batch_size)\n inputs = {\n input_feature_name: input_feature.create_sample_input(batch_size=batch_size).to(trainer.device)\n for input_feature_name, input_feature in trainer.model.input_features.items()\n }\n targets = {\n output_feature_name: output_feature.create_sample_output(batch_size=batch_size).to(trainer.device)\n for output_feature_name, output_feature in trainer.model.output_features.items()\n }\n with torch.no_grad():\n trainer.dist_model((inputs, targets))\n\n return _PredictBatchSizeEvaluator()\n\n def run_evaluation(\n self,\n training_set,\n validation_set,\n test_set,\n progress_tracker: ProgressTracker,\n train_summary_writer,\n validation_summary_writer,\n test_summary_writer,\n model_hyperparameters_path,\n output_features,\n metrics_names,\n save_path,\n loss: torch.Tensor,\n all_losses: dict[str, torch.Tensor],\n early_stopping_steps: int,\n checkpoint_manager: CheckpointManager,\n ) -> bool:\n \"\"\"Runs evaluation over training, validation, and test sets.\n\n Also:\n - Prints results, saves results to the progress tracker.\n - Saves the model if the validation score is the best so far\n - If there is no validation set, the model is always saved.\n\n Returns whether the trainer should early stop, based on validation metrics history.\n \"\"\"\n start_time = time.time()\n self.callback(lambda c: c.on_eval_start(self, progress_tracker, save_path))\n\n if self.is_coordinator():\n logger.info(f\"\\nRunning evaluation for step: {progress_tracker.steps}, epoch: {progress_tracker.epoch}\")\n\n # ================ Eval ================\n # eval metrics on train\n self.eval_batch_size = max(self.eval_batch_size, progress_tracker.batch_size)\n\n if self.evaluate_training_set:\n # Run a separate pass over the training data to compute metrics\n self.evaluation(\n training_set, \"train\", progress_tracker.train_metrics, self.eval_batch_size, progress_tracker\n )\n else:\n # Use metrics accumulated during training\n metrics = self.model.get_metrics()\n append_metrics(self.model, \"train\", metrics, progress_tracker.train_metrics, progress_tracker)\n self.model.reset_metrics()\n\n self.write_eval_summary(\n summary_writer=train_summary_writer,\n metrics=progress_tracker.train_metrics,\n step=progress_tracker.steps,\n )\n\n if validation_set is not None:\n self.callback(lambda c: c.on_validation_start(self, progress_tracker, save_path))\n\n # eval metrics on validation set\n self.evaluation(\n validation_set,\n VALIDATION,\n progress_tracker.validation_metrics,\n self.eval_batch_size,\n progress_tracker,\n )\n\n llm_eval_examples = progress_tracker.llm_eval_examples\n dict_save_dir = os.path.join(os.path.dirname(checkpoint_manager.directory), \"llm_eval_examples\")\n os.makedirs(dict_save_dir, exist_ok=True)\n dict_save_path = os.path.join(dict_save_dir, f\"{progress_tracker.checkpoint_number}.csv\")\n llm_eval_examples = pd.DataFrame(llm_eval_examples).to_dict(orient=\"records\")\n with open(dict_save_path, \"w\", encoding=\"utf-8\") as outfile:\n writer = csv.DictWriter(outfile, fieldnames=[\"inputs\", \"targets\", \"outputs\"])\n writer.writeheader()\n writer.writerows(llm_eval_examples)\n\n self.write_eval_summary(\n summary_writer=validation_summary_writer,\n metrics=progress_tracker.validation_metrics,\n step=progress_tracker.steps,\n )\n\n self.callback(lambda c: c.on_validation_end(self, progress_tracker, save_path))\n\n if test_set is not None:\n self.callback(lambda c: c.on_test_start(self, progress_tracker, save_path))\n\n # eval metrics on test set\n self.evaluation(test_set, TEST, progress_tracker.test_metrics, self.eval_batch_size, progress_tracker)\n\n self.write_eval_summary(\n summary_writer=test_summary_writer,\n metrics=progress_tracker.test_metrics,\n step=progress_tracker.steps,\n )\n\n self.callback(lambda c: c.on_test_end(self, progress_tracker, save_path))\n\n elapsed_time = (time.time() - start_time) * 1000.0\n\n if self.is_coordinator():\n logger.info(f\"Evaluation took {time_utils.strdelta(elapsed_time)}\\n\")\n print_metrics_table(\n output_features,\n progress_tracker.train_metrics,\n progress_tracker.validation_metrics,\n progress_tracker.test_metrics,\n )\n\n # ================ Validation Logic ================\n should_break = False\n if validation_set is not None and validation_set.size > 0:\n should_break = self.check_progress_on_validation(\n progress_tracker,\n self.validation_field,\n self.validation_metric,\n save_path,\n model_hyperparameters_path,\n self.increase_batch_size_on_plateau,\n self.increase_batch_size_on_plateau_patience,\n self.increase_batch_size_on_plateau_rate,\n self.max_batch_size,\n self.increase_batch_size_eval_metric,\n self.increase_batch_size_eval_split,\n early_stopping_steps,\n self.skip_save_model,\n checkpoint_manager,\n )\n else:\n # There's no validation, so we save the model.\n if not self.skip_save_model:\n if self.is_coordinator():\n logger.info(\"Saving model.\\n\")\n checkpoint_manager.save_best(progress_tracker.steps)\n self.callback(lambda c: c.on_save_best_checkpoint(self, progress_tracker, save_path))\n\n # Trigger eval end callback after any model weights save for complete checkpoint\n self.callback(lambda c: c.on_eval_end(self, progress_tracker, save_path))\n\n # Clear the CUDA cache to free up memory\n torch.cuda.empty_cache()\n\n return should_break\n\n def save_checkpoint(self, progress_tracker: ProgressTracker, save_path: str, checkpoint_manager: CheckpointManager):\n \"\"\"Checkpoints the model, progress tracker, and invokes the checkpoint callback.\"\"\"\n progress_tracker.increment_checkpoint()\n\n checkpoint_manager.save(progress_tracker.steps)\n if self.is_coordinator():\n progress_tracker.save(os.path.join(save_path, TRAINING_PROGRESS_TRACKER_FILE_NAME))\n\n # Callback that the checkpoint was reached, regardless of whether the model was evaluated.\n self.callback(lambda c: c.on_checkpoint(self, progress_tracker))\n\n def create_checkpoint_handle(self):\n return self.distributed.create_checkpoint_handle(\n dist_model=self.dist_model, model=self.model, optimizer=self.optimizer, scheduler=self.scheduler\n )\n\n def train(\n self,\n training_set,\n validation_set=None,\n test_set=None,\n save_path=MODEL_FILE_NAME,\n return_state_dict: bool = False,\n **kwargs,\n ):\n \"\"\"Trains a model with a set of hyperparameters listed below. Customizable.\n\n :param training_set: The training set\n :param validation_set: The validation dataset\n :param test_set: The test dataset\n :param save_path: The directory that will contain the saved model\n :param return_state_dict: Whether to return the state dict of the model instead of the model itself\n \"\"\"\n # ====== General setup =======\n output_features = self.model.output_features\n\n # Only use signals when on the main thread to avoid issues with CherryPy\n # https://github.com/ludwig-ai/ludwig/issues/286\n if threading.current_thread() == threading.main_thread():\n # set the original sigint signal handler\n # as we want to restore it at the end of training\n self.original_sigint_handler = signal.getsignal(signal.SIGINT)\n signal.signal(signal.SIGINT, self.set_steps_to_1_or_quit)\n\n metrics_names = get_metric_names(output_features)\n\n # ====== Setup file names =======\n model_hyperparameters_path = None\n tensorboard_log_dir = None\n if self.is_coordinator():\n os.makedirs(save_path, exist_ok=True)\n model_hyperparameters_path = os.path.join(save_path, MODEL_HYPERPARAMETERS_FILE_NAME)\n tensorboard_log_dir = os.path.join(save_path, \"logs\")\n\n # Sync save_path across the workers\n save_path = self.distributed.broadcast_object(save_path or \"\")\n\n training_progress_tracker_path = None\n training_checkpoints_path = None\n if save_path:\n training_progress_tracker_path = os.path.join(save_path, TRAINING_PROGRESS_TRACKER_FILE_NAME)\n training_checkpoints_path = os.path.join(save_path, TRAINING_CHECKPOINTS_DIR_PATH)\n\n self.callback(\n lambda c: c.on_trainer_train_setup(self, save_path, self.is_coordinator()), coordinator_only=False\n )\n\n # ====== Setup session =======\n checkpoint = self.create_checkpoint_handle()\n checkpoint_manager = CheckpointManager(checkpoint, training_checkpoints_path, device=self.device)\n\n # ====== Setup Tensorboard writers =======\n train_summary_writer = None\n validation_summary_writer = None\n test_summary_writer = None\n if self.is_coordinator() and not self.skip_save_log and tensorboard_log_dir:\n train_summary_writer = SummaryWriter(os.path.join(tensorboard_log_dir, TRAINING))\n if validation_set is not None and validation_set.size > 0:\n validation_summary_writer = SummaryWriter(os.path.join(tensorboard_log_dir, VALIDATION))\n if test_set is not None and test_set.size > 0:\n test_summary_writer = SummaryWriter(os.path.join(tensorboard_log_dir, TEST))\n\n # ================ Resume logic ================\n self.callback(lambda c: c.on_resume_training(self.is_coordinator()))\n\n should_resume = self.resume and self.resume_files_exist(\n training_progress_tracker_path, training_checkpoints_path\n )\n # make sure all workers are on the same page about resuming.\n should_resume = self.distributed.broadcast_object(should_resume, name=\"should_resume\")\n\n if should_resume:\n try:\n progress_tracker = self.resume_training_progress_tracker(training_progress_tracker_path)\n self.resume_weights_and_optimizer(training_checkpoints_path, checkpoint)\n if self.is_coordinator():\n logger.info(\"Resuming training from previous run.\")\n except Exception:\n # This may happen if model training is interrupted after the progress tracker is initialized\n # but before any real training progress is made.\n progress_tracker = get_new_progress_tracker(\n batch_size=self.batch_size,\n learning_rate=self.base_learning_rate,\n best_eval_metric_value=get_initial_validation_value(self.validation_metric),\n best_increase_batch_size_eval_metric=get_initial_validation_value(\n self.increase_batch_size_eval_metric\n ),\n output_features=output_features,\n )\n if self.is_coordinator():\n logger.info(\"Failed to resume training from previous run. Creating fresh model training run.\")\n else:\n progress_tracker = get_new_progress_tracker(\n batch_size=self.batch_size,\n learning_rate=self.base_learning_rate,\n best_eval_metric_value=get_initial_validation_value(self.validation_metric),\n best_increase_batch_size_eval_metric=get_initial_validation_value(self.increase_batch_size_eval_metric),\n output_features=output_features,\n )\n if self.is_coordinator():\n logger.info(\"Creating fresh model training run.\")\n\n # Distributed: broadcast initial variable states from rank 0 to all other processes.\n # This is necessary to ensure consistent initialization of all workers when\n # training is started with random weights or restored from a checkpoint.\n self.distributed.sync_model(self.dist_model)\n self.distributed.sync_optimizer(self.optimizer)\n self.scheduler.load_state_dict(self.distributed.broadcast_object(self.scheduler.state_dict()))\n\n # For DeepSpeed, we need to set the batch size here in case it was modfied during auto-tuning\n self.distributed.set_batch_size(self.dist_model, self.batch_size)\n\n set_random_seed(self.random_seed)\n\n if self.enable_profiling:\n logger.warning(\"Full torch profiler is enabled. Training may be significantly slower.\")\n profiler = torch.profiler.profile(\n schedule=torch.profiler.schedule(\n wait=self.config.profiler.wait,\n warmup=self.config.profiler.warmup,\n active=self.config.profiler.active,\n repeat=self.config.profiler.repeat,\n ),\n on_trace_ready=torch.profiler.tensorboard_trace_handler(os.path.join(tensorboard_log_dir, \"profiling\")),\n record_shapes=True,\n with_stack=True,\n profile_memory=True,\n )\n else:\n profiler = None\n\n try:\n with training_set.initialize_batcher(\n batch_size=self.batch_size,\n should_shuffle=self.should_shuffle,\n random_seed=self.random_seed,\n distributed=self.distributed,\n ignore_last=True,\n augmentation_pipeline=self.model.get_augmentation_pipelines(),\n ) as batcher:\n # ================ Training Loop ================\n self.steps_per_epoch = batcher.steps_per_epoch\n self.total_steps = get_total_steps(self.epochs, batcher.steps_per_epoch, self.train_steps)\n # NOTE(geoffrey): this ensures that the total number of epochs coincides with the number of\n # times `batcher.set_epoch` is called.\n old_epochs = self.epochs\n self.epochs = math.ceil(self.total_steps / self.steps_per_epoch)\n if old_epochs != self.epochs:\n logger.warning(\n f\"The number of epochs has been adjusted from config-specified {old_epochs} \"\n f\"to {self.epochs} to match the total number of steps.\"\n )\n\n # Get the terminal steps per checkpoint.\n final_steps_per_checkpoint = get_final_steps_per_checkpoint(\n batcher.steps_per_epoch,\n self.steps_per_checkpoint,\n self.checkpoints_per_epoch,\n self.is_coordinator(),\n )\n final_steps_per_checkpoint = min(final_steps_per_checkpoint, self.total_steps)\n early_stopping_steps = final_steps_per_checkpoint * self.early_stop\n if not self.skip_save_progress:\n self.total_expected_checkpoints = get_total_expected_checkpoints(\n self.total_steps, final_steps_per_checkpoint, self.epochs\n )\n\n # Initialize the learning rate scheduler.\n self.scheduler = LRScheduler(\n self.config.learning_rate_scheduler,\n self.optimizer,\n steps_per_checkpoint=final_steps_per_checkpoint,\n total_steps=self.total_steps,\n )\n\n if self.is_coordinator():\n logger.info(\n f\"Training for {self.total_steps} step(s), approximately \"\n f\"{int(self.total_steps / batcher.steps_per_epoch)} epoch(s).\"\n )\n if self.early_stop < 0:\n logger.info(\"Early stopping policy: None\")\n else:\n logger.info(\n f\"Early stopping policy: {self.early_stop} round(s) of evaluation, or \"\n f\"{early_stopping_steps} step(s), approximately \"\n f\"{int(early_stopping_steps / batcher.steps_per_epoch)} epoch(s).\\n\"\n )\n logger.info(f\"Starting with step {progress_tracker.steps}, epoch: {progress_tracker.epoch}\")\n\n progress_bar_config = {\n \"desc\": \"Training\",\n \"initial\": progress_tracker.steps,\n \"total\": self.total_steps,\n \"disable\": is_progressbar_disabled(),\n \"file\": sys.stdout,\n }\n progress_bar = LudwigProgressBar(self.report_tqdm_to_ray, progress_bar_config, self.is_coordinator())\n\n if profiler:\n profiler.start()\n\n while progress_tracker.steps < self.total_steps:\n # note that batch size may change over epochs\n batcher.set_epoch(progress_tracker.epoch, progress_tracker.batch_size)\n\n # epoch init\n start_time = time.time()\n\n # Reset the metrics at the start of the next epoch\n self.dist_model.train() # Sets model to training mode.\n self.model.reset_metrics()\n\n self.callback(lambda c: c.on_epoch_start(self, progress_tracker, save_path))\n\n # Trains over a full epoch of data or up to the last training step, whichever is sooner.\n should_break, has_nan_or_inf_tensors = self._train_loop(\n batcher,\n progress_tracker,\n save_path,\n train_summary_writer,\n progress_bar,\n training_set,\n validation_set,\n test_set,\n start_time,\n validation_summary_writer,\n test_summary_writer,\n model_hyperparameters_path,\n output_features,\n metrics_names,\n checkpoint_manager,\n final_steps_per_checkpoint,\n early_stopping_steps,\n profiler,\n )\n if self.is_coordinator():\n # ========== Save training progress ==========\n logger.debug(\n f\"Epoch {progress_tracker.epoch} took: \"\n f\"{time_utils.strdelta((time.time() - start_time) * 1000.0)}.\"\n )\n\n # Skip saving progress if we're not saving the model. We should do this so as to not overwrite the\n # best model checkpoint from the previous round of evaluation so that the previous best model\n # weights can be used for inference instead of the current weights which are in a bad state.\n if has_nan_or_inf_tensors:\n break\n\n if not self.skip_save_progress:\n self.save_checkpoint(progress_tracker, save_path, checkpoint_manager)\n\n if not self.skip_save_model and self.skip_all_evaluation:\n # All evaluation was skipped, so save the current step as the best so far.\n checkpoint_manager.save_best(progress_tracker.steps)\n\n # Early stop if needed.\n if should_break:\n break\n finally:\n # ================ Finished Training ================\n self.callback(\n lambda c: c.on_trainer_train_teardown(self, progress_tracker, save_path, self.is_coordinator()),\n coordinator_only=False,\n )\n\n # Deactivate any forward hooks for the model used at training time.\n self.compiled_model._deactivate_forward_hooks()\n\n # Stop the profiler.\n if profiler:\n profiler.stop()\n\n # Close the summary writers.\n if train_summary_writer is not None:\n train_summary_writer.close()\n if validation_summary_writer is not None:\n validation_summary_writer.close()\n if test_summary_writer is not None:\n test_summary_writer.close()\n\n if not self.skip_save_model and self.skip_all_evaluation and not has_nan_or_inf_tensors:\n # All evaluation was skipped, so save the current step as the best so far.\n checkpoint_manager.save_best(progress_tracker.steps)\n\n if not self.skip_save_progress:\n checkpoint_manager.close()\n\n # Load the best weights from saved checkpoint\n state_dict = None\n if self.distributed.is_coordinator():\n if not self.skip_save_model:\n state_dict = checkpoint_manager.get_best_checkpoint_state_for_inference(self.return_device)\n if not state_dict:\n error_message = \"Training ran into an error. No checkpoint was saved.\"\n if has_nan_or_inf_tensors:\n error_message += (\n \" This is because training was terminated early due to the presence of NaN or \"\n \"Inf values in the model weights before a single valid checkpoint could be saved.\"\n )\n raise RuntimeError(error_message)\n if not return_state_dict:\n if self.distributed.is_model_parallel():\n # Assume the full weights cannot fit in memory on GPU\n self.model = self.model.cpu()\n\n # For a full explanation of this 8-bit workaround, see https://github.com/ludwig-ai/ludwig/pull/3606\n # TODO (jeffkinnison): Determine why `SCB` and `CB` are deleted from parameter state\n quantization = get_quantization(self.model.config_obj)\n uses_quantization = bool(quantization) if not isinstance(quantization, list) else any(quantization)\n if uses_quantization and 8 in quantization:\n # If the model was previously placed on GPU, 8-bit parameter state will be updated with several\n # matrices containing quantization information. These are recorded matrices are recorded in the\n # training checkpoint state dicts, but do not necessarily exist in the parameter object, leading\n # to a RuntimeError in `load_state_dict`. Explicitly call `model.cuda()` to make sure the\n # matrices are part of model state. This workaround is necessary because the matrices are\n # deleted during the model's forward pass.\n if self.model.config_obj.model_type == MODEL_LLM and self.model.model.device.type == \"cuda\":\n self.model.model.cuda()\n elif self.model.config_obj.model_type == MODEL_ECD and self.model.device.type == \"cuda\":\n self.model.cuda()\n _, unexpected_keys = self.model.load_state_dict(state_dict, strict=False)\n only_weights_format_keys = [\"weights_format\" in k for k in unexpected_keys]\n\n # bitsandbytes adds a number of `weights_format` metadata fields to the state dict in\n # `Linear8bitLt._save_to_state_dict`. These contain information about how the 8-bit tensors\n # are tiled, but the fields themselves never exist in the module and get returned as unexpected\n # keys when loading the state dict. The\n assert (\n unexpected_keys == [] or only_weights_format_keys\n ), f\"Unexpected keys found in state dict: {unexpected_keys}\"\n else:\n _, unexpected_keys = self.model.load_state_dict(state_dict, strict=False)\n assert unexpected_keys == [], f\"Unexpected keys found in state dict: {unexpected_keys}\"\n elif return_state_dict:\n state_dict = self.model.cpu().state_dict()\n\n # When running with Ray, we only need to return the state dict, as it's faster and cheaper to send the\n # state dict over the network than to load the model state here, serialize it back to a state dict, then\n # load it back on the head node.\n return_value = self.model if not return_state_dict else state_dict\n\n # restore original sigint signal handler\n if self.original_sigint_handler and threading.current_thread() == threading.main_thread():\n signal.signal(signal.SIGINT, self.original_sigint_handler)\n\n return (\n return_value,\n progress_tracker.train_metrics,\n progress_tracker.validation_metrics,\n progress_tracker.test_metrics,\n )\n\n def _train_loop(\n self,\n batcher,\n progress_tracker: ProgressTracker,\n save_path,\n train_summary_writer,\n progress_bar: LudwigProgressBar,\n training_set,\n validation_set,\n test_set,\n start_time,\n validation_summary_writer,\n test_summary_writer,\n model_hyperparameters_path,\n output_features,\n metrics_names,\n checkpoint_manager: CheckpointManager,\n final_steps_per_checkpoint: int,\n early_stopping_steps: int,\n profiler: torch.profiler.profile | None,\n ) -> tuple[bool, bool]:\n \"\"\"Completes up to one epoch through the data.\n\n This function completes a single pass (epoch) through the training data and returns\n two boolean values:\n\n Returns:\n should_break (bool):\n Indicates whether the training loop should be terminated prematurely.\n\n has_nan_or_inf_tensors (bool):\n Indicates whether the model weights contain NaN or Inf values.\n \"\"\"\n self.distributed.zero_grad(self.optimizer)\n batch_idx = 0\n should_break = False\n has_nan_or_inf_tensors = False\n while not batcher.last_batch() and progress_tracker.steps < self.total_steps and not should_break:\n progress_tracker.learning_rate = self.optimizer.param_groups[0][\"lr\"]\n self.callback(lambda c: c.on_batch_start(self, progress_tracker, save_path))\n\n # obtain batch\n batch = batcher.next_batch()\n\n # determine whether we need to accumulate gradients as trigger a full parameter update\n should_sync_grads = (batch_idx + 1) % self.gradient_accumulation_steps == 0\n is_checkpoint_step = (progress_tracker.steps + 1) % final_steps_per_checkpoint == 0\n should_step = should_sync_grads or is_checkpoint_step\n batch_idx += 1\n\n # Move tensors to cuda here.\n inputs = {\n i_feat.feature_name: torch.from_numpy(np.array(batch[i_feat.proc_column], copy=True)).to(self.device)\n for i_feat in self.model.input_features.values()\n }\n targets = {\n o_feat.feature_name: torch.from_numpy(np.array(batch[o_feat.proc_column], copy=True)).to(self.device)\n for o_feat in self.model.output_features.values()\n }\n\n loss, all_losses, used_tokens = self.train_step(inputs, targets, should_step=should_step, profiler=profiler)\n\n # Update LR schduler here instead of train loop to avoid updating during batch size tuning, etc.\n self.scheduler.step()\n\n # Update progress tracker with token information.\n progress_tracker.set_token_usage_for_this_step(used_tokens)\n\n if self.is_coordinator() and not self.skip_save_log:\n self.write_step_summary(\n train_summary_writer=train_summary_writer,\n combined_loss=loss.detach().float(),\n all_losses=all_losses,\n step=progress_tracker.steps,\n used_tokens=used_tokens,\n total_tokens_used=progress_tracker.total_tokens_used,\n learning_rate=progress_tracker.learning_rate,\n )\n\n progress_tracker.steps += 1\n progress_bar.set_postfix({\"loss\": loss.detach().item()})\n progress_bar.update(1)\n if self.is_coordinator():\n logger.debug(\n \"training: completed batch %s memory used: %.2fMB\",\n progress_bar.total_steps,\n psutil.Process(os.getpid()).memory_info()[0] / 1e6,\n )\n\n # Executing `on_batch_end` calls before `run_evaluation` enables more accurate\n # batch duration measurements when using timer callbacks.\n self.callback(lambda c: c.on_batch_end(self, progress_tracker, save_path, sync_step=should_step))\n\n # If this is the last batch in the epoch, increment before running evaluation so that metrics are reported\n # with the correct epoch.\n if batcher.last_batch():\n progress_tracker.epoch += 1\n\n if progress_tracker.steps % final_steps_per_checkpoint == 0:\n # Before continuing to evaluation or skipping evaluation altogether, we should use this point to\n # ensure that the model weights are not NaN or Inf.\n has_nan_or_inf_tensors = self._has_nan_or_inf_weights(self.dist_model)\n # If a nan/inf tensor is detected, we should break out of the training loop immediately and raise an #\n # error. There is no point in running evaluation for this step as the model weights are already in\n # a bad state. Theere is also no point in continuing to train the model since the loss will always be\n # NaN or Inf from this point forward.\n if has_nan_or_inf_tensors:\n return True, has_nan_or_inf_tensors\n\n if not self.skip_all_evaluation:\n # Publishes metrics to MLFLow if there are any MLFlow callbacks.\n should_break = self.run_evaluation(\n training_set,\n validation_set,\n test_set,\n progress_tracker,\n train_summary_writer,\n validation_summary_writer,\n test_summary_writer,\n model_hyperparameters_path,\n output_features,\n metrics_names,\n save_path,\n loss,\n all_losses,\n early_stopping_steps,\n checkpoint_manager,\n )\n else:\n should_break = False\n\n # Checkpoint the model.\n # NOTE: Ideally we would do this before evaluation, but for some reason DeepSpeed will complain\n # about inflight params if we do that, which is why we checkpoint after eval instead. In practice,\n # this should not make a difference, except in the unlikely event an error occurs during eval and we\n # want to resume from the last checkpoint, in which case we will lose slightly more progress this way.\n if not self.skip_save_progress:\n self.save_checkpoint(progress_tracker, save_path, checkpoint_manager)\n\n # If this was the last batch, then increment the epoch counter and invoke the `on_epoch_end` callback.\n if batcher.last_batch():\n self.callback(lambda c: c.on_epoch_end(self, progress_tracker, save_path))\n\n return should_break, has_nan_or_inf_tensors\n\n def _has_nan_or_inf_weights(self, model: torch.nn.Module) -> bool:\n \"\"\"Check for NaN or infinity (inf) values in the weights (parameters and buffers) of a PyTorch model in a\n local or distributed training environment. It is called to ensure the model's numerical stability during\n training. It works for both model parallel and data parallel training.\n\n This function recursively inspects the model's parameters and buffers to identify NaN or inf values. It\n communicates and aggregates the results across all distributed processes using the `all_reduce` operation. If\n any process finds NaN or inf values, it is considered a critical error, and the main coordinator process will\n return True to halt training in the main training loop.\n\n Parameters:\n model (torch.nn.Module): The PyTorch model to check for NaN or inf weights.\n\n Returns:\n bool: Returns True if any NaN or inf tensors are found in the model's weights. Otherwise, returns False.\n \"\"\"\n local_has_nan_or_inf = contains_nan_or_inf_tensors(model)\n\n # Use all_reduce to aggregate local_has_nan across all processes and sum the result into global_has_nan, which\n # will be a tensor with a single element on all processes after the all_reduce operation.\n global_has_nan_or_inf = torch.tensor(int(local_has_nan_or_inf), device=self.device)\n self.distributed.allreduce(global_has_nan_or_inf)\n\n # The main coordinator process will raise a runtime error if any of the processes found NaN or inf weights.\n if self.distributed.local_rank() == 0:\n if global_has_nan_or_inf.item() > 0:\n logger.warning(\"NaN or inf tensors found in the model. Stopping training.\")\n return True\n return False\n\n def train_online(self, dataset):\n self.dist_model.train() # Sets model training mode.\n with dataset.initialize_batcher(\n batch_size=self.batch_size,\n should_shuffle=self.should_shuffle,\n distributed=self.distributed,\n ignore_last=True,\n ) as batcher:\n # training step loop\n progress_bar_config = {\n \"desc\": \"Training online\",\n \"total\": batcher.steps_per_epoch,\n \"file\": sys.stdout,\n \"disable\": is_progressbar_disabled(),\n }\n progress_bar = LudwigProgressBar(self.report_tqdm_to_ray, progress_bar_config, self.is_coordinator())\n\n while not batcher.last_batch():\n batch = batcher.next_batch()\n inputs = {\n i_feat.feature_name: torch.from_numpy(np.array(batch[i_feat.proc_column], copy=True)).to(\n self.device\n )\n for i_feat in self.model.input_features.values()\n }\n targets = {\n o_feat.feature_name: torch.from_numpy(np.array(batch[o_feat.proc_column], copy=True)).to(\n self.device\n )\n for o_feat in self.model.output_features.values()\n }\n\n self.train_step(\n inputs,\n targets,\n )\n\n progress_bar.update(1)\n\n progress_bar.close()\n return self.model\n\n @property\n def validation_field(self):\n return self._validation_field\n\n @property\n def validation_metric(self):\n return self._validation_metric\n\n def evaluation(self, dataset, dataset_name, metrics_log, batch_size, progress_tracker):\n predictor = Predictor(\n self.dist_model,\n batch_size=batch_size,\n distributed=self.distributed,\n report_tqdm_to_ray=self.report_tqdm_to_ray,\n model=self.model,\n )\n metrics, _ = predictor.batch_evaluation(dataset, collect_predictions=False, dataset_name=dataset_name)\n\n return append_metrics(self.model, dataset_name, metrics, metrics_log, progress_tracker)\n\n def check_progress_on_validation(\n self,\n progress_tracker,\n validation_output_feature_name,\n validation_metric: str,\n save_path,\n model_hyperparameters_path,\n increase_batch_size_on_plateau,\n increase_batch_size_on_plateau_patience,\n increase_batch_size_on_plateau_rate,\n increase_batch_size_on_plateau_max,\n increase_batch_size_eval_metric,\n increase_batch_size_eval_split,\n early_stopping_steps: int,\n skip_save_model,\n checkpoint_manager: CheckpointManager,\n ) -> bool:\n \"\"\"Checks the history of validation scores.\n\n Uses history of validation scores to reduce learning rate, increase batch size, and decide whether training\n should stop.\n\n Saves the model if scores have improved.\n\n Returns whether the model should stop training.\n \"\"\"\n should_break = False\n improved_fn = get_improved_fn(validation_metric)\n\n all_validation_metrics = progress_tracker.validation_metrics[validation_output_feature_name]\n # The most recent validation_metric metric.\n eval_metric: TrainerMetric = all_validation_metrics[validation_metric][-1]\n eval_metric_value = eval_metric[-1]\n\n if eval_metric_value != eval_metric_value:\n # Fallback to 0 if the validation metric value is a NaN.\n # This is potentially relevant for small datasets like those used in testing where if there's only a\n # single output label, some metrics like ROC may turn out to be NaN.\n # However, we want to guarantee that the model will be saved at least once over a full\n # training-checkpoint-eval-loop.\n eval_metric_value = 0\n\n if improved_fn(eval_metric_value, progress_tracker.best_eval_metric_value):\n previous_best_eval_metric_value = progress_tracker.best_eval_metric_value\n\n # Save the value, steps, epoch, and checkpoint number.\n progress_tracker.best_eval_metric_value = eval_metric_value\n progress_tracker.best_eval_metric_steps = progress_tracker.steps\n progress_tracker.best_eval_metric_epoch = progress_tracker.epoch\n progress_tracker.best_eval_metric_checkpoint_number = progress_tracker.checkpoint_number\n\n # Save best metrics for all data subsets.\n progress_tracker.best_eval_train_metrics = get_latest_metrics_dict(progress_tracker.train_metrics)\n progress_tracker.best_eval_validation_metrics = get_latest_metrics_dict(progress_tracker.validation_metrics)\n progress_tracker.best_eval_test_metrics = get_latest_metrics_dict(progress_tracker.test_metrics)\n\n if self.is_coordinator():\n logger.info(\n f\"Evaluation validation metric: '{validation_output_feature_name}' '{validation_metric}' improved.\"\n )\n absolute_eval_metric_value_change = round(\n abs(previous_best_eval_metric_value - progress_tracker.best_eval_metric_value), 3\n )\n if get_metric_objective(validation_metric) == MINIMIZE:\n logger.info(\n f\"'{validation_output_feature_name}' '{validation_metric}' decreased by \"\n f\"{absolute_eval_metric_value_change}.\"\n )\n else:\n logger.info(\n f\"'{validation_output_feature_name}' '{validation_metric}' increased by \"\n f\"{absolute_eval_metric_value_change}.\"\n )\n\n # Save the model.\n if not skip_save_model:\n logger.info(\"New best model saved.\\n\")\n checkpoint_manager.save_best(progress_tracker.steps)\n self.callback(lambda c: c.on_save_best_checkpoint(self, progress_tracker, save_path))\n\n last_improvement_in_steps = progress_tracker.steps - progress_tracker.best_eval_metric_steps\n progress_tracker.last_improvement_steps = last_improvement_in_steps\n\n if last_improvement_in_steps != 0 and self.is_coordinator():\n logger.info(\n f\"Last improvement of {validation_output_feature_name} validation {validation_metric} happened \"\n + f\"{last_improvement_in_steps} step(s) ago.\\n\"\n )\n\n # ========== Learning Rate Schedule evaluation updates ========\n self.scheduler.eval_step(progress_tracker, validation_output_feature_name)\n\n # ========== Increase Batch Size Plateau logic =========\n if increase_batch_size_on_plateau > 0:\n self.increase_batch_size(\n progress_tracker,\n validation_output_feature_name,\n increase_batch_size_on_plateau,\n increase_batch_size_on_plateau_patience,\n increase_batch_size_on_plateau_rate,\n increase_batch_size_on_plateau_max,\n increase_batch_size_eval_metric,\n increase_batch_size_eval_split,\n )\n progress_tracker.last_increase_batch_size = (\n progress_tracker.steps - progress_tracker.last_increase_batch_size_steps\n )\n if (\n progress_tracker.last_increase_batch_size > 0\n and progress_tracker.last_increase_batch_size_eval_metric_improvement > 0\n and not progress_tracker.num_increases_batch_size >= increase_batch_size_on_plateau\n and not progress_tracker.batch_size >= increase_batch_size_on_plateau_max\n ):\n logger.info(\n \"Last batch size increase \"\n f\"happened {progress_tracker.last_increase_batch_size} step(s) ago, \"\n f\"improvement of {validation_output_feature_name} {increase_batch_size_eval_split} \"\n f\"{increase_batch_size_eval_metric} happened \"\n f\"{progress_tracker.last_increase_batch_size_eval_metric_improvement} step(s) ago.\"\n )\n\n # ========== Early Stop logic ==========\n # If any early stopping condition is satisfied, either lack of improvement for many steps, or via callbacks on\n # any worker, then trigger early stopping.\n early_stop_bool = 0 < early_stopping_steps <= last_improvement_in_steps\n if not early_stop_bool:\n for callback in self.callbacks:\n if callback.should_early_stop(self, progress_tracker, self.is_coordinator()):\n early_stop_bool = True\n break\n\n should_early_stop = torch.as_tensor([early_stop_bool], dtype=torch.int, device=self.device)\n should_early_stop = self.distributed.allreduce(should_early_stop)\n if should_early_stop.item():\n if self.is_coordinator():\n logger.info(\n f\"\\nEARLY STOPPING due to lack of validation improvement. It has been {last_improvement_in_steps} \"\n \"step(s) since last validation improvement.\"\n )\n should_break = True\n return should_break\n\n def set_steps_to_1_or_quit(self, signum, frame):\n \"\"\"Custom SIGINT handler used to elegantly exit training.\n\n A single SIGINT will stop training after the next training step. A second SIGINT will stop training immediately.\n \"\"\"\n if not self.received_sigint:\n self.total_steps = 1\n self.received_sigint = True\n logger.critical(\"\\nReceived SIGINT, will finish this training step and then conclude training.\")\n logger.critical(\"Send another SIGINT to immediately interrupt the process.\")\n else:\n logger.critical(\"\\nReceived a second SIGINT, will now quit\")\n if self.original_sigint_handler:\n signal.signal(signal.SIGINT, self.original_sigint_handler)\n sys.exit(1)\n\n @staticmethod\n def resume_files_exist(\n training_progress_tracker_path: str,\n training_checkpoint_path: str,\n ) -> bool:\n missing_files = []\n # training_progress.json\n if not path_exists(training_progress_tracker_path):\n missing_files.append(training_progress_tracker_path)\n # latest.ckpt in training_checkpoints/\n latest_ckpt = os.path.join(training_checkpoint_path, \"latest.ckpt\")\n if not path_exists(latest_ckpt):\n missing_files.append(latest_ckpt)\n if missing_files:\n logger.warning(f\"Could not find {missing_files} while trying to resume model training.\")\n return False\n return True\n\n def resume_training_progress_tracker(self, training_progress_tracker_path):\n progress_tracker_dict = None\n if self.is_coordinator():\n logger.info(f\"Loading progress tracker for model: {training_progress_tracker_path}\")\n progress_tracker_dict = load_json(training_progress_tracker_path)\n\n logger.debug(\"Broadcasting model progress tracker dict to all workers\")\n progress_tracker_dict = self.distributed.broadcast_object(\n progress_tracker_dict, name=\"broadcast_progress_tracker\"\n )\n\n progress_tracker = ProgressTracker.load(progress_tracker_dict)\n return progress_tracker\n\n def resume_weights_and_optimizer(\n self,\n model_weights_progress_path: str,\n checkpoint: Checkpoint,\n ):\n CheckpointManager.load_latest_checkpoint(checkpoint, model_weights_progress_path, self.device)\n\n def increase_batch_size(\n self,\n progress_tracker: ProgressTracker,\n validation_output_feature_name: str,\n increase_batch_size_on_plateau: int,\n increase_batch_size_on_plateau_patience: int,\n increase_batch_size_on_plateau_rate: float,\n increase_batch_size_on_plateau_max: int,\n increase_batch_size_eval_metric: str = LOSS,\n increase_batch_size_eval_split: str = TRAINING,\n ):\n \"\"\"Uses the progress tracker to determine if the batch size should be increased.\"\"\"\n if (\n not progress_tracker.num_increases_batch_size >= increase_batch_size_on_plateau\n and not progress_tracker.batch_size == increase_batch_size_on_plateau_max\n ):\n if increase_batch_size_eval_split == TRAINING:\n split_metrics = progress_tracker.train_metrics\n elif increase_batch_size_eval_split == VALIDATION:\n split_metrics = progress_tracker.validation_metrics\n else: # if increase_batch_size_eval_split == TEST:\n split_metrics = progress_tracker.test_metrics\n\n validation_metric = increase_batch_size_eval_metric\n last_metric = split_metrics[validation_output_feature_name][validation_metric][-1]\n last_metric_value = last_metric[-1]\n\n improved_fn = get_improved_fn(validation_metric)\n is_improved = improved_fn(last_metric_value, progress_tracker.best_increase_batch_size_eval_metric)\n if is_improved:\n # We update the best metric value and set it to the current one, and reset last\n # improvement step count\n progress_tracker.best_increase_batch_size_eval_metric = last_metric_value\n progress_tracker.last_increase_batch_size_eval_metric_improvement = 0\n else:\n progress_tracker.last_increase_batch_size_eval_metric_improvement += 1\n if not is_improved and (\n # Batch size increase happened more than N steps ago\n progress_tracker.last_increase_batch_size >= increase_batch_size_on_plateau_patience\n and (\n # No improvement of the evaluation metric since more than N steps ago\n progress_tracker.last_increase_batch_size_eval_metric_improvement\n >= increase_batch_size_on_plateau_patience\n )\n ):\n progress_tracker.batch_size = min(\n int(increase_batch_size_on_plateau_rate * progress_tracker.batch_size),\n increase_batch_size_on_plateau_max,\n )\n\n if self.is_coordinator():\n logger.info(\n f\"PLATEAU REACHED, increasing batch size to {progress_tracker.batch_size} due to lack of \"\n f\"improvement of {validation_output_feature_name} {increase_batch_size_eval_split} \"\n f\"{validation_metric}.\"\n )\n\n progress_tracker.last_increase_batch_size_steps = progress_tracker.steps\n progress_tracker.last_increase_batch_size = 0\n progress_tracker.num_increases_batch_size += 1\n\n if progress_tracker.num_increases_batch_size >= increase_batch_size_on_plateau:\n if self.is_coordinator():\n logger.info(\n f\"Batch size was already increased {progress_tracker.num_increases_batch_size} times, \"\n \"not increasing it anymore.\"\n )\n elif progress_tracker.batch_size >= increase_batch_size_on_plateau_max:\n if self.is_coordinator():\n logger.info(\n f\"Batch size was already increased {progress_tracker.num_increases_batch_size} times, \"\n f\"currently it is {progress_tracker.batch_size}, the maximum allowed.\"\n )\n\n def is_coordinator(self):\n return self.distributed.rank() == 0\n\n @property\n def local_rank(self) -> int:\n return self.distributed.local_rank()\n\n def barrier(self):\n self.distributed.barrier()\n\n def callback(self, fn, coordinator_only=True):\n if not coordinator_only or self.is_coordinator():\n for callback in self.callbacks:\n fn(callback)\n\n @property\n def return_device(self):\n return self.device\n\n\nclass RemoteTrainer(Trainer):\n def __init__(self, gpus=None, gpu_memory_limit=None, allow_parallel_threads=True, **kwargs):\n super().__init__(**kwargs)\n\n # Only return results from rank 0 to reduce network overhead\n self.train = self.distributed.return_first(self.train)\n self.train_online = self.distributed.return_first(self.train_online)\n\n @property\n def return_device(self):\n # When returning the model weights from remote to driver, place them on CPU,\n # as the driver likely doesn't have a GPU.\n return \"cpu\"\n\n\nlearning_rate_scale_fns = {\n \"linear\": lambda n: n,\n \"sqrt\": lambda n: math.sqrt(n),\n \"constant\": lambda n: 1,\n}\n"
},
{
"path": "ludwig/trainers/trainer_llm.py",
"content": "import logging\nimport os\nimport time\nfrom collections.abc import Callable\nfrom typing import Union\n\nfrom torch.utils.tensorboard import SummaryWriter\n\nfrom ludwig.constants import MINIMUM_BATCH_SIZE, TEST, TRAINING, VALIDATION\nfrom ludwig.data.dataset.base import Dataset\nfrom ludwig.distributed.base import DistributedStrategy, LocalStrategy\nfrom ludwig.features.feature_utils import LudwigFeatureDict\nfrom ludwig.globals import MODEL_FILE_NAME\nfrom ludwig.models.llm import LLM\nfrom ludwig.models.predictor import LlmFineTunePredictor, LlmPredictor\nfrom ludwig.modules.metric_modules import get_initial_validation_value\nfrom ludwig.schema.trainer import BaseTrainerConfig, FineTuneTrainerConfig, NoneTrainerConfig\nfrom ludwig.trainers.base import BaseTrainer\nfrom ludwig.trainers.registry import register_llm_ray_trainer, register_llm_trainer\nfrom ludwig.trainers.trainer import Trainer\nfrom ludwig.types import ModelConfigDict\nfrom ludwig.utils import time_utils\nfrom ludwig.utils.batch_size_tuner import (\n BatchSizeEvaluator,\n LLMFinetunePredictBatchSizeEvaluator,\n LLMFinetuneTrainerBatchSizeEvaluator,\n)\nfrom ludwig.utils.defaults import default_random_seed\nfrom ludwig.utils.metric_utils import TrainerMetric\nfrom ludwig.utils.metrics_printed_table import print_metrics_table\nfrom ludwig.utils.misc_utils import set_random_seed\nfrom ludwig.utils.torch_utils import get_torch_device\nfrom ludwig.utils.trainer_utils import append_metrics, get_new_progress_tracker, ProgressTracker\n\nlogger = logging.getLogger(__name__)\n\nMAX_EVALUATION_EXAMPLES = 1000\nMAX_EVALUATION_EXAMPLES_SHOWN = 5\n\n\n@register_llm_trainer(\"none\")\n@register_llm_ray_trainer(\"none\")\nclass NoneTrainer(BaseTrainer):\n \"\"\"NoneTrainer is a trainer that does not train a model, only runs evaluation.\"\"\"\n\n def __init__(\n self,\n config: NoneTrainerConfig,\n model: LLM,\n resume: float = False,\n skip_save_model: bool = False,\n skip_save_progress: bool = False,\n skip_save_log: bool = False,\n callbacks: list = None,\n report_tqdm_to_ray=False,\n random_seed: float = default_random_seed,\n distributed: DistributedStrategy | None = None,\n device: str | None = None,\n **kwargs,\n ):\n \"\"\"\n :param config: `ludwig.schema.trainer.NoneTrainerConfig` instance that specifies training hyperparameters\n (default: `ludwig.schema.trainer.NoneTrainerConfig()`).\n :param model: Underlying Ludwig model\n :type model: `ludwig.models.llm.LLM`\n :param resume: Resume training a model that was being trained. (default: False).\n :type resume: Boolean\n :param skip_save_model: Disables saving model weights and hyperparameters each time the model improves. By\n default Ludwig saves model weights after each round of evaluation the validation metric (improves, but\n if the model is really big that can be time consuming. If you do not want to keep the weights and just\n find out what performance a model can get with a set of hyperparameters, use this parameter to skip it,\n but the model will not be loadable later on. (default: False).\n :type skip_save_model: Boolean\n :param skip_save_progress: Disables saving progress each round of evaluation. By default Ludwig saves weights\n and stats after each round of evaluation for enabling resuming of training, but if the model is really\n big that can be time consuming and will uses twice as much space, use this parameter to skip it, but\n training cannot be resumed later on. (default: False).\n :type skip_save_progress: Boolean\n :param skip_save_log: Disables saving TensorBoard logs. By default Ludwig saves logs for the TensorBoard, but if\n it is not needed turning it off can slightly increase the overall speed. (default: False).\n :type skip_save_log: Boolean\n :param callbacks: List of `ludwig.callbacks.Callback` objects that provide hooks into the Ludwig pipeline.\n (default: None).\n :type callbacks: list\n :param report_tqdm_to_ray: Enables using the ray based tqdm Callback for progress bar reporting\n :param random_seed: Default initialization for the random seeds (default: 42).\n :type random_seed: Float\n :param distributed: Distributed strategy (default: None).\n :type distributed: `DistributedStrategy`\n :param device: Device to load the model on from a saved checkpoint (default: None).\n :type device: str\n \"\"\"\n\n super().__init__()\n\n # Ensure distributed strategy is initialized for metric sync_context.\n # NoneTrainer may run on the head node (not in a Ray Train worker),\n # so init_dist_strategy may not have been called yet.\n from ludwig.distributed import init_dist_strategy\n\n init_dist_strategy(\"local\")\n\n self.config = config\n self.distributed = distributed if distributed is not None else LocalStrategy()\n self.skip_save_log = skip_save_log\n self.resume = resume\n self.skip_save_model = skip_save_model\n self.skip_save_progress = skip_save_progress\n self.random_seed = random_seed\n self.callbacks = callbacks or []\n self.report_tqdm_to_ray = report_tqdm_to_ray\n\n self.device = device if device is not None else get_torch_device()\n self.model = model.to_device(self.device)\n self.model.metrics_to_device(self.device)\n\n # Since we are only running evaluation without training, set the model to evaluation mode.\n self.model.eval()\n\n self.batch_size = self.config.batch_size\n self.eval_batch_size = self.config.eval_batch_size\n self.base_learning_rate = self.config.base_learning_rate\n self.should_shuffle = self.config.should_shuffle\n self.epochs = self.config.epochs\n self.train_steps = self.config.train_steps\n self.steps_per_checkpoint = self.config.steps_per_checkpoint\n self.checkpoints_per_epoch = self.config.checkpoints_per_epoch\n self.early_stop = self.config.early_stop\n self.evaluate_training_set = self.config.evaluate_training_set\n self.skip_all_evaluation = self.config.skip_all_evaluation\n\n def close_writers(\n self, progress_tracker, save_path, train_summary_writer, validation_summary_writer, test_summary_writer\n ):\n # ================ Finished Training ================\n self.callback(\n lambda c: c.on_trainer_train_teardown(self, progress_tracker, save_path, self.is_coordinator()),\n coordinator_only=False,\n )\n\n if train_summary_writer is not None:\n train_summary_writer.close()\n if validation_summary_writer is not None:\n validation_summary_writer.close()\n if test_summary_writer is not None:\n test_summary_writer.close()\n\n def train(\n self,\n training_set: Dataset,\n validation_set: Dataset | None = None,\n test_set: Dataset | None = None,\n save_path: str = MODEL_FILE_NAME,\n return_state_dict: bool = False,\n **kwargs,\n ):\n output_features = self.model.output_features\n\n # ====== Setup file names =======\n tensorboard_log_dir = None\n if self.is_coordinator():\n os.makedirs(save_path, exist_ok=True)\n tensorboard_log_dir = os.path.join(save_path, \"logs\")\n\n self.callback(\n lambda c: c.on_trainer_train_setup(self, save_path, self.is_coordinator()), coordinator_only=False\n )\n\n train_summary_writer = None\n validation_summary_writer = None\n test_summary_writer = None\n if self.is_coordinator() and not self.skip_save_log and tensorboard_log_dir:\n train_summary_writer = SummaryWriter(os.path.join(tensorboard_log_dir, TRAINING))\n if validation_set is not None and validation_set.size > 0:\n validation_summary_writer = SummaryWriter(os.path.join(tensorboard_log_dir, VALIDATION))\n if test_set is not None and test_set.size > 0:\n test_summary_writer = SummaryWriter(os.path.join(tensorboard_log_dir, TEST))\n\n set_random_seed(self.random_seed)\n\n progress_tracker = get_new_progress_tracker(\n batch_size=self.batch_size,\n learning_rate=self.base_learning_rate,\n best_eval_metric_value=get_initial_validation_value(self.validation_metric),\n best_increase_batch_size_eval_metric=get_initial_validation_value(self.validation_metric),\n output_features=output_features,\n )\n\n # When running with Ray, we only need to return the state dict, as it's faster and cheaper to send the\n # state dict over the network than to load the model state here, serialize it back to a state dict, then\n # load it back on the head node.\n return_value = self.model if not return_state_dict else self.model.cpu().state_dict()\n\n if self.skip_all_evaluation:\n self.close_writers(\n progress_tracker, save_path, train_summary_writer, validation_summary_writer, test_summary_writer\n )\n return (\n return_value,\n progress_tracker.train_metrics,\n progress_tracker.validation_metrics,\n progress_tracker.test_metrics,\n )\n\n try:\n self.run_evaluation(\n training_set,\n validation_set,\n test_set,\n progress_tracker,\n train_summary_writer,\n validation_summary_writer,\n test_summary_writer,\n output_features,\n save_path,\n )\n finally:\n self.close_writers(\n progress_tracker, save_path, train_summary_writer, validation_summary_writer, test_summary_writer\n )\n\n return (\n return_value,\n progress_tracker.train_metrics,\n progress_tracker.validation_metrics,\n progress_tracker.test_metrics,\n )\n\n def train_online(\n self,\n dataset,\n ):\n pass\n\n def tune_batch_size(\n self,\n config: ModelConfigDict,\n training_set: Dataset,\n random_seed: int = default_random_seed,\n max_trials: int = 20,\n halving_limit: int = 3,\n snapshot_weights: bool = True,\n on_best_batch_size_updated: Callable[[int, float, int], None] | None = None,\n tune_for_training: bool = True,\n ) -> int:\n # TODO: Implement batch size tuning for LLM, currently just returns the default batch size\n # Compared to ECD, this just requires forward passes till we OOM.\n # https://github.com/ludwig-ai/ludwig/issues/3525\n return MINIMUM_BATCH_SIZE\n\n @property\n def validation_field(self):\n return self.config.validation_field\n\n @property\n def validation_metric(self):\n return self.config.validation_metric\n\n # Remote implementations may override this\n def shutdown(self):\n pass\n\n @property\n def local_rank(self) -> int:\n return 0\n\n def barrier(self):\n pass\n\n # Functions needed to treat Trainer as a context manager\n def __enter__(self):\n return self\n\n def __exit__(self, exc_type, exc_val, exc_tb):\n self.shutdown()\n\n @staticmethod\n def get_schema_cls() -> BaseTrainerConfig:\n return NoneTrainerConfig\n\n def is_coordinator(self) -> bool:\n return self.distributed.rank() == 0\n\n def callback(self, fn, coordinator_only=True):\n if not coordinator_only or self.is_coordinator():\n for callback in self.callbacks:\n fn(callback)\n\n def evaluation(\n self,\n dataset: \"Dataset\", # noqa: F821\n dataset_name: str,\n metrics_log: dict[str, dict[str, list[TrainerMetric]]],\n batch_size: int,\n progress_tracker: ProgressTracker,\n ):\n predictor = LlmPredictor(\n self.model, batch_size=batch_size, distributed=self.distributed, report_tqdm_to_ray=self.report_tqdm_to_ray\n )\n metrics, _ = predictor.batch_evaluation(dataset, collect_predictions=False, dataset_name=dataset_name)\n\n return append_metrics(self.model, dataset_name, metrics, metrics_log, progress_tracker)\n\n @classmethod\n def write_eval_summary(\n cls,\n summary_writer,\n metrics,\n step,\n ):\n if not summary_writer:\n return\n\n for feature_name, output_feature in metrics.items():\n for metric_name, metrics in output_feature.items():\n if metrics:\n metric_tag = f\"{feature_name}/epoch_{metric_name}\"\n metric_val = metrics[-1][-1]\n summary_writer.add_scalar(metric_tag, metric_val, global_step=step)\n summary_writer.flush()\n\n def run_evaluation(\n self,\n training_set: Union[\"Dataset\", \"RayDataset\"], # noqa: F821\n validation_set: Union[\"Dataset\", \"RayDataset\"] | None, # noqa: F821\n test_set: Union[\"Dataset\", \"RayDataset\"] | None, # noqa: F821\n progress_tracker: ProgressTracker,\n train_summary_writer: SummaryWriter,\n validation_summary_writer: SummaryWriter,\n test_summary_writer: SummaryWriter,\n output_features: LudwigFeatureDict,\n save_path: str,\n ) -> bool:\n \"\"\"Runs evaluation over training, validation, and test sets.\n\n Also:\n - Prints results, saves results to the progress tracker.\n - Saves the model if the validation score is the best so far\n - If there is no validation set, the model is always saved.\n\n Returns whether the trainer should early stop, based on validation metrics history.\n \"\"\"\n start_time = time.time()\n self.callback(lambda c: c.on_eval_start(self, progress_tracker, save_path))\n\n progress_tracker.checkpoint_number += 1\n if self.is_coordinator():\n logger.info(f\"\\nRunning evaluation for step: {progress_tracker.steps}, epoch: {progress_tracker.epoch}\")\n\n # ================ Eval ================\n # Run a separate pass over the training data to compute metrics\n # Appends results to progress_tracker.train_metrics.\n if self.evaluate_training_set:\n self.evaluation(\n training_set, \"train\", progress_tracker.train_metrics, self.eval_batch_size, progress_tracker\n )\n\n self.write_eval_summary(\n summary_writer=train_summary_writer,\n metrics=progress_tracker.train_metrics,\n step=progress_tracker.steps,\n )\n\n if validation_set is not None:\n self.callback(lambda c: c.on_validation_start(self, progress_tracker, save_path))\n\n # eval metrics on validation set\n self.evaluation(\n validation_set,\n VALIDATION,\n progress_tracker.validation_metrics,\n self.eval_batch_size,\n progress_tracker,\n )\n\n self.write_eval_summary(\n summary_writer=validation_summary_writer,\n metrics=progress_tracker.validation_metrics,\n step=progress_tracker.steps,\n )\n\n self.callback(lambda c: c.on_validation_end(self, progress_tracker, save_path))\n\n if test_set is not None:\n self.callback(lambda c: c.on_test_start(self, progress_tracker, save_path))\n\n # eval metrics on test set\n self.evaluation(test_set, TEST, progress_tracker.test_metrics, self.eval_batch_size, progress_tracker)\n\n self.write_eval_summary(\n summary_writer=test_summary_writer,\n metrics=progress_tracker.test_metrics,\n step=progress_tracker.steps,\n )\n\n self.callback(lambda c: c.on_test_end(self, progress_tracker, save_path))\n\n elapsed_time = (time.time() - start_time) * 1000.0\n\n if self.is_coordinator():\n logger.info(f\"Evaluation took {time_utils.strdelta(elapsed_time)}\\n\")\n print_metrics_table(\n output_features,\n progress_tracker.train_metrics,\n progress_tracker.validation_metrics,\n progress_tracker.test_metrics,\n )\n\n # Trigger eval end callback after any model weights save for complete checkpoint\n self.callback(lambda c: c.on_eval_end(self, progress_tracker, save_path))\n\n return False\n\n\n@register_llm_trainer(\"finetune\")\nclass FineTuneTrainer(Trainer):\n @staticmethod\n def get_schema_cls():\n return FineTuneTrainerConfig\n\n def __init__(\n self,\n config: FineTuneTrainerConfig,\n model: LLM,\n resume: float = False,\n skip_save_model: bool = False,\n skip_save_progress: bool = False,\n skip_save_log: bool = False,\n callbacks: list = None,\n report_tqdm_to_ray=False,\n random_seed: int = default_random_seed,\n distributed: DistributedStrategy | None = None,\n device: str | None = None,\n **kwargs,\n ):\n super().__init__(\n config,\n model,\n resume,\n skip_save_model,\n skip_save_progress,\n skip_save_log,\n callbacks,\n report_tqdm_to_ray,\n random_seed,\n distributed,\n device,\n **kwargs,\n )\n\n def evaluation(self, dataset, dataset_name, metrics_log, batch_size, progress_tracker):\n predictor = LlmFineTunePredictor(\n self.model, batch_size=batch_size, distributed=self.distributed, report_tqdm_to_ray=self.report_tqdm_to_ray\n )\n metrics, _, input_target_output_dict = predictor.batch_evaluation(\n dataset, collect_predictions=False, dataset_name=dataset_name\n )\n # Setting collect_predictions=True currently causes an error when doing batch evaluation because the outputs\n # can be of variable sizes but we try to concatenate them into a single tensor.\n\n tokenizer = self.dist_model.tokenizer\n\n # There should only be one key in the dict for LLMs\n input_key = list(input_target_output_dict[\"inputs\"].keys())[0]\n num_examples = min(len(input_target_output_dict[\"inputs\"][input_key]), MAX_EVALUATION_EXAMPLES)\n\n llm_eval_examples = {\"inputs\": [], \"targets\": [], \"outputs\": []}\n for key in input_target_output_dict[\"inputs\"]:\n for inp in input_target_output_dict[\"inputs\"][key][:num_examples]:\n llm_eval_examples[\"inputs\"].append(tokenizer.decode(inp, skip_special_tokens=True))\n\n for key in input_target_output_dict[\"targets\"]:\n for tar in input_target_output_dict[\"targets\"][key][:num_examples]:\n llm_eval_examples[\"targets\"].append(tokenizer.decode(tar, skip_special_tokens=True))\n\n for key in input_target_output_dict[\"outputs\"]:\n for out in input_target_output_dict[\"outputs\"][key][:num_examples]:\n llm_eval_examples[\"outputs\"].append(tokenizer.decode(out, skip_special_tokens=True))\n\n num_examples_shown = min(len(llm_eval_examples[\"inputs\"]), MAX_EVALUATION_EXAMPLES_SHOWN)\n for i in range(num_examples_shown):\n logger.info(f\"Input: {llm_eval_examples['inputs'][i].strip()}\")\n logger.info(f\"Output: {llm_eval_examples['outputs'][i].strip()}\")\n logger.info(\"--------------------\")\n\n progress_tracker.llm_eval_examples = llm_eval_examples\n return append_metrics(self.model, dataset_name, metrics, metrics_log, progress_tracker)\n\n def tune_batch_size(\n self,\n config: ModelConfigDict,\n training_set: Dataset,\n random_seed: int = default_random_seed,\n max_trials: int = 20,\n halving_limit: int = 3,\n snapshot_weights: bool = True,\n on_best_batch_size_updated: Callable[[int, float, int], None] | None = None,\n tune_for_training: bool = True,\n global_max_sequence_length: int | None = None,\n ) -> int:\n if global_max_sequence_length is None:\n global_max_sequence_length = self.model.global_max_sequence_length\n return super().tune_batch_size(\n config,\n training_set,\n random_seed,\n max_trials,\n halving_limit,\n snapshot_weights,\n on_best_batch_size_updated,\n tune_for_training,\n global_max_sequence_length,\n )\n\n def _create_batch_size_evaluator(self) -> BatchSizeEvaluator:\n return LLMFinetuneTrainerBatchSizeEvaluator(self)\n\n def _create_predict_batch_size_evaluator(self) -> BatchSizeEvaluator:\n return LLMFinetunePredictBatchSizeEvaluator(self)\n\n\nclass RemoteLLMTrainer(NoneTrainer):\n def __init__(self, gpus=None, gpu_memory_limit=None, allow_parallel_threads=True, **kwargs):\n super().__init__(**kwargs)\n\n # Only return results from rank 0 to reduce network overhead\n self.train = self.distributed.return_first(self.train)\n self.train_online = self.distributed.return_first(self.train_online)\n\n\nclass RemoteLLMFineTuneTrainer(FineTuneTrainer):\n def __init__(self, gpus=None, gpu_memory_limit=None, allow_parallel_threads=True, **kwargs):\n super().__init__(**kwargs)\n\n # Only return results from rank 0 to reduce network overhead\n self.train = self.distributed.return_first(self.train)\n self.train_online = self.distributed.return_first(self.train_online)\n"
},
{
"path": "ludwig/types.py",
"content": "\"\"\"Public API: Common typing for Ludwig dictionary parameters.\"\"\"\n\nfrom typing import Any\n\nFeatureConfigDict = dict[str, Any]\n\"\"\"Dictionary of parameters used to configure an input or output feature.\n\nhttps://ludwig.ai/latest/configuration/features/supported_data_types/\n\"\"\"\n\nModelConfigDict = dict[str, Any]\n\"\"\"Dictionary representation of the ModelConfig object.\n\nhttps://ludwig.ai/latest/configuration/\n\"\"\"\n\nTrainingSetMetadataDict = dict[str, Any]\n\"\"\"Training set metadata, which consists of internal configuration parameters.\"\"\"\n\nPreprocessingConfigDict = dict[str, Any]\n\"\"\"Dictionary of parameters used to configure preprocessing.\n\nMay be type-defaults global preprocessing or feature-specific preprocessing.\nhttps://ludwig.ai/latest/configuration/preprocessing/\n\"\"\"\n\nHyperoptConfigDict = dict[str, Any]\n\"\"\"Dictionary of parameters used to configure hyperopt.\n\nhttps://ludwig.ai/latest/configuration/hyperparameter_optimization/\n\"\"\"\n\nTrainerConfigDict = dict[str, Any]\n\"\"\"Dictionary of parameters used to configure training.\n\nhttps://ludwig.ai/latest/configuration/trainer/\n\"\"\"\n\nFeatureTypeDefaultsDict = dict[str, FeatureConfigDict]\n\"\"\"Dictionary of type to parameters that configure the defaults for that feature type.\n\nhttps://ludwig.ai/latest/configuration/defaults/\n\"\"\"\n\nFeatureMetadataDict = dict[str, Any]\n\"\"\"Metadata for a specific feature like idx2str.\"\"\"\n\nFeaturePostProcessingOutputDict = dict[str, Any]\n\"\"\"Output from feature post-processing.\"\"\"\n"
},
{
"path": "ludwig/upload.py",
"content": "import argparse\nimport logging\nimport os\nimport sys\n\nfrom ludwig.globals import MODEL_FILE_NAME, MODEL_HYPERPARAMETERS_FILE_NAME, MODEL_WEIGHTS_FILE_NAME\nfrom ludwig.utils.print_utils import get_logging_level_registry\nfrom ludwig.utils.upload_utils import HuggingFaceHub, Predibase\n\nlogger = logging.getLogger(__name__)\n\n\ndef get_upload_registry():\n return {\n \"hf_hub\": HuggingFaceHub,\n \"predibase\": Predibase,\n }\n\n\ndef upload_cli(\n service: str,\n repo_id: str,\n model_path: str,\n repo_type: str = \"model\",\n private: bool = False,\n commit_message: str = \"Upload trained [Ludwig](https://ludwig.ai/latest/) model weights\",\n commit_description: str | None = None,\n dataset_file: str | None = None,\n dataset_name: str | None = None,\n **kwargs,\n) -> None:\n \"\"\"Create an empty repo on the HuggingFace Hub and upload trained model artifacts to that repo.\n\n Args:\n service (`str`):\n Name of the hosted model service to push the trained artifacts to.\n Currently, this only supports `hf_hub` and `predibase`.\n repo_id (`str`):\n A namespace (user or an organization) and a repo name separated\n by a `/`.\n model_path (`str`):\n The path of the saved model. This is the parent-folder of the folder\n where the 'model_weights' folder and the 'model_hyperparameters.json' file\n are stored.\n private (`bool`, *optional*, defaults to `False`):\n Whether the model repo should be private.\n repo_type (`str`, *optional*):\n Set to `\"dataset\"` or `\"space\"` if uploading to a dataset or\n space, `None` or `\"model\"` if uploading to a model. Default is\n `None`.\n commit_message (`str`, *optional*):\n The summary / title / first line of the generated commit. Defaults to:\n `f\"Upload {path_in_repo} with huggingface_hub\"`\n commit_description (`str` *optional*):\n The description of the generated commit\n dataset_file (`str`, *optional*):\n The path to the dataset file. Required if `service` is set to\n `\"predibase\"` for new model repos.\n dataset_name (`str`, *optional*):\n The name of the dataset. Used by the `service`\n `\"predibase\"`.\n \"\"\"\n model_service = get_upload_registry().get(service, \"hf_hub\")\n hub: HuggingFaceHub = model_service()\n if os.path.exists(os.path.join(model_path, MODEL_FILE_NAME, MODEL_WEIGHTS_FILE_NAME)) and os.path.exists(\n os.path.join(model_path, MODEL_FILE_NAME, MODEL_HYPERPARAMETERS_FILE_NAME)\n ):\n experiment_path = model_path\n elif os.path.exists(os.path.join(model_path, MODEL_WEIGHTS_FILE_NAME)) and os.path.exists(\n os.path.join(model_path, MODEL_HYPERPARAMETERS_FILE_NAME)\n ):\n experiment_path = os.path.normpath(os.path.join(model_path, \"..\"))\n else:\n raise ValueError(\n f\"Can't find 'model_weights' and '{MODEL_HYPERPARAMETERS_FILE_NAME}' either at \"\n f\"'{model_path}' or at '{model_path}/model'\"\n )\n hub.upload(\n repo_id=repo_id,\n model_path=experiment_path,\n repo_type=repo_type,\n private=private,\n commit_message=commit_message,\n commit_description=commit_description,\n dataset_file=dataset_file,\n dataset_name=dataset_name,\n )\n\n\ndef cli(sys_argv):\n parser = argparse.ArgumentParser(\n description=\"This script pushes a trained model to a hosted model repository service\",\n prog=\"ludwig upload\",\n usage=\"%(prog)s [options]\",\n )\n\n # ---------------\n # Required parameters\n # ---------------\n parser.add_argument(\n \"service\",\n help=\"Name of the model repository service.\",\n default=\"hf_hub\",\n choices=[\"hf_hub\", \"predibase\"],\n )\n\n parser.add_argument(\n \"-r\",\n \"--repo_id\",\n help=\"Name of the repo. This will be created if it doesn't exist. Format: username/repo_name\",\n required=True,\n )\n\n parser.add_argument(\"-m\", \"--model_path\", help=\"Path of the trained model on disk\", required=True)\n\n # ---------------\n # Optional parameters\n # ---------------\n parser.add_argument(\"-p\", \"--private\", help=\"Make the repo private\", default=False, choices=[True, False])\n\n parser.add_argument(\n \"-t\", \"--repo_type\", help=\"Type of repo\", default=\"model\", choices=[\"model\", \"space\", \"dataset\"]\n )\n\n parser.add_argument(\n \"-c\",\n \"--commit_message\",\n help=\"The summary / title / first line of the generated commit.\",\n default=\"Upload trained [Ludwig](https://ludwig.ai/latest/) model weights\",\n )\n\n parser.add_argument(\"-d\", \"--commit_description\", help=\"The description of the generated commit\", default=None)\n\n parser.add_argument(\n \"-l\",\n \"--logging_level\",\n default=\"info\",\n help=\"The level of logging to use\",\n choices=[\"critical\", \"error\", \"warning\", \"info\", \"debug\", \"notset\"],\n )\n\n parser.add_argument(\"-df\", \"--dataset_file\", help=\"The location of the dataset file\", default=None)\n parser.add_argument(\n \"-dn\", \"--dataset_name\", help=\"(Optional) The name of the dataset in the Provider\", default=None\n )\n\n args = parser.parse_args(sys_argv)\n\n args.logging_level = get_logging_level_registry()[args.logging_level]\n logging.getLogger(\"ludwig\").setLevel(args.logging_level)\n global logger\n logger = logging.getLogger(\"ludwig.upload\")\n\n upload_cli(**vars(args))\n\n\nif __name__ == \"__main__\":\n cli(sys.argv[1:])\n"
},
{
"path": "ludwig/utils/__init__.py",
"content": ""
},
{
"path": "ludwig/utils/algorithms_utils.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nfrom ludwig.constants import TIED\n\n\ndef topological_sort(graph_unsorted):\n \"\"\"Repeatedly go through all of the nodes in the graph, moving each of the nodes that has all its edges\n resolved, onto a sequence that forms our sorted graph.\n\n A node has all of its edges resolved and can be moved once all the nodes its edges point to, have been moved from\n the unsorted graph onto the sorted one.\n \"\"\"\n\n # This is the list we'll return, that stores each node/edges pair\n # in topological order.\n graph_sorted = []\n\n # Convert the unsorted graph into a hash table. This gives us\n # constant-time lookup for checking if edges are unresolved, and\n # for removing nodes from the unsorted graph.\n graph_unsorted = dict(graph_unsorted)\n\n # Run until the unsorted graph is empty.\n while graph_unsorted:\n # Go through each of the node/edges pairs in the unsorted\n # graph. If a set of edges does not contain any nodes that\n # haven't been resolved, that is, that are still in the\n # unsorted graph, remove the pair from the unsorted graph,\n # and append it to the sorted graph. Note here that by using\n # using the items() method for iterating, a copy of the\n # unsorted graph is used, allowing us to modify the unsorted\n # graph as we move through it. We also keep a flag for\n # checking that that graph is acyclic, which is true if any\n # nodes are resolved during each pass through the graph. If\n # not, we need to bail out as the graph therefore can't be\n # sorted.\n acyclic = False\n for node, edges in list(graph_unsorted.items()):\n if edges is None:\n edges = []\n for edge in edges:\n if edge in graph_unsorted:\n break\n else:\n acyclic = True\n del graph_unsorted[node]\n graph_sorted.append((node, edges))\n\n if not acyclic:\n # Uh oh, we've passed through all the unsorted nodes and\n # weren't able to resolve any of them, which means there\n # are nodes with cyclic edges that will never be resolved,\n # so we bail out with an error.\n raise RuntimeError(\"A cyclic dependency occurred\")\n\n return graph_sorted\n\n\ndef topological_sort_feature_dependencies(features):\n # topological sorting of output features for resolving dependencies\n dependencies_graph = {}\n output_features_dict = {}\n for feature in features:\n dependencies = []\n if \"dependencies\" in feature:\n dependencies.extend(feature[\"dependencies\"])\n if TIED in feature:\n dependencies.append(feature[TIED])\n dependencies_graph[feature[\"name\"]] = dependencies\n output_features_dict[feature[\"name\"]] = feature\n return [output_features_dict[node[0]] for node in topological_sort(dependencies_graph)]\n"
},
{
"path": "ludwig/utils/audio_utils.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport functools\nimport logging\nfrom io import BytesIO\nfrom typing import Any\n\nimport torch\nimport torch.nn.functional as F\nimport torchaudio\nfrom packaging import version\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import DEFAULT_AUDIO_TENSOR_LENGTH\nfrom ludwig.utils.types import TorchAudioTuple\n\nlogger = logging.getLogger(__name__)\n\n# https://github.com/pytorch/audio/blob/main/torchaudio/csrc/sox/types.cpp\nAUDIO_EXTENSIONS = (\".wav\", \".amb\", \".mp3\", \".ogg\", \".vorbis\", \".flac\", \".opus\", \".sphere\")\n\n\n_TORCH_AUDIO_210 = version.parse(torchaudio.__version__) >= version.parse(\"2.1.0\")\n_TORCH_AUDIO_201 = version.parse(torchaudio.__version__) >= version.parse(\"2.0.1\")\n\n\n@DeveloperAPI\ndef is_torch_audio_tuple(audio: Any) -> bool:\n if isinstance(audio, tuple):\n if len(audio) == 2 and isinstance(audio[0], torch.Tensor) and isinstance(audio[1], int):\n return True\n return False\n\n\n@DeveloperAPI\ndef get_default_audio(audio_lst: list[TorchAudioTuple]) -> TorchAudioTuple:\n if not audio_lst:\n # Return a silent audio tensor as default when no valid audio is available\n default_audio_tensor = torch.zeros(1, DEFAULT_AUDIO_TENSOR_LENGTH)\n return default_audio_tensor, 16000\n\n sampling_rates = [audio[1] for audio in audio_lst]\n tensor_list = [audio[0] for audio in audio_lst]\n\n for i, tensor in enumerate(tensor_list):\n if tensor.shape[1] > DEFAULT_AUDIO_TENSOR_LENGTH:\n tensor_list[i] = tensor[:, :DEFAULT_AUDIO_TENSOR_LENGTH]\n else:\n pad_size = DEFAULT_AUDIO_TENSOR_LENGTH - tensor.shape[1]\n tensor_list[i] = F.pad(tensor, (0, pad_size))\n default_audio_tensor = torch.mean(torch.stack(tensor_list), dim=0)\n default_sampling_rate = calculate_mean(sum(sampling_rates), len(sampling_rates))\n\n return default_audio_tensor, default_sampling_rate\n\n\n@DeveloperAPI\ndef read_audio_from_path(path: str) -> TorchAudioTuple | None:\n \"\"\"Reads audio from path.\n\n Useful for reading from a small number of paths. For more intensive reads, use backend.read_binary_files instead.\n \"\"\"\n try:\n if _TORCH_AUDIO_210:\n return torchaudio.load(path, backend=\"sox\")\n elif _TORCH_AUDIO_201:\n return torchaudio.backend.sox_io_backend.load(path)\n else:\n return torchaudio.backend.sox_backend.load(path)\n except Exception as e:\n logger.warning(e)\n return None\n\n\n@DeveloperAPI\n@functools.lru_cache(maxsize=32)\ndef read_audio_from_bytes_obj(bytes_obj: bytes) -> TorchAudioTuple | None:\n try:\n f = BytesIO(bytes_obj)\n return torchaudio.load(f)\n except Exception as e:\n logger.warning(e)\n return None\n\n\ndef _pre_emphasize_data(data: torch.Tensor, emphasize_value: float = 0.97):\n # Increase precision in order to achieve parity with scipy.signal.lfilter implementation\n filter_window = torch.tensor([1.0, -emphasize_value], dtype=torch.float64, device=data.device)\n a_coeffs = torch.tensor([1, 0], dtype=torch.float64, device=data.device)\n pre_emphasized_data = torchaudio.functional.lfilter(\n data.to(dtype=torch.float64),\n a_coeffs,\n filter_window,\n clamp=False,\n ).to(torch.float32)\n return pre_emphasized_data\n\n\n@DeveloperAPI\ndef get_length_in_samp(sampling_rate_in_hz: float | int, length_in_s: float | int) -> int:\n return int(sampling_rate_in_hz * length_in_s)\n\n\n@DeveloperAPI\ndef get_group_delay(\n raw_data: torch.Tensor,\n sampling_rate_in_hz: int,\n window_length_in_s: float,\n window_shift_in_s: float,\n num_fft_points: int,\n window_type: str,\n):\n X_stft_transform = _get_stft(\n raw_data, sampling_rate_in_hz, window_length_in_s, window_shift_in_s, num_fft_points, window_type=window_type\n )\n Y_stft_transform = _get_stft(\n raw_data,\n sampling_rate_in_hz,\n window_length_in_s,\n window_shift_in_s,\n num_fft_points,\n window_type=window_type,\n data_transformation=\"group_delay\",\n )\n X_stft_transform_real = torch.real(X_stft_transform)\n X_stft_transform_imag = torch.imag(X_stft_transform)\n Y_stft_transform_real = torch.real(Y_stft_transform)\n Y_stft_transform_imag = torch.imag(Y_stft_transform)\n nominator = torch.multiply(X_stft_transform_real, Y_stft_transform_real) + torch.multiply(\n X_stft_transform_imag, Y_stft_transform_imag\n )\n denominator = torch.square(torch.abs(X_stft_transform))\n group_delay = torch.divide(nominator, denominator + 1e-10)\n assert not torch.isnan(group_delay).any(), \"There are NaN values in group delay\"\n return torch.transpose(group_delay, 0, 1)\n\n\n@DeveloperAPI\ndef get_phase_stft_magnitude(\n raw_data: torch.Tensor,\n sampling_rate_in_hz: int,\n window_length_in_s: float,\n window_shift_in_s: float,\n num_fft_points: int,\n window_type: str,\n) -> torch.Tensor:\n stft = _get_stft(\n raw_data, sampling_rate_in_hz, window_length_in_s, window_shift_in_s, num_fft_points, window_type=window_type\n )\n abs_stft = torch.abs(stft)\n phase = torch.angle(stft)\n stft_phase = torch.cat([phase, abs_stft], dim=1)\n return torch.transpose(stft_phase, 0, 1)\n\n\n@DeveloperAPI\ndef get_stft_magnitude(\n raw_data: torch.Tensor,\n sampling_rate_in_hz: int,\n window_length_in_s: float,\n window_shift_in_s: float,\n num_fft_points: int,\n window_type: str,\n):\n stft = _get_stft(\n raw_data, sampling_rate_in_hz, window_length_in_s, window_shift_in_s, num_fft_points, window_type=window_type\n )\n stft_magnitude = torch.abs(stft)\n return torch.transpose(stft_magnitude, 0, 1)\n\n\n################################################################################\n# The following code for FBank is adapted from jameslyons/python_speech_features\n# MIT licensed implementation\n# https://github.com/jameslyons/python_speech_features/blob/40c590269b57c64a8c1f1ddaaff2162008d1850c/python_speech_features/base.py#L84################################################################################\n################################################################################\n@DeveloperAPI\ndef get_fbank(\n raw_data: torch.Tensor,\n sampling_rate_in_hz: int,\n window_length_in_s: float,\n window_shift_in_s: float,\n num_fft_points: int,\n window_type: str,\n num_filter_bands: int,\n) -> torch.Tensor:\n stft = _get_stft(\n raw_data,\n sampling_rate_in_hz,\n window_length_in_s,\n window_shift_in_s,\n num_fft_points,\n window_type=window_type,\n zero_mean_offset=True,\n )\n stft_power = torch.abs(stft) ** 2\n upper_limit_freq = int(sampling_rate_in_hz / 2)\n upper_limit_mel = _convert_hz_to_mel(upper_limit_freq)\n lower_limit_mel = 0\n list_mel_points = torch.linspace(lower_limit_mel, upper_limit_mel, num_filter_bands + 2, device=raw_data.device)\n mel_fbank_matrix = _get_mel_fbank_matrix(list_mel_points, num_filter_bands, num_fft_points, sampling_rate_in_hz)\n mel_fbank_feature = torch.matmul(stft_power, torch.transpose(mel_fbank_matrix, 0, 1))\n log_mel_fbank_feature = torch.log(mel_fbank_feature + 1.0e-10)\n return torch.transpose(log_mel_fbank_feature, 0, 1)\n\n\ndef _get_mel_fbank_matrix(\n list_mel_points: torch.Tensor, num_filter_bands: int, num_fft_points: int, sampling_rate_in_hz: int\n) -> torch.Tensor:\n num_ess_fft_points = get_non_symmetric_length(num_fft_points)\n freq_scale = (num_fft_points + 1) / sampling_rate_in_hz\n freq_bins_on_mel_scale = torch.floor(freq_scale * _convert_mel_to_hz(list_mel_points))\n mel_scaled_fbank = torch.zeros(\n (num_filter_bands, num_ess_fft_points), dtype=torch.float32, device=list_mel_points.device\n )\n for filt_idx in range(num_filter_bands):\n start_bin_freq = freq_bins_on_mel_scale[filt_idx]\n middle_bin_freq = freq_bins_on_mel_scale[filt_idx + 1]\n end_bin_freq = freq_bins_on_mel_scale[filt_idx + 2]\n mel_scaled_fbank[filt_idx] = _create_triangular_filter(\n start_bin_freq, middle_bin_freq, end_bin_freq, num_ess_fft_points\n )\n return mel_scaled_fbank\n\n\ndef _create_triangular_filter(\n start_bin_freq: torch.Tensor, middle_bin_freq: torch.Tensor, end_bin_freq: torch.Tensor, num_ess_fft_points: int\n):\n filter_window = torch.zeros(num_ess_fft_points, dtype=torch.float32, device=start_bin_freq.device)\n filt_support_begin = middle_bin_freq - start_bin_freq\n filt_support_end = end_bin_freq - middle_bin_freq\n for freq in range(int(start_bin_freq), int(middle_bin_freq)):\n filter_window[freq] = (freq - start_bin_freq) / filt_support_begin\n for freq in range(int(middle_bin_freq), int(end_bin_freq)):\n filter_window[freq] = (end_bin_freq - freq) / filt_support_end\n return filter_window\n\n\ndef _convert_hz_to_mel(hz: int) -> float:\n return float(2595.0 * torch.log10(torch.tensor(1 + hz / 700.0)))\n\n\ndef _convert_mel_to_hz(mel):\n return 700.0 * (10 ** (mel / 2595.0) - 1)\n\n\ndef _get_stft(\n raw_data: torch.Tensor,\n sampling_rate_in_hz: int,\n window_length_in_s: float,\n window_shift_in_s: float,\n num_fft_points: int,\n window_type: str,\n data_transformation: str | None = None,\n zero_mean_offset: bool = False,\n) -> torch.Tensor:\n pre_emphasized_data = _pre_emphasize_data(raw_data)\n stft = _short_time_fourier_transform(\n pre_emphasized_data,\n sampling_rate_in_hz,\n window_length_in_s,\n window_shift_in_s,\n num_fft_points,\n window_type,\n data_transformation,\n zero_mean_offset,\n )\n non_symmetric_stft = get_non_symmetric_data(stft)\n return non_symmetric_stft\n\n\ndef _short_time_fourier_transform(\n data: torch.Tensor,\n sampling_rate_in_hz: int,\n window_length_in_s: float,\n window_shift_in_s: float,\n num_fft_points: int,\n window_type: str,\n data_transformation: str | None = None,\n zero_mean_offset: bool = False,\n) -> torch.Tensor:\n window_length_in_samp: int = get_length_in_samp(window_length_in_s, sampling_rate_in_hz)\n window_shift_in_samp: int = get_length_in_samp(window_shift_in_s, sampling_rate_in_hz)\n preprocessed_data_matrix = _preprocess_to_padded_matrix(\n data[0], window_length_in_samp, window_shift_in_samp, zero_mean_offset=zero_mean_offset\n )\n weighted_data_matrix = _weight_data_matrix(\n preprocessed_data_matrix, window_type, data_transformation=data_transformation\n )\n fft = torch.fft.fft(weighted_data_matrix, n=num_fft_points)\n return fft\n\n\ndef _preprocess_to_padded_matrix(\n data: torch.Tensor, window_length_in_samp: int, window_shift_in_samp: int, zero_mean_offset: bool = False\n) -> torch.Tensor:\n num_input = data.shape[0]\n num_output = get_num_output_padded_to_fit_input(num_input, window_length_in_samp, window_shift_in_samp)\n zero_padded_matrix = torch.zeros((num_output, window_length_in_samp), dtype=torch.float32, device=data.device)\n for num_output_idx in range(num_output):\n start_idx = window_shift_in_samp * num_output_idx\n is_last_output = num_output_idx == num_output - 1\n end_idx = start_idx + window_length_in_samp if not is_last_output else num_input\n end_padded_idx = window_length_in_samp if not is_last_output else end_idx - start_idx\n window_data = data[start_idx:end_idx]\n if zero_mean_offset:\n window_data = window_data - torch.mean(window_data)\n zero_padded_matrix[num_output_idx, :end_padded_idx] = window_data\n return zero_padded_matrix\n\n\n@DeveloperAPI\ndef get_num_output_padded_to_fit_input(num_input: int, window_length_in_samp: int, window_shift_in_samp: int) -> int:\n num_output_valid = torch.tensor((num_input - window_length_in_samp) / window_shift_in_samp + 1)\n return int(torch.ceil(num_output_valid))\n\n\n@DeveloperAPI\ndef get_window(window_type: str, window_length_in_samp: int, device: torch.device | None = None) -> torch.Tensor:\n # Increase precision in order to achieve parity with scipy.signal.windows.get_window implementation\n if window_type == \"bartlett\":\n return torch.bartlett_window(window_length_in_samp, periodic=False, dtype=torch.float64, device=device).to(\n torch.float32\n )\n elif window_type == \"blackman\":\n return torch.blackman_window(window_length_in_samp, periodic=False, dtype=torch.float64, device=device).to(\n torch.float32\n )\n elif window_type == \"hamming\":\n return torch.hamming_window(window_length_in_samp, periodic=False, dtype=torch.float64, device=device).to(\n torch.float32\n )\n elif window_type == \"hann\":\n return torch.hann_window(window_length_in_samp, periodic=False, dtype=torch.float64, device=device).to(\n torch.float32\n )\n else:\n raise ValueError(f\"Unknown window type: {window_type}\")\n\n\n@DeveloperAPI\ndef is_audio_score(src_path):\n # Used for AutoML\n return int(isinstance(src_path, str) and src_path.lower().endswith(AUDIO_EXTENSIONS))\n\n\ndef _weight_data_matrix(\n data_matrix: torch.Tensor, window_type: str, data_transformation: str | None = None\n) -> torch.Tensor:\n window_length_in_samp = data_matrix[0].shape[0]\n window = get_window(window_type, window_length_in_samp, device=data_matrix.device)\n if data_transformation is not None and data_transformation == \"group_delay\":\n window *= torch.arange(window_length_in_samp, device=data_matrix.device).float()\n return data_matrix * window\n\n\n@DeveloperAPI\ndef get_non_symmetric_length(symmetric_length: int) -> int:\n return int(symmetric_length / 2) + 1\n\n\n@DeveloperAPI\ndef get_non_symmetric_data(data: torch.Tensor) -> torch.Tensor:\n num_fft_points = data.shape[-1]\n num_ess_fft_points = get_non_symmetric_length(num_fft_points)\n return data[:, :num_ess_fft_points]\n\n\n@DeveloperAPI\ndef get_max_length_stft_based(length_in_samp, window_length_in_s, window_shift_in_s, sampling_rate_in_hz):\n window_length_in_samp = get_length_in_samp(window_length_in_s, sampling_rate_in_hz)\n window_shift_in_samp = get_length_in_samp(window_shift_in_s, sampling_rate_in_hz)\n return get_num_output_padded_to_fit_input(length_in_samp, window_length_in_samp, window_shift_in_samp)\n\n\n@DeveloperAPI\ndef calculate_incr_var(var_prev, mean_prev, mean, length):\n return var_prev + (length - mean_prev) * (length - mean)\n\n\n@DeveloperAPI\ndef calculate_incr_mean(count, mean, length):\n return mean + (length - mean) / float(count)\n\n\n@DeveloperAPI\ndef calculate_var(sum1, sum2, count):\n return (sum2 - ((sum1 * sum1) / float(count))) / float(count - 1) if count > 1 else 0.0\n\n\n@DeveloperAPI\ndef calculate_mean(sum1, count):\n return sum1 / float(count)\n"
},
{
"path": "ludwig/utils/augmentation_utils.py",
"content": "from ludwig.api_annotations import DeveloperAPI\nfrom ludwig.utils.registry import Registry\n\n###\n# Registry for augmentation operations\n# Each augmentation operation is registered with the feature type it is applicable to\n# and the name of the operation.\n###\n_augmentation_op_registry = Registry()\n\n\n@DeveloperAPI\ndef get_augmentation_op_registry() -> Registry:\n return _augmentation_op_registry\n\n\n@DeveloperAPI\ndef register_augmentation_op(name: str, features: str | list[str]):\n if isinstance(features, str):\n features = [features]\n\n def wrap(cls):\n for feature in features:\n augmentation_op_registry = get_augmentation_op_registry().get(feature, {})\n augmentation_op_registry[name] = cls\n get_augmentation_op_registry()[feature] = augmentation_op_registry\n return cls\n\n return wrap\n\n\n@DeveloperAPI\ndef get_augmentation_op(feature_type: str, op_name: str):\n return get_augmentation_op_registry()[feature_type][op_name]\n\n\nclass AugmentationPipelines:\n \"\"\"Container holding augmentation pipelines defined in the model.\"\"\"\n\n def __init__(self, augmentation_pipelines: dict):\n self.augmentation_pipelines = augmentation_pipelines\n\n def __getitem__(self, key):\n return self.augmentation_pipelines[key]\n\n def __contains__(self, key):\n return key in self.augmentation_pipelines\n\n def __len__(self):\n return len(self.augmentation_pipelines)\n\n def __iter__(self):\n return self.augmentation_pipelines.__iter__()\n\n def items(self):\n return self.augmentation_pipelines.items()\n"
},
{
"path": "ludwig/utils/automl/__init__.py",
"content": ""
},
{
"path": "ludwig/utils/automl/data_source.py",
"content": "from abc import ABC, abstractmethod\n\nimport dask.dataframe as dd\nimport pandas as pd\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.utils.audio_utils import is_audio_score\nfrom ludwig.utils.automl.utils import avg_num_tokens\nfrom ludwig.utils.image_utils import is_image_score\nfrom ludwig.utils.misc_utils import memoized_method\nfrom ludwig.utils.types import DataFrame\n\n\n@DeveloperAPI\nclass DataSource(ABC):\n @property\n @abstractmethod\n def columns(self) -> list[str]:\n raise NotImplementedError()\n\n @abstractmethod\n def get_dtype(self, column: str) -> str:\n raise NotImplementedError()\n\n @abstractmethod\n def get_distinct_values(self, column: str, max_values_to_return: int) -> tuple[int, list[str], float]:\n raise NotImplementedError()\n\n @abstractmethod\n def get_nonnull_values(self, column: str) -> int:\n raise NotImplementedError()\n\n @abstractmethod\n def get_avg_num_tokens(self, column: str) -> int:\n raise NotImplementedError()\n\n @abstractmethod\n def is_string_type(self, dtype: str) -> bool:\n raise NotImplementedError()\n\n @abstractmethod\n def size_bytes(self) -> int:\n raise NotImplementedError()\n\n @abstractmethod\n def __len__(self) -> int:\n raise NotImplementedError()\n\n\n@DeveloperAPI\nclass DataframeSourceMixin:\n df: DataFrame\n\n @property\n def columns(self) -> list[str]:\n return self.df.columns\n\n def get_dtype(self, column: str) -> str:\n return self.df[column].dtype.name\n\n def get_distinct_values(self, column, max_values_to_return: int) -> tuple[int, list[str], float]:\n unique_values = self.df[column].dropna().unique()\n num_unique_values = len(unique_values)\n unique_values_counts = self.df[column].value_counts()\n if len(unique_values_counts) != 0:\n unique_majority_values = unique_values_counts[unique_values_counts.idxmax()]\n unique_minority_values = unique_values_counts[unique_values_counts.idxmin()]\n unique_values_balance = unique_minority_values / unique_majority_values\n else:\n unique_values_balance = 1.0\n return num_unique_values, unique_values[:max_values_to_return], unique_values_balance\n\n def get_nonnull_values(self, column: str) -> int:\n return len(self.df[column].notnull())\n\n def get_image_values(self, column: str, sample_size: int = 10) -> int:\n return int(sum(is_image_score(x) for x in self.df[column].head(sample_size)))\n\n def get_audio_values(self, column: str, sample_size: int = 10) -> int:\n return int(sum(is_audio_score(x) for x in self.df[column].head(sample_size)))\n\n def get_avg_num_tokens(self, column: str) -> int:\n return avg_num_tokens(self.df[column])\n\n def is_string_type(self, dtype: str) -> bool:\n return dtype in [\"str\", \"string\", \"object\"]\n\n def size_bytes(self) -> int:\n return sum(self.df.memory_usage(deep=True))\n\n def __len__(self) -> int:\n return len(self.df)\n\n\n@DeveloperAPI\nclass DataframeSource(DataframeSourceMixin, DataSource):\n def __init__(self, df):\n self.df = df\n\n\n@DeveloperAPI\nclass DaskDataSource(DataframeSource):\n @memoized_method(maxsize=1)\n def get_sample(self) -> pd.DataFrame:\n # TODO: uniform random sample\n return self.df.head(10000)\n\n @property\n def sample(self) -> pd.DataFrame:\n return self.get_sample()\n\n def get_distinct_values(self, column, max_values_to_return) -> tuple[int, list[str], float]:\n unique_values = self.df[column].drop_duplicates().dropna().persist()\n num_unique_values = len(unique_values)\n\n # TODO(travis): implement imbalance ratio\n imbalance_ratio = 1.0\n return num_unique_values, unique_values.head(max_values_to_return), imbalance_ratio\n\n def get_nonnull_values(self, column) -> int:\n return self.df[column].notnull().sum().compute()\n\n def get_image_values(self, column: str, sample_size: int = 10) -> int:\n return int(sum(is_image_score(x) for x in self.sample[column].head(sample_size)))\n\n def get_audio_values(self, column: str, sample_size: int = 10) -> int:\n return int(sum(is_audio_score(x) for x in self.sample[column].head(sample_size)))\n\n def get_avg_num_tokens(self, column) -> int:\n return avg_num_tokens(self.sample[column])\n\n\n@DeveloperAPI\ndef wrap_data_source(df: DataFrame) -> DataSource:\n if isinstance(df, dd.DataFrame):\n return DaskDataSource(df)\n return DataframeSource(df)\n"
},
{
"path": "ludwig/utils/automl/field_info.py",
"content": "from dataclasses import dataclass\n\nfrom dataclasses_json import dataclass_json, LetterCase\n\nfrom ludwig.api_annotations import DeveloperAPI\n\n\n@DeveloperAPI\n@dataclass_json(letter_case=LetterCase.CAMEL)\n@dataclass\nclass FieldInfo:\n name: str\n dtype: str\n key: str = None\n distinct_values: list = None\n distinct_values_balance: float = 1.0\n num_distinct_values: int = 0\n nonnull_values: int = 0\n image_values: int = 0\n audio_values: int = 0\n avg_words: int = None\n\n\n@DeveloperAPI\n@dataclass_json(letter_case=LetterCase.CAMEL)\n@dataclass\nclass FieldConfig:\n name: str\n column: str\n type: str\n\n\n@DeveloperAPI\n@dataclass_json(letter_case=LetterCase.CAMEL)\n@dataclass\nclass FieldMetadata:\n name: str\n config: FieldConfig\n excluded: bool\n mode: str\n missing_values: float\n imbalance_ratio: float\n"
},
{
"path": "ludwig/utils/automl/ray_utils.py",
"content": "import os\n\nfrom ludwig.backend.ray import initialize_ray\n\ntry:\n import ray\nexcept ImportError:\n raise ImportError(\" ray is not installed. \" \"In order to use auto_train please run \" \"pip install ludwig[ray]\")\n\n\ndef _ray_init():\n if ray.is_initialized():\n return\n\n # Forcibly terminate trial requested to stop after this amount of time passes\n os.environ.setdefault(\"TUNE_FORCE_TRIAL_CLEANUP_S\", \"120\")\n\n initialize_ray()\n"
},
{
"path": "ludwig/utils/automl/type_inference.py",
"content": "import logging\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import AUDIO, BINARY, CATEGORY, DATE, IMAGE, NUMBER, TEXT\nfrom ludwig.utils import strings_utils\nfrom ludwig.utils.automl.field_info import FieldInfo\n\n# For a given feature, the highest percentage of distinct values out of the total number of rows that we might still\n# assign the CATEGORY type.\nCATEGORY_TYPE_DISTINCT_VALUE_PERCENTAGE_CUTOFF = 0.5\n\n# Consider the field a valid text field if it has at least 5 average words. Fewer than this and it may be a cateogry\n# or an ID field (like a name or place) of some kind.\nTEXT_AVG_WORDS_CUTOFF = 5\n\n\n@DeveloperAPI\ndef infer_type(field: FieldInfo, missing_value_percent: float, row_count: int) -> str:\n \"\"\"Perform type inference on field.\n\n # Inputs\n :param field: (FieldInfo) object describing field\n :param missing_value_percent: (float) percent of missing values in the column\n :param row_count: (int) total number of entries in original dataset # Return\n :return: (str) feature type\n \"\"\"\n if field.dtype == DATE or field.dtype.startswith(\"datetime\"):\n return DATE\n\n num_distinct_values = field.num_distinct_values\n distinct_values = field.distinct_values\n\n if num_distinct_values <= 1:\n return CATEGORY\n\n if num_distinct_values == 2 and missing_value_percent == 0:\n # Check that all distinct values are conventional bools.\n if strings_utils.are_conventional_bools(distinct_values):\n return BINARY\n\n if field.image_values >= 3:\n return IMAGE\n\n if field.audio_values >= 3:\n return AUDIO\n\n if strings_utils.are_all_datetimes(distinct_values):\n return DATE\n\n # Use CATEGORY if:\n # - The number of distinct values is significantly less than the total number of examples.\n # - The distinct values are not all numbers.\n # - The distinct values are all numbers but comprise of a perfectly sequential list of integers that suggests the\n # values represent categories.\n valid_row_count = row_count * (1.0 - missing_value_percent)\n if num_distinct_values < valid_row_count * CATEGORY_TYPE_DISTINCT_VALUE_PERCENTAGE_CUTOFF and (\n (not strings_utils.are_all_numbers(distinct_values)) or strings_utils.are_sequential_integers(distinct_values)\n ):\n return CATEGORY\n\n # Use NUMBER if all of the distinct values are numbers.\n if strings_utils.are_all_numbers(distinct_values):\n return NUMBER\n\n # TODO (ASN): add other modalities (image, etc. )\n # Fallback to TEXT.\n return TEXT\n\n\n@DeveloperAPI\ndef should_exclude(\n idx: int, field: FieldInfo, dtype: str, column_count: int, row_count: int, targets: set[str]\n) -> bool:\n if field.key == \"PRI\":\n logging.info(f\"Exclude {field.name} ({dtype}): primary key\")\n return True\n\n if field.name in targets:\n return False\n\n if field.num_distinct_values <= 1:\n logging.info(f\"Exclude {field.name} ({dtype}): less than 2 distinct values\")\n return True\n\n distinct_value_percent = float(field.num_distinct_values) / row_count\n if distinct_value_percent == 1.0:\n upper_name = field.name.upper()\n if (\n (idx == 0 and \"INDEX\" in upper_name and dtype == NUMBER)\n or upper_name.endswith(\"ID\")\n or upper_name.startswith(\"ID\")\n ):\n logging.info(f\"Exclude {field.name} ({dtype}): unique ID column\")\n return True\n\n # For TEXT fields, we only want to use them if they appear \"interesting\", otherwise we would rather exclude\n # them and treat the problem as a tabular problem\n if column_count > 3 and dtype == TEXT and (field.avg_words or 0) < TEXT_AVG_WORDS_CUTOFF:\n logging.info(f\"Exclude {field.name} ({dtype}): too few average words\")\n return True\n\n return False\n"
},
{
"path": "ludwig/utils/automl/utils.py",
"content": "import bisect\nimport logging\n\nfrom numpy import nan_to_num\nfrom pandas import Series\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import (\n BINARY,\n CATEGORY,\n COMBINER,\n CONFIG,\n HYPEROPT,\n IMBALANCE_DETECTION_RATIO,\n NAME,\n NUMBER,\n PARAMETERS,\n SEARCH_ALG,\n TRAINER,\n TYPE,\n)\nfrom ludwig.features.feature_registries import get_output_type_registry\nfrom ludwig.modules.metric_registry import get_metric_objective\nfrom ludwig.schema.combiners.utils import get_combiner_jsonschema\n\nlogger = logging.getLogger(__name__)\n\n\n@DeveloperAPI\ndef avg_num_tokens_decoder(x):\n if x is None:\n return None\n if type(x) is bytes:\n return x.decode(\"utf-8\")\n return str(x)\n\n\n@DeveloperAPI\ndef avg_num_tokens(field: Series) -> int:\n logger.info(f\"Calculating average number tokens for field {field.name} using sample of 100 rows.\")\n field_sample = field.head(100).apply(avg_num_tokens_decoder)\n\n unique_entries = field_sample.unique()\n avg_words = round(nan_to_num(Series(unique_entries).str.split().str.len().mean()))\n return avg_words\n\n\n@DeveloperAPI\ndef get_model_type(config: dict) -> str:\n if (\n \"input_features\" in config\n and len(config[\"input_features\"]) == 1\n and \"type\" in config[\"input_features\"][0]\n and config[\"input_features\"][0][\"type\"] == \"text\"\n ):\n model_type = \"text\"\n elif COMBINER in config and TYPE in config[COMBINER]:\n model_type = config[COMBINER][TYPE]\n else:\n default_combiner_type = get_combiner_jsonschema()[\"properties\"][\"type\"][\"default\"]\n model_type = default_combiner_type\n return model_type\n\n\n# ref_configs comes from a file storing the config for a high-performing model per reference dataset.\n# If the automl model type matches that of any reference models, set the initial point_to_evaluate\n# in the automl hyperparameter search to the config of the reference model with the closest-matching\n# input number columns ratio. This model config \"transfer learning\" can improve the automl search.\ndef _add_transfer_config(base_config: dict, ref_configs: dict) -> dict:\n base_model_type = base_config[COMBINER][TYPE]\n base_model_numeric_ratio = _get_ratio_numeric_input_features(base_config[\"input_features\"])\n min_numeric_ratio_distance = 1.0\n min_dataset = None\n\n for dataset in ref_configs[\"datasets\"]:\n dataset_config = dataset[CONFIG]\n if base_model_type == dataset_config[COMBINER][TYPE]:\n dataset_numeric_ratio = _get_ratio_numeric_input_features(dataset_config[\"input_features\"])\n ratio_distance = abs(base_model_numeric_ratio - dataset_numeric_ratio)\n if ratio_distance <= min_numeric_ratio_distance:\n min_numeric_ratio_distance = ratio_distance\n min_dataset = dataset\n\n if min_dataset is not None:\n logger.info(\"Transfer config from dataset {}\".format(min_dataset[\"name\"]))\n min_dataset_config = min_dataset[CONFIG]\n hyperopt_params = base_config[HYPEROPT][PARAMETERS]\n point_to_evaluate = {}\n _add_option_to_evaluate(point_to_evaluate, min_dataset_config, hyperopt_params, COMBINER)\n _add_option_to_evaluate(point_to_evaluate, min_dataset_config, hyperopt_params, TRAINER)\n base_config[HYPEROPT][SEARCH_ALG][\"points_to_evaluate\"] = [point_to_evaluate]\n return base_config\n\n\ndef _get_ratio_numeric_input_features(input_features: dict) -> float:\n num_input_features = len(input_features)\n num_numeric_input = 0\n for input_feature in input_features:\n if input_feature[TYPE] == NUMBER:\n num_numeric_input = num_numeric_input + 1\n return num_numeric_input / num_input_features\n\n\n# Update point_to_evaluate w/option value from dataset_config for options in hyperopt_params.\n# Also, add option value to associated categories list if it is not already included.\ndef _add_option_to_evaluate(\n point_to_evaluate: dict, dataset_config: dict, hyperopt_params: dict, option_type: str\n) -> dict:\n options = dataset_config[option_type]\n for option in options.keys():\n option_param = option_type + \".\" + option\n if option_param in hyperopt_params.keys():\n option_val = options[option]\n point_to_evaluate[option_param] = option_val\n if option_val not in hyperopt_params[option_param][\"categories\"]:\n bisect.insort(hyperopt_params[option_param][\"categories\"], option_val)\n return point_to_evaluate\n\n\n@DeveloperAPI\ndef set_output_feature_metric(base_config):\n \"\"\"If single output feature, set trainer and hyperopt metric and goal for that feature if not set.\"\"\"\n if len(base_config[\"output_features\"]) != 1:\n # If multiple output features, ludwig uses the goal of minimizing combined loss;\n # this could be revisited/refined in the future.\n return base_config\n output_name = base_config[\"output_features\"][0][NAME]\n output_type = base_config[\"output_features\"][0][TYPE]\n output_metric = get_output_type_registry()[output_type].get_schema_cls().default_validation_metric\n output_goal = get_metric_objective(output_metric)\n if \"validation_field\" not in base_config[TRAINER] and \"validation_metric\" not in base_config[TRAINER]:\n base_config[TRAINER][\"validation_field\"] = output_name\n base_config[TRAINER][\"validation_metric\"] = output_metric\n if (\n \"output_feature\" not in base_config[HYPEROPT]\n and \"metric\" not in base_config[HYPEROPT]\n and \"goal\" not in base_config[HYPEROPT]\n ):\n base_config[HYPEROPT][\"output_feature\"] = output_name\n base_config[HYPEROPT][\"metric\"] = output_metric\n base_config[HYPEROPT][\"goal\"] = output_goal\n return base_config\n\n\n@DeveloperAPI\ndef has_imbalanced_output(base_config, features_metadata) -> bool:\n \"\"\"Check binary and category output feature(s) for imbalance, i.e., low minority/majority instance count\n ratio.\"\"\"\n imbalanced_output = False\n for output_feature in base_config[\"output_features\"]:\n if output_feature[TYPE] == BINARY or output_feature[TYPE] == CATEGORY:\n for feature_metadata in features_metadata:\n if output_feature[NAME] == feature_metadata.name:\n if feature_metadata.imbalance_ratio < IMBALANCE_DETECTION_RATIO:\n logger.info(\n f\"Imbalance in {output_feature[NAME]}: minority/majority={feature_metadata.imbalance_ratio}\"\n )\n imbalanced_output = True\n break\n return imbalanced_output\n"
},
{
"path": "ludwig/utils/backward_compatibility.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport copy\nimport logging\nimport warnings\nfrom collections.abc import Callable\nfrom typing import Any\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import (\n AUDIO,\n BIAS,\n CLASS_WEIGHTS,\n COLUMN,\n CONV_BIAS,\n CONV_USE_BIAS,\n DECODER,\n DEFAULT_BIAS,\n DEFAULT_USE_BIAS,\n DEFAULTS,\n ENCODER,\n EVAL_BATCH_SIZE,\n EXECUTOR,\n FORCE_SPLIT,\n HEIGHT,\n HYPEROPT,\n IMAGE,\n INPUT_FEATURES,\n LOSS,\n MISSING_VALUE_STRATEGY,\n MODEL_ECD,\n NAME,\n NUM_SAMPLES,\n NUMBER,\n OUTPUT_FEATURES,\n PARAMETERS,\n PREPROCESSING,\n PROBABILITIES,\n RANDOM,\n RAY,\n SAMPLER,\n SCHEDULER,\n SEARCH_ALG,\n SEQUENCE,\n SPLIT,\n SPLIT_PROBABILITIES,\n STRATIFY,\n TEXT,\n TIMESERIES,\n TRAINER,\n TRAINING,\n TYPE,\n USE_BIAS,\n WIDTH,\n)\nfrom ludwig.features.feature_registries import get_base_type_registry, get_input_type_registry, get_output_type_registry\nfrom ludwig.globals import LUDWIG_VERSION\nfrom ludwig.schema.encoders.utils import get_encoder_cls\nfrom ludwig.types import (\n FeatureConfigDict,\n FeatureTypeDefaultsDict,\n HyperoptConfigDict,\n ModelConfigDict,\n PreprocessingConfigDict,\n TrainerConfigDict,\n TrainingSetMetadataDict,\n)\nfrom ludwig.utils.metric_utils import TrainerMetric\nfrom ludwig.utils.misc_utils import get_from_registry, merge_dict\nfrom ludwig.utils.version_transformation import VersionTransformation, VersionTransformationRegistry\n\nconfig_transformation_registry = VersionTransformationRegistry()\n\n\n@DeveloperAPI\ndef register_config_transformation(version: str, prefixes: str | list[str] = []) -> Callable:\n \"\"\"This decorator registers a transformation function for a config version. Version is the first version which\n requires the transform. For example, since \"training\" is renamed to \"trainer\" in 0.5, this change should be\n registered with 0.5. from_version < version <= to_version.\n\n Args:\n version: The version to register this transformation with. The earliest ludwig version which requires this\n transformation.\n prefixes: A list of keypath prefixes to apply this transformation to. If not specified, transforms the entire\n config dict. If a prefix indicates a list, i.e. \"input_features\", the transformation is applied to\n each element of the list (each input feature).\n \"\"\"\n if isinstance(prefixes, str):\n prefixes = [prefixes]\n\n def wrap(fn: Callable[[dict], dict]):\n config_transformation_registry.register(VersionTransformation(transform=fn, version=version, prefixes=prefixes))\n return fn\n\n return wrap\n\n\n@DeveloperAPI\ndef upgrade_config_dict_to_latest_version(config: ModelConfigDict) -> ModelConfigDict:\n \"\"\"Updates config from an older version of Ludwig to the current version. If config does not have a\n \"ludwig_version\" key, all updates are applied.\n\n Args:\n config: A config saved by an older version of Ludwig.\n\n Returns A new copy of config, upgraded to the current Ludwig version. Returns config if config has no\n \"ludwig_version\".\n \"\"\"\n return config_transformation_registry.update_config(\n config, from_version=config.get(\"ludwig_version\", \"0.0\"), to_version=LUDWIG_VERSION\n )\n\n\ndef upgrade_model_progress(model_progress: dict) -> dict:\n \"\"\"Updates model progress info to be compatible with latest ProgressTracker implementation.\n\n Notably, we convert epoch-based stats to their step-based equivalents and reformat metrics into `TrainerMetric`\n tuples.\n \"\"\"\n ret = copy.deepcopy(model_progress)\n\n if \"last_improvement_epoch\" in ret:\n ret[\"last_improvement_steps\"] = ret[\"last_improvement_epoch\"] * ret[\"batch_size\"]\n del ret[\"last_improvement_epoch\"]\n\n if \"last_learning_rate_reduction_epoch\" in ret:\n ret[\"last_learning_rate_reduction_steps\"] = ret[\"last_learning_rate_reduction_epoch\"] * ret[\"batch_size\"]\n del ret[\"last_learning_rate_reduction_epoch\"]\n\n if \"last_increase_batch_size_epoch\" in ret:\n ret[\"last_increase_batch_size_steps\"] = ret[\"last_increase_batch_size_epoch\"] * ret[\"batch_size\"]\n del ret[\"last_increase_batch_size_epoch\"]\n\n if \"vali_metrics\" in ret:\n ret[\"validation_metrics\"] = ret[\"vali_metrics\"]\n del ret[\"vali_metrics\"]\n\n for metric_group in (\"train_metrics\", \"test_metrics\", \"validation_metrics\"):\n if metric_group not in ret:\n continue\n for tgt in ret[metric_group]:\n for metric in ret[metric_group][tgt]:\n if len(ret[metric_group][tgt][metric]) == 0 or isinstance(\n ret[metric_group][tgt][metric][0], (tuple, list)\n ):\n continue\n\n ret[metric_group][tgt][metric] = [\n TrainerMetric(i + 1, (i + 1) * ret[\"batch_size\"], val)\n for i, val in enumerate(ret[metric_group][tgt][metric])\n ]\n\n if \"tune_checkpoint_num\" not in ret:\n ret[\"tune_checkpoint_num\"] = 0\n\n # Upgrades related to extending progress tracker with explicit bests.\n if \"checkpoint_number\" not in ret:\n ret[\"checkpoint_number\"] = 0\n\n if \"best_eval_metric_steps\" not in ret:\n ret[\"best_eval_metric_steps\"] = 0\n\n if \"best_eval_metric_epoch\" not in ret:\n ret[\"best_eval_metric_epoch\"] = 0\n\n if \"best_eval_metric_checkpoint_number\" not in ret:\n ret[\"best_eval_metric_checkpoint_number\"] = 0\n\n if \"best_eval_train_metrics\" not in ret:\n ret[\"best_eval_train_metrics\"] = {}\n\n if \"best_eval_validation_metrics\" not in ret:\n ret[\"best_eval_validation_metrics\"] = {}\n\n if \"best_eval_test_metrics\" not in ret:\n ret[\"best_eval_test_metrics\"] = {}\n\n if \"best_eval_metric\" in ret:\n ret[\"best_eval_metric_value\"] = ret[\"best_eval_metric\"]\n del ret[\"best_eval_metric\"]\n\n if \"last_improvement\" in ret:\n del ret[\"last_improvement\"]\n\n # Delete learning-rate related fields removed in https://github.com/ludwig-ai/ludwig/pull/2877.\n if \"best_reduce_learning_rate_eval_metric\" in ret:\n del ret[\"best_reduce_learning_rate_eval_metric\"]\n\n if \"last_reduce_learning_rate_eval_metric_improvement\" in ret:\n del ret[\"last_reduce_learning_rate_eval_metric_improvement\"]\n\n return ret\n\n\ndef _traverse_dicts(config: Any, f: Callable[[dict], None]):\n \"\"\"Recursively applies function f to every dictionary contained in config.\n\n f should in-place modify the config dict. f will be called on leaves first, root last.\n \"\"\"\n if isinstance(config, dict):\n for k, v in config.items():\n _traverse_dicts(v, f)\n f(config)\n elif isinstance(config, list):\n for v in config:\n _traverse_dicts(v, f)\n\n\n@register_config_transformation(\"0.6\", \"backend\")\ndef _update_backend_cache_credentials(backend: dict[str, Any]) -> dict[str, Any]:\n if \"cache_credentials\" in backend:\n credentials = backend.get(\"credentials\", {})\n if \"cache\" in credentials:\n warnings.warn(\"`cache` already found in `backend.credentials`, ignoring `cache_credentials`\")\n else:\n warnings.warn(\n \"`backend.cache_credentials` has been renamed `backend.credentials.cache`\", DeprecationWarning\n )\n credentials[\"cache\"] = backend.pop(\"cache_credentials\")\n backend[\"credentials\"] = credentials\n return backend\n\n\n@register_config_transformation(\"0.6\", [\"output_features\"])\ndef update_class_weights_in_features(feature: FeatureConfigDict) -> FeatureConfigDict:\n if LOSS in feature:\n class_weights = feature[LOSS].get(CLASS_WEIGHTS, None)\n if not isinstance(class_weights, (list, dict)):\n class_weights = None\n feature[LOSS][CLASS_WEIGHTS] = class_weights\n\n return feature\n\n\n@register_config_transformation(\"0.4\")\ndef _update_level_metadata(config: ModelConfigDict) -> ModelConfigDict:\n # Replace parameters represented as keys with params represented as values.\n # Precedence is defined by first in the dictionary order, so if multiple\n # provided keys map to the same value, the one that appears earlier in this\n # dictionary will take priority.\n drop_params = {\n \"sequence_length_limit\": \"max_sequence_length\",\n \"word_most_common\": \"most_common\",\n \"word_sequence_length_limit\": \"max_sequence_length\",\n \"word_tokenizer\": \"tokenizer\",\n \"word_vocab_file\": \"vocab_file\",\n \"char_most_common\": \"most_common\",\n \"char_sequence_length_limit\": \"max_sequence_length\",\n \"char_tokenizer\": \"tokenizer\",\n \"char_vocab_file\": \"vocab_file\",\n }\n\n def upgrade_params(params):\n for key, value in drop_params.items():\n if key in params:\n if value in params:\n warnings.warn(\n f\"Removing deprecated config preprocessing parameter {key} as new param {value} already \"\n f\"present in the config\",\n DeprecationWarning,\n )\n else:\n warnings.warn(\n f\"Renaming deprecated config preprocessing parameter {key} to {value}\",\n DeprecationWarning,\n )\n params[value] = params[key]\n del params[key]\n\n sequence_types = [SEQUENCE, TEXT, AUDIO, TIMESERIES]\n for dtype in sequence_types:\n params = config.get(PREPROCESSING, {}).get(dtype, {})\n upgrade_params(params)\n\n for feature in config[INPUT_FEATURES]:\n if feature.get(TYPE) not in sequence_types:\n continue\n params = feature.get(PREPROCESSING, {})\n upgrade_params(params)\n\n return config\n\n\n@register_config_transformation(\"0.5\")\ndef rename_training_to_trainer(config: ModelConfigDict) -> ModelConfigDict:\n if TRAINING in config:\n warnings.warn(\n 'Config section \"training\" renamed to \"trainer\" and will be removed in a future version', DeprecationWarning\n )\n config[TRAINER] = config[TRAINING]\n del config[TRAINING]\n return config\n\n\n@register_config_transformation(\"0.5\", [\"input_features\", \"output_features\"])\ndef _upgrade_use_bias_in_features(feature: FeatureConfigDict) -> FeatureConfigDict:\n def upgrade_use_bias(config):\n if BIAS in config:\n warnings.warn(\n 'Parameter \"bias\" renamed to \"use_bias\" and will be removed in a future version', DeprecationWarning\n )\n config[USE_BIAS] = config[BIAS]\n del config[BIAS]\n if CONV_BIAS in config:\n warnings.warn(\n 'Parameter \"conv_bias\" renamed to \"conv_use_bias\" and will be removed in a future version',\n DeprecationWarning,\n )\n config[CONV_USE_BIAS] = config[CONV_BIAS]\n del config[CONV_BIAS]\n if DEFAULT_BIAS in config:\n warnings.warn(\n 'Parameter \"default_bias\" renamed to \"default_use_bias\" and will be removed in a future version',\n DeprecationWarning,\n )\n config[DEFAULT_USE_BIAS] = config[DEFAULT_BIAS]\n del config[DEFAULT_BIAS]\n\n _traverse_dicts(feature, upgrade_use_bias)\n return feature\n\n\n@register_config_transformation(\"0.5\", [\"input_features\", \"output_features\"])\ndef _upgrade_feature(feature: FeatureConfigDict) -> FeatureConfigDict:\n \"\"\"Upgrades feature config (in-place)\"\"\"\n if feature.get(TYPE) == \"numerical\":\n warnings.warn(\n 'Feature type \"numerical\" renamed to \"number\" and will be removed in a future version', DeprecationWarning\n )\n feature[TYPE] = NUMBER\n if feature.get(TYPE) == AUDIO:\n if PREPROCESSING in feature:\n feature[PREPROCESSING] = upgrade_audio_preprocessing(feature[PREPROCESSING])\n warnings.warn(\n \"Parameters specified at the `audio_feature` parameter level have been unnested and should now \"\n \"be specified at the preprocessing level. Support for `audio_feature` will be removed in a future version\",\n DeprecationWarning,\n )\n return feature\n\n\ndef upgrade_audio_preprocessing(preproc_dict: PreprocessingConfigDict) -> PreprocessingConfigDict:\n if \"audio_feature\" in preproc_dict:\n for k, v in preproc_dict[\"audio_feature\"].items():\n preproc_dict[k] = v\n del preproc_dict[\"audio_feature\"]\n return preproc_dict\n\n\n@register_config_transformation(\"0.6\", [\"input_features\"])\ndef _upgrade_encoder_params(feature: FeatureConfigDict) -> FeatureConfigDict:\n return _upgrade_encoder_decoder_params(feature, True)\n\n\n@register_config_transformation(\"0.6\", [\"output_features\"])\ndef _upgrade_decoder_params(feature: FeatureConfigDict) -> FeatureConfigDict:\n return _upgrade_encoder_decoder_params(feature, False)\n\n\ndef _upgrade_encoder_decoder_params(feature: FeatureConfigDict, input_feature: bool) -> FeatureConfigDict:\n \"\"\"\n This function nests un-nested encoder/decoder parameters to conform with the new config structure for 0.6\n Args:\n feature (Dict): Feature to nest encoder/decoder params for.\n input_feature (Bool): Whether this feature is an input feature or not.\n \"\"\"\n if TYPE not in feature:\n return feature\n\n try:\n if input_feature:\n module_type = ENCODER\n feature_cls = get_from_registry(feature[TYPE], get_input_type_registry())\n else:\n module_type = DECODER\n feature_cls = get_from_registry(feature[TYPE], get_output_type_registry())\n except ValueError:\n logging.exception(\"Failed to obtain encoder / decoder from registry\")\n return feature\n\n feature_schema_cls = feature_cls.get_schema_cls()\n feature_keys = feature_schema_cls.get_valid_field_names()\n\n # These keys have been renamed from the form below to `fc_` in the new config\n fc_layer_keys = [\n \"fc_layers\",\n \"output_size\",\n \"use_bias\",\n \"weights_initializer\",\n \"bias_initializer\",\n \"norm\",\n \"norm_params\",\n \"activation\",\n \"dropout\",\n ]\n\n module = feature.get(module_type, {})\n\n warn = False\n if isinstance(module, str):\n module = {TYPE: module}\n feature[module_type] = module\n warn = True\n\n nested_params = []\n for k, v in feature.items():\n if k not in feature_keys:\n module[k] = v\n if k in fc_layer_keys and module_type == DECODER:\n module[f\"fc_{k}\"] = v\n nested_params.append(k)\n warn = True\n\n if module:\n if module_type in feature:\n feature[module_type].update(module)\n else:\n feature[module_type] = module\n\n for k in nested_params:\n del feature[k]\n\n if warn:\n warnings.warn(\n f\"{module_type} specific parameters should now be nested within a dictionary under the '{module_type}' \"\n f\"parameter. Support for un-nested {module_type} specific parameters will be removed in a future version\",\n DeprecationWarning,\n )\n return feature\n\n\n@register_config_transformation(\"0.5\", [\"hyperopt\"])\ndef _upgrade_hyperopt(hyperopt: HyperoptConfigDict) -> HyperoptConfigDict:\n \"\"\"Upgrades hyperopt config (in-place)\"\"\"\n # check for use of legacy \"training\" reference, if any found convert to \"trainer\"\n if PARAMETERS in hyperopt:\n hparams = hyperopt[PARAMETERS]\n for k, v in list(hparams.items()):\n substr = \"training.\"\n if k.startswith(substr):\n warnings.warn(\n 'Config section \"training\" renamed to \"trainer\" and will be removed in a future version',\n DeprecationWarning,\n )\n hparams[\"trainer.\" + k[len(substr) :]] = v\n del hparams[k]\n\n # check for legacy parameters in \"executor\"\n if EXECUTOR in hyperopt:\n hpexecutor = hyperopt[EXECUTOR]\n executor_type = hpexecutor.get(TYPE, None)\n if executor_type is not None and executor_type != RAY:\n warnings.warn(\n f'executor type \"{executor_type}\" not supported, converted to \"ray\" will be flagged as error '\n \"in a future version\",\n DeprecationWarning,\n )\n hpexecutor[TYPE] = RAY\n\n # if search_alg not at top level and is present in executor, promote to top level\n if SEARCH_ALG in hpexecutor:\n # promote only if not in top-level, otherwise use current top-level\n if SEARCH_ALG not in hyperopt:\n hyperopt[SEARCH_ALG] = hpexecutor[SEARCH_ALG]\n if isinstance(hyperopt[SEARCH_ALG], str):\n hyperopt[SEARCH_ALG] = {TYPE: hyperopt[SEARCH_ALG]}\n del hpexecutor[SEARCH_ALG]\n else:\n warnings.warn(\n 'Missing \"executor\" section, adding \"ray\" executor will be flagged as error in a future version',\n DeprecationWarning,\n )\n hyperopt[EXECUTOR] = {TYPE: RAY}\n\n # check for legacy \"sampler\" section\n if SAMPLER in hyperopt:\n warnings.warn(\n f'\"{SAMPLER}\" is no longer supported, converted to \"{SEARCH_ALG}\". \"{SAMPLER}\" will be flagged as '\n \"error in a future version\",\n DeprecationWarning,\n )\n if SEARCH_ALG in hyperopt[SAMPLER]:\n if SEARCH_ALG not in hyperopt:\n hyperopt[SEARCH_ALG] = hyperopt[SAMPLER][SEARCH_ALG]\n if isinstance(hyperopt[SEARCH_ALG], str):\n hyperopt[SEARCH_ALG] = {TYPE: hyperopt[SEARCH_ALG]}\n warnings.warn('Moved \"search_alg\" to hyperopt config top-level', DeprecationWarning)\n\n # if num_samples or scheduler exist in SAMPLER move to EXECUTOR Section\n if NUM_SAMPLES in hyperopt[SAMPLER] and NUM_SAMPLES not in hyperopt[EXECUTOR]:\n hyperopt[EXECUTOR][NUM_SAMPLES] = hyperopt[SAMPLER][NUM_SAMPLES]\n warnings.warn('Moved \"num_samples\" from \"sampler\" to \"executor\"', DeprecationWarning)\n\n if SCHEDULER in hyperopt[SAMPLER] and SCHEDULER not in hyperopt[EXECUTOR]:\n hyperopt[EXECUTOR][SCHEDULER] = hyperopt[SAMPLER][SCHEDULER]\n warnings.warn('Moved \"scheduler\" from \"sampler\" to \"executor\"', DeprecationWarning)\n\n if SCHEDULER in hyperopt[EXECUTOR] and len(hyperopt[EXECUTOR][SCHEDULER].keys()) == 0:\n del hyperopt[EXECUTOR][SCHEDULER]\n\n # remove legacy section\n del hyperopt[SAMPLER]\n\n if SEARCH_ALG not in hyperopt:\n # make top-level as search_alg, if missing put in default value\n hyperopt[SEARCH_ALG] = {TYPE: \"variant_generator\"}\n warnings.warn(\n 'Missing \"search_alg\" at hyperopt top-level, adding in default value, will be flagged as error '\n \"in a future version\",\n DeprecationWarning,\n )\n return hyperopt\n\n\n@register_config_transformation(\"0.5\", [\"trainer\"])\ndef _upgrade_trainer(trainer: TrainerConfigDict) -> TrainerConfigDict:\n \"\"\"Upgrades trainer config (in-place)\"\"\"\n eval_batch_size = trainer.get(EVAL_BATCH_SIZE)\n if eval_batch_size == 0:\n warnings.warn(\n \"`trainer.eval_batch_size` value `0` changed to `None`, will be unsupported in a future version\",\n DeprecationWarning,\n )\n trainer[EVAL_BATCH_SIZE] = None\n return trainer\n\n\n@register_config_transformation(\"0.5\")\ndef _upgrade_preprocessing_defaults(config: ModelConfigDict) -> ModelConfigDict:\n \"\"\"Move feature-specific preprocessing parameters into defaults in config (in-place)\"\"\"\n type_specific_preprocessing_params = dict()\n\n # If preprocessing section specified and it contains feature specific preprocessing parameters,\n # make a copy and delete it from the preprocessing section\n for parameter in list(config.get(PREPROCESSING, {})):\n if parameter in get_base_type_registry():\n warnings.warn(\n f\"Moving preprocessing configuration for `{parameter}` feature type from `preprocessing` section\"\n \" to `defaults` section in Ludwig config. This will be unsupported in a future version.\",\n DeprecationWarning,\n )\n type_specific_preprocessing_params[parameter] = config[PREPROCESSING].pop(parameter)\n\n if parameter == \"numerical\":\n warnings.warn(\n f\"Moving preprocessing configuration for `{parameter}` feature type from `preprocessing` section\"\n \" to `defaults` section in Ludwig config. This will be unsupported in a future version.\",\n DeprecationWarning,\n )\n type_specific_preprocessing_params[NUMBER] = config[PREPROCESSING].pop(parameter)\n\n # Delete empty preprocessing section if no other preprocessing parameters specified\n if PREPROCESSING in config and not config[PREPROCESSING]:\n del config[PREPROCESSING]\n\n # Update defaults with the default feature specific preprocessing parameters\n defaults = config.get(DEFAULTS, {})\n for feature_type, preprocessing_param in type_specific_preprocessing_params.items():\n if PREPROCESSING in preprocessing_param:\n preprocessing_param = preprocessing_param[PREPROCESSING]\n\n if feature_type == AUDIO:\n preprocessing_param = upgrade_audio_preprocessing(preprocessing_param)\n\n # If defaults was empty, then create a new key with feature type\n if feature_type not in defaults:\n defaults[feature_type] = {PREPROCESSING: preprocessing_param}\n # Feature type exists but preprocessing hasn't be specified\n elif PREPROCESSING not in defaults[feature_type]:\n defaults[feature_type][PREPROCESSING] = preprocessing_param\n # Update default feature specific preprocessing with parameters from config\n else:\n defaults[feature_type][PREPROCESSING].update(\n merge_dict(defaults[feature_type][PREPROCESSING], preprocessing_param)\n )\n\n if defaults:\n config[DEFAULTS] = defaults\n\n return config\n\n\n@register_config_transformation(\"0.5\", \"preprocessing\")\ndef _upgrade_preprocessing_split(preprocessing: PreprocessingConfigDict) -> PreprocessingConfigDict:\n \"\"\"Upgrade split related parameters in preprocessing.\"\"\"\n split_params = {}\n\n force_split = preprocessing.pop(FORCE_SPLIT, None)\n split_probabilities = preprocessing.pop(SPLIT_PROBABILITIES, None)\n stratify = preprocessing.pop(STRATIFY, None)\n\n if split_probabilities is not None:\n split_params[PROBABILITIES] = split_probabilities\n warnings.warn(\n \"`preprocessing.split_probabilities` has been replaced by `preprocessing.split.probabilities`, \"\n \"will be flagged as error in a future version\",\n DeprecationWarning,\n )\n\n if stratify is not None:\n split_params[TYPE] = STRATIFY\n split_params[COLUMN] = stratify\n warnings.warn(\n \"`preprocessing.stratify` has been replaced by `preprocessing.split.column` \"\n 'when setting `preprocessing.split.type` to \"stratify\", '\n \"will be flagged as error in a future version\",\n DeprecationWarning,\n )\n\n if force_split is not None:\n warnings.warn(\n \"`preprocessing.force_split` has been replaced by `preprocessing.split.type`, \"\n \"will be flagged as error in a future version\",\n DeprecationWarning,\n )\n\n if TYPE not in split_params:\n split_params[TYPE] = RANDOM\n\n if split_params:\n preprocessing[SPLIT] = split_params\n\n if AUDIO in preprocessing:\n if \"audio_feature\" in preprocessing[AUDIO]:\n for k, v in preprocessing[AUDIO][\"audio_feature\"].items():\n preprocessing[AUDIO][k] = v\n del preprocessing[AUDIO][\"audio_feature\"]\n warnings.warn(\n \"Parameters specified at the `audio_feature` parameter level have been unnested and should now \"\n \"be specified at the preprocessing level. Support for `audio_feature` will be removed in a future version\",\n DeprecationWarning,\n )\n return preprocessing\n\n\n@register_config_transformation(\"0.5\")\ndef update_training(config: ModelConfigDict) -> ModelConfigDict:\n if TRAINING in config:\n warnings.warn(\n 'Config section \"training\" renamed to \"trainer\" and will be removed in a future version', DeprecationWarning\n )\n config[TRAINER] = config[TRAINING]\n del config[TRAINING]\n return config\n\n\n@register_config_transformation(\"0.6\")\ndef upgrade_missing_value_strategy(config: ModelConfigDict) -> ModelConfigDict:\n for input_feature in config.get(INPUT_FEATURES, []):\n if _is_old_missing_value_strategy(input_feature):\n _update_old_missing_value_strategy(input_feature)\n\n for output_feature in config.get(OUTPUT_FEATURES, []):\n if _is_old_missing_value_strategy(output_feature):\n _update_old_missing_value_strategy(output_feature)\n\n for feature, feature_defaults in config.get(DEFAULTS, {}).items():\n if _is_old_missing_value_strategy(feature_defaults):\n _update_old_missing_value_strategy(config.get(DEFAULTS).get(feature))\n\n return config\n\n\n@register_config_transformation(\"0.6\", [\"trainer\"])\ndef _upgrade_max_batch_size(trainer: TrainerConfigDict) -> TrainerConfigDict:\n if \"increase_batch_size_on_plateau_max\" in trainer:\n warnings.warn(\n 'Config param \"increase_batch_size_on_plateau_max\" renamed to \"max_batch_size\" and will be '\n \"removed in a future version\",\n DeprecationWarning,\n )\n increase_batch_size_on_plateau_max_val = trainer.pop(\"increase_batch_size_on_plateau_max\")\n if \"max_batch_size\" in trainer:\n warnings.warn('\"max_batch_size\" config param already set. Discarding \"increase_batch_size_on_plateau_max\".')\n else:\n warnings.warn(\n f'Setting \"max_batch_size\" config param to \"increase_batch_size_on_plateau_max\" value '\n f'({increase_batch_size_on_plateau_max_val}) and discarding \"increase_batch_size_on_plateau_max\"'\n )\n trainer[\"max_batch_size\"] = increase_batch_size_on_plateau_max_val\n return trainer\n\n\n@register_config_transformation(\"0.6\")\ndef remove_trainer_type(config: ModelConfigDict) -> ModelConfigDict:\n # LLM Model types support different trainer types\n if config.get(\"model_type\", None) == \"llm\":\n return config\n\n if TYPE in config.get(\"trainer\", {}):\n warnings.warn(\n \"Config param `type` has been removed from the trainer. The trainer type is determined by the top level \"\n \" `model_type` parameter. Support for the `type` params in trainer will be removed in a future version\",\n DeprecationWarning,\n )\n del config[\"trainer\"][TYPE]\n\n return config\n\n\n@register_config_transformation(\"0.7\", [\"trainer\"])\ndef learning_rate_scheduler(trainer: TrainerConfigDict) -> TrainerConfigDict:\n key_mapping = {\n \"reduce_learning_rate_on_plateau\": \"reduce_on_plateau\",\n \"reduce_learning_rate_on_plateau_patience\": \"reduce_on_plateau_patience\",\n \"reduce_learning_rate_on_plateau_rate\": \"reduce_on_plateau_rate\",\n \"reduce_learning_rate_eval_metric\": \"reduce_eval_metric\",\n \"reduce_learning_rate_eval_split\": \"reduce_eval_split\",\n \"decay\": \"decay\",\n \"decay_steps\": \"decay_steps\",\n \"decay_rate\": \"decay_rate\",\n \"staircase\": \"staircase\",\n \"learning_rate_warmup_epochs\": \"warmup_evaluations\",\n }\n\n lr_scheduler = trainer.get(\"learning_rate_scheduler\", {})\n for old_key, new_key in key_mapping.items():\n if old_key in trainer:\n warnings.warn(\n f\"Config param `trainer.{old_key}` has been moved to `trainer.learning_rate_scheduler.{new_key}`.\",\n DeprecationWarning,\n )\n if new_key in lr_scheduler:\n warnings.warn(\n f\"`trainer.learning_rate_scheduler.{new_key}` config param already set. \"\n f\"Discarding `trainer.{old_key}`.\"\n )\n else:\n value = trainer[old_key]\n if old_key == \"decay\" and isinstance(value, bool):\n # Decay has changed from a bool to an optional enum\n lr_scheduler[new_key] = \"exponential\" if value else None\n elif old_key == \"reduce_learning_rate_on_plateau\":\n lr_scheduler[new_key] = int(value)\n else:\n lr_scheduler[new_key] = value\n del trainer[old_key]\n\n if lr_scheduler:\n trainer[\"learning_rate_scheduler\"] = lr_scheduler\n\n return trainer\n\n\n@register_config_transformation(\"0.7\", [\"input_features\"])\ndef _upgrade_legacy_image_encoders(feature: FeatureConfigDict) -> FeatureConfigDict:\n if feature.get(TYPE) != IMAGE:\n return feature\n\n encoder_mapping = {\n \"resnet\": \"_resnet_legacy\",\n \"vit\": \"_vit_legacy\",\n }\n\n encoder = feature.get(ENCODER, {})\n encoder_type = encoder.get(TYPE)\n if encoder_type not in encoder_mapping:\n return feature\n\n # For this version of Ludwig, only ECD supported these encoders.\n new_encoder_cls = get_encoder_cls(MODEL_ECD, feature[TYPE], encoder_type)\n new_encoder_fields = new_encoder_cls.get_valid_field_names()\n\n legacy_encoder_cls = get_encoder_cls(MODEL_ECD, feature[TYPE], encoder_mapping[encoder_type])\n legacy_encoder_fields = legacy_encoder_cls.get_valid_field_names()\n\n user_fields = set(encoder.keys())\n user_fields.remove(TYPE)\n\n removed_fields = legacy_encoder_fields.difference(new_encoder_fields)\n added_fields = new_encoder_fields.difference(legacy_encoder_fields)\n\n user_legacy_fields = user_fields.intersection(removed_fields)\n user_new_fields = user_fields.intersection(added_fields)\n if len(user_legacy_fields) > 0:\n if len(user_new_fields) > 0:\n raise ValueError(\n f\"Intended encoder type is ambiguous. \"\n f\"Provided encoder fields matching encoder '{encoder_type}' {user_new_fields} and \"\n f\"legacy encoder '{encoder_mapping[encoder_type]}' {user_legacy_fields}. \"\n f\"Please remove features unique to one of these encoder types from your configuration.\"\n )\n\n warnings.warn(\n f\"Encoder '{encoder_type}' with params '{user_legacy_fields}' has been renamed to \"\n f\"'{encoder_mapping[encoder_type]}'. Please upgrade your config to use the new '{encoder_type}' as \"\n f\"support for '{encoder_mapping[encoder_type]}' is not guaranteed in future versions.\",\n DeprecationWarning,\n )\n\n # User provided legacy fields and no new fields, so we assume they intended to use the legacy encoder\n encoder[TYPE] = encoder_mapping[encoder_type]\n\n return feature\n\n\n@register_config_transformation(\"0.7\")\ndef upgrade_missing_hyperopt(config: ModelConfigDict) -> ModelConfigDict:\n hyperopt = config.get(HYPEROPT)\n if hyperopt == {}:\n # This is a deprecated form of providing a missing hyperopt section, as it violates the schema definition\n warnings.warn(\n \"Config section `hyperopt: {}` is deprecated, please set `hyperopt: null` to disable hyperopt.\",\n DeprecationWarning,\n )\n del config[HYPEROPT]\n return config\n\n\n@register_config_transformation(\"0.7\", \"defaults\")\ndef remove_extra_type_param_in_defaults_config(defaults: FeatureTypeDefaultsDict) -> FeatureTypeDefaultsDict:\n \"\"\"Fixes a bug introduced before 0.7.3.\n\n [1] and subsequent refactors accidentally introduced a bug where a `type` param was added to every feature in the\n defaults config. It was removed by [2], but made it into one of the patch releases. This transformation removes that\n `type` param from each section of the defaults config if it exists.\n\n [1]: https://github.com/ludwig-ai/ludwig/pull/3223\n [2]: https://github.com/ludwig-ai/ludwig/pull/3258\n \"\"\"\n defaults_copy = copy.deepcopy(defaults)\n for feature_type, feature_config in defaults.items():\n if TYPE in feature_config:\n del defaults_copy[feature_type][TYPE]\n return defaults_copy\n\n\ndef upgrade_metadata(metadata: TrainingSetMetadataDict) -> TrainingSetMetadataDict:\n # TODO(travis): stopgap solution, we should make it so we don't need to do this\n # by decoupling config and metadata\n metadata = copy.deepcopy(metadata)\n _upgrade_metadata_missing_values(metadata)\n return metadata\n\n\ndef _upgrade_metadata_missing_values(metadata: TrainingSetMetadataDict):\n for k, v in metadata.items():\n if isinstance(v, dict) and _is_old_missing_value_strategy(v):\n _update_old_missing_value_strategy(v)\n elif isinstance(v, dict) and _is_image_feature(v):\n _update_old_image_preprocessing(v)\n\n\ndef _update_old_missing_value_strategy(feature_config: FeatureConfigDict):\n missing_value_strategy = feature_config.get(PREPROCESSING).get(MISSING_VALUE_STRATEGY)\n replacement_strategy = \"bfill\" if missing_value_strategy == \"backfill\" else \"ffill\"\n feature_name = feature_config.get(NAME)\n warnings.warn(\n f\"Using `{replacement_strategy}` instead of `{missing_value_strategy}` as the missing value strategy\"\n f\" for `{feature_name}`. These are identical. `{missing_value_strategy}` will be removed in a future version\",\n DeprecationWarning,\n )\n feature_config[PREPROCESSING].update({MISSING_VALUE_STRATEGY: replacement_strategy})\n\n\ndef _is_old_missing_value_strategy(feature_config: FeatureConfigDict):\n if PREPROCESSING not in feature_config:\n return False\n missing_value_strategy = feature_config.get(PREPROCESSING).get(MISSING_VALUE_STRATEGY, None)\n if not missing_value_strategy or missing_value_strategy not in (\"backfill\", \"pad\"):\n return False\n return True\n\n\ndef _is_image_feature(feature_config: FeatureConfigDict):\n preproc = feature_config.get(PREPROCESSING, {})\n return HEIGHT in preproc and WIDTH in preproc\n\n\ndef _update_old_image_preprocessing(feature_config: FeatureConfigDict):\n preprocessing = feature_config.get(PREPROCESSING)\n if not preprocessing:\n return\n preprocessing[\"standardize_image\"] = preprocessing.get(\"standardize_image\")\n"
},
{
"path": "ludwig/utils/batch_size_tuner.py",
"content": "import gc\nimport logging\nimport statistics\nimport time\nfrom abc import ABC\n\nimport torch\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import MAX_BATCH_SIZE_DATASET_FRACTION, MIN_POSSIBLE_BATCH_SIZE\n\nlogger = logging.getLogger(__name__)\n\nTOTAL_STEPS = 5\n\n\n@DeveloperAPI\nclass BatchSizeEvaluator(ABC):\n def select_best_batch_size(\n self,\n dataset_len: int,\n max_batch_size: int | None = None,\n max_trials: int = 20,\n is_coordinator: bool | None = True,\n global_max_sequence_length: int | None = None,\n ) -> int:\n \"\"\"Returns optimal batch size as measured by throughput (samples / sec).\"\"\"\n logger.info(\"Tuning batch size...\")\n\n max_batch_size = max_batch_size or dataset_len\n\n def _is_valid_batch_size(batch_size):\n # make sure that batch size is valid (e.g. less than 20% of ds size and max_batch_size)\n is_smaller_than_training_set = batch_size <= MAX_BATCH_SIZE_DATASET_FRACTION * dataset_len\n is_under_max_batch_size = batch_size <= max_batch_size\n is_valid = is_smaller_than_training_set and is_under_max_batch_size\n if not is_valid and is_coordinator:\n logger.info(\n f\"Batch size {batch_size} is invalid, must be less than or equal to \"\n f\"{MAX_BATCH_SIZE_DATASET_FRACTION * 100}% dataset size \"\n f\"({int(MAX_BATCH_SIZE_DATASET_FRACTION * dataset_len)} samples \"\n f\"of {dataset_len}) and less than or equal to max batch size {max_batch_size}\"\n )\n return is_valid\n\n batch_size = MIN_POSSIBLE_BATCH_SIZE\n best_samples_per_sec = 0\n best_batch_size = None\n count = 0\n while count < max_trials and _is_valid_batch_size(batch_size):\n if is_coordinator:\n logger.info(f\"Exploring batch_size={batch_size}\")\n gc.collect()\n\n try:\n samples_per_sec = self.evaluate(\n batch_size, total_steps=TOTAL_STEPS, global_max_sequence_length=global_max_sequence_length\n )\n if is_coordinator:\n logger.info(f\"Throughput at batch_size={batch_size}: {samples_per_sec:.5f} samples/s\")\n if samples_per_sec < best_samples_per_sec:\n # We assume that once the throughput starts degrading, it won't go up again\n if is_coordinator:\n logger.info(f\"Throughput decrease at batch_size={batch_size}\")\n break\n\n best_samples_per_sec = samples_per_sec\n best_batch_size = batch_size\n count += 1\n\n # double batch size\n batch_size *= 2\n except RuntimeError as e:\n # PyTorch only generates Runtime errors for CUDA OOM.\n gc.collect()\n if \"CUDA out of memory\" in str(e) or isinstance(e, torch.cuda.OutOfMemoryError):\n if is_coordinator:\n logger.info(f\"OOM at batch_size={batch_size}\")\n else:\n # Not a CUDA error\n raise\n break\n\n # Ensure that some batch size is found.\n # `best_batch_size` can be None if the first batch size is invalid.\n if best_batch_size is None:\n if is_coordinator:\n logger.info(f\"Could not tune batch size, using minimum batch size of {MIN_POSSIBLE_BATCH_SIZE}\")\n best_batch_size = MIN_POSSIBLE_BATCH_SIZE\n\n if is_coordinator:\n logger.info(f\"Selected batch_size={best_batch_size}\")\n return best_batch_size\n\n def evaluate(self, batch_size: int, total_steps: int = 5, global_max_sequence_length: int | None = None) -> float:\n \"\"\"Evaluates throughput of the given batch size.\n\n Return:\n Median throughput in samples / sec.\n \"\"\"\n durations = []\n for _ in range(total_steps):\n self.reset()\n start_ts = time.time()\n self.step(batch_size, global_max_sequence_length=global_max_sequence_length)\n durations.append(time.time() - start_ts)\n\n med_duration_s = statistics.median(durations)\n if med_duration_s == 0.0:\n return float(\"inf\")\n\n return batch_size / med_duration_s\n\n def reset(self):\n \"\"\"Called at the beginning of each evaluation step.\"\"\"\n\n def step(self, batch_size: int, global_max_sequence_length: int | None = None):\n \"\"\"Called each step to evaluate the given batch size.\"\"\"\n raise NotImplementedError(\"`step` must be implemented by concrete evaluator.\")\n\n\nclass BaseLLMBatchSizeEvaluator(BatchSizeEvaluator):\n \"\"\"Base class for batch size evaluators for LLM models.\"\"\"\n\n def __init__(self, trainer):\n self.trainer = trainer\n self.input_feature_name, self.input_feature = list(trainer.model.input_features.items())[0]\n self.output_feature_name, self.output_feature = list(trainer.model.output_features.items())[0]\n\n # Get the length of the longest input sequence from the training data\n self.input_msl = self.input_feature.input_shape[0]\n if trainer.model.config_obj.input_features[0].preprocessing.max_sequence_length:\n self.input_msl = trainer.model.config_obj.input_features[0].preprocessing.max_sequence_length\n\n # Get the length of the longest output sequence from the training data\n self.output_msl = self.output_feature.output_shape[0]\n if trainer.model.config_obj.output_features[0].preprocessing.max_sequence_length:\n self.output_msl = trainer.model.config_obj.output_features[0].preprocessing.max_sequence_length\n\n # This is useful to create the synthetic input and target data which will be a\n # random sequence of integers between 0 and vocab_size\n self.vocab_size = len(trainer.model.config_obj.input_features[0].encoder.vocab)\n\n def reset(self):\n self.trainer.model.reset_metrics()\n self.trainer.optimizer.zero_grad()\n\n def step(self, batch_size: int, global_max_sequence_length: int | None = None):\n if global_max_sequence_length and self.input_msl + self.output_msl > global_max_sequence_length:\n # In this case, we just need to make sure that the length of the synthetic data exceeds\n # max_sequence_length by at most a small amount\n self.input_msl = global_max_sequence_length // 2 + 1\n self.output_msl = global_max_sequence_length // 2 + 1\n\n inputs = {\n self.input_feature_name: torch.randint(0, self.vocab_size, size=(batch_size, self.input_msl))\n .to(self.input_feature.input_dtype)\n .to(self.trainer.device)\n }\n targets = {\n self.output_feature_name: torch.randint(0, self.vocab_size, size=(batch_size, self.output_msl))\n .to(self.output_feature.get_output_dtype())\n .to(self.trainer.device)\n }\n\n self.perform_step(inputs, targets)\n\n def perform_step(self, inputs, targets):\n raise NotImplementedError(\"perform_step method must be implemented in subclasses\")\n\n\nclass LLMFinetuneTrainerBatchSizeEvaluator(BaseLLMBatchSizeEvaluator):\n \"\"\"Batch size evaluator for training batch size for LLM finetuning.\"\"\"\n\n def perform_step(self, inputs, targets):\n self.trainer.train_step(inputs, targets)\n\n\nclass LLMFinetunePredictBatchSizeEvaluator(BaseLLMBatchSizeEvaluator):\n \"\"\"Batch size evaluator for prediction/evaluation batch size for LLM finetuning.\"\"\"\n\n def perform_step(self, inputs, targets):\n with torch.no_grad():\n self.trainer.dist_model((inputs, targets))\n"
},
{
"path": "ludwig/utils/calibration.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nimport logging\nfrom abc import ABC, abstractmethod\nfrom dataclasses import dataclass\n\nimport numpy as np\nimport torch\nimport torch.nn as nn\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import BINARY, CATEGORY\nfrom ludwig.utils.registry import DEFAULT_KEYS, Registry\n\nlogger = logging.getLogger(__name__)\n\ncalibration_registry = Registry()\n\n\n@DeveloperAPI\ndef register_calibration(name: str, features: str | list[str], default=False):\n \"\"\"Registers a calibration implementation for a list of features.\"\"\"\n if isinstance(features, str):\n features = [features]\n\n def wrap(cls):\n for feature in features:\n feature_registry = calibration_registry.get(feature, {})\n feature_registry[name] = cls\n if default:\n for key in DEFAULT_KEYS:\n feature_registry[key] = cls\n calibration_registry[feature] = feature_registry\n return cls\n\n return wrap\n\n\n@DeveloperAPI\ndef get_calibration_cls(feature: str, calibration_method: str) -> type[\"CalibrationModule\"]:\n \"\"\"Get calibration class for specified feature type and calibration method.\"\"\"\n if not calibration_method:\n return None\n if feature in calibration_registry:\n if calibration_method in calibration_registry[feature]:\n return calibration_registry[feature][calibration_method]\n else:\n raise ValueError(f\"Calibration method {calibration_method} not supported for {feature} output features\")\n else:\n raise ValueError(f\"Calibration not yet supported for {feature} output features\")\n return None\n\n\n@DeveloperAPI\nclass ECELoss(nn.Module):\n \"\"\"Calculates the Expected Calibration Error of a model.\n\n The input to this loss is the logits of a model, NOT the softmax scores.\n This divides the confidence outputs into equally-sized interval bins.\n In each bin, we compute the confidence gap:\n\n bin_gap = | avg_confidence_in_bin - accuracy_in_bin |\n\n We then return an average of the gaps, weighted by the number of samples in each bin.\n\n References:\n Naeini, Mahdi Pakdaman, Gregory F. Cooper, and Milos Hauskrecht\n \"Obtaining Well Calibrated Probabilities Using Bayesian Binning.\" AAAI. 2015.\n\n Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger\n \"On Calibration of Modern Neural Networks.\" PMLR 2017.\n \"\"\"\n\n def __init__(self, n_bins: int = 15):\n \"\"\"n_bins (int): number of confidence interval bins.\"\"\"\n super().__init__()\n bin_boundaries = torch.linspace(0, 1, n_bins + 1)\n self.bin_lowers = bin_boundaries[:-1]\n self.bin_uppers = bin_boundaries[1:]\n\n def forward(self, logits: torch.Tensor, one_hot_labels: torch.Tensor) -> torch.Tensor:\n softmaxes = nn.functional.softmax(logits, dim=1)\n confidences, predictions = torch.max(softmaxes, 1)\n labels = torch.argmax(one_hot_labels, 1)\n accuracies = predictions.eq(labels)\n ece = torch.zeros(1, device=logits.device)\n for bin_lower, bin_upper in zip(self.bin_lowers, self.bin_uppers):\n # Calculates |confidence - accuracy| in each bin\n in_bin = confidences.gt(bin_lower.item()) * confidences.le(bin_upper.item())\n prop_in_bin = in_bin.float().mean()\n if prop_in_bin.item() > 0:\n accuracy_in_bin = accuracies[in_bin].float().mean()\n avg_confidence_in_bin = confidences[in_bin].mean()\n ece += torch.abs(avg_confidence_in_bin - accuracy_in_bin) * prop_in_bin\n return ece\n\n\n@DeveloperAPI\n@dataclass\nclass CalibrationResult:\n \"\"\"Tracks results of probability calibration.\"\"\"\n\n before_calibration_nll: float\n before_calibration_ece: float\n after_calibration_nll: float\n after_calibration_ece: float\n\n\n@DeveloperAPI\nclass CalibrationModule(nn.Module, ABC):\n @abstractmethod\n def train_calibration(\n self, logits: torch.Tensor | np.ndarray, labels: torch.Tensor | np.ndarray\n ) -> CalibrationResult:\n \"\"\"Calibrate output probabilities using logits and labels from validation set.\"\"\"\n return NotImplementedError()\n\n\n@DeveloperAPI\n@register_calibration(\"temperature_scaling\", [BINARY, CATEGORY], default=True)\nclass TemperatureScaling(CalibrationModule):\n \"\"\"Implements temperature scaling of logits. Based on results from \"On Calibration of Modern Neural Networks\":\n https://arxiv.org/abs/1706.04599. Temperature scaling scales all logits by the same constant factor. Though it\n may modify output probabilities it will never change argmax or categorical top-n predictions. In the case of\n binary classification with a threshold, however, calibration may change predictions.\n\n Implementation inspired by https://github.com/gpleiss/temperature_scaling\n\n Args:\n num_classes: The number of classes. Must be 2 if binary is True.\n binary: If binary is true, logits is expected to be a 1-dimensional array. If false, logits is a 2-dimensional\n array of shape (num_examples, num_classes).\n \"\"\"\n\n def __init__(self, num_classes: int = 2, binary: bool = False):\n super().__init__()\n self.num_classes = 2 if binary else num_classes\n self.binary = binary\n self.device = \"cuda\" if torch.cuda.is_available() and torch.cuda.device_count() > 0 else \"cpu\"\n self.temperature = nn.Parameter(torch.ones(1), requires_grad=False).to(self.device)\n\n def train_calibration(\n self, logits: torch.Tensor | np.ndarray, labels: torch.Tensor | np.ndarray\n ) -> CalibrationResult:\n logits = torch.as_tensor(logits, dtype=torch.float32, device=self.device)\n labels = torch.as_tensor(labels, dtype=torch.int64, device=self.device)\n one_hot_labels = nn.functional.one_hot(labels, self.num_classes).float()\n if self.binary:\n # Treat binary classification as multi-class with 2 classes to re-use code.\n # The math works out the same: softmax([0, a])[1] == sigmoid(a)\n logits = torch.stack([torch.zeros_like(logits), logits], axis=-1)\n nll_criterion = nn.CrossEntropyLoss().to(self.device)\n ece_criterion = ECELoss().to(self.device)\n # Saves the original temperature parameter, in case something goes wrong in optimization.\n original_temperature = self.temperature.clone().detach()\n self.temperature.requires_grad = True\n # Calculate NLL and ECE before temperature scaling\n before_calibration_nll = nll_criterion(logits, one_hot_labels).item()\n before_calibration_ece = ece_criterion(logits, one_hot_labels).item()\n logger.info(\n \"Before temperature scaling:\\n\"\n \" Negative log-likelihood: %.3f\\n\"\n \" Expected Calibration Error: %.3f\" % (before_calibration_nll, before_calibration_ece)\n )\n\n # Optimizes the temperature to minimize NLL\n optimizer = torch.optim.LBFGS([self.temperature], lr=0.01, max_iter=50, line_search_fn=\"strong_wolfe\")\n\n def eval():\n optimizer.zero_grad()\n loss = nll_criterion(self.scale_logits(logits), one_hot_labels)\n loss.backward()\n return loss\n\n optimizer.step(eval)\n\n # Calculate NLL and ECE after temperature scaling\n after_calibration_nll = nll_criterion(self.scale_logits(logits), one_hot_labels).item()\n after_calibration_ece = ece_criterion(self.scale_logits(logits), one_hot_labels).item()\n logger.info(\"Optimal temperature: %.3f\" % self.temperature.item())\n logger.info(\n \"After temperature scaling:\\n\"\n \" Negative log-likelihood: %.3f\\n\"\n \" Expected Calibration Error: %.3f\" % (after_calibration_nll, after_calibration_ece)\n )\n self.temperature.requires_grad = False\n # This should never happen, but if expected calibration error is higher after optimizing temperature, revert.\n if after_calibration_ece > before_calibration_ece:\n logger.warning(\n \"Expected calibration error higher after scaling, \"\n \"reverting to temperature=%.3f.\" % original_temperature.item()\n )\n with torch.no_grad():\n self.temperature.data = original_temperature.data\n return CalibrationResult(\n before_calibration_nll, before_calibration_ece, after_calibration_nll, after_calibration_ece\n )\n\n def scale_logits(self, logits: torch.Tensor) -> torch.Tensor:\n return torch.div(logits, self.temperature)\n\n def forward(self, logits: torch.Tensor) -> torch.Tensor:\n \"\"\"Converts logits to probabilities.\"\"\"\n scaled_logits = self.scale_logits(logits)\n if self.binary:\n return torch.sigmoid(scaled_logits)\n else:\n return torch.softmax(scaled_logits, -1)\n\n\n@DeveloperAPI\n@register_calibration(\"matrix_scaling\", CATEGORY, default=False)\nclass MatrixScaling(CalibrationModule):\n \"\"\"Implements matrix scaling of logits, as described in Beyond temperature scaling: Obtaining well-calibrated\n multiclass probabilities with Dirichlet calibration https://arxiv.org/abs/1910.12656.\n\n Unlike temperature scaling which has only one free parameter, matrix scaling has n_classes x (n_classes + 1)\n parameters. Use this only with a large validation set, as matrix scaling has a tendency to overfit small datasets.\n Also, unlike temperature scaling, matrix scaling can change the argmax or top-n predictions.\n\n NOTE: Matrix Scaling is not exposed in the UI or config yet, though it may be in a future release after testing.\n\n Args:\n num_classes: The number of classes.\n off_diagonal_l2: The regularization weight for off-diagonal matrix entries.\n mu: The regularization weight for bias vector. Defaults to off_diagonal_l2 if not specified.\n \"\"\"\n\n def __init__(self, num_classes: int = 2, off_diagonal_l2: float = 0.01, mu: float = None):\n super().__init__()\n self.num_classes = num_classes\n self.device = \"cuda\" if torch.cuda.is_available() and torch.cuda.device_count() > 0 else \"cpu\"\n self.w = nn.Parameter(torch.eye(self.num_classes), requires_grad=False).to(self.device)\n self.b = nn.Parameter(torch.zeros(self.num_classes), requires_grad=False).to(self.device)\n self.off_diagonal_l2 = off_diagonal_l2\n self.mu = off_diagonal_l2 if mu is None else mu\n\n def train_calibration(\n self, logits: torch.Tensor | np.ndarray, labels: torch.Tensor | np.ndarray\n ) -> CalibrationResult:\n logits = torch.as_tensor(logits, dtype=torch.float32, device=self.device)\n labels = torch.as_tensor(labels, dtype=torch.int64, device=self.device)\n one_hot_labels = nn.functional.one_hot(labels, self.num_classes).float()\n nll_criterion = nn.CrossEntropyLoss().to(self.device)\n ece_criterion = ECELoss().to(self.device)\n self.w.requires_grad = True\n self.b.requires_grad = True\n # Calculate NLL and ECE before temperature scaling\n before_calibration_nll = nll_criterion(logits, one_hot_labels).item()\n before_calibration_ece = ece_criterion(logits, one_hot_labels).item()\n logger.info(\n \"Before matrix scaling:\\n\"\n \" Negative log-likelihood: %.3f\\n\"\n \" Expected Calibration Error: %.3f\" % (before_calibration_nll, before_calibration_ece)\n )\n\n # Optimizes the linear transform to minimize NLL\n optimizer = torch.optim.LBFGS([self.w, self.b], lr=0.001, max_iter=200, line_search_fn=\"strong_wolfe\")\n\n def eval():\n optimizer.zero_grad()\n loss = nll_criterion(self.scale_logits(logits), one_hot_labels) + self.regularization_terms()\n loss.backward()\n return loss\n\n optimizer.step(eval)\n\n # Calculate NLL and ECE after matrix scaling\n after_calibration_nll = nll_criterion(self.scale_logits(logits), one_hot_labels).item()\n after_calibration_ece = ece_criterion(self.scale_logits(logits), one_hot_labels).item()\n logger.info(\n \"After matrix scaling:\\n\"\n \" Negative log-likelihood: %.3f\\n\"\n \" Expected Calibration Error: %.3f\" % (after_calibration_nll, after_calibration_ece)\n )\n self.w.requires_grad = False\n self.b.requires_grad = False\n # This should never happen, but if expected calibration error is higher after optimizing matrix, revert.\n if after_calibration_ece > before_calibration_ece:\n logger.warning(\"Expected calibration error higher after matrix scaling, reverting to identity.\")\n with torch.no_grad():\n self.w.data = torch.eye(self.num_classes)\n self.b.data = torch.zeros(self.num_classes)\n return CalibrationResult(\n before_calibration_nll, before_calibration_ece, after_calibration_nll, after_calibration_ece\n )\n\n def regularization_terms(self) -> torch.Tensor:\n \"\"\"Off-Diagonal and Intercept Regularisation (ODIR).\n\n Described in \"Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet\n calibration\"\n https://proceedings.neurips.cc/paper/2019/file/8ca01ea920679a0fe3728441494041b9-Paper.pdf\n \"\"\"\n off_diagonal_entries = torch.masked_select(\n self.w, ~torch.eye(self.num_classes, dtype=bool, device=self.w.device)\n )\n weight_matrix_loss = self.off_diagonal_l2 * torch.linalg.vector_norm(off_diagonal_entries)\n bias_vector_loss = self.mu * torch.linalg.vector_norm(self.b, 2)\n return bias_vector_loss + weight_matrix_loss\n\n def scale_logits(self, logits: torch.Tensor) -> torch.Tensor:\n return torch.matmul(self.w, logits.T).T + self.b\n\n def forward(self, logits: torch.Tensor) -> torch.Tensor:\n \"\"\"Converts logits to probabilities.\"\"\"\n return torch.softmax(self.scale_logits(logits), -1)\n"
},
{
"path": "ludwig/utils/carton_utils.py",
"content": "import asyncio\nimport importlib.util\nimport logging\nimport os\nimport shutil\nimport sys\nimport tempfile\nimport traceback\nfrom typing import Any\n\nimport torch\n\nfrom ludwig.api import LudwigModel\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import NAME\nfrom ludwig.types import ModelConfigDict\nfrom ludwig.utils.fs_utils import open_file\n\nlogger = logging.getLogger(__name__)\n\n\nINFERENCE_MODULE_TEMPLATE = \"\"\"\nfrom typing import Any, Dict, List, Tuple, Union\nimport torch\nfrom ludwig.utils.types import TorchscriptPreprocessingInput\n\nclass GeneratedInferenceModule(torch.nn.Module):\n def __init__(self, inference_module):\n super().__init__()\n self.inference_module = inference_module\n\n def forward(self, inputs: Dict[str, Any]):\n retyped_inputs: Dict[str, TorchscriptPreprocessingInput] = {{}}\n for k, v in inputs.items():\n assert isinstance(v, TorchscriptPreprocessingInput)\n retyped_inputs[k] = v\n\n results = self.inference_module(retyped_inputs)\n return {output_dicts}\n\"\"\"\n\n\ndef _get_output_dicts(config: ModelConfigDict) -> str:\n results = []\n for feature in config[\"output_features\"]:\n name = feature[NAME]\n results.append(f'\"{name}\": results[\"{name}\"][\"predictions\"]')\n return \"{\" + \", \".join(results) + \"}\"\n\n\n@DeveloperAPI\ndef generate_carton_torchscript(model: LudwigModel):\n config = model.config\n inference_module = model.to_torchscript()\n with tempfile.TemporaryDirectory() as tmpdir:\n ts_path = os.path.join(tmpdir, \"generated.py\")\n with open_file(ts_path, \"w\") as f:\n f.write(\n INFERENCE_MODULE_TEMPLATE.format(\n output_dicts=_get_output_dicts(config),\n )\n )\n\n spec = importlib.util.spec_from_file_location(\"generated.ts\", ts_path)\n gen_ts = importlib.util.module_from_spec(spec)\n spec.loader.exec_module(gen_ts)\n\n gen_module = gen_ts.GeneratedInferenceModule(inference_module)\n scripted_module = torch.jit.script(gen_module)\n return scripted_module\n\n\ndef _get_input_spec(model: LudwigModel) -> list[dict[str, Any]]:\n from cartonml import TensorSpec\n\n spec = []\n for feature_name, feature in model.model.input_features.items():\n metadata = model.training_set_metadata[feature_name]\n spec.append(\n TensorSpec(\n name=feature.feature_name, dtype=feature.get_preproc_input_dtype(metadata), shape=(\"batch_size\",)\n )\n )\n return spec\n\n\ndef _get_output_spec(model: LudwigModel) -> list[dict[str, Any]]:\n from cartonml import TensorSpec\n\n spec = []\n for feature_name, feature in model.model.output_features.items():\n metadata = model.training_set_metadata[feature_name]\n spec.append(\n TensorSpec(\n name=feature.feature_name, dtype=feature.get_postproc_output_dtype(metadata), shape=(\"batch_size\",)\n )\n )\n return spec\n\n\n@DeveloperAPI\ndef export_carton(model: LudwigModel, carton_path: str, carton_model_name=\"ludwig_model\"):\n try:\n import cartonml as carton\n except ImportError:\n raise RuntimeError('The \"cartonml-nightly\" package is not installed in your environment.')\n\n # Generate a torchscript model\n model_ts = generate_carton_torchscript(model)\n\n with tempfile.TemporaryDirectory() as tmpdir:\n # Save the model to a temp dir\n input_model_path: str = os.path.join(tmpdir, \"model.pt\")\n torch.jit.save(model_ts, input_model_path)\n\n # carton.pack is an async function so we run it and wait until it's complete\n # See https://pyo3.rs/v0.20.0/ecosystem/async-await#a-note-about-asynciorun for why we wrap it\n # in another function\n async def pack() -> str:\n try:\n return await carton.pack(\n path=input_model_path,\n runner_name=\"torchscript\",\n # Any 2.x.x version is okay\n # TODO: improve this\n required_framework_version=\"=2\",\n model_name=carton_model_name,\n inputs=_get_input_spec(model),\n outputs=_get_output_spec(model),\n )\n except Exception as e:\n exception_message: str = 'An Exception inside \"pack()\" occurred.\\n'\n exception_traceback: str = traceback.format_exc()\n exception_message += f'{type(e).__name__}: \"{str(e)}\". Traceback: \"{exception_traceback}\".'\n sys.stderr.write(exception_message)\n sys.stderr.flush()\n raise ValueError(exception_message) from e # Re-raise error for calling function to handle.\n\n try:\n tmp_out_path: str = asyncio.run(pack())\n # Move it to the output path\n shutil.move(tmp_out_path, carton_path)\n except Exception as e:\n exception_message: str = 'An Exception inside \"export_carton()\" occurred.\\n'\n exception_traceback: str = traceback.format_exc()\n exception_message += f'{type(e).__name__}: \"{str(e)}\". Traceback: \"{exception_traceback}\".'\n sys.stderr.write(exception_message)\n sys.stderr.flush()\n raise SystemExit(exception_message) from e # Make sure error is fatal.\n"
},
{
"path": "ludwig/utils/checkpoint_utils.py",
"content": "\"\"\"Implements similar functionality as tf.train.Checkpoint and tf.train.CheckpointManager.\n\nhttps://gist.github.com/kevinzakka/5d345421f7abefd5dbaf6a77f829e70a.\n\"\"\"\n\nimport errno\nimport logging\nimport os\nimport re\nimport shutil\nimport signal\nimport tempfile\nimport uuid\nfrom abc import ABC, abstractmethod\nfrom collections.abc import Mapping\nfrom glob import glob\nfrom typing import Any, TYPE_CHECKING\n\nimport torch\nfrom torch.optim import Optimizer\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.globals import MODEL_WEIGHTS_FILE_NAME\nfrom ludwig.modules.lr_scheduler import LRScheduler\n\nif TYPE_CHECKING:\n from ludwig.distributed.base import DistributedStrategy\n from ludwig.models.base import BaseModel\n\n\nlogger = logging.getLogger(__name__)\nLATEST = \"latest\"\nBEST = \"best\"\n\n\n@DeveloperAPI\ndef mkdir(s):\n \"\"\"Create a directory if it doesn't already exist.\"\"\"\n if not os.path.exists(s):\n os.makedirs(s)\n\n\n@DeveloperAPI\ndef get_files(d, pattern, sort=True):\n \"\"\"Return a list of files in a given directory.\n\n Args:\n d (str): The path to the directory.\n pattern (str): The wildcard to filter files with.\n sort (bool): Whether to sort the returned list. Assumes filenames contain a number value to sort by (tmp-001).\n \"\"\"\n files = glob(os.path.join(d, pattern))\n files = [f for f in files if os.path.isfile(f)]\n if sort:\n\n def filter_numeric(s):\n return re.sub(\"[^0-9]\", \"\", s)\n\n files.sort(key=lambda x: int(filter_numeric(os.path.basename(x).split(\".\")[0])))\n return files\n\n\n@DeveloperAPI\ndef get_latest_checkpoint_path(directory: str) -> str:\n latest_path = os.path.join(directory, f\"{LATEST}.ckpt\")\n if os.path.exists(latest_path):\n return latest_path\n\n # Legacy codepath for checkpoints saved by global step number\n ckpts = get_files(directory, \"*.ckpt\")\n if ckpts:\n return ckpts[-1]\n\n return None\n\n\n@DeveloperAPI\nclass Checkpoint(ABC):\n \"\"\"Save and restore model and optimizer states.\"\"\"\n\n def __init__(\n self,\n distributed: \"DistributedStrategy\",\n model: \"BaseModel\",\n optimizer: Optimizer | None = None,\n scheduler: LRScheduler | None = None,\n ):\n \"\"\"Constructor.\"\"\"\n self.distributed = distributed\n self.model = model\n self.optimizer = optimizer\n self.scheduler = scheduler\n self.global_step = 0\n\n def prepare(self, directory: str):\n # create checkpoint directory if it doesn't\n # already exist\n mkdir(directory)\n\n @abstractmethod\n def load(self, save_path: str, device: torch.device | None = None) -> bool:\n pass\n\n @abstractmethod\n def get_state_for_inference(self, save_path: str, device: torch.device | None = None) -> Mapping[str, Any]:\n pass\n\n @abstractmethod\n def save(self, save_path: str, global_step: int):\n pass\n\n def _get_global_step(self, state: dict[str, Any], save_path: str) -> int:\n global_step = state.get(\"global_step\")\n if global_step is None:\n # Legacy step detection for older checkpoint format which encoded the\n # step number in the checkpoint filename.\n return int(os.path.basename(save_path).split(\".\")[0])\n return global_step\n\n\n@DeveloperAPI\nclass MultiNodeCheckpoint(Checkpoint):\n def prepare(self, directory: str):\n if self.is_local_rank_0():\n super().prepare(directory)\n self.distributed.barrier()\n\n def load(self, save_path: str, device: torch.device | None = None) -> bool:\n \"\"\"Load state from a saved checkpoint.\n\n Args:\n save_path (str): The filepath to the saved checkpoint.\n device (torch.device): The device on which to\n load the state.\n\n Returns:\n True if the checkpoint was sucessfully loaded, False if the checkpoint file\n could not be found.\n \"\"\"\n try:\n state = torch.load(save_path, map_location=device)\n try:\n self.global_step = self._get_global_step(state, save_path)\n _, unexpected_keys = self.model.load_state_dict(state[MODEL_WEIGHTS_FILE_NAME], strict=False)\n assert unexpected_keys == [], f\"Unexpected keys found in state dict: {unexpected_keys}\"\n if self.optimizer is not None:\n self.optimizer.load_state_dict(state[\"optim_state\"])\n if self.scheduler is not None and \"scheduler_state\" in state:\n self.scheduler.load_state_dict(state[\"scheduler_state\"])\n logger.info(f\"Successfully loaded model weights from {save_path}.\")\n return True\n except Exception as e:\n # there was an issue loading the state which means\n # either the model definition and saved weights\n # do not agree or they were not saved in the first\n # place.\n # since this is a severe issue, we raise an error\n # rather than allowing the program to proceed.\n raise e\n except FileNotFoundError as e:\n logger.error(e)\n return False\n\n def get_state_for_inference(self, save_path: str, device: torch.device | None = None) -> Mapping[str, Any]:\n state = torch.load(save_path, map_location=device)\n return state[MODEL_WEIGHTS_FILE_NAME]\n\n def save(self, save_path: str, global_step: int):\n \"\"\"Save a state to disk.\n\n Modified from brentyi/fannypack.\n\n Args:\n save_path (str): The name of the checkpoint to save.\n global_step (int): The iteration number which will be used\n to name the checkpoint.\n \"\"\"\n if self.is_local_rank_0():\n state = {\n \"global_step\": global_step,\n MODEL_WEIGHTS_FILE_NAME: self.get_model_state_dict(),\n }\n if self.optimizer is not None:\n state[\"optim_state\"] = self.optimizer.state_dict()\n if self.scheduler is not None:\n state[\"scheduler_state\"] = self.scheduler.state_dict()\n\n # ignore ctrl+c while saving\n try:\n orig_handler = signal.getsignal(signal.SIGINT)\n signal.signal(signal.SIGINT, lambda _sig, _frame: None)\n except ValueError:\n # signal throws a ValueError if we're not in the main thread\n orig_handler = None\n\n try:\n # atomic save\n with tempfile.TemporaryDirectory() as tmpdir:\n # Save to a temporary directory outside of the checkpoint dir so\n # async processes do not try and copy a partially-written checkpoint.\n # See Ray Tune and MLFlow for examples of background processes that\n # are affected by this.\n tmp_path = os.path.join(tmpdir, \"temp.ckpt\")\n torch.save(state, tmp_path)\n\n self.safe_move_file(tmp_path, save_path)\n logger.debug(f\"Saved checkpoint at {save_path}.\")\n finally:\n # restore SIGINT handler\n if orig_handler is not None:\n signal.signal(signal.SIGINT, orig_handler)\n self.distributed.barrier()\n\n def get_model_state_dict(self) -> dict[str, Any]:\n state = self.model.state_dict()\n\n # Remove frozen parameter weights from state_dict for adapters and pretrained models\n for n, p in self.model.named_parameters():\n if n in state and not p.requires_grad:\n del state[n]\n\n return state\n\n def is_local_rank_0(self) -> bool:\n return self.distributed.local_rank() == 0\n\n def safe_move_file(self, src: str, dst: str):\n \"\"\"Move a file from one directory to another, possibly across filesystems.\n\n This implementation specifically addresses the following issue with distributed training:\n\n 1. The `save_path` is a directory local to the node, in which case every node should write\n checkpoints separately.\n 2. The `save_path` is a remote / global filesystem like NFS, in which case only the coordinator\n should write checkpoints.\n \"\"\"\n try:\n os.replace(src, dst)\n except OSError as err:\n if err.errno == errno.EXDEV:\n # Tried to move to an external filesystem. This means we should only run this on the coordinator\n if not self.distributed.is_coordinator():\n logger.info(\n f\"Skipping writing checkpoint from rank {self.distributed.rank()} as it is not the coordinator \"\n f\"and the destination filesystem is remote.\"\n )\n return\n\n # Generate a unique ID, and copy `` to the target directory with a temporary name `..tmp`.\n # Because we're copying across a filesystem boundary, this initial copy may not be atomic. We insert a\n # random UUID so if different processes are copying into ``, they don't overlap in their tmp\n # copies.\n copy_id = uuid.uuid4()\n tmp_dst = f\"{dst}.{copy_id}.tmp\"\n shutil.copyfile(src, tmp_dst)\n\n # Atomic replace file onto the new name, and clean up original source file.\n os.replace(tmp_dst, dst)\n os.unlink(src)\n else:\n raise\n\n\n@DeveloperAPI\nclass CheckpointManager:\n \"\"\"A model and optimizer checkpoint manager.\"\"\"\n\n def __init__(self, checkpoint: Checkpoint, directory: str, device: torch.device):\n \"\"\"Constructor.\n\n Args:\n checkpoint (Checkpoint): An instance of `Checkpoint`.\n directory (str): The directory in which checkpoints will be saved.\n device (torch.device): The computing device on which to restore\n checkpoints.\n \"\"\"\n self.checkpoint = checkpoint\n self.directory = directory\n self.device = device\n self.latest_checkpoint = None\n self.checkpoint.prepare(self.directory)\n\n def restore_or_initialize(self) -> int:\n \"\"\"Restore items in checkpoint from the latest checkpoint file.\n\n Returns:\n The global iteration step. This is parsed from the latest\n checkpoint file if one is found, else 0 is returned.\n \"\"\"\n last_ckpt = get_latest_checkpoint_path(self.directory)\n if last_ckpt:\n status = self.checkpoint.load(last_ckpt, self.device)\n if not status:\n logger.warning(\"Could not restore latest checkpoint file.\")\n return 0\n self.latest_checkpoint = last_ckpt\n return self.checkpoint.global_step\n return 0\n\n def save(self, global_step: int, tag: str = LATEST):\n \"\"\"Create a new checkpoint.\n\n Args:\n global_step (int): The iteration number which will be used\n to name the checkpoint.\n \"\"\"\n save_path = os.path.join(self.directory, f\"{tag}.ckpt\")\n self.checkpoint.save(save_path, global_step)\n self.latest_checkpoint = save_path\n\n def save_best(self, global_step: int):\n self.save(global_step, BEST)\n\n def load(self, tag: str = LATEST):\n \"\"\"Load a checkpoint.\n\n Args:\n tag (str): The tag of the checkpoint to load.\n \"\"\"\n save_path = os.path.join(self.directory, f\"{tag}.ckpt\")\n self.checkpoint.load(save_path, self.device)\n\n def get_best_checkpoint_state_for_inference(self, device: torch.device) -> tuple[Mapping[str, Any], None]:\n save_path = os.path.join(self.directory, f\"{BEST}.ckpt\")\n try:\n return self.checkpoint.get_state_for_inference(save_path, device)\n except Exception:\n # This exception may be hit if the best checkpoint does not exist. This can happen if the model runs into\n # NaN loss because of NaN or inf values in the weights before the first checkpoint is saved. In this case,\n logger.error(f\"Could not load best checkpoint state from {save_path}. Best checkpoint may not exist.\")\n return None\n\n def close(self):\n pass\n\n @staticmethod\n def load_latest_checkpoint(checkpoint: Checkpoint, directory: str, device: torch.device):\n last_ckpt = get_latest_checkpoint_path(directory)\n if last_ckpt:\n checkpoint.load(last_ckpt, device)\n else:\n raise FileNotFoundError(f\"No checkpoints found in {directory}.\")\n"
},
{
"path": "ludwig/utils/config_utils.py",
"content": "from typing import Any\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import (\n DECODER,\n ENCODER,\n IMAGE,\n INPUT_FEATURES,\n MODEL_ECD,\n MODEL_LLM,\n MODEL_TYPE,\n PREPROCESSING,\n SEQUENCE,\n TEXT,\n TIMESERIES,\n TYPE,\n)\nfrom ludwig.features.feature_registries import get_input_type_registry\nfrom ludwig.schema.model_config import ModelConfig\nfrom ludwig.types import FeatureConfigDict, FeatureTypeDefaultsDict, PreprocessingConfigDict\n\n\n@DeveloperAPI\ndef get_feature_type_parameter_values_from_section(\n config: ModelConfig, features_section: str, feature_type: str, parameter_name: str\n) -> set:\n \"\"\"Returns the set of all parameter values used for the given features_section, feature_type, and\n parameter_name.\"\"\"\n parameter_values = set()\n for feature in config[features_section]:\n if feature[TYPE] == feature_type:\n if parameter_name in feature:\n parameter_values.add(feature[parameter_name])\n elif parameter_name in feature[ENCODER]:\n parameter_values.add(feature[ENCODER][parameter_name])\n elif parameter_name in feature[DECODER]:\n parameter_values.add(feature[DECODER][parameter_name])\n return parameter_values\n\n\n@DeveloperAPI\ndef get_defaults_section_for_feature_type(\n feature_type: str,\n config_defaults: FeatureTypeDefaultsDict,\n config_defaults_section: str,\n) -> FeatureConfigDict:\n \"\"\"Returns a dictionary of all default parameter values specified in the global defaults section for the\n config_defaults_section of the feature_type.\"\"\"\n\n if feature_type not in config_defaults:\n return {}\n\n if config_defaults_section not in config_defaults[feature_type]:\n return {}\n\n return config_defaults[feature_type][config_defaults_section]\n\n\ndef _to_dict(obj) -> dict:\n \"\"\"Convert a config object or dict to a plain dict.\"\"\"\n if isinstance(obj, dict):\n return obj\n return obj.to_dict()\n\n\ndef get_preprocessing_params(config_obj: ModelConfig) -> PreprocessingConfigDict:\n \"\"\"Returns a new dictionary that merges preprocessing section of config with type-specific preprocessing\n parameters from config defaults.\"\"\"\n preprocessing_params = {}\n preprocessing_params.update(_to_dict(config_obj.preprocessing))\n for feat_type in get_input_type_registry().keys():\n if hasattr(config_obj.defaults, feat_type):\n feat_defaults = getattr(config_obj.defaults, feat_type)\n preprocessing = (\n feat_defaults.preprocessing\n if not isinstance(feat_defaults, dict)\n else feat_defaults.get(\"preprocessing\", {})\n )\n preprocessing_params[feat_type] = _to_dict(preprocessing)\n return preprocessing_params\n\n\n@DeveloperAPI\ndef merge_config_preprocessing_with_feature_specific_defaults(\n config_preprocessing: PreprocessingConfigDict, config_defaults: FeatureTypeDefaultsDict\n) -> PreprocessingConfigDict:\n \"\"\"Returns a new dictionary that merges preprocessing section of config with type-specific preprocessing\n parameters from config defaults.\"\"\"\n preprocessing_params = {}\n preprocessing_params.update(config_preprocessing)\n for feature_type in config_defaults:\n preprocessing_params[feature_type] = config_defaults[feature_type].get(PREPROCESSING, {})\n return preprocessing_params\n\n\ndef has_trainable_encoder(config: ModelConfig) -> bool:\n for feature in config.input_features.to_list():\n encoder = feature.get(\"encoder\", {})\n if encoder.get(\"trainable\", False):\n # TODO(travis): we assume here that False is always the default, which may not be true. We should dervice\n # this from the schema.\n return True\n\n return False\n\n\ndef has_unstructured_input_feature(config: ModelConfig) -> bool:\n for feature in config.input_features.to_list():\n if feature.get(\"type\", None) in {TEXT, IMAGE, SEQUENCE, TIMESERIES}:\n return True\n return False\n\n\ndef has_pretrained_encoder(config: ModelConfig) -> bool:\n for feature in config.input_features:\n if feature.encoder.is_pretrained():\n return True\n return False\n\n\ndef config_uses_llm(config: dict[str, Any] | ModelConfig) -> bool:\n \"\"\"Determine if a config uses an LLM.\n\n Args:\n config: Ludwig config object or dictionary\n\n Returns:\n True if the model type is LLM or if the model uses and LLM encoder, otherwise False.\n \"\"\"\n uses_llm = False\n\n # For a valid config, model_type LLM is automatically True\n # ECD models need to be checked for at least one LLM text encoder\n if isinstance(config, ModelConfig):\n if config.model_type == MODEL_LLM:\n uses_llm = True\n else:\n for feature in config.input_features:\n if feature.encoder and feature.encoder.type == MODEL_LLM:\n uses_llm = True\n break\n elif isinstance(config, dict) and config:\n if config.get(MODEL_TYPE, MODEL_ECD) == MODEL_LLM:\n uses_llm = True\n elif INPUT_FEATURES in config:\n for feature in config.get(INPUT_FEATURES, []):\n if feature.get(ENCODER, {}).get(TYPE) == MODEL_LLM:\n uses_llm = True\n break\n else:\n raise ValueError(\n \"Invalid config cannot be checked for LLM usage because it has no input features.\" f\"Config: {config}\"\n )\n else:\n raise ValueError(f\"Invalid config cannot be checked for LLM usage. Config: {config}\")\n\n return uses_llm\n\n\ndef get_quantization(config: dict[str, Any] | ModelConfig) -> list[int | None]:\n \"\"\"Get the quantization specified in a config at any level.\n\n Args:\n config: Ludwig config object or dictionary\n\n Returns:\n For LLM models, the value of quantization.bits or None if it is not specified.\n For ECD models, the list of values of quantization.bits for each encoder. If the encoder does not\n support quantization or no quantization config is specified, the list entry is None.\n \"\"\"\n if isinstance(config, ModelConfig):\n if config.model_type == MODEL_LLM:\n return [config.quantization.bits] if config.quantization else [None]\n else:\n quantization_bits = []\n for feature in config.input_features:\n try:\n quantization = feature.encoder.quantization.bits\n except AttributeError:\n quantization = None\n quantization_bits.append(quantization)\n return quantization_bits\n elif isinstance(config, dict) and config:\n if config.get(MODEL_TYPE, MODEL_ECD) == MODEL_LLM:\n return [config.get(\"quantization\", {}).get(\"bits\")]\n elif INPUT_FEATURES in config:\n quantization_bits = []\n for feature in config.get(INPUT_FEATURES, []):\n quantization_bits.append(feature.get(ENCODER, {}).get(\"quantization\", {}).get(\"bits\"))\n return quantization_bits\n else:\n raise ValueError(\n \"Invalid config cannot be checked for quantization because it has no input features.\"\n f\"Config: {config}\"\n )\n else:\n raise ValueError(f\"Invalid config cannot be checked for quantization. Config: {config}\")\n"
},
{
"path": "ludwig/utils/data_utils.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport base64\nimport collections.abc\nimport contextlib\nimport csv\nimport dataclasses\nimport functools\nimport hashlib\nimport json\nimport logging\nimport os\nimport os.path\nimport pickle\nimport random\nimport re\nimport tempfile\nimport threading\nfrom itertools import islice\nfrom typing import Any\n\nimport numpy as np\nimport pandas as pd\nimport pyarrow as pa\nimport yaml\nfrom fsspec.config import conf, set_conf_files\nfrom pandas.errors import ParserError\nfrom sklearn.model_selection import KFold\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import PREPROCESSING, SPLIT\nfrom ludwig.data.cache.types import CacheableDataset\nfrom ludwig.globals import MODEL_HYPERPARAMETERS_FILE_NAME, MODEL_WEIGHTS_FILE_NAME, TRAIN_SET_METADATA_FILE_NAME\nfrom ludwig.utils.dataframe_utils import from_numpy_dataset, is_dask_lib, to_numpy_dataset\nfrom ludwig.utils.fs_utils import download_h5, has_remote_protocol, open_file, upload_h5\nfrom ludwig.utils.math_utils import cumsum\nfrom ludwig.utils.misc_utils import get_from_registry\nfrom ludwig.utils.types import DataFrame\n\ntry:\n import dask\n import dask.dataframe as dd\n\n DASK_DF_FORMATS = {dd.DataFrame}\nexcept ImportError:\n DASK_DF_FORMATS = set()\n dd = None\n\nlogger = logging.getLogger(__name__)\n\nDATASET_SPLIT_URL = \"dataset_{}_fp\"\nDATA_PROCESSED_CACHE_DIR = \"data_processed_cache_dir\"\nDATA_TRAIN_HDF5_FP = \"data_train_hdf5_fp\"\n\nDATA_TRAIN_PARQUET_FP = \"data_train_parquet_fp\"\nDATA_VALIDATION_PARQUET_FP = \"data_validation_parquet_fp\"\nDATA_TEST_PARQUET_FP = \"data_test_parquet_fp\"\n\nHDF5_COLUMNS_KEY = \"columns\"\nDICT_FORMATS = {\"dict\", \"dictionary\", dict}\nDATAFRAME_FORMATS = {\"dataframe\", \"df\", pd.DataFrame} | DASK_DF_FORMATS\nCSV_FORMATS = {\"csv\"}\nTSV_FORMATS = {\"tsv\"}\nJSON_FORMATS = {\"json\"}\nJSONL_FORMATS = {\"jsonl\"}\nEXCEL_FORMATS = {\"excel\"}\nPARQUET_FORMATS = {\"parquet\"}\nPICKLE_FORMATS = {\"pickle\"}\nFEATHER_FORMATS = {\"feather\"}\nFWF_FORMATS = {\"fwf\"}\nHTML_FORMATS = {\"html\"}\nORC_FORMATS = {\"orc\"}\nSAS_FORMATS = {\"sas\"}\nSPSS_FORMATS = {\"spss\"}\nSTATA_FORMATS = {\"stata\"}\nHDF5_FORMATS = {\"hdf5\", \"h5\"}\nCACHEABLE_FORMATS = set.union(\n *(\n CSV_FORMATS,\n TSV_FORMATS,\n JSON_FORMATS,\n JSONL_FORMATS,\n EXCEL_FORMATS,\n PARQUET_FORMATS,\n PICKLE_FORMATS,\n FEATHER_FORMATS,\n FWF_FORMATS,\n HTML_FORMATS,\n ORC_FORMATS,\n SAS_FORMATS,\n SPSS_FORMATS,\n STATA_FORMATS,\n DATAFRAME_FORMATS,\n )\n)\n\nPANDAS_DF = pd\n\n\n# Lock over the entire interpreter as we can only have one set\n# of credentials scoped to the interpreter at once.\nGLOBAL_CRED_LOCK = threading.Lock()\n\n\n@DeveloperAPI\ndef get_parquet_filename(n: int):\n \"\"\"Left pads the partition number with zeros to preserve order in downstream reads.\n\n Downstream reads use the filename to determine the lexical order of the partitions.\n \"\"\"\n return f\"part.{str(n).zfill(8)}.parquet\"\n\n\n@DeveloperAPI\ndef get_split_path(dataset_fp):\n return os.path.splitext(dataset_fp)[0] + \".split.parquet\"\n\n\n@DeveloperAPI\ndef get_abs_path(src_path, file_path):\n if has_remote_protocol(file_path):\n return file_path\n elif src_path is not None:\n return os.path.join(src_path, file_path)\n else:\n return file_path\n\n\n@DeveloperAPI\ndef load_csv(data_fp):\n with open_file(data_fp, \"rb\") as f:\n data = list(csv.reader(f))\n return data\n\n\n# Decorator used to encourage Dask on Ray to spread out data loading across workers\n@DeveloperAPI\ndef spread(fn):\n def wrapped_fn(*args, **kwargs):\n if dd is None or not hasattr(dask, \"annotate\"):\n return fn(*args, **kwargs)\n\n with dask.annotate(ray_remote_args=dict(scheduling_strategy=\"SPREAD\")):\n return fn(*args, **kwargs)\n\n return wrapped_fn\n\n\ndef read_nrows_via_chunksize(fp, read_fn, **kwargs):\n chunksize = kwargs.pop(\"nrows\", None)\n ret = read_fn(fp, chunksize=chunksize, **kwargs)\n\n if isinstance(ret, collections.abc.Iterator):\n return next(ret)\n\n return ret\n\n\n@DeveloperAPI\n@spread\ndef read_xsv(data_fp, df_lib=PANDAS_DF, separator=\",\", header=0, nrows=None, skiprows=None, dtype=object, **kwargs):\n \"\"\"Helper method to read a csv file. Wraps around pd.read_csv to handle some exceptions. Can extend to cover\n cases as necessary.\n\n :param data_fp: path to the xsv file\n :param df_lib: DataFrame library used to read in the CSV\n :param separator: defaults separator to use for splitting\n :param header: header argument for pandas to read the csv\n :param nrows: number of rows to read from the csv, None means all\n :param skiprows: number of rows to skip from the csv, None means no skips\n :param dtype: dtype to use for columns. Defaults to object to disable type inference.\n :return: Pandas dataframe with the data\n \"\"\"\n with open_file(data_fp, \"r\", encoding=\"utf8\") as csvfile:\n try:\n dialect = csv.Sniffer().sniff(csvfile.read(1024 * 100), delimiters=[\",\", \"\\t\", \"|\"])\n separator = dialect.delimiter\n except csv.Error:\n # Could not conclude the delimiter, defaulting to user provided\n pass\n\n # NOTE: by default we read all XSV columns in as dtype=object, bypassing all type inference. This is to avoid silent\n # issues related to incorrect type inference (e.g. NaNs in bool columns). Convert data to correct types after\n # reading in.\n kwargs = dict(sep=separator, header=header, skiprows=skiprows, dtype=dtype, **kwargs)\n\n if nrows is not None:\n kwargs[\"nrows\"] = nrows\n\n try:\n df = df_lib.read_csv(data_fp, **kwargs)\n except ParserError:\n logger.warning(\"Failed to parse the CSV with pandas default way, trying \\\\ as escape character.\")\n df = df_lib.read_csv(data_fp, escapechar=\"\\\\\", **kwargs)\n\n return df\n\n\nread_csv = functools.partial(read_xsv, separator=\",\")\nread_tsv = functools.partial(read_xsv, separator=\"\\t\")\n\n\n@DeveloperAPI\n@spread\ndef read_json(data_fp, df_lib, normalize=False, **kwargs):\n # Not supported unless lines=True\n kwargs.pop(\"nrows\", None)\n\n if normalize:\n return df_lib.json_normalize(load_json(data_fp))\n else:\n return df_lib.read_json(data_fp, **kwargs)\n\n\n@DeveloperAPI\n@spread\ndef read_jsonl(data_fp, df_lib, **kwargs):\n return df_lib.read_json(data_fp, lines=True, **kwargs)\n\n\n@DeveloperAPI\n@spread\ndef read_excel(data_fp, df_lib, **kwargs):\n fp_split = os.path.splitext(data_fp)\n if fp_split[1] == \".xls\":\n excel_engine = \"xlrd\"\n else:\n excel_engine = \"openpyxl\"\n\n # https://github.com/dask/dask/issues/9055\n if is_dask_lib(df_lib):\n logger.warning(\"Falling back to pd.read_excel() since dask backend does not support it\")\n return dd.from_pandas(pd.read_excel(data_fp, engine=excel_engine, **kwargs), npartitions=1)\n return df_lib.read_excel(data_fp, engine=excel_engine, **kwargs)\n\n\n@DeveloperAPI\n@spread\ndef read_parquet(data_fp, df_lib, nrows=None, **kwargs):\n if nrows is not None:\n import pyarrow.parquet as pq\n\n from ludwig.utils.fs_utils import get_fs_and_path\n\n fs, _ = get_fs_and_path(data_fp)\n dataset = pq.ParquetDataset(data_fp, filesystem=fs).fragments[0]\n\n preview = dataset.head(nrows).to_pandas()\n\n if is_dask_lib(df_lib):\n return df_lib.from_pandas(preview, npartitions=1)\n return preview\n\n return df_lib.read_parquet(data_fp, **kwargs)\n\n\n@DeveloperAPI\n@spread\ndef read_pickle(data_fp, df_lib, **kwargs):\n # Chunking is not supported for pickle files:\n kwargs.pop(\"nrows\", None)\n\n # https://github.com/dask/dask/issues/9055\n if is_dask_lib(df_lib):\n logger.warning(\"Falling back to pd.read_pickle() since dask backend does not support it\")\n return dd.from_pandas(pd.read_pickle(data_fp), npartitions=1)\n return df_lib.read_pickle(data_fp)\n\n\n@DeveloperAPI\n@spread\ndef read_fwf(data_fp, df_lib, **kwargs):\n return df_lib.read_fwf(data_fp, **kwargs)\n\n\n@DeveloperAPI\n@spread\ndef read_feather(data_fp, df_lib, **kwargs):\n # Chunking is not supported for feather files:\n kwargs.pop(\"nrows\", None)\n\n # https://github.com/dask/dask/issues/9055\n if is_dask_lib(df_lib):\n logger.warning(\"Falling back to pd.read_feather() since dask backend does not support it\")\n return dd.from_pandas(pd.read_feather(data_fp), npartitions=1)\n return df_lib.read_feather(data_fp)\n\n\n@DeveloperAPI\n@spread\ndef read_html(data_fp, df_lib, **kwargs):\n # Chunking is not supported for html files:\n kwargs.pop(\"nrows\", None)\n\n # Wrap literal HTML strings in StringIO to avoid pandas FutureWarning\n from io import StringIO\n\n if isinstance(data_fp, str) and not os.path.isfile(data_fp):\n data_fp = StringIO(data_fp)\n\n # https://github.com/dask/dask/issues/9055\n if is_dask_lib(df_lib):\n logger.warning(\"Falling back to pd.read_html() since dask backend does not support it\")\n return dd.from_pandas(pd.read_html(data_fp)[0], npartitions=1)\n return df_lib.read_html(data_fp)[0]\n\n\n@DeveloperAPI\n@spread\ndef read_orc(data_fp, df_lib, **kwargs):\n # Chunking is not supported for orc files:\n kwargs.pop(\"nrows\", None)\n\n return df_lib.read_orc(data_fp, **kwargs)\n\n\n@DeveloperAPI\n@spread\ndef read_sas(data_fp, df_lib, **kwargs):\n # https://github.com/dask/dask/issues/9055\n if is_dask_lib(df_lib):\n logger.warning(\"Falling back to pd.read_sas() since dask backend does not support it\")\n return dd.from_pandas(read_nrows_via_chunksize(data_fp, df_lib.read_sas, **kwargs), npartitions=1)\n return read_nrows_via_chunksize(data_fp, df_lib.read_sas, **kwargs)\n\n\n@DeveloperAPI\n@spread\ndef read_spss(data_fp, df_lib, **kwargs):\n # Chunking is not supported for spss files:\n kwargs.pop(\"nrows\", None)\n\n # https://github.com/dask/dask/issues/9055\n if is_dask_lib(df_lib):\n logger.warning(\"Falling back to pd.read_spss() since dask backend does not support it\")\n return dd.from_pandas(pd.read_spss(data_fp), npartitions=1)\n return df_lib.read_spss(data_fp)\n\n\n@DeveloperAPI\n@spread\ndef read_stata(data_fp, df_lib, **kwargs):\n # https://github.com/dask/dask/issues/9055\n if is_dask_lib(df_lib):\n logger.warning(\"Falling back to pd.read_stata() since dask backend does not support it\")\n return dd.from_pandas(read_nrows_via_chunksize(data_fp, df_lib.read_stata, **kwargs), npartitions=1)\n return read_nrows_via_chunksize(data_fp, df_lib.read_stata, **kwargs)\n\n\n@DeveloperAPI\n@spread\ndef read_hdf5(data_fp, **_kwargs):\n return load_hdf5(data_fp, clean_cols=True)\n\n\n@DeveloperAPI\n@spread\ndef read_buffer(buf, fname):\n \"\"\"Reads data in from a binary buffer by first writing the data to a temporary file, and then processes it\n based on its format (hdf5, csv, tsv etc).\n\n Useful if object is a binary buffer coming from streaming data.\n \"\"\"\n data_format = figure_data_format_dataset(fname)\n reader_fn = data_reader_registry[data_format]\n with tempfile.TemporaryDirectory() as tmpdir:\n temp_name = os.path.join(tmpdir, \"dataset\")\n with open(temp_name, \"wb\") as f:\n f.write(buf.read())\n return reader_fn(temp_name, pd)\n\n\n@DeveloperAPI\n@spread\ndef read_fname(fname, data_format=None, df_lib=pd, **kwargs):\n \"\"\"This function reads data from fname using the df_lib data processing library (defaults to pandas).\n\n Useful if you don't know the file type extension in advance.\n \"\"\"\n data_format = data_format or figure_data_format_dataset(fname)\n reader_fn = data_reader_registry[data_format]\n return reader_fn(fname, df_lib, **kwargs)\n\n\n@DeveloperAPI\ndef save_csv(data_fp, data):\n with open_file(data_fp, \"w\", encoding=\"utf-8\") as csv_file:\n writer = csv.writer(csv_file)\n for row in data:\n if not isinstance(row, collections.abc.Iterable) or isinstance(row, str):\n row = [row]\n writer.writerow(row)\n\n\n@DeveloperAPI\ndef csv_contains_column(data_fp, column_name):\n return column_name in read_csv(data_fp, nrows=0) # only loads header\n\n\n@DeveloperAPI\ndef load_yaml(yaml_fp):\n with open_file(yaml_fp, \"r\") as f:\n return yaml.safe_load(f)\n\n\n@DeveloperAPI\ndef load_config_from_str(config):\n \"\"\"Load the config as either a serialized string or a path to a YAML file.\"\"\"\n config = yaml.safe_load(config)\n if isinstance(config, str):\n # Assume the caller provided a path name\n with open(config, encoding=\"utf-8\") as f:\n config = yaml.safe_load(f)\n return config\n\n\n@DeveloperAPI\ndef load_json(data_fp):\n with open_file(data_fp, \"r\") as input_file:\n data = json.load(input_file)\n return data\n\n\n@DeveloperAPI\ndef save_json(data_fp, data, sort_keys=True, indent=4):\n with open_file(data_fp, \"w\") as output_file:\n json.dump(data, output_file, cls=NumpyEncoder, sort_keys=sort_keys, indent=indent)\n\n\n@DeveloperAPI\ndef hash_dict(d: dict, max_length: int | None = 6) -> bytes:\n \"\"\"Function that maps a dictionary into a unique hash.\n\n Known limitation: All values and keys of the dict must have an ordering. If not, there's no guarantee to obtain the\n same hash. For instance, values that are sets will potentially lead to different hashed when run on different\n machines or in different python sessions. Replacing them with sorted lists is suggested.\n \"\"\"\n s = json.dumps(d, cls=NumpyEncoder, sort_keys=True, ensure_ascii=True)\n h = hashlib.md5(s.encode())\n d = h.digest()\n b = base64.b64encode(d, altchars=b\"__\")\n return b[:max_length]\n\n\n@DeveloperAPI\ndef to_json_dict(d):\n \"\"\"Converts Python dict to pure JSON ready format.\"\"\"\n return json.loads(json.dumps(d, cls=NumpyEncoder))\n\n\n@DeveloperAPI\ndef chunk_dict(data, chunk_size=100):\n \"\"\"Split large dictionary into chunks.\n\n Source: https://stackoverflow.com/a/22878842\n \"\"\"\n it = iter(data)\n for _ in range(0, len(data), chunk_size):\n yield {k: data[k] for k in islice(it, chunk_size)}\n\n\n@DeveloperAPI\ndef flatten_dict(d, parent_key=\"\", sep=\".\"):\n \"\"\"Based on https://www.geeksforgeeks.org/python-convert-nested-dictionary-into-flattened-dictionary/\"\"\"\n items = []\n for k, v in d.items():\n new_key = parent_key + sep + k if parent_key else k\n\n if isinstance(v, collections.abc.MutableMapping):\n items.extend(flatten_dict(v, new_key, sep=sep).items())\n elif isinstance(v, list):\n list_mapping = {str(i): item for i, item in enumerate(v)}\n items.extend(flatten_dict(list_mapping, new_key, sep=sep).items())\n else:\n items.append((new_key, v))\n return dict(items)\n\n\n@DeveloperAPI\ndef save_hdf5(data_fp, data):\n numpy_dataset = to_numpy_dataset(data)\n with upload_h5(data_fp) as h5_file:\n h5_file.create_dataset(HDF5_COLUMNS_KEY, data=np.array(data.columns.values, dtype=\"S\"))\n for column in data.columns:\n h5_file.create_dataset(column, data=numpy_dataset[column])\n\n\n@DeveloperAPI\ndef load_hdf5(data_fp, clean_cols: bool = False):\n with download_h5(data_fp) as hdf5_data:\n columns = [s.decode(\"utf-8\") for s in hdf5_data[HDF5_COLUMNS_KEY][()].tolist()]\n\n numpy_dataset = {}\n for column in columns:\n # Column names from training hdf5 will be in the form 'Survived_a2fv4'\n np_col = column.rsplit(\"_\", 1)[0] if clean_cols else column\n numpy_dataset[np_col] = hdf5_data[column][()]\n\n return from_numpy_dataset(numpy_dataset)\n\n\n@DeveloperAPI\ndef load_object(object_fp):\n with open_file(object_fp, \"rb\") as f:\n return pickle.load(f)\n\n\n@DeveloperAPI\ndef save_object(object_fp, obj):\n with open_file(object_fp, \"wb\") as f:\n pickle.dump(obj, f)\n\n\n@DeveloperAPI\ndef load_array(data_fp, dtype=float):\n list_num = []\n with open_file(data_fp, \"r\") as input_file:\n for x in input_file:\n list_num.append(dtype(x.strip()))\n return np.array(list_num)\n\n\n@DeveloperAPI\ndef load_matrix(data_fp, dtype=float):\n list_num = []\n with open_file(data_fp, \"r\") as input_file:\n for row in input_file:\n list_num.append([dtype(elem) for elem in row.strip().split()])\n return np.squeeze(np.array(list_num))\n\n\n@DeveloperAPI\ndef save_array(data_fp, array):\n with open_file(data_fp, \"w\") as output_file:\n for x in np.nditer(array):\n output_file.write(str(x) + \"\\n\")\n\n\n# TODO(shreya): Confirm types of args\n@DeveloperAPI\ndef load_pretrained_embeddings(embeddings_path: str, vocab: list[str]) -> np.ndarray:\n \"\"\"Create an embedding matrix of all words in vocab.\"\"\"\n embeddings, embeddings_size = load_glove(embeddings_path, return_embedding_size=True)\n\n # calculate an average embedding, to use for initializing missing words\n avg_embedding = [embeddings[w] for w in vocab if w in embeddings]\n avg_embedding = sum(avg_embedding) / len(avg_embedding)\n\n # create the embedding matrix\n embeddings_vectors = []\n for word in vocab:\n if word in embeddings:\n embeddings_vectors.append(embeddings[word])\n else:\n embeddings_vectors.append(avg_embedding + np.random.uniform(-0.01, 0.01, embeddings_size))\n embeddings_matrix = np.stack(embeddings_vectors)\n\n # let's help the garbage collector free some memory\n embeddings = None\n\n return embeddings_matrix\n\n\n@DeveloperAPI\n@functools.lru_cache(1)\ndef load_glove(file_path: str, return_embedding_size: bool = False) -> dict[str, np.ndarray]:\n \"\"\"Loads Glove embeddings for each word.\n\n Returns:\n Mapping between word and numpy array of size embedding_size as set by\n first line of file.\n \"\"\"\n logger.info(f\" Loading Glove format file {file_path}\")\n embeddings = {}\n embedding_size = 0\n\n # collect embeddings size assuming the first line is correct\n with open_file(file_path, \"r\", encoding=\"utf-8\") as f:\n found_line = False\n while not found_line:\n line = f.readline()\n if line:\n embedding_size = len(line.split()) - 1\n found_line = True\n\n # collect embeddings\n with open_file(file_path, \"r\", encoding=\"utf-8\") as f:\n for line_number, line in enumerate(f):\n if line:\n try:\n split = line.split()\n if len(split) != embedding_size + 1:\n raise ValueError(\n f\"Line {line_number} is of length {len(split)}, \"\n f\"while expected length is {embedding_size + 1}.\"\n )\n word = split[0]\n embedding = np.array([float(val) for val in split[-embedding_size:]])\n embeddings[word] = embedding\n except ValueError:\n logger.warning(f\"Line {line_number} in the GloVe file {file_path} is malformed, skipping it\")\n logger.info(f\" {len(embeddings)} embeddings loaded\")\n\n if return_embedding_size:\n return embeddings, embedding_size\n return embeddings\n\n\n@DeveloperAPI\ndef split_data(split: float, data: list) -> tuple[list, list]:\n split_length = int(round(split * len(data)))\n random.shuffle(data)\n return data[:split_length], data[split_length:]\n\n\n@DeveloperAPI\ndef split_by_slices(slices: list[Any], n: int, probabilities: list[float]) -> list[Any]:\n splits = []\n indices = cumsum([int(x * n) for x in probabilities])\n start = 0\n for end in indices:\n splits.append(slices[start:end])\n start = end\n return splits\n\n\n@DeveloperAPI\ndef shuffle_unison_inplace(list_of_lists, random_state=None):\n if list_of_lists:\n assert all(len(single_list) == len(list_of_lists[0]) for single_list in list_of_lists)\n if random_state is not None:\n p = random_state.permutation(len(list_of_lists[0]))\n else:\n p = np.random.permutation(len(list_of_lists[0]))\n return [single_list[p] for single_list in list_of_lists]\n return None\n\n\n@DeveloperAPI\ndef shuffle_dict_unison_inplace(np_dict, random_state=None):\n keys = list(np_dict.keys())\n list_of_lists = list(np_dict.values())\n\n # shuffle up the list of lists according to previous fct\n shuffled_list = shuffle_unison_inplace(list_of_lists, random_state)\n\n recon = {}\n for ii, dkey in enumerate(keys):\n recon[dkey] = shuffled_list[ii]\n\n # we've shuffled the dictionary in place!\n return recon\n\n\n@DeveloperAPI\ndef split_dataset_ttv(dataset, split):\n # Obtain distinct splits from the split column. If\n # a split is not present in this set, then we can skip generating\n # the dataframe for that split.\n if dataset[split].dtype != int:\n dataset[split] = dataset[split].astype(int)\n\n distinct_values = dataset[split].drop_duplicates()\n if hasattr(distinct_values, \"compute\"):\n distinct_values = distinct_values.compute()\n distinct_values = set(distinct_values.values.tolist())\n\n training_set = split_dataset(dataset, split, 0) if 0 in distinct_values else None\n validation_set = split_dataset(dataset, split, 1) if 1 in distinct_values else None\n test_set = split_dataset(dataset, split, 2) if 2 in distinct_values else None\n return training_set, test_set, validation_set\n\n\n@DeveloperAPI\ndef split_dataset(dataset, split, value_to_split=0):\n split_df = dataset[dataset[split] == value_to_split]\n return split_df\n\n\n@DeveloperAPI\ndef collapse_rare_labels(labels, labels_limit):\n if labels_limit > 0:\n labels[labels >= labels_limit] = labels_limit\n return labels\n\n\n@DeveloperAPI\ndef class_counts(dataset, labels_field):\n return np.bincount(dataset[labels_field].flatten()).tolist()\n\n\n@DeveloperAPI\ndef load_from_file(file_name, field=None, dtype=int, ground_truth_split=2):\n \"\"\"Load experiment data from supported file formats.\n\n Experiment data can be test/train statistics, model predictions, probability, ground truth, ground truth metadata.\n :param file_name: Path to file to be loaded\n :param field: Target Prediction field.\n :param dtype:\n :param ground_truth_split: Ground truth split filter where 0 is train 1 is validation and 2 is test split. By\n default test split is used when loading ground truth from hdf5.\n :return: Experiment data as array\n \"\"\"\n if file_name.endswith(\".hdf5\") and field is not None:\n dataset = pd.read_hdf(file_name, key=HDF5_COLUMNS_KEY)\n column = dataset[field]\n array = column[dataset[SPLIT] == ground_truth_split].values # ground truth\n elif file_name.endswith(\".npy\"):\n array = np.load(file_name)\n elif file_name.endswith(\".csv\"):\n array = read_csv(file_name, header=None).values\n else:\n array = load_matrix(file_name, dtype)\n return array\n\n\n@DeveloperAPI\ndef replace_file_extension(file_path, extension):\n \"\"\"Return a file path for a file with same name but different format. a.csv, json -> a.json a.csv, hdf5 ->\n a.hdf5.\n\n :param file_path: original file path\n :param extension: file extension\n :return: file path with same name but different format\n \"\"\"\n if file_path is None:\n return None\n extension = extension.strip()\n if extension.startswith(\".\"):\n # Handle the case if the user calls with '.hdf5' instead of 'hdf5'\n extension = extension[1:]\n\n return os.path.splitext(file_path)[0] + \".\" + extension\n\n\n@DeveloperAPI\ndef file_exists_with_diff_extension(file_path, extension):\n return file_path is None or os.path.isfile(replace_file_extension(file_path, extension))\n\n\n@DeveloperAPI\ndef add_sequence_feature_column(df, col_name, seq_length):\n \"\"\"Adds a new column to the dataframe computed from an existing column. Values in the new column are space-\n delimited strings composed of preceding values of the same column up to seq_length. For example values of the\n i-th row of the new column will be a space-delimited string of df[col_name][i-seq_length].\n\n :param df: input dataframe\n :param col_name: column name containing sequential data\n :param seq_length: length of an array of preceeding column values to use\n \"\"\"\n if col_name not in df.columns.values:\n logger.error(f\"{col_name} column does not exist\")\n return\n\n new_col_name = col_name + \"_feature\"\n if new_col_name in df.columns.values:\n logger.warning(f\"{new_col_name} column already exists, values will be overridden\")\n\n new_data = [None] * seq_length\n old_data = np.array(df[col_name])\n\n for i in range(seq_length, len(df)):\n new_data.append(\" \".join(str(j) for j in old_data[i - seq_length : i]))\n\n df[new_col_name] = new_data\n df[new_col_name] = df[new_col_name].bfill()\n\n\n@DeveloperAPI\ndef override_in_memory_flag(input_features, override_value):\n num_overrides = 0\n for feature in input_features:\n if PREPROCESSING in feature:\n if \"in_memory\" in feature[PREPROCESSING]:\n feature[PREPROCESSING][\"in_memory\"] = override_value\n num_overrides += 1\n return num_overrides\n\n\n@DeveloperAPI\ndef normalize_numpy(obj):\n if isinstance(obj, np.integer):\n return int(obj)\n elif isinstance(obj, np.floating):\n return float(obj)\n elif isinstance(obj, np.ndarray):\n return normalize_numpy(obj.tolist())\n elif isinstance(obj, list):\n return [normalize_numpy(v) for v in obj]\n else:\n return obj\n\n\n@DeveloperAPI\nclass NumpyEncoder(json.JSONEncoder):\n \"\"\"Custom JSON encoder for handling NumPy objects.\n\n This encoder extends the `json.JSONEncoder` class and provides\n custom serialization for NumPy objects. It converts NumPy arrays,\n sets, tuples, integers, floating-point numbers, booleans, and\n dataclasses to their JSON serializable equivalents.\n\n Attributes:\n None\n\n Methods:\n default: Overrides the default method of `json.JSONEncoder`\n to provide custom serialization for NumPy objects.\n\n Usage:\n Use this encoder when serializing objects that contain NumPy\n arrays or other NumPy objects to JSON.\n\n Example:\n encoder = NumpyEncoder()\n json_data = encoder.encode(data)\n \"\"\"\n\n def default(self, o):\n if isinstance(o, (set, tuple)):\n return list(o)\n elif isinstance(o, np.bool_):\n return bool(o)\n elif isinstance(o, np.integer):\n return int(o)\n elif isinstance(o, np.floating):\n return float(o)\n elif isinstance(o, np.ndarray):\n return o.tolist()\n elif dataclasses.is_dataclass(o):\n return dataclasses.asdict(o)\n elif hasattr(o, \"to_dict\"):\n return o.to_dict()\n else:\n return json.JSONEncoder.default(self, o)\n\n\n@DeveloperAPI\ndef generate_kfold_splits(data_df, num_folds, random_state):\n kf = KFold(n_splits=num_folds, shuffle=True, random_state=random_state)\n fold_num = 0\n for train_indices, test_indices in kf.split(data_df):\n fold_num += 1\n yield train_indices, test_indices, fold_num\n\n\n@DeveloperAPI\ndef get_path_size(start_path, regex_accept=None, regex_reject=None):\n total_size = 0\n pattern_accept = re.compile(regex_accept) if regex_accept else None\n pattern_reject = re.compile(regex_reject) if regex_reject else None\n\n for dirpath, _, filenames in os.walk(start_path):\n for filename in filenames:\n filepath = os.path.join(dirpath, filename)\n if not os.path.islink(filepath):\n accepted = True\n if pattern_accept:\n accepted = accepted and pattern_accept.match(filename)\n if pattern_reject:\n accepted = accepted and not pattern_reject.match(filename)\n if accepted:\n total_size += os.path.getsize(filepath)\n\n return total_size\n\n\n@DeveloperAPI\ndef clear_data_cache():\n \"\"\"Clears any cached data objects (e.g., embeddings)\"\"\"\n load_glove.cache_clear()\n\n\n@DeveloperAPI\ndef figure_data_format_dataset(dataset):\n if isinstance(dataset, CacheableDataset):\n return figure_data_format_dataset(dataset.unwrap())\n elif isinstance(dataset, pd.DataFrame):\n return pd.DataFrame\n elif dd and isinstance(dataset, dd.DataFrame):\n return dd.DataFrame\n elif isinstance(dataset, dict):\n return dict\n elif isinstance(dataset, str):\n dataset = dataset.strip()\n if dataset.startswith(\"ludwig://\"):\n return \"ludwig\"\n if dataset.startswith(\"hf://\"):\n return \"hf\"\n\n dataset = dataset.lower()\n if dataset.endswith(\".csv\"):\n return \"csv\"\n elif dataset.endswith(\".tsv\"):\n return \"tsv\"\n elif dataset.endswith(\".json\"):\n return \"json\"\n elif dataset.endswith(\".jsonl\"):\n return \"jsonl\"\n elif (\n dataset.endswith(\".xls\")\n or dataset.endswith(\".xlsx\")\n or dataset.endswith(\".xlsm\")\n or dataset.endswith(\".xlsb\")\n or dataset.endswith(\".odf\")\n or dataset.endswith(\".ods\")\n or dataset.endswith(\".odt\")\n ):\n return \"excel\"\n elif dataset.endswith(\".parquet\"):\n return \"parquet\"\n elif dataset.endswith(\".pickle\") or dataset.endswith(\".p\"):\n return \"pickle\"\n elif dataset.endswith(\".feather\"):\n return \"feather\"\n elif dataset.endswith(\".fwf\"):\n return \"fwf\"\n elif dataset.endswith(\".html\"):\n return \"html\"\n elif dataset.endswith(\".orc\"):\n return \"orc\"\n elif dataset.endswith(\".sas\"):\n return \"sas\"\n elif dataset.endswith(\".spss\"):\n return \"spss\"\n elif dataset.endswith(\".dta\") or dataset.endswith(\".stata\"):\n return \"stata\"\n elif dataset.endswith(\".h5\") or dataset.endswith(\".hdf5\"):\n return \"hdf5\"\n else:\n raise ValueError(f\"Dataset path string {dataset} does not contain a valid extension\")\n else:\n raise ValueError(f\"Cannot figure out the format of dataset {dataset}\")\n\n\n@DeveloperAPI\ndef figure_data_format(dataset=None, training_set=None, validation_set=None, test_set=None):\n if dataset is not None:\n data_format = figure_data_format_dataset(dataset)\n elif training_set is not None:\n data_formats = [figure_data_format_dataset(training_set)]\n if validation_set is not None:\n data_formats.append(figure_data_format_dataset(validation_set))\n if test_set is not None:\n data_formats.append(figure_data_format_dataset(test_set))\n data_formats_set = set(data_formats)\n if len(data_formats_set) > 1:\n error_message = \"Datasets have different formats. Training: \"\n error_message += str(data_formats[0])\n if validation_set:\n error_message = \", Validation: \"\n error_message += str(data_formats[1])\n if test_set:\n error_message = \", Test: \"\n error_message += str(data_formats[-1])\n raise ValueError(error_message)\n else:\n data_format = next(iter(data_formats_set))\n else:\n raise ValueError(\"At least one between dataset and training_set must be not None\")\n return data_format\n\n\n@DeveloperAPI\ndef is_model_dir(path: str) -> bool:\n hyperparameters_fn = os.path.join(path, MODEL_HYPERPARAMETERS_FILE_NAME)\n ts_metadata_fn = os.path.join(path, TRAIN_SET_METADATA_FILE_NAME)\n is_a_model_dir = False\n if os.path.isdir(path) and os.path.isfile(hyperparameters_fn) and os.path.isfile(ts_metadata_fn):\n weights_files_count = 0\n for file_name in os.listdir(path):\n if file_name.startswith(MODEL_WEIGHTS_FILE_NAME):\n weights_files_count += 1\n if weights_files_count >= 2:\n is_a_model_dir = True\n return is_a_model_dir\n\n\n@DeveloperAPI\ndef ndarray2string(parm_array):\n # convert numpy.ndarray to ludwig custom string format\n if isinstance(parm_array, np.ndarray):\n return f\"__ndarray__{json.dumps(parm_array.tolist())}\"\n else:\n raise ValueError(f\"Argument must be numpy.ndarray. Instead argument found to be {type(parm_array)}\")\n\n\n@DeveloperAPI\ndef string2ndarray(parm_string):\n # convert ludwig custom ndarray string to numpy.ndarray\n if isinstance(parm_string, str) and parm_string[:11] == \"__ndarray__\":\n return np.array(json.loads(parm_string[11:]))\n else:\n raise ValueError(\"Argument must be Ludwig custom string format for numpy.ndarray\")\n\n\n@DeveloperAPI\ndef is_ludwig_ndarray_string(parm_string):\n # tests if parameter is a Ludwig custom ndarray string\n return isinstance(parm_string, str) and parm_string[:11] == \"__ndarray__\"\n\n\n@DeveloperAPI\ndef get_pa_dtype(obj: Any):\n if np.isscalar(obj):\n return pa.from_numpy_dtype(np.array(obj).dtype)\n elif isinstance(obj, np.ndarray) or isinstance(obj, list) or isinstance(obj, tuple):\n return pa.list_(get_pa_dtype(obj[0]))\n else:\n raise ValueError(f\"Unsupported type for pyarrow dtype: {type(obj)}\")\n\n\n@DeveloperAPI\ndef get_pa_schema(df: DataFrame):\n \"\"\"Gets the pyarrow schema associated with a given DataFrame.\n\n This will fail in very specific conditions worth enumerating:\n 1. If the DataFrame is a Dask DataFrame which has a partition of size 1 and its only sample is a NaN, then the\n `schema` dict will not contain the associated key. The value in this case will be inferred (likely incorrectly)\n as a float64 downstream.\n 2. If the DataFrame contains NaNs in some column and the presence of NaNs changes the overall dtype of the column.\n For example, if a number feature column contains some NaN-like value, then its dtype will be changed by the\n below `fillna` call from float32 to float64. This will cause `to_parquet` to fail downstream.\n \"\"\"\n head = df.head(100)\n\n schema = {}\n for k, v in head.items():\n if sum(v.isna()) > 0:\n v = v.fillna(np.nan).replace([np.nan], [None]) # Only fill NaNs if they are present\n v = v.values\n\n for val in v:\n if val is not None and k not in schema:\n schema[k] = get_pa_dtype(val)\n break\n return pa.schema(list(schema.items()))\n\n\ndata_reader_registry = {\n **{fmt: read_csv for fmt in CSV_FORMATS},\n **{fmt: read_tsv for fmt in TSV_FORMATS},\n **{fmt: read_json for fmt in JSON_FORMATS},\n **{fmt: read_jsonl for fmt in JSONL_FORMATS},\n **{fmt: read_excel for fmt in EXCEL_FORMATS},\n **{fmt: read_parquet for fmt in PARQUET_FORMATS},\n **{fmt: read_pickle for fmt in PICKLE_FORMATS},\n **{fmt: read_fwf for fmt in FWF_FORMATS},\n **{fmt: read_feather for fmt in FEATHER_FORMATS},\n **{fmt: read_html for fmt in HTML_FORMATS},\n **{fmt: read_orc for fmt in ORC_FORMATS},\n **{fmt: read_sas for fmt in SAS_FORMATS},\n **{fmt: read_spss for fmt in SPSS_FORMATS},\n **{fmt: read_stata for fmt in STATA_FORMATS},\n **{fmt: read_hdf5 for fmt in HDF5_FORMATS},\n}\n\n\n@DeveloperAPI\ndef load_dataset(dataset, data_format=None, df_lib=PANDAS_DF):\n if not data_format or data_format == \"auto\":\n data_format = figure_data_format(dataset)\n\n # use appropriate reader to create dataframe\n if data_format in DATAFRAME_FORMATS:\n return dataset\n elif data_format in DICT_FORMATS:\n return pd.DataFrame(dataset)\n elif data_format in CACHEABLE_FORMATS:\n data_reader = get_from_registry(data_format, data_reader_registry)\n return data_reader(dataset, df_lib)\n else:\n raise ValueError(f\"{data_format} format is not supported\")\n\n\n@DeveloperAPI\n@contextlib.contextmanager\ndef use_credentials(creds):\n if creds is None:\n with contextlib.nullcontext():\n yield\n return\n\n # https://filesystem-spec.readthedocs.io/en/latest/features.html#configuration\n # This allows us to avoid having to plumb the `storage_options` kwargs through\n # every remote FS call in Ludwig. This implementation is restricted to one thread\n # in the process acquiring the lock at once.\n with GLOBAL_CRED_LOCK:\n with tempfile.TemporaryDirectory() as tmpdir:\n fname = os.path.join(tmpdir, \"conf.json\")\n with open(fname, \"w\", encoding=\"utf-8\") as f:\n json.dump(creds, f)\n\n # Backup any existing credentials\n old_conf = dict(**conf)\n\n conf.clear()\n set_conf_files(tmpdir, conf)\n try:\n yield\n finally:\n # Restore previous credentials\n with open(fname, \"w\", encoding=\"utf-8\") as f:\n json.dump(old_conf, f)\n conf.clear()\n set_conf_files(tmpdir, conf)\n\n\ndef get_sanitized_feature_name(feature_name: str) -> str:\n \"\"\"Replaces non-word characters (anything other than alphanumeric or _) with _.\n\n Used in model config initialization and sanitize_column_names(), which is called during dataset building.\n \"\"\"\n return re.sub(r\"[(){}.:\\\"\\\"\\'\\'\\[\\]]\", \"_\", feature_name)\n\n\ndef sanitize_column_names(df: DataFrame) -> DataFrame:\n \"\"\"Renames df columns with non-word characters (anything other than alphanumeric or _) to _.\"\"\"\n safe_column_names = [get_sanitized_feature_name(col) for col in df.columns]\n return df.rename(columns=dict(zip(df.columns, safe_column_names)))\n"
},
{
"path": "ludwig/utils/dataframe_utils.py",
"content": "from typing import Optional\n\nimport numpy as np\nimport pandas as pd\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import DASK_MODULE_NAME\nfrom ludwig.data.dataframe.base import DataFrameEngine\nfrom ludwig.utils.types import DataFrame\n\n\n@DeveloperAPI\ndef is_dask_lib(df_lib) -> bool:\n \"\"\"Returns whether the dataframe library is dask.\"\"\"\n return df_lib.__name__ == DASK_MODULE_NAME\n\n\n@DeveloperAPI\ndef is_dask_backend(backend: Optional[\"Backend\"]) -> bool: # noqa: F821\n \"\"\"Returns whether the backend's dataframe is dask.\"\"\"\n return backend is not None and is_dask_lib(backend.df_engine.df_lib)\n\n\n@DeveloperAPI\ndef is_dask_series_or_df(df: DataFrame, backend: Optional[\"Backend\"]) -> bool: # noqa: F821\n if is_dask_backend(backend):\n import dask.dataframe as dd\n\n return isinstance(df, dd.Series) or isinstance(df, dd.DataFrame)\n return False\n\n\n@DeveloperAPI\ndef flatten_df(df: DataFrame, df_engine: DataFrameEngine) -> tuple[DataFrame, dict[str, tuple]]: # noqa: F821\n \"\"\"Returns a flattened dataframe with a dictionary of the original shapes, keyed by dataframe columns.\"\"\"\n # Workaround for: https://issues.apache.org/jira/browse/ARROW-5645\n column_shapes = {}\n for c in df.columns:\n df = df_engine.persist(df)\n shape = df_engine.compute(\n df_engine.map_objects(\n df[c],\n lambda x: np.array(x).shape,\n ).max()\n )\n\n if len(shape) > 1:\n column_shapes[c] = shape\n df[c] = df_engine.map_objects(df[c], lambda x: np.array(x).reshape(-1))\n return df, column_shapes\n\n\n@DeveloperAPI\ndef unflatten_df(df: DataFrame, column_shapes: dict[str, tuple], df_engine: DataFrameEngine) -> DataFrame: # noqa: F821\n \"\"\"Returns an unflattened dataframe, the reverse of flatten_df.\"\"\"\n for c in df.columns:\n shape = column_shapes.get(c)\n if shape:\n df[c] = df_engine.map_objects(df[c], lambda x: np.array(x).reshape(shape))\n return df\n\n\n@DeveloperAPI\ndef to_numpy_dataset(df: DataFrame, backend: Optional[\"Backend\"] = None) -> dict[str, np.ndarray]: # noqa: F821\n \"\"\"Returns a dictionary of numpy arrays, keyed by the columns of the given dataframe.\"\"\"\n # Compute Dask DataFrames to pandas first to avoid issues with extension dtypes\n # (e.g. TensorDtype) that Dask-expr's metadata system cannot handle.\n if backend and is_dask_backend(backend):\n df = backend.df_engine.compute(df)\n dataset = {}\n for col in df.columns:\n if len(df.index) != 0:\n dataset[col] = np.stack(df[col].to_numpy())\n else:\n # Dataframe is empty.\n # Use to_list() directly, as np.stack() requires at least one array to stack.\n dataset[col] = df[col].to_list()\n return dataset\n\n\n@DeveloperAPI\ndef from_numpy_dataset(dataset) -> pd.DataFrame:\n \"\"\"Returns a pandas dataframe from the dataset.\"\"\"\n col_mapping = {}\n for k, v in dataset.items():\n if len(v.shape) > 1:\n # unstacking, needed for ndarrays of dimension 2 and more\n (*vals,) = v\n else:\n # not unstacking. Needed because otherwise pandas casts types\n # the way it wants, like converting a list of float32 scalats\n # to a column of float64\n vals = v\n col_mapping[k] = vals\n return pd.DataFrame.from_dict(col_mapping)\n\n\n@DeveloperAPI\ndef set_index_name(pd_df: pd.DataFrame, name: str) -> pd.DataFrame:\n pd_df.index.name = name\n return pd_df\n\n\n@DeveloperAPI\ndef to_batches(df: pd.DataFrame, batch_size: int) -> list[pd.DataFrame]:\n n_rows = len(df)\n return [df[i : i + batch_size].copy() for i in range(0, n_rows, batch_size)]\n\n\n@DeveloperAPI\ndef from_batches(batches: list[pd.DataFrame]) -> pd.DataFrame:\n return pd.concat(batches)\n\n\n@DeveloperAPI\ndef to_scalar_df(df: pd.DataFrame) -> pd.DataFrame:\n \"\"\"Converts all columns in a pd.DataFrame to be scalar types.\n\n For object columns of lists, each element of the list is expanded into its own column named {column}_{index}. We\n assume all object columns are lists of the same length (i.e., tensor format output from preprocessing). It's also\n important that the relative order of the columns is preserved, to maintain consistency with other conversions like\n the one for Hummingbird.\n \"\"\"\n scalar_df = df\n column_ordering = []\n for c, s in df.items():\n if s.dtype == \"object\":\n s_list = s.to_list()\n try:\n ncols = s_list[0].shape[0]\n split_cols = [f\"{c}_{k}\" for k in range(ncols)]\n sdf = pd.DataFrame(s_list, columns=split_cols)\n scalar_df = pd.concat([scalar_df, sdf], axis=1)\n column_ordering += split_cols\n except AttributeError as e:\n raise ValueError(f\"Expected series of lists, but found {s_list[0]}\") from e\n else:\n column_ordering.append(c)\n return scalar_df[column_ordering]\n"
},
{
"path": "ludwig/utils/dataset_utils.py",
"content": "import pandas as pd\nfrom sklearn.model_selection import train_test_split\n\nfrom ludwig.api_annotations import PublicAPI\nfrom ludwig.constants import TEST_SPLIT, TRAIN_SPLIT, VALIDATION_SPLIT\nfrom ludwig.data.dataset.base import Dataset\nfrom ludwig.utils.defaults import default_random_seed\n\n\n@PublicAPI\ndef get_repeatable_train_val_test_split(\n df_input, stratify_colname=\"\", random_seed=default_random_seed, frac_train=0.7, frac_val=0.1, frac_test=0.2\n):\n \"\"\"Return df_input with split column containing (if possible) non-zero rows in the train, validation, and test\n data subset categories.\n\n If the input dataframe does not contain an existing split column or if the\n number of rows in both the validation and test split is 0 and non-empty\n stratify_colname specified, return df_input with split column set according\n to frac_ and stratify_colname.\n\n Else stratify_colname is ignored, and:\n If the input dataframe contains an existing split column and non-zero row\n counts for all three split types, return df_input.\n If the input dataframe contains an existing split column but only one of\n validation and test split has non-zero row counts, return df_input with\n missing split getting rows from train split as per frac_.\n\n Parameters\n ----------\n df_input : Pandas dataframe\n Input dataframe to be split.\n stratify_colname : str\n The column used for stratification (if desired); usually the label column.\n random_seed : int\n Seed used to get repeatable split.\n frac_train : float\n frac_val : float\n frac_test : float\n The ratios with which to split the dataframe into train, val, and test data;\n should sum to 1.0.\n\n Returns\n -------\n df_split :\n Dataframe containing the three splits.\n \"\"\"\n\n if frac_train + frac_val + frac_test != 1.0:\n raise ValueError(f\"fractions {frac_train:f}, {frac_val:f}, {frac_test:f} do not add up to 1.0\")\n if stratify_colname:\n do_stratify_split = True\n if stratify_colname not in df_input.columns:\n raise ValueError(\"%s is not a column in the dataframe\" % (stratify_colname))\n else:\n do_stratify_split = False\n if \"split\" not in df_input.columns:\n df_input[\"split\"] = 0 # set up for non-stratified split path\n\n if \"split\" in df_input.columns:\n df_train = df_input[df_input[\"split\"] == TRAIN_SPLIT].copy()\n df_val = df_input[df_input[\"split\"] == VALIDATION_SPLIT].copy()\n df_test = df_input[df_input[\"split\"] == TEST_SPLIT].copy()\n if not do_stratify_split or len(df_val) != 0 or len(df_test) != 0:\n if len(df_val) == 0:\n df_val = df_train.sample(frac=frac_val, replace=False, random_state=random_seed)\n df_train = df_train.drop(df_val.index)\n if len(df_test) == 0:\n df_test = df_train.sample(frac=frac_test, replace=False, random_state=random_seed)\n df_train = df_train.drop(df_test.index)\n do_stratify_split = False\n\n if do_stratify_split:\n # Make sure the `stratify_colname` doesn't have any NaNs.\n df_input = df_input[df_input[stratify_colname].notna()]\n\n # Split original dataframe into train and temp dataframes.\n y = df_input[[stratify_colname]] # Dataframe of just the column on which to stratify.\n df_train, df_temp, y_train, y_temp = train_test_split(\n df_input, y, stratify=y, test_size=(1.0 - frac_train), random_state=random_seed\n )\n # Split the temp dataframe into val and test dataframes.\n relative_frac_test = frac_test / (frac_val + frac_test)\n df_val, df_test, y_val, y_test = train_test_split(\n df_temp, y_temp, stratify=y_temp, test_size=relative_frac_test, random_state=random_seed\n )\n\n assert len(df_input) == len(df_train) + len(df_val) + len(df_test)\n df_train[\"split\"] = TRAIN_SPLIT\n df_val[\"split\"] = VALIDATION_SPLIT\n df_test[\"split\"] = TEST_SPLIT\n df_split = pd.concat([df_train, df_val, df_test], ignore_index=True)\n return df_split\n\n\ndef generate_dataset_statistics(\n training_set: Dataset,\n validation_set: str | dict | pd.DataFrame | Dataset | None,\n test_set: str | dict | pd.DataFrame | Dataset | None,\n) -> list[tuple[str, int, int]]:\n from ludwig.benchmarking.utils import format_memory\n\n dataset_statistics = [\n [\"Dataset\", \"Size (Rows)\", \"Size (In Memory)\"],\n [\"Training\", len(training_set), format_memory(training_set.in_memory_size_bytes)],\n ]\n if validation_set is not None:\n dataset_statistics.append(\n [\"Validation\", len(validation_set), format_memory(validation_set.in_memory_size_bytes)]\n )\n if test_set is not None:\n dataset_statistics.append([\"Test\", len(test_set), format_memory(test_set.in_memory_size_bytes)])\n return dataset_statistics\n"
},
{
"path": "ludwig/utils/date_utils.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport time\nfrom datetime import date, datetime, timezone\n\nimport numpy as np\nfrom dateutil.parser import parse, ParserError\n\nfrom ludwig.api_annotations import DeveloperAPI\n\nSCALE_S = np.floor(np.log10(time.time()))\n\n\n@DeveloperAPI\ndef create_vector_from_datetime_obj(datetime_obj):\n yearday = datetime_obj.toordinal() - date(datetime_obj.year, 1, 1).toordinal() + 1\n\n midnight = datetime_obj.replace(hour=0, minute=0, second=0, microsecond=0)\n second_of_day = (datetime_obj - midnight).seconds\n\n return [\n datetime_obj.year,\n datetime_obj.month,\n datetime_obj.day,\n datetime_obj.weekday(),\n yearday,\n datetime_obj.hour,\n datetime_obj.minute,\n datetime_obj.second,\n second_of_day,\n ]\n\n\n@DeveloperAPI\ndef parse_datetime(timestamp: float | int | str) -> datetime:\n \"\"\"Parse a datetime from a string or a numeric timestamp.\n\n Args:\n timestamp: A datetime string or numeric timestamp.\n\n Returns:\n A datetime representation of `timestamp`.\n \"\"\"\n try:\n dt = parse(timestamp)\n except (OverflowError, ParserError, TypeError):\n dt = convert_number_to_datetime(timestamp)\n\n return dt\n\n\n@DeveloperAPI\ndef convert_number_to_datetime(timestamp: float | int | str) -> datetime:\n \"\"\"Convert a numeric timestamp to a datetime object.\n\n `datetime` objects can be created from POSIX timestamps like those returned by `time.time()`.\n\n Args:\n timestamp: A numeric timestamp.\n\n Returns:\n A datetime representation of `timestamp`.\n\n Raises:\n ValueError: Raised if `timestamp` is not a number or not a valid datetime.\n \"\"\"\n try:\n timestamp = float(timestamp)\n except TypeError:\n raise ValueError(f\"Provided value {timestamp} is not a valid numeric timestamp\")\n\n # Determine the unit of the timestamp\n ts_scale = np.floor(np.log10(timestamp))\n\n # `datetime.datetime.fromtimestamp` expects a timestamp in seconds. Rescale the timestamp if it is not in seconds.\n if SCALE_S < ts_scale:\n delta = ts_scale - SCALE_S\n timestamp = timestamp / np.power(10, delta)\n\n # Convert the timestamp to a datetime object. If it is not a valid timestamp, `ValueError` is raised.\n dt = datetime.fromtimestamp(timestamp, tz=timezone.utc).replace(tzinfo=None)\n return dt\n"
},
{
"path": "ludwig/utils/defaults.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport argparse\nimport copy\nimport logging\n\nimport yaml\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.contrib import add_contrib_callback_args\nfrom ludwig.features.feature_registries import get_input_type_registry\nfrom ludwig.globals import LUDWIG_VERSION\nfrom ludwig.schema.model_config import ModelConfig\nfrom ludwig.schema.preprocessing import PreprocessingConfig\nfrom ludwig.utils.backward_compatibility import upgrade_config_dict_to_latest_version\nfrom ludwig.utils.data_utils import load_config_from_str, load_yaml\nfrom ludwig.utils.fs_utils import open_file\nfrom ludwig.utils.print_utils import print_ludwig\n\nlogger = logging.getLogger(__name__)\n\ndefault_random_seed = 42\n\n# Still needed for preprocessing TODO(Connor): Refactor ludwig/data/preprocessing to use schema\n# TODO(travis): remove this, make type a protected string for each subclass\ndefault_feature_specific_preprocessing_parameters = {\n name: preproc_sect.get_schema_cls()(name=\"__tmp__\", type=name).preprocessing.to_dict()\n for name, preproc_sect in get_input_type_registry().items()\n}\n\ndefault_training_preprocessing_parameters = copy.deepcopy(default_feature_specific_preprocessing_parameters)\ndefault_training_preprocessing_parameters.update(PreprocessingConfig().to_dict())\n\ndefault_prediction_preprocessing_parameters = copy.deepcopy(default_feature_specific_preprocessing_parameters)\n\n\n@DeveloperAPI\ndef render_config(config=None, output=None, **kwargs):\n upgraded_config = upgrade_config_dict_to_latest_version(config)\n output_config = ModelConfig.from_dict(upgraded_config).to_dict()\n\n if output is None:\n print(yaml.safe_dump(output_config, None, sort_keys=False))\n else:\n with open_file(output, \"w\") as f:\n yaml.safe_dump(output_config, f, sort_keys=False)\n\n\n@DeveloperAPI\ndef cli_render_config(sys_argv):\n parser = argparse.ArgumentParser(\n description=\"This script renders the full config from a user config.\",\n prog=\"ludwig render_config\",\n usage=\"%(prog)s [options]\",\n )\n parser.add_argument(\n \"-c\",\n \"--config\",\n type=load_yaml,\n help=\"Path to the YAML file containing the model configuration\",\n )\n parser.add_argument(\n \"-cs\",\n \"--config_str\",\n dest=\"config\",\n type=load_config_from_str,\n help=\"JSON or YAML serialized string of the model configuration\",\n )\n parser.add_argument(\n \"-o\",\n \"--output\",\n type=str,\n help=\"output rendered YAML config path\",\n required=False,\n )\n\n add_contrib_callback_args(parser)\n args = parser.parse_args(sys_argv)\n\n args.callbacks = args.callbacks or []\n for callback in args.callbacks:\n callback.on_cmdline(\"render_config\", *sys_argv)\n\n print_ludwig(\"Render Config\", LUDWIG_VERSION)\n render_config(**vars(args))\n"
},
{
"path": "ludwig/utils/entmax/LICENSE",
"content": "MIT License\n\nCopyright (c) 2019 DeepSPIN\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
},
{
"path": "ludwig/utils/entmax/README.md",
"content": "# entmax\n\n______________________________________________________________________\n\nThis package provides a pytorch implementation of entmax and entmax losses:\na sparse family of probability mappings and corresponding loss functions,\ngeneralizing softmax / cross-entropy.\n\n*Features:*\n\n- Exact partial-sort algorithms for 1.5-entmax and 2-entmax (sparsemax).\n- A bisection-based algorithm for generic alpha-entmax.\n- Gradients w.r.t. alpha for adaptive, learned sparsity!\n\n*Requirements:* python 3, pytorch >= 1.0 (and pytest for unit tests)\n\n## Example\n\n```python\nimport torch\nfrom torch.nn.functional import softmax\n\nfrom entmax import sparsemax, entmax15\n\nx = torch.tensor([-2, 0, 0.5])\n\nprint(softmax(x, dim=0))\n# tensor([0.0486, 0.3592, 0.5922])\n\nprint(sparsemax(x, dim=0))\n# tensor([0.0000, 0.2500, 0.7500])\n\nprint(entmax15(x, dim=0))\n# tensor([0.0000, 0.3260, 0.6740])\n```\n\nGradients w.r.t. alpha (continued):\n\n```python\nimport torch\nfrom torch.autograd import grad\n\nfrom entmax import entmax_bisect\n\nx = torch.tensor([[-1, 0, 0.5], [1, 2, 3.5]])\n\nalpha = torch.tensor(1.33, requires_grad=True)\n\np = entmax_bisect(x, alpha)\n\nprint(p)\n# tensor([[0.0460, 0.3276, 0.6264],\n# [0.0026, 0.1012, 0.8963]], grad_fn=)\n\nprint(grad(p[0, 0], alpha))\n# (tensor(-0.2562),)\n```\n\n## Installation\n\n```\npip install entmax\n```\n\n## Citations\n\n[Sparse Sequence-to-Sequence Models](https://www.aclweb.org/anthology/P19-1146)\n\n```\n@inproceedings{entmax,\n author = {Peters, Ben and Niculae, Vlad and Martins, Andr{\\'e} FT},\n title = {Sparse Sequence-to-Sequence Models},\n booktitle = {Proc. ACL},\n year = {2019},\n url = {https://www.aclweb.org/anthology/P19-1146}\n}\n```\n\n[Adaptively Sparse Transformers](https://arxiv.org/pdf/1909.00015.pdf)\n\n```\n@inproceedings{correia19adaptively,\n author = {Correia, Gon\\c{c}alo M and Niculae, Vlad and Martins, Andr{\\'e} FT},\n title = {Adaptively Sparse Transformers},\n booktitle = {Proc. EMNLP-IJCNLP (to appear)},\n year = {2019},\n}\n```\n\nFurther reading:\n\n- Blondel, Martins, and Niculae, 2019. [Learning with Fenchel-Young Losses](https://arxiv.org/abs/1901.02324).\n- Martins and Astudillo, 2016. [From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification](https://arxiv.org/abs/1602.02068).\n- Peters and Martins, 2019 [IT-IST at the SIGMORPHON 2019 Shared Task: Sparse Two-headed Models for Inflection](https://www.aclweb.org/anthology/W19-4207).\n"
},
{
"path": "ludwig/utils/entmax/__init__.py",
"content": "__version__ = \"1.1.dev0\"\n\nfrom ludwig.utils.entmax.activations import entmax15, Entmax15, sparsemax, Sparsemax\nfrom ludwig.utils.entmax.losses import (\n entmax15_loss,\n Entmax15Loss,\n entmax_bisect_loss,\n EntmaxBisectLoss,\n sparsemax_bisect_loss,\n sparsemax_loss,\n SparsemaxBisectLoss,\n SparsemaxLoss,\n)\nfrom ludwig.utils.entmax.root_finding import entmax_bisect, EntmaxBisect, sparsemax_bisect, SparsemaxBisect\n\n__all__ = [\n \"entmax15\",\n \"Entmax15\",\n \"sparsemax\",\n \"Sparsemax\",\n \"entmax15_loss\",\n \"Entmax15Loss\",\n \"entmax_bisect_loss\",\n \"EntmaxBisectLoss\",\n \"sparsemax_bisect_loss\",\n \"sparsemax_loss\",\n \"SparsemaxBisectLoss\",\n \"SparsemaxLoss\",\n \"entmax_bisect\",\n \"EntmaxBisect\",\n \"sparsemax_bisect\",\n \"SparsemaxBisect\",\n]\n"
},
{
"path": "ludwig/utils/entmax/activations.py",
"content": "\"\"\"An implementation of entmax (Peters et al., 2019). See https://arxiv.org/pdf/1905.05702 for detailed\ndescription.\n\nThis builds on previous work with sparsemax (Martins & Astudillo, 2016). See https://arxiv.org/pdf/1602.02068.\n\"\"\"\n\n# Author: Ben Peters\n# Author: Vlad Niculae \n# License: MIT\n\nimport torch\nimport torch.nn as nn\nfrom torch.autograd import Function\n\n\ndef _make_ix_like(X, dim):\n d = X.size(dim)\n rho = torch.arange(1, d + 1, device=X.device, dtype=X.dtype)\n view = [1] * X.dim()\n view[0] = -1\n return rho.view(view).transpose(0, dim)\n\n\ndef _roll_last(X, dim):\n if dim == -1:\n return X\n elif dim < 0:\n dim = X.dim() - dim\n\n perm = [i for i in range(X.dim()) if i != dim] + [dim]\n return X.permute(perm)\n\n\ndef _sparsemax_threshold_and_support(X, dim=-1, k=None):\n \"\"\"Core computation for sparsemax: optimal threshold and support size.\n\n Parameters\n ----------\n X : torch.Tensor\n The input tensor to compute thresholds over.\n\n dim : int\n The dimension along which to apply sparsemax.\n\n k : int or None\n number of largest elements to partial-sort over. For optimal\n performance, should be slightly bigger than the expected number of\n nonzeros in the solution. If the solution is more than k-sparse,\n this function is recursively called with a 2*k schedule.\n If `None`, full sorting is performed from the beginning.\n\n Returns\n -------\n tau : torch.Tensor like `X`, with all but the `dim` dimension intact\n the threshold value for each vector\n support_size : torch LongTensor, shape like `tau`\n the number of nonzeros in each vector.\n \"\"\"\n\n if k is None or k >= X.shape[dim]: # do full sort\n topk, _ = torch.sort(X, dim=dim, descending=True)\n else:\n topk, _ = torch.topk(X, k=k, dim=dim)\n\n topk_cumsum = topk.cumsum(dim) - 1\n rhos = _make_ix_like(topk, dim)\n support = rhos * topk > topk_cumsum\n\n support_size = support.sum(dim=dim).unsqueeze(dim)\n tau = topk_cumsum.gather(dim, support_size - 1)\n tau /= support_size.to(X.dtype)\n\n if k is not None and k < X.shape[dim]:\n unsolved = (support_size == k).squeeze(dim)\n\n if torch.any(unsolved):\n in_ = _roll_last(X, dim)[unsolved]\n tau_, ss_ = _sparsemax_threshold_and_support(in_, dim=-1, k=2 * k)\n _roll_last(tau, dim)[unsolved] = tau_\n _roll_last(support_size, dim)[unsolved] = ss_\n\n return tau, support_size\n\n\ndef _entmax_threshold_and_support(X, dim=-1, k=None):\n \"\"\"Core computation for 1.5-entmax: optimal threshold and support size.\n\n Parameters\n ----------\n X : torch.Tensor\n The input tensor to compute thresholds over.\n\n dim : int\n The dimension along which to apply 1.5-entmax.\n\n k : int or None\n number of largest elements to partial-sort over. For optimal\n performance, should be slightly bigger than the expected number of\n nonzeros in the solution. If the solution is more than k-sparse,\n this function is recursively called with a 2*k schedule.\n If `None`, full sorting is performed from the beginning.\n\n Returns\n -------\n tau : torch.Tensor like `X`, with all but the `dim` dimension intact\n the threshold value for each vector\n support_size : torch LongTensor, shape like `tau`\n the number of nonzeros in each vector.\n \"\"\"\n\n if k is None or k >= X.shape[dim]: # do full sort\n Xsrt, _ = torch.sort(X, dim=dim, descending=True)\n else:\n Xsrt, _ = torch.topk(X, k=k, dim=dim)\n\n rho = _make_ix_like(Xsrt, dim)\n mean = Xsrt.cumsum(dim) / rho\n mean_sq = (Xsrt**2).cumsum(dim) / rho\n ss = rho * (mean_sq - mean**2)\n delta = (1 - ss) / rho\n\n # NOTE this is not exactly the same as in reference algo\n # Fortunately it seems the clamped values never wrongly\n # get selected by tau <= sorted_z. Prove this!\n delta_nz = torch.clamp(delta, 0)\n tau = mean - torch.sqrt(delta_nz)\n\n support_size = (tau <= Xsrt).sum(dim).unsqueeze(dim)\n tau_star = tau.gather(dim, support_size - 1)\n\n if k is not None and k < X.shape[dim]:\n unsolved = (support_size == k).squeeze(dim)\n\n if torch.any(unsolved):\n X_ = _roll_last(X, dim)[unsolved]\n tau_, ss_ = _entmax_threshold_and_support(X_, dim=-1, k=2 * k)\n _roll_last(tau_star, dim)[unsolved] = tau_\n _roll_last(support_size, dim)[unsolved] = ss_\n\n return tau_star, support_size\n\n\nclass SparsemaxFunction(Function):\n @classmethod\n def forward(cls, ctx, X, dim=-1, k=None):\n ctx.dim = dim\n output, backwards_kwargs = _sparsemax_forward(X, dim, k)\n ctx.save_for_backward(backwards_kwargs[\"supp_size\"], output)\n return output\n\n @classmethod\n def backward(cls, ctx, grad_output):\n supp_size, output = ctx.saved_tensors\n dim = ctx.dim\n grad_input = grad_output.clone()\n grad_input[output == 0] = 0\n\n v_hat = grad_input.sum(dim=dim) / supp_size.to(output.dtype).squeeze(dim)\n v_hat = v_hat.unsqueeze(dim)\n grad_input = torch.where(output != 0, grad_input - v_hat, grad_input)\n return grad_input, None, None\n\n\ndef _sparsemax_forward(X, dim, k):\n max_val, _ = X.max(dim=dim, keepdim=True)\n X = X - max_val # same numerical stability trick as softmax\n tau, supp_size = _sparsemax_threshold_and_support(X, dim=dim, k=k)\n output = torch.clamp(X - tau, min=0)\n return output, {\"supp_size\": supp_size}\n\n\nclass Entmax15Function(Function):\n @classmethod\n def forward(cls, ctx, X, dim=0, k=None):\n ctx.dim = dim\n Y, _ = _entmax15_forward(X, dim, k)\n ctx.save_for_backward(Y)\n return Y\n\n @classmethod\n def backward(cls, ctx, dY):\n (Y,) = ctx.saved_tensors\n gppr = Y.sqrt() # = 1 / g'' (Y)\n dX = dY * gppr\n q = dX.sum(ctx.dim) / gppr.sum(ctx.dim)\n q = q.unsqueeze(ctx.dim)\n dX -= q * gppr\n return dX, None, None\n\n\ndef _entmax15_forward(X, dim, k):\n max_val, _ = X.max(dim=dim, keepdim=True)\n X = X - max_val # same numerical stability trick as for softmax\n X = X / 2 # divide by 2 to solve actual Entmax\n\n tau_star, _ = _entmax_threshold_and_support(X, dim=dim, k=k)\n\n Y = torch.clamp(X - tau_star, min=0) ** 2\n return Y, {}\n\n\ndef sparsemax(X, dim=-1, k=None, training=True):\n \"\"\"sparsemax: normalizing sparse transform (a la softmax).\n\n Solves the projection:\n\n min_p ||x - p||_2 s.t. p >= 0, sum(p) == 1.\n\n Parameters\n ----------\n X : torch.Tensor\n The input tensor.\n\n dim : int\n The dimension along which to apply sparsemax.\n\n k : int or None\n number of largest elements to partial-sort over. For optimal\n performance, should be slightly bigger than the expected number of\n nonzeros in the solution. If the solution is more than k-sparse,\n this function is recursively called with a 2*k schedule.\n If `None`, full sorting is performed from the beginning.\n\n Returns\n -------\n P : torch tensor, same shape as X\n The projection result, such that P.sum(dim=dim) == 1 elementwise.\n \"\"\"\n # Avoids call to custom autograd.Function during eval to ensure torchscript compatibility\n # custom autograd.Function is not scriptable: https://github.com/pytorch/pytorch/issues/22329#issuecomment-506608053\n if not training:\n output, _ = _sparsemax_forward(X, dim, k)\n return output\n return SparsemaxFunction.apply(X, dim, k)\n\n\ndef entmax15(X, dim=-1, k=None, training=True):\n \"\"\"1.5-entmax: normalizing sparse transform (a la softmax).\n\n Solves the optimization problem:\n\n max_p - H_1.5(p) s.t. p >= 0, sum(p) == 1.\n\n where H_1.5(p) is the Tsallis alpha-entropy with alpha=1.5.\n\n Parameters\n ----------\n X : torch.Tensor\n The input tensor.\n\n dim : int\n The dimension along which to apply 1.5-entmax.\n\n k : int or None\n number of largest elements to partial-sort over. For optimal\n performance, should be slightly bigger than the expected number of\n nonzeros in the solution. If the solution is more than k-sparse,\n this function is recursively called with a 2*k schedule.\n If `None`, full sorting is performed from the beginning.\n\n Returns\n -------\n P : torch tensor, same shape as X\n The projection result, such that P.sum(dim=dim) == 1 elementwise.\n \"\"\"\n # Avoids call to custom autograd.Function during eval to ensure torchscript compatibility\n # custom autograd.Function is not scriptable: https://github.com/pytorch/pytorch/issues/22329#issuecomment-506608053\n if not training:\n output, _ = _entmax15_forward(X, dim, k)\n return output\n return Entmax15Function.apply(X, dim, k)\n\n\nclass Sparsemax(nn.Module):\n def __init__(self, dim=-1, k=None):\n \"\"\"sparsemax: normalizing sparse transform (a la softmax).\n\n Solves the projection:\n\n min_p ||x - p||_2 s.t. p >= 0, sum(p) == 1.\n\n Parameters\n ----------\n dim : int\n The dimension along which to apply sparsemax.\n\n k : int or None\n number of largest elements to partial-sort over. For optimal\n performance, should be slightly bigger than the expected number of\n nonzeros in the solution. If the solution is more than k-sparse,\n this function is recursively called with a 2*k schedule.\n If `None`, full sorting is performed from the beginning.\n \"\"\"\n self.dim = dim\n self.k = k\n super().__init__()\n\n def forward(self, X):\n return sparsemax(X, dim=self.dim, k=self.k, training=self.training)\n\n\nclass Entmax15(nn.Module):\n def __init__(self, dim=-1, k=None):\n \"\"\"1.5-entmax: normalizing sparse transform (a la softmax).\n\n Solves the optimization problem:\n\n max_p - H_1.5(p) s.t. p >= 0, sum(p) == 1.\n\n where H_1.5(p) is the Tsallis alpha-entropy with alpha=1.5.\n\n Parameters\n ----------\n dim : int\n The dimension along which to apply 1.5-entmax.\n\n k : int or None\n number of largest elements to partial-sort over. For optimal\n performance, should be slightly bigger than the expected number of\n nonzeros in the solution. If the solution is more than k-sparse,\n this function is recursively called with a 2*k schedule.\n If `None`, full sorting is performed from the beginning.\n \"\"\"\n self.dim = dim\n self.k = k\n super().__init__()\n\n def forward(self, X):\n return entmax15(X, dim=self.dim, k=self.k, training=self.training)\n"
},
{
"path": "ludwig/utils/entmax/losses.py",
"content": "import torch\nimport torch.nn as nn\nfrom torch.autograd import Function\n\nfrom ludwig.constants import IGNORE_INDEX_TOKEN_ID\nfrom ludwig.utils.entmax.activations import entmax15, sparsemax\nfrom ludwig.utils.entmax.root_finding import entmax_bisect, sparsemax_bisect\n\n\nclass _GenericLoss(nn.Module):\n def __init__(self, ignore_index=IGNORE_INDEX_TOKEN_ID, reduction=\"elementwise_mean\"):\n assert reduction in [\"elementwise_mean\", \"sum\", \"none\"]\n self.reduction = reduction\n self.ignore_index = ignore_index\n super().__init__()\n\n def forward(self, X, target):\n loss = self.loss(X, target)\n if self.ignore_index >= 0:\n ignored_positions = target == self.ignore_index\n size = (target.size(0) - ignored_positions.sum()).item()\n loss.masked_fill_(ignored_positions, 0.0)\n else:\n size = target.size(0)\n if self.reduction == \"sum\":\n loss = loss.sum()\n elif self.reduction == \"elementwise_mean\":\n if size == 0:\n # Returns zero loss and zero gradient in the rare case that all row targets are ignored.\n loss = loss.sum() * 0.0\n else:\n loss = loss.sum() / float(size)\n return loss\n\n\nclass _GenericLossFunction(Function):\n @classmethod\n def forward(cls, ctx, X, target, alpha, proj_args):\n \"\"\"X (FloatTensor): n x num_classes target (LongTensor): n, the indices of the target classes.\"\"\"\n assert X.shape[0] == target.shape[0]\n\n p_star = cls.project(X, alpha, **proj_args)\n loss = cls.omega(p_star, alpha)\n\n p_star.scatter_add_(1, target.unsqueeze(1), torch.full_like(p_star, -1))\n loss += torch.einsum(\"ij,ij->i\", p_star, X)\n ctx.save_for_backward(p_star)\n\n return loss\n\n @classmethod\n def backward(cls, ctx, grad_output):\n (p_star,) = ctx.saved_tensors\n grad = grad_output.unsqueeze(1) * p_star\n ret = (grad,)\n\n # pad with as many Nones as needed\n return ret + (None,) * (1 + cls.n_fwd_args)\n\n\nclass SparsemaxLossFunction(_GenericLossFunction):\n n_fwd_args = 1\n\n @classmethod\n def project(cls, X, alpha, k):\n return sparsemax(X, dim=-1, k=k)\n\n @classmethod\n def omega(cls, p_star, alpha):\n return (1 - (p_star**2).sum(dim=1)) / 2\n\n @classmethod\n def forward(cls, ctx, X, target, k=None):\n return super().forward(ctx, X, target, alpha=2, proj_args=dict(k=k))\n\n\nclass SparsemaxBisectLossFunction(_GenericLossFunction):\n n_fwd_args = 1\n\n @classmethod\n def project(cls, X, alpha, n_iter):\n return sparsemax_bisect(X, n_iter=n_iter)\n\n @classmethod\n def omega(cls, p_star, alpha):\n return (1 - (p_star**2).sum(dim=1)) / 2\n\n @classmethod\n def forward(cls, ctx, X, target, n_iter=50):\n return super().forward(ctx, X, target, alpha=2, proj_args=dict(n_iter=n_iter))\n\n\nclass Entmax15LossFunction(_GenericLossFunction):\n n_fwd_args = 1\n\n @classmethod\n def project(cls, X, alpha, k=None):\n return entmax15(X, dim=-1, k=k)\n\n @classmethod\n def omega(cls, p_star, alpha):\n return (1 - (p_star * torch.sqrt(p_star)).sum(dim=1)) / 0.75\n\n @classmethod\n def forward(cls, ctx, X, target, k=None):\n return super().forward(ctx, X, target, alpha=1.5, proj_args=dict(k=k))\n\n\nclass EntmaxBisectLossFunction(_GenericLossFunction):\n n_fwd_args = 2\n\n @classmethod\n def project(cls, X, alpha, n_iter):\n return entmax_bisect(X, alpha=alpha, n_iter=n_iter, ensure_sum_one=True)\n\n @classmethod\n def omega(cls, p_star, alpha):\n return (1 - (p_star**alpha).sum(dim=1)) / (alpha * (alpha - 1))\n\n @classmethod\n def forward(cls, ctx, X, target, alpha=1.5, n_iter=50):\n return super().forward(ctx, X, target, alpha, proj_args=dict(n_iter=n_iter))\n\n\ndef sparsemax_loss(X, target, k=None):\n \"\"\"sparsemax loss: sparse alternative to cross-entropy.\n\n Computed using a partial sorting strategy.\n\n Parameters\n ----------\n X : torch.Tensor, shape=(n_samples, n_classes)\n The input 2D tensor of predicted scores\n\n target : torch.LongTensor, shape=(n_samples,)\n The ground truth labels, 0 <= target < n_classes.\n\n k : int or None\n number of largest elements to partial-sort over. For optimal\n performance, should be slightly bigger than the expected number of\n nonzeros in the solution. If the solution is more than k-sparse,\n this function is recursively called with a 2*k schedule.\n If `None`, full sorting is performed from the beginning.\n\n Returns\n -------\n losses, torch.Tensor, shape=(n_samples,)\n The loss incurred at each sample.\n \"\"\"\n return SparsemaxLossFunction.apply(X, target, k)\n\n\ndef sparsemax_bisect_loss(X, target, n_iter=50):\n \"\"\"sparsemax loss: sparse alternative to cross-entropy.\n\n Computed using bisection.\n\n Parameters\n ----------\n X : torch.Tensor, shape=(n_samples, n_classes)\n The input 2D tensor of predicted scores\n\n target : torch.LongTensor, shape=(n_samples,)\n The ground truth labels, 0 <= target < n_classes.\n\n n_iter : int\n Number of bisection iterations. For float32, 24 iterations should\n suffice for machine precision.\n\n Returns\n -------\n losses, torch.Tensor, shape=(n_samples,)\n The loss incurred at each sample.\n \"\"\"\n return SparsemaxBisectLossFunction.apply(X, target, n_iter)\n\n\ndef entmax15_loss(X, target, k=None):\n \"\"\"1.5-entmax loss: sparse alternative to cross-entropy\n\n Computed using a partial sorting strategy.\n\n Parameters\n ----------\n X : torch.Tensor, shape=(n_samples, n_classes)\n The input 2D tensor of predicted scores\n\n target : torch.LongTensor, shape=(n_samples,)\n The ground truth labels, 0 <= target < n_classes.\n\n k : int or None\n number of largest elements to partial-sort over. For optimal\n performance, should be slightly bigger than the expected number of\n nonzeros in the solution. If the solution is more than k-sparse,\n this function is recursively called with a 2*k schedule.\n If `None`, full sorting is performed from the beginning.\n\n Returns\n -------\n losses, torch.Tensor, shape=(n_samples,)\n The loss incurred at each sample.\n \"\"\"\n return Entmax15LossFunction.apply(X, target, k)\n\n\ndef entmax_bisect_loss(X, target, alpha=1.5, n_iter=50):\n \"\"\"alpha-entmax loss: sparse alternative to cross-entropy.\n\n Computed using bisection, supporting arbitrary alpha > 1.\n\n Parameters\n ----------\n X : torch.Tensor, shape=(n_samples, n_classes)\n The input 2D tensor of predicted scores\n\n target : torch.LongTensor, shape=(n_samples,)\n The ground truth labels, 0 <= target < n_classes.\n\n alpha : float or torch.Tensor\n Tensor of alpha parameters (> 1) to use for each row of X. If scalar\n or python float, the same value is used for all rows. A value of\n alpha=2 corresponds to sparsemax, and alpha=1 would in theory recover\n softmax. For numeric reasons, this algorithm does not work with `alpha=1`:\n if you want softmax, we recommend `torch.nn.softmax`\n\n n_iter : int\n Number of bisection iterations. For float32, 24 iterations should\n suffice for machine precision.\n\n Returns\n -------\n losses, torch.Tensor, shape=(n_samples,)\n The loss incurred at each sample.\n \"\"\"\n return EntmaxBisectLossFunction.apply(X, target, alpha, n_iter)\n\n\nclass SparsemaxBisectLoss(_GenericLoss):\n def __init__(self, n_iter=50, ignore_index=IGNORE_INDEX_TOKEN_ID, reduction=\"elementwise_mean\"):\n self.n_iter = n_iter\n super().__init__(ignore_index, reduction)\n\n def loss(self, X, target):\n return sparsemax_bisect_loss(X, target, self.n_iter)\n\n\nclass SparsemaxLoss(_GenericLoss):\n def __init__(self, k=None, ignore_index=IGNORE_INDEX_TOKEN_ID, reduction=\"elementwise_mean\"):\n self.k = k\n super().__init__(ignore_index, reduction)\n\n def loss(self, X, target):\n return sparsemax_loss(X, target, self.k)\n\n\nclass EntmaxBisectLoss(_GenericLoss):\n def __init__(\n self,\n alpha=1.5,\n n_iter=50,\n ignore_index=IGNORE_INDEX_TOKEN_ID,\n reduction=\"elementwise_mean\",\n ):\n self.alpha = alpha\n self.n_iter = n_iter\n super().__init__(ignore_index, reduction)\n\n def loss(self, X, target):\n return entmax_bisect_loss(X, target, self.alpha, self.n_iter)\n\n\nclass Entmax15Loss(_GenericLoss):\n def __init__(self, k=100, ignore_index=IGNORE_INDEX_TOKEN_ID, reduction=\"elementwise_mean\"):\n self.k = k\n super().__init__(ignore_index, reduction)\n\n def loss(self, X, target):\n return entmax15_loss(X, target, self.k)\n"
},
{
"path": "ludwig/utils/entmax/root_finding.py",
"content": "\"\"\"Bisection implementation of alpha-entmax (Peters et al., 2019).\n\nBackward pass wrt alpha per (Correia et al., 2019). See https://arxiv.org/pdf/1905.05702 for detailed description.\n\"\"\"\n\n# Author: Goncalo M Correia\n# Author: Ben Peters\n# Author: Vlad Niculae \n\nimport torch\nimport torch.nn as nn\nfrom torch.autograd import Function\n\n\nclass EntmaxBisectFunction(Function):\n @classmethod\n def _gp(cls, x, alpha):\n return x ** (alpha - 1)\n\n @classmethod\n def _gp_inv(cls, y, alpha):\n return y ** (1 / (alpha - 1))\n\n @classmethod\n def _p(cls, X, alpha):\n return cls._gp_inv(torch.clamp(X, min=0), alpha)\n\n @classmethod\n def forward(cls, ctx, X, alpha=1.5, dim=-1, n_iter=50, ensure_sum_one=True):\n p_m, backward_kwargs = _entmax_bisect_forward(X, alpha, dim, n_iter, ensure_sum_one, cls)\n\n ctx.alpha = backward_kwargs[\"alpha\"]\n ctx.dim = backward_kwargs[\"dim\"]\n ctx.save_for_backward(p_m)\n return p_m\n\n @classmethod\n def backward(cls, ctx, dY):\n (Y,) = ctx.saved_tensors\n\n gppr = torch.where(Y > 0, Y ** (2 - ctx.alpha), Y.new_zeros(1))\n\n dX = dY * gppr\n q = dX.sum(ctx.dim) / gppr.sum(ctx.dim)\n q = q.unsqueeze(ctx.dim)\n dX -= q * gppr\n\n d_alpha = None\n if ctx.needs_input_grad[1]:\n # alpha gradient computation\n # d_alpha = (partial_y / partial_alpha) * dY\n # NOTE: ensure alpha is not close to 1\n # since there is an indetermination\n # batch_size, _ = dY.shape\n\n # shannon terms\n S = torch.where(Y > 0, Y * torch.log(Y), Y.new_zeros(1))\n # shannon entropy\n ent = S.sum(ctx.dim).unsqueeze(ctx.dim)\n Y_skewed = gppr / gppr.sum(ctx.dim).unsqueeze(ctx.dim)\n\n d_alpha = dY * (Y - Y_skewed) / ((ctx.alpha - 1) ** 2)\n d_alpha -= dY * (S - Y_skewed * ent) / (ctx.alpha - 1)\n d_alpha = d_alpha.sum(ctx.dim).unsqueeze(ctx.dim)\n\n return dX, d_alpha, None, None, None\n\n\ndef _entmax_bisect_forward(X, alpha, dim, n_iter, ensure_sum_one, cls=EntmaxBisectFunction):\n if not isinstance(alpha, torch.Tensor):\n alpha = torch.tensor(alpha, dtype=X.dtype, device=X.device)\n\n alpha_shape = list(X.shape)\n alpha_shape[dim] = 1\n alpha = alpha.expand(*alpha_shape)\n\n d = X.shape[dim]\n\n max_val, _ = X.max(dim=dim, keepdim=True)\n X = X * (alpha - 1)\n max_val = max_val * (alpha - 1)\n\n # Note: when alpha < 1, tau_lo > tau_hi. This still works since dm < 0.\n tau_lo = max_val - cls._gp(1, alpha)\n tau_hi = max_val - cls._gp(1 / d, alpha)\n\n f_lo = cls._p(X - tau_lo, alpha).sum(dim) - 1\n\n dm = tau_hi - tau_lo\n\n for it in range(n_iter):\n dm /= 2\n tau_m = tau_lo + dm\n p_m = cls._p(X - tau_m, alpha)\n f_m = p_m.sum(dim) - 1\n\n mask = (f_m * f_lo >= 0).unsqueeze(dim)\n tau_lo = torch.where(mask, tau_m, tau_lo)\n\n if ensure_sum_one:\n p_m /= p_m.sum(dim=dim).unsqueeze(dim=dim)\n\n return p_m, {\"alpha\": alpha, \"dim\": dim}\n\n\n# slightly more efficient special case for sparsemax\nclass SparsemaxBisectFunction(EntmaxBisectFunction):\n @classmethod\n def _gp(cls, x, alpha):\n return x\n\n @classmethod\n def _gp_inv(cls, y, alpha):\n return y\n\n @classmethod\n def _p(cls, x, alpha):\n return torch.clamp(x, min=0)\n\n @classmethod\n def forward(cls, ctx, X, dim=-1, n_iter=50, ensure_sum_one=True):\n p_m, backward_kwargs = _sparsemax_bisect_forward(X, dim, n_iter, ensure_sum_one)\n\n ctx.alpha = backward_kwargs[\"alpha\"]\n ctx.dim = backward_kwargs[\"dim\"]\n ctx.save_for_backward(p_m)\n return p_m\n\n @classmethod\n def backward(cls, ctx, dY):\n (Y,) = ctx.saved_tensors\n gppr = (Y > 0).to(dtype=dY.dtype)\n dX = dY * gppr\n q = dX.sum(ctx.dim) / gppr.sum(ctx.dim)\n q = q.unsqueeze(ctx.dim)\n dX -= q * gppr\n return dX, None, None, None\n\n\ndef _sparsemax_bisect_forward(X, dim, n_iter, ensure_sum_one):\n return _entmax_bisect_forward(X, alpha=2, dim=dim, n_iter=50, ensure_sum_one=True, cls=SparsemaxBisectFunction)\n\n\ndef entmax_bisect(X, alpha=1.5, dim=-1, n_iter=50, ensure_sum_one=True, training=True):\n \"\"\"alpha-entmax: normalizing sparse transform (a la softmax).\n\n Solves the optimization problem:\n\n max_p - H_a(p) s.t. p >= 0, sum(p) == 1.\n\n where H_a(p) is the Tsallis alpha-entropy with custom alpha >= 1,\n using a bisection (root finding, binary search) algorithm.\n\n This function is differentiable with respect to both X and alpha.\n\n Parameters\n ----------\n X : torch.Tensor\n The input tensor.\n\n alpha : float or torch.Tensor\n Tensor of alpha parameters (> 1) to use. If scalar\n or python float, the same value is used for all rows, otherwise,\n it must have shape (or be expandable to)\n alpha.shape[j] == (X.shape[j] if j != dim else 1)\n A value of alpha=2 corresponds to sparsemax, and alpha=1 would in theory recover\n softmax. For numeric reasons, this algorithm does not work with `alpha=1`: if you\n want softmax, we recommend `torch.nn.softmax`.\n\n dim : int\n The dimension along which to apply alpha-entmax.\n\n n_iter : int\n Number of bisection iterations. For float32, 24 iterations should\n suffice for machine precision.\n\n ensure_sum_one : bool,\n Whether to divide the result by its sum. If false, the result might\n sum to close but not exactly 1, which might cause downstream problems.\n\n Returns\n -------\n P : torch tensor, same shape as X\n The projection result, such that P.sum(dim=dim) == 1 elementwise.\n \"\"\"\n # Avoids call to custom autograd.Function during eval to ensure torchscript compatibility\n # custom autograd.Function is not scriptable: https://github.com/pytorch/pytorch/issues/22329#issuecomment-506608053\n if not training:\n output, _ = _entmax_bisect_forward(X, alpha, dim, n_iter, ensure_sum_one)\n return output\n return EntmaxBisectFunction.apply(X, alpha, dim, n_iter, ensure_sum_one)\n\n\ndef sparsemax_bisect(X, dim=-1, n_iter=50, ensure_sum_one=True, training=True):\n \"\"\"sparsemax: normalizing sparse transform (a la softmax), via bisection.\n\n Solves the projection:\n\n min_p ||x - p||_2 s.t. p >= 0, sum(p) == 1.\n\n Parameters\n ----------\n X : torch.Tensor\n The input tensor.\n\n dim : int\n The dimension along which to apply sparsemax.\n\n n_iter : int\n Number of bisection iterations. For float32, 24 iterations should\n suffice for machine precision.\n\n ensure_sum_one : bool,\n Whether to divide the result by its sum. If false, the result might\n sum to close but not exactly 1, which might cause downstream problems.\n\n Note: This function does not yet support normalizing along anything except\n the last dimension. Please use transposing and views to achieve more\n general behavior.\n\n Returns\n -------\n P : torch tensor, same shape as X\n The projection result, such that P.sum(dim=dim) == 1 elementwise.\n \"\"\"\n # Avoids call to custom autograd.Function during eval to ensure torchscript compatibility\n # custom autograd.Function is not scriptable: https://github.com/pytorch/pytorch/issues/22329#issuecomment-506608053\n if not training:\n output, _ = _sparsemax_bisect_forward(X, dim, n_iter, ensure_sum_one)\n return output\n return SparsemaxBisectFunction.apply(X, dim, n_iter, ensure_sum_one)\n\n\nclass SparsemaxBisect(nn.Module):\n def __init__(self, dim=-1, n_iter=None):\n \"\"\"sparsemax: normalizing sparse transform (a la softmax) via bisection\n\n Solves the projection:\n\n min_p ||x - p||_2 s.t. p >= 0, sum(p) == 1.\n\n Parameters\n ----------\n dim : int\n The dimension along which to apply sparsemax.\n\n n_iter : int\n Number of bisection iterations. For float32, 24 iterations should\n suffice for machine precision.\n \"\"\"\n self.dim = dim\n self.n_iter = n_iter\n super().__init__()\n\n def forward(self, X):\n return sparsemax_bisect(X, dim=self.dim, n_iter=self.n_iter, training=self.training)\n\n\nclass EntmaxBisect(nn.Module):\n def __init__(self, alpha=1.5, dim=-1, n_iter=50):\n \"\"\"alpha-entmax: normalizing sparse map (a la softmax) via bisection.\n\n Solves the optimization problem:\n\n max_p - H_a(p) s.t. p >= 0, sum(p) == 1.\n\n where H_a(p) is the Tsallis alpha-entropy with custom alpha >= 1,\n using a bisection (root finding, binary search) algorithm.\n\n Parameters\n ----------\n alpha : float or torch.Tensor\n Tensor of alpha parameters (> 1) to use. If scalar\n or python float, the same value is used for all rows, otherwise,\n it must have shape (or be expandable to)\n alpha.shape[j] == (X.shape[j] if j != dim else 1)\n A value of alpha=2 corresponds to sparsemax; and alpha=1 would in theory recover\n softmax. For numeric reasons, this algorithm does not work with `alpha=1`; if you\n want softmax, we recommend `torch.nn.softmax`.\n\n dim : int\n The dimension along which to apply alpha-entmax.\n\n n_iter : int\n Number of bisection iterations. For float32, 24 iterations should\n suffice for machine precision.\n\n \"\"\"\n super().__init__()\n self.dim = dim\n self.n_iter = n_iter\n if isinstance(alpha, torch.Tensor):\n self.register_buffer(\"alpha\", alpha)\n else:\n self.alpha = alpha\n\n def forward(self, X):\n return entmax_bisect(X, alpha=self.alpha, dim=self.dim, n_iter=self.n_iter, training=self.training)\n"
},
{
"path": "ludwig/utils/error_handling_utils.py",
"content": "import logging\nfrom functools import partial\n\nfrom retry.api import retry, retry_call\n\nimport ludwig.constants as const\n\nlogger = logging.getLogger(__name__)\n\n\ndefault_retry_call = partial(\n retry_call, tries=const.TRIES, backoff=const.BACKOFF, delay=const.DELAY, jitter=const.JITTER, logger=logger\n)\n\n\ndefault_retry = partial(\n retry, tries=const.TRIES, backoff=const.BACKOFF, delay=const.DELAY, jitter=const.JITTER, logger=logger\n)\n"
},
{
"path": "ludwig/utils/eval_utils.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nfrom collections import OrderedDict\n\nimport numpy as np\nfrom sklearn import metrics\nfrom sklearn.metrics import confusion_matrix\n\nlogger = logging.getLogger(__name__)\n\n\nclass ConfusionMatrix:\n def __init__(self, conditions, predictions, labels=None, sample_weight=None):\n # assert (len(predictions) == len(conditions))\n min_length = min(len(predictions), len(conditions))\n self.predictions = predictions[:min_length]\n self.conditions = conditions[:min_length]\n\n if labels is not None:\n self.label2idx = {label: idx for idx, label in enumerate(labels)}\n self.idx2label = {idx: label for idx, label in enumerate(labels)}\n labels = list(range(len(labels)))\n else:\n self.label2idx = {\n str(label): idx for idx, label in enumerate(np.unique([self.predictions, self.conditions]))\n }\n self.idx2label = {\n idx: str(label) for idx, label in enumerate(np.unique([self.predictions, self.conditions]))\n }\n self.cm = confusion_matrix(self.conditions, self.predictions, labels=labels, sample_weight=sample_weight)\n\n # if labels is not None:\n # self.labels_dict = {label: idx for idx, label in enumerate(labels)}\n # else:\n # if conditions.dtype.char == 'S': # it's an array of strings\n # self.labels_dict = {str(label): idx for idx, label in\n # enumerate(np.unique([predictions, conditions]))}\n # else: # number\n # max_label = np.concatenate([predictions, conditions]).max()\n # self.labels_dict = {str(i): i for i in range(max_label + 1)}\n # labels = [str(i) for i in range(max_label + 1)]\n # self.cm = confusion_matrix(conditions, predictions, labels, sample_weight)\n\n self.sum_predictions = np.sum(self.cm, axis=0)\n self.sum_conditions = np.sum(self.cm, axis=1)\n self.all = np.sum(self.cm)\n\n def label_to_idx(self, label):\n return self.label2idx[label]\n\n def true_positives(self, idx):\n return self.cm[idx, idx]\n\n def true_negatives(self, idx):\n return self.all - self.sum_predictions[idx] - self.sum_conditions[idx] + self.true_positives(idx)\n\n def false_positives(self, idx):\n return self.sum_predictions[idx] - self.true_positives(idx)\n\n def false_negatives(self, idx):\n return self.sum_conditions[idx] - self.true_positives(idx)\n\n def true_positive_rate(self, idx):\n nom = self.true_positives(idx)\n den = self.sum_conditions[idx]\n if den == 0 or den == np.nan:\n return 0\n else:\n return nom / den\n\n def true_negative_rate(self, idx):\n nom = tn = self.true_negatives(idx)\n den = tn + self.false_positives(idx)\n if den == 0 or den == np.nan:\n return 0\n else:\n return nom / den\n\n def positive_predictive_value(self, idx):\n nom = self.true_positives(idx)\n den = self.sum_predictions[idx]\n if den == 0 or den == np.nan:\n return 0\n else:\n return nom / den\n\n def negative_predictive_value(self, idx):\n nom = tn = self.true_negatives(idx)\n den = tn + self.false_negatives(idx)\n if den == 0 or den == np.nan:\n return 0\n else:\n return nom / den\n\n def false_negative_rate(self, idx):\n return 1.0 - self.true_positive_rate(idx)\n\n def false_positive_rate(self, idx):\n return 1.0 - self.true_negative_rate(idx)\n\n def false_discovery_rate(self, idx):\n return 1.0 - self.positive_predictive_value(idx)\n\n def false_omission_rate(self, idx):\n return 1.0 - self.negative_predictive_value(idx)\n\n def accuracy(self, idx):\n nom = self.true_positives(idx) + self.true_negatives(idx)\n den = self.all\n if den == 0 or den == np.nan:\n return 0\n else:\n return nom / den\n\n def precision(self, idx):\n return self.positive_predictive_value(idx)\n\n def recall(self, idx):\n return self.true_positive_rate(idx)\n\n def fbeta_score(self, beta, idx):\n beta_2 = np.power(beta, 2)\n precision = self.precision(idx)\n recall = self.recall(idx)\n nom = (1 + beta_2) * precision * recall\n den = (beta_2 * precision) + recall\n if den == 0 or den == np.nan:\n return 0\n else:\n return nom / den\n\n def f1_score(self, idx):\n return self.fbeta_score(1, idx)\n\n def sensitivity(self, idx):\n return self.true_positive_rate(idx)\n\n def specificity(self, idx):\n return self.true_negative_rate(idx)\n\n def hit_rate(self, idx):\n return self.true_positive_rate(idx)\n\n def miss_rate(self, idx):\n return self.false_negative_rate(idx)\n\n def fall_out(self, idx):\n return self.false_positive_rate(idx)\n\n def matthews_correlation_coefficient(self, idx):\n tp = self.true_positives(idx)\n tn = self.true_negatives(idx)\n fp = self.false_positives(idx)\n fn = self.false_negatives(idx)\n nom = tp * tn - fp * fn\n den = np.sqrt((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn))\n if den == 0 or den == np.nan:\n return 0\n else:\n return nom / den\n\n def informedness(self, idx):\n return self.true_positive_rate(idx) + self.true_negative_rate(idx) - 1\n\n def markedness(self, idx):\n return self.positive_predictive_value(idx) + self.negative_predictive_value(idx) - 1\n\n def token_accuracy(self):\n return metrics.accuracy_score(self.conditions, self.predictions)\n\n def avg_precision(self, average=\"macro\"):\n return metrics.precision_score(self.conditions, self.predictions, average=average)\n\n def avg_recall(self, average=\"macro\"):\n return metrics.recall_score(self.conditions, self.predictions, average=average)\n\n def avg_f1_score(self, average=\"macro\"):\n return metrics.f1_score(self.conditions, self.predictions, average=average)\n\n def avg_fbeta_score(self, beta, average=\"macro\"):\n return metrics.fbeta_score(self.conditions, self.predictions, beta=beta, average=average)\n\n def kappa_score(self):\n return metrics.cohen_kappa_score(self.conditions, self.predictions)\n\n def class_stats(self, idx):\n return {\n \"true_positives\": self.true_positives(idx),\n \"true_negatives\": self.true_negatives(idx),\n \"false_positives\": self.false_positives(idx),\n \"false_negatives\": self.false_negatives(idx),\n \"true_positive_rate\": self.true_positive_rate(idx),\n \"true_negative_rate\": self.true_negative_rate(idx),\n \"positive_predictive_value\": self.positive_predictive_value(idx),\n \"negative_predictive_value\": self.negative_predictive_value(idx),\n \"false_negative_rate\": self.false_negative_rate(idx),\n \"false_positive_rate\": self.false_positive_rate(idx),\n \"false_discovery_rate\": self.false_discovery_rate(idx),\n \"false_omission_rate\": self.false_omission_rate(idx),\n \"accuracy\": self.accuracy(idx),\n \"precision\": self.precision(idx),\n \"recall\": self.recall(idx),\n \"f1_score\": self.f1_score(idx),\n \"sensitivity\": self.sensitivity(idx),\n \"specificity\": self.specificity(idx),\n \"hit_rate\": self.hit_rate(idx),\n \"miss_rate\": self.miss_rate(idx),\n \"fall_out\": self.fall_out(idx),\n \"matthews_correlation_coefficient\": self.matthews_correlation_coefficient(idx),\n \"informedness\": self.informedness(idx),\n \"markedness\": self.markedness(idx),\n }\n\n def per_class_stats(self):\n stats = OrderedDict()\n for idx in sorted(self.idx2label.keys()):\n stats[self.idx2label[idx]] = self.class_stats(idx)\n return stats\n\n def stats(self):\n return {\n \"token_accuracy\": self.token_accuracy(),\n \"avg_precision_macro\": self.avg_precision(average=\"macro\"),\n \"avg_recall_macro\": self.avg_recall(average=\"macro\"),\n \"avg_f1_score_macro\": self.avg_f1_score(average=\"macro\"),\n \"avg_precision_micro\": self.avg_precision(average=\"micro\"),\n \"avg_recall_micro\": self.avg_recall(average=\"micro\"),\n \"avg_f1_score_micro\": self.avg_f1_score(average=\"micro\"),\n \"avg_precision_weighted\": self.avg_precision(average=\"micro\"),\n \"avg_recall_weighted\": self.avg_recall(average=\"micro\"),\n \"avg_f1_score_weighted\": self.avg_f1_score(average=\"weighted\"),\n \"kappa_score\": self.kappa_score(),\n }\n\n\ndef roc_curve(conditions, prediction_scores, pos_label=None, sample_weight=None):\n return metrics.roc_curve(conditions, prediction_scores, pos_label=pos_label, sample_weight=sample_weight)\n\n\ndef roc_auc_score(conditions, prediction_scores, average=\"micro\", sample_weight=None):\n try:\n return metrics.roc_auc_score(conditions, prediction_scores, average=average, sample_weight=sample_weight)\n except ValueError as ve:\n logger.info(ve)\n\n\ndef precision_recall_curve(conditions, prediction_scores, pos_label=None, sample_weight=None):\n return metrics.precision_recall_curve(\n conditions, prediction_scores, pos_label=pos_label, sample_weight=sample_weight\n )\n\n\ndef average_precision_score(conditions, prediction_scores, average=\"micro\", sample_weight=None):\n # average == [micro, macro, sampled, weidhted]\n return metrics.average_precision_score(conditions, prediction_scores, average=average, sample_weight=sample_weight)\n"
},
{
"path": "ludwig/utils/fs_utils.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2021 Linux Foundation.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nimport contextlib\nimport errno\nimport functools\nimport logging\nimport os\nimport pathlib\nimport shutil\nimport tempfile\nimport uuid\nfrom urllib.parse import unquote, urlparse\n\nimport certifi\nimport fsspec\nimport h5py\nimport pyarrow.fs\nimport urllib3\nfrom filelock import FileLock\nfrom fsspec.core import split_protocol\n\nfrom ludwig.api_annotations import DeveloperAPI\n\nlogger = logging.getLogger(__name__)\n\n\n@DeveloperAPI\ndef get_default_cache_location() -> str:\n \"\"\"Returns a path to the default LUDWIG_CACHE location, or $HOME/.ludwig_cache.\"\"\"\n cache_path = None\n if \"LUDWIG_CACHE\" in os.environ and os.environ[\"LUDWIG_CACHE\"]:\n cache_path = os.environ[\"LUDWIG_CACHE\"]\n else:\n cache_path = str(pathlib.Path.home().joinpath(\".ludwig_cache\"))\n\n # Check if the cache path exists, if not create it\n if not os.path.exists(cache_path):\n os.makedirs(cache_path)\n return cache_path\n\n\n@DeveloperAPI\ndef get_fs_and_path(url):\n protocol, path = split_protocol(url)\n # Parse the url to get only the escaped url path\n path = unquote(urlparse(path).path)\n # Create a windows compatible path from url path\n path = os.fspath(pathlib.PurePosixPath(path))\n fs = fsspec.filesystem(protocol)\n return fs, path\n\n\n@DeveloperAPI\ndef has_remote_protocol(url):\n protocol, _ = split_protocol(url)\n return protocol and protocol != \"file\"\n\n\n@DeveloperAPI\ndef is_http(urlpath):\n protocol, _ = split_protocol(urlpath)\n return protocol == \"http\" or protocol == \"https\"\n\n\n@DeveloperAPI\ndef upgrade_http(urlpath):\n protocol, url = split_protocol(urlpath)\n if protocol == \"http\":\n return \"https://\" + url\n return None\n\n\n@DeveloperAPI\n@functools.lru_cache(maxsize=32)\ndef get_bytes_obj_from_path(path: str) -> bytes | None:\n if is_http(path):\n try:\n return get_bytes_obj_from_http_path(path)\n except Exception as e:\n logger.warning(e)\n return None\n else:\n try:\n with open_file(path) as f:\n return f.read()\n except OSError as e:\n logger.warning(e)\n return None\n\n\n@DeveloperAPI\ndef stream_http_get_request(path: str) -> urllib3.response.HTTPResponse:\n if upgrade_http(path):\n http = urllib3.PoolManager()\n else:\n http = urllib3.PoolManager(ca_certs=certifi.where())\n resp = http.request(\"GET\", path, preload_content=False)\n return resp\n\n\n@DeveloperAPI\n@functools.lru_cache(maxsize=32)\ndef get_bytes_obj_from_http_path(path: str) -> bytes:\n resp = stream_http_get_request(path)\n if resp.status == 404:\n upgraded = upgrade_http(path)\n if upgraded:\n logger.info(f\"reading url {path} failed. upgrading to https and retrying\")\n return get_bytes_obj_from_http_path(upgraded)\n else:\n raise urllib3.exceptions.HTTPError(f\"reading url {path} failed and cannot be upgraded to https\")\n\n # stream data\n data = b\"\"\n for chunk in resp.stream(1024):\n data += chunk\n return data\n\n\n@DeveloperAPI\ndef find_non_existing_dir_by_adding_suffix(directory_name):\n fs, _ = get_fs_and_path(directory_name)\n suffix = 0\n curr_directory_name = directory_name\n while fs.exists(curr_directory_name):\n curr_directory_name = directory_name + \"_\" + str(suffix)\n suffix += 1\n return curr_directory_name\n\n\n@DeveloperAPI\ndef abspath(url):\n protocol, _ = split_protocol(url)\n if protocol is not None:\n # we assume any path containing an explicit protovol is fully qualified\n return url\n return os.path.abspath(url)\n\n\n@DeveloperAPI\ndef path_exists(url):\n fs, path = get_fs_and_path(url)\n return fs.exists(path)\n\n\n@DeveloperAPI\ndef listdir(url):\n fs, path = get_fs_and_path(url)\n return fs.listdir(path)\n\n\n@DeveloperAPI\ndef safe_move_file(src, dst):\n \"\"\"Rename a file from `src` to `dst`. Inspired by: https://alexwlchan.net/2019/03/atomic-cross-filesystem-\n moves-in-python/\n\n * Moves must be atomic. `shutil.move()` is not atomic.\n\n * Moves must work across filesystems. Sometimes temp directories and the\n model directories live on different filesystems. `os.replace()` will\n throw errors if run across filesystems.\n\n So we try `os.replace()`, but if we detect a cross-filesystem copy, we\n switch to `shutil.move()` with some wrappers to make it atomic.\n \"\"\"\n try:\n os.replace(src, dst)\n except OSError as err:\n if err.errno == errno.EXDEV:\n # Generate a unique ID, and copy `` to the target directory with a temporary name `..tmp`.\n # Because we're copying across a filesystem boundary, this initial copy may not be atomic. We insert a\n # random UUID so if different processes are copying into ``, they don't overlap in their tmp copies.\n copy_id = uuid.uuid4()\n tmp_dst = f\"{dst}.{copy_id}.tmp\"\n shutil.copyfile(src, tmp_dst)\n\n # Atomic replace file onto the new name, and clean up original source file.\n os.replace(tmp_dst, dst)\n os.unlink(src)\n else:\n raise\n\n\n@DeveloperAPI\ndef safe_move_directory(src, dst):\n \"\"\"Recursively moves files from src directory to dst directory and removes src directory.\n\n If dst directory does not exist, it will be created.\n \"\"\"\n try:\n os.replace(src, dst)\n except OSError as err:\n if err.errno == errno.EXDEV:\n # Generate a unique ID, and copy `` to the target directory with a temporary name `..tmp`.\n # Because we're copying across a filesystem boundary, this initial copy may not be atomic. We insert a\n # random UUID so if different processes are copying into ``, they don't overlap in their tmp copies.\n copy_id = uuid.uuid4()\n tmp_dst = f\"{dst}.{copy_id}.tmp\"\n shutil.copytree(src, tmp_dst)\n\n # Atomic replace directory name onto the new name, and clean up original source directory.\n os.replace(tmp_dst, dst)\n os.unlink(src)\n else:\n raise\n\n\n@DeveloperAPI\ndef rename(src, tgt):\n protocol, _ = split_protocol(tgt)\n if protocol is not None:\n fs = fsspec.filesystem(protocol)\n fs.mv(src, tgt, recursive=True)\n else:\n safe_move_file(src, tgt)\n\n\n@DeveloperAPI\ndef upload_file(src, tgt):\n protocol, _ = split_protocol(tgt)\n fs = fsspec.filesystem(protocol)\n fs.put(src, tgt)\n\n\n@DeveloperAPI\ndef copy(src, tgt, recursive=False):\n protocol, _ = split_protocol(tgt)\n fs = fsspec.filesystem(protocol)\n fs.copy(src, tgt, recursive=recursive)\n\n\n@DeveloperAPI\ndef makedirs(url, exist_ok=False):\n fs, path = get_fs_and_path(url)\n fs.makedirs(path, exist_ok=exist_ok)\n\n\n@DeveloperAPI\ndef delete(url, recursive=False):\n fs, path = get_fs_and_path(url)\n return fs.delete(path, recursive=recursive)\n\n\n@DeveloperAPI\ndef upload(lpath, rpath):\n fs, path = get_fs_and_path(rpath)\n pyarrow.fs.copy_files(lpath, path, destination_filesystem=pyarrow.fs.PyFileSystem(pyarrow.fs.FSSpecHandler(fs)))\n\n\n@DeveloperAPI\ndef download(rpath, lpath):\n fs, path = get_fs_and_path(rpath)\n pyarrow.fs.copy_files(path, lpath, source_filesystem=pyarrow.fs.PyFileSystem(pyarrow.fs.FSSpecHandler(fs)))\n\n\n@DeveloperAPI\ndef checksum(url):\n fs, path = get_fs_and_path(url)\n return fs.checksum(path)\n\n\n@DeveloperAPI\ndef to_url(path):\n protocol, _ = split_protocol(path)\n if protocol is not None:\n return path\n return pathlib.Path(os.path.abspath(path)).as_uri()\n\n\n@DeveloperAPI\n@contextlib.contextmanager\ndef upload_output_directory(url):\n if url is None:\n yield None, None\n return\n\n if has_remote_protocol(url):\n # To avoid extra network load, write all output files locally at runtime,\n # then upload to the remote fs at the end.\n with tempfile.TemporaryDirectory() as tmpdir:\n fs, remote_path = get_fs_and_path(url)\n\n # In cases where we are resuming from a previous run, we first need to download\n # the artifacts from the remote filesystem\n if path_exists(url):\n fs.get(url, tmpdir + \"/\", recursive=True)\n\n def put_fn():\n # Use pyarrow API here as fs.put() is inconsistent in where it uploads the file\n # See: https://github.com/fsspec/filesystem_spec/issues/1062\n pyarrow.fs.copy_files(\n tmpdir, remote_path, destination_filesystem=pyarrow.fs.PyFileSystem(pyarrow.fs.FSSpecHandler(fs))\n )\n\n # Write to temp directory locally\n yield tmpdir, put_fn\n\n # Upload to remote when finished\n put_fn()\n else:\n # For local paths (including file:// URIs), use the path directly.\n _, local_path = get_fs_and_path(url)\n makedirs(local_path, exist_ok=True)\n yield local_path, None\n\n\n@DeveloperAPI\n@contextlib.contextmanager\ndef open_file(url, *args, **kwargs):\n fs, path = get_fs_and_path(url)\n with fs.open(path, *args, **kwargs) as f:\n yield f\n\n\n@DeveloperAPI\n@contextlib.contextmanager\ndef download_h5(url):\n with tempfile.TemporaryDirectory() as tmpdir:\n local_path = os.path.join(tmpdir, os.path.basename(url))\n fs, path = get_fs_and_path(url)\n fs.get(path, local_path)\n with h5py.File(local_path, \"r\") as f:\n yield f\n\n\n@DeveloperAPI\n@contextlib.contextmanager\ndef upload_h5(url):\n with upload_output_file(url) as local_fname:\n mode = \"w\"\n if url == local_fname and path_exists(url):\n mode = \"r+\"\n\n with h5py.File(local_fname, mode) as f:\n yield f\n\n\n@DeveloperAPI\n@contextlib.contextmanager\ndef upload_output_file(url):\n \"\"\"Takes a remote URL as input, returns a temp filename, then uploads it when done.\"\"\"\n protocol, _ = split_protocol(url)\n if protocol is not None:\n fs = fsspec.filesystem(protocol)\n with tempfile.TemporaryDirectory() as tmpdir:\n local_fname = os.path.join(tmpdir, \"tmpfile\")\n yield local_fname\n fs.put(local_fname, url, recursive=True)\n else:\n yield url\n\n\n@DeveloperAPI\nclass file_lock(contextlib.AbstractContextManager):\n \"\"\"File lock based on filelock package.\"\"\"\n\n def __init__(self, path: str, ignore_remote_protocol: bool = True, lock_file: str = \".lock\") -> None:\n if not isinstance(path, (str, os.PathLike, pathlib.Path)):\n self.lock = None\n else:\n path = os.path.join(path, lock_file) if os.path.isdir(path) else f\"{path}./{lock_file}\"\n if ignore_remote_protocol and has_remote_protocol(path):\n self.lock = None\n else:\n self.lock = FileLock(path, timeout=-1)\n\n def __enter__(self, *args, **kwargs):\n if self.lock:\n return self.lock.__enter__(*args, **kwargs)\n\n def __exit__(self, *args, **kwargs):\n if self.lock:\n return self.lock.__exit__(*args, **kwargs)\n\n\n@DeveloperAPI\ndef list_file_names_in_directory(directory_name: str) -> list[str]:\n file_path: pathlib.Path # noqa [F842] # incorrect flagging of \"local variable is annotated but never used\"\n file_names: list[str] = [\n file_path.name for file_path in pathlib.Path(directory_name).iterdir() if file_path.is_file()\n ]\n return file_names\n"
},
{
"path": "ludwig/utils/h3_util.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nfrom typing import NamedTuple\n\n\nclass H3Data(NamedTuple):\n mode: int\n edge: int\n resolution: int\n base_cell: int\n cells: list[int]\n\n\ndef set_bit(v, index, x):\n \"\"\"Set the index:th bit of v to 1 if x is truthy, else to 0, and return the new value.\"\"\"\n mask = 1 << index # Compute mask, an integer with just bit 'index' set.\n v &= ~mask # Clear the bit indicated by the mask (if x is False)\n if x:\n v |= mask # If x was True, set the bit indicated by the mask.\n return v # Return the result, we're done.\n\n\ndef set_bits(v, start_bit, slice_length, x):\n bin_x = bin(x)\n for i, index in enumerate(range(start_bit, start_bit + slice_length)):\n val = int(bin_x[-(i + 1)]) if 2 + i < len(bin_x) else 0\n v = set_bit(v, index, val)\n return v\n\n\ndef components_to_h3(components):\n h3 = 18446744073709551615\n h3 = set_bits(h3, 64 - 5, 4, components[\"mode\"])\n h3 = set_bits(h3, 64 - 8, 3, components[\"edge\"])\n h3 = set_bits(h3, 64 - 12, 4, components[\"resolution\"])\n h3 = set_bits(h3, 64 - 19, 7, components[\"base_cell\"])\n for i, cell in enumerate(components[\"cells\"]):\n h3 = set_bits(h3, 64 - 19 - (i + 1) * 3, 3, cell)\n h3 = set_bits(h3, 64 - 1, 4, 0)\n return h3\n\n\ndef bitslice(x: int, start_bit: int, slice_length: int) -> int:\n ones_mask: int = int(2**slice_length - 1)\n return (x & (ones_mask << start_bit)) >> start_bit\n\n\ndef h3_index_mode(h3_long: int) -> int:\n return bitslice(h3_long, 64 - 5, 4)\n\n\ndef h3_edge(h3_long: int) -> int:\n return bitslice(h3_long, 64 - 8, 3)\n\n\ndef h3_resolution(h3_long: int) -> int:\n return bitslice(h3_long, 64 - 12, 4)\n\n\ndef h3_base_cell(h3_long: int) -> int:\n return bitslice(h3_long, 64 - 19, 7)\n\n\ndef h3_octal_components(h3_long):\n res = h3_resolution(h3_long)\n return \"{0:0{w}o}\".format(bitslice(h3_long + 2**63, 64 - 19 - 3 * res, 3 * res), w=res)\n\n\ndef h3_component(h3_long: int, i: int) -> int:\n return bitslice(h3_long, 64 - 19 - 3 * i, 3)\n\n\ndef h3_components(h3_long: int) -> list[int]:\n return [h3_component(h3_long, i) for i in range(1, h3_resolution(h3_long) + 1)]\n\n\ndef h3_to_components(h3_value: int) -> H3Data:\n \"\"\"Extract the values from an H3 hexadecimal value Refer to this for the bit layout:\n\n https://uber.github.io/h3/#/documentation/core-library/h3-index-representations\n \"\"\"\n # lat_long = (0, 0) # h3ToGeo(h3_value)\n return H3Data(\n mode=h3_index_mode(h3_value),\n edge=h3_edge(h3_value),\n resolution=h3_resolution(h3_value),\n base_cell=h3_base_cell(h3_value),\n cells=h3_components(h3_value),\n )\n\n\nif __name__ == \"__main__\":\n value = 622236723497533439\n components = h3_to_components(value)\n h3 = components_to_h3(components)\n components2 = h3_to_components(h3)\n print(value)\n print(components)\n print(h3)\n print(components2)\n"
},
{
"path": "ludwig/utils/heuristics.py",
"content": "from ludwig.schema.model_config import ModelConfig\nfrom ludwig.utils.config_utils import has_pretrained_encoder, has_trainable_encoder, has_unstructured_input_feature\n\n\ndef get_auto_learning_rate(config: ModelConfig) -> float:\n \"\"\"Uses config heuristics to determine an appropriate learning rate.\n\n The main idea behind the following heuristics is that smaller learning rates are more\n suitable for features with larger encoders, which are typically used with unstructured features.\n Note that these are meant to be rough heuristics that are solely based on feature types and the\n type of the corresponding encoder. More factors could be taken into consideration such as model\n size, dataset size, batch size, number of features, etc.\n\n Args:\n config: Ludwig config used to train the model.\n \"\"\"\n if not has_unstructured_input_feature(config):\n return 0.001\n\n if not has_pretrained_encoder(config):\n return 0.0001\n\n if has_trainable_encoder(config):\n return 0.00001\n\n return 0.00002\n"
},
{
"path": "ludwig/utils/hf_utils.py",
"content": "import logging\nimport os\nimport tempfile\nfrom os import PathLike\n\nfrom transformers import AutoTokenizer, PreTrainedModel\nfrom transformers.tokenization_utils import PreTrainedTokenizer\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.utils.error_handling_utils import default_retry\nfrom ludwig.utils.fs_utils import download, path_exists\nfrom ludwig.utils.upload_utils import hf_hub_login\n\nlogger = logging.getLogger(__name__)\n\n\n@default_retry()\ndef load_pretrained_hf_model_from_hub(\n model_class: type,\n pretrained_model_name_or_path: str | PathLike | None,\n **pretrained_kwargs,\n) -> PreTrainedModel:\n \"\"\"Download a HuggingFace model.\n\n Downloads a model from the HuggingFace zoo with retry on failure.\n Args:\n model_class: Class of the model to download.\n pretrained_model_name_or_path: Name of the model to download.\n pretrained_kwargs: Additional arguments to pass to the model constructor.\n Returns:\n The pretrained model object.\n \"\"\"\n return model_class.from_pretrained(pretrained_model_name_or_path, **pretrained_kwargs)\n\n\n@default_retry()\ndef load_pretrained_hf_tokenizer(\n pretrained_model_name_or_path: str | PathLike | None, **pretrained_kwargs\n) -> PreTrainedTokenizer:\n \"\"\"Download a HuggingFace tokenizer.\n\n Args:\n pretrained_model_name_or_path: Name of the tokenizer to download.\n pretrained_kwargs: Additional arguments to pass to the tokenizer constructor.\n Returns:\n The pretrained tokenizer object.\n \"\"\"\n return AutoTokenizer.from_pretrained(pretrained_model_name_or_path, **pretrained_kwargs)\n\n\ndef _load_pretrained_hf_model_from_dir(\n model_class: type,\n pretrained_model_name_or_path: str | PathLike | None,\n **pretrained_kwargs,\n) -> PreTrainedModel:\n \"\"\"Downloads a model to a local temporary directory, and Loads a pretrained HF model from a local directory.\"\"\"\n with tempfile.TemporaryDirectory() as tmpdir:\n download(pretrained_model_name_or_path, tmpdir)\n return model_class.from_pretrained(tmpdir, **pretrained_kwargs)\n\n\n@DeveloperAPI\ndef load_pretrained_hf_model_with_hub_fallback(\n model_class: type,\n pretrained_model_name_or_path: str | PathLike | None,\n **pretrained_kwargs,\n) -> tuple[PreTrainedModel, bool]:\n \"\"\"Returns the model and a boolean indicating whether the model was downloaded from the HuggingFace hub.\n\n If the `LUDWIG_PRETRAINED_MODELS_DIR` environment variable is set, we attempt to load the HF model from this\n directory, falling back to downloading from the HF hub if the model is not found, downloading fails, or if model\n initialization fails.\n\n `LUDWIG_PRETRAINED_MODELS_DIR` can be an s3 path. Weights are copied to a local temporary directory, and the model\n is loaded from there.\n\n The expected structure of the `LUDWIG_PRETRAINED_MODELS_DIR` directory is:\n {LUDWIG_PRETRAINED_MODELS_DIR}/{pretrained_model_name_or_path}/pytorch_model.bin\n {LUDWIG_PRETRAINED_MODELS_DIR}/{pretrained_model_name_or_path}/config.json\n\n For example, if `LUDWIG_PRETRAINED_MODELS_DIR` is set to `s3://my-bucket/pretrained-models`, and\n `pretrained_model_name_or_path` is set to `bert-base-uncased`, we expect to find the following files:\n s3://my-bucket/bert-base-uncased/\n - pytorch_model.bin\n - config.json\n\n If the `LUDWIG_PRETRAINED_MODELS_DIR` environment variable is not set, we download the model from the HF hub.\n \"\"\"\n pretrained_models_dir = os.environ.get(\"LUDWIG_PRETRAINED_MODELS_DIR\")\n if pretrained_models_dir:\n pretrained_model_path = os.path.join(pretrained_models_dir, pretrained_model_name_or_path)\n if path_exists(pretrained_model_path):\n try:\n logger.info(\n f\"Found existing pretrained model artifact {pretrained_model_name_or_path} in directory \"\n f\"{pretrained_models_dir}. Downloading.\"\n )\n return (\n _load_pretrained_hf_model_from_dir(model_class, pretrained_model_path, **pretrained_kwargs),\n False,\n )\n except Exception as e:\n logger.warning(\n f\"Failed to download pretrained model from {pretrained_models_dir} with error {e}. \"\n \"Falling back to HuggingFace model hub.\"\n )\n\n # Fallback to HF hub.\n return load_pretrained_hf_model_from_hub(model_class, pretrained_model_name_or_path, **pretrained_kwargs), True\n\n\ndef upload_folder_to_hfhub(\n repo_id: str,\n folder_path: str,\n repo_type: str | None = \"model\",\n private: bool | None = False,\n path_in_repo: str | None = None, # defaults to root of repo\n commit_message: str | None = None,\n commit_description: str | None = None,\n) -> None:\n \"\"\"Uploads a local folder to the Hugging Face Model Hub.\n\n Args:\n repo_id (str): The ID of the target repository on the Hugging Face Model Hub.\n folder_path (str): The local path to the folder to be uploaded.\n repo_type (str, optional): The type of the repository ('model', 'dataset', or 'space').\n Defaults to 'model'.\n private (bool, optional): If True, the repository will be private; otherwise, it will be public.\n Defaults to False.\n path_in_repo (str, optional): The relative path within the repository where the folder should be uploaded.\n Defaults to None, which means the root of the repository.\n commit_message (str, optional): A message for the commit associated with the upload.\n commit_description (str, optional): A description for the commit associated with the upload.\n\n Raises:\n FileNotFoundError: If the specified folder does not exist.\n ValueError: If the specified folder is empty, a file, or if an invalid 'repo_type' is provided.\n ValueError: If the upload process fails for any reason.\n\n Returns:\n None\n \"\"\"\n # Make sure the folder exists\n if not os.path.exists(folder_path):\n raise FileNotFoundError(f\"Folder {folder_path} does not exist.\")\n\n # Make sure the folder is not a file\n if os.path.isfile(folder_path):\n raise ValueError(f\"Folder {folder_path} is a file. Please provide a folder.\")\n\n # Make sure the folder is not empty\n if not os.listdir(folder_path):\n raise ValueError(f\"Folder {folder_path} is empty.\")\n\n if repo_type not in {\"model\", \"dataset\", \"space\"}:\n raise ValueError(f\"Invalid repo_type {repo_type}. Valid values are 'model', 'dataset', and 'space'.\")\n\n # Login to the hub\n api = hf_hub_login()\n\n # Create the repo if it doesn't exist. This is a no-op if the repo already exists\n # This is required because the API doesn't allow uploading to a non-existent repo\n if not api.repo_exists(repo_id, repo_type=repo_type):\n logger.info(f\"{repo_id} does not exist. Creating.\")\n api.create_repo(repo_id, private=private, exist_ok=True, repo_type=repo_type)\n\n # Upload the folder\n try:\n logger.info(f\"Uploading folder {folder_path} to repo {repo_id}.\")\n api.upload_folder(\n repo_id=repo_id,\n folder_path=folder_path,\n repo_type=repo_type,\n path_in_repo=path_in_repo,\n commit_message=commit_message,\n commit_description=commit_description,\n )\n except Exception as e:\n raise ValueError(f\"Failed to upload folder {folder_path} to repo {repo_id}\") from e\n"
},
{
"path": "ludwig/utils/html_utils.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nimport re\nfrom html.parser import HTMLParser\n\nfrom ludwig.utils import strings_utils\n\nlogger = logging.getLogger(__name__)\n\n\nclass HTMLStripper(HTMLParser):\n def __init__(self):\n super().__init__()\n self.reset()\n self.strict = False\n self.convert_charrefs = True\n self.fed = []\n\n def handle_data(self, data):\n self.fed.append(data)\n\n def get_data(self):\n return \"\".join(self.fed)\n\n def error(self, message):\n logger.error(message)\n\n\ndef strip_tags(html):\n stripper = HTMLStripper()\n stripper.feed(html)\n return stripper.get_data()\n\n\n# regular expressions for cleaning text\nres_pre = [(re.compile(r\"([^.:;\\?\\!>])( )\"), r\"\\1.\\2\"), (re.compile(r\" \"), r\" \")]\nres_post = [\n (re.compile(r\"[ \\t\\0]\"), r\" \"),\n (re.compile(r\"[–_]\"), r\"-\"),\n (\n re.compile(r\"[\\’\\‘]\"),\n r\"\"\"),\n (re.compile(r'[”“]]'), r\"\"\",\n ),\n (re.compile(r\"℅\"), r\"%\"),\n (re.compile(r\"([^.>])( )\"), r\"\\1.\\2\"),\n (re.compile(r\"\\\\\\\\[NnRr]\"), r\" \"),\n (re.compile(r\"\\\\[NnRr]\"), r\" \"),\n (re.compile(r\"[\\n\\r]\"), r\" \"),\n (re.compile(r\"\\\\\\\\\"), r\" / \"),\n (re.compile(r\" \"), r\" \"),\n (re.compile(r\"\\\\\\\\\" \"\"), r\"\\'\"),\n (re.compile(r\"^\\'([^\\']+)$\"), r\"\\1\"),\n (re.compile(r\"([\\<\\>\\{\\}\\[\\]\\(\\)\\-\\+\\=:;,\\./\\?\\!\\$%&£#@\\'₹ ])\\1+\"), r\"\\1\"),\n (\n re.compile(\n r\"[^qwertyuiopasdfghjklzxcvbnmQWERTYUIOPASDFGHJKLZXCVBNM1234567890\\<\\>\\{\\}\\[\\]\\(\\)\\-\\+\\=:;,\\./\\?\\!\\$%&£#@\\'₹ ]\" # noqa\n ),\n r\" \",\n ),\n (re.compile(r\"\\s{2,}\"), r\" \"),\n]\n\n\ndef clean_html(html_text):\n # print()\n # print(html_text)\n html_text, matched = strings_utils.match_replace(html_text, res_pre)\n # print(html_text)\n html_text = strip_tags(html_text)\n # print(html_text)\n html_text = strings_utils.strip_accents(html_text)\n # print(html_text)\n # result = html_text.strip(\n # 'qwertyuiopasdfghjklzxcvbnmQWERTYUIOPASDFGHJKLZXCVBNM1234567890\\<\\>\\{\\}\\[\\]\\(\\)\\-\\+\\=:;,\\./\\?\\!\\$%&€£#@'₹\\' ')\n # if result:\n # print(result)\n html_text, matched = strings_utils.match_replace(html_text, res_post)\n # print(matched)\n # print(html_text)\n return html_text\n"
},
{
"path": "ludwig/utils/image_utils.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nimport warnings\nfrom collections.abc import Callable, Iterable\nfrom dataclasses import dataclass\nfrom io import BytesIO\n\nimport numpy as np\nimport torch\nimport torchvision.transforms.functional as F\nfrom torchvision.io import decode_image, ImageReadMode\nfrom torchvision.models._api import WeightsEnum\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import CROP_OR_PAD, IMAGE_MAX_CLASSES, INTERPOLATE\nfrom ludwig.utils.data_utils import get_abs_path\nfrom ludwig.utils.fs_utils import get_bytes_obj_from_path\nfrom ludwig.utils.registry import Registry\n\n\n@dataclass\nclass TVModelVariant:\n # Model variant identifier\n variant_id: str | int\n\n # TorchVision function to create model class\n create_model_function: Callable\n\n # Torchvision class for model weights\n model_weights: WeightsEnum\n\n\nlogger = logging.getLogger(__name__)\n\nIMAGE_EXTENSIONS = (\".png\", \".jpg\", \".jpeg\", \".tiff\", \".tif\", \".bmp\", \".gif\")\n\n\n@DeveloperAPI\nclass ResizeChannels(torch.nn.Module):\n def __init__(self, num_channels: int):\n super().__init__()\n self.num_channels = num_channels\n\n def forward(self, imgs: torch.Tensor):\n original_imgs_shape = imgs.shape\n if len(original_imgs_shape) == 3: # if shape is (C, H, W), add batch dimension\n imgs = imgs.unsqueeze(0)\n\n channels = imgs.shape[1]\n if channels > self.num_channels:\n # take the first `self.num_channels` channels\n imgs = imgs[:, : self.num_channels, :, :]\n elif channels < self.num_channels:\n # repeat and use the first `self.num_channels` channels\n imgs = imgs.repeat(1, (self.num_channels // channels) + 1, 1, 1)[:, : self.num_channels, :, :]\n\n if len(original_imgs_shape) == 3: # if shape was (C, H, W), remove batch dimension\n return imgs[0]\n return imgs\n\n\n@DeveloperAPI\ndef get_gray_default_image(num_channels: int, height: int, width: int) -> np.ndarray:\n return np.full((num_channels, height, width), 128, dtype=np.float32)\n\n\n@DeveloperAPI\ndef get_average_image(image_lst: list[np.ndarray]) -> np.array:\n return np.mean([x for x in image_lst if x is not None], axis=(0), dtype=np.float32)\n\n\n@DeveloperAPI\ndef is_bytes_image(bytes_obj) -> bool:\n \"\"\"Check if a bytes object is an image using PIL.\"\"\"\n try:\n from io import BytesIO\n\n from PIL import Image\n\n if isinstance(bytes_obj, bytes):\n bytes_obj = BytesIO(bytes_obj)\n Image.open(bytes_obj).verify()\n return True\n except Exception:\n return False\n\n\ndef is_image(src_path: str, img_entry: bytes | str, column: str) -> bool:\n if not isinstance(img_entry, str):\n return False\n try:\n from io import BytesIO\n\n from PIL import Image\n\n path = get_abs_path(src_path, img_entry)\n bytes_obj = get_bytes_obj_from_path(path)\n if isinstance(bytes_obj, bytes):\n bytes_obj = BytesIO(bytes_obj)\n Image.open(bytes_obj).verify()\n return True\n except Exception as e:\n logger.warning(f\"While assessing potential image in is_image() for column {column}, encountered exception: {e}\")\n return False\n\n\n@DeveloperAPI\ndef is_image_score(path):\n return int(isinstance(path, str) and path.lower().endswith(IMAGE_EXTENSIONS))\n\n\n@DeveloperAPI\ndef get_image_read_mode_from_num_channels(num_channels: int) -> ImageReadMode:\n \"\"\"Returns the torchvision.io.ImageReadMode corresponding to the number of channels.\n\n If num_channels is not recognized, returns ImageReadMode.UNCHANGED.\n \"\"\"\n mode = ImageReadMode.UNCHANGED\n if num_channels == 1:\n mode = ImageReadMode.GRAY\n elif num_channels == 2:\n mode = ImageReadMode.GRAY_ALPHA\n elif num_channels == 3:\n mode = ImageReadMode.RGB\n elif num_channels == 4:\n mode = ImageReadMode.RGB_ALPHA\n return mode\n\n\n@DeveloperAPI\ndef read_image_from_path(\n path: str, num_channels: int | None = None, return_num_bytes=False\n) -> torch.Tensor | None | tuple[torch.Tensor | None, int]:\n \"\"\"Reads image from path.\n\n Useful for reading from a small number of paths. For more intensive reads, use backend.read_binary_files instead. If\n `return_num_bytes` is True, returns a tuple of (image, num_bytes).\n \"\"\"\n bytes_obj = get_bytes_obj_from_path(path)\n image = read_image_from_bytes_obj(bytes_obj, num_channels)\n if return_num_bytes:\n if bytes_obj is not None:\n num_bytes = len(bytes_obj)\n else:\n num_bytes = None\n return image, num_bytes\n else:\n return image\n\n\n@DeveloperAPI\ndef read_image_from_bytes_obj(bytes_obj: bytes | None = None, num_channels: int | None = None) -> torch.Tensor | None:\n \"\"\"Tries to read image as a tensor from the path.\n\n If the path is not decodable as a PNG, attempts to read as a numpy file. If neither of these work, returns None.\n \"\"\"\n if bytes_obj is None:\n return None\n mode = get_image_read_mode_from_num_channels(num_channels)\n\n image = read_image_as_png(bytes_obj, mode)\n if image is None:\n image = read_image_as_numpy(bytes_obj)\n if image is None:\n image = read_image_as_tif(bytes_obj)\n if image is None:\n warnings.warn(\"Unable to read image from bytes object.\")\n return image\n\n\n@DeveloperAPI\ndef read_image_as_png(bytes_obj: bytes, mode: ImageReadMode = ImageReadMode.UNCHANGED) -> torch.Tensor | None:\n \"\"\"Reads image from bytes object from a PNG file.\"\"\"\n try:\n with BytesIO(bytes_obj) as buffer:\n buffer_view = buffer.getbuffer()\n if len(buffer_view) == 0:\n del buffer_view\n raise Exception(\"Bytes object is empty. This could be due to a failed load from storage.\")\n image = decode_image(torch.frombuffer(buffer_view, dtype=torch.uint8), mode=mode)\n del buffer_view\n return image\n except Exception as e:\n warnings.warn(f\"Failed to read image from PNG file. Original exception: {e}\")\n return None\n\n\n@DeveloperAPI\ndef read_image_as_numpy(bytes_obj: bytes) -> torch.Tensor | None:\n \"\"\"Reads image from bytes object from a numpy file.\"\"\"\n try:\n with BytesIO(bytes_obj) as buffer:\n image = np.load(buffer)\n return torch.from_numpy(image)\n except Exception as e:\n warnings.warn(f\"Failed to read image from numpy file. Original exception: {e}\")\n return None\n\n\n@DeveloperAPI\ndef read_image_as_tif(bytes_obj: bytes) -> torch.Tensor | None:\n \"\"\"Reads image from bytes object from a tif file.\"\"\"\n try:\n import tifffile\n\n with BytesIO(bytes_obj) as buffer:\n image = tifffile.imread(buffer)\n if image.dtype == np.uint16:\n image = image.astype(np.int32)\n image = torch.from_numpy(image)\n if len(image.shape) == 2:\n image = torch.unsqueeze(image, dim=0)\n return image\n except Exception as e:\n warnings.warn(f\"Failed to read image from tif file. Original exception: {e}\")\n return None\n\n\n@DeveloperAPI\ndef pad(\n img: torch.Tensor,\n new_size: int | tuple[int, int],\n) -> torch.Tensor:\n \"\"\"Torchscript-compatible implementation of pad.\n\n Args:\n img (torch.Tensor): image with shape [..., height, width] to pad\n new_size (Union[int, Tuple[int, int]]): size to pad to. If int, resizes to square image of that size.\n\n Returns:\n torch.Tensor: padded image of size [..., size[0], size[1]] or [..., size, size] if size is int.\n \"\"\"\n new_size = to_tuple(new_size)\n old_size = img.shape[-2:]\n pad_size = (torch.tensor(new_size) - torch.tensor(old_size)) / 2\n padding = torch.cat((torch.floor(pad_size), torch.ceil(pad_size)))\n padding[padding < 0] = 0\n padding = [int(x) for x in padding]\n return F.pad(img, padding=padding, padding_mode=\"edge\")\n\n\n@DeveloperAPI\ndef crop(\n img: torch.Tensor,\n new_size: int | tuple[int, int],\n) -> torch.Tensor:\n \"\"\"Torchscript-compatible implementation of crop.\n\n Args:\n img (torch.Tensor): image with shape [..., height, width] to crop\n size (Union[int, Tuple[int, int]]): size to crop to. If int, crops to square image of that size.\n\n Returns:\n torch.Tensor: cropped image of size [..., size[0], size[1]] or [..., size, size] if size is int.\n \"\"\"\n new_size = to_tuple(new_size)\n return F.center_crop(img, output_size=new_size)\n\n\n@DeveloperAPI\ndef crop_or_pad(img: torch.Tensor, new_size: int | tuple[int, int]):\n \"\"\"Torchscript-compatible implementation of resize using constants.CROP_OR_PAD.\n\n Args:\n img (torch.Tensor): image with shape [..., height, width] to resize\n new_size (Union[int, Tuple[int, int]]): size to resize to. If int, resizes to square image of that size.\n\n Returns:\n torch.Tensor: resized image of size [..., size[0], size[1]] or [..., size, size] if size is int.\n \"\"\"\n new_size = to_tuple(new_size)\n if list(new_size) == list(img.shape[-2:]):\n return img\n img = pad(img, new_size)\n img = crop(img, new_size)\n return img\n\n\n@DeveloperAPI\ndef resize_image(\n img: torch.Tensor,\n new_size: int | tuple[int, int],\n resize_method: str,\n crop_or_pad_constant: str = CROP_OR_PAD,\n interpolate_constant: str = INTERPOLATE,\n) -> torch.Tensor:\n \"\"\"Torchscript-compatible implementation of resize.\n\n Args:\n img (torch.Tensor): image with shape [..., height, width] to resize\n new_size (Union[int, Tuple[int, int]]): size to resize to. If int, resizes to square image of that size.\n resize_method (str): method to use for resizing. Either constants.CROP_OR_PAD or constants.INTERPOLATE.\n\n Returns:\n torch.Tensor: resized image of size [..., size[0], size[1]] or [..., size, size] if size is int.\n \"\"\"\n new_size = to_tuple(new_size)\n if list(img.shape[-2:]) != list(new_size):\n if resize_method == crop_or_pad_constant:\n return crop_or_pad(img, new_size)\n elif resize_method == interpolate_constant:\n return F.resize(img, new_size)\n raise ValueError(f\"Invalid image resize method: {resize_method}\")\n return img\n\n\n@DeveloperAPI\ndef grayscale(img: torch.Tensor) -> torch.Tensor:\n \"\"\"Grayscales RGB image.\"\"\"\n return F.rgb_to_grayscale(img)\n\n\n@DeveloperAPI\ndef num_channels_in_image(img: torch.Tensor):\n \"\"\"Returns number of channels in image.\"\"\"\n if img is None or img.ndim < 2:\n raise ValueError(\"Invalid image data\")\n\n if img.ndim == 2:\n return 1\n else:\n return img.shape[0]\n\n\n@DeveloperAPI\ndef get_unique_channels(\n image_sample: list[torch.Tensor],\n num_channels: int,\n num_classes: int = None,\n) -> torch.Tensor:\n \"\"\"Returns a tensor of unique channel values from a list of images.\n Args:\n image_sample: A list of images of dimensions [C x H x W] or [H x W], where C is the channel dimension\n num_channels: The expected number of channels\n num_classes: The expected number of classes or None\n\n Return:\n channel_class_map: A tensor mapping channel values to classes, where dim=0 is the class.\n \"\"\"\n n_images = 0\n no_new_class = 0\n channel_class_map = None\n for img in image_sample:\n if img.ndim < 2:\n raise ValueError(\"Invalid image dimensions {img.ndim}\")\n if img.ndim == 2:\n img = img.unsqueeze(0)\n if num_channels == 1 and num_channels_in_image(img) != 1:\n img = grayscale(img)\n if num_classes == 2 and num_channels_in_image(img) == 1:\n img = img.type(torch.float32) / 255\n img = img.round() * 255\n img = img.type(torch.uint8)\n\n img = img.flatten(1, 2)\n img = img.permute(1, 0)\n uniq_chans = img.unique(dim=0)\n\n if channel_class_map is None:\n channel_class_map = uniq_chans\n else:\n channel_class_map = torch.concat((channel_class_map, uniq_chans)).unique(dim=0)\n if channel_class_map.shape[0] > IMAGE_MAX_CLASSES:\n raise ValueError(\n f\"Images inferred num classes {channel_class_map.shape[0]} exceeds \" f\"max classes {IMAGE_MAX_CLASSES}.\"\n )\n\n n_images += 1\n if n_images % 25 == 0:\n logger.info(f\"Processed the first {n_images} images inferring {channel_class_map.shape[0]} classes...\")\n\n if channel_class_map.shape[0] == uniq_chans.shape[0]:\n no_new_class += 1\n if no_new_class >= 4 and channel_class_map.shape[0] == num_classes:\n break # early loop exit\n else:\n no_new_class = 0\n\n logger.info(f\"Inferred {channel_class_map.shape[0]} classes from the first {n_images} images.\")\n return channel_class_map.type(torch.uint8)\n\n\n@DeveloperAPI\ndef get_class_mask_from_image(\n channel_class_map: torch.Tensor,\n img: torch.Tensor,\n) -> torch.Tensor:\n \"\"\"Returns a masked image where each mask value is the channel class of the input.\n Args:\n channel_class_map: A tensor mapping channel values to classes, where dim=0 is the class.\n img: An input image of dimensions [C x H x W] or [H x W], where C is the channel dimension\n\n Return:\n [mask] A masked image of dimensions [H x W] where each value is the channel class of the input\n \"\"\"\n num_classes = channel_class_map.shape[0]\n mask = torch.full((img.shape[-2], img.shape[-1]), num_classes, dtype=torch.uint8)\n if img.ndim == 2:\n img = img.unsqueeze(0)\n if num_classes == 2 and num_channels_in_image(img) == 1:\n img = img.type(torch.float32) / 255\n img = img.round() * 255\n img = img.type(torch.uint8)\n img = img.permute(1, 2, 0)\n for nclass, value in enumerate(channel_class_map):\n mask[(img == value).all(-1)] = nclass\n\n if torch.any(mask.ge(num_classes)):\n raise ValueError(\n f\"Image channel could not be mapped to a class because an unknown channel value was detected. \"\n f\"{num_classes} classes were inferred from the first set of images. This image has a channel \"\n f\"value that was not previously seen in the first set of images. Check preprocessing parameters \"\n f\"for image resizing, num channels, num classes and num samples. Image resizing may affect \"\n f\"channel values. \"\n )\n\n return mask\n\n\n@DeveloperAPI\ndef get_image_from_class_mask(\n channel_class_map: torch.Tensor,\n mask: np.ndarray,\n) -> np.ndarray:\n \"\"\"Returns an image with channel values determined from a corresponding mask.\n Args:\n channel_class_map: An tensor mapping channel values to classes, where dim=0 is the class.\n mask: A masked image of dimensions [H x W] where each value is the channel class of the final image\n\n Return:\n [img] An image of dimensions [C x H x W], where C is the channel dimension\n \"\"\"\n mask = torch.from_numpy(mask)\n img = torch.zeros(channel_class_map.shape[1], mask.shape[-2], mask.shape[-1], dtype=torch.uint8)\n img = img.permute(1, 2, 0)\n mask = mask.unsqueeze(0)\n mask = mask.permute(1, 2, 0)\n for nclass, value in enumerate(channel_class_map):\n img[(mask == nclass).all(-1)] = value\n img = img.permute(2, 0, 1)\n\n return img.numpy()\n\n\n@DeveloperAPI\ndef to_tuple(v: int | tuple[int, int]) -> tuple[int, int]:\n \"\"\"Converts int or tuple to tuple of ints.\"\"\"\n if torch.jit.isinstance(v, int):\n return v, v\n else:\n return v\n\n\n@DeveloperAPI\ndef to_np_tuple(prop: int | Iterable) -> np.ndarray:\n \"\"\"Creates a np array of length 2 from a Conv2D property.\n\n E.g., stride=(2, 3) gets converted into np.array([2, 3]), where the height_stride = 2 and width_stride = 3. stride=2\n gets converted into np.array([2, 2]).\n \"\"\"\n if isinstance(prop, int):\n return np.ones(2).astype(int) * prop\n elif isinstance(prop, np.ndarray) and prop.size == 2:\n return prop.astype(int)\n elif isinstance(prop, Iterable) and len(prop) == 2:\n return np.array(list(prop)).astype(int)\n else:\n raise TypeError(f\"prop must be int or iterable of length 2, but is {prop}.\")\n\n\n@DeveloperAPI\ndef get_img_output_shape(\n img_height: int,\n img_width: int,\n kernel_size: int | tuple[int],\n stride: int | tuple[int],\n padding: int | tuple[int] | str,\n dilation: int | tuple[int],\n) -> tuple[int]:\n \"\"\"Returns the height and width of an image after a 2D img op.\n\n Currently supported for Conv2D, MaxPool2D and AvgPool2d ops.\n \"\"\"\n if padding == \"same\":\n return (img_height, img_width)\n elif padding == \"valid\":\n padding = np.zeros(2)\n else:\n padding = to_np_tuple(padding)\n\n kernel_size = to_np_tuple(kernel_size)\n stride = to_np_tuple(stride)\n dilation = to_np_tuple(dilation)\n shape = np.array([img_height, img_width])\n\n out_shape = np.floor(((shape + 2 * padding - dilation * (kernel_size - 1) - 1) / stride) + 1)\n\n return tuple(out_shape.astype(int))\n\n\ntorchvision_model_registry = Registry()\n\n\ndef register_torchvision_model_variants(variants: list[TVModelVariant]):\n def wrap(cls):\n # prime with empty placeholder\n torchvision_model_registry[cls.torchvision_model_type] = {}\n\n # register each variant\n for variant in variants:\n torchvision_model_registry[cls.torchvision_model_type][variant.variant_id] = variant\n return cls\n\n return wrap\n"
},
{
"path": "ludwig/utils/inference_utils.py",
"content": "from datetime import datetime\n\nimport pandas as pd\nimport torch\n\nfrom ludwig.constants import (\n AUDIO,\n BAG,\n BINARY,\n CATEGORY,\n COLUMN,\n DATE,\n IMAGE,\n NAME,\n POSTPROCESSOR,\n PREDICTOR,\n PREPROCESSOR,\n SEQUENCE,\n SET,\n TEXT,\n TIMESERIES,\n TYPE,\n VECTOR,\n)\nfrom ludwig.types import FeatureConfigDict, ModelConfigDict\nfrom ludwig.utils.audio_utils import read_audio_from_path\nfrom ludwig.utils.date_utils import create_vector_from_datetime_obj\nfrom ludwig.utils.image_utils import read_image_from_path\nfrom ludwig.utils.torch_utils import place_on_device\nfrom ludwig.utils.types import TorchDevice, TorchscriptPreprocessingInput\n\nFEATURES_TO_CAST_AS_STRINGS = {BINARY, CATEGORY, BAG, SET, TEXT, SEQUENCE, TIMESERIES, VECTOR}\n\n\ndef get_filename_from_stage(stage: str, device: TorchDevice) -> str:\n \"\"\"Returns the filename for a stage of inference.\"\"\"\n if stage not in [PREPROCESSOR, PREDICTOR, POSTPROCESSOR]:\n raise ValueError(f\"Invalid stage: {stage}.\")\n # device is only tracked for predictor stage\n if stage == PREDICTOR:\n return f\"inference_{stage}-{device}.pt\"\n else:\n return f\"inference_{stage}.pt\"\n\n\ndef to_inference_module_input_from_dataframe(\n dataset: pd.DataFrame, config: ModelConfigDict, load_paths: bool = False, device: torch.device | None = None\n) -> dict[str, TorchscriptPreprocessingInput]:\n \"\"\"Converts a pandas DataFrame to be compatible with a torchscripted InferenceModule forward pass.\"\"\"\n inputs = {}\n for if_config in config[\"input_features\"]:\n feature_inputs = to_inference_model_input_from_series(\n dataset[if_config[COLUMN]],\n if_config[TYPE],\n load_paths=load_paths,\n feature_config=if_config,\n )\n feature_inputs = place_on_device(feature_inputs, device)\n inputs[if_config[NAME]] = feature_inputs\n return inputs\n\n\ndef to_inference_model_input_from_series(\n s: pd.Series, feature_type: str, load_paths: bool = False, feature_config: FeatureConfigDict | None = None\n) -> TorchscriptPreprocessingInput:\n \"\"\"Converts a pandas Series to be compatible with a torchscripted InferenceModule forward pass.\"\"\"\n if feature_type == IMAGE:\n if load_paths:\n return [read_image_from_path(v) if isinstance(v, str) else v for v in s]\n elif feature_type == AUDIO:\n if load_paths:\n return [read_audio_from_path(v) if isinstance(v, str) else v for v in s]\n elif feature_type == DATE:\n if feature_config is None:\n raise ValueError('\"date\" feature type requires the associated feature config to be provided.')\n datetime_format = feature_config[\"preprocessing\"][\"datetime_format\"]\n return [torch.tensor(create_vector_from_datetime_obj(datetime.strptime(v, datetime_format))) for v in s]\n elif feature_type in FEATURES_TO_CAST_AS_STRINGS:\n return s.astype(str).to_list()\n return torch.from_numpy(s.to_numpy())\n"
},
{
"path": "ludwig/utils/llm_quantization_utils.py",
"content": "import torch\nfrom torch import nn\n\ntry:\n from bitsandbytes.functional import dequantize_4bit\n from bitsandbytes.nn.modules import Linear4bit\nexcept Exception:\n dequantize_4bit = None\n Linear4bit = None\n\nfrom ludwig.api_annotations import DeveloperAPI\n\n\n@DeveloperAPI\ndef linear4bit_to_linear(linear4bit_layer):\n \"\"\"Converts a Linear4Bit layer to a standard Linear layer by dequantizing the weight values and copying the\n dequantized weights to a new Linear layer.\n\n Args:\n linear4bit_layer (Linear4bit): The input Linear4Bit layer.\n\n Returns:\n nn.Linear: A new Linear layer with dequantized weights and biases.\n \"\"\"\n # Create a new Linear layer with the same shape\n new_linear_layer = nn.Linear(\n linear4bit_layer.in_features,\n linear4bit_layer.out_features,\n bias=linear4bit_layer.bias is not None,\n dtype=torch.float16,\n )\n\n # Dequantize the weight and bias from the Linear4bit layer and perform an in-place tensor replacement\n # to update the weights and bias in the new Linear layer. This is done to avoid creating a new tensor\n # and copying the data, which is slow.\n new_linear_layer.weight.data.copy_(\n dequantize_4bit(linear4bit_layer.weight.data, linear4bit_layer.weight.quant_state)\n )\n if linear4bit_layer.bias is not None:\n new_linear_layer.bias.data.copy_(linear4bit_layer.bias.data)\n\n return new_linear_layer\n\n\n@DeveloperAPI\ndef convert_quantized_linear_to_linear(module):\n \"\"\"Recursively converts Linear4Bit layers to standard Linear layers in a given module.\n\n Args:\n module (nn.Module): The input module containing potentially nested Linear4Bit layers.\n\n Returns:\n None\n \"\"\"\n for name, child in module.named_children():\n if isinstance(child, Linear4bit):\n # Replace Linear4Bit layer with a new Linear layer\n setattr(module, name, linear4bit_to_linear(child))\n else:\n # Recursively apply the conversion for nested modules\n convert_quantized_linear_to_linear(child)\n"
},
{
"path": "ludwig/utils/llm_utils.py",
"content": "import copy\nimport logging\nimport tempfile\nfrom typing import TYPE_CHECKING, Union\n\nimport torch\nimport torch.nn.functional as F\nimport transformers\nfrom packaging import version\n\ntry:\n from bitsandbytes.nn.modules import Embedding as BnbEmbedding\nexcept Exception:\n BnbEmbedding = None\nfrom transformers import AutoConfig, AutoModelForCausalLM, PreTrainedModel, PreTrainedTokenizer, TextStreamer\n\nfrom ludwig.constants import IGNORE_INDEX_TOKEN_ID, LOGITS, PREDICTIONS, PROBABILITIES\nfrom ludwig.schema.trainer import LLMTrainerConfig\nfrom ludwig.utils.error_handling_utils import default_retry\nfrom ludwig.utils.logging_utils import log_once\nfrom ludwig.utils.model_utils import find_embedding_layer_with_path\n\nif TYPE_CHECKING:\n from ludwig.schema.encoders.text_encoders import LLMEncoderConfig\n from ludwig.schema.model_types.llm import LLMModelConfig\n\n\nlogger = logging.getLogger(__name__)\n\ntransformers_436 = version.parse(transformers.__version__) >= version.parse(\"4.36.0\")\n\nFALLBACK_CONTEXT_LEN = 2048\n\n_MODELS_WITH_DEVICE_MAP_AUTO_EXCLUSION = set()\n\n\n@default_retry(tries=8)\ndef load_pretrained_from_config(\n config_obj: Union[\"LLMModelConfig\", \"LLMEncoderConfig\"],\n model_config: AutoConfig | None = None,\n weights_save_path: str | None = None,\n) -> PreTrainedModel:\n load_kwargs = {}\n if config_obj.quantization:\n # Apply quantization configuration at model load time\n load_kwargs[\"dtype\"] = getattr(torch, config_obj.quantization.bnb_4bit_compute_dtype)\n load_kwargs[\"quantization_config\"] = config_obj.quantization.to_bitsandbytes()\n load_kwargs[\"device_map\"] = \"auto\"\n\n if transformers_436:\n load_kwargs[\"attn_implementation\"] = \"eager\"\n else:\n # Load in float32 by default to avoid CUBLAS errors with small hidden sizes\n # and to ensure numerical stability during training without mixed-precision.\n load_kwargs[\"dtype\"] = torch.float32\n\n config_modified = False\n if config_obj.model_parameters:\n # Add any model specific parameters to the load kwargs\n for param_name, param_value in config_obj.model_parameters.to_dict().items():\n # Not all parameters are supported by all models, so we only add the parameter to the load kwargs\n # if it is supported by the model.\n if param_value is None:\n continue\n\n if hasattr(model_config, param_name):\n if isinstance(param_value, dict):\n # For nested dict params (e.g. rope_scaling), merge with existing\n # config values to preserve defaults like rope_theta.\n existing = getattr(model_config, param_name, {}) or {}\n existing.update(param_value)\n setattr(model_config, param_name, existing)\n config_modified = True\n else:\n load_kwargs[param_name] = param_value\n else:\n logger.warning(f\"Parameter {param_name} is not supported by {config_obj.base_model}. Skipping.\")\n\n # Only pass config= when we've directly modified it (e.g. rope_scaling merge).\n if config_modified:\n load_kwargs[\"config\"] = model_config\n\n logger.info(\"Loading large language model...\")\n pretrained_model_name_or_path = weights_save_path or config_obj.base_model\n trust_remote_code = getattr(config_obj, \"trust_remote_code\", False)\n model: PreTrainedModel = AutoModelForCausalLM.from_pretrained(\n pretrained_model_name_or_path, trust_remote_code=trust_remote_code, **load_kwargs\n )\n return model\n\n\ndef to_device(\n model: PreTrainedModel,\n device: str | torch.DeviceObjType,\n config_obj: \"LLMModelConfig\", # noqa F821\n curr_device: torch.DeviceObjType,\n) -> tuple[PreTrainedModel, torch.DeviceObjType]:\n \"\"\"Move an LLM to the requested device, accounting for sharding and adapters.\n\n Args:\n model: Pretrained model to put on device\n config_obj: LLM config\n curr_device: The current device that the model is on\n\n Returns:\n `model` moved to `device`\n \"\"\"\n device = torch.device(device)\n\n if device.type == curr_device.type:\n log_once(f\"Model already on device'{device}'.\")\n return model, device\n else:\n log_once(f\"Moving LLM from '{curr_device}' to '{device}'.\")\n\n model_kwargs = {}\n num_gpus = torch.cuda.device_count()\n if device == torch.device(\"cuda\") and num_gpus > 1:\n # TODO: make this configurable in the future. These parameters are from FastChat:\n # https://github.com/lm-sys/FastChat/blob/0e958b852a14f4bef5f0e9d7a5e7373477329cf2/fastchat/serve/inference.py#L90 # noqa\n # TODO: Wrap device_map=\"auto\" in a try-except block since it may not be supported for all models (E.g. BertLMHead) # noqa\n # We don't add quantization here (float16 or bfloat16) since we may not always want to quantize. We should\n # make quantization configurable in the future via the trainer config.\n model_kwargs.update(\n dict(\n low_cpu_mem_usage=True,\n max_memory={i: \"13GiB\" for i in range(num_gpus)},\n )\n )\n\n if config_obj.base_model not in _MODELS_WITH_DEVICE_MAP_AUTO_EXCLUSION:\n model_kwargs[\"device_map\"] = \"auto\"\n\n if config_obj.quantization:\n model_kwargs[\"quantization_config\"] = config_obj.quantization.to_bitsandbytes()\n\n # we save and reload the weights to ensure that they can be sharded across the GPUs using `from_pretrained`\n with tempfile.TemporaryDirectory() as tmpdir:\n model.save_pretrained(tmpdir)\n\n if config_obj.adapter:\n model = AutoModelForCausalLM.from_pretrained(\n config_obj.base_model,\n trust_remote_code=getattr(config_obj, \"trust_remote_code\", False),\n **model_kwargs,\n )\n\n # Leave this import inline to support a minimal install of Ludwig\n from peft import PeftModel # noqa\n\n model = PeftModel.from_pretrained(\n model,\n tmpdir,\n torch_dtype=torch.float16,\n )\n else:\n model = AutoModelForCausalLM.from_pretrained(\n tmpdir,\n trust_remote_code=getattr(config_obj, \"trust_remote_code\", False),\n **model_kwargs,\n )\n else:\n model = model.to(device)\n\n return model, device\n\n\ndef _load_peft_config(pretrained_adapter_weights: str):\n \"\"\"Load a PeftConfig, fixing known compatibility issues with newer PEFT versions.\"\"\"\n import json\n\n from huggingface_hub import hf_hub_download\n from peft import PeftConfig\n\n config_file = hf_hub_download(pretrained_adapter_weights, \"adapter_config.json\")\n with open(config_file) as f:\n config_dict = json.load(f)\n\n # AdaLoRA requires total_step > 0 in newer PEFT versions, but pretrained\n # configs may have total_step=None.\n if config_dict.get(\"peft_type\") == \"ADALORA\" and not config_dict.get(\"total_step\"):\n config_dict[\"total_step\"] = 10000\n\n return PeftConfig.from_peft_type(**config_dict)\n\n\ndef initialize_adapter(\n model: PreTrainedModel, config_obj: \"LLMModelConfig\" # noqa F821\n) -> Union[\"PeftModel\", PreTrainedModel]: # noqa F821\n \"\"\"Wrap a pretrained model with a PEFT model for fine-tuning.\n\n Args:\n model: Pretrained model to fine-tune with an adapter.\n config_obj: LLM config\n\n Returns:\n `model` wrapped in a PEFT model if an adapter config was provided, otherwise `model`.\n \"\"\"\n # Only load a PEFT model if the config specifies an adapter, otherwise return the model unaltered.\n if config_obj.adapter:\n if config_obj.adapter.pretrained_adapter_weights:\n # Load pretrained adapter weights if specified.\n logger.info(f\"Using pretrained adapter weights: {config_obj.adapter.pretrained_adapter_weights}\")\n\n # Leave this import inline to support a minimal install of Ludwig\n from peft import MODEL_TYPE_TO_PEFT_MODEL_MAPPING, PeftConfig # noqa\n\n peft_config = _load_peft_config(config_obj.adapter.pretrained_adapter_weights)\n\n model = MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type].from_pretrained(\n model, config_obj.adapter.pretrained_adapter_weights, config=peft_config\n )\n else:\n # Leave this import inline to support a minimal install of Ludwig\n from peft import get_peft_model, TaskType # noqa\n\n # If no pretrained adapter is provided, we want to load untrained weights into the model\n peft_config = config_obj.adapter.to_config(\n task_type=TaskType.CAUSAL_LM, tokenizer_name_or_path=config_obj.base_model\n )\n\n model = get_peft_model(model, peft_config)\n\n return model\n\n\ndef get_context_len(model_config: AutoConfig):\n \"\"\"Determines the maximum length of the context (input + output tokens) based on the provided model\n configuration.\n\n Args:\n model_config (AutoConfig): The model configuration object containing information about the model's properties.\n\n Returns:\n int: The maximum context length, which can be derived from the model configuration. If no relevant attribute\n is found, the default value of 2048 is returned.\n\n This function examines the provided model configuration object to identify the attribute that specifies the maximum\n context length. It checks for attributes in the following order of preference:\n 1. 'max_sequence_length': If this attribute is present in the model configuration, its value is returned.\n 2. 'max_position_embeddings': If 'max_sequence_length' is not found but 'max_position_embeddings' is present, its\n value is returned.\n 3. 'n_positions': If neither 'max_sequence_length' nor 'max_position_embeddings' are found, and 'n_positions' is\n present, its value is returned.\n 4. Default: If none of the relevant attributes are present, the function returns a default value of 2048.\n\n Note:\n - The maximum context length is important for defining the size of input and output sequences in a model.\n\n Example Usage:\n >>> config = AutoConfig.from_pretrained(\"bert-base-uncased\")\n >>> context_len = get_context_len(config)\n >>> print(context_len)\n 512\n \"\"\"\n if hasattr(model_config, \"max_sequence_length\"):\n return model_config.max_sequence_length\n elif hasattr(model_config, \"max_position_embeddings\"):\n return model_config.max_position_embeddings\n elif hasattr(model_config, \"n_positions\"):\n return model_config.n_positions\n else:\n return FALLBACK_CONTEXT_LEN\n\n\ndef has_padding_token(input_tensor: torch.Tensor, tokenizer: PreTrainedTokenizer):\n \"\"\"Checks if the input tensor contains any padding tokens.\n\n Args:\n input_tensor (torch.Tensor): The input tensor.\n tokenizer (PreTrainedTokenizer): The tokenizer used to encode the input.\n\n Returns:\n bool: True if the input tensor contains any padding tokens, False otherwise.\n\n Example:\n >>> import torch\n >>> from transformers import PreTrainedTokenizer\n >>> tokenizer = PreTrainedTokenizer.from_pretrained('bert-base-uncased')\n >>> input_sentence = \"This is an example sentence.\"\n >>> input_ids = tokenizer.encode(input_sentence, add_special_tokens=True)\n >>> padded_input_ids = torch.nn.functional.pad(input_ids, (0, 10 - len(input_ids)))\n >>> has_padding = has_padding_token(padded_input_ids, tokenizer)\n >>> has_padding\n True\n \"\"\"\n if input_tensor.dim() == 1:\n return torch.any(input_tensor == tokenizer.pad_token_id).item()\n elif input_tensor.dim() == 2:\n return torch.any(input_tensor == tokenizer.pad_token_id, dim=-1).item()\n else:\n raise ValueError(\"Input tensor must be 1D or 2D\")\n\n\ndef remove_left_padding(input_ids_sample: torch.Tensor, tokenizer: PreTrainedTokenizer):\n \"\"\"Removes left padding and other tokens until the first BOS token from the input_ids tensor.\n\n Args:\n input_ids_sample (torch.Tensor): The input tensor with padding and other tokens.\n tokenizer (PreTrainedTokenizer): The tokenizer used to encode the input.\n\n Returns:\n torch.Tensor: The input tensor without left padding and other tokens until the first BOS token.\n\n Example:\n >>> import torch\n >>> from transformers import PreTrainedTokenizer\n >>> tokenizer = PreTrainedTokenizer.from_pretrained('bert-base-uncased')\n >>> input_sentence = \"This is an example sentence.\"\n >>> input_ids = tokenizer.encode(input_sentence, add_special_tokens=True)\n >>> padded_input_ids = torch.nn.functional.pad(input_ids, (10 - len(input_ids), 0))\n >>> input_ids_no_padding = remove_left_padding(padded_input_ids, tokenizer)\n >>> input_ids_no_padding\n tensor([[1, 2, 3]])\n \"\"\"\n # Remove all PAD tokens\n pad_idxs = torch.where(input_ids_sample == tokenizer.pad_token_id)[0] # all PAD token locations\n input_ids_no_padding = input_ids_sample\n if len(pad_idxs) != 0:\n pad_idx = pad_idxs[-1] # get last PAD token location\n input_ids_no_padding = input_ids_sample[pad_idx + 1 :]\n\n # Start from the first BOS token\n bos_idxs = torch.where(input_ids_no_padding == tokenizer.bos_token_id)[0] # all BOS token locations\n if len(bos_idxs) != 0:\n bos_idx = bos_idxs[0] # get first BOS token location\n else:\n bos_idx = 0\n\n input_ids_no_bos = input_ids_no_padding[bos_idx:].unsqueeze(0)\n return input_ids_no_bos\n\n\ndef add_left_padding(input_ids, max_length, pad_value=0):\n \"\"\"Adds left padding to the input_ids tensor.\n\n Args:\n input_ids (torch.Tensor): The input tensor.\n max_length (int): The maximum length of the tensor after padding.\n pad_value (int, optional): The value used for padding. Defaults to 0.\n\n Returns:\n torch.Tensor: The input_ids tensor with left padding.\n\n Example:\n >>> input_ids = torch.tensor([1, 2, 3])\n >>> max_length = 5\n >>> padded_tensor = add_left_padding(input_ids, max_length)\n >>> padded_tensor\n tensor([0, 0, 1, 2, 3])\n \"\"\"\n padding = torch.tensor([pad_value] * (max_length - input_ids.shape[0]), dtype=torch.int64, device=input_ids.device)\n return torch.cat((padding, input_ids), dim=-1)\n\n\ndef create_attention_mask(input_ids: torch.Tensor, tokenizer: PreTrainedTokenizer):\n \"\"\"Creates an attention mask for the input_ids tensor. This also sets the last padding token ID to 1 if it\n exists.\n\n Args:\n input_ids (torch.Tensor): The input tensor.\n tokenizer (PreTrainedTokenizer): The tokenizer used to encode the input.\n\n Returns:\n torch.Tensor: The attention mask tensor.\n\n Example:\n >>> import torch # noqa\n >>> from transformers import PreTrainedTokenizer\n >>> tokenizer = PreTrainedTokenizer.from_pretrained('bert-base-uncased')\n >>> input_sentence = \"This is an example sentence.\"\n >>> input_ids = tokenizer.encode(input_sentence, add_special_tokens=True)\n >>> attention_mask = create_attention_mask(input_ids, tokenizer)\n >>> attention_mask\n tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])\n \"\"\"\n attention_mask = input_ids != tokenizer.pad_token_id\n # Last token may not be padding if we've already hit the max sequence length\n if not attention_mask[-1]:\n # last token is padding, always attended to even if it is padding\n attention_mask[-1] = 1\n attention_mask = attention_mask.to(torch.int64)\n return attention_mask\n\n\ndef find_last_matching_index(tensor_a: torch.Tensor, tensor_b: torch.Tensor):\n \"\"\"Returns the last index of `tensor_a` that matches `tensor_b`. Specifically, this checks whether the tensor_b\n is in the last tensor_b.shape[0] elements of tensor_a.\n\n Args:\n tensor_a (torch.Tensor): The first tensor.\n tensor_b (torch.Tensor): The second tensor.\n\n Returns:\n int: The last index of `tensor_a` that matches `tensor_b`. Returns -1 if there is no matching index.\n\n Example:\n >>> import torch\n >>> tensor_a = torch.tensor([1, 2, 3, 4, 5, 6, 7, 8])\n >>> tensor_b = torch.tensor([6, 7, 8])\n >>> last_matching_index = find_last_matching_index(tensor_a, tensor_b)\n >>> last_matching_index\n 5\n \"\"\"\n last_index = -1\n\n tensor_a_length = tensor_a.shape[0]\n tensor_b_length = tensor_b.shape[0]\n\n # Get the last tensor_b_length elements of tensor_a.\n tensor_a_truncated = tensor_a[-tensor_b_length:]\n\n # Find the last matching index.\n for i in range(tensor_b_length):\n if torch.equal(tensor_a_truncated[i:], tensor_b[: tensor_b_length - i]):\n last_index = tensor_a_length - tensor_b_length + i\n break\n\n return last_index\n\n\ndef pad_target_tensor_for_fine_tuning(\n targets: dict[str, torch.Tensor],\n predictions: dict[str, torch.Tensor],\n model_inputs: torch.Tensor,\n of_name: str,\n) -> dict[str, torch.Tensor]:\n \"\"\"Pad and adjust target tensors for fine-tuning LLMS models.\n\n This function is used to pad and adjust the target tensors with IGNORE_INDEX_TOKEN_ID based on the model inputs and\n predictions during the fine-tuning process of Language Models. Here's what this function does:\n 1. If none of the tokens from the target were in the model inputs, we create a tensor of the length of model\n inputs with value IGNORE_INDEX_TOKEN_IDs. This ignores this row from affecting loss.\n 2. If the target tokens were entirely inside the model inputs, we want to pad all the tokens in model_inputs\n coming from the input with IGNORE_INDEX_TOKEN_IDs and leave the target tokens as is. This ensures that all\n of the target tokens are used during loss computation.\n 3. In the scenario that only some part of the target tokens were in the model inputs, we want to pad the model\n inputs until that point and only leave the partial tokens of the target as is. This ensures that we will\n only compute loss on the target tokens that were in the model inputs.\n\n Args:\n targets (Dict[str, torch.Tensor]): A dictionary containing the target tensors.\n predictions (Dict[str, torch.Tensor]): A dictionary containing the predicted tensors.\n model_inputs (torch.Tensor): The input tensor passed into the model's forward pass.\n of_name (str): The name of the target tensor to be padded and adjusted.\n\n Returns:\n Dict[str, torch.Tensor]: A dictionary containing the updated target\n dictionaries.\n \"\"\"\n target_length = targets.get(of_name).size()[1]\n prediction_length = predictions[of_name].get(PREDICTIONS).size()[1]\n\n if target_length == prediction_length:\n return targets\n\n updated_targets = []\n for idx, target in enumerate(targets[of_name]):\n # Remove any leading IGNORE_INDEX_TOKEN_IDs in the target that were temporarily added for alignment\n end_index = (target != IGNORE_INDEX_TOKEN_ID).nonzero()[0]\n target = target[end_index:]\n target_device = target.device\n\n # See if any part of the target was in the tensor passed into the model's forward pass\n last_matching_index = find_last_matching_index(model_inputs[idx], target)\n\n # If the last matching index is -1, it means that the input tensor passed into the model was truncated\n # and did not contain the target tensor. In this case, we need to truncate the target tensors as well\n # and just set it to a tensor of IGNORE_INDEX_TOKEN_ID so that we don't compute loss on this target tensor.\n if last_matching_index == -1:\n updated_targets.append(torch.full((prediction_length,), IGNORE_INDEX_TOKEN_ID).to(device=target_device))\n\n # If the last matching index is not -1, it means that the input tensor passed into the model was not\n # truncated and contained either a part of the target tensor or the entire target tensor. In this case,\n # we need to set the target tensor to the part of the target tensor that was passed into the model while\n # also padding it to the correct length with IGNORE_INDEX_TOKEN_ID.\n else:\n padding = torch.full((last_matching_index,), IGNORE_INDEX_TOKEN_ID).to(device=target_device)\n updated_targets.append(torch.cat((padding, target), dim=-1)[:prediction_length])\n\n targets[of_name] = torch.stack(updated_targets).to(device=targets.get(of_name).device, dtype=torch.int64)\n\n return targets\n\n\ndef generate_merged_ids(\n input_ids: torch.tensor, target_ids: torch.tensor, tokenizer: PreTrainedTokenizer, max_sequence_length: int = None\n):\n \"\"\"Generate merged input and target IDs tensor.\n\n This function merges the input_ids and target_ids together to create a unified tensor\n to pass into the model. It also returns attention masks for the merged tensors.\n\n Args:\n input_ids (torch.Tensor): The input IDs tensor.\n target_ids (torch.Tensor or None): The target IDs tensor or None.\n max_sequence_length (int or None): The maximum sequence length to pad or truncate to.\n tokenizer (PreTrainedTokenizer): The tokenizer used to encode the input_ids and target_ids.\n\n Returns:\n torch.Tensor: The merged input and target IDs tensor.\n torch.Tensor: The attention masks for the merged tensor.\n \"\"\"\n merged_input_and_targets = []\n lengths = []\n\n eos_tensor = torch.tensor([tokenizer.eos_token_id]).to(target_ids[0].device)\n\n # Merge input_ids and target_ids by concatenating them together.\n # We remove the left padding from both input_ids and target_ids before concatenating them.\n for input_id_sample, target_id_sample in zip(input_ids, target_ids):\n input_id_sample_no_padding = remove_left_padding(input_id_sample, tokenizer)[0]\n target_id_sample_no_padding = remove_left_padding(target_id_sample, tokenizer)[0]\n target_id_sample_no_padding = torch.cat((target_id_sample_no_padding, eos_tensor), dim=-1)\n\n merged_sample_ids = torch.cat((input_id_sample_no_padding, target_id_sample_no_padding), dim=-1)\n # If the merged tensor is longer than the maximum sequence length, we truncate it.\n if max_sequence_length and merged_sample_ids.shape[0] > max_sequence_length:\n merged_sample_ids = merged_sample_ids[:max_sequence_length]\n\n merged_input_and_targets.append(merged_sample_ids)\n lengths.append(merged_sample_ids.shape[0])\n\n # Since we remove the left padding from the target_ids, the merged input_ids and target_ids\n # may not have the same lengths. We need to align them to the same length by adding left padding\n # and generate an attention mask for just the part of the input that is not padding.\n max_length = max(lengths)\n attention_masks = []\n for i, merged_sample_ids in enumerate(merged_input_and_targets):\n merged_input_and_targets[i] = add_left_padding(merged_sample_ids, max_length)\n attention_masks.append(create_attention_mask(merged_input_and_targets[i], tokenizer))\n\n return torch.stack(merged_input_and_targets), torch.stack(attention_masks)\n\n\ndef _get_decoded_targets_and_predictions(\n targets: dict[str, torch.Tensor],\n predictions: dict[str, dict[str, torch.Tensor]],\n tokenizer: PreTrainedTokenizer,\n of_name: str,\n):\n \"\"\"Returns the decoded targets and predictions, accounting for IGNORE_INDEX_TOKEN_ID.\"\"\"\n target_tensor = targets[of_name]\n pred_tensor = predictions[of_name][PREDICTIONS]\n # Ensure targets and predictions are on the same device\n if target_tensor.device != pred_tensor.device:\n target_tensor = target_tensor.to(pred_tensor.device)\n sanitized_targets = torch.where(target_tensor != IGNORE_INDEX_TOKEN_ID, target_tensor, tokenizer.pad_token_id)\n sanitized_predictions = torch.where(\n pred_tensor != IGNORE_INDEX_TOKEN_ID,\n pred_tensor,\n tokenizer.pad_token_id,\n )\n decoded_targets = tokenizer.batch_decode(sanitized_targets, skip_special_tokens=True)\n decoded_predictions = tokenizer.batch_decode(sanitized_predictions, skip_special_tokens=True)\n return decoded_targets, decoded_predictions\n\n\ndef get_realigned_target_and_prediction_tensors_for_inference(\n targets: dict[str, torch.Tensor],\n predictions: dict[str, dict[str, torch.Tensor]],\n of_name: str,\n tokenizer: PreTrainedTokenizer,\n pad_value: int = None,\n) -> tuple[dict[str, torch.Tensor], dict[str, torch.Tensor]]:\n \"\"\"Realigns the target tensor with the predictions.\n\n This is necessary for text metrics that require the target and prediction to be of the same length.\n\n Args:\n targets: The target tensor.\n predictions: The prediction tensor.\n of_name: The output feature's name.\n tokenizer: The HF tokenizer.\n pad_direction: The direction to pad the tensors. Can be 'left' or 'right'.\n Defaults to 'right'.\n\n Returns:\n Tuple of realigned (targets, decoded_targets, predictions, decoded_predictions).\n - targets is a map of feature name -> tensor of token ids.\n - predictions is a map from output feature name -> map of tensors with the following items:\n - \"predictions\": tensor of token ids.\n - \"probabilities\": tensor of probabilities.\n - \"logits\": tensor of logits.\n \"\"\"\n target_length = targets.get(of_name).size()[1]\n prediction_length = predictions[of_name].get(PREDICTIONS).size()[1]\n\n if target_length == prediction_length:\n return targets, predictions\n\n if not pad_value:\n pad_value = tokenizer.pad_token_id if tokenizer.pad_token_id is not None else tokenizer.eos_token_id\n\n zeros_to_add = (\n target_length - prediction_length if target_length > prediction_length else prediction_length - target_length\n )\n\n # We don't want to modify the original targets and predictions tensors, so we create a copy of them.\n _targets = copy.deepcopy(targets)\n _predictions = copy.deepcopy(predictions)\n\n # Align target and prediction tensors for text to text metric computation\n if target_length > prediction_length:\n # Pad the predictions.\n _predictions[of_name][PREDICTIONS] = F.pad(\n _predictions[of_name][PREDICTIONS], (0, zeros_to_add), value=pad_value\n ).to(torch.int64)\n\n _predictions[of_name][PROBABILITIES] = F.pad(_predictions[of_name][PROBABILITIES], (0, 0, 0, zeros_to_add)).to(\n torch.float32\n )\n\n _predictions[of_name][LOGITS] = F.pad(_predictions[of_name][LOGITS], (0, 0, 0, zeros_to_add)).to(torch.float32)\n else:\n _targets[of_name] = F.pad(_targets[of_name], (0, zeros_to_add), value=pad_value).to(torch.int64)\n\n return _targets, _predictions\n\n\ndef update_embedding_layer(model: AutoModelForCausalLM, config_obj: LLMTrainerConfig) -> AutoModelForCausalLM:\n \"\"\"Updates the embedding layer of the model to use the 8-bit embedding layer from bitsandbytes.nn.modules.\n\n This is necessary when using 8-bit optimizers from bitsandbytes.\n See: https://github.com/TimDettmers/bitsandbytes#tldr\n \"\"\"\n # If we're using an 8-bit optimizer, we need to replace the embedding layer with a custom embedding layer from\n # bnb.nn.modules.Embedding.\n if hasattr(config_obj, \"optimizer\") and config_obj.optimizer.is_8bit:\n embedding_layer, module_path = find_embedding_layer_with_path(model)\n if embedding_layer is None:\n raise ValueError(\n \"Could not find an embedding layer in the model. This is required when using 8-bit optimizers\"\n \" since a custom 8-bit embedding layer is used in place of the original embedding layer.\"\n )\n\n # Initialize the BNB embedding layer with the same parameters and weights as the original embedding layer.\n bnb_embedding = BnbEmbedding(\n num_embeddings=embedding_layer.num_embeddings,\n embedding_dim=embedding_layer.embedding_dim,\n padding_idx=embedding_layer.padding_idx,\n max_norm=embedding_layer.max_norm,\n norm_type=embedding_layer.norm_type,\n scale_grad_by_freq=embedding_layer.scale_grad_by_freq,\n sparse=embedding_layer.sparse,\n _weight=embedding_layer.weight,\n device=model.device,\n )\n\n # Update the model's original embedding layer to use the BNB embedding layer using the module_path\n # returned by find_embedding_layer_with_path.\n module_path = module_path.split(\".\")\n module = model\n for module_name in module_path[:-1]:\n module = getattr(module, module_name)\n setattr(module, module_path[-1], bnb_embedding)\n\n # Set the get input embeddings lambda function to return the BNB embedding layer\n model.get_input_embeddings = lambda: bnb_embedding\n\n logger.info(\"Updated the pretrained embedding layer to use the embedding layer from bitsandbytes.\")\n\n return model\n\n\ndef create_text_streamer(tokenizer: PreTrainedTokenizer) -> TextStreamer:\n \"\"\"Creates a TextStreamer object for streaming text to stdout during generation.\"\"\"\n return TextStreamer(tokenizer=tokenizer, skip_prompt=True)\n"
},
{
"path": "ludwig/utils/logging_utils.py",
"content": "_logged = set()\n\n\ndef log_once(key: str) -> bool:\n \"\"\"Returns True if this is the \"first\" call for a given key.\n\n Example:\n if log_once(\"some_key\"):\n logger.info(\"Some verbose logging statement\") # noqa\n \"\"\"\n\n if key not in _logged:\n _logged.add(key)\n return True\n return False\n"
},
{
"path": "ludwig/utils/loss_utils.py",
"content": "import torch\n\n\ndef rmspe_loss(targets: torch.Tensor, predictions: torch.Tensor) -> torch.Tensor:\n \"\"\"Root mean square percentage error.\n\n Bad predictions can lead to arbitrarily large RMSPE values, especially if some values of targets are very close to\n zero. We return a large value instead of inf when (some) targets are zero.\n \"\"\"\n epsilon = 1e-4\n # add epsilon if targets are zero to avoid division by zero\n denominator = targets + epsilon * (targets == 0).float()\n loss = torch.sqrt(torch.mean(((targets - predictions).float() / denominator) ** 2))\n return loss\n\n\ndef mean_confidence_penalty(probabilities: torch.Tensor, num_classes: int) -> torch.Tensor:\n max_entropy = torch.log(torch.tensor(num_classes))\n # clipping needed for avoiding log(0) = -inf\n entropy_per_class, _ = torch.max(-probabilities * torch.log(torch.clamp(probabilities, 1e-10, 1)), dim=0)\n entropy = torch.sum(entropy_per_class, -1)\n penalty = (max_entropy - entropy) / max_entropy\n return torch.mean(penalty)\n"
},
{
"path": "ludwig/utils/math_utils.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport math\n\nimport numpy as np\n\n\ndef softmax(x, temperature=1.0):\n e_x = np.exp((x - np.max(x)) / temperature)\n return e_x / e_x.sum()\n\n\ndef int_type(number):\n if number <= np.iinfo(np.int8).max:\n return np.int8\n elif number <= np.iinfo(np.int16).max:\n return np.int16\n elif number <= np.iinfo(np.int32).max:\n return np.int32\n else: # if number <= np.iinfo(np.int64).max:\n return np.int64\n\n\ndef convert_size(size_bytes):\n if size_bytes == 0:\n return \"0B\"\n size_name = (\"B\", \"KB\", \"MB\", \"GB\", \"TB\", \"PB\", \"EB\", \"ZB\", \"YB\")\n i = int(math.floor(math.log(size_bytes, 1024)))\n p = math.pow(1024, i)\n s = round(size_bytes / p, 2)\n return f\"{s} {size_name[i]}\"\n\n\ndef round2precision(val, precision: int = 0, which: str = \"\"):\n assert precision >= 0\n val *= 10**precision\n round_callback = round\n if which.lower() == \"up\":\n round_callback = math.ceil\n if which.lower() == \"down\":\n round_callback = math.floor\n return \"{1:.{0}f}\".format(precision, round_callback(val) / 10**precision)\n\n\ndef cumsum(x: list[int]) -> list[int]:\n results = []\n j = 0\n for i in range(0, len(x)):\n j += x[i]\n results.append(j)\n return results\n"
},
{
"path": "ludwig/utils/metric_utils.py",
"content": "from collections import defaultdict, namedtuple\n\nimport torch\nfrom torch import Tensor\nfrom torchmetrics.metric import Metric\n\nfrom ludwig.constants import COMBINED, LOSS, NAME, TYPE\nfrom ludwig.modules.metric_registry import get_metric_names_for_type\nfrom ludwig.types import FeatureConfigDict\n\n\ndef sequence_mask(lengths: Tensor, maxlen: int | None = None, dtype=torch.bool) -> Tensor:\n \"\"\"Implements tf.sequence_mask in torch.\n\n From https://discuss.pytorch.org/t/pytorch-equivalent-for-tf-sequence-mask/39036/2.\n \"\"\"\n if maxlen is None:\n maxlen = lengths.max()\n row_vector = torch.arange(0, maxlen, 1).to(lengths.device)\n matrix = torch.unsqueeze(lengths, dim=-1)\n mask = row_vector < matrix\n\n return mask.type(dtype)\n\n\ndef dynamic_partition(data: Tensor, partitions: Tensor, num_partitions: int) -> list[Tensor]:\n \"\"\"Implements tf.dynamic_partition in torch.\n\n From https://discuss.pytorch.org/t/equivalent-of-tf-dynamic-partition/53735.\n \"\"\"\n assert data.size() == partitions.size()\n\n # Flatten data into 1D vectors to do partitioning correctly.\n data = data.view(-1)\n partitions = partitions.view(-1)\n result = []\n for i in range(num_partitions):\n result += [data[(partitions == i).nonzero().squeeze(1)]]\n return result\n\n\ndef masked_correct_predictions(targets: Tensor, preds: Tensor, targets_sequence_lengths: Tensor) -> Tensor:\n \"\"\"Masks out special symbols, and returns tensor of correct predictions.\n\n Args:\n targets: 2D tensor [batch_size, sequence_length]\n preds: 2D tensor [batch_size, sequence_length]\n\n Returns:\n 1D tensor of all correct predictions.\n \"\"\"\n correct_preds = preds == targets\n\n mask = sequence_mask(lengths=targets_sequence_lengths, maxlen=correct_preds.shape[1], dtype=torch.int32)\n _, masked_correct_preds = dynamic_partition(data=correct_preds, partitions=mask, num_partitions=2)\n\n return masked_correct_preds.type(torch.float32)\n\n\ndef get_scalar_from_ludwig_metric(metric: Metric) -> float:\n \"\"\"Returns the scalar value of a Ludwig metric.\n\n Params:\n metric: Metric object\n\n Returns:\n float: scalar value of the metric\n \"\"\"\n return metric.compute().detach().cpu().numpy().item()\n\n\n# Data for training and evaluation metrics.\nTrainerMetric = namedtuple(\"TrainerMetric\", (\"epoch\", \"step\", \"value\"))\n\n\ndef reduce_trainer_metrics_dict(\n dict_dict_trainer_metrics: dict[str, dict[str, list[TrainerMetric]]],\n) -> dict[str, dict[str, list[float]]]:\n \"\"\"Reduces Dict[feature_name, Dict[metric_name, List[TrainerMetric]]] to Dict[feature_name, Dict[metric_name,\n List[float]]].\n\n Used for flattening the results returned by trainer.py::train(), which come from ProgressTracker.\n \"\"\"\n flattened_dict = defaultdict(lambda: defaultdict(list))\n for feature_name, trainer_metric_dict in dict_dict_trainer_metrics.items():\n for metric_name, trainer_metrics in trainer_metric_dict.items():\n for trainer_metric in trainer_metrics:\n flattened_dict[feature_name][metric_name].append(trainer_metric[-1])\n # Convert defaultdict to dict so JSON serialization works with dataclasses.asdict().\n return {k: dict(v) for k, v in flattened_dict.items()}\n\n\ndef get_metric_names(output_features: dict[str, \"OutputFeature\"]) -> dict[str, list[str]]: # noqa\n \"\"\"Returns a dict of output_feature_name -> list of metric names.\"\"\"\n metrics_names = {}\n for output_feature_name, output_feature in output_features.items():\n metrics_names[output_feature_name] = sorted(list(get_metric_names_for_type(output_feature.type())))\n # Add combined loss.\n metrics_names[COMBINED] = [LOSS]\n return metrics_names\n\n\ndef get_feature_to_metric_names_map(output_features: list[FeatureConfigDict]) -> dict[str, list[str]]:\n \"\"\"Returns a dict of output_feature_name -> list of metric names.\"\"\"\n metrics_names = {}\n for output_feature in output_features:\n output_feature_name = output_feature[NAME]\n output_feature_type = output_feature[TYPE]\n metrics_names[output_feature_name] = get_metric_names_for_type(output_feature_type)\n metrics_names[COMBINED] = [LOSS]\n return metrics_names\n\n\ndef get_feature_to_metric_names_map_from_feature_collection(\n output_features: \"FeatureCollection\", # noqa\n) -> dict[str, list[str]]:\n \"\"\"Returns a dict of output_feature_name -> list of metric names.\"\"\"\n metrics_names = {\n output_feature.name: get_metric_names_for_type(output_feature.type) for output_feature in output_features\n }\n metrics_names[COMBINED] = [LOSS]\n return metrics_names\n"
},
{
"path": "ludwig/utils/metrics_printed_table.py",
"content": "import logging\n\nfrom tabulate import tabulate\n\nfrom ludwig.constants import COMBINED, LOSS\nfrom ludwig.utils.metric_utils import TrainerMetric\n\nlogger = logging.getLogger(__name__)\n\n\ndef get_metric_value_or_empty(metrics_log: dict[str, list[TrainerMetric]], metric_name: str):\n \"\"\"Returns the metric value if it exists or empty.\"\"\"\n if metric_name not in metrics_log:\n return \"\"\n return metrics_log[metric_name][-1][-1]\n\n\ndef print_table_for_single_output_feature(\n train_metrics_log: dict[str, list[TrainerMetric]],\n validation_metrics_log: dict[str, list[TrainerMetric]],\n test_metrics_log: dict[str, list[TrainerMetric]],\n combined_loss_for_each_split: list[float],\n) -> None:\n \"\"\"Prints the metrics table for a single output feature.\n\n Args:\n train_metrics_log: Dict from metric name to list of TrainerMetric.\n validation_metrics_log: Dict from metric name to list of TrainerMetric.\n test_metrics_log: Dict from metric name to list of TrainerMetric.\n \"\"\"\n # Get the superset of metric names across all splits.\n all_metric_names = set()\n all_metric_names.update(train_metrics_log.keys())\n all_metric_names.update(validation_metrics_log.keys())\n all_metric_names.update(test_metrics_log.keys())\n all_metric_names = sorted(list(all_metric_names))\n\n # Assemble the printed table.\n # Each item in the printed_table corresponds to a row in the printed table.\n printed_table = [[\"train\", \"validation\", \"test\"]]\n for metric_name in all_metric_names:\n metrics_for_each_split = [\n get_metric_value_or_empty(train_metrics_log, metric_name),\n get_metric_value_or_empty(validation_metrics_log, metric_name),\n get_metric_value_or_empty(test_metrics_log, metric_name),\n ]\n printed_table.append([metric_name] + metrics_for_each_split)\n\n # Add combined loss.\n printed_table.append([\"combined_loss\"] + combined_loss_for_each_split)\n\n logger.info(tabulate(printed_table, headers=\"firstrow\", tablefmt=\"fancy_grid\", floatfmt=\".4f\"))\n\n\ndef print_metrics_table(\n output_features: dict[str, \"OutputFeature\"], # noqa\n train_metrics_log: dict[str, dict[str, list[TrainerMetric]]],\n validation_metrics_log: dict[str, dict[str, list[TrainerMetric]]],\n test_metrics_log: dict[str, dict[str, list[TrainerMetric]]],\n):\n \"\"\"Prints a table of metrics table for each output feature, for each split.\n\n Example:\n ╒═══════════════╤═════════╤══════════════╤════════╕\n │ │ train │ validation │ test │\n ╞═══════════════╪═════════╪══════════════╪════════╡\n │ accuracy │ 0.8157 │ 0.6966 │ 0.8090 │\n ├───────────────┼─────────┼──────────────┼────────┤\n │ loss │ 0.4619 │ 0.5039 │ 0.4488 │\n ├───────────────┼─────────┼──────────────┼────────┤\n │ precision │ 0.8274 │ 0.6250 │ 0.7818 │\n ├───────────────┼─────────┼──────────────┼────────┤\n │ recall │ 0.6680 │ 0.4545 │ 0.6615 │\n ├───────────────┼─────────┼──────────────┼────────┤\n │ roc_auc │ 0.8471 │ 0.7706 │ 0.8592 │\n ├───────────────┼─────────┼──────────────┼────────┤\n │ specificity │ 0.9105 │ 0.8393 │ 0.8938 │\n ├───────────────┼─────────┼──────────────┼────────┤\n │ combined_loss │ 0.4619 │ 0.5039 │ 0.4488 │\n ╘═══════════════╧═════════╧══════════════╧════════╛\n \"\"\"\n # Obtain the combined loss, which is the same across all output features.\n combined_loss_for_each_split = [\n get_metric_value_or_empty(train_metrics_log[COMBINED], LOSS),\n get_metric_value_or_empty(validation_metrics_log[COMBINED], LOSS),\n get_metric_value_or_empty(test_metrics_log[COMBINED], LOSS),\n ]\n\n for output_feature_name in sorted(output_features.keys()):\n if output_feature_name == COMBINED:\n # Skip the combined output feature. The combined loss will be added to each output feature's table.\n continue\n print_table_for_single_output_feature(\n train_metrics_log[output_feature_name],\n validation_metrics_log[output_feature_name],\n test_metrics_log[output_feature_name],\n combined_loss_for_each_split,\n )\n"
},
{
"path": "ludwig/utils/misc_utils.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport copy\nimport functools\nimport os\nimport random\nimport subprocess\nimport weakref\nfrom collections import OrderedDict\nfrom collections.abc import Mapping\nfrom typing import Any, TYPE_CHECKING\n\nimport numpy\nimport torch\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import PROC_COLUMN\nfrom ludwig.globals import DESCRIPTION_FILE_NAME, MODEL_FILE_NAME\nfrom ludwig.utils import fs_utils\nfrom ludwig.utils.fs_utils import find_non_existing_dir_by_adding_suffix\n\nif TYPE_CHECKING:\n from ludwig.schema.model_types.base import ModelConfig\n\n\n@DeveloperAPI\ndef set_random_seed(random_seed):\n os.environ[\"PYTHONHASHSEED\"] = str(random_seed)\n random.seed(random_seed)\n numpy.random.seed(random_seed)\n torch.manual_seed(random_seed)\n if torch.cuda.is_available() and torch.cuda.device_count() > 0:\n torch.cuda.manual_seed(random_seed)\n\n\n@DeveloperAPI\ndef merge_dict(dct, merge_dct):\n \"\"\"Recursive dict merge. Inspired by :meth:``dict.update()``, instead of updating only top-level keys,\n dict_merge recurses down into dicts nested to an arbitrary depth, updating keys. The ``merge_dct`` is merged\n into ``dct``.\n\n :param dct: dict onto which the merge is executed\n :param merge_dct: dct merged into dct\n :return: None\n \"\"\"\n dct = copy.deepcopy(dct)\n for k, v in merge_dct.items():\n if k in dct and isinstance(dct[k], dict) and isinstance(merge_dct[k], Mapping):\n dct[k] = merge_dict(dct[k], merge_dct[k])\n else:\n dct[k] = merge_dct[k]\n return dct\n\n\n@DeveloperAPI\ndef sum_dicts(dicts, dict_type=dict):\n summed_dict = dict_type()\n for d in dicts:\n for key, value in d.items():\n if key in summed_dict:\n prev_value = summed_dict[key]\n if isinstance(value, (dict, OrderedDict)):\n summed_dict[key] = sum_dicts([prev_value, value], dict_type=type(value))\n elif isinstance(value, numpy.ndarray):\n summed_dict[key] = numpy.concatenate((prev_value, value))\n else:\n summed_dict[key] = prev_value + value\n else:\n summed_dict[key] = value\n return summed_dict\n\n\n@DeveloperAPI\ndef get_from_registry(key, registry):\n if hasattr(key, \"lower\"):\n key = key.lower()\n if key in registry:\n return registry[key]\n else:\n raise ValueError(f\"Key '{key}' not in registry, available options: {registry.keys()}\")\n\n\n@DeveloperAPI\ndef set_default_value(dictionary, key, value):\n if key not in dictionary:\n dictionary[key] = value\n\n\n@DeveloperAPI\ndef set_default_values(dictionary: dict, default_value_dictionary: dict):\n \"\"\"This function sets multiple default values recursively for various areas of the config. By using the helper\n function set_default_value, It parses input values that contain nested dictionaries, only setting values for\n parameters that have not already been defined by the user.\n\n Args:\n dictionary (dict): The dictionary to set default values for, generally a section of the config.\n default_value_dictionary (dict): The dictionary containing the default values for the config.\n \"\"\"\n for key, value in default_value_dictionary.items():\n if key not in dictionary: # Event where the key is not in the dictionary yet\n dictionary[key] = value\n elif value == {}: # Event where dict is empty\n set_default_value(dictionary, key, value)\n elif isinstance(value, dict) and value: # Event where dictionary is nested - recursive call\n set_default_values(dictionary[key], value)\n else:\n set_default_value(dictionary, key, value)\n\n\n@DeveloperAPI\ndef get_class_attributes(c):\n return {i for i in dir(c) if not callable(getattr(c, i)) and not i.startswith(\"_\")}\n\n\n@DeveloperAPI\ndef get_output_directory(output_directory, experiment_name, model_name=\"run\"):\n base_dir_name = os.path.join(output_directory, experiment_name + (\"_\" if model_name else \"\") + (model_name or \"\"))\n return fs_utils.abspath(find_non_existing_dir_by_adding_suffix(base_dir_name))\n\n\n@DeveloperAPI\ndef get_file_names(output_directory):\n description_fn = os.path.join(output_directory, DESCRIPTION_FILE_NAME)\n training_stats_fn = os.path.join(output_directory, \"training_statistics.json\")\n\n model_dir = os.path.join(output_directory, MODEL_FILE_NAME)\n\n return description_fn, training_stats_fn, model_dir\n\n\n@DeveloperAPI\ndef get_combined_features(config):\n return config[\"input_features\"] + config[\"output_features\"]\n\n\n@DeveloperAPI\ndef get_proc_features(config):\n return get_proc_features_from_lists(config[\"input_features\"], config[\"output_features\"])\n\n\n@DeveloperAPI\ndef get_proc_features_from_lists(*args):\n return {feature[PROC_COLUMN]: feature for features in args for feature in features}\n\n\n@DeveloperAPI\ndef set_saved_weights_in_checkpoint_flag(config_obj: \"ModelConfig\"):\n \"\"\"Adds a flag to all input feature encoder configs indicating that the weights are saved in the checkpoint.\n\n Next time the model is loaded we will restore pre-trained encoder weights from ludwig model (and not load from cache\n or model hub).\n \"\"\"\n for input_feature in config_obj.input_features:\n encoder_obj = input_feature.encoder\n encoder_obj.saved_weights_in_checkpoint = True\n\n\n@DeveloperAPI\ndef remove_empty_lines(str):\n return \"\\n\".join([line.rstrip() for line in str.split(\"\\n\") if line.rstrip()])\n\n\n@DeveloperAPI\ndef memoized_method(*lru_args, **lru_kwargs):\n def decorator(func):\n @functools.wraps(func)\n def wrapped_func(self, *args, **kwargs):\n # We're storing the wrapped method inside the instance. If we had\n # a strong reference to self the instance would never die.\n self_weak = weakref.ref(self)\n\n @functools.wraps(func)\n @functools.lru_cache(*lru_args, **lru_kwargs)\n def cached_method(*args, **kwargs):\n return func(self_weak(), *args, **kwargs)\n\n setattr(self, func.__name__, cached_method)\n return cached_method(*args, **kwargs)\n\n return wrapped_func\n\n return decorator\n\n\n@DeveloperAPI\ndef get_commit_hash():\n \"\"\"If Ludwig is run from a git repository, get the commit hash of the current HEAD.\n\n Returns None if git is not executable in the current environment or Ludwig is not run in a git repo.\n \"\"\"\n try:\n with open(os.devnull, \"w\") as devnull:\n is_a_git_repo = subprocess.call([\"git\", \"branch\"], stderr=subprocess.STDOUT, stdout=devnull) == 0\n if is_a_git_repo:\n commit_hash = subprocess.check_output([\"git\", \"rev-parse\", \"HEAD\"]).decode(\"utf-8\")\n return commit_hash\n except: # noqa: E722\n pass\n return None\n\n\n@DeveloperAPI\ndef scrub_creds(config_dict: dict[str, Any]) -> dict[str, Any]:\n \"\"\"Returns a copy of a config dict with all sensitive fields scrubbed.\"\"\"\n if config_dict.get(\"backend\", {}) and \"credentials\" in config_dict.get(\"backend\", {}):\n config_dict[\"backend\"][\"credentials\"] = {}\n return config_dict\n"
},
{
"path": "ludwig/utils/model_utils.py",
"content": "import logging\nfrom collections import OrderedDict\n\nimport numpy as np\nimport torch\n\nlogger = logging.getLogger(__name__)\n\nNUMPY_TO_TORCH_DTYPE = {\n bool: torch.bool,\n np.bool_: torch.bool,\n np.uint8: torch.uint8,\n np.int8: torch.int8,\n np.int16: torch.int16,\n np.int32: torch.int32,\n np.int64: torch.int64,\n np.float16: torch.float16,\n np.float32: torch.float32,\n np.float64: torch.float64,\n np.complex64: torch.complex64,\n np.complex128: torch.complex128,\n}\n\n\ndef extract_tensors(model: torch.nn.Module) -> tuple[torch.nn.Module, list[dict]]:\n \"\"\"Remove the tensors from a PyTorch model, convert them to NumPy arrays, and return the stripped model and\n tensors.\n\n Reference implementation: https://medium.com/ibm-data-ai/how-to-load-pytorch-models-340-times-faster-with-\n ray-8be751a6944c # noqa\n \"\"\"\n\n tensors = []\n for _, module in model.named_modules():\n # Store the tensors as numpy arrays in Python dictionaries\n # Delete the same tensors since we no longer need them and we want to reduce memory pressure.\n # This ensures that throughout this process, we keep memory nearly linear w.r.t model parameters.\n params = OrderedDict()\n buffers = OrderedDict()\n for name, param in module.named_parameters(recurse=False):\n params[name] = torch.clone(param).detach().numpy()\n del param\n for name, buf in module.named_buffers(recurse=False):\n buffers[name] = torch.clone(buf).detach().numpy()\n del buf\n tensors.append({\"params\": params, \"buffers\": buffers})\n\n # Strip all tensors and buffers out of the original model.\n for _, module in model.named_modules():\n for name in [name for name, _ in module.named_parameters(recurse=False)] + [\n name for name, _ in module.named_buffers(recurse=False)\n ]:\n setattr(module, name, None)\n\n return model, tensors\n\n\ndef replace_tensors(m: torch.nn.Module, tensors: list[dict], device: torch.device):\n \"\"\"Restore the tensors that extract_tensors() stripped out of a PyTorch model. This operation is performed in\n place.\n\n Reference implementation: https://medium.com/ibm-data-ai/how-to-load-pytorch-models-340-times-faster-with-\n ray-8be751a6944c # noqa\n \"\"\"\n modules = [module for _, module in m.named_modules()]\n for module, tensor_dict in zip(modules, tensors):\n # There are separate APIs to set parameters and buffers.\n for name, array in tensor_dict[\"params\"].items():\n module.register_parameter(\n name,\n torch.nn.Parameter(torch.as_tensor(array, device=device, dtype=NUMPY_TO_TORCH_DTYPE.get(array.dtype))),\n )\n\n for name, array in tensor_dict[\"buffers\"].items():\n module.register_buffer(\n name,\n torch.as_tensor(array, device=device, dtype=NUMPY_TO_TORCH_DTYPE.get(array.dtype)),\n )\n\n\ndef find_embedding_layer_with_path(module, module_names=[]):\n \"\"\"Recursively search through a module to find an embedding layer and its module path.\n\n Returns a tuple containing the embedding layer and its module path.\n \"\"\"\n for name, child_module in module.named_children():\n if isinstance(child_module, torch.nn.Embedding):\n # If an embedding layer is found, return it along with the module path\n return child_module, \".\".join(module_names + [name])\n else:\n # Recursively search in the child module and update the module_names list\n found, path = find_embedding_layer_with_path(child_module, module_names + [name])\n if found is not None:\n return found, path\n return None, None\n\n\ndef contains_nan_or_inf_tensors(module: torch.nn.Module) -> bool:\n \"\"\"Check for NaN or infinity (inf) values in the tensors (parameters and buffers) of a PyTorch module. This\n function recursively inspects the module's parameters and buffers to identify NaN or inf values. It is designed\n to ensure the numerical stability of the model by detecting any irregularities in the tensor values.\n\n Parameters:\n module (torch.nn.Module): The PyTorch module to check for NaN or inf values.\n\n Returns:\n bool: Returns True if any NaN or inf values are found in the module's tensors. Otherwise, returns False.\n \"\"\"\n for name, param in module.named_parameters():\n if param.requires_grad and (torch.isnan(param).any() or torch.isinf(param).any()):\n logger.info(f\"Found NaN or inf values in parameter '{name}' of module '{module.__class__.__name__}'\")\n return True\n\n for name, buffer in module.named_buffers():\n if torch.isnan(buffer).any() or torch.isinf(buffer).any():\n logger.info(f\"Found NaN or inf values in buffer '{name}' of module '{module.__class__.__name__}'\")\n return True\n\n for name, submodule in module.named_children():\n if contains_nan_or_inf_tensors(submodule):\n logger.info(f\"Found NaN or inf values in submodule '{name}' of module '{module.__class__.__name__}'\")\n return True\n\n return False\n"
},
{
"path": "ludwig/utils/nlp_utils.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nimport sys\n\nlogger = logging.getLogger(__name__)\n\nnlp_pipelines = {\n \"en\": None,\n \"it\": None,\n \"es\": None,\n \"de\": None,\n \"fr\": None,\n \"pt\": None,\n \"nl\": None,\n \"el\": None,\n \"nb\": None,\n \"lt\": None,\n \"da\": None,\n \"pl\": None,\n \"ro\": None,\n \"ja\": None,\n \"zh\": None,\n \"xx\": None,\n}\nlanguage_module_registry = {\n \"en\": \"en_core_web_sm\",\n \"it\": \"it_core_news_sm\",\n \"es\": \"es_core_news_sm\",\n \"de\": \"de_core_news_sm\",\n \"fr\": \"fr_core_news_sm\",\n \"pt\": \"pt_core_news_sm\",\n \"nl\": \"nl_core_news_sm\",\n \"el\": \"el_core_news_sm\",\n \"nb\": \"nb_core_news_sm\",\n \"lt\": \"lt_core_news_sm\",\n \"da\": \"da_core_news_sm\",\n \"pl\": \"pl_core_news_sm\",\n \"ro\": \"ro_core_news_sm\",\n \"ja\": \"ja_core_news_sm\",\n \"zh\": \"zh_core_web_sm\",\n \"xx\": \"xx_ent_wiki_sm\",\n}\ndefault_characters = [\n \" \",\n \"a\",\n \"b\",\n \"c\",\n \"d\",\n \"e\",\n \"f\",\n \"g\",\n \"h\",\n \"i\",\n \"j\",\n \"k\",\n \"l\",\n \"m\",\n \"n\",\n \"o\",\n \"p\",\n \"q\",\n \"r\",\n \"s\",\n \"t\",\n \"u\",\n \"v\",\n \"w\",\n \"x\",\n \"y\",\n \"z\",\n \"0\",\n \"1\",\n \"2\",\n \"3\",\n \"4\",\n \"5\",\n \"6\",\n \"8\",\n \"9\",\n \"-\",\n \",\",\n \";\",\n \".\",\n \"!\",\n \"?\",\n \":\",\n \"'\",\n \"'\",\n \"/\",\n \"\\\\\",\n \"|\",\n \"_\",\n \"@\",\n \"#\",\n \"$\",\n \"%\",\n \"^\",\n \"&\",\n \"*\",\n \"~\",\n \"`\",\n \"+\",\n \"-\",\n \"=\",\n \"<\",\n \">\",\n \"(\",\n \")\",\n \"[\",\n \"]\",\n \"{\",\n \"}\",\n]\npunctuation = {\".\", \",\", \"@\", \"$\", \"%\", \"/\", \":\", \";\", \"+\", \"=\"}\n\n\ndef load_nlp_pipeline(language=\"xx\"):\n if language not in language_module_registry:\n logger.error(\n \"Language {} is not supported.\"\n \"Suported languages are: {}\".format(language, language_module_registry.keys())\n )\n raise ValueError\n else:\n spacy_module_name = language_module_registry[language]\n if nlp_pipelines[language] is None:\n logger.info(\"Loading NLP pipeline\")\n try:\n import spacy\n except ImportError:\n logger.error(\n \" spacy is not installed. \"\n \"In order to install all text feature dependencies run \"\n \"pip install ludwig[text]\"\n )\n sys.exit(-1)\n\n try:\n nlp_pipelines[language] = spacy.load(spacy_module_name, disable=[\"parser\", \"tagger\", \"ner\"])\n except OSError:\n logger.info(\" spaCy {} model is missing, downloading it \" \"(this will only happen once)\")\n from spacy.cli import download\n\n download(spacy_module_name)\n nlp_pipelines[language] = spacy.load(spacy_module_name, disable=[\"parser\", \"tagger\", \"ner\"])\n\n return nlp_pipelines[language]\n\n\ndef pass_filters(\n token, filter_numbers=False, filter_punctuation=False, filter_short_tokens=False, filter_stopwords=False\n):\n passes_filters = True\n if filter_numbers:\n passes_filters = not token.like_num\n if passes_filters and filter_punctuation:\n passes_filters = not bool(set(token.orth_) & punctuation)\n if passes_filters and filter_short_tokens:\n passes_filters = len(token) > 2\n if passes_filters and filter_stopwords:\n passes_filters = not token.is_stop\n return passes_filters\n\n\ndef process_text(\n text,\n nlp_pipeline,\n return_lemma=False,\n filter_numbers=False,\n filter_punctuation=False,\n filter_short_tokens=False,\n filter_stopwords=False,\n):\n doc = nlp_pipeline(text)\n return [\n token.lemma_ if return_lemma else token.text\n for token in doc\n if pass_filters(token, filter_numbers, filter_punctuation, filter_short_tokens, filter_stopwords)\n ]\n\n\nif __name__ == \"__main__\":\n text = (\n \"Hello John, how are you doing my good old friend? Are you still number 732 in the list? Did you pay $32.43 or \"\n \"54.21 for the book?\"\n )\n print(process_text(text, load_nlp_pipeline()))\n print(\n process_text(text, load_nlp_pipeline(), filter_numbers=True, filter_punctuation=True, filter_short_tokens=True)\n )\n print(process_text(text, load_nlp_pipeline(), filter_stopwords=True))\n print(process_text(text, load_nlp_pipeline(), return_lemma=True))\n print(\n process_text(\n text,\n load_nlp_pipeline(),\n return_lemma=True,\n filter_numbers=True,\n filter_punctuation=True,\n filter_short_tokens=True,\n )\n )\n print(process_text(text, load_nlp_pipeline(), return_lemma=True, filter_stopwords=True))\n"
},
{
"path": "ludwig/utils/numerical_test_utils.py",
"content": "# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nfrom typing import Any\n\nimport numpy as np\n\n\ndef _dict_like(x):\n \"\"\"Returns true if an object is a dict or convertible to one, false if not.\"\"\"\n try:\n _ = dict(x)\n except (TypeError, ValueError):\n return False\n return True\n\n\ndef _enumerable(x):\n \"\"\"Returns true if an object is enumerable, false if not.\"\"\"\n try:\n _ = enumerate(x)\n except (TypeError, ValueError):\n return False\n return True\n\n\ndef assert_all_finite(x: Any, keypath=\"\"):\n \"\"\"Ensures that all scalars at all levels of the dictionary, list, array, or scalar are finite.\n\n keypath is only used for logging error messages, to indicate where the non-finite value was detected.\n \"\"\"\n path_description = f\" at {keypath} \" if keypath else \" \"\n if np.isscalar(x):\n assert np.isfinite(x), f\"Value{path_description}should be finite, but is {str(x)}.\"\n elif isinstance(x, np.ndarray):\n non_finite_indices = np.nonzero(~np.isfinite(x))\n non_finite_values = x[non_finite_indices]\n assert np.all(np.isfinite(x)), (\n f\"All values{path_description}should be finite, but found {str(non_finite_values)} \"\n \"at positions {str(np.array(non_finite_indices).flatten())}.\"\n )\n elif _dict_like(x):\n # x is either a dict or convertible to one\n for k, v in dict(x).items():\n assert_all_finite(v, keypath=keypath + \".\" + str(k) if keypath else str(k))\n elif _enumerable(x):\n # x is a list, set or other enumerable type, but not a string, dict, or numpy array.\n for i, v in enumerate(x):\n assert_all_finite(v, keypath=keypath + f\"[{i}]\")\n else:\n assert False, f\"Unhandled type {str(type(x))} for value{path_description}\"\n"
},
{
"path": "ludwig/utils/output_feature_utils.py",
"content": "\"\"\"Utilities used for managing output feature dicts.\"\"\"\n\nimport numpy as np\nimport torch\n\nfrom ludwig.utils.torch_utils import sequence_length_3D, sequence_mask\n\n\ndef get_feature_concat_name(feature_name: str, tensor_name: str) -> str:\n return feature_name + \"::\" + tensor_name\n\n\ndef get_tensor_name_from_concat_name(concat_name: str) -> str:\n return concat_name.split(\"::\")[-1]\n\n\ndef get_feature_name_from_concat_name(concat_name: str) -> str:\n return \"::\".join(concat_name.split(\"::\")[:-1])\n\n\ndef get_single_output_feature_tensors(\n output_feature_dict: dict[str, torch.Tensor], feature_name: str\n) -> dict[str, torch.Tensor]:\n \"\"\"Returns a map of tensors related to the given feature_name.\"\"\"\n single_output_feature_tensors = {}\n for concat_name, tensor in output_feature_dict.items():\n if get_feature_name_from_concat_name(concat_name) == feature_name:\n single_output_feature_tensors[get_tensor_name_from_concat_name(concat_name)] = tensor\n return single_output_feature_tensors\n\n\ndef get_output_feature_tensor(\n output_dict: dict[str, torch.Tensor], feature_name: str, tensor_name: str\n) -> torch.Tensor:\n \"\"\"Returns a tensor related for the given feature_name and tensor_name.\"\"\"\n concat_name = get_feature_concat_name(feature_name, tensor_name)\n if concat_name not in output_dict:\n raise ValueError(\n f\"Could not find {tensor_name} for {feature_name} in the output_dict with keys: {output_dict.keys()}\"\n )\n return output_dict[get_feature_concat_name(feature_name, tensor_name)]\n\n\ndef set_output_feature_tensor(\n output_dict: dict[str, torch.Tensor], feature_name: str, tensor_name: str, tensor: torch.Tensor\n):\n \"\"\"Adds tensor for the given feature_name and tensor_name to the tensor dict.\"\"\"\n output_dict[get_feature_concat_name(feature_name, tensor_name)] = tensor\n\n\ndef concat_dependencies(\n feature_name: str,\n dependencies: list[str],\n dependency_reducers: torch.ModuleDict,\n combiner_hidden_state: torch.Tensor,\n other_output_feature_states: dict[str, torch.Tensor],\n) -> torch.Tensor:\n \"\"\"Concatenates combiner_hidden_state with other output feature hidden states based on listed dependencies.\"\"\"\n # No dependencies.\n if not dependencies:\n return combiner_hidden_state\n\n dependency_hidden_states = []\n for feature_name in dependencies:\n # The dependent feature should be present since ECD does a topological sort over output features.\n feature_hidden_state = other_output_feature_states[feature_name]\n\n # This feature is sequential.\n if len(combiner_hidden_state.shape) > 2:\n if len(feature_hidden_state.shape) > 2:\n # The dependent feature is also sequential.\n # matrix matrix -> concat\n assert combiner_hidden_state.shape[1] == feature_hidden_state.shape[1]\n dependency_hidden_states.append(feature_hidden_state)\n else:\n # The dependent feature is not sequential.\n # matrix vector -> tile concat\n sequence_max_length = combiner_hidden_state.shape[1]\n multipliers = (1, sequence_max_length, 1)\n tiled_representation = torch.tile(torch.unsqueeze(feature_hidden_state, 1), multipliers)\n\n sequence_length = sequence_length_3D(combiner_hidden_state)\n mask = sequence_mask(sequence_length, sequence_max_length)\n tiled_representation = torch.mul(\n tiled_representation,\n mask[:, :, np.newaxis].type(torch.float32),\n )\n\n dependency_hidden_states.append(tiled_representation)\n\n else:\n # This feature is not sequential.\n if len(feature_hidden_state.shape) > 2:\n # The dependent feature is sequential.\n # vector matrix -> reduce concat\n reducer = dependency_reducers[feature_name]\n dependency_hidden_states.append(reducer(feature_hidden_state))\n else:\n # The dependent feature is not sequential.\n # vector vector -> concat\n dependency_hidden_states.append(feature_hidden_state)\n\n try:\n hidden = torch.cat([combiner_hidden_state] + dependency_hidden_states, dim=-1)\n except Exception as e:\n raise ValueError(\n f\"Shape mismatch {e} while concatenating dependent features of {feature_name}: \"\n f\"{dependencies}. Concatenating the feature activations tensor {combiner_hidden_state} \"\n f\"with activation tensors of dependencies: {dependency_hidden_states}. The error is \"\n \"likely due to a mismatch of the second dimension (sequence length) or a \"\n \"difference in ranks. Likely solutions are setting the maximum_sequence_length \"\n \"of all sequential features to be the same, or reduce the output of some \"\n \"features, or disabling the bucketing setting bucketing_field to None / null, \"\n \"as activating it will reduce the length of the field the bucketing is \"\n \"performed on.\"\n )\n return hidden\n"
},
{
"path": "ludwig/utils/package_utils.py",
"content": "import importlib\nimport types\n\n\nclass LazyLoader(types.ModuleType):\n \"\"\"Lazily import a module, mainly to avoid pulling in large dependencies.\n\n `contrib`, and `ffmpeg` are examples of modules that are large and not always\n needed, and this allows them to only be loaded when they are used.\n\n Copied from: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/util/lazy_loader.py\n \"\"\"\n\n # The lint error here is incorrect.\n def __init__(self, local_name, parent_module_globals, name): # pylint: disable=super-on-old-class\n self._local_name = local_name\n self._parent_module_globals = parent_module_globals\n\n super().__init__(name)\n\n def _load(self):\n # Import the target module and insert it into the parent's namespace\n module = importlib.import_module(self.__name__)\n self._parent_module_globals[self._local_name] = module\n\n # Update this object's dict so that if someone keeps a reference to the\n # LazyLoader, lookups are efficient (__getattr__ is only called on lookups\n # that fail).\n self.__dict__.update(module.__dict__)\n\n return module\n\n def __getattr__(self, item):\n module = self._load()\n return getattr(module, item)\n\n def __dir__(self):\n module = self._load()\n return dir(module)\n"
},
{
"path": "ludwig/utils/print_utils.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nfrom collections import OrderedDict\nfrom pprint import pformat\n\nfrom ludwig.api_annotations import DeveloperAPI\n\nlogger = logging.getLogger(__name__)\n\n\n@DeveloperAPI\ndef get_logging_level_registry() -> dict[str, int]:\n return {\n \"critical\": logging.CRITICAL,\n \"error\": logging.ERROR,\n \"warning\": logging.WARNING,\n \"info\": logging.INFO,\n \"debug\": logging.DEBUG,\n \"notset\": logging.NOTSET,\n }\n\n\n@DeveloperAPI\ndef get_logo(message, ludwig_version):\n return \"\\n\".join(\n [\n \"███████████████████████\",\n \"█ █ █ █ ▜█ █ █ █ █ █\",\n \"█ █ █ █ █ █ █ █ █ █ ███\",\n \"█ █ █ █ █ █ █ █ █ ▌ █\",\n \"█ █████ █ █ █ █ █ █ █ █\",\n \"█ █ ▟█ █ █ █\",\n \"███████████████████████\",\n f\"ludwig v{ludwig_version} - {message}\",\n \"\",\n ]\n )\n\n\n@DeveloperAPI\ndef print_ludwig(message, ludwig_version):\n logger.info(get_logo(message, ludwig_version))\n\n\n@DeveloperAPI\ndef print_boxed(text, print_fun=logger.info):\n box_width = len(text) + 2\n print_fun(\"\")\n print_fun(\"╒{}╕\".format(\"═\" * box_width))\n print_fun(f\"│ {text.upper()} │\")\n print_fun(\"╘{}╛\".format(\"═\" * box_width))\n print_fun(\"\")\n\n\n@DeveloperAPI\ndef repr_ordered_dict(d: OrderedDict):\n return \"{\" + \",\\n \".join(f\"{x}: {pformat(y, indent=4)}\" for x, y in d.items()) + \"}\"\n\n\n@DeveloperAPI\ndef query_yes_no(question: str, default: str | None = \"yes\"):\n \"\"\"Ask a yes/no question via raw_input() and return their answer.\n\n Args:\n question: String presented to the user\n default: The presumed answer from the user. Must be \"yes\", \"no\", or None (Answer is required)\n\n Returns: Boolean based on prompt response\n \"\"\"\n valid = {\"yes\": True, \"y\": True, \"ye\": True, \"no\": False, \"n\": False}\n if default is None:\n prompt = \" [y/n] \"\n elif default == \"yes\":\n prompt = \" [Y/n] \"\n elif default == \"no\":\n prompt = \" [y/N] \"\n else:\n raise ValueError(\"invalid default answer: '%s'\" % default)\n\n while True:\n logger.info(question + prompt)\n choice = input().lower()\n if default is not None and choice == \"\":\n return valid[default]\n elif choice in valid:\n return valid[choice]\n else:\n logger.info(\"Please respond with 'yes' or 'no' \" \"(or 'y' or 'n').\\n\")\n"
},
{
"path": "ludwig/utils/registry.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2020 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nfrom collections import UserDict\nfrom typing import Generic, TypeVar\n\nDEFAULT_KEYS = [\"None\", \"none\", \"null\", None]\n\n\nT = TypeVar(\"T\")\n\n\nclass Registry(UserDict, Generic[T]):\n \"\"\"Registry is like a normal dict, but with an optional parent dict.\n\n Items are considered to exist in the registry if they are added to either the registry itself, or its parent.\n \"\"\"\n\n def __init__(self, source=None):\n init_data = None\n parent = {}\n if isinstance(source, Registry):\n parent = source\n else:\n init_data = source\n\n self.parent = parent\n super().__init__(init_data)\n\n def __getitem__(self, key: str) -> T:\n if self.parent and key not in self.data:\n return self.parent.__getitem__(key)\n return self.data.__getitem__(key)\n\n def __contains__(self, key: str):\n return key in self.data or key in self.parent\n\n def __len__(self) -> int:\n return len(self.data) + len(self.parent)\n\n def __iter__(self):\n return self._merged().__iter__()\n\n def keys(self):\n return self._merged().keys()\n\n def values(self):\n return self._merged().values()\n\n def items(self):\n return self._merged().items()\n\n def _merged(self):\n return {**self.parent, **self.data}\n\n def register(self, name: str, default: bool = False):\n def wrap(cls):\n self[name] = cls\n if default:\n for key in DEFAULT_KEYS:\n self[key] = cls\n return cls\n\n return wrap\n"
},
{
"path": "ludwig/utils/server_utils.py",
"content": "import json\nimport os\nimport tempfile\nfrom typing import Any\n\nimport numpy as np\nimport pandas as pd\nfrom starlette.datastructures import UploadFile\nfrom starlette.responses import JSONResponse\n\nfrom ludwig.utils.data_utils import NumpyEncoder\n\n\ndef serialize_payload(data_source: pd.DataFrame | pd.Series) -> tuple:\n \"\"\"\n Generates two dictionaries to be sent via REST API for Ludwig prediction\n service.\n First dictionary created is payload_dict. Keys found in payload_dict:\n raw_data: this is json string created by pandas to_json() method\n source_type: indicates if the data_source is either a pandas dataframe or\n pandas series. This is needed to know how to rebuild the structure.\n ndarray_dtype: this is a dictionary where each entry is for any ndarray\n data found in the data_source. This could be an empty dictioinary if no\n ndarray objects are present in data_source. Key for this dictionary is\n column name if data_source is dataframe or index name if data_source is\n series. The value portion of the dictionary is the dtype of the\n ndarray. This value is used to set the correct dtype when rebuilding\n the entry.\n\n Second dictionary created is called payload_files, this contains information\n and content for files to be sent to the server. NOTE: if no files are to be\n sent, this will be an empty dictionary.\n Entries in this dictionary:\n Key: file path string for file to be sent to server\n Value: tuple(file path string, byte encoded file content,\n 'application/octet-stream')\n\n Args:\n data_source: input features to be sent to Ludwig server\n\n Returns: tuple(payload_dict, payload_files)\n\n \"\"\"\n payload_dict = {}\n payload_dict[\"ndarray_dtype\"] = {}\n payload_files = {}\n if isinstance(data_source, pd.DataFrame):\n payload_dict[\"raw_data\"] = data_source.to_json(orient=\"columns\")\n payload_dict[\"source_type\"] = \"dataframe\"\n for col in data_source.columns:\n if isinstance(data_source[col].iloc[0], np.ndarray):\n # if we have any ndarray columns, record dtype\n payload_dict[\"ndarray_dtype\"][col] = str(data_source[col].iloc[0].dtype)\n elif isinstance(data_source[col].iloc[0], str) and os.path.exists(data_source[col].iloc[0]):\n # if we have file path feature, prepare file for transport\n for v in data_source[col]:\n payload_files[v] = (v, open(v, \"rb\"), \"application/octet-stream\")\n elif isinstance(data_source, pd.Series):\n payload_dict[\"raw_data\"] = data_source.to_json(orient=\"index\")\n payload_dict[\"source_type\"] = \"series\"\n for col in data_source.index:\n if isinstance(data_source[col], np.ndarray):\n # for ndarrays record dtype for reconstruction\n payload_dict[\"ndarray_dtype\"][col] = str(data_source[col].dtype)\n elif isinstance(data_source[col], str) and os.path.exists(data_source[col]):\n # if we have file path feature, prepare file for transport\n v = data_source[col]\n payload_files[v] = (v, open(v, \"rb\"), \"application/octet-stream\")\n else:\n ValueError(\n '\"data_source\" must be either a pandas DataFrame or Series, '\n \"format found to be {}\".format(type(data_source))\n )\n\n return payload_dict, payload_files\n\n\ndef _write_file(v, files):\n # Convert UploadFile to a NamedTemporaryFile to ensure it's on the disk\n suffix = os.path.splitext(v.filename)[1]\n named_file = tempfile.NamedTemporaryFile(delete=False, suffix=suffix)\n files.append(named_file)\n named_file.write(v.file.read())\n named_file.close()\n return named_file.name\n\n\ndef deserialize_payload(json_string: str) -> pd.DataFrame:\n \"\"\"This function performs the inverse of the serialize_payload function and rebuilds the object represented in\n json_string to a pandas DataFrame.\n\n Args:\n json_string: representing object to be rebuilt.\n\n Returns: pandas.DataFrame\n \"\"\"\n payload_dict = json.loads(json_string)\n\n # extract raw data from json string\n raw_data_dict = json.loads(payload_dict[\"raw_data\"])\n # rebuild based on original data source\n if payload_dict[\"source_type\"] == \"dataframe\":\n # reconstitute the pandas dataframe\n df = pd.DataFrame.from_dict(raw_data_dict, orient=\"columns\")\n elif payload_dict[\"source_type\"] == \"series\":\n # reconstitute series into single row dataframe\n df = pd.DataFrame(pd.Series(raw_data_dict)).T\n else:\n ValueError(\n 'Unknown \"source_type\" found. Valid values are \"dataframe\" or '\n '\"series\". Instead found {}'.format(payload_dict[\"source_type\"])\n )\n\n # if source has ndarrays, rebuild those from list and set\n # original dtype.\n if payload_dict[\"ndarray_dtype\"]:\n # yes, now covert list representation to ndarray representation\n for col in payload_dict[\"ndarray_dtype\"]:\n dtype = payload_dict[\"ndarray_dtype\"][col]\n df[col] = df[col].apply(lambda x: np.array(x).astype(dtype))\n\n return df\n\n\ndef deserialize_request(form) -> tuple:\n \"\"\"This function will deserialize the REST API request packet to create a pandas dataframe that is input to the\n Ludwig predict method and a list of files that will be cleaned up at the end of processing.\n\n Args:\n form: REST API provide form data\n\n Returns: tuple(pandas.DataFrame, list of temporary files to clean up)\n \"\"\"\n files = []\n file_index = {}\n for k, v in form.multi_items():\n if type(v) is UploadFile:\n file_index[v.filename] = _write_file(v, files)\n\n # reconstruct the dataframe\n df = deserialize_payload(form[\"payload\"])\n\n # insert files paths of the temporary files in place of the original\n # file paths specified by the user.\n # pd.DataFrame.replace() method is used to replace file path string\n # specified by the user context with the file path string where a\n # temporary file containing the same content.\n # parameters for replace() method:\n # to_replace: list of file path strings that the user provided\n # value: list of temporary files created for each input file\n #\n # IMPORTANT: There is a one-to-one correspondence of the to_replace list\n # and the value list. Each list must be the same size.\n df.replace(to_replace=list(file_index.keys()), value=list(file_index.values()), inplace=True)\n\n return df, files\n\n\nclass NumpyJSONResponse(JSONResponse):\n def render(self, content: dict[str, Any]) -> str:\n \"\"\"Override the default JSONResponse behavior to encode numpy arrays.\n\n Args:\n content: JSON object to be serialized.\n\n Returns: str\n \"\"\"\n return json.dumps(\n content, ensure_ascii=False, allow_nan=False, indent=None, separators=(\",\", \":\"), cls=NumpyEncoder\n ).encode(\"utf-8\")\n"
},
{
"path": "ludwig/utils/state_dict_backward_compatibility.py",
"content": "# Copyright (c) 2023 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\n\ndef _update_transformers_to_freeze_module(state_dict):\n \"\"\"Updates pre-trained encoders which were saved prior to the addition of FreezeModule.\"\"\"\n return {\n (\n k.replace(\"encoder_obj.transformer.\", \"encoder_obj.transformer.module.\")\n if \"encoder_obj.transformer.module\" not in k\n else k\n ): v\n for k, v in state_dict.items()\n }\n\n\ndef _update_combiner_no_input_features(state_dict):\n \"\"\"Removed combiner.input_features from state_dict following DeepSpeed integration.\"\"\"\n return {k: v for k, v in state_dict.items() if not k.startswith(\"combiner.input_features.\")}\n\n\ndef _update_combiner_no_device_tensor(state_dict):\n \"\"\"Removed device_tensor from state_dict following DeepSpeed integration.\"\"\"\n return {k: v for k, v in state_dict.items() if not k.endswith(\"device_tensor\")}\n\n\ndef update_state_dict(state_dict):\n \"\"\"Checks state_dict on load, updates state dict if needed.\"\"\"\n state_dict = _update_transformers_to_freeze_module(state_dict)\n state_dict = _update_combiner_no_input_features(state_dict)\n state_dict = _update_combiner_no_device_tensor(state_dict)\n return state_dict\n"
},
{
"path": "ludwig/utils/strings_utils.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nimport re\nimport unicodedata\nfrom collections import Counter\nfrom dataclasses import dataclass\nfrom enum import Enum\n\nimport numpy as np\nfrom dateutil.parser import parse as parse_datetime\n\nfrom ludwig.constants import PADDING_SYMBOL, START_SYMBOL, STOP_SYMBOL, UNKNOWN_SYMBOL\nfrom ludwig.data.dataframe.base import DataFrameEngine\nfrom ludwig.data.dataframe.pandas import PANDAS\nfrom ludwig.utils.fs_utils import open_file\nfrom ludwig.utils.math_utils import int_type\nfrom ludwig.utils.tokenizers import get_tokenizer_from_registry\nfrom ludwig.utils.types import Series\n\nPANDAS_TRUE_STRS = {\"true\"}\nPANDAS_FALSE_STRS = {\"false\"}\n\nBOOL_TRUE_STRS = {\"yes\", \"y\", \"true\", \"t\", \"1\", \"1.0\"}\nBOOL_FALSE_STRS = {\"no\", \"n\", \"false\", \"f\", \"0\", \"0.0\", \"-1\", \"-1.0\"}\n\nlogger = logging.getLogger(__name__)\n\n\nclass SpecialSymbol(Enum):\n \"\"\"Special symbols used for text features.\"\"\"\n\n STOP = 0\n START = 1\n PADDING = 2\n UNKNOWN = 3\n\n\ndef all_bool_strs():\n \"\"\"Returns all valid boolean strings, with varied capitalization.\"\"\"\n fns = [lambda x: x, lambda x: x.upper(), lambda x: x.capitalize()]\n return sorted({fn(x) for fn in fns for x in BOOL_TRUE_STRS | BOOL_FALSE_STRS})\n\n\ndef make_safe_filename(s):\n def safe_char(c):\n if c.isalnum():\n return c\n else:\n return \"_\"\n\n return \"\".join(safe_char(c) for c in s).rstrip(\"_\")\n\n\ndef strip_accents(s):\n return \"\".join(c for c in unicodedata.normalize(\"NFD\", s) if unicodedata.category(c) != \"Mn\")\n\n\ndef str2bool(v: str, fallback_true_label=None) -> bool:\n \"\"\"Returns bool representation of the given value v.\n\n Check the value against global bool string lists.\n Fallback to using fallback_true_label as True if the value isn't in the global bool lists.\n\n args:\n v: Value to get the bool representation for.\n fallback_true_label: (str) label to use as 'True'.\n \"\"\"\n v_str = str(v).lower()\n if v_str in BOOL_TRUE_STRS:\n return True\n if v_str in BOOL_FALSE_STRS:\n return False\n if fallback_true_label is None:\n raise ValueError(\n f\"Cannot automatically map value '{v}' to a boolean and no `preprocessing.fallback_true_label` specified\"\n )\n return v == fallback_true_label\n\n\ndef values_are_pandas_numbers(values: list[str]):\n \"\"\"Returns True if values would be read by pandas as dtype float or int.\"\"\"\n for v in values:\n try:\n float(v)\n except ValueError:\n return False\n return True\n\n\ndef values_are_pandas_bools(values: list[str]):\n \"\"\"Returns True if values would be read by pandas as dtype bool.\"\"\"\n lowercase_values_set = {str(v).lower() for v in values}\n return lowercase_values_set.issubset(PANDAS_FALSE_STRS | PANDAS_TRUE_STRS)\n\n\ndef are_conventional_bools(values: list[str | bool]) -> bool:\n \"\"\"Returns whether all values are conventional booleans.\"\"\"\n for value in values:\n lower_value = str(value).lower()\n if lower_value not in BOOL_TRUE_STRS and lower_value not in BOOL_FALSE_STRS:\n return False\n return True\n\n\ndef is_number(s: str | int | float):\n \"\"\"Returns whether specified value is number.\"\"\"\n if isinstance(s, str) and s.lower() == \"nan\":\n return True\n try:\n float(s)\n return True\n except ValueError:\n return False\n\n\ndef is_datetime(s: str | int | float):\n \"\"\"Returns whether specified value is datetime.\"\"\"\n if is_number(s):\n return False\n\n try:\n parse_datetime(s)\n return True\n except Exception:\n return False\n\n\ndef are_all_datetimes(values: list[str | int | float]):\n \"\"\"Returns whether all values are datetimes.\"\"\"\n for value in values:\n if not is_datetime(value):\n return False\n return True\n\n\ndef are_all_numbers(values: list[str | int | float]):\n \"\"\"Returns whether all values are numbers.\"\"\"\n for value in values:\n if not is_number(value):\n return False\n return True\n\n\ndef is_integer(s: str | int | float):\n \"\"\"Returns whether specified value is an integer.\"\"\"\n try:\n float(s)\n except ValueError:\n return False\n else:\n return float(s).is_integer() and not np.isnan(float(s))\n\n\ndef are_sequential_integers(values: list[str | int | float]):\n \"\"\"Returns whether distinct values form sequential integer list.\"\"\"\n int_list = []\n for value in values:\n if not is_integer(value):\n return False\n int_list.append(int(float(value)))\n return (max(int_list) - min(int_list) + 1) == len(int_list)\n\n\ndef match_replace(string_to_match, list_regex):\n \"\"\"Matches strings against regular expressions.\n\n arguments:\n string_to_match -- the string to match\n\n returns:\n string_to_match -- the cleaned string\n matched -- the list of regular expressions that matched\n \"\"\"\n matched = []\n for regex in list_regex:\n match = re.search(regex[0], string_to_match)\n if match:\n string_to_match = re.sub(regex[0], regex[1], string_to_match)\n matched.append(regex[0].pattern)\n return string_to_match, matched\n\n\ndef load_vocabulary(vocab_file):\n with open_file(vocab_file, \"r\", encoding=\"utf-8\") as f:\n vocabulary = []\n for line in f:\n line = line.strip()\n if \" \" in line:\n line = line.split(\" \")[0]\n vocabulary.append(line)\n return vocabulary\n\n\ndef add_or_move_symbol(vocab_list: list[str], vocab_set: set[str], symbol: str, index: int):\n \"\"\"Inserts or moves the symbol to the specified index.\"\"\"\n if symbol in vocab_set:\n vocab_list.remove(symbol)\n vocab_list.insert(index, symbol)\n\n\n@dataclass\nclass Vocabulary:\n vocab: list[str]\n \"\"\"List of strings representing the computed vocabulary.\"\"\"\n\n str2idx: dict[str, int]\n \"\"\"Map of symbol to index.\"\"\"\n\n str2freq: dict[str, int]\n \"\"\"Map of symbol to frequency.\"\"\"\n\n str2idf: dict[str, int] | None\n \"\"\"Map of symbol to inverse document frequency.\"\"\"\n\n max_sequence_length: int\n \"\"\"Maximum sequence length.\"\"\"\n\n sequence_length_99ptile: int\n \"\"\"99th percentile of maximum sequence length.\"\"\"\n\n pad_idx: int\n \"\"\"Index to padding symbol.\"\"\"\n\n padding_symbol: str\n \"\"\"Actual padding symbol.\"\"\"\n\n unknown_symbol: str\n \"\"\"Actual unknown symbol.\"\"\"\n\n prompt_template_num_tokens: int = 0\n \"\"\"The number of tokens in the prompt template.\n\n If -1, then there is no prompt template.\n \"\"\"\n\n\ndef _get_vocab_from_dict(vocab: dict[str, int]) -> list[str]:\n \"\"\"Returns a vocab in list format from a vocab token=>idx dictionary.\"\"\"\n vocab_values = list(vocab.values())\n if len(set(vocab_values)) != len(vocab_values):\n raise ValueError(\"Vocabulary has duplicate mappings in its vocabulary. This should never happen.\")\n\n # construct a vocab that is a list that reflects the token=>index mapping in HF's vocab\n # pre-allocate a list to make sure each index is inited to prevent OBO errors caused by missing indices\n max_idx = max(vocab_values)\n vocab_list = [None for _ in range(max_idx + 1)]\n for token, idx in vocab.items():\n vocab_list[idx] = token\n return vocab_list\n\n\ndef _get_vocabulary(\n tokenizer_type: str,\n tokenizer,\n vocab_file: str,\n unknown_symbol: str,\n add_special_symbols: bool,\n padding_symbol: str,\n unit_counts: Counter,\n num_most_frequent: int,\n) -> list[str] | None:\n \"\"\"Returns the vocabulary from the tokenizer_type, tokenizer, or vocab_file.\n\n If the `tokenizer_type` is 'hf_tokenizer', then the set vocabulary from the tokenizer is used.\n\n If there's no vocab_file or if the tokenizer has no set vocabulary (e.g. space_punct), then the vocabulary is\n determined from the tokenized data (unit_counts).\n\n The UNKNOWN special symbol is always included in the final vocabulary. Additional special symbols (PADDING, START,\n STOP) are added if add_special_symbols=True. If the tokenizer is a pre-trained huggingface tokenizer, then the\n special symbols are taken from the tokenizer's vocabulary.\n \"\"\"\n # Pre-trained huggingface tokenizer. Use the pre-existing vocabulary and special symbols.\n if tokenizer_type == \"hf_tokenizer\":\n try:\n return _get_vocab_from_dict(tokenizer.get_vocab())\n except NotImplementedError:\n logger.warning(\n \"HuggingFace tokenizer does not have a get_vocab() method. \"\n + \"Using tokenizer.tokenizer.vocab_size and tokenizer.tokenizer._convert_id_to_token \"\n + \"to build the vocabulary.\"\n )\n vocab = []\n for idx in range(tokenizer.tokenizer.vocab_size):\n vocab.append(tokenizer.tokenizer._convert_id_to_token(idx))\n vocab += tokenizer.tokenizer.added_tokens_encoder.keys()\n return vocab\n\n # The tokenizer has a preset vocabulary.\n if hasattr(tokenizer, \"get_vocab\"):\n return _get_vocab_from_dict(tokenizer.get_vocab())\n\n # Load the vocabulary from the vocab file.\n if vocab_file is not None:\n return load_vocabulary(vocab_file)\n\n # The tokenizer had no preset vocabulary, for example space_punct.\n # Compute the vocabulary from tokenized data.\n return [unit for unit, _ in unit_counts.most_common(num_most_frequent)]\n\n\ndef remove_bracketed_elements(prompt_template: str) -> str:\n \"\"\"Example: -> .\"\"\"\n pattern = r\"\\{.*?\\}\"\n return re.sub(pattern, \"\", prompt_template)\n\n\ndef create_vocabulary(\n data: Series,\n tokenizer_type: str = \"space\",\n lowercase: bool = True,\n num_most_frequent: int = None,\n vocab_file: str = None,\n add_special_symbols: bool = True,\n unknown_symbol: str = UNKNOWN_SYMBOL,\n padding_symbol: str = PADDING_SYMBOL,\n start_symbol: str = START_SYMBOL,\n stop_symbol: str = STOP_SYMBOL,\n pretrained_model_name_or_path: str = None,\n ngram_size: int | None = None,\n compute_idf: bool = False,\n processor: DataFrameEngine = PANDAS,\n prompt_template: str = \"\",\n) -> Vocabulary:\n \"\"\"Computes a vocabulary over the provided data frame.\n\n This function is used when the data consists of multiple tokens within one example. E.g., words in a text feature,\n items in a set feature, etc. If the feature only contains a single token like for category features,\n `create_vocabulary_single_token` should be used instead, as it is more efficient.\n\n A tokenizer is specified using the `tokenizer_type`. The tokenizer will be used to process all of the data\n provided, producing an indexed vocabulary with frequency counts. If the `tokenizer_type` is 'hf_tokenizer',\n then a pre-trained huggingface tokenizer is loaded from `pretrained_model_name_or_path` and that vocabulary is\n used directly.\n\n The UNKNOWN special symbol is always included in the final vocabulary. Additional special symbols (PADDING, START,\n STOP) are added if add_special_symbols=True. If the tokenizer is a pre-trained huggingface tokenizer, then the\n special symbols are taken from the tokenizer's vocabulary.\n\n Args:\n prompt_template: The prompt template for the model. Applicable only to LLMs.\n data: Series of string data.\n tokenizer_type: Tokenizer type. Can be a tokenizer registry value or 'hf_tokenizer' for huggingface.\n lowercase: Whether to lowercase all strings.\n num_most_frequent: Upper limit on vocabulary size.,\n add_special_symbols: If True, START, STOP, PADDING special symbols are added to the vocabulary. UNKNOWN is\n always added.\n unknown_symbol: String representation for the UNKNOWN symbol.\n padding_symbol: String representation for the PADDING symbol.\n start_symbol: String representation for the START symbol.\n stop_symbol: String representation for the STOP symbol.\n pretrained_model_name_or_path: Name/path to huggingface model.\n ngram_size: Size of the n-gram when using `ngram` tokenizer.\n compute_idf: If True, computes the inverse document frequency for each token.\n processor: Which processor to use to process data.\n\n Returns:\n Vocabulary object containing metadata about the vocab.\n\n TODO(Justin): Clean up pad_idx, padding_symbol, unknown_symbol return, as no one seems to be using it.\n \"\"\"\n tokenizer = get_tokenizer_from_registry(tokenizer_type)(\n vocab_file=vocab_file,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n ngram_size=ngram_size,\n )\n\n # Number of tokens in template.\n prompt_template_num_tokens = -1\n if prompt_template:\n prompt_without_bracketed_elements = remove_bracketed_elements(prompt_template)\n prompt_template_num_tokens = len(tokenizer(prompt_without_bracketed_elements))\n\n # Tokenize the data.\n def process_line(line):\n return tokenizer(line.lower() if lowercase else line)\n\n processed_lines = processor.map_objects(data, process_line)\n processed_counts = processed_lines.explode().value_counts(sort=False)\n processed_counts = processor.compute(processed_counts)\n unit_counts = Counter(dict(processed_counts))\n max_sequence_length = processor.compute(processed_lines.map(len).max())\n sequence_length_99ptile = processor.compute(processed_lines.map(len).quantile(0.99))\n\n if tokenizer_type != \"hf_tokenizer\":\n # For non-HF tokenizers, add 2 for start and stop symbols.\n max_sequence_length += 2\n sequence_length_99ptile += 2\n\n pad_idx = None\n if tokenizer_type == \"hf_tokenizer\":\n # Replace the special symbols with the ones from the tokenizer.\n unknown_symbol = tokenizer.get_unk_token()\n padding_symbol = tokenizer.get_pad_token()\n pad_idx = tokenizer.convert_token_to_id(padding_symbol)\n\n vocab: list[str] = _get_vocabulary(\n tokenizer_type,\n tokenizer,\n vocab_file,\n unknown_symbol,\n add_special_symbols,\n padding_symbol,\n unit_counts,\n num_most_frequent,\n )\n vocab_set = set(vocab)\n\n doc_unit_counts = None\n if compute_idf:\n # The document frequency used for TF-IDF. Similar to unit_counts, but de-duped by document.\n document_counts = processed_lines.map(lambda x: set(x)).explode().value_counts(sort=False)\n document_counts = processor.compute(document_counts)\n doc_unit_counts = Counter(dict(document_counts))\n\n if tokenizer_type != \"hf_tokenizer\":\n if add_special_symbols:\n add_or_move_symbol(vocab, vocab_set, stop_symbol, SpecialSymbol.STOP.value)\n add_or_move_symbol(vocab, vocab_set, start_symbol, SpecialSymbol.START.value)\n add_or_move_symbol(vocab, vocab_set, padding_symbol, SpecialSymbol.PADDING.value)\n # Always add the UNKNOWN symbol if we're using our own tokenizer.\n add_or_move_symbol(vocab, vocab_set, unknown_symbol, SpecialSymbol.UNKNOWN.value)\n\n str2idx = {unit: i for i, unit in enumerate(vocab)}\n str2freq = {unit: unit_counts.get(unit) if unit in unit_counts else 0 for unit in vocab}\n str2idf = (\n {unit: np.log(len(vocab) / (1 + doc_unit_counts.get(unit))) if unit in doc_unit_counts else 0 for unit in vocab}\n if compute_idf\n else None\n )\n\n if pad_idx is None and padding_symbol in str2idx.keys():\n pad_idx = str2idx[padding_symbol]\n\n return Vocabulary(\n vocab=vocab,\n str2idx=str2idx,\n str2freq=str2freq,\n str2idf=str2idf,\n max_sequence_length=max_sequence_length,\n sequence_length_99ptile=sequence_length_99ptile,\n pad_idx=pad_idx,\n padding_symbol=padding_symbol,\n unknown_symbol=unknown_symbol,\n prompt_template_num_tokens=prompt_template_num_tokens,\n )\n\n\ndef create_vocabulary_single_token(\n data: Series,\n num_most_frequent: int | None = None,\n processor: DataFrameEngine = PANDAS,\n unknown_symbol: str = UNKNOWN_SYMBOL,\n):\n \"\"\"Computes a vocabulary over the provided data frame.\n\n This function should be used iff the values in each row of data should be considered as a single token, e.g.,\n category features (\"interested\", \"not interested\", \"somewhat interested\").\n\n This assumption allows us to be more efficient than `create_vocabulary()` as we can skip tokenization and\n computing the maximum sequence length, which are unnecessary for category features.\n\n Args:\n data: Series of string data.\n num_most_frequent: Upper limit on vocabulary size.\n unknown_symbol: String representation for the UNKNOWN symbol.\n processor: Which processor to use to process data.\n\n Returns:\n Tuple of:\n vocab: List of strings representing the computed vocabulary.\n str2idx: Map of symbol to index.\n str2freq: Map of symbol to frequency.\n \"\"\"\n processed_counts = data.str.strip().value_counts(sort=True)\n processed_counts = processor.compute(processed_counts)\n full_vocab = processed_counts.index.tolist()\n # Only add unknown symbol if num most frequent tokens is less than total number of unique tokens\n if num_most_frequent < len(full_vocab):\n vocab = [unknown_symbol] + full_vocab[:num_most_frequent]\n else:\n vocab = full_vocab\n str2idx = {unit: i for i, unit in enumerate(vocab)}\n str2freq = processed_counts.to_dict()\n str2freq = {k: str2freq.get(k, 0) for k in vocab}\n return vocab, str2idx, str2freq\n\n\ndef _get_sequence_vector(\n sequence, tokenizer, tokenizer_type, format_dtype, unit_to_id, lowercase=True, unknown_symbol=UNKNOWN_SYMBOL\n) -> np.ndarray:\n unit_sequence = tokenizer(sequence.lower() if lowercase else sequence)\n\n unit_indices_vector = np.empty(len(unit_sequence), dtype=format_dtype)\n for i in range(len(unit_sequence)):\n curr_unit = unit_sequence[i]\n if tokenizer_type == \"hf_tokenizer\":\n unit_indices_vector[i] = curr_unit\n else:\n if curr_unit in unit_to_id:\n unit_indices_vector[i] = unit_to_id[curr_unit]\n else:\n unit_indices_vector[i] = unit_to_id[unknown_symbol]\n\n # Add start and stop symbols.\n # Huggingface's pretrained tokenizers take care of this implicitly:\n # https://huggingface.co/docs/transformers/preprocessing\n if tokenizer_type != \"hf_tokenizer\":\n unit_indices_vector = np.append(unit_indices_vector, unit_to_id[STOP_SYMBOL])\n unit_indices_vector = np.insert(unit_indices_vector, 0, unit_to_id[START_SYMBOL])\n return unit_indices_vector\n\n\ndef build_sequence_matrix(\n sequences, # pd.core.series.Series\n inverse_vocabulary,\n tokenizer_type,\n length_limit,\n padding_symbol=PADDING_SYMBOL,\n padding=\"right\",\n unknown_symbol=UNKNOWN_SYMBOL,\n lowercase=True,\n tokenizer_vocab_file=None,\n pretrained_model_name_or_path=None,\n processor=PANDAS,\n) -> np.ndarray:\n tokenizer = get_tokenizer_from_registry(tokenizer_type)(\n vocab_file=tokenizer_vocab_file,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n )\n\n format_dtype = int_type(len(inverse_vocabulary) - 1)\n\n unit_vectors = sequences.map(\n lambda sequence: _get_sequence_vector(\n sequence,\n tokenizer,\n tokenizer_type,\n format_dtype,\n inverse_vocabulary,\n lowercase=lowercase,\n unknown_symbol=unknown_symbol,\n )\n )\n\n max_length = processor.compute(unit_vectors.map(len).max())\n if max_length < length_limit:\n logger.debug(f\"max length of {format}: {max_length} < limit: {length_limit}\")\n max_length = length_limit\n\n if tokenizer_type == \"hf_tokenizer\":\n padding_symbol = tokenizer.get_pad_token()\n pad_token_id = tokenizer.convert_token_to_id(padding_symbol)\n else:\n pad_token_id = inverse_vocabulary[padding_symbol]\n\n def pad(vector):\n sequence = np.full((int(max_length),), pad_token_id, dtype=format_dtype)\n limit = min(vector.shape[0], max_length)\n if padding == \"right\":\n sequence[:limit] = vector[:limit]\n else: # if padding == 'left\n sequence[max_length - limit :] = vector[:limit]\n return sequence\n\n padded = processor.map_objects(unit_vectors, pad)\n return padded\n\n\ndef get_tokenizer(tokenizer_type: str, tokenizer_vocab_file: str, pretrained_model_name_or_path: str):\n \"\"\"Returns a tokenizer object based on the tokenizer type.\"\"\"\n return get_tokenizer_from_registry(tokenizer_type)(\n vocab_file=tokenizer_vocab_file,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n )\n"
},
{
"path": "ludwig/utils/structural_warning.py",
"content": "import warnings\n\nfrom ludwig.utils.logging_utils import log_once\n\n\ndef warn_structure_refactor(old_module: str, new_module: str, direct: bool = True) -> None:\n \"\"\"Create structure refactor warning to indicate modules new location post.\n\n Only creates a warning once per module.\n \"\"\"\n old_module = old_module.replace(\".py\", \"\")\n if log_once(old_module):\n warning = (\n f\"The module `{old_module}` has been moved to `{new_module}` and the old \"\n f\"location will be deprecated soon. Please adjust your imports to point \"\n f\"to the new location.\"\n )\n\n if direct:\n warning += f\" Example: Do a global search and \" f\"replace `{old_module}` with `{new_module}`.\"\n else:\n warning += (\n f\"\\nATTENTION: This module may have been split or refactored. Please \"\n f\"check the contents of `{new_module}` before making changes.\"\n )\n\n with warnings.catch_warnings():\n warnings.simplefilter(\"always\")\n warnings.warn(warning, DeprecationWarning, stacklevel=3)\n"
},
{
"path": "ludwig/utils/system_utils.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2022 Predibase, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nfrom dataclasses import dataclass\n\nfrom ludwig.api_annotations import DeveloperAPI\n\n\n@DeveloperAPI\n@dataclass\nclass Resources:\n cpus: int\n gpus: int\n"
},
{
"path": "ludwig/utils/time_utils.py",
"content": "#! /usr/bin/env python\n# Copyright (c) 2023 Predibase, Inc., 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\nimport logging\nimport time\nfrom datetime import datetime, timedelta\n\nlogger = logging.getLogger(__name__)\n\n\nclass WithTimer:\n def __init__(self, title=\"\", quiet=False):\n self.title = title\n self.quiet = quiet\n\n def elapsed(self):\n return time.time() - self.wall, time.process_time() - self.proc\n\n def enter(self):\n \"\"\"Manually trigger enter.\"\"\"\n self.__enter__()\n\n def __enter__(self):\n self.proc = time.process_time()\n self.wall = time.time()\n return self\n\n def __exit__(self, *args):\n if not self.quiet:\n elapsed_wp = self.elapsed()\n logger.info(f\"Elapsed {self.title}: wall {elapsed_wp[0]:.06f}, sys {elapsed_wp[1]:.06f}\")\n\n\nclass Timer:\n def __init__(self):\n self.reset()\n\n def reset(self):\n self._proc = time.process_time()\n self._wall = time.time()\n\n def elapsed(self):\n return self.wall(), self.proc()\n\n def elapsed_str(self):\n return strdelta(self.wall() * 1000.0), strdelta(self.proc() * 1000.0)\n\n def wall(self):\n return time.time() - self._wall\n\n def proc(self):\n return time.process_time() - self._proc\n\n def tic(self):\n \"\"\"Like Matlab tic/toc for wall time and processor time.\"\"\"\n self.reset()\n\n def toc(self):\n \"\"\"Like Matlab tic/toc for wall time.\"\"\"\n return self.wall()\n\n def tocproc(self):\n \"\"\"Like Matlab tic/toc, but for processor time.\"\"\"\n return self.proc()\n\n\ndef timestamp():\n return f\"{datetime.now():%Y_%m_%d_%H_%M_%S}\"\n\n\ndef strdelta(tdelta):\n if isinstance(tdelta, (int, float)):\n tdelta = timedelta(milliseconds=tdelta)\n d = {\"D\": tdelta.days}\n d[\"H\"], rem = divmod(tdelta.seconds, 3600)\n d[\"M\"], d[\"S\"] = divmod(rem, 60)\n d[\"f\"] = str(tdelta.microseconds)[0:4]\n if d[\"D\"] > 0:\n t = \"{D}d {H}h {M}m {S}.{f}s\"\n elif d[\"H\"] > 0:\n t = \"{H}h {M}m {S}.{f}s\"\n elif d[\"M\"] > 0:\n t = \"{M}m {S}.{f}s\"\n else:\n t = \"{S}.{f}s\"\n return t.format(**d)\n"
},
{
"path": "ludwig/utils/tokenizers.py",
"content": "\"\"\"Ludwig string tokenizers including string-based, spacy-based, and huggingface-based implementations.\n\nTo add a new tokenizer, 1) implement a subclass of BaseTokenizer and 2) add it to the tokenizer_registry.\n\nOnce it's in the registry, tokenizers can be used in a ludwig config, e.g..\n\n```\ninput_features:\n - name: title\n type: text\n preprocessing:\n tokenizer: \n```\n\"\"\"\n\nimport logging\nimport re\nfrom abc import abstractmethod\nfrom typing import Any\n\nimport torch\n\nfrom ludwig.utils.nlp_utils import load_nlp_pipeline, process_text\n\nlogger = logging.getLogger(__name__)\n\n\nSPACE_PUNCTUATION_REGEX = re.compile(r\"\\w+|[^\\w\\s]\")\nCOMMA_REGEX = re.compile(r\"\\s*,\\s*\")\nUNDERSCORE_REGEX = re.compile(r\"\\s*_\\s*\")\n\nTORCHSCRIPT_COMPATIBLE_TOKENIZERS = {\"space\", \"space_punct\"}\n\n\nclass BaseTokenizer:\n @abstractmethod\n def __init__(self, **kwargs):\n pass\n\n @abstractmethod\n def __call__(self, text: str):\n pass\n\n\nclass CharactersToListTokenizer(BaseTokenizer):\n def __call__(self, text):\n return [char for char in text]\n\n\nclass SpaceStringToListTokenizer(torch.nn.Module):\n \"\"\"Implements torchscript-compatible whitespace tokenization.\"\"\"\n\n def __init__(self, **kwargs):\n super().__init__()\n\n def forward(self, v: str | list[str] | torch.Tensor) -> Any:\n if isinstance(v, torch.Tensor):\n raise ValueError(f\"Unsupported input: {v}\")\n\n inputs: list[str] = []\n # Ludwig calls map on List[str] objects, so we need to handle individual strings as well.\n if isinstance(v, str):\n inputs.append(v)\n else:\n inputs.extend(v)\n\n tokens: list[list[str]] = []\n for sequence in inputs:\n split_sequence = sequence.strip().split(\" \")\n token_sequence: list[str] = []\n for token in split_sequence:\n if len(token) > 0:\n token_sequence.append(token)\n tokens.append(token_sequence)\n\n return tokens[0] if isinstance(v, str) else tokens\n\n\nclass SpacePunctuationStringToListTokenizer(torch.nn.Module):\n \"\"\"Implements torchscript-compatible space_punct tokenization.\"\"\"\n\n def __init__(self, **kwargs):\n super().__init__()\n\n def is_regex_w(self, c: str) -> bool:\n return c.isalnum() or c == \"_\"\n\n def forward(self, v: str | list[str] | torch.Tensor) -> Any:\n if isinstance(v, torch.Tensor):\n raise ValueError(f\"Unsupported input: {v}\")\n\n inputs: list[str] = []\n # Ludwig calls map on List[str] objects, so we need to handle individual strings as well.\n if isinstance(v, str):\n inputs.append(v)\n else:\n inputs.extend(v)\n\n tokens: list[list[str]] = []\n for sequence in inputs:\n token_sequence: list[str] = []\n word: list[str] = []\n for c in sequence:\n if self.is_regex_w(c):\n word.append(c)\n elif len(word) > 0: # if non-empty word and non-alphanumeric char, append word to token sequence\n token_sequence.append(\"\".join(word))\n word.clear()\n\n if not self.is_regex_w(c) and not c.isspace(): # non-alphanumeric, non-space char is punctuation\n token_sequence.append(c)\n\n if len(word) > 0: # add last word\n token_sequence.append(\"\".join(word))\n\n tokens.append(token_sequence)\n\n return tokens[0] if isinstance(v, str) else tokens\n\n\nclass StringSplitTokenizer(BaseTokenizer):\n \"\"\"Splits a string by a given separator.\"\"\"\n\n def __init__(self, separator: str = \" \", **kwargs):\n self.separator = separator\n\n def __call__(self, text):\n return text.split(self.separator)\n\n\nclass NgramTokenizer(BaseTokenizer):\n \"\"\"Tokenizes text into unigrams + ngrams up to n.\"\"\"\n\n def __init__(self, n: int = 2, **kwargs):\n self.n = n\n\n def __call__(self, text):\n tokens = text.strip().split()\n result = list(tokens)\n for i in range(2, self.n + 1):\n for j in range(len(tokens) - i + 1):\n result.append(\" \".join(tokens[j : j + i]))\n return result\n\n\nclass UnderscoreStringToListTokenizer(BaseTokenizer):\n def __call__(self, text):\n return UNDERSCORE_REGEX.split(text.strip())\n\n\nclass CommaStringToListTokenizer(BaseTokenizer):\n def __call__(self, text):\n return COMMA_REGEX.split(text.strip())\n\n\nclass UntokenizedStringToListTokenizer(BaseTokenizer):\n def __call__(self, text):\n return [text]\n\n\nclass StrippedStringToListTokenizer(BaseTokenizer):\n def __call__(self, text):\n return [text.strip()]\n\n\nclass EnglishTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"en\"))\n\n\nclass EnglishFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text, load_nlp_pipeline(\"en\"), filter_numbers=True, filter_punctuation=True, filter_short_tokens=True\n )\n\n\nclass EnglishRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"en\"), filter_stopwords=True)\n\n\nclass EnglishLemmatizeTokenizer(BaseTokenizer):\n def __call__(self, text):\n process_text(text, load_nlp_pipeline(\"en\"), return_lemma=True)\n\n\nclass EnglishLemmatizeFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text,\n load_nlp_pipeline(\"en\"),\n return_lemma=True,\n filter_numbers=True,\n filter_punctuation=True,\n filter_short_tokens=True,\n )\n\n\nclass EnglishLemmatizeRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"en\"), return_lemma=True, filter_stopwords=True)\n\n\nclass ItalianTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"it\"))\n\n\nclass ItalianFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text, load_nlp_pipeline(\"it\"), filter_numbers=True, filter_punctuation=True, filter_short_tokens=True\n )\n\n\nclass ItalianRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"it\"), filter_stopwords=True)\n\n\nclass ItalianLemmatizeTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"it\"), return_lemma=True)\n\n\nclass ItalianLemmatizeFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text,\n load_nlp_pipeline(\"it\"),\n return_lemma=True,\n filter_numbers=True,\n filter_punctuation=True,\n filter_short_tokens=True,\n )\n\n\nclass ItalianLemmatizeRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"it\"), return_lemma=True, filter_stopwords=True)\n\n\nclass SpanishTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"es\"))\n\n\nclass SpanishFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text, load_nlp_pipeline(\"es\"), filter_numbers=True, filter_punctuation=True, filter_short_tokens=True\n )\n\n\nclass SpanishRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"es\"), filter_stopwords=True)\n\n\nclass SpanishLemmatizeTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"es\"), return_lemma=True)\n\n\nclass SpanishLemmatizeFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text,\n load_nlp_pipeline(\"es\"),\n return_lemma=True,\n filter_numbers=True,\n filter_punctuation=True,\n filter_short_tokens=True,\n )\n\n\nclass SpanishLemmatizeRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"es\"), return_lemma=True, filter_stopwords=True)\n\n\nclass GermanTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"de\"))\n\n\nclass GermanFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text, load_nlp_pipeline(\"de\"), filter_numbers=True, filter_punctuation=True, filter_short_tokens=True\n )\n\n\nclass GermanRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"de\"), filter_stopwords=True)\n\n\nclass GermanLemmatizeTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"de\"), return_lemma=True)\n\n\nclass GermanLemmatizeFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text,\n load_nlp_pipeline(\"de\"),\n return_lemma=True,\n filter_numbers=True,\n filter_punctuation=True,\n filter_short_tokens=True,\n )\n\n\nclass GermanLemmatizeRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"de\"), return_lemma=True, filter_stopwords=True)\n\n\nclass FrenchTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"fr\"))\n\n\nclass FrenchFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text, load_nlp_pipeline(\"fr\"), filter_numbers=True, filter_punctuation=True, filter_short_tokens=True\n )\n\n\nclass FrenchRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"fr\"), filter_stopwords=True)\n\n\nclass FrenchLemmatizeTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"fr\"), return_lemma=True)\n\n\nclass FrenchLemmatizeFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text,\n load_nlp_pipeline(\"fr\"),\n return_lemma=True,\n filter_numbers=True,\n filter_punctuation=True,\n filter_short_tokens=True,\n )\n\n\nclass FrenchLemmatizeRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"fr\"), return_lemma=True, filter_stopwords=True)\n\n\nclass PortugueseTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"pt\"))\n\n\nclass PortugueseFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text, load_nlp_pipeline(\"pt\"), filter_numbers=True, filter_punctuation=True, filter_short_tokens=True\n )\n\n\nclass PortugueseRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"pt\"), filter_stopwords=True)\n\n\nclass PortugueseLemmatizeTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"pt\"), return_lemma=True)\n\n\nclass PortugueseLemmatizeFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text,\n load_nlp_pipeline(\"pt\"),\n return_lemma=True,\n filter_numbers=True,\n filter_punctuation=True,\n filter_short_tokens=True,\n )\n\n\nclass PortugueseLemmatizeRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"pt\"), return_lemma=True, filter_stopwords=True)\n\n\nclass DutchTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"nl\"))\n\n\nclass DutchFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text, load_nlp_pipeline(\"nl\"), filter_numbers=True, filter_punctuation=True, filter_short_tokens=True\n )\n\n\nclass DutchRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"nl\"), filter_stopwords=True)\n\n\nclass DutchLemmatizeTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"nl\"), return_lemma=True)\n\n\nclass DutchLemmatizeFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text,\n load_nlp_pipeline(\"nl\"),\n return_lemma=True,\n filter_numbers=True,\n filter_punctuation=True,\n filter_short_tokens=True,\n )\n\n\nclass DutchLemmatizeRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"nl\"), return_lemma=True, filter_stopwords=True)\n\n\nclass GreekTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"el\"))\n\n\nclass GreekFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text, load_nlp_pipeline(\"el\"), filter_numbers=True, filter_punctuation=True, filter_short_tokens=True\n )\n\n\nclass GreekRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"el\"), filter_stopwords=True)\n\n\nclass GreekLemmatizeTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"el\"), return_lemma=True)\n\n\nclass GreekLemmatizeFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text,\n load_nlp_pipeline(\"el\"),\n return_lemma=True,\n filter_numbers=True,\n filter_punctuation=True,\n filter_short_tokens=True,\n )\n\n\nclass GreekLemmatizeRemoveStopwordsFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"el\"), return_lemma=True, filter_stopwords=True)\n\n\nclass NorwegianTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"nb\"))\n\n\nclass NorwegianFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text, load_nlp_pipeline(\"nb\"), filter_numbers=True, filter_punctuation=True, filter_short_tokens=True\n )\n\n\nclass NorwegianRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"nb\"), filter_stopwords=True)\n\n\nclass NorwegianLemmatizeTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"nb\"), return_lemma=True)\n\n\nclass NorwegianLemmatizeFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text,\n load_nlp_pipeline(\"nb\"),\n return_lemma=True,\n filter_numbers=True,\n filter_punctuation=True,\n filter_short_tokens=True,\n )\n\n\nclass NorwegianLemmatizeRemoveStopwordsFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"nb\"), return_lemma=True, filter_stopwords=True)\n\n\nclass LithuanianTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"lt\"))\n\n\nclass LithuanianFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text, load_nlp_pipeline(\"lt\"), filter_numbers=True, filter_punctuation=True, filter_short_tokens=True\n )\n\n\nclass LithuanianRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"lt\"), filter_stopwords=True)\n\n\nclass LithuanianLemmatizeTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"lt\"), return_lemma=True)\n\n\nclass LithuanianLemmatizeFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text,\n load_nlp_pipeline(\"lt\"),\n return_lemma=True,\n filter_numbers=True,\n filter_punctuation=True,\n filter_short_tokens=True,\n )\n\n\nclass LithuanianLemmatizeRemoveStopwordsFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"lt\"), return_lemma=True, filter_stopwords=True)\n\n\nclass DanishTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"da\"))\n\n\nclass DanishFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text, load_nlp_pipeline(\"da\"), filter_numbers=True, filter_punctuation=True, filter_short_tokens=True\n )\n\n\nclass DanishRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"da\"), filter_stopwords=True)\n\n\nclass DanishLemmatizeTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"da\"), return_lemma=True)\n\n\nclass DanishLemmatizeFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text,\n load_nlp_pipeline(\"da\"),\n return_lemma=True,\n filter_numbers=True,\n filter_punctuation=True,\n filter_short_tokens=True,\n )\n\n\nclass DanishLemmatizeRemoveStopwordsFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"da\"), return_lemma=True, filter_stopwords=True)\n\n\nclass PolishTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"pl\"))\n\n\nclass PolishFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text, load_nlp_pipeline(\"pl\"), filter_numbers=True, filter_punctuation=True, filter_short_tokens=True\n )\n\n\nclass PolishRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"pl\"), filter_stopwords=True)\n\n\nclass PolishLemmatizeTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"pl\"), return_lemma=True)\n\n\nclass PolishLemmatizeFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text,\n load_nlp_pipeline(\"pl\"),\n return_lemma=True,\n filter_numbers=True,\n filter_punctuation=True,\n filter_short_tokens=True,\n )\n\n\nclass PolishLemmatizeRemoveStopwordsFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"pl\"), return_lemma=True, filter_stopwords=True)\n\n\nclass RomanianTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"ro\"))\n\n\nclass RomanianFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text, load_nlp_pipeline(\"ro\"), filter_numbers=True, filter_punctuation=True, filter_short_tokens=True\n )\n\n\nclass RomanianRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"ro\"), filter_stopwords=True)\n\n\nclass RomanianLemmatizeTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"ro\"), return_lemma=True)\n\n\nclass RomanianLemmatizeFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text,\n load_nlp_pipeline(\"ro\"),\n return_lemma=True,\n filter_numbers=True,\n filter_punctuation=True,\n filter_short_tokens=True,\n )\n\n\nclass RomanianLemmatizeRemoveStopwordsFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"ro\"), return_lemma=True, filter_stopwords=True)\n\n\nclass JapaneseTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"jp\"))\n\n\nclass JapaneseFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text, load_nlp_pipeline(\"jp\"), filter_numbers=True, filter_punctuation=True, filter_short_tokens=True\n )\n\n\nclass JapaneseRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"jp\"), filter_stopwords=True)\n\n\nclass JapaneseLemmatizeTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"jp\"), return_lemma=True)\n\n\nclass JapaneseLemmatizeFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text,\n load_nlp_pipeline(\"jp\"),\n return_lemma=True,\n filter_numbers=True,\n filter_punctuation=True,\n filter_short_tokens=True,\n )\n\n\nclass JapaneseLemmatizeRemoveStopwordsFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"jp\"), return_lemma=True, filter_stopwords=True)\n\n\nclass ChineseTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"zh\"))\n\n\nclass ChineseFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text, load_nlp_pipeline(\"zh\"), filter_numbers=True, filter_punctuation=True, filter_short_tokens=True\n )\n\n\nclass ChineseRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"zh\"), filter_stopwords=True)\n\n\nclass ChineseLemmatizeTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"zh\"), return_lemma=True)\n\n\nclass ChineseLemmatizeFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text,\n load_nlp_pipeline(\"zh\"),\n return_lemma=True,\n filter_numbers=True,\n filter_punctuation=True,\n filter_short_tokens=True,\n )\n\n\nclass ChineseLemmatizeRemoveStopwordsFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"zh\"), return_lemma=True, filter_stopwords=True)\n\n\nclass MultiTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text, load_nlp_pipeline(\"xx\"), filter_numbers=True, filter_punctuation=True, filter_short_tokens=True\n )\n\n\nclass MultiFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text, load_nlp_pipeline(\"xx\"), filter_numbers=True, filter_punctuation=True, filter_short_tokens=True\n )\n\n\nclass MultiRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"xx\"), filter_stopwords=True)\n\n\nclass MultiLemmatizeTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"xx\"), return_lemma=True)\n\n\nclass MultiLemmatizeFilterTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(\n text,\n load_nlp_pipeline(\"xx\"),\n return_lemma=True,\n filter_numbers=True,\n filter_punctuation=True,\n filter_short_tokens=True,\n )\n\n\nclass MultiLemmatizeRemoveStopwordsTokenizer(BaseTokenizer):\n def __call__(self, text):\n return process_text(text, load_nlp_pipeline(\"xx\"), return_lemma=True, filter_stopwords=True)\n\n\nclass HFTokenizer(BaseTokenizer):\n def __init__(self, pretrained_model_name_or_path, **kwargs):\n super().__init__()\n from transformers import AutoTokenizer\n\n self.tokenizer = AutoTokenizer.from_pretrained(\n pretrained_model_name_or_path,\n trust_remote_code=kwargs.get(\"trust_remote_code\", False),\n )\n # Some models (e.g. LLaMA) don't have a pad_token by default.\n # Set it to eos_token to avoid NoneType errors in preprocessing.\n if self.tokenizer.pad_token is None and self.tokenizer.eos_token is not None:\n self.tokenizer.pad_token = self.tokenizer.eos_token\n\n def __call__(self, text):\n return self.tokenizer.encode(text, truncation=True)\n\n def get_vocab(self):\n return self.tokenizer.get_vocab()\n\n def get_pad_token(self) -> str:\n return self.tokenizer.pad_token\n\n def get_unk_token(self) -> str:\n return self.tokenizer.unk_token\n\n def convert_token_to_id(self, token: str) -> int:\n if token is None:\n return 0\n return self.tokenizer.convert_tokens_to_ids(token)\n\n\ntokenizer_registry = {\n # Torchscript-compatible tokenizers.\n \"space\": SpaceStringToListTokenizer,\n \"space_punct\": SpacePunctuationStringToListTokenizer,\n # Tokenizers not compatible with torchscript\n \"characters\": CharactersToListTokenizer,\n \"underscore\": UnderscoreStringToListTokenizer,\n \"comma\": CommaStringToListTokenizer,\n \"untokenized\": UntokenizedStringToListTokenizer,\n \"stripped\": StrippedStringToListTokenizer,\n \"english_tokenize\": EnglishTokenizer,\n \"english_tokenize_filter\": EnglishFilterTokenizer,\n \"english_tokenize_remove_stopwords\": EnglishRemoveStopwordsTokenizer,\n \"english_lemmatize\": EnglishLemmatizeTokenizer,\n \"english_lemmatize_filter\": EnglishLemmatizeFilterTokenizer,\n \"english_lemmatize_remove_stopwords\": EnglishLemmatizeRemoveStopwordsTokenizer,\n \"italian_tokenize\": ItalianTokenizer,\n \"italian_tokenize_filter\": ItalianFilterTokenizer,\n \"italian_tokenize_remove_stopwords\": ItalianRemoveStopwordsTokenizer,\n \"italian_lemmatize\": ItalianLemmatizeTokenizer,\n \"italian_lemmatize_filter\": ItalianLemmatizeFilterTokenizer,\n \"italian_lemmatize_remove_stopwords\": ItalianLemmatizeRemoveStopwordsTokenizer,\n \"spanish_tokenize\": SpanishTokenizer,\n \"spanish_tokenize_filter\": SpanishFilterTokenizer,\n \"spanish_tokenize_remove_stopwords\": SpanishRemoveStopwordsTokenizer,\n \"spanish_lemmatize\": SpanishLemmatizeTokenizer,\n \"spanish_lemmatize_filter\": SpanishLemmatizeFilterTokenizer,\n \"spanish_lemmatize_remove_stopwords\": SpanishLemmatizeRemoveStopwordsTokenizer,\n \"german_tokenize\": GermanTokenizer,\n \"german_tokenize_filter\": GermanFilterTokenizer,\n \"german_tokenize_remove_stopwords\": GermanRemoveStopwordsTokenizer,\n \"german_lemmatize\": GermanLemmatizeTokenizer,\n \"german_lemmatize_filter\": GermanLemmatizeFilterTokenizer,\n \"german_lemmatize_remove_stopwords\": GermanLemmatizeRemoveStopwordsTokenizer,\n \"french_tokenize\": FrenchTokenizer,\n \"french_tokenize_filter\": FrenchFilterTokenizer,\n \"french_tokenize_remove_stopwords\": FrenchRemoveStopwordsTokenizer,\n \"french_lemmatize\": FrenchLemmatizeTokenizer,\n \"french_lemmatize_filter\": FrenchLemmatizeFilterTokenizer,\n \"french_lemmatize_remove_stopwords\": FrenchLemmatizeRemoveStopwordsTokenizer,\n \"portuguese_tokenize\": PortugueseTokenizer,\n \"portuguese_tokenize_filter\": PortugueseFilterTokenizer,\n \"portuguese_tokenize_remove_stopwords\": PortugueseRemoveStopwordsTokenizer,\n \"portuguese_lemmatize\": PortugueseLemmatizeTokenizer,\n \"portuguese_lemmatize_filter\": PortugueseLemmatizeFilterTokenizer,\n \"portuguese_lemmatize_remove_stopwords\": PortugueseLemmatizeRemoveStopwordsTokenizer,\n \"dutch_tokenize\": DutchTokenizer,\n \"dutch_tokenize_filter\": DutchFilterTokenizer,\n \"dutch_tokenize_remove_stopwords\": DutchRemoveStopwordsTokenizer,\n \"dutch_lemmatize\": DutchLemmatizeTokenizer,\n \"dutch_lemmatize_filter\": DutchLemmatizeFilterTokenizer,\n \"dutch_lemmatize_remove_stopwords\": DutchLemmatizeRemoveStopwordsTokenizer,\n \"greek_tokenize\": GreekTokenizer,\n \"greek_tokenize_filter\": GreekFilterTokenizer,\n \"greek_tokenize_remove_stopwords\": GreekRemoveStopwordsTokenizer,\n \"greek_lemmatize\": GreekLemmatizeTokenizer,\n \"greek_lemmatize_filter\": GreekLemmatizeFilterTokenizer,\n \"greek_lemmatize_remove_stopwords\": GreekLemmatizeRemoveStopwordsFilterTokenizer,\n \"norwegian_tokenize\": NorwegianTokenizer,\n \"norwegian_tokenize_filter\": NorwegianFilterTokenizer,\n \"norwegian_tokenize_remove_stopwords\": NorwegianRemoveStopwordsTokenizer,\n \"norwegian_lemmatize\": NorwegianLemmatizeTokenizer,\n \"norwegian_lemmatize_filter\": NorwegianLemmatizeFilterTokenizer,\n \"norwegian_lemmatize_remove_stopwords\": NorwegianLemmatizeRemoveStopwordsFilterTokenizer,\n \"lithuanian_tokenize\": LithuanianTokenizer,\n \"lithuanian_tokenize_filter\": LithuanianFilterTokenizer,\n \"lithuanian_tokenize_remove_stopwords\": LithuanianRemoveStopwordsTokenizer,\n \"lithuanian_lemmatize\": LithuanianLemmatizeTokenizer,\n \"lithuanian_lemmatize_filter\": LithuanianLemmatizeFilterTokenizer,\n \"lithuanian_lemmatize_remove_stopwords\": LithuanianLemmatizeRemoveStopwordsFilterTokenizer,\n \"danish_tokenize\": DanishTokenizer,\n \"danish_tokenize_filter\": DanishFilterTokenizer,\n \"danish_tokenize_remove_stopwords\": DanishRemoveStopwordsTokenizer,\n \"danish_lemmatize\": DanishLemmatizeTokenizer,\n \"danish_lemmatize_filter\": DanishLemmatizeFilterTokenizer,\n \"danish_lemmatize_remove_stopwords\": DanishLemmatizeRemoveStopwordsFilterTokenizer,\n \"polish_tokenize\": PolishTokenizer,\n \"polish_tokenize_filter\": PolishFilterTokenizer,\n \"polish_tokenize_remove_stopwords\": PolishRemoveStopwordsTokenizer,\n \"polish_lemmatize\": PolishLemmatizeTokenizer,\n \"polish_lemmatize_filter\": PolishLemmatizeFilterTokenizer,\n \"polish_lemmatize_remove_stopwords\": PolishLemmatizeRemoveStopwordsFilterTokenizer,\n \"romanian_tokenize\": RomanianTokenizer,\n \"romanian_tokenize_filter\": RomanianFilterTokenizer,\n \"romanian_tokenize_remove_stopwords\": RomanianRemoveStopwordsTokenizer,\n \"romanian_lemmatize\": RomanianLemmatizeTokenizer,\n \"romanian_lemmatize_filter\": RomanianLemmatizeFilterTokenizer,\n \"romanian_lemmatize_remove_stopwords\": RomanianLemmatizeRemoveStopwordsFilterTokenizer,\n \"japanese_tokenize\": JapaneseTokenizer,\n \"japanese_tokenize_filter\": JapaneseFilterTokenizer,\n \"japanese_tokenize_remove_stopwords\": JapaneseRemoveStopwordsTokenizer,\n \"japanese_lemmatize\": JapaneseLemmatizeTokenizer,\n \"japanese_lemmatize_filter\": JapaneseLemmatizeFilterTokenizer,\n \"japanese_lemmatize_remove_stopwords\": JapaneseLemmatizeRemoveStopwordsFilterTokenizer,\n \"chinese_tokenize\": ChineseTokenizer,\n \"chinese_tokenize_filter\": ChineseFilterTokenizer,\n \"chinese_tokenize_remove_stopwords\": ChineseRemoveStopwordsTokenizer,\n \"chinese_lemmatize\": ChineseLemmatizeTokenizer,\n \"chinese_lemmatize_filter\": ChineseLemmatizeFilterTokenizer,\n \"chinese_lemmatize_remove_stopwords\": ChineseLemmatizeRemoveStopwordsFilterTokenizer,\n \"multi_tokenize\": MultiTokenizer,\n \"multi_tokenize_filter\": MultiFilterTokenizer,\n \"multi_tokenize_remove_stopwords\": MultiRemoveStopwordsTokenizer,\n \"multi_lemmatize\": MultiLemmatizeTokenizer,\n \"multi_lemmatize_filter\": MultiLemmatizeFilterTokenizer,\n \"multi_lemmatize_remove_stopwords\": MultiLemmatizeRemoveStopwordsTokenizer,\n}\n\n\nclass SentencePieceTokenizer(torch.nn.Module):\n \"\"\"SentencePiece tokenizer using HuggingFace transformers (XLMR-based).\"\"\"\n\n def __init__(self, pretrained_model_name_or_path: str | None = None, **kwargs):\n super().__init__()\n from transformers import AutoTokenizer\n\n if pretrained_model_name_or_path is None:\n pretrained_model_name_or_path = \"xlm-roberta-base\"\n self.tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path)\n\n def forward(self, v: str | list[str] | torch.Tensor):\n if isinstance(v, torch.Tensor):\n raise ValueError(f\"Unsupported input: {v}\")\n if isinstance(v, str):\n return self.tokenizer.tokenize(v)\n return [self.tokenizer.tokenize(s) for s in v]\n\n\nclass CLIPTokenizer(torch.nn.Module):\n \"\"\"CLIP tokenizer using HuggingFace transformers.\"\"\"\n\n def __init__(self, pretrained_model_name_or_path: str | None = None, **kwargs):\n super().__init__()\n from transformers import CLIPTokenizer as HFCLIPTokenizer\n\n if pretrained_model_name_or_path is None:\n pretrained_model_name_or_path = \"openai/clip-vit-base-patch32\"\n self.tokenizer = HFCLIPTokenizer.from_pretrained(pretrained_model_name_or_path)\n\n def __call__(self, text):\n if isinstance(text, str):\n return self.tokenizer.tokenize(text)\n return [self.tokenizer.tokenize(t) for t in text]\n\n def get_vocab(self):\n return self.tokenizer.get_vocab()\n\n\nclass GPT2BPETokenizer(torch.nn.Module):\n \"\"\"GPT-2 BPE tokenizer using HuggingFace transformers.\"\"\"\n\n def __init__(self, pretrained_model_name_or_path: str | None = None, **kwargs):\n super().__init__()\n from transformers import GPT2Tokenizer\n\n if pretrained_model_name_or_path is None:\n pretrained_model_name_or_path = \"gpt2\"\n self.tokenizer = GPT2Tokenizer.from_pretrained(pretrained_model_name_or_path)\n\n def __call__(self, text):\n if isinstance(text, str):\n return self.tokenizer.tokenize(text)\n return [self.tokenizer.tokenize(t) for t in text]\n\n def get_vocab(self):\n return self.tokenizer.get_vocab()\n\n\nclass BERTTokenizer(torch.nn.Module):\n \"\"\"BERT tokenizer using HuggingFace transformers.\"\"\"\n\n def __init__(\n self,\n vocab_file: str | None = None,\n pretrained_model_name_or_path: str | None = None,\n is_hf_tokenizer: bool | None = False,\n do_lower_case: bool | None = None,\n **kwargs,\n ):\n super().__init__()\n from transformers import BertTokenizer\n\n if pretrained_model_name_or_path is None:\n pretrained_model_name_or_path = \"bert-base-uncased\"\n tokenizer_kwargs = {}\n if do_lower_case is not None:\n tokenizer_kwargs[\"do_lower_case\"] = do_lower_case\n self.tokenizer = BertTokenizer.from_pretrained(pretrained_model_name_or_path, **tokenizer_kwargs)\n self.is_hf_tokenizer = is_hf_tokenizer\n self.pad_token = self.tokenizer.pad_token\n self.unk_token = self.tokenizer.unk_token\n self.cls_token_id = self.tokenizer.cls_token_id\n self.sep_token_id = self.tokenizer.sep_token_id\n\n def __call__(self, text):\n if isinstance(text, str):\n texts = [text]\n else:\n texts = text\n\n if self.is_hf_tokenizer:\n results = [self.tokenizer.encode(t) for t in texts]\n else:\n results = [self.tokenizer.tokenize(t) for t in texts]\n\n return results[0] if isinstance(text, str) else results\n\n def get_vocab(self):\n return self.tokenizer.get_vocab()\n\n def get_pad_token(self) -> str:\n return self.pad_token\n\n def get_unk_token(self) -> str:\n return self.unk_token\n\n def convert_token_to_id(self, token: str) -> int:\n if token is None:\n return 0\n return self.tokenizer.convert_tokens_to_ids(token)\n\n\ntokenizer_registry.update(\n {\n \"sentencepiece\": SentencePieceTokenizer,\n \"clip\": CLIPTokenizer,\n \"gpt2bpe\": GPT2BPETokenizer,\n \"bert\": BERTTokenizer,\n }\n)\n\n\ndef get_hf_tokenizer(pretrained_model_name_or_path, **kwargs):\n \"\"\"Gets a HuggingFace-based tokenizer that follows HF convention.\n\n Args:\n pretrained_model_name_or_path: Name of the model in the HF repo. Example: \"bert-base-uncased\".\n Returns:\n A HF tokenizer.\n \"\"\"\n model_name_lower = pretrained_model_name_or_path.lower()\n # Use BERTTokenizer only for actual BERT models, not for models like albert/roberta\n # that have \"bert\" in their name but use different tokenization (SentencePiece, BPE, etc.)\n if \"bert\" in model_name_lower and not any(x in model_name_lower for x in (\"albert\", \"roberta\", \"distilbert\")):\n logger.info(f\"Loading BERT tokenizer for {pretrained_model_name_or_path}\")\n return BERTTokenizer(pretrained_model_name_or_path=pretrained_model_name_or_path, is_hf_tokenizer=True)\n\n logger.info(f\"Loading HuggingFace tokenizer for {pretrained_model_name_or_path}\")\n return HFTokenizer(pretrained_model_name_or_path)\n\n\ntokenizer_registry.update(\n {\n \"hf_tokenizer\": get_hf_tokenizer,\n }\n)\n\n\ndef get_tokenizer_from_registry(tokenizer_name: str) -> torch.nn.Module:\n \"\"\"Returns the appropriate tokenizer from the tokenizer registry.\"\"\"\n if tokenizer_name in tokenizer_registry:\n return tokenizer_registry[tokenizer_name]\n raise KeyError(f\"Invalid tokenizer name: '{tokenizer_name}'. Available tokenizers: {tokenizer_registry.keys()}\")\n"
},
{
"path": "ludwig/utils/torch_utils.py",
"content": "import math\nimport os\nimport warnings\nfrom abc import abstractmethod\nfrom functools import lru_cache\n\nimport torch\nfrom torch import nn\nfrom torch.nn import Module, ModuleDict\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import ENCODER_OUTPUT\nfrom ludwig.utils.strings_utils import SpecialSymbol\n\n_TORCH_INIT_PARAMS: tuple | None = None\n\n\n@DeveloperAPI\ndef get_torch_device():\n if torch.cuda.is_available() and torch.cuda.device_count() > 0:\n # Use cublasLt for batched GEMM operations. The default cublas library has known\n # bugs with cublasSgemmStridedBatched on certain GPU/driver combinations.\n torch.backends.cuda.preferred_blas_library(\"cublaslt\")\n return \"cuda\"\n\n if bool(os.environ.get(\"LUDWIG_ENABLE_MPS\")):\n if torch.backends.mps.is_available() and torch.backends.mps.is_built():\n if not bool(os.environ.get(\"PYTORCH_ENABLE_MPS_FALLBACK\")):\n warnings.warn(\n \"LUDWIG_ENABLE_MPS is set and MPS is available, but PYTORCH_ENABLE_MPS_FALLBACK has not been set. \"\n \"Depending on your model config, some operations may not be compatible. If errors occur, try \"\n \"setting `PYTORCH_ENABLE_MPS_FALLBACK=1` and resubmitting.\"\n )\n return \"mps\"\n else:\n warnings.warn(\"LUDWIG_ENABLE_MPS is set but MPS is not available, falling back to CPU.\")\n\n return \"cpu\"\n\n\nDEVICE = get_torch_device()\n\n\n@DeveloperAPI\ndef place_on_device(x, device):\n \"\"\"Recursively places the input on the specified device.\"\"\"\n if isinstance(x, list):\n return [place_on_device(xi, device) for xi in x]\n elif isinstance(x, dict):\n return {k: place_on_device(v, device) for k, v in x.items()}\n elif isinstance(x, set):\n return {place_on_device(xi, device) for xi in x}\n elif isinstance(x, tuple):\n return tuple(place_on_device(xi, device) for xi in x)\n elif isinstance(x, torch.Tensor):\n return x.to(device)\n else:\n return x\n\n\n@DeveloperAPI\ndef sequence_length_2D(sequence: torch.Tensor) -> torch.Tensor:\n \"\"\"Returns the number of non-padding elements per sequence in batch.\n\n :param sequence: (torch.Tensor) A 2D tensor of shape [batch size x max sequence length]. # Return\n :returns: (torch.Tensor) The count on non-zero elements per sequence.\n \"\"\"\n used = (sequence != SpecialSymbol.PADDING.value).type(torch.int32)\n length = torch.sum(used, 1)\n return length\n\n\n@DeveloperAPI\ndef sequence_length_3D(sequence: torch.Tensor) -> torch.Tensor:\n \"\"\"Returns the number of non-zero elements per sequence in batch.\n\n :param sequence: (torch.Tensor) A 3D tensor of shape [batch size x max sequence length x hidden size]. # Return\n :returns: (torch.Tensor) The count on non-zero elements per sequence.\n \"\"\"\n used = torch.sign(torch.amax(torch.abs(sequence), dim=2))\n length = torch.sum(used, 1)\n length = length.int()\n return length\n\n\n@DeveloperAPI\ndef sequence_mask(lengths: torch.Tensor, maxlen: int | None = None, dtype: torch.dtype = torch.bool):\n \"\"\"Returns a mask of shape (batch_size x maxlen), where mask[i] is True for each element up to lengths[i],\n otherwise False i.e. if maxlen=5 and lengths[i] = 3, mask[i] = [True, True True, False False].\n\n :param lengths: (torch.Tensor) A 1d integer tensor of shape [batch size].\n :param maxlen: (Optional[int]) The maximum sequence length. If not specified, the max(lengths) is used.\n :param dtype: (type) The type to output. # Return\n :returns: (torch.Tensor) A sequence mask tensor of shape (batch_size x maxlen).\n \"\"\"\n if maxlen is None:\n maxlen = lengths.max()\n matrix = torch.unsqueeze(lengths, dim=-1)\n row_vector = torch.arange(0, maxlen, 1, device=lengths.device)\n mask = row_vector < matrix\n mask = mask.type(dtype)\n return mask\n\n\n@DeveloperAPI\ndef periodic(inputs: torch.Tensor, period: int) -> torch.Tensor:\n \"\"\"Returns periodic representation assuming 0 is start of period.\"\"\"\n return torch.cos(inputs * 2 * math.pi / period)\n\n\ninitializer_registry = {\n \"uniform\": nn.init.uniform_,\n \"normal\": nn.init.normal_,\n \"constant\": nn.init.constant_,\n \"ones\": nn.init.ones_,\n \"zeros\": nn.init.zeros_,\n \"eye\": nn.init.eye_,\n \"dirac\": nn.init.dirac_,\n \"xavier_uniform\": nn.init.xavier_uniform_,\n \"xavier_normal\": nn.init.xavier_normal_,\n \"kaiming_uniform\": nn.init.kaiming_uniform_,\n \"kaiming_normal\": nn.init.kaiming_normal_,\n \"orthogonal\": nn.init.orthogonal_,\n \"sparse\": nn.init.sparse_,\n \"identity\": nn.init.eye_,\n}\n\nactivations = {\n \"elu\": nn.ELU,\n \"leakyRelu\": nn.LeakyReLU,\n \"logSigmoid\": nn.LogSigmoid,\n \"relu\": nn.ReLU,\n \"sigmoid\": nn.Sigmoid,\n \"tanh\": nn.Tanh,\n \"softmax\": nn.Softmax,\n None: nn.Identity,\n}\n\n\n@DeveloperAPI\ndef get_activation(activation):\n return activations[activation]()\n\n\n@DeveloperAPI\ndef reg_loss(model: nn.Module, regularizer: str, l1: float = 0.01, l2: float = 0.01):\n \"\"\"Computes the regularization loss for a given model.\n\n Parameters:\n model: torch.nn.Module object to compute regularization loss for.\n regularizer: regularizer to use (currently l1, l2 and l1_l2 supported).\n l1: L1 regularization coefficient.\n l2: L2 regularization coefficient.\n\n Returns:\n Regularization loss for the model (float).\n \"\"\"\n\n if regularizer == \"l1\":\n l1_reg = l1 * sum(torch.abs(p).sum() for p in model.parameters())\n return l1_reg\n if regularizer == \"l2\":\n l2_reg = l2 * sum(torch.square(p).sum() for p in model.parameters())\n return l2_reg\n if regularizer == \"l1_l2\":\n l1_reg = l1 * sum(torch.abs(p).sum() for p in model.parameters())\n l2_reg = l2 * sum(torch.square(p).sum() for p in model.parameters())\n return l1_reg + l2_reg\n\n\n@DeveloperAPI\nclass LudwigModule(Module):\n def __init__(self):\n super().__init__()\n self._losses = {}\n self.register_buffer(\"device_tensor\", torch.zeros(0), persistent=False)\n\n @property\n def device(self):\n return self.device_tensor.device\n\n def prepare_for_training(self):\n \"\"\"This is called from within the Trainer object to do any final instantiation before model training.\"\"\"\n\n def losses(self):\n collected_losses = []\n for loss in self._losses.values():\n collected_losses.append(loss)\n\n for child in self.children():\n if isinstance(child, LudwigModule):\n collected_losses.extend(child.losses())\n elif isinstance(child, ModuleDict):\n for c in child.values():\n if hasattr(c, \"losses\"): # Some modules, i.e. SequenceReducers, don't have losses.\n collected_losses.extend(c.losses())\n elif isinstance(child, Module):\n pass\n else:\n raise ValueError\n\n return collected_losses\n\n def update_loss(self, key: str, loss: torch.Tensor):\n \"\"\"This should be called in the forward pass to add a custom loss term to the combined loss.\"\"\"\n self._losses[key] = loss\n\n @property\n def input_dtype(self):\n return torch.float32\n\n @property\n @abstractmethod\n def input_shape(self) -> torch.Size:\n \"\"\"Returns size of the input tensor without the batch dimension.\"\"\"\n # raise NotImplementedError(\"Abstract class.\")\n\n @property\n def output_shape(self) -> torch.Size:\n \"\"\"Returns size of the output tensor without the batch dimension.\"\"\"\n return self._computed_output_shape()\n\n @lru_cache(maxsize=1)\n def _computed_output_shape(self) -> torch.Size:\n dummy_input = torch.rand(2, *self.input_shape, device=self.device)\n output_tensor = self.forward(dummy_input.type(self.input_dtype))\n\n if isinstance(output_tensor, torch.Tensor):\n return output_tensor.size()[1:]\n elif isinstance(output_tensor, dict) and ENCODER_OUTPUT in output_tensor:\n return output_tensor[ENCODER_OUTPUT].size()[1:]\n else:\n raise ValueError(\"Unknown output tensor type.\")\n\n\ndef freeze_parameters(module: nn.Module):\n \"\"\"Freezes the parameters of a torch module.\"\"\"\n for p in module.parameters():\n p.requires_grad = False\n\n\n@DeveloperAPI\nclass FreezeModule(nn.Module):\n def __init__(self, module: nn.Module, frozen: bool):\n super().__init__()\n if frozen:\n freeze_parameters(module)\n module.eval()\n else:\n module.train()\n self.module = module\n self.frozen = frozen\n\n def train(self, mode: bool = True):\n if self.frozen:\n # Ignores any attempt to set params trainable\n return self\n\n return super().train(mode)\n\n\n@DeveloperAPI\nclass Dense(LudwigModule):\n def __init__(\n self,\n input_size,\n output_size,\n use_bias=True,\n weights_initializer=\"xavier_uniform\",\n bias_initializer=\"zeros\",\n ):\n super().__init__()\n self.dense = nn.Linear(in_features=input_size, out_features=output_size, bias=use_bias)\n weights_initializer = initializer_registry[weights_initializer]\n weights_initializer(self.dense.weight)\n\n if use_bias:\n bias_initializer = initializer_registry[bias_initializer]\n bias_initializer(self.dense.bias)\n\n @property\n def input_shape(self) -> torch.Size:\n return self.dense.input_shape\n\n def forward(self, input: torch.Tensor) -> torch.Tensor:\n output = torch.squeeze(self.dense(input), dim=-1)\n return output\n\n\n@DeveloperAPI\ndef initialize_pytorch(\n gpus: int | str | list[int] | None = None,\n gpu_memory_limit: float | None = None,\n allow_parallel_threads: bool = True,\n):\n param_tuple = (gpus, gpu_memory_limit, allow_parallel_threads)\n if _TORCH_INIT_PARAMS is not None:\n if _TORCH_INIT_PARAMS != param_tuple:\n warnings.warn(\n \"PyTorch has already been initialized. Changes to `gpus`, \"\n \"`gpu_memory_limit`, and `allow_parallel_threads` will be ignored. \"\n \"Start a new Python process to modify these values.\"\n )\n return\n\n # For reproducivility / determinism, set parallel threads to 1.\n # For performance, leave unset to allow PyTorch to select the best value automatically.\n if not allow_parallel_threads:\n torch.set_num_threads(1)\n torch.set_num_interop_threads(1)\n if torch.cuda.is_available() and torch.cuda.device_count() > 0:\n torch.backends.cudnn.deterministic = True\n torch.backends.cudnn.benchmark = False\n\n if isinstance(gpus, int):\n gpus = [gpus]\n elif isinstance(gpus, str):\n gpus = gpus.strip()\n gpus = [int(g) for g in gpus.split(\",\")]\n\n if gpus and len(gpus) == 1 and gpus[0] == -1:\n # CUDA_VISIBLE_DEVICES syntax for disabling all GPUs\n os.environ[\"CUDA_VISIBLE_DEVICES\"] = \"\"\n elif torch.cuda.is_available() and torch.cuda.device_count() > 0:\n # Set visible devices so GPU utilization is isolated\n # (no GPU contention between workers).\n if gpus is not None:\n if len(gpus) == 1:\n torch.cuda.set_device(gpus[0])\n elif len(gpus) > 1:\n os.environ[\"CUDA_VISIBLE_DEVICES\"] = \",\".join(str(i) for i in gpus)\n\n # Limit the amount of memory that can be consumed per GPU\n if gpu_memory_limit is not None:\n for gpu in gpus or range(torch.cuda.device_count()):\n torch.cuda.memory.set_per_process_memory_fraction(gpu_memory_limit, gpu)\n\n _set_torch_init_params(param_tuple)\n\n\ndef _set_torch_init_params(params: tuple | None):\n global _TORCH_INIT_PARAMS\n _TORCH_INIT_PARAMS = params\n\n\ndef _get_torch_init_params() -> tuple | None:\n return _TORCH_INIT_PARAMS\n\n\n@DeveloperAPI\ndef model_size(model: nn.Module):\n \"\"\"Computes PyTorch model size in bytes.\"\"\"\n size = 0\n size += sum(param.nelement() * param.element_size() for param in model.parameters())\n size += sum(buffer.nelement() * buffer.element_size() for buffer in model.buffers())\n return size\n"
},
{
"path": "ludwig/utils/trainer_utils.py",
"content": "import logging\nimport re\nfrom collections import defaultdict\nfrom typing import TYPE_CHECKING\n\ntry:\n from typing import Literal\nexcept ImportError:\n from typing import Literal\n\nfrom ludwig.api_annotations import DeveloperAPI\nfrom ludwig.constants import AUTO, COMBINED, LOSS\nfrom ludwig.models.base import BaseModel\nfrom ludwig.models.ecd import ECD\nfrom ludwig.models.llm import LLM\nfrom ludwig.modules.metric_modules import get_best_function\nfrom ludwig.schema.trainer import ECDTrainerConfig, FineTuneTrainerConfig\nfrom ludwig.utils.data_utils import save_json\nfrom ludwig.utils.metric_utils import TrainerMetric\n\nif TYPE_CHECKING:\n from ludwig.features.base_feature import OutputFeature\n from ludwig.schema.trainer import BaseTrainerConfig\n\n\nlogger = logging.getLogger(__name__)\n\n\n@DeveloperAPI\ndef initialize_trainer_metric_dict(output_features) -> dict[str, dict[str, list[TrainerMetric]]]:\n \"\"\"Returns a dict of dict of metrics, output_feature_name -> metric_name -> List[TrainerMetric].\"\"\"\n metrics = defaultdict(lambda: defaultdict(list))\n return metrics\n\n\ndef get_latest_metrics_dict(\n progress_tracker_metrics: dict[str, dict[str, list[TrainerMetric]]],\n) -> dict[str, dict[str, float]]:\n \"\"\"Returns a dict of field name -> metric name -> latest metric value.\"\"\"\n latest_metrics_dict = defaultdict(dict)\n for feature_name, metrics_dict in progress_tracker_metrics.items():\n for metric_name, metrics in metrics_dict.items():\n if metrics:\n # Metrics may be missing if computing metrics was excepted, if the metrics are entirely empty\n # due to a missing subset, or if evaluate_training_set is False.\n latest_metrics_dict[feature_name][metric_name] = metrics[-1][-1]\n return latest_metrics_dict\n\n\n@DeveloperAPI\ndef get_new_progress_tracker(\n batch_size: int,\n best_eval_metric_value: float,\n best_increase_batch_size_eval_metric: float,\n learning_rate: float,\n output_features: dict[str, \"OutputFeature\"],\n):\n \"\"\"Returns a new instance of a ProgressTracker with empty metrics.\"\"\"\n return ProgressTracker(\n epoch=0,\n batch_size=batch_size,\n steps=0,\n tune_checkpoint_num=0,\n checkpoint_number=0,\n best_eval_metric_steps=0,\n best_eval_metric_epoch=0,\n best_eval_metric_checkpoint_number=0,\n last_learning_rate_reduction_steps=0,\n last_increase_batch_size_steps=0,\n last_improvement_steps=0,\n best_eval_metric_value=best_eval_metric_value,\n best_increase_batch_size_eval_metric=best_increase_batch_size_eval_metric,\n last_increase_batch_size_eval_metric_improvement=0,\n learning_rate=learning_rate,\n num_reductions_learning_rate=0,\n num_increases_batch_size=0,\n train_metrics=initialize_trainer_metric_dict(output_features),\n validation_metrics=initialize_trainer_metric_dict(output_features),\n test_metrics=initialize_trainer_metric_dict(output_features),\n last_learning_rate_reduction=0,\n last_increase_batch_size=0,\n best_eval_train_metrics={},\n best_eval_validation_metrics={},\n best_eval_test_metrics={},\n llm_eval_examples={},\n checkpoint_to_step={},\n checkpoint_to_epoch={},\n incremental_step_token_usage={},\n cumulative_step_token_usage={},\n incremental_checkpoint_token_usage={},\n cumulative_checkpoint_token_usage={},\n total_tokens_used=0,\n )\n\n\n@DeveloperAPI\nclass ProgressTracker:\n def __init__(\n self,\n epoch: int,\n batch_size: int,\n steps: int,\n tune_checkpoint_num: int,\n checkpoint_number: int,\n best_eval_metric_steps: int,\n best_eval_metric_epoch: int,\n best_eval_metric_checkpoint_number: int,\n last_improvement_steps: int,\n last_learning_rate_reduction_steps: int,\n last_increase_batch_size_steps: int,\n best_eval_metric_value: float,\n best_increase_batch_size_eval_metric: float,\n last_increase_batch_size_eval_metric_improvement: int,\n learning_rate: float,\n num_reductions_learning_rate: int,\n num_increases_batch_size: int,\n train_metrics: dict[str, dict[str, list[TrainerMetric]]],\n validation_metrics: dict[str, dict[str, list[TrainerMetric]]],\n test_metrics: dict[str, dict[str, list[TrainerMetric]]],\n last_learning_rate_reduction: int,\n last_increase_batch_size: int,\n best_eval_train_metrics: dict[str, dict[str, float]],\n best_eval_validation_metrics: dict[str, dict[str, float]],\n best_eval_test_metrics: dict[str, dict[str, float]],\n llm_eval_examples: dict[str, list[str]] = None,\n checkpoint_to_step: dict[str, int] = None,\n checkpoint_to_epoch: dict[str, int] = None,\n incremental_step_token_usage: dict[str, int] = None,\n cumulative_step_token_usage: dict[str, int] = None,\n incremental_checkpoint_token_usage: dict[str, int] = None,\n cumulative_checkpoint_token_usage: dict[str, int] = None,\n total_tokens_used: int = 0,\n ):\n \"\"\"JSON-serializable holder object that stores information related to training progress.\n\n [train/vali/test]_metrics is a nested dictionary of TrainerMetrics: feature_name -> metric_name ->\n List[TrainerMetrics], with one entry per training checkpoint.\n\n When the model is saved, all of the progress tracker's attributes are serialized to JSON as\n `training_progress.json` under the model output directory.\n\n JSON serialization automatically converts all dictionary top-level keys to strings, and the string typing\n is preserved when the progress tracker is deserialized from JSON when model resumes training from a checkpoint.\n\n For this reason, all of the dictionary attributes of the progress tracker are keyed by strings to ensure a\n consistent interface before or after deserialization. For example, the `tokens` dictionaries are keyed by steps,\n as strings.\n\n When the progress tracker is deserialized from JSON like when a model resumes training from a checkpoint, the\n TrainerMetrics namedtuples are automatically converted into regular (epoch, steps, value) tuples, which is why\n in trainer.py, we often use `[-1]` to index into the last element of the TrainerMetric namedtuple to get the\n actual metric value instead of the named field.\n\n Args:\n epoch: The current epoch number.\n steps: The current step of training.\n batch_size: The current batch size.\n tune_checkpoint_num: The hyperopt checkpoint number (Ray Tune).\n checkpoint_number: The current checkpoint number.\n\n best_eval_metric_steps: The step of training that has the best evaluation so far.\n best_eval_metric_epoch: The epoch of training that has the best evaluation so far.\n best_eval_metric_checkpoint_number: The checkpoint number that has the best evaluation so far.\n\n last_improvement_steps: The number of steps since the last improvement.\n last_learning_rate_reduction_steps: The training step of the last learning rate reduction.\n last_increase_batch_size_steps: The training_step of the the last batch size increase.\n\n best_eval_metric_value: The metric value of the best evaluation so far.\n best_increase_batch_size_eval_metric:\n The metric value of the best evaluation so far, for increasing the batch size.\n\n last_learning_rate_reduction: The number of steps since the last learning rate reduction.\n last_increase_batch_size: The number of steps since the last batch size increase.\n\n last_increase_batch_size_eval_metric_improvement:\n The number of checkpoints since the last batch size increase.\n\n num_reductions_learning_rate: The number of total reductions in learning rate.\n num_increases_batch_size: The number of total increases in batch size.\n\n train_metrics: Training metrics.
![](\"https://github.com/ludwig-ai/ludwig-docs/raw/main/docs/images/ludwig_hero_smaller.jpg\")
\n \n