[
  {
    "path": ".github/ISSUE_TEMPLATE/bug_report.md",
    "content": "---\nname: 🐛发现一个bug\nabout: 需提交版本号、触发代码、错误日志\ntitle: ''\nlabels: bug\nassignees: hankcs\n\n---\n\n<!--\n感谢找出bug，请认真填写下表：\n-->\n\n**Describe the bug**\nA clear and concise description of what the bug is.\n\n**Code to reproduce the issue**\nProvide a reproducible test case that is the bare minimum necessary to generate the problem.\n\n```python\n```\n\n**Describe the current behavior**\nA clear and concise description of what happened.\n\n**Expected behavior**\nA clear and concise description of what you expected to happen.\n\n**System information**\n- OS Platform and Distribution (e.g., Linux Ubuntu 16.04):\n- Python version:\n- HanLP version:\n\n**Other info / logs**\nInclude any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.\n\n* [ ] I've completed this form and searched the web for solutions.\n<!-- ⬆️此处务必勾选，否则你的issue会被机器人自动删除！ -->\n<!-- ⬆️此处务必勾选，否则你的issue会被机器人自动删除！ -->\n<!-- ⬆️此处务必勾选，否则你的issue会被机器人自动删除！ -->"
  },
  {
    "path": ".github/ISSUE_TEMPLATE/config.yml",
    "content": "blank_issues_enabled: false\ncontact_links:\n  - name: ⁉️ 提问求助请上论坛\n    url: https://bbs.hankcs.com/\n    about: 欢迎前往蝴蝶效应论坛求助\n"
  },
  {
    "path": ".github/ISSUE_TEMPLATE/feature_request.md",
    "content": "---\nname: 🚀新功能请愿\nabout: 建议增加一个新功能\ntitle: ''\nlabels: feature request\nassignees: hankcs\n\n---\n\n<!--\n提问请上论坛，不要发这里！\n提问请上论坛，不要发这里！\n提问请上论坛，不要发这里！\n\n以下必填，否则直接关闭。\n-->\n\n**Describe the feature and the current behavior/state.**\n\n**Will this change the current api? How?**\n\n**Who will benefit with this feature?**\n\n**Are you willing to contribute it (Yes/No):**\n\n**System information**\n- OS Platform and Distribution (e.g., Linux Ubuntu 16.04):\n- Python version:\n- HanLP version:\n\n**Any other info**\n\n* [ ] I've carefully completed this form.\n<!-- 发表前先搜索，此处一定要勾选！ -->\n<!-- 发表前先搜索，此处一定要勾选！ -->\n<!-- 发表前先搜索，此处一定要勾选！ -->"
  },
  {
    "path": ".github/pull_request_template.md",
    "content": "<!--\nThank you for being interested in contributing to HanLP! You are awesome ✨.\n⚠️Changes must be made on dev branch.\n-->\n\n# Title of Your Pull Request\n\n## Description\n\nPlease include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.\n\nFixes # (issue)\n\n## Type of Change\n\nPlease check any relevant options and delete the rest.\n\n- [ ] Bug fix (non-breaking change which fixes an issue)\n- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)\n- [ ] New feature (non-breaking change which adds functionality)\n- [ ] This change requires a documentation update\n\n## How Has This Been Tested?\n\nPlease describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration\n\n## Checklist\n\nCheck all items that apply.\n\n- [ ] ⚠️Changes **must** be made on `dev` branch instead of `master`\n- [ ] I have added tests that prove my fix is effective or that my feature works\n- [ ] New and existing unit tests pass locally with my changes\n- [ ] My code follows the style guidelines of this project\n- [ ] I have commented my code, particularly in hard-to-understand areas\n- [ ] I have made corresponding changes to the documentation\n- [ ] My changes generate no new warnings\n- [ ] I have checked my code and corrected any misspellings\n"
  },
  {
    "path": ".github/workflows/unit-tests.yml",
    "content": "name: Unit Tests\n\non:\n  push:\n    branches: [ \"**\" ]\n  pull_request:\n    branches: [ \"**\" ]\n\njobs:\n  build:\n\n    runs-on: ${{ matrix.os }}\n    env:\n      HANLP_HOME: ${{ github.workspace }}/data\n    strategy:\n      fail-fast: false\n      matrix:\n        os: [ ubuntu-latest, macos-latest, windows-latest ]\n        python-version: [ 3.6, 3.7, 3.8, 3.9, '3.10' ]\n        exclude:\n          # GHA doesn't list 3.6 for ubuntu-22.04\n          - os: ubuntu-latest\n            python-version: \"3.6\"\n\n          # MacOS 14.4.1 for arm64 doesn't support Python < 3.8\n          - os: macos-latest\n            python-version: \"3.6\"\n          - os: macos-latest\n            python-version: \"3.7\"\n\n        include:\n          # GHA doesn't list 3.6 for ubuntu-22\n          - os: ubuntu-20.04\n            python-version: \"3.6\"\n\n          # MacOS 13 required for Python < 3.8\n          - os: macos-13\n            python-version: \"3.6\"\n          - os: macos-13\n            python-version: \"3.7\"\n\n    steps:\n      - uses: actions/checkout@v3\n\n      - name: Set up Python ${{ matrix.python-version }}\n        uses: actions/setup-python@v3\n        with:\n          python-version: ${{ matrix.python-version }}\n\n      - name: Install dependencies\n        shell: bash\n        run: |\n          python -m pip install -e plugins/hanlp_trie\n          python -m pip install -e plugins/hanlp_common\n          python -m pip install -e .\n          python -m pip install pytest\n\n      - name: Cache data\n        uses: actions/cache@v3\n        with:\n          path: ${{ env.HANLP_HOME }}\n          key: hanlp-data\n\n      - name: Test with pytest\n        shell: bash\n        run: |\n          pytest tests\n          pytest plugins/hanlp_trie/tests\n  deploy:\n    needs: build\n    if: github.event_name == 'push' && github.ref == 'refs/heads/master'\n    runs-on: ubuntu-latest\n\n    steps:\n      - uses: actions/checkout@v3\n      - name: Install dependencies\n        run: |\n          python -m pip install setuptools wheel twine\n      - name: Deploy to PyPI\n        run: |\n          python setup.py sdist bdist_wheel\n          python -m twine upload dist/*\n        env:\n          TWINE_USERNAME: __token__\n          TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}\n          TWINE_REPOSITORY: pypi\n"
  },
  {
    "path": ".gitignore",
    "content": "# Created by .ignore support plugin (hsz.mobi)\n### Python template\n# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\nwheels/\npip-wheel-metadata/\nshare/python-wheels/\n*.egg-info/\n.installed.cfg\n*.egg\nMANIFEST\n\n# PyInstaller\n#  Usually these files are written by a python script from a template\n#  before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.nox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*.cover\n.hypothesis/\n.pytest_cache/\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\nlocal_settings.py\ndb.sqlite3\ndb.sqlite3-journal\n\n# Flask stuff:\ninstance/\n.webassets-cache\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\ntarget/\n\n# Jupyter Notebook\n.ipynb_checkpoints\n\n# IPython\nprofile_default/\nipython_config.py\n\n# pyenv\n.python-version\n\n# pipenv\n#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.\n#   However, in case of collaboration, if having platform-specific dependencies or dependencies\n#   having no cross-platform support, pipenv may install dependencies that don't work, or not\n#   install all needed dependencies.\n#Pipfile.lock\n\n# celery beat schedule file\ncelerybeat-schedule\n\n# SageMath parsed files\n*.sage.py\n\n# Environments\n.env\n.venv\nenv/\nvenv/\nENV/\nenv.bak/\nvenv.bak/\n\n# Spyder project settings\n.spyderproject\n.spyproject\n\n# Rope project settings\n.ropeproject\n\n# mkdocs documentation\n/site\n\n# mypy\n.mypy_cache/\n.dmypy.json\ndmypy.json\n\n# Pyre type checker\n.pyre/\n\n### Java template\n# Compiled class file\n*.class\n\n# Log file\n\n# BlueJ files\n*.ctxt\n\n# Mobile Tools for Java (J2ME)\n.mtj.tmp/\n\n# Package Files #\n*.jar\n*.war\n*.nar\n*.ear\n*.zip\n*.tar.gz\n*.rar\n\n# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml\nhs_err_pid*\n\n### Eclipse template\n.metadata\nbin/\ntmp/\n*.tmp\n*.bak\n*.swp\n*~.nib\nlocal.properties\n.settings/\n.loadpath\n.recommenders\n\n# External tool builders\n.externalToolBuilders/\n\n# Locally stored \"Eclipse launch configurations\"\n*.launch\n\n# PyDev specific (Python IDE for Eclipse)\n*.pydevproject\n\n# CDT-specific (C/C++ Development Tooling)\n.cproject\n\n# CDT- autotools\n.autotools\n\n# Java annotation processor (APT)\n.factorypath\n\n# PDT-specific (PHP Development Tools)\n.buildpath\n\n# sbteclipse plugin\n.target\n\n# Tern plugin\n.tern-project\n\n# TeXlipse plugin\n.texlipse\n\n# STS (Spring Tool Suite)\n.springBeans\n\n# Code Recommenders\n.recommenders/\n\n# Annotation Processing\n.apt_generated/\n\n# Scala IDE specific (Scala & Java development for Eclipse)\n.cache-main\n.scala_dependencies\n.worksheet\n\n### VisualStudioCode template\n.vscode/*\n!.vscode/settings.json\n!.vscode/tasks.json\n!.vscode/launch.json\n!.vscode/extensions.json\n\n### JetBrains template\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio and WebStorm\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n.idea/**/workspace.xml\n.idea/**/tasks.xml\n.idea/**/usage.statistics.xml\n.idea/**/dictionaries\n.idea/**/shelf\n\n# Generated files\n.idea/**/contentModel.xml\n\n# Sensitive or high-churn files\n.idea/**/dataSources/\n.idea/**/dataSources.ids\n.idea/**/dataSources.local.xml\n.idea/**/sqlDataSources.xml\n.idea/**/dynamic.xml\n.idea/**/uiDesigner.xml\n.idea/**/dbnavigator.xml\n\n# Gradle\n.idea/**/gradle.xml\n.idea/**/libraries\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\ncmake-build-*/\n\n# Mongo Explorer plugin\n.idea/**/mongoSettings.xml\n\n# File-based project format\n*.iws\n\n# IntelliJ\nout/\n\n# mpeltonen/sbt-idea plugin\n.idea_modules/\n\n# JIRA plugin\natlassian-ide-plugin.xml\n\n# Cursive Clojure plugin\n.idea/replstate.xml\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\ncom_crashlytics_export_strings.xml\ncrashlytics.properties\ncrashlytics-build.properties\nfabric.properties\n\n# Editor-based Rest HanLPClient\n.idea/httpRequests\n\n# Android studio 3.1+ serialized cache file\n.idea/caches/build_file_checksums.ser\n.idea\n*.iml\ndata\n.vscode\n*.pkl\n*.pdf\n_static/\n_build/\n_templates/"
  },
  {
    "path": "CITATION.cff",
    "content": "cff-version: 1.2.0\nmessage: \"If you use this software, please cite it as below.\"\nauthors:\n- family-names: He\n  given-names: Han\n  orcid: \"https://orcid.org/0009-0005-1778-917X\"\ntitle: \"HanLP: Han Language Processing\"\nversion: 2.1\ndate-released: 2015-05-27\nurl: \"https://github.com/hankcs/HanLP\"\npreferred-citation:\n  type: conference-paper\n  authors:\n    - family-names: He\n      given-names: Han\n    - family-names: Choi\n      given-names: Jinho D.\n  title: \"The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders\"\n  editors:\n    - family-names: Moens\n      given-names: Marie-Francine\n    - family-names: Huang\n      given-names: Xuanjing\n    - family-names: Specia\n      given-names: Lucia\n    - family-names: Yih\n      given-names: Scott Wen-tau\n  year: 2021\n  month: 11\n  date-released: 2021-11\n  conference:\n    name: \"2021 Conference on Empirical Methods in Natural Language Processing\"\n    place: \"Online and Punta Cana, Dominican Republic\"\n    url: \"https://aclanthology.org/2021.emnlp-main.451\"\n  doi: \"10.18653/v1/2021.emnlp-main.451\"\n  url: \"https://aclanthology.org/2021.emnlp-main.451\"\n  publisher: \"Association for Computational Linguistics\"\n  booktitle: \"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing\"\n  location: \"Online and Punta Cana, Dominican Republic\"\n  pages: \"5555-5577\"\n"
  },
  {
    "path": "LICENSE",
    "content": "                                 Apache License\n                           Version 2.0, January 2004\n                        http://www.apache.org/licenses/\n\n   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n   1. Definitions.\n\n      \"License\" shall mean the terms and conditions for use, reproduction,\n      and distribution as defined by Sections 1 through 9 of this document.\n\n      \"Licensor\" shall mean the copyright owner or entity authorized by\n      the copyright owner that is granting the License.\n\n      \"Legal Entity\" shall mean the union of the acting entity and all\n      other entities that control, are controlled by, or are under common\n      control with that entity. For the purposes of this definition,\n      \"control\" means (i) the power, direct or indirect, to cause the\n      direction or management of such entity, whether by contract or\n      otherwise, or (ii) ownership of fifty percent (50%) or more of the\n      outstanding shares, or (iii) beneficial ownership of such entity.\n\n      \"You\" (or \"Your\") shall mean an individual or Legal Entity\n      exercising permissions granted by this License.\n\n      \"Source\" form shall mean the preferred form for making modifications,\n      including but not limited to software source code, documentation\n      source, and configuration files.\n\n      \"Object\" form shall mean any form resulting from mechanical\n      transformation or translation of a Source form, including but\n      not limited to compiled object code, generated documentation,\n      and conversions to other media types.\n\n      \"Work\" shall mean the work of authorship, whether in Source or\n      Object form, made available under the License, as indicated by a\n      copyright notice that is included in or attached to the work\n      (an example is provided in the Appendix below).\n\n      \"Derivative Works\" shall mean any work, whether in Source or Object\n      form, that is based on (or derived from) the Work and for which the\n      editorial revisions, annotations, elaborations, or other modifications\n      represent, as a whole, an original work of authorship. For the purposes\n      of this License, Derivative Works shall not include works that remain\n      separable from, or merely link (or bind by name) to the interfaces of,\n      the Work and Derivative Works thereof.\n\n      \"Contribution\" shall mean any work of authorship, including\n      the original version of the Work and any modifications or additions\n      to that Work or Derivative Works thereof, that is intentionally\n      submitted to Licensor for inclusion in the Work by the copyright owner\n      or by an individual or Legal Entity authorized to submit on behalf of\n      the copyright owner. For the purposes of this definition, \"submitted\"\n      means any form of electronic, verbal, or written communication sent\n      to the Licensor or its representatives, including but not limited to\n      communication on electronic mailing lists, source code control systems,\n      and issue tracking systems that are managed by, or on behalf of, the\n      Licensor for the purpose of discussing and improving the Work, but\n      excluding communication that is conspicuously marked or otherwise\n      designated in writing by the copyright owner as \"Not a Contribution.\"\n\n      \"Contributor\" shall mean Licensor and any individual or Legal Entity\n      on behalf of whom a Contribution has been received by Licensor and\n      subsequently incorporated within the Work.\n\n   2. Grant of Copyright License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      copyright license to reproduce, prepare Derivative Works of,\n      publicly display, publicly perform, sublicense, and distribute the\n      Work and such Derivative Works in Source or Object form.\n\n   3. Grant of Patent License. Subject to the terms and conditions of\n      this License, each Contributor hereby grants to You a perpetual,\n      worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n      (except as stated in this section) patent license to make, have made,\n      use, offer to sell, sell, import, and otherwise transfer the Work,\n      where such license applies only to those patent claims licensable\n      by such Contributor that are necessarily infringed by their\n      Contribution(s) alone or by combination of their Contribution(s)\n      with the Work to which such Contribution(s) was submitted. If You\n      institute patent litigation against any entity (including a\n      cross-claim or counterclaim in a lawsuit) alleging that the Work\n      or a Contribution incorporated within the Work constitutes direct\n      or contributory patent infringement, then any patent licenses\n      granted to You under this License for that Work shall terminate\n      as of the date such litigation is filed.\n\n   4. Redistribution. You may reproduce and distribute copies of the\n      Work or Derivative Works thereof in any medium, with or without\n      modifications, and in Source or Object form, provided that You\n      meet the following conditions:\n\n      (a) You must give any other recipients of the Work or\n          Derivative Works a copy of this License; and\n\n      (b) You must cause any modified files to carry prominent notices\n          stating that You changed the files; and\n\n      (c) You must retain, in the Source form of any Derivative Works\n          that You distribute, all copyright, patent, trademark, and\n          attribution notices from the Source form of the Work,\n          excluding those notices that do not pertain to any part of\n          the Derivative Works; and\n\n      (d) If the Work includes a \"NOTICE\" text file as part of its\n          distribution, then any Derivative Works that You distribute must\n          include a readable copy of the attribution notices contained\n          within such NOTICE file, excluding those notices that do not\n          pertain to any part of the Derivative Works, in at least one\n          of the following places: within a NOTICE text file distributed\n          as part of the Derivative Works; within the Source form or\n          documentation, if provided along with the Derivative Works; or,\n          within a display generated by the Derivative Works, if and\n          wherever such third-party notices normally appear. The contents\n          of the NOTICE file are for informational purposes only and\n          do not modify the License. You may add Your own attribution\n          notices within Derivative Works that You distribute, alongside\n          or as an addendum to the NOTICE text from the Work, provided\n          that such additional attribution notices cannot be construed\n          as modifying the License.\n\n      You may add Your own copyright statement to Your modifications and\n      may provide additional or different license terms and conditions\n      for use, reproduction, or distribution of Your modifications, or\n      for any such Derivative Works as a whole, provided Your use,\n      reproduction, and distribution of the Work otherwise complies with\n      the conditions stated in this License.\n\n   5. Submission of Contributions. Unless You explicitly state otherwise,\n      any Contribution intentionally submitted for inclusion in the Work\n      by You to the Licensor shall be under the terms and conditions of\n      this License, without any additional terms or conditions.\n      Notwithstanding the above, nothing herein shall supersede or modify\n      the terms of any separate license agreement you may have executed\n      with Licensor regarding such Contributions.\n\n   6. Trademarks. This License does not grant permission to use the trade\n      names, trademarks, service marks, or product names of the Licensor,\n      except as required for reasonable and customary use in describing the\n      origin of the Work and reproducing the content of the NOTICE file.\n\n   7. Disclaimer of Warranty. Unless required by applicable law or\n      agreed to in writing, Licensor provides the Work (and each\n      Contributor provides its Contributions) on an \"AS IS\" BASIS,\n      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n      implied, including, without limitation, any warranties or conditions\n      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n      PARTICULAR PURPOSE. You are solely responsible for determining the\n      appropriateness of using or redistributing the Work and assume any\n      risks associated with Your exercise of permissions under this License.\n\n   8. Limitation of Liability. In no event and under no legal theory,\n      whether in tort (including negligence), contract, or otherwise,\n      unless required by applicable law (such as deliberate and grossly\n      negligent acts) or agreed to in writing, shall any Contributor be\n      liable to You for damages, including any direct, indirect, special,\n      incidental, or consequential damages of any character arising as a\n      result of this License or out of the use or inability to use the\n      Work (including but not limited to damages for loss of goodwill,\n      work stoppage, computer failure or malfunction, or any and all\n      other commercial damages or losses), even if such Contributor\n      has been advised of the possibility of such damages.\n\n   9. Accepting Warranty or Additional Liability. While redistributing\n      the Work or Derivative Works thereof, You may choose to offer,\n      and charge a fee for, acceptance of support, warranty, indemnity,\n      or other liability obligations and/or rights consistent with this\n      License. However, in accepting such obligations, You may act only\n      on Your own behalf and on Your sole responsibility, not on behalf\n      of any other Contributor, and only if You agree to indemnify,\n      defend, and hold each Contributor harmless for any liability\n      incurred by, or claims asserted against, such Contributor by reason\n      of your accepting any such warranty or additional liability.\n\n   END OF TERMS AND CONDITIONS\n\n   APPENDIX: How to apply the Apache License to your work.\n\n      To apply the Apache License to your work, attach the following\n      boilerplate notice, with the fields enclosed by brackets \"[]\"\n      replaced with your own identifying information. (Don't include\n      the brackets!)  The text should be enclosed in the appropriate\n      comment syntax for the file format. We also recommend that a\n      file or class name and description of purpose be included on the\n      same \"printed page\" as the copyright notice for easier\n      identification within third-party archives.\n\n   Copyright [yyyy] [name of copyright owner]\n\n   Licensed under the Apache License, Version 2.0 (the \"License\");\n   you may not use this file except in compliance with the License.\n   You may obtain a copy of the License at\n\n       http://www.apache.org/licenses/LICENSE-2.0\n\n   Unless required by applicable law or agreed to in writing, software\n   distributed under the License is distributed on an \"AS IS\" BASIS,\n   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n   See the License for the specific language governing permissions and\n   limitations under the License.\n"
  },
  {
    "path": "README.md",
    "content": "<h2 align=\"center\">HanLP: Han Language Processing</h2>\n\n<div align=\"center\">\n    <a href=\"https://github.com/hankcs/HanLP/actions/workflows/unit-tests.yml\">\n       <img alt=\"Unit Tests\" src=\"https://github.com/hankcs/hanlp/actions/workflows/unit-tests.yml/badge.svg?branch=master\">\n    </a>\n    <a href=\"https://pypi.org/project/hanlp/\">\n        <img alt=\"PyPI Version\" src=\"https://img.shields.io/pypi/v/hanlp?color=blue\">\n    </a>\n    <a href=\"https://pypi.org/project/hanlp/\">\n        <img alt=\"Python Versions\" src=\"https://img.shields.io/pypi/pyversions/hanlp?colorB=blue\">\n    </a>\n    <a href=\"https://pepy.tech/project/hanlp\">\n        <img alt=\"Downloads\" src=\"https://static.pepy.tech/badge/hanlp\">\n    </a>\n    <a href=\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Ftutorial.ipynb\">\n        <img alt=\"在线运行\" src=\"https://mybinder.org/badge_logo.svg\">\n    </a>\n</div>\n<h4 align=\"center\">\n    <a href=\"https://github.com/hankcs/HanLP/tree/master\">English</a> |\n    <a href=\"https://github.com/hankcs/HanLP/tree/doc-ja\">日本語</a> |\n    <a href=\"https://hanlp.hankcs.com/docs/\">文档</a> |\n    <a href=\"https://bbs.hankcs.com/t/topic/3940\">论文</a> |\n    <a href=\"https://bbs.hankcs.com/\">论坛</a> |\n    <a href=\"https://github.com/wangedison/hanlp-jupyterlab-docker\">docker</a> |\n    <a href=\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Ftutorial.ipynb\">▶️在线运行</a>\n</h4>\n\n\n\n面向生产环境的多语种自然语言处理工具包，基于PyTorch和TensorFlow 2.x双引擎，目标是普及落地最前沿的NLP技术。HanLP具备功能完善、精度准确、性能高效、语料时新、架构清晰、可自定义的特点。\n\n[![demo](https://raw.githubusercontent.com/hankcs/OpenCC-to-HanLP/img/demo.gif)](https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Ftutorial.ipynb)\n\n借助世界上最大的多语种语料库，HanLP2.1支持包括简繁中英日俄法德在内的[130种语言](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/mtl.html#hanlp.pretrained.mtl.UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_MMINILMV2L6)上的10种联合任务以及多种单任务。HanLP预训练了十几种任务上的数十个模型并且正在持续迭代语料库与模型：\n\n<div align=\"center\">\n\n| 功能                                                         | RESTful                                                      | 多任务                                                       | 单任务                                                       | 模型                                                         | 标注标准                                                     |\n| ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ |\n| [分词](https://hanlp.hankcs.com/demos/tok.html)              | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/tok_restful.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/tok_mtl.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/tok_stl.ipynb) | [tok](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/tok.html) | [粗分](https://hanlp.hankcs.com/docs/annotations/tok/msr.html)、[细分](https://hanlp.hankcs.com/docs/annotations/tok/ctb.html) |\n| [词性标注](https://hanlp.hankcs.com/demos/pos.html)          | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/pos_restful.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/pos_mtl.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/pos_stl.ipynb) | [pos](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/pos.html) | [CTB](https://hanlp.hankcs.com/docs/annotations/pos/ctb.html)、[PKU](https://hanlp.hankcs.com/docs/annotations/pos/pku.html)、[863](https://hanlp.hankcs.com/docs/annotations/pos/863.html) |\n| [命名实体识别](https://hanlp.hankcs.com/demos/ner.html)      | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/ner_restful.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/ner_mtl.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/ner_stl.ipynb) | [ner](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/ner.html) | [PKU](https://hanlp.hankcs.com/docs/annotations/ner/pku.html)、[MSRA](https://hanlp.hankcs.com/docs/annotations/ner/msra.html)、[OntoNotes](https://hanlp.hankcs.com/docs/annotations/ner/ontonotes.html) |\n| [依存句法分析](https://hanlp.hankcs.com/demos/dep.html)      | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/dep_restful.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/dep_mtl.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/dep_stl.ipynb) | [dep](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/dep.html) | [SD](https://hanlp.hankcs.com/docs/annotations/dep/sd_zh.html)、[UD](https://hanlp.hankcs.com/docs/annotations/dep/ud.html#chinese)、[PMT](https://hanlp.hankcs.com/docs/annotations/dep/pmt.html) |\n| [成分句法分析](https://hanlp.hankcs.com/demos/con.html)      | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/con_restful.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/con_mtl.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/con_stl.ipynb) | [con](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/constituency.html) | [Chinese Tree Bank](https://hanlp.hankcs.com/docs/annotations/constituency/ctb.html) |\n| [语义依存分析](https://hanlp.hankcs.com/demos/sdp.html)      | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/sdp_restful.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/sdp_mtl.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/sdp_stl.ipynb) | [sdp](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/sdp.html) | [CSDP](https://hanlp.hankcs.com/docs/annotations/sdp/semeval16.html#) |\n| [语义角色标注](https://hanlp.hankcs.com/demos/srl.html)      | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/srl_restful.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/srl_mtl.ipynb) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/srl_stl.ipynb) | [srl](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/srl.html) | [Chinese Proposition Bank](https://hanlp.hankcs.com/docs/annotations/srl/cpb.html) |\n| [抽象意义表示](https://hanlp.hankcs.com/demos/amr.html)      | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/amr_restful.ipynb) | 暂无                                                         | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/amr_stl.ipynb) | [amr](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/amr.html) | [CAMR](https://www.hankcs.com/nlp/corpus/introduction-to-chinese-abstract-meaning-representation.html) |\n| [指代消解](https://hanlp.hankcs.com/demos/cor.html)          | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/cor_restful.ipynb) | 暂无                                                         | 暂无                                                         | 暂无                                                         | OntoNotes                                                    |\n| [语义文本相似度](https://hanlp.hankcs.com/demos/sts.html)    | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/sts_restful.ipynb) | 暂无                                                         | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/sts_stl.ipynb) | [sts](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/sts.html) | 暂无                                                         |\n| [文本风格转换](https://hanlp.hankcs.com/demos/tst.html)      | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/tst_restful.ipynb) | 暂无                                                         | 暂无                                                         | 暂无                                                         | 暂无                                                         |\n| [关键词短语提取](https://hanlp.hankcs.com/demos/keyphrase.html) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/keyphrase_restful.ipynb) | 暂无                                                         | 暂无                                                         | 暂无                                                         | 暂无                                                         |\n| [抽取式自动摘要](https://hanlp.hankcs.com/demos/exsum.html)  | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/extractive_summarization_restful.ipynb) | 暂无                                                         | 暂无                                                         | 暂无                                                         | 暂无                                                         |\n| [生成式自动摘要](https://hanlp.hankcs.com/demos/absum.html)  | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/abstractive_summarization_restful.ipynb) | 暂无                                                         | 暂无                                                         | 暂无                                                         | 暂无                                                         |\n| [文本语法纠错](https://hanlp.hankcs.com/demos/gec.html)      | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/gec_restful.ipynb) | 暂无                                                         | 暂无                                                         | 暂无                                                         | 暂无                                                         |\n| [文本分类](https://hanlp.hankcs.com/demos/classification.html) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/classification_restful.ipynb) | 暂无                                                         | 暂无                                                         | 暂无                                                         | 暂无                                                         |\n| [情感分析](https://hanlp.hankcs.com/demos/sentiment.html)    | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/sentiment_restful.ipynb) | 暂无                                                         | 暂无                                                         | 暂无                                                         | `[-1,+1]`                                                    |\n| [语种检测](https://hanlp.hankcs.com/demos/classification.html) | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/lid_restful.ipynb) | 暂无                                                         | [教程](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/lid_stl.ipynb) | 暂无                                                         | [ISO 639-1编码](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) |\n\n</div>\n\n- 词干提取、词法语法特征提取请参考[英文教程](https://hanlp.hankcs.com/docs/tutorial.html)；[词向量](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/word2vec.html)和[完形填空](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/mlm.html)请参考相应文档。\n- 简繁转换、拼音、新词发现、文本聚类请参考[1.x教程](https://github.com/hankcs/HanLP/tree/1.x)。\n\n量体裁衣，HanLP提供**RESTful**和**native**两种API，分别面向轻量级和海量级两种场景。无论何种API何种语言，HanLP接口在语义上保持一致，在代码上坚持开源。如果您在研究中使用了HanLP，请引用我们的[EMNLP论文](https://aclanthology.org/2021.emnlp-main.451/)。\n\n### 轻量级RESTful API\n\n仅数KB，适合敏捷开发、移动APP等场景。简单易用，无需GPU配环境，秒速安装。语料更多、模型更大、精度更高，**强烈推荐**。服务器GPU算力有限，匿名用户配额较少，[建议申请**免费公益**API秘钥`auth`](https://bbs.hanlp.com/t/hanlp2-1-restful-api/53)。\n\n#### Python\n\n```shell\npip install hanlp_restful\n```\n\n创建客户端，填入服务器地址和秘钥：\n\n```python\nfrom hanlp_restful import HanLPClient\nHanLP = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh') # auth不填则匿名，zh中文，mul多语种\n```\n\n#### Golang\n\n安装 `go get -u github.com/hankcs/gohanlp@main` ，创建客户端，填入服务器地址和秘钥：\n\n```go\nHanLP := hanlp.HanLPClient(hanlp.WithAuth(\"\"),hanlp.WithLanguage(\"zh\")) // auth不填则匿名，zh中文，mul多语种\n```\n\n#### Java\n\n在`pom.xml`中添加依赖：\n\n```xml\n<dependency>\n    <groupId>com.hankcs.hanlp.restful</groupId>\n    <artifactId>hanlp-restful</artifactId>\n    <version>0.0.12</version>\n</dependency>\n```\n\n创建客户端，填入服务器地址和秘钥：\n\n```java\nHanLPClient HanLP = new HanLPClient(\"https://www.hanlp.com/api\", null, \"zh\"); // auth不填则匿名，zh中文，mul多语种\n```\n\n#### 快速上手\n\n无论何种开发语言，调用`parse`接口，传入一篇文章，得到HanLP精准的分析结果。\n\n```java\nHanLP.parse(\"2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。阿婆主来到北京立方庭参观自然语义科技公司。\")\n```\n\n更多功能包括语义相似度、风格转换、指代消解等，请参考[文档](https://hanlp.hankcs.com/docs/api/restful.html)和[测试用例](https://github.com/hankcs/HanLP/blob/master/plugins/hanlp_restful/tests/test_client.py)。\n\n### 海量级native API\n\n依赖PyTorch、TensorFlow等深度学习技术，适合**专业**NLP工程师、研究者以及本地海量数据场景。要求Python 3.6至3.10，支持Windows，推荐*nix。可以在CPU上运行，推荐GPU/TPU。安装PyTorch版：\n\n```bash\npip install hanlp\n```\n\n- HanLP每次发布都通过了Linux、macOS和Windows上Python3.6至3.10的[单元测试](https://github.com/hankcs/HanLP/actions?query=branch%3Amaster)，不存在安装问题。\n\nHanLP发布的模型分为多任务和单任务两种，多任务速度快省显存，单任务精度高更灵活。\n\n#### 多任务模型\n\nHanLP的工作流程为加载模型然后将其当作函数调用，例如下列联合多任务模型：\n\n```python\nimport hanlp\nHanLP = hanlp.load(hanlp.pretrained.mtl.CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH) # 世界最大中文语料库\nHanLP(['2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', '阿婆主来到北京立方庭参观自然语义科技公司。'])\n```\n\nNative API的输入单位为句子，需使用[多语种分句模型](https://github.com/hankcs/HanLP/blob/master/plugins/hanlp_demo/hanlp_demo/sent_split.py)或[基于规则的分句函数](https://github.com/hankcs/HanLP/blob/master/hanlp/utils/rules.py#L19)先行分句。RESTful和native两种API的语义设计完全一致，用户可以无缝互换。简洁的接口也支持灵活的参数，常用的技巧有：\n\n- 灵活的`tasks`任务调度，任务越少，速度越快，详见[教程](https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Ftutorial.ipynb)。在内存有限的场景下，用户还可以[删除不需要的任务](https://github.com/hankcs/HanLP/blob/master/plugins/hanlp_demo/hanlp_demo/zh/demo_del_tasks.py)达到模型瘦身的效果。\n- 高效的trie树自定义词典，以及强制、合并、校正3种规则，请参考[demo](https://github.com/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/tok_mtl.ipynb)和[文档](https://hanlp.hankcs.com/docs/api/hanlp/components/tokenizers/transformer.html)。规则系统的效果将无缝应用到后续统计模型，从而快速适应新领域。\n\n#### 单任务模型\n\n根据我们的[最新研究](https://aclanthology.org/2021.emnlp-main.451)，多任务学习的优势在于速度和显存，然而精度往往不如单任务模型。所以，HanLP预训练了许多单任务模型并设计了优雅的[流水线模式](https://hanlp.hankcs.com/docs/api/hanlp/components/pipeline.html#hanlp.components.pipeline.Pipeline)将其组装起来。\n\n```python\nimport hanlp\nHanLP = hanlp.pipeline() \\\n    .append(hanlp.utils.rules.split_sentence, output_key='sentences') \\\n    .append(hanlp.load('FINE_ELECTRA_SMALL_ZH'), output_key='tok') \\\n    .append(hanlp.load('CTB9_POS_ELECTRA_SMALL'), output_key='pos') \\\n    .append(hanlp.load('MSRA_NER_ELECTRA_SMALL_ZH'), output_key='ner', input_key='tok') \\\n    .append(hanlp.load('CTB9_DEP_ELECTRA_SMALL', conll=0), output_key='dep', input_key='tok')\\\n    .append(hanlp.load('CTB9_CON_ELECTRA_SMALL'), output_key='con', input_key='tok')\nHanLP('2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。阿婆主来到北京立方庭参观自然语义科技公司。')\n```\n\n更多功能，请参考[demo](https://github.com/hankcs/HanLP/tree/doc-zh/plugins/hanlp_demo/hanlp_demo/zh)和[文档](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/index.html)了解更多模型与用法。\n\n### 输出格式\n\n无论何种API何种开发语言何种自然语言，HanLP的输出统一为`json`格式兼容`dict`的[`Document`](https://hanlp.hankcs.com/docs/api/common/document.html):\n\n```json\n{\n  \"tok/fine\": [\n    [\"2021年\", \"HanLPv2.1\", \"为\", \"生产\", \"环境\", \"带来\", \"次\", \"世代\", \"最\", \"先进\", \"的\", \"多\", \"语种\", \"NLP\", \"技术\", \"。\"],\n    [\"阿婆主\", \"来到\", \"北京\", \"立方庭\", \"参观\", \"自然\", \"语义\", \"科技\", \"公司\", \"。\"]\n  ],\n  \"tok/coarse\": [\n    [\"2021年\", \"HanLPv2.1\", \"为\", \"生产\", \"环境\", \"带来\", \"次世代\", \"最\", \"先进\", \"的\", \"多语种\", \"NLP\", \"技术\", \"。\"],\n    [\"阿婆主\", \"来到\", \"北京立方庭\", \"参观\", \"自然语义科技公司\", \"。\"]\n  ],\n  \"pos/ctb\": [\n    [\"NT\", \"NR\", \"P\", \"NN\", \"NN\", \"VV\", \"JJ\", \"NN\", \"AD\", \"JJ\", \"DEG\", \"CD\", \"NN\", \"NR\", \"NN\", \"PU\"],\n    [\"NN\", \"VV\", \"NR\", \"NR\", \"VV\", \"NN\", \"NN\", \"NN\", \"NN\", \"PU\"]\n  ],\n  \"pos/pku\": [\n    [\"t\", \"nx\", \"p\", \"vn\", \"n\", \"v\", \"b\", \"n\", \"d\", \"a\", \"u\", \"a\", \"n\", \"nx\", \"n\", \"w\"],\n    [\"n\", \"v\", \"ns\", \"ns\", \"v\", \"n\", \"n\", \"n\", \"n\", \"w\"]\n  ],\n  \"pos/863\": [\n    [\"nt\", \"w\", \"p\", \"v\", \"n\", \"v\", \"a\", \"nt\", \"d\", \"a\", \"u\", \"a\", \"n\", \"ws\", \"n\", \"w\"],\n    [\"n\", \"v\", \"ns\", \"n\", \"v\", \"n\", \"n\", \"n\", \"n\", \"w\"]\n  ],\n  \"ner/pku\": [\n    [],\n    [[\"北京立方庭\", \"ns\", 2, 4], [\"自然语义科技公司\", \"nt\", 5, 9]]\n  ],\n  \"ner/msra\": [\n    [[\"2021年\", \"DATE\", 0, 1], [\"HanLPv2.1\", \"ORGANIZATION\", 1, 2]],\n    [[\"北京\", \"LOCATION\", 2, 3], [\"立方庭\", \"LOCATION\", 3, 4], [\"自然语义科技公司\", \"ORGANIZATION\", 5, 9]]\n  ],\n  \"ner/ontonotes\": [\n    [[\"2021年\", \"DATE\", 0, 1], [\"HanLPv2.1\", \"ORG\", 1, 2]],\n    [[\"北京立方庭\", \"FAC\", 2, 4], [\"自然语义科技公司\", \"ORG\", 5, 9]]\n  ],\n  \"srl\": [\n    [[[\"2021年\", \"ARGM-TMP\", 0, 1], [\"HanLPv2.1\", \"ARG0\", 1, 2], [\"为生产环境\", \"ARG2\", 2, 5], [\"带来\", \"PRED\", 5, 6], [\"次世代最先进的多语种NLP技术\", \"ARG1\", 6, 15]], [[\"最\", \"ARGM-ADV\", 8, 9], [\"先进\", \"PRED\", 9, 10], [\"技术\", \"ARG0\", 14, 15]]],\n    [[[\"阿婆主\", \"ARG0\", 0, 1], [\"来到\", \"PRED\", 1, 2], [\"北京立方庭\", \"ARG1\", 2, 4]], [[\"阿婆主\", \"ARG0\", 0, 1], [\"参观\", \"PRED\", 4, 5], [\"自然语义科技公司\", \"ARG1\", 5, 9]]]\n  ],\n  \"dep\": [\n    [[6, \"tmod\"], [6, \"nsubj\"], [6, \"prep\"], [5, \"nn\"], [3, \"pobj\"], [0, \"root\"], [8, \"amod\"], [15, \"nn\"], [10, \"advmod\"], [15, \"rcmod\"], [10, \"assm\"], [13, \"nummod\"], [15, \"nn\"], [15, \"nn\"], [6, \"dobj\"], [6, \"punct\"]],\n    [[2, \"nsubj\"], [0, \"root\"], [4, \"nn\"], [2, \"dobj\"], [2, \"conj\"], [9, \"nn\"], [9, \"nn\"], [9, \"nn\"], [5, \"dobj\"], [2, \"punct\"]]\n  ],\n  \"sdp\": [\n    [[[6, \"Time\"]], [[6, \"Exp\"]], [[5, \"mPrep\"]], [[5, \"Desc\"]], [[6, \"Datv\"]], [[13, \"dDesc\"]], [[0, \"Root\"], [8, \"Desc\"], [13, \"Desc\"]], [[15, \"Time\"]], [[10, \"mDegr\"]], [[15, \"Desc\"]], [[10, \"mAux\"]], [[8, \"Quan\"], [13, \"Quan\"]], [[15, \"Desc\"]], [[15, \"Nmod\"]], [[6, \"Pat\"]], [[6, \"mPunc\"]]],\n    [[[2, \"Agt\"], [5, \"Agt\"]], [[0, \"Root\"]], [[4, \"Loc\"]], [[2, \"Lfin\"]], [[2, \"ePurp\"]], [[8, \"Nmod\"]], [[9, \"Nmod\"]], [[9, \"Nmod\"]], [[5, \"Datv\"]], [[5, \"mPunc\"]]]\n  ],\n  \"con\": [\n    [\"TOP\", [[\"IP\", [[\"NP\", [[\"NT\", [\"2021年\"]]]], [\"NP\", [[\"NR\", [\"HanLPv2.1\"]]]], [\"VP\", [[\"PP\", [[\"P\", [\"为\"]], [\"NP\", [[\"NN\", [\"生产\"]], [\"NN\", [\"环境\"]]]]]], [\"VP\", [[\"VV\", [\"带来\"]], [\"NP\", [[\"ADJP\", [[\"NP\", [[\"ADJP\", [[\"JJ\", [\"次\"]]]], [\"NP\", [[\"NN\", [\"世代\"]]]]]], [\"ADVP\", [[\"AD\", [\"最\"]]]], [\"VP\", [[\"JJ\", [\"先进\"]]]]]], [\"DEG\", [\"的\"]], [\"NP\", [[\"QP\", [[\"CD\", [\"多\"]]]], [\"NP\", [[\"NN\", [\"语种\"]]]]]], [\"NP\", [[\"NR\", [\"NLP\"]], [\"NN\", [\"技术\"]]]]]]]]]], [\"PU\", [\"。\"]]]]]],\n    [\"TOP\", [[\"IP\", [[\"NP\", [[\"NN\", [\"阿婆主\"]]]], [\"VP\", [[\"VP\", [[\"VV\", [\"来到\"]], [\"NP\", [[\"NR\", [\"北京\"]], [\"NR\", [\"立方庭\"]]]]]], [\"VP\", [[\"VV\", [\"参观\"]], [\"NP\", [[\"NN\", [\"自然\"]], [\"NN\", [\"语义\"]], [\"NN\", [\"科技\"]], [\"NN\", [\"公司\"]]]]]]]], [\"PU\", [\"。\"]]]]]]\n  ]\n}\n```\n\n特别地，Python RESTful和native API支持基于等宽字体的[可视化](https://hanlp.hankcs.com/docs/tutorial.html#visualization)，能够直接将语言学结构在控制台内可视化出来：\n\n```python\nHanLP(['2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', '阿婆主来到北京立方庭参观自然语义科技公司。']).pretty_print()\n\nDep Tree    \tToken    \tRelati\tPoS\tTok      \tNER Type        \tTok      \tSRL PA1     \tTok      \tSRL PA2     \tTok      \tPoS    3       4       5       6       7       8       9 \n────────────\t─────────\t──────\t───\t─────────\t────────────────\t─────────\t────────────\t─────────\t────────────\t─────────\t─────────────────────────────────────────────────────────\n ┌─────────►\t2021年    \ttmod  \tNT \t2021年    \t───►DATE        \t2021年    \t───►ARGM-TMP\t2021年    \t            \t2021年    \tNT ───────────────────────────────────────────►NP ───┐   \n │┌────────►\tHanLPv2.1\tnsubj \tNR \tHanLPv2.1\t───►ORGANIZATION\tHanLPv2.1\t───►ARG0    \tHanLPv2.1\t            \tHanLPv2.1\tNR ───────────────────────────────────────────►NP────┤   \n ││┌─►┌─────\t为        \tprep  \tP  \t为        \t                \t为        \t◄─┐         \t为        \t            \t为        \tP ───────────┐                                       │   \n │││  │  ┌─►\t生产       \tnn    \tNN \t生产       \t                \t生产       \t  ├►ARG2    \t生产       \t            \t生产       \tNN ──┐       ├────────────────────────►PP ───┐       │   \n │││  └─►└──\t环境       \tpobj  \tNN \t环境       \t                \t环境       \t◄─┘         \t环境       \t            \t环境       \tNN ──┴►NP ───┘                               │       │   \n┌┼┴┴────────\t带来       \troot  \tVV \t带来       \t                \t带来       \t╟──►PRED    \t带来       \t            \t带来       \tVV ──────────────────────────────────┐       │       │   \n││       ┌─►\t次        \tamod  \tJJ \t次        \t                \t次        \t◄─┐         \t次        \t            \t次        \tJJ ───►ADJP──┐                       │       ├►VP────┤   \n││  ┌───►└──\t世代       \tnn    \tNN \t世代       \t                \t世代       \t  │         \t世代       \t            \t世代       \tNN ───►NP ───┴►NP ───┐               │       │       │   \n││  │    ┌─►\t最        \tadvmod\tAD \t最        \t                \t最        \t  │         \t最        \t───►ARGM-ADV\t最        \tAD ───────────►ADVP──┼►ADJP──┐       ├►VP ───┘       ├►IP\n││  │┌──►├──\t先进       \trcmod \tJJ \t先进       \t                \t先进       \t  │         \t先进       \t╟──►PRED    \t先进       \tJJ ───────────►VP ───┘       │       │               │   \n││  ││   └─►\t的        \tassm  \tDEG\t的        \t                \t的        \t  ├►ARG1    \t的        \t            \t的        \tDEG──────────────────────────┤       │               │   \n││  ││   ┌─►\t多        \tnummod\tCD \t多        \t                \t多        \t  │         \t多        \t            \t多        \tCD ───►QP ───┐               ├►NP ───┘               │   \n││  ││┌─►└──\t语种       \tnn    \tNN \t语种       \t                \t语种       \t  │         \t语种       \t            \t语种       \tNN ───►NP ───┴────────►NP────┤                       │   \n││  │││  ┌─►\tNLP      \tnn    \tNR \tNLP      \t                \tNLP      \t  │         \tNLP      \t            \tNLP      \tNR ──┐                       │                       │   \n│└─►└┴┴──┴──\t技术       \tdobj  \tNN \t技术       \t                \t技术       \t◄─┘         \t技术       \t───►ARG0    \t技术       \tNN ──┴────────────────►NP ───┘                       │   \n└──────────►\t。        \tpunct \tPU \t。        \t                \t。        \t            \t。        \t            \t。        \tPU ──────────────────────────────────────────────────┘   \n\nDep Tree    \tTok\tRelat\tPo\tTok\tNER Type        \tTok\tSRL PA1 \tTok\tSRL PA2 \tTok\tPo    3       4       5       6 \n────────────\t───\t─────\t──\t───\t────────────────\t───\t────────\t───\t────────\t───\t────────────────────────────────\n         ┌─►\t阿婆主\tnsubj\tNN\t阿婆主\t                \t阿婆主\t───►ARG0\t阿婆主\t───►ARG0\t阿婆主\tNN───────────────────►NP ───┐   \n┌┬────┬──┴──\t来到 \troot \tVV\t来到 \t                \t来到 \t╟──►PRED\t来到 \t        \t来到 \tVV──────────┐               │   \n││    │  ┌─►\t北京 \tnn   \tNR\t北京 \t───►LOCATION    \t北京 \t◄─┐     \t北京 \t        \t北京 \tNR──┐       ├►VP ───┐       │   \n││    └─►└──\t立方庭\tdobj \tNR\t立方庭\t───►LOCATION    \t立方庭\t◄─┴►ARG1\t立方庭\t        \t立方庭\tNR──┴►NP ───┘       │       │   \n│└─►┌───────\t参观 \tconj \tVV\t参观 \t                \t参观 \t        \t参观 \t╟──►PRED\t参观 \tVV──────────┐       ├►VP────┤   \n│   │  ┌───►\t自然 \tnn   \tNN\t自然 \t◄─┐             \t自然 \t        \t自然 \t◄─┐     \t自然 \tNN──┐       │       │       ├►IP\n│   │  │┌──►\t语义 \tnn   \tNN\t语义 \t  │             \t语义 \t        \t语义 \t  │     \t语义 \tNN  │       ├►VP ───┘       │   \n│   │  ││┌─►\t科技 \tnn   \tNN\t科技 \t  ├►ORGANIZATION\t科技 \t        \t科技 \t  ├►ARG1\t科技 \tNN  ├►NP ───┘               │   \n│   └─►└┴┴──\t公司 \tdobj \tNN\t公司 \t◄─┘             \t公司 \t        \t公司 \t◄─┘     \t公司 \tNN──┘                       │   \n└──────────►\t。  \tpunct\tPU\t。  \t                \t。  \t        \t。  \t        \t。  \tPU──────────────────────────┘   \n```\n\n关于标注集含义，请参考[《语言学标注规范》](https://hanlp.hankcs.com/docs/annotations/index.html)及[《格式规范》](https://hanlp.hankcs.com/docs/data_format.html)。我们购买、标注或采用了世界上量级最大、种类最多的语料库用于联合多语种多任务学习，所以HanLP的标注集也是覆盖面最广的。\n\n## 训练你自己的领域模型\n\n写深度学习模型一点都不难，难的是复现较高的准确率。下列[代码](https://github.com/hankcs/HanLP/blob/master/plugins/hanlp_demo/hanlp_demo/zh/train_sota_bert_pku.py)展示了如何在sighan2005 PKU语料库上花6分钟训练一个超越学术界state-of-the-art的中文分词模型。\n\n```python\ntokenizer = TransformerTaggingTokenizer()\nsave_dir = 'data/model/cws/sighan2005_pku_bert_base_96.73'\ntokenizer.fit(\n    SIGHAN2005_PKU_TRAIN_ALL,\n    SIGHAN2005_PKU_TEST,  # Conventionally, no devset is used. See Tian et al. (2020).\n    save_dir,\n    'bert-base-chinese',\n    max_seq_len=300,\n    char_level=True,\n    hard_constraint=True,\n    sampler_builder=SortingSamplerBuilder(batch_size=32),\n    epochs=3,\n    adam_epsilon=1e-6,\n    warmup_steps=0.1,\n    weight_decay=0.01,\n    word_dropout=0.1,\n    seed=1660853059,\n)\ntokenizer.evaluate(SIGHAN2005_PKU_TEST, save_dir)\n```\n\n其中，由于指定了随机数种子，结果一定是`96.73`。不同于那些虚假宣传的学术论文或商业项目，HanLP保证所有结果可复现。如果你有任何质疑，我们将当作最高优先级的致命性bug第一时间排查问题。\n\n请参考[demo](https://github.com/hankcs/HanLP/tree/master/plugins/hanlp_demo/hanlp_demo/zh/train)了解更多训练脚本。\n\n## 性能\n\n<table><thead><tr><th rowspan=\"2\">lang</th><th rowspan=\"2\">corpora</th><th rowspan=\"2\">model</th><th colspan=\"2\">tok</th><th colspan=\"4\">pos</th><th colspan=\"3\">ner</th><th rowspan=\"2\">dep</th><th rowspan=\"2\">con</th><th rowspan=\"2\">srl</th><th colspan=\"4\">sdp</th><th rowspan=\"2\">lem</th><th rowspan=\"2\">fea</th><th rowspan=\"2\">amr</th></tr><tr><th>fine</th><th>coarse</th><th>ctb</th><th>pku</th><th>863</th><th>ud</th><th>pku</th><th>msra</th><th>ontonotes</th><th>SemEval16</th><th>DM</th><th>PAS</th><th>PSD</th></tr></thead><tbody><tr><td rowspan=\"2\">mul</td><td rowspan=\"2\">UD2.7<br>OntoNotes5</td><td>small</td><td>98.62</td><td>-</td><td>-</td><td>-</td><td>-</td><td>93.23</td><td>-</td><td>-</td><td>74.42</td><td>79.10</td><td>76.85</td><td>70.63</td><td>-</td><td>91.19</td><td>93.67</td><td>85.34</td><td>87.71</td><td>84.51</td><td>-</td></tr><tr><td>base</td><td>98.97</td><td>-</td><td>-</td><td>-</td><td>-</td><td>90.32</td><td>-</td><td>-</td><td>80.32</td><td>78.74</td><td>71.23</td><td>73.63</td><td>-</td><td>92.60</td><td>96.04</td><td>81.19</td><td>85.08</td><td>82.13</td><td>-</td></tr><tr><td rowspan=\"5\">zh</td><td rowspan=\"2\">open</td><td>small</td><td>97.25</td><td>-</td><td>96.66</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>95.00</td><td>84.57</td><td>87.62</td><td>73.40</td><td>84.57</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>base</td><td>97.50</td><td>-</td><td>97.07</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>96.04</td><td>87.11</td><td>89.84</td><td>77.78</td><td>87.11</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td rowspan=\"3\">close</td><td>small</td><td>96.70</td><td>95.93</td><td>96.87</td><td>97.56</td><td>95.05</td><td>-</td><td>96.22</td><td>95.74</td><td>76.79</td><td>84.44</td><td>88.13</td><td>75.81</td><td>74.28</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>base</td><td>97.52</td><td>96.44</td><td>96.99</td><td>97.59</td><td>95.29</td><td>-</td><td>96.48</td><td>95.72</td><td>77.77</td><td>85.29</td><td>88.57</td><td>76.52</td><td>73.76</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>ernie</td><td>96.95</td><td>97.29</td><td>96.76</td><td>97.64</td><td>95.22</td><td>-</td><td>97.31</td><td>96.47</td><td>77.95</td><td>85.67</td><td>89.17</td><td>78.51</td><td>74.10</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr></tbody></table>\n\n- 根据我们的[最新研究](https://aclanthology.org/2021.emnlp-main.451)，单任务学习的性能往往优于多任务学习。在乎精度甚于速度的话，建议使用[单任务模型](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/index.html)。\n\nHanLP采用的数据预处理与拆分比例与流行方法未必相同，比如HanLP采用了[完整版的MSRA命名实体识别语料](https://bbs.hankcs.com/t/topic/3033)，而非大众使用的阉割版；HanLP使用了语法覆盖更广的[Stanford Dependencies标准](https://hanlp.hankcs.com/docs/annotations/dep/sd_zh.html)，而非学术界沿用的Zhang and Clark (2008)标准；HanLP提出了[均匀分割CTB的方法](https://bbs.hankcs.com/t/topic/3024)，而不采用学术界不均匀且遗漏了51个黄金文件的方法。HanLP开源了[一整套语料预处理脚本与相应语料库](https://github.com/hankcs/HanLP/blob/master/plugins/hanlp_demo/hanlp_demo/zh/train/open_small.py)，力图推动中文NLP的透明化。\n\n总之，HanLP只做我们认为正确、先进的事情，而不一定是流行、权威的事情。\n\n## 引用\n\n如果你在研究中使用了HanLP，请按如下格式引用：\n\n```bibtex\n@inproceedings{he-choi-2021-stem,\n    title = \"The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders\",\n    author = \"He, Han and Choi, Jinho D.\",\n    booktitle = \"Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing\",\n    month = nov,\n    year = \"2021\",\n    address = \"Online and Punta Cana, Dominican Republic\",\n    publisher = \"Association for Computational Linguistics\",\n    url = \"https://aclanthology.org/2021.emnlp-main.451\",\n    pages = \"5555--5577\",\n    abstract = \"Multi-task learning with transformer encoders (MTL) has emerged as a powerful technique to improve performance on closely-related tasks for both accuracy and efficiency while a question still remains whether or not it would perform as well on tasks that are distinct in nature. We first present MTL results on five NLP tasks, POS, NER, DEP, CON, and SRL, and depict its deficiency over single-task learning. We then conduct an extensive pruning analysis to show that a certain set of attention heads get claimed by most tasks during MTL, who interfere with one another to fine-tune those heads for their own objectives. Based on this finding, we propose the Stem Cell Hypothesis to reveal the existence of attention heads naturally talented for many tasks that cannot be jointly trained to create adequate embeddings for all of those tasks. Finally, we design novel parameter-free probes to justify our hypothesis and demonstrate how attention heads are transformed across the five tasks during MTL through label analysis.\",\n}\n```\n\n## License\n\n### 源代码\n\nHanLP源代码的授权协议为 **Apache License 2.0**，可免费用做商业用途。请在产品说明中附加HanLP的链接和授权协议。HanLP受版权法保护，侵权必究。\n\n##### 自然语义（青岛）科技有限公司\n\nHanLP从v1.7版起独立运作，由自然语义（青岛）科技有限公司作为项目主体，主导后续版本的开发，并拥有后续版本的版权。\n\n##### 上海林原公司\n\nHanLP 早期得到了上海林原公司的大力支持，并拥有1.28及前序版本的版权，相关版本也曾在上海林原公司网站发布。\n\n### 预训练模型\n\n机器学习模型的授权在法律上没有定论，但本着尊重开源语料库原始授权的精神，如不特别说明，HanLP的多语种模型授权沿用[CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)，中文模型授权为仅供研究与教学使用。\n\n## References\n\nhttps://hanlp.hankcs.com/docs/references.html\n\n"
  },
  {
    "path": "docs/Makefile",
    "content": "# Minimal makefile for Sphinx documentation\n#\n\n# You can set these variables from the command line, and also\n# from the environment for the first two.\nSPHINXOPTS    ?=\nSPHINXBUILD   ?= sphinx-build\nSOURCEDIR     = .\nBUILDDIR      = _build\n\n# Put it first so that \"make\" without argument is like \"make help\".\nhelp:\n\t@$(SPHINXBUILD) -M help \"$(SOURCEDIR)\" \"$(BUILDDIR)\" $(SPHINXOPTS) $(O)\n\n.PHONY: help Makefile\n\n# Catch-all target: route all unknown targets to Sphinx using the new\n# \"make mode\" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).\n%: Makefile\n\t@$(SPHINXBUILD) -M $@ \"$(SOURCEDIR)\" \"$(BUILDDIR)\" $(SPHINXOPTS) $(O)\n"
  },
  {
    "path": "docs/annotations/constituency/ctb.md",
    "content": "<!--\n# ========================================================================\n# Copyright 2020 hankcs\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n# ========================================================================\n-->\n\n# Chinese Tree Bank\n\nSee also [The Bracketing Guidelines for the Penn Chinese Treebank (3.0)](https://repository.upenn.edu/cgi/viewcontent.cgi?article=1040&context=ircs_reports).\n\n| Tag  | Definition                             | 定义                                     | 例子            |\n|------|----------------------------------------------|----------------------------------------------------|-------------------|\n| ADJP | adjective phrase                             | 形容词短语，以形容词为中心词                                     | 不完全、大型            |\n| ADVP | adverbial phrase headed by AD (adverb)       | 副词短语，以副词为中心词                                       | 非常、很              |\n| CLP  | classifier phrase                            | 由量词构成的短语                                           | 系列、大批             |\n| CP   | clause headed by C (complementizer)          | 从句，通过带补语（如“的”、“吗”等）                                | 张三喜欢李四吗？          |\n| DNP  | phrase formed by ‘‘XP + DEG’’                | 结构为XP + DEG(的)的短语，其中XP可以是ADJP、DP、QP、PP等等，用于修饰名词短语。 | 大型的、前几年的、五年的、在上海的 |\n| DP   | determiner phrase                            | 限定词短语，通常由限定词和数量词构成                                 | 这三个、任何            |\n| DVP  | phrase formed by ‘‘XP + DEV’’                | 结构为XP+地的短评，用于修饰动词短语VP                              | 心情失落地、大批地         |\n| FRAG | fragment                                     | 片段                                                 | (完）               |\n| INTJ | interjection                                 | 插话，感叹语                                             | 哈哈、切              |\n| IP   | simple clause headed by I (INFL)             | 简单子句或句子，通常不带补语（如“的”、“吗”等）                          | 张三喜欢李四。           |\n| LCP  | phrase formed by ‘‘XP + LC’’                 | 用于表本地点+方位词（LC)的短语                                  | 生活中、田野上           |\n| LST  | list marker                                  | 列表短语，包括标点符号                                        | 一.                |\n| MSP  | some particles                               | 其他小品词                                              | 所、而、来、去           |\n| NN   | common noun                                  | 名词                                                 | HanLP、技术          |\n| NP   | noun phrase                                  | 名词短语，中心词通常为名词                                      | 美好生活、经济水平         |\n| PP   | preposition phrase                           | 介词短语，中心词通常为介词                                      | 在北京、据报道           |\n| PRN  | parenthetical                                | 插入语                                                | ，（张三说)，           |\n| QP   | quantifier phrase                            | 量词短语                                               | 三个、五百辆            |\n| TOP  | root node                                    | 根节点                                                | 根节点               |\n| UCP  | unidentical coordination phrase              | 不对称的并列短语，指并列词两侧的短语类型不致                             | (养老、医疗）保险         |\n| VCD  | coordinated verb compound                    | 复合动词                                               | 出版发行              |\n| VCP  | verb compounds formed by VV + VC             | VV + VC形式的动词短语                                     | 看作是               |\n| VNV  | verb compounds formed by A-not-A or A-one-A  | V不V形式的动词短语                                         | 能不能、信不信           |\n| VP   | verb phrase                                  | 动词短语，中心词通常为动词                                      | 完成任务、努力工作         |\n| VPT  | potential form V-de-R or V-bu-R              | V不R、V得R形式的动词短语                                     | 打不赢、打得过           |\n| VRD  | verb resultative compound                    | 动补结构短语                                             | 研制成功、降下来          |\n| VSB  | verb compounds formed by a modifier + a head | 修饰语+中心词构成的动词短语                                     | 拿来支付、仰头望去         |"
  },
  {
    "path": "docs/annotations/constituency/index.md",
    "content": "# Constituency Parsing\n\n## Chinese\n```{toctree}\nctb\n```\n\n## English\n```{toctree}\nptb\n```\n\n## Japanese\n```{toctree}\nnpcmj\n```\n\n"
  },
  {
    "path": "docs/annotations/constituency/npcmj.md",
    "content": "<!--\n# ========================================================================\n# Copyright 2020 hankcs\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n# ========================================================================\n-->\n\n# NPCMJ\n\n| Tag             | Description                             |\n|-----------------|-----------------------------------------|\n| ADVP            | adverb phrase                           |\n| ADVP-CMPL       | complement adverb phrase                |\n| ADVP-MSR        | measurement adverb phrase               |\n| ADVP-PRD        | predicate adverb phrase                 |\n| ADVP-TMP        | temporal adverb phrase                  |\n| CONJP           | conjunction phrase                      |\n| CP-EXL          | exclamative                             |\n| CP-IMP          | imperative                              |\n| CP-FINAL        | projection for sentence final particle  |\n| CP-QUE          | question (direct or indirect)           |\n| CP-QUE-ADV      | question used adverbially               |\n| CP-QUE-OB1      | question used as object                 |\n| CP-QUE-PRD      | question used as a nominal predicate    |\n| CP-QUE-SBJ      | question used as subject                |\n| CP-THT          | complementizer clause                   |\n| CP-THT-ADV      | complementizer clause used adverbially  |\n| CP-THT-OB1      | complementizer clause used as object    |\n| CP-THT-PRD      | complementizer clause used as predicate |\n| CP-THT-PRP      | purposive complementizer clause         |\n| CP-THT-SBJ      | complementizer clause used as subject   |\n| FRAG            | fragment                                |\n| FS              | false start                             |\n| INTJP           | interjection phrase                     |\n| IP-ADV          | adverbial clause                        |\n| IP-ADV-CONJ     | coordinated clause                      |\n| IP-ADV-PRD      | adverbial clause used as predicate      |\n| IP-ADV-SCON     | subordinate clause                      |\n| IP-ADV-SCON-CND |                                         |\n| conditional     | clause                                  |\n| IP-EMB          | gapless noun-modifying clause           |\n| IP-IMP          | imperative clause                       |\n| IP-MAT          | matrix clause                           |\n| IP-NMZ          | nominalized clause                      |\n| IP-NMZ-PRD      | nominalized clause used as predicate    |\n| IP-REL          | relative clause                         |\n| IP-SMC          | small clause                            |\n| IP-SMC-CNT      | small clause in continuative form       |\n| IP-SMC-OB1      | small clause used as object             |\n| IP-SMC-SBJ      | small clause used as subject            |\n| IP-SUB          | clause under CP* layer                  |\n| multi-sentence  | multiple sentence                       |\n| NML             | intermediate nominal layer              |\n| NP              | noun phrase                             |\n| NP-ADV          | adverbial noun phrase                   |\n| NP-CZZ          | causee noun phrase                      |\n| NP-DOB1         | derived primary object noun phrase      |\n| NP-DSBJ         | derived subject noun phrase             |\n| NP-LGS          | logical subject noun phrase             |\n| NP-LOC          | locational noun phrase                  |\n| NP-MSR          | measure noun phrase                     |\n| NP-OB1          | primary object noun phrase              |\n| NP-OB2          | secondary object noun phrase            |\n| NP-POS          | possessive noun phrase                  |\n| NP-PRD          | predicate noun phrase                   |\n| NP-SBJ          | subject noun phrase                     |\n| NP-SBJ2         | secondary subject noun phrase           |\n| NP-TMP          | temporal noun phrase                    |\n| NP-TPC          | topic noun phrase                       |\n| NP-VOC          | vocative noun phrase                    |\n| NUMCLP          | numeral-classifier phrase               |\n| PNLP            | prenominal phrase                       |\n| PP              | particle phrase                         |\n| PP-ADV          | adverbial particle phrase               |\n| PP-CMPL         | complement particle phrase              |\n| PP-CONJ         | coordination particle phrase            |\n| PP-CZZ          | causee particle phrase                  |\n| PP-DOB1         | derived primary object particle phrase  |\n| PP-DSBJ         | derived subject particle phrase         |\n| PP-LGS          | logical subject particle phrase         |\n| PP-LOC          | locational particle phrase              |\n| PP-MSR          | measure particle phrase                 |\n| PP-OB1          | primary object particle phrase          |\n| PP-OB2          | secondary object particle phrase        |\n| PP-PRD          | predicate particle phrase               |\n| PP-PRP          | purpositive particle phrase             |\n| PP-SBJ          | subject particle phrase                 |\n| PP-SBJ2         | secondary subject particle phrase       |\n| PP-SCON         | subordination particle phrase           |\n| PP-SCON-CND     | conditional particle phrase             |\n| PP-TMP          | temporal particle phrase                |\n| PP-TPC          | topic particle phrase                   |\n| PP-VOC          | vocative particle phrase                |\n| PRN             | parenthetical                           |"
  },
  {
    "path": "docs/annotations/constituency/ptb.md",
    "content": "<!--\n# ========================================================================\n# Copyright 2020 hankcs\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n# ========================================================================\n-->\n\n# Penn Treebank\n\n| Tag    | Description                                                                                                                                                                                                         |\n|--------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| ADJP   | Adjective Phrase.                                                                                                                                                                                                   |\n| ADVP   | Adverb Phrase.                                                                                                                                                                                                      |\n| CONJP  | Conjunction Phrase.                                                                                                                                                                                                 |\n| FRAG   | Fragment.                                                                                                                                                                                                           |\n| INTJ   | Interjection. Corresponds approximately to the part-of-speech tag UH.                                                                                                                                               |\n| LST    | List marker. Includes surrounding punctuation.                                                                                                                                                                      |\n| NAC    | Not a Constituent; used to show the scope of certain prenominal modifiers within an NP.                                                                                                                             |\n| NP     | Noun Phrase.                                                                                                                                                                                                        |\n| NX     | - Used within certain complex NPs to mark the head of the NP. Corresponds very roughly to N-bar level but used quite differently.                                                                                   |\n| PP     | Prepositional Phrase.                                                                                                                                                                                               |\n| PRN    | Parenthetical                                                                                                                                                                                                       |\n| PRT    | Particle. Category for words that should be tagged RP.                                                                                                                                                              |\n| QP     | Quantifier Phrase (i.e. complex measure/amount phrase); used within NP.                                                                                                                                             |\n| ROOT   | No description                                                                                                                                                                                                      |\n| RRC    | Reduced Relative Clause.                                                                                                                                                                                            |\n| S      | conjunction or a wh-word and that does not exhibit subject-verb inversion.                                                                                                                                          |\n| SBAR   | Clause introduced by a (possibly empty) subordinating conjunction.                                                                                                                                                  |\n| SBARQ  | - Direct question introduced by a wh-word or a wh-phrase. Indirect questions and relative clauses should be bracketed as SBAR, not SBARQ.                                                                           |\n| SINV   | - Inverted declarative sentence, i.e. one in which the subject follows the tensed verb or modal.                                                                                                                    |\n| SQ     | Inverted yes/no question, or main clause of a wh-question, following the wh-phrase in SBARQ.                                                                                                                        |\n| UCP    | Unlike Coordinated Phrase.                                                                                                                                                                                          |\n| VP     | Verb Phrase.                                                                                                                                                                                                       |\n| WHADJP | Wh-adjective Phrase. Adjectival phrase containing a wh-adverb, as in how hot.                                                                                                                                       |\n| WHADVP | - Wh-adverb Phrase. Introduces a clause with an NP gap. May be null (containing the 0 complementizer) or lexical, containing a wh-adverb such as how or why.                                                        |\n| WHNP   | - Wh-noun Phrase. Introduces a clause with an NP gap. May be null (containing the 0 complementizer) or lexical, containing some wh-word, e.g. who, which book, whose daughter, none of which, or how many leopards. |\n| WHPP   | - Wh-prepositional Phrase. Prepositional phrase containing a wh-noun phrase (such as of which or by whose authority) that either introduces a PP gap or is contained by a WHNP.                                     |\n| X      | - Unknown, uncertain, or unbracketable. X is often used for bracketing typos and in bracketing the…​the-constructions.                                                                                              |\n\n"
  },
  {
    "path": "docs/annotations/dep/index.md",
    "content": "# Dependency Parsing\n\n## Chinese\n\n```{toctree}\nsd_zh\npmt\n```\n\n## English\n\n```{toctree}\nsd_en\n```\n\n## Multilingual\n\n```{toctree}\nud\n```\n"
  },
  {
    "path": "docs/annotations/dep/pmt.md",
    "content": "<!--\n# ========================================================================\n# Copyright 2020 hankcs\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n# ========================================================================\n-->\n\n# PKU Multi-view Chinese Treebank\n\n\n```{eval-rst}\n\nSee also :cite:`qiu-etal-2014-multi`.\n    \n```\n\n| Tag  | Description                                 | 依存关系       |\n| ---- | ------------------------------------------- | -------------- |\n| ACT  | action object                               | 行为宾语       |\n| ADV  | adverbial                                   | 状语           |\n| APP  | appositive element                          | 同位           |\n| ATT  | attribute                                   | 定语           |\n| CMP  | complement                                  | 补语           |\n| COO  | other coordination element                  | 一般并列       |\n| COS  | share-right-child coordination element      | 共享并列       |\n| DE   | de (modifier of 的(special function word))  | 的字           |\n| DEI  | dei (modifier of 得(special function word)) | 得字           |\n| DI   | di (modifier of 地(special function word))  | 地字           |\n| FOC  | focus                                       | 强调           |\n| HED  | root of a sentence                          | 核心           |\n| IC   | independent clause                          | 小句           |\n| IOB  | indirect object                             | 间接宾语       |\n| IS   | independent structure                       | 独立结构       |\n| ISC  | non-shared independent structure            | 并列式独立结构 |\n| LAD  | left additive                               | 前附加         |\n| MT   | modality and time                           | 时体           |\n| NUM  | number                                      | 数字           |\n| POB  | propositional object                        | 介宾           |\n| PUN  | punctuation                                 | 标点           |\n| PUS  | cross-clause punctuation                    | 跨句标点       |\n| QUC  | post-positional quantity                    | 数量补语       |\n| QUCC | non-shared post-positional quantity         | 非共享数量补语 |\n| QUN  | quantity                                    | 数量           |\n| RAD  | right additive                              | 后附加         |\n| RADC | non-shared right additive                   | 非共享后附加   |\n| RED  | reduplicate element                         | 重叠           |\n| SBV  | subject                                     | 主语           |\n| TPC  | topic                                       | 话题           |\n| VOB  | direct object                               | 宾语           |\n| VV   | serial verb construction                    | 连动           |\n"
  },
  {
    "path": "docs/annotations/dep/sd_en.md",
    "content": "<!--\n# ========================================================================\n# Copyright 2020 hankcs\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n# ========================================================================\n-->\n\n# Stanford Dependencies English\n\nSee also [Stanford typed dependencies manual](https://nlp.stanford.edu/software/dependencies_manual.pdf).\n\n| Tag        | Description                       |\n|------------|-----------------------------------|\n| abbrev     | abbreviation modifier             |\n| acomp      | adjectival complement             |\n| advcl      | adverbial clause modifier         |\n| advmod     | adverbial modifier                |\n| agent      | agent                             |\n| amod       | adjectival modifier               |\n| appos      | appositional modifier             |\n| arg        | argument                          |\n| attr       | attributive                       |\n| aux        | auxiliary                         |\n| auxpass    | passive auxiliary                 |\n| cc         | coordination                      |\n| ccomp      | clausal complement                |\n| comp       | complement                        |\n| complm     | complementizer                    |\n| conj       | conjunct                          |\n| cop        | copula                            |\n| csubj      | clausal subject                   |\n| csubjpass  | clausal passive subject           |\n| dep        | dependent                         |\n| det        | determiner                        |\n| discourse  | discourse element                 |\n| dobj       | direct object                     |\n| expl       | expletive                         |\n| goeswith   | goes with                         |\n| iobj       | indirect object                   |\n| mark       | marker                            |\n| mod        | modifier                          |\n| mwe        | multi-word expression             |\n| neg        | negation modifier                 |\n| nn         | noun compound modifier            |\n| npadvmod   | noun phrase as adverbial modifier |\n| nsubj      | nominal subject                   |\n| nsubjpass  | passive nominal subject           |\n| num        | numeric modifier                  |\n| number     | element of compound number        |\n| obj        | object                            |\n| parataxis  | parataxis                         |\n| pcomp      | prepositional complement          |\n| pobj       | object of a preposition           |\n| poss       | possession modifier               |\n| possessive | possessive modifier               |\n| preconj    | preconjunct                       |\n| pred       | predicate                         |\n| predet     | predeterminer                     |\n| prep       | prepositional modifier            |\n| prepc      | prepositional clausal modifier    |\n| prt        | phrasal verb particle             |\n| punct      | punctuation                       |\n| purpcl     | purpose clause modifier           |\n| quantmod   | quantifier phrase modifier        |\n| rcmod      | relative clause modifier          |\n| ref        | referent                          |\n| rel        | relative                          |\n| root       | root                              |\n| sdep       | semantic dependent                |\n| subj       | subject                           |\n| tmod       | temporal modifier                 |\n| vmod       | verb modifier                     |\n| xcomp      | open clausal complement           |\n| xsubj      | controlling subject               |"
  },
  {
    "path": "docs/annotations/dep/sd_zh.md",
    "content": "<!--\n# ========================================================================\n# Copyright 2020 hankcs\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n# ========================================================================\n-->\n\n# Stanford Dependencies Chinese\n\n\n```{eval-rst}\n\nSee also :cite:`chang-etal-2009-discriminative`.\n    \n```\n\n|Tag|Description|中文简称|例句|依存弧|\n| ---- | ---- | ---- | ---- | ---- |\n|nn|noun compound modifier|复合名词修饰|服务中心|nn(中心，服务）|\n|punct|punctuation|标点符号|海关统计表明，|punct(表明，，)|\n|nsubj|nominal subject|名词性主语|梅花盛开|nsubj (盛开，梅花）|\n|conj|conjunct (links two conjuncts)|连接性状语|设备和原材料|conj(原材料，设备）|\n|dobj|direct object|直接宾语|浦东颁布了七十一件文件|dobj(颁布，文件）|\n|advmod|adverbial modifier|副词性状语|部门先送上文件|advmod(送上，先）|\n|prep|prepositional modifier|介词性修饰语|在实践中逐步完善|prep(完善，在）|\n|nummod|number modifier|数词修饰语|七十一件文件|nummod(件，七十一）|\n|amod|adjectival modifier|形容词修饰语|跨世纪工程|amod(工程，跨世纪）|\n|pobj|prepositional object|介词性宾语|根据有关规定|pobj (根据，规定）|\n|rcmod|relative clause modifier|关系从句修饰语|不曾遇到过的情况|rcmod(情况，遇到）|\n|cpm|complementizer|补语|开发浦东的经济活动|cpm(开发，的）|\n|assm|associative marker|关联标记|企业的商品|assm(企业，的）|\n|assmod|associative modifier|关联修饰|企业的商品|assmod(商品，企业）|\n|cc|coordinating conjunction|并列关系|设备和原材料|cc(原材料，和）|\n|clf|classifier modifier|类别修饰|七十一件文件|clf(文件，件）|\n|ccomp|clausal complement|从句补充|银行决定先取得信用评级|ccomp(决定，取得）|\n|det|determiner|限定语|这些经济活动|det(活动，这些）|\n|lobj|localizer object|范围宾语|近年来|lobj(来，近年）|\n|range|dative object that is a quantifier phrase|数量词间接宾语|成交药品一亿多元|range(成交，元）|\n|asp|aspect marker|时态标记|发挥了作用|asp(发挥，了）|\n|tmod|temporal modifier|时间修饰语|以前不曾遇到过|tmod(遇到，以前）|\n|plmod|localizer modifier of a preposition|介词性地点修饰|在这片热土上|plmod(在，上）|\n|attr|attributive|属性|贸易额为二百亿美元|attr(为，美元）|\n|mmod|modal verb modifier|情态动词|利益能得到保障|mmod(得到，能）|\n|loc|localizer|位置补语|占九成以上|loc(占，以上）|\n|top|topic|主题|建筑是主要活动|top(是，建筑）|\n|pccomp|clausal complement of a preposition|介词补语|据有关部门介绍|pccomp(据，介绍）|\n|etc|etc modifier|省略关系|科技、文教等领域|etc(文教，等）|\n|lccomp|clausal complement of a localizer|位置补语|中国对外开放中升起的明星|lccomp(中，开放）|\n|ordmod|ordinal number modifier|量词修饰|第七个机构|ordmod(个，第七）|\n|xsubj|controlling subject|控制主语|银行决定先取得信用评级|xsubj (取得，银行）|\n|neg|negative modifier|否定修饰|以前不曾遇到过|neg(遇到，不）|\n|rcomp|resultative complement|结果补语|研究成功|rcomp(研究，成功）|\n|comod|coordinated verb compound modifier|并列联合动词|颁布实行|comod(颁布，实行）|\n|vmod|verb modifier|动词修饰|其在支持外商企业方面的作用|vmod(方面，支持）|\n|prtmod|particles such as 所，以，来，而|小品词|在产业化所取得的成就|prtmod(取得，所）|\n|ba|“ba” construction|把字关系|把注意力转向市场|ba(转向，把）|\n|dvpm|manner DE(地）modifier|地字修饰|有效地防止流失|dvpm(有效，地）|\n|dvpmod|a \"XP+DEV\", phrase that modifies VP|地字动词短语|有效地防止流失|dvpmod(防止，有效）|\n|prnmod|parenthetical modifier|插入词修饰|八五期间（1990-1995 )|pmmod(期间，1995)|\n|cop|copular|系动词|原是自给自足的经济|cop(自给自足，是）|\n|pass|passive marker|被动标记|被认定为高技术产业|pass(认定，被）|\n|nsubjpass|nominal passive subject|被动名词主语|镍被称作现代工业的维生素|nsubjpass(称作，镍）|\n|dep|dependent|其他依赖关系|新华社北京二月十二日电|dep(电，新华社）|\n"
  },
  {
    "path": "docs/annotations/dep/ud.md",
    "content": "<!--\n# ========================================================================\n# Copyright 2020 hankcs\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n# ========================================================================\n-->\n\n# Universal Dependencies\n\n## Cross-Linguistic\n\nSee also [Universal Dependencies](https://universaldependencies.org/docs/u/dep/index.html).\n\n| Tag        | Description                                  |\n|------------|----------------------------------------------|\n| acl        | clausal modifier of noun (adjectival clause) |\n| advcl      | adverbial clause modifier                    |\n| advmod     | adverbial modifier                           |\n| amod       | adjectival modifier                          |\n| appos      | appositional modifier                        |\n| aux        | auxiliary                                    |\n| auxpass    | passive auxiliary                            |\n| case       | case marking                                 |\n| cc         | coordinating conjunction                     |\n| ccomp      | clausal complement                           |\n| compound   | compound                                     |\n| conj       | conjunct                                     |\n| cop        | copula                                       |\n| csubj      | clausal subject                              |\n| csubjpass  | clausal passive subject                      |\n| dep        | unspecified dependency                       |\n| det        | determiner                                   |\n| discourse  | discourse element                            |\n| dislocated | dislocated elements                          |\n| dobj       | direct object                                |\n| expl       | expletive                                    |\n| foreign    | foreign words                                |\n| goeswith   | goes with                                    |\n| iobj       | indirect object                              |\n| list       | list                                         |\n| mark       | marker                                       |\n| mwe        | multi-word expression                        |\n| name       | name                                         |\n| neg        | negation modifier                            |\n| nmod       | nominal modifier                             |\n| nsubj      | nominal subject                              |\n| nsubjpass  | passive nominal subject                      |\n| nummod     | numeric modifier                             |\n| parataxis  | parataxis                                    |\n| punct      | punctuation                                  |\n| remnant    | remnant in ellipsis                          |\n| reparandum | overridden disfluency                        |\n| root       | root                                         |\n| vocative   | vocative                                     |\n| xcomp      | open clausal complement                      |\n\n\n## Localization\n\n### Chinese\n\n| Tag              |       简称 |                                                         例句 |\n| :--------------- |---------:| -----------------------------------------------------------: |\n| acl              |    形容词子句 | ![acl](https://file.hankcs.com/img/ud/1303b5cbe9413044cb800b3c3514b70b.svg) |\n| advcl:loc        |  状语从句修饰语 | ![advcl:loc](https://file.hankcs.com/img/ud/e8865563caf0eda7a80043eda8cc43a6.svg) |\n| advmod           |       状语 | ![advmod](https://file.hankcs.com/img/ud/3ce9276f4e18d92edb48e58956bbaee7.svg) |\n| advmod:dvp       |     状语:地 | ![advmod:dvp](https://file.hankcs.com/img/ud/e90870682b9f0a80736d25977565f96a.svg) |\n| advmod:loc       |    状语:限定 | ![advmod:loc](https://file.hankcs.com/img/ud/135e9143e73e5f45290d204d4ad5b30e.svg) |\n| advmod:rcomp     |    状语:因果 | ![advmod:rcomp](https://file.hankcs.com/img/ud/aa75be342648bed0846f54a88f71e7a7.svg) |\n| amod             |       形容 | ![amod](https://file.hankcs.com/img/ud/dee0097c244c1bd0a1d1ed117932346d.svg) |\n| amod:ordmod      |    形容:数量 | ![amod:ordmod](https://file.hankcs.com/img/ud/8bb79245311a4190836dce8439591e91.svg) |\n| appos            |       同位 | ![appos](https://file.hankcs.com/img/ud/a74f6a31f68ba5697d0a8906e8476b47.svg) |\n| aux:asp          |    助语:时态 | ![aux:asp](https://file.hankcs.com/img/ud/8c32de9b4858c0e4d24ee6da5fb80a6e.svg) |\n| aux:ba           |     助语:把 | ![aux:ba](https://file.hankcs.com/img/ud/2c712e3af49fcdbd5914398895904f3c.svg) |\n| aux:modal        |    助语:情态 | ![aux:modal](https://file.hankcs.com/img/ud/606946c569e4bfbacbb1b9e13336e247.svg) |\n| aux:prtmod       |    助语:分词 | ![aux:prtmod](https://file.hankcs.com/img/ud/fc49d338487dd63687941433a0633f5d.svg) |\n| auxpass          |       被动 | ![auxpass](https://file.hankcs.com/img/ud/a6e4a8aabb7bb1bb5c4e9cdf7876e3f7.svg) |\n| case             |       条件 | ![case](https://file.hankcs.com/img/ud/35a021e15a9355880cb8720ba34ed936.svg) |\n| cc               |     并列连词 | ![cc](https://file.hankcs.com/img/ud/18c6a22520cec2ba60ce636bb410f651.svg) |\n| ccomp            |     从句补语 | ![ccomp](https://file.hankcs.com/img/ud/8cc4ea0c6a090f1ba03d02926240c35b.svg) |\n| compound:nn      |     复合名词 | ![compound:nn](https://file.hankcs.com/img/ud/587e12141aa42aa9862ea0ac0eb30e09.svg) |\n| compound:vc      |     复合动词 | ![compound:vc](https://file.hankcs.com/img/ud/f72cedcb6cec8563d88063b118544a9d.svg) |\n| conj             |       连接 | ![conj](https://file.hankcs.com/img/ud/fc924f495d1d5a3a828a0e2262da06cd.svg) |\n| cop              |       系动 | ![cop](https://file.hankcs.com/img/ud/a7da58f57adbe9e6bd166ecb514f2d1c.svg) |\n| csubj            |     从句主语 | ![csubj](https://file.hankcs.com/img/ud/0adda481e81b3765ed7f4f9d55c153c4.svg) |\n| dep              |      未定义 | ![dep](https://file.hankcs.com/img/ud/db15b792f1bfd5e42982832b04c65a79.svg) |\n| det              |       限定 | ![det](https://file.hankcs.com/img/ud/17376d13a4e7b0677cd18d13e0990dab.svg) |\n| discourse        |       语气 | ![discourse](https://file.hankcs.com/img/ud/d7eb37d5fd13462b237140a08f0ed9a4.svg) |\n| dobj             |     直接宾语 | ![dobj](https://file.hankcs.com/img/ud/f5e801103ddc57a9aeff0e272b8f7b44.svg) |\n| etc              |       省略 | ![etc](https://file.hankcs.com/img/ud/86d3fd24cae9f585b7730119edaa0248.svg) |\n| mark             |       标记 | ![mark](https://file.hankcs.com/img/ud/b17b4027ab368c76a3b6f085d5b561d9.svg) |\n| mark:clf         |    标记:量词 | ![mark:clf](https://file.hankcs.com/img/ud/5974c92e3587aa64ba1d572243b9c5cc.svg) |\n| name             |       名称 | ![name](https://file.hankcs.com/img/ud/63ea082457dfe6f4fc04f635a8c019f3.svg) |\n| neg              |       否定 | ![neg](https://file.hankcs.com/img/ud/e38814231ff9a31dcce5672556375c94.svg) |\n| nmod             |     名词修饰 | ![nmod](https://file.hankcs.com/img/ud/e948a8dbcd43984d14c257f0ace1753d.svg) |\n| nmod:assmod      |  名词修饰:关联 | ![nmod:assmod](https://file.hankcs.com/img/ud/76349f30cef2c4978a03118d65ac6c81.svg) |\n| nmod:poss        | 名词修饰:所有格 | ![nmod:poss](https://file.hankcs.com/img/ud/5b4937dbea42cdff7054e9dd0904bedb.svg) |\n| nmod:prep        |  名词修饰:介词 | ![nmod:prep](https://file.hankcs.com/img/ud/63b92981638b758681a82e9f4a9aa04c.svg) |\n| nmod:range       |  名词修饰:范围 | ![nmod:range](https://file.hankcs.com/img/ud/217ec98756cfe3750c76f5e5e89b7f54.svg) |\n| nmod:tmod        |  名词修饰:时间 | ![nmod:tmod](https://file.hankcs.com/img/ud/166e3b8fb72db52f0ec332d444ea017f.svg) |\n| nmod:topic       |  名词修饰:主题 | ![nmod:topic](https://file.hankcs.com/img/ud/93c83c98c188b131211ac5e9ff5242c0.svg) |\n| nsubj            |     名词主语 | ![nsubj](https://file.hankcs.com/img/ud/63e3902d4a3045d1d696a0c4ed203563.svg) |\n| nsubj:xsubj      | 名词主语: 补语 | ![nsubj:xsubj](https://file.hankcs.com/img/ud/80cb355b9f9732fd888186a1f658b0ac.svg) |\n| nsubjpass        |    被动态主语 | ![nsubjpass](https://file.hankcs.com/img/ud/6327fab58ab42d5a417b2e5c7018ac3a.svg) |\n| nummod           |       数量 | ![nummod](https://file.hankcs.com/img/ud/0fd20559645265c2c937f06631aa74df.svg) |\n| parataxis:prnmod |       并列 | ![parataxis:prnmod](https://file.hankcs.com/img/ud/783a0faf4cd935bb61f5d225a388b79e.svg) |\n| punct            |     标点符号 | ![punct](https://file.hankcs.com/img/ud/983410055658352080ae476a5d85e6b5.svg) |\n| root             |        根 | ![root](https://file.hankcs.com/img/ud/588101bec0440ffb769172f8b7e9f98e.svg) |\n| xcomp            |     从句补语 | ![xcomp](https://file.hankcs.com/img/ud/c72071875f1c01e51acb9e1ec4893113.svg) |\n"
  },
  {
    "path": "docs/annotations/index.md",
    "content": "# Annotations\n\n\n```{toctree}\ntok/index\npos/index\nner/index\ndep/index\nsdp/index\nsrl/index\nconstituency/index\n```\n\n"
  },
  {
    "path": "docs/annotations/ner/index.md",
    "content": "# Named Entity Recognition\n\n## Chinese\n\n```{toctree}\npku\nmsra\n```\n\n## Multilingual\n\n```{toctree}\nontonotes\n```\n"
  },
  {
    "path": "docs/annotations/ner/msra.md",
    "content": "<!--\n# ========================================================================\n# Copyright 2020 hankcs\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n# ========================================================================\n-->\n\n# msra\n\n| Category | Subcategory    | Tag-set of Format-1 | Tag-set of Format-2 |\n|----------|----------------|---------------------|---------------------|\n| NAMEX    | Person         | P                   | PERSON              |\n|          | Location       | L                   | LOCATION            |\n|          | Organization   | 〇                   | ORGANIZATION        |\n| TIMEX    | Date           | dat                 | DATE                |\n|          | Duration       | dur                 | DURATION            |\n|          | Time           | tim                 | TIME                |\n| NUMEX    | Percent        | per                 | PERCENT             |\n|          | Money          | mon                 | MONEY               |\n|          | Frequency      | fre                 | FREQUENCY           |\n|          | Integer        | int                 | INTEGER             |\n|          | Fraction       | fra                 | FRACTION            |\n|          | Decimal        | dec                 | DECIMAL             |\n|          | Ordinal        | ord                 | ORDINAL             |\n|          | Rate           | rat                 | RATE                |\n| MEASUREX | Age            | age                 | AGE                 |\n|          | Weight         | wei                 | WEIGHT              |\n|          | Length         | len                 | LENGTH              |\n|          | Temperature    | tem                 | TEMPERATURE         |\n|          | Angle          | ang                 | ANGLE               |\n|          | Area           | are                 | AREA                |\n|          | Capacity       | cap                 | CAPACITY            |\n|          | Speed          | spe                 | SPEED               |\n|          | Acceleration   | acc                 | ACCELERATION        |\n|          | Other measures | mea                 | MEASURE             |\n| ADDREX   | Email          | ema                 | EMAIL               |\n|          | Phone          | pho                 | PHONE               |\n|          | Fax            | fax                 | FAX                 |\n|          | Telex          | tel                 | TELEX               |\n|          | WWW            | WWW                 | WWW                 |\n|          | Postalcode     | pos                 | POSTALCODE          |\n\n"
  },
  {
    "path": "docs/annotations/ner/ontonotes.md",
    "content": "<!--\n# ========================================================================\n# Copyright 2020 hankcs\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n# ========================================================================\n-->\n\n# ontonotes\n\n| TAG       | Description                        |\n|--------------|------------------------------------------------------|\n| PERSON       | People, including fictional                          |\n| NORP         | Nationalities or religious or political groups       |\n| FACILITY     | Buildings, airports, highways, bridges, etc.         |\n| ORGANIZATION | Companies, agencies, institutions, etc.              |\n| GPE          | Countries, cities, states                            |\n| LOCATION     | Non-GPE locations, mountain ranges, bodies of water  |\n| PRODUCT      | Vehicles, weapons, foods, etc. (Not services)        |\n| EVENT        | Named hurricanes, battles, wars, sports events, etc. |\n| WORK OF ART  | Titles of books, songs, etc.                         |\n| LAW          | Named documents made into laws                       |\n| DATE     | Absolute or relative dates or periods        |\n| TIME     | Times smaller than a day                     |\n| PERCENT  | Percentage                        |\n| MONEY    | Monetary values, including unit              |\n| QUANTITY | Measurements, as of weight or distance       |\n| ORDINAL  | “first”, “second”                             |\n| CARDINAL | Numerals that do not fall under another type |\n\n"
  },
  {
    "path": "docs/annotations/ner/pku.md",
    "content": "<!--\n# ========================================================================\n# Copyright 2020 hankcs\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n# ========================================================================\n-->\n\n# pku\n\n| 序号 | 词性 | 名称     | 帮助记忆的诠释                                         | 例子及注解                                                   |\n| ---- | ---- | -------- | ------------------------------------------------------ | ------------------------------------------------------------ |\n| 1   | nr   | 人名     | 名词代码n和“人(ren)”的声母并在一起。                   | 1. 汉族人及与汉族起名方式相同的非汉族人的姓和名单独切分，并分别标注为nr。张/nr 仁伟/nr， 欧阳/nr 修/nr， 阮/nr 志雄/nr， 朴/nr 贞爱/nr汉族人除有单姓和复姓外，还有双姓，即有的女子出嫁后，在原来的姓上加上丈夫的姓。如：陈方安生。这种情况切分、标注为：陈/nr 方/nr 安生/nr；唐姜氏，切分、标注为：唐/nr 姜氏/nr。2. 姓名后的职务、职称或称呼要分开。江/nr 主席/n， 小平/nr 同志/n， 江/nr 总书记/n，张/nr 教授/n， 王/nr 部长/n， 陈/nr 老总/n， 李/nr 大娘/n， 刘/nr 阿姨/n， 龙/nr 姑姑/n3. 对人的简称、尊称等若为两个字，则合为一个切分单位，并标以nr。老张/nr， 大李/nr， 小郝/nr， 郭老/nr， 陈总/nr4. 明显带排行的亲属称谓要切分开，分不清楚的则不切开。三/m 哥/n， 大婶/n， 大/a 女儿/n， 大哥/n， 小弟/n， 老爸/n5. 一些著名作者的或不易区分姓和名的笔名通常作为一个切分单位。鲁迅/nr， 茅盾/nr， 巴金/nr， 三毛/nr， 琼瑶/nr， 白桦/nr6. 外国人或少数民族的译名（包括日本人的姓名）不予切分，标注为nr。克林顿/nr， 叶利钦/nr， 才旦卓玛/nr， 小林多喜二/nr， 北研二/nr，华盛顿/nr， 爱因斯坦/nr有些西方人的姓名中有小圆点，也不分开。卡尔·马克思/nr |\n| 2   | ns   | 地名     | 名词代码n和处所词代码s并在一起。                       | 安徽/ns，深圳/ns，杭州/ns，拉萨/ns，哈尔滨/ns， 呼和浩特/ns， 乌鲁木齐/ns，长江/ns，黄海/ns，太平洋/ns， 泰山/ns， 华山/ns，亚洲/ns， 海南岛/ns，太湖/ns，白洋淀/ns， 俄罗斯/ns，哈萨克斯坦/ns，彼得堡/ns， 伏尔加格勒/ns 1. 国名不论长短，作为一个切分单位。中国/ns， 中华人民共和国/ns， 日本国/ns， 美利坚合众国/ns， 美国/ns2. 地名后有“省”、“市”、“县”、“区”、“乡”、“镇”、“村”、“旗”、“州”、“都”、“府”、“道”等单字的行政区划名称时，不切分开，作为一个切分单位。四川省/ns， 天津市/ns，景德镇/ns沙市市/ns， 牡丹江市/ns，正定县/ns，海淀区/ns， 通州区/ns，东升乡/ns， 双桥镇/ns 南化村/ns，华盛顿州/ns，俄亥俄州/ns，东京都/ns， 大阪府/ns，北海道/ns， 长野县/ns，开封府/ns，宣城县/ns3. 地名后的行政区划有两个以上的汉字，则将地名同行政区划名称切开，不过要将地名同行政区划名称用方括号括起来，并标以短语NS。[芜湖/ns 专区/n] NS，[宣城/ns 地区/n]ns，[内蒙古/ns 自治区/n]NS，[深圳/ns 特区/n]NS， [厦门/ns 经济/n 特区/n]NS， [香港/ns 特别/a 行政区/n]NS，[香港/ns 特区/n]NS， [华盛顿/ns 特区/n]NS，4. 地名后有表示地形地貌的一个字的普通名词，如“江、河、山、洋、海、岛、峰、湖”等，不予切分。鸭绿江/ns，亚马逊河/ns， 喜马拉雅山/ns， 珠穆朗玛峰/ns，地中海/ns，大西洋/ns，洞庭湖/ns， 塞普路斯岛/ns 5. 地名后接的表示地形地貌的普通名词若有两个以上汉字，则应切开。然后将地名同该普通名词标成短语NS。[台湾/ns 海峡/n]NS，[华北/ns 平原/n]NS，[帕米尔/ns 高原/n]NS， [南沙/ns 群岛/n]NS，[京东/ns 大/a 峡谷/n]NS [横断/b 山脉/n]NS6．地名后有表示自然区划的一个字的普通名词，如“ 街，路，道，巷，里，町，庄，村，弄，堡”等，不予切分。 中关村/ns，长安街/ns，学院路/ns， 景德镇/ns， 吴家堡/ns， 庞各庄/ns， 三元里/ns，彼得堡/ns， 北菜市巷/ns， 7．地名后接的表示自然区划的普通名词若有两个以上汉字，则应切开。然后将地名同自然区划名词标成短语NS。[米市/ns 大街/n]NS， [蒋家/nz 胡同/n]NS ， [陶然亭/ns 公园/n]NS ， 8． 大小地名相连时的标注方式为：北京市/ns 海淀区/ns 海淀镇/ns [南/f 大街/n]NS [蒋家/nz 胡同/n]NS 24/m 号/q ， |\n| 3   | nt   | 机构团体 | “团”的声母为t，名词代码n和t并在一起。                  | （参见2。短语标记说明--NT）联合国/nt，中共中央/nt，国务院/nt， 北京大学/nt1．大多数团体、机构、组织的专有名称一般是短语型的，较长，且含有地名或人名等专名，再组合，标注为短语NT。[中国/ns 计算机/n 学会/n]NT， [香港/ns 钟表业/n 总会/n]NT， [烟台/ns 大学/n]NT， [香港/ns 理工大学/n]NT， [华东/ns 理工大学/n]NT，[合肥/ns 师范/n 学院/n]NT， [北京/ns 图书馆/n]NT， [富士通/nz 株式会社/n]NT， [香山/ns 植物园/n]NT， [安娜/nz 美容院/n]NT，[上海/ns 手表/n 厂/n]NT， [永和/nz 烧饼铺/n]NT，[北京/ns 国安/nz 队/n]NT，2. 对于在国际或中国范围内的知名的唯一的团体、机构、组织的名称即使前面没有专名，也标为nt或NT。联合国/nt，国务院/nt，外交部/nt， 财政部/nt，教育部/nt， 国防部/nt，[世界/n 贸易/n 组织/n]NT， [国家/n 教育/vn 委员会/n]NT，[信息/n 产业/n 部/n]NT，[全国/n 信息/n 技术/n 标准化/vn 委员会/n]NT，[全国/n 总/b 工会/n]NT，[全国/n 人民/n 代表/n 大会/n]NT，美国的“国务院”，其他国家的“外交部、财政部、教育部”，必须在其所属国的国名之后出现时，才联合标注为NT。[美国/ns 国务院/n]NT，[法国/ns 外交部/n]NT，[美/j 国会/n]NT，日本有些政府机构名称很特别，无论是否出现在“日本”国名之后都标为nt。[日本/ns 外务省/nt]NT，[日/j 通产省/nt]NT通产省/nt 3. 前后相连有上下位关系的团体机构组织名称的处理方式如下:[联合国/nt 教科文/j 组织/n]NT， [中国/ns 银行/n 北京/ns 分行/n]NT，[河北省/ns 正定县/ns 西平乐乡/ns 南化村/ns 党支部/n]NT， 当下位名称含有专名（如“北京/ns 分行/n”、“南化村/ns 党支部/n”、“昌平/ns 分校/n”）时，也可脱离前面的上位名称单独标注为NT。[中国/ns 银行/n]NT [北京/ns 分行/n]NT，北京大学/nt [昌平/ns 分校/n]NT，4. 团体、机构、组织名称中用圆括号加注简称时:[宝山/ns 钢铁/n （/w 宝钢/j ）/w 总/b 公司/n]NT，[宝山/ns 钢铁/n 总/b 公司/n]NT，（/w 宝钢/j ）/w |"
  },
  {
    "path": "docs/annotations/pos/863.md",
    "content": "<!--\n# ========================================================================\n# Copyright 2020 hankcs\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n# ========================================================================\n-->\n\n# 863\n\n| 词性  |   名称   |              说明              |                                                                                                                                                                                                                                    例子                                                                                                                                                                                                                                    |\n| :-- | -----: | ---------------------------: | -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |\n| a   |    形容词 |        取英语形容词adjective的第1个字母 |                                                                                                                                                                                                                                                                                                                                                                                                                                         [重要/a 步伐/n]NP ，美丽/a ，看似/v 抽象/a ， |\n| c   |     连词 |      取英语连词conjunction的第1个字母。 |                                                                                                                                                                                                                                                                                                                                                                                                                                                           合作/vn 与/c 伙伴/n |\n| d   |     副词 | 取adverb的第2个字母，因其第1个字母已用于形容词。 |                                                                                                                                                                                                                                                                                                                                                                                                                                                             进一步/d 发展/v ， |\n| e   |     叹词 |      取英语叹词exclamation的第1个字母。 |                                                                                                                                                                                                                                                                                                                                                                                                                                             啊/e ，/w 那/r 金灿灿/z 的/u 麦穗/n ， |\n| f   |    方位词 |                      取汉字“方”。 |                                                                                                                                                                                                                                                                                                                                                                                                                                    军人/n 的/u 眼睛/n 里/f 不/d 是/v 没有/v 风景/n ， |\n| g   |    语素字 |                              |                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |\n| h   |   前接成分 |               取英语head的第1个字母。 |                                                                                                                                                                                                                                                                                                                                                                                                          许多/m 非/h 主角/n 人物/n ，办事处/n 的/u “/w 准/h 政府/n ”/w 功能/n 不断/d 加强/v ， |\n| i   |     成语 |            取英语成语idiom的第1个字母。 |                                                                                                                                                                                                                                                                                                                                                                                                                                                         一言一行/i ，义无反顾/i ， |\n| j   |   简称略语 |                   取汉字“简”的声母。 |                                                                                                                                                                                                                                                                                                                                                                                                                                                     [德/j 外长/n]NP ，文教/j ， |\n| k   |   后接成分 |                        后接成分。 |                                                                                                                                                                                                                                                                                                                                                                                                                                         少年儿童/l 朋友/n 们/k ，身体/n 健康/a 者/k ， |\n| m   |     数词 |    取英语numeral的第3个字母，n，u已有他用。 | 1．数量词组应切分为数词和量词。 三/m 个/q， 10/m 公斤/q， 一/m 盒/q 点心/n ，但少数数量词已是词典的登录单位，则不再切分。 一个/m ， 一些/m ，2. 基数、序数、小数、分数、百分数一律不予切分，为一个切分单位，标注为 m 。一百二十三/m，20万/m， 123.54/m， 一个/m， 第一/m， 第三十五/m， 20%/m， 三分之二/m， 千分之三十/m， 几十/m 人/n， 十几万/m 元/q， 第一百零一/m 个/q ，3. 约数，前加副词、形容词或后加“来、多、左右”等助数词的应予分开。约/d 一百/m 多/m 万/m，仅/d 一百/m 个/q， 四十/m 来/m 个/q，二十/m 余/m 只/q， 十几/m 个/q，三十/m 左右/m ，两个数词相连的及“成百”、“上千”等则不予切分。五六/m 年/q， 七八/m 天/q，十七八/m 岁/q， 成百/m 学生/n，上千/m 人/n， 4．表序关系的“数＋名”结构，应予切分。二/m 连/n ，　三/m 部/n ， |\n| mq  |    数量词 |                              |                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |\n| n   |     名词 |             取英语名词noun的第1个字母。 |                                                                                                                                                                                                                                                                                                                                                                                                                        （参见 动词--v）岗位/n ， 城市/n ， 机会/n ，她/r 是/v 责任/n 编辑/n ， |\n| nd  |   方位名词 |           方位名词（nd），表示位置的相对方向 |                                                                                                                                                                                                                                                                                                                                                                                                                  上  下  左  右  前  后  里  外  中  东  西  南  北前边  左面  里头  中间  外部 |\n| nh  |     人名 |           人名（nh），表示人的名称的专有名词 |                                                                                                                                                                                                                                                                                                                                                                                                                                        华罗庚  阿凡提  诸葛亮  司马相如  松赞干布  卡尔·马克思 |\n| nhf |      姓 |                              |                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |\n| nhs |      名 |                              |                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |\n| ni  |    机构名 |    机构名（ni），表示团体、组织、机构名称的专有名词 |                                                                                                                                                                                                                                                                                                                                                                                                                                                    联合国  教育部  北京大学  中国科学院 |\n| nl  |   处所名词 |                处所名词（nl），表示处所 |                                                                                                                                                                                                                                                                                                                                                                                                                                           空中  高处  隔壁  门口  附近  边疆  一旁  野外 |\n| ns  |     地名 |         地名（ns），表示地理区域名称的专有名词 |                                                                                                                                                                                                                                                                                                                                                                                                                       亚洲  大西洋  地中海  阿尔卑斯山  加拿大中国  北京  浙江  景德镇  呼和浩特  中关村 |\n| nt  |   时间名词 |          时间名词（nt），包括一般所说的时量词 |                                                                                                                                                                                                                                                                                                                                                                                                                                 年  月  日  分  秒现在  过去  昨天  去年  将来  宋朝  星期一 |\n| nz  | 其他专有名词 |                   其他专有名词（nz） |                                                                                                                                                                                                                                                                                                                                                                                                                                                           五粮液  宫爆鸡丁  桑塔纳 |\n| o   |    拟声词 |    取英语拟声词onomatopoeia的第1个字母。 |                                                                                                                                                                                                                                                                                                                                                                                                                                          哈哈/o 一/m 笑/v ，装载机/n 隆隆/o 推进/v ， |\n| p   |     介词 |    取英语介词prepositional的第1个字母。 |                                                                                                                                                                                                                                                                                                                                                                                对/p 子孙后代/n 负责/v ，以/p 煤/n 养/v 农/Ng ，为/p 治理/v 荒山/n 服务/v ， 把/p 青年/n 推/v 上/v 了/u 领导/vn 岗位/n ， |\n| q   |     量词 |           取英语quantity的第1个字母。 |                                                                                                                                                                                                                                                                                                                                                                                                                                                （参见数词m）首/m 批/q ，一/m 年/q ， |\n| r   |     代词 |  取英语代词pronoun的第2个字母，因p已用于介词。 |                                                                                                                                                                                                                                                                                                                                                                           单音节代词“本”、“每”、“各”、“诸”后接单音节名词时，和后接的单音节名词合为代词；当后接双音节名词时，应予切分。本报/r， 每人/r， 本社/r， 本/r 地区/n， 各/r 部门/n |\n| u   |     助词 |              取英语助词auxiliary。 |                                                                                                                                                                                                                                                                                                                                                                   [[俄罗斯/ns 和/c 北约/j]NP-BL 之间/f [战略/n 伙伴/n 关系/n]NP 的/u 建立/vn]NP 填平/v 了/u [[欧洲/ns 安全/a 政治/n]NP 的/u 鸿沟/n]NP |\n| v   |     动词 |             取英语动词verb的第一个字母。 |                                                                                                                                                                                                                                                                                                                （参见 名词--n）[[[欧盟/j 扩大/v]S 的/u [历史性/n 决定/n]NP]NP 和/c [北约/j 开放/v]S]NP-BL [为/p [创建/v [一/m 种/q 新/a 的/u 欧洲/ns 安全/a 格局/n]NP]VP-SBI]PP-MD [奠定/v 了/u 基础/n]V-SBI ，， |\n| vd  |   趋向动词 |                趋向动词（vd），表示趋向 |                                                                                                                                                                                                                                                                                                                                                                                                                      （走）上   （趴）下   （进）来   （回）去（跑）上来  （掉）下去  （提）起来  （扔）过去 |\n| vl  |   联系动词 |             联系动词（vl），表示关系的判断 |                                                                                                                                                                                                                                                                                                                                                                                                                                                                        是 |\n| vu  |   能愿动词 |             能愿动词（vu），表示可能、意愿 |                                                                                                                                                                                                                                                                                                                                                                                                                                             能够  能  应该  可以  可能  情愿  愿意  要 |\n| w   |   标点符号 |                              |                                                                                                                                                                                                                                                                                                                                                                                                                                                                  ”/w ：/w |\n| ws  | 非汉字字符串 |                非汉字字符串（ws），如： |                                                                                                                                                                                                                                                                                                                                                                                                                                                    HanLP office  windows |\n| x   |   非语素字 |  非语素字只是一个符号，字母x通常用于代表未知数、符号。 | \n"
  },
  {
    "path": "docs/annotations/pos/ctb.md",
    "content": "<!--\n# ========================================================================\n# Copyright 2020 hankcs\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n# ========================================================================\n-->\n\n# ctb\n\n See also [The Part-Of-Speech Tagging Guidelines for the Penn Chinese Treebank (3.0)](https://repository.upenn.edu/cgi/viewcontent.cgi?article=1039&context=ircs_reports).\n\n| Tag  | Description                                          | Chinese | Chinese Description                                                      | Examples              |\n|-----|------------------------------------------------------|---------|---------------------------------------------------------|------------------------|\n| AD  | adverb                                               | 副词      | 副词                                                      | 仍然、很、大大、约              |\n| AS  | aspect marker                                        | 动态助词    | 助词                                                      | 了、着、过                  |\n| BA  | `bǎ` in ba-construction                              | 把字句     | 当“把”、“将”出现在结构“NP0 + BA + NP1+VP”时的词性                    | 把、将                    |\n| CC  | coordinating conjunction                             | 并列连接词   | 并列连词                                                    | 与、和、或者、还是              |\n| CD  | cardinal number                                      | 概数词     | 数词或表达数量的词                                               | 一百、好些、若干               |\n| CS  | subordinating conjunction                            | 从属连词    | 从属连词                                                    | 如果、那么、就                |\n| DEC | `de` as a complementizer or a nominalizer            | 补语成分“的” | 当“的”或“之”作补语标记或名词化标记时的词性，其结构为：S/VP DEC {NP}，如，喜欢旅游的大学生   | 的、之                    |\n| DEG | `de` as a genitive marker and an associative marker  | 属格“的”   | 当“的”或“之”作所有格时的词性，其结构为:NP/PP/JJ/DT DEG {NP}， 如，他的车、经济的发展 | 的、之                    |\n| DER | resultative `de`, `de` in V-de const and V-de-R      | 表结果的“得” | 当“得”出现在结构“V-得-R”时的词性，如，他跑得很快                            | 得                      |\n| DEV | manner `de`, `de` before VP                          | 表方式的“地” | 当“地”出现在结构“X-地-VP”时的词性，如，高兴地说                            | 地                      |\n| DT  | determiner                                           | 限定词     | 代冠词，通常用来修饰名词                                            | 这、那、该、每、各              |\n| ETC | for words like \"etc.\"                                | 表示省略    | “等”、“等等”的词性                                             | 等、等等              |\n| EM  | emoji                                                | 表情符     | 表情符、或称颜文字                                      | ：）             |\n| FW  | foreign words                                        | 外来语     | 外来词                                                     | 卡拉、A型                  |\n| IC  | incomplete component                                 | 不完整成分   | 不完整成分，尤指ASR导致的错误                         | 好*xin*、那个*ba*  |\n| IJ  | interjection                                         | 句首感叹词   | 感叹词，通常出现在句子首部                                           | 啊                      |\n| JJ  | other noun-modifier                                  | 其他名词修饰语 | 形容词                                                     | 共同、新                   |\n| LB  | `bèi` in long bei-const                              | 长句式表被动  | 当“被”、“叫”、“给”出现在结构“NP0 + LB + NP1+ VP”结构时 的词性，如，他被我训了一顿  | 被、叫、给                  |\n| LC  | localizer                                            | 方位词     | 方位词以及表示范围的限定词                                                     | 前、旁、到、在内、以来、为止               |\n| M   | measure word                                         | 量词      | 量词                                                      | 个、群、公里                 |\n| MSP | other particle                                       | 其他小品词   | 其他虚词，包括“所”、“以”、“来”和“而”等出现在VP前的词                         | 所、以、来、而                |\n| NN  | common noun                                          | 其他名词    | 除专有名词和时间名词外的所有名词                                        | 桌子、生活、经济               |\n| NOI | noise that characters are written in the wrong order | 噪声      | 汉字顺序颠倒产生的噪声                    | 事/NOI 类/NOI 各/NOI 故/NOI |\n| NR  | proper noun                                          | 专有名词    | 专有名词，通常表示地名、人名、机构名等                                     | 北京、乔丹、微软               |\n| NT  | temporal noun                                        | 时间名词    | 表示时间概念的名词                                               | 一月、汉朝、当今               |\n| OD  | ordinal number                                       | 序数词     | 序列词                                                     | 第一百                    |\n| ON  | onomatopoeia                                         | 象声词     | 象声词                                                     | 哗哗、呼、咯吱              |\n| P   | preposition e.g., \"from\" and \"to\"                    | 介词      | 介词                                                      | 从、对、根据                 |\n| PN  | pronoun                                              | 代词      | 代词，通常用来指代名词                                             | 我、这些、其、自己              |\n| PU  | punctuation                                          | 标点符号    | 标点符号                                                    | ?、。、；                  |\n| SB  | `bèi` in short bei-const                             | 短句式表被动  | 当“被”、“给”出现在NP0 +SB+ VP结果时的词性，如，他被训了 一顿                  | 被、叫                    |\n| SP  | sentence final particle                              | 句末助词    | 经常出现在句尾的词                                               | 吧、呢、啊、啊                |\n| URL | web address                                          | 网址      | 网址                                                      | www.hankcs.com         |\n| VA  | predicative adjective                                | 表语形容词   | 可以接在“很”后面的形容词谓语                                         | 雪白、厉害                  |\n| VC  | copula, be words                                     | 系动词     | 系动词，表示“是”或“非”概念的动词                                       | 是、为、非                  |\n| VE  | `yǒu` as the main verb                               | 动词有无    | 表示“有”或“无”概念的动词                                          | 有、没有、无                 |\n| VV  | other verb                                           | 其他动词    | 其他普通动词，包括情态词、控制动词、动作动词、心理动词等等                           | 可能、要、走、喜欢              |\n"
  },
  {
    "path": "docs/annotations/pos/index.md",
    "content": "# Part-of-Speech Tagging\n\n## Chinese\n```{toctree}\nctb\npku\n863\n```\n\n## Japanese\n```{toctree}\nnpcmj\n```\n\n## Multilingual\n\n```{toctree}\nud\n```\n\n\n\n"
  },
  {
    "path": "docs/annotations/pos/npcmj.md",
    "content": "<!--\n# ========================================================================\n# Copyright 2020 hankcs\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n# ========================================================================\n-->\n\n# NPCMJ\n\n\n| Tag       | Description                       |\n|-----------|-----------------------------------|\n| ADJI      | イ-adjective                      |\n| ADJI-MD   | modal イ-adjective                |\n| ADJN      | ナ-adjective                      |\n| ADJN-MD   | modal ナ-adjective                |\n| ADV       | adverb                            |\n| AX        | auxiliary verb (including copula) |\n| AXD       | auxiliary verb, past tense        |\n| CL        | classifier                        |\n| CONJ      | coordinating conjunction          |\n| D         | determiner                        |\n| FN        | formal noun                       |\n| FW        | foreign word                      |\n| INTJ      | interjection                      |\n| MD        | modal element                     |\n| N         | noun                              |\n| N-MENTION | mentioned expression              |\n| NEG       | negation                          |\n| NPR       | proper noun                       |\n| NUM       | numeral                           |\n| P-COMP    | complementizer particle           |\n| P-CONN    | conjunctional particle            |\n| P-FINAL   | final particle                    |\n| P-INTJ    | interjectional particle           |\n| P-OPTR    | toritate particle                 |\n| P-ROLE    | role particle                     |\n| PASS      | direct passive                    |\n| PASS2     | indirect passive                  |\n| PNL       | prenominal                        |\n| PRO       | pronoun                           |\n| PU        | punctuation                       |\n| PUL       | left bracket                      |\n| PUR       | right bracket                     |\n| Q         | quantifier                        |\n| QUOT      | quote                             |\n| SYM       | symbol                            |\n| VB        | verb (or verb stem)               |\n| VB0       | light verb                        |\n| VB2       | secondary verb                    |\n| WADV      | indeterminate adverb              |\n| WD        | indeterminate determiner          |\n| WNUM      | indeterminate numeral             |\n| WPRO      | indeterminate pronoun             |"
  },
  {
    "path": "docs/annotations/pos/pku.md",
    "content": "<!--\n# ========================================================================\n# Copyright 2020 hankcs\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n# ========================================================================\n-->\n\n# pku\n\n| 序号 | 词性 | 名称     | 帮助记忆的诠释                                         | 例子及注解                                                   |\n| ---- | ---- | -------- | ------------------------------------------------------ | ------------------------------------------------------------ |\n| 1    | Ag   | 形语素   | 形容词性语素。形容词代码为a，语素代码ｇ前面置以A。     | 绿色/n 似/d 锦/Ag ，                                         |\n| 2    | a    | 形容词   | 取英语形容词adjective的第1个字母                       | [重要/a 步伐/n]NP ，美丽/a ，看似/v 抽象/a ，                |\n| 3    | ad   | 副形词   | 直接作状语的形容词。形容词代码a和副词代码d并在一起。   | [积极/ad 谋求/v]V-ZZ ，幻象/n 易/ad 逝/Vg ，                 |\n| 4    | an   | 名形词   | 具有名词功能的形容词。形容词代码a和名词代码n并在一起。 | [外交/n 和/c 安全/an]NP-BL ，                                |\n| 5    | Bg   | 区别语素 | 区别词性语素。区别词代码为b，语素代码ｇ前面置以B。     | 赤/Ag 橙/Bg 黄/a 绿/a 青/a 蓝/a 紫/a ，                      |\n| 6    | b    | 区别词   | 取汉字“别”的声母。                                     | 女/b 司机/n， 金/b 手镯/n， 慢性/b 胃炎/n， 古/b 钱币/n， 副/b 主任/n， 总/b 公司/n 单音节区别词和单音节名词或名语素组合，作为一个词，并标以名词词性n。 |\n| 7    | c    | 连词     | 取英语连词conjunction的第1个字母。                     | 合作/vn 与/c 伙伴/n                                          |\n| 8    | Dg   | 副语素   | 副词性语素。副词代码为d，语素代码ｇ前面置以D。         | 了解/v 甚/Dg 深/a ，煞/Dg 是/v 喜人/a ，                     |\n| 9    | d    | 副词     | 取adverb的第2个字母，因其第1个字母已用于形容词。       | 进一步/d 发展/v ，                                           |\n| 10   | e    | 叹词     | 取英语叹词exclamation的第1个字母。                     | 啊/e ，/w 那/r 金灿灿/z 的/u 麦穗/n ，                       |\n| 11   | f    | 方位词   | 取汉字“方”。                                           | 军人/n 的/u 眼睛/n 里/f 不/d 是/v 没有/v 风景/n ，           |\n| 12   | h    | 前接成分 | 取英语head的第1个字母。                                | 许多/m 非/h 主角/n 人物/n ，办事处/n 的/u “/w 准/h 政府/n ”/w 功能/n 不断/d 加强/v ， |\n| 13   | i    | 成语     | 取英语成语idiom的第1个字母。                           | 一言一行/i ，义无反顾/i ，                                   |\n| 14   | j    | 简称略语 | 取汉字“简”的声母。                                     | [德/j 外长/n]NP ，文教/j ，                                  |\n| 15   | k    | 后接成分 | 后接成分。                                             | 少年儿童/l 朋友/n 们/k ，身体/n 健康/a 者/k ，               |\n| 16   | l    | 习用语   | 习用语尚未成为成语，有点“临时性”，取“临”的声母。       | 少年儿童/l 朋友/n 们/k ，落到实处/l ，                       |\n| 17   | Mg   | 数语素   | 数词性语素。数词代码为m，语素代码ｇ前面置以M。         | 甲/Mg 减下/v 的/u 人/n 让/v 乙/Mg 背上/v ，凡/d “/w 寅/Mg 年/n ”/w 中/f 出生/v 的/u 人/n 生肖/n 都/d 属/v 虎/n ， |\n| 18   | m    | 数词     | 取英语numeral的第3个字母，n，u已有他用。               | 1．数量词组应切分为数词和量词。 三/m 个/q， 10/m 公斤/q， 一/m 盒/q 点心/n ，但少数数量词已是词典的登录单位，则不再切分。 一个/m ， 一些/m ，2. 基数、序数、小数、分数、百分数一律不予切分，为一个切分单位，标注为 m 。一百二十三/m，20万/m， 123.54/m， 一个/m， 第一/m， 第三十五/m， 20%/m， 三分之二/m， 千分之三十/m， 几十/m 人/n， 十几万/m 元/q， 第一百零一/m 个/q ，3. 约数，前加副词、形容词或后加“来、多、左右”等助数词的应予分开。约/d 一百/m 多/m 万/m，仅/d 一百/m 个/q， 四十/m 来/m 个/q，二十/m 余/m 只/q， 十几/m 个/q，三十/m 左右/m ，两个数词相连的及“成百”、“上千”等则不予切分。五六/m 年/q， 七八/m 天/q，十七八/m 岁/q， 成百/m 学生/n，上千/m 人/n， 4．表序关系的“数＋名”结构，应予切分。二/m 连/n ，　三/m 部/n ， |\n| 19   | Ng   | 名语素   | 名词性语素。名词代码为n，语素代码ｇ前面置以N。         | 出/v 过/u 两/m 天/q 差/Ng， 理/v 了/u 一/m 次/q 发/Ng，      |\n| 20   | n    | 名词     | 取英语名词noun的第1个字母。                            | （参见 动词--v）岗位/n ， 城市/n ， 机会/n ，她/r 是/v 责任/n 编辑/n ， |\n| 21   | nr   | 人名     | 名词代码n和“人(ren)”的声母并在一起。                   | 1. 汉族人及与汉族起名方式相同的非汉族人的姓和名单独切分，并分别标注为nr。张/nr 仁伟/nr， 欧阳/nr 修/nr， 阮/nr 志雄/nr， 朴/nr 贞爱/nr汉族人除有单姓和复姓外，还有双姓，即有的女子出嫁后，在原来的姓上加上丈夫的姓。如：陈方安生。这种情况切分、标注为：陈/nr 方/nr 安生/nr；唐姜氏，切分、标注为：唐/nr 姜氏/nr。2. 姓名后的职务、职称或称呼要分开。江/nr 主席/n， 小平/nr 同志/n， 江/nr 总书记/n，张/nr 教授/n， 王/nr 部长/n， 陈/nr 老总/n， 李/nr 大娘/n， 刘/nr 阿姨/n， 龙/nr 姑姑/n3. 对人的简称、尊称等若为两个字，则合为一个切分单位，并标以nr。老张/nr， 大李/nr， 小郝/nr， 郭老/nr， 陈总/nr4. 明显带排行的亲属称谓要切分开，分不清楚的则不切开。三/m 哥/n， 大婶/n， 大/a 女儿/n， 大哥/n， 小弟/n， 老爸/n5. 一些著名作者的或不易区分姓和名的笔名通常作为一个切分单位。鲁迅/nr， 茅盾/nr， 巴金/nr， 三毛/nr， 琼瑶/nr， 白桦/nr6. 外国人或少数民族的译名（包括日本人的姓名）不予切分，标注为nr。克林顿/nr， 叶利钦/nr， 才旦卓玛/nr， 小林多喜二/nr， 北研二/nr，华盛顿/nr， 爱因斯坦/nr有些西方人的姓名中有小圆点，也不分开。卡尔·马克思/nr |\n| 22   | ns   | 地名     | 名词代码n和处所词代码s并在一起。                       | 安徽/ns，深圳/ns，杭州/ns，拉萨/ns，哈尔滨/ns， 呼和浩特/ns， 乌鲁木齐/ns，长江/ns，黄海/ns，太平洋/ns， 泰山/ns， 华山/ns，亚洲/ns， 海南岛/ns，太湖/ns，白洋淀/ns， 俄罗斯/ns，哈萨克斯坦/ns，彼得堡/ns， 伏尔加格勒/ns 1. 国名不论长短，作为一个切分单位。中国/ns， 中华人民共和国/ns， 日本国/ns， 美利坚合众国/ns， 美国/ns2. 地名后有“省”、“市”、“县”、“区”、“乡”、“镇”、“村”、“旗”、“州”、“都”、“府”、“道”等单字的行政区划名称时，不切分开，作为一个切分单位。四川省/ns， 天津市/ns，景德镇/ns沙市市/ns， 牡丹江市/ns，正定县/ns，海淀区/ns， 通州区/ns，东升乡/ns， 双桥镇/ns 南化村/ns，华盛顿州/ns，俄亥俄州/ns，东京都/ns， 大阪府/ns，北海道/ns， 长野县/ns，开封府/ns，宣城县/ns3. 地名后的行政区划有两个以上的汉字，则将地名同行政区划名称切开，不过要将地名同行政区划名称用方括号括起来，并标以短语NS。[芜湖/ns 专区/n] NS，[宣城/ns 地区/n]ns，[内蒙古/ns 自治区/n]NS，[深圳/ns 特区/n]NS， [厦门/ns 经济/n 特区/n]NS， [香港/ns 特别/a 行政区/n]NS，[香港/ns 特区/n]NS， [华盛顿/ns 特区/n]NS，4. 地名后有表示地形地貌的一个字的普通名词，如“江、河、山、洋、海、岛、峰、湖”等，不予切分。鸭绿江/ns，亚马逊河/ns， 喜马拉雅山/ns， 珠穆朗玛峰/ns，地中海/ns，大西洋/ns，洞庭湖/ns， 塞普路斯岛/ns 5. 地名后接的表示地形地貌的普通名词若有两个以上汉字，则应切开。然后将地名同该普通名词标成短语NS。[台湾/ns 海峡/n]NS，[华北/ns 平原/n]NS，[帕米尔/ns 高原/n]NS， [南沙/ns 群岛/n]NS，[京东/ns 大/a 峡谷/n]NS [横断/b 山脉/n]NS6．地名后有表示自然区划的一个字的普通名词，如“ 街，路，道，巷，里，町，庄，村，弄，堡”等，不予切分。 中关村/ns，长安街/ns，学院路/ns， 景德镇/ns， 吴家堡/ns， 庞各庄/ns， 三元里/ns，彼得堡/ns， 北菜市巷/ns， 7．地名后接的表示自然区划的普通名词若有两个以上汉字，则应切开。然后将地名同自然区划名词标成短语NS。[米市/ns 大街/n]NS， [蒋家/nz 胡同/n]NS ， [陶然亭/ns 公园/n]NS ， 8． 大小地名相连时的标注方式为：北京市/ns 海淀区/ns 海淀镇/ns [南/f 大街/n]NS [蒋家/nz 胡同/n]NS 24/m 号/q ， |\n| 23   | nt   | 机构团体 | “团”的声母为t，名词代码n和t并在一起。                  | （参见2。短语标记说明--NT）联合国/nt，中共中央/nt，国务院/nt， 北京大学/nt1．大多数团体、机构、组织的专有名称一般是短语型的，较长，且含有地名或人名等专名，再组合，标注为短语NT。[中国/ns 计算机/n 学会/n]NT， [香港/ns 钟表业/n 总会/n]NT， [烟台/ns 大学/n]NT， [香港/ns 理工大学/n]NT， [华东/ns 理工大学/n]NT，[合肥/ns 师范/n 学院/n]NT， [北京/ns 图书馆/n]NT， [富士通/nz 株式会社/n]NT， [香山/ns 植物园/n]NT， [安娜/nz 美容院/n]NT，[上海/ns 手表/n 厂/n]NT， [永和/nz 烧饼铺/n]NT，[北京/ns 国安/nz 队/n]NT，2. 对于在国际或中国范围内的知名的唯一的团体、机构、组织的名称即使前面没有专名，也标为nt或NT。联合国/nt，国务院/nt，外交部/nt， 财政部/nt，教育部/nt， 国防部/nt，[世界/n 贸易/n 组织/n]NT， [国家/n 教育/vn 委员会/n]NT，[信息/n 产业/n 部/n]NT，[全国/n 信息/n 技术/n 标准化/vn 委员会/n]NT，[全国/n 总/b 工会/n]NT，[全国/n 人民/n 代表/n 大会/n]NT，美国的“国务院”，其他国家的“外交部、财政部、教育部”，必须在其所属国的国名之后出现时，才联合标注为NT。[美国/ns 国务院/n]NT，[法国/ns 外交部/n]NT，[美/j 国会/n]NT，日本有些政府机构名称很特别，无论是否出现在“日本”国名之后都标为nt。[日本/ns 外务省/nt]NT，[日/j 通产省/nt]NT通产省/nt 3. 前后相连有上下位关系的团体机构组织名称的处理方式如下:[联合国/nt 教科文/j 组织/n]NT， [中国/ns 银行/n 北京/ns 分行/n]NT，[河北省/ns 正定县/ns 西平乐乡/ns 南化村/ns 党支部/n]NT， 当下位名称含有专名（如“北京/ns 分行/n”、“南化村/ns 党支部/n”、“昌平/ns 分校/n”）时，也可脱离前面的上位名称单独标注为NT。[中国/ns 银行/n]NT [北京/ns 分行/n]NT，北京大学/nt [昌平/ns 分校/n]NT，4. 团体、机构、组织名称中用圆括号加注简称时:[宝山/ns 钢铁/n （/w 宝钢/j ）/w 总/b 公司/n]NT，[宝山/ns 钢铁/n 总/b 公司/n]NT，（/w 宝钢/j ）/w |\n| 24   | nx   | 外文字符 | 外文字符。                                             | A/nx 公司/n ，B/nx 先生/n ，X/nx 君/Ng ，24/m K/nx 镀金/n ，C/nx 是/v 光速/n ，Windows98/nx ，PentiumIV/nx ，I LOVE THIS GAME/nx ，HanLP/nx |\n| 25   | nz   | 其他专名 | “专”的声母的第1个字母为z，名词代码n和z并在一起。       | （参见2。短语标记说明--NZ）除人名、国名、地名、团体、机构、组织以外的其他专有名词都标以nz。满族/nz，俄罗斯族/nz，汉语/nz，罗马利亚语/nz， 捷克语/nz，中文/nz， 英文/nz， 满人/nz， 哈萨克人/nz， 诺贝尔奖/nz， 茅盾奖/nz， 1.包含专有名称（或简称）的交通线，标以nz；短语型的，标为NZ。津浦路/nz， 石太线/nz， [京/j 九/j 铁路/n]NZ， [京/j 津/j 高速/b 公路/n]NZ， 2. 历史上重要事件、运动等专有名称一般是短语型的，按短语型专有名称处理，标以NZ。[卢沟桥/ns 事件/n]NZ， [西安/ns 事变/n]NZ，[五四/t 运动/n]NZ， [明治/nz 维新/n]NZ，[甲午/t 战争/n]NZ，3.专有名称后接多音节的名词，如“语言”、“文学”、“文化”、“方式”、“精神”等，失去专指性，则应分开。欧洲/ns 语言/n， 法国/ns 文学/n， 西方/ns 文化/n， 贝多芬/nr 交响乐/n， 雷锋/nr 精神/n， 美国/ns 方式/n，日本/ns 料理/n， 宋朝/t 古董/n 4. 商标（包括专名及后接的“牌”、“型”等）是专指的，标以nz，但其后所接的商品仍标以普通名词n。康师傅/nr 方便面/n， 中华牌/nz 香烟/n， 牡丹III型/nz 电视机/n， 联想/nz 电脑/n， 鳄鱼/nz 衬衣/n， 耐克/nz 鞋/n5. 以序号命名的名称一般不认为是专有名称。2/m 号/q 国道/n ，十一/m 届/q 三中全会/j如果前面有专名，合起来作为短语型专名。[中国/ns 101/m 国道/n]NZ， [中共/j 十一/m 届/q 三中全会/j]NZ，6. 书、报、杂志、文档、报告、协议、合同等的名称通常有书名号加以标识，不作为专有名词。由于这些名字往往较长，名字本身按常规处理。《/w 宁波/ns 日报/n 》/w ，《/w 鲁迅/nr 全集/n 》/w，中华/nz 读书/vn 报/n， 杜甫/nr 诗选/n，少数书名、报刊名等专有名称，则不切分。红楼梦/nz， 人民日报/nz，儒林外史/nz 7. 当有些专名无法分辨它们是人名还是地名或机构名时，暂标以nz。[巴黎/ns 贝尔希/nz 体育馆/n]NT，其中“贝尔希”只好暂标为nz。 |\n| 26   | o    | 拟声词   | 取英语拟声词onomatopoeia的第1个字母。                  | 哈哈/o 一/m 笑/v ，装载机/n 隆隆/o 推进/v ，                 |\n| 27   | p    | 介词     | 取英语介词prepositional的第1个字母。                   | 对/p 子孙后代/n 负责/v ，以/p 煤/n 养/v 农/Ng ，为/p 治理/v 荒山/n 服务/v ， 把/p 青年/n 推/v 上/v 了/u 领导/vn 岗位/n ， |\n| 28   | q    | 量词     | 取英语quantity的第1个字母。                            | （参见数词m）首/m 批/q ，一/m 年/q ，                        |\n| 29   | Rg   | 代语素   | 代词性语素。代词代码为r，在语素的代码g前面置以R。      | 读者/n 就/d 是/v 这/r 两/m 棵/q 小树/n 扎根/v 于/p 斯/Rg 、/w 成长/v 于/p 斯/Rg 的/u 肥田/n 沃土/n ， |\n| 30   | r    | 代词     | 取英语代词pronoun的第2个字母，因p已用于介词。          | 单音节代词“本”、“每”、“各”、“诸”后接单音节名词时，和后接的单音节名词合为代词；当后接双音节名词时，应予切分。本报/r， 每人/r， 本社/r， 本/r 地区/n， 各/r 部门/n |\n| 31   | s    | 处所词   | 取英语space的第1个字母。                               | 家里/s 的/u 电脑/n 都/d 联通/v 了/u 国际/n 互联网/n ，西部/s 交通/n 咽喉/n ， |\n| 32   | Tg   | 时语素   | 时间词性语素。时间词代码为t，在语素的代码g前面置以T。  | ３日/t 晚/Tg 在/p 总统府/n 发表/v 声明/n ，尊重/v 现/Tg 执政/vn 当局/n 的/u 权威/n ， |\n| 33   | t    | 时间词   | 取英语time的第1个字母。                                | 1. 年月日时分秒，按年、月、日、时、分、秒切分，标注为t 。1997年/t 3月/t 19日/t 下午/t 2时/t 18分/t若数字后无表示时间的“年、月、日、时、分、秒”等的标为数词m。1998/m 中文/n 信息/n 处理/vn 国际/n 会议/n 2. 历史朝代的名称虽然有专有名词的性质，仍标注为t。西周/t， 秦朝/t， 东汉/t， 南北朝/t， 清代/t“牛年、虎年”等一律不予切分，标注为：牛年/t， 虎年/t， 甲午年/t， 甲午/t 战争/n， 庚子/t 赔款/n， 戊戌/t 变法/n |\n| 34   | u    | 助词     | 取英语助词auxiliary。                                  | [[俄罗斯/ns 和/c 北约/j]NP-BL 之间/f [战略/n 伙伴/n 关系/n]NP 的/u 建立/vn]NP 填平/v 了/u [[欧洲/ns 安全/a 政治/n]NP 的/u 鸿沟/n]NP |\n| 35   | Vg   | 动语素   | 动词性语素。动词代码为v。在语素的代码g前面置以V。      | 洗/v 了/u 一个/m 舒舒服服/z 的/u 澡/Vg                       |\n| 36   | v    | 动词     | 取英语动词verb的第一个字母。                           | （参见 名词--n）[[[欧盟/j 扩大/v]S 的/u [历史性/n 决定/n]NP]NP 和/c [北约/j 开放/v]S]NP-BL [为/p [创建/v [一/m 种/q 新/a 的/u 欧洲/ns 安全/a 格局/n]NP]VP-SBI]PP-MD [奠定/v 了/u 基础/n]V-SBI ，， |\n| 37   | vd   | 副动词   | 直接作状语的动词。动词和副词的代码并在一起。           | 形势/n 会/v 持续/vd 好转/v ，认为/v 是/v 电话局/n 收/v 错/vd 了/u 费/n ， |\n| 38   | vn   | 名动词   | 指具有名词功能的动词。动词和名词的代码并在一起。       | 引起/v 人们/n 的/u 关注/vn 和/c 思考/vn ，收费/vn 电话/n 的/u 号码/n ， |\n| 39   | w    | 标点符号 |                                                        | ”/w ：/w                                                     |\n| 40   | x    | 非语素字 | 非语素字只是一个符号，字母x通常用于代表未知数、符号。  |                                                              |\n| 41   | Yg   | 语气语素 | 语气词性语素。语气词代码为y。在语素的代码g前面置以Y。  | 唯/d 大力/d 者/k 能/v 致/v 之/u 耳/Yg                        |\n| 42   | y    | 语气词   | 取汉字“语”的声母。                                     | 会/v 泄露/v 用户/n 隐私/n 吗/y ，又/d 何在/v 呢/y ？         |\n| 43   | z    | 状态词   | 取汉字“状”的声母的前一个字母。                         | 取得/v 扎扎实实/z 的/u 突破性/n 进展/vn ，四季/n 常青/z 的/u 热带/n 树木/n ，短短/z 几/m 年/q 间， |"
  },
  {
    "path": "docs/annotations/pos/ud.md",
    "content": "<!--\n# ========================================================================\n# Copyright 2020 hankcs\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n# ========================================================================\n-->\n\n# Universal Dependencies\n\nSee also [Universal Dependencies](https://universaldependencies.org/u/pos/).\n\n| Tag        | Description                                  |\n|------------|----------------------------------------------|\n| ADJ   | adjective                 |\n| ADP   | adposition                |\n| ADV   | adverb                    |\n| AUX   | auxiliary                 |\n| CCONJ | coordinating conjunction  |\n| DET   | determiner                |\n| INTJ  | interjection              |\n| NOUN  | noun                      |\n| NUM   | numeral                   |\n| PART  | particle                  |\n| PRON  | pronoun                   |\n| PROPN | proper noun               |\n| PUNCT | punctuation               |\n| SCONJ | subordinating conjunction |\n| SYM   | symbol                    |\n| VERB  | verb                      |\n| X     | other                     |"
  },
  {
    "path": "docs/annotations/sdp/dm.md",
    "content": "# The reduction of Minimal Recursion Semantics\n\nPlease refer to [Minimal Recursion Semantics An Introduction](https://www.cl.cam.ac.uk/~aac10/papers/mrs.pdf).\n"
  },
  {
    "path": "docs/annotations/sdp/index.md",
    "content": "# Semantic Dependency Parsing\n\n## Chinese\n\n```{toctree}\nsemeval16\n```\n\n## English\n\n```{toctree}\ndm\npas\npsd\n```\n\n"
  },
  {
    "path": "docs/annotations/sdp/pas.md",
    "content": "# Predicate-Argument Structures\n\nPlease refer to [Probabilistic disambiguation models for wide-coverage HPSG parsing](https://www.aclweb.org/anthology/P05-1011.pdf).\n"
  },
  {
    "path": "docs/annotations/sdp/psd.md",
    "content": "# Prague Czech-English Dependency Treebank\n\nPlease refer to [Prague Czech-English Dependency Treebank](http://ufal.mff.cuni.cz/pcedt2.0/en/index.html).\n"
  },
  {
    "path": "docs/annotations/sdp/semeval16.md",
    "content": "<!--\n# ========================================================================\n# Copyright 2020 hankcs\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n# ========================================================================\n-->\n\n# SemEval2016\n\n## CSDP\n\nSemEval2016 adopts the CSDP guideline listed as follows.\n\n### 语义关系标注标签集\n\n| 分类         |              |                 |                                                              |\n| ------------ | ------------ | --------------- | ------------------------------------------------------------ |\n| 语义周边角色 | 主体角色     | 施事AGT；       | 施事Agt；感事Aft                                             |\n|              |              | 当事EXP；       | 当事Exp；领事Poss                                            |\n|              | 客体角色     | 受事PAT；       | 受事Pat                                                      |\n|              |              | 客事CONT；      | 客事Cont；成事Prod；结局Cons                                 |\n|              |              | 涉事DATV；      | 涉事Datv；比较Comp；源事Orig                                 |\n|              |              | 系事LINK；      | 类事Clas；属事Belg                                           |\n|              | 情境角色     | 工具TOOL；      | 工具Tool                                                     |\n|              |              | 材料MATL；      | 材料Matl                                                     |\n|              |              | 方式MANN；      | 方式Mann；依据Accd                                           |\n|              |              | 范围SCO；       | 范围Sco                                                      |\n|              |              | 缘由REAS；      | 缘故Reas；意图Int                                            |\n|              |              | 时间TIME；      | 时间Time；时间起点Tini；时间终点Tfin；时段Tdur；时距Trang    |\n|              |              | 空间LOC；       | 空间Loc；原处所Lini；终处所Lfin；通过处所Lthru；趋向Dir      |\n|              |              | 度量MEAS；      | 数量Quan；起始量Nini；终止量Nfin；数量短语Qp；频率Freq；顺序Seq；变化量Nvar |\n|              |              | 状态STAT；      | 状态Stat；起始状态Sini；终止状态Sfin；历经状态Sproc          |\n|              |              | 修饰FEAT；      | 描写Desc；宿主Host；名词修饰语Nmod；时间修饰语Tmod           |\n| 语义结构关系 | 反关系       | 反施事rAGT；    | 反施事rAgt；反感事rAft                                       |\n|              |              | 反当事rEXP。    | 反当事rExp；反领事rPoss                                      |\n|              |              | 反受事rPAT；    | 反受事rPat                                                   |\n|              |              | 反客事rCONT；   | 反客事rCont；反成事rProd；反结局rCons                        |\n|              |              | 反涉事rDATV；   | 反涉事rDatv；反比较rComp；反源事rOrig                        |\n|              |              | 反系事rLINK。   | 反类事rClas；反属事rBelg                                     |\n|              |              | 反工具rTOOL；   | 反工具rTool                                                  |\n|              |              | 反材料rMATL；   | 反材料rMatl                                                  |\n|              |              | 反方式RMANN；   | 反方式rMann；反依据rAccd                                     |\n|              |              | 反范围rSCO；    | 反范围rSco                                                   |\n|              |              | 反缘由rREAS；   | 反缘故rReas；反意图rInt                                      |\n|              |              | 反时间rTIME；   | 反时间rTime；反时间起点rTini；反时间终点rTfin；反时段rTdur；反时距rTrang |\n|              |              | 反空间rLOC；    | 反空间rLoc；反原处所rLini；反终处所rLfin；反通过处所rLthru；反趋向rDir |\n|              |              | 反度量rMEAS；   | 反数量rQuan；反起始量rNini；反终止量rNfin；反数量短语rQp；反频率rFreq；反顺序rSeq；反变化量rNvar |\n|              |              | 反状态rSTAT；   | 反状态rStat；反起始状态rSini；反终止状态rSfin；反历经状态rSproc |\n|              |              | 反修饰rFEAT；   | 反描写rDesc；反宿主rHost; 反名词修饰语rNmod; 反时间修饰语rTmod |\n|              | 嵌套事件关系 | 嵌套施事dAGT；  | 嵌套施事dAgt；嵌套感事dAft                                   |\n|              |              | 嵌套当事dEXP。  | 嵌套当事dExp；嵌套领事dPoss                                  |\n|              |              | 嵌套受事dPAT；  | 嵌套受事dPat                                                 |\n|              |              | 嵌套客事dCONT； | 嵌套客事dCont；嵌套成事dProd；嵌套结局dCons                  |\n|              |              | 嵌套涉事dDATV； | 嵌套涉事dDatv；嵌套比较dComp；嵌套源事dOrig                  |\n|              |              | 嵌套系事dLINK。 | 嵌套类事dClas；嵌套属事dBelg                                 |\n|              |              | 嵌套工具dTOOL； | 嵌套工具dTool                                                |\n|              |              | 嵌套材料dMATL； | 嵌套材料dMatl                                                |\n|              |              | 嵌套方式dMANN； | 嵌套方式dMann；嵌套依据dAccd                                 |\n|              |              | 嵌套范围dSCO；  | 嵌套范围dSco                                                 |\n|              |              | 嵌套缘由dREAS； | 嵌套缘故dReas；嵌套意图dInt                                  |\n|              |              | 嵌套时间dTIME； | 嵌套时间dTime；嵌套时间起点dTini；嵌套时间终点dTfin；嵌套时段dTdur；嵌套时距dTrang |\n|              |              | 嵌套空间dLOC；  | 嵌套空间dLoc；嵌套原处所dLini；嵌套终处所dLfin；嵌套通过处所dLthru；嵌套趋向dDir |\n|              |              | 嵌套度量dMEAS； | 嵌套数量dQuan；嵌套起始量dNini；嵌套终止量dNfin；嵌套数量短语dQp；嵌套频率dFreq；嵌套顺序dSeq；嵌套变化量dNvar |\n|              |              | 嵌套状态dSTAT； | 嵌套状态dStat；嵌套起始状态dSini；嵌套终止状态dSfin；嵌套历经状态dSproc |\n|              |              | 嵌套修饰dFEAT； | 嵌套描写dDesc；嵌套宿主dHost; 嵌套名词修饰语dNmod; 嵌套时间修饰语dTmod |\n|              | 事件关系     | 并列关系eCOO；  | 并列eCoo；等同eEqu；分叙eRect；选择eSelt;割舍eAban；选取ePref；总括eSum |\n|              |              | 先行关系ePREC； | 先行ePrec；原因eCau；条件eCond；假设eSupp；手段eMetd；让步eConc |\n|              |              | 后继关系eSUCC； | 后继eSucc；递进eProg；转折 eAdvt；目的ePurp；结果eResu；推论eInf |\n| 语义依附标记 | 标点标记     | 标点标记mPUNC； | 标点标记mPunc                                                |\n|              | 依附标记     | 否定标记mNEG；  | 否定标记mNeg                                                 |\n|              |              | 关系标记mRELA； | 连词标记mConj；介词标记mPrep                                 |\n|              |              | 依附标记mDEPD； | 语气标记mTone；时间标记mTime;范围标记mRang；情态标记mMod； 频率标记mFreq；程度标记mDegr；趋向标记mDir；的字标记mAux； 多数标记mMaj；插入语标记mPars；离合标记mSepa；实词虚化标记mVain 重复标记mRept |\n\n## SemEval2016\n\nThe following table is a subset of CSDP but offers some examples to illustrate the idea.\n\n| 关系类型   | Tag           | Description        | Example                     |\n|--------|---------------|--------------------|-----------------------------|\n| 施事关系   | Agt           | Agent              | 我送她一束花 (我 <– 送)             |\n| 当事关系   | Exp           | Experiencer        | 我跑得快 (跑 –> 我)               |\n| 感事关系   | Aft           | Affection          | 我思念家乡 (思念 –> 我)             |\n| 领事关系   | Poss          | Possessor          | 他有一本好读 (他 <– 有)             |\n| 受事关系   | Pat           | Patient            | 他打了小明 (打 –> 小明)             |\n| 客事关系   | Cont          | Content            | 他听到鞭炮声 (听 –> 鞭炮声)           |\n| 成事关系   | Prod          | Product            | 他写了本小说 (写 –> 小说)            |\n| 源事关系   | Orig          | Origin             | 我军缴获敌人四辆坦克 (缴获 –> 坦克)       |\n| 涉事关系   | Datv          | Dative             | 他告诉我个秘密 ( 告诉 –> 我 )         |\n| 比较角色   | Comp          | Comitative         | 他成绩比我好 (他 –> 我)             |\n| 属事角色   | Belg          | Belongings         | 老赵有俩女儿 (老赵 <– 有)            |\n| 类事角色   | Clas          | Classification     | 他是中学生 (是 –> 中学生)            |\n| 依据角色   | Accd          | According          | 本庭依法宣判 (依法 <– 宣判)           |\n| 缘故角色   | Reas          | Reason             | 他在愁女儿婚事 (愁 –> 婚事)           |\n| 意图角色   | Int           | Intention          | 为了金牌他拼命努力 (金牌 <– 努力)        |\n| 结局角色   | Cons          | Consequence        | 他跑了满头大汗 (跑 –> 满头大汗)         |\n| 方式角色   | Mann          | Manner             | 球慢慢滚进空门 (慢慢 <– 滚)           |\n| 工具角色   | Tool          | Tool               | 她用砂锅熬粥 (砂锅 <– 熬粥)           |\n| 材料角色   | Malt          | Material           | 她用小米熬粥 (小米 <– 熬粥)           |\n| 时间角色   | Time          | Time               | 唐朝有个李白 (唐朝 <– 有)            |\n| 空间角色   | Loc           | Location           | 这房子朝南 (朝 –> 南)              |\n| 历程角色   | Proc          | Process            | 火车正在过长江大桥 (过 –> 大桥)         |\n| 趋向角色   | Dir           | Direction          | 部队奔向南方 (奔 –> 南)             |\n| 范围角色   | Sco           | Scope              | 产品应该比质量 (比 –> 质量)           |\n| 数量角色   | Quan          | Quantity           | 一年有365天 (有 –> 天)            |\n| 数量数组   | Qp            | Quantity-phrase    | 三本书 (三 –> 本)                |\n| 频率角色   | Freq          | Frequency          | 他每天看书 (每天 <– 看)             |\n| 顺序角色   | Seq           | Sequence           | 他跑第一 (跑 –> 第一)              |\n| 描写角色   | Desc(Feat)    | Description        | 他长得胖 (长 –> 胖)               |\n| 宿主角色   | Host          | Host               | 住房面积 (住房 <– 面积)             |\n| 名字修饰角色 | Nmod          | Name-modifier      | 果戈里大街 (果戈里 <– 大街)           |\n| 时间修饰角色 | Tmod          | Time-modifier      | 星期一上午 (星期一 <– 上午)           |\n| 反角色    | r + main role |                    | 打篮球的小姑娘 (打篮球 <– 姑娘)         |\n| 嵌套角色   | d + main role |                    | 爷爷看见孙子在跑 (看见 –> 跑)          |\n| 并列关系   | eCoo          | event Coordination | 我喜欢唱歌和跳舞 (唱歌 –> 跳舞)         |\n| 选择关系   | eSelt         | event Selection    | 您是喝茶还是喝咖啡 (茶 –> 咖啡)         |\n| 等同关系   | eEqu          | event Equivalent   | 他们三个人一起走 (他们 –> 三个人)        |\n| 先行关系   | ePrec         | event Precedent    | 首先，先                        |\n| 顺承关系   | eSucc         | event Successor    | 随后，然后                       |\n| 递进关系   | eProg         | event Progression  | 况且，并且                       |\n| 转折关系   | eAdvt         | event adversative  | 却，然而                        |\n| 原因关系   | eCau          | event Cause        | 因为，既然                       |\n| 结果关系   | eResu         | event Result       | 因此，以致                       |\n| 推论关系   | eInf          | event Inference    | 才，则                         |\n| 条件关系   | eCond         | event Condition    | 只要，除非                       |\n| 假设关系   | eSupp         | event Supposition  | 如果，要是                       |\n| 让步关系   | eConc         | event Concession   | 纵使，哪怕                       |\n| 手段关系   | eMetd         | event Method       |                             |\n| 目的关系   | ePurp         | event Purpose      | 为了，以便                       |\n| 割舍关系   | eAban         | event Abandonment  | 与其，也不                       |\n| 选取关系   | ePref         | event Preference   | 不如，宁愿                       |\n| 总括关系   | eSum          | event Summary      | 总而言之                        |\n| 分叙关系   | eRect         | event Recount      | 例如，比方说                      |\n| 连词标记   | mConj         | Conjunction        | 和，或                         |\n| 的字标记   | mAux          | Auxiliary          | 的，地，得                       |\n| 介词标记   | mPrep         | Preposition        | 把，被                         |\n| 语气标记   | mTone         | Tone               | 吗，呢                         |\n| 时间标记   | mTime         | Time               | 才，曾经                        |\n| 范围标记   | mRang         | Range              | 都，到处                        |\n| 程度标记   | mDegr         | Degree             | 很，稍微                        |\n| 频率标记   | mFreq         | Frequency Marker   | 再，常常                        |\n| 趋向标记   | mDir          | Direction Marker   | 上去，下来                       |\n| 插入语标记  | mPars         | Parenthesis Marker | 总的来说，众所周知                   |\n| 否定标记   | mNeg          | Negation Marker    | 不，没，未                       |\n| 情态标记   | mMod          | Modal Marker       | 幸亏，会，能                      |\n| 标点标记   | mPunc         | Punctuation Marker | ，。！                         |\n| 重复标记   | mPept         | Repetition Marker  | 走啊走 (走 –> 走)                |\n| 多数标记   | mMaj          | Majority Marker    | 们，等                         |\n| 实词虚化标记 | mVain         | Vain Marker        |                             |\n| 离合标记   | mSepa         | Seperation Marker  | 吃了个饭 (吃 –> 饭) 洗了个澡 (洗 –> 澡) |\n| 根节点    | Root          | Root               | 全句核心节点                      |\n\nSee also [SemEval-2016 Task 9](https://www.hankcs.com/nlp/sdp-corpus.html) and [CSDP](https://csdp-doc.readthedocs.io/zh_CN/latest/%E9%99%84%E5%BD%95/).\n"
  },
  {
    "path": "docs/annotations/srl/cpb.md",
    "content": "<!--\n# ========================================================================\n# Copyright 2020 hankcs\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n# ========================================================================\n-->\n\n# Chinese Proposition Bank\n\n|      | 标签       | 角色    | 例子                      |\n|------|----------|-------|-------------------------|\n| 中心角色 | ARG0     | 施事者   | (ARG0中国政府)提供援助         |\n|      | ARG1     | 受事者   | 中国政府提供(ARG1援助)          |\n|      | ARG2     | 依谓词而定 | 失业率控制(ARG2在百分之十内)       |\n|      | ARG3     | 依谓词而定 | (ARG3从城市)扩大到农村          |\n|      | ARG4     | 依谓词而定 | 提高(ARG4百分之二十)          |\n| 附属角色 | ARGM-ADV | 状语    | (ARGM-ADV共同)承担          |\n|      | ARGM-BNF | 受益者   | (ARGM-BNF为其他国家)进行融资     |\n|      | ARGM-CND | 条件    | (ARGM-CND如果成功)，他就留下     |\n|      | ARGM-DIR | 方向    | (ARGM-DIR向和平)迈出一大步      |\n|      | ARGM-EXT | 范围    | 在北京逗留(ARGM-EXT两天)      |\n|      | ARGM-FRQ | 频率    | 每半年执行(ARGM-FRQ一次)      |\n|      | ARGM-LOC | 地点、位置 | (ARGM-LOC在机场)被捕获        |\n|      | ARGM-MNR | 方式    | (ARGM-MNR以中英文)发行        |\n|      | ARGM-PRP | 目的或原因 | (ARGM-PRP由于危机)而破产       |\n|      | ARGM-TMP | 时间    | 公司(ARGM-TMP去年)成立       |\n|      | ARGM-TPC | 主题    | (ARGM-TPC稳定政策)，核心是...   |\n|      | ARGM-DIS | 话语标记  | (ARGM-DIS因此)，他感到不公      |\n|      | ARGM-CRD | 并列论元  | (ARGM-CRD与台湾)非正式接触      |\n|      | ARGM-PRD | 次谓词   | 指控廉政公署五人(ARGM-PRD接受贿赂) |\n\n\n```{note}\nAlthough ARG0 and ARG1 share general definitions across all predicates, word sense disambiguation is required to find \nthe coresponding definition of semantic roles. Given the word sense of `变化`, say `变化-2`, \n[its second frameset](http://verbs.colorado.edu/chinese/cpb/html_frames/0183-bian-hua.html) can \nbe found which defines the following 2 arguments:\n\n1.    ARG0: agent/cause\n2.    ARG1: entity arg0 changes\n\nThese definitions are different from that of frameset `变化-1`:\n\n1.    ARG0: entity undergoing change\n   \nSometimes, the number of arguments and definitions can vary a lot across framesets. \nIn summary, word sense disambiguation is essential if SRL is to be used to best effect in practical applications  \n```"
  },
  {
    "path": "docs/annotations/srl/index.md",
    "content": "# Semantic Role Labeling\n\n## Chinese\n```{toctree}\ncpb\n```\n\n## English\n```{toctree}\npropbank\n```\n\n"
  },
  {
    "path": "docs/annotations/srl/propbank.md",
    "content": "<!--\n# ========================================================================\n# Copyright 2020 hankcs\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n# ========================================================================\n-->\n\n# English PropBank\n\n| Role | Description                                  |\n|------|----------------------------------------|\n| ARG0 | agent                                  |\n| ARG1 | patient                                |\n| ARG2 | instrument, benefactive, attribute     |\n| ARG3 | starting point, benefactive, attribute |\n| ARG4 | ending point                           |\n| ARGM | modifier                               |\n| COM  | Comitative                             |\n| LOC  | Locative                               |\n| DIR  | Directional                            |\n| GOL  | Goal                                   |\n| MNR  | Manner                                 |\n| TMP  | Temporal                               |\n| EXT  | Extent                                 |\n| REC  | Reciprocals                            |\n| PRD  | Secondary Predication                  |\n| PRP  | Purpose                                |\n| CAU  | Cause                                  |\n| DIS  | Discourse                              |\n| ADV  | Adverbials                             |\n| ADJ  | Adjectival                             |\n| MOD  | Modal                                  |\n| NEG  | Negation                               |\n| DSP  | Direct Speech                          |\n| LVB  | Light Verb                             |\n| CXN  | Construction                           |\n\n"
  },
  {
    "path": "docs/annotations/tok/ctb.md",
    "content": "The Segmentation Guidelines for the Penn Chinese Treebank (3.0)\n===============================================================\n\nFei Xia\n\n*University of Pennsylvania*\n\nThis is an OCR version. See also the [PDF version](https://repository.upenn.edu/cgi/viewcontent.cgi?article=1038&context=ircs_reports).\n\n## Abstract\n\n\nThis document describes the segmentation guidelines for the Penn Chinese Treebank Project. The goal of the project is the creation of a 100-thousand-word corpus of Mandarin Chinese text with syntactic bracketing. The Chinese Treebank has been released via the Linguistic Data Consortium (LDC) and is available to the public.\n\nThe segmentation guidelines have been revised several times during the two-year period of the project. The previous two versions were completed in December 1998 and March 1999, respectively. This document is the third and final version. We have added an introduction chapter in order to explain some rationale behind certain decisions in the guidelines. We also include the English gloss to the Chinese words in the guidelines.\n\n\nIn this document, we first discuss the notion of word and tests for wordhood that have been proposed in the literature. Then we give the specification for word segmentation. The specification is organized according to the potential Part-of-Speech tag of an expression and the internal structure of the expression. Next, we specify the treatment for some common collocations. Finally, we compare our guidelines with two segmentation standards: the first (Liu et al., 1993) is used in Mainland China and the second (CKIP, 1996) is used in Academia Sinica in Taiwan.\n\n## Chapter 1 Introduction\n\nThis document is designed for the Penn Chinese Treebank Project [XPX+ 00]. The goal of the project is the creation of a 100-thousand word corpus of Mandarin Chinese text with syntactic bracketing. The annotation consists of two stages: the first phrase is word segmentation and part-of-speech (POS) tagging and the second phrase is syntactic bracketing. Each stage includes at least two passes, that is, the data are annotated by one annotator, then the resulting files are checked by another annotator. \n\nThe segmentation guidelines, like POS guidelines and bracketing guidelines, have been revised several times during the project. So far, we have released all three versions on our web site: the first draft was completed in December 1998, after the first pass of word segmentation and POS tagging; the second draft in March 1999, after the second pass of word segmentation and POS tagging. This document, which is the third draft, is revised after the second pass of bracketing. The major changes in the third draft, compared with the previous two drafts, are (1) we add an introduction chapter in order to explain some rationale behind the guideline, (2) we add the gloss to the Chinese words in the guidelines,1 and (3) we also turn the guidelines into a technical report, which is published by the Institute for Research in Cognitive Science (IRCS) of the University of Pennsylvania.\n\n### 1.1 Notion of *word*\n\nThe difficulty in defining the notion of word is not unique to Chinese,2 but the problem is certainly more severe for Chinese for a number of reasons. First, Chinese is not written with word delimiters so segmenting a sentence into \"words\" is not a natural task even for a native speaker. Second, Chinese has little inflectional morphology to ease word identification. Third, there is little consensus in the community on difficult constructions that could affect word segmentation. For instance, the segmentation of verb resultative compounds depends on the syntactic analysis of the construction. One view on how a verb resultative compound is formed says that a simple sentence with a compound is actually bi-clausal and the compound is formed by movement, therefore, the compound should be treated as two words. Another view believes that the compound is formed in the lexicon, and therefore should be one word. The segmentation of the verb resultative compounds depends on which view we adopt for this construction. Fourth, many monosyllabic morphemes that used to be able to stand alone in non-Modern Chinese become bound in Modern Chinese. The influence of non-Modern Chinese makes it difficult to draw the line between bound morphemes and free morphemes, the notions which could otherwise have been very useful for deciding word boundaries.\n\n\nOur approach is based on both linguistic and engineering consideration. The notion word in our Treebank is roughly a syntactic atom as defined in [SW87], that is, anything that can be inserted into an X° position in syntax. This includes both compounds and simple words.\n\n### 1.2 Tests of wordhood\n\n\nWhat tests can be used to decide whether a string of hanzi[Chinese character] is a word or not? Without loss of generalization, we assume the string that we are trying to segment is X-Y, which has two morphemes X and Y. The following tests for establishing word boundaries have been proposed by various authors:\n\n\n- Bound morpheme: a bound morpheme should be attached to its neighboring morpheme to form a word when possible.\n\n\n- Productivity: if a rule that combines the expression X-Y does not apply generally (i.e., it is not productive), then X-Y is likely to be a word.\n\n\n- Frequency of co-occurrence: if the expression X-Y occurs very often, it is likely to be a word.\n\n\n- Complex internal structure: strings with complex internal structures should be segmented when possible.\n\n\n- Compositionality: if the meaning of X-Y is not compositional, it is likely to be a word.\n\n\n- Insertion: if another morpheme can be inserted between X and Y, then X-Y is unlikely to be a word.\n\n\n- XP-substitution: if a morpheme can not be replaced by a phrase of the same type, then it is likely to be part of a word.\n\n\n- The number of syllables: several guidelines [LTS93, Chi96] have used syllable numbers on certain cases. For example, in [LTS93], a verb resultative compound is treated as one word if the resultative part is monosyllabic, and it is treated as two words if the resultative part has more than one syllable.\n\n\nAll of these tests are very useful. However, none of them is sufficient by itself for covering the entire range of difficult cases. Either the test is applicable only to limited cases (e.g., the XP-substitution test) or there is no objective way to perform the test as the test refers to vaguely defined properties (e.g., in the productive test, it is not clear where to draw the line between a productive rule and a non-productive rule). For more discussion on this topic from the linguistics point of view, please refer to [Pac98, SW87].\n\n\nSince no single test is sufficient, we chose a set of tests for our segmentation guidelines which includes all of the ones mentioned except for the productivity test and the frequency test. Rather than have the annotators try to memorize the entire set and make each decision from these principles, in the guidelines we spell out what the results of applying the tests would be for all of the relevant phenomena. For example, for the treatment of verb resultative compounds, we select the relevant tests (e.g., the number of syllables and the insertion test), and give several examples of the results of applying these tests to verb resultative compounds. This makes it straightforward, and thus efficient, for the annotators to follow the guidelines.\n\n### 1.3 Compatibility with other guidelines\n\n\nWe have studied other groups, guidelines, such as the Segmentation Standard in China [LTS93] and the one in Taiwan [Chi96], and tried to accommodate them in our guidelines if possible.\n\n\nSince the final result of the Treebank is a list of bracketed sentences, our guidelines have some flexibility with regards to the segmentation of certain constructions. For example, the string 走上来[walk up] is treated as two segments in [LTS93], but one segment in [Chi96]. In our Treebank, we will segment it into two parts, and then group them together as a compound ——that is, (走[walk]/V 上来[up]/V)/V. We call 走上来 a word with internal structures. Out annotation, in this case, is compatible with both [LTS93] and [Chi96]. The comparisons of these three guidelines can be found in Appendix A.\n\n\nNote: For the sake of annotation efficiency, the grouping of the words with internal structure is done at bracketing stage, rather than at the segmentation stage. In this document, we show the grouping format, but keep in mind that the format is the one AFTER the bracketing is completed. For example, we consider 走上来[walk up] 2us one word. It is segmented into “走[walk]/V 上来[up]/V” at the segmentation stage, and it will be grouped into (走[walk]/V 上来[up]/V)/V at the bracketing stage. In the paper, we just say 走上来[walk up] should be annotated as (走[walk]/V 上来[up]/V)/V.\n\n\nMost disagreements among these three guidelines do not make much difference to parsing or sentence interpretation. For most patterns for which the guidelines give different treatments (e.g., numbers and reduplication strings), simple conversion programs can be written to convert the data from one format to another.\n\n\nOur goal is: in the final output, the word boundary (the highest-level X° in the parse tree) should be as accurate as possible, while the internal structure serves as a bridge for the resource sharing with other systems.\n\n### 1.4 Treatment for unclear cases\n\nThere are two types of unclear cases:\n\n- A construction is easy to identify but there is no consensus on its treatment. \n  Ex: A-not-A, V-de construction, V-R, potential form (i.e., V-de-R).Our approach: we will choose one analysis, and annotate the data according to that analysis. Make sure that the annotation is easy to convert to the structures for other analyses if necessary.\n- Two constructions are difficult to tell apart by existing tests.\n  Ex: some N+N are compounds, others are phrases.\n\n\nOur approach: for the sake of consistency and efficiency, we don^ disambiguate the two constructions unless making the distinction is crucial for various reasons.\n\n### 1.5 Organization of this guidelines\n\n\nThe guidelines are organized according to the internal structure of the corresponding expressions (e.g., a verb resultative compound is represented as V+V, while a verb-object expression is as V+N), so it is easy for the annotators to search the guidelines for reference. The Part-of-speech tags used in this paper are identical to the ones used in the POS tagging task except that the tags for verbs are merged into V and the ones for nouns are merged into N. For the descriptions of the complete POS tagset, please refer to our Part-of-Speech Tagging Guidelines for the Penn Chinese Treebank (3.0). The list of POS tags can be found in Appendix B.\n\n\nIn this guidelines, we list mainly the decision for each case without going into detail elaborating other alternatives and the reasoning behind each decision.\n\nChapter 2 Specification\n---------\n\n\nIn this chapter, we assume that a sentence has been segmented into large chunks, and the next step is to decide whether each chunk should be further divided. The chapter is arranged by the potential POS of the chunk if the chunk is a word. To search through the section, first use the ^POS^ of the chunk to find the subsection, then use the ^word^ formation information to find the subsection; or simply use the “word” formation information.\n\n### 2.1    Common noun: NN\n\n#### 2.1.1    Name of relative\n\n\nTreat it as one word.\n\n\nEx:三叔[uncle]/NN，表叔[uncle]/NN,.大姑父[uncle]/NN.\n\n#### 2.1.2 CD+N\n\n\nIf a measure word can be inserted between CD and N without changing the meaning, tag it as CD+N; otherwise, tag it as one word (N).\n\n\nOne word:三排[the third platoon]/NN，一方[one side]/NN,三者[three entities]/NN, 一行[a group traveling together]/NN，2 1 世纪[the 21st century]/NT.\n\n\nTwo words: — [one]/CD 学生[student]/NN.\n\n#### 2.1.3 DT+N\n\n\nTreat it as one word if both DT and N are monosyllabic and either DT or N is bound; otherwise, treat it as two words.\n\n\nSometimes, it is difficult to decide whether a morpheme is bound or not because of the influence of non-Modern Chinese. To be consistent, we maintain a list of nouns and a list of determiners. If a morpheme is in one of the lists, we consider it as bound:\n\n\n- monosyllabic bound nouns: /^.[school], ^ (when it means the earth).\n\n\n- monosyllabic bound determiners:当[this/that]\n\n\nWe also treat 本人[oneself]/NN as one word and tag it as NN.\n\n\nOne word:本人[oneself]/NN,本校[our school]/NN,全球[whole world]/NN,当地[the place mentioned]/NN,当今[present time]/NT,当代[the contemporary era]/NN.\n\n\nTwo words:本[one’s]/DT 单位[organization]/NN.\n\n#### 2.1.4    PN+N\n\n\nTreat it as one word if both PN and N are monosyllabic and N is bound; otherwise, treat it as two words.\n\n\nIn this case, the current list of bound nouns is:校[school].\n\n\nOne word:我校[my school]/NN.\n\n\nTwo words:我[my]/PN 单祆[organization]/NN.\n\n#### 2.1.5    JJ+N\n\n\nThe pattern is: X+N, where X modifies the N, and X is either a JJ or a prefix.\n\n\nNote: JJ+N can be a phrase. For example, in one of the files we annotated,全国性[nationwide]/JJ 网络[network]/NN is extended into “全国性[nationwide]/JJ 观测[observe]/VV 苏梅克一列桌/NR 9 号[number 9]/NN 彗星[comet]/NN 撞击[hit]/W 木星[Jupiter]/NN 的/DEC 网络[network]/NN”.\n\n\nSegment X+N according to the type of X:\n\n\n- X is a prefix: treat X+N as one word.[1](#bookmark93) A list of prefixes:啊，非[non-].\n\n\nEx:啊爸[father]/NN,非商业化[non-commercial]/JJ 宗旨[purpose]/NN.\n\n\nA list of JJs:原[former],前[former]\n\n\nEx:原[former]/JJ 在[at]/P 华[China]/NR 老挝[Laos]/NR 难民[refugee]/NN;\n\n\n前[former]/JJ 民主德国[German Democratic Republic]/NR.\n\n\n- X is a non-predicate adjective:[2](#bookmark94) if both JJ and N are monosyllabic, tag it as one word; otherwise, treat it as JJ+N.\n\n\nOne word:女人[woman]/NN.\n\n\nTwo words:共同[mutual]/JJ 利益[interest]/NN.\n\n\n- X is an adjective: treat it as one word if X or N is bound or the meaning of X+N is non-compositional. For unclear cases, if both JJ and N are monosyllabic, treat JJ+N as one word (e.g” 鲜花[fresh flower]/NN,强队[strong team]/NN, •红茶[black tea]/NN,好评[favorable comment]/NN).\n\n\nOne word:小媳妇[daughter-in-law]/NN,大洲[continent]/NN,大海[sea]/NN.\n\n\nTwo words:厚[thick]/JJ 书[book]/NN.\n\n#### 2.1.6    LC+N\n\n\nIf both LC and N are monosyllabic, treat the string as one word, and tag it as NN or NT according to its meaning.\n\n\nEx:前院[front yard]/NN,前天[day before yesterday]/NT,左肩[left shoulder]/NN.\n\n#### 2.1.7    N+LC\n\n\nTreat N+LC as one word if:[3](#bookmark95)\n\n\n- the N and LC are monosyllabic; and\n\n\n- in this context, the N is non-referential or bound; and\n\n\n- in this context, the N can not be modified by Det-M or other modifiers.\n\n\nOtherwise, treat it as two words.\n\n\n- One word (some of them might be two words in other context):室内[indoor](室内[indoor]/NN 训练[training]/NN),台下[off stage],眼前[at present],境外[foreign](境外[foreign]/NN 集团[group]/NN 境内外[domestic and international /NN,海外[oversea](海外[oversea]/NN 市场[market]/NN)，背后[at the back]/NN,天下[world]/NN,国内[domestic]/NN,午后[afternoon]/NT,赛前[before the contest]/NT.\n- Two words:中午[noon]/NT 以后[afterwards]/LC.\n\n#### 2.1.8    N+N: N1 modifies N2\n\n\nIf it is 1-hl or 2+1 (i.e., N1 has one or two hanzi and N2 has one hanzi), treat N1+N2 as one word (i.e.，we treat all monosyllabic nouns as potential “接尾词. If a noun with no more than 2 hanzi is followed by multiple    \"接尾词\"    monosyllabic noun attaches to the preceding    the whole string is treated as one word (e.g•，物理学家[physicist]/NN).\n\n\nFor other cases, the string is treated as two words.\n\n\n- One word:北京市[Beijing]/NR,研究室[research lab]/NN,发展史[developmental history]/NN,始祖鸟[proto-bird]/NN, 残疾人[the physically challenged]/NN, 清晰度[visibility]/NN, [sense of urgency]/NN, 大奖赛[tournament]/NN,太阳系[the solar system]/NN.\n- Two words:北京[Beijing]/NR 大学[University]/NN,坑具[toy]/NN 工厂[factory]/NN,合作[collaboration]/NN, 领城[area]/NN,史学[history]/NN 研究[research]/NN.\n\n#### 2.1.9    PN+LC\n\n\nIf both PN and LC are monosyllabic, treat PN+LC as one word and tag it as NT or NN.\n\n\nOne word:此间[here]/NN,此前[before this]/NN,其中[among them]/NN,何时[when]/NT.\n\n\nTwo word:这[this]/PN 以后[after]/LC.\n\n#### 2.1.10    V+N\n\n\nIn this pattern, we assume V is VV (For VA+N, please refer to the section for JJ+N) If V modifies N, treat V+N as one word and tag it as a noun.\n\n\none word:烤肉[barbecue]/NN，炒菜[stir-fried dishes]/NN,证明信[certificate]/NN,讨论会[symposium]/NN.[4](#bookmark96)\n\n### 2.2 Proper Noun: NR\n\n\nCurrently, if the proper noun is composed of multiple words, we don^ group them.\n\n#### 2.2.1    Personal name\n\n\nTreat it as one word. Don't give the internal structure unless there is a space between two names (in foreign alphabet).\n\n\nEx:张胜利/NR,卡尔[Karl].马克斯[Maxx]/NR, John/NR Smith/NR.\n\n#### 2.2.2    Personal name with affixes\n\n\nTreat it as one word.\n\n\nEx:老张/NR,张老/NR\n\n#### 2.2.3    Personal name + title\n\n\nTreat it as two words.\n\n\nEx:张/NR 教授[professor]/NN,张/NR 李/NR 两[two]/CD 位/M 教授[professor\"^\n\n#### 2.2.4    Name of Organization/Country/School/..\n\n\nIf the pattern is N1+N2, where N2 is a common noun, then if N2 is monosyllabic, treat N1+N2 as one word, else treat N1+N2 as two words.\n\n\nSimple names:北京市[Beijing]/NR,黄河[the Yellow River]/NR,沙市[Sha City]/NR,黑龙江省[Heilongji^ Province]/NR.\n\n\nComplex names:北京[Beijing]/NR.大学[University]/NN,北京[Beijing]/NR 第一[First]/OD 服装厂[Clothing Factory]/NN，美国[the United States]/NR 国会[Congress]/NN.\n\n#### 2.2.5 NR+NR: coordination without conjunction\n\n\nTreat it as two words.\n\n\nEx:中[China]/NR 美[the United States]/NR,中[China]/NR 美[the United States]/NR 关系[relation]/NN, 东[Eastern Asia]/NR 新[Singapore]/NR 澳[Macao]/NR.\n\n### 2.3 Temporal noun: NT\n\n\nThe names of years/months/day/hour and so on axe words.\n\n\nEx: 1998年[1998]/NT 3月[March]/NT 21 日[21st]/NT, 5点钟[5 o’clock]/NT，初一[the first day of a lunar month]NT，i年[last year]/NT.\n\n#### 2.3.1 CD+N\n\n\nIf CD+N is the name of a time, treat it as one word (NT). If it is the count of the time, treat it as two words (CD+M).\n\n\nOne word: 1998年[1998]/NT, 5点钟[5 o，clock]/NT, 9 0 年代[the 90s]/NT,\n\n\nTwo words: 3/CD 年[year]/M, 3/CD 个/M 月[month]/NN.\n\n### 2.4 Localizer: LC\n\n\nLocalizers are separated from the noun that it attaxJies to except for the case mentioned in Section 2.1.7 (i.e., N+LC).\n\n\nA localizer is either one or two syllables:\n\n\n- monosyllabic localizers: e.g.内[in],后[after].\n\n\n- bisyUabic localizers: e.g.之间[between],以来[since],以后[afterwards],左右[around].\n\n### 2.5 Pronoun: PN\n\n\nTreat it as one word.\n\n\nEx:他们[they]/PN,他自己piimself]/PN，自己[self]/PN.\n\n### 2.6    Determiner: DT\n\n\nWe separate DTs from the succeeding words.\n\n\nEx:这[this]/DT 三[three]/CD 个/M 人[people]/NN,各[each]/DT 国[nation]/NN.\n\n\nCurrently, we treat 这些[these] as one word, and tag it as DT.\n\n\nSome examples of bisyllabic DTs:全体[all]，其余[the rest], —切[all],这些[these],那些[those],所\n\n### 2.7    Cardinal number: CD\n\n\nTreat it as one word. Note: the internal structure of a CD is very easy to recover if needed.\n\n\nSome examples:\n\n\n- Pure numbers: 一亿三千万[one hundred and thirty million]/CD, 30.1/CD, 123,456/CD, 35.6%/CD, 30万[three hundred thousand]/CD, 30几[thirty odd]/CD.\n\n\n- Estimation:三四十[between thirty and forty-nine]/CD 岁[years old]/M.\n\n\n- CD + X + CD(5.5.4): X is a morpheme such as 余[odd],分之[fraction]，点[point]•三十几亿[three billion odd]/CD,三分之一[one third]/CD,三点一[three point one]/CD,好几[multiple]/CD 个/M.\n\n\n- CD+X: X is a morpheme such as 余[odd],来[over/odd]:四千一百余[four thousand and one hundred odd]/CD 人[people]/NN,三十雇[about thirty]/CD 个/M.\n\n### 2.8 Ordinal number: OD\n\n\nTreat it as one word.\n\n\nEx:第一[first]/OD,第三十一[thirty-first]/OD.\n\n### 2.9 Measure word: M\n\n\nTreat the measure word, including a reduplicated or a compound measure word, as one word. Treat the string such as 分钟[minute] as one word.\n\n\nEx:杯[cup]/M,杯杯[cup-cup]/M,架次[number of flights]/M,分钟[minute]/M.\n\n### 2.10 Verb: VA, VC, VE, and VV\n\n#### 2.10.1 Reduplication: A A, ABAB, A ABB, A AB, ABB，ABAC\n\n\nTreat it as one word.\n\n\n- AA, A is a verb: AA/V \n  Ex:看看[see]/W，红红[vivid red]/VA.\n\n\n- ABAB: AB is a verb: ABAB/V\n  Ex:研究研究[research]/VV,雪白雪白[snow white]/VA.\n\n\n- AABB, AB is a verb: AABB/V\n  Ex:来来往往[come and go]/W,高髙兴兴[happy]/VA Note: most of the time, AA or BB is not a word.\n\n\n- AAB(except for AA-看 in 2.10.2):AAB/V \n  Ex:蒙蒙亮\n  Note: most of the time, AA or B is not a word.\n\n\n- ABB: ABB/V\n  Ex:绿油油[bright green]/VA，红彤彤[bright red]/VA.\n  Note: most of the time, A or BB is not a word.\n\n\n- ABAC, etc.: ABAC/V\n  Ex:马里马虎[careless]/VA,有条有理[orderly]/VA，一清二楚[very clear]/VA.\n\n#### 2.10.2 “Reduplication”： AA-kan, A-one-A, A-le-one-A，A-le-A\n\n\nTreat it as one word with internal structure.\n\n\n- AA-看：（AA/V 看/V)/V\n  Ex:(说说[say]/W 看/VV)/V.\n  The basic meaning of the word 看 is to “see”，but in this context，it roughly means \"try to do something\".\n\n\n- A-one-A: (A/V one/CD A/V)/V \n  Ex:(想[think]/W — [one]/CD 想[think]/VV)/V.  \n\n- A-le-A: (A/V le/AS A/V)/V \n  Ex:(想[think]/W 了/AS 想[think]/W)/V.\n\n\n- A-l^on^A: (A/V le/AS one/CD A/V)/V \n  Ex:(想[think]/W 了/AS — [one]/CD 想[think]/W)/V.\n\n\nNote: V+CD+M is treated as three words, e.g. [look]/V [one]/CD [eye]/M (take a look).\n\n#### 2.10.3 A-not-A\n\nTreat it as one word with internal structure.\n\nEx:(来[come]/VV 没[not]/AD 来[come]/VV)/V，（高[happy]/VA 不[not]/AD 高兴[happy]/VA)/V， (喜[like]/VV 不[not]/AD 喜欢[like]/VV)/V.\n\n#### 2.10.4 AD+V\n\n\nIf one or more of the following hold, treat AD+V as one word (V):\n\n\n- no free word can intervene between AD and V,\n\n\n- the V cannot be a predicate without the AD,\n\n\n- the subcategorization frame of AD+V is different from that of the V.\n\n\nOtherwise, treat it as two words.\n\n- One word:胡说[talk nonsense],胡来[mess things up],敬献[present with great respect],尚余[remain]\n  [(尚余[still remain]/VV 七十五[75]/^D 名)M 难民[refugee]〉NN)，历任[have served successively as],并列[tie处 不喪[not afraid o月.\n\n- Two words:已经[already]/AD 采取[take]/VV,不[not]/AD 应该[should]/VV，没[not]/AD 完成[complete]/VV.\n\n#### 2.10.5 MSP+V\n\n\nIf the V can not be a predicate without the MSP, treat MSP+V as one word (V).\n\n\nOne word:以期[in order to]/W (以期[in order to]/W 在[at] 与[with] 美国[the United States]、 瑞典[Sweden]、挪威[Norway]、这些 [these]、世界[world]、强队[strong teams] 、交锋[competition] 、中[during]...).\n\n#### 2.10.6 N+V\n\n\nSome subject-predicate strings Coin be either a phrase or a word depending on the context.\n\n\nIf a VP-modifier can be inserted between the subject and the predicate part and the “subject” is referential, then the string is a phrase, otherwise it is a word.\n\n\nOne word:头疼[headache]/VA in “他[he]/PN 让[make]/VV 我[me]/PN 很[very]/AD 〈He gives me a headache}”.\n\n\nTwo words:头[head]/NN 疼[ache]/VA in “我[I]/PN 头[head]/NN {很[very]/AD}疼[ache]/VA〈I have a headache}’’.\n\n#### 2.10.7 V+N\n\n\nIf the V and the N axe separated (by the aspect markers, by the modifiers of the N, or because the V is reduplicated), treat V+N as two words.\n\n\nIf the V and the N are adjacent,[6](#bookmark98)\n\n\n- If V-N is semantically transitive and its object can occur after N only when VN are adjacent (therefore the V is not a ditransitive verb)，treat V+N as one word (e.g.,投资[invest]/VV, 出席[be present]/W,关心[care]/VV,为期[scheduled for a specific duration of time]/W).\n\n\n- If V and VN have similar meaning and both axe semantically intransitive, treat VN as one word (e.g.,睡觉[sleep]/VV).\n\n\n- If N is “bound”, treat VN as one word (e.g.,游泳[swim]/VV,无望[hopeless]/VV,无效[invalid]/VV, 无法[unable to]/VV,辞职[resign]/W).\n\n\n- If V-N is 1+1 AND the meaning is non-compositional，treat V-N as one word (e.g.,念书[study]/VV, 流血[bleed]/VV).  \n\n\nExamples of V-N as two words:访[visit]/VY 华[China]/NR in the sentence 他[he]/PN 曾[previously]/AD 七[seven]/CD 次[time]/M 访[visit]/W 华[China]/NR〈He has visited China seven times、\n\n#### 2.10.8    V+R\n\n\nThe tests for verb resultative compounds (V-Hs): both V and R are verbs and the potential forms (V-de-R, V-not-R) exist. So our definition of V-R includes resultative and directional verb com-pounds (e.g.，看见[see] and 走上来[walk up])，but it does NOT include words such as 改善[improve] and 鼓动[agitate].\n\n- We treat it as one word. For the sake of compatibility with other guidelines, we give the internal structure for the words if they have more than 2 syllables or if the R is the following:完[finish]/W.\n\n- Words without internal structure:吃掉[eat up]/VV,看见[see]/W，擦净[wipe clean]/VV.\n\n- Words with internal structures:(做[do]/VV 完[finish]/W)/V,(擦[wipe]/VV 干净[clean]/VV)/V, (认识[realize]/W 到[reach]/VV)/V.\n\n#### 2.10.9    Potential form: V-de/bu-R\n\n\nWe treat it as one word.\n\n- If V-R exists, give the internal structure of V-de/bu-R, otherwise, don^ give one.\n  Ex: words with internal structure:(擦[wipe]/VV 不[not]/AD 冷[clean]/VA)/V，（擦[wipe]/VV 得/DER 净[clean]/VA)/V.    \"\n\n- words without internal structure:吃不了 [unable to eat anymore]/W，买不起[cannot afford]/VV.\n\n\nNote: the string WV de R,? can be ambiguous between potential form and V-de construction. For example, “这[this]张[M]桌子[table]擦[wipe]得pER]干净[clean]吗[SP]?’’ can either be a potential form (which means Can this table be wiped clean?), or it could be a V-de construction (which means Has the table been wiped clean?). The two constructions have different syntactic structures. Normally, we can tell them apart by meaning, by the position of the object or by checking whether adverbs can be inserted between the de and the R.\n\n#### 2.10.10    V+DIR\n\n\nSee Section 2.10.8 (i.e., the section for V+R).\n\n\nWords with internal structure:(走[walk]/VV 出去[out]/VV)/V,(走[walk]/VV 不[not]/AD 出去[o叫 Words without internal structure:走出[walk out of]/VV,想出[think of]/VV.\n\n#### 2.10.11    V+AS\n\n\nTreat it as two words.[7](#bookmark99)\n\n\nEx:走[walk]/VV 了/AS.\n\n#### 2.10.12 V+DER\n\n\nThe pattern is V-de in V-de construction. We treat V-de as two words.[8](#bookmark100) Ex:走[walk]/VV 得/DER (走[walk]/W 得/DER 很[very]/AD 快[fast]/VA).\n\n#### 2.10.13 Verb coordination without conjunctive words\n\n\nIf the pattern is 1+1, treat it as a word; otherwise, treat it as multiple words.\n\n\nOne word:修建[build]/VV.\n\n\nTwo words:宣传[propagate]/VV 鼓动[agitate]/VV.\n\n#### 2.10.14 V+coverb\n\n\nThe pattern is V+X, where X is monosyllabic and it is either a P or a V.[9](#bookmark101)\n\n- We first decide whether V+X is a word. If it is, we use its syllable count to decide whether to show its internal structure. That is, if V is monosyllabic, don^ give the internal structure;\n\n\notherwise, give the internal structure.\n\n\n- treat V+X as one word if X is in the following list:给[give];为[become],成[become]，作[treat as],到[arrive],出[out];自[from],向[toward],入[in],以[with].\n  Ex:\n\n  - 给[give]:送给[give/send to]/VV，交给[hand in]/VV，（赠送[give as a gift to]/VV 给[give]/VV)/V.\n  - 为[to],成[become/into]，作[do/as],到[arrive],出[out]:(翻译[translate]/VV 成[become] 当作[treat as]/VV,起到[take effect]>V，找到[find]/VV,(认识[realize]/VV 到[reach]/VV)/V,决出[decide victors]/VV.\n  - 自[from],向[toward],入[in],以[with]:来自[come from]面向[face toward]/ into]/VV,迈向[step toward]，VV,报以[respond with]/VV，加以[supplement with]/VV.\n\n\n\n- treat V+X as two words if X is in the following list:在[at]，似[like].\n\n  - Ex:生[to be born]/W 在[at]/P，坐[sit]/W 在[at]/P，留[stay]/W 在[at]/P，深[deep]/VA 似pike]/P 海[sea]〉NN.\n\n-  treat V+X as one word or two words (V+P) according to the meaning of the X, if X is in the following list:于[at].\n\n  - If 于 in V + 于 can be replaced by 在[at], tag V+于 £us two words (V+P). Otherwise, tag it as one word.\n  - One word:等于[equal to]/VV,缘于[due to]/VV,大于[bigger than]/VV,小于[smaller than]/VV, 无助于[of no help to]/VY 低于[lower than]/W,利于[be beneficial for]/W,有利于[be beneficial for]/VV.\n  - Two words:生[to be born]/W 于[at]/P,建[build]/VV 于[at]/P.\n\n\n#### 2.10.15 Others\n\nGenerally, in X+V(or V+X) where X modifies V, if X cannot modify other verbs, or V cannot be a predicate without the X, treat X+V as one word.\n\n- Ex:以期[in order to]/W\n\n### 2.11 Adverb: AD\n\n\nAdverbs are separated from the XP that it modifies.\n\n\nAdverbs that modify numbers:近[almost]/AD 三十[thirty]/CD，5[five]/CD 分[minute]/M 多[odd]^ 钟[minute]/NN.[10](#bookmark102)\n\n\nThe string such as fe^[extremely big] is an adverb when it modifies VPs, not AD+VA, because the VA(大[big]) cannot modify VPs without the AD(极[extremely]).\n\n#### 2.11.1    Reduplication\n\n\nWhen VA(or AD) reduplicates, the resulting word can be an AD.\n\n\nEx:妤好[well]/AD 干[do]/W，常常[always]/AD,仅仅[only]/AD.\n\n#### 2.11.2    DT+M/N\n\n\nThe following are tagged as ADs when they modify VP/S:这样[this way]/AD (这样[this way]/AD 做[do]/W),同机[on the same airplane]/AD (同机[on the same airplane]/AD 到达[arrive]/W).\n\n#### 2.11.3    P+PN\n\n\nWe treat the following as two words:为[for]/P 此[this]/PN.\n\n#### 2.11.4   P+N\n\n\nThe following can be seen as frozen PPs. Since they have the same function as the ADs, we treat them as words, and tag them as ADs:迄今[until now],沿途[on the way],即席[impromptu]， 为何[why](为何[why]/AD 愈演愈洩[get worse and worse]/VA),为什么[why]/AD 来[come]/VV\n\n#### 2.11.5 PN+LC\n\n\nIf a PN+LC totally loses the function of an NP and the string acts like an adverb, treat it as an adverb.\n\n\nWe treat the following as ADs:此外[in addition]/AD.\n\n#### 2.11.6 Others\n\n\nIf in that context a string totally loses the function of the XP(where X is the head of the string) and the string behaves like an adverb, tag it as AD.\n\n\nWe treat the following as ADs:进一步[a step further]/AD.\n\n### 2.12 Preposition: P\n\n\nSeparate it from NP/S that follows it.\n\n\nMost prepositions are monosyllabic. Some common bisyllabic prepositions are:为了 [in order to],随着[along with],沿着[along],本着[in conformity with],鉴于[due to],除了[except],经过[through]，\n\n\n作为[being/regard as],截止[until].\n\n\nWhen a coverb follows a verb, we have to decide whether the word is part of a verb compound. A list of such coverbs are:于，给，为， See Section 2.10.14 for details.\n\n### 2.13 Subordinating Conjunction: CS\n\n\nSeparate it from the XP that follows it.\n\n\nStrings such as 只有[only] is ambiguous:\n\n\n- CS:只有[only if]/CS ...才[then]/AD ....\n\n\n- AD+VE:他[he]只[only]/AD 有[have]/VE 三[three]/CD 块/M 钱[money]/NN〈He only has three dollars).\n\n### 2.14 Conjunction: CC\n\n\nSeparate it from the XPs that it conjoins.\n\n\nEx:和[and]/CC,与\n\n### 2.15 Particle: DEC, DEG, DEV, DER，AS, SP，ETC，and MSP\n\n\nSeparate it from the XP that it attaches to.[11](#bookmark103)\n\n\nMost particles axe monosyllabic. One of bisyllabic particles is 的话[if so]/SP.\n\n### 2.16 Interjection: IJ\n\n\nTreat it as one word.\n\n\nEx: 哈[expressing satisfaction and so on]/IJ.\n\n### 2.17 Onomatopoeia: ON\n\n\nTreat it as one word.\n\n\nEx:哈哈[sound of laughter]/ON,哔啦啦[sound of water/rain]/ON\n\n### 2.18 Other noun-modifier: JJ\n\n\nSeparate it from the measure word (M) or the noun (N) that it modifies. Ex:三[three]/CD 大[big]/JJ 杯[glass]/M 水[water]/NN\n\n\n\"When JJs modify nouns, the JJs can be adjectives,区别词(非谓形容词)，or “phrasal words”. Most of the <4phrasal words,? have two parts: X+Y, both X and Y are monosyllabic, and X or Y is the short-form of the corresponding words. Some examples of the \"phrasal words\" are as follows:\n\n#### 2.18.1 V+N\n\n\nV+N:随军[being with the army]/JJ.妓女[prostitute]/NN,旅英[having studied in England]/JJ 学者[scholar]/N^ 成套[forming a complete set]/JJ 设备[equipment]/NN,.发稿[sending manuscripts to press]/JJ 时间[time]/NN， ^^[receiving award]/JJ #i[scholar]/NN, 驻华[being stationed in China]/JJ  使馆[embassy]/NN, ^4[giving benefit]/JJ 国家[nation]/NN，\n\n#### 2.18.2 AD+VA\n\n\nAD+VA:最新[the newest]/JJ 消息[news]/NN,超大[extra-large]/JJ 规模[scale]/NN 集成[integrate]/NN 电路[circuit]/NN，较大[relatively big]/JJ 增长[growth]/NN.\n\n\nThe common “AD”：最[the most],超[extra-],较[relatively].\n\n#### 2.18.3 VA+N\n\n\nVA+N/M:高层[high-ranking]/JJ 人士[official]/NN,高速[high speed]/JJ 公路[highway]/NN，大幅[big size]/JJ 标语[slogan]/NN.\n\n#### 2.18.4 CD+N\n\n\nCD+N/M:两国[two~nation]/JJ 关系[relation]/NN，多国[multi-nation|/JJ 部队[troop]/NN\n\n#### 2.18.5 P+N\n\n\nP+N/LC:对外[foreign]/JJ 政策[policy]/NN\n\n#### 2.18.6 Others\n\n\nothers:关贸[tariff and trade]/JJ 总协定[treaty]/NN,年均[annual average]/JJ 增长率[growth rate]/NN, 上述[aforementioned]/JJ 三[three]/CD 国[nation]/NN,历届[all previous sessions]/JJ 世界[world]/NN 体操[gymnastics]/NN 大赛[championship]/NN,有关[related]/JJ 方面[parties]/]S[N.\n\n### 2.19 Punctuation: PU\n\n\nTreat it as one word, except when it is part of another word; for example, 4V? in a number (e.g., 123,456/CD) or in proper names，（e.g.，卡尔[Karl].马克斯[Marx]/NR).\n\n### 2.20 Foreign word: FW\n\n\nTreat it as one word, except when it is part of another word (e.g.,    [Karaoke]/NN).\n\n### 2.21 Others\n\n#### 2.21.1 Idioms\n\n\nThe frozen idioms (成语）axe treated as words when they function as an NP or a VP.\n\n\nEx:各有所好[each has his likes and dislikes]/V, 一比高低[compete]/V.\n\n#### 2.21.2 Telescopic strings\n\n\nTelescopic strings are treated as one word if they are not too long (less than four characters). K it is too long, segment them according to pauses.\n\n\nShort strings:进出口[imports and exports]/NN 贸易[trade]/NN,国内外[foreign and domestic]/NN 形勢[situation] /NN.\n\n\nLong strings:交响[symphony]/JJ 乐团[orchestra]/NN，北京[Beijing]/NR 市长[mayor]/NN.\n\n#### 2.21.3 Short form\n\n\nEx:三好[three-merit]/JJ 学生[student]/NN，教科文[education，science,紐d culture]/NN 组织[organization] (UNESCO),七中[the seventh central government]/NN 全会[convention]/NN.\n\n  \n\nShortened part is treated as one word. If the shortened part is longer than 3 syllables, segment them according to phonologic evidence (e.g., pauses). The structure of the short form might be different from that of the full form.\n\n\n\n\nChapter 3 Collocation with Some Morphemes\n---------\n\n### 3.1 Strings with zhe5\n\n\nSome prepositions end with 着.\n\n\nEx:随着[along with]/P.\n\n### 3.2 Strings with zhi1\n\n\nzhi+LC, where LC is monosyllabic, is treated as one word (LC).\n\n- Ex:之外[aside from]/LC,之中[among]/LC.\n- zhi1+CD is treated as DEG+CD (e.g.,方法[method]/NN 之/DEG 一[one]/CD,方法[method]/NN 之/DEG 三[three]/CD).\n\n\nFor simplicity,之一 in a sentence such as 中国是发展中国家之一 is treated as one word and tagged as an NN.\n\n\nzhi1+N is treated as DEG+N (e.g.,少年[Children]/NN 之/DEG 家[Club/Center]/NN).\n\n### 3.3 Strings with bu4\n\n\nIf X in X+不[not] (or 不[not]+X) must co-occur with bu4 or the meaning of X+不[not] is not com-positional, we treat X+bu4 as one word.\n\n\nWords that include bu4(不[not]):不到[less than](不到[less than] 5 分钟[minutes],不足[less than] (不足[less than] 5 公斤[kilogram])，不便[inconvenient]，不久[not before long].\n\n### 3.4 Strings with shi4\n\n\nFor simplicity, we treat 特别是[particularly]/AD as one word.\n\n### 3.5 Strings with xiel\n\n\nThe following axe treated as one word:    [these]/PN(or DT),    [some]/CD.\n\n### 3.6 Strings with you3\n\nV+有[have] is often a verb; for example,刻有[engraved with]/VV,真有possess]/VV,富有[rich]/VV. \n\nmei2you3(没有) is always treated as one word(VV or VE or SP).\n\n\nMany idioms include the word 有[have]; for example,若有所思[as if lost in thought]/W.\n\n\nThe following are two words:有[have]/V 所/MSP,仅[only]/AD 有[have]/V,有[have]/V 可能[possibility]/NN.\n\n\nThe following are ambiguous without the context:\n\n\n- you3-dian3(有点)：V[have]+M or AD[a little bit]\n  It is V+M when 点 can be dropped or replaced by 一点[a little bit].\n  you3-dian3 is an AD when it can be replaced by other degree adverbs such as ^[very] or when it is followed by a VP.\n\n  - 他[he]/PN 有点[a little bit]/AD 下不了 [unable to get off]/VV 台[stage]/NN〈He felt embarrassed}.\n  - 这[this]/DT 本/M 书[book]/NN 有[have]/V 点/M 意思[meaning]/NN〈This book is interesting〉.\n  - 这[this]/DT 本/M 书[book]/NN 有[have]/V 点/M 看头[worth reading]/NN〈This book is worth reading).\n\n\n\n- you3-de5(有的)：V[have]+DEC or DT[some]\n\n  - 他[he]有[have]/V 的/DEC 书[book]我[I]也[also]有[have]〈The books that lie has, I have, too〉.\n  - 有的[some]/DT 人[people]已经[already]走[leave] 了[AS]〈Some people have already left〉.\n\n\n\n- you3-xie1 (有些)：V[have]+M or DT[some]:\n\n  - 我[I]只[only]有[have]/VV 些[some]/M 旧书[old books]〈I only have some old books.}\n  - 他[he]不[not]像[like]有些[certain]/DT 人[people]专门[especially]爱[like]抬杜[argue]〈沿 like certain people who especially like to argue).\n\n\n\n- zhi3-you3(只有)：AD[only]+V[have] or CS[only if]:\n\n  - 你[you]只有[only]/CS 学习[learn]才[then]/AD 能[able to]改进[improve]工作[work]〈You can only improve your work by learning).\n  - 他[he]只[only]/AD 有[have]/VV 10 块[M].钱[dollars]〈He only has ten dollars〉.\n\n\n### 3.7    Strings with zai4\n\n\nOne word:正在[in the process of]/AD.\n\n### 3.8    Strings with zi4ji3\n\n\nAlways treat PN+zi4ji3 (自己[self]) as one word. Ex:他自己/PN.\n\nChapter 4 Common Collocations\n---------\n\n### 4.1 As one word\n\n\n- AD:迄今为止[皿til today]，迄今[皿til now]，进一步[one step further],越来越[more and more],同机[on the same airplane]，沿途[on the way],即席[impromptu].\n\n\n- DT:这些[these].\n\n\n- JJ:对外[foreign] (e.g.,对外[foreign]/JJ 政策[policy]/NN),各界[all circles]/JJ.\n\n\n- LC:之间[between]，在内[inside].\n\n\n- NN:其中[among them], —行[group traveling together].\n\n\n- P:为了[in order to].\n\n\n- V:来自[come from],面向[face toward],流入[flow in],迈向[step toward],报以[respond with],为期[scheduled for a specific duration of time]，有利于[be beneficial for].\n\n### 4.2 As two words\n\n\n- AD-like:并[yet]/AD 未[not]/AD.\n\n\n- CC-like:及[and]/CC 其[his/its/her]/PN，而[and]/CC 又[in addition]/AD.\n\n\n- DT-like:各[each]/DT 个/M.\n\n\n- NN-like:超大[extra-large]/JJ 祝模[scale]/NN，我[our]/PN 国[nation]/NN.\n\n\n- NT-like:零点[midnight]/NT 零一分[one]/NT〈one minute past midnight〉.\n\n### 4.3 Other cases\n\n\nV-V:(迎上[step forward]/W 前去[go forward]/VV)/V.\n\nAppendix A Comparison with Other Guidelines\n----------\n\n\nIn this appendix, we compare our guidelines with the guidelines from PRC [LTS93] and from Rocling [Chi96]. The grouping of words in our system is done in bracketing stage.\n\n\n|  | Ours | PRC | Rocling | Example |\n| --- | --- | --- | --- | --- |\n| Verb |  |  |  |  |\n| AA | AA | AA | AA | 看看 |\n| ABAB | ABAB | ABAB | ABAB | 研究研究 |\n| AABB | AABB | AABB | AABB | 高高兴兴 |\n| ABB | ABB | ABB | ABB | 绿油油 |\n| AAB(excl AA-看） | AAB | AAB | AAB | 蒙蒙亮 |\n| ABAC etc. | ABAC | ABAC | ABAC | 有条有理 |\n| AA-看 | (AA/V kan/V)/V | AA kan | AA kan |  |\n| A-yi-A | (A/V yi/CD A/V)/V | AyiA | AyiA | 走一走 |\n| A-l^A | (A/V le/AS A/V)/V | A le A | A le A | 走了走 |\n| A-le-yi-A | (A/V le/AS yi/CD A/V)/V | A le yi A | A le yi A | 走了一走 |\n| nonreduced A-not-A | (A/V not/AD A/V)/V | A not A | A not A | 喜欢不喜欢 |\n| reduced A-not-A | (A/V not/AD A/V)/V | A-not-A | A-not-A | 喜不喜欢 |\n| V-R(R is monosyl.) | v-r except v/V 完/V | v-r | v-r | 打破 |\n| V-R(R is bisyl.) | (v/V r/V)/V | v r | v r | 扫千净 |\n| V-de/bu-R | (v/V de/DER r/v)/V | v de r | v de r | 打得破 |\n| (V-R exists) | (v/V bu4/AD r/v)/V | v bu r | y bu r | 打不破 |\n| V-de/bu-R | y-de-r/V | ?? | y-de-r | 来得及 |\n| (V-R doesn’t exist) | v-bu-r/V | ?? | y-bu-r | 来不及 |\n| V-DIR | (v/V dix/V)/V | v dir | v-dir | 走上来 |\n| V-x-0 | v/V x/X o/N | v x n | v x n | 吃了饭 |\n| VO | depends | depends | depends | 关心，吃饭 |\n| V-de | y/V de/DER | v de5 | v de5 | 走得 |\n| V-AS | y/V as/AS | v as | v as | 走了 |\n\n**Table A. 1: Comparison with PRC’s and Rocling’s Guidelines**\n\n\n|  | Ours | PRC | Rocling | Example |\n| --- | --- | --- | --- | --- |\n| Nouns\n| Proper Names (NR) \nLstNm+Fst Nm | one seg | two segs | one seg | 王鸣 |\n| IstNm+title | name/NR title/NN | name title | name title | 王市长 |\n| NR +接尾词 | nr-nn/NR | depends | nr-nn | 北京市 |\n| NR + common noun | nr/NR nn/NN | nr nn | nr nn | 北京大学 |\n| complex names | several segs | depends | several segs | 北京第一服装厂 |\n| Common nouns N+men5 | one seg | one seg | two segs | 学生们 |\n| VA+N | depends | depends | depends | 小媳妇 |\n| N+N | depends | depends | depends | 牛肉 |\n| Temporal nouns name of time | cd-year/NT | cd year | cd-year | 1998年 |\n| count of time | cd/CD year/NN | cd year | cd year | 3年 |\n| DP-related\nCD | one seg | ?? | one seg | 一万三千 |\n| CD+X+CD | one seg | several | one seg | 三分之一 |\n| AD + CD | ad/AD + cd/CD | ad cd | ad cd | 约三百 |\n| CD + X | cd-X/CD | cdX | cd-X | 三百多 |\n| di4-CD | di 娈 cd/OD | di4 cd | di4-cd | 第一 |\n| CD+M | cd/CD m/M | cd m | cd m | 这个 |\n| M + M | m-m/M | m-m | m-m | 片片 |\n| yi1+M+M | yi1/CD m-m/M | yi1 m-m | yi1-mm | 一片片 |\n| yi1-M-yi1-M | yi1/CD m/M yi1/CD m/M | ?? | yi1 m yi1 m | —^^~-个 |\n| Markers\nV-AS | v/V as/AS | v AS | v AS | 打了 |\n| V-de | v/V de/DER | v de5 | v de5 | 走得 |\n| SP | one seg | one seg | one seg | 吗 |\n| de5(的，地） | one seg | one seg | one seg | 我的，高兴地 |\n| zhi1(之)+CD/N | two segs | two segs | two segs | 方法之三 |\n| zhi1(之)+LOC | one seg | ?? | one seg |  |\n| Others\n成语(no insertion) | one seg | one seg | one seg | 鼠目寸光 |\n| ACROM | one seg | one seg | one seg | 北大 |\n\n**Table A.2: Comparison with PRC and Rocling^ Guidelines(Ctd)**\n\nAppendix B Treebank Part-of-Speech Tagset\n----------\n\n\nThe following is the Part-of-Speech Tagset used in our Penn Chinese Treebank.\n\n\n|  |  |  |\n| --- | --- | --- |\n| AD | adverb | 还 |\n| AS | aspect marker | 着 |\n| BA | 把 in barconstmction | 把，将 |\n| CC | coordinating conjunction | 和 |\n| CD | cardinal number | 一百 |\n| CS | subordinating conjunction | 虽然 |\n| DEC | 的 in a relative-clause | 的 |\n| DEG | associative 的 | 的 |\n| DER | 得 in V-de const, and V-de-R | 得 |\n| DEV | 地 before VP | 地 |\n| DT | determiner | 这 |\n| ETC | for words等，等等 | 等，等等 |\n| FW | foreign words | ISO |\n| IJ | interjection | 啊 |\n| JJ | other noun-modifier | 男，共同 |\n| LB | 被 in long bei-const | 被^给 |\n| LC | localizer | 里 |\n| M | measure word | 个 |\n| MSP | other particle | 所 |\n| NN | common noun | 书 |\n| NR | proper noun | 美国 |\n| NT | temporal noun | 今天 |\n| OD | ordinal number | 第一 |\n| ON | onomatopoeia | 哔哔 |\n| P | preposition excl.被 and 把 | 从 |\n| PN | pronoun | 他 |\n| PU | punctuation |  |\n| SB | 被 in short bei-const | 被^给 |\n| SP | sentence-final particle | 吗 |\n| VA | predicative adjective | •红 |\n| VC | 是 | 是 |\n| VE | # as the main verb | 有 |\n| VV | other verb | 走 |\n\n**Table B.l: Our POS tagset in alphabetical order**\n\nBibliography\n------------\n\n\n[Chi96] Chinese Knowledge Information Processing Group. Shouwen Jiezi - A study of Chinese Word Boundaries and Segmentation Standard for Information Processing (in Chinese). Technical report, Taipei: Academia Sinica, 1996.\n\n\n[1D92] John Xiang ling Dai. The Head in Wo Pao De Kuai. Journal of Chinese Linguistics, 1992.\n\n\n[LTS93] Y. Liu, Q. Tan, and X. Shen. Segmentation Standard for Modern Chinese Information Processing and Automatic Segmentation Methodology, 1993.\n\n\n[Pac98] Jerome L. Packard, editor. New Approaches to Chinese Word Formation, Mouton de Gruyter, 1998.\n\n\n[SW87] Anna Maria Di Sciullo and Edwin Williams. On the Definition of Word. The MIT Press, 1987.\n\n\n[XPX+00] Fei Xia, Martha Palmer, Nianwen Xue, Mary Ellen Okurowski, John Kovarik, Shizhe Huang, Tony Kroch, and Mitch Marcus. Developing Guidelines and Ensuring Consistency for Chinese Text Annotation. In Proc. of the 2nd International Conference on Language Resources and Evaluation (LREC-2000)^ Athens, Greece, 2000.\n\n\n\n\n\n[1](#footnote1)\n\n\nThe difference between a JJ and a prefix is that the latter, not the former, is bound. As mentioned before, sometimes, it is difficult to tell whether a morpheme is bound or not, so we keep a list of morphemes that we regard as prefixes. In this case, if the N in X+N can be replaced with, an NP, we treat X as a JJ, ratter than a prefix.\n\n\n[2](#footnote2)\n\n\n A word is a non-predicate adjective if it can not appear as a predicate after the subject without the help of 是...的.\n\n\n[3](#footnote3)\n\n\nN+LC1+LC2, where LC1 and LC2 denote opposite directions, is treated similarly.\n\n\n[4](#footnote4)\n\n\nIn either of the last two examples, the first morpheme is bisyllabic, and it could be tagged as nouns in some context. Because the second morpheme is mono-syllabic, the expression should be treated as one word regardless of the POS tag of the first morpheme.\n\n[5](#footnote5)\n\n\n[6](#footnote6)\n\n\nThe V+N combination is among the hardest cases for the word definition. The tests proposed here are not perfect. They tend to treat idiomatic phrases (similar to \"kick the bucket\" in English) as words. However, Those errors can be easily corrected if later a dictionary becomes available.\n\n[7](#footnote7)\n\n\nIt has been argued that aspect markers are affixes (e.g., [1D92]). Right now, we do not group the V and the AS together.\n\n\n[8](#footnote8)\n\n\nThe function of de in the V-de construction is controversial. It ranges from an affix, a particle, to a verb. We will not get into details here.\n\n\n[9](#footnote9)\n\n\n Many of Xs in this pattern are ^coverbs^ and it is highly debated which tag, V or P, X should Lave in this pattern and whether V+X forms a word by the process such as reanalysis.\n\n\n[10](#footnote10)\n\n\nNote: 50 多分钟 is segmented as 50 多[50\\_odd]/CD 分钟/M.\n\n\n[11](#footnote11)\n\n\nIn the literature(e.g., [1D92]), it has been argued that some of the particles such as 得，了 are  affixes. For the sake of compatibility with other guidelines and also because it is very easy to automatically group these particles with preceding words, we separate the particles from the preceding words.\n\n\n\n"
  },
  {
    "path": "docs/annotations/tok/index.md",
    "content": "# Tokenization\n\n## Chinese\n```{toctree}\nctb\nmsr\n```"
  },
  {
    "path": "docs/annotations/tok/msr.md",
    "content": "# MSR中文文本标注规范 (5.0 版)\n\n[**Tokenization Guidelines of Chinese Text (V5.0)**](http://sighan.cs.uchicago.edu/bakeoff2006/MSRAsamp/msra-spec.pdf)\n\n黄昌宁 李玉梅 朱晓丹\n\nChang-Ning Huang, Yumei Li, and Xiaodan Zhu\n\n微软亚洲研究院\n\nMicrosoft Research Asia\n\n２００6 年 3 月 27 日\n\nMarch 27, 2006\n\n微软《中文文本标注规范（5.0 版）》\n\n## 第一章 概述\n\n### 1.1 版本说明\n\n微软亚洲研究院《命名实体标注规范》3.0版是为30万词《人民日报》语料的命名实体（NE)标注任务制定的。其英文版‘Guideline on Chinese Named Entity Annotation’成稿于2003年2月，用于LSP(Lexical Service Platform)课题。当时在研究院，命名实体识别（Name Entity Recognition)和自动分词（Word Segmentation)是文本处理中互相独立的两个过程，所以未曾深入考虑分词词表（lexicon)对命名实体标注带来的影响。2005年3月至7月在准备第二届国际自动分词评测（SIGHANBakeoff2005)的237万词训练语料的过程中修订了该规范，形成4.0版。《命名实体标注规范》4.0版的一个最大特点是把命名实体识别有机地融入到中文自动分词的整体过程中去。因此，除了命名实体自身的定义以外，还需要系统地阐明词表词和各类实体之间的复杂关系。本规范是在微软亚洲研究院《命名实体标注规范》4.0版的基础上编制的。由于规范实际上涵盖了文本中词语和各类实体的标注规则与实例，所以更名为《中文文本标注规范》(Tokenization Guidelines of Chinese Text) 5.0版。\n\n### 1.2导读\n\n规范的第一章（概述)、第二章（专有名词标注总则）、第六章（数字串标注总则）以及第九章（分词歧义消解细则）是每个标注人员必读的材料。其它章节收集了大量的实体标注规则与实例，用以补充各类实体定义的不足。凭借这些具有上下文信息的词例化实例可以进一步提高文本标注的精度和一致性，所以它们是供标注人员经常查阅的参考资料。诚恳欢迎读者对本规范和带标语料中的错误提出宝贵意见，以便及时更正。批评和意见请寄[黄昌宁](mailto:cnhuang@msrchina.research.microsoft.com)。\n\n### 1.3标注格式\n\nformat-1是面向标注人员的格式：\n\n/十月九日/上午/ ->/[dat十月九日]/[tim上午]/\n\nformat-2是基于XML的标注格式：\n\n\n/十月九日/上午/ -> `<w><TIMEX TYPE=“DATE\" >十月九日</TIMEX></w><w> <TIMEXTYPE=“TIME”>上午</TIMEX></w>`\n\n\n*TIMEX* 是时间表达式，日期 *DATE* 和时间 *TIME* 是它的两个子类。\n\n考虑到本规范主要是为标注人员编写的，以后的例子主要以第一种格式（format-1）表示。想了解更多 XML 格式的读者，请参见 MET-2 Guideline1。\n\n1MET(MultipleEntityTask)是1997年第七届MUC(Meassage Understading Conference)会议多实体识别任务的简称。MET-2是当年美国NIST公布的命名实体标注规范。可查阅：http://www.itl.nist.gov/iaui/894.02/related_projects/muc/proceedings/ne_task.html\n\n### 1.4命名实体标记集\n\n表1-1是本规范定义的全部命名实体标记，包括专有名词（*NAMEX*）、时间表达式（*TIMEX*）、数字表达式（*NUMEX*）、度量表达式（*MEASUREX*）和地址表达式（*ADDREX*）等类五大类及其下属的三十个子类。\n\n| 大类 | 子类 | Format-1标注集 | Format-2标注集 |\n| --- | --- | --- | --- |\n| NAMEX | Person | P | PERSON |\n| Location | L | LOCATION |\n| Organization | O | ORGANIZATION |\n| TIMEX | Date | dat | DATE |\n| Duration | dur | DURATION |\n| Time | tim | TIME |\n| NUMEX | Percent | per | PERCENT |\n| Money | mon | MONEY |\n| Frequency | fre | FREQUENCY |\n| Integer | int | INTEGER |\n| Fraction | fra | FRACTION |\n| Decimal | dec | DECIMAL |\n| Ordinal | ord | ORDINAL |\n| Rate | rat | RATE |\n| MEASUREX | Age | age | AGE |\n| Weight | wei | WEIGHT |\n| Length | len | LENGTH |\n| Temperature | tem | TEMPERATURE |\n| Angle | ang | ANGLE |\n| Area | are | AREA |\n| Capacity | cap | CAPACITY |\n| Speed | spe | SPEED |\n| Acceleration | acc | ACCELERATION |\n| Othermeasures | mea | MEASURE |\n| ADDREX | Email | ema | EMAIL |\n| Phone | pho | PHONE |\n| Fax | fax | FAX |\n| Telex | tel | TELEX |\n| WWW | www | WWW |\n| Postalcode | pos | POSTALCODE |\n\n**表1-1命名实体的标记集**\n\n### 1.5基本原则\n\n#### 1.5.1基本考虑\n\n通用性：尽量遵循国际标准MET-2和ER-992，不同之处在本规范中阐明。\n\n实用性：可用于LSP (Lexical Service Platform), TTS (Text To Speech conversion), IR (Information Retrieval), IE (Information Extraction), QA (Question Answering), IME(Input Method Editor)等应用系统。\n\n#### 1.5.2标注对象\n\n##### 1.5.2.1词表词与未登录词\n\n本规范认为：文本中的任何一个词要么是词表词（LW），要么是未登录词（OOV）。两者都是文本的标注对象。未登录词可以进一步分成命名实体（NE）、词法派生词（MDW）和新词（NW）等三部分。本规范定义的命名实体是未登录词的主体。\n\n（1）命名实体（NE）\n\n命名实体可以进一步分成如下五大类共三十个子类（详见表1-1）：\n\n- 专有名词（*NAMEX*）包括人名（*P*）、地名（*L*）和机构名（*O*）等3种。\n- 时间表达式（*TIMEX*）包括日期（*dat*）、时间（*tim*）和时段（*dur*）等3种。\n- 数字表达式（*NUMEX*）包括百分数（*per*）、钱款（*mon*）、频度（*fre*）、整数（*int*）、分数（*fra*）、小数（*dec*）、序数（*ord*）和比率（*rat*）等8种。\n- 度量表达式（*MEASUREX*）包括年龄（*age*）、温度（*tem*）、角度（*ang*）、长度（*len*）、\n- 面积（*are*）、容积（*cap*）、重量（*wei*）、速度（*spe*）、加速度（acc）和其它（*mea*）等10种。\n- 地址表达式（*ADDREX*）包括电子邮箱（*ema*）、电话（*pho*）、传真（*fax*）、电报挂号（*tel*）、邮政编码（*pos*）和网址（*www*）等6种。\n\n在标注过的文本中，词的边界一律用斜线（slash）表示。除了词表词以外，每个独立的命名实体（即非嵌入到词表词内部的实体，见1.5.2.3）也被视为一个词，其标注符号及形式详见本规范。\n\n（2）词法派生词（MDW）\n\n以词表词AB的重迭形式AABB和AB/AB为例：\n\n/*转轨*/*哪*/*有*/*像*/*人*/*说*/*得*/*那般*/*轻轻松松*/？\n\n/*积累*/*多*/*了*/*，*/*抽出*/*时间*/*，*/*认真*/*整理*/*整理*/，\n\n（3）新词（NW）\n\n一个新词的左右两侧用符号&标示，其内部的切分符保留3，如：\n\n/&*桑拿*&/*浴*/\n\n/*天时地利*/&*人*/*和*&/*；*/\n\n/[L*罗*]/*货币*/&*列*/*伊*&/\n\n以下是一些真实的例句，例句中的实体标注符号请参阅表1-1。\n\n[Example-1]\n\n```\n/[dat ６月２９日]/、/[dat ３０日]/[tim 晚上]/，/[L 北京市]/下/了/[int 两场]/大雨/，/笔者/\n居住/的/宿舍/楼/前/，/宽/[len 六七米]/、/长/[len ３０多米]/的/路/上/积水/达/膝盖/之上/。\n<sentence>\n<w><TIMEX TYPE=\"DATE\"> ６月２９日 </TIMEX></w><w> 、 </w><w><TIMEX\nTYPE=\"DATE\"> ３０日 </TIMEX></w><w><TIMEX TYPE=\"TIME\"> 晚 上\n</TIMEX></w><w>，</w><w><NAMEX TYPE=\"LOCATION\">北京市</NAMEX></w><w>\n下 </w><w> 了 </w><w><NUMEX TYPE=\"INTEGER\"> 两 场 </NUMEX></w><w> 大 雨\n</w><w>，</w><w>笔者</w><w>居住</w><w>的</w><w>宿舍</w><w>楼</w><w>前\n</w><w> ， </w><w> 宽 </w><w><MEASUREX TYPE=\"LENGTH\"> 六七米\n</MEASUREX></w><w>、</w><w>长</w><w><MEASUREX TYPE=\"LENGTH\">３０多米\n</MEASUREX></w><w>的</w><w>路</w><w>上</w><w>，</w><w>积水</w><w>达\n</w><w>膝盖</w><w>之上</w><w>。</w>\n</sentence> \n```\n\n\n[Example-2]\n\n```\n/[dat ６月中下旬]/，/笔者/到/[L 意大利]/、/ [L 西班牙]/等/国/访问/时/，/一个/很/深/的/感受\n/是/[L 意]/、/[L 西]/两国/的/高速公路/非常/发达/，/东西南北/，/纵横/成/网/，/.四通八达/。\n<sentence>\n<w><TIMEX TYPE=\"DATE\">６月中下旬</TIMEX></w><w>，</w><w>笔者</w><w>到\n</w><w><NAMEX TYPE=\"LOCATION\">意大利</NAMEX></w><w>、</w><w><NAMEX\nTYPE=\"LOCATION\">西班牙</NAMEX></w><w>等</w><w>国</w><w>访问</w><w>时\n</w><w> ， </w><w> 一 个 </w><w> 很 </w><w> 深 </w><w> 的 </w><w> 感 受 </w><w> 是\n</w><w><NAMEX TYPE=\"LOCATION\"> 意 </NAMEX></w><w> 、 </w><w><NAMEX\nTYPE=\"LOCATION\">西</NAMEX></w><w>两国</w><w>的</w><w>高速公路</w><w>非\n常</w><w>发达</w><w>，</w><w>东西南北</w><w>，</w><w>纵横</w><w>成</w><w>\n网</w><w>，</w><w>四通八达</w><w>。</w>\n</sentence> \n```\n\n[Example-3]\n\n```\n/[O 县委]/决定/选派/任/了/[dur 八年]/[O 城建局]/长/的/[P 周欣光]/担任/[O 老干部局]/长/。\n<sentence>\n<w><NAMEX TYPE=\"ORGANIZATION\"> 县 委 </NAMEX></w><w> 决 定 </w><w> 选 派\n</w><w> 任 </w><w> 了 </w><w><TIMEX TYPE=\"DURATION\"> 八 年\n</TIMEX></w><w><NAMEX TYPE=\"ORGANIZATION\"> 城建局 </NAMEX></w><w> 长\n</w><w> 的 </w><w><NAMEX TYPE=\"PERSON\"> 周欣光 </NAMEX></w><w> 担 任\n</w><w><NAMEX TYPE=\"ORGANIZATION\">老干部局</NAMEX></w><w>长</w><w>。\n</w>\n</sentence>\n```\n\n\n[Example-4]\n\n```\n/[L喇嘛寺村]/地处/[L承德避暑山庄]/，/[L山庄]/寺庙/林立/，/僧侣/穿梭/，/[L山庄]/[L外八庙]/的/[ord第一个]/庙/就/是/[L喇嘛寺]/。\n\n<sentence>\n<w><NAMEX TYPE=\"LOCATION\">喇嘛寺村</NAMEX></w><w>地处</w><w><NAMEX\nTYPE=\"LOCATION\"> 承德避暑山庄 </NAMEX></w><w> ， </w><w>><NAMEX\nTYPE=\"LOCATION\">山庄</NAMEX></w><w>寺庙</w><w>林立</w><w>，</w><w>僧侣\n</w><w> 穿 梭 </w><w> ， </w><w>><NAMEX TYPE=\"LOCATION\"> 山 庄\n</NAMEX></w><w><NAMEX TYPE=\"LOCATION\"> 外八庙 </NAMEX></w><w> 的\n</w><w><NUMEX TYPE=\"ORDINAL\">第一个</NUMEX></w><w>庙</w><w>就</w><w>\n是</w><w><NAMEX TYPE=\"LOCATION\">喇嘛寺</NAMEX></w><w>。</w>\n</sentence> \n```\n\n##### 1.5.2.2*L*, *P*,*O*, *dat*,*tim*,*dur*等实体的边界允许跨越多个词表词\n\n例如：\n\n/[L*瑞典*]/[O*斯德哥尔摩国际和平研究所*]/ /[O*中国工商银行上海市分行*]/\n\n/[tim*下午当地时间*5*时*59*分*]/\n\n1.5.2.3专名的标记（L，P，O）可以插入到一个词表词的内部\n\n例如，词表词抗日战争*和*事后诸葛亮*中的地名和人名应分别予以标注。\n\n/*抗*[L*日*]*战争*/----正确标注。\n\n/*抗日战争*/*----未标出*L，是错误标注。\n\n/*抗*/[L*日*]/*战争*/ ----插入分词标记，是错误标注。\n\n/*事后*[P*诸葛亮*]/\n\n##### 1.5.2.4数字串（除专名以外的其他四类表达式）的标记不得插入到词表词的内部\n\n###### 1.5.2.4.1dat，tim等标记不得插入到一个词表词的内部\n\n词表词*夏令营、*春耕、*冬训*、*早出晚归*中的*夏、春、冬、早、晚*等词素都有*dat*和*tim*的意思，但不得标注。例如，\n\n/[dat*冬*]*训*/ ---错误标注。\n\n/[tim*早*]*出*[dat*晚*]*归* /  ---错误标注。\n\n然而词表词被整体标注为*dat*和*tim*的情况是常有的，例如：\n\n/[dat*初冬*]/ ----*初冬*是词表词。\n\n/*[dat*夏季*]/*----*夏季*是词表词。\n\n/告别/*了*/[dat*冬日*]/*的*/*凝重*/*、*/[dat*春天*]/*的*/*轻盈*/*、*/[dat*夏日*]/*的*/*浪漫*/，\n\n- 注：在文本中具有比喻意义的*春、夏、秋、冬*、历史上的*今天、昨天、明天*不作标注。例如：\n\n/[dat*今年*]/*又*/*迎来*/*了*/*一个*/*科学*/*的*/*春天*/ /\"/*在*/*陆地*/*资源*/*日渐*/*减少*/*的*/*今天*/*，*/\n\n/*他们*/*的*/*明天*/*将*/*更加*/*辉煌*/*。*/\n\n###### 1.5.2.4.2int,ord等标记不得插入到到一个词表词的内部\n\n词表词*五湖四海*、*不管三七二十一*、*三纲五常中的数词不允许标注*int（整数）。例如，\n\n/*[int*五*]*湖*[int*四*]*海*/*----错误标注。\n\n/*十年动乱*/*结束*/*不久*/*，*/ ----*十年动乱*是词表词。*十年*不标。\n\n/*不管三七二十一*/ /*三纲五常*/\n\n##### 1.5.2.5数词首、半、双、两等\n\n###### 1.5.2.5.1序数词素首\n\n词表中有许多词含有词素*首*，如*首创、首倡、首选、首发、首航、首飞、首演、首映、首战、首展、首席代表、首席科学家、首席执行官、首富、榜首、魁首、居首*等。但不可把词表词中的词素*首*单独作为*ord*（序数）来标注。\n\n/*首席执行官*/----正确标注。\n\n/*[ord首席]执行官*/----在词表词中插标*ord*是错误的。\n\n以下的词表词属于\"首+量词\"结构，可以整体作为*ord*标注。例如：\n\n*[ord*首届*]*，*[ord*首次*]*，*[ord*首批*]*，*[ord*首位*]*，*[ord*首例*]*\n\n###### 1.5.2.5.2分数词素半\n\n词表中有许多词含有词素*半**如半价、半票、半饱、半身、半世、半辈子、上半时、下半场、半边*等，但不可把上述词表词中的词素*半*标注为*fra*（分数)。\n\n/*下半场*/*比赛*/[O*中国队*]/*未进*/*一*/*球*/\n\n/*上半时*/\n\n/*下*[fra*半*]*场*/----在词表词中插标*fra*是错误的。\n\n以下的词表词可作为不同的数字串（*dur*，*tim*，*fra*，*int*，*age*）标注：\n\n*[dur*半年*]*，*[dur*半天*]*，*[tim|dur*半夜*]*，*[fra*半个*]*，*[int|age*半百*]*\n\n- 注：半个西瓜中的半个，与四半中的半概念不一样，前一个半是指二分之一，\n\n后一个半是量词，所以标注也不同！！/*[int*一个*]*/*西瓜*/*分为*/[int*四半*]/ /[fra*半个*]/*西瓜*/\n\n###### 1.5.2.5.3整数词素双\n\n当数词双成为词表词的一个词素时，如\"双方、双边、双手、双打、双杠、双轨、双层、双目、双亲\"等，一律不作为整数(*int*)标注。对于非词表词，只标[*int双*]。例如：\n\n/*窗外*/*又*/*起风*/*了*/*，*/*双层*/*的*/*窗*/*硬是*/*阻挡*/*不住*/*沙尘*/*的*/*侵扰*/*。*/\n\n/*双方*/*认为*/*，*/[L*中*][L*美*]/*两国*/*应该*/*从*/*战略*/*的*/*高度*/*和*/*长远*/*的*/*角度*/\n\n- 注：一般情况下，数词和\"方\"之间不切分整体标为*int*。但\"四方\"是词表词所以不标。\n\n[*int三方*]/*会谈*/ /*举行*/*四方*/*会谈*/\n\n以下是相关的例子：\n\n/*用*/*任何*/*一*/*部*/[*int双*]/*音频*/*电话*/*只需*/*拨打*/[pho*２５８０*]/*就*/*可以*/\n\n/*部队*/*进行*/*的*/*海上*/*训练*/*、*/[int*双*]/*机*/*穿云*/*、*/*超低空*/*等*/*高难*/*课目*/*训练*/ /*全村*/[are*７００亩*]/*旱地*/*都*/*种上*/*了*/[int*双*]/*膜*/*棉*/*，*/\n\n###### 1.5.2.5.4整数词素两\n\n当数词\"两\"成为词表词的一个词素时，如\"两国、两会、两地、两者、两头、 两手、两边、两旁、两侧\"等，一律不作为数位串（*int*）标注。例如：\n\n/*使*/*两国*/*的*/*友好*/*合作*/*得到*/*巩固*/*和*/*发展*/。\n\n- 注：一般情况下，数词和\"国\"之间是要切分的，如：[*int五*]/*国*/*元首*/\n\n/*前*/*些*/*年*/*我*/*对*/*参加*/*『*/*两会*/*』*/*总是*/*有点*/*发怵*/*。*/\n\n/*大街*/*两旁*/*店铺*/*林立*/\n\n/*戏台*/*两侧*/*立柱*/*上*/*有*/*这样*/*一*/*副*/*对联*/*：*/\n\n/*中间*/[int*两间*]/*是*/*客厅*/*，*/*两边*/*是*/*卧室*/*和*/*书房*/*。*/\n\n/*对*/*分散*/*居住*/*的*/*\"*/*五保*/*\"*/*户*/*，*/*镇*/*、*/*村*/[int*两*]/*级*/*拨*/*专款*/\n\n/[int*两*]/*车*/*饮料*/*以及*/*办公*/*桌椅*/*，*/\n\n- 注：临时量词\"车、船、床、桌、屋子、院子\"等不进入int标注。/*成为*/[O*议会*]/[int*两*]/*院*/*审议*/*的*/*重点*/*和*/*舆论*/*关注*/*的*/*焦点*/\n\n- 注：两院不是词表词，所以应当切分标注如上。\n\n/*及早*/*进行*/*政治*/*谈判*/*推动*/[L*两岸*]/*关系*/*发展*/ /*沿江*/[int*两*]/*岸*/*苗家*/*吊脚楼*/*上*/*的*/*观众*/\n\n- 注：词表词两岸是专指台湾海峡两岸的地名。如果泛指江河两岸，则不作为 地名标注，而且要切分并标数词\"两\"为*int*。\n\n/*一下*/*进*/*了*/[int*两*]/*球*/\n\n#### 1.5.3基本规定\n\n1)标注时，不得在原来的文本中加入回车换行符。\n\n2)对于NIST制定的两个中文NE标准：MET-2和ER-99。前者已有系统参加评测，它们的评测结果可供后来者参考；后者是前者的修订版,但尚未有系统参加测试。本规范与这两种标准不同之处将尽可能在注释中加以说明。例如：\n\n/[dat*去年上半年*]/\n\n- 注：MET-2把去年上半年*整体视为*dat*；ER-99则只将上半年*视为*dat*。\n\n3)对于微软研究院根据自己的需要而加入的标记，本规范也将在注释中加以说明。比如本规范要求的如下标注：\n\n/[P*邓小平*]/*理论*/\n\n- 注：MET-2和ER-99规定，*理论，主义，思想，定律*等词前面的人名均不作为专名标注（见2.8）。\n\n## 第二章 专有名词标注总则 \n\n### 2.1专有名词（NAMEX）标注通则\n\n对于人名、地名和机构名这三类专有名词，MET-2和ER-99之间的差异甚微，在它们给出的示例中，只有两处不同：中南美*和长江流域*（具体情况见后）。所以在制订人名、地名、机构名的标注规范时，我们没有刻意去区分这两个标准，而是力图把它们统一地融入本规范。\n\n下面给出人名、地名、机构名的定义。\n\n### 2.2专有名词是具体的、特定的，而不是抽象的、泛指的\n\n比如：*上苍、老外、姑娘，小镇，企业*等就不应视为专有名词。\n\n### 2.3复合专有名词的标注不允许嵌套\n\n在MET-2和ER-99标准中，任何命名实体都不允许嵌套。换句话讲，只标一个实体的最长边界，不标其内部包含的其它实体。\n\n### 2.4人名、地名、机构名中的数字串不单独标出\n\n例如:\n\n/[P*龟山一郎*]/\n\n/[L*德富路二四一至二六三号*]/\n\n/[O*北京*101*中学*]/\n\n/[O*北京*[ord*四*]*中*]/ ----这种嵌套式的的标注是错误的！\n\n### 2.5含有外文和数字的命名实体应整体一起标注\n\n例如：\n\n/[O*American航空公司*]/\n\n/[O*SONY公司*]/\n\n### 2.6当两个实体用虚词的连接时应分别标注为两个实体\n\n例如:\n\n/[L*美国*]/*的*/[L*纽约*]/\n\n/[L*美国*]/*的*/[P*理查德本森*]/\n\n但当*的*成为实体的一部分时，要整体一起标注。例如：\n\n/[O*美的电器集团*]/\n\n### 2.7实体前后有引号或书名号的情况\n\n如果一个命名实体中间有引号或书名号，则引号或书名号是该实体的一部分。如果一个实体被外面的引号或书名号括起来，那么其引号或书名号就不作为实体的一部分标注。例如：\n\n/[O\"*阿克布拉克*\"*中哈合资企业*]/ \n\n/[O*美国《幸福》杂志*]/\n\n/*《*/[O*星岛日报*]/*》*/*的*/*社论*/*说*/\n\n### 2.8短语内部包含实体、但整体又不是命名实体的情况\n\nER-99规定：如果一个短语内部包含实体、但整体又不是命名实体，则一律不作标注。本规范则要求对该短语中的实体部分加以标注。例如:\n\n/[L*巴拿马运河*]/*条约*/\n\n- 注：ER-99认为,巴拿马运河条约*整体不能分解，其中的地名不应标注。但本规范把其中的巴拿马运河*单独标为地名。\n\n/[L*巴拿马运河*]/*----巴拿马运河*单独出现时，作为地名标注。\n\n/[L*香港*]/*脚*/\n\n- 注：英文为\"HongKongfoot\"，类似于\"athletesfoot\"，不可分解，所以ER-99规定整体不标。本规范，仍将*香港*标为地名。\n\n/[L*美国*]/*小姐*/\n\n- 注：原文为\"MissAmerica\"，指选美活动中获全美第一名的小姐。对此ER-99规定整体不标。本规范，仍将*美国*标为地名。\n\n/[L*美国*]/*姑娘*/*----ER-99对本例的美国*也是标注的。\n\n/[ord*第四十六届*]/[O*太平洋亚洲旅行协会*]/*年会*/\n\n- 注：此例在ER-99中整体不标，理由是不可分解。本规范认为找不出充分理由说明其不可分解。所以我们把太平洋亚洲旅行协会*标为机构名。第四十六届太平洋亚洲旅行协会年会*整体不是机构名。\n\n/[P*毛泽东*]*思想*/ /[P*马克思*]*主义*/\n\n/[P*马克思*]/*主义*/ ----*错误标注！因为*马克思主义*是词表词。\n\n/[P*阿佛加罗*]/*定律*/\n\n- 注：ER-99规定，在理论、主义、思想、定律*等词前面出现人名时，是整体不可分解的字符串；因此该字符串和其中的人名都不标注。但本规范仍将标注其中的人名。\n\n### 2.9与军队相关的情况\n\n当泛指某个国家的军队（如英军、美军*等）时，不是机构名；当指一个具体的军种（如空军、陆军、海军*等）时，要标注为机构名。例如：\n\n/[L*美*]/*军*/*飞机*/\n\n/[O*斯里兰卡空军*]/ \n\n/[O*英国皇家空军*]/\n\n但是，有如下特殊情况：\n\n*[L*济南军区*]/ ----*军区是*L*而不是*O*。\n\n/[L*彼得森空军基地*]/ -----军事基地是L而不是O。\n\n/[L*西非*]/&*维*/*和*&/*部队*/ ------部队不作为机构名标准。\n\n### 2.10多媒体、产品和条约中的人名、地名、机构名\n\nER-99规定：当人名、地名、机构名属于多媒体、产品和条约时，均不加标注。但本规范对上述实体名还是要标注的。例如：\n\n/[P*邓小平*]/*一*/*片*/*的*/*播出*/\n\n- 注：ER-99规定，电视节目的名字邓小平*不标。本规范仍把它标为人名。此外，邓小平*作为片名，在规范的文本中应当用书名号括出，如《邓小平》。\n\n/*二战*/ ----*二战*是事件,所以不标注。\n\n/[L*香港*]/*百*/*题*/*今天*/*为*/*您*/*解答*/\n\n- 注：ER-99规定，香港百题是电视片的标题，所以专名香港不予标注。但本规范仍把香港标为地名。下面其它的例子就不一一解释了。\n\n/*这*/*本*/*介绍*/[P*毛泽东*]/*的*/*小说*/ ----*毛泽东*要标注。\n\n/*这*/*本*/*名*/*为*/[P*毛泽东*]/*的*/*小说*/ ----ER-99*毛泽东*不标。\n\n/[L*广州*]/*条约*/ ----ER-99*广州*不标。\n\n/[L*辽*][L*沈*]*战役*/ ----ER-99*辽沈*不标。\n\n本规范在后面还要对人名、地名、机构名中不加标注的情况作专门的说明，详见下面的各章节标注细则。\n\n### 2.11别名或简称的标注\n\n对人名、地名、机构名的别名或简称要标注。例如：\n\n/[O*IBM*]/\n\n/[L*深*]/[L*沪*]/*股市*/ \n\n/[O*北约*]/\n\n/[L*中*][L*美*]/*首脑*/*互访*/\n\n/[L*中*]/[L*文*]/*双方*/*一致*/*认为*/\n\n- 注：由于中美是词表词，标注地名时不可插入分词标记。中文也是词表词，但这里是指中国和文莱，所以标成地名时需要在两个简称中插入分词符号。这样的词表词还有中意、意中、中巴、日中、中肯、中非等。巴中是一个地名，但表示巴基斯坦和中国时需要用分词符号把两个简称分隔开。\n\n- 注：对于简称中嵌套的人名、地名、机构名不予标注，如：\n\n/[O*中共*]/ ----*中*指*中国*，但不标。\n\n/[O*中共中央政治局*]/ - ---同理，不标注*中*。\n\n## 第三章人名\n\n人名一般包含姓和名两部分，姓是表明家族的字，有单姓和复姓之别；名也就是名字，是一种称号，由一个或几个字组成，跟姓合在一起，用来代表一个人，以区别于别的人。下面将对人名的标注规则进行详解。\n\n### 3.1人名标注规则\n\n正常情况下，人名一般包含姓和名两部分，标注规则如下表所示：\n\n| **序号** | **情况** | **标记方法** |例子 |\n| --- | --- | --- | --- |\n| 1 | 只含姓，没有名 | 标出姓氏部分 | *[P*庄*]*、*[P*欧阳*]*、*[P*司马*]* |\n| 2 | 只包含名字 | 标出名字部分 | *[P*育焜*]* |\n| 3 | 姓名 | 姓名整体标出 | *[P*苏宗哲*]*、*[P*萝莉胡吉温*]* |\n| 4 | 姓名|姓|名+称谓称谓+姓名|\n| 5 | 前缀+姓名|姓|名姓名|\n| 6 | 姓名+姓名 | 分开来标 | *[P*李向东*]/[P*李向阳*]* |\n| 7 | 外国人名 | 作为一个整体来标 | *[P*罗马里奥*]*[P*马拉多纳*]*[P*比尔*•*盖茨*]* |\n\n- 说明：当人名中包含•时，整体标注为人名，如[P*比尔•盖茨*]。\n\n### 3.2人名标注细则\n\n#### 3.2.1人名的示例和详细说明\n\n#### 3.2.1.1人名实例\n\n/[P*颜惠忠*]/ \n\n/[P*连战*]/ \n\n/[P*凡*•*高*]/\n\n/[P*陈方安生*]/\n\n---当妻子与丈夫的名字写在一起时，要作为一个人名整体标注为P！\n\n#### 3.2.1.2称谓、绰号、官职不作为人名的一部分\n\n称谓、绰号、官职(如先生、总理等)不作为人名的一部分。例如，\n\n/[P*张*]/*经理*/ \n\n/[P*李*]/*市长*/\n\n/[P*陈*]/*姓*/*游客*/*说*/ \n\n/[P*刘*]/[ord*二*]/*嫂*/ /[P*周*]/*总理*/\n\n/[P*雷锋*]/*同志*/\n\n/[P*奥尔布赖特*]/*国务卿*/\n\n#### 3.2.1.3当称谓和姓名不可分时应整体标注为人名\n\n/[P*李主席登辉*]/*先生*/\n\n/*处*/[P*李犯清龙*]/*死刑*/*，*/\n\n/[P*李犯*×*龙*]/*持*/*刀*/*行凶*/*杀害*/*无辜*/*青年*/*，*/\n\n#### 3.2.1.4几世、几代要作为人名的一部分\n\n/[P*十四世达赖丹增加措*]/\n\n/[L*英国*]/*女王*/[P*伊丽莎白二世*]/\n\n#### 3.2.1.5家族实体\n\n/[P*蒋*]/*氏*/*父子*/\n\n/[P*西迪*]/*兄弟*/\n\n#### 3.2.1.6圣人和宗教人物要标注为人名\n\n/[P*释迦穆尼*]/ \n\n/[P*达赖*]/*喇嘛*/\n\n### 3.3虚构的人物、动物的名字要标注为人名\n\n#### 3.3.1在童话、小说中虚构人物要标注为人名\n\n/[P*孙悟空*]/ \n\n/[P*玉皇大帝*]/\n\n#### 3.3.2虚构的动物和非人的人物要标注为人名\n\n/[P*唐老鸭*]/\n\n/[P*花仙子*]/\n\n/\"/[P*盼盼*]/\"/*是*/*国内外*/*著名*/*的*/*熊猫*/*明星*/*，*/\n\n/*争相*/*目睹*/*狮*/*王*/[P*木法沙*]/*和*/*王后*/[P*色拉碧*]*产下*/*的*/*小*/*王子*/[P*辛巴*]/\n\n/*走进*/*一家*/*饭馆*/*，*/*发现*/*老板*/*就*/*是*/*大*/*灰*/*狼*/[P*罗克*]/*。*/\n\n#### 3.3.3用称谓或朝代等名号来指称特定人时要标注为人名\n\n例如：\n\n/[P*康熙*]/ \n\n/[P*乾隆*]/ \n\n/[P*秦始皇*]/ \n\n/[P*老子*]/ \n\n/[P*孔子*]/\n\n### 3.4不标注为人名的各种情况\n\n#### 3.4.1虚构的非人的植物的名字不作为人名标注\n\n如：\n\n/\"/*彩霞*/*，*/\"/*石子*/*小声*/*嘟哝*/*着*/*，*/\"/*多*/*恶心*/*的*/*名字*/*！*/\"/\n\n/*电磨*/*姐姐*/*故意*/*气*/*气*/*小*/*毛驴*/*，*/*说*/*：*/\"/*输*/*了*/*，*/*可*/*不能*/*哭鼻子*/*。*/\"/ \n\n/\"/*卡车*/*哥哥*/*，*/*我*/*和*/*你*/*来*/*比*/*一*/*比*/*谁*/*运*/*得*/*多*/*，*/*怎么样*/*？*/\"/\n\n/*好像*/*在*/*说*/*：*/\"/*荷花*/*姐姐*/*，*/*你*/*好*/*！*/\n\n#### 3.4.2对于嵌套在地名和机构名中的人名，不作标注\n\n如：\n\n/[L*嘉诚广场*]/\n\n/[O*中山大学*]/\n\n/[O*宋庆龄基金会*]/\n\n#### 3.4.3作为书名或画名的人名\n\n作为书名或画名的人名ER-99不标（见2.8），但本规范是要作标注的。如：\n\n/*世界*/*名画*/*《*/[P*蒙娜莉萨*]/*》*/\n\n*/《/*[P*蒋介石*]/*与*/[P*毛泽东*]/*》*/\n\n3.4.4法律、法庭事件、天气形成、疾病和奖金等五种情况\n\n当人名后面紧跟法律名、法庭事件、天气形成、疾病、奖金这五种情况时，人名不标注。\n\n例如：\n\n/*里*/*氏*/[ord*六点二级*]/ -----*里*不标。\n\n/*专家*/*呼吁*/*人们*/*要*/*注意*/*沙*/*氏*/*杆菌*/ -----*沙*不标。\n\n/[P*诺贝尔*]*奖*/ -----ER-99*诺贝尔*不标。\n\n#### 3.4.5在人名后面出现基金会时要整体标注为机构名\n\n/[O*李嘉诚基金会*]/\n\n所以基金会*和奖、奖金*是不同的两种情况，需加以区别。又如\n\n/[O*李嘉诚股份有限公司*]/ \n\n/[O*诺贝尔股份有限公司*]/\n\n## 第四章 地名\n\n地名包括洲、海洋、国家、省、市、县、地区、街道、乡、镇、村、机场、军事基地、军区、铁路、公路、桥梁、海峡、海湾、港湾、河流、湖、公园、草原、煤矿、牧场、养殖场、音乐厅、剧院、教堂、寺庙、图书馆、博物馆、美术馆、展览中心、公园、动物园、植物园、火车站、广场、大厦、大楼、体育场(馆)、游泳馆(池)、赛车场、商城、超市、书店(城)等城市公共设施，还包括某些特定的城市建筑和虚构的处所。详见下表。\n\n### 4.1地名标注规则\n\n| **序号** | **情况** | 标记方法 |例子 |\n| --- | --- | --- | --- |\n| 1 | 只是单独地名 | 标出地名部分 | *[L*中国*]*[L*竹塘乡*] |\n| 2 | 地名+地理(行政)单位 | 作为整体标出 | *[L*北京市*]*[L*台北县*]*地理单位如：省、地区、市、县、乡、镇、村、店、庙、沟、屯、坟、崖、海洋、河、川、江、峡谷、海湾、港湾、丘陵、湖、半岛、三角洲、区、街、路、街、街道、社区、小区、公寓、音乐厅、剧院、图书馆、博物馆、美术馆、展览馆、公园、动(植)物园、火车站、广场、大厦、大楼、体育场(馆)、游泳馆(池)、赛车场、商城、超市、书店(城)等城市公共设施及象征性建筑物、军事基地、军区等。*[L*天安门广场*]*[L*艾菲尔铁塔*]* |\n| 3 | 包含上、下位的地名（即合成地名）以及并列的地名 | 一律分别单独标出 | *[L*山东省*]/[L*青岛市*]/[L*胜利广场*]*[L*青岛市*]/[L*孙中山广场*]*[L*北京市*]/[L*海淀区*]/[L*知春路*]/[L*希格玛大厦*]*[L*北京*]/*、*/[L*天津*]/*、*/[L*上海*] |\n| 4 | 地名简称 | 单独标出 | *[L*鲁*]/*、*/[L*冀*]/*、*/[L*京*]* |\n| 5 | 并列的简称 | 单独标出 | [L*中*]/[L*俄*]/*两国*/*领导人*/*进行*/*了*/*会晤*[L*港*][L*澳*][L*台*]/地区 |\n| 6 | 地名包含人名以及地名包含地名的情况 | 地名中的人名、地名不标 | *[L*李嘉诚广场*]*[L*南京路*] |\n| 7 | 地名+地名关键词表达一个完整的概念时 | 相对完整的地名 | *[L*南非共和国*]*[L*宁夏回族自治区*]*[L*香港特别行政区*]* |\n\n### 4.2地名标注细则\n\n#### 4.2.1地名实体示例\n\n/[L*北京*]/ \n\n/[L*亚洲*]/\n\n/[dat*2008年*]/[L*奥*]*运会*/*，*/[L*中国*]*人*/\n\n/[L*中国*]*人民*/  ----*中国人、中国人民*都是词表词。\n\n/[L*朝鲜*]/*南北*/*对话*/ ----*不标注南*,北。\n\n- 注：词表词\"京剧、京白、京腔、京味儿\"中的\"京\"字要标注为：\n\n/[L*京*]*剧*/*、*/[L*京*]*白、*/[L*京*]*腔*/*、*/[L*京*]*味儿*/\n\n/[L*台东火车站*]/ \n\n/[L*卑南文化公园*]/\n\n/[L*基隆文化中心广场*]/\n\n/[L*高雄港第一港口*]/\n\n/[L*苏澳镇*]/[L*南方澳渔港*]/\n\n/*环*/[L*渤海湾*]/*地区*/*的*/*天然气*/*市场*/\n\n/*来自*/[L*沈阳军区*]/*各*/*集团军*/\n\n/[L*梅狮路后段*]/\n\n/[L*中横公路天祥段*]/\n\n/[L*华禄溪*]/*及*/[L*碧绿隧道*]/ \n\n/[L*南二高*]/[L*高雄支线*]/\n\n/[L*台廿一线*]/\n\n/[L*美国空军基地*]/\n\n/[L*上海*]/[L*国际航运大厦*]/ \n\n/[L*上海*]/[L*虹口足球场*]/\n\n/[L*上海博物馆*]/\n\n/[L*上海*]/[L*城市规划展示馆*]/ \n\n/[L*石家庄*]/[L*富强电力新村*]/\n\n/[L*西安第二长途通讯大楼*]/\n\n/[L*北京市*]/[L*王府井百货大楼*]/\n\n/[L*广深铁路*]/*以及*/[O*深圳发展银行*]/*部分*/*高官*/*也*/*被*/*免职*/ \n\n/[L*汉江*]/*上*/*的*/[L*圣水大桥*]/\n\n/[L*新亚欧大陆桥*]/ \n\n---从世界知识知道此处大陆桥的名字叫*新亚欧大陆桥*，是不可分解的。\n\n#### 4.2.2地名指示词（如国、省、市等）视为地名的一部分一起标注\n\n地名指示词（如国、省、市等）视为地名的一部分一起标注。复杂的、具有包含关系的地名要分开标注，但分开标注时不可把一个有完整意义的地名拆散。以下是正确的标注：\n\n/[L*德国联邦*]/*政府*/*总理*/\n\n/[L*基隆市*]/ \n\n/[L*台东县*]/ \n\n/[L*南山部落*]/\n\n/[L*美国*]/ [L*马里兰州*]/\n\n/[L*约旦河*]/\n\n/[L*朝鲜半岛*]/ \n\n/[L*长江三角洲*]/ -----*长江三角洲*是词表词。\n\n/[L*吉林省*]/[L*延边朝鲜族自治州*]/[L*图们市*]/\n\n以下两例均为错误的标注，因为*延边朝鲜族自治州*是具有完整意义的地名：\n\n/[L*吉林省*]/[L*延边*]/[L*朝鲜族自治州*]/[L*图们市*]/\n\n/[L*吉林省延边朝鲜族自治州*]/[L*图们市*]/\n\n- 注：在ER-99的标准测试集中，把中国西昌卫星发射基地*整体标为地名。我们认为这是错误的，因为在一个地名中不应当包含具有上、下位关系的另一地名。正确的标注是：\n\n/[L*中国*]/[L*西昌卫星发射基地*]/\n\n/[L*美国洛克希德·马丁卫星测控中心*]/*和*/[L*中卫公司测控站*]/\n\n/*从*/[L*法*]/*属*/[L*圭亚那*]/[L*库鲁航天中心*]/*发射*/\n\n- 注：本规范不采用ER-99的标注：*[L*法属圭亚那库鲁航天中心*]*。\n\n/[L*武汉*]/[L*长江大桥*]/\n\n/[L*上海*]/[L*中山公园*]/\n\n- 注：尽管其它城市也有长江大桥和中山公园，但在当地它们已构成完整的地名，所以应单独标注。\n\n/*位于*/[L*朝阳门*]/*外*/*商务*/*区*/*之中*/*，*/\n\n/[L*盛华公寓*]/*坐落*/*于*/[L*西直门*]/*内*/[L*冠英园小区*]/\n\n- 注：内、外都不在标注范围之内，但如果地名中的内、外去掉不能说明是一个完整的地名时，内、外要标注在地名内。如：\n\n/[O*外交部*]/*位于*/[L*北京市*]/[L*朝阳门内南小街*52*号*]/\n\n/[L*西直门外大街*71*号*]/\n\n4.2.3并列的地名应分别标注\n\n对于并列的多个地名应分别标注。对于嵌套在地名中的人名、地名和机构名不再单独标注。例如：\n\n/[L*中*]/[L*意*]/*双方*/ ----*中意*是词表词，作为国名时要切开。\n\n/[L*香港*]/*和*/[L*澳门特别行政区*]/\n\n/*目前*/*已*/*有*/[int*１２个*]/[L*中*]/*、*/[L*东欧*]/*国家*/\n\n/[L*北京*]/[L*上海*]/\n\n/[L*科*]/[L*伊*]/*边境*/\n\n#### 4.2.4跨国家的和国家内部的地名\n\n/[L*西非*]/*国家领导人*/\n\n/*从*/[L*陕*]/[L*甘*]/*革命*/*老区*/*到*/*沿海*/*经济特区*/*，*/\n\n/[L*亚太*]/----亚太是词表词，它是一个地名，而不是两个地名。\n\n/[L*近东*]/*和*/[L*北非*]/\n\n##### 4.2.4.1表示地理方位的名词\n\n一些表示地理方位的名词如*南半球、北半球、江南、江北、西南、西北、华南、华北、华中、东北*等虽然不完全具备确指性，也要作为地名标注为*L*。\n\n/[L*汉水*]/*流域*/*、*[L*西南*]/*地区*/*东部*/\n\n/[L*江南*]/*大*/*部*/*、*[L*华南*]/*有*/*小*/*到*/*中雨*/\n\n/*近*/[dur*两天*]/*造成*/[L*东北*]/*、*/[L*华北*]/*地区*/*的*/*降雨*/*天气*/*系统*/\n\n/*迫使*/[L*北半球*]/*的*/*副热带*/*高压带*/*在*/[L*青*][L*藏*]/*地区*/\"/*断裂*/\"/\n\n- 注：上述地名后面的方位词*南部、北部、东部、西部*不应包括在地名的括号里，\n\n因为其所指的区域是更不确定的。\n\n##### 4.2.4.2方位词修饰地名实体时要整体标注为L\n\n/[L*东西九龙*]/ ----这是一个并列的地名。\n\n/*一代*/*又*/*一代*/*海*/*测*/*官兵*/*犁*/*波*/*耕*/*浪*/*于*/[L*南中国海*]/*，*\n\n/[L*北爱尔兰*]/\n\n/[L*中西伯利亚*]/\n\n- 注：ER-99将此例标为*中*/ [L*西伯利亚*]。我们认为它整体是一个专指性的地名。\n\n/[L*中南美*]/\n\n/[L*东南亚*]/\n\n- 注：ER-99要求把上面两个地名分别标注为[L*中*]/[L*南美*]*和*/[L*东*]/[L*南亚*]/*。其实中南美*指*中美*和*南美*两个地名，而*东南亚*是一个地名。这样的细节需要专门的地理知识才能做出判断。所以我们不遵循ER-99的这条规则。\n\n#### 4.2.5地名实体受时间词修饰时，时间词不标\n\n/*前*/[L*苏联*]/\n\n/*前*/[L*南*]/*地区*----*南*指南斯拉夫，时间词*前*不标。\n\n#### 4.2.6 只有经纬度在一起时才能标注为 **L**\n\n只有经纬度在一起时才能标注为L，否则经度或纬度单独标为角度*ang*。如：\n\n/*震*/*中*/*位于*/[L*北纬三十六点二零度，东经九十点二九度*]/\n\n/*并*/*将*/*卫星*/*定点*/*在*/[L*东经*110.5*度赤道*]/*上空*/*。*/ \n\n/*震*/*中*/*位于*/[ang*北纬*30.5*度*]/*，*/\n\n#### 4.2.7天体的标注\n\n/[L*宇宙*]/ \n\n/[L*地球*]/ \n\n/[L*太阳*]/\n\n/[L*太阳系*]/ \n\n/[L*银河*]/\n\n/[L*银河系*]/\n\n/[L*月亮*]/ \n\n/[L*海王星*]/\n\n/[L*东方红三号*]/\n\n/[L\"*鑫诺１号*\"*卫星*]/\n\n- 注：火箭只是卫星的发射工具，故火箭型号不作为星体标注。\n\n/[dat*９６年２月１５日*]/*长征*/[ord*三号乙*]/*火箭*/*发射*/*失利*/，\n\n/*长*/[ord*二*]/*捆*/*火箭*/ ----*全名为*\"*长征二号捆绑式运载火箭*\"。\n\n### 4.3不作地名标注的示例\n\n/[L*阿*]/[L*以*]/*冲突*/\n\n- 注：ER-99和MT-2认为阿（阿拉伯）不是一个特定国家的简称，本规范不采纳他们的规定。\n\n/*回答*/*了*/[L*中*]*外*/*记者*/*的*/*提问*/ ---*外*不标。\n\n#### 4.3.1地区一般不作为地名的一部分标注\n\n仅当*地区*特指行政单位时，才被视为地名的一部分。一般情况下，*地区*泛指一片地方，不是地名的一部分。若不能确定时，*地区*不作为地名的一部分标注。\n\n/[L*港*][L*澳*][L*台*]/*地区*/ -----*港澳台*是词表词。\n\n/[L*巴尔干*]*地区*/\n\n/[L*临沂*]/*地区*/*现*/*更名*/*为*/[L*临沂市*]/\n\n#### 4.3.2平原、山脉、山区、盆地、沙漠、流域不在标注范围内\n\n*平原、山脉、山区、盆地、沙漠、戈壁、流域、故里、故居、纪念馆、风景区、开发区、经济区*等都不在地名标注范围内。但当某某故居、故里、纪念馆成为一个对外开放的旅游景点时，才作为地名标注。如：\n\n/[L*云*][L*贵*]*高原*/\n\n----*云贵高原*是词表词不可分割，但云、贵要分别标注*L*。\n\n/[L*成都*]/*平原*/\n\n/[L*秦岭山*]/*脉*/\n\n/[L*秦*]/[L*巴*]/*山区*/\n\n/[L*四川*]/*盆地*/ \n\n/[L*撒哈拉*]*沙漠*/ ----*撒哈拉沙漠*是词表词。\n\n/[L*长江*]/*流域*/\n\n/[L*毕加索故居*]/\n\n/*造型*/*典雅*//*毗邻*/[L*青云岩*]/*风景区*/*及*/[L*北山湾*]/*旅游区*/\n\n*[L*约旦河西岸*]*----因为*约旦河西岸*是专指。\n\n/[L*海峡两岸*] /  ----指*台湾湾海峡两岸*。\n\n/[L*两岸*]/\n\n- 注：词表词*两岸*只有在表示台湾海峡两岸时，才作为地名标注为*L*，当作为*江河、湖泊*的两岸时，*两岸*要切分标注。如：\n\n/[L*长江*]/*的*/*丰姿*/*和*/[int*两*]/*岸*/*的*/*美景*/*尽收眼底*/*。*/\n\n/*祖国*/[L*大陆*]/\n\n- 注：内地虽然指中国大陆，但不作为地名标注，这里遵从了ER-99的规定。特区只有在确指是香港和澳门时才作标注。如：\n\n/*来自*/*内地*/*和*/[L*香港特区*]/\n\n/[L*特区*]/*政府*/*和*/[L*香港*]/*同胞*/*正*/*以*/*喜悦*/*的*/*心情*/ \n\n/[L*中国*]/[L*厦门*]/*经济特区*/\n\n#### 4.3.3对语言文字前的单音节地名不标，双音节的地名标注为L\n\n*英语*----对*英*不标注。\n\n*汉语*----对*汉*不标注。\n\n*中文*----对*中*不标注。\n\n/*对*/[L*西藏*]/*地区*/*的*/*藏语*/*广播*/\n\n/*主张*/*台语*/*在*/[L*台*]/\n\n/*用*/[L*四川*]/*话*/ ----如果*语、文*前面的地名为双音节时，就要标注。\n\n/[L*荷兰*]/*语*/\n\n#### 4.3.4以族或裔结尾的词组中地名也要标注\n\nMT-2和ER-99规定：以族或裔结尾的词组中的地名不标注。因此*华裔*、*汉族*中的*华*和*汉（指汉族）*都不作为地名标，但*华人、华侨、华商、中医、中草药、中餐馆、亚运会、奥运会*里的*华、中、亚、奥*仍需标注*L*。本规范不采用这一规则。作为民族的名字，单音节的不标，双音节的标*L*。\n\n下面是一些标准实例：\n\n/[L*美*]*籍*[L*华*]人----\"美籍华人\"是词表词。\n\n/*目的*/*是*/*促进*/[L*塞浦路斯*]/*西*/*族*/*与*/*土*/*族*/*的*/*和解*\n\n/*她*/*和*/*同*/*是*/[L*日*]/*裔*/[int*三*]/*世*/*的*/*男*/*友*/\n\n/*通过*/*在*/[L*中*]*医药*/*宝库*/*里*/*寻找*/*线索*/\n\n/*人们*/*纷纷*/*拥向*/[L*中*]*餐*/*馆*/*，*/*一时间*/*人满为患*/\n\n/[L*吉普赛*]/*人*/----*吉普赛*不是词表词。\n\n/[L*印地安*]/*民族*/*；*/ ----*印地安人*是词表词。\n\n## 第五章 机构名\n\n机构名包括：股票（证券）交易所、国家或国际组织、商业团体（公司、企业、工厂）、电视台、广播电台、报刊杂志、出版社、政党或党派、学校、科研院所、医院、诊所、邮电局、乐队、体育运动队、联盟、议会或代表大会、军队、咖啡厅、酒吧、饭店、旅馆，以及虚构的机构等。\n\n### 5.1机构名标注规则\n\n机构名的后缀应视为机构名的一部分。\n\n| **序号** | **情况** | 标记方法 |例子 |\n| --- | --- | --- | --- |\n| 1 | 普通名字+机构名 | 整体标出 | *[O*板桥市胜捷公司*]* |\n| 2 | 地名+机构名 | 机构名整体标出 | [O*北京市电信局*]*[O*台北县立莺歌高职*]*[O*台北看守所*]*[O*基隆长庚医院*]*[O*东直门敬老院*]机构名的关键词如：幼儿园、各级学校、科学院、部委、实验室、工厂、公司、报刊杂志、出版社、大使馆、领事馆、咖啡店、快餐店、饭店、酒店、旅馆等 |\n| 3 | 人名+机构名 | 机构名整体标出 | *[O*李嘉诚基金会*]* |\n| 4 | 简称 | 一律整体标注 | *[O*北约*]*[O*上轮集团*]----*指上海轮胎集团*[O*白宫*]/*官员*/表示 |\n\n### 5.2机构名标注细则\n\n#### 5.2.1机构名标注实体示例\n\n/[O*国防部*]/*长*/[P*迟浩田*]/ \n\n/[O*美国国防部*]/*长*/[P*佩里*]/\n\n/[O*台北县地政局地权课*]/\n\n/[O*地政局*]/\n\n/[O*政风室*]/*接*/*获*/*检举*/*调查*/\n\n/[O*国军北投医院*]/\n\n/[O*三重地政事务所*]/\n\n/[O*台湾银行宜兰分行*]/ \n\n/[O*省立关山工商*]/\n\n/[O*基隆市光隆家商*]/\n\n/[O*东信国小*]/ \n\n/[O*安乐国中*]/\n\n/[O*原住民委员会*]/\n\n/[O*连萧全国竞选总部*]/\n\n/[O*北京钓鱼台国宾馆*]/ \n\n/[L*浙江*]/[O*温州大酒店*]/ \n\n/[O*松下电工株式会社*]/\n\n/[O*公司*]/*英文*/*名称*/[O *HUNAN* FORE *SCAPE* TECHNOLOGY*CO*．，*LTD*]/\n\n/[O*朝鲜人民武装力量部*]/*副*/*部长*/\n\n/[O*美国海军*]/ \n\n/[O*欧共体*]/\n\n/[O*中国国家生育委员会*]/ \n\n/[O*中国奥林匹克队*]/\n\n/[O*披头四*]/ \n\n/[O*飞虎队*]/\n\n/*敢死队*/ -----泛指不标。\n\n/*但是*/[O*共和党*]/*人*/*说*/\n\n/[O*土耳其议会外交关系委员会*]/ \n\n/[O*终战*50*周年国会议员联盟*]/\n\n/*记者*/*来到*/[O*中山医科大学第一附属医院住院部*]/\n\n/[O*中共中央政治局*]/*常委*/*、*/[O*中央纪委*]/*书记*/[P*尉健行*]/\n\n- 注：中国共产党的简称中共或共要标注为O。例如：\n\n/[ord*第二次*]/[O*国*]/[O*共*]/*合作*/\n\n- 注：类似的简称党，由于专指性不强，不标，如：\n\n/但/这种/现象/的/产生/，/是/同/党/和/国家/尊师重教/的/方针/背道而驰/的/，\n\n/全国/\"/[dat三八]/\"/红旗手/、/全国/优秀/共青团员/\n\n- 注：\"三八红旗手\"是词表词。但如果\"三八\"在文中被双引号断开，就要单独表为dat。另外，词表词共青团员、共产党员、共产党人、中的机构名不确指，所以一律不标。\n\n/[O*中共中央政治局常委会*]/\n\n- 注：常委会可以是机构名，常委则不是。\n\n/*党*/*的*/[O*十四大*]/*以来*/\n\n- 注：中共的*X中全会*不是机构名，除了词表词*三中全会*什么也不标以外，数词*X*应单独标注为*ord*。例如：\n\n/*根据*/*党*/*的*/[ord*十五届*]/[ord*二*]/*中*/*全会*/\n\n/[O*八届全国人大*]/*代表*/[P*陈妙珍*]/\n\n/[O*西藏政协*]/*委员*/*强调*/*，*/*必须*/*旗帜*/*鲜明*/*地*/*反对*/*民族*/*分裂*/\n\n[O*澳门中华总商会*]/*会*/*董*/*兼*/[O*青年委员会*]/*副*/*主任*/\n\n/[O*足协*]/*杯赛*/*冠军*/[O*北京国安队*]/ ----*杯赛*是词表词。\n\n/[O*以国家电视一台*]/ ----指以色列国家电视一台\n\n/[L*汉城*]/[O*路透*]/*电*/\n\n/*前*/[L*苏联*]/[O*切尔诺贝利核电站*]/*泄漏*/*事件*/\n\n/*参加*/*这次*/*比赛*/*的*/*还有*/[O*日本*]/*、*/[O*俄罗斯*]/*、*/[O*美国*]/*、*/[O*德国*]/\n\n和/[O*意大利队*]/*。*/\n\n/*前往*/[O*解放军驻港部队总部*]/*慰问*/*驻军*/\n\n/[O*第四届和平小天使台湾访问团*]/*抵达*/[L*重庆直辖市*]/\n\n/[O*塔里班*]/*部队*/*已经*/*到达*/[P*杜斯塔姆*]/*将军*/*的*/*家乡*/\n\n/*用*/*公款*/*购买*/[O*靖国神社*]/*和*/[O*护国神社*]/*的*/*祭祀*/*品*/\n\n/*纪念币*/*正面*/*是*/*由*/[O*解放军*]/*军徽*/*光，*/*八一南昌起义*/*和*/[O*解放军*]/[O*陆*]/[O*海*]/[O*空*]/*三军*/*战士*/*的*/*图案*/\n\n- 注：词表词八一南昌起义*是一个事件，不是机构名。三军*是词表词，所以数字*三*不作为*int*标注。\n\n- 注：股市报导中的企业和公司名不论其前后有没有外文字符，一律作为一个整体\n\n标注成*O*。例如：\n\n/[O*ＳＴ辽物资*]/[dec*１４．１４１*]/[O*宁波中百*]/[dec*２０．３５４*]/\n\n/[O*ＤＲ沪港机*]/[dec*１１．１９４*]/[O*鲁北化工*]/[dec*８．０５１*]/\n\n- 注：商城或百货公司本应标注为L，但作为股市中时企业时应标注为O。\n\n- 注：股票指数在没有明确说明是多少元的情况下一律标注为*int*或*dec*。\n\n- 注：被命名的轮船、飞机、机车应标注为*O*。例如：\n\n/*却*/*购*/*回*/*了*/[int*３张*]/ [*O\"长月\"号轮船*]/*船票*/*，*/ \n\n/[O*泰坦尼克号游轮*]/*上*/*的*/*这*/*对*/*情人*/*实在*/*浅*/*得*/*很*/*。*/ \n\n/[O*美国\"哥伦比亚\"号航天飞机*]/*上*/*的*/*宇航员*/\n\n#### 5.2.2机构名的后缀是机构名的一部分\n\n机构名的后缀是机构名的一部分，即要准确的标出机构名的最长边界（机构名的全称）。机构名中可以包含人名、地名和机构名，但对于它们不再单独标注。例如：\n\n/[O*苗栗县环保局*]/\n\n/[O*卫生署桃园医院*]/\n\n/[O*兰阳民生医院*]/*前身*/*为*/[O*吴外妇科*]/\n\n/[O*台北爱乐青年管弦乐团*]/\n\n/[O*行政院农委会林业试验所福山分所*]/ \n\n/[O*宋庆龄基金会*]/\n\n/[O*上海轮胎橡胶（集团）股份有限公司*]/ \n\n/[O*中国驻日本大使馆*]/\n\n/[O*美国白宫*]/\n\n/*前*/[O*中国新华社香港分社*]/*社长*/[P*许家屯*]/ \n\n[O*清华大学计算机系人工智能实验室*]/\n\n[O*中保财产保险四川省分公司*]/\n\n#### 5.2.3国家（或国际）立法部门或行政部门标注为机构名\n\n/*当选*/[O*国会*]/*议员*/\n\n/[O*内阁*]/*改组*/*将*/*会*/*在*/[dat*八月底*]/*前*/*完成*/\n\n/*前*/[O*内阁官房*]/*长官*/[P*山静六*]/\n\n/[P*刹瓦什*]/*向*/[O*宪政法庭*]/*提出*/*动议*/\n\n5.2.4地名和机构名紧邻时的情况\n\n地名和机构名的关系一般有以下两种情况：\n\n(1)表示所属关系（如：法国航空航天局，航空航天局隶属法国）。\n\n(2)表示地理位置关系（如：北京邮电大学表示大学位于北京，而不是隶属于北京）。\n\n地名和机构名之间还可能有更复杂的情况，这里不予讨论。\n\n##### 5.2.4.1规则一\n\n如果机构名以一个地名开头，而且删除这个地名后所剩部分不再是一个具有特指性的机构名，那么该地名必须留在机构名中作为该机构名的一部分标注；\n\n/[O*北京大学*]/ \n\n/[O*深圳中学*]/\n\n/[O*复旦大学专用集成电路与系统实验室*]/\n\n/[O*东南大学*]/[O*深圳宝安设计院*]/\n\n##### 5.2.4.2规则二\n\n如果机构名前面还有一个或多个地名，那么该机构名与前面紧邻的地名应当分开标注。\n\n如：\n\n/[L*中国*]/[O*北京大学*]/\n\n/[L*中国*]/[L*广东*]/[O*深圳中学*]/\n\n/[L*北京*]/[L*昌平*]/[O*十三陵抽水蓄能电站*]/\n\n##### 5.2.4.3规则三\n\n如果一个机构名的开头不是地名，那么当它前面邻接一个或多个地名时，只有其中与该机构名紧邻的那个地名需一起标注。例如：\n\n/[O*上海同济大学*]/\n\n/[L*中国*]/[O*上海同济大学*]/ \n\n/[O*湖北省武钢三中*]/\n\n##### 5.2.4.4规则四\n\n如果一个机构名本身以两个或两个以上并列的地名开头，则这些地名都要留在该机构名中。如果在它前面再出现其它地名时，一律同该机构名分开标注。但是如果上一级地名不能管辖下一级地名时，要把上一级地名标注在机构名内。\n\n例如：\n\n/[L*洛杉矶*]/[O*亚太法律中心*]/ \n\n/[L*香港*]/[O*中港贸易协会*]/ \n\n/[O*广东亚洲大酒店*]/\n\n/[O*澳大利亚维多利亚投资公司上海办事处*]/*》*/，\n\n/[O*澳大利亚维多利亚投资公司*]/*》*/\n\n- 注：\"广东\"与\"亚洲、澳大利亚与维多利亚\"都不属于上、下级管辖关系，所以要把上一级地名标注在机构名内。\n\n##### 5.2.4.5更复杂的情况\n\n在更复杂的情况下，我们可能无法判定某机构名究竟是以一个还是两个地名开头的。这时可按规则5.2.5和5.2.6来处理。\n\n例如，*洛杉矶台北经济文化办事处*\n\n究竟是A：*[L*洛杉矶*]/[O*台北经济文化办事处*]*\n\n还是B：*[O*洛杉矶台北经济文化办事处*]*\n\n这时，默认的标注方式是B(理由见5.2.8)。\n\n##### 5.2.4.6地名概念比较模糊的情况\n\n如果该地名比较模糊，而标注者又没有足够的知识来判断某机构名的开头是否是一个地名。就标注到一个比较明确的地名，\n\n例如：*印度尼西亚莫巴蒂努山打腊航空公司*中的*莫巴蒂*·*努山打腊*不知道是不是地名。但至少知道一旦拿走了这个字符串，剩下的字符串已不构成专指性的地名。此时，按规则2.5的标注方式应是：\n\n/[L*印度尼西亚*]/[O*莫巴蒂*·*努山打腊航空公司*]/\n\n/[O*河北沙岭子电厂*]/\n\n----*沙岭子*是一个乡镇的地名，河北和内蒙古都有一个沙岭子镇，地名的概念比较模糊，故标注在机构名内。\n\n/*国际*/*著名*/*的*/[O*加拿大*B＋*H国际建筑师事务所*]/\n\n##### 5.2.4.7紧邻的地名和机构名不构成修饰关系的情况\n\n一个地名后紧邻一个机构名，但它们不构成修饰关系，则一律分开标注。\n\n/*促进*/*了*/[L*中国*]/[O*东盟*]/*的*/*合作*/\n\n/*在*/[L*日内瓦*]/[O*联合国*]/&*人*/*权*&/*会议*/*上*/\n\n更典型的例子需借助上下文来判断，如：\n\n/*促进*/*了*/[L*中国*]/[O*微软*]*的合作*/\n\n/[O*中国微软*]/*即将*/*发布*/*新产品*/\n\n- 注：如果标注者不能判断它们是不是修饰关系，则默认为分开标注，如：\n\n/[L*中国*]/[O*微软*]/\n\n/[O*美国众议院*]/\n\n/[L*重庆*]/[O*长江救助打捞公司*]/ /[L*日本*]/[O*东京股市*]/ ----错误标注！\n\n/[L*日本*]/[L*东京*]/*股市*/ ----正确标注。\n\n/[L*美国*]/[L*华盛顿*]/[O*三普证券公司*]/ ----错误标注!\n\n/[L*美国*]/[O*华盛顿三普证券公司*]/ ----正确标注。\n\n/[L*华盛顿*]/[O*美国国务院*]/\n\n/[L*瑞典*]/[O*斯德哥尔摩国际和平研究所*]/\n\n#### 5.2.5会议、晚会、运动会等以会结尾的短语是事件，不作机构名标注\n\n/*泛*/[L*美*]/*运动会*/\n\n/[L*中国*]/[ord*第一届*]/*人工智能*/*大会*/ \n\n/[ord*第四届*]/[L*中*]/[L*法*]/*经济*/*研讨会*/\n\n/[ord*第三届*]/[L*海峡两岸*]/*水利*/*科技*/*交流*/*研讨会*/\n\n----以上几例为事件，不是机构名。\n\n/[O*中国人工智能协会*]/\n\n/[O*中国人工智能联合会*]/ ----为机构名。\n\n当会议指议会(congress)或代表大会(chamberofdeputies)时，应视为机构名。但是要注意:虽然议会或代表大会是机构名，但是议会或代表大会中的某一次会议是一个事件，不是机构名。为了更明确的区分各种情况，我们用以下例子说明：\n\n/*通报*/*了*/[O*八届政协*]/[ord*五次*]/*会议*/*的*/*各*/*项*/*安排*/\n\n/[O*全国政协*]/[ord*八届*]/[[ord*五次*]/*会议*/*将*/*于*/\n\n/*听取*/*和*/*审议*/[O*全国政协八届五次会议常务委员会*]/*报告*/\n\n/*审议*/[ord*八届*]/[ord*五次*]/*会议*/*提案*/*审查*/*情况*/*的*/*报告*/\n\n- 注：*八届五次会议*、*五次会议*是一个事件，不应标注为机构名。但是这次会议的组委会、委员会应视为机构名。例如：\n\n/[O*八届全国人大*]/[ord*五次*]/*会议*/\n\n/[O*政协九届一次会议*]/ --错误标注！\n\n/[O*中国共产党第十五次全国代表大会*]/\n\n/[O*九届人大*]/[ord*一次*]/*会议*/\n\n/[O*中国全国人大*]/ \n\n/[O*中共十五大*]/ \n\n/*各级*/*人大*/*常委会*/ --不是专指，故不标。\n\n/[O*中国科协*]/[ord*第五次*]/*全国代表大会*/\n\n/[L*湖南省*]/[ord*六届*]/[ord*二次*]/*全*/*委*/*会议*/\n\n/*向*/*同级*/*人民代表大会*/*或*/*人民代表大会常务委员会*/*提请*/*审议*/\n\n- 注：*全国人民代表大会*和确指的省、市人民代表大会及其常委会、常务委员会需作为机构名标注。泛指的人大、中央银行、人民银行、&*农*/*发*/*行*&不作为机构名标注。\n\n/[O*临澧县人大*]/*抓*/*村*/*级*/*财务监督*/*一瞥*/*（*/*监督*/*广角*/）\n\n/*由于*/*各级*/*人大*/*代表*/*的*/*有效*/*监督*/*，*/[dat*去年*]/*以来*/*该*/*县*/*各*/*村*/*村*/*务*/*情况*/*出现*/*好转*/*，*/\n\n- 注：在地名*国会大厦*中，*国会*不可作为机构名标注，否则就出现嵌套了。\n\n/[L*国会大厦*]/\n\n- 注：\"联合国大会\"及其简称\"联大\"都是词表词，但不可整体标为O。如：\n\n/[O*联合国*]*大会*/*于*/[dat*１９９２年*]/*批准*/*了*/*这*/*一*/*条约*/*。*\n\n/[P*沈国放*]/[dat*２７日*]/*在*/[O*联*]*大*/*全体*/*会议*/*上*/*表示*/，\n\n- 注：*会*也可能出现在一般的机构名中，如：\n\n/[O*红十字协会*]/\n\n#### 5.2.6用我们、我等代词修饰的机构名，只对机构名进行标注\n\n/*我国*/[O*共产党*]/ \n\n/*我们*/[O*清华大学*]/\n\n- 注：根据上下文是确指的某公司、单位名称的简称要标注为机构，否则不标注！但如果在公司、集团等词前面有本、我、该等字样时，此处的公司、集团不进行标注。其他特殊情况依据上下文进行标注。如：\n\n/*凡*/*《*/[O*克罗伏特缓冲器股份有限公司*]/*股份*/*》*/*记名*/*的*/*持有*/*人*/*均*/*为*/*本*/*公司*/*股东*/*。*/\n\n/*我*/*公司*/*出资*/*总额*/[mon*50万元*]/\n\n/[O*港资陕西华懋实业公司*]/*总经理*/[P*商铭渔*]/*，*/*受*/[O*公司董事会*]/*委托*/*来到*/[O*咸阳市西北地勘局二一五医院*]/*看望*/[O*公司*]/*保安*/*员*/[P*韩玉刚*]/*，*/\n\n#### 5.2.7大使馆和领事馆的标注\n\n当大使馆(或领事馆或其它外交使团)所代表的国家和所在地区相连时，整体标为机构名。如：\n\n/*后来*/*调*/*任*/[O*美国驻洪都拉斯大使馆*]/\n\n当大使馆(或领事馆或其它外交使团)所代表的国家或所在地没有出现在上下文中，或者在描述范围内不连续，那么存在两种情况：\n\n（1）大使馆所代表的国家和大使馆（领事馆）相连，此地名和大使馆一起标记 为机构名。如：\n\n/*前往*/[L*香港*]/*的*/[O*洪都拉斯领事馆*]/\n\n（2）大使馆所在地和大使馆（领事馆）相连，此地名应单独标记，整体不作为机构名。如：\n\n/[L*美国*]/*在*/*通过*/*驻*/[L*金沙萨*]/*大使馆*/*和*/*其它*/*正常*/*管道*/\n\n- 注：虽然*驻金沙萨大使馆*是一个连续的短语，但它的实际意思是*美国（或*X*国）驻金沙萨大使馆*，而不是什么*金沙萨（的）大使馆*。因此在这里*大使馆*不视为机构名。\n\n#### 5.2.8生产厂家要标注为机构名，产品则不标\n\n这里定义的产品范围较广，不仅包括生产厂家生产出来的产品（如自行车等），还包括计算出来的产品（如：股票指数）、媒体产品（如：电视节目）\n\n/[O*道琼*]/*工业*/*平均*/*指数*/\n\n----因为股票指数可以视为产品，那么*道琼*就可以视为生产厂家。\n\n/[O*纳斯达克*]/*指数*/ ---原因同前。\n\n/[O*太原刚玉*]/[dec*１０．５８１*]/ \n\n/[O*咸阳偏转*]/[dec*１６．１１２*]/ \n\n/[O*深华发Ａ*]/[dec*１５．６６３*]/ \n\n/[O*渝开发*A]/\n\n#### 5.2.9报纸、广播电台、电视台和杂志的名字要标为机构名\n\n新闻媒体（如：报纸、广播电台、电视台和杂志等）的名字要标为*O*，但报刊、电视栏目的名字不标。例如：\n\n/[O*美国之音*]/*记者*/*表示*/\n\n/[O*人民日报*]/*海外*/*版*/[ord*第三版*]/\n\n/*《*/[O*泰晤士报*]/*》*/*援引*/*一个*/*国际*/*专家*/*委员会*/\n\n/[O*中央电视台*]/*《*/*焦点*/*访谈*/*》*/*、*/*《*/*东方*/*时空*/*》*/*主持人*/\n\n/[O*武汉电视台*]/*《*/*科技*/*之*/*光*/*》*/*栏目*/*的*/*《*/*科学家*/*，*/*您好*/*》*/*专栏*/ \n\n/[O*美国《科学》杂志*]/\n\n/[O*美国探索电视网*]/\n\n/*创办*/*《*/[O*深圳房地产快讯*]/*》*/ \n\n/*办*/*好*/*《*/[O*中外房地产导报*]/*》*/\n\n#### 5.2.10特殊情况\n\n***民族不作为机构名***\n\n***泛指的*部队不作为机构名**\n\n***政府不作为机构名***\n\n***学术或商务会议(conference,meeting)不作为机构名***\n\n***交易会不作为机构名***\n\n***运动会不作为机构名***\n\n***联赛不作为机构名***\n\n#### 5.2.11特殊情况示例\n\n/[L*中国*]/[L*天津*]/*出口*/*商品*/*交易会*/\n\n/[L*中国*]/[O*天津出口商品交易会*]/ ----错误标注！\n\n/[L*中国*]/*政府*/ ----*不把政府*标为机构名。\n\n/[L*非洲*]/*维持*/*和平*/*部队*/ ----*不把部队*标为机构名。\n\n/[L*中国*]/*公安*/*部门*/ ----*不把部门*标为机构名。\n\n/[O*中国公安部门*]/ ----错误的标注！\n\n- 注：标注并列的机构名（*O*）时，连接词和标点符号不进入标注范围。例如：\n\n/[O*上海*]/*、*/[O*北京人类基因组研究中心*]/\n\n/[P*贺国中*]/*分别*/*任*/[O*一*]/*、*/[O*四*]/*、*/[O*七团*]/*党代表*/\n\n- 注：上述情况和标注并列的序数（*ord*）不同，连接词和标点符号是否进入标注范围取决于序数词所修饰的词语。例如：\n\n/*获得*/*个人*/[ord*一、二、三等*]/*奖*/\n\n/[ord*一*]/*、*/[ord*二*]/*、*/[ord*三*]/*产业*/\n\n/*书店*/[ord*三、四层*]/\n\n- 注：*中央*不作为机构名，但党中央*标为机构名。\n\n/*在*/*中央*/*的*/*领导*/*下*/\n\n/*以*/[P*胡锦涛*]/*同志*/*为*/*核心*/*的*/[O*党中央*]/*周围*/\n\n#### 5.2.12地名和机构名容易混淆的情况\n\n/[L*人民大会堂*]  ----地名。\n\n/[O*五角大楼*]/*发言人*/*说*/*，*\n\n/[O*白宫*]  ----机构名。\n\n/[O*克里姆林宫*]/*表示*  ----机构名。\n\n/*在*/[L*总统府*]/*分别*/*约见*/*了*/*多*/*位*/[O*国民党*]/*中*/*常委*/*检察官*/\n\n- 注：*总统府*标注为L而不是*O*。这是因为有的国家有多处总统府，所以不能把它们视为国家或政府的唯一代表。\n\n- 注：下面的例子中出现的类似单位名称的，因不是确指，而且是出现在各种条令、合同中，适合任何一个省、市、县的单位机构名称，所以不能作为一个机构名称标注为*O*。如：\n\n/*本*/*合同*/*正本*/[int*三份*]/*，*/*出租*/*人*/*、*/*承租*/*人*/*、*/*市*/*公证处*/*各*/*执*/*一*/*份*/*。*/*副本*/*若干*/*份*/*，*/*报*/*市*/*经济*/*委员会*/*、*/*市*/*经济体制*/*改革*/*委员会*/*、*/*市*/*财政*/*局*/*、*/*劳动局*/*、*/*税务局*/*、*/*审计*/*局*/*、*/*工商*/*行政管理*/*局*/*、*/[O*中国人民银行*]/*市*/*分行*/*、*/[O*中国工商银行*]/*、*/*市*/*分行*/*等*/*有关*/*部门*/*备案*/ */*本*/*合同*/*在*/*履行*/*中*/*如*/*发生*/*争议*/*，*/*双方*/*应*/*协商*/*解决*/*；/*协商*/*不*/*成*/*时*/  /*任何*/*一方*/*均*/*可*/*向*/*工商*/*行政管理*/*局*/*合同*/*仲裁*/*委员会*/*申请*/*调解*/*或*/*仲裁*/\n\n## 第六章 数字串标注总则\n\n数字串（**Factoid**）包括时间表达式（**TIMEX**） 、数字表达式（ **NUMEX** ）、度量表达式（**MEASUREX**）和地址表达式（**ADDREX**）等**4**大类，***27***个小类，详见表**1-1**。标注数字串的一条重要原则就是：它的标记不得插入到词表词的内部（见**1.5.2.4**）。\n\n### 6.1时间表达式\n\n时间表达式（*TIMEX*）包括日期（*dat*）、时间（*tim*）和时段（*dur*）三小类。所有小于一天的时间都被定义为时间（*tim*），如秒，分，小时。一天或者大于一天的时间单位则属于日期（*dat*），如*天，日，星期，礼拜，月，季度，年，五年，十年，世纪*等。时段（dur）通常也使用日期和时间中的单位，如月、年、时、分*等。对此标注者要注意区分。\n\n将日期、时间同时段区分开来有时是困难的，下面分别给出它们的定义。\n\n#### 6.1.1日期（dat）和时间（tim）的定义\n\n日期和时间在一维的时间坐标轴上有相对确定的位置。小于一天的时间都被定义为时间。一天或者大于一天的时间则属于日期。\n\n/[tim*8*点30*分*]/\n\n/[dat*今天*]/[tim*晚上*]/ ----*晚上*是词表词。\n\n/[dat*昨天*]/[tim*夜里*]/ ----*昨天*和*夜里*都是词表词。\n\n/[tim*昨夜*]/ ----*昨夜、昨晚*都是词表词，只能整体标*tim*。\n\n/[dat*昨*]/[tim*晚*]/ ----错误的标注！\n\n/[dat*春节*]/---在每一年中，是比较固定一天或几天。\n\n/[dat*1999*年*]/---以*年*为单位，与别的年份相区别。\n\n/*在*/\"/[dat*六五*]/\"/*中*/---以*五年*为单位，与别的*五年*相区别\n\n- 注：严格地说，每一个*dat*或*tim*都占据了一个时间段，因此这里出现的*期间*和*中*，不能作为标注时段的理由。\n\n/\"/[dat*九五*]/\"/计划\n\n/[dat\"*九五*\"*初*]/\n\n/*仅*/*\"*[dat*八五*]/*\"*/*期间*/*就*/*达*/[mon一百一十五亿元]/。\n\n/[dat*下半年*]/---以*半年*为单位，与*上半年*相区别。\n\n/[dat*二十世纪*]/---以一百年为单位，与别的*世纪*相区别。\n\n/*为*/*庆祝*/[O*北京大学*]/*建*/*校*/[dat*１００周年*]/*，*/\n\n/[dat*民国八十六年*]/\n\n/[dat*民国六十年代*]/\n\n/[dat*八十八年下半年*]/*及*/[dat*八十九年*]/*中央*/*统筹*/*分配*/*款*/*，*/ \n\n/[dat*公元二千年*]/\n\n/[dat*今年九月*]/\n\n/*\"*/[O*迈特兴华*]/*\"*/*杯*/[ord*首届*]/*全国*/*象棋*/*大师*/*赛*/*于*/[dat*今日*]/*收*/*秤*\n\n/[dat*１９９７年下半年*]/*，*/ \n\n/*可*/*于*/[dat*农历年*]/*前*/*迁居*/*。*/ \n\n/[tim*第七十三分钟*]/\n\n/[tim*中午*12*点*]/\n\n/[tim*格林威治时间*5*时*59*分*]/----含有地名。\n\n/[dat*第二天*]/[tim*一大早*]/*，*----*一大早*是词表词。\n\n/*在*/[dat*今年暑期*]/*大学生*/*送*/*科技*/*下乡*/*活动*/*中*/，\n\n/*大约*/[tim*七点*]/*到达*/*----大约*不标。\n\n/[tim*晚上大约七点*]/*到达*/\n\n- 注：*大约*被两个*tim*包围，分割不开，所以整体标上。这条标注规则遵照了ER-99和MET-2的标准。\n\n- 注：事件戊戌变法、辛亥革命、甲午战争、五四运动等都是词表词，其中的日期不标注。但当戊戌、辛亥、五四单独出现时，应作为日期来标注。例如：\n\n/*与*/*稍*/*后*/*的*/*辛亥革命*/*，*/*都*/*有*/*相通*/*的*/*地方*/，\n\n/*在*/[L*香港*]/*回归*/[dat*周年*]/*前夕*/*和*/*\"*/*七七事变*/*\"*/*纪念日*/*，*/[dat*戊戌*]/*思潮*/*与*/*前此*/*的*/*洋务运动*/，\n\n#### 6.1.2时段（dur）的定义\n\n时段既可以长于一天，也可以短于一天。它不同于日期和时间，在一维的时间坐标轴上没有确定的位置。例如：\n\n/[dur*三年*]/ \n\n/[dur*半年*]/\n\n/[dur*四分之一个世纪*]/\n\n/[dur*廿四个月*]/\n\n/*时间*/*长*/*达*/[dur*六分钟*]/ \n\n/[dur*两个星期*]/\n\n/[dur*一个月*]/*后*/\n\n/*曾*/*在*/[dur*５、６年*]/*前*/*撰文*/*陈述*/\n\n/*早产*/[dur*十二周*]/*左右*/\n\n/*大水*/[dur*十天*]/*后*/*才*/*退*/*尽*/ /[dur*一至两年*]/\n\n/[dur*一小时卅分钟*]/ /*这*/[dur*几天*]/\n\n/[dur*卅天*]/*会期*/*只*/*开*/*了*/[dur*九天*]/\n\n/*虽*/*经*/[dur*一整天*]/*磋商*/*，*----*一整天*不是词表词，但要标为*dur*。\n\n与*/*洪水*/*奋战*/[dur*一天一夜*]/*，*----*一天一夜*也不是词表词。\n\n时间表达式的标注细则详见第七章。\n\n### 6.2数字表达式\n\n数字表达式（*NUMEX*）包括百分数（*per*）、钱款（*mon*）、频度（*fre*）、整数（*int*）、分数（*fra*）、小数（*dec*）、序数（*ord*）、比率（*rat*）等8小类。\n\n#### 6.2.1百分数（per）\n\n/[per*百分之二十五*]/\n\n/[per*百分之一点七*]/ ---虽然是小数，但要标作per。\n\n/[per*六点五百分点*]/ /[per*五成*]/*以上*/ /[per*六折*]/\n\n/[fra*百万分之八*]/  ----注意标的是*fra*而不是*per*。\n\n/*大约／*[per5%]/ ----约数*大约*不进入标注。\n\n6.2.2钱款（mon）\n\n/[mon*四亿元台币*]/\n\n/[mon*43.6亿美元*]/ \n\n/[mon*卅万元*]/\n\n/[mon*四万五千块钱*]/ \n\n/[mon*四万五千元人民币*]/\n\n/*只*/*增加*/*了*/[mon*几元钱*]/*的*/*成本*/\n\n/*决不*/*乱*/*花*/*国家*/*的*/[mon*一分钱*]/。\n\n- 注：同一笔钱的不同货币形式需分开标注。货币中的地名不标。\n\n[mon*26万英镑*]/ (/[mon*43.6亿美元*]/)/\n\n- 注：*约*是一个不确切的概念，故不标注。但*上*、*数*、*好*要和数字串捆绑在一起标注。但*近*作为特例，不与数词捆绑!!\n\n/*约*/[mon*十万元*]/\n\n/*大概*/*需要*/*花费*/[mon*上千万美元*]/*的*/*投资*/*和*/[dur*3*年*]/*左右*/*时间*/*，*/ \n\n/*多*/*收入*/[mon*好几十元*]/\n\n#### 6.2.3频度（fre）\n\n/[fre*数度*]/ /[fre*两次*]/ /[fre*26次*]/ /[fre*十多次*]/ /[fre*多次*]/\n\n- 注：动量词次除了一次不标注以外，其余的全部标注为*fre*。\n\n/[fre*一次次*]/\n\n/[fre*再次*]/ /[fre*无数次*]/ /[fre*数次*]/\n\n#### 6.2.4整数（int）\n\n*int*标注的是数词和量词组合成数量词组。\n\n/[int*卅七件*]/ /[int*一百卅项*]/ /[int*三种*]/\n\n/[int*九个*]/*课室*/ /[int*几家*]/\n\n/*后*/[int*几名*]/ /[int*十*]/*多*/*人*/ /[int*四条*]/*断层*/ /[int*五十户*]/ /[int*百余名*]/ /[int*上万*]/*人潮*/\n\n/*\"*/[int*双*]/[int*百*]/*\"*/*方针*/，\n\n- 注：\"双百方针\"是词表词，由于文中\"双百\"用引号括起，而且它们是两个数字，所以要分别按数字串标注。类似情况还有词表词\"五四运动\"，这是个事件不标。但是如果文中日期\"五四\"被引号括起，就要单独标为：/\"/[dat五四]/\"/运动/。又如\"六一儿童节、六一国际儿童节、六一节\"都是词表词。由于\"六一\"和\"儿童节\"是同一个日期，即使在文中\"六一\"被引号括起，也可以整体标为dat，如：/[dat\"六一\"儿童节]/。\n\n- 注：人次应标注为*mea*而不是*int*，例如：\n\n/*近*/[dur*３年*]/*中*/*，*/*该*/*市*/*采取*/*多*/*形式*/*的*/*农技*/*培训*/*近*/[mea*万人次*]/，\n\n- 注：\"*数词*+*强*\"不一定表示序数，因此只单独标注数词为*int*。例如：\n\n*/*在*/*这次*/*从*/[int*十六*]/*强*/*到*/*冠*/*、*/*亚军*/*的*/*一次性*/*竞猜*/*中*/*，*/*\n\n/[O*宝钢*]/*为*/*跻身*/*世界*/[int*５００*]/*强*/*而*/*采取*/*的*/*重要*/*步骤*/*。*/\n\n#### 6.2.5分数（fra）\n\n/[fra*数倍*]/ \n\n/[fra*一半*]/ \n\n/[fra*千百倍*]/ \n\n/[fra*3/4]/\n\n/[fra*四分之三*]/\n\n/[fra*百万分之三百六十四*]/ *----*注意标记是*fra*而不是*per*。\n\n/[fra*半个*]/ /[fra*4倍半*]/ *----*倍数是分数的一种表示，应标*fra*。\n\n/[fra*4倍半*]/\n\n/[fra*4.5倍*]/ ----*虽然* *4.5*是个小数，但不标*dec*。\n\n/*有效*/*载*/*力*/*提高*/[fra*２至３倍*]/\n\n注：\"过半数\"是词表词，因此不作为分数fra标注。例如：*/*都*/*难以*/*获得*/*过半数*/*的*/[int*２０７张*]/*选票*/*，*/*\n\n#### 6.2.6小数（dec）\n\n/[dec*3.14]/\n\n/[dec*三点一四*]/\n\n/*看*/*了*/*那么*/*长*/*时间*/*的*/*电视*/*，*/*视力*/*依旧*/[dec*1*．*5*]/\n\n/*我*/*有着*/*足以*/*令*/*我*/*自豪*/*的*/[dec*1*．*2*]/*视力*/\n\n----视力的多少是一个量级，没有单位，故按数量标注整数或小数。/*并*/*以*/[dec*6139.69点*]/*收盘*/\n\n/*以*/ [dec*33.8*]/*收盘*/ /*比重*/*：*/[dec*1.02*]/\n\n#### 6.2.7序数（ord）\n\n/[ord*第一任*]/\n\n/[ord*第一期*]/ \n\n/[ord*十六楼*]/\n\n/[ord*第三次*]/*世界大战*/\n\n/[ord*首*]/*日*/*销售*/*欠佳*/ \n\n/[ord*第二*]/*故乡*/\n\n/[ord*三等*]/*奖*/\n\n/[ord*前*6*名*]/\n\n/*地震烈度*/*不*/*超过*/[ord*8度*]/ \n\n/*这*/[ord*第二条*]/*尤为*/*重要*/ \n\n/*位居*/*金牌*/*榜*/[ord*第二名*]/\n\n/[O*北京市*]/[ord*首家*]/*就业*/*与*/*创业*/*组合*/*市场*/ \n\n/[ord*1174号*]/*文件*/\n\n/[ord*6*路*]/*汽车*/ /[ord*六年级*]/*学生*/\n\n/[dat*今年*]/*读*/[ord*大三*]/\n\n/*发展*/*第一产业*/* ----第一产业*是词表词。\n\n/*发展*/[ord*第一*]*产业*/ ----错误的标注。\n\n/*阵风*/[ord*五级*]\n\n/[ord*一、二、三等*]/*奖*/。\n\n/*他*/*亲手*/*接*/*治*/[L*墨西哥*]/[ord*首例*]/*艾滋病*/*患*/*儿*/\n\n#### 6.2.8比率（rat）\n\n/ [rat*一比廿五*]/\n\n/*以*/[rat*０∶６*]/*失利*/\n\n/*上*/*一*/*届*/*世界杯*/*赛*/*就*/*以*/[rat*１∶０*]/*胜过*/*。*/\n\n/*最终*/*以*/[rat*三比三*]/*握手言和*/*。*/ /*用*/*原液*/*与*/*水*/*稀释*/[rat*1*∶*20*倍*]/*。*/\n\n数字表达式的标注细则详见第八章。\n\n### 6.3度量表达式\n\n度量表达式（*MEASUREX*）包括年龄（*age*）、温度（*tem*）、角度（*ang*）、长度（*len*）、面积（*are*）、容积（*cap*）、重量（*wei*）、速度（*spe*）、加速度（acc）和其它（*mea*）等10小类。\n\n#### 6.3.1年龄（age）\n\n/[age*卅五岁*]/\n\n/[age*廿一岁*]/ \n\n/[age*六十五岁*]/ \n\n/[age*34岁*]/ \n\n/[age*六十寿辰*]/\n\n/[age*花甲*]/*老人*/ ----*花甲*是词表词。\n\n/*如同*/[age*年过半百*]/*的*/*老*/*妇*/*。*/ ----*年过半百*是词表词。\n\n/[P*李元*]/*、*/[P*卞德培*]/[int*两位*]/*先生*/*都*/*已*/[age*年逾古稀*]/。\n\n#### 6.3.2温度（tem）\n\n/*寒流*/*耍*/*酷*/*平地*/[tem*6℃*]/ \n\n/*才*/*会*/*微*/*升*/[tem*6.1℃*]/\n\n/*但*/*平地*/*温度*/*还*/*会*/*下*/*探*/[tem*5℃*]/*左右*/\n\n/*积温*/*高*/*（*/[tem*2800度*]/*）*/----注意！\n\n/[tem*零下*5*到*6*摄氏度*]/\n\n- 注：数字范围的标注方式详见7.1.1。\n\n/*大约*/[tem*5~7℃*]/\n\n/*低温*/*反而*/*只*/*有*/[tem*10℃*]/~/[tem*12℃*]/ \n\n/[tem*摄氏19*－*24度*]/\n\n/[tem*摄氏*19*度*]/ -/[tem*24度*]/\n\n#### 6.3.3角度（ang）\n\n/*钝角*/*就*/*是*/*大于*/ [ang*90度*]/*的*/*角*/\n\n/*并*/*将*/*卫星*/*定点*/*在*/[agn*东经*110.5*度*]/[L*赤道*]/*上空*/*。*/\n\n/*震*/*中*/*位于*/[ang*北纬*30.5*度*]/*，*/ ----详见4.2.4.2\n\n#### 6.3.4长度（len）\n\n/*开掘*/*到*/ [len*一米六七*]/*深度*/*时*/ \n\n/*高*/ [len*五米*]/*宽*/ [len*一百米*]/ /[len*109×78厘米*]/\n\n/[len*1纳米*]/=/[len*十的负九次方米*]/\n\n/*应用*/*于*/*紧*/*固*/*件*/*直径*/*为*/[len*1*／*4″*]/*（*/[len6m]/*）*/\n\n/*最高*/*速度*/*每*/*秒*/  [len*360米*]/\n\n/*发生*/*每*/*秒*/*速度*/*达*/[len*四十二米*]/*的*/*大风*/*。*/\n\n/[L*三峡*]/*截流*/*落差*/*在*/[len*0.7-0.8米*]/*之间*/*，*/\n\n#### 6.3.5面积（are）\n\n/[are*廿七公顷*]/*土地*/\n\n/*占*/*地*/[are*六百多公顷*]/\n\n/*兴建*/[are*五千坪*]/*大*/*的*/*厂房*/ /[are*七百余坪*]/\n\n/*每*/*套*/*住宅*/*面积*/[are*140*－*160m2*]/*，*/ \n\n/[are*997万平方公里*]/\n\n/*农田*/ [are*20万亩*]/\n\n#### 6.3.6容积（cap）\n\n/*运输量*/*为*/ [cap*34个立方*]/\n\n/[cap*一两箩*]/*谷子*/\n\n/*选定*/*的*/*设计*/*流量*/*是*/*每*/*秒*/[cap*1.4*万至*1.9万立方米*]/*。*/ \n\n/*工程*/*已*/*完成*/*土方*/[cap*２３００多万方*]/*，*/\n\n/*全国*/*消费*/*了*/[cap*２５万升*]/*啤酒*/*。*/\n\n#### 6.3.7重量（wei）\n\n/[wei*九百至一千吨*]/\n\n/[wei*零点三公克*/]\n\n/[wei*三千二百英吨*]/\n\n/*重*/*约*/[wei*五、六公斤*]/*的*/*鲤鱼*/\n\n/[wei*十台斤*]/\n\n/[wei*三点五公吨*]/\n\n/*产量*/*达到*/ [wei*数千万吨*]/ /[wei*几万吨*]/\n\n/[wei*二十万吨*]/*级*/*以上*/\n\n- 注：ER-99把上例标为：[wei*二十万吨级*]/以上。\n\n#### 6.3.8速度（spe）\n\n/*最高*/*速度*/ [spe*360米每秒*]/\n\n/*打印*/*速度*/*：*/[spe12cps]/\n\n----\"*cps*\"表示\"characterspersecond（每秒字符数）\"。\n\n#### 6.3.9加速度（acc）\n\n/*抗震*/*能力*/*：*/*地面*/*水平*/*加速度*/*≤*/[acc*0.4m*／*s2*]/ /*地面*/*垂直*/*加速度*/*≤*/[acc*0.2m*／*s2*]/\n\n#### 6.3.10其它度量表达式（mea）\n\n除了上面提到的度量单位元之外，物理、化学及其它度量单位的统一标注为*mea*。/*额定*/*电压*/*至*/[mea*660V]/\n\n/[mea*5.5瓦特*]/\n\n/*参观*/*人数*/*达*/[mea*620万人次*]/ /*工资*/[mea*3500元*/*人*/*月*]/\n\n/[mea*25元*/*公斤*]/\n\n/*风*/*压*/*不*/*超过*/[mea*700Pa*]/*（*/*相当于*/*风速*/[spe*34m*／*s*]/*）*/*。*\n\n/*迁移*/*到*/[mea*千兆比特*]/*的*/*能力*/*能够*/*降低*/*拥有*/*总*/*成本*/*的*/*管理*/*方案*/\n\n/*这些*/*快速*/*以太*/*网*/*和*/[mea*千兆位*]/*以太*/*网*/*服务器*/\n\n#### 6.4地址表达式\n\n地址表达式（*ADDREX*）包括电子邮箱（*ema*）、电话（*pho*）、传真（*fax*）、电报挂号（*tel*）、邮政编码（*pos*）和网址（*www*）等6种。\n\n#### 6.4.1电子邮箱（ema）\n\n/[ema *exp@email.com.cn*]/\n\n/[ema*cnhuang@msrchina.research.microsoft.com*]/\n\n#### 6.4.2电话（pho）\n\n在标注电话号码时，要把国际区号、国内区号、本地区号等作为一个整体标注。如果有分机号码也要一并标注。当有多个分机号码时，要分别标注。如：\n\n*预约*/*订*/*位*/*电话*/[pho*九五一八六二八*]/ \n\n/*洽*/*询*/*电话*/[pho*二四九三一零二零*]/ \n\n/*订*/*席*/*专线*/[pho（*8610*）-78906617]/\n\n/*查询*/*电话*/*是*/(/[pho*零三八六二一一零零转二五二*]/)/\n\n/*查询*/*电话*/[pho*三六九九七二一转二三三一*]/*或*/[pho*二三三二*]/\n\n/[pho*120*]/\n\n/[pho*119*]/\n\n#### 6.4.3传真（fax）\n\n/*全国*/*客户*/*服务*/*传真*/*：*/[fax*010-58722727*]/\n\n/*传真*/*号码*/:/[fax*86-10-66665555*]/\n\n/*公司*/*传真*/*：*/[fax*86-10-66665555*]/\n\n#### 6.4.4电报挂号（tel）\n\n/[O*搜狐公司*]/*电报挂号*/*是*/*：*/[tel(8610)*62726666*]/ \n\n/*电报挂号*/*：*/[tel*86-10-66665555*]/\n\n/*联系*/*电话*/*：*/[tel*86-10-66665555*]/\n\n#### 6.4.5邮政编码（pos）\n\n/[O*清华大学*]/*的*/*邮政编码*/*是*/*：*/ [pos*100080*]/\n\n/[L*安徽*]/[L*阜阳*]*/*地区*/*的*/*邮政编码*/*是*/*：/[pos*233600*]/\n\n#### 6.4.6网址（www）\n\n/*活动*/*报名*/*网址*/*：*/[www http:www.acer.net/event/apply]/\n\n/[O*蕃薯藤*]/*购物*/*网*/*（*/[www http:shopping.yam.com]/*）*/\n\n## 第七章 时间表达式标注细则\n\n**时间表达式（**TIMEX***）包括日期（***dat**）** 、时间（ tim ）和时段（ dur ）三小 **类。**\n\n### 7.1日期（dat）\n\n/[dat*明治三十九年*]/*（*/[dat*公元一九零六年*]/*）*/ \n\n/[dat*大正十四年*]/*（*/[dat*公元一九二五年*]/*）*/\n\n/[dat*昭和二年*]/*（*/[dat*公元一九二七年*]/*）*/\n\n/[dat*清*]/[dat*道光十四年*]/ \n\n/[dat*清*]/[dat*咸丰十一年*]/\n\n/[dat*民国六十八年*]/*拆除*/*后*/*迁到*/[L*芦洲*]/*，*/[dat*八十一年*]/*间*/*又*/*扩建*/ \n\n/[dat*一九九九*]/\n\n/[dat*一九九九年十二月三十号*]/ \n\n/[dat*公元*1990*年*4*月*22*日*]/ \n\n/[dat*旧石器时代*]/\n\n/[dat*八十年代*]/\n\n/[dat*下半年*]/\n\n/[dat*1989财年*]/ ----注意!\n\n/[dat*1989*财年第三季度*]/\n\n/[dat*1990*上半财年*]/\n\n/[dat*1991*财政年度*]/\n\n/[dat*秋季*]/*报告*/ \n\n/[dat*第四季度*]/\n\n/[dat*十五世纪*]/\n\n/*努力*/*成为*/*一*/*名*/*高*/*素质*/*的*/[dat*跨世纪*]/*人才*/*。*/ \n\n/*值*/*此*/[dat*世纪之交*]/*的*/*时候*/*，*\n\n/*走*/*向*/[dat*新世纪*]/*的*/[L*中国*]/*律师*/*业*/ \n\n/[dat*新旧世纪交替*]/*之际*/\n\n/*黑色*/[dat*星期一*]----注意！\n\n/[*Ｌ北京］*/*在*/[dat*23号*]/*发表*/*了*/*报告*/\n\n- 注：数字串*23号*若不表示日期，则不标。\n\n/[dat*五月上旬*]/  ----*上、中、下旬*要标注。\n\n/*科技*/*之*/[dat*夏*]/  ----注意！\n\n/[dat*夏*]/[dat*秋*]/*之间*/\n\n/[dur*一年*]/*中*/*四季*/*分明*/  ----*四季*是词表词不标注。\n\n/[L*南极*]/*的*/[dat*夏季*]/ \n\n/[L*中国*]/[dat*汉代*]/\n\n/[dat*春节*]/  ----日期确定的节日要标注。\n\n/[dat*肉孜节*]/\n\n/[dat*开斋节*]/\n\n/[dat*中秋*]/*时节*/\n\n----注意*时节*不标。\n\n/[L*美国*]/*的*/[dat*独立日*]/----美国独立日为每年7月4日。\n\n/[dat*27年*]/*是*/*一个*/*多*/*事*/*的*/*年份*/\n\n- 注：*27*年*可能表示时段，标注者须根据上下文注意区分。\n\n/*现在*/*是*/[dat*26号*]/*，*/[dat*星期三*]/\n\n----同一个时间的不同表达，要分开标注。\n\n/*现在*/*是*/[dat*二月九号*]/*，*/[dat*农历大年初三*]/\n\n/*大约*/[dat*五月四日*]/*----大约，大致，大概*等词不标。\n\n/[dat*第二个十年*]/ /[dat*第二年３月*]/ \n\n/[dat*当年*9*月*]/ \n\n/[dat*今春*]/  ----*今春*不是词表词。\n\n#### 7.1.1日期起讫表达式的标注\n\n当日期表达式中有至、到和连结符－时，处在至、到和连结符－前后的日期表达式分别叫做前式和后式。如果前式和后式都是完整的日期表达式，则它们应分别进行*da*t标注；否则前、后式要整体标注为*dat*。\n\n这条规则同样适合于其它各类数字串的标注，如：*tim*，*dur*，*int*，*tem*，*wei*，*mon*等。其一般表达式为：\n\n/X+量词/到/X+量词/\n\n/X+量词/至/X+量词/\n\n/X+量词/－/X+量词/\n\n/X+至+X+量词/\n\n/X+到+X+量词/\n\n/X+－+X+量词/\n\n/X+、+X+量词/\n\n例如：\n\n/[dat*三月三日*]/*至*/[dat*三月卅一日*]/ \n\n/[dat*一月十八日*]/*到*/[dat*廿一日*]/ \n\n/[dat*三月三至廿一日*]/\n\n/[dat*二月十八日*]/-/[dat*廿一日*]/ \n\n/*于*/[dat*今明两年*]/*陆续*/*推出*/*。*/ \n\n/[dat*民国五十五、五十六年*]/ \n\n/[dat*今明两天*]/\n\n/[dat*今*]/*、*/[dat*明*]/[dur*两日*]/\n\n/[dat*1980年*]*到*[dat*1990*年*1月*]/\n\n- 注：含有比喻意义的今天、昨天、明天、今日、昨日、明日全不标注。\n\n*/*\"*/*一失足成千古恨*/*，*/*同学*/*们*/*，*/*看到*/*今天*/*的*/*我*/*，*/*你们*/*是否*/*感悟*/*到*/了*/*什么*/*？*/*\"*/\n\n/*尽管*/*炮火*/*已*/*消失*/*在*/*昨天*/*那*/*段*/*苦难*/*，*/\n\n/[O\"四方\"集团*]/*的*/*明天*/*将*/*会*/*更加*/*灿烂*/*美好*/*。*/\n\n- 注：当年、同年、当月等词语后有具体的日期时，要整体标注dat，如果当年、当月、同年等词语单独出现，而其前后有确指的日期时也要标注为dat，否则不作标注!当日、当天等词后有具体的时间时标注为dat，否则不作标注!如：\n\n/[dat*当年７月*]/*在*/[L*莫斯科*]/*举行*/ \n\n/*然后*/*于*/[dat*同年８月*]/*奉调*/*回国*/*。*/\n\n/[P*克林顿*]/*在*/[dat*当月１３日*]/*表示*/*，*/\n\n/*那*/*是*/[dat*当天*]/[tim*中午１时*]/*的*/*汇率*/\n\n/*发言人*/*于*/[dat*当日*]/[tim*午夜*]/*发表*/*声明*/\n\n#### 7.1.2前、头、下+时段（dur）应整体标注为dat\n\n/[dat*头两个礼拜*]/ \n\n/[dat*前３天*]/\n\n/[dat*今年头四个月*]/\n\n/*比*/[dat*上一年*]/*增长*/[per*１０．４％*]/*。*/\n\n/*集中*/*研究*/*解决*/[dat*下半年*]/*纠风*/*工作*/*如何*/*突出*/*重点*/*，*/\n\n/[dur*两周*]/*前*/\n\n/[dat*1993年之初*]/ ----注意!\n\n/[dat*公元之初*]/\n\n#### 7.1.3当乾隆、康熙、道光等表示年代时标注为dat\n\n当乾隆、康熙、道光*等表示年代时标注为*dat*，而当*乾隆、康熙、道光*等表示皇帝本人的名字时标为P。如：\n\n/*最近*/*发现*/*一*/*张*/*在*/*农家*/*珍藏*/*的*/[dat*清代*]/[P*康熙*]/*、*/[P*雍正*]/*、*/[P*乾隆*]/*、*/[P*嘉庆*]/*、*/[P*道光*]/[int*五*]/*皇帝*/*诰封*/*圣旨*/[int*九道*]/*，*/ /*收藏*/*了*/*自*/[dat*清代*]/[dat*乾隆*]/*年间*/*至今*/*各个*/*历史*/*时期*/*的*/*鼻烟壶*/*艺术*/*珍品*/*，*/\n\n#### 7.1.4朝代名的默认值为dat\n\n当朝代名被上下文确认为国家名时标注*L*，否则默认为*dat*。如：\n\n/*如果*/[P*刘伯温*]/*不是*/*一直*/*压抑*/*着*/*对*/[dat*元*]/*王朝*/*的*/*不满*/*，*/ \n\n/[dat*楚*]/*霸王*/[P*项羽*]/*带领*/[int*两万*]/*兵*/*将*/*，*/\n\n/*只*/*带*/[dur*三天*]/*粮食*/*，*/*渡过*/[L*漳河*]/*去*/*与*/*强大*/*的*/[dat*秦*]/*兵*/*作战*/*。*/*结果*/*，*/[dat*楚*]/*军*/*大败*/[dat*秦*]/*军*/*。*/\n\n/[dat*吴*]/*王*/[*P*夫差]/*战胜*/*了*/[dat*越*]/*王*/[P*勾践*]/，\n\n/[dat*战国*]/*时*/[L*赵国*]/*良*/*相*/[P*蔺相如*]/*曾*/*为*/[L*赵国*]/*立*/*下*/*汗马功劳*/；\n\n*[P*唐睢*]/*出使*/[L*秦国*]/*，*/*\n\n《*/[L*水浒*]/*全传*/*》*/*描述*/*的*/*是*/[dat*北宋末年*]/*震撼*/[dat*宋*]/*室*/*江山*/*的*/[P*宋江*]*起义*/*。*/*\n\n/*从*/*侧面*/*表现*/*了*/[dat*清*]/*政府*/*的*/*腐败*/*无能*/*，*/*激起*/*了*/*深*/*埋*/*在*/*人们*/*心底*\n\n/*对*/*侵略者*/*的*/*敌视*/*和*/*对*/[dat*清*]/*政府*/*的*/*愤怒*/*，*/\n\n/*但是*/*，*/*战争*/*最终*/*因*/[dat*清*]/*政府*/*的*/*妥协*/*、*/*投降*/*而*/*告*/*失败*/*。*/ /*无奈*/*夜郎自大*/*、*/*腐败*/*不堪*/*的*/*大*/[L*清国*]/*武器*/*太*/*落后*/*，*/\n\n#### 7.1.5在\"过去、今后、未来+时段（dur）\"等修饰成分不进入标注范围\n\n/*过去*/[dur*３年*]/*中*/*，*\n\n/*将*/*在*/*未来*/[dur*几年*]/*内*/*出现*/\n\n/*未来*/[dur*两天*]/*沿江*/*地区*/*仍*/*有*/*中*/*到*/*大雨*/*，*/\n\n/[dat*今年七八月*]/*间*/\n\n#### 7.1.6词表词近年来、近些年、近几年来、近几年、几年来等均不标注\n\n按规定，词表词*近年来、近几年、近几年、几年来、多年来、近些年*等内部的*dat*、*tim*、*dur*都是不标的。但对非词表词则要分开标注。例如：\n\n/[L*瑞士*]/*多年来*/*是*/[ord*第一次*]/*。*\n\n/近几年/，/[L中]/[L菲]/关系/\n\n/*近*/[dur*五年*]/*来*/\n\n/*时至今日*/*仍*/*在*/*缓刑*/*期间*/*。*/*-----时至今日*是词表词。\n\n### 7.2时间\n\n/[tim*凌晨零时*]/\n\n/[tim*清晨六时卅五分*]/*到*/[tim*四十分*]/ \n\n/[tim*凌晨二至四点*]/\n\n/[tim*中午十二时*]/-/[tim*晚上九时*]/ \n\n/[tim*上午十一时*]/*至*/[tim*下午二时*]/ \n\n/[tim*第七十三分钟*]/\n\n/[tim*格林威治时间*5*时*59*分*]/ ----含有地名。\n\n/[tim*下午当地时间*5*时*59*分*]/\n\n/[tim*九点整*]/*到达*/[L*北京站*]/\n\n/[dat*九月十三日*]/*大约*/[tim*七点*]/*到达*/[L*北京*]/\n\n- 注：这里*大约*不标。因为它虽被一个*dat*和一个*tim*包围，但是仍可以分割开。\n\n### 7.3时段\n\n/[dur*两个星期*]/ \n\n/[dur*一个月*]/*后*/\n\n/*曾*/*在*/[dur*５、６年*]/*前*/*撰文*/*陈述*/\n\n/*早产*/[dur*十二周*]/*左右*/\n\n/*大水*/[dur*十天*]/*后*/*才*/*退*/*尽*/\n\n/[dur*一至两年*]/ /[dur*一小时卅分钟*]/ \n\n/*这*/[dur*几天*]/\n\n/[dur*卅天*]/*会期*/*只*/*开*/*了*/[dur*九天*]/ \n\n/[dur*10个月*]/\n\n/*虽*/*经*/[dur*一整天*]/*磋商*/ ----*一整天*不是词表词，但要标为*dur*。\n\n/*与*/*洪水*/*奋战*/[dur*一天一夜*]/*，*/ ----*一天一夜*也不是词表词。\n\n*/*历经*/[dur*一二十年*]/*创建*/*了*/*庞大*/*的*/*船队*/*，*/*\n\n/*让*/*我们*/*全家*/*人*/*感动*/*了*/[dur*好几天*]/\n\n/*在*/*水门*/*丑闻*/ [dur*四分之一世纪*]/*时*/*发表*/*的*/*评论。*/\n\n- 注：按照前面的原则：*水门*/*丑闻*/ [dur*四分之一世纪*]/*时*在时间坐标轴上有比较固定的位置，因此应当标为*dat*。但这种与事件（水门丑闻）相关的时间表达，在ER-99和MET-2中都是不标注的。这样，只有*四分之一世纪*需要标注为*dur*。\n\n/[dur*十多年*]/\n\n/[dur*几年*]/*以来*/\n\n/*在*/[dur*半年*]/*时间*/*内*/----注意：*上半年*是*dat*。\n\n/*在*/*总结*/[dur*14年*]/*改革开放*/*经验*/*的*/*基础*/*上*/\n\n- 注：*14年*、*30*年*也可能表示dat。标注者要注意区分。*/*我们*/*在*/*美国*/*奔波*/*了*/[dur*30年*]/\n\n/[dur*27年*]/*的*/*军旅*/*生涯*/\n\n/*整整*/[dur*十五年*]/  ----*整整*不标。\n\n/*大约／*[dur*十年*]*／的／时间*/ ----*大约*不标。\n\n/[dur*十年*]/*来*/\n\n/[dur*十几年*]/*的*/*时间*/ ----注意！\n\n/[dur*十几年*]/*来*/ /[dur*十来年*]/ /[dur*数年*]/\n\n/[dur*多年*]/ ----ER99不标。\n\n#### 7.3.1一年都标为dur\n\n/*新*/*的*/[dur*一年*]/*即将*/*开始*/\n\n/*硬*/*是*/*在*/*地下室*/*干*/*了*/ [dur*一年*]/*的*/*公司*/\n\n/[dur*一年*]/*创*/*产值*/*效益*/…/…/\n\n/*聘金*/*为*/[dur*一年*]/ [mon*900万美元*]/*的*/*价码*/\n\n- 注：*/*这*/*一年*/*、/*那*/*一年*/*中的一年不是确指不作标注。\n\n*/*这*/*一年*/*，*/*企业*/*增收节支*/*达*/[mon*１１０万元*]/\n\n/*在*/[O*北大*]/*就读*/*的*/*那*/*一年*/*，*/\n\n- 注：整天、整日、整夜一律标注为*dur*，如：\n\n/[dur*整天*]/*都*/*很*/*安静*/*，*/\n\n/*还*/*东奔西走*/[dur*整日*]/*忙*/*个*/*不停*/*，*/\n\n/*让*/*人*/[dur*整夜*]/*不得*/*入睡*/\n\n- 注：当年、月、日、周等词修饰后面的工资、交易（销售）额、创汇等词语时，要作为时段（*dur*）来标注。如：\n\n/[dur*月*]/*收入*/*就*/*在*/[mon*千元*]/*以上*/\n\n/[dur*年*]/*交易额*/*近*/[mon*１０００亿元*]/*。*\n\n/*这*/*一*/*工程*/[dur*日*]/*处理*/*污水*/[cap*２万立方米*]/*。*\n\n#### 7.3.2一天的标注有以下三种情况，需区别对待：\n\n##### 7.3.2.1\"前一天\"，不论其前面有没有定语修饰统统标注为dat（参见7.4.1）：\n\n/[dat*前一天*]/*还*/*静止*/*的*/*电梯*/[dat*今天*]/*动*/*起来*/*了*/*，*/\n\n/[L*香港*]/[O*恒生*]/*指数*/*比*/[dat*前一天*]/*下跌*/[int*４１２点*]/*，*/\n\n/*这次*/[L*中*]/[L*韩*]/*足球*/*对抗赛*/*是*/*在*/[O*韩国队*]/*准备*/*赴*/[L*法*]/*出征*/\n\n*世界杯*/*的*/[dat*前一天*]/*举行*/*的*/*，*/\n\n##### 7.3.2.2\"一天\"的意思是指时间段（24小时），标注为dur：\n\n/*每人*/*每月*/*接待*/*来访*/[dur*一天*]/*，*\n\n/[P*汤*]/[P*尤*]/*杯*/[dur*一天*]/*不*/*拿*/*回来*/*，*/\n\n/*仅*/[dat*５月３１日*]/[dur*一天*]/*，*/[L*莫斯科市*]/*税*/*警*/*就*/*查出*/[int*１６００个*]/*违法*/*经营者*/*。*/\n\n/*青年人*/*辛苦*/*忙碌*/*了*/[dur*一天*]/*来*/*此*/*坐*/*坐*/*，*/ /*在*/[L*墨西哥*]/*最后*/[dur*一天*]/*的*/*访问*/*中*/*，*/\n\n/*每*/*枚*/*多*/*赚*/[mon*７分钱*]/*，*/[dur*一天*]/*下来*/*能*/*多*/*收入*/[mon*好几十元*]/*。*\n\n##### 7.3.2.3\"一天\"的意思相当于\"有一天\"，由于不是确指的日期所以什么也不标：\n\n/*但愿*/*有一天*/*我们*/*轻松*/*地*/*说*/*：*/*消费*/*着*/*是*/*美丽*/*的*/*。*/\n\n/[dat*１９９７年*]/*的*/*一天*/*，*/[P*吴佩民*]/*在*/*办公室*/*热情*/*接待*/*了*/*一个*/*素不相识*/*的*/*中年*/*妇女*/*。*/\n\n/*一天*/[tim*下午*]/*，*/*记者*/*到*/*那*/*店*/*里*/*专门*/*拜访*/*了*/[P*佛朗科*]/*师傅*/*。*/\n\n/*一天*/*，*/[P*列宁*]/*收到*/*一*/*封*/*前线*/*发*/*来*/*的*/*要求*/*支援*/*武器*/*和*/*服装*/*的*/*电报*/*。*/\n\n/*一天*/*上*/*晚*/*自习*/*回来*/*，*/*有*/*一*/*条*/*狗*/*总*/*跟着*/*她*/*，*/\n\n/*一天*/[tim*深夜*]/*，*/*一*/*人*/*酒后*/*拦截*/*过往*/*的*/*外地*/*车辆*/*，*/ /*一天*/*，*/*我*/*走过*/*他*/*的*/*门前*/*，*/\n\n/*一天*/[tim*晚上*]/*，*/*新*/*上任*/*的*/[L*河北省*]/[O*栾城县委*]/*书记*/[dat*六月八日*]/*，*/\n\n- 注：\"这/一天、那/一天\"中的\"一天\"也非确指，所以也不标。\n\n/*记住*/*这*/*一天*/*，*/*也是*/*表达*/*我*/*对*/[L*香港*]/*回归祖国*/*的*/*预祝*/*。*/ \n\n/[P*王龙雨*]/*从*/*上任*/*的*/*那*/*一天*/*起*/*，*/\n\n### 7.4有关时间表达式的规则\n\n#### 7.4.1前(后)+日期|时间要整体标注\n\n/[dat*今年前五个月*]/ \n\n/[dat*前三天*]/\n\n- 注：以下的标注是正确的：\n\n/*在*/*上半时*/*结束*/*前*/[dur*1分钟*]/  ----*上半时*是词表词。\n\n/*比赛*/*前*/[dur*十分钟*]/\n\n/*在*/*上*/*半场*/[tim*第２７分钟*]/*时*/\n\n#### 7.4.2反例——不应该标注的例子\n\n刚才、最近、开始军备谈判以来、一会儿*等表示不确定时间的词语，不标。如果节日没有确定的时间，也不标。如：\n\n/[L*印度*]/*国际*/*电影节*/\n\n/[L*中国*]/*旅游年*/\n\n#### 7.4.3特例\n\n若两个短语属于不同的子类*dat*和*tim*，就需分开标注。\n\n/[dat*2*月*12日*]/[tim*上午*8*点*]/\n\n/[dat*星期一*]/[tim*8点*]/\n\n- 注1：时间中的地名，如北京时间下午*5*点，在ER-99中不标注，而在NET-2中要标注。本规范按NET-2标注（参照前面的例子）。如果*dat*和*tim*分不开，就整体标注。\n\n/[tim*北京时间*1997*年*2*月*9*号*19*点*28*分*]/\n\n- 注2：*去年、昨天、今早*等词在MET-2中要标，在ER-99中不标。本规范只参照MET-2:\n\n/[dat*去年上半年*]/ \n\n/[dat*今年夏天*]/\n\n/[dat*今年三月一日*]/ \n\n/[dat*去年春夏之交*]/\n\n/[dat*昨天*]/[tim*夜里*]/ ---*夜里*是词表词。\n\n/[dat*今天*]/[tim*晚上*]/ ---*晚上*是词表词。\n\n/[dat*今*]/[tim*早六点*]/ ---*今早*不是词表词。\n\n/[tim*早上六点*]/ ---*早上*是词表词。\n\n/[dat*５月份*]/*产品*/*出口*/*和*/*转口*/*总值*/*比*/[dat*去年同月*]/*下降*/[per*３．２％*]/*，*\n\n/[dat*同一天*][tim*晚上*]/\n\n/[dat*当日*]/[tim*下午*]/\n\n- 注3：当日是词表词。如果在上下文中能确定*当日、当天*或*同一天*的具体日期时，就标注；否则不标。\n\n/*每日*/[tim*上午１１时*]/*至*/[tim*深夜３时*]/ ----*深夜*是词表词。\n\n/[tim*昨夜*/]/ ----*昨夜*是词表词。\n\n/*每*/[dat*周四，二，一*]/\n\n- 注：MET-2和ER-99对*早上六点*的标注是相同的。但ER-99认为*早上六点*与*今早六点*不同。原因可以从英语的表达来理解：前者是\"6：00am\"，后者是\"6：00thismorning\"。\"thismorning\"在ER-99中被视为\"相对时间\"，不标注。但在MET-2中，\"相对时间\"是要标的。本规范遵循MET-2。\n\n/[dat*11月２４至２７日*]/\n\n/[dat*3*月*15日*]/*至*/[dat*17日*]/ \n\n/[dat*1949年*]/-/[dat*1972年*]/\n\n/[L*美国*]/*南北战争*/*（*/[dat*１８６１—１８６５年*]/*）*/*中*/\n\n/*软件*/*最*/*长*/*的*/*寿命*/*为*/[dur*两到三年*]/*，*/\n\n---清注意这里日期范围的标注方式。\n\n*迄今*----*词表词不标-，MET-2标今*。\n\n*今后*----*词表词不标-，MET-2标今。\n\n*晨练*----*词表词中的*晨*不标。*-\n\n*晚宴*----*词表词中的*晚*不标。*-\n\n*春联*----词表词中的*春*不标。\n\n*他们*/*的*/*今天*/*，*/*仿佛*/*就是*/*我们*/*的*/*明天*/*。*----泛指不标。\n\n*参加*/*半决赛*----*半决赛*是词表词，*半*不标。\n\n*双边*/*会谈*----*双边*是词表词，因此*双*不标。\n\n#### 7.4.4每年和年不标注\n\n本规则也适用于*月，天，小时*等其它时间单位。例如：\n\n*/*年产值*/*…*/*…*/*\n\n/*每年*/*创*/*产值*/*效益*/*…*/*…*/ \n\n/*每年*/*收入*/*…*/*…*/\n\n## 第八章 数字表达式标注细则\n\n数字表达式（*NUMEX*）包括百分数（*per*）、钱款（*mon*）、频度（*fre*）、整数（*int*）、分数（*fra*）、小数（*dec*）、序数（*ord*）、比率（*rat*）等8小类。以下是数字表达式的一些标注规则。\n\n### 8.1如果整数、分数、小数、序数后面有量词，数量短语要整体标注\n\n例如：\n\n/[int*几千万盆*]/ \n\n/[int*几家*]/*工厂*/\n\n/*一*/*家*/ [int*5*]/*人*/\n\n/*一*/*家*/ [int*5口*]/*人*/\n\n/*铁人*/[int*三项*]/*比赛*/*是*/*多*/*项目*/*的*/*综合*/*运动*/*，*\n\n/*计算机*/*配置*/*：*/586/*以上*/*，*/[int*8兆*]/*内存*/*以上*/ \n\n/*打印*/*分辨率*/*：*/[mea*180dpi*]/\n\n注：*dpi*表示每英寸的点数，所以作为*mea*标注。\n\n/*评为*/*\"*/[int*十*]/*星*/*级*/*乡镇*/*\"*/*、*/*\"*/[int*十*]/*星*/*级*/*支部*/*\"*\n\n### 8.2单纯的数字、词表词（包括俗语）中的数字都不作标注\n\n如：\n\n/*自然数*/5/*和*/6/*都是*/*整数*/\n\n/*大家*/*听*/*口令*/*，*/*齐步走*/*，*/*一*/*二*/*一*/*，一*/*二*/*一*/*，*/*一*/*二*/*三*/*四*/*，/*\n\n/*但是*/*卷子*/*上*/*的*/\"/6/\"/*还是*/*颠*/*巍巍*/*地*/*变成*/*了*/\"/8/\"/*。*/\n\n/[L*瑞士*]/*、*/[L*西班牙*]/*、*/[L*比利时*]/*、*/[L*丹麦*]/[int*四*]/*国*/\n\n/*并*/*促进*/*了*/[L*中*][L*美*]/*两国*/*的*/*交流*/*与*/*合作*/ ----*两国*是词表词。\n\n/*并*/*促进*/*了*/[L*中*][L*美*]/[int*两*]*国*/*的*/*交流*/*与*/*合作*/*，* ----错误！\n\n/*垄断*/*了*/[L*神奈川*]/*、*/[L*青森*]/*等*/[int*５*]/*县*/*的*/*交通*/*信号*/*维修*/*业务*/*。*\n\n/[L*两岸*]/*经济*/*合作*/*和*/*直接*/*三通*/ ----*三通*是词表词。\n\n/[L*两岸*]/*经济*/*合作*/*和*/*直接*/[int*三*]*通*/ ----错误!\n\n/*到*/[L*云*]/[L*贵*]/[L*川*]/*的*/*大三线*/*地区*/*，----大三线*是词表词。\n\n/*到*/[L*云*]/[L*贵*]/[L*川*]/*的*/*大*[int*三*]*线*/*地区*/*，----错误！\n\n/*十年寒窗*/ ----*词表词中的十年*不标。\n\n/*千载难逢*/ ----*词表词中的千载*不标。\n\n/*十*/*年*/*九*/*旱*/*----非词表词。虚指的十年*不标。\n\n/*眼*/*观*/*六*/*路*/*，耳*/*听*/*八*/*方*/ ----非词表词。虚指的六、八不标。\n\n/*利*/*在*/*千秋*/*的*/*大事*/ ----*虚指的*\"*千秋*\"不标。\n\n/*十*/*年*/*如*/*一*/*日*/ ---*-虚指的十年*和*一日*，不标。\n\n/*万里*[L*长城*]/ ---*-虚指的万里*，不标。\n\n/*三皇五帝*/----*三皇五帝*是词表词。\n\n/*乌七八糟*/*的*/*东西*/*几乎*/*扫荡*/*殆尽*/*----乌七八糟*是词表词。\n\n/*三大球*/*在*/*走*/*向*/*市场*/*时*/----*三大球*是词表词。\n\n/*第二次世界大战*/*的*/*反法西斯*/*斗争*/----*第二次世界大战*是词表词。\n\n/*三五成群*/*地*/*散落*/*着*/*警察*/*，*----*三五成群*是词表词。\n\n- 注：*一会儿，一起，唯一，付之一炬，一流，千方百计，一分为二，一切，二娃*等词表词中的数字一律不标。\n\n/*本职*/*创*/\"/*一流*/\"/*活动*/ /[int*亿万*]/*人民*/\n\n/[int*百万*]/*民众*/\n\n- 注：按照ER-99，*亿万、百万*不是一个抽象的数字，因此是要标注的。\n\n### 8.3约、近是一个不确切概念，故不同后面的数字串一起标注\n\n*上*、*数*、*几*、*好*则要和数字串捆绑在一起标注，而*约、近*作为特例，不与数词捆绑。\n\n/*大约*/[int*12亿*]/*人口*/\n\n/*约*/[int*四五千*]/*人*/*在*/[L*金边奥林匹克运动中心*]/*举行*/*集会*/*，*/ \n\n/*约*/[mon*十万元*]/\n\n/*近*/[mon*千万元*]/\n\n/*大概*/*需要*/*花费*/[mon*上千万美元*]/*的*/*投资*/*和*/[dur*3年*]/*左右*/*时间*/*，*/\n\n/[O*省电力公司*]/*还*/*投资*/[mon*好几百万元*]/*，*/\n\n/*多于*/[mon$90,000]/ /[mon*几百万新元*]/\n\n/*统计*/*了*/[int*上百种*]/*数字*/*，*/\n\n/*每年*/*都*/*要*/*花费*/*大量*/*外汇*/*引进*/[int*上百套*]/*系统*/\n\n/*每年*/*搞*/[int*一两个*]/*工程*/*，*/\n\n/*邀请*/*全国*/*近*/[int*百名*]/*书法*/*名家*/*，*/\n\n/*近*/[int*千名*]/*员工*/\n\n- 注：余、多本不应标注，但当它们位于量词前分割不开，所以整体加以标注。\n\n/[mon*二十七万余元*]/ \n\n/[mon*五百多万元*]/\n\n### 8.4钱款式中的地名\n\n钱款表达式中的地名不论是单音节还是多音节的，Er-99和MET-2都不标，否则就形成嵌套。\n\n如果货币字符串在文本中单独出现，字符串中没有数字修饰，那么双音节的地名要标注为*L*，单音节的地名不标注。例如非词表词*泰铢*中的*泰*不标。注意词表词*日元*、*美元*中的单音节的地名也不标。\n\n/[mon*2000新元*]/\n\n/[mon*2000新加坡元*]/\n\n/*泰*/*铢*/*汇率*/*稳定*/*在*/[mon*３８铢*]/—/[mon*３９铢*]/*兑*/[mon*美元*]/*水平*/\n\n/*纷纷*/*抛*/*出*/*日元*/*购*/*进*/[L*德国*]/*马克*/*，*/ \n\n/[L*菲律宾*]/*比索*/*对*/*美元*/*汇率*/*也*/*下跌*/*。*/\n\n### 8.5钱款标注中的特例\n\nMET-2规定：如果没有表示钱款的单位，则不标。ER-99则不然。本规范采用ER-99的规定。\n\n/*这*/*辆*/*汽车*/*值*/[mon*20万*]/\n\n/*卷标*/*上*/*的*/*价格*/*是*/ [mon*50*]/\n\n/[O*纳斯达克*]/*跌*/ [int*140*]/\n\n### 8.6频率的特例\n\n/[fre*四年一度*]/ ----*四年一度*并非词表词，但整体标注为*fre*。\n\n/[fre*一年一度*]/\n\n----*一年一度*是词表词，整体标注为*fre*。\n\n*/*主要*/*在*/*交流*/[fre*50Hz*]/*，/*额定*/*电压*/*至*/[mea*660V*]/\n\n*---*交流电的频率是*50Hz*（赫兹）*,*即每秒变化*50*周。所以理应标成*fre*而不是*mea*。\n\n/*频率*/*高*/*（*/[fre*30*－*60KHz*]/）\n\n/*卫星*/*每年*/*发射*/[fre*６至７次*]/。\n\n- 注：又一次、再一次全部标注为fre，但/*一次*/*又*/*一次*/例外，不作标注。\n\n如：\n\n/*此间*/*舆论*/[fre*又一次*]/*注意*/*到*/[L*亚*]/[L*非*]/*足球*/*的*/*差距*/ \n\n/*精湛*/*演技*/*，*/[fre*再一次*]/*赢得*/*了*/*首都*/*观众*/*的*/*由衷*/*赞赏*/\n\n### 8.7名词方没有与之搭配的量词，因此可以和前面的数词直接结合\n\n在我方、校方中的名词方没有与之搭配的量词，因此可以和前面的数词直接结合，如：\n\n/[int*三方*]/*已*/*就*/[O*劳斯莱斯*]/*汽车*/*的*/*前景*/*达成*/*协定*/*，*/\n\n### 8.8一相当于英语的冠词a，一般不标\n\n一相当于英语的冠词a，一般不标，但一倍是例外,要标fra。例如：\n\n/*一个*/*条件*/\n\n/*一*/*座*/*城市*/\n\n/*最大*/*的*/*企业*/*之一*/\n\n/*荣立*/*一等功*/ ----*一等功*是词表词，不可标注。\n\n/荣立*/[ord*一等*]*功*/----错误的标注！\n\n/*获*/*县*/*政府*/*新技术*/*推广*/[ord*一、二等*]/*奖*/*。*/\n\n/*我*/*的*/*收入*/*是*/*她*/*的*/[fra*一倍*]/ ----*一倍*是要标的。\n\n### 8.9一（1）+量词不标注*int*\n\n#### 8.9.1一+量词是词表词的情况\n\n词表词一个、一种、一类、一批、一次、一套、一阵等作为数量短语不予切分，也不标注*int*。其中有些量词重迭形式也是词表词，如一个个、一天天，应保持其整词形式，而其它非词表词的数量短语和量词重迭形式都是要切开的。\n\n/*一个*/*人*/\n\n/*一个个*/*观众*/ \n\n/*一种*/*算法*/\n\n/*一套*/*特种*/*邮票*/\n\n/*一次*/*讨论*/\n\n/*一*/*匹*/*黄骠马*/ \n\n/*一*/*栋*/*栋*/*楼房*/\n\n/*一天天*/*暖和*/*起来*/\n\n#### 8.9.2词表词一起、一块、一道、一面用作数量短语时应切开\n\n词表词*一起、一块、一道、一面*有副词和其它词性的用法，但当它们用作数量短语时一律切开，而且不标注*int*。\n\n/*一*/*块*/*石头*/\n\n/*一*/*起*/*交通*/*事故*/\n\n/*一*/*面*/*镜子*/\n\n### 8.10一（1)\"+物理单位元需按度量表达式标注\n\n一（1)\"+物理单位元（如米、公斤、摄氏度等）需按度量表达式（见6.3）标注。如：\n\n/[wei*一公斤*]/*大米*/\n\n/[mea*一度*]/*电*/\n\n### 8.11分数词素半\n\n#### 8.11.1词表词中的词素半不可标注为fra（分数)\n\n词表词*如半价、半票、半饱、半身、半世、半辈子、上半时、下半场、半边*等，但不可把上述词表词中的词素*半*标注为*fra*（分数)。\n\n/*上*/[fra*半*]/*场*/*比赛*/[L*中国*]*队*/*未进*/*一*/*球*/\n\n/*下半场*/----词表词，是正确标注。\n\n/*下*[fra*半*]*场*/----在词表词中插标*fra*是错误的。\n\n/*目前*/*还*/*空闲*/*着*/[fra*一大半*]/*的*/*营业*/*面积*/*。*/\n\n/*他们*/*之中*/*肯定*/*有*/[fra*一多半*]/*人*/*没有*/*球*/*票*/ \n\n/*有*/[fra*大半个*]/*篮球*/*场*/*那么*/*大*/\n\n- 注：当半作为一个独立的词时要标注，标注的原则是：半+量词或名词时标注，半+动词或形容词时不作标注，如：\n\n/*下半场*/*后*/[fra*半*]/*段*/\n\n/*地处*/*偏僻*/[fra*半*]/*山区*/\n\n/*部分*/*企业*/*停产*/*或*/*半*/*停产*/\n\n/*而*/*处于*/*半*/*死亡*/*或*/*休眠*/*状态*/*，*/\n\n/*干旱*/*半*/*干旱*/*地区*/*径流*/*造林*/*技术*/*、*/\n\n#### 8.11.2以下的词表词不作为分数标注，而作为其它不同的数字串标注\n\n/[dur*半年*]/ \n\n/[dur*半天*]/\n\n/[tim|dur*半夜*]/ \n\n/[int|age*半百*]/\n\n#### 8.11.3例外\n\n半个西瓜中的半个，与四半中的半概念不一样，前一个半是指二分之一，后一个半是量词，所以标注也不同。\n\n/[fra*半个*]/*西瓜*/\n\n/[int*一个*]/*西瓜*/*分为*/[int*四半*]/\n\n### 8.12序数词素首\n\n#### 8.12.1词表词中的词素首不可标注为ord（序数)\n\n词表中有许多词含有词素*首*，如*首创、首倡、首选、首发、首航、首飞、首演、首映、首战、首展、首席代表、首席科学家、首席执行官、首富、榜首、魁首、居首*等。但不可把词表词中的词素*首*单独作为*ord*（序数）来标注。\n\n/*首席执行官*/----正确标注。\n\n/[*ord首席*]*执行官*/----在词表词中插标*ord*是错误的。\n\n#### 8.12.2具有首+量词结构的词表词或非词表词，应整体作为ord标注\n\n具有\"首+量词\"结构的词表词有：*[ord*首届*]*，*[ord*首次*]*，*[ord*首批*]*，*[ord*首位*]*，[ord*首例*]等。\n\n具有首+量词结构的非词表词，如：\n\n/[L*北京市*]/[ord*首家*]/*就业*/*与*/*创业*/*组合*/*市场*/ \n\n/[P*满文军*]/*则*/*以*/*自己*/*的*/[ord*首张*]/*个人*/*专辑*/ \n\n/[dat*首日*]/*销售*/*欠佳*/\n\n----这里首日不能作序数词来标注，应标注为日期*dat*。(详见7.1)。\n\n- 注：头版、头条是词表词。它们和头一回统统标注为*ord*。如：\n\n/*在*/[dat*４月１１日*]/*的*/*《*/[O*人民日报*]/*》*/[ord*头版*]/[ord*头条*]/*社论*/*位置*/*发表*/*出来*/*，*\n\n/*由于*/*是*/[ord*头一回*]/*，*/*总*/*怕*/*有*/*个*/*闪失*/*，*\n\n- 注：\"头\"的上述标注不可类推到其它词组中，例如，\n\n*上*/*半场*/*表现*/*不好*/*，*/*头*/[dur*１０分钟*]/*甚至*/*有些*/*拖泥带水*/*。*\n\n*----*注：这里半场时词表词，但不标注为*fra*。\n\n### 8.13序数词+量词结构，应整体作为ord标注\n\n/[ord*第一期*]/ \n\n/[ord*第二*]/*故乡*/\n\n/[ord*三等*]/*奖*/\n\n/[dat*第一天*]/ *---*相对日期，标*dat,*而不是** [ord*第一*]/*天*。\n\n/[dat*第二年*]/ *---*相对日期，标*dat,*而不是** [ord*第二*]/*年*。\n\n/[O*波音*]/747 */*  ----*产品序号不标*。\n\n/*地震烈度*/*不*/*超过*/[ord*8度*]//\n\n/*这*/[ord*第二条*]/*尤为*/*重要*/*，*/ \n\n/*位居*/*金牌*/*榜*/[ord*第二名*]/*。*/\n\n/*作为*/*大豆*/*行动*/*计划*/*的*/[ord*第二步*]/\n\n/[ord*1174号*]/*文件*/\n\n/[ord*6路*]/*汽车*/ /[ord*六年级*]/*学生*/\n\n/[dat*今年*]/*读*/[ord*大三*]/\n\n/*发展*/*第一产业*/  ----*第一产业*是词表词。\n\n/*发展*/[ord*第一*]*产业*/ ----错误的标注。\n\n/*阵风*/[ord*五级］*/\n\n/*通过*/*大学*/*英语*/[ord*六级*]/\n\n- 注：联赛中的A/组、B/组等不作为序数字串标注。如：\n\n/*在*/[L*里昂*]/*进行*/*的*/*世界杯*/*Ｇ*/*组*/*比赛*/*中*/\n\n- 注：\"甲级、甲/Ａ、乙/级、乙/Ａ\"等不作为序数ord标注。如：\n\n*/*当即*/*停止*/*该*/*场*/*比赛*/*主*/*裁判员*/*执法*/*全国*/*足球*/*甲*/*Ａ*/*联赛*/*；\n\n/*获得*/[ord*前两名*]/*的*/*球队*/*晋级*/*甲*/*Ａ*/*行列*/*。*/\n\n/[dat*１９９８年*]/*全国*/[O*男篮*]/*甲*/*Ｂ*/*联赛*/\n\n/*判处*/*以*/[P*东条英机*]/*为首*/*的*/[int*７名*]/*甲级*/*战犯*/*死刑*/*。*/\n\n### 8.14仅当形容词前表示比赛名次时才和后面的序数结构一起标注\n\n仅当形容词前表示比赛名次，如前*6*名、前四（指前四名）时，才和后面的序数结构一起标注为*ord*。其余的情况如前两次、前三组、前三场、前两项等，前都不得进入被标注的数字表达式。\n\n/*获得*/[ord*前十名*]/*的*/*是*/*：*/*在*/*前*/[int*两轮*]/*小组*/*赛*/*中*/\n\n/*列*/*前*/[int*两位*]/*的*/*是*/[O*澳大利亚队*]/*和*/[O*日本队*]/*。*\n\n### 8.15文本中表示标号的数字不标\n\n规范、条例中的条款标号，包括一、二、三、Ⅰ、Ⅱ、Ⅲ、1，2，3、第一条、第二条、第三条等，一律不予标注。只有当这些条款被正文引用时，才作为序号ord被标注。例如：\n\n/*第二*/*，*/*制定*/*必要*/*的*/*行规*/*、*/*行约*/*，*/*共同*/*规范*/*，*/*共同*/*遵守*/*，*/\n\n/*一*/*无*/*资金*/*，*/*二*/*无*/*场地*/\n\n/*一*/*靠*/*政策*/*调动*/*农民*/*的*/*积极性；*/ /*二*/*靠*/*科技；*/\n\n/*一*/*是*/*继续*/*加强*/*农业；*/\n\n/*二*/*是*/*采取*/*措施*/*稳定*/*物价*/*，*/*抑制*/*通货膨胀；*/ \n\n/*１*/*．*/*自卑*/*的*/*羞耻*/*感*/*。*/\n\n/*２*/*．*/*依赖*/*的*/*恐惧*/*感*/*。*/\n\n/*（１）*/*加强*/*爱国主义*/*的*/*宣传*/*教育。*/\n\n/*（２）*/*加强*/*正确*/*的*/*理想*/*、*/*信念*/*、*/*人生观*/*、*/*价值观*/*的*/*宣传*/*教育*/*。*\n\n\"*第*+*数词*+*条*\"视为词表词，但作为文中陈述的标号时不标注*ord*。仅当其在文中被引用时才作为*ord*标注。例如：\n\n/*第一条*/*、*/*消费者*/*永远*/*是*/*对*/*的*/*；*\n\n/*第二条*/*、*/*如果*/*消费者*/*真*/*的*/*错*/*了*/*，*/*清*/*参照*/[ord*第一条*]/*。*/\n\n- 注：当上述数字表示等级序号时，则要标注为*ord*。例如：\n\n*污秽*/*等级*/*：*/[ord*Ⅰ*]/*、*/[ord*Ⅱ*]/*、*/[ord*Ⅲ*]/*、*/[ord*Ⅳ*]/*。*\n\n### 8.16人名、地名、机构名中的数字，不单独标注int\n\n/[P*佐腾一郎*]/\n\n/[L*梅竹蹊六十七号茶花庄*]/ \n\n/[O*子弟一中*]/\n\n/[O*三明市*]/\n\n/*任*/*队长*/*的*/ [O*1205钻井队*]/\n\n### 8.17外文字符串的标注\n\n由于外文的词与词之间都有空格作为分隔符，因此无需再去切分，只在标点符号的前后加切分标记。遇到字母词、名称缩写等情况也不作切分，如：/COM/经济/（网络经济）、/E/产品/（电子产品）、/卡拉/OK/等。\n\n/Good morning/  ,/everyone/./\n\n/*最近*/*引进*/*一*/*台*/JT-ESWL-*Ⅲ*/*型*/*体*/*外*/*震波*/*粉碎*/*肾结石*/*机*/*，*/\n\n\"*/[L *ZHONG* HUA *REN* MIN *GONG* HE*GUO]/\"/*，*/*这是*/[L*中华人民共和国*]/*的*/*汉语拼音*/*。*/*\n\n\"*/Brother/*，*/I *love* you *all* the *time/*，*/  Thank *you* very *much/*！*/\"/ \"/Happy *birthday* to*you/*！*/\"\n\n/Dip *one* end *of* a *straw* in *the* solution/./Blow *gently* through *the* straw/./ */A* soap *bubble* forms/./What *happens* when *you* keep *on* blowing/?/\n\n/The *bubble* bursts *because* the *pressure* inside *the* bubble *is* more *than* the*pressure *outside* the*bubble/./\n\n### 8.18数学公式和机型标号均作为一个整体来切分和标注\n\n例如：\n\n/*△*S/=/[len*12*（*S1*＋*S2*）*mm*]/\n\n/*IEC298*．*265*．*129*．*694. *420*．*56.* 529*．*932*/\n\n/*GB3804.* *3906*．*11022*/\n\n/IEC60129A2/*（*/[dat*1996*]/）*UES*－*K3*／*2/ /UEMC40K8U*／*1*/ */1* V/FJ220001R2/\n\n/*SFL12*／*17.5*/IVD *P575303RI/  /S*FL24A/IVDP5753/O2RI/\n\n## 第九章 分词歧义消解细则\n\n本章中的歧义切分实例是从微软亚洲研究院237万词训练语料、10万词测试语料和20万词散页语料中抽取出来的。这些歧义字段可粗分为交集型歧义(OAS)和组合型歧义(CAS)两大类。交集型歧义又包含用正反向最大匹配（ＭＭ）算法侦查不到的所谓隐藏的CAS。下面就分别介绍不同歧义字段的消解规则。\n\n### 9.1交集型歧义字段（OAS）\n\n#### 9.1.1交集型歧义字段示例\n\n由于交集型歧义字段的例子太多，不便穷举，所以下面只列举少量实例供参考。\n\n（１）/矛头/所/指/正是/以/包/代/管/、/负/盈/不/负/亏/、/\n\n（２）/[L四川]/一/私营企业/家/向/下岗/女工/捐款/\n\n（３）/柚木/购/进/后/市场价格/大/跌/，/\n\n（４）/图/为/[O保险公司]/向/受灾/企业/赔/付/现场/\n\n（５）/地方政府/亟需/在/加强/压/锭/监管/力度/方面/下功夫/，/\n\n（６）/与/厂/内/存留/的/旧/纱/机/一并/销毁/。/\n\n（７）经/请示/，/自行/将/本/厂/经/改造/的/应/压缩/设备\n\n（８）擅自/新/增/棉纺/生产能力/，\n\n（９）/有人/钻/政策/空子/、/骗/财政补贴/。\n\n（１０）日益/猖獗/的/走私/犯罪/活动/，\n\n（１１）/他/在/教务/活动/中/积极/研究/、\n\n（１２）对/全/山/的/商业/网点/和/摊/区/重新/进行/了/规划/和/建设/。\n\n（１３）/加强/各级/领导班子/建设/。\n\n（１４）全体/员工/开展/了/\"/人家/学/我们/，/我们/怎么办/\"/的/大/讨论/，\n\n（１５）/[O欧佩克]/提高/原油/配额/和/暖冬/等/因素/影响/，\n\n（１６）/保护/国家/和/人民群众/的/生命/财产/安全/。\n\n（１７）/以/维护/民族团结/为/己任/，\n\n（１８）/各级/领导干部/要/站/在/党/和/国家/全局/的/高度/，\n\n（１９）/只有/坚持/解放/思想/、/实事求是/的/思想/路线/，\n\n（２０）/呈现/了/\"/部队/添/战斗力/，/企业/增/生产力/，\n\n（２１）/共建/双方/通过/自上而下/层/层/签约/，\n\n（２２）/电力/部门/还/专门/建立/了/正规/的/转业军人/业务/培训/机制/，\n\n（２３）/这/条/线/不/停电/，/官兵/跳伞/太/危险/了/。\n\n（２４）如同/[L华中]/电网/强大/的/发电机/群/按照/同一/频率/转动/一样/，\n\n（２５）通过/举办/一些/全/集团/参与/的/拥军/活动/，\n\n（２６）/在/全国/工业/[ord５００]/强/中/名列前茅/的/大型/企业集团/。\n\n（２７）/本/次/检测/中/性能/系数/最高/者/。\n\n（２８）/不是/主张/所有/的/会议/都/开/成/电视电话/会议/。\n\n#### 9.1.2隐藏的交集型歧义字段\n\n隐藏的交集型歧义字段是指那些用正、反向最大匹配（ＭＭ）算法无法侦查到的交集型歧义字段。\n\n注：以下例句中，双百分号右面为改正后的切分。\n\n（1）/[L新疆]/经济/社会/发展/一定/会展/现出/越来越/美好/的/前景/\n\n/[L新疆]/经济/社会/发展/一定/会/展现/出/越来越/美好/的/前景/\n\n（2）/成立/了/专/司空/中和/地面/服务/质量/监管/的/服务/质量/督察/办公室/，/\n\n/成立/了/专/司/空中/和/地面/服务/质量/监管/的/服务/质量/督察/办公室/，/\n\n（4）/其内/容或/规则/已/译/成/[int１５]/国/语言/，/\n\n/其/内容/或/规则/已/译/成/[int１５]/国/语言/，/\n\n（5）/这/一发/现有/可能/加速/艾滋病/新药/和/疫苗/的/研制/。/\n\n这/一/发现/有/可能/加速/艾滋病/新药/和/疫苗/的/研制/。/\n\n（6）/[L韩国]/对/日出/口中/，/\n\n/[L韩国]/对/[L日]/出口/中/，/\n\n（7）/恰/在/此时/，/奉/党委/派赴/[O共产国际]/工作/的/[P张太雷]/于/[dat８月]/回国/\n\n/恰/在/此时/，/奉/党/委派/赴/[O共产国际]/工作/的/[P张太雷]/于/[dat８月]/回国\n\n（8）/站/在建/设有/[L中国]/特色/社会主义/全局/\n\n/站/在/建设/有/[L中国]/特色/社会主义/全局/\n\n（9）/金融/危机/就/可能/会演/变为/经济危机/\n\n金融/危机/就/可能/会/演变/为/经济危机/\n\n（10）/如/少数/司机/在/东侧/门楼/外道/路上/违章/占/道/停车/，/\n\n如/少数/司机/在/东侧/门楼/外/道路/上/违章/占/道/停车/，/\n\n（11）/有关/部门/要/下决心/下力/气管/好/电子/游戏/室/。/\n\n有关/部门/要/下决心/下/力气/管/好/电子/游戏/室/。/\n\n（12）/相近/似的/设施/化/保护/菜地/面积/达/[are１３００万亩]/；/\n\n/相/近似/的/设施/化/保护/菜地/面积/达/[are１３００万亩]/；/\n\n（13）/表明/了/财政部/门对/落实/科教/兴/国/战略/采取/的/实际/行动/。/\n\n/表明/了/财政/部门/对/落实/科教/兴/国/战略/采取/的/实际/行动/。/\n\n（14）儿时/站/在家/门口/向/四面/望/，/\n\n儿时/站/在/家门/口/向/四面/望/，/\n\n（15）/就/不可能/正确/地理/解和/执行/党/的/路线/方针/政策/，/\n\n/就/不可能/正确/地/理解/和/执行/党/的/路线/方针/政策/，/\n\n（16）/使得/高校/中原/有的/个别/的/知识/物化/行为/迅速/扩展/为/一种/专门/职能/。/\n\n/使得/高校/中/原有/的/个别/的/知识/物化/行为/迅速/扩展/为/一种/专门/职能/。/\n\n（17）/这是/大都/市里/的/一个/皮货/修理/店/，/\n\n/这是/大/都市/里/的/一个/皮货/修理/店/，/\n\n（18）/需/招集/体制/女性/业务员/[int四名]/，/\n\n/需/招/集体/制/女性/业务员/[int四名]/，/\n\n（19）/连同/应/交费/用以/划/支票/寄/还/。/\n\n/连同/应/交/费用/以/划/支票/寄/还/。/\n\n（20）/我们/一口气/跑/到家/门口/的/一/棵/大树/前/，/\n\n/我们/一口气/跑/到/家门/口/的/一/棵/大树/前/，/\n\n（21）/\"/唉/，/要是/好/好/复习/，/可不/会考/得/这样/糟/。/\"/\n\n\"/唉/，/要是/好/好/复习/，/可/不会/考/得/这样/糟/。/\"/\n\n（22）/特制/定本/规定/。/\n\n/特/制定/本/规定/。/\n\n（23）/股东/会所/议事/项/作/成/会议/记录/，/\n\n/股东/会/所/议/事项/作/成/会议/记录/，/\n\n### 9.2组合型歧义字段(CAS)\n\n组合型歧义字段在真实文本中大量出现，有的是比较常见的，有的是非常罕见的。尤其是有的CAS即使根据上下文也很难判断其正确切分，如正在、就是、还是、只有、只是、一道、一起等等。因此有必要针对那些高频的CAS逐条加以说明。\n\n#### 9.2.1常见的组合型歧义字段\n\n下面对一些常见的组合型歧义字段加以解释。\n\n##### 9.2.1.1数词一和量词组成的CAS\n\n词表词一个、一种、一类、一批、一次、一套、一阵等作为数量短语不予切分，也不标注int。其中有些量词重迭形式也是词表词，如一个个、一天天，应保持其整词形式，而其它非词表词的数量短语和量词重迭形式都是要切开的。（详见8.9）\n\n/*一个*/*人*/\n\n/*一个个*/*观众*/\n\n/*一天天*/*暖和*/*起来*/\n\n/*一套*/*特种*/*邮票*/\n\n/*一次*/*讨论*/\n\n/*一*/*匹*/*黄骠马*/\n\n/*一*/*栋*/*栋*/*楼房*/\n\n词表词一起、一道、一样、一手、一面、一口、一头、一气等既可以用作连词、副词、名词或形容词等，又可以切分开来成为数量短语。但像一套这样的词表词，除了数量短语的用法以外，不再有其它用法，因此不存在切分问题。词表词有一套是有本事的意思时，也不切分。这类词的切分问题只能逐个加以描述。\n\n##### 9.2.1.2动量词次与频率int的标注\n\n动量词中只有*次*被标注为频率*fre*，如[fre*再次*]*、*[fre*数次*]*、*[fre*一次次*]*、*[fre*无数次*]*、*[fre*好几次*]*，而*遍、回、趟*不标注为频率，一*/*遍、一*/*回、一*/*趟、一次（词表词不切分）、一*/*遍*/*又*/*一*/*遍、一*/*回*/*又*/*一*/*回、一*/*趟*/*又*/*一*/*趟，一次*/*又*/*一次*也不标注为*fre*。这条规则的理由如下：\n\n(1)遍表达的是动作从开始到结束的全过程；次、回描写动作的重复；趟只用于表示行走意义的动词。*去一趟*可以说成*去一次*、*去一回*，但*做一次*、*做一回*不能说成*做一趟*。\n\n*遍、次、回*有时可通用，如*你再唱一遍*，可以说成*你再唱一次*或*你再唱一回*而意思不变。但单纯表示动作数量时，只用*次*，不用*遍*，如*他表示了多次*、*敌人的三次进攻都被击退了*。\n\n*次*与*回*区别在于，*次*既用于书面语又用于口语；*回*只用于口语。如*多次、数次*等带文言色彩的短语，就不能说成*多回、数回*。\n\n(2)*这本书我看了一遍*，是指从书的开头到末尾的全过程。*这本书我看了一次*，着重指看的次数，不指看的全过程。\n\n##### 9.2.1.3一(1)＋物理单位元量词构成度量表达式\n\n当一(1)后面是长度、重量等物理单位元时应分别按度量表达式标注为*len*，*wei*，如[*len*一米*]*、*[wei*1*公斤*]（见8.10）。\n\n#### 9.2.2CAS示例\n\n下面是一些常见CAS的切分规则和示例。\n\n（1）人为作形容词时不切分。\n\n（1a）而是/深究/灾难/中/的/人为/因素/。（1b）以/人/为/本/\n\n（2）为人：\n\n（2a）也/包括/最/基本/的/为人/处事/的/行为/准则/\n\n（2b）/始终/主宰/着/他/的/为人/之/道/和/为/艺/之/方/。（2c）/我/把/不大/为/人/所/知/的/一些/往事/写/下来/，\n\n（3）一起：作名词和副词使用时不切分，作为\"数+量词\"时切分。（3a）/和/市民/一起/聊天/，/听取/群众/反映/。\n\n（3b）/[dat４月１７日]/发生/在/[L北京]/[L海淀区]/[L阜石路]/的/一/起/车祸/，/\n\n（4）一点:形容词，意思是少许，不切分。但作为数量短语时要切开。（4a）/文/中/还有/一点/小/差误/，/也/顺便/提/提/。/\n\n（4b）/都/清楚/地/意识到/了/这/一/点/，/\n\n（5）一道：作副词使用时不切分；作为\"数+量词\"时切分。\n\n（5a）/而且/要求/未来/的/丈夫/同/她/一道/挑起/照顾/[P穆]/大爷/的/担子/。\n\n（5b）/已/成为/百里/油田/的/一/道/风景/线/。\n\n（5c）/在/我/的/前额/刻下/了/一/道/道/弯曲/的/青春/印记/。\n\n（6）一面：作名词和副词使用时不切分，作为\"数+量词\"时切分。\n\n（6a）/虽然/在/现代汉语/里/含有/贬义/，/但/其/积极/的/一面/应该/肯定/。\n\n（6b）/一面/学习/，/一面/实践/，/贯彻/到/筹组/[L澳门特别行政区]/的/工作/中/去/。（6c）/爱/是/一/面/辽阔/光滑/的/回音壁/，/微小/的/爱/意/反复/回响/着/，\n\n（7）一口：作形容词和副词使用时不切分，作为\"数+量词\"时切分。（7a）/[P崔]/又/一口/回绝/并/与/其/发生/争吵/。\n\n（7b）要不/则/是/一/脸/匪/相/或者/一口/痞/气/，\n\n（7c）不由得/倒/吸/了/一/口/冷气/打/了/一个/寒战/，\n\n（7d）一/口/大/锅/解决/了/[int两家]/的/再就业/难题/。\n\n（8）一手既有名词和副词的用时，又有\"数+量词\"的用法，但在文本中一律不予区分。\n\n（8a）所有/的/为/官/为/政/者/都/能/写/一手/好/文章/，（8b）/整个/事件/是/他/一手/策划/的/，\n\n（8c）/他/一手/划水/，/一手/搂/着/女/青年/游/向/岸边/。\n\n（9）一头：作名词和副词使用时不切分，作为\"数+量词\"时切分。（9a）/二来/街道/一头/联/着/片/内/的/企业单位/，/一头/联/着/居民/，\n\n（9b）/此后/，/他/一头/钻进/常年/云雾/缭绕/的/云雾山/，/拜访/民间/郎中/，（9c）/一/头/经过/救助/已/恢复/健康/的/灰/鲸/『/[P杰杰]/』/\n\n（10）一路：作名词和副词使用时不切分；作为\"数+量词\"时切分。\n\n（10a）有时/公共汽车/挤/不/上/，/干脆/快步/当/车/一路/小跑/。\n\n（10b）我们/一路/攀登/来到/[P王永祥]/简陋/的/护林/小屋/。\n\n（10c）另/一/路/是/探索/[L火星]/、/[L木星]/等/星球/。\n\n（11）一下：用作副词和数量词使用时不切分；当一作副词下作动词时要切分。\n\n（11a）/只要/通融/一下/，/既/能/得到/一/笔/大钱/，/又/能/保持/友情/。\n\n（11b）/相互/拍打/一下/：/\"/你/猜/[rat几比几]/？/\"\n\n（11c）/书包/斜/背/在/肩/上/，/带子/太/长/，/随着/步子/一/上/一/下/跳跃/着/拍打/在/屁股/上/。\n\n（12）一片：作形容词使用时不切分；作数量短语时切分。\n\n（12a）/台上/台/下/那/一片/亲切/和谐/的/气氛/，\n\n（12b）/融入/一片/[dat夏日]/的/浓绿/之中\n\n（12c）/地板/上/看/不/到/一/片/碎/纸屑/。\n\n（12d）/宽宽大大/的/粽/叶/，/她/总/要/一/片/片/洗/净/。\n\n（13）一则：作副词使用时不切分；作数量短语时切分。\n\n（13a）/一则/表达/对/同乡/画/马/大师/[P徐悲鸿]/的/敬仰/，/二/则/愿/家乡/建设/如/骏马/奔腾/一日千里/。\n\n（13b）/[L法国]/报纸/刊/出/一/则/特写/，\n\n（14）不见：是动词见的否定形式不切分。当它同前面的动词形成V/*得*/*见*、*V/*不*/*见*的*\n\n可能式动补结构时，要切分。类似的可能式动补结构还有*V/*得*/*下去*/*、*V/*不*/*下去*/*，\n\n*V/*得*/*来*/*、*V/*不*/*来*，* *V/*得*/*起、*V/*不*/*起*，** V/*得*/*了*/*、*V/*不*/*了*/*，*V/*得*/*成*/*、*V/*不*/*成*/*，*长*/*得*/*大*/*、长*/*不*/*大*/*等*。\n\n（14a）/全/都是/\"/不见/兔子/不/撒/鹰/\"/。\n\n（14b）/人武部/就/看/不/见/一/盏/长明灯/，\n\n（15）不对：作形容词表示不正确时不切分；如果对作为介词，就要切开。\n\n（15a）父母/这么/想/当然/不对/，/可/也/不能/全/怪/他们/的/愚钝/和/落后/。\n\n（15b）/中国/主张/和平/的/外交/政策/，/中国/不/对/任何/国家/构成/威胁/。\n\n（16）不等：作形容词表示不相同时不切分；如果等作为动词，就要切开。\n\n（16a）/按照/用户/要求/生产/大小/不等/的/编织/塑料/袋/，\n\n（16b）不/等/妻子/说/什么/，/他/自己/悄悄/地/找/开/了/出路/。\n\n（16c）/时间/不/等/人/\n\n（17）不下：表示不少于时不切；作为动词下的否定式和可能式动补结构（见14），就要切开。\n\n（17a）/每天/她/经手/的/业务/不下/[int百笔]/，\n\n（17b）/架子/还/放/不/下/，/面子/还/丢/不/开/，\n\n（17c）/刑/不/上/大夫/，/礼/不/下/庶人/\n\n（17d）/[L俄罗斯]/整个/国家/开支/居/高/不/下/，\n\n（18）不成：作动词、形容词和助词使用时不切分；当它作为可能式动补结构（见14）时，一律切开。\n\n（18a）/难道/自己/这/一辈子/就/这么/过/不成/？\n\n（18b）/毛虾/已/不成/汛/，\n\n（18c）攀/\"/亲/\"/不成/反/折本/，\n\n（18d）/往往/是/有/点/而/形/不/成/网/，\n\n（19）上下：用作动词时一律切开，如\"上/下/火车\"；用作名词（包括并列意义）时则不切，如\"上下/两册\"。\n\n（19a）/经过/上下/的/共同/努力/，\n\n（19b）/上/下/车/、/船/，/须/待/车/、/船/停/稳/后/先/下/客/后/上/客/，\n\n（20）从前：作时间名词时不切分；如果从作介词前作方位词，就要切开。\n\n（20a）/有的/是/从前/在/队/中/当/板凳/球员/，\n\n（20b）/导致/美元/对/马克/的/汇价/从/前/一/交易/日/的/[rat１比１·７７６６]/降/至/[rat１比１·７７６２]/。\n\n（20c）/从/前/不久/[L深圳]/一家/公司/大规模/地/恶意/抢/注/商标/案/，\n\n（21）以为：作动词时不切分。\n\n（21a）/以为/强大/的/[P卡斯珀罗夫]/恢复/了/他/的/本来面目/。\n\n（21b）有/一些/干部/想/不/通/，/以为/是/搞/形式/，/出风头/。\n\n（21c）/我们/引/以/为/自豪/的/风格/多少/应/有些/改变/了/。\n\n（21d）/代表/们/以/为/人民/高度/负责/的/精神/，/提出/批评/和/意见/。\n\n（22）正当：作形容词时不切分。\n\n（22a）/我们/是否/能/以/某/种/不/正当/的/方式/反对/，\n\n（22b）/正/当/禾苗/生长/关键/时期/，/\n\n（23）正在：\n\n（23a）/对/各地/已/建成/尚未/售出/和/正在/建设/的/住房/，\n\n（23b）/记者/正/在/回/[L巴黎]/的/高速/列车/上/。\n\n（23c）/[O世界卫生组织]/正/在/[L科特迪瓦]/召开/国际/会议/，\n\n（24）会上：\n\n（24a）/[L苏州]/等/省市/及/有关单位/在/会上/介绍/了/经验/，\n\n（24b）/在/[O国际泳联]/[dat二十四日]/举行/的/听证/会/上/，\n\n（25）台上：\n\n（25a）/表演/完/节目/后/竟/在/台上/掩/面/而/泣/。/\n\n（25b）/预赛/是/在/有/围/绳/的/拳击/台/上/\n\n（26）走向：用作名词时不切分；用作动词+介词时，一律切分。\n\n（26a）/[L北京]/输/气/管道/工程/线路/走向/示意图/（/示意图/：/[P孙伟]/绘/）/\n\n（26b）/迈出/了/我国/航天/事业/走/向/世界/的/[ord第一步]/。\n\n（27）才能：用作名词时不切分；用作副词+能愿动词时，必须切开。\n\n（27a）/但/如果/施展/才能/的/空间/很/大/，/而且/能/充分/发挥/所/学/专长/，/不妨/一/试/。\n\n（27b）/勤奋/才/能/有/真知灼见/；\n\n（28）人才：用作名词时不切分。\n\n（28a）要/想/成为/[dat跨世纪]/人才/，/光/有/专业知识/不够/，\n\n（28b）/这/恐怕/只有/浪漫/的/[L法国]/人/才/想/得/出来/。\n\n（29）上来：作趋向动词和动词时不切分；当上作方位词时，就要切开。\n\n（29a）/一/届/新/班子/上来/以后/，/\n\n（29b）/把/工作/重点/转移/到/社会主义/现代化/建设/上/来/，/\n\n(30)上去：作趋向动词和动词时不切分；当上作方位词时，就要切开。\n\n（30a）/显然/是/上/了/学/的/[L瑶族]/娃子/写/上去/的/。/\n\n（30b）/把/科研/技术/成果/转移/到/社会/应用/上/去/。/\n\n（31）上前：\n\n（31a）/他/的/四个/弟兄/挨次/伸出/手/来/上前/祝贺/\n\n（31b）/当即/冲/上/前/去/，/扭/住/一/名/歹徒/不/放/，/\n\n（32）上路：作动词时不切分；上作动词是要切分。\n\n（32a）/我/背/起/你/的/薄被/送/你/上路/，\n\n（32b）/过去/村里/也是/上/路/打场/，/\n\n（33）得了：取助词用法时不切分；但作为动词+助词（了）和可能式动补结构（见14）时要切分。\n\n（33a）/没/叫/到/你/的/时候/，/安心/等/着/就/得了/。\n\n（33b）/经/医生/诊断/他/得/了/胃癌/。\n\n（33c）/书记/何以/承受/得/了/，\n\n（34）得出：作动词时不切；作可能式动补结构（见14）时要切分。\n\n（34a）[L天津]/近几年/的/实践/得出/了/肯定/的/答案/。\n\n（34b）为了/让/[dat今年]/蒜农/的/产品/卖/得/出/、/卖/出/好/价钱/，/\n\n（35）人称：作名词时不切分；当称作动词是要切分。\n\n（35a）作者/用/第一/人称/的/叙述/手法/，\n\n（35b）据/用/过/的/人/称/，/打/国际/长途/如/从/[L北京]/到/[L美国]/，/每/分钟/只需/传统/电话/费用/的/[fra１／４]/。\n\n（36）同行：用作名词时不切分，读作tonghang；用作动词时读作tongxing，一律切分。\n\n（36a）/这时/恰/有/同行/到来/，/只好/借/[mon一元钱]/给/他/。\n\n（36b）/笔者/与/她/骑车/同/行/。\n\n（37）从小：\n\n（37a）/图文并茂/、/声/形/兼备/的/写作/能力/将要/从小/培养/，\n\n（37b）/企业/从无到有/，/从/小/到/大/，\n\n（38）中学：\n\n（38a）/在/长期/的/中学/教学/实践/中/我/体会/到/，\n\n（38b）/引导/他们/在/实践/中/学/会/正确/行使/民主/权利/。\n\n（39）上门：\n\n（39a）/营业员/们/便/主动/上门/收款/。\n\n（39b）/打/出/了/名气/，/找/上/门/来/的/工程/一个/接/一个/。\n\n（39c）我/冲/出/门/去/，/随手/拉/上/门/。/\n\n（40）声响：作名词使用时不切分。\n\n（40a）/而/轻轻/地/挪动/椅子/走开/，/无/一点/声响/。/\n\n（40b）/\"/哗哗/\"/的/潮水/声/响/成/一片/，\n\n（41）就此：作副词使用时不切分。\n\n（41a）/国际/足球/界/一些/有识之士/就此/产生/一种/忧虑/，\n\n（41b）我们/也/欢迎/科技界/人士/就/此/问题/发表/意见/和/建议/。\n\n（42）高层次作形容词时不切分；当该词前有副词修饰时需切分。\n\n（42a）/着眼点/放/在/培养/造就/大批/高层次/科技/人才/上/。\n\n（42b）/实现/更/大/规模/、/更/高/层次/的/扩张/和/发展/。/\n\n（43）有的：\n\n（43a）/有的/用/汉语/，/有的/用/俄语/，\n\n（43b）/是/[L北大荒]/独/有/的/风味/。\n\n（44）的话：作助词使用时不切。\n\n（44a）/如果/要/使/谈判/取得/迅速/进展/的话/，\n\n（44b）/[P卡比拉]/先生/对/我/的/话/是/持/认真/态度/的/。\n\n（45）话说：整体作动词使用时不切。\n\n（45a）/话说/当年/，/他/言语/铿锵/：/\"/在/当时/，/一切/都/得/打破常规/。\n\n（45b）用/他/自己/的/话/说/，\n\n（46）标本：意思为生物/标本时不切；表示\"直接和根本\"并列的意思时要切开。\n\n（46a）/他们/还/结合/挂图/、/标本/进行/讲解/。\n\n（46b）/反/腐败/要/坚持/标/本/兼/治/，\n\n（47）上将：作为军衔使用时不切分。\n\n（47a）/[O中央军委]/副/主席/、/国务委员/兼/[O国防部]/长/[P迟浩田]/上将/\n\n（47b）/[L中国]/在/人口/问题/上/将/面临/新/的/挑战/。\n\n（48）将军：作为军衔使用时不切分。\n\n（48a）/党/和/国家/领导人/、/解放军/元帅/、/将军/、/政府/省/部级/干部\n\n（48b）/将/军：将/军/体/与/群体/紧密/结合/，/开办/体育/知识/讲座\n\n（49）之一：\n\n（49a）/企业/领导班子/不/适应/社会主义/市场经济/的/要求/是/主要/原因/之一/。\n\n（49b）/游人/视线/随/之/一/收/，/\"/[L太和宫]/\"/[int三个]/大字/豁然/在/目/。/\n\n（49c）/我/不禁/为/之/一/震/。\n\n（50）到家：作为形容词不切分。\n\n（50a）/现在/不行/，/你/技术/不/过关/，/说明/练/得/还/不/到家/，/\n\n（50b）/果不其然/，/此/儿/到/家/就/猝不及防/地/给/了/他/妈/一/刀/。\n\n（50c）/[P赵匡胤]/终于/将/义/妹/[P京娘]/送/到/家/。\n\n（51）在家：\n\n（51a）/一直/在家/等待/厂子/通知/上班/的/她/再/也/沉/不住/气/了/\n\n（51b）/实现/访客/在/家/门口/与/住户/可/视/\n\n（51c）/把/她/一/人/放/在/家/中/[P孙威锋]/放心不下/，\n\n（52）人均：\n\n（52a）/学生/拥有/计算机/的/人均/占有率/最高/\n\n（52b）/[int两]/人/均/未/达到/[fra２／３]/的/当选/票/数/，\n\n（53）中用：\n\n（53a）/\"/察/古/知/今/\"/基本上/不/中用/了/，\n\n（53b）/天文学/上/把/[L宇宙]/中/用/光学/方法/看/不/到/的/物质/称/做/暗/物质/，\n\n（53c）/西/体/[L中]/用/我/也/反对/，\n\n（54）前去：\n\n（54a）/让/[P胡洁青]/前去/扶持/、/帮忙/。\n\n（54b）一/名/应邀/到会/的/[L北京]/小学生/激动/地/跑/上/前/去/请/他/签名/。\n\n（55）词表词\"受过\"只有\"代人受过\"的意思。当动词受和助词过构成\"动+助\"结构时，一律切开。\n\n（55a）/它们/代/四/奸/受过/，\n\n（55b）/[P鲁迅]/虽然/在/[dat二十年代中期]/受/过/[P托洛茨基]/的/一定/影响/，\n\n（56）结果：有名词和动词两种用法，都不切分，动词结果的意思是杀死，而不是结出果实的意思。作为后一个意思，名词果是动词结的宾语，所以需切分。\n\n（56a）/矫枉过正/的/结果/，/是/大家/几乎/忘/了/怎么/吃/，/\n\n（56b）/种/果树/一般/要/三年/才/能/结/果/，\n\n#### 9.2.4就是、只有、只是、还是的切分规则\n\n##### 9.2.4.1就是\n\n就是作副词、连词、助词使用时不切分。但作动词时，就是副词，是是动词，一律切分。\n\n(A)作副词时，共有6个义项：\n\n（i）单用，表示同意，对；\n\n(ii)表示坚决，不可更改；\n\n(iii)强调肯定某种性质和状态，含有反驳意味；(iv)强调迅速果断；\n\n(v)确定范围，排除其它；(vi)表示没有别的情况；。\n\n（1）我/一定/办到/，您/放心/就是/。/\n\n（2）/反正/姥爷/就是/看/我/不/顺心/，/一点/也/不/喜欢/我/。/\n\n（3）/望/着/车/来/车/往/的/马路/，/一/站/就是/[int几个小时]/。/\n\n（4）/就是/节目/诉/求/为/非常/鲜明/的/单一/主题/，\n\n(B)作连词有2个义项：\n\n(i)表示假设的让步；即使(后面常用也作呼应)；\n\n(ii)表示一种极端情况；纵然。如：\n\n(5)不是/播种/，/就是/锄地/；/不是/下/田/挖/野菜/，/就是/上山/打柴/。\n\n(6)这个/建议/好/倒/是/好/，/就是/远水/不解/近/渴/。/\n\n(C)就是作动词时，一律切分，就是副词，是是动词。如：\n\n（7）/[O光华国中]/职员/[P杨一中]/就/是/买/菜/变成/[O慈德]/会员/的/一/例/。\n\n（8）/多元化/的/意思/就/是/有/了/更/多/的/选择/，\n\n（9）/最/特别/的/就/是/黄金/压制/的/邮票/。\n\n（10）/最/关键/的/原则/就/是/「/避/凶/趋/吉/」/，\n\n（11）这/就/是/[L海尔-波普彗星]/。/\n\n##### 9.2.4.2只有\n\n只有作为一个词表词有副词和连词两个义项。当他用作动词时，一律切开。（A）只有做副词相当于只好，表示唯一的选择。如：\n\n（1）/家属/最后/只有/寄/望/对岸/[O海协会]/能/请/[L大陆]/渔船/协/寻/。\n\n（2）/[L中国]/的/体育/长期/是/国家/一/家/办/，/发达国家/是/国家/不/办/，/只有/社会/办/，/现在/国际/体育/的/潮流/是/国家/与/社会/共同/兴办/。/\n\n（3）无/雪/的/[dat冬天]/是/难挨/的/，/我/只有/在/心中/落/着/一/场/场/大雪/。/\n\n（4）协办员/和/见习员/在/通过/[int三道]/关/后/，/还要/经由/主办员/挑选/，/没有/主办员/\n\n挑选/的/也/只有/待岗/。/\n\n（5）如果/\"/邪恶/的/敌人/对/[L伊]/发动/侵略/，/[L伊拉克]/将/别无选择/，/只有/用/其/全部/的/潜力/、/经验/和/信仰/进行/自卫/\"/。/\n\n(B)\"只有\"作连词用表示必要条件，下文常用副词才、方呼应。如：\n\n（6）只有/掌握/了/最/先进/的/科学/，/我们/才/能/有/巩固/的/国防/。/\n\n（7）高尚/的/世界/只/对/高尚/的/人们/存在/，/高尚/的/精神/境界/只有/高尚/的/人们/才/有/\n\n资格/领略/。/\n\n（C）\"只有\"用作动词时一律切开。这时\"只\"做副词、\"有\"作动词。如：\n\n（8）/完成/管理/的/比率/只/有/[per百分之八十九]/，\n\n（9）/车行/时速/只/有/[len卅到五十公里]/左右/；\n\n（10）/目前/[O基隆邮局]/只/有/一个/集邮/柜台/，\n\n（11）/因/[O中嵙国小]/整个/学区/只/有/一个/[L中嵙里]/，\n\n##### 9.2.4.3还是\n\n还是有三种用法：连词、副词和动词。作动词时一律切分。\n\n（A）作连词用时表示选择，通常跟无论、不管等连用。带连词还是的句子，除疑问句外，还是都可以换成或者，意思不变。例如：\n\n（1）无论是/说/新/话/，/提/新/观点/，/还是/放弃/前人/和/本本/上/的/过时/的/观点/、/错误/的/结论/，/都/需要/勇气/。/\n\n（2）农民/[P张戎梅]/说/：/\"/我们/村/不论/是/养猪/还是/种菜/的/，/现在/都/把/眼睛/盯/在/了/铁路/两头/。/\"/\n\n（3）不管/是/开工/还是/竣工/，/既/有/庆典/，/又/有/报导/，/或/称/世纪/工程/，\n\n（4）他们/不但/是/我们/公司/发展/的/\"/动力/之/源/\"/，/还是/我们/学习/的/好榜样/！/\n\n（B）还是作副词用时有三个义项：\n\n（i）表示行为、动作或状态继续保持不变，相当于\"仍然\"、\"依然\"。\n\n（ii）表示经过比较后做出的选择。\n\n（iii）加强语气，相当于\"到底\"、\"究竟\"、\"毕竟\"。\n\n还是/d用在动词、形容词前，可以省作还，而用在主语前不能省作还。\n\n（5）/但/现场/交通/还是/十分/杂乱/。\n\n（6）/该/基金/还是/可以/支应/灾民/最高/[mon一百万元]/的/贷款/额/，\n\n（7）/很多/居民/还是/使用/地下水/，\n\n（8）/[P陈]/还是/不/改/顽皮/个性/，\n\n（9）/多数人/还是/喜欢/为/宝宝/选/个/金/饰/，\n\n（10）/[P陈小弟]/的/父/母亲/还是/勇敢/地/生/下/他/，\n\n(C)还是用作动词时一律都要切开，即还作副词使用，有作动词使用。句型\"是/v……的\"可以帮助我们判断还是在句中是不是一种动词的用法。\n\n（11）/关键/还/是/在/府/会/双方/态度/，\n\n（12）/初/到/部队/，/[age十五六岁]/，/还/是/个/没/见/过/世面/的/毛孩子/。/\n\n（13）/她/不/相信/歌剧/这/门/综合/艺术/会/落入/低谷/，/认为/关键/还/是/提高/歌剧/自身/的\n\n/品质/。/\n\n（14）但/在/日常/工作/中/，/我/深感/除了/忙/还/是/忙/，/搞/得/焦头烂额/，/一天到晚/自己/不/属于/自己/。/\n\n##### 9.2.4.4只是\n\n只是有三种用法：副词、连词和动词。作动词时统统切分。\n\n（A）只是作副词使用时有两个义项：\n\n(i)表示限定某种情况或范围，相当于仅仅是。句末用而已或罢了等配合，\n\n表示语气更为缓和。\n\n(ii)强调在任何条件下情况都不变，有总是的意思。\n\n（1）/只是/作为/预定/分娩/日/的/参考/。\n\n（2）/只是/没有/焢/窑/经验/的/[P张]/课/长/，\n\n（3）/他/虽/表示/民意调查/结果/数据/只是/具有/参考/价值/，\n\n（4）/施工/初期/只是/修剪/树枝/，\n\n（B）只是作连词用，用在后一分句，表示轻微的转折，补充修正上文的意思，与不过的用法相近。\n\n（5）/记者/在/重灾区/[L大河乡]/注意/到/，/群众/有/饭/吃/，/有/衣/穿/，/有/伤病/能/医治/，/只是/搭建/的/小/窝棚/难以/抵御/坝上/呼啸/的/寒风/。/\n\n（6）[dat唐朝]/著名/诗人/[P李商隐]/『/夕阳/无限/好/，/只是/近/黄昏/』/的/诗句/是/对/黄昏/的/叹息/和/无奈/，/\n\n(C)只是用作动词时一律要切开，即只作副词，是作动词。如：\n\n（7）/他/只/是/[dur一个月]/领/[mon二万多元]/的/工人/\n\n（8）/事实上/[L盐埔乡]/公所/的/薪水/无/着落/只/是/冰山/一/角/，\n\n（9）/这些/需求/不只/是/钱/或/资源/，/\n"
  },
  {
    "path": "docs/api/common/configurable.rst",
    "content": ".. _api/configurable:\n\nconfigurable\n====================\n\n\n.. autoclass:: hanlp_common.configurable.Configurable\n\t:members:\n\n.. autoclass:: hanlp_common.configurable.AutoConfigurable\n\t:members:\n"
  },
  {
    "path": "docs/api/common/conll.rst",
    "content": ".. _api/conll:\n\nconll\n====================\n\n\n.. autoclass:: hanlp_common.conll.CoNLLWord\n\t:members:\n\n.. autoclass:: hanlp_common.conll.CoNLLUWord\n\t:members:\n\n.. autoclass:: hanlp_common.conll.CoNLLSentence\n\t:members:"
  },
  {
    "path": "docs/api/common/constant.rst",
    "content": "constant\n====================\n\n\n.. automodule:: hanlp_common.constant\n\t:members:\n"
  },
  {
    "path": "docs/api/common/document.rst",
    "content": ".. _api/document:\n\ndocument\n====================\n\n.. currentmodule:: hanlp_common\n\n.. autoclass:: hanlp_common.document.Document\n\t:members:\n"
  },
  {
    "path": "docs/api/common/index.md",
    "content": "# hanlp_common\n\nCommon APIs shared between `hanlp` and `restful`.\n\n```{toctree}\ndocument\nconll\nconfigurable\nconstant\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/common/component.rst",
    "content": "component\n=================\n\n.. currentmodule:: hanlp.common\n\n.. autoclass:: hanlp.common.component.Component\n\t:members:\n"
  },
  {
    "path": "docs/api/hanlp/common/dataset.md",
    "content": "# dataset\n\nThis module provides base definition for datasets, dataloaders and samplers.\n\n## datasets\n\n```{eval-rst}\n.. currentmodule:: hanlp.common\n\n.. autoclass:: hanlp.common.dataset.Transformable\n\t:members:\n\n.. autoclass:: hanlp.common.dataset.TransformableDataset\n\t:members:\n\t:special-members:\n\t:exclude-members: __init__, __repr__\n```\n\n## dataloaders\n\n```{eval-rst}\n.. currentmodule:: hanlp.common\n\n.. autoclass:: hanlp.common.dataset.PadSequenceDataLoader\n\t:members:\n\t:special-members:\n\t:exclude-members: __init__, __repr__\n\n.. autoclass:: hanlp.common.dataset.PrefetchDataLoader\n\t:members:\n\t:special-members:\n\t:exclude-members: __init__, __repr__\n```\n\n## samplers\n\n```{eval-rst}\n.. currentmodule:: hanlp.common\n\n.. autoclass:: hanlp.common.dataset.BucketSampler\n\t:members:\n\n.. autoclass:: hanlp.common.dataset.KMeansSampler\n\t:members:\n\n.. autoclass:: hanlp.common.dataset.SortingSampler\n\t:members:\n```\n\n## sampler builders\n\n```{eval-rst}\n.. currentmodule:: hanlp.common\n\n.. autoclass:: hanlp.common.dataset.SamplerBuilder\n\t:members:\n\n.. autoclass:: hanlp.common.dataset.SortingSamplerBuilder\n\t:members:\n\n.. autoclass:: hanlp.common.dataset.KMeansSamplerBuilder\n\t:members:\n\n```"
  },
  {
    "path": "docs/api/hanlp/common/index.md",
    "content": "# common\n\nCommon base classes.\n\n```{toctree}\nstructure\nvocab\ntransform\ndataset\ncomponent\ntorch_component\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/common/structure.md",
    "content": "# structure\n\n```{eval-rst}\n.. currentmodule:: hanlp.common\n\n.. autoclass:: hanlp.common.structure.ConfigTracker\n\t:members:\n\n.. autoclass:: hanlp.common.structure.History\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/common/torch_component.md",
    "content": "# torch_component\n\n```{eval-rst}\n.. currentmodule:: hanlp.common.torch_component\n\n.. autoclass:: hanlp.common.torch_component.TorchComponent\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/common/transform.md",
    "content": "# transform\n\n```{eval-rst}\n.. currentmodule:: hanlp.common\n\n.. autoclass:: hanlp.common.transform.VocabDict\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/common/vocab.md",
    "content": "# vocab\n\n```{eval-rst}\n.. currentmodule:: hanlp.common\n\n.. autoclass:: hanlp.common.transform.Vocab\n\t:members:\n\t:special-members:\n\t:exclude-members: __init__, __repr__, __call__, __str__\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/classifiers.md",
    "content": "# classifiers\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.classifiers\n\n.. autoclass:: hanlp.components.classifiers.transformer_classifier.TransformerClassifier\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/eos.md",
    "content": "# eos\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.eos\n\n.. autoclass:: hanlp.components.eos.ngram.NgramSentenceBoundaryDetector\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/index.md",
    "content": "# components\n\nNLP components.\n\n```{toctree}\nmtl/index\nclassifiers\neos\ntokenizers/index\nlemmatizer\ntaggers/index\nner/index\nparsers/index\nsrl/index\npipeline\nsts\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/components/lemmatizer.md",
    "content": "# lemmatizer\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.lemmatizer\n\n.. autoclass:: TransformerLemmatizer\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/mtl/index.md",
    "content": "# mtl\n\nMulti-Task Learning (MTL) framework.\n\n```{toctree}\nmtl\ntasks/index\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/components/mtl/mtl.md",
    "content": "# MultiTaskLearning\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.mtl\n\n.. autoclass:: hanlp.components.mtl.multi_task_learning.MultiTaskLearning\n\t:members:\n\t:special-members:\n\t:exclude-members: __init__, __repr__\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/mtl/tasks/constituency.md",
    "content": "# con\n\nConstituency parsing.\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.mtl\n\n.. autoclass:: hanlp.components.mtl.tasks.constituency.CRFConstituencyParsing\n\t:members:\n\t:exclude-members: execute_training_loop, fit_dataloader\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/mtl/tasks/dep.md",
    "content": "# dep\n\nDependency parsing.\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.mtl\n\n.. autoclass:: hanlp.components.mtl.tasks.dep.BiaffineDependencyParsing\n\t:members:\n\t:exclude-members: execute_training_loop, fit_dataloader\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/mtl/tasks/index.md",
    "content": "# tasks\n\nMulti-Task Learning (MTL) tasks.\n\n```{toctree}\ntask\nconstituency\ndep\nsdp\nud\nlem\npos\ntok\nner/index\nsrl/index\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/components/mtl/tasks/lem.md",
    "content": "# lem\n\nLemmatization.\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.mtl\n\n.. autoclass:: hanlp.components.mtl.tasks.lem.TransformerLemmatization\n\t:members:\n\t:exclude-members: execute_training_loop, fit_dataloader\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/mtl/tasks/ner/biaffine_ner.md",
    "content": "# biaffine_ner\n\nBiaffine Named Entity Recognition.\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.mtl\n\n.. autoclass:: hanlp.components.mtl.tasks.ner.biaffine_ner.BiaffineNamedEntityRecognition\n\t:members:\n\t:exclude-members: execute_training_loop, fit_dataloader\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/mtl/tasks/ner/index.md",
    "content": "# ner\n\nNamed Entity Recognition.\n\n```{toctree}\ntag_ner\nbiaffine_ner\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/components/mtl/tasks/ner/tag_ner.md",
    "content": "# tag_ner\n\nTagging based Named Entity Recognition.\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.mtl\n\n.. autoclass:: hanlp.components.mtl.tasks.ner.tag_ner.TaggingNamedEntityRecognition\n\t:members:\n\t:exclude-members: execute_training_loop, fit_dataloader\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/mtl/tasks/pos.md",
    "content": "# pos\n\nPart-of-speech tagging.\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.mtl\n\n.. autoclass:: hanlp.components.mtl.tasks.pos.TransformerTagging\n\t:members:\n\t:exclude-members: execute_training_loop, fit_dataloader\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/mtl/tasks/sdp.md",
    "content": "# sdp\n\nSemantic Dependency Parsing.\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.mtl\n\n.. autoclass:: hanlp.components.mtl.tasks.sdp.BiaffineSemanticDependencyParsing\n\t:members:\n\t:exclude-members: execute_training_loop, fit_dataloader\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/mtl/tasks/srl/bio_srl.md",
    "content": "# bio_srl\n\nBIO Tagging based Semantic Role Labeling.\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.mtl\n\n.. autoclass:: hanlp.components.mtl.tasks.srl.bio_srl.SpanBIOSemanticRoleLabeling\n\t:members:\n\t:exclude-members: execute_training_loop, fit_dataloader\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/mtl/tasks/srl/index.md",
    "content": "# srl\n\nSemantic Role Labeling.\n\n```{toctree}\nbio_srl\nrank_srl\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/components/mtl/tasks/srl/rank_srl.md",
    "content": "# rank_srl\n\nSpan Ranking Semantic Role Labeling.\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.mtl\n\n.. autoclass:: hanlp.components.mtl.tasks.srl.rank_srl.SpanRankingSemanticRoleLabeling\n\t:members:\n\t:exclude-members: execute_training_loop, fit_dataloader\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/mtl/tasks/task.md",
    "content": "# Task\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.mtl\n\n.. autoclass:: hanlp.components.mtl.tasks.Task\n\t:members:\n\t:exclude-members: execute_training_loop, fit_dataloader\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/mtl/tasks/tok.md",
    "content": "# tok\n\nTokenization.\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.mtl\n\n.. autoclass:: hanlp.components.mtl.tasks.tok.tag_tok.TaggingTokenization\n\t:members:\n\t:exclude-members: execute_training_loop, fit_dataloader\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/mtl/tasks/ud.md",
    "content": "# ud\n\nUniversal Dependencies Parsing (lemmatization, features, PoS tagging and dependency parsing).\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.mtl\n\n.. autoclass:: hanlp.components.mtl.tasks.ud.UniversalDependenciesParsing\n\t:members:\n\t:exclude-members: execute_training_loop, fit_dataloader\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/ner/biaffine_ner.md",
    "content": "# biaffine_ner\n\nBiaffine Named Entity Recognition.\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.ner.transformer_ner\n\n.. autoclass:: hanlp.components.ner.biaffine_ner.biaffine_ner.BiaffineNamedEntityRecognizer\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/ner/index.md",
    "content": "# ner\n\nNamed Entity Recognition.\n\n```{toctree}\ntransformer_ner\nrnn_ner\nbiaffine_ner\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/components/ner/rnn_ner.md",
    "content": "# rnn_ner\n\nTagging based Named Entity Recognition.\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.ner.rnn_ner\n\n.. autoclass:: hanlp.components.ner.rnn_ner.RNNNamedEntityRecognizer\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/ner/transformer_ner.md",
    "content": "# transformer_ner\n\nTagging based Named Entity Recognition.\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.ner.transformer_ner\n\n.. autoclass:: hanlp.components.ner.transformer_ner.TransformerNamedEntityRecognizer\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/parsers/biaffine_dep.md",
    "content": "# biaffine_dep\n\nBiaffine dependency parser.\n\n```{eval-rst}\n.. currentmodule:: hanlp.components\n\n.. autoclass:: hanlp.components.parsers.biaffine.biaffine_dep.BiaffineDependencyParser\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/parsers/biaffine_sdp.md",
    "content": "# biaffine_sdp\n\nBiaffine dependency parser.\n\n```{eval-rst}\n.. currentmodule:: hanlp.components\n\n.. autoclass:: hanlp.components.parsers.biaffine.biaffine_sdp.BiaffineSemanticDependencyParser\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/parsers/crf_constituency_parser.md",
    "content": "# crf_constituency_parser\n\nBiaffine dependency parser.\n\n```{eval-rst}\n.. currentmodule:: hanlp.components\n\n.. autoclass:: hanlp.components.parsers.constituency.crf_constituency_parser.CRFConstituencyParser\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/parsers/index.md",
    "content": "# parsers\n\nParsers.\n\n```{toctree}\nbiaffine_dep\nbiaffine_sdp\nud_parser\ncrf_constituency_parser\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/components/parsers/ud_parser.md",
    "content": "# ud_parser\n\nUniversal Dependencies Parsing (lemmatization, features, PoS tagging and dependency parsing).\n\n```{eval-rst}\n.. currentmodule:: hanlp.components\n\n.. autoclass:: hanlp.components.parsers.ud.ud_parser.UniversalDependenciesParser\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/pipeline.md",
    "content": "# pipeline\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.pipeline\n\n.. autoclass:: hanlp.components.pipeline.Pipe\n\t:members:\n\t\n.. autoclass:: hanlp.components.pipeline.Pipeline\n\t:members:\n\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/components/srl/index.md",
    "content": "# srl\n\nSemantic Role Labelers.\n\n```{toctree}\nspan_rank\nspan_bio\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/components/srl/span_bio.md",
    "content": "# span_bio\n\nSpan BIO tagging based SRL.\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.srl.span_bio.span_bio\n\n.. autoclass:: SpanBIOSemanticRoleLabeler\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/srl/span_rank.md",
    "content": "# span_rank\n\nSpan Rank based SRL.\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.srl.span_rank.span_rank\n\n.. autoclass:: SpanRankingSemanticRoleLabeler\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/sts.md",
    "content": "# sts\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.sts\n\n.. autoclass:: hanlp.components.sts.transformer_sts.TransformerSemanticTextualSimilarity\n\t:members:\n\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/components/taggers/index.md",
    "content": "# taggers\n\nTaggers.\n\n```{toctree}\ntransformer_tagger\nrnn_tagger\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/components/taggers/rnn_tagger.md",
    "content": "# rnn_tagger\n\nRNN based tagger.\n\n```{eval-rst}\n.. currentmodule:: hanlp.components\n\n.. autoclass:: hanlp.components.taggers.rnn_tagger.RNNTagger\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/taggers/transformer_tagger.md",
    "content": "# transformer_tagger\n\nTransformer based tagger.\n\n```{eval-rst}\n.. currentmodule:: hanlp.components\n\n.. autoclass:: hanlp.components.taggers.transformers.transformer_tagger.TransformerTagger\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/tokenizers/index.md",
    "content": "# tokenizers\n\nTokenizers.\n\n```{toctree}\ntransformer\nmulti_criteria\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/components/tokenizers/multi_criteria.md",
    "content": "# multi_criteria\n\nTransformer based Multi-Criteria Word tokenizer.\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.tokenizers.multi_criteria_cws_transformer\n\n.. autoclass:: hanlp.components.tokenizers.multi_criteria_cws_transformer.MultiCriteriaTransformerTaggingTokenizer\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/components/tokenizers/transformer.md",
    "content": "# transformer\n\nTransformer based tokenizer.\n\n```{eval-rst}\n.. currentmodule:: hanlp.components.tokenizers.transformer\n\n.. autoclass:: hanlp.components.tokenizers.transformer.TransformerTaggingTokenizer\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/datasets/constituency/constituency_dataset.md",
    "content": "# constituency_dataset\n\n```{eval-rst}\n\n.. autoclass:: hanlp.datasets.parsing.loaders.constituency_dataset.ConstituencyDataset\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/datasets/constituency/index.md",
    "content": "# con\n\nConstituency parsing datasets.\n\n```{toctree}\nconstituency_dataset\nresources\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/datasets/constituency/resources.md",
    "content": "# resources\n\n## Chinese Treebank\n\n\n### CTB8\n\n\n\n````{margin} **Discussion**\n```{seealso}\nAbout our data split on [our forum](https://bbs.hankcs.com/t/topic/3024).\n```\n````\n\n```{eval-rst}\n\n\n.. autodata:: hanlp.datasets.parsing.ctb8.CTB8_BRACKET_LINE_NOEC_TRAIN\n.. autodata:: hanlp.datasets.parsing.ctb8.CTB8_BRACKET_LINE_NOEC_DEV\n.. autodata:: hanlp.datasets.parsing.ctb8.CTB8_BRACKET_LINE_NOEC_TEST\n\n```\n\n### CTB9\n\n````{margin} **Discussion**\n```{seealso}\nAbout our data split on [our forum](https://bbs.hankcs.com/t/topic/3024).\n```\n````\n\n```{eval-rst}\n\n\n.. autodata:: hanlp.datasets.parsing.ctb9.CTB9_BRACKET_LINE_NOEC_TRAIN\n.. autodata:: hanlp.datasets.parsing.ctb9.CTB9_BRACKET_LINE_NOEC_DEV\n.. autodata:: hanlp.datasets.parsing.ctb9.CTB9_BRACKET_LINE_NOEC_TEST\n\n```\n\n## English Treebank\n\n### PTB\n\n```{eval-rst}\n\n.. autodata:: hanlp.datasets.parsing.ptb.PTB_TRAIN\n.. autodata:: hanlp.datasets.parsing.ptb.PTB_DEV\n.. autodata:: hanlp.datasets.parsing.ptb.PTB_TEST\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/datasets/dep/conll_dataset.md",
    "content": "# conll\n\n```{eval-rst}\n.. currentmodule:: hanlp.datasets.parsing.loaders.conll_dataset \n\n\n.. autoclass:: CoNLLParsingDataset\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/datasets/dep/index.md",
    "content": "# dep\n\nDependency parsing datasets.\n\n```{toctree}\nconll_dataset\nresources\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/datasets/dep/resources.md",
    "content": "# resources\n\n## PKU Multiview Treebank\n\nPKU Multi-view Chinese Treebank, released by PKU-ICL. It contains the sentences from People's Daily(19980101-19980110).\nThe number of sentences in it is 14463.\n\n```{eval-rst}\n\n.. automodule:: hanlp.datasets.parsing.pmt1\n    :members:\n\n```\n\n## Chinese Treebank\n\n### CTB5\n\n```{eval-rst}\n\n.. automodule:: hanlp.datasets.parsing.ctb5\n    :members:\n\n```\n\n### CTB7\n\n```{eval-rst}\n\n.. automodule:: hanlp.datasets.parsing.ctb7\n    :members:\n\n```\n\n### CTB8\n\n```{eval-rst}\n\n.. Attention::\n\n    We propose a new data split for CTB which is different from the academia conventions with the following 3 advantages.\n    \n    - Easy to reproduce. Files ending with ``8`` go to dev set, ending with ``9`` go to the test set, otherwise go to the training set.\n    - Full use of CTB8. The academia conventional split omits 50 gold files while we recall them.\n    - More balanced split across genres. Proportions of samples in each genres are similar.\n    \n    We also use Stanford Dependencies 3.3.0 which offers fine-grained relations and more grammars than the conventional\n    head finding rules introduced by :cite:`zhang-clark-2008-tale`.\n    \n    Therefore, scores on our preprocessed CTB8 are not directly comparable to those in most literatures. We have \n    experimented the same model on the conventionally baked CTB8 and the scores could be 4~5 points higher. \n    We believe it's worthy since HanLP is made for practical purposes, not just for producing pretty numbers.\n    \n```\n\n````{margin} **Discussion**\n```{seealso}\nWe have a discussion on [our forum](https://bbs.hankcs.com/t/topic/3024).\n```\n````\n\n```{eval-rst}\n\n\n.. autodata:: hanlp.datasets.parsing.ctb8.CTB8_SD330_TRAIN\n.. autodata:: hanlp.datasets.parsing.ctb8.CTB8_SD330_DEV\n.. autodata:: hanlp.datasets.parsing.ctb8.CTB8_SD330_TEST\n\n```\n\n### CTB9\n\n```{eval-rst}\n\n.. Attention::\n\n    Similar preprocessing and splits with CTB8 are applied. See the notice above.\n    \n```\n\n```{eval-rst}\n\n\n.. autodata:: hanlp.datasets.parsing.ctb9.CTB9_SD330_TRAIN\n.. autodata:: hanlp.datasets.parsing.ctb9.CTB9_SD330_DEV\n.. autodata:: hanlp.datasets.parsing.ctb9.CTB9_SD330_TEST\n\n```\n\n## English Treebank\n\n### PTB\n\n```{eval-rst}\n\n.. autodata:: hanlp.datasets.parsing.ptb.PTB_SD330_TRAIN\n.. autodata:: hanlp.datasets.parsing.ptb.PTB_SD330_DEV\n.. autodata:: hanlp.datasets.parsing.ptb.PTB_SD330_TEST\n\n```\n\n## Universal Dependencies\n\n### Languages\n\n```{eval-rst}\n\n.. automodule:: hanlp.datasets.parsing.ud.ud27\n    :members:\n\n```\n\n### Multilingual\n\n```{eval-rst}\n\n.. automodule:: hanlp.datasets.parsing.ud.ud27m\n    :members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/datasets/eos/eos.md",
    "content": "# eos\n\n```{eval-rst}\n.. currentmodule:: hanlp.datasets.eos.eos\n\n.. autoclass:: SentenceBoundaryDetectionDataset\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/datasets/eos/index.md",
    "content": "# eos\n\nSentence boundary detection datasets.\n\n```{toctree}\neos\nresources\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/datasets/eos/resources.md",
    "content": "# resources\n\n## nn_eos\n\n```{eval-rst}\n\n.. automodule:: hanlp.datasets.eos.loaders.nn_eos\n    :members:\n\n```"
  },
  {
    "path": "docs/api/hanlp/datasets/index.md",
    "content": "# datasets\n\n```{eval-rst}\nNLP datasets grouped by tasks. For each task, we provide at least one ``torch.utils.data.Dataset`` compatible class\nand several open-source resources. Their file format and description can be found in their ``Dataset.load_file`` \ndocuments. Their contents are split into ``TRAIN``, ``DEV`` and ``TEST`` portions, each of them is stored in\na Python constant which can be fetched using :meth:`~hanlp.utils.io_util.get_resource`.  \n``` \n\n````{margin} **Professionals use Linux**\n```{note}\nMany preprocessing scripts written by professionals make heavy use of Linux/Unix tool chains like shell, perl, gcc, \netc., which is not available or buggy on Windows. You may need a *nix evironment to run these scripts.\n```\n````\n\n```{toctree}\neos/index\ntok/index\npos/index\nner/index\ndep/index\nsrl/index\nconstituency/index\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/datasets/ner/index.md",
    "content": "# ner\n\nNER datasets.\n\n```{toctree}\ntsv\njson\nresources\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/datasets/ner/json.md",
    "content": "# json\n\n```{eval-rst}\n.. currentmodule:: hanlp.datasets.ner.loaders.json_ner\n\n.. autoclass:: JsonNERDataset\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/datasets/ner/resources.md",
    "content": "# resources\n\n## CoNLL 2003\n\n```{eval-rst}\n\n.. automodule:: hanlp.datasets.ner.conll03\n    :members:\n\n```\n\n## MSRA\n\n```{eval-rst}\n\n.. automodule:: hanlp.datasets.ner.msra\n    :members:\n\n```\n\n## OntoNotes5\n\n```{eval-rst}\n\n.. autodata:: hanlp.datasets.srl.ontonotes5.chinese.ONTONOTES5_CONLL12_CHINESE_TRAIN\n.. autodata:: hanlp.datasets.srl.ontonotes5.chinese.ONTONOTES5_CONLL12_CHINESE_DEV\n.. autodata:: hanlp.datasets.srl.ontonotes5.chinese.ONTONOTES5_CONLL12_CHINESE_TEST\n\n.. autodata:: hanlp.datasets.srl.ontonotes5.chinese.ONTONOTES5_CONLL12_NER_CHINESE_TRAIN\n.. autodata:: hanlp.datasets.srl.ontonotes5.chinese.ONTONOTES5_CONLL12_NER_CHINESE_DEV\n.. autodata:: hanlp.datasets.srl.ontonotes5.chinese.ONTONOTES5_CONLL12_NER_CHINESE_TEST\n\n```\n\n## Resume\n\n```{eval-rst}\n\n.. automodule:: hanlp.datasets.ner.resume\n    :members:\n```\n\n## Weibo\n\n\n```{eval-rst}\n\n.. automodule:: hanlp.datasets.ner.weibo\n    :members:\n```"
  },
  {
    "path": "docs/api/hanlp/datasets/ner/tsv.md",
    "content": "# tsv\n\n```{eval-rst}\n.. currentmodule:: hanlp.datasets.ner.loaders.tsv\n\n.. autoclass:: TSVTaggingDataset\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/datasets/pos/index.md",
    "content": "# pos\n\nPoS datasets. \n\n```{eval-rst}\nPoS is a normal tagging task which uses :class:`hanlp.datasets.ner.loaders.tsv.TSVTaggingDataset` for loading.\n```\n\n```{toctree}\nresources\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/datasets/pos/resources.md",
    "content": "# resources\n\n## CTB5\n\n```{eval-rst}\n\n.. automodule:: hanlp.datasets.pos.ctb5\n    :members:\n\n```\n\n## CTB8\n\n```{eval-rst}\n\n.. autodata:: hanlp.datasets.parsing.ctb8.CTB8_POS_TRAIN\n.. autodata:: hanlp.datasets.parsing.ctb8.CTB8_POS_DEV\n.. autodata:: hanlp.datasets.parsing.ctb8.CTB8_POS_TEST\n\n```\n\n## CTB9\n\n\n```{eval-rst}\n\n\n.. autodata:: hanlp.datasets.parsing.ctb9.CTB9_POS_TRAIN\n.. autodata:: hanlp.datasets.parsing.ctb9.CTB9_POS_DEV\n.. autodata:: hanlp.datasets.parsing.ctb9.CTB9_POS_TEST\n\n```"
  },
  {
    "path": "docs/api/hanlp/datasets/srl/conll2012_dataset.md",
    "content": "# conll2012_dataset\n\n```{eval-rst}\n\n.. autoclass:: hanlp.datasets.srl.loaders.conll2012.CoNLL2012SRLDataset\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/datasets/srl/index.md",
    "content": "# srl\n\nSemantic Role Labeling datasets.\n\n```{toctree}\nconll2012_dataset\nresources\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/datasets/srl/resources.md",
    "content": "# resources\n\n## OntoNotes 5\n\n### Chinese\n\n```{eval-rst}\n\n.. autodata:: hanlp.datasets.srl.ontonotes5.chinese.ONTONOTES5_CONLL12_CHINESE_TRAIN\n    :noindex:\n.. autodata:: hanlp.datasets.srl.ontonotes5.chinese.ONTONOTES5_CONLL12_CHINESE_DEV\n    :noindex:\n.. autodata:: hanlp.datasets.srl.ontonotes5.chinese.ONTONOTES5_CONLL12_CHINESE_TEST\n    :noindex:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/datasets/tok/index.md",
    "content": "# tok\n\nTokenization datasets.\n\n```{toctree}\ntxt\nmcws_dataset\nresources\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/datasets/tok/mcws_dataset.md",
    "content": "# mcws_dataset\n\n```{eval-rst}\n.. currentmodule:: hanlp.datasets.tokenization.loaders.multi_criteria_cws.mcws_dataset\n\n.. autoclass:: MultiCriteriaTextTokenizingDataset\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/datasets/tok/resources.md",
    "content": "# resources\n\n## sighan2005\n\n[The Second International Chinese Word Segmentation Bakeoff](http://sighan.cs.uchicago.edu/bakeoff2005/) took place over the summer of 2005.\n\n### pku\n\n```{eval-rst}\n\n.. automodule:: hanlp.datasets.tokenization.sighan2005.pku\n    :members:\n\n```\n\n### msr\n\n```{eval-rst}\n\n.. automodule:: hanlp.datasets.tokenization.sighan2005.msr\n    :members:\n\n```\n\n### as\n\n```{eval-rst}\n\n.. automodule:: hanlp.datasets.tokenization.sighan2005.as_\n    :members:\n\n```\n\n### cityu\n\n```{eval-rst}\n\n.. automodule:: hanlp.datasets.tokenization.sighan2005.cityu\n    :members:\n\n```\n\n## CTB6\n\n```{eval-rst}\n\n.. automodule:: hanlp.datasets.tokenization.ctb6\n    :members:\n\n```\n\n## CTB8\n\n\n```{eval-rst}\n\n.. automodule:: hanlp.datasets.parsing.ctb8\n\n.. autodata:: CTB8_CWS_TRAIN\n.. autodata:: CTB8_CWS_DEV\n.. autodata:: CTB8_CWS_TEST\n\n```\n\n## CTB9\n\n\n```{eval-rst}\n\n.. automodule:: hanlp.datasets.parsing.ctb9\n\n.. autodata:: CTB9_CWS_TRAIN\n.. autodata:: CTB9_CWS_DEV\n.. autodata:: CTB9_CWS_TEST\n\n```"
  },
  {
    "path": "docs/api/hanlp/datasets/tok/txt.md",
    "content": "# txt\n\n```{eval-rst}\n.. currentmodule:: hanlp.datasets.tokenization.loaders.txt\n\n.. autoclass:: TextTokenizingDataset\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/hanlp.rst",
    "content": ".. _api/main:\n\nhanlp\n==========\n\n.. currentmodule:: hanlp\n\n.. autofunction:: load\n\n.. autofunction:: pipeline"
  },
  {
    "path": "docs/api/hanlp/index.md",
    "content": "# hanlp\n\nCore APIs for `hanlp`.\n\n```{toctree}\nhanlp\ncommon/index\ncomponents/index\npretrained/index\ndatasets/index\nutils/index\nlayers/index\n```"
  },
  {
    "path": "docs/api/hanlp/layers/decoders/biaffine_ner.md",
    "content": "# biaffine_ner\n\n\n```{eval-rst}\n\n.. autoclass:: hanlp.components.ner.biaffine_ner.biaffine_ner_model.BiaffineNamedEntityRecognitionDecoder\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/layers/decoders/index.md",
    "content": "# decoders\n\n```{toctree}\nlinear_crf\nbiaffine_ner\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/layers/decoders/linear_crf.md",
    "content": "# linear_crf\n\n\n```{eval-rst}\n\n.. autoclass:: hanlp.components.mtl.tasks.pos.LinearCRFDecoder\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/layers/embeddings/char_cnn.md",
    "content": "# char_cnn\n\n\n```{eval-rst}\n\n.. autoclass:: hanlp.layers.embeddings.char_cnn.CharCNN\n\t:members:\n\n.. autoclass:: hanlp.layers.embeddings.char_cnn.CharCNNEmbedding\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/layers/embeddings/char_rnn.md",
    "content": "# char_rnn\n\n\n```{eval-rst}\n\n.. autoclass:: hanlp.layers.embeddings.char_rnn.CharRNN\n\t:members:\n\n.. autoclass:: hanlp.layers.embeddings.char_rnn.CharRNNEmbedding\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/layers/embeddings/embedding.md",
    "content": "# embedding\n\n\n```{eval-rst}\n\n.. autoclass:: hanlp.layers.embeddings.embedding.Embedding\n\t:members:\n\n.. autoclass:: hanlp.layers.embeddings.embedding.ConcatModuleList\n\t:members:\n\n.. autoclass:: hanlp.layers.embeddings.embedding.EmbeddingList\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/layers/embeddings/fasttext.md",
    "content": "# fasttext\n\n```{eval-rst}\n\n.. autoclass:: hanlp.layers.embeddings.fast_text.FastTextEmbedding\n\t:members:\n\n.. autoclass:: hanlp.layers.embeddings.fast_text.FastTextEmbeddingModule\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/layers/embeddings/index.md",
    "content": "# embeddings\n\n```{toctree}\nembedding\nword2vec\nfasttext\nchar_cnn\nchar_rnn\ntransformer\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/layers/embeddings/transformer.md",
    "content": "# transformer\n\n\n```{eval-rst}\n\n.. autoclass:: hanlp.layers.embeddings.contextual_word_embedding.ContextualWordEmbedding\n\t:members:\n\n.. autoclass:: hanlp.layers.embeddings.contextual_word_embedding.ContextualWordEmbeddingModule\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/layers/embeddings/word2vec.md",
    "content": "# word2vec\n\n```{eval-rst}\n\n.. autoclass:: hanlp.layers.embeddings.word2vec.Word2VecEmbedding\n\t:members:\n\n.. autoclass:: hanlp.layers.embeddings.word2vec.Word2VecEmbeddingModule\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/layers/index.md",
    "content": "# layers\n\n```{toctree}\nembeddings/index\ntransformers/index\ndecoders/index\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/layers/transformers/encoder.md",
    "content": "# encoder\n\n\n```{eval-rst}\n\n.. autoclass:: hanlp.layers.transformers.encoder.TransformerEncoder\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/layers/transformers/index.md",
    "content": "# transformers\n\n```{toctree}\nencoder\ntokenizer\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/layers/transformers/tokenizer.md",
    "content": "# tokenizer\n\n\n```{eval-rst}\n\n.. autoclass:: hanlp.transform.transformer_tokenizer.TransformerSequenceTokenizer\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/hanlp/pretrained/amr.md",
    "content": "---\njupytext:\n  formats: ipynb,md:myst\n  text_representation:\n    extension: .md\n    format_name: myst\n    format_version: '0.8'\n    jupytext_version: 1.4.2\nkernelspec:\n  display_name: Python 3\n  language: python\n  name: python3\n---\n# amr\n\nAMR captures “who is doing what to whom” in a sentence. Each sentence is represented as a rooted, directed, acyclic graph with labels on edges (relations) and leaves (concepts).\nBefore loading an AMR model, make sure to install HanLP with the `amr` dependencies:\n\n```shell\npip install hanlp[amr] -U\n```\n\nTo parse a raw sentence into AMR:\n\n```{eval-rst}\n.. margin:: Batching is Faster\n\n    .. Hint:: Parse multiple sentences at once for faster speed! \n```\n\n\n```{code-cell} ipython3\n:tags: [output_scroll]\nimport hanlp\n\namr_parser = hanlp.load(hanlp.pretrained.amr.AMR3_SEQ2SEQ_BART_LARGE)\namr = amr_parser('The boy wants the girl to believe him.')\nprint(amr)\n```\n\nAll the pre-trained parsers and their scores are listed below.\n\n```{eval-rst}\n\n.. automodule:: hanlp.pretrained.amr\n    :members:\n\n```"
  },
  {
    "path": "docs/api/hanlp/pretrained/amr2text.md",
    "content": "---\njupytext:\n  formats: ipynb,md:myst\n  text_representation:\n    extension: .md\n    format_name: myst\n    format_version: '0.8'\n    jupytext_version: 1.4.2\nkernelspec:\n  display_name: Python 3\n  language: python\n  name: python3\n---\n# amr2text\n\nAMR captures “who is doing what to whom” in a sentence. Each sentence is represented as a rooted, directed, acyclic graph with labels on edges (relations) and leaves (concepts).\nThe goal of AMR-to-Text Generation is to recover the original sentence realization given an AMR. This task can be seen as the reverse of the structured prediction found in AMR parsing.\nBefore loading an AMR model, make sure to install HanLP with the `amr` dependencies:\n\n```shell\npip install hanlp[amr] -U\n```\n\nTo generate a sentence given an AMR:\n\n```{eval-rst}\n.. margin:: Batching is Faster\n\n    .. Hint:: Generate multiple sentences at once for faster speed! \n```\n\n\n```{code-cell} ipython3\n:tags: [output_scroll]\nimport hanlp\n\ngeneration = hanlp.load(hanlp.pretrained.amr2text.AMR3_GRAPH_PRETRAIN_GENERATION)\nprint(generation('''\n(z0 / want-01\n    :ARG0 (z1 / boy)\n    :ARG1 (z2 / believe-01\n              :ARG0 (z3 / girl)\n              :ARG1 z1))\n'''))\n```\n\nAll the pre-trained parsers and their scores are listed below.\n\n```{eval-rst}\n\n.. automodule:: hanlp.pretrained.amr2text\n    :members:\n\n```"
  },
  {
    "path": "docs/api/hanlp/pretrained/constituency.md",
    "content": "---\njupytext:\n  formats: ipynb,md:myst\n  text_representation:\n    extension: .md\n    format_name: myst\n    format_version: '0.8'\n    jupytext_version: 1.4.2\nkernelspec:\n  display_name: Python 3\n  language: python\n  name: python3\n---\n\n# constituency\n\nConstituency Parsing is the process of analyzing the sentences by breaking down it into sub-phrases also known as constituents.\n\nTo parse a tokenized sentence into constituency tree, first load a parser:\n\n```{eval-rst}\n.. margin:: Batching is Faster\n\n    .. Hint:: To speed up, parse multiple sentences at once, and use a GPU.\n```\n\n```{code-cell} ipython3\n:tags: [output_scroll]\nimport hanlp\n\ncon = hanlp.load(hanlp.pretrained.constituency.CTB9_CON_FULL_TAG_ELECTRA_SMALL)\n```\n\nThen parse a sequence or multiple sequences of tokens to it. \n\n```{code-cell} ipython3\n:tags: [output_scroll]\ntree = con([\"2021年\", \"HanLPv2.1\", \"带来\", \"最\", \"先进\", \"的\", \"多\", \"语种\", \"NLP\", \"技术\", \"。\"])\n```\n\nThe constituency tree is a nested list of constituencies:\n\n```{code-cell} ipython3\n:tags: [output_scroll]\ntree\n```\n\nYou can `str` or `print` it to get its bracketed form:\n\n```{code-cell} ipython3\n:tags: [output_scroll]\nprint(tree)\n```\n\nAll the pre-trained parsers and their scores are listed below.\n\n```{eval-rst}\n\n.. automodule:: hanlp.pretrained.constituency\n    :members:\n\n```"
  },
  {
    "path": "docs/api/hanlp/pretrained/dep.md",
    "content": "# dep\n\n```{eval-rst}\n\n.. automodule:: hanlp.pretrained.dep\n    :members:\n\n```"
  },
  {
    "path": "docs/api/hanlp/pretrained/eos.md",
    "content": "# eos\n\n\n```{eval-rst}\n\n.. automodule:: hanlp.pretrained.eos\n    :members:\n\n```"
  },
  {
    "path": "docs/api/hanlp/pretrained/fasttext.md",
    "content": "# fasttext\n\n```{eval-rst}\n\n.. automodule:: hanlp.pretrained.fasttext\n    :members:\n\n```"
  },
  {
    "path": "docs/api/hanlp/pretrained/glove.md",
    "content": "# glove\n\n```{eval-rst}\n\n.. automodule:: hanlp.pretrained.glove\n    :members:\n\n```"
  },
  {
    "path": "docs/api/hanlp/pretrained/index.md",
    "content": "# pretrained\n\n```{eval-rst}\nNLP components grouped by tasks. For each task, we provide at least one :class:`~hanlp.common.component.Component` \ncompatible class and several pretrained models. Each of them is stored in a Python constant which can be fetched \nusing :meth:`hanlp.load`.  \n``` \n\n```{toctree}\nmtl\neos\ntok\npos\nner\ndep\nconstituency\nsrl\nsdp\namr\namr2text\nsts\nword2vec\nglove\nfasttext\nmlm\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/pretrained/mlm.md",
    "content": "---\njupytext:\n  formats: ipynb,md:myst\n  text_representation:\n    extension: .md\n    format_name: myst\n    format_version: '0.8'\n    jupytext_version: 1.4.2\nkernelspec:\n  display_name: Python 3\n  language: python\n  name: python3\n---\n\n# mlm\n\nMasked Language Model (MLM) predicts words that were originally hidden intentionally in a sentence.\nTo perform such prediction, first load a pre-trained MLM (e.g., `bert-base-chinese`):\n\n````{margin} Batching is Faster\n```{hint}\nPredict multiple sentences in batch mode for faster speed! \n```\n````\n\n````{margin} Multilingual Support\n```{note}\nHanLP always support multilingual. Feel free to use a multilingual model listed [here](https://huggingface.co/models?pipeline_tag=fill-mask&sort=downloads).\n```\n````\n\n```{code-cell} ipython3\n:tags: [output_scroll]\nfrom hanlp.components.lm.mlm import MaskedLanguageModel\nmlm = MaskedLanguageModel()\nmlm.load('bert-base-chinese')\n```\n\nRepresent blanks (masked tokens) with `[MASK]` and let MLM fills them:\n\n```{code-cell} ipython3\n:tags: [output_scroll]\nmlm('生活的真谛是[MASK]。')\n```\n\nBatching is always faster:\n\n```{code-cell} ipython3\n:tags: [output_scroll]\nmlm(['生活的真谛是[MASK]。', '巴黎是[MASK][MASK]的首都。'])\n```\n\n\nAll the pre-trained MLM models and their details are listed in the [docs](https://huggingface.co/models?pipeline_tag=fill-mask&sort=downloads) of Hugging Face 🤗 Transformers."
  },
  {
    "path": "docs/api/hanlp/pretrained/mtl.md",
    "content": "# mtl\n\n```{eval-rst}\n\n.. automodule:: hanlp.pretrained.mtl\n    :members:\n\n```"
  },
  {
    "path": "docs/api/hanlp/pretrained/ner.md",
    "content": "# ner\n\n```{eval-rst}\n\n.. automodule:: hanlp.pretrained.ner\n    :members:\n\n```"
  },
  {
    "path": "docs/api/hanlp/pretrained/pos.md",
    "content": "---\njupytext:\n  formats: ipynb,md:myst\n  text_representation:\n    extension: .md\n    format_name: myst\n    format_version: '0.8'\n    jupytext_version: 1.4.2\nkernelspec:\n  display_name: Python 3\n  language: python\n  name: python3\n---\n\n# pos\n\nThe process of classifying words into their **parts of speech** and labeling them accordingly is known as **part-of-speech tagging**, **POS-tagging**, or simply **tagging**. \n\nTo tag a tokenized sentence:\n\n````{margin} Batching is Faster\n```{hint}\nTag multiple sentences at once for faster speed! \n```\n````\n\n\n```{code-cell} ipython3\n:tags: [output_scroll]\nimport hanlp\n\npos = hanlp.load(hanlp.pretrained.pos.CTB9_POS_ELECTRA_SMALL)\npos(['我', '的', '希望', '是', '希望', '世界', '和平'])\n```\n\n````{margin} Custom Dictionary Supported\n```{seealso}\nSee [this tutorial](https://github.com/hankcs/HanLP/blob/master/plugins/hanlp_demo/hanlp_demo/zh/demo_pos_dict.py) for custom dictionary.\n```\n````\n\nAll the pre-trained taggers and their details are listed below.\n\n```{eval-rst}\n\n.. automodule:: hanlp.pretrained.pos\n    :members:\n\n```"
  },
  {
    "path": "docs/api/hanlp/pretrained/sdp.md",
    "content": "# sdp\n\n```{eval-rst}\n\n.. automodule:: hanlp.pretrained.sdp\n    :members:\n\n```"
  },
  {
    "path": "docs/api/hanlp/pretrained/srl.md",
    "content": "---\njupytext:\n  formats: ipynb,md:myst\n  text_representation:\n    extension: .md\n    format_name: myst\n    format_version: '0.8'\n    jupytext_version: 1.4.2\nkernelspec:\n  display_name: Python 3\n  language: python\n  name: python3\n---\n\n# srl\n\nSemantic Role Labeling (SRL) is one shallow semantic parsing that produces predicate-argument structures which are semantic roles (or participants) such as agent, patient, and theme associated with verbs.\n\nInputs to SRL are tokenized sentences:\n\n````{margin} Batching is Faster\n```{hint}\nFeed in multiple sentences at once for faster speed! \n```\n````\n\n\n```{code-cell} ipython3\n:tags: [output_scroll]\nimport hanlp\n\nsrl = hanlp.load(hanlp.pretrained.srl.CPB3_SRL_ELECTRA_SMALL)\nsrl(['男孩', '希望', '女孩', '相信', '他', '。'])\n```\n\nAll the pre-trained labelers and their details are listed below.\n\n```{eval-rst}\n\n.. automodule:: hanlp.pretrained.srl\n    :members:\n\n```"
  },
  {
    "path": "docs/api/hanlp/pretrained/sts.md",
    "content": "---\njupytext:\n  formats: ipynb,md:myst\n  text_representation:\n    extension: .md\n    format_name: myst\n    format_version: '0.8'\n    jupytext_version: 1.4.2\nkernelspec:\n  display_name: Python 3\n  language: python\n  name: python3\n---\n\n# sts\n\n`sts` package holds pre-trained Semantic Textual Similarity (STS) models. We surveyed both supervised and unsupervised\nmodels and we believe that unsupervised models are still immature at this moment. Unsupervised STS is good for IR but \nnot NLP especially on sentences with little lexical overlap.\n \n\n```{eval-rst}\n\n.. automodule:: hanlp.pretrained.sts\n    :members:\n\n```\n\n```{code-cell} ipython3\nimport hanlp\n\nsim = hanlp.load(hanlp.pretrained.sts.STS_ELECTRA_BASE_ZH)\nsim([\n    ['看图猜一电影名', '看图猜电影'],\n    ['无线路由器怎么无线上网', '无线上网卡和无线路由器怎么用'],\n    ['北京到上海的动车票', '上海到北京的动车票'],\n])\n```"
  },
  {
    "path": "docs/api/hanlp/pretrained/tok.md",
    "content": "---\njupytext:\n  formats: ipynb,md:myst\n  text_representation:\n    extension: .md\n    format_name: myst\n    format_version: '0.8'\n    jupytext_version: 1.4.2\nkernelspec:\n  display_name: Python 3\n  language: python\n  name: python3\n---\n\n# tok\n\nTokenization is a way of separating a sentence into smaller units called tokens. In lexical analysis, tokens usually refer to words.\n\n````{margin} Batching is Faster\n```{hint}\nTokenize multiple sentences at once for faster speed! \n```\n````\n````{margin} Custom Dictionary Supported\n```{seealso}\nSee [this tutorial](https://github.com/hankcs/HanLP/blob/master/plugins/hanlp_demo/hanlp_demo/zh/demo_custom_dict.py) for custom dictionary.\n```\n````\n\nTo tokenize raw sentences:\n\n\n```{code-cell} ipython3\n:tags: [output_scroll]\nimport hanlp\n\ntok = hanlp.load(hanlp.pretrained.tok.COARSE_ELECTRA_SMALL_ZH)\ntok(['商品和服务。', '晓美焰来到北京立方庭参观自然语义科技公司'])\n```\n\nAll the pre-trained tokenizers and their details are listed below.\n\n\n```{eval-rst}\n\n.. automodule:: hanlp.pretrained.tok\n    :members:\n\n```\n\n"
  },
  {
    "path": "docs/api/hanlp/pretrained/word2vec.md",
    "content": "---\njupytext:\n  formats: ipynb,md:myst\n  text_representation:\n    extension: .md\n    format_name: myst\n    format_version: '0.8'\n    jupytext_version: 1.4.2\nkernelspec:\n  display_name: Python 3\n  language: python\n  name: python3\n---\n\n# word2vec\n\nWord2Vec is a family of model architectures and optimizations that can be used to learn word embeddings from large unlabeled datasets. In this document, it is narrowly  defined as a component to map discrete words to distributed representations which are dense vectors.\n\nTo perform such mapping:\n\n````{margin} Batching is Faster\n```{hint}\nMap multiple tokens in batch mode for faster speed! \n```\n````\n\n````{margin} Multilingual Support\n```{note}\nHanLP always support multilingual. Feel free to use a multilingual model listed [here](http://vectors.nlpl.eu/repository/).\n```\n````\n\n```{code-cell} ipython3\n:tags: [output_scroll]\nimport hanlp\nword2vec = hanlp.load(hanlp.pretrained.word2vec.CONVSEG_W2V_NEWS_TENSITE_WORD_PKU)\nword2vec('先进')\n```\n\nThese vectors have already been normalized to facilitate similarity computation:\n\n```{code-cell} ipython3\n:tags: [output_scroll]\nimport torch\nprint(torch.nn.functional.cosine_similarity(word2vec('先进'), word2vec('优秀'), dim=0))\nprint(torch.nn.functional.cosine_similarity(word2vec('先进'), word2vec('水果'), dim=0))\n```\n\nUsing these similarity scores, the most similar words can be found:\n\n```{code-cell} ipython3\n:tags: [output_scroll]\nword2vec.most_similar('上海')\n```\n\nWord2Vec usually can not process OOV or phrases:\n\n```{code-cell} ipython3\n:tags: [output_scroll]\n\nword2vec.most_similar('非常寒冷') # phrases are usually OOV\n```\n\nDoc2Vec, as opposite to Word2Vec model, can create a vectorised representation by averaging a group of words. To enable Doc2Vec for OOV and phrases, pass `doc2vec=True`:\n\n```{code-cell} ipython3\n:tags: [output_scroll]\n\nword2vec.most_similar('非常寒冷', doc2vec=True)\n```\n\nAll the pre-trained word2vec models and their details are listed below.\n\n```{eval-rst}\n\n.. automodule:: hanlp.pretrained.word2vec\n    :members:\n\n```"
  },
  {
    "path": "docs/api/hanlp/utils/index.md",
    "content": "# utils\n\nUtilities.\n\n```{toctree}\nio_util\n```\n"
  },
  {
    "path": "docs/api/hanlp/utils/io_util.md",
    "content": "# io_util\n\n```{eval-rst}\n\n.. currentmodule:: hanlp.utils\n\n.. automodule:: hanlp.utils.io_util\n\t:members:\n\n```\n"
  },
  {
    "path": "docs/api/restful.rst",
    "content": ".. _api/hanlp_restful:\n\nhanlp_restful\n====================\n\n.. currentmodule:: hanlp_restful\n\n.. autoclass:: HanLPClient\n\t:members:\n\t:special-members:\n\t:exclude-members: __init__, __repr__, __weakref__"
  },
  {
    "path": "docs/api/restful_golang.md",
    "content": "# Golang RESTful API\n\n## Install\n\n```shell script\ngo get -u github.com/hankcs/gohanlp@main\n```\n\n## Quick Start \n\nObtain an `auth` from any compatible service provider like our [free service](https://bbs.hankcs.com/t/apply-for-free-hanlp-restful-apis/3178), then initiate a `HanLPClient` and call its `Parse` interface.\n\n```java\npackage main\n\nimport (\n\t\"fmt\"\n\t\"github.com/hankcs/gohanlp/hanlp\"\n)\n\nfunc main() {\n    client := hanlp.HanLPClient(hanlp.WithAuth(\"The auth you applied for\")) // anonymous users can skip auth\n    s, _ := client.Parse(\"In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques to production environments.\",hanlp.WithLanguage(\"mul\"))\n    fmt.Println(s)\n}\n```\n\nRefer to our [testcases](https://github.com/hankcs/gohanlp/blob/main/main_test.go) and [data format](../data_format) for more details.\n\n"
  },
  {
    "path": "docs/api/restful_java.md",
    "content": "# Java RESTful API\n\nAdd the following dependency into the `pom.xml` file of your project. \n\n```xml\n<dependency>\n  <groupId>com.hankcs.hanlp.restful</groupId>\n  <artifactId>hanlp-restful</artifactId>\n  <version>0.0.15</version>\n</dependency>\n```\n\nObtain an `auth` from any compatible service provider like our [free service](https://bbs.hankcs.com/t/apply-for-free-hanlp-restful-apis/3178), then initiate a `HanLPClient` and call its `parse` interface.\n\n```java\nHanLPClient client = new HanLPClient(\"https://hanlp.hankcs.com/api\", null); // Replace null with your auth\nSystem.out.println(client.parse(\"2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。晓美焰来到北京立方庭参观自然语义科技公司。\"));\n```\n\nRefer to our [testcases](https://github.com/hankcs/HanLP/blob/master/plugins/hanlp_restful_java/src/test/java/com/hankcs/hanlp/restful/HanLPClientTest.java) and [data format](../data_format) for more details.\n\n"
  },
  {
    "path": "docs/api/trie/dictionary.md",
    "content": "# dictionary\n\n```{eval-rst}\n.. currentmodule:: hanlp_trie\n\n.. autoclass:: hanlp_trie.dictionary.DictInterface\n\t:members:\n\n.. autoclass:: hanlp_trie.dictionary.TrieDict\n\t:members:\n```\n"
  },
  {
    "path": "docs/api/trie/index.md",
    "content": "# hanlp_trie\n\nHanLP trie/dictionary interface and referential implementation.\n\n```{toctree}\ntrie\ndictionary\n```\n\n"
  },
  {
    "path": "docs/api/trie/trie.md",
    "content": "# trie\n\n```{eval-rst}\n.. currentmodule:: hanlp_trie\n\n.. autoclass:: hanlp_trie.trie.Node\n\t:members:\n\n.. autoclass:: hanlp_trie.trie.Trie\n\t:members:\n```\n"
  },
  {
    "path": "docs/conf.py",
    "content": "# -- Project information -----------------------------------------------------\nimport sys\nimport os\nfrom datetime import datetime\n\nsys.path.append(os.path.abspath('..'))\nsys.path.append(os.path.abspath('../plugins/hanlp_common'))\nsys.path.append(os.path.abspath('../plugins/hanlp_trie'))\nsys.path.append(os.path.abspath('../plugins/hanlp_restful'))\nimport hanlp\n\nproject = 'HanLP'\ncopyright = f'2020-{datetime.now().year}, hankcs'\nauthor = 'hankcs'\n\n# The short X.Y version.\nversion = hanlp.__version__\n# The full version, including alpha/beta/rc tags.\nrelease = hanlp.__version__\n\n# The language for content autogenerated by Sphinx. Refer to documentation\n# for a list of supported languages.\nlanguage = 'en'\n\nmaster_doc = \"index\"\n\n# -- General configuration ---------------------------------------------------\n\n# Add any Sphinx extension module names here, as strings. They can be\n# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom\n# ones.\nextensions = [\n    \"myst_nb\",\n    \"sphinx_copybutton\",\n    \"sphinx_togglebutton\",\n    \"sphinxcontrib.bibtex\",\n    'sphinx_astrorefs',  # astrophysics style, similar to ACL\n    \"sphinx_thebe\",\n    \"sphinx.ext.autodoc\",\n    \"sphinx.ext.intersphinx\",\n    \"sphinx.ext.viewcode\",\n    \"ablog\",\n    'sphinx.ext.napoleon',\n]\n\n# Add any paths that contain templates here, relative to this directory.\ntemplates_path = [\"_templates\"]\n\n# List of patterns, relative to source directory, that match files and\n# directories to ignore when looking for source files.\n# This pattern also affects html_static_path and html_extra_path.\nexclude_patterns = [\"_build\", \"Thumbs.db\", \".DS_Store\"]\n\nintersphinx_mapping = {\n    \"python\": (\"https://docs.python.org/3.8\", None),\n    \"sphinx\": (\"https://www.sphinx-doc.org/en/3.x\", None),\n}\nnitpick_ignore = [\n    (\"py:class\", \"docutils.nodes.document\"),\n    (\"py:class\", \"docutils.parsers.rst.directives.body.Sidebar\"),\n]\nautoclass_content = 'both'\n\nnumfig = True\n\nmyst_admonition_enable = True\nmyst_deflist_enable = True\nmyst_url_schemes = (\"http\", \"https\", \"mailto\")\npanels_add_bootstrap_css = False\n\n# -- Options for HTML output -------------------------------------------------\n\n# The theme to use for HTML and HTML Help pages.  See the documentation for\n# a list of builtin themes.\n#\nhtml_theme = \"sphinx_book_theme\"\nhtml_title = \"HanLP Documentation\"\nhtml_logo = \"_static/logo.png\"\nhtml_favicon = \"_static/favicon.png\"\nhtml_copy_source = True\nhtml_sourcelink_suffix = \"\"\n\nhtml_sidebars = {\n    # \"reference/blog/*\": [\n    #     \"sidebar-search-bs.html\",\n    #     \"postcard.html\",\n    #     \"recentposts.html\",\n    #     \"tagcloud.html\",\n    #     \"categories.html\",\n    #     \"archives.html\",\n    #     \"sbt-sidebar-nav.html\",\n    #     \"sbt-sidebar-footer.html\",\n    # ]\n}\n# Add any paths that contain custom static files (such as style sheets) here,\n# relative to this directory. They are copied after the builtin static files,\n# so a file named \"default.css\" will overwrite the builtin \"default.css\".\nhtml_static_path = [\"_static\"]\njupyter_execute_notebooks = \"cache\"\nthebe_config = {\n    \"repository_url\": \"https://github.com/binder-examples/jupyter-stacks-datascience\",\n    \"repository_branch\": \"master\",\n}\n\nhtml_theme_options = {\n    \"theme_dev_mode\": False,\n    \"path_to_docs\": \"docs\",\n    \"repository_url\": \"https://github.com/hankcs/HanLP\",\n    # \"repository_branch\": \"gh-pages\",  # For testing\n    # \"launch_buttons\": {\n    #     # \"binderhub_url\": \"https://mybinder.org\",\n    #     # \"jupyterhub_url\": \"https://datahub.berkeley.edu\",  # For testing\n    #     \"colab_url\": \"https://colab.research.google.com/\",\n    #     \"notebook_interface\": \"jupyterlab\",\n    #     \"thebe\": True,\n    # },\n    \"use_edit_page_button\": True,\n    \"use_issues_button\": True,\n    \"use_repository_button\": True,\n    \"use_download_button\": True,\n    # For testing\n    # \"home_page_in_toc\": True,\n    # \"single_page\": True,\n    # \"extra_footer\": \"<a href='https://google.com'>Test</a>\",  # DEPRECATED KEY\n    # \"extra_navbar\": \"<a href='https://google.com'>Test</a>\",\n}\nhtml_baseurl = \"https://hanlp.hankcs.com/docs/\"\n\n# -- ABlog config -------------------------------------------------\nblog_path = \"reference/blog\"\nblog_post_pattern = \"reference/blog/*.md\"\nblog_baseurl = \"https://hanlp.hankcs.com/docs/\"\nfontawesome_included = True\npost_auto_image = 1\npost_auto_excerpt = 2\nexecution_show_tb = \"READTHEDOCS\" in os.environ\n\n# Localization\nnb_render_priority = {\n    \"gettext\": (\n        \"application/vnd.jupyter.widget-view+json\",\n        \"application/javascript\",\n        \"text/html\",\n        \"image/svg+xml\",\n        \"image/png\",\n        \"image/jpeg\",\n        \"text/markdown\",\n        \"text/latex\",\n        \"text/plain\",\n    )\n}\n\nlocale_dirs = ['locale/']\n\n# bibtex\nbibtex_default_style = 'unsrtalpha'\n"
  },
  {
    "path": "docs/configure.md",
    "content": "# Configuration\n\n## Customize ``HANLP_HOME``\n\nAll resources HanLP use will be cached into a directory called `HANLP_HOME`. \nIt is an environment variable which you can customize to any path you like. \nBy default, `HANLP_HOME` resolves to `~/.hanlp` and `%appdata%\\hanlp` on *nix and Windows respectively. \nIf you want to redirect `HANLP_HOME` to a different location, say `/data/hanlp`, the following shell command can be very helpful.\n\n```bash\nexport HANLP_HOME=/data/hanlp\n```\n\n## Use GPUs\n\nBy default, HanLP tries to use the least occupied GPU so that mostly you don't need to worry about it, HanLP makes the best choice for you. This behavior is very useful when you're using a public server shared across your lab or company with your colleagues. \n\nHanLP also honors the ``CUDA_VISIBLE_DEVICES`` used by PyTorch and TensorFlow to limit which devices HanLP can choose from. For example, the following command will only keep the `0`th and `1`st GPUs.\n\n```bash\nexport CUDA_VISIBLE_DEVICES=0,1\n```\n\n```{eval-rst}\nIf you need fine grained control over each component, ``hanlp.load(..., devices=...)`` is what you're looking for.\nSee documents for :meth:`hanlp.load`.\n```\n\n### External Resources\n\nFor deep learning beginners, you might need to learn how to set up a working GPU environment first. Here are some \nresources.\n\n- [CUDA Toolkit](https://developer.nvidia.com/cuda-toolkit)\n    - It's a good practice to install the driver shipped with a CUDA package. \n- [PyTorch](https://pytorch.org/get-started/locally/)\n    - If no existing PyTorch found, `pip install hanlp` will have the CPU-only PyTorch installed, which is universal and assumes no GPU or CUDA dependencies. \n    - You will need to install a GPU-enabled PyTorch according to your CUDA and OS versions.\n- Cloud servers\n    - There are many cloud services providing out-of-the-box deep learning images. HanLP works fine on these platforms. \n        They could save your time and efforts.\n- Google Colab\n    - Colab allows you to write excutable notebooks with full GPU support. PyTorch and TensorFlow have been pre-installed and configured to the best state.\n    - In fact, you can click [![Open In Colab](https://file.hankcs.com/img/colab-badge.svg)](https://colab.research.google.com/drive/1KPX6t1y36TOzRIeB4Kt3uJ1twuj6WuFv?usp=sharing) to play with the GPU-enabled HanLP tutorial right now.\n\n\n## Use Mirror Sites\n\nBy default, models are downloaded from a global CDN we maintain. However, in some regions the downloading speed can \nbe slow occasionally. If you happen to be in one of those regions, you can find some third party mirror sites \non our [bbs](https://bbs.hankcs.com/). When you find a working URL, say \n[https://ftp.hankcs.com/hanlp/](https://ftp.hankcs.com/hanlp/), you can set a `HANLP_URL` \nenvironment variable and HanLP will pick it up at the next startup.\n\n```bash\nexport HANLP_URL=https://ftp.hankcs.com/hanlp/\n```\n\n## Control Verbosity\n\nBy default, HanLP will print progressive message to the console when you load a model. If you want to silence it, use the \nfollowing environment variable.\n\n```bash\nexport HANLP_VERBOSE=0\n```\n"
  },
  {
    "path": "docs/contributing.md",
    "content": "# Contributing Guide\n\nThank you for being interested in contributing to `HanLP`! You\nare awesome ✨.\n\nThis guideline contains information about our conventions around coding style, pull request workflow, commit messages and more.\n\nThis page also contains information to help you get started with development on this\nproject.\n\n## Development\n\n### Set-up\n\nGet the source code of this project using git:\n\n```bash\ngit clone https://github.com/hankcs/HanLP --branch master\ncd HanLP\npip install -e plugins/hanlp_trie\npip install -e plugins/hanlp_common\npip install -e plugins/hanlp_restful\npip install -e .\n```\n\nTo work on this project, you need Python 3.6 or newer.\n\n### Running Tests\n\nThis project has a test suite to ensure certain important APIs work properly. The tests can be run using:\n\n```bash\npython -m unittest discover ./tests\n```\n\n```{tip}\nIt's hard to cover every API especially those of deep learning models, due to the limited computation resource of CI. However, we suggest all inference APIs to be tested at least.\n```\n\n## Repository Structure\n\nThis repository is a split into a few critical folders:\n\nhanlp/\n: The HanLP core package, containing the Python code.\n\nplugins/\n: Contains codes shared across several individual packages or non core APIs.\n\ndocs/\n: The documentation for HanLP, which is in markdown format mostly.\n: The build configuration is contained in `conf.py`.\n\ntests/\n: Testing infrastructure that uses `unittest` to ensure the output of API is what we expect it to be.\n\n.github/\n: Contains Continuous-integration (CI) workflows, run on commits/PRs to the GitHub repository.\n\n"
  },
  {
    "path": "docs/data_format.md",
    "content": "---\njupytext:\n  formats: ipynb,md:myst\n  text_representation:\n    extension: .md\n    format_name: myst\n    format_version: '0.8'\n    jupytext_version: 1.4.2\nkernelspec:\n  display_name: Python 3\n  language: python\n  name: python3\n---\n\n# Data Format\n\n\n## Input Format\n\n### RESTful Input\n\n#### Definition\n\nTo make a RESTful call, one needs to send a `json` HTTP POST request to the server, which contains at least a `text` \nfield or a `tokens` field. The input to RESTful API is very flexible. It can be one of the following 3 formats:\n\n1. It can be a document of raw `str` filled into `text`. The server will split it into sentences.\n1. It can be a `list` of sentences, each sentence is a raw `str`, filled into `text`.\n1. It can be a `list` of tokenized sentences, each sentence is a list of `str` typed tokens, filled into `tokens`.\n\n```{eval-rst}\nAdditionally, fine-grained controls are performed with the arguments defined in \n:meth:`hanlp_restful.HanLPClient.parse`.\n```\n\n\n#### Examples\n\n```shell script\ncurl -X 'POST' \\\n  'https://hanlp.hankcs.com/api/parse' \\\n  -H 'accept: application/json' \\\n  -H 'Content-Type: application/json' \\\n  -d '{\n  \"language\": \"zh\",\n  \"text\": \"HanLP为生产环境带来次世代最先进的多语种NLP技术。晓美焰来到北京参观自然语义科技公司。\"\n}'\n```\n\n### Model Input\n\n````{margin} **How about training inputs?**\n```{seealso}\nWe mostly follow the conventional file format of each NLP task instead of re-inventing them. Thus, we use `.tsv` for tagging and \n`.conllu` for parsing etc. For more details, refer to [datasets](https://hanlp.hankcs.com/docs/api/hanlp/datasets/index.html).   \n```\n````\n\nThe input format to models is specified per model and per task. Generally speaking, if a model has no tokenizer built in, then its input is\na sentence in `list[str]` form (a list of tokens), or multiple such sentences nested in a `list`.\n\nIf a model has a tokenizer built in, each sentence is in `str` form. \nAdditionally, you can use `skip_tasks='tok*'` to ask the model to use your tokenized inputs instead of tokenizing \nthem, in which case, each of your sentence needs to be in `list[str]` form, as if there was no tokenizer.\n\n```{eval-rst}\nFor any model, its input is of sentence level, which means you have to split a document into sentences beforehand. \nYou may want to try :class:`~hanlp.components.eos.ngram.NgramSentenceBoundaryDetector` for sentence splitting.\n```\n\n## Output Format\n\n\n```{eval-rst}\nThe outputs of both :class:`~hanlp_restful.HanLPClient` and \n:class:`~hanlp.components.mtl.multi_task_learning.MultiTaskLearning` are unified as the same \n:class:`~hanlp_common.document.Document` format.\n```\n\nFor example, the following RESTful codes will output such an instance.\n\n```{code-cell} ipython3\n:tags: [output_scroll]\nfrom hanlp_restful import HanLPClient\nHanLP = HanLPClient('https://hanlp.hankcs.com/api', auth=None)  # Fill in your auth\nprint(HanLP('2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。晓美焰来到北京立方庭参观自然语义科技公司。'))\n```\n\nThe outputs above is represented as a `json` dictionary where each key is a task name and its value is \nthe output of the corresponding task.\nFor each output, if it's a nested `list` then it contains multiple sentences otherwise it's just one single sentence.\n\nWe make the following naming convention of NLP tasks, each consists of 3 letters.\n\n````{margin} **How about annotations?**\n```{seealso}\nEach NLP task can exploit multiple datasets with their annotations, see our [annotations](annotations/index) for details.\n```\n````\n\n### Naming Convention \n\n| key  | Task                                                         | Chinese      |\n| ---- | ------------------------------------------------------------ | ------------ |\n| tok  | Tokenization. Each element is a token.                       | 分词         |\n| pos  | Part-of-Speech Tagging. Each element is a tag.               | 词性标注     |\n| lem  | Lemmatization. Each element is a lemma.                      | 词干提取     |\n| fea  | Features of Universal Dependencies. Each element is a feature. | 词法语法特征 |\n| ner  | Named Entity Recognition. Each element is a tuple of `(entity, type, begin, end)`, where `end`s are exclusive offsets. | 命名实体识别 |\n| dep  | Dependency Parsing. Each element is a tuple of `(head, relation)` where `head` starts with index `0` (which is `ROOT`). | 依存句法分析 |\n| con  | Constituency Parsing. Each list is a bracketed constituent.  | 短语成分分析 |\n| srl  | Semantic Role Labeling. Similar to `ner`, each element is a tuple of `(arg/pred, label, begin, end)`, where the predicate is labeled as `PRED`. | 语义角色标注 |\n| sdp  | Semantic Dependency Parsing. Similar to `dep`, however each token can have any number (including zero) of heads and corresponding relations. | 语义依存分析 |\n| amr  | Abstract Meaning Representation. Each AMR graph is represented as list of logical triples. See [AMR guidelines](https://github.com/amrisi/amr-guidelines/blob/master/amr.md#example). | 抽象意义表示 |\n\nWhen there are multiple models performing the same task, their keys are appended with a secondary identifier. \nFor example, `tok/fine` and `tok/corase` means a fine-grained tokenization model and a coarse-grained one respectively."
  },
  {
    "path": "docs/index.md",
    "content": "# HanLP: Han Language Processing\n\n[![GitHub stars](https://img.shields.io/github/stars/hankcs/HanLP)](https://github.com/hankcs/HanLP/stargazers) [![GitHub forks](https://img.shields.io/github/forks/hankcs/HanLP)](https://github.com/hankcs/HanLP/network) ![pypi](https://img.shields.io/pypi/v/HanLP) [![Downloads](https://static.pepy.tech/badge/HanLP)](https://pepy.tech/project/HanLP) [![GitHub license](https://img.shields.io/github/license/hankcs/HanLP)](https://github.com/hankcs/HanLP/blob/master/LICENSE) [![Open In Colab](https://file.hankcs.com/img/colab-badge.svg)](https://colab.research.google.com/drive/1KPX6t1y36TOzRIeB4Kt3uJ1twuj6WuFv?usp=sharing)\n\nThe multilingual NLP library for researchers and companies, built on PyTorch and TensorFlow 2.x, for advancing \nstate-of-the-art deep learning techniques in both academia and industry. HanLP was designed from day one to be \nefficient, user friendly and extendable. It comes with pretrained models for various human languages \nincluding English, Chinese, Japanese and many others.\n\n\n\n## Tutorials\n\n```{toctree}\n:maxdepth: 1\n:caption: Introduction\n\ntutorial\ninstall\nconfigure\ndata_format\nannotations/index\ncontributing\nLive Demo <https://hanlp.hankcs.com/>\n```\n\n## Python API\n\n```{toctree}\n:caption: Python API\n:maxdepth: 2\n\napi/hanlp/index\napi/common/index\napi/restful\napi/trie/index\n```\n\n## Java API\n\n```{toctree}\n:maxdepth: 1\n:caption: Java API\n\n1.x API <https://github.com/hankcs/HanLP/tree/1.x>\napi/restful_java\n```\n\n## Golang API\n\n```{toctree}\n:maxdepth: 1\n:caption: Golang API\n\napi/restful_golang\n```\n\n## References\n\n```{toctree}\n:caption: References\n:maxdepth: 2\n\nreferences\n```\n\n\n## Acknowledgements\n\nHanLPv2.1 is heavily inspired by [AllenNLP](https://allennlp.org/) and [SuPar](https://pypi.org/project/supar/). \n\n[pypi-badge]: https://img.shields.io/pypi/v/hanlp.svg\n[pypi-link]: https://pypi.org/project/hanlp\n\n"
  },
  {
    "path": "docs/install.md",
    "content": "# Install\n\n```{figure} _static/install-versions.svg\n---\nwidth: 100%\nfigclass: caption\nalt: HanLP versions\nname: hanlp-versions\n---\nChoose your HanLP version\n```\n\n## Install RESTful Packages\n\n[![Downloads](https://static.pepy.tech/badge/hanlp-restful)](https://pepy.tech/project/hanlp-restful) [![Downloads](https://static.pepy.tech/badge/hanlp-restful/month)](https://pepy.tech/project/hanlp-restful) [![Downloads](https://static.pepy.tech/badge/hanlp-restful/week)](https://pepy.tech/project/hanlp-restful) \n\n```{eval-rst}\n.. margin:: **Beginners Attention**\n\n    .. Hint:: New to NLP? Just install RESTful packages and call :meth:`~hanlp_restful.HanLPClient.parse` without pain.\n```\n\nFor beginners, the recommended RESTful packages are easier to start with. \nThe only requirement is [an auth key](https://bbs.hankcs.com/t/apply-for-free-hanlp-restful-apis/3178). \nWe officially released the following language bindings:\n\n### Python\n\n```shell script\npip install hanlp_restful\n```\n\n### Java\n\nSee [Java instructions](https://hanlp.hankcs.com/docs/api/restful_java.html).\n\n### Golang\n\nSee [Golang instructions](https://hanlp.hankcs.com/docs/api/restful_golang.html).\n\n## Install Native Package\n\n[![Downloads](https://static.pepy.tech/badge/hanlp)](https://pepy.tech/project/hanlp) [![Downloads](https://static.pepy.tech/badge/hanlp/month)](https://pepy.tech/project/hanlp) [![Downloads](https://static.pepy.tech/badge/hanlp/week)](https://pepy.tech/project/hanlp)  \n\nThe native package running locally can be installed via pip.\n\n````{margin} **Install from Source**\n```{note}\nSee [developer guideline](https://hanlp.hankcs.com/docs/contributing.html#development).\n```\n````\n\n```\npip install hanlp\n```\n\nHanLP requires Python 3.6 or later. GPU/TPU is suggested but not mandatory. Depending on your preference, HanLP offers the following flavors:\n\n````{margin} **Windows Support**\n```{note}\nInstallation on Windows is **perfectly** supported. No need to install Microsoft Visual C++ Build Tools anymore. \n```\n````\n\n````{margin} **Apple Silicon**\n```{note}\nHanLP also perfectly supports accelerating on Apple Silicon M1 chips, see [tutorial](https://www.hankcs.com/nlp/hanlp-official-m1-support.html).\n```\n````\n\n| Flavor  | Description                                                  |\n| ------- | ------------------------------------------------------------ |\n| default | This installs the default version which delivers the most commonly used functionalities. However, some heavy dependencies like TensorFlow are not installed. |\n| tf      | This installs TensorFlow and fastText.                       |\n| amr     | To support Abstract Meaning Representation (AMR) models, this installs AMR related dependencies like `penman`. |\n| full    | For experts who seek to maximize the efficiency via TensorFlow and C++ extensions, `pip install hanlp[full]` installs all the above dependencies. |\n\n\n## Install Models\n\nIn short, you don't need to manually install any model. Instead, they are automatically downloaded to a directory called [`HANLP_HOME`](https://hanlp.hankcs.com/docs/configure.html#customize-hanlp-home) when you call `hanlp.load`.\nOccasionally, some errors might occur the first time you load a model, in which case you can refer to the following tips.\n\n### Download Error\n\n#### HanLP Models\n\nIf the auto-download of a HanLP model fails, you can either:\n\n1. Retry as our file server might be busy serving users from all over the world.\n1. Follow the message on your terminal, which often guides you to manually download a `zip` file to a particular path. \n1. Use a [mirror site](https://hanlp.hankcs.com/docs/configure.html#use-mirror-sites) which could be faster and stabler in your region.\n\n#### Hugging Face 🤗 Transformers Models\n\nIf the auto-download of a Hugging Face 🤗 Transformers model fails, e.g., the following exception is threw out:\n\n```bash\nlib/python3.8/site-packages/transformers/file_utils.py\", line 2102, in get_from_cache\n    raise ValueError(\nValueError: Connection error, and we cannot find the requested files in the cached \npath. Please try again or make sure your Internet connection is on.\n```\n\nYou can either:\n\n1. Retry as the Internet is quite unstable in some regions (e.g., China).\n\n2. Force Hugging Face 🤗 Transformers to use cached models instead of checking updates from the Internet **if you have ever successfully loaded it before**, by setting the following environment variable:\n\n   ```bash\n   export TRANSFORMERS_OFFLINE=1\n   ```\n\n### Server without Internet\n\nIf your server has no Internet access at all, just debug your codes on your local PC and copy the following directories to your server via a USB disk or something.\n\n1. `~/.hanlp`: the home directory for HanLP models.\n1. `~/.cache/huggingface`: the home directory for Hugging Face 🤗 Transformers.\n\n\n### Import Error\n\nSome TensorFlow/fastText models will ask you to install the missing TensorFlow/fastText modules, in which case you'll need to install the full version:\n\n```shell script\npip install hanlp[full]\n```\n\n```{danger}\nNEVER install thirdparty packages (TensorFlow/fastText etc.) by yourself, as higher or lower versions of thirparty packages have not been tested and might not work properly.\n```"
  },
  {
    "path": "docs/references.bib",
    "content": "%% This BibTeX bibliography file was created using BibDesk.\n%% https://bibdesk.sourceforge.io/\n\n%% Created for hankcs at 2022-12-07 15:02:16 -0500 \n\n\n%% Saved with string encoding Unicode (UTF-8) \n\n\n\n@inproceedings{bai-etal-2022-graph,\n\taddress = {Dublin, Ireland},\n\tauthor = {Bai, Xuefeng and Chen, Yulong and Zhang, Yue},\n\tbooktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},\n\tdate-added = {2022-12-07 15:02:15 -0500},\n\tdate-modified = {2022-12-07 15:02:15 -0500},\n\tmonth = may,\n\tpages = {6001--6015},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {Graph Pre-training for {AMR} Parsing and Generation},\n\turl = {https://aclanthology.org/2022.acl-long.415},\n\tyear = {2022},\n\tbdsk-url-1 = {https://aclanthology.org/2022.acl-long.415}}\n\n@inproceedings{wang-etal-2021-minilmv2,\n\taddress = {Online},\n\tauthor = {Wang, Wenhui and Bao, Hangbo and Huang, Shaohan and Dong, Li and Wei, Furu},\n\tbooktitle = {Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021},\n\tdate-added = {2022-06-14 20:10:18 -0400},\n\tdate-modified = {2022-06-14 20:10:18 -0400},\n\tdoi = {10.18653/v1/2021.findings-acl.188},\n\tmonth = aug,\n\tpages = {2140--2151},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {{M}ini{LM}v2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers},\n\turl = {https://aclanthology.org/2021.findings-acl.188},\n\tyear = {2021},\n\tbdsk-url-1 = {https://aclanthology.org/2021.findings-acl.188},\n\tbdsk-url-2 = {https://doi.org/10.18653/v1/2021.findings-acl.188}}\n\n@article{zhang2021mengzi,\n\tauthor = {Zhang, Zhuosheng and Zhang, Hanqing and Chen, Keming and Guo, Yuhang and Hua, Jingyun and Wang, Yulong and Zhou, Ming},\n\tdate-added = {2022-04-15 10:32:14 -0400},\n\tdate-modified = {2022-04-15 10:32:14 -0400},\n\tjournal = {arXiv preprint arXiv:2110.06696},\n\ttitle = {Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinese},\n\tyear = {2021}}\n\n@inproceedings{samuel-straka-2020-ufal,\n\tabstract = {We present PERIN, a novel permutation-invariant approach to sentence-to-graph semantic parsing. PERIN is a versatile, cross-framework and language independent architecture for universal modeling of semantic structures. Our system participated in the CoNLL 2020 shared task, Cross-Framework Meaning Representation Parsing (MRP 2020), where it was evaluated on five different frameworks (AMR, DRG, EDS, PTG and UCCA) across four languages. PERIN was one of the winners of the shared task. The source code and pretrained models are available at http://www.github.com/ufal/perin.},\n\taddress = {Online},\n\tauthor = {Samuel, David and Straka, Milan},\n\tbooktitle = {Proceedings of the CoNLL 2020 Shared Task: Cross-Framework Meaning Representation Parsing},\n\tdate-added = {2022-04-12 22:36:23 -0400},\n\tdate-modified = {2022-04-12 22:36:23 -0400},\n\tdoi = {10.18653/v1/2020.conll-shared.5},\n\tmonth = nov,\n\tpages = {53--64},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {{{\\'U}FAL} at {MRP} 2020: Permutation-invariant Semantic Parsing in {PERIN}},\n\turl = {https://aclanthology.org/2020.conll-shared.5},\n\tyear = {2020},\n\tbdsk-url-1 = {https://aclanthology.org/2020.conll-shared.5},\n\tbdsk-url-2 = {https://doi.org/10.18653/v1/2020.conll-shared.5}}\n\n@inproceedings{qiu-etal-2014-multi,\n\taddress = {Dublin, Ireland},\n\tauthor = {Qiu, Likun and Zhang, Yue and Jin, Peng and Wang, Houfeng},\n\tbooktitle = {Proceedings of {COLING} 2014, the 25th International Conference on Computational Linguistics: Technical Papers},\n\tdate-added = {2022-02-15 04:42:58 -0500},\n\tdate-modified = {2022-02-15 04:42:58 -0500},\n\tmonth = aug,\n\tpages = {257--268},\n\tpublisher = {Dublin City University and Association for Computational Linguistics},\n\ttitle = {Multi-view {C}hinese Treebanking},\n\turl = {https://aclanthology.org/C14-1026},\n\tyear = {2014},\n\tbdsk-url-1 = {https://aclanthology.org/C14-1026}}\n\n@inproceedings{li-etal-2018-analogical,\n\tabstract = {Analogical reasoning is effective in capturing linguistic regularities. This paper proposes an analogical reasoning task on Chinese. After delving into Chinese lexical knowledge, we sketch 68 implicit morphological relations and 28 explicit semantic relations. A big and balanced dataset CA8 is then built for this task, including 17813 questions. Furthermore, we systematically explore the influences of vector representations, context features, and corpora on analogical reasoning. With the experiments, CA8 is proved to be a reliable benchmark for evaluating Chinese word embeddings.},\n\taddress = {Melbourne, Australia},\n\tauthor = {Li, Shen and Zhao, Zhe and Hu, Renfen and Li, Wensi and Liu, Tao and Du, Xiaoyong},\n\tbooktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},\n\tdate-added = {2022-01-30 22:52:52 -0500},\n\tdate-modified = {2022-01-30 22:52:52 -0500},\n\tdoi = {10.18653/v1/P18-2023},\n\tmonth = jul,\n\tpages = {138--143},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {Analogical Reasoning on {C}hinese Morphological and Semantic Relations},\n\turl = {https://aclanthology.org/P18-2023},\n\tyear = {2018},\n\tbdsk-url-1 = {https://aclanthology.org/P18-2023},\n\tbdsk-url-2 = {https://doi.org/10.18653/v1/P18-2023}}\n\n@inproceedings{NIPS2013_9aa42b31,\n\tauthor = {Mikolov, Tomas and Sutskever, Ilya and Chen, Kai and Corrado, Greg S and Dean, Jeff},\n\tbooktitle = {Advances in Neural Information Processing Systems},\n\tdate-added = {2022-01-30 18:17:28 -0500},\n\tdate-modified = {2022-01-30 18:17:28 -0500},\n\teditor = {C. J. C. Burges and L. Bottou and M. Welling and Z. Ghahramani and K. Q. Weinberger},\n\tpublisher = {Curran Associates, Inc.},\n\ttitle = {Distributed Representations of Words and Phrases and their Compositionality},\n\turl = {https://proceedings.neurips.cc/paper/2013/file/9aa42b31882ec039965f3c4923ce901b-Paper.pdf},\n\tvolume = {26},\n\tyear = {2013},\n\tbdsk-url-1 = {https://proceedings.neurips.cc/paper/2013/file/9aa42b31882ec039965f3c4923ce901b-Paper.pdf}}\n\n@inproceedings{bevilacqua-etal-2021-one,\n\tauthor = {Bevilacqua, Michele and Blloshmi, Rexhina and Navigli, Roberto},\n\tbooktitle = {Proceedings of AAAI},\n\tdate-added = {2022-01-25 11:58:03 -0500},\n\tdate-modified = {2022-01-25 11:58:03 -0500},\n\ttitle = {One {SPRING} to Rule Them Both: {S}ymmetric {AMR} Semantic Parsing and Generation without a Complex Pipeline},\n\tyear = {2021}}\n\n@inproceedings{lewis-etal-2020-bart,\n\tabstract = {We present BART, a denoising autoencoder for pretraining sequence-to-sequence models. BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Tranformer-based neural machine translation architecture which, despite its simplicity, can be seen as generalizing BERT (due to the bidirectional encoder), GPT (with the left-to-right decoder), and other recent pretraining schemes. We evaluate a number of noising approaches, finding the best performance by both randomly shuffling the order of sentences and using a novel in-filling scheme, where spans of text are replaced with a single mask token. BART is particularly effective when fine tuned for text generation but also works well for comprehension tasks. It matches the performance of RoBERTa on GLUE and SQuAD, and achieves new state-of-the-art results on a range of abstractive dialogue, question answering, and summarization tasks, with gains of up to 3.5 ROUGE. BART also provides a 1.1 BLEU increase over a back-translation system for machine translation, with only target language pretraining. We also replicate other pretraining schemes within the BART framework, to understand their effect on end-task performance.},\n\taddress = {Online},\n\tauthor = {Lewis, Mike and Liu, Yinhan and Goyal, Naman and Ghazvininejad, Marjan and Mohamed, Abdelrahman and Levy, Omer and Stoyanov, Veselin and Zettlemoyer, Luke},\n\tbooktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},\n\tdate-added = {2022-01-25 11:56:10 -0500},\n\tdate-modified = {2022-01-25 11:56:10 -0500},\n\tdoi = {10.18653/v1/2020.acl-main.703},\n\tmonth = jul,\n\tpages = {7871--7880},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {{BART}: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension},\n\turl = {https://www.aclweb.org/anthology/2020.acl-main.703},\n\tyear = {2020},\n\tbdsk-url-1 = {https://www.aclweb.org/anthology/2020.acl-main.703},\n\tbdsk-url-2 = {https://doi.org/10.18653/v1/2020.acl-main.703}}\n\n@article{knight2014abstract,\n\tauthor = {Knight, Kevin and Baranescu, Lauren and Bonial, Claire and Georgescu, Madalina and Griffitt, Kira and Hermjakob, Ulf and Marcu, Daniel and Palmer, Martha and Schneifer, Nathan},\n\tdate-added = {2022-01-25 11:54:11 -0500},\n\tdate-modified = {2022-01-25 11:54:11 -0500},\n\tjournal = {Web download},\n\ttitle = {Abstract meaning representation (amr) annotation release 1.0},\n\tyear = {2014}}\n\n@inproceedings{he-choi-2021-stem,\n\tabstract = {Multi-task learning with transformer encoders (MTL) has emerged as a powerful technique to improve performance on closely-related tasks for both accuracy and efficiency while a question still remains whether or not it would perform as well on tasks that are distinct in nature. We first present MTL results on five NLP tasks, POS, NER, DEP, CON, and SRL, and depict its deficiency over single-task learning. We then conduct an extensive pruning analysis to show that a certain set of attention heads get claimed by most tasks during MTL, who interfere with one another to fine-tune those heads for their own objectives. Based on this finding, we propose the Stem Cell Hypothesis to reveal the existence of attention heads naturally talented for many tasks that cannot be jointly trained to create adequate embeddings for all of those tasks. Finally, we design novel parameter-free probes to justify our hypothesis and demonstrate how attention heads are transformed across the five tasks during MTL through label analysis.},\n\taddress = {Online and Punta Cana, Dominican Republic},\n\tauthor = {He, Han and Choi, Jinho D.},\n\tbooktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},\n\tdate-added = {2021-11-06 18:24:44 -0400},\n\tdate-modified = {2021-11-06 18:24:44 -0400},\n\tmonth = nov,\n\tpages = {5555--5577},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders},\n\turl = {https://aclanthology.org/2021.emnlp-main.451},\n\tyear = {2021},\n\tbdsk-url-1 = {https://aclanthology.org/2021.emnlp-main.451}}\n\n@inproceedings{he-choi-2019,\n\tabstract = {This paper presents new state-of-the-art models for three tasks, part-of-speech tagging, syntactic parsing, and semantic parsing, using the cutting-edge contextualized embedding framework known as BERT. For each task, we first replicate and simplify the current state-of-the-art approach to enhance its model efficiency. We then evaluate our simplified approaches on those three tasks using token embeddings generated by BERT. 12 datasets in both English and Chinese are used for our experiments. The BERT models outperform the previously best-performing models by 2.5\\% on average (7.5\\% for the most significant case). All models and source codes are available in public so that researchers can improve upon and utilize them to establish strong baselines for the next decade.},\n\tauthor = {Han He and Jinho Choi},\n\tbooktitle = {The Thirty-Third International Flairs Conference},\n\tconference = {Florida Artificial Intelligence Research Society Conference},\n\tdate-added = {2021-10-16 21:09:00 -0400},\n\tdate-modified = {2021-10-16 21:09:00 -0400},\n\tkeywords = {part-of-speech tagging, syntactic parsing, semantic parsing, Transformer, BERT},\n\ttitle = {Establishing Strong Baselines for the New Decade: Sequence Tagging, Syntactic and Semantic Parsing with BERT},\n\turl = {https://www.aaai.org/ocs/index.php/FLAIRS/FLAIRS20/paper/view/18438},\n\tyear = {2020},\n\tbdsk-url-1 = {https://www.aaai.org/ocs/index.php/FLAIRS/FLAIRS20/paper/view/18438}}\n\n@inproceedings{xiao-etal-2021-ernie,\n\tabstract = {Coarse-grained linguistic information, such as named entities or phrases, facilitates adequately representation learning in pre-training. Previous works mainly focus on extending the objective of BERT{'}s Masked Language Modeling (MLM) from masking individual tokens to contiguous sequences of n tokens. We argue that such contiguously masking method neglects to model the intra-dependencies and inter-relation of coarse-grained linguistic information. As an alternative, we propose ERNIE-Gram, an explicitly n-gram masking method to enhance the integration of coarse-grained information into pre-training. In ERNIE-Gram, n-grams are masked and predicted directly using explicit n-gram identities rather than contiguous sequences of n tokens. Furthermore, ERNIE-Gram employs a generator model to sample plausible n-gram identities as optional n-gram masks and predict them in both coarse-grained and fine-grained manners to enable comprehensive n-gram prediction and relation modeling. We pre-train ERNIE-Gram on English and Chinese text corpora and fine-tune on 19 downstream tasks. Experimental results show that ERNIE-Gram outperforms previous pre-training models like XLNet and RoBERTa by a large margin, and achieves comparable results with state-of-the-art methods. The source codes and pre-trained models have been released at https://github.com/PaddlePaddle/ERNIE.},\n\taddress = {Online},\n\tauthor = {Xiao, Dongling and Li, Yu-Kun and Zhang, Han and Sun, Yu and Tian, Hao and Wu, Hua and Wang, Haifeng},\n\tbooktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},\n\tdate-added = {2021-09-04 14:09:52 -0400},\n\tdate-modified = {2021-09-04 14:09:52 -0400},\n\tdoi = {10.18653/v1/2021.naacl-main.136},\n\tmonth = jun,\n\tpages = {1702--1715},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {{ERNIE}-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding},\n\turl = {https://aclanthology.org/2021.naacl-main.136},\n\tyear = {2021},\n\tbdsk-url-1 = {https://aclanthology.org/2021.naacl-main.136},\n\tbdsk-url-2 = {https://doi.org/10.18653/v1/2021.naacl-main.136}}\n\n@inproceedings{akbik-etal-2018-contextual,\n\tabstract = {Recent advances in language modeling using recurrent neural networks have made it viable to model language as distributions over characters. By learning to predict the next character on the basis of previous characters, such models have been shown to automatically internalize linguistic concepts such as words, sentences, subclauses and even sentiment. In this paper, we propose to leverage the internal states of a trained character language model to produce a novel type of word embedding which we refer to as contextual string embeddings. Our proposed embeddings have the distinct properties that they (a) are trained without any explicit notion of words and thus fundamentally model words as sequences of characters, and (b) are contextualized by their surrounding text, meaning that the same word will have different embeddings depending on its contextual use. We conduct a comparative evaluation against previous embeddings and find that our embeddings are highly useful for downstream tasks: across four classic sequence labeling tasks we consistently outperform the previous state-of-the-art. In particular, we significantly outperform previous work on English and German named entity recognition (NER), allowing us to report new state-of-the-art F1-scores on the CoNLL03 shared task. We release all code and pre-trained language models in a simple-to-use framework to the research community, to enable reproduction of these experiments and application of our proposed embeddings to other tasks: https://github.com/zalandoresearch/flair},\n\taddress = {Santa Fe, New Mexico, USA},\n\tauthor = {Akbik, Alan and Blythe, Duncan and Vollgraf, Roland},\n\tbooktitle = {Proceedings of the 27th International Conference on Computational Linguistics},\n\tdate-added = {2021-09-01 13:10:59 -0400},\n\tdate-modified = {2021-09-01 13:10:59 -0400},\n\tmonth = aug,\n\tpages = {1638--1649},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {Contextual String Embeddings for Sequence Labeling},\n\turl = {https://aclanthology.org/C18-1139},\n\tyear = {2018},\n\tbdsk-url-1 = {https://aclanthology.org/C18-1139}}\n\n@inproceedings{he-choi-2021-levi,\n\tabstract = {Coupled with biaffine decoders, transformers have been effectively adapted to text-to-graph transduction and achieved state-of-the-art performance on AMR parsing. Many prior works, however, rely on the biaffine decoder for either or both arc and label predictions although most features used by the decoder may be learned by the transformer already. This paper presents a novel approach to AMR parsing by combining heterogeneous data (tokens, concepts, labels) as one input to a transformer to learn attention, and use only attention matrices from the transformer to predict all elements in AMR graphs (concepts, arcs, labels). Although our models use significantly fewer parameters than the previous state-of-the-art graph parser, they show similar or better accuracy on AMR 2.0 and 3.0.},\n\taddress = {Online},\n\tauthor = {He, Han and Choi, Jinho D.},\n\tbooktitle = {Proceedings of the 17th International Conference on Parsing Technologies and the IWPT 2021 Shared Task on Parsing into Enhanced Universal Dependencies (IWPT 2021)},\n\tdate-added = {2021-09-01 13:09:14 -0400},\n\tdate-modified = {2021-09-01 13:09:14 -0400},\n\tdoi = {10.18653/v1/2021.iwpt-1.5},\n\tmonth = aug,\n\tpages = {50--57},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {Levi Graph {AMR} Parser using Heterogeneous Attention},\n\turl = {https://aclanthology.org/2021.iwpt-1.5},\n\tyear = {2021},\n\tbdsk-url-1 = {https://aclanthology.org/2021.iwpt-1.5},\n\tbdsk-url-2 = {https://doi.org/10.18653/v1/2021.iwpt-1.5}}\n\n@inproceedings{conneau-etal-2020-unsupervised,\n\tabstract = {This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred languages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +14.6{\\%} average accuracy on XNLI, +13{\\%} average F1 score on MLQA, and +2.4{\\%} F1 score on NER. XLM-R performs particularly well on low-resource languages, improving 15.7{\\%} in XNLI accuracy for Swahili and 11.4{\\%} for Urdu over previous XLM models. We also present a detailed empirical analysis of the key factors that are required to achieve these gains, including the trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing per-language performance; XLM-R is very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We will make our code and models publicly available.},\n\taddress = {Online},\n\tauthor = {Conneau, Alexis and Khandelwal, Kartikay and Goyal, Naman and Chaudhary, Vishrav and Wenzek, Guillaume and Guzm{\\'a}n, Francisco and Grave, Edouard and Ott, Myle and Zettlemoyer, Luke and Stoyanov, Veselin},\n\tbooktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},\n\tdate-added = {2021-09-01 12:41:50 -0400},\n\tdate-modified = {2021-09-01 12:41:50 -0400},\n\tdoi = {10.18653/v1/2020.acl-main.747},\n\tmonth = jul,\n\tpages = {8440--8451},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {Unsupervised Cross-lingual Representation Learning at Scale},\n\turl = {https://aclanthology.org/2020.acl-main.747},\n\tyear = {2020},\n\tbdsk-url-1 = {https://aclanthology.org/2020.acl-main.747},\n\tbdsk-url-2 = {https://doi.org/10.18653/v1/2020.acl-main.747}}\n\n@inproceedings{xue-etal-2021-mt5,\n\tabstract = {The recent {``}Text-to-Text Transfer Transformer{''} (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual benchmarks. We also describe a simple technique to prevent {``}accidental translation{''} in the zero-shot setting, where a generative model chooses to (partially) translate its prediction into the wrong language. All of the code and model checkpoints used in this work are publicly available.},\n\taddress = {Online},\n\tauthor = {Xue, Linting and Constant, Noah and Roberts, Adam and Kale, Mihir and Al-Rfou, Rami and Siddhant, Aditya and Barua, Aditya and Raffel, Colin},\n\tbooktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},\n\tdate-added = {2021-09-01 12:40:34 -0400},\n\tdate-modified = {2021-09-01 12:40:34 -0400},\n\tdoi = {10.18653/v1/2021.naacl-main.41},\n\tmonth = jun,\n\tpages = {483--498},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {m{T}5: A Massively Multilingual Pre-trained Text-to-Text Transformer},\n\turl = {https://aclanthology.org/2021.naacl-main.41},\n\tyear = {2021},\n\tbdsk-url-1 = {https://aclanthology.org/2021.naacl-main.41},\n\tbdsk-url-2 = {https://doi.org/10.18653/v1/2021.naacl-main.41}}\n\n@misc{https://doi.org/10.35111/gvd0-xk91,\n\tauthor = {Xue, Nianwen and {Zhang, Xiuhong} and {Jiang, Zixin} and {Palmer, Martha} and {Xia, Fei} and {Chiou, Fu-Dong} and {Chang, Meiyu}},\n\tdate-added = {2021-09-01 12:32:05 -0400},\n\tdate-modified = {2021-09-01 12:36:22 -0400},\n\tdoi = {10.35111/GVD0-XK91},\n\tpublisher = {Linguistic Data Consortium},\n\ttitle = {Chinese Treebank 9.0},\n\turl = {https://catalog.ldc.upenn.edu/LDC2016T13},\n\tyear = {2016},\n\tbdsk-url-1 = {https://catalog.ldc.upenn.edu/LDC2016T13},\n\tbdsk-url-2 = {https://doi.org/10.35111/GVD0-XK91}}\n\n@inproceedings{clark2020electra,\n\tauthor = {Kevin Clark and Minh-Thang Luong and Quoc V. Le and Christopher D. Manning},\n\tbooktitle = {ICLR},\n\tdate-added = {2021-08-07 15:53:27 -0400},\n\tdate-modified = {2021-08-07 15:53:27 -0400},\n\ttitle = {{ELECTRA}: Pre-training Text Encoders as Discriminators Rather Than Generators},\n\turl = {https://openreview.net/pdf?id=r1xMH1BtvB},\n\tyear = {2020},\n\tbdsk-url-1 = {https://openreview.net/pdf?id=r1xMH1BtvB}}\n\n@inproceedings{chang-etal-2009-discriminative,\n\taddress = {Boulder, Colorado},\n\tauthor = {Chang, Pi-Chuan and Tseng, Huihsin and Jurafsky, Dan and Manning, Christopher D.},\n\tbooktitle = {Proceedings of the Third Workshop on Syntax and Structure in Statistical Translation ({SSST}-3) at {NAACL} {HLT} 2009},\n\tdate-added = {2021-03-17 13:37:03 -0400},\n\tdate-modified = {2021-03-17 13:37:03 -0400},\n\tmonth = jun,\n\tpages = {51--59},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {Discriminative Reordering with {C}hinese Grammatical Relations Features},\n\turl = {https://www.aclweb.org/anthology/W09-2307},\n\tyear = {2009},\n\tbdsk-url-1 = {https://www.aclweb.org/anthology/W09-2307}}\n\n@inproceedings{pennington-etal-2014-glove,\n\taddress = {Doha, Qatar},\n\tauthor = {Pennington, Jeffrey and Socher, Richard and Manning, Christopher},\n\tbooktitle = {Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ({EMNLP})},\n\tdate-added = {2020-12-31 15:07:29 -0500},\n\tdate-modified = {2020-12-31 15:07:29 -0500},\n\tdoi = {10.3115/v1/D14-1162},\n\tmonth = oct,\n\tpages = {1532--1543},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {{G}lo{V}e: Global Vectors for Word Representation},\n\turl = {https://www.aclweb.org/anthology/D14-1162},\n\tyear = {2014},\n\tbdsk-url-1 = {https://www.aclweb.org/anthology/D14-1162},\n\tbdsk-url-2 = {https://doi.org/10.3115/v1/D14-1162}}\n\n@incollection{he2018dual,\n\tauthor = {He, Han and Wu, Lei and Yang, Xiaokun and Yan, Hua and Gao, Zhimin and Feng, Yi and Townsend, George},\n\tbooktitle = {Information Technology-New Generations},\n\tdate-added = {2020-12-31 15:03:58 -0500},\n\tdate-modified = {2020-12-31 15:03:58 -0500},\n\tpages = {421--426},\n\tpublisher = {Springer},\n\ttitle = {Dual long short-term memory networks for sub-character representation learning},\n\tyear = {2018}}\n\n@inproceedings{devlin-etal-2019-bert,\n\tabstract = {We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a; Radford et al., 2018), BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7{\\%} (4.6{\\%} absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).},\n\taddress = {Minneapolis, Minnesota},\n\tauthor = {Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},\n\tbooktitle = {Proceedings of the 2019 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},\n\tdate-added = {2020-12-31 14:46:54 -0500},\n\tdate-modified = {2020-12-31 14:46:54 -0500},\n\tdoi = {10.18653/v1/N19-1423},\n\tmonth = jun,\n\tpages = {4171--4186},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {{BERT}: Pre-training of Deep Bidirectional Transformers for Language Understanding},\n\turl = {https://www.aclweb.org/anthology/N19-1423},\n\tyear = {2019},\n\tbdsk-url-1 = {https://www.aclweb.org/anthology/N19-1423},\n\tbdsk-url-2 = {https://doi.org/10.18653/v1/N19-1423}}\n\n@inproceedings{Lan2020ALBERT:,\n\tauthor = {Zhenzhong Lan and Mingda Chen and Sebastian Goodman and Kevin Gimpel and Piyush Sharma and Radu Soricut},\n\tbooktitle = {International Conference on Learning Representations},\n\tdate-added = {2020-12-31 14:44:52 -0500},\n\tdate-modified = {2020-12-31 14:44:52 -0500},\n\ttitle = {ALBERT: A Lite BERT for Self-supervised Learning of Language Representations},\n\turl = {https://openreview.net/forum?id=H1eA7AEtvS},\n\tyear = {2020},\n\tbdsk-url-1 = {https://openreview.net/forum?id=H1eA7AEtvS}}\n\n@inproceedings{wang-xu-2017-convolutional,\n\tabstract = {Character-based sequence labeling framework is flexible and efficient for Chinese word segmentation (CWS). Recently, many character-based neural models have been applied to CWS. While they obtain good performance, they have two obvious weaknesses. The first is that they heavily rely on manually designed bigram feature, i.e. they are not good at capturing $n$-gram features automatically. The second is that they make no use of full word information. For the first weakness, we propose a convolutional neural model, which is able to capture rich $n$-gram features without any feature engineering. For the second one, we propose an effective approach to integrate the proposed model with word embeddings. We evaluate the model on two benchmark datasets: PKU and MSR. Without any feature engineering, the model obtains competitive performance {---} 95.7{\\%} on PKU and 97.3{\\%} on MSR. Armed with word embeddings, the model achieves state-of-the-art performance on both datasets {---} 96.5{\\%} on PKU and 98.0{\\%} on MSR, without using any external labeled resource.},\n\taddress = {Taipei, Taiwan},\n\tauthor = {Wang, Chunqi and Xu, Bo},\n\tbooktitle = {Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},\n\tdate-added = {2020-12-31 14:42:35 -0500},\n\tdate-modified = {2020-12-31 14:42:35 -0500},\n\tmonth = nov,\n\tpages = {163--172},\n\tpublisher = {Asian Federation of Natural Language Processing},\n\ttitle = {Convolutional Neural Network with Word Embeddings for {C}hinese Word Segmentation},\n\turl = {https://www.aclweb.org/anthology/I17-1017},\n\tyear = {2017},\n\tbdsk-url-1 = {https://www.aclweb.org/anthology/I17-1017}}\n\n@article{bojanowski2017enriching,\n\tauthor = {Bojanowski, Piotr and Grave, Edouard and Joulin, Armand and Mikolov, Tomas},\n\tdate-added = {2020-12-25 22:31:59 -0500},\n\tdate-modified = {2020-12-25 22:31:59 -0500},\n\tissn = {2307-387X},\n\tjournal = {Transactions of the Association for Computational Linguistics},\n\tpages = {135--146},\n\ttitle = {Enriching Word Vectors with Subword Information},\n\tvolume = {5},\n\tyear = {2017}}\n\n@article{collins-koo-2005-discriminative,\n\tauthor = {Collins, Michael and Koo, Terry},\n\tdate-added = {2020-12-25 17:25:59 -0500},\n\tdate-modified = {2020-12-25 17:25:59 -0500},\n\tdoi = {10.1162/0891201053630273},\n\tjournal = {Computational Linguistics},\n\tnumber = {1},\n\tpages = {25--70},\n\ttitle = {Discriminative Reranking for Natural Language Parsing},\n\turl = {https://www.aclweb.org/anthology/J05-1003},\n\tvolume = {31},\n\tyear = {2005},\n\tbdsk-url-1 = {https://www.aclweb.org/anthology/J05-1003},\n\tbdsk-url-2 = {https://doi.org/10.1162/0891201053630273}}\n\n@inproceedings{zhang-clark-2008-tale,\n\taddress = {Honolulu, Hawaii},\n\tauthor = {Zhang, Yue and Clark, Stephen},\n\tbooktitle = {Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing},\n\tdate-added = {2020-12-25 15:10:10 -0500},\n\tdate-modified = {2020-12-25 15:10:10 -0500},\n\tmonth = oct,\n\tpages = {562--571},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {A Tale of Two Parsers: {I}nvestigating and Combining Graph-based and Transition-based Dependency Parsing},\n\turl = {https://www.aclweb.org/anthology/D08-1059},\n\tyear = {2008},\n\tbdsk-url-1 = {https://www.aclweb.org/anthology/D08-1059}}\n\n@inproceedings{pradhan-etal-2012-conll,\n\taddress = {Jeju Island, Korea},\n\tauthor = {Pradhan, Sameer and Moschitti, Alessandro and Xue, Nianwen and Uryupina, Olga and Zhang, Yuchen},\n\tbooktitle = {Joint Conference on {EMNLP} and {C}o{NLL} - Shared Task},\n\tdate-added = {2020-12-24 23:42:41 -0500},\n\tdate-modified = {2020-12-24 23:42:41 -0500},\n\tmonth = jul,\n\tpages = {1--40},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {{C}o{NLL}-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in {O}nto{N}otes},\n\turl = {https://www.aclweb.org/anthology/W12-4501},\n\tyear = {2012},\n\tbdsk-url-1 = {https://www.aclweb.org/anthology/W12-4501}}\n\n@inproceedings{levow-2006-third,\n\taddress = {Sydney, Australia},\n\tauthor = {Levow, Gina-Anne},\n\tbooktitle = {Proceedings of the Fifth {SIGHAN} Workshop on {C}hinese Language Processing},\n\tdate-added = {2020-12-24 23:21:14 -0500},\n\tdate-modified = {2020-12-24 23:21:14 -0500},\n\tmonth = jul,\n\tpages = {108--117},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {The Third International {C}hinese Language Processing Bakeoff: Word Segmentation and Named Entity Recognition},\n\turl = {https://www.aclweb.org/anthology/W06-0115},\n\tyear = {2006},\n\tbdsk-url-1 = {https://www.aclweb.org/anthology/W06-0115}}\n\n@inproceedings{tjong-kim-sang-de-meulder-2003-introduction,\n\tauthor = {Tjong Kim Sang, Erik F. and De Meulder, Fien},\n\tbooktitle = {Proceedings of the Seventh Conference on Natural Language Learning at {HLT}-{NAACL} 2003},\n\tdate-added = {2020-12-24 23:19:00 -0500},\n\tdate-modified = {2020-12-24 23:19:00 -0500},\n\tpages = {142--147},\n\ttitle = {Introduction to the {C}o{NLL}-2003 Shared Task: Language-Independent Named Entity Recognition},\n\turl = {https://www.aclweb.org/anthology/W03-0419},\n\tyear = {2003},\n\tbdsk-url-1 = {https://www.aclweb.org/anthology/W03-0419}}\n\n@inproceedings{koehn2005europarl,\n\tauthor = {Koehn, Philipp},\n\tbooktitle = {MT summit},\n\tdate-added = {2020-12-24 23:06:03 -0500},\n\tdate-modified = {2020-12-24 23:06:03 -0500},\n\torganization = {Citeseer},\n\tpages = {79--86},\n\ttitle = {Europarl: A parallel corpus for statistical machine translation},\n\tvolume = {5},\n\tyear = {2005}}\n\n@inproceedings{Schweter:Ahmed:2019,\n\tauthor = {Stefan Schweter and Sajawel Ahmed},\n\tbooktitle = {Proceedings of the 15th Conference on Natural Language Processing (KONVENS)},\n\tdate-added = {2020-12-24 23:03:23 -0500},\n\tdate-modified = {2020-12-24 23:03:23 -0500},\n\tlocation = {Erlangen, Germany},\n\tnote = {accepted},\n\ttitle = {{Deep-EOS: General-Purpose Neural Networks for Sentence Boundary Detection}},\n\tyear = 2019}\n\n@incollection{he2019effective,\n\tauthor = {He, Han and Wu, Lei and Yan, Hua and Gao, Zhimin and Feng, Yi and Townsend, George},\n\tbooktitle = {Smart Intelligent Computing and Applications},\n\tdate-added = {2020-12-24 19:35:03 -0500},\n\tdate-modified = {2020-12-24 19:35:03 -0500},\n\tpages = {133--142},\n\tpublisher = {Springer},\n\ttitle = {Effective neural solution for multi-criteria word segmentation},\n\tyear = {2019}}\n\n@inproceedings{dozat2017stanford,\n\tauthor = {Dozat, Timothy and Qi, Peng and Manning, Christopher D},\n\tbooktitle = {Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies},\n\tdate-added = {2020-12-24 15:02:18 -0500},\n\tdate-modified = {2020-12-24 15:02:18 -0500},\n\tpages = {20--30},\n\ttitle = {Stanford's graph-based neural dependency parser at the conll 2017 shared task},\n\tyear = {2017}}\n\n@inproceedings{he-etal-2018-jointly,\n\tabstract = {Recent BIO-tagging-based neural semantic role labeling models are very high performing, but assume gold predicates as part of the input and cannot incorporate span-level features. We propose an end-to-end approach for jointly predicting all predicates, arguments spans, and the relations between them. The model makes independent decisions about what relationship, if any, holds between every possible word-span pair, and learns contextualized span representations that provide rich, shared input features for each decision. Experiments demonstrate that this approach sets a new state of the art on PropBank SRL without gold predicates.},\n\taddress = {Melbourne, Australia},\n\tauthor = {He, Luheng and Lee, Kenton and Levy, Omer and Zettlemoyer, Luke},\n\tbooktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},\n\tdate-added = {2020-12-24 14:23:45 -0500},\n\tdate-modified = {2020-12-24 14:23:45 -0500},\n\tdoi = {10.18653/v1/P18-2058},\n\tmonth = jul,\n\tpages = {364--369},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling},\n\turl = {https://www.aclweb.org/anthology/P18-2058},\n\tyear = {2018},\n\tbdsk-url-1 = {https://www.aclweb.org/anthology/P18-2058},\n\tbdsk-url-2 = {https://doi.org/10.18653/v1/P18-2058}}\n\n@inproceedings{yu-etal-2020-named,\n\tabstract = {Named Entity Recognition (NER) is a fundamental task in Natural Language Processing, concerned with identifying spans of text expressing references to entities. NER research is often focused on flat entities only (flat NER), ignoring the fact that entity references can be nested, as in [Bank of [China]] (Finkel and Manning, 2009). In this paper, we use ideas from graph-based dependency parsing to provide our model a global view on the input via a biaffine model (Dozat and Manning, 2017). The biaffine model scores pairs of start and end tokens in a sentence which we use to explore all spans, so that the model is able to predict named entities accurately. We show that the model works well for both nested and flat NER through evaluation on 8 corpora and achieving SoTA performance on all of them, with accuracy gains of up to 2.2 percentage points.},\n\taddress = {Online},\n\tauthor = {Yu, Juntao and Bohnet, Bernd and Poesio, Massimo},\n\tbooktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},\n\tdate-added = {2020-12-24 13:35:09 -0500},\n\tdate-modified = {2020-12-24 13:35:09 -0500},\n\tdoi = {10.18653/v1/2020.acl-main.577},\n\tmonth = jul,\n\tpages = {6470--6476},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {Named Entity Recognition as Dependency Parsing},\n\turl = {https://www.aclweb.org/anthology/2020.acl-main.577},\n\tyear = {2020},\n\tbdsk-url-1 = {https://www.aclweb.org/anthology/2020.acl-main.577},\n\tbdsk-url-2 = {https://doi.org/10.18653/v1/2020.acl-main.577}}\n\n@inproceedings{10.1145/1457838.1457895,\n\tabstract = {Many computer applications require the storage of large amounts of information within the computer's memory where it will be readily available for reference and updating. Quite commonly, more storage space is required than is available in the computer's high-speed working memory. It is, therefore, a common practice to equip computers with magnetic tapes, disks, or drums, or a combination of these to provide additional storage. This additional storage is always slower in operation than the computer's working memory and therefore care must be taken when using it to avoid excessive operating time.},\n\taddress = {New York, NY, USA},\n\tauthor = {De La Briandais, Rene},\n\tbooktitle = {Papers Presented at the the March 3-5, 1959, Western Joint Computer Conference},\n\tdate-added = {2020-12-24 13:07:31 -0500},\n\tdate-modified = {2020-12-24 13:07:31 -0500},\n\tdoi = {10.1145/1457838.1457895},\n\tisbn = {9781450378659},\n\tlocation = {San Francisco, California},\n\tnumpages = {4},\n\tpages = {295--298},\n\tpublisher = {Association for Computing Machinery},\n\tseries = {IRE-AIEE-ACM '59 (Western)},\n\ttitle = {File Searching Using Variable Length Keys},\n\turl = {https://doi.org/10.1145/1457838.1457895},\n\tyear = {1959},\n\tbdsk-url-1 = {https://doi.org/10.1145/1457838.1457895}}\n\n@article{lafferty2001conditional,\n\tauthor = {Lafferty, John and McCallum, Andrew and Pereira, Fernando CN},\n\tdate-added = {2020-12-24 11:46:30 -0500},\n\tdate-modified = {2020-12-24 12:08:29 -0500},\n\tjournal = {Departmental Papers (CIS)},\n\ttitle = {Conditional random fields: Probabilistic models for segmenting and labeling sequence data},\n\tyear = {2001}}\n\n@inproceedings{clark-etal-2019-bam,\n\tabstract = {It can be challenging to train multi-task neural networks that outperform or even match their single-task counterparts. To help address this, we propose using knowledge distillation where single-task models teach a multi-task model. We enhance this training with teacher annealing, a novel method that gradually transitions the model from distillation to supervised learning, helping the multi-task model surpass its single-task teachers. We evaluate our approach by multi-task fine-tuning BERT on the GLUE benchmark. Our method consistently improves over standard single-task and multi-task training.},\n\taddress = {Florence, Italy},\n\tauthor = {Clark, Kevin and Luong, Minh-Thang and Khandelwal, Urvashi and Manning, Christopher D. and Le, Quoc V.},\n\tbooktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},\n\tdate-added = {2020-12-24 11:26:54 -0500},\n\tdate-modified = {2020-12-24 11:26:54 -0500},\n\tdoi = {10.18653/v1/P19-1595},\n\tmonth = jul,\n\tpages = {5931--5937},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {{BAM}! Born-Again Multi-Task Networks for Natural Language Understanding},\n\turl = {https://www.aclweb.org/anthology/P19-1595},\n\tyear = {2019},\n\tbdsk-url-1 = {https://www.aclweb.org/anthology/P19-1595},\n\tbdsk-url-2 = {https://doi.org/10.18653/v1/P19-1595}}\n\n@inproceedings{kondratyuk-straka-2019-75,\n\taddress = {Hong Kong, China},\n\tauthor = {Kondratyuk, Dan and Straka, Milan},\n\tbooktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},\n\tdate-added = {2020-12-23 23:51:07 -0500},\n\tdate-modified = {2020-12-23 23:51:07 -0500},\n\tpages = {2779--2795},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {75 Languages, 1 Model: Parsing Universal Dependencies Universally},\n\turl = {https://www.aclweb.org/anthology/D19-1279},\n\tyear = {2019},\n\tbdsk-url-1 = {https://www.aclweb.org/anthology/D19-1279}}\n\n@inproceedings{dozat:17a,\n\tauthor = {Dozat, Timothy and Manning, Christopher D.},\n\tbooktitle = {Proceedings of the 5th International Conference on Learning Representations},\n\tdate-added = {2020-12-23 23:46:20 -0500},\n\tdate-modified = {2020-12-23 23:46:20 -0500},\n\tseries = {ICLR'17},\n\ttitle = {{Deep Biaffine Attention for Neural Dependency Parsing}},\n\turl = {https://openreview.net/pdf?id=Hk95PK9le},\n\tyear = {2017},\n\tbdsk-url-1 = {http://arxiv.org/abs/1611.01734},\n\tbdsk-url-2 = {https://openreview.net/pdf?id=Hk95PK9le}}\n\n@inproceedings{smith-smith-2007-probabilistic,\n\taddress = {Prague, Czech Republic},\n\tauthor = {Smith, David A. and Smith, Noah A.},\n\tbooktitle = {Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning ({EMNLP}-{C}o{NLL})},\n\tdate-added = {2020-12-23 21:46:06 -0500},\n\tdate-modified = {2020-12-23 21:46:06 -0500},\n\tmonth = jun,\n\tpages = {132--140},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {Probabilistic Models of Nonprojective Dependency Trees},\n\turl = {https://www.aclweb.org/anthology/D07-1014},\n\tyear = {2007},\n\tbdsk-url-1 = {https://www.aclweb.org/anthology/D07-1014}}\n\n@inproceedings{ijcai2020-560,\n\tauthor = {Zhang, Yu and Zhou, Houquan and Li, Zhenghua},\n\tbooktitle = {Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, {IJCAI-20}},\n\tdate-added = {2020-12-23 21:36:56 -0500},\n\tdate-modified = {2020-12-23 21:36:56 -0500},\n\tdoi = {10.24963/ijcai.2020/560},\n\teditor = {Christian Bessiere},\n\tmonth = {7},\n\tnote = {Main track},\n\tpages = {4046--4053},\n\tpublisher = {International Joint Conferences on Artificial Intelligence Organization},\n\ttitle = {Fast and Accurate Neural CRF Constituency Parsing},\n\turl = {https://doi.org/10.24963/ijcai.2020/560},\n\tyear = {2020},\n\tbdsk-url-1 = {https://doi.org/10.24963/ijcai.2020/560}}\n\n@inproceedings{buchholz-marsi-2006-conll,\n\taddress = {New York City},\n\tauthor = {Buchholz, Sabine and Marsi, Erwin},\n\tbooktitle = {Proceedings of the Tenth Conference on Computational Natural Language Learning ({C}o{NLL}-X)},\n\tdate-added = {2020-12-22 22:57:41 -0500},\n\tdate-modified = {2020-12-22 22:57:41 -0500},\n\tmonth = jun,\n\tpages = {149--164},\n\tpublisher = {Association for Computational Linguistics},\n\ttitle = {{C}o{NLL}-{X} Shared Task on Multilingual Dependency Parsing},\n\turl = {https://www.aclweb.org/anthology/W06-2920},\n\tyear = {2006},\n\tbdsk-url-1 = {https://www.aclweb.org/anthology/W06-2920}}\n"
  },
  {
    "path": "docs/references.rst",
    "content": "References\n==================\n\n.. bibliography:: references.bib\n\t:cited:\n\t:style: astrostyle"
  },
  {
    "path": "docs/tutorial.md",
    "content": "---\njupytext:\n  formats: ipynb,md:myst\n  text_representation:\n    extension: .md\n    format_name: myst\n    format_version: '0.8'\n    jupytext_version: 1.4.2\nkernelspec:\n  display_name: Python 3\n  language: python\n  name: python3\n---\n\n# Tutorial\n\nNatural Language Processing is an exciting field consisting of many closely related tasks like lexical analysis \nand parsing. Each task involves many datasets and models, all requiring a high degree of expertise. \nThings become even more complex when dealing with multilingual text, as there's simply no datasets for some \nlow-resource languages. However, with HanLP 2.1, core NLP tasks have been made easy to access and efficient in \nproduction environments. In this tutorial, we'll walk through the APIs in HanLP step by step. \n\nHanLP offers out-of-the-box RESTful API and native Python API which share very similar interfaces \nwhile they are designed for different scenes.\n\n```{code-cell} ipython3\n:tags: [remove_cell]\n\nimport hanlp_common.constant\n\nhanlp_common.constant.IPYTHON = False  # Avoid pretty_print prints html which doesn't play well with this theme\n```\n\n## RESTful API\n\nRESTful API is an endpoint where you send your documents to then get the parsed annotations back. \nWe are hosting a **non-commercial** API service and you are welcome to [apply for an auth key](https://bbs.hankcs.com/t/apply-for-free-hanlp-restful-apis/3178). \nAn auth key is a password which gives you access to our API and protects our server from being abused. \nOnce obtained such an auth key, you can parse your document with our RESTful client which can be installed via:\n\n````{margin} **Non-Commercial**\n```{seealso}\nOur models and RESTful APIs are under the [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) licence.\n```\n````\n\n````{margin} **Zero-Shot Learning**\n```{note}\nAlthough UD covers 104 languages, OntoNotes (NER, CON, SRL) covers only English, Chinese and Arabic.\nSo NER/CON/SRL of languages other than the 3 are considered as Zero-Shot and their accuracies can be very low.  \n```\n````\n\n```bash\npip install hanlp_restful\n```\n\n```{eval-rst}\nThen initiate a :class:`~hanlp_restful.HanLPClient` with your auth key and send a document to have it parsed.\n```\n\n```{code-cell} ipython3\n:tags: [output_scroll]\nfrom hanlp_restful import HanLPClient\n# Fill in your auth, set language='zh' to use Chinese models\nHanLP = HanLPClient('https://hanlp.hankcs.com/api', auth=None, language='mul')\ndoc = HanLP('In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques to production environments. ' \\\n            '2021年、HanLPv2.1は次世代の最先端多言語NLP技術を本番環境に導入します。' \\\n            '2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。')\nprint(doc)\n```\n````{margin} **But what do these annotations mean?**\n```{seealso}\nSee our [data format](data_format) and [annotations](annotations/index) for details.\n```\n````\n\n\n## Visualization\n\n```{eval-rst}\nThe returned :class:`~hanlp_common.document.Document` has a handy method :meth:`~hanlp_common.document.Document.pretty_print` \nwhich offers visualization in any mono-width text environment. \n```\n\n````{margin} **Non-ASCII**\n```{note}\nNon-ASCII text might be skewed in terminals but in Jupyter Notebook it will align correctly. \nYou can also use our [live demo](https://hanlp.hankcs.com/).\n```\n````\n\n````{margin} **Non-Projective**\n```{note}\nNon-projective dependency trees cannot be visualized and won't be printed out at this moment.\n```\n````\n\n```{code-cell} ipython3\ndoc.pretty_print()\n```\n\n## Native API\n\n### Multi-Task Learning\n\nIf you want to run our models locally or you want to implement your own RESTful server, \nyou can [install the native API](https://hanlp.hankcs.com/docs/install.html#install-native-package) \nand call it just like the RESTful one.\n\n````{margin} **Sentences Required**\n```{seealso}\nAs MTL doesn't predict sentence boundaries, inputs have to be split beforehand. \nSee our [data format](data_format) for details.\n```\n````\n\n```{code-cell} ipython3\n:tags: [output_scroll]\nimport hanlp\nHanLP = hanlp.load(hanlp.pretrained.mtl.UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_XLMR_BASE)\nprint(HanLP(['In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques to production environments.',\n             '2021年、HanLPv2.1は次世代の最先端多言語NLP技術を本番環境に導入します。',\n             '2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。']))\n```\n\nDue to the fact that the service provider is very likely running a different model or having different settings, the\nRESTful and native results might be slightly different. \n\nTo process Chinese or Japanese, HanLP provides mono-lingual models in each language which significantly outperform the multi-lingual model. See [docs](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/mtl.html) for the list of models.\n\n### Single-Task Learning\n\nHanLP also provides a full spectrum of single-task learning models for core NLP tasks including tagging and parsing. Please refer to the documentations of  [`pretrained`](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/index.html) models for details."
  },
  {
    "path": "hanlp/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-06-13 18:05\nimport hanlp.common\nimport hanlp.components\nimport hanlp.pretrained\nimport hanlp.utils\nfrom hanlp.version import __version__\n\nhanlp.utils.ls_resource_in_module(hanlp.pretrained)\n\n\ndef load(save_dir: str, verbose=None, **kwargs) -> hanlp.common.component.Component:\n    \"\"\"Load a pretrained component from an identifier.\n\n    Args:\n      save_dir (str): The identifier to the saved component. It could be a remote URL or a local path.\n      verbose: ``True`` to print loading progress.\n      **kwargs: Arguments passed to :func:`hanlp.common.torch_component.TorchComponent.load`, e.g.,\n        ``devices`` is a useful argument to specify which GPU devices a PyTorch component will use.\n\n    Examples::\n\n        import hanlp\n        # Load component onto the 0-th GPU.\n        hanlp.load(..., devices=0)\n        # Load component onto the 0-th and 1-st GPUs using data parallelization.\n        hanlp.load(..., devices=[0, 1])\n\n    .. Note::\n        A component can have dependencies on other components or resources, which will be recursively loaded. So it's\n        common to see multiple downloading messages per single load.\n\n    Returns:\n      hanlp.common.component.Component: A pretrained component.\n\n    \"\"\"\n    save_dir = hanlp.pretrained.ALL.get(save_dir, save_dir)\n    from hanlp.utils.component_util import load_from_meta_file\n    if verbose is None:\n        from hanlp_common.constant import HANLP_VERBOSE\n        verbose = HANLP_VERBOSE\n    return load_from_meta_file(save_dir, 'meta.json', verbose=verbose, **kwargs)\n\n\ndef pipeline(*pipes) -> hanlp.components.pipeline.Pipeline:\n    \"\"\"Creates a pipeline of components. It's made for bundling `KerasComponents`. For `TorchComponent`, use\n    :class:`~hanlp.components.mtl.multi_task_learning.MultiTaskLearning` instead.\n\n    Args:\n      *pipes: Components if pre-defined any.\n\n    Returns:\n      hanlp.components.pipeline.Pipeline: A pipeline, which is a list of components in order.\n\n    \"\"\"\n    return hanlp.components.pipeline.Pipeline(*pipes)\n"
  },
  {
    "path": "hanlp/callbacks/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-05 02:10"
  },
  {
    "path": "hanlp/callbacks/fine_csv_logger.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-05 02:12\nimport copy\nfrom io import TextIOWrapper\nfrom typing import List\n\nimport numpy as np\nimport tensorflow as tf\n\n\nclass StreamTableFormatter(object):\n\n    def __init__(self) -> None:\n        super().__init__()\n        self.col_widths = None\n\n    def format_row(self, cells) -> List[str]:\n        if not isinstance(cells, list):\n            cells = list(cells)\n        if not self.col_widths:\n            self.col_widths = [0] * len([_ for _ in cells])\n        for i, c in enumerate(cells):\n            self.col_widths[i] = max(self.col_widths[i], len(self.format_cell(c, self.col_widths[i])))\n        return list(self.format_cell(cell, width) for cell, width in zip(cells, self.col_widths))\n\n    def format_cell(self, cell: str, min_width) -> str:\n        if isinstance(cell, (np.float32, np.float)):\n            return '{:>{}.4f}'.format(cell, min_width)\n        return '{:>{}}'.format(cell, min_width)\n\n\nclass FineCSVLogger(tf.keras.callbacks.History):\n\n    def __init__(self, filename, separator=',', append=False):\n        super().__init__()\n        self.append = append\n        self.separator = separator\n        self.filename = filename\n        self.out: TextIOWrapper = None\n        self.keys = []\n        self.formatter = StreamTableFormatter()\n\n    def on_train_begin(self, logs=None):\n        super().on_train_begin(logs)\n        self.out = open(self.filename, 'a' if self.append else 'w')\n\n    def on_train_end(self, logs=None):\n        self.out.close()\n\n    def on_epoch_end(self, epoch, logs=None):\n        super().on_epoch_end(epoch, logs)\n        if not self.keys:\n            self.keys = sorted(logs.keys())\n\n            if getattr(self.model, 'stop_training', None):\n                # We set NA so that csv parsers do not fail for this last epoch.\n                logs = dict([(k, logs[k]) if k in logs else (k, 'NA') for k in self.keys])\n\n            # feed them twice to decide the actual width\n            values = self.formatter.format_row([epoch + 1] + [logs.get(k, 'NA') for k in self.keys])\n            headers = self.formatter.format_row(['epoch'] + self.keys)\n            # print headers and bars\n            self.out.write(self.separator.join(headers) + '\\n')\n            # bars for markdown style\n            bars = [''.join(['-'] * width) for width in self.formatter.col_widths]\n            self.out.write(self.separator.join(bars) + '\\n')\n\n        values = self.formatter.format_row([epoch + 1] + [logs.get(k, 'NA') for k in self.keys])\n        self.out.write(self.separator.join(values) + '\\n')\n        self.out.flush()\n"
  },
  {
    "path": "hanlp/common/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-08-26 14:45\n"
  },
  {
    "path": "hanlp/common/component.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-08-26 14:45\nimport inspect\nfrom abc import ABC, abstractmethod\nfrom typing import Any\n\nfrom hanlp_common.configurable import Configurable\n\n\nclass Component(Configurable, ABC):\n    @abstractmethod\n    def predict(self, *args, **kwargs):\n        \"\"\"Predict on data. This is the base class for all components, including rule based and statistical ones.\n\n        Args:\n          *args: Any type of data subject to sub-classes\n          **kwargs: Additional arguments\n\n        Returns: Any predicted annotations.\n\n        \"\"\"\n        raise NotImplementedError('%s.%s()' % (self.__class__.__name__, inspect.stack()[0][3]))\n\n    def __call__(self, *args, **kwargs):\n        \"\"\"\n        A shortcut for :func:`~hanlp.common.component.predict`.\n\n        Args:\n          *args: Any type of data subject to sub-classes\n          **kwargs: Additional arguments\n\n        Returns: Any predicted annotations.\n\n        \"\"\"\n        return self.predict(*args, **kwargs)\n"
  },
  {
    "path": "hanlp/common/dataset.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-09 20:27\nimport math\nimport os\nimport random\nimport tempfile\nimport warnings\nfrom abc import ABC, abstractmethod\nfrom copy import copy\nfrom logging import Logger\nfrom typing import Union, List, Callable, Iterable, Dict, Any\n\nimport torch\nimport torch.multiprocessing as mp\nfrom hanlp.common.transform import TransformList, VocabDict, EmbeddingNamedTransform\nfrom hanlp.common.vocab import Vocab\nfrom hanlp.components.parsers.alg import kmeans\nfrom hanlp.utils.io_util import read_cells, get_resource\nfrom hanlp.utils.time_util import CountdownTimer\nfrom hanlp.utils.torch_util import dtype_of\nfrom hanlp_common.configurable import AutoConfigurable\nfrom hanlp_common.constant import IDX, HANLP_VERBOSE\nfrom hanlp_common.util import isdebugging, merge_list_of_dict, k_fold\nfrom torch.nn.utils.rnn import pad_sequence\nfrom torch.utils.data import Dataset, DataLoader, Sampler\nfrom torch.utils.data.dataset import IterableDataset\n\n\nclass Transformable(ABC):\n    def __init__(self, transform: Union[Callable, List] = None) -> None:\n        \"\"\"An object which can be transformed with a list of functions. It is the final result of an object being passed\n        through a list of functions, while these functions are kept in a list.\n\n        Args:\n            transform: A transform function or a list of functions.\n        \"\"\"\n        super().__init__()\n        if isinstance(transform, list) and not isinstance(transform, TransformList):\n            transform = TransformList(*transform)\n        self.transform: Union[Callable, TransformList] = transform\n\n    def append_transform(self, transform: Callable):\n        \"\"\"Append a transform to its list of transforms.\n\n        Args:\n            transform: A new transform to be appended.\n\n        Returns:\n            Itself.\n\n        \"\"\"\n        assert transform is not None, 'None transform not allowed'\n        if not self.transform:\n            self.transform = TransformList(transform)\n        elif not isinstance(self.transform, TransformList):\n            if self.transform != transform:\n                self.transform = TransformList(self.transform, transform)\n        else:\n            if transform not in self.transform:\n                self.transform.append(transform)\n        return self\n\n    def insert_transform(self, index: int, transform: Callable):\n        \"\"\"Insert a transform to a certain position.\n\n        Args:\n            index: A certain position.\n            transform: A new transform.\n\n        Returns:\n            Itself.\n\n        \"\"\"\n        assert transform is not None, 'None transform not allowed'\n        if not self.transform:\n            self.transform = TransformList(transform)\n        elif not isinstance(self.transform, TransformList):\n            if self.transform != transform:\n                self.transform = TransformList(self.transform)\n                self.transform.insert(index, transform)\n        else:\n            if transform not in self.transform:\n                self.transform.insert(index, transform)\n        return self\n\n    def transform_sample(self, sample: dict, inplace=False) -> dict:\n        \"\"\"Apply transforms to a sample.\n\n        Args:\n            sample: A sample, which is a ``dict`` holding features.\n            inplace: ``True`` to apply transforms inplace.\n\n        .. Attention::\n            If any transform modifies existing features, it will modify again and again when ``inplace=True``.\n            For example, if a transform insert a ``BOS`` token to a list inplace, and it is called twice,\n            then 2 ``BOS`` will be inserted which might not be an intended result.\n\n        Returns:\n            Transformed sample.\n        \"\"\"\n        if not inplace:\n            sample = copy(sample)\n        if self.transform:\n            sample = self.transform(sample)\n        return sample\n\n\nclass TransformableDataset(Transformable, Dataset, ABC):\n\n    def __init__(self,\n                 data: Union[str, List],\n                 transform: Union[Callable, List] = None,\n                 cache=None,\n                 generate_idx=None) -> None:\n        \"\"\"A :class:`~torch.utils.data.Dataset` which can be applied with a list of transform functions.\n\n        Args:\n            data: The local or remote path to a dataset, or a list of samples where each sample is a dict.\n            transform: Predefined transform(s).\n            cache: ``True`` to enable caching, so that transforms won't be called twice.\n            generate_idx: Create a :const:`~hanlp_common.constants.IDX` field for each sample to store its order in dataset. Useful for prediction when\n                samples are re-ordered by a sampler.\n        \"\"\"\n        super().__init__(transform)\n        if generate_idx is None:\n            generate_idx = isinstance(data, list)\n        data_ = self.load_data(data, generate_idx)\n        # assert data_, f'No samples loaded from {data}'\n        if data_:\n            assert isinstance(data_[0], dict\n                              ), f'TransformDataset expects each sample to be a dict but got {type(data_[0])} instead.'\n        self.data = data_\n        if cache:\n            self.cache = [None] * len(data_)\n        else:\n            self.cache = None\n\n    def load_data(self, data, generate_idx=False):\n        \"\"\"A intermediate step between constructor and calling the actual file loading method.\n\n        Args:\n            data: If data is a file, this method calls :meth:`~hanlp.common.dataset.TransformableDataset.load_file`\n                to load it.\n            generate_idx: Create a :const:`~hanlp_common.constants.IDX` field for each sample to store its order in dataset. Useful for prediction when\n                samples are re-ordered by a sampler.\n\n        Returns: Loaded samples.\n\n        \"\"\"\n        if self.should_load_file(data):\n            if isinstance(data, str):\n                data = get_resource(data)\n            data = list(self.load_file(data))\n        if generate_idx:\n            for i, each in enumerate(data):\n                each[IDX] = i\n        # elif isinstance(data, list):\n        #     data = self.load_list(data)\n        return data\n\n    # noinspection PyMethodMayBeStatic\n    # def load_list(self, data: list) -> List[Dict[str, Any]]:\n    #     return data\n\n    def should_load_file(self, data) -> bool:\n        \"\"\"Determines whether data is a filepath.\n\n        Args:\n            data: Data to check.\n\n        Returns: ``True`` to indicate it's a filepath.\n\n        \"\"\"\n        return isinstance(data, str)\n\n    @abstractmethod\n    def load_file(self, filepath: str):\n        \"\"\"The actual file loading logic.\n\n        Args:\n            filepath: The path to a dataset.\n        \"\"\"\n        pass\n\n    def __getitem__(self, index: Union[int, slice]) -> Union[dict, List[dict]]:\n        \"\"\" Get the index-th sample in this dataset.\n\n        Args:\n            index: Either a integer index of a list of indices.\n\n        Returns: Either a sample or or list of samples depending on how many indices are passed in.\n\n        \"\"\"\n        # if isinstance(index, (list, tuple)):\n        #     assert len(index) == 1\n        #     index = index[0]\n        if isinstance(index, slice):\n            indices = range(*index.indices(len(self)))\n            return [self[i] for i in indices]\n\n        if self.cache:\n            cache = self.cache[index]\n            if cache:\n                return cache\n        sample = self.data[index]\n        sample = self.transform_sample(sample)\n        if self.cache:\n            self.cache[index] = sample\n        return sample\n\n    def __len__(self) -> int:\n        return len(self.data)\n\n    def __repr__(self) -> str:\n        return f'{len(self)} samples: {self[0] if len(self) else \"\"} ...'\n\n    def purge_cache(self):\n        \"\"\"Purges all cache. If cache is not enabled, this method enables it.\n        \"\"\"\n        self.cache = [None] * len(self.data)\n\n    def split(self, *ratios):\n        \"\"\"Split dataset into subsets.\n\n        Args:\n            *ratios: The ratios for each subset. They can be any type of numbers which will be normalized. For example,\n                    ``8, 1, 1`` are equivalent to ``0.8, 0.1, 0.1``.\n\n        Returns:\n            list[TransformableDataset]: A list of subsets.\n        \"\"\"\n        ratios = [x / sum(ratios) for x in ratios]\n        chunks = []\n        prev = 0\n        for r in ratios:\n            cur = prev + math.ceil(len(self) * r)\n            chunks.append([prev, cur])\n            prev = cur\n        chunks[-1][1] = len(self)\n        outputs = []\n        for b, e in chunks:\n            dataset = copy(self)\n            dataset.data = dataset.data[b:e]\n            if dataset.cache:\n                dataset.cache = dataset.cache[b:e]\n            outputs.append(dataset)\n        return outputs\n\n    def k_fold(self, k, i):\n        \"\"\"Perform k-fold sampling.\n\n        Args:\n            k (int): Number of folds.\n            i (int): The i-th fold.\n\n        Returns:\n            TransformableDataset: The i-th fold subset of this dataset.\n\n        \"\"\"\n        assert 0 <= i <= k, f'Invalid split {i}'\n        train_indices, test_indices = k_fold(k, len(self), i)\n        return self.subset(train_indices), self.subset(test_indices)\n\n    def subset(self, indices):\n        \"\"\"Create a subset given indices of samples.\n\n        Args:\n            indices: Indices of samples.\n\n        Returns:\n            TransformableDataset: The a subset of this dataset.\n        \"\"\"\n        dataset = copy(self)\n        dataset.data = [dataset.data[i] for i in indices]\n        if dataset.cache:\n            dataset.cache = [dataset.cache[i] for i in indices]\n        return dataset\n\n    def shuffle(self):\n        \"\"\"Shuffle this dataset inplace.\n        \"\"\"\n        if not self.cache:\n            random.shuffle(self.data)\n        else:\n            z = list(zip(self.data, self.cache))\n            random.shuffle(z)\n            self.data, self.cache = zip(*z)\n\n    def prune(self, criterion: Callable, logger: Logger = None):\n        \"\"\"Prune (to discard) samples according to a criterion.\n\n        Args:\n            criterion: A functions takes a sample as input and output ``True`` if the sample needs to be pruned.\n            logger: If any, log statistical messages using it.\n\n        Returns:\n            int: Size before pruning.\n        \"\"\"\n        # noinspection PyTypeChecker\n        size_before = len(self)\n        good_ones = [i for i, s in enumerate(self) if not criterion(s)]\n        self.data = [self.data[i] for i in good_ones]\n        if self.cache:\n            self.cache = [self.cache[i] for i in good_ones]\n        if logger:\n            size_after = len(self)\n            num_pruned = size_before - size_after\n            logger.info(f'Pruned [yellow]{num_pruned} ({num_pruned / size_before:.1%})[/yellow] '\n                        f'samples out of {size_before}.')\n        return size_before\n\n\nclass TransformSequentialDataset(Transformable, IterableDataset, ABC):\n    pass\n\n\nclass DeviceDataLoader(DataLoader):\n    def __init__(self, dataset, batch_size=32, shuffle=False, sampler=None,\n                 batch_sampler=None, num_workers=None, collate_fn=None,\n                 pin_memory=False, drop_last=False, timeout=0,\n                 worker_init_fn=None, multiprocessing_context=None,\n                 device=None, **kwargs):\n        if batch_sampler is not None:\n            batch_size = 1\n        if num_workers is None:\n            if isdebugging():\n                num_workers = 0\n            else:\n                num_workers = 2\n        # noinspection PyArgumentList\n        super(DeviceDataLoader, self).__init__(dataset=dataset, batch_size=batch_size, shuffle=shuffle,\n                                               sampler=sampler,\n                                               batch_sampler=batch_sampler, num_workers=num_workers,\n                                               collate_fn=collate_fn,\n                                               pin_memory=pin_memory, drop_last=drop_last, timeout=timeout,\n                                               worker_init_fn=worker_init_fn,\n                                               multiprocessing_context=multiprocessing_context, **kwargs)\n        self.device = device\n\n    def __iter__(self):\n        for raw_batch in super(DeviceDataLoader, self).__iter__():\n            if self.device is not None:\n                for field, data in raw_batch.items():\n                    if isinstance(data, torch.Tensor):\n                        data = data.to(self.device)\n                        raw_batch[field] = data\n            yield raw_batch\n\n    def collate_fn(self, samples):\n        return merge_list_of_dict(samples)\n\n\nclass PadSequenceDataLoader(DataLoader):\n\n    def __init__(self, dataset, batch_size=32, shuffle=False, sampler=None,\n                 batch_sampler=None, num_workers=0, collate_fn=None,\n                 pin_memory=False, drop_last=False, timeout=0,\n                 worker_init_fn=None, multiprocessing_context=None,\n                 pad: dict = None, vocabs: VocabDict = None, device=None, **kwargs):\n        \"\"\" A dataloader commonly used for NLP tasks. It offers the following convenience.\n\n        - Bachify each field of samples into a :class:`~torch.Tensor` if the field name satisfies the following criterion.\n            - Name ends with _id, _ids, _count, _offset, _span, mask\n            - Name is in `pad` dict.\n\n        - Pad each field according to field name, the vocabs and pad dict.\n        - Move :class:`~torch.Tensor` onto device.\n\n        Args:\n            dataset: A :class:`~torch.utils.data.Dataset` to be bachified.\n            batch_size: Max size of each batch.\n            shuffle: ``True`` to shuffle batches.\n            sampler: A :class:`~torch.utils.data.Sampler` to sample samples from data.\n            batch_sampler: A :class:`~torch.utils.data.Sampler` to sample batches form all batches.\n            num_workers: Number of workers for multi-thread loading. Note that multi-thread loading aren't always\n                faster.\n            collate_fn: A function to perform batchifying. It must be set to ``None`` in order to make use of the\n                 features this class offers.\n            pin_memory: If samples are loaded in the Dataset on CPU and would like to be pushed to\n                    the GPU, enabling pin_memory can speed up the transfer. It's not useful since most data field are\n                    not in Tensor type.\n            drop_last: Drop the last batch since it could be half-empty.\n            timeout: For multi-worker loading, set a timeout to wait for a worker.\n            worker_init_fn: Init function for multi-worker.\n            multiprocessing_context: Context for multiprocessing.\n            pad: A dict holding field names and their padding values.\n            vocabs: A dict of vocabs so padding value can be fetched from it.\n            device: The device tensors will be moved onto.\n            **kwargs: Other arguments will be passed to :meth:`torch.utils.data.Dataset.__init__`\n        \"\"\"\n        if device == -1:\n            device = None\n        if collate_fn is None:\n            collate_fn = self.collate_fn\n        if num_workers is None:\n            if isdebugging():\n                num_workers = 0\n            else:\n                num_workers = 2\n        if batch_sampler is None:\n            assert batch_size, 'batch_size has to be specified when batch_sampler is None'\n        else:\n            batch_size = 1\n            shuffle = None\n            drop_last = None\n        # noinspection PyArgumentList\n        super(PadSequenceDataLoader, self).__init__(dataset=dataset, batch_size=batch_size, shuffle=shuffle,\n                                                    sampler=sampler,\n                                                    batch_sampler=batch_sampler, num_workers=num_workers,\n                                                    collate_fn=collate_fn,\n                                                    pin_memory=pin_memory, drop_last=drop_last, timeout=timeout,\n                                                    worker_init_fn=worker_init_fn,\n                                                    multiprocessing_context=multiprocessing_context, **kwargs)\n        self.vocabs = vocabs\n        if isinstance(dataset, TransformableDataset) and dataset.transform:\n            transform = dataset.transform\n            if not isinstance(transform, TransformList):\n                transform = []\n            for each in transform:\n                if isinstance(each, EmbeddingNamedTransform):\n                    if pad is None:\n                        pad = {}\n                    if each.dst not in pad:\n                        pad[each.dst] = 0\n        self.pad = pad\n        self.device = device\n\n    def __iter__(self):\n        for raw_batch in super(PadSequenceDataLoader, self).__iter__():\n            yield self.tensorize(raw_batch, vocabs=self.vocabs, pad_dict=self.pad, device=self.device)\n\n    @staticmethod\n    def tensorize(raw_batch: Dict[str, Any], vocabs: VocabDict, pad_dict: Dict[str, int] = None, device=None):\n        for field, data in raw_batch.items():\n            if isinstance(data, torch.Tensor):\n                continue\n            vocab_key = field[:-len('_id')] if field.endswith('_id') else None\n            vocab: Vocab = vocabs.get(vocab_key, None) if vocabs and vocab_key else None\n            if vocab:\n                pad = vocab.safe_pad_token_idx\n                dtype = torch.long\n            elif pad_dict is not None and pad_dict.get(field, None) is not None:\n                pad = pad_dict[field]\n                dtype = dtype_of(pad)\n            elif field.endswith('_offset') or field.endswith('_id') or field.endswith(\n                    '_count') or field.endswith('_ids') or field.endswith('_score') or field.endswith(\n                '_length') or field.endswith('_span'):\n                # guess some common fields to pad\n                pad = 0\n                dtype = torch.long\n            elif field.endswith('_mask'):\n                pad = False\n                dtype = torch.bool\n            else:\n                # no need to pad\n                continue\n            data = PadSequenceDataLoader.pad_data(data, pad, dtype)\n            raw_batch[field] = data\n        if device is not None:\n            for field, data in raw_batch.items():\n                if isinstance(data, torch.Tensor):\n                    data = data.to(device)\n                    raw_batch[field] = data\n        return raw_batch\n\n    @staticmethod\n    def pad_data(data: Union[torch.Tensor, Iterable], pad, dtype=None, device=None):\n        \"\"\"Perform the actual padding for a given data.\n\n        Args:\n            data: Data to be padded.\n            pad: Padding value.\n            dtype: Data type.\n            device: Device to be moved onto.\n\n        Returns:\n            torch.Tensor: A ``torch.Tensor``.\n        \"\"\"\n        if isinstance(data[0], torch.Tensor):\n            data = pad_sequence(data, True, pad)\n        elif isinstance(data[0], Iterable):\n            inner_is_iterable = False\n            for each in data:\n                if len(each):\n                    if isinstance(each[0], Iterable):\n                        inner_is_iterable = True\n                        if len(each[0]):\n                            if not dtype:\n                                dtype = dtype_of(each[0][0])\n                    else:\n                        inner_is_iterable = False\n                        if not dtype:\n                            dtype = dtype_of(each[0])\n                    break\n            if inner_is_iterable:\n                max_seq_len = len(max(data, key=len))\n                max_word_len = len(max([chars for words in data for chars in words], key=len))\n                ids = torch.zeros(len(data), max_seq_len, max_word_len, dtype=dtype, device=device)\n                for i, words in enumerate(data):\n                    for j, chars in enumerate(words):\n                        ids[i][j][:len(chars)] = torch.tensor(chars, dtype=dtype, device=device)\n                data = ids\n            else:\n                data = pad_sequence([torch.tensor(x, dtype=dtype, device=device) for x in data], True, pad)\n        elif isinstance(data, list):\n            data = torch.tensor(data, dtype=dtype, device=device)\n        return data\n\n    def collate_fn(self, samples):\n        return merge_list_of_dict(samples)\n\n\nclass CachedDataLoader(object):\n    def __init__(self, dataloader: torch.utils.data.DataLoader, filename=None):\n        if not filename:\n            filename = tempfile.NamedTemporaryFile(prefix='hanlp-cache-', delete=False).name\n        self.filename = filename\n        self.size = len(dataloader)\n        self._build_cache(dataloader)\n\n    def _build_cache(self, dataset, verbose=HANLP_VERBOSE):\n        timer = CountdownTimer(self.size)\n        with open(self.filename, \"wb\") as f:\n            for i, batch in enumerate(dataset):\n                torch.save(batch, f, _use_new_zipfile_serialization=False)\n                if verbose:\n                    timer.log(f'Caching {self.filename} [blink][yellow]...[/yellow][/blink]')\n\n    def close(self):\n        if os.path.isfile(self.filename):\n            os.remove(self.filename)\n\n    def __iter__(self):\n        with open(self.filename, \"rb\") as f:\n            for i in range(self.size):\n                batch = torch.load(f)\n                yield batch\n\n    def __len__(self):\n        return self.size\n\n\ndef _prefetch_generator(dataloader, queue, batchify=None):\n    while True:\n        for batch in dataloader:\n            if batchify:\n                batch = batchify(batch)\n            queue.put(batch)\n\n\nclass PrefetchDataLoader(DataLoader):\n    def __init__(self, dataloader: torch.utils.data.DataLoader, prefetch: int = 10, batchify: Callable = None) -> None:\n        \"\"\" A dataloader wrapper which speeds up bachifying using multi-processing. It works best for dataloaders\n        of which the bachify takes very long time. But it introduces extra GPU memory consumption since prefetched\n        batches are stored in a ``Queue`` on GPU.\n\n        .. Caution::\n\n            PrefetchDataLoader only works in spawn mode with the following initialization code:\n\n            Examples::\n\n                if __name__ == '__main__':\n                    import torch\n\n                    torch.multiprocessing.set_start_method('spawn')\n\n            And these 2 lines **MUST** be put into ``if __name__ == '__main__':`` block.\n\n        Args:\n            dataloader: A :class:`~torch.utils.data.DatasetLoader` to be prefetched.\n            prefetch: Number of batches to prefetch.\n            batchify: A bachify function called on each batch of samples. In which case, the inner dataloader shall\n                    return samples without really bachify them.\n        \"\"\"\n        super().__init__(dataset=dataloader)\n        self._batchify = batchify\n        self.prefetch = None if isdebugging() else prefetch\n        if self.prefetch:\n            self._fire_process(dataloader, prefetch)\n\n    def _fire_process(self, dataloader, prefetch):\n        self.queue = mp.Queue(prefetch)\n        self.process = mp.Process(target=_prefetch_generator, args=(dataloader, self.queue, self._batchify))\n        self.process.start()\n\n    def __iter__(self):\n        if not self.prefetch:\n            for batch in self.dataset:\n                if self._batchify:\n                    batch = self._batchify(batch)\n                yield batch\n        else:\n            size = len(self)\n            while size:\n                batch = self.queue.get()\n                yield batch\n                size -= 1\n\n    def close(self):\n        \"\"\"Close this dataloader and terminates internal processes and queue. It's recommended to call this method to\n            ensure a program can gracefully shutdown.\n        \"\"\"\n        if self.prefetch:\n            self.queue.close()\n            self.process.terminate()\n\n    @property\n    def batchify(self):\n        return self._batchify\n\n    @batchify.setter\n    def batchify(self, batchify):\n        self._batchify = batchify\n        if not self.prefetch:\n            prefetch = vars(self.queue).get('maxsize', 10)\n            self.close()\n            self._fire_process(self.dataset, prefetch)\n\n\nclass BucketSampler(Sampler):\n    # noinspection PyMissingConstructor\n    def __init__(self, buckets: Dict[float, List[int]], batch_max_tokens, batch_size=None, shuffle=False):\n        \"\"\"A bucketing based sampler which groups samples into buckets then creates batches from each bucket.\n\n        Args:\n            buckets: A dict of which keys are some statistical numbers of each bucket, and values are the indices of\n                samples in each bucket.\n            batch_max_tokens: Maximum tokens per batch.\n            batch_size: Maximum samples per batch.\n            shuffle: ``True`` to shuffle batches and samples in a batch.\n        \"\"\"\n        self.shuffle = shuffle\n        self.sizes, self.buckets = zip(*[\n            (size, bucket) for size, bucket in buckets.items()\n        ])\n        # the number of chunks in each bucket, which is clipped by\n        # range [1, len(bucket)]\n        if batch_size:\n            self.chunks = [\n                max(batch_size, min(len(bucket), max(round(size * len(bucket) / batch_max_tokens), 1)))\n                for size, bucket in zip(self.sizes, self.buckets)\n            ]\n        else:\n            self.chunks = [\n                min(len(bucket), max(round(size * len(bucket) / batch_max_tokens), 1))\n                for size, bucket in zip(self.sizes, self.buckets)\n            ]\n\n    def __iter__(self):\n        # if shuffle, shuffle both the buckets and samples in each bucket\n        range_fn = torch.randperm if self.shuffle else torch.arange\n        for i in range_fn(len(self.buckets)).tolist():\n            split_sizes = [(len(self.buckets[i]) - j - 1) // self.chunks[i] + 1 for j in range(self.chunks[i])]\n            # DON'T use `torch.chunk` which may return wrong number of chunks\n            for batch in range_fn(len(self.buckets[i])).split(split_sizes):\n                yield [self.buckets[i][j] for j in batch.tolist()]\n\n    def __len__(self):\n        return sum(self.chunks)\n\n\nclass KMeansSampler(BucketSampler):\n    def __init__(self, lengths, batch_max_tokens, batch_size=None, shuffle=False, n_buckets=1):\n        \"\"\"A bucket sampler which groups samples using KMeans on their lengths.\n\n        Args:\n            lengths: Lengths of each sample, usually measured by number of tokens.\n            batch_max_tokens: Maximum tokens per batch.\n            batch_size: Maximum samples per batch.\n            shuffle: ``True`` to shuffle batches. Samples in the same batch won't be shuffled since the ordered sequence\n                    is helpful to speed up RNNs.\n            n_buckets: Number of buckets. Clusters in terms of KMeans.\n        \"\"\"\n        if n_buckets > len(lengths):\n            n_buckets = 1\n        self.n_buckets = n_buckets\n        self.lengths = lengths\n        buckets = dict(zip(*kmeans(self.lengths, n_buckets)))\n        super().__init__(buckets, batch_max_tokens, batch_size, shuffle)\n\n\nclass SortingSampler(Sampler):\n    # noinspection PyMissingConstructor\n    def __init__(self, lengths: List[int], batch_size=None, batch_max_tokens=None, use_effective_tokens=False,\n                 shuffle=False) -> None:\n        \"\"\"A sampler which sorts samples according to their lengths. It takes a continuous chunk of sorted samples to\n        make a batch. The effective batch size is determined by ``batch_size``, ``batch_max_tokens`` and\n        ``use_effective_tokens``.\n\n        Args:\n            lengths: Lengths of each sample, usually measured by number of tokens.\n            batch_max_tokens: Maximum tokens per batch.\n            use_effective_tokens: Whether to calculate the effective number of tokens after padding when applying the\n                ``batch_max_tokens``.\n            batch_size: Maximum samples per batch.\n            shuffle: ``True`` to shuffle batches and samples in a batch.\n        \"\"\"\n        # assert any([batch_size, batch_max_tokens]), 'At least one of batch_size and batch_max_tokens is required'\n        self.shuffle = shuffle\n        self.batch_size = batch_size\n        # self.batch_max_tokens = batch_max_tokens\n        self.batch_indices = []\n        num_tokens = 0\n        mini_batch = []\n        for i in torch.argsort(torch.tensor(lengths), descending=True).tolist():\n            # if batch_max_tokens:\n            effective_tokens = lengths[i] if (not mini_batch or not use_effective_tokens) else lengths[mini_batch[0]]\n            if (batch_max_tokens is None or num_tokens + effective_tokens <= batch_max_tokens) and (\n                    batch_size is None or len(mini_batch) < batch_size):\n                mini_batch.append(i)\n                num_tokens += effective_tokens\n            else:\n                if not mini_batch:  # this sequence is longer than  batch_max_tokens\n                    mini_batch.append(i)\n                    self.batch_indices.append(mini_batch)\n                    mini_batch = []\n                    num_tokens = 0\n                else:\n                    self.batch_indices.append(mini_batch)\n                    mini_batch = [i]\n                    num_tokens = effective_tokens\n        if mini_batch:\n            self.batch_indices.append(mini_batch)\n        # print(len(max(self.batch_indices, key=len)))\n\n    def __iter__(self):\n        if self.shuffle:\n            random.shuffle(self.batch_indices)\n        for batch in self.batch_indices:\n            yield batch\n\n    def __len__(self) -> int:\n        return len(self.batch_indices)\n\n\nclass SamplerBuilder(AutoConfigurable, ABC):\n    @abstractmethod\n    def build(self, lengths: List[int], shuffle=False, gradient_accumulation=1, **kwargs) -> Sampler:\n        \"\"\"Build a ``Sampler`` given statistics of samples and other arguments.\n\n        Args:\n            lengths: The lengths of samples.\n            shuffle: ``True`` to shuffle batches. Note samples in each mini-batch are not necessarily shuffled.\n            gradient_accumulation: Number of mini-batches per update step.\n            **kwargs: Other arguments to be passed to the constructor of the sampler.\n        \"\"\"\n        pass\n\n    def __call__(self, lengths: List[int], shuffle=False, **kwargs) -> Sampler:\n        return self.build(lengths, shuffle, **kwargs)\n\n    def scale(self, gradient_accumulation):\n        r\"\"\"Scale down the ``batch_size`` and ``batch_max_tokens`` to :math:`\\frac{1}{\\text{gradient_accumulation}}`\n        of them respectively.\n\n        Args:\n            gradient_accumulation: Number of mini-batches per update step.\n\n        Returns:\n            tuple(int,int): batch_size, batch_max_tokens\n        \"\"\"\n        batch_size = self.batch_size\n        batch_max_tokens = self.batch_max_tokens\n        if gradient_accumulation:\n            if batch_size:\n                batch_size //= gradient_accumulation\n            if batch_max_tokens:\n                batch_max_tokens //= gradient_accumulation\n        return batch_size, batch_max_tokens\n\n\nclass SortingSamplerBuilder(SortingSampler, SamplerBuilder):\n    # noinspection PyMissingConstructor\n    def __init__(self, batch_size=None, batch_max_tokens=None, use_effective_tokens=False) -> None:\n        \"\"\"Builds a :class:`~hanlp.common.dataset.SortingSampler`.\n\n        Args:\n            batch_max_tokens: Maximum tokens per batch.\n            use_effective_tokens: Whether to calculate effective number of tokens when applying the `batch_max_tokens`.\n            batch_size: Maximum samples per batch.\n        \"\"\"\n        self.use_effective_tokens = use_effective_tokens\n        self.batch_max_tokens = batch_max_tokens\n        self.batch_size = batch_size\n\n    def build(self, lengths: List[int], shuffle=False, gradient_accumulation=1, **kwargs) -> Sampler:\n        batch_size, batch_max_tokens = self.scale(gradient_accumulation)\n        return SortingSampler(lengths, batch_size, batch_max_tokens, shuffle)\n\n    def __len__(self) -> int:\n        return 1\n\n\nclass KMeansSamplerBuilder(KMeansSampler, SamplerBuilder):\n    # noinspection PyMissingConstructor\n    def __init__(self, batch_max_tokens, batch_size=None, n_buckets=1):\n        \"\"\"Builds a :class:`~hanlp.common.dataset.KMeansSampler`.\n\n        Args:\n            batch_max_tokens: Maximum tokens per batch.\n            batch_size: Maximum samples per batch.\n            n_buckets: Number of buckets. Clusters in terms of KMeans.\n        \"\"\"\n        self.n_buckets = n_buckets\n        self.batch_size = batch_size\n        self.batch_max_tokens = batch_max_tokens\n\n    def build(self, lengths: List[int], shuffle=False, gradient_accumulation=1, **kwargs) -> Sampler:\n        batch_size, batch_max_tokens = self.scale(gradient_accumulation)\n        return KMeansSampler(lengths, batch_max_tokens, batch_size, shuffle, self.n_buckets)\n\n    def __len__(self) -> int:\n        return 1\n\n\nclass TableDataset(TransformableDataset):\n    def __init__(self,\n                 data: Union[str, List],\n                 transform: Union[Callable, List] = None,\n                 cache=None,\n                 delimiter='auto',\n                 strip=True,\n                 headers=None) -> None:\n        self.headers = headers\n        self.strip = strip\n        self.delimiter = delimiter\n        super().__init__(data, transform, cache)\n\n    def load_file(self, filepath: str):\n        for idx, cells in enumerate(read_cells(filepath, strip=self.strip, delimiter=self.delimiter)):\n            if not idx and not self.headers:\n                self.headers = cells\n                if any(len(h) > 32 for h in self.headers):\n                    warnings.warn('As you did not pass in `headers` to `TableDataset`, the first line is regarded as '\n                                  'headers. However, the length for some headers are too long (>32), which might be '\n                                  'wrong. To make sure, pass `headers=...` explicitly.')\n            else:\n                yield dict(zip(self.headers, cells))\n"
  },
  {
    "path": "hanlp/common/keras_component.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-08-26 14:45\nimport logging\nimport math\nimport os\nimport sys\nfrom abc import ABC, abstractmethod\nfrom typing import Optional, List, Any, Dict\n\nimport numpy as np\nimport tensorflow as tf\n\nimport hanlp.utils\nfrom hanlp_common.io import save_json, load_json\nfrom hanlp.callbacks.fine_csv_logger import FineCSVLogger\nfrom hanlp.common.component import Component\nfrom hanlp.common.transform_tf import Transform\nfrom hanlp.common.vocab_tf import VocabTF\nfrom hanlp.metrics.chunking.iobes_tf import IOBES_F1_TF\nfrom hanlp.optimizers.adamw import AdamWeightDecay\nfrom hanlp.utils import io_util\nfrom hanlp.utils.io_util import get_resource, tempdir_human\nfrom hanlp.utils.log_util import init_logger, logger\nfrom hanlp.utils.string_util import format_scores\nfrom hanlp.utils.tf_util import format_metrics, size_of_dataset, summary_of_model, get_callback_by_class, NumpyEncoder\nfrom hanlp.utils.time_util import Timer, now_datetime\nfrom hanlp_common.reflection import str_to_type, classpath_of\nfrom hanlp_common.structure import SerializableDict\nfrom hanlp_common.util import merge_dict\n\n\nclass KerasComponent(Component, ABC):\n    def __init__(self, transform: Transform) -> None:\n        super().__init__()\n        self.meta = {\n            'class_path': classpath_of(self),\n            'hanlp_version': hanlp.version.__version__,\n        }\n        self.model: Optional[tf.keras.Model] = None\n        self.config = SerializableDict()\n        self.transform = transform\n        # share config with transform for convenience, so we don't need to pass args around\n        if self.transform.config:\n            for k, v in self.transform.config.items():\n                self.config[k] = v\n        self.transform.config = self.config\n\n    def evaluate(self, input_path: str, save_dir=None, output=False, batch_size=128, logger: logging.Logger = None,\n                 callbacks: List[tf.keras.callbacks.Callback] = None, warm_up=True, verbose=True, **kwargs):\n        input_path = get_resource(input_path)\n        file_prefix, ext = os.path.splitext(input_path)\n        name = os.path.basename(file_prefix)\n        if not name:\n            name = 'evaluate'\n        if save_dir and not logger:\n            logger = init_logger(name=name, root_dir=save_dir, level=logging.INFO if verbose else logging.WARN,\n                                 mode='w')\n        tst_data = self.transform.file_to_dataset(input_path, batch_size=batch_size)\n        samples = self.num_samples_in(tst_data)\n        num_batches = math.ceil(samples / batch_size)\n        if warm_up:\n            for x, y in tst_data:\n                self.model.predict_on_batch(x)\n                break\n        if output:\n            assert save_dir, 'Must pass save_dir in order to output'\n            if isinstance(output, bool):\n                output = os.path.join(save_dir, name) + '.predict' + ext\n            elif isinstance(output, str):\n                output = output\n            else:\n                raise RuntimeError('output ({}) must be of type bool or str'.format(repr(output)))\n        timer = Timer()\n        eval_outputs = self.evaluate_dataset(tst_data, callbacks, output, num_batches, **kwargs)\n        loss, score, output = eval_outputs[0], eval_outputs[1], eval_outputs[2]\n        delta_time = timer.stop()\n        speed = samples / delta_time.delta_seconds\n\n        if logger:\n            f1: IOBES_F1_TF = None\n            for metric in self.model.metrics:\n                if isinstance(metric, IOBES_F1_TF):\n                    f1 = metric\n                    break\n            extra_report = ''\n            if f1:\n                overall, by_type, extra_report = f1.state.result(full=True, verbose=False)\n                extra_report = ' \\n' + extra_report\n            logger.info('Evaluation results for {} - '\n                        'loss: {:.4f} - {} - speed: {:.2f} sample/sec{}'\n                        .format(name + ext, loss,\n                                format_scores(score) if isinstance(score, dict) else format_metrics(self.model.metrics),\n                                speed, extra_report))\n        if output:\n            logger.info('Saving output to {}'.format(output))\n            with open(output, 'w', encoding='utf-8') as out:\n                self.evaluate_output(tst_data, out, num_batches, self.model.metrics)\n\n        return loss, score, speed\n\n    def num_samples_in(self, dataset):\n        return size_of_dataset(dataset)\n\n    def evaluate_dataset(self, tst_data, callbacks, output, num_batches, **kwargs):\n        loss, score = self.model.evaluate(tst_data, callbacks=callbacks, steps=num_batches)\n        return loss, score, output\n\n    def evaluate_output(self, tst_data, out, num_batches, metrics: List[tf.keras.metrics.Metric]):\n        # out.write('x\\ty_true\\ty_pred\\n')\n        for metric in metrics:\n            metric.reset_states()\n        for idx, batch in enumerate(tst_data):\n            outputs = self.model.predict_on_batch(batch[0])\n            for metric in metrics:\n                metric(batch[1], outputs, outputs._keras_mask if hasattr(outputs, '_keras_mask') else None)\n            self.evaluate_output_to_file(batch, outputs, out)\n            print('\\r{}/{} {}'.format(idx + 1, num_batches, format_metrics(metrics)), end='')\n        print()\n\n    def evaluate_output_to_file(self, batch, outputs, out):\n        for x, y_gold, y_pred in zip(self.transform.X_to_inputs(batch[0]),\n                                     self.transform.Y_to_outputs(batch[1], gold=True),\n                                     self.transform.Y_to_outputs(outputs, gold=False)):\n            out.write(self.transform.input_truth_output_to_str(x, y_gold, y_pred))\n\n    def _capture_config(self, config: Dict,\n                        exclude=(\n                                'trn_data', 'dev_data', 'save_dir', 'kwargs', 'self', 'logger', 'verbose',\n                                'dev_batch_size', '__class__')):\n        \"\"\"\n        Save arguments to config\n\n        Parameters\n        ----------\n        config\n            `locals()`\n        exclude\n        \"\"\"\n        if 'kwargs' in config:\n            config.update(config['kwargs'])\n        config = dict(\n            (key, tf.keras.utils.serialize_keras_object(value)) if hasattr(value, 'get_config') else (key, value) for\n            key, value in config.items())\n        for key in exclude:\n            config.pop(key, None)\n        self.config.update(config)\n\n    def save_meta(self, save_dir, filename='meta.json', **kwargs):\n        self.meta['create_time']: now_datetime()\n        self.meta.update(kwargs)\n        save_json(self.meta, os.path.join(save_dir, filename))\n\n    def load_meta(self, save_dir, filename='meta.json'):\n        save_dir = get_resource(save_dir)\n        metapath = os.path.join(save_dir, filename)\n        if os.path.isfile(metapath):\n            self.meta.update(load_json(metapath))\n\n    def save_config(self, save_dir, filename='config.json'):\n        self.config.save_json(os.path.join(save_dir, filename))\n\n    def load_config(self, save_dir, filename='config.json'):\n        save_dir = get_resource(save_dir)\n        self.config.load_json(os.path.join(save_dir, filename))\n\n    def save_weights(self, save_dir, filename='model.h5'):\n        self.model.save_weights(os.path.join(save_dir, filename))\n\n    def load_weights(self, save_dir, filename='model.h5', **kwargs):\n        assert self.model.built or self.model.weights, 'You must call self.model.built() in build_model() ' \\\n                                                       'in order to load it'\n        save_dir = get_resource(save_dir)\n        self.model.load_weights(os.path.join(save_dir, filename))\n\n    def save_vocabs(self, save_dir, filename='vocabs.json'):\n        vocabs = SerializableDict()\n        for key, value in vars(self.transform).items():\n            if isinstance(value, VocabTF):\n                vocabs[key] = value.to_dict()\n        vocabs.save_json(os.path.join(save_dir, filename))\n\n    def load_vocabs(self, save_dir, filename='vocabs.json'):\n        save_dir = get_resource(save_dir)\n        vocabs = SerializableDict()\n        vocabs.load_json(os.path.join(save_dir, filename))\n        for key, value in vocabs.items():\n            vocab = VocabTF()\n            vocab.copy_from(value)\n            setattr(self.transform, key, vocab)\n\n    def load_transform(self, save_dir) -> Transform:\n        \"\"\"\n        Try to load transform only. This method might fail due to the fact it avoids building the model.\n        If it do fail, then you have to use `load` which might be too heavy but that's the best we can do.\n        :param save_dir: The path to load.\n        \"\"\"\n        save_dir = get_resource(save_dir)\n        self.load_config(save_dir)\n        self.load_vocabs(save_dir)\n        self.transform.build_config()\n        self.transform.lock_vocabs()\n        return self.transform\n\n    def save(self, save_dir: str, **kwargs):\n        self.save_config(save_dir)\n        self.save_vocabs(save_dir)\n        self.save_weights(save_dir)\n\n    def load(self, save_dir: str, logger=hanlp.utils.log_util.logger, **kwargs):\n        self.meta['load_path'] = save_dir\n        save_dir = get_resource(save_dir)\n        self.load_config(save_dir)\n        self.load_vocabs(save_dir)\n        self.build(**merge_dict(self.config, training=False, logger=logger, **kwargs, overwrite=True, inplace=True))\n        self.load_weights(save_dir, **kwargs)\n        self.load_meta(save_dir)\n\n    @property\n    def input_shape(self) -> List:\n        return self.transform.output_shapes[0]\n\n    def build(self, logger, **kwargs):\n        self.transform.build_config()\n        self.model = self.build_model(**merge_dict(self.config, training=kwargs.get('training', None),\n                                                   loss=kwargs.get('loss', None)))\n        self.transform.lock_vocabs()\n        optimizer = self.build_optimizer(**self.config)\n        loss = self.build_loss(\n            **self.config if 'loss' in self.config else dict(list(self.config.items()) + [('loss', None)]))\n        # allow for different\n        metrics = self.build_metrics(**merge_dict(self.config, metrics=kwargs.get('metrics', 'accuracy'),\n                                                  logger=logger, overwrite=True))\n        if not isinstance(metrics, list):\n            if isinstance(metrics, tf.keras.metrics.Metric):\n                metrics = [metrics]\n        if not self.model.built:\n            sample_inputs = self.sample_data\n            if sample_inputs is not None:\n                self.model(sample_inputs)\n            else:\n                if len(self.transform.output_shapes[0]) == 1 and self.transform.output_shapes[0][0] is None:\n                    x_shape = self.transform.output_shapes[0]\n                else:\n                    x_shape = list(self.transform.output_shapes[0])\n                    for i, shape in enumerate(x_shape):\n                        x_shape[i] = [None] + shape  # batch + X.shape\n                self.model.build(input_shape=x_shape)\n        self.compile_model(optimizer, loss, metrics)\n        return self.model, optimizer, loss, metrics\n\n    def compile_model(self, optimizer, loss, metrics):\n        try:\n            self.model.compile(optimizer=optimizer, loss=loss, metrics=metrics, run_eagerly=self.config.run_eagerly)\n        except ValueError:\n            from keras.saving.object_registration import CustomObjectScope\n            with CustomObjectScope({'adamweightdecay': AdamWeightDecay}):\n                self.model.compile(optimizer=optimizer, loss=loss, metrics=metrics, run_eagerly=self.config.run_eagerly)\n\n    def build_optimizer(self, optimizer, **kwargs) -> tf.keras.optimizers.Optimizer:\n        if isinstance(optimizer, (str, dict)):\n            custom_objects = {'AdamWeightDecay': AdamWeightDecay}\n            try:\n                optimizer = tf.keras.utils.deserialize_keras_object(optimizer, module_objects=vars(tf.keras.optimizers),\n                                                                    custom_objects=custom_objects)\n            except ValueError:\n                optimizer['config'].pop('decay', None)\n                optimizer = tf.keras.utils.deserialize_keras_object(optimizer, module_objects=vars(tf.keras.optimizers),\n                                                                    custom_objects=custom_objects)\n        self.config.optimizer = tf.keras.utils.serialize_keras_object(optimizer)\n        return optimizer\n\n    def build_loss(self, loss, **kwargs):\n        if not loss:\n            loss = tf.keras.losses.SparseCategoricalCrossentropy(\n                reduction=tf.keras.losses.Reduction.SUM_OVER_BATCH_SIZE,\n                from_logits=True)\n        elif isinstance(loss, (str, dict)):\n            loss = tf.keras.utils.deserialize_keras_object(loss, module_objects=vars(tf.keras.losses))\n        if isinstance(loss, tf.keras.losses.Loss):\n            self.config.loss = tf.keras.utils.serialize_keras_object(loss)\n        return loss\n\n    def build_transform(self, **kwargs):\n        return self.transform\n\n    def build_vocab(self, trn_data, logger):\n        train_examples = self.transform.fit(trn_data, **self.config)\n        self.transform.summarize_vocabs(logger)\n        return train_examples\n\n    def build_metrics(self, metrics, logger: logging.Logger, **kwargs):\n        metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')\n        return [metric]\n\n    @abstractmethod\n    def build_model(self, **kwargs) -> tf.keras.Model:\n        pass\n\n    def fit(self, trn_data, dev_data, save_dir, batch_size, epochs, run_eagerly=False, logger=None, verbose=True,\n            finetune: str = None, **kwargs):\n        self._capture_config(locals())\n        if sys.version_info >= (3, 10):\n            logger.warning(f'Training with TensorFlow {tf.__version__} has not been tested on Python '\n                           f'{sys.version_info.major}.{sys.version_info.minor}. Please downgrade to '\n                           f'Python<=3.9 in case any compatibility issues arise.')\n        self.transform = self.build_transform(**self.config)\n        if not save_dir:\n            save_dir = tempdir_human()\n        if not logger:\n            logger = init_logger(name='train', root_dir=save_dir, level=logging.INFO if verbose else logging.WARN)\n        logger.info('Hyperparameter:\\n' + self.config.to_json())\n        num_examples = self.build_vocab(trn_data, logger)\n        # assert num_examples, 'You forgot to return the number of training examples in your build_vocab'\n        logger.info('Building...')\n        train_steps_per_epoch = math.ceil(num_examples / batch_size) if num_examples else None\n        self.config.train_steps = train_steps_per_epoch * epochs if num_examples else None\n        model, optimizer, loss, metrics = self.build(**merge_dict(self.config, logger=logger, training=True))\n        logger.info('Model built:\\n' + summary_of_model(self.model))\n        if finetune:\n            finetune = get_resource(finetune)\n            if os.path.isdir(finetune):\n                finetune = os.path.join(finetune, 'model.h5')\n            model.load_weights(finetune, by_name=True, skip_mismatch=True)\n            logger.info(f'Loaded pretrained weights from {finetune} for finetuning')\n        self.save_config(save_dir)\n        self.save_vocabs(save_dir)\n        self.save_meta(save_dir)\n        trn_data = self.build_train_dataset(trn_data, batch_size, num_examples)\n        dev_data = self.build_valid_dataset(dev_data, batch_size)\n        callbacks = self.build_callbacks(save_dir, **merge_dict(self.config, overwrite=True, logger=logger))\n        # need to know #batches, otherwise progbar crashes\n        dev_steps = math.ceil(self.num_samples_in(dev_data) / batch_size)\n        checkpoint = get_callback_by_class(callbacks, tf.keras.callbacks.ModelCheckpoint)\n        timer = Timer()\n        try:\n            history = self.train_loop(**merge_dict(self.config, trn_data=trn_data, dev_data=dev_data, epochs=epochs,\n                                                   num_examples=num_examples,\n                                                   train_steps_per_epoch=train_steps_per_epoch, dev_steps=dev_steps,\n                                                   callbacks=callbacks, logger=logger, model=model, optimizer=optimizer,\n                                                   loss=loss,\n                                                   metrics=metrics, overwrite=True))\n        except KeyboardInterrupt:\n            print()\n            if not checkpoint or checkpoint.best in (np.Inf, -np.Inf):\n                self.save_weights(save_dir)\n                logger.info('Aborted with model saved')\n            else:\n                logger.info(f'Aborted with model saved with best {checkpoint.monitor} = {checkpoint.best:.4f}')\n            # noinspection PyTypeChecker\n            history: tf.keras.callbacks.History() = get_callback_by_class(callbacks, tf.keras.callbacks.History)\n        delta_time = timer.stop()\n        best_epoch_ago = 0\n        if history and hasattr(history, 'epoch'):\n            trained_epoch = len(history.epoch)\n            logger.info('Trained {} epochs in {}, each epoch takes {}'.\n                        format(trained_epoch, delta_time, delta_time / trained_epoch if trained_epoch else delta_time))\n            save_json(history.history, io_util.path_join(save_dir, 'history.json'), cls=NumpyEncoder)\n            monitor_history: List = history.history.get(checkpoint.monitor, None)\n            if monitor_history:\n                best_epoch_ago = len(monitor_history) - monitor_history.index(checkpoint.best)\n            if checkpoint and monitor_history and checkpoint.best != monitor_history[-1]:\n                logger.info(f'Restored the best model saved with best '\n                            f'{checkpoint.monitor} = {checkpoint.best:.4f} '\n                            f'saved {best_epoch_ago} epochs ago')\n                self.load_weights(save_dir)  # restore best model\n        return history\n\n    def train_loop(self, trn_data, dev_data, epochs, num_examples, train_steps_per_epoch, dev_steps, model, optimizer,\n                   loss, metrics, callbacks,\n                   logger, **kwargs):\n        history = self.model.fit(trn_data, epochs=epochs, steps_per_epoch=train_steps_per_epoch,\n                                 validation_data=dev_data,\n                                 callbacks=callbacks,\n                                 validation_steps=dev_steps,\n                                 )  # type:tf.keras.callbacks.History\n        return history\n\n    def build_valid_dataset(self, dev_data, batch_size):\n        dev_data = self.transform.file_to_dataset(dev_data, batch_size=batch_size, shuffle=False)\n        return dev_data\n\n    def build_train_dataset(self, trn_data, batch_size, num_examples):\n        trn_data = self.transform.file_to_dataset(trn_data, batch_size=batch_size,\n                                                  shuffle=True,\n                                                  repeat=-1 if self.config.train_steps else None)\n        return trn_data\n\n    def build_callbacks(self, save_dir, logger, **kwargs):\n        metrics = kwargs.get('metrics', 'accuracy')\n        if isinstance(metrics, (list, tuple)):\n            metrics = metrics[-1]\n        monitor = f'val_{metrics}'\n        checkpoint = tf.keras.callbacks.ModelCheckpoint(\n            os.path.join(save_dir, 'model.h5'),\n            # verbose=1,\n            monitor=monitor, save_best_only=True,\n            mode='max',\n            save_weights_only=True)\n        logger.debug(f'Monitor {checkpoint.monitor} for checkpoint')\n        tensorboard_callback = tf.keras.callbacks.TensorBoard(\n            log_dir=io_util.makedirs(io_util.path_join(save_dir, 'logs')))\n        csv_logger = FineCSVLogger(os.path.join(save_dir, 'train.log'), separator=' | ', append=True)\n        callbacks = [checkpoint, tensorboard_callback, csv_logger]\n        lr_decay_per_epoch = self.config.get('lr_decay_per_epoch', None)\n        if lr_decay_per_epoch:\n            learning_rate = self.model.optimizer.get_config().get('learning_rate', None)\n            if not learning_rate:\n                logger.warning('Learning rate decay not supported for optimizer={}'.format(repr(self.model.optimizer)))\n            else:\n                logger.debug(f'Created LearningRateScheduler with lr_decay_per_epoch={lr_decay_per_epoch}')\n                callbacks.append(tf.keras.callbacks.LearningRateScheduler(\n                    lambda epoch: learning_rate / (1 + lr_decay_per_epoch * epoch)))\n        anneal_factor = self.config.get('anneal_factor', None)\n        if anneal_factor:\n            callbacks.append(tf.keras.callbacks.ReduceLROnPlateau(factor=anneal_factor,\n                                                                  patience=self.config.get('anneal_patience', 10)))\n        early_stopping_patience = self.config.get('early_stopping_patience', None)\n        if early_stopping_patience:\n            callbacks.append(tf.keras.callbacks.EarlyStopping(monitor=monitor, mode='max',\n                                                              verbose=1,\n                                                              patience=early_stopping_patience))\n        return callbacks\n\n    def on_train_begin(self):\n        \"\"\"\n        Callback before the training starts\n        \"\"\"\n        pass\n\n    def predict(self, data: Any, batch_size=None, **kwargs):\n        assert self.model, 'Please call fit or load before predict'\n        if not data:\n            return []\n        data, flat = self.transform.input_to_inputs(data)\n\n        if not batch_size:\n            batch_size = self.config.batch_size\n\n        dataset = self.transform.inputs_to_dataset(data, batch_size=batch_size, gold=kwargs.get('gold', False))\n\n        results = []\n        num_samples = 0\n        data_is_list = isinstance(data, list)\n        for idx, batch in enumerate(dataset):\n            samples_in_batch = tf.shape(batch[-1] if isinstance(batch[-1], tf.Tensor) else batch[-1][0])[0]\n            if data_is_list:\n                inputs = data[num_samples:num_samples + samples_in_batch]\n            else:\n                inputs = None  # if data is a generator, it's usually one-time, not able to transform into a list\n            for output in self.predict_batch(batch, inputs=inputs, **kwargs):\n                results.append(output)\n            num_samples += samples_in_batch\n        self.transform.cleanup()\n\n        if flat:\n            return results[0]\n        return results\n\n    def predict_batch(self, batch, inputs=None, **kwargs):\n        X = batch[0]\n        Y = self.model.predict_on_batch(X)\n        for output in self.transform.Y_to_outputs(Y, X=X, inputs=inputs, batch=batch, **kwargs):\n            yield output\n\n    @property\n    def sample_data(self):\n        return None\n\n    @staticmethod\n    def from_meta(meta: dict, **kwargs):\n        \"\"\"\n\n        Parameters\n        ----------\n        meta\n        kwargs\n\n        Returns\n        -------\n        KerasComponent\n\n        \"\"\"\n        cls = str_to_type(meta['class_path'])\n        obj: KerasComponent = cls()\n        assert 'load_path' in meta, f'{meta} doesn\\'t contain load_path field'\n        obj.load(meta['load_path'])\n        return obj\n\n    def export_model_for_serving(self, export_dir=None, version=1, overwrite=False, show_hint=False):\n        assert self.model, 'You have to fit or load a model before exporting it'\n        if not export_dir:\n            assert 'load_path' in self.meta, 'When not specifying save_dir, load_path has to present'\n            export_dir = get_resource(self.meta['load_path'])\n        model_path = os.path.join(export_dir, str(version))\n        if os.path.isdir(model_path) and not overwrite:\n            logger.info(f'{model_path} exists, skip since overwrite = {overwrite}')\n            return export_dir\n        logger.info(f'Exporting to {export_dir} ...')\n        tf.saved_model.save(self.model, model_path)\n        logger.info(f'Successfully exported model to {export_dir}')\n        if show_hint:\n            logger.info(f'You can serve it through \\n'\n                        f'tensorflow_model_server --model_name={os.path.splitext(os.path.basename(self.meta[\"load_path\"]))[0]} '\n                        f'--model_base_path={export_dir} --rest_api_port=8888')\n        return export_dir\n\n    def serve(self, export_dir=None, grpc_port=8500, rest_api_port=0, overwrite=False, dry_run=False):\n        export_dir = self.export_model_for_serving(export_dir, show_hint=False, overwrite=overwrite)\n        if not dry_run:\n            del self.model  # free memory\n        logger.info('The inputs of exported model is shown below.')\n        os.system(f'saved_model_cli show --all --dir {export_dir}/1')\n        cmd = f'nohup tensorflow_model_server --model_name={os.path.splitext(os.path.basename(self.meta[\"load_path\"]))[0]} ' \\\n              f'--model_base_path={export_dir} --port={grpc_port} --rest_api_port={rest_api_port} ' \\\n              f'>serve.log 2>&1 &'\n        logger.info(f'Running ...\\n{cmd}')\n        if not dry_run:\n            os.system(cmd)\n"
  },
  {
    "path": "hanlp/common/structure.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-08-26 14:58\nfrom typing import Dict\n\nfrom hanlp_common.configurable import Configurable\nfrom hanlp_common.reflection import classpath_of\nfrom hanlp_common.structure import SerializableDict\n\n\nclass ConfigTracker(Configurable):\n\n    def __init__(self, locals_: Dict, exclude=('kwargs', 'self', '__class__', 'locals_')) -> None:\n        \"\"\"This base class helps sub-classes to capture their arguments passed to ``__init__``, and also their types so\n        that they can be deserialized from a config in dict form.\n\n        Args:\n            locals_: Obtained by :meth:`locals`.\n            exclude: Arguments to be excluded.\n\n        Examples:\n            >>> class MyClass(ConfigTracker):\n            >>>     def __init__(self, i_need_this='yes') -> None:\n            >>>         super().__init__(locals())\n            >>> obj = MyClass()\n            >>> print(obj.config)\n            {'i_need_this': 'yes', 'classpath': 'test_config_tracker.MyClass'}\n\n        \"\"\"\n        if 'kwargs' in locals_:\n            locals_.update(locals_['kwargs'])\n        self.config = SerializableDict(\n            (k, v.config if hasattr(v, 'config') else v) for k, v in locals_.items() if k not in exclude)\n        self.config['classpath'] = classpath_of(self)\n\n\nclass History(object):\n    def __init__(self):\n        \"\"\" A history of training context. It records how many steps have passed and provides methods to decide whether\n        an update should be performed, and to caculate number of training steps given dataloader size and\n        ``gradient_accumulation``.\n        \"\"\"\n        self.num_mini_batches = 0\n\n    def step(self, gradient_accumulation):\n        \"\"\" Whether the training procedure should perform an update.\n\n        Args:\n            gradient_accumulation: Number of batches per update.\n\n        Returns:\n            bool: ``True`` to update.\n        \"\"\"\n        self.num_mini_batches += 1\n        return self.num_mini_batches % gradient_accumulation == 0\n\n    def num_training_steps(self, num_batches, gradient_accumulation):\n        \"\"\" Caculate number of training steps.\n\n        Args:\n            num_batches: Size of dataloader.\n            gradient_accumulation: Number of batches per update.\n\n        Returns:\n\n        \"\"\"\n        return len(\n            [i for i in range(self.num_mini_batches + 1, self.num_mini_batches + num_batches + 1) if\n             i % gradient_accumulation == 0])\n"
  },
  {
    "path": "hanlp/common/torch_component.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-08 21:20\nimport logging\nimport os\nimport re\nimport time\nfrom abc import ABC, abstractmethod\nfrom typing import Optional, Dict, List, Union, Callable\n\nimport torch\nfrom torch import nn\nfrom torch.utils.data import DataLoader\n\nimport hanlp\nfrom hanlp.common.component import Component\nfrom hanlp.common.dataset import TransformableDataset\nfrom hanlp.common.transform import VocabDict\nfrom hanlp.utils.io_util import get_resource, basename_no_ext\nfrom hanlp.utils.log_util import init_logger, flash\nfrom hanlp.utils.torch_util import cuda_devices, set_seed\nfrom hanlp_common.configurable import Configurable\nfrom hanlp_common.constant import IDX, HANLP_VERBOSE\nfrom hanlp_common.reflection import classpath_of\nfrom hanlp_common.structure import SerializableDict\nfrom hanlp_common.util import merge_dict, isdebugging\n\n\nclass TorchComponent(Component, ABC):\n    def __init__(self, **kwargs) -> None:\n        \"\"\"The base class for all components using PyTorch as backend. It provides common workflows of building vocabs,\n        datasets, dataloaders and models. These workflows are more of a conventional guideline than en-forced\n        protocols, which means subclass has the freedom to override or completely skip some steps.\n\n        Args:\n            **kwargs: Addtional arguments to be stored in the ``config`` property.\n        \"\"\"\n        super().__init__()\n        self.model: Optional[torch.nn.Module] = None\n        self.config = SerializableDict(**kwargs)\n        self.vocabs = VocabDict()\n\n    def _capture_config(self, locals_: Dict,\n                        exclude=(\n                                'trn_data', 'dev_data', 'save_dir', 'kwargs', 'self', 'logger', 'verbose',\n                                'dev_batch_size', '__class__', 'devices', 'eval_trn')):\n        \"\"\"Save arguments to config\n\n        Args:\n          locals_: Dict: \n          exclude:  (Default value = ('trn_data')\n          'dev_data': \n          'save_dir': \n          'kwargs': \n          'self': \n          'logger': \n          'verbose': \n          'dev_batch_size': \n          '__class__': \n          'devices'): \n\n        Returns:\n\n        \n        \"\"\"\n        if 'kwargs' in locals_:\n            locals_.update(locals_['kwargs'])\n        locals_ = dict((k, v) for k, v in locals_.items() if k not in exclude and not k.startswith('_'))\n        self.config.update(locals_)\n        return self.config\n\n    def save_weights(self, save_dir, filename='model.pt', trainable_only=True, **kwargs):\n        \"\"\"Save model weights to a directory.\n\n        Args:\n            save_dir: The directory to save weights into.\n            filename: A file name for weights.\n            trainable_only: ``True`` to only save trainable weights. Useful when the model contains lots of static\n                embeddings.\n            **kwargs: Not used for now.\n        \"\"\"\n        model = self.model_\n        state_dict = model.state_dict()\n        if trainable_only:\n            trainable_names = set(n for n, p in model.named_parameters() if p.requires_grad)\n            state_dict = dict((n, p) for n, p in state_dict.items() if n in trainable_names)\n        torch.save(state_dict, os.path.join(save_dir, filename))\n\n    def load_weights(self, save_dir, filename='model.pt', **kwargs):\n        \"\"\"Load weights from a directory.\n\n        Args:\n            save_dir: The directory to load weights from.\n            filename: A file name for weights.\n            **kwargs: Not used.\n        \"\"\"\n        save_dir = get_resource(save_dir)\n        filename = os.path.join(save_dir, filename)\n        # flash(f'Loading model: {filename} [blink]...[/blink][/yellow]')\n        try:\n            self.model_.load_state_dict(torch.load(filename, map_location='cpu', weights_only=True), strict=False)\n        except TypeError:\n            self.model_.load_state_dict(torch.load(filename, map_location='cpu'), strict=False)\n        # flash('')\n\n    def save_config(self, save_dir, filename='config.json'):\n        \"\"\"Save config into a directory.\n\n        Args:\n            save_dir: The directory to save config.\n            filename: A file name for config.\n        \"\"\"\n        self._savable_config.save_json(os.path.join(save_dir, filename))\n\n    def load_config(self, save_dir, filename='config.json', **kwargs):\n        \"\"\"Load config from a directory.\n\n        Args:\n            save_dir: The directory to load config.\n            filename: A file name for config.\n            **kwargs: K-V pairs to override config.\n        \"\"\"\n        save_dir = get_resource(save_dir)\n        self.config.load_json(os.path.join(save_dir, filename))\n        self.config.update(kwargs)  # overwrite config loaded from disk\n        for k, v in self.config.items():\n            if isinstance(v, dict) and 'classpath' in v:\n                self.config[k] = Configurable.from_config(v)\n        self.on_config_ready(**self.config, save_dir=save_dir)\n\n    def save_vocabs(self, save_dir, filename='vocabs.json'):\n        \"\"\"Save vocabularies to a directory.\n\n        Args:\n            save_dir: The directory to save vocabularies.\n            filename:  The name for vocabularies.\n        \"\"\"\n        if hasattr(self, 'vocabs'):\n            self.vocabs.save_vocabs(save_dir, filename)\n\n    def load_vocabs(self, save_dir, filename='vocabs.json'):\n        \"\"\"Load vocabularies from a directory.\n\n        Args:\n            save_dir: The directory to load vocabularies.\n            filename:  The name for vocabularies.\n        \"\"\"\n        if hasattr(self, 'vocabs'):\n            self.vocabs = VocabDict()\n            self.vocabs.load_vocabs(save_dir, filename)\n\n    def save(self, save_dir: str, **kwargs):\n        \"\"\"Save this component to a directory.\n\n        Args:\n            save_dir: The directory to save this component.\n            **kwargs: Not used.\n        \"\"\"\n        self.save_config(save_dir)\n        self.save_vocabs(save_dir)\n        self.save_weights(save_dir)\n\n    def load(self, save_dir: str, devices=None, verbose=HANLP_VERBOSE, **kwargs):\n        \"\"\"Load from a local/remote component.\n\n        Args:\n            save_dir: An identifier which can be a local path or a remote URL or a pre-defined string.\n            devices: The devices this component will be moved onto.\n            verbose: ``True`` to log loading progress.\n            **kwargs: To override some configs.\n        \"\"\"\n        save_dir = get_resource(save_dir)\n        # flash('Loading config and vocabs [blink][yellow]...[/yellow][/blink]')\n        if devices is None and self.model:\n            devices = self.devices\n        self.load_config(save_dir, **kwargs)\n        self.load_vocabs(save_dir)\n        if verbose:\n            flash('Building model [blink][yellow]...[/yellow][/blink]')\n        self.config.pop('training', None)  # Some legacy versions accidentally put training into config file\n        self.model = self.build_model(\n            **merge_dict(self.config, **kwargs, overwrite=True, inplace=True), training=False, save_dir=save_dir)\n        if verbose:\n            flash('')\n        self.load_weights(save_dir, **kwargs)\n        self.to(devices, verbose=verbose)\n        self.model.eval()\n\n    def fit(self,\n            trn_data,\n            dev_data,\n            save_dir,\n            batch_size,\n            epochs,\n            devices=None,\n            logger=None,\n            seed=None,\n            finetune: Union[bool, str] = False,\n            eval_trn=True,\n            _device_placeholder=False,\n            **kwargs):\n        \"\"\"Fit to data, triggers the training procedure. For training set and dev set, they shall be local or remote\n        files.\n\n        Args:\n            trn_data: Training set.\n            dev_data: Development set.\n            save_dir: The directory to save trained component.\n            batch_size: The number of samples in a batch.\n            epochs: Number of epochs.\n            devices: Devices this component will live on.\n            logger: Any :class:`logging.Logger` instance.\n            seed: Random seed to reproduce this training.\n            finetune: ``True`` to load from ``save_dir`` instead of creating a randomly initialized component. ``str``\n                to specify a different ``save_dir`` to load from.\n            eval_trn: Evaluate training set after each update. This can slow down the training but provides a quick\n                diagnostic for debugging.\n            _device_placeholder: ``True`` to create a placeholder tensor which triggers PyTorch to occupy devices so\n                other components won't take these devices as first choices.\n            **kwargs: Hyperparameters used by sub-classes.\n\n        Returns:\n            Any results sub-classes would like to return. Usually the best metrics on training set.\n\n        \"\"\"\n        # Common initialization steps\n        config = self._capture_config(locals())\n        if not logger:\n            logger = self.build_logger('train', save_dir)\n        if seed is None:\n            self.config.seed = 233 if isdebugging() else int(time.time())\n        set_seed(self.config.seed)\n        logger.info(self._savable_config.to_json(sort=True))\n        if isinstance(devices, list) or devices is None or isinstance(devices, float):\n            flash('[yellow]Querying CUDA devices [blink]...[/blink][/yellow]')\n            devices = -1 if isdebugging() else cuda_devices(devices)\n            flash('')\n        # flash(f'Available GPUs: {devices}')\n        if isinstance(devices, list):\n            first_device = (devices[0] if devices else -1)\n        elif isinstance(devices, dict):\n            first_device = next(iter(devices.values()))\n        elif isinstance(devices, int):\n            first_device = devices\n        else:\n            first_device = -1\n        if _device_placeholder and first_device >= 0:\n            _dummy_placeholder = self._create_dummy_placeholder_on(first_device)\n        if finetune:\n            if isinstance(finetune, str):\n                self.load(finetune, devices=devices)\n            else:\n                self.load(save_dir, devices=devices)\n            self.config.finetune = finetune\n            self.vocabs.unlock()  # For extending vocabs\n            logger.info(\n                f'Finetune model loaded with {sum(p.numel() for p in self.model.parameters() if p.requires_grad)}'\n                f'/{sum(p.numel() for p in self.model.parameters())} trainable/total parameters.')\n        self.on_config_ready(**self.config, save_dir=save_dir)\n        trn = self.build_dataloader(**merge_dict(config, data=trn_data, batch_size=batch_size, shuffle=True,\n                                                 training=True, device=first_device, logger=logger, vocabs=self.vocabs,\n                                                 overwrite=True))\n        dev = self.build_dataloader(**merge_dict(config, data=dev_data, batch_size=batch_size, shuffle=False,\n                                                 training=None, device=first_device, logger=logger, vocabs=self.vocabs,\n                                                 overwrite=True)) if dev_data else None\n        flash('[yellow]Building model [blink]...[/blink][/yellow]')\n        self.model = self.build_model(**merge_dict(config, training=True), logger=logger)\n        flash('')\n        logger.info(f'Model built with {sum(p.numel() for p in self.model.parameters() if p.requires_grad)}'\n                    f'/{sum(p.numel() for p in self.model.parameters())} trainable/total parameters.')\n        assert self.model, 'build_model is not properly implemented.'\n        _description = repr(self.model)\n        if len(_description.split('\\n')) < 10:\n            logger.info(_description)\n        self.save_config(save_dir)\n        self.save_vocabs(save_dir)\n        self.to(devices, logger)\n        if _device_placeholder and first_device >= 0:\n            del _dummy_placeholder\n        criterion = self.build_criterion(**merge_dict(config, trn=trn))\n        optimizer = self.build_optimizer(**merge_dict(config, trn=trn, criterion=criterion))\n        metric = self.build_metric(**self.config)\n        if hasattr(trn, 'dataset') and dev and hasattr(dev, 'dataset'):\n            if trn.dataset and dev.dataset:\n                logger.info(f'{len(trn.dataset)}/{len(dev.dataset)} samples in trn/dev set.')\n        if hasattr(trn, '__len__') and dev and hasattr(dev, '__len__'):\n            trn_size = len(trn) // self.config.get('gradient_accumulation', 1)\n            ratio_width = len(f'{trn_size}/{trn_size}')\n        else:\n            ratio_width = None\n        return self.execute_training_loop(**merge_dict(config, trn=trn, dev=dev, epochs=epochs, criterion=criterion,\n                                                       optimizer=optimizer, metric=metric, logger=logger,\n                                                       save_dir=save_dir,\n                                                       devices=devices,\n                                                       ratio_width=ratio_width,\n                                                       trn_data=trn_data,\n                                                       dev_data=dev_data,\n                                                       eval_trn=eval_trn,\n                                                       overwrite=True))\n\n    def build_logger(self, name, save_dir):\n        \"\"\"Build a :class:`logging.Logger`.\n\n        Args:\n            name: The name of this logger.\n            save_dir: The directory this logger should save logs into.\n\n        Returns:\n            logging.Logger: A logger.\n        \"\"\"\n        logger = init_logger(name=name, root_dir=save_dir, level=logging.INFO, fmt=\"%(message)s\")\n        return logger\n\n    @abstractmethod\n    def build_dataloader(self, data, batch_size, shuffle=False, device=None, logger: logging.Logger = None,\n                         **kwargs) -> DataLoader:\n        \"\"\"Build dataloader for training, dev and test sets. It's suggested to build vocabs in this method if they are\n        not built yet.\n\n        Args:\n            data: Data representing samples, which can be a path or a list of samples.\n            batch_size: Number of samples per batch.\n            shuffle: Whether to shuffle this dataloader.\n            device: Device tensors should be loaded onto.\n            logger: Logger for reporting some message if dataloader takes a long time or if vocabs has to be built.\n            **kwargs: Arguments from ``**self.config``.\n        \"\"\"\n        pass\n\n    def build_vocabs(self, trn: torch.utils.data.Dataset, logger: logging.Logger):\n        \"\"\"Override this method to build vocabs.\n\n        Args:\n            trn: Training set.\n            logger: Logger for reporting progress.\n        \"\"\"\n        pass\n\n    @property\n    def _savable_config(self):\n        def convert(k, v):\n            if not isinstance(v, SerializableDict) and hasattr(v, 'config'):\n                v = v.config\n            elif isinstance(v, (set, tuple)):\n                v = list(v)\n            if isinstance(v, dict):\n                v = dict(convert(_k, _v) for _k, _v in v.items())\n            return k, v\n\n        config = SerializableDict(\n            convert(k, v) for k, v in sorted(self.config.items()))\n        config.update({\n            # 'create_time': now_datetime(),\n            'classpath': classpath_of(self),\n            'hanlp_version': hanlp.__version__,\n        })\n        return config\n\n    @abstractmethod\n    def build_optimizer(self, **kwargs):\n        \"\"\"Implement this method to build an optimizer.\n\n        Args:\n            **kwargs: The subclass decides the method signature.\n        \"\"\"\n        pass\n\n    @abstractmethod\n    def build_criterion(self, **kwargs):\n        \"\"\"Implement this method to build criterion (loss function).\n\n        Args:\n            **kwargs: The subclass decides the method signature.\n        \"\"\"\n        pass\n\n    @abstractmethod\n    def build_metric(self, **kwargs):\n        \"\"\"Implement this to build metric(s).\n\n        Args:\n            **kwargs: The subclass decides the method signature.\n        \"\"\"\n        pass\n\n    @abstractmethod\n    def execute_training_loop(self, trn: DataLoader, dev: DataLoader, epochs, criterion, optimizer, metric, save_dir,\n                              logger: logging.Logger, devices, ratio_width=None,\n                              **kwargs):\n        \"\"\"Implement this to run training loop.\n\n        Args:\n            trn: Training set.\n            dev: Development set.\n            epochs: Number of epochs.\n            criterion: Loss function.\n            optimizer: Optimizer(s).\n            metric: Metric(s)\n            save_dir: The directory to save this component.\n            logger: Logger for reporting progress.\n            devices: Devices this component and dataloader will live on.\n            ratio_width: The width of dataset size measured in number of characters. Used for logger to align messages.\n            **kwargs: Other hyper-parameters passed from sub-class.\n        \"\"\"\n        pass\n\n    @abstractmethod\n    def fit_dataloader(self, trn: DataLoader, criterion, optimizer, metric, logger: logging.Logger, **kwargs):\n        \"\"\"Fit onto a dataloader.\n\n        Args:\n            trn: Training set.\n            criterion: Loss function.\n            optimizer: Optimizer.\n            metric: Metric(s).\n            logger: Logger for reporting progress.\n            **kwargs: Other hyper-parameters passed from sub-class.\n        \"\"\"\n        pass\n\n    @abstractmethod\n    def evaluate_dataloader(self, data: DataLoader, criterion: Callable, metric=None, output=False, **kwargs):\n        \"\"\"Evaluate on a dataloader.\n\n        Args:\n            data: Dataloader which can build from any data source.\n            criterion: Loss function.\n            metric: Metric(s).\n            output: Whether to save outputs into some file.\n            **kwargs: Not used.\n        \"\"\"\n        pass\n\n    @abstractmethod\n    def build_model(self, training=True, **kwargs) -> torch.nn.Module:\n        \"\"\"Build model.\n\n        Args:\n            training: ``True`` if called during training.\n            **kwargs: ``**self.config``.\n        \"\"\"\n        raise NotImplementedError\n\n    def evaluate(self, tst_data, save_dir=None, logger: logging.Logger = None, batch_size=None, output=False, **kwargs):\n        \"\"\"Evaluate test set.\n\n        Args:\n            tst_data: Test set, which is usually a file path.\n            save_dir: The directory to save evaluation scores or predictions.\n            logger: Logger for reporting progress.\n            batch_size: Batch size for test dataloader.\n            output: Whether to save outputs into some file.\n            **kwargs: Not used.\n\n        Returns:\n            (metric, outputs) where outputs are the return values of ``evaluate_dataloader``.\n        \"\"\"\n        if not self.model:\n            raise RuntimeError('Call fit or load before evaluate.')\n        if isinstance(tst_data, str):\n            tst_data = get_resource(tst_data)\n            filename = os.path.basename(tst_data)\n        else:\n            filename = None\n        if output is True:\n            output = self.generate_prediction_filename(tst_data if isinstance(tst_data, str) else 'test.txt', save_dir)\n        if logger is None:\n            _logger_name = basename_no_ext(filename) if filename else None\n            logger = self.build_logger(_logger_name, save_dir)\n        if not batch_size:\n            batch_size = self.config.get('batch_size', 32)\n        data = self.build_dataloader(**merge_dict(self.config, data=tst_data, batch_size=batch_size, shuffle=False,\n                                                  device=self.devices[0], logger=logger, overwrite=True))\n        dataset = data\n        while dataset and hasattr(dataset, 'dataset'):\n            dataset = dataset.dataset\n        num_samples = len(dataset) if dataset else None\n        if output and isinstance(dataset, TransformableDataset):\n            def add_idx(samples):\n                for idx, sample in enumerate(samples):\n                    if sample:\n                        sample[IDX] = idx\n\n            add_idx(dataset.data)\n            if dataset.cache:\n                add_idx(dataset.cache)\n\n        criterion = self.build_criterion(**self.config)\n        metric = self.build_metric(**self.config)\n        start = time.time()\n        outputs = self.evaluate_dataloader(data, criterion=criterion, filename=filename, output=output, input=tst_data,\n                                           save_dir=save_dir,\n                                           test=True,\n                                           num_samples=num_samples,\n                                           **merge_dict(self.config, batch_size=batch_size, metric=metric,\n                                                        logger=logger, **kwargs))\n        elapsed = time.time() - start\n        if logger:\n            if num_samples:\n                logger.info(f'speed: {num_samples / elapsed:.0f} samples/second')\n            else:\n                logger.info(f'speed: {len(data) / elapsed:.0f} batches/second')\n        return metric, outputs\n\n    def generate_prediction_filename(self, tst_data, save_dir):\n        assert isinstance(tst_data,\n                          str), 'tst_data has be a str in order to infer the output name'\n        output = os.path.splitext(os.path.basename(tst_data))\n        output = os.path.join(save_dir, output[0] + '.pred' + output[1])\n        return output\n\n    def to(self,\n           devices: Union[int, float, List[int], Dict[str, Union[int, torch.device]]] = None,\n           logger: logging.Logger = None, verbose=HANLP_VERBOSE):\n        \"\"\"Move this component to devices.\n\n        Args:\n            devices: Target devices.\n            logger: Logger for printing progress report, as copying a model from CPU to GPU can takes several seconds.\n            verbose: ``True`` to print progress when logger is None.\n        \"\"\"\n        if devices is None:\n            # if getattr(torch, 'has_mps', None):  # mac M1 chips\n            #     devices = torch.device('mps:0')\n            # else:\n            devices = cuda_devices(devices)\n        elif devices == -1 or devices == [-1]:\n            devices = []\n        elif isinstance(devices, (int, float)):\n            devices = cuda_devices(devices)\n        if devices:\n            if logger:\n                logger.info(f'Using GPUs: [on_blue][cyan][bold]{devices}[/bold][/cyan][/on_blue]')\n            if isinstance(devices, list):\n                if verbose:\n                    flash(f'Moving model to GPUs {devices} [blink][yellow]...[/yellow][/blink]')\n                self.model = self.model.to(devices[0])\n                if len(devices) > 1 and not isdebugging() and not isinstance(self.model, nn.DataParallel):\n                    self.model = self.parallelize(devices)\n            elif isinstance(devices, dict):\n                for name, module in self.model.named_modules():\n                    for regex, device in devices.items():\n                        try:\n                            on_device: torch.device = next(module.parameters()).device\n                        except StopIteration:\n                            continue\n                        if on_device == device:\n                            continue\n                        if isinstance(device, int):\n                            if on_device.index == device:\n                                continue\n                        if re.match(regex, name):\n                            if not name:\n                                name = '*'\n                            flash(f'Moving module [yellow]{name}[/yellow] to [on_yellow][magenta][bold]{device}'\n                                  f'[/bold][/magenta][/on_yellow]: [red]{regex}[/red]\\n')\n                            module.to(device)\n            elif isinstance(devices, torch.device):\n                if verbose:\n                    flash(f'Moving model to {devices} [blink][yellow]...[/yellow][/blink]')\n                self.model = self.model.to(devices)\n            else:\n                raise ValueError(f'Unrecognized devices {devices}')\n            if verbose:\n                flash('')\n        else:\n            if logger:\n                logger.info('Using [red]CPU[/red]')\n\n    def parallelize(self, devices: List[Union[int, torch.device]]):\n        return nn.DataParallel(self.model, device_ids=devices)\n\n    @property\n    def devices(self):\n        \"\"\"The devices this component lives on.\n        \"\"\"\n        if self.model is None:\n            return None\n        # next(parser.model.parameters()).device\n        if hasattr(self.model, 'device_ids'):\n            return self.model.device_ids\n        device: torch.device = next(self.model.parameters()).device\n        return [device]\n\n    @property\n    def device(self):\n        \"\"\"The first device this component lives on.\n        \"\"\"\n        devices = self.devices\n        if not devices:\n            return None\n        return devices[0]\n\n    def on_config_ready(self, **kwargs):\n        \"\"\"Called when config is ready, either during ``fit`` or ``load``. Subclass can perform extra initialization\n        tasks in this callback.\n\n        Args:\n            **kwargs: Not used.\n        \"\"\"\n        pass\n\n    @property\n    def model_(self) -> nn.Module:\n        \"\"\"\n        The actual model when it's wrapped by a `DataParallel`\n\n        Returns: The \"real\" model\n\n        \"\"\"\n        if isinstance(self.model, nn.DataParallel):\n            return self.model.module\n        return self.model\n\n    # noinspection PyMethodOverriding\n    @abstractmethod\n    def predict(self, *args, **kwargs):\n        \"\"\"Predict on data fed by user. Users shall avoid directly call this method since it is not guarded with\n        ``torch.no_grad`` and will introduces unnecessary gradient computation. Use ``__call__`` instead.\n\n        Args:\n            *args: Sentences or tokens.\n            **kwargs: Used in sub-classes.\n        \"\"\"\n        pass\n\n    @staticmethod\n    def _create_dummy_placeholder_on(device):\n        if device < 0:\n            device = 'cpu:0'\n        return torch.zeros(16, 16, device=device)\n\n    @torch.no_grad()\n    def __call__(self, *args, **kwargs):\n        \"\"\"Predict on data fed by user. This method calls :meth:`~hanlp.common.torch_component.predict` but decorates\n        it with ``torch.no_grad``.\n\n        Args:\n            *args: Sentences or tokens.\n            **kwargs: Used in sub-classes.\n        \"\"\"\n        return super().__call__(*args, **merge_dict(self.config, overwrite=True, **kwargs))\n"
  },
  {
    "path": "hanlp/common/transform.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-03 14:44\nimport logging\nimport os\nfrom abc import ABC, abstractmethod\nfrom typing import Tuple, Union, List\n\nfrom hanlp_common.constant import EOS, PAD\nfrom hanlp_common.structure import SerializableDict\nfrom hanlp_common.configurable import Configurable\nfrom hanlp.common.vocab import Vocab\nfrom hanlp.utils.io_util import get_resource\nfrom hanlp_common.io import load_json\nfrom hanlp_common.reflection import classpath_of, str_to_type\nfrom hanlp.utils.string_util import ispunct\n\n\nclass ToIndex(ABC):\n\n    def __init__(self, vocab: Vocab = None) -> None:\n        super().__init__()\n        if vocab is None:\n            vocab = Vocab()\n        self.vocab = vocab\n\n    @abstractmethod\n    def __call__(self, sample):\n        pass\n\n    def save_vocab(self, save_dir, filename='vocab.json'):\n        vocab = SerializableDict()\n        vocab.update(self.vocab.to_dict())\n        vocab.save_json(os.path.join(save_dir, filename))\n\n    def load_vocab(self, save_dir, filename='vocab.json'):\n        save_dir = get_resource(save_dir)\n        vocab = SerializableDict()\n        vocab.load_json(os.path.join(save_dir, filename))\n        self.vocab.copy_from(vocab)\n\n\nclass FieldToIndex(ToIndex):\n\n    def __init__(self, src, vocab: Vocab, dst=None) -> None:\n        super().__init__(vocab)\n        self.src = src\n        if not dst:\n            dst = f'{src}_id'\n        self.dst = dst\n\n    def __call__(self, sample: dict):\n        sample[self.dst] = self.vocab(sample[self.src])\n        return sample\n\n    def save_vocab(self, save_dir, filename=None):\n        if not filename:\n            filename = f'{self.dst}_vocab.json'\n        super().save_vocab(save_dir, filename)\n\n    def load_vocab(self, save_dir, filename=None):\n        if not filename:\n            filename = f'{self.dst}_vocab.json'\n        super().load_vocab(save_dir, filename)\n\n\nclass VocabList(list):\n\n    def __init__(self, *fields) -> None:\n        super().__init__()\n        for each in fields:\n            self.append(FieldToIndex(each))\n\n    def append(self, item: Union[str, Tuple[str, Vocab], Tuple[str, str, Vocab], FieldToIndex]) -> None:\n        if isinstance(item, str):\n            item = FieldToIndex(item)\n        elif isinstance(item, (list, tuple)):\n            if len(item) == 2:\n                item = FieldToIndex(src=item[0], vocab=item[1])\n            elif len(item) == 3:\n                item = FieldToIndex(src=item[0], dst=item[1], vocab=item[2])\n            else:\n                raise ValueError(f'Unsupported argument length: {item}')\n        elif isinstance(item, FieldToIndex):\n            pass\n        else:\n            raise ValueError(f'Unsupported argument type: {item}')\n        super(self).append(item)\n\n    def save_vocab(self, save_dir):\n        for each in self:\n            each.save_vocab(save_dir, None)\n\n    def load_vocab(self, save_dir):\n        for each in self:\n            each.load_vocab(save_dir, None)\n\n\nclass VocabDict(SerializableDict):\n\n    def __init__(self, *args, **kwargs) -> None:\n        \"\"\"A dict holding :class:`hanlp.common.vocab.Vocab` instances. When used as a transform, it transforms the field\n        corresponding to each :class:`hanlp.common.vocab.Vocab` into indices.\n\n        Args:\n            *args: A list of vocab names.\n            **kwargs: Names and corresponding :class:`hanlp.common.vocab.Vocab` instances.\n        \"\"\"\n        vocabs = dict(kwargs)\n        for each in args:\n            vocabs[each] = Vocab()\n        super().__init__(vocabs)\n\n    def save_vocabs(self, save_dir, filename='vocabs.json'):\n        \"\"\"Save vocabularies to a directory.\n\n        Args:\n            save_dir: The directory to save vocabularies.\n            filename:  The name for vocabularies.\n        \"\"\"\n        vocabs = SerializableDict()\n        for key, value in self.items():\n            if isinstance(value, Vocab):\n                vocabs[key] = value.to_dict()\n        vocabs.save_json(os.path.join(save_dir, filename))\n\n    def load_vocabs(self, save_dir, filename='vocabs.json', vocab_cls=Vocab):\n        \"\"\"Load vocabularies from a directory.\n\n        Args:\n            save_dir: The directory to load vocabularies.\n            filename:  The name for vocabularies.\n        \"\"\"\n        save_dir = get_resource(save_dir)\n        vocabs = SerializableDict()\n        vocabs.load_json(os.path.join(save_dir, filename))\n        self._load_vocabs(self, vocabs, vocab_cls)\n\n    @staticmethod\n    def _load_vocabs(vd, vocabs: dict, vocab_cls=Vocab):\n        \"\"\"\n\n        Args:\n            vd:\n            vocabs:\n            vocab_cls: Default class for the new vocab\n        \"\"\"\n        for key, value in vocabs.items():\n            if 'idx_to_token' in value:\n                cls = value.get('type', None)\n                if cls:\n                    cls = str_to_type(cls)\n                else:\n                    cls = vocab_cls\n                vocab = cls()\n                vocab.copy_from(value)\n                vd[key] = vocab\n            else:  # nested Vocab\n                # noinspection PyTypeChecker\n                vd[key] = nested = VocabDict()\n                VocabDict._load_vocabs(nested, value, vocab_cls)\n\n    def lock(self):\n        \"\"\"\n        Lock each vocab.\n        \"\"\"\n        for key, value in self.items():\n            if isinstance(value, Vocab):\n                value.lock()\n\n    def unlock(self):\n        \"\"\"\n        Unlock each vocab.\n        \"\"\"\n        for key, value in self.items():\n            if isinstance(value, Vocab):\n                value.unlock()\n\n    @property\n    def mutable(self):\n        status = [v.mutable for v in self.values() if isinstance(v, Vocab)]\n        return len(status) == 0 or any(status)\n\n    def __call__(self, sample: dict):\n        for key, value in self.items():\n            if isinstance(value, Vocab):\n                field = sample.get(key, None)\n                if field is not None:\n                    sample[f'{key}_id'] = value(field)\n        return sample\n\n    def __getattr__(self, key):\n        if key.startswith('__'):\n            return dict.__getattr__(key)\n        return self.__getitem__(key)\n\n    def __setattr__(self, key, value):\n        return self.__setitem__(key, value)\n\n    def __getitem__(self, k: str) -> Vocab:\n        return super().__getitem__(k)\n\n    def __setitem__(self, k: str, v: Vocab) -> None:\n        super().__setitem__(k, v)\n\n    def summary(self, logger: logging.Logger = None):\n        \"\"\"Log a summary of vocabs using a given logger.\n\n        Args:\n            logger: The logger to use.\n        \"\"\"\n        for key, value in self.items():\n            if isinstance(value, Vocab):\n                report = value.summary(verbose=False)\n                if logger:\n                    logger.info(f'{key}{report}')\n                else:\n                    print(f'{key}{report}')\n\n    def put(self, **kwargs):\n        \"\"\"Put names and corresponding :class:`hanlp.common.vocab.Vocab` instances into self.\n\n        Args:\n            **kwargs: Names and corresponding :class:`hanlp.common.vocab.Vocab` instances.\n        \"\"\"\n        for k, v in kwargs.items():\n            self[k] = v\n\n\nclass NamedTransform(ABC):\n    def __init__(self, src: str, dst: str = None) -> None:\n        if dst is None:\n            dst = src\n        self.dst = dst\n        self.src = src\n\n    @abstractmethod\n    def __call__(self, sample: dict) -> dict:\n        return sample\n\n\nclass ConfigurableTransform(Configurable, ABC):\n    @property\n    def config(self):\n        return dict([('classpath', classpath_of(self))] +\n                    [(k, v) for k, v in self.__dict__.items() if not k.startswith('_')])\n\n    @classmethod\n    def from_config(cls, config: dict):\n        \"\"\"\n\n        Args:\n          config: \n          kwargs: \n          config: dict: \n\n        Returns:\n\n        \n        \"\"\"\n        cls = config.get('classpath', None)\n        assert cls, f'{config} doesn\\'t contain classpath field'\n        cls = str_to_type(cls)\n        config = dict(config)\n        config.pop('classpath')\n        return cls(**config)\n\n\nclass ConfigurableNamedTransform(NamedTransform, ConfigurableTransform, ABC):\n    pass\n\n\nclass EmbeddingNamedTransform(ConfigurableNamedTransform, ABC):\n\n    def __init__(self, output_dim: int, src: str, dst: str) -> None:\n        super().__init__(src, dst)\n        self.output_dim = output_dim\n\n\nclass RenameField(NamedTransform):\n\n    def __call__(self, sample: dict):\n        sample[self.dst] = sample.pop(self.src)\n        return sample\n\n\nclass CopyField(object):\n    def __init__(self, src, dst) -> None:\n        self.dst = dst\n        self.src = src\n\n    def __call__(self, sample: dict) -> dict:\n        sample[self.dst] = sample[self.src]\n        return sample\n\n\nclass FilterField(object):\n    def __init__(self, *keys) -> None:\n        self.keys = keys\n\n    def __call__(self, sample: dict):\n        sample = dict((k, sample[k]) for k in self.keys)\n        return sample\n\n\nclass TransformList(list):\n    \"\"\"Composes several transforms together.\n\n    Args:\n      transforms(list of ``Transform`` objects): list of transforms to compose.\n    Example:\n\n    Returns:\n\n    >>> transforms.TransformList(\n        >>>     transforms.CenterCrop(10),\n        >>>     transforms.ToTensor(),\n        >>> )\n    \"\"\"\n\n    def __init__(self, *transforms) -> None:\n        super().__init__()\n        self.extend(transforms)\n\n    def __call__(self, sample):\n        for t in self:\n            sample = t(sample)\n        return sample\n\n    def index_by_type(self, t):\n        for i, trans in enumerate(self):\n            if isinstance(trans, t):\n                return i\n\n\nclass LowerCase(object):\n    def __init__(self, src, dst=None) -> None:\n        if dst is None:\n            dst = src\n        self.src = src\n        self.dst = dst\n\n    def __call__(self, sample: dict) -> dict:\n        src = sample[self.src]\n        if isinstance(src, str):\n            sample[self.dst] = src.lower()\n        elif isinstance(src, list):\n            sample[self.dst] = [x.lower() for x in src]\n        return sample\n\n\nclass LowerCase3D(LowerCase):\n\n    def __call__(self, sample: dict) -> dict:\n        src = sample[self.src]\n        sample[self.dst] = [[y.lower() for y in x] for x in src]\n        return sample\n\n\nclass ToChar(object):\n    def __init__(self, src, dst='char', max_word_length=None, min_word_length=None, pad=PAD) -> None:\n        if dst is None:\n            dst = src\n        self.src = src\n        self.dst = dst\n        self.max_word_length = max_word_length\n        self.min_word_length = min_word_length\n        self.pad = pad\n\n    def __call__(self, sample: dict) -> dict:\n        src = sample[self.src]\n        if isinstance(src, str):\n            sample[self.dst] = self.to_chars(src)\n        elif isinstance(src, list):\n            sample[self.dst] = [self.to_chars(x) for x in src]\n        return sample\n\n    def to_chars(self, word: str):\n        chars = list(word)\n        if self.min_word_length and len(chars) < self.min_word_length:\n            chars = chars + [self.pad] * (self.min_word_length - len(chars))\n        if self.max_word_length:\n            chars = chars[:self.max_word_length]\n        return chars\n\n\nclass AppendEOS(NamedTransform):\n\n    def __init__(self, src: str, dst: str = None, eos=EOS) -> None:\n        super().__init__(src, dst)\n        self.eos = eos\n\n    def __call__(self, sample: dict) -> dict:\n        sample[self.dst] = sample[self.src] + [self.eos]\n        return sample\n\n\nclass WhitespaceTokenizer(NamedTransform):\n\n    def __call__(self, sample: dict) -> dict:\n        src = sample[self.src]\n        if isinstance(src, str):\n            sample[self.dst] = self.tokenize(src)\n        elif isinstance(src, list):\n            sample[self.dst] = [self.tokenize(x) for x in src]\n        return sample\n\n    @staticmethod\n    def tokenize(text: str):\n        return text.split()\n\n\nclass NormalizeDigit(object):\n    def __init__(self, src, dst=None) -> None:\n        if dst is None:\n            dst = src\n        self.src = src\n        self.dst = dst\n\n    @staticmethod\n    def transform(word: str):\n        new_word = \"\"\n        for char in word:\n            if char.isdigit():\n                new_word += '0'\n            else:\n                new_word += char\n        return new_word\n\n    def __call__(self, sample: dict) -> dict:\n        src = sample[self.src]\n        if isinstance(src, str):\n            sample[self.dst] = self.transform(src)\n        elif isinstance(src, list):\n            sample[self.dst] = [self.transform(x) for x in src]\n        return sample\n\n\nclass Bigram(NamedTransform):\n\n    def __init__(self, src: str, dst: str = None) -> None:\n        if not dst:\n            dst = f'{src}_bigram'\n        super().__init__(src, dst)\n\n    def __call__(self, sample: dict) -> dict:\n        src: List = sample[self.src]\n        dst = src + [EOS]\n        dst = [dst[i] + dst[i + 1] for i in range(len(src))]\n        sample[self.dst] = dst\n        return sample\n\n\nclass FieldLength(NamedTransform):\n\n    def __init__(self, src: str, dst: str = None, delta=0) -> None:\n        self.delta = delta\n        if not dst:\n            dst = f'{src}_length'\n        super().__init__(src, dst)\n\n    def __call__(self, sample: dict) -> dict:\n        sample[self.dst] = len(sample[self.src]) + self.delta\n        return sample\n\n\nclass BMESOtoIOBES(object):\n    def __init__(self, field='tag') -> None:\n        self.field = field\n\n    def __call__(self, sample: dict) -> dict:\n        sample[self.field] = [self.convert(y) for y in sample[self.field]]\n        return sample\n\n    @staticmethod\n    def convert(y: str):\n        if y.startswith('M-'):\n            return 'I-'\n        return y\n\n\nclass NormalizeToken(ConfigurableNamedTransform):\n\n    def __init__(self, mapper: Union[str, dict], src: str, dst: str = None) -> None:\n        super().__init__(src, dst)\n        self.mapper = mapper\n        if isinstance(mapper, str):\n            mapper = get_resource(mapper)\n        if isinstance(mapper, str):\n            self._table = load_json(mapper)\n        elif isinstance(mapper, dict):\n            self._table = mapper\n        else:\n            raise ValueError(f'Unrecognized mapper type {mapper}')\n\n    def __call__(self, sample: dict) -> dict:\n        src = sample[self.src]\n        if self.src == self.dst:\n            sample[f'{self.src}_'] = src\n        if isinstance(src, str):\n            src = self.convert(src)\n        else:\n            src = [self.convert(x) for x in src]\n        sample[self.dst] = src\n        return sample\n\n    def convert(self, token) -> str:\n        return self._table.get(token, token)\n\n\nclass PunctuationMask(ConfigurableNamedTransform):\n    def __init__(self, src: str, dst: str = None) -> None:\n        \"\"\"Mask out all punctuations (set mask of punctuations to False)\n\n        Args:\n          src:\n          dst:\n\n        Returns:\n\n        \"\"\"\n        if not dst:\n            dst = f'{src}_punct_mask'\n        super().__init__(src, dst)\n\n    def __call__(self, sample: dict) -> dict:\n        src = sample[self.src]\n        if isinstance(src, str):\n            dst = not ispunct(src)\n        else:\n            dst = [not ispunct(x) for x in src]\n        sample[self.dst] = dst\n        return sample\n\n\nclass NormalizeCharacter(NormalizeToken):\n    def convert(self, token) -> str:\n        return ''.join([NormalizeToken.convert(self, c) for c in token])\n"
  },
  {
    "path": "hanlp/common/transform_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-10-27 14:22\nimport inspect\nfrom abc import ABC, abstractmethod\nfrom typing import Generator, Tuple, Union, Iterable, Any\n\nimport tensorflow as tf\n\nfrom hanlp_common.structure import SerializableDict\nfrom hanlp.common.vocab_tf import VocabTF\nfrom hanlp.utils.io_util import get_resource\nfrom hanlp.utils.log_util import logger\n\n\nclass Transform(ABC):\n\n    def __init__(self, config: SerializableDict = None, map_x=True, map_y=True, **kwargs) -> None:\n        super().__init__()\n        self.map_y = map_y\n        self.map_x = map_x\n        if kwargs:\n            if not config:\n                config = SerializableDict()\n            for k, v in kwargs.items():\n                config[k] = v\n        self.config = config\n        self.output_types = None\n        self.output_shapes = None\n        self.padding_values = None\n        # Fix tf memory leak: https://github.com/tensorflow/tensorflow/issues/37653#issuecomment-1000517720\n        self.py_func_set_to_cleanup = set()\n\n    @abstractmethod\n    def fit(self, trn_path: str, **kwargs) -> int:\n        \"\"\"\n        Build the vocabulary from training file\n\n        Parameters\n        ----------\n        trn_path : path to training set\n        kwargs\n\n        Returns\n        -------\n        int\n            How many samples in the training set\n        \"\"\"\n        raise NotImplementedError('%s.%s()' % (self.__class__.__name__, inspect.stack()[0][3]))\n\n    def build_config(self):\n        \"\"\"\n        By default, call build_types_shapes_values, usually called in component's build method.\n        You can perform other building task here. Remember to call super().build_config\n        \"\"\"\n        self.output_types, self.output_shapes, self.padding_values = self.create_types_shapes_values()\n        # We prefer list over shape here, as it's easier to type [] than ()\n        # if isinstance(self.output_shapes, tuple):\n        #     self.output_shapes = list(self.output_shapes)\n        # for i, shapes in enumerate(self.output_shapes):\n        #     if isinstance(shapes, tuple):\n        #         self.output_shapes[i] = list(shapes)\n        #     for j, shape in enumerate(shapes):\n        #         if isinstance(shape, tuple):\n        #             shapes[j] = list(shape)\n\n    @abstractmethod\n    def create_types_shapes_values(self) -> Tuple[Tuple, Tuple, Tuple]:\n        \"\"\"\n        Create dataset related values,\n        \"\"\"\n        raise NotImplementedError('%s.%s()' % (self.__class__.__name__, inspect.stack()[0][3]))\n\n    @abstractmethod\n    def file_to_inputs(self, filepath: str, gold=True):\n        \"\"\"\n        Transform file to inputs. The inputs are defined as raw features (e.g. words) to be processed into more\n        features (e.g. forms and characters)\n\n        Parameters\n        ----------\n        filepath\n        gold\n        \"\"\"\n        raise NotImplementedError('%s.%s()' % (self.__class__.__name__, inspect.stack()[0][3]))\n\n    def inputs_to_samples(self, inputs, gold=False):\n        if gold:\n            yield from inputs\n        else:\n            for x in inputs:\n                yield x, self.padding_values[-1]\n\n    def file_to_samples(self, filepath: str, gold=True):\n        \"\"\"\n        Transform file to samples\n        Parameters\n        ----------\n        filepath\n        gold\n        \"\"\"\n        filepath = get_resource(filepath)\n        inputs = self.file_to_inputs(filepath, gold)\n        yield from self.inputs_to_samples(inputs, gold)\n\n    def file_to_dataset(self, filepath: str, gold=True, map_x=None, map_y=None, batch_size=32, shuffle=None,\n                        repeat=None,\n                        drop_remainder=False,\n                        prefetch=1,\n                        cache=True,\n                        **kwargs) -> tf.data.Dataset:\n        \"\"\"\n        Transform file to dataset\n\n        Parameters\n        ----------\n        filepath\n        gold : bool\n            Whether it's processing gold data or not. Example: there is usually a column for gold answer\n            when gold = True.\n        map_x : bool\n            Whether call map_x or not. Default to self.map_x\n        map_y : bool\n            Whether call map_y or not. Default to self.map_y\n        batch_size\n        shuffle\n        repeat\n        prefetch\n        kwargs\n\n        Returns\n        -------\n\n        \"\"\"\n\n        # debug\n        # for sample in self.file_to_samples(filepath):\n        #     pass\n\n        def generator():\n            inputs = self.file_to_inputs(filepath, gold)\n            samples = self.inputs_to_samples(inputs, gold)\n            yield from samples\n\n        return self.samples_to_dataset(generator, map_x, map_y, batch_size, shuffle, repeat, drop_remainder, prefetch,\n                                       cache)\n\n    def inputs_to_dataset(self, inputs, gold=False, map_x=None, map_y=None, batch_size=32, shuffle=None, repeat=None,\n                          drop_remainder=False,\n                          prefetch=1, cache=False, **kwargs) -> tf.data.Dataset:\n        # debug\n        # for sample in self.inputs_to_samples(inputs):\n        #     pass\n\n        def generator():\n            samples = self.inputs_to_samples(inputs, gold)\n            yield from samples\n\n        return self.samples_to_dataset(generator, map_x, map_y, batch_size, shuffle, repeat, drop_remainder, prefetch,\n                                       cache)\n\n    def samples_to_dataset(self, samples: Generator, map_x=None, map_y=None, batch_size=32, shuffle=None, repeat=None,\n                           drop_remainder=False,\n                           prefetch=1, cache=True) -> tf.data.Dataset:\n        output_types, output_shapes, padding_values = self.output_types, self.output_shapes, self.padding_values\n        if not all(v for v in [output_shapes, output_shapes,\n                               padding_values]):\n            # print('Did you forget to call build_config() on your transform?')\n            self.build_config()\n            output_types, output_shapes, padding_values = self.output_types, self.output_shapes, self.padding_values\n        assert all(v for v in [output_shapes, output_shapes,\n                               padding_values]), 'Your create_types_shapes_values returns None, which is not allowed'\n        # if not callable(samples):\n        #     samples = Transform.generator_to_callable(samples)\n        if not hasattr(tf.compat.v1.get_default_graph(), '_py_funcs_used_in_graph'):\n            tf.compat.v1.get_default_graph()._py_funcs_used_in_graph = []\n        py_func_set_before = set(tf.compat.v1.get_default_graph()._py_funcs_used_in_graph)\n        dataset = tf.data.Dataset.from_generator(samples, output_types=output_types, output_shapes=output_shapes)\n        if cache:\n            logger.debug('Dataset cache enabled')\n            dataset = dataset.cache(cache if isinstance(cache, str) else '')\n        if shuffle:\n            if isinstance(shuffle, bool):\n                shuffle = 1024\n            dataset = dataset.shuffle(shuffle)\n        if repeat:\n            dataset = dataset.repeat(repeat)\n        if batch_size:\n            dataset = dataset.padded_batch(batch_size, output_shapes, padding_values, drop_remainder)\n        if prefetch:\n            dataset = dataset.prefetch(prefetch)\n        if map_x is None:\n            map_x = self.map_x\n        if map_y is None:\n            map_y = self.map_y\n        if map_x or map_y:\n            def mapper(X, Y):\n                if map_x:\n                    X = self.x_to_idx(X)\n                if map_y:\n                    Y = self.y_to_idx(Y)\n                return X, Y\n\n            dataset = dataset.map(mapper, num_parallel_calls=tf.data.experimental.AUTOTUNE)\n        py_func_set_after = set(tf.compat.v1.get_default_graph()._py_funcs_used_in_graph) - py_func_set_before\n        self.py_func_set_to_cleanup |= py_func_set_after\n        return dataset\n\n    @abstractmethod\n    def x_to_idx(self, x) -> Union[tf.Tensor, Tuple]:\n        raise NotImplementedError('%s.%s()' % (self.__class__.__name__, inspect.stack()[0][3]))\n\n    @abstractmethod\n    def y_to_idx(self, y) -> tf.Tensor:\n        raise NotImplementedError('%s.%s()' % (self.__class__.__name__, inspect.stack()[0][3]))\n\n    def lock_vocabs(self):\n        for key, value in vars(self).items():\n            if isinstance(value, VocabTF):\n                value.lock()\n\n    def summarize_vocabs(self, logger=None, header='Vocab summary:'):\n        output = header + '\\n'\n        vocabs = {}\n        for key, value in vars(self).items():\n            if isinstance(value, VocabTF):\n                vocabs[key] = value\n        # tag vocab comes last usually\n        for key, value in sorted(vocabs.items(), key=lambda kv: len(kv[1]), reverse=True):\n            output += f'{key}' + value.summary(verbose=False) + '\\n'\n        output = output.strip()\n        if logger:\n            logger.info(output)\n        else:\n            print(output)\n\n    @staticmethod\n    def generator_to_callable(generator: Generator):\n        return lambda: (x for x in generator)\n\n    def str_to_idx(self, X, Y) -> Tuple[Union[tf.Tensor, Tuple], tf.Tensor]:\n        return self.x_to_idx(X), self.y_to_idx(Y)\n\n    def X_to_inputs(self, X: Union[tf.Tensor, Tuple[tf.Tensor]]) -> Iterable:\n        return [repr(x) for x in X]\n\n    def Y_to_outputs(self, Y: Union[tf.Tensor, Tuple[tf.Tensor]], gold=False, inputs=None, X=None,\n                     batch=None) -> Iterable:\n        return [repr(y) for y in Y]\n\n    def XY_to_inputs_outputs(self, X: Union[tf.Tensor, Tuple[tf.Tensor]],\n                             Y: Union[tf.Tensor, Tuple[tf.Tensor]], gold=False) -> Iterable:\n        \"\"\"\n        Convert predicted tensors to outputs\n\n        Parameters\n        ----------\n        X : Union[tf.Tensor, Tuple[tf.Tensor]]\n            The inputs of model\n        Y : Union[tf.Tensor, Tuple[tf.Tensor]]\n            The outputs of model\n\n        Returns\n        -------\n\n        \"\"\"\n        return [(x, y) for x, y in zip(self.X_to_inputs(X), self.Y_to_outputs(Y, gold))]\n\n    def input_is_single_sample(self, input: Any) -> bool:\n        return False\n\n    def input_to_inputs(self, input: Any) -> Tuple[Any, bool]:\n        \"\"\"\n        If input is one sample, convert it to a list which contains this unique sample\n\n        Parameters\n        ----------\n        input :\n            sample or samples\n\n        Returns\n        -------\n        (inputs, converted) : Tuple[Any, bool]\n\n        \"\"\"\n        flat = self.input_is_single_sample(input)\n        if flat:\n            input = [input]\n        return input, flat\n\n    def input_truth_output_to_str(self, input, truth, output):\n        \"\"\"\n        Convert input truth output to string representation, usually for writing to file during evaluation\n\n        Parameters\n        ----------\n        input\n        truth\n        output\n\n        Returns\n        -------\n\n        \"\"\"\n        return '\\t'.join([input, truth, output]) + '\\n'\n\n    def cleanup(self):\n        new_py_funcs = set(tf.compat.v1.get_default_graph()._py_funcs_used_in_graph) - self.py_func_set_to_cleanup\n        tf.compat.v1.get_default_graph()._py_funcs_used_in_graph = list(new_py_funcs)\n        self.py_func_set_to_cleanup = set()\n"
  },
  {
    "path": "hanlp/common/vocab.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-06-13 22:42\nfrom collections import Counter\nfrom typing import List, Dict, Union, Iterable\n\nfrom hanlp_common.constant import UNK, PAD\nfrom hanlp_common.structure import Serializable\nfrom hanlp_common.reflection import classpath_of\n\n\nclass Vocab(Serializable):\n    def __init__(self, idx_to_token: List[str] = None, token_to_idx: Dict = None, mutable=True, pad_token=PAD,\n                 unk_token=UNK) -> None:\n        \"\"\"Vocabulary base class which converts tokens to indices and vice versa.\n\n        Args:\n            idx_to_token: id to token mapping.\n            token_to_idx: token to id mapping.\n            mutable: ``True`` to allow adding new tokens, ``False`` to map OOV to ``unk``.\n            pad_token: The token representing padding.\n            unk_token: The token representing OOV.\n        \"\"\"\n        super().__init__()\n        if idx_to_token:\n            t2i = dict((token, idx) for idx, token in enumerate(idx_to_token))\n            if token_to_idx:\n                t2i.update(token_to_idx)\n            token_to_idx = t2i\n        if token_to_idx is None:\n            token_to_idx = {}\n            if pad_token is not None:\n                token_to_idx[pad_token] = len(token_to_idx)\n            if unk_token is not None:\n                token_to_idx[unk_token] = token_to_idx.get(unk_token, len(token_to_idx))\n        self.token_to_idx = token_to_idx\n        self.idx_to_token: List[str] = None\n        self.mutable = mutable\n        self.pad_token = pad_token\n        self.unk_token = unk_token\n\n    def __setitem__(self, token: str, idx: int):\n        assert self.mutable, 'Update an immutable Vocab object is not allowed'\n        self.token_to_idx[token] = idx\n\n    def __getitem__(self, key: Union[str, int, List]) -> Union[int, str, List]:\n        \"\"\" Get the index/indices associated with a token or a list of tokens or vice versa.\n\n        Args:\n            key: ``str`` for token(s) and ``int`` for index/indices.\n\n        Returns: Associated indices or tokens.\n\n        \"\"\"\n        if isinstance(key, str):\n            return self.get_idx(key)\n        elif isinstance(key, int):\n            return self.get_token(key)\n        elif isinstance(key, list):\n            if len(key) == 0:\n                return []\n            elif isinstance(key[0], str):\n                return [self.get_idx(x) for x in key]\n            elif isinstance(key[0], int):\n                return [self.get_token(x) for x in key]\n\n    def __contains__(self, key: Union[str, int]):\n        if isinstance(key, str):\n            return key in self.token_to_idx\n        elif isinstance(key, int):\n            return 0 <= key < len(self.idx_to_token)\n        else:\n            return False\n\n    def add(self, token: str) -> int:\n        \"\"\" Tries to add a token into a vocab and returns its id. If it has already been there, its id will be returned\n        and the vocab won't be updated. If the vocab is locked, an assertion failure will occur.\n\n        Args:\n            token: A new or existing token.\n\n        Returns:\n            Its associated id.\n\n        \"\"\"\n        assert self.mutable, 'It is not allowed to call add on an immutable Vocab'\n        assert isinstance(token, str), f'Token type must be str but got {type(token)} from {token}'\n        assert token is not None, 'Token must not be None'\n        idx = self.token_to_idx.get(token, None)\n        if idx is None:\n            idx = len(self.token_to_idx)\n            self.token_to_idx[token] = idx\n        return idx\n\n    def update(self, tokens: Iterable[str]) -> None:\n        \"\"\"Update the vocab with these tokens by adding them to vocab one by one.\n\n        Args:\n          tokens (Iterable[str]): A list of tokens.\n        \"\"\"\n        assert self.mutable, 'It is not allowed to update an immutable Vocab'\n        for token in tokens:\n            self.add(token)\n\n    def get_idx(self, token: str) -> int:\n        \"\"\"Get the idx of a token. If it's not there, it will be added to the vocab when the vocab is locked otherwise\n        the id of UNK will be returned.\n\n        Args:\n            token: A token.\n\n        Returns:\n            The id of that token.\n\n        \"\"\"\n        assert isinstance(token, str), 'token has to be `str`'\n        idx = self.token_to_idx.get(token, None)\n        if idx is None:\n            if self.mutable:\n                idx = len(self.token_to_idx)\n                self.token_to_idx[token] = idx\n            else:\n                idx = self.token_to_idx.get(self.unk_token, None)\n        return idx\n\n    def get_idx_without_add(self, token: str) -> int:\n        idx = self.token_to_idx.get(token, None)\n        if idx is None:\n            idx = self.token_to_idx.get(self.safe_unk_token, None)\n        return idx\n\n    def get_token(self, idx: int) -> str:\n        \"\"\"Get the token using its index.\n\n        Args:\n            idx: The index to a token.\n\n        Returns:\n\n        \"\"\"\n        if self.idx_to_token:\n            return self.idx_to_token[idx]\n\n        if self.mutable:\n            for token in self.token_to_idx:\n                if self.token_to_idx[token] == idx:\n                    return token\n\n    def has_key(self, token):\n        return token in self.token_to_idx\n\n    def __len__(self):\n        return len(self.token_to_idx)\n\n    def lock(self):\n        \"\"\"Lock this vocab up so that it won't accept new tokens.\n\n        Returns:\n            Itself.\n\n        \"\"\"\n        if self.locked:\n            return self\n        self.mutable = False\n        self.build_idx_to_token()\n        return self\n\n    def build_idx_to_token(self):\n        max_idx = max(self.token_to_idx.values())\n        self.idx_to_token = [None] * (max_idx + 1)\n        for token, idx in self.token_to_idx.items():\n            self.idx_to_token[idx] = token\n\n    def unlock(self):\n        \"\"\"Unlock this vocab so that new tokens can be added in.\n\n        Returns:\n            Itself.\n\n        \"\"\"\n        if not self.locked:\n            return\n        self.mutable = True\n        self.idx_to_token = None\n        return self\n\n    @property\n    def locked(self):\n        \"\"\"\n        ``True`` indicates this vocab is locked.\n        \"\"\"\n        return not self.mutable\n\n    @property\n    def unk_idx(self):\n        \"\"\"\n        The index of ``UNK`` token.\n        \"\"\"\n        if self.unk_token is None:\n            return None\n        else:\n            return self.token_to_idx.get(self.unk_token, None)\n\n    @property\n    def pad_idx(self):\n        \"\"\"\n        The index of ``PAD`` token.\n        \"\"\"\n        if self.pad_token is None:\n            return None\n        else:\n            return self.token_to_idx.get(self.pad_token, None)\n\n    @property\n    def tokens(self):\n        \"\"\"\n        A set of all tokens in this vocab.\n        \"\"\"\n        return self.token_to_idx.keys()\n\n    def __str__(self) -> str:\n        return self.token_to_idx.__str__()\n\n    def summary(self, verbose=True) -> str:\n        \"\"\"Get or print a summary of this vocab.\n\n        Args:\n            verbose: ``True`` to print the summary to stdout.\n\n        Returns:\n            Summary in text form.\n\n        \"\"\"\n        # report = 'Length: {}\\n'.format(len(self))\n        # report += 'Samples: {}\\n'.format(str(list(self.token_to_idx.keys())[:min(50, len(self))]))\n        # report += 'Mutable: {}'.format(self.mutable)\n        # report = report.strip()\n        report = '[{}] = '.format(len(self))\n        report += str(list(self.token_to_idx.keys())[:min(50, len(self))])\n        if verbose:\n            print(report)\n        return report\n\n    def __call__(self, some_token: Union[str, Iterable[str]]) -> Union[int, List[int]]:\n        if isinstance(some_token, (list, tuple, set)):\n            indices = []\n            if len(some_token) and isinstance(some_token[0], (list, tuple, set)):\n                for sent in some_token:\n                    inside = []\n                    for token in sent:\n                        inside.append(self.get_idx(token))\n                    indices.append(inside)\n                return indices\n            for token in some_token:\n                indices.append(self.get_idx(token))\n            return indices\n        else:\n            return self.get_idx(some_token)\n\n    def to_dict(self) -> dict:\n        \"\"\"Convert this vocab to a dict so that it can be json serialized.\n\n        Returns:\n            A dict.\n\n        \"\"\"\n        idx_to_token = self.idx_to_token\n        pad_token = self.pad_token\n        unk_token = self.unk_token\n        mutable = self.mutable\n        items = locals().copy()\n        items.pop('self')\n        return items\n\n    def copy_from(self, item: dict):\n        \"\"\"Copy properties from a dict so that it can json de-serialized.\n\n        Args:\n            item: A dict holding ``token_to_idx``\n\n        Returns:\n            Itself.\n\n        \"\"\"\n        for key, value in item.items():\n            setattr(self, key, value)\n        self.token_to_idx = {k: v for v, k in enumerate(self.idx_to_token)}\n        return self\n\n    def lower(self):\n        \"\"\"Convert all tokens to lower case.\n\n        Returns:\n            Itself.\n\n        \"\"\"\n        self.unlock()\n        token_to_idx = self.token_to_idx\n        self.token_to_idx = {}\n        for token in token_to_idx.keys():\n            self.add(token.lower())\n        return self\n\n    @property\n    def first_token(self):\n        \"\"\"The first token in this vocab.\n        \"\"\"\n        if self.idx_to_token:\n            return self.idx_to_token[0]\n        if self.token_to_idx:\n            return next(iter(self.token_to_idx))\n        return None\n\n    def merge(self, other):\n        \"\"\"Merge this with another vocab inplace.\n\n        Args:\n            other (Vocab): Another vocab.\n        \"\"\"\n        for word, idx in other.token_to_idx.items():\n            self.get_idx(word)\n\n    @property\n    def safe_pad_token(self) -> str:\n        \"\"\"Get the pad token safely. It always returns a pad token, which is the pad token or the first token\n        if pad does not present in the vocab.\n        \"\"\"\n        if self.pad_token:\n            return self.pad_token\n        if self.first_token:\n            return self.first_token\n        return PAD\n\n    @property\n    def safe_pad_token_idx(self) -> int:\n        \"\"\"Get the idx to the pad token safely. It always returns an index, which corresponds to the pad token or the\n        first token if pad does not present in the vocab.\n        \"\"\"\n        return self.token_to_idx.get(self.safe_pad_token, 0)\n\n    @property\n    def safe_unk_token(self) -> str:\n        \"\"\"Get the unk token safely. It always returns a unk token, which is the unk token or the first token if unk\n        does not presented in the vocab.\n        \"\"\"\n        if self.unk_token:\n            return self.unk_token\n        if self.first_token:\n            return self.first_token\n        return UNK\n\n    def __repr__(self) -> str:\n        if self.idx_to_token is not None:\n            return self.idx_to_token.__repr__()\n        return self.token_to_idx.__repr__()\n\n    def extend(self, tokens: Iterable[str]):\n        self.unlock()\n        self(tokens)\n\n    def reload_idx_to_token(self, idx_to_token: List[str], pad_idx=0, unk_idx=1):\n        self.idx_to_token = idx_to_token\n        self.token_to_idx = dict((s, i) for i, s in enumerate(idx_to_token))\n        if pad_idx is not None:\n            self.pad_token = idx_to_token[pad_idx]\n        if unk_idx is not None:\n            self.unk_token = idx_to_token[unk_idx]\n\n    def set_unk_as_safe_unk(self):\n        \"\"\"Set ``self.unk_token = self.safe_unk_token``. It's useful when the dev/test set contains OOV labels.\n        \"\"\"\n        self.unk_token = self.safe_unk_token\n\n    def clear(self):\n        self.unlock()\n        self.token_to_idx.clear()\n\n\nclass CustomVocab(Vocab):\n    def to_dict(self) -> dict:\n        d = super().to_dict()\n        d['type'] = classpath_of(self)\n        return d\n\n\nclass LowercaseVocab(CustomVocab):\n    def get_idx(self, token: str) -> int:\n        idx = self.token_to_idx.get(token, None)\n        if idx is None:\n            idx = self.token_to_idx.get(token.lower(), None)\n        if idx is None:\n            if self.mutable:\n                idx = len(self.token_to_idx)\n                self.token_to_idx[token] = idx\n            else:\n                idx = self.token_to_idx.get(self.unk_token, None)\n        return idx\n\n\nclass VocabWithNone(CustomVocab):\n    def get_idx(self, token: str) -> int:\n        if token is None:\n            return -1\n        return super().get_idx(token)\n\n\nclass VocabWithFrequency(CustomVocab):\n\n    def __init__(self, counter: Counter = None, min_occur_cnt=0, pad_token=PAD, unk_token=UNK, specials=None) -> None:\n        super().__init__(None, None, True, pad_token, unk_token)\n        if specials:\n            for each in specials:\n                counter.pop(each, None)\n                self.add(each)\n        self.frequencies = [1] * len(self)\n        if counter:\n            for token, freq in counter.most_common():\n                if freq >= min_occur_cnt:\n                    self.add(token)\n                    self.frequencies.append(freq)\n        self.lock()\n\n    def to_dict(self) -> dict:\n        d = super().to_dict()\n        d['frequencies'] = self.frequencies\n        return d\n\n    def copy_from(self, item: dict):\n        super().copy_from(item)\n        self.frequencies = item['frequencies']\n\n    def get_frequency(self, token):\n        idx = self.get_idx(token)\n        if idx is not None:\n            return self.frequencies[idx]\n        return 0\n\n\nclass VocabCounter(CustomVocab):\n\n    def __init__(self, idx_to_token: List[str] = None, token_to_idx: Dict = None, mutable=True, pad_token=PAD,\n                 unk_token=UNK) -> None:\n        super().__init__(idx_to_token, token_to_idx, mutable, pad_token, unk_token)\n        self.counter = Counter()\n\n    def get_idx(self, token: str) -> int:\n        if self.mutable:\n            self.counter[token] += 1\n        return super().get_idx(token)\n\n    def trim(self, min_frequency):\n        assert self.mutable\n        specials = {self.unk_token, self.pad_token}\n        survivors = list((token, freq) for token, freq in self.counter.most_common()\n                         if freq >= min_frequency and token not in specials)\n        survivors = [(x, -1) for x in specials if x] + survivors\n        self.counter = Counter(dict(survivors))\n        self.token_to_idx = dict()\n        self.idx_to_token = None\n        for token, freq in survivors:\n            idx = len(self.token_to_idx)\n            self.token_to_idx[token] = idx\n\n    def copy_from(self, item: dict):\n        super().copy_from(item)\n        self.counter = Counter(item['counter'].items()) if 'counter' in item else Counter()\n\n    def to_dict(self) -> dict:\n        d = super().to_dict()\n        d['counter'] = dict(self.counter.items())\n        return d\n\n\nclass Vocab3D(CustomVocab):\n    def __call__(self, some_token: Union[str, Iterable[str], Iterable[Iterable[str]]]) \\\n            -> Union[int, List[int], List[List[int]]]:\n        \"\"\"It supports 3D arrays of tokens.\n\n        Args:\n            some_token: Tokens of 1D to 3D\n\n        Returns:\n            A list of indices.\n\n        \"\"\"\n        if isinstance(some_token, (list, tuple, set)):\n            indices = []\n            if len(some_token) and isinstance(some_token[0], (list, tuple, set)):\n                for sent in some_token:\n                    inside = []\n                    for token in sent:\n                        inside.append(self.get_idx(token))\n                    indices.append(inside)\n                return indices\n            for token in some_token:\n                if isinstance(token, str):\n                    indices.append(self.get_idx(token))\n                else:\n                    indices.append([self.get_idx(x) for x in token])\n            return indices\n        else:\n            return self.get_idx(some_token)\n\n\ndef create_label_vocab() -> Vocab:\n    return Vocab(pad_token=None, unk_token=None)\n"
  },
  {
    "path": "hanlp/common/vocab_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-06-13 22:42\nfrom typing import List, Dict, Union, Iterable\n\nfrom hanlp_common.structure import Serializable\nfrom hanlp_common.constant import PAD, UNK\nimport tensorflow as tf\nfrom tensorflow.python.ops.lookup_ops import index_table_from_tensor\n\n\nclass VocabTF(Serializable):\n    def __init__(self, idx_to_token: List[str] = None, token_to_idx: Dict = None, mutable=True, pad_token=PAD,\n                 unk_token=UNK) -> None:\n        super().__init__()\n        if idx_to_token:\n            t2i = dict((token, idx) for idx, token in enumerate(idx_to_token))\n            if token_to_idx:\n                t2i.update(token_to_idx)\n            token_to_idx = t2i\n        if token_to_idx is None:\n            token_to_idx = {}\n            if pad_token:\n                token_to_idx[pad_token] = len(token_to_idx)\n            if unk_token:\n                token_to_idx[unk_token] = len(token_to_idx)\n        self.token_to_idx = token_to_idx\n        self.idx_to_token: list = None\n        self.mutable = mutable\n        self.pad_token = pad_token\n        self.unk_token = unk_token\n        self.token_to_idx_table: tf.lookup.StaticHashTable = None\n        self.idx_to_token_table = None\n\n    def __setitem__(self, token: str, idx: int):\n        assert self.mutable, 'Update an immutable Vocab object is not allowed'\n        self.token_to_idx[token] = idx\n\n    def __getitem__(self, key: Union[str, int, List]) -> Union[int, str, List]:\n        if isinstance(key, str):\n            return self.get_idx(key)\n        elif isinstance(key, int):\n            return self.get_token(key)\n        elif isinstance(key, list):\n            if len(key) == 0:\n                return []\n            elif isinstance(key[0], str):\n                return [self.get_idx(x) for x in key]\n            elif isinstance(key[0], int):\n                return [self.get_token(x) for x in key]\n\n    def __contains__(self, key: Union[str, int]):\n        if isinstance(key, str):\n            return key in self.token_to_idx\n        elif isinstance(key, int):\n            return 0 <= key < len(self.idx_to_token)\n        else:\n            return False\n\n    def add(self, token: str) -> int:\n        assert self.mutable, 'It is not allowed to call add on an immutable Vocab'\n        assert isinstance(token, str), f'Token type must be str but got {type(token)} from {token}'\n        assert token, 'Token must not be None or length 0'\n        idx = self.token_to_idx.get(token, None)\n        if idx is None:\n            idx = len(self.token_to_idx)\n            self.token_to_idx[token] = idx\n        return idx\n\n    def update(self, tokens: Iterable[str]) -> None:\n        \"\"\"Update the vocab with these tokens by adding them to vocab one by one.\n\n        Args:\n          tokens: Iterable[str]: \n\n        Returns:\n\n        \n        \"\"\"\n        assert self.mutable, 'It is not allowed to update an immutable Vocab'\n        for token in tokens:\n            self.add(token)\n\n    def get_idx(self, token: str) -> int:\n        idx = self.token_to_idx.get(token, None)\n        if idx is None:\n            if self.mutable:\n                idx = len(self.token_to_idx)\n                self.token_to_idx[token] = idx\n            else:\n                idx = self.token_to_idx.get(self.unk_token, None)\n        return idx\n\n    def get_idx_without_add(self, token: str) -> int:\n        idx = self.token_to_idx.get(token, None)\n        if idx is None:\n            idx = self.token_to_idx.get(self.safe_unk_token, None)\n        return idx\n\n    def get_token(self, idx: int) -> str:\n        if self.idx_to_token:\n            return self.idx_to_token[idx]\n\n        if self.mutable:\n            for token in self.token_to_idx:\n                if self.token_to_idx[token] == idx:\n                    return token\n\n    def has_key(self, token):\n        return token in self.token_to_idx\n\n    def __len__(self):\n        return len(self.token_to_idx)\n\n    def lock(self):\n        if self.locked:\n            return self\n        self.mutable = False\n        self.build_idx_to_token()\n        self.build_lookup_table()\n        return self\n\n    def build_idx_to_token(self):\n        max_idx = max(self.token_to_idx.values())\n        self.idx_to_token = [None] * (max_idx + 1)\n        for token, idx in self.token_to_idx.items():\n            self.idx_to_token[idx] = token\n\n    def build_lookup_table(self):\n        tensor = tf.constant(self.idx_to_token, dtype=tf.string)\n        self.token_to_idx_table = index_table_from_tensor(tensor, num_oov_buckets=1 if self.unk_idx is None else 0,\n                                                          default_value=-1 if self.unk_idx is None else self.unk_idx)\n        # self.idx_to_token_table = index_to_string_table_from_tensor(self.idx_to_token, self.safe_unk_token)\n\n    def unlock(self):\n        if not self.locked:\n            return\n        self.mutable = True\n        self.idx_to_token = None\n        self.idx_to_token_table = None\n        self.token_to_idx_table = None\n        return self\n\n    @property\n    def locked(self):\n        return not self.mutable\n\n    @property\n    def unk_idx(self):\n        if self.unk_token is None:\n            return None\n        else:\n            return self.token_to_idx.get(self.unk_token, None)\n\n    @property\n    def pad_idx(self):\n        if self.pad_token is None:\n            return None\n        else:\n            return self.token_to_idx.get(self.pad_token, None)\n\n    @property\n    def tokens(self):\n        return self.token_to_idx.keys()\n\n    def __str__(self) -> str:\n        return self.token_to_idx.__str__()\n\n    def summary(self, verbose=True) -> str:\n        # report = 'Length: {}\\n'.format(len(self))\n        # report += 'Samples: {}\\n'.format(str(list(self.token_to_idx.keys())[:min(50, len(self))]))\n        # report += 'Mutable: {}'.format(self.mutable)\n        # report = report.strip()\n        report = '[{}] = '.format(len(self))\n        report += str(list(self.token_to_idx.keys())[:min(50, len(self))])\n        if verbose:\n            print(report)\n        return report\n\n    def __call__(self, some_token: Union[str, List[str]]) -> Union[int, List[int]]:\n        if isinstance(some_token, list):\n            indices = []\n            for token in some_token:\n                indices.append(self.get_idx(token))\n            return indices\n        else:\n            return self.get_idx(some_token)\n\n    def lookup(self, token_tensor: tf.Tensor) -> tf.Tensor:\n        if self.mutable:\n            self.lock()\n        return self.token_to_idx_table.lookup(token_tensor)\n\n    def to_dict(self) -> dict:\n        idx_to_token = self.idx_to_token\n        pad_token = self.pad_token\n        unk_token = self.unk_token\n        mutable = self.mutable\n        items = locals().copy()\n        items.pop('self')\n        return items\n\n    def copy_from(self, item: dict):\n        for key, value in item.items():\n            setattr(self, key, value)\n        self.token_to_idx = {k: v for v, k in enumerate(self.idx_to_token)}\n        if not self.mutable:\n            self.build_lookup_table()\n\n    def lower(self):\n        self.unlock()\n        token_to_idx = self.token_to_idx\n        self.token_to_idx = {}\n        for token in token_to_idx.keys():\n            self.add(token.lower())\n        return self\n\n    @property\n    def first_token(self):\n        if self.idx_to_token:\n            return self.idx_to_token[0]\n        if self.token_to_idx:\n            return next(iter(self.token_to_idx))\n        return None\n\n    def merge(self, other):\n        for word, idx in other.token_to_idx.items():\n            self.get_idx(word)\n\n    @property\n    def safe_pad_token(self) -> str:\n        \"\"\"Get the pad token safely. It always returns a pad token, which is the token\n        closest to pad if not presented in the vocab.\n\n        Args:\n\n        Returns:\n\n        \n        \"\"\"\n        if self.pad_token:\n            return self.pad_token\n        if self.first_token:\n            return self.first_token\n        return PAD\n\n    @property\n    def safe_pad_token_idx(self) -> int:\n        return self.token_to_idx.get(self.safe_pad_token, 0)\n\n    @property\n    def safe_unk_token(self) -> str:\n        \"\"\"Get the unk token safely. It always returns a unk token, which is the token\n        closest to unk if not presented in the vocab.\n\n        Args:\n\n        Returns:\n\n        \n        \"\"\"\n        if self.unk_token:\n            return self.unk_token\n        if self.first_token:\n            return self.first_token\n        return UNK\n\n\ndef create_label_vocab() -> VocabTF:\n    return VocabTF(pad_token=None, unk_token=None)\n"
  },
  {
    "path": "hanlp/components/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-08-26 16:10\nfrom .pipeline import Pipeline"
  },
  {
    "path": "hanlp/components/amr/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-20 17:35\n"
  },
  {
    "path": "hanlp/components/amr/amrbart/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2022-12-05 17:53\n"
  },
  {
    "path": "hanlp/components/amr/amrbart/bart_amr_generation.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2022-12-05 17:56\nimport logging\nimport os.path\nfrom typing import Callable, Union, List\n\nimport penman\nimport torch\nfrom torch.utils.data import DataLoader\n\nfrom hanlp.components.amr.amrbart.data_interface.dataset import AMR2TextDataSet\nfrom hanlp.common.dataset import SortingSamplerBuilder, PadSequenceDataLoader\nfrom hanlp.common.torch_component import TorchComponent\nfrom hanlp.components.amr.seq2seq.dataset.dataset import AMRDataset\nfrom hanlp.layers.transformers.pt_imports import AutoConfig_\nfrom hanlp.utils.time_util import CountdownTimer\nfrom hanlp_common.constant import IDX\nfrom hanlp_common.util import reorder\nfrom hanlp.components.amr.amrbart.model_interface.modeling_bart import BartForConditionalGeneration\nfrom hanlp.components.amr.amrbart.model_interface.tokenization_bart import AMRBartTokenizer\nfrom hanlp.components.amr.amrbart.preprocess.read_and_process import dfs_linearize\n\n\nclass BART_AMR_Generation(TorchComponent):\n    def __init__(self, **kwargs) -> None:\n        super().__init__(**kwargs)\n        self.tokenizer: AMRBartTokenizer = None\n        self.transformer_config = None\n        self.model: BartForConditionalGeneration = None\n\n    def build_dataloader(self, data, batch_size=32, shuffle=False, device=None, logger: logging.Logger = None,\n                         sampler_builder=None,\n                         **kwargs) -> DataLoader:\n        dataset = AMRDataset(data, generate_idx=True, cache=True)\n        dataset.append_transform(lambda x: {**x, 'lamr': ' '.join(dfs_linearize(x['amr']))})\n        dataset.append_transform(\n            lambda x: AMR2TextDataSet.tokenize(x, tokenizer=self.tokenizer, text='text', amr='lamr')\n        )\n        if not sampler_builder:\n            sampler_builder = SortingSamplerBuilder(batch_max_tokens=500)\n        sampler = sampler_builder.build([len(x['input_ids']) for x in dataset], shuffle, 1)\n        return PadSequenceDataLoader(dataset, batch_size, shuffle, device=device, batch_sampler=sampler,\n                                     pad={'input_ids': self.transformer_config.pad_token_id,\n                                          'labels': self.transformer_config.pad_token_id})\n\n    def build_optimizer(self, **kwargs):\n        pass\n\n    def build_criterion(self, **kwargs):\n        pass\n\n    def build_metric(self, **kwargs):\n        pass\n\n    def execute_training_loop(self, trn: DataLoader, dev: DataLoader, epochs, criterion, optimizer, metric, save_dir,\n                              logger: logging.Logger, devices, ratio_width=None, **kwargs):\n        pass\n\n    def fit_dataloader(self, trn: DataLoader, criterion, optimizer, metric, logger: logging.Logger, **kwargs):\n        pass\n\n    def evaluate_dataloader(self, data: DataLoader, criterion: Callable, metric=None, output=False, **kwargs):\n        pass\n\n    def build_model(self, training=True, transformer=None, **kwargs) -> torch.nn.Module:\n        model = BartForConditionalGeneration.from_pretrained(\n            transformer,\n            config=self.transformer_config,\n        )\n        if not training:\n            model.eval()\n        model.resize_token_embeddings(len(self.tokenizer))\n        return model\n\n    def input_is_flat(self, data):\n        return isinstance(data, (str, penman.Graph))\n\n    def predict(\n            self,\n            data: Union[str, List[str]], num_beams=5, max_length=1024, beautiful_amr_graph=True, verbose=False,\n            **kwargs\n    ):\n        flat = self.input_is_flat(data)\n        if flat:\n            data = [data]\n        dataloader = self.build_dataloader([{'amr': penman.loads(x)[0] if isinstance(x, str) else x} for x in data],\n                                           **self.config, device=self.device)\n        orders = []\n        results = []\n        if verbose:\n            timer = CountdownTimer(len(dataloader))\n        for batch in dataloader:\n            pieces = self.predict_batch(batch, num_beams, max_length)\n            results.extend(pieces)\n            orders.extend(batch[IDX])\n            if verbose:\n                # noinspection PyUnboundLocalVariable\n                timer.log()\n        results = reorder(results, orders)\n        if flat:\n            results = results[0]\n        return results\n\n    def predict_batch(self, batch, num_beams, max_length):\n        tokenizer = self.tokenizer\n        input_ids = batch['input_ids']\n        preds = self.model.generate(\n            input_ids,\n            num_beams=num_beams,\n            use_cache=True,\n            decoder_start_token_id=tokenizer.eos_token_id,\n            eos_token_id=tokenizer.eos_token_id,\n            no_repeat_ngram_size=0,\n            max_length=max_length,\n            min_length=0,\n            length_penalty=1.0,\n        )\n        # tokens = batch['tgt']\n        decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)\n        decoded_preds = [x.strip() for x in decoded_preds]\n        return decoded_preds\n\n    def load_config(self, save_dir: str, filename='config.json', **kwargs):\n        if os.path.isdir(save_dir):\n            super().load_config(save_dir, filename, **kwargs)\n            transformer = self.config.transformer\n        else:\n            self.config.transformer = transformer = save_dir\n        self.transformer_config = AutoConfig_.from_pretrained(transformer)\n\n    def load_vocabs(self, save_dir, filename='vocabs.json'):\n        self.tokenizer = AMRBartTokenizer.from_pretrained(\n            self.config.transformer,\n            use_fast=True,\n        )\n\n    def load_weights(self, save_dir, filename='model.pt', **kwargs):\n        pass\n"
  },
  {
    "path": "hanlp/components/amr/amrbart/bart_amr_parser.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2022-12-05 17:56\nimport logging\nimport os.path\nfrom typing import Callable, Union, List\n\nimport datetime\nimport torch\nfrom torch.utils.data import DataLoader\n\nfrom hanlp.components.amr.amrbart.data_interface.dataset import AMRParsingDataSet\nfrom hanlp.common.dataset import SortingSamplerBuilder, PadSequenceDataLoader\nfrom hanlp.common.torch_component import TorchComponent\nfrom hanlp.components.amr.seq2seq.dataset.dataset import AMRDataset\nfrom hanlp.components.amr.seq2seq.dataset.penman import AMRGraph\nfrom hanlp.components.amr.seq2seq.evaluation import write_predictions, compute_smatch\nfrom hanlp.layers.transformers.pt_imports import AutoConfig_\nfrom hanlp.metrics.amr.smatch_eval import smatch_eval\nfrom hanlp.metrics.mtl import MetricDict\nfrom hanlp.utils.time_util import CountdownTimer\nfrom hanlp_common.constant import IDX\nfrom hanlp_common.util import reorder\nfrom hanlp.components.amr.amrbart.model_interface.modeling_bart import BartForConditionalGeneration\nfrom hanlp.components.amr.amrbart.model_interface.tokenization_bart import AMRBartTokenizer\n\n\nclass BART_AMR_Parser(TorchComponent):\n    def __init__(self, **kwargs) -> None:\n        super().__init__(**kwargs)\n        self.tokenizer: AMRBartTokenizer = None\n        self.transformer_config = None\n        self.model: BartForConditionalGeneration = None\n\n    def build_dataloader(self, data, batch_size=32, shuffle=False, device=None, logger: logging.Logger = None,\n                         sampler_builder=None,\n                         **kwargs) -> DataLoader:\n        dataset = AMRDataset(data, generate_idx=True, cache=True)\n        if isinstance(data, str):\n            dataset.append_transform(lambda x: {**x, 'text': x['amr'].metadata['snt']})\n        dataset.append_transform(\n            lambda x: AMRParsingDataSet.tokenize(x, tokenizer=self.tokenizer, text='text')\n        )\n        if not sampler_builder:\n            sampler_builder = SortingSamplerBuilder(batch_max_tokens=500)\n        sampler = sampler_builder.build([len(x['input_ids']) for x in dataset], shuffle, 1)\n        return PadSequenceDataLoader(dataset, batch_size, shuffle, device=device, batch_sampler=sampler,\n                                     pad={'input_ids': self.transformer_config.pad_token_id,\n                                          'labels': self.transformer_config.pad_token_id})\n\n    def build_optimizer(self, **kwargs):\n        pass\n\n    def build_criterion(self, **kwargs):\n        pass\n\n    def build_metric(self, **kwargs):\n        pass\n\n    def execute_training_loop(self, trn: DataLoader, dev: DataLoader, epochs, criterion, optimizer, metric, save_dir,\n                              logger: logging.Logger, devices, ratio_width=None, **kwargs):\n        pass\n\n    def fit_dataloader(self, trn: DataLoader, criterion, optimizer, metric, logger: logging.Logger, **kwargs):\n        pass\n\n    def build_model(self, training=True, transformer=None, **kwargs) -> torch.nn.Module:\n        model = BartForConditionalGeneration.from_pretrained(\n            transformer,\n            config=self.transformer_config,\n        )\n        if not training:\n            model.eval()\n        model.resize_token_embeddings(len(self.tokenizer))\n        return model\n\n    def input_is_flat(self, data):\n        return isinstance(data, str)\n\n    def predict(\n            self,\n            data: Union[str, List[str]], num_beams=5, max_length=1024, beautiful_amr_graph=True, verbose=False,\n            **kwargs\n    ):\n        flat = self.input_is_flat(data)\n        if flat:\n            data = [data]\n        dataloader = self.build_dataloader([{'text': x} for x in data], **self.config, device=self.device)\n        orders = []\n        results = []\n        # inputs, logits, labels, loss = torch.load('/local/scratch/hhe43/amrbart/batch.pt')\n        if verbose:\n            timer = CountdownTimer(len(dataloader))\n        for batch in dataloader:\n            pieces = self.predict_batch(batch, num_beams, max_length)\n            results.extend(pieces)\n            orders.extend(batch[IDX])\n            if verbose:\n                # noinspection PyUnboundLocalVariable\n                timer.log()\n        results = reorder(results, orders)\n        if flat:\n            results = results[0]\n        return results\n\n    def predict_batch(self, batch, num_beams, max_length):\n        tokenizer = self.tokenizer\n        input_ids = batch['input_ids']\n        preds = self.model.generate(\n            input_ids,\n            num_beams=num_beams,\n            num_return_sequences=num_beams,\n            use_cache=True,\n            decoder_start_token_id=tokenizer.amr_bos_token_id,\n            eos_token_id=tokenizer.amr_eos_token_id,\n            no_repeat_ngram_size=0,\n            max_length=max_length,\n            min_length=0,\n            length_penalty=1.0,\n        ).tolist()\n        # tokens = batch['tgt']\n        graphs = []\n        for i in range(0, len(preds), num_beams):\n            graphs_same_source = []\n            for j in range(i, i + num_beams):\n                ith_pred = preds[j]\n                ith_pred[0] = tokenizer.bos_token_id\n                ith_pred = [\n                    tokenizer.eos_token_id if itm == tokenizer.amr_eos_token_id else itm\n                    for itm in ith_pred if itm != tokenizer.pad_token_id\n                ]\n\n                graph, status, (lin, backr) = tokenizer.decode_amr(\n                    ith_pred, restore_name_ops=False\n                )\n                graph.status = status\n                graph.nodes = lin\n                graph.backreferences = backr\n                graph.tokens = ith_pred\n                graphs_same_source.append(graph)\n            graphs_same_source[:] = \\\n                tuple(zip(*sorted(enumerate(graphs_same_source), key=lambda x: (x[1].status.value, x[0]))))[1]\n            graphs.append(graphs_same_source)\n        # assert len(graphs) == len(tokens), f\"inconsistent lengths {len(graphs)} vs {len(tokens)}\"\n        # for idx, gps, snt in zip(batch[IDX], graphs, tokens):\n        #     for gp in gps:\n        #         gp.metadata = {\"id\": str(idx), \"annotator\": \"bart-amr\",\n        #                        \"snt\": snt.replace(\"<AMR>\", '').replace(\"</AMR>\", '').strip()}\n        pieces = [AMRGraph(g.triples, g.top, g.epidata, g.metadata) for g in [gs[0] for gs in graphs]]\n        return pieces\n\n    def load_config(self, save_dir: str, filename='config.json', **kwargs):\n        if os.path.isdir(save_dir):\n            super().load_config(save_dir, filename, **kwargs)\n            transformer = self.config.transformer\n        else:\n            self.config.transformer = transformer = save_dir\n        self.transformer_config = AutoConfig_.from_pretrained(transformer)\n\n    def load_vocabs(self, save_dir, filename='vocabs.json'):\n        self.tokenizer = AMRBartTokenizer.from_pretrained(\n            self.config.transformer,\n            use_fast=True,\n        )\n\n    def load_weights(self, save_dir, filename='model.pt', **kwargs):\n        pass\n\n    @torch.no_grad()\n    def evaluate_dataloader(self, data: DataLoader, criterion: Callable, metric=None, output=False, ratio_width=None,\n                            logger=None, input=None, use_fast=False, num_beams=5, max_length=1024,\n                            **kwargs):\n        self.model.eval()\n        timer = CountdownTimer(len(data))\n        graphs = []\n        orders = []\n        smatch = 0\n        for idx, batch in enumerate(data):\n            graphs_per_batch = self.predict_batch(batch, num_beams, max_length)\n            # Copy meta data from gold graph\n            for gp, gg in zip(graphs_per_batch, batch['amr']):\n                metadata = gg.metadata.copy()\n                metadata['annotator'] = f'{self.transformer_config.name_or_path}-amr'\n                metadata['date'] = str(datetime.datetime.now())\n                if 'save-date' in metadata:\n                    del metadata['save-date']\n                gp.metadata = metadata\n            graphs.extend(graphs_per_batch)\n            orders.extend(batch[IDX])\n            if idx == timer.total - 1:\n                graphs = reorder(graphs, orders)\n                write_predictions(output, None, graphs)\n                try:\n                    if use_fast:\n                        smatch = compute_smatch(output, input)\n                    else:\n                        smatch = smatch_eval(output, input, use_fast=False)\n                except:\n                    pass\n                timer.log(smatch.cstr() if isinstance(smatch, MetricDict) else f'{smatch:.2%}', ratio_percentage=False,\n                          logger=logger)\n            else:\n                timer.log(ratio_percentage=False, logger=logger)\n\n        return smatch\n\n    def evaluate(self, tst_data, save_dir=None, logger: logging.Logger = None, batch_size=None, output=True, **kwargs):\n        return super().evaluate(tst_data, save_dir, logger, batch_size, output, **kwargs)\n"
  },
  {
    "path": "hanlp/components/amr/amrbart/common/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2022-12-05 17:53\n"
  },
  {
    "path": "hanlp/components/amr/amrbart/common/constant.py",
    "content": "# coding:utf-8\n# MIT License\n#\n# Copyright (c) 2022 xfbai\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\n\nfrom transformers import (\n    AutoTokenizer,\n    AutoModelForSeq2SeqLM,\n    BartTokenizer,\n    BartForConditionalGeneration,\n    T5Tokenizer,\n    T5Model,\n    T5ForConditionalGeneration,\n)\nfrom transformers.optimization import (\n    get_cosine_schedule_with_warmup,\n    get_cosine_with_hard_restarts_schedule_with_warmup,\n    get_linear_schedule_with_warmup,\n    get_polynomial_decay_schedule_with_warmup,\n    get_constant_schedule_with_warmup,\n)\n\nraw_special_tokens = ['Ġcause-01', 'Ġpossible-01', 'Ġcontrast-01', 'Ġsay-01', 'Ġhave-03', 'Ġgovern-01', 'Ġstate-01',\n                      'Ġthink-01', 'Ġdo-02', 'Ġwant-01', 'Ġknow-01', 'Ġrecommend-01', 'Ġsee-01', 'Ġresemble-01',\n                      'Ġmean-01', 'Ġobligate-01', 'Ġuse-01', 'Ġgood-02', 'Ġneed-01', 'Ġwork-01', 'Ġpay-01', 'Ġget-01',\n                      'Ġattack-01', 'Ġreal-04', 'Ġbelieve-01', 'Ġsupport-01', 'Ġreport-01', 'Ġtry-01', 'Ġsame-01',\n                      'Ġtax-01', 'Ġoppose-01', 'Ġlive-01', 'Ġtell-01', 'Ġmake-02', 'Ġdie-01', 'Ġkill-01', 'Ġnew-01',\n                      'Ġgive-01', 'Ġincrease-01', 'Ġagree-01', 'Ġactual-02', 'Ġgo-02', 'Ġright-05', 'Ġvote-01',\n                      'Ġmake-01', 'Ġtake-01', 'Ġseem-01', 'Ġtalk-01', 'Ġissue-02', 'Ġbecome-01', 'Ġpost-01', 'Ġhelp-01',\n                      'Ġstart-01', 'Ġend-01', 'Ġdevelop-02', 'Ġdecide-01', 'Ġfind-01', 'Ġclaim-01', 'Ġdefend-01',\n                      'Ġlead-02', 'Ġhigh-02', 'Ġcontrol-01', 'Ġfree-04', 'Ġtraffic-01', 'Ġlong-03', 'Ġprovide-01',\n                      'Ġcome-01', 'Ġplan-01', 'Ġproduce-01', 'Ġchange-01', 'Ġdiffer-02', 'Ġmarry-01', 'Ġemploy-01',\n                      'Ġchoose-01', 'Ġfight-01', 'Ġmeet-03', 'Ġcall-01', 'Ġread-01', 'Ġunderstand-01', 'Ġsure-02',\n                      'Ġcapable-01', 'Ġallow-01', 'Ġcrime-02', 'Ġinclude-01', 'Ġsell-01', 'Ġinfer-01', 'Ġshow-01',\n                      'Ġfeel-01', 'Ġwar-01', 'Ġquestion-01', 'Ġlook-01', 'Ġopine-01', 'Ġlegal-02', 'Ġlose-02',\n                      'Ġstop-01', 'Ġcreate-01', 'Ġcost-01', 'Ġcontinue-01', 'Ġbad-07', 'Ġact-02', 'Ġcare-03', 'Ġwin-01',\n                      'Ġdiscuss-01', 'Ġdestroy-01', 'Ġpolicy-01', 'Ġelect-01', 'Ġgo-01', 'Ġtrue-01', 'Ġlie-08',\n                      'Ġbase-02', 'Ġinsure-02', 'Ġinvest-01', 'Ġfund-01', 'Ġliberal-02', 'Ġtrade-01', 'Ġspeak-01',\n                      'Ġinvolve-01', 'Ġfail-01', 'Ġhear-01', 'Ġlet-01', 'Ġhope-01', 'Ġinterest-01', 'Ġthreaten-01',\n                      'Ġgrow-01', 'Ġdeal-01', 'Ġspend-01', 'Ġexist-01', 'Ġbegin-01', 'Ġdepend-01', 'Ġarrest-01',\n                      'Ġprove-01', 'Ġbuy-01', 'Ġput-01', 'Ġget-05', 'Ġactivity-06', 'Ġoffer-01', 'Ġpersonal-02',\n                      'Ġprotect-01', 'Ġquote-01', 'Ġwrite-01', 'Ġown-01', 'Ġbuild-01', 'Ġbenefit-01', 'Ġrelation-03',\n                      'Ġequal-01', 'Ġsurrender-01', 'Ġexpect-01', 'Ġlike-01', 'Ġcooperate-01', 'Ġmove-01', 'Ġexcept-01',\n                      'Ġrealize-01', 'Ġstrong-02', 'Ġhate-01', 'Ġargue-01', 'Ġask-01', 'Ġanswer-01', 'Ġlow-04',\n                      'Ġcase-03', 'Ġresult-01', 'Ġeasy-05', 'Ġhard-02', 'Ġconcern-01', 'Ġsuspect-01', 'Ġbear-02',\n                      'Ġserve-01', 'Ġaccept-01', 'Ġclear-06', 'Ġlove-01', 'Ġdemand-01', 'Ġlaunch-01', 'Ġexplain-01',\n                      'Ġwrong-04', 'Ġright-06', 'Ġrequire-01', 'Ġaffect-01', 'Ġeffort-01', 'Ġforce-01', 'Ġlook-02',\n                      'Ġwatch-01', 'Ġout-06', 'Ġoperate-01', 'Ġattempt-01', 'Ġban-01', 'Ġstudy-01', 'Ġsuggest-01',\n                      'Ġlikely-01', 'Ġconcern-02', 'Ġthank-01', 'Ġpublic-02', 'Ġwork-09', 'Ġexemplify-01', 'Ġintend-01',\n                      'Ġprice-01', 'Ġrespond-01', 'Ġpropose-01', 'Ġvisit-01', 'Ġcomplete-02', 'Ġtransfer-01',\n                      'Ġaccuse-01', 'Ġcounter-01', 'Ġcut-02', 'Ġsimple-02', 'Ġcare-01', 'Ġcharge-05', 'Ġrepresent-01',\n                      'Ġsucceed-01', 'Ġlocal-02', 'Ġmurder-01', 'Ġremember-01', 'Ġsend-01', 'Ġevidence-01',\n                      'Ġresearch-01', 'Ġmajor-02', 'Ġwait-01', 'Ġestablish-01', 'Ġremain-01', 'Ġtest-01', 'Ġkeep-02',\n                      'Ġexport-01', 'Ġannounce-01', 'Ġbomb-01', 'Ġfavor-01', 'Ġdeny-01', 'Ġrun-01', 'Ġexperience-01',\n                      'Ġexpert-01', 'Ġprevent-01', 'Ġfair-01', 'Ġknow-02', 'Ġgeneral-02', 'Ġapprove-01', 'Ġwhite-02',\n                      'Ġdescribe-01', 'Ġshare-01', 'Ġconsider-01', 'Ġcase-04', 'Ġreceive-01', 'Ġignore-01', 'Ġlink-01',\n                      'Ġkeep-01', 'Ġcomment-01', 'Ġsex-01', 'Ġlaugh-01', 'Ġinvestigate-01', 'Ġview-02',\n                      'Ġproliferate-01', 'Ġrefuse-01', 'Ġfear-01', 'Ġget-03', 'Ġwill-02', 'Ġrape-01', 'Ġallege-01',\n                      'Ġget-04', 'Ġstay-01', 'Ġrise-01', 'Ġsupply-01', 'Ġdirect-02', 'Ġhonest-01', 'Ġdebate-01',\n                      'Ġobvious-01', 'Ġappear-02', 'Ġcampaign-01', 'Ġblack-05', 'Ġreduce-01', 'Ġask-02',\n                      'Ġcriticize-01', 'Ġguess-01', 'Ġlearn-01', 'Ġseek-01', 'Ġaccess-01', 'Ġsafe-01', 'Ġwish-01',\n                      'Ġwrong-02', 'Ġeducate-01', 'Ġconflict-01', 'Ġrespect-01', 'Ġreach-01', 'Ġage-01', 'Ġmention-01',\n                      'Ġexecute-01', 'Ġfind-02', 'Ġjudge-01', 'Ġbring-01', 'Ġblame-01', 'Ġhead-01', 'Ġwell-09',\n                      'Ġensure-01', 'Ġarm-01', 'Ġcover-01', 'Ġserious-02', 'Ġtreat-01', 'Ġteach-01', 'Ġdoubt-01',\n                      'Ġimmigrate-01', 'Ġinvade-01', 'Ġsmuggle-01', 'Ġlack-01', 'Ġearn-01', 'Ġhold-01', 'Ġlimit-01',\n                      'Ġparticipate-01', 'Ġsentence-01', 'Ġdamage-01', 'Ġconsider-02', 'Ġname-01', 'Ġsorry-01',\n                      'Ġrelate-01', 'Ġcriminal-03', 'Ġleft-19', 'Ġadmit-01', 'Ġadministrate-01', 'Ġtarget-01',\n                      'Ġrun-02', 'Ġgo-06', 'Ġimprove-01', 'Ġconstruct-01', 'Ġmoral-02', 'Ġfollow-01', 'Ġcorrect-02',\n                      'Ġprotest-01', 'Ġleave-11', 'Ġaid-01', 'Ġvalue-01', 'Ġsense-02', 'Ġdrop-01', 'Ġface-01',\n                      'Ġserious-01', 'Ġseize-01', 'Ġtrain-01', 'Ġwarn-01', 'Ġavoid-01', 'Ġeffective-04', 'Ġdeserve-01',\n                      'Ġplay-01', 'Ġenter-01', 'Ġregulate-01', 'Ġnear-02', 'Ġborder-01', 'Ġsolve-01', 'Ġprefer-01',\n                      'Ġviolate-01', 'Ġrelease-01', 'Ġcite-01', 'Ġfocus-01', 'Ġadvise-01', 'Ġsound-01', 'Ġrisk-01',\n                      'Ġreturn-01', 'Ġlist-01', 'Ġsignificant-02', 'Ġhire-01', 'Ġsurprise-01', 'Ġopen-01', 'Ġnice-01',\n                      'Ġraise-01', 'Ġmaintain-01', 'Ġprivate-03', 'Ġimplement-01', 'Ġassist-01', 'Ġcall-02',\n                      'Ġcompare-01', 'Ġprofit-01', 'Ġcontribute-01', 'Ġhave-to-do-with-04', 'Ġcorrupt-01', 'Ġclose-10',\n                      'Ġsuffer-01', 'Ġexpand-01', 'Ġwonder-01', 'Ġresponsible-01', 'Ġtotal-01', 'Ġspecific-02',\n                      'Ġpass-01', 'Ġhappy-01', 'Ġassume-02', 'Ġchance-02', 'Ġremove-01', 'Ġadd-02', 'Ġmanufacture-01',\n                      'Ġexpress-01', 'Ġinspect-01', 'Ġwalk-01', 'Ġgood-03', 'Ġrule-01', 'Ġmanage-01', 'Ġhold-04',\n                      'Ġspecial-02', 'Ġinfluence-01', 'Ġexchange-01', 'Ġtake-10', 'Ġconvict-01', 'Ġprocess-02',\n                      'Ġtravel-01', 'Ġcarry-01', 'Ġdefine-01', 'Ġdisagree-01', 'Ġsave-02', 'Ġpermit-01', 'Ġestimate-01',\n                      'Ġrate-01', 'Ġcall-03', 'Ġsingle-02', 'Ġabuse-01', 'Ġsign-01', 'Ġrule-03', 'Ġact-01',\n                      'Ġachieve-01', 'Ġintervene-01', 'Ġfall-01', 'Ġattend-02', 'Ġfeel-02', 'Ġadopt-01', 'Ġfollow-02',\n                      'Ġgo-on-15', 'Ġloan-01', 'Ġnegotiate-01', 'Ġhit-01', 'Ġcondition-01', 'Ġshort-07', 'Ġpromise-01',\n                      'Ġrebel-01', 'Ġpromote-02', 'Ġstrengthen-01', 'Ġsanction-02', 'Ġwarm-01', 'Ġbehave-01',\n                      'Ġhave-06', 'Ġsuffice-01', 'Ġlead-03', 'Ġtry-02', 'Ġlike-02', 'Ġfire-01', 'Ġdrive-01', 'Ġfly-01',\n                      'Ġgain-02', 'Ġafford-01', 'Ġexplode-01', 'Ġpoint-out-02', 'Ġconsume-01', 'Ġmeasure-02',\n                      'Ġreform-01', 'Ġenjoy-01', 'Ġsit-01', 'Ġavailable-02', 'Ġstrike-01', 'Ġsign-02', 'Ġcome-03',\n                      'Ġnatural-03', 'Ġorganize-01', 'Ġprepare-02', 'Ġreplace-01', 'Ġhanging-07', 'Ġleave-15',\n                      'Ġretire-01', 'Ġimport-01', 'Ġrange-01', 'Ġokay-04', 'Ġcover-03', 'Ġimagine-01', 'Ġkey-02',\n                      'Ġsurvive-01', 'Ġfree-03', 'Ġbase-01', 'Ġcomplain-01', 'Ġnormal-02', 'Ġcomplete-01', 'Ġreveal-01',\n                      'Ġenforce-01', 'Ġdetermine-01', 'Ġvictimize-01', 'Ġrepeat-01', 'Ġinterview-01', 'Ġmake-05',\n                      'Ġdonate-01', 'Ġsteal-01', 'Ġquick-02', 'Ġattract-01', 'Ġanalyze-01', 'Ġally-01', 'Ġsuppose-01',\n                      'Ġresponsible-03', 'Ġclose-01', 'Ġcombat-01', 'Ġidentify-01', 'Ġsuppose-02', 'Ġrecord-01',\n                      'Ġnominate-01', 'Ġrely-01', 'Ġturn-02', 'Ġhandle-01', 'Ġprocess-01', 'Ġpredict-01', 'Ġdeploy-01',\n                      'Ġfortunate-01', 'Ġeat-01', 'Ġjustify-01', 'Ġexpend-01', 'Ġbullshit-01', 'Ġdiscover-01',\n                      'Ġenrich-01', 'Ġcommit-02', 'Ġshoot-02', 'Ġcheap-02', 'Ġreject-01', 'Ġweak-02', 'Ġpowerful-02',\n                      'Ġdispute-01', 'Ġlegislate-01', 'Ġissue-01', 'Ġarrive-01', 'Ġjoin-01', 'Ġapply-02',\n                      'Ġindicate-01', 'Ġengage-01', 'Ġinnocent-01', 'Ġfast-02', 'Ġpressure-01', 'Ġpublish-01',\n                      'Ġobtain-01', 'Ġsad-02', 'Ġconfirm-01', 'Ġtreat-03', 'Ġlead-01', 'Ġlisten-01', 'Ġoffend-01',\n                      'Ġaddress-02', 'Ġword-01', 'Ġright-08', 'Ġnote-01', 'Ġcontain-01', 'Ġpurchase-01', 'Ġrequest-01',\n                      'Ġgood-04', 'Ġdesign-01', 'Ġnotice-01', 'Ġpresent-01', 'Ġshock-01', 'Ġright-02', 'Ġtransport-01',\n                      'Ġdeliver-01', 'Ġburn-01', 'Ġfault-01', 'Ġmatter-01', 'Ġabort-01', 'Ġstick-01', 'Ġconnect-01',\n                      'Ġconclude-01', 'Ġcontract-02', 'Ġpossess-01', 'Ġend-up-03', 'Ġsearch-01', 'Ġget-02',\n                      'Ġqualify-02', 'Ġreact-01', 'Ġconfuse-01', 'Ġanger-01', 'Ġpursue-01', 'Ġreside-01',\n                      'Ġrelevant-01', 'Ġoccupy-01', 'Ġwithdraw-01', 'Ġokay-01', 'Ġconform-01', 'Ġdemonstrate-01',\n                      'Ġwear-01', 'Ġhave-04', 'Ġdecrease-01', 'Ġpunish-01', 'Ġpractice-01', 'Ġcapture-01', 'Ġgo-03',\n                      'Ġpoll-01', 'Ġshow-04', 'Ġrefer-01', 'Ġcommit-01', 'Ġdisarm-01', 'Ġbelong-01', 'Ġdivide-02',\n                      'Ġdrink-01', 'Ġdesire-01', 'Ġsave-01', 'Ġignorant-02', 'Ġperfect-02', 'Ġposition-02', 'Ġcrap-01',\n                      'Ġinsult-01', 'Ġprivate-02', 'Ġwaste-01', 'Ġguilty-01', 'Ġeliminate-01', 'Ġmortgage-01',\n                      'Ġworth-01', 'Ġinherit-01', 'Ġthrow-01', 'Ġtour-01', 'Ġsuspend-01', 'Ġharm-01', 'Ġimpose-01',\n                      'Ġimprison-01', 'Ġrecognize-01', 'Ġprosecute-01', 'Ġview-01', 'Ġforget-01', 'Ġfound-01',\n                      'Ġchallenge-01', 'Ġtrouble-01', 'Ġsecure-02', 'Ġorder-01', 'Ġpartner-01', 'Ġspend-02',\n                      'Ġprogressive-02', 'Ġaccount-01', 'Ġblock-01', 'Ġguarantee-01', 'Ġconvince-01', 'Ġworry-02',\n                      'Ġendanger-01', 'Ġmovement-07', 'Ġfuck-01', 'Ġextend-01', 'Ġseparate-02', 'Ġbalance-01',\n                      'Ġlose-03', 'Ġpower-01', 'Ġsue-02', 'Ġurge-01', 'Ġcheck-01', 'Ġpoint-01', 'Ġturn-01',\n                      'Ġprogress-01', 'Ġrecover-01', 'Ġridiculous-02', 'Ġaccompany-01', 'Ġappear-01', 'Ġworry-01',\n                      'Ġplace-01', 'Ġattend-01', 'Ġsleep-01', 'Ġbreak-01', 'Ġfind-out-03', 'Ġbias-01', 'Ġaccord-03',\n                      'Ġwide-02', 'Ġenable-01', 'Ġaffair-02', 'Ġhide-01', 'Ġhold-02', 'Ġrecognize-02', 'Ġback-01',\n                      'Ġbet-01', 'Ġhack-04', 'Ġacquire-01', 'Ġtake-04', 'Ġpenalize-01', 'Ġmessage-01', 'Ġready-02',\n                      'Ġcease-01', 'Ġcrazy-03', 'Ġbad-04', 'Ġcompete-02', 'Ġcontact-01', 'Ġsource-01', 'Ġset-up-03',\n                      'Ġrestrict-01', 'Ġregard-01', 'Ġwitness-01', 'Ġlabor-01', 'Ġsmoke-02', 'Ġkick-01', 'Ġcompete-01',\n                      'Ġhouse-01', 'Ġhurt-01', 'Ġimprovise-01', 'Ġfinance-01', 'Ġinsist-01', 'Ġfarm-01', 'Ġapply-01',\n                      'Ġstep-01', 'Ġdeep-02', 'Ġpride-01', 'Ġbill-01', 'Ġpretend-01', 'Ġfill-01', 'Ġfine-04',\n                      'Ġstop-03', 'Ġoffend-03', 'Ġadvertise-01', 'Ġstand-01', 'Ġaim-02', 'Ġimpact-01', 'Ġfeed-01',\n                      'Ġgrant-01', 'Ġlast-01', 'Ġform-01', 'Ġdrive-02', 'Ġengineer-01', 'Ġinjure-01', 'Ġdevelop-01',\n                      'Ġpresent-02', 'Ġsubsidize-01', 'Ġbring-up-02', 'Ġintelligent-01', 'Ġwelcome-01', 'Ġtake-away-05',\n                      'Ġresolve-01', 'Ġappropriate-02', 'Ġencourage-01', 'Ġperform-02', 'Ġgo-back-19', 'Ġdeclare-02',\n                      'Ġfull-09', 'Ġhopeful-03', 'Ġconduct-01', 'Ġsurgery-01', 'Ġdetain-01', 'Ġrelative-05',\n                      'Ġcount-01', 'Ġglad-02', 'Ġrare-02', 'Ġcome-out-09', 'Ġapproach-02', 'Ġrace-02', 'Ġbattle-01',\n                      'Ġcross-02', 'Ġmove-02', 'Ġquestion-03', 'Ġadminister-01', 'Ġgrow-03', 'Ġmeet-02', 'Ġdown-03',\n                      'Ġmeet-01', 'Ġcondemn-01', 'Ġreason-01', 'Ġcarry-out-03', 'Ġworth-02', 'Ġinform-01', 'Ġstable-03',\n                      'Ġstand-11', 'Ġutilize-01', 'Ġperpetrate-01', 'Ġassociate-01', 'Ġapologize-01', 'Ġcredit-01',\n                      'Ġdisgust-01', 'Ġspread-03', 'Ġcommand-02', 'Ġsense-01', 'Ġdetail-01', 'Ġdefeat-01',\n                      'Ġdistribute-01', 'Ġgive-up-07', 'Ġpain-01', 'Ġship-01', 'Ġkeep-04', 'Ġaddict-01',\n                      'Ġcompromise-01', 'Ġlegitimate-02', 'Ġregular-02', 'Ġpick-01', 'Ġsource-02', 'Ġraid-01',\n                      'Ġhard-04', 'Ġrain-01', 'Ġcommunicate-01', 'Ġmarket-01', 'Ġlower-05', 'Ġill-01', 'Ġdefraud-01',\n                      'Ġposition-01', 'Ġterrible-01', 'Ġdivorce-01', 'Ġamaze-01', 'Ġedit-01', 'Ġspread-02',\n                      'Ġclarify-10', 'Ġargue-02', 'Ġpush-01', 'Ġmiss-01', 'Ġimply-01', 'Ġdiscriminate-02', 'Ġlight-06',\n                      'Ġappoint-01', 'Ġdelay-01', 'Ġgross-03', 'Ġput-03', 'Ġintroduce-02', 'Ġstandard-02', 'Ġpull-01',\n                      'Ġdraw-02', 'Ġgo-08', 'Ġaim-01', 'Ġmodern-02', 'Ġdare-01', 'Ġneighbor-01', 'Ġconfront-01',\n                      'Ġsuperior-01', 'Ġreasonable-02', 'Ġschedule-01', 'Ġadd-01', 'Ġnew-02', 'Ġlend-01', 'Ġdouble-01',\n                      'Ġfinish-01', 'Ġraise-03', 'Ġexcuse-02', 'Ġmonitor-01', 'Ġobserve-01', 'Ġpopular-02',\n                      'Ġcharge-01', 'Ġbudget-01', 'Ġnegative-03', 'Ġdirect-01', 'Ġrid-01', 'Ġmake-18', 'Ġmean-02',\n                      'Ġfame-01', 'Ġjoke-01', 'Ġbeautiful-02', 'Ġtend-02', 'Ġrob-01', 'Ġriot-01', 'Ġsponsor-01',\n                      'Ġentitle-01', 'Ġlobby-01', 'Ġbad-02', 'Ġcollapse-01', 'Ġexpose-01', 'Ġemphasize-01',\n                      'Ġfriendly-01', 'Ġplay-02', 'Ġinitiate-01', 'Ġappreciate-02', 'Ġremind-01', 'Ġblack-04',\n                      'Ġefficient-01', 'Ġconverse-01', 'Ġresponsible-02', 'Ġmeasure-01', 'Ġcome-04', 'Ġeffect-03',\n                      'Ġsubject-01', 'Ġmistake-02', 'Ġpass-03', 'Ġsignal-07', 'Ġguard-01', 'Ġopen-04', 'Ġset-02',\n                      'Ġfun-01', 'Ġcome-up-11', 'Ġflee-05', 'Ġlabel-01', 'Ġsize-01', 'Ġconfident-01', 'Ġsmart-06',\n                      'Ġhost-01', 'Ġtough-02', 'Ġrecall-02', 'Ġscare-01', 'Ġdream-01', 'Ġassault-01', 'Ġfreeze-02',\n                      'Ġtake-over-12', 'Ġrecession-02', 'Ġfunction-01', 'Ġwhine-01', 'Ġshort-06', 'Ġprosper-01',\n                      'Ġadvanced-02', 'Ġvalue-02', 'Ġbother-01', 'Ġcomply-01', 'Ġright-04', 'Ġrevolution-03',\n                      'Ġaccomplish-01', 'Ġgo-out-17', 'Ġfigure-out-05', 'Ġslow-05', 'Ġaccountable-02', 'Ġcool-01',\n                      'Ġdocument-01', 'Ġauthorize-01', 'Ġembargo-01', 'Ġvolunteer-01', 'Ġregister-02', 'Ġfrequent-02',\n                      'Ġrank-01', 'Ġresist-01', 'Ġbreak-up-08', 'Ġred-02', 'Ġcomfortable-02', 'Ġexamine-01',\n                      'Ġadjust-01', 'Ġoriginate-01', 'Ġreply-01', 'Ġbreak-18', 'Ġshoot-01', 'Ġmiss-02', 'Ġdismiss-01',\n                      'Ġcollect-01', 'Ġdraft-01', 'Ġsubmit-01', 'Ġrelieve-01', 'Ġembarrass-01', 'Ġreturn-02',\n                      'Ġvoluntary-02', 'Ġpure-02', 'Ġbeat-01', 'Ġbear-01', 'Ġvary-01', 'Ġsick-05', 'Ġaffair-01',\n                      'Ġtypical-02', 'Ġnegative-02', 'Ġserve-02', 'Ġeradicate-01', 'Ġrealize-02', 'Ġperceive-01',\n                      'Ġleave-14', 'Ġgive-16', 'Ġback-up-04', 'Ġgenerate-01', 'Ġbail-out-02', 'Ġtouch-01',\n                      'Ġcultivate-01', 'Ġconvert-01', 'Ġdismantle-01', 'Ġservice-05', 'Ġstraight-04', 'Ġbad-05',\n                      'Ġforce-04', 'Ġadvocate-01', 'Ġpray-01', 'Ġdecline-01', 'Ġinfect-01', 'Ġtitle-01',\n                      'Ġdesperate-02', 'Ġupset-01', 'Ġtolerate-01', 'Ġprohibit-01', 'Ġmind-05', 'Ġbeat-03', 'Ġveto-01',\n                      'Ġcrash-01', 'Ġside-01', 'Ġcombine-01', 'Ġclose-13', 'Ġgo-10', 'Ġequip-01', 'Ġrant-01',\n                      'Ġjail-01', 'Ġcopy-01', 'Ġdrop-05', 'Ġconsistent-02', 'Ġspend-04', 'Ġsend-03', 'Ġcritical-02',\n                      'Ġcarry-on-02', 'Ġraise-02', 'Ġmotivate-01', 'Ġguide-01', 'Ġwonderful-03', 'Ġtrust-01',\n                      'Ġreverse-01', 'Ġjust-02', 'Ġclaim-02', 'Ġsurvey-01', 'Ġspy-01', 'Ġget-22', 'Ġhave-05',\n                      'Ġcool-04', 'Ġpicture-01', 'Ġunion-02', 'Ġmanage-02', 'Ġinstruct-01', 'Ġblow-03', 'Ġsacrifice-01',\n                      'Ġowe-01', 'Ġappeal-01', 'Ġexceed-01', 'Ġradiate-01', 'Ġhonor-01', 'Ġseparate-01', 'Ġarrange-01',\n                      'Ġdominate-01', 'Ġtransact-01', 'Ġgrow-up-04', 'Ġverify-01', 'Ġgo-05', 'Ġfamiliarize-01',\n                      'Ġrenew-01', 'Ġfire-02', 'Ġtake-out-11', 'Ġinterpret-01', 'Ġvalid-02', 'Ġshow-up-02',\n                      'Ġconfiscate-01', 'Ġshut-down-05', 'Ġcheat-03', 'Ġharass-01', 'Ġtie-01', 'Ġabuse-02',\n                      'Ġassess-01', 'Ġcompensate-01', 'Ġsensitive-03', 'Ġsettle-02', 'Ġencounter-01', 'Ġmatch-01',\n                      'Ġrecover-02', 'Ġtrust-02', 'Ġperform-01', 'Ġborrow-01', 'Ġselect-01', 'Ġbetray-01', 'Ġride-01',\n                      'Ġuseful-05', 'Ġsplit-01', 'Ġshift-01', 'Ġannoy-01', 'Ġmind-01', 'Ġfair-04', 'Ġoppress-01',\n                      'Ġinterfere-01', 'Ġcredit-02', 'Ġlaunder-01', 'Ġamount-01', 'Ġleave-13', 'Ġrescue-01',\n                      'Ġstaff-01', 'Ġplay-11', 'Ġkind-01', 'Ġauthor-01', 'Ġsympathize-01', 'Ġupgrade-02',\n                      'Ġsuppress-01', 'Ġwake-up-02', 'Ġinvite-01', 'Ġcome-12', 'Ġdeter-01', 'Ġbrainwash-01', 'Ġshit-01',\n                      'Ġfix-02', 'Ġwhite-03', 'Ġgroup-01', 'Ġabsent-01', 'Ġarmor-01', 'Ġup-03', 'Ġpraise-01',\n                      'Ġreview-01', 'Ġdry-02', 'Ġintercept-01', 'Ġbroadcast-01', 'Ġworship-01', 'Ġterm-01',\n                      'Ġobject-01', 'Ġpledge-01', 'Ġprepare-01', 'Ġopen-up-03', 'Ġlay-01', 'Ġfile-01', 'Ġcheck-out-05',\n                      'Ġattach-01', 'Ġsatisfy-01', 'Ġdepart-01', 'Ġopposite-01', 'Ġworsen-01', 'Ġaward-01',\n                      'Ġpollute-01', 'Ġretaliate-01', 'Ġdisrupt-01', 'Ġreturn-05', 'Ġpopulate-01', 'Ġenvision-01',\n                      'Ġplease-01', 'Ġrepair-01', 'Ġslaughter-01', 'Ġsin-01', 'Ġconstitute-01', 'Ġshop-01',\n                      'Ġtranslate-01', 'Ġassure-01', 'Ġpay-off-02', 'Ġstimulate-01', 'Ġdamn-01', 'Ġswitch-01',\n                      'Ġdisappear-01', 'Ġreelect-01', 'Ġspin-03', 'Ġtestify-01', 'Ġlegalize-01', 'Ġprint-01',\n                      'Ġaverage-01', 'Ġright-03', 'Ġfix-03', 'Ġundermine-01', 'Ġcome-on-25', 'Ġlicense-01',\n                      'Ġindict-01', 'Ġtransit-01', 'Ġwash-01', 'Ġbreathe-01', 'Ġbroad-02', 'Ġleave-17', 'Ġorder-02',\n                      'Ġhead-02', 'Ġsing-01', 'Ġentertain-01', 'Ġcomplicate-01', 'Ġpush-02', 'Ġrealistic-03',\n                      'Ġdisappoint-01', 'Ġbother-02', 'Ġtough-03', 'Ġdisplay-01', 'Ġflow-01', 'Ġdiffer-01', 'Ġlie-07',\n                      'Ġpremise-01', 'Ġrelocate-01', 'Ġcorrect-01', 'Ġcoordinate-01', 'Ġabandon-01', 'Ġdictate-01',\n                      'Ġplay-08', 'Ġrebuild-01', 'Ġclean-04', 'Ġwork-out-02', 'Ġrun-13', 'Ġcurious-01', 'Ġpromote-01',\n                      'Ġspecialize-01', 'Ġstarve-01', 'Ġshame-02', 'Ġfit-06', 'Ġflaw-01', 'Ġfigure-01', 'Ġhunt-01',\n                      'Ġexperiment-01', 'Ġmix-01', 'Ġregular-03', 'Ġfree-01', 'Ġdeclare-01', 'Ġescape-01', 'Ġput-02',\n                      'Ġobsess-01', 'Ġbuild-up-05', 'Ġshut-up-06', 'Ġrally-01', 'Ġdissent-01', 'Ġprogram-01',\n                      'Ġamend-01', 'Ġinvent-01', 'Ġleak-01', 'Ġtrigger-01', 'Ġdistinguish-01', 'Ġsymbolize-01',\n                      'Ġexcellent-02', 'Ġlook-04', 'Ġcry-02', 'Ġassign-01', 'Ġrecruit-01', 'Ġcope-01', 'Ġmigrate-01',\n                      'Ġtake-on-09', 'Ġbless-01', 'Ġsharp-02', 'Ġuse-02', 'Ġdisturb-01', 'Ġconsult-01', 'Ġlay-off-02',\n                      'Ġbid-01', 'Ġaccord-02', 'Ġbusy-01', 'Ġprovoke-01', 'Ġisolate-01', 'Ġdirty-02', 'Ġblind-02',\n                      'Ġstage-01', 'Ġboost-01', 'Ġoutrage-01', 'Ġtrack-01', 'Ġretard-01', 'Ġexclude-01', 'Ġpatent-01',\n                      'Ġblog-01', 'Ġtorture-01', 'Ġplot-01', 'Ġcut-01', 'Ġhunger-01', 'Ġoverwhelm-01', 'Ġexploit-01',\n                      'Ġland-01', 'Ġreserve-01', 'Ġbetter-01', 'Ġup-02', 'Ġremark-01', 'Ġpiss-03', 'Ġexcuse-01',\n                      'Ġparalyze-01', 'Ġsummarize-01', 'Ġload-01', 'Ġdevote-01', 'Ġbury-01', 'Ġsurround-01',\n                      'Ġdance-01', 'Ġdistort-01', 'Ġretain-01', 'Ġoverthrow-01', 'Ġrival-01', 'Ġready-01', 'Ġevolve-01',\n                      'Ġimpoverish-01', 'Ġalarm-01', 'Ġunify-01', 'Ġrepay-01', 'Ġassume-01', 'Ġclose-06', 'Ġadmire-01',\n                      'Ġvow-01', 'Ġaverage-04', 'Ġsight-01', 'Ġinflate-01', 'Ġreference-04', 'Ġlook-up-05',\n                      'Ġcivilize-01', 'Ġsuitable-04', 'Ġdetect-01', 'Ġpiss-off-02', 'Ġassassinate-01', 'Ġopen-05',\n                      'Ġshave-01', 'Ġemail-01', 'Ġfuel-01', 'Ġincentivize-01', 'Ġmark-01', 'Ġsustain-01',\n                      'Ġspeculate-01', 'Ġsurveil-01', 'Ġswim-01', 'Ġconquer-01', 'Ġgenocide-01', 'Ġhoax-01',\n                      'Ġnotice-03', 'Ġbe-done-08', 'Ġopt-01', 'Ġbait-01', 'Ġcompile-01', 'Ġinnovate-01', 'Ġallocate-01',\n                      'Ġshelter-01', 'Ġcontrary-01', 'Ġburden-01', 'Ġfreeze-01', 'Ġinspire-01', 'Ġgraduate-01',\n                      'Ġwipe-out-02', 'Ġfall-05', 'Ġcover-up-04', 'Ġrepute-01', 'Ġenhance-01', 'Ġclassify-01',\n                      'Ġgreen-03', 'Ġscore-01', 'Ġmodify-01', 'Ġreflect-01', 'Ġforce-02', 'Ġequate-01',\n                      'Ġmerchandise-01', 'Ġregret-01', 'Ġovercome-01', 'Ġprocure-01', 'Ġscam-01', 'Ġquit-01',\n                      'Ġdrill-01', 'Ġdisable-01', 'Ġgrasp-01', 'Ġorbit-01', 'Ġlaughable-03', 'Ġconsent-01',\n                      'Ġendorse-01', 'Ġcatch-02', 'Ġleave-02', 'Ġweigh-01', 'Ġroll-01', 'Ġrestore-01', 'Ġshape-01',\n                      'Ġcomprehend-01', 'Ġtrip-03', 'Ġget-away-08', 'Ġsingle-03', 'Ġphone-01', 'Ġintimidate-01',\n                      'Ġinstall-01', 'Ġsuck-03', 'Ġback-02', 'Ġdeem-01', 'Ġmake-up-10', 'Ġplant-01', 'Ġhand-out-03',\n                      'Ġgo-off-16', 'Ġspeed-01', 'Ġrefute-01', 'Ġimplicate-01', 'Ġdock-01', 'Ġcrack-down-06',\n                      'Ġforecast-01', 'Ġrush-01', 'Ġgenerous-01', 'Ġunite-01', 'Ġgrab-01', 'Ġcompetent-01',\n                      'Ġground-02', 'Ġevaluate-01', 'Ġadvance-01', 'Ġmainstream-02', 'Ġdiagnose-01', 'Ġpass-05',\n                      'Ġuphold-01', 'Ġhalt-01', 'Ġhinder-01', 'Ġbefriend-01', 'Ġconvene-01', 'Ġawe-01', 'Ġapplaud-01',\n                      'Ġmodernize-01', 'Ġintegrate-01', 'Ġexecute-02', 'Ġwound-01', 'Ġprostitute-01', 'Ġexercise-01',\n                      'Ġbind-01', 'Ġphotograph-01', 'Ġfascinate-01', 'Ġreward-01', 'Ġclean-up-02', 'Ġrepeal-01',\n                      'Ġtwist-01', 'Ġmodel-01', 'Ġmandate-01', 'Ġconspire-01', 'Ġtear-01', 'Ġbrutal-02', 'Ġcharge-08',\n                      'Ġdry-08', 'Ġwow-01', 'Ġbank-01', 'Ġfuck-up-02', 'Ġstand-up-07', 'Ġportray-01', 'Ġnationalize-01',\n                      'Ġliberate-01', 'Ġexempt-01', 'Ġdefy-01', 'Ġshout-01', 'Ġdevastate-01', 'Ġhijack-01',\n                      'Ġacknowledge-01', 'Ġcompromise-02', 'Ġconsist-01', 'Ġcoach-01', 'Ġintense-02', 'Ġdrag-01',\n                      'Ġminor-01', 'Ġfulfill-01', 'Ġclear-01', 'Ġdeceive-01', 'Ġshake-01', 'Ġcold-01', 'Ġalign-01',\n                      'Ġsupervise-01', 'Ġinternal-02', 'Ġgift-01', 'Ġstruggle-01', 'Ġcast-01', 'Ġfeature-01',\n                      'Ġharsh-02', 'Ġemerge-01', 'Ġfollow-04', 'Ġcut-off-04', 'Ġmistake-01', 'Ġlocate-01', 'Ġslow-01',\n                      'Ġaccelerate-01', 'Ġcover-02', 'Ġsoft-02', 'Ġidentical-01', 'Ġsail-01', 'Ġjump-03',\n                      'Ġfacilitate-01', 'Ġexcessive-02', 'Ġalter-01', 'Ġescalate-01', 'Ġmad-04', 'Ġkid-01', 'Ġfloat-01',\n                      'Ġmess-up-02', 'Ġkidnap-01', 'Ġbore-02', 'Ġclean-01', 'Ġforgive-01', 'Ġgo-through-20', 'Ġcare-04',\n                      'Ġmeet-up-04', 'Ġmoisturize-01', 'Ġhighlight-01', 'Ġdislike-01', 'Ġboom-02', 'Ġblow-up-06',\n                      'Ġappeal-02', 'Ġadhere-02', 'Ġcontradict-01', 'Ġleave-12', 'Ġdialogue-01', 'Ġpush-04',\n                      'Ġcontaminate-01', 'Ġfinalize-01', 'Ġtape-02', 'Ġpatrol-01', 'Ġincite-01', 'Ġrenounce-01',\n                      'Ġhallucinate-01', 'Ġundertake-01', 'Ġaverage-03', 'Ġcompel-01', 'Ġstruggle-02', 'Ġgo-12',\n                      'Ġtrap-01', 'Ġquiet-04', 'Ġconvey-01', 'Ġopen-02', 'Ġclothe-01', 'Ġexclusive-02', 'Ġgather-03',\n                      'Ġextensive-03', 'Ġapproach-01', 'Ġmanipulate-02', 'Ġinfringe-01', 'Ġruin-01', 'Ġstrive-01',\n                      'Ġproductive-03', 'Ġexplore-01', 'Ġinhabit-01', 'Ġpress-01', 'Ġforbid-01', 'Ġhit-02',\n                      'Ġabolish-01', 'Ġimpress-01', 'Ġprospect-02', 'Ġgoogle-01', 'Ġsink-01', 'Ġresign-01',\n                      'Ġpull-out-02', 'Ġstation-01', 'Ġcenter-02', 'Ġindustrialize-01', 'Ġcounsel-01', 'Ġpropel-01',\n                      'Ġsmell-01', 'Ġmoderate-03', 'Ġpresume-01', 'Ġrun-09', 'Ġkeep-up-10', 'Ġdeal-03', 'Ġapprehend-01',\n                      'Ġsick-02', 'Ġsmell-02', 'Ġhave-11', 'Ġfrustrate-01', 'Ġcatch-01', 'Ġimpression-03',\n                      'Ġspecify-01', 'Ġemploy-02', 'Ġthankful-02', 'Ġman-01', 'Ġprioritize-01', 'Ġattribute-01',\n                      'Ġproject-01', 'Ġparrot-01', 'Ġbitch-01', 'Ġstand-04', 'Ġvoice-01', 'Ġpreserve-01',\n                      'Ġpublicize-01', 'Ġexhibit-01', 'Ġundergo-28', 'Ġhelp-02', 'Ġbankrupt-01', 'Ġflood-01',\n                      'Ġprecede-01', 'Ġreinforce-01', 'Ġtask-01', 'Ġtype-03', 'Ġtransform-01', 'Ġdespair-01',\n                      'Ġchase-01', 'Ġspread-01', 'Ġappall-01', 'Ġrestrain-01', 'Ġterrify-01', 'Ġfool-01', 'Ġaspire-01',\n                      'Ġwarm-07', 'Ġbring-up-08', 'Ġbleed-01', 'Ġdepress-01', 'Ġcare-02', 'Ġalert-01', 'Ġwonder-02',\n                      'Ġdrop-out-04', 'Ġspoil-01', 'Ġstink-01', 'Ġdrug-01', 'Ġoverturn-01', 'Ġheat-01', 'Ġmerge-01',\n                      'Ġpeak-01', 'Ġset-01', 'Ġsolid-02', 'Ġinteract-01', 'Ġthrow-out-06', 'Ġholiday-01', 'Ġrefine-01',\n                      'Ġallow-02', 'Ġsign-up-03', 'Ġbribe-01', 'Ġappease-01', 'Ġstress-02', 'Ġfine-01', 'Ġminor-02',\n                      'Ġmine-01', 'Ġlove-02', 'Ġnetwork-01', 'Ġdeposit-01', 'Ġstore-01', 'Ġextract-01',\n                      'Ġinterrogate-01', 'Ġturn-out-11', 'Ġimpregnate-01', 'Ġfake-02', 'Ġwhore-01', 'Ġconceal-01',\n                      'Ġfire-03', 'Ġlean-01', 'Ġharmful-02', 'Ġout-05', 'Ġfall-07', 'Ġdodge-01', 'Ġorient-01',\n                      'Ġbrand-01', 'Ġsocial-03', 'Ġcut-03', 'Ġcap-01', 'Ġoverpay-01', 'Ġbridge-01', 'Ġcollaborate-01',\n                      'Ġaddress-03', 'Ġdivert-01', 'Ġpull-09', 'Ġrevise-01', 'Ġmolest-01', 'Ġextradite-01',\n                      'Ġdismiss-02', 'Ġreprocess-01', 'Ġaccumulate-01', 'Ġoccasion-02', 'Ġobstruct-01',\n                      'Ġbreak-down-12', 'Ġrumor-01', 'Ġfirm-03', 'Ġsettle-03', 'Ġorder-03', 'Ġstipulate-01',\n                      'Ġaudit-01', 'Ġenact-01', 'Ġcelebrate-02', 'Ġbargain-01', 'Ġsucceed-03', 'Ġinject-01',\n                      'Ġexcite-01', 'Ġgreet-01', 'Ġblack-07', 'Ġterminate-01', 'Ġdescend-01', 'Ġemerge-02', 'Ġwreck-01',\n                      'Ġabsorb-01', 'Ġblow-01', 'Ġfine-03', 'Ġcirculate-01', 'Ġtight-05', 'Ġoffense-02', 'Ġactivate-01',\n                      'Ġsecure-01', 'Ġpass-by-17', 'Ġbash-01', 'Ġprop-up-01', 'Ġcount-04', 'Ġslap-01', 'Ġbring-down-03',\n                      'Ġamuse-01', 'Ġfilm-01', 'Ġintroduce-01', 'Ġdesignate-01', 'Ġhang-01', 'Ġwave-04',\n                      'Ġprivilege-01', 'Ġtake-02', 'Ġcycle-02', 'Ġcancel-01', 'Ġbuy-05', 'Ġsweep-01', 'Ġhelp-out-03',\n                      'Ġleft-20', 'Ġsuit-01', 'Ġenslave-01', 'Ġrest-01', 'Ġambush-01', 'Ġmean-04', 'Ġdistract-01',\n                      'Ġmatch-03', 'Ġwarrant-01', 'Ġdisguise-01', 'Ġmake-up-07', 'Ġparty-01', 'Ġclose-11', 'Ġfall-10',\n                      'Ġpump-01', 'Ġresort-01', 'Ġget-back-10', 'Ġregain-01', 'Ġlose-01', 'Ġerr-01', 'Ġrun-out-05',\n                      'Ġthat-is-it-00', 'Ġaggravate-01', 'Ġloot-01', 'Ġhappen-02', 'Ġscrew-02', 'Ġmake-it-14',\n                      'Ġpick-up-04', 'Ġrefer-02', 'Ġbreak-13', 'Ġupdate-01', 'Ġshine-01', 'Ġcongratulate-01',\n                      'Ġpilot-01', 'Ġdisgrace-01', 'Ġfabricate-01', 'Ġsicken-01', 'Ġcriticism-04', 'Ġpreach-01',\n                      'Ġdeport-01', 'Ġdeal-02', 'Ġinflict-01', 'Ġgain-01', 'Ġresume-01', 'Ġoutlaw-01', 'Ġshoot-down-05',\n                      'Ġpartition-01', 'Ġaddress-01', 'Ġenvy-01', 'Ġbreak-02', 'Ġspeak-out-03', 'Ġbroaden-01',\n                      'Ġstress-01', 'Ġinfiltrate-01', 'Ġflat-06', 'Ġimpeach-01', 'Ġtransgress-01', 'Ġpardon-01',\n                      'Ġuncover-01', 'Ġcomprise-01', 'Ġreconstruct-01', 'Ġlibel-01', 'Ġhand-01', 'Ġhint-01',\n                      'Ġencourage-02', 'Ġprevail-02', 'Ġbrave-02', 'Ġforesee-01', 'Ġconcede-01', 'Ġdeteriorate-01',\n                      'Ġtopple-01', 'Ġmobile-02', 'Ġpanic-01', 'Ġmisunderstand-01', 'Ġtire-01', 'Ġenthusiastic-03',\n                      'Ġexercise-02', 'Ġpersist-01', 'Ġinferior-01', 'Ġbrilliant-01', 'Ġbuild-02', 'Ġscream-01',\n                      'Ġanticipate-01', 'Ġout-03', 'Ġration-01', 'Ġcount-02', 'Ġconsistent-01', 'Ġawait-01',\n                      'Ġschool-01', 'Ġrent-01', 'Ġarise-02', 'Ġappeal-03', 'Ġhelpful-04', 'Ġsee-03', 'Ġlock-01',\n                      'Ġstereotype-01', 'Ġjoin-in-05', 'Ġscrew-up-01', 'Ġwithhold-01', 'Ġmoderate-01', 'Ġaffiliate-01',\n                      'Ġwaive-01', 'Ġsuck-01', 'Ġgolf-01', 'Ġturn-out-17', 'Ġput-up-11', 'Ġkeep-up-05', 'Ġstraight-05',\n                      'Ġdress-01', 'Ġdig-01', 'Ġplead-02', 'Ġlecture-01', 'Ġgo-09', 'Ġpervert-01', 'Ġcry-01',\n                      'Ġmitigate-01', 'Ġsubstitute-01', 'Ġsend-02', 'Ġdown-01', 'Ġwesternize-01', 'Ġcolor-01',\n                      'Ġrefer-03', 'Ġpersecute-01', 'Ġscheme-01', 'Ġreactionary-02', 'Ġsubscribe-01', 'Ġshield-01',\n                      'Ġexile-01', 'Ġdetonate-01', 'Ġstall-01', 'Ġbroker-01', 'Ġcalculate-01', 'Ġnarrow-02',\n                      'Ġstock-01', 'Ġturn-down-05', 'Ġparole-01', 'Ġjoin-04', 'Ġinstitute-01', 'Ġdisprove-01',\n                      'Ġpass-20', 'Ġspew-01', 'Ġbid-03', 'Ġwage-01', 'Ġsample-01', 'Ġretail-01', 'Ġratify-01',\n                      'Ġspank-01', 'Ġdispatch-01', 'Ġharvest-01', 'Ġrot-01', 'Ġdelude-01', 'Ġclimb-01', 'Ġfrighten-01',\n                      'Ġyell-01', 'Ġcoerce-01', 'Ġscary-03', 'Ġstretch-01', 'Ġdestabilize-01', 'Ġblood-02',\n                      'Ġconfine-01', 'Ġoutrageous-02', 'Ġbeg-01', 'Ġwield-01', 'Ġscrap-01', 'Ġprivatize-01', 'Ġcure-01',\n                      'Ġmature-02', 'Ġcoexist-01', 'Ġassert-02', 'Ġget-along-18', 'Ġreunify-01', 'Ġlook-forward-03',\n                      'Ġnumber-01', 'Ġtrash-01', 'Ġrun-04', 'Ġgive-up-08', 'Ġbright-02', 'Ġout-01', 'Ġheal-01',\n                      'Ġmassacre-01', 'Ġtackle-01', 'Ġstake-01', 'Ġopen-09', 'Ġknow-04', 'Ġcorrespond-02',\n                      'Ġdisregard-01', 'Ġalienate-01', 'Ġinsure-01', 'Ġdisapprove-01', 'Ġdrain-01', 'Ġdeflect-01',\n                      'Ġexit-01', 'Ġvacation-01', 'Ġcook-01', 'Ġadapt-01', 'Ġdissolve-01', 'Ġlift-01', 'Ġclose-down-04',\n                      'Ġcome-down-23', 'Ġbully-01', 'Ġdenounce-01', 'Ġstab-01', 'Ġexpel-01', 'Ġabstain-01',\n                      'Ġcut-out-06', 'Ġswallow-01', 'Ġcome-in-07', 'Ġstep-in-02', 'Ġseek-out-02', 'Ġpace-01', 'Ġwed-01',\n                      'Ġgo-on-25', 'Ġsave-03', 'Ġcome-up-13', 'Ġsort-out-02', 'Ġtattoo-01', 'Ġleave-out-03', 'Ġkiss-01',\n                      'Ġchance-01', 'Ġprolong-01', 'Ġtroll-01', 'Ġconcentrate-01', 'Ġchannel-01', 'Ġrecreation-02',\n                      'Ġcenter-01', 'Ġweaponize-01', 'Ġexplicit-03', 'Ġdraft-02', 'Ġpose-02', 'Ġcrush-01',\n                      'Ġdiscredit-01', 'Ġfurther-01', 'Ġdedicate-01', 'Ġsit-down-02', 'Ġleave-10', 'Ġforge-02',\n                      'Ġcensor-01', 'Ġparade-02', 'Ġpaint-02', 'Ġcatch-03', 'Ġremortgage-01', 'Ġslow-down-03',\n                      'Ġadmit-02', 'Ġbreak-19', 'Ġcounterfeit-01', 'Ġrun-10', 'Ġupgrade-01', 'Ġdeduct-01',\n                      'Ġconfess-01', 'Ġdecline-02', 'Ġbar-01', 'Ġbrief-01', 'Ġconduct-02', 'Ġlynch-01', 'Ġacquit-01',\n                      'Ġhyperlink-01', 'Ġlight-04', 'Ġconcrete-02', 'Ġreach-02', 'Ġmarch-01', 'Ġpurport-01',\n                      'Ġcall-on-05', 'Ġpaddle-01', 'Ġfilter-02', 'Ġstrip-01', 'Ġcompose-01', 'Ġerupt-01', 'Ġwipe-01',\n                      'Ġtrace-02', 'Ġdespise-01', 'Ġminimize-01', 'Ġneglect-01', 'Ġloyal-01', 'Ġslip-01', 'Ġrevive-01',\n                      'Ġwork-07', 'Ġbeat-up-05', 'Ġdetermined-02', 'Ġpass-07', 'Ġprescribe-02', 'Ġfuss-01',\n                      'Ġdemolish-01', 'Ġavail-01', 'Ġput-in-05', 'Ġlease-01', 'Ġembrace-01', 'Ġmerit-01',\n                      'Ġintensify-01', 'Ġhearing-02', 'Ġweaken-01', 'Ġcolonize-01', 'Ġoffset-01', 'Ġgather-01',\n                      'Ġtake-off-07', 'Ġbright-03', 'Ġextend-02', 'Ġget-30', 'Ġpreexist-01', 'Ġsnow-01', 'Ġstrike-02',\n                      'Ġgross-06', 'Ġdiminish-01', 'Ġprejudice-01', 'Ġrage-02', 'Ġnotify-01', 'Ġcontest-02', 'Ġhype-01',\n                      'Ġrevisit-01', 'Ġdark-02', 'Ġstand-08', 'Ġcertify-01', 'Ġoversee-01', 'Ġname-02', 'Ġlock-up-03',\n                      'Ġknow-03', 'Ġminimal-02', 'Ġtell-02', 'Ġrotate-01', 'Ġoperate-02', 'Ġfat-03', 'Ġindulge-01',\n                      'Ġfeel-06', 'Ġset-08', 'Ġsurpass-01', 'Ġpull-06', 'Ġget-06', 'Ġcamp-02', 'Ġgut-01', 'Ġchair-01',\n                      'Ġqualify-01', 'Ġspare-01', 'Ġblunt-02', 'Ġproceed-01', 'Ġdump-01', 'Ġreckon-01', 'Ġpierce-01',\n                      'Ġmelt-01', 'Ġfeel-05', 'Ġstand-03', 'Ġelaborate-01', 'Ġreach-03', 'Ġspark-01', 'Ġcoincide-01',\n                      'Ġslander-01', 'Ġjoin-up-02', 'Ġshame-01', 'Ġboard-01', 'Ġrule-out-02', 'Ġblockade-01',\n                      'Ġincinerate-01', 'Ġderive-01', 'Ġget-by-17', 'Ġcharacterize-01', 'Ġstockpile-01', 'Ġpersuade-01',\n                      'Ġdecapitate-01', 'Ġrun-08', 'Ġpack-01', 'Ġbust-01', 'Ġpolice-01', 'Ġtrick-01', 'Ġblast-05',\n                      'Ġtreat-04', 'Ġrun-off-24', 'Ġapprentice-01', 'Ġdispose-01', 'Ġinhibit-01', 'Ġwire-01', 'Ġtop-01',\n                      'Ġhand-over-02', 'Ġknow-06', 'Ġabet-01', 'Ġcatch-up-04', 'Ġsleep-02', 'Ġslam-02', 'Ġbreed-01',\n                      'Ġcontend-02', 'Ġperjure-01', 'Ġmanipulate-01', 'Ġprobe-01', 'Ġtrend-01', 'Ġtighten-01',\n                      'Ġboycott-01', 'Ġtable-01', 'Ġindoctrinate-01', 'Ġsafeguard-01', 'Ġevacuate-01', 'Ġinterdict-01',\n                      'Ġpetition-01', 'Ġformulate-01', 'Ġpartake-01', 'Ġpass-04', 'Ġoverride-01', 'Ġemit-01',\n                      'Ġcharacteristic-02', 'Ġtimely-03', 'Ġstun-01', 'Ġcrumble-01', 'Ġmaximize-01', 'Ġpass-away-16',\n                      'Ġrun-07', 'Ġsmile-01', 'Ġinquire-01', 'Ġlag-01', 'Ġlive-up-04', 'Ġdistance-01', 'Ġcold-02',\n                      'Ġdeep-03', 'Ġrelax-01', 'Ġill-02', 'Ġsignify-01', 'Ġhold-back-07', 'Ġtransplant-01', 'Ġsmoke-01',\n                      'Ġcurb-01', 'Ġdelegate-01', 'Ġseal-01', 'Ġlure-01', 'Ġintimate-02', 'Ġfresh-04', 'Ġseat-01',\n                      'Ġmove-03', 'Ġkeep-03', 'Ġoutweigh-01', 'Ġrevere-01', 'Ġclone-01', 'Ġenlist-01', 'Ġclick-01',\n                      'Ġempty-02', 'Ġfire-04', 'Ġcontend-01', 'Ġabide-01', 'Ġcraft-01', 'Ġtip-05', 'Ġwrap-01',\n                      'Ġbite-01', 'Ġtoss-01', 'Ġpolite-01', 'Ġdesirable-02', 'Ġdefuse-01', 'Ġthrill-01', 'Ġproduce-02',\n                      'Ġoblige-02', 'Ġdate-02', 'Ġalternate-01', 'Ġget-on-21', 'Ġramble-02', 'Ġhurt-02', 'Ġdistant-02',\n                      'Ġhot-05', 'Ġpale-03', 'Ġproclaim-01', 'Ġclass-01', 'Ġcome-across-21', 'Ġsneak-01', 'Ġerode-01',\n                      'Ġchampion-01', 'Ġneutral-02', 'Ġalien-01', 'Ġgrieve-01', 'Ġswear-01', 'Ġgo-21',\n                      'Ġunderestimate-01', 'Ġaddictive-02', 'Ġpropagate-01', 'Ġlast-04', 'Ġcommence-01', 'Ġair-01',\n                      'Ġmark-02', 'Ġaccommodate-01', 'Ġdemonize-01', 'Ġmock-01', 'Ġnuke-01', 'Ġswell-01', 'Ġbrag-01',\n                      'Ġassert-03', 'Ġdisrespect-01', 'Ġwork-12', 'Ġremarkable-02', 'Ġpool-01', 'Ġpaint-03', 'Ġpour-01',\n                      'Ġdecommission-01', 'Ġamplify-01', 'Ġmad-02', 'Ġcorrelate-01', 'Ġautomate-01', 'Ġmoney-01',\n                      'Ġcontent-02', 'Ġstorm-01', 'Ġthrive-01', 'Ġliable-01', 'Ġhopeful-02', 'Ġexpire-01', 'Ġwork-06',\n                      'Ġdisperse-01', 'Ġlay-04', 'Ġfall-apart-09', 'Ġterror-02', 'Ġphilander-01', 'Ġscrutinize-01',\n                      'Ġfathom-01', 'Ġmake-up-08', 'Ġhumiliate-01', 'Ġcharge-06', 'Ġnatural-02', 'Ġfollow-up-03',\n                      'Ġbend-01', 'Ġgrade-01', 'Ġenter-02', 'Ġpend-01', 'Ġprey-01', 'Ġmediate-01', 'Ġconclude-02',\n                      'Ġmask-01', 'Ġreactivate-01', 'Ġevolve-02', 'Ġrestart-01', 'Ġencrypt-01', 'Ġget-through-12',\n                      'Ġgrow-02', 'Ġbestow-01', 'Ġput-out-10', 'Ġdisplace-01', 'Ġcount-03', 'Ġstabilize-01',\n                      'Ġembezzle-01', 'Ġpass-on-09', 'Ġform-02', 'Ġroot-02', 'Ġtrample-01', 'Ġmake-out-23',\n                      'Ġfit-in-02', 'Ġhospitalize-01', 'Ġcut-down-11', 'Ġconstrain-01', 'Ġclash-01', 'Ġconsolidate-01',\n                      'Ġmeddle-01', 'Ġreproduce-01', 'Ġclever-01', 'Ġdiversify-01', 'Ġpostpone-01', 'Ġstructure-01',\n                      'Ġnarrow-01', 'Ġincur-01', 'Ġdraw-up-03', 'Ġdrive-04', 'Ġpin-01', 'Ġdelight-01', 'Ġput-on-08',\n                      'Ġcoverage-06', 'Ġbring-about-05', 'Ġstir-up-04', 'Ġlet-down-04', 'Ġsigh-02', 'Ġspace-01',\n                      'Ġcheat-02', 'Ġlessen-01', 'Ġrender-02', 'Ġrender-01', 'Ġmenace-01', 'Ġprevail-01', 'Ġreclaim-01',\n                      'Ġpuzzle-01', 'Ġhesitate-01', 'Ġgo-23', 'Ġcharm-01', 'Ġturn-over-12', 'Ġwander-01',\n                      'Ġrenovate-01', 'Ġpackage-01', 'Ġheadquarter-01', 'Ġline-01', 'Ġstraight-06', 'Ġpark-01',\n                      'Ġturn-on-13', 'Ġarbitrary-02', 'Ġconceive-01', 'Ġexert-01', 'Ġspell-01', 'Ġdye-01', 'Ġtune-01',\n                      'Ġrip-01', 'Ġgarner-01', 'Ġsick-04', 'Ġshove-01', 'Ġwave-01', 'Ġrust-01', 'Ġkneel-01',\n                      'Ġcelebrate-01', 'Ġmisrepresent-01', 'Ġincarcerate-01', 'Ġawake-03', 'Ġup-01', 'Ġslip-02',\n                      'Ġconcentrate-02', 'Ġround-05', 'Ġloose-04', 'Ġcripple-01', 'Ġpart-01', 'Ġhoard-01', 'Ġchain-01',\n                      'Ġtricky-02', 'Ġhook-up-02', 'Ġtype-01', 'Ġglance-01', 'Ġprize-01', 'Ġtransmit-01', 'Ġhold-03',\n                      'Ġsurge-01', 'Ġheadline-01', 'Ġvote-02', 'Ġdraw-01', 'Ġtext-01', 'Ġshower-01', 'Ġcalm-down-02',\n                      'Ġfeed-up-03', 'Ġslide-01', 'Ġgo-down-27', 'Ġforward-01', 'Ġproject-02', 'Ġempower-01',\n                      'Ġmind-04', 'Ġpass-02', 'Ġneutralize-01', 'Ġrepress-01', 'Ġserve-04', 'Ġeye-01',\n                      'Ġdiscriminate-01', 'Ġoverlook-01', 'Ġtop-02', 'Ġmobilize-01', 'Ġstart-out-05', 'Ġpunishable-02',\n                      'Ġunderlie-01', 'Ġpenetrate-01', 'Ġgrind-01', 'Ġjump-01', 'Ġpertain-01', 'Ġincline-01',\n                      'Ġhumble-01', 'Ġmoderate-02', 'Ġmeaningful-05', 'Ġmislead-01', 'Ġfinish-07', 'Ġdisgruntle-01',\n                      'Ġturn-up-15', 'Ġknock-01', 'Ġtake-03', 'Ġlunch-01', 'Ġadd-03', 'Ġcommend-01', 'Ġpatient-01',\n                      'Ġattain-01', 'Ġhike-02', 'Ġlurk-01', 'Ġbe-02', 'Ġblackmail-01', 'Ġdubious-02', 'Ġentrench-01',\n                      'Ġget-off-23', 'Ġflame-01', 'Ġstand-02', 'Ġsurvive-02', 'Ġafford-02', 'Ġlive-02', 'Ġmoan-01',\n                      'Ġportion-01', 'Ġslash-02', 'Ġbreak-through-22', 'Ġplague-01', 'Ġblunt-01', 'Ġabominable-02',\n                      'Ġhonorable-03', 'Ġrelated-04', 'Ġdeprive-01', 'Ġdecay-01', 'Ġdistress-01', 'Ġredistribute-01',\n                      'Ġforeclose-01', 'Ġwarm-06', 'Ġjealous-02', 'Ġcohere-01', 'Ġpaste-01', 'Ġprompt-01',\n                      'Ġcurtail-01', 'Ġtrack-down-02', 'Ġpity-01', 'Ġticket-02', 'Ġtransition-01', 'Ġburst-02',\n                      'Ġbroke-23', 'Ġrewrite-01', 'Ġdeliberate-01', 'Ġdisclose-01', 'Ġsituate-01', 'Ġreiterate-01',\n                      'Ġprofess-01', 'Ġbabble-01', 'Ġlift-02', 'Ġdeclassify-01', 'Ġremand-01', 'Ġreconcile-01',\n                      'Ġassemble-01', 'Ġextort-01', 'Ġcorroborate-01', 'Ġsnip-01', 'Ġnormalize-01', 'Ġclose-03',\n                      'Ġremit-01', 'Ġsweep-06', 'Ġbreach-01', 'Ġbehead-01', 'Ġsimulate-01', 'Ġastonish-01',\n                      'Ġdeviate-01', 'Ġsmear-02', 'Ġgive-away-02', 'Ġdifferentiate-01', 'Ġintersect-01', 'Ġrectify-01',\n                      'Ġlose-out-06', 'Ġtelephone-01', 'Ġrevolutionary-04', 'Ġblow-14', 'Ġexaggerate-01', 'Ġsoar-01',\n                      'Ġcontent-01', 'Ġpreside-01', 'Ġcheck-07', 'Ġrefrain-01', 'Ġcrack-02', 'Ġdisintegrate-01',\n                      'Ġexterminate-01', 'Ġridicule-01', 'Ġobey-01', 'Ġbundle-01', 'Ġcompound-01', 'Ġwine-01',\n                      'Ġdine-01', 'Ġresent-01', 'Ġjeopardize-01', 'Ġusher-in-01', 'Ġcrowd-01', 'Ġelevate-01',\n                      'Ġtear-down-05', 'Ġresolve-02', 'Ġearnest-01', 'Ġirritate-01', 'Ġgreen-02', 'Ġheed-01',\n                      'Ġplay-10', 'Ġspread-out-04', 'Ġcruise-01', 'Ġcater-01', 'Ġstay-on-02', 'Ġstick-around-03',\n                      'Ġcall-13', 'Ġbicker-01', 'Ġcurse-02', 'Ġopen-07', 'Ġrun-up-19', 'Ġtrump-01', 'Ġhappy-02',\n                      'Ġredeem-01', 'Ġstrike-04', 'Ġbring-on-06', 'Ġenlighten-01', 'Ġgray-02', 'Ġnote-02', 'Ġshred-01',\n                      'Ġgas-03', 'Ġlevy-01', 'Ġturn-18', 'Ġlevel-04', 'Ġbow-01', 'Ġturn-14', 'Ġrehabilitate-01',\n                      'Ġcouple-01', 'Ġdent-01', 'Ġcautious-02', 'Ġbust-02', 'Ġshut-01', 'Ġflip-01', 'Ġvalidate-01',\n                      'Ġkill-03', 'Ġhot-04', 'Ġchat-01', 'Ġcurious-02', 'Ġlump-01', 'Ġexacerbate-01', 'Ġsneaky-03',\n                      'Ġconviction-02', 'Ġproceeding-02', 'Ġreorganize-01', 'Ġfit-05', 'Ġsee-05', 'Ġacquaint-01',\n                      'Ġvile-02', 'Ġzap-01', 'Ġuniform-01', 'Ġreplicate-01', 'Ġintent-02', 'Ġgrip-01', 'Ġswear-02',\n                      'Ġdecry-01', 'Ġsegregate-01', 'Ġspur-01', 'Ġstorm-02', 'Ġcap-02', 'Ġslant-01', 'Ġspan-01',\n                      'Ġcut-back-05', 'Ġfledge-01', 'Ġfoster-01', 'Ġgripe-01', 'Ġquest-01', 'Ġpunch-01',\n                      'Ġderegulate-01', 'Ġloathe-01', 'Ġimitate-01', 'Ġhang-out-06', 'Ġbaffle-01', 'Ġsuck-up-04',\n                      'Ġtempt-01', 'Ġcondone-01', 'Ġassemble-02', 'Ġoust-01', 'Ġvent-01', 'Ġspout-01', 'Ġsound-02',\n                      'Ġevade-01', 'Ġendure-01', 'Ġinvoke-01', 'Ġdevalue-01', 'Ġpose-01', 'Ġbear-06', 'Ġhypothesize-01',\n                      'Ġspot-01', 'Ġdiscount-02', 'Ġrail-01', 'Ġhaul-01', 'Ġgauge-01', 'Ġcopyright-01', 'Ġgive-in-09',\n                      'Ġimpede-01', 'Ġblast-01', 'Ġtrue-02', 'Ġbeware-01', 'Ġrestore-02', 'Ġnegative-05', 'Ġsteady-01',\n                      'Ġfluctuate-01', 'Ġdate-01', 'Ġbathe-01', 'Ġgo-22', 'Ġrestructure-01', 'Ġpile-01', 'Ġspin-01',\n                      'Ġtake-down-22', 'Ġbake-01', 'Ġtriple-01', 'Ġdowngrade-02', 'Ġordain-01', 'Ġmultiply-01',\n                      'Ġskip-01', 'Ġincorporate-02', 'Ġsettle-01', 'Ġpass-on-14', 'Ġcreepy-04', 'Ġstuff-01',\n                      'Ġline-up-02', 'Ġimmune-02', 'Ġlust-01', 'Ġnotable-04', 'Ġbuy-into-04', 'Ġimpair-01',\n                      'Ġfigure-04', 'Ġpiss-01', 'Ġgive-back-03', 'Ġboast-01', 'Ġlay-off-06', 'Ġdive-01', 'Ġcommute-02',\n                      'Ġracket-02', 'Ġdip-01', 'Ġrotate-02', 'Ġdemagogue-01', 'Ġchange-02', 'Ġbarter-01', 'Ġalike-05',\n                      'Ġbind-03', 'Ġwhip-up-03', 'Ġmanifest-01', 'Ġcheck-03', 'Ġyield-01', 'Ġslay-01', 'Ġtally-01',\n                      'Ġget-through-13', 'Ġbreak-through-26', 'Ġrelinquish-01', 'Ġreopen-01', 'Ġdefame-01',\n                      'Ġinterrupt-01', 'Ġcast-03', 'Ġpattern-01', 'Ġdose-01', 'Ġreenter-01', 'Ġmotivate-02',\n                      'Ġstandardize-01', 'Ġdate-entity', 'Ġgovernment-organization', 'Ġtemporal-quantity',\n                      'Ġamr-unknown', 'Ġmulti-sentence', 'Ġpolitical-party', 'Ġ:compared-to', 'Ġmonetary-quantity',\n                      'Ġordinal-entity', 'Ġreligious-group', 'Ġpercentage-entity', 'Ġworld-region', 'Ġ:consist',\n                      'Ġurl-entity', 'Ġpolitical-movement', 'Ġet-cetera', 'Ġat-least', 'Ġmass-quantity',\n                      'Ġhave-org-role-91', 'Ġhave-rel-role-91', 'Ġinclude-91', 'Ġhave-concession-91',\n                      'Ġhave-condition-91', 'Ġbe-located-at-91', 'Ġrate-entity-91', 'Ġinstead-of-91', 'Ġhyperlink-91',\n                      'Ġrequest-confirmation-91', 'Ġhave-purpose-91', 'Ġbe-temporally-at-91', 'Ġregardless-91',\n                      'Ġhave-polarity-91', 'Ġbyline-91', 'Ġhave-manner-91', 'Ġhave-part-91', 'Ġhave-quant-91',\n                      'Ġpublication-91', 'Ġbe-from-91', 'Ġhave-mod-91', 'Ġhave-frequency-91', 'Ġscore-on-scale-91',\n                      'Ġhave-li-91', 'Ġbe-compared-to-91', 'Ġbe-destined-for-91', 'Ġcourse-91', 'Ġhave-subevent-91',\n                      'Ġstreet-address-91', 'Ġhave-extent-91', 'Ġstatistical-test-91', 'Ġhave-instrument-91',\n                      'Ġhave-name-91', 'Ġbe-polite-91', '-00', '-01', '-02', '-03', '-04', '-05', '-06', '-07', '-08',\n                      '-09', '-10', '-11', '-12', '-13', '-14', '-15', '-16', '-17', '-18', '-19', '-20', '-21', '-22',\n                      '-23', '-24', '-25', '-26', '-27', '-28', '-29', '-20', '-31', '-32', '-33', '-34', '-35', '-36',\n                      '-37', '-38', '-39', '-40', '-41', '-42', '-43', '-44', '-45', '-46', '-47', '-48', '-49', '-50',\n                      '-51', '-52', '-53', '-54', '-55', '-56', '-57', '-58', '-59', '-60', '-61', '-62', '-63', '-64',\n                      '-65', '-66', '-67', '-68', '-69', '-70', '-71', '-72', '-73', '-74', '-75', '-76', '-77', '-78',\n                      '-79', '-80', '-81', '-82', '-83', '-84', '-85', '-86', '-87', '-88', '-89', '-90', '-91', '-92',\n                      '-93', '-94', '-95', '-96', '-97', '-98', '-of', 'Ġ:op1', 'Ġ:op2', 'Ġ:op3', 'Ġ:op4', 'Ġ:op5',\n                      'Ġ:ARG0', 'Ġ:ARG1', 'Ġ:ARG2', 'Ġ:ARG3', 'Ġ:ARG4', 'Ġ:ARG5', 'Ġ:ARG6', 'Ġ:ARG7', 'Ġ:ARG8',\n                      'Ġ:ARG9', 'Ġ:ARG10', 'Ġ:ARG11', 'Ġ:ARG12', 'Ġ:ARG13', 'Ġ:ARG14', 'Ġ:ARG15', 'Ġ:ARG16', 'Ġ:ARG17',\n                      'Ġ:ARG18', 'Ġ:ARG19', 'Ġ:ARG20', 'Ġ:accompanier', 'Ġ:age', 'Ġ:beneficiary', 'Ġ:calendar',\n                      'Ġ:cause', 'Ġ:century', 'Ġ:concession', 'Ġ:condition', 'Ġ:conj-as-if', 'Ġ:consist-of', 'Ġ:cost',\n                      'Ġ:day', 'Ġ:dayperiod', 'Ġ:decade', 'Ġ:degree', 'Ġ:destination', 'Ġ:direction', 'Ġ:domain',\n                      'Ġ:duration', 'Ġ:employed-by', 'Ġ:era', 'Ġ:example', 'Ġ:extent', 'Ġ:frequency', 'Ġ:instrument',\n                      'Ġ:li', 'Ġ:location', 'Ġ:manner', 'Ġ:meaning', 'Ġ:medium', 'Ġ:mod', 'Ġ:mode', 'Ġ:month', 'Ġ:name',\n                      'Ġ:ord', 'Ġ:part', 'Ġ:path', 'Ġ:polarity', 'Ġ:polite', 'Ġ:poss', 'Ġ:purpose', 'Ġ:quant',\n                      'Ġ:quarter', 'Ġ:range', 'Ġ:relation', 'Ġ:role', 'Ġ:scale', 'Ġ:season', 'Ġ:source', 'Ġ:subevent',\n                      'Ġ:subset', 'Ġ:superset', 'Ġ:time', 'Ġ:timezone', 'Ġ:topic', 'Ġ:unit', 'Ġ:value', 'Ġ:weekday',\n                      'Ġ:wiki', 'Ġ:year', 'Ġ:year2', 'Ġ:snt0', 'Ġ:snt1', 'Ġ:snt2', 'Ġ:snt3', 'Ġ:snt4', 'Ġ:snt5',\n                      'ĠCOUNTRY', 'ĠQUANTITY', 'ĠORGANIZATION', 'ĠDATE_ATTRS', 'ĠNATIONALITY', 'ĠLOCATION', 'ĠENTITY',\n                      'ĠMISC', 'ĠORDINAL_ENTITY', 'ĠIDEOLOGY', 'ĠRELIGION', 'ĠSTATE_OR_PROVINCE', 'ĠCAUSE_OF_DEATH',\n                      'ĠTITLE', 'ĠDATE', 'ĠNUMBER', 'ĠHANDLE', 'ĠSCORE_ENTITY', 'ĠDURATION', 'ĠORDINAL', 'ĠMONEY',\n                      'ĠCRIMINAL_CHARGE', '_1', '_2', '_3', '_4', '_2', '_5', '_6', '_7', '_8', '_9', '_10', '_11',\n                      '_12', '_13', '_14', '_15', 'Ġ<pointer:0>', 'Ġ<pointer:1>', 'Ġ<pointer:2>', 'Ġ<pointer:3>',\n                      'Ġ<pointer:4>', 'Ġ<pointer:5>', 'Ġ<pointer:6>', 'Ġ<pointer:7>', 'Ġ<pointer:8>', 'Ġ<pointer:9>',\n                      'Ġ<pointer:10>', 'Ġ<pointer:11>', 'Ġ<pointer:12>', 'Ġ<pointer:13>', 'Ġ<pointer:14>',\n                      'Ġ<pointer:15>', 'Ġ<pointer:16>', 'Ġ<pointer:17>', 'Ġ<pointer:18>', 'Ġ<pointer:19>',\n                      'Ġ<pointer:20>', 'Ġ<pointer:21>', 'Ġ<pointer:22>', 'Ġ<pointer:23>', 'Ġ<pointer:24>',\n                      'Ġ<pointer:25>', 'Ġ<pointer:26>', 'Ġ<pointer:27>', 'Ġ<pointer:28>', 'Ġ<pointer:29>',\n                      'Ġ<pointer:30>', 'Ġ<pointer:31>', 'Ġ<pointer:32>', 'Ġ<pointer:33>', 'Ġ<pointer:34>',\n                      'Ġ<pointer:35>', 'Ġ<pointer:36>', 'Ġ<pointer:37>', 'Ġ<pointer:38>', 'Ġ<pointer:39>',\n                      'Ġ<pointer:40>', 'Ġ<pointer:41>', 'Ġ<pointer:42>', 'Ġ<pointer:43>', 'Ġ<pointer:44>',\n                      'Ġ<pointer:45>', 'Ġ<pointer:46>', 'Ġ<pointer:47>', 'Ġ<pointer:48>', 'Ġ<pointer:49>',\n                      'Ġ<pointer:50>', 'Ġ<pointer:51>', 'Ġ<pointer:52>', 'Ġ<pointer:53>', 'Ġ<pointer:54>',\n                      'Ġ<pointer:55>', 'Ġ<pointer:56>', 'Ġ<pointer:57>', 'Ġ<pointer:58>', 'Ġ<pointer:59>',\n                      'Ġ<pointer:60>', 'Ġ<pointer:61>', 'Ġ<pointer:62>', 'Ġ<pointer:63>', 'Ġ<pointer:64>',\n                      'Ġ<pointer:65>', 'Ġ<pointer:66>', 'Ġ<pointer:67>', 'Ġ<pointer:68>', 'Ġ<pointer:69>',\n                      'Ġ<pointer:70>', 'Ġ<pointer:71>', 'Ġ<pointer:72>', 'Ġ<pointer:73>', 'Ġ<pointer:74>',\n                      'Ġ<pointer:75>', 'Ġ<pointer:76>', 'Ġ<pointer:77>', 'Ġ<pointer:78>', 'Ġ<pointer:79>',\n                      'Ġ<pointer:80>', 'Ġ<pointer:81>', 'Ġ<pointer:82>', 'Ġ<pointer:83>', 'Ġ<pointer:84>',\n                      'Ġ<pointer:85>', 'Ġ<pointer:86>', 'Ġ<pointer:87>', 'Ġ<pointer:88>', 'Ġ<pointer:89>',\n                      'Ġ<pointer:90>', 'Ġ<pointer:91>', 'Ġ<pointer:92>', 'Ġ<pointer:93>', 'Ġ<pointer:94>',\n                      'Ġ<pointer:95>', 'Ġ<pointer:96>', 'Ġ<pointer:97>', 'Ġ<pointer:98>', 'Ġ<pointer:99>',\n                      'Ġ<pointer:100>', 'Ġ<pointer:101>', 'Ġ<pointer:102>', 'Ġ<pointer:103>', 'Ġ<pointer:104>',\n                      'Ġ<pointer:105>', 'Ġ<pointer:106>', 'Ġ<pointer:107>', 'Ġ<pointer:108>', 'Ġ<pointer:109>',\n                      'Ġ<pointer:110>', 'Ġ<pointer:111>', 'Ġ<pointer:112>', 'Ġ<pointer:113>', 'Ġ<pointer:114>',\n                      'Ġ<pointer:115>', 'Ġ<pointer:116>', 'Ġ<pointer:117>', 'Ġ<pointer:118>', 'Ġ<pointer:119>',\n                      'Ġ<pointer:120>', 'Ġ<pointer:121>', 'Ġ<pointer:122>', 'Ġ<pointer:123>', 'Ġ<pointer:124>',\n                      'Ġ<pointer:125>', 'Ġ<pointer:126>', 'Ġ<pointer:127>', 'Ġ<pointer:128>', 'Ġ<pointer:129>',\n                      'Ġ<pointer:130>', 'Ġ<pointer:131>', 'Ġ<pointer:132>', 'Ġ<pointer:133>', 'Ġ<pointer:134>',\n                      'Ġ<pointer:135>', 'Ġ<pointer:136>', 'Ġ<pointer:137>', 'Ġ<pointer:138>', 'Ġ<pointer:139>',\n                      'Ġ<pointer:140>', 'Ġ<pointer:141>', 'Ġ<pointer:142>', 'Ġ<pointer:143>', 'Ġ<pointer:144>',\n                      'Ġ<pointer:145>', 'Ġ<pointer:146>', 'Ġ<pointer:147>', 'Ġ<pointer:148>', 'Ġ<pointer:149>',\n                      'Ġ<pointer:150>', 'Ġ<pointer>', 'Ġ<stop>', 'Ġ<lit>', 'Ġ</lit>', 'Ġ<backr:src:XXX>',\n                      'Ġ<backr:trg:XXX>', '<AMR>', '</AMR>']\nspecial_tokens = [itm.lstrip(\"Ġ\") for itm in raw_special_tokens]\n\nrecategorizations = [\n    \"\\u0120COUNTRY\",\n    \"\\u0120QUANTITY\",\n    \"\\u0120ORGANIZATION\",\n    \"\\u0120DATE_ATTRS\",\n    \"\\u0120NATIONALITY\",\n    \"\\u0120LOCATION\",\n    \"\\u0120ENTITY\",\n    \"\\u0120MISC\",\n    \"\\u0120ORDINAL_ENTITY\",\n    \"\\u0120IDEOLOGY\",\n    \"\\u0120RELIGION\",\n    \"\\u0120STATE_OR_PROVINCE\",\n    \"\\u0120CAUSE_OF_DEATH\",\n    \"\\u0120TITLE\",\n    \"\\u0120DATE\",\n    \"\\u0120NUMBER\",\n    \"\\u0120HANDLE\",\n    \"\\u0120SCORE_ENTITY\",\n    \"\\u0120DURATION\",\n    \"\\u0120ORDINAL\",\n    \"\\u0120MONEY\",\n    \"\\u0120CRIMINAL_CHARGE\",\n]\n\n# special_tokens = [\"<AMR>\", \"</AMR>\"]\n\narg_to_scheduler = {\n    \"linear\": get_linear_schedule_with_warmup,\n    \"cosine\": get_cosine_schedule_with_warmup,\n    \"cosine_w_restarts\": get_cosine_with_hard_restarts_schedule_with_warmup,\n    \"polynomial\": get_polynomial_decay_schedule_with_warmup,\n    \"constant\": get_constant_schedule_with_warmup,\n}\narg_to_scheduler_choices = sorted(arg_to_scheduler.keys())\narg_to_scheduler_metavar = \"{\" + \", \".join(arg_to_scheduler_choices) + \"}\"\n\nROUGE_KEYS = [\"rouge1\", \"rouge2\", \"rougeL\", \"rougeLsum\"]\n\narg_to_tokenizer = {\n    \"AutoTokenizer\": AutoTokenizer,\n    \"BartTokenizer\": BartTokenizer,\n    \"T5Tokenizer\": T5Tokenizer,\n}\narg_to_plm_model = {\n    \"AutoModelForSeq2SeqLM\": AutoModelForSeq2SeqLM,\n    \"BartForConditionalGeneration\": BartForConditionalGeneration,\n    \"T5Model\": T5Model,\n    \"T5ForConditionalGeneration\": T5ForConditionalGeneration,\n}\n"
  },
  {
    "path": "hanlp/components/amr/amrbart/common/penman_interface.py",
    "content": "# coding:utf-8\n# MIT License\n#\n# Copyright (c) 2022 xfbai\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\nfrom penman import load as load_, Graph, Triple\nfrom penman import loads as loads_\nfrom penman import encode as encode_\nfrom penman.model import Model\nfrom penman.models.noop import NoOpModel\nfrom penman.models import amr\n\nop_model = Model()\nnoop_model = NoOpModel()\namr_model = amr.model\nDEFAULT = op_model\n\n\ndef _get_model(dereify):\n    if dereify is None:\n        return DEFAULT\n\n    elif dereify:\n        return op_model\n\n    else:\n        return noop_model\n\n\ndef _remove_wiki(graph):\n    metadata = graph.metadata\n    triples = []\n    for t in graph.triples:\n        v1, rel, v2 = t\n        if rel == \":wiki\":\n            t = Triple(v1, rel, \"+\")\n        triples.append(t)\n    graph = Graph(triples)\n    graph.metadata = metadata\n    return graph\n\n\ndef load(source, dereify=None, remove_wiki=False):\n    model = _get_model(dereify)\n    out = load_(source=source, model=model)\n    if remove_wiki:\n        for i in range(len(out)):\n            out[i] = _remove_wiki(out[i])\n    return out\n\n\ndef loads(string, dereify=None, remove_wiki=False):\n    model = _get_model(dereify)\n    out = loads_(string=string, model=model)\n    if remove_wiki:\n        for i in range(len(out)):\n            out[i] = _remove_wiki(out[i])\n    return out\n\n\ndef encode(g, top=None, indent=-1, compact=False):\n    model = amr_model\n    return encode_(g=g, top=top, indent=indent, compact=compact, model=model)\n"
  },
  {
    "path": "hanlp/components/amr/amrbart/common/postprocessing.py",
    "content": "# coding:utf-8\n# MIT License\n#\n# Copyright (c) 2022 xfbai\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\nimport re\nimport enum\nimport penman\nimport networkx as nx\nfrom hanlp.components.amr.amrbart.common.penman_interface import encode\nfrom collections import defaultdict, Counter\n\nBACKOFF = penman.Graph(\n    [\n        penman.Triple(\"d2\", \":instance\", \"dog\"),\n        penman.Triple(\"b1\", \":instance\", \"bark-01\"),\n        penman.Triple(\"b1\", \":ARG0\", \"d2\"),\n    ]\n)\n\n\ndef token_processing(tok):\n    if tok is None:\n        return None\n    elif tok.isdigit():\n        try:\n            return eval(tok)\n        except:\n            return tok\n    elif tok.startswith('\"') and (not tok.endswith('\"')):\n        return tok + '\"'\n    elif tok.endswith('\"') and (not tok.startswith('\"')):\n        return '\"' + tok\n    else:\n        return tok\n\n\ndef decode_into_node_and_backreferences(subtoken_ids, tokenizer):\n    rex_arg = re.compile(f\"^{tokenizer.INIT}(op|snt|conj|prep)\")\n    rex_spc = re.compile(r\"<(s|/s|lit|/lit|stop|unk|pad|mask)>\")\n    \n    # subtoken_ids.insert(1,36)           # add \"(\" id\n    # subtoken_ids.insert(-1, 4839)       # add \")\" id\n\n    # get strings\n    subtokens = [tokenizer.decoder.get(t) for t in subtoken_ids]\n    # print(\"subtokens:\", subtokens)\n    # fix backreferences\n    \n    subtoken_backreferences = [max(t - len(tokenizer.encoder), -1) for t in subtoken_ids]\n    # strip padding\n    subtokens, subtoken_backreferences = zip(\n        *[\n            (s, b)\n            for s, b in zip(subtokens, subtoken_backreferences)\n            if s != (\"<pad>\")\n        ]\n    )\n\n    # subword collapse\n    tokens = []\n    backreferences = []\n    subword_to_token_map = {}\n    current_token_i = 0\n    for subw_i, (subw_backr, subtok) in enumerate(zip(subtoken_backreferences, subtokens)):\n        subword_to_token_map[subw_i] = current_token_i\n\n        # if empty you cannot do anything but add a new word\n        if not tokens:\n            tokens.append(subtok.lstrip(tokenizer.INIT))\n            backreferences.append(-1)\n            current_token_i += 1\n\n        # backref can't be splitted\n        elif subw_backr > -1:\n            tokens.append(None)\n            backreferences.append(subword_to_token_map[subw_backr])\n            current_token_i += 1\n\n        # after a special token release\n        elif isinstance(tokens[-1], str) and rex_spc.match(tokens[-1]):\n            tokens.append(subtok.lstrip(tokenizer.INIT))\n            backreferences.append(-1)\n            current_token_i += 1\n\n        # after a subtoken ':' (which should be followed by the rest of the edge) ignore tokenizer.INIT\n        # TODO: this is an ugly patch due to the fact that BART tokenizer splits after ':'\n        elif (tokens[-1] == \":\") and rex_arg.match(subtok):\n            tokens[-1] = tokens[-1] + subtok[1:]\n\n        # leading tokenizer.INIT\n        elif subtok.startswith(tokenizer.INIT):\n            tokens.append(subtok.lstrip(tokenizer.INIT))\n            backreferences.append(-1)\n            current_token_i += 1\n\n        # very ugly patch for some cases in which tokenizer.INIT is not in the following token to the edge\n        elif (\n            isinstance(tokens[-1], str)\n            and tokens[-1].startswith(\":\")\n            and tokens[-1][-1].isdigit()\n            and (subtok != \"-of\")\n        ):\n            tokens.append(subtok.lstrip(tokenizer.INIT))\n            backreferences.append(-1)\n            current_token_i += 1\n\n        # in any other case attach to the previous\n        else:\n            tokens[-1] = tokens[-1] + subtok\n\n    # strip INIT and fix byte-level\n    tokens = [\n        tokenizer.convert_tokens_to_string(list(t)).lstrip() if isinstance(t, str) else t\n        for t in tokens\n    ]\n    # tokens = [t.replace(tokenizer.INIT, '') if isinstance(t, str) else t for t in tokens]\n\n    # unks are substituted with thing\n    tokens = [t if t != \"<unk>\" else \"thing\" for t in tokens]\n\n    old_tokens = tokens\n    old_backreferences = backreferences\n\n    # <lit> Barack Obama </lit> -> \"Barack Obama\"\n    tokens = []\n    backreferences = []\n    token_to_token_map = {}\n    start_search = 0\n    removed = 0\n    while True:\n        try:\n\n            lit_start = old_tokens.index(\"<lit>\", start_search)\n            token_addition = old_tokens[start_search:lit_start]\n            for i, t in enumerate(token_addition, start=start_search):\n                token_to_token_map[i] = i - removed\n            tokens += token_addition\n\n            backreferences_addition = [\n                token_to_token_map[b] if b > -1 else -1\n                for b in old_backreferences[start_search:lit_start]\n            ]\n            backreferences += backreferences_addition\n\n            lit_end = min(lit_start + 2, len(old_tokens) - 1)\n\n            while lit_end < len(old_tokens):\n                old_tok = old_tokens[lit_end]\n\n                if isinstance(old_tok, str) and (\n                    (old_tok.startswith(\":\") and len(old_tok) > 3) or (old_tok == \"<stop>\")\n                ):\n                    res_tok = old_tokens[lit_start + 1 : lit_end]\n                    for i in range(lit_start, lit_end):\n                        token_to_token_map[i] = len(tokens)\n\n                    # Remove possible wrong None\n                    res = old_tokens[lit_start + 1 : lit_end]\n                    res = [str(r) for r in res if r is not None]\n                    res = '\"' + \"_\".join(res) + '\"'\n\n                    removed += len(res_tok)\n                    start_search = lit_end\n                    tokens += [res, old_tok]\n                    backreferences += [-1, -1]\n                    break\n\n                elif old_tok == \"</lit>\":\n                    res_tok = old_tokens[lit_start + 1 : lit_end]\n                    for i in range(lit_start, lit_end + 1):\n                        token_to_token_map[i] = len(tokens)\n\n                    # Remove possible wrong None\n                    res = old_tokens[lit_start + 1 : lit_end]\n                    res = [str(r) for r in res if r is not None]\n                    res = '\"' + \"_\".join(res) + '\"'\n\n                    removed += len(res_tok) + 1\n                    start_search = lit_end + 1\n                    tokens.append(res)\n                    backreferences.append(-1)\n                    break\n\n                else:\n                    lit_end += 1\n                    start_search = lit_end\n\n        except ValueError:\n            token_addition = old_tokens[start_search:]\n            for i, t in enumerate(token_addition, start=start_search):\n                token_to_token_map[i] = i - removed\n            backreferences_addition = [\n                token_to_token_map[b] if b > -1 else b for b in old_backreferences[start_search:]\n            ]\n            tokens += token_addition\n            backreferences += backreferences_addition\n            break\n\n    tokens = [token_processing(t) for t in tokens]\n\n    shift = 1\n    if tokens[1] == \"<s>\":\n        shift = 2\n\n    tokens = tokens[shift:]\n    backreferences = [b if b == -1 else b - shift for b in backreferences[shift:]]\n\n    if tokens[-1] == \"</s>\":\n        tokens.pop()\n        backreferences.pop()\n\n    return tokens, backreferences\n\n\ndef index_of(element, iterable, default=None, start=None, end=None):\n    if not callable(element):\n\n        def check(x):\n            return element == x\n\n    else:\n        check = element\n    if start is None:\n        start = 0\n    if end is None:\n        end = len(iterable)\n    item = start\n    while item < end:\n        if check(iterable[item]):\n            return item\n        item += 1\n    return default\n\n\ndef separate_edges_nodes(edges_nodes_slice, *other):\n    is_arg = lambda x: isinstance(x, str) and x.startswith(\":\")\n    start = 0\n    edges = []\n    nodes = []\n    l = len(edges_nodes_slice)\n    while start < l:\n        edge_index = index_of(is_arg, edges_nodes_slice, start=start)\n        if edge_index is None or edge_index == (l - 1):\n            break\n        if is_arg(edges_nodes_slice[edge_index + 1]):\n            start = edge_index + 1\n            continue\n        edges.append(edge_index)\n        nodes.append(edge_index + 1)\n        start = edge_index + 2\n    ret = []\n    for oth in other:\n        edges_oth = [oth[i] for i in edges]\n        nodes_oth = [oth[i] for i in nodes]\n        ret.append((edges_oth, nodes_oth))\n    return ret\n\n\ndef _split_name_ops(graph):\n    # identify name triples\n    name_vars = {}\n    for i, (v1, rel, v2) in enumerate(graph.triples):\n        if rel == \":instance\" and v2 == \"name\":\n            name_vars[v1] = 1\n\n    # check if they have ops\n    name_vars_to_ops = defaultdict(list)\n    for i, (v1, rel, v2) in enumerate(graph.triples):\n        if v1 in name_vars and rel.startswith(\":op\"):\n            name_vars_to_ops[v1].append((i, rel, v2.strip('\"')))\n\n    triples = graph.triples.copy()\n    for nv, ops in name_vars_to_ops.items():\n        ops = sorted(ops, key=lambda x: int(x[1][3:]))\n        idx, _, lits = zip(*ops)\n        for i in idx:\n            triples[i] = None\n\n        lits = ['\"' + l + '\"' for lit in lits for l in lit.split(\"_\")]\n\n        tt = []\n        for i, l in enumerate(lits, start=1):\n            rel = \":op\" + str(i)\n            tt.append(penman.Triple(nv, rel, l))\n\n        triples[min(idx)] = tt\n\n    triples = [t if isinstance(t, list) else [t] for t in triples if t is not None]\n    triples = [t for tt in triples for t in tt]\n\n    graph_ = penman.Graph(triples)\n    graph_.metadata = graph.metadata\n    return graph_\n\n\ndef _reconstruct_graph_from_nodes(nodes, backreferences):\n    triples = []\n    triples_added = set()\n\n    variable2index = {}\n    index2variable = {}\n    start_index = 0\n\n    cnt = defaultdict(Counter)\n\n    while start_index < len(nodes):\n        stop_index = index_of(\"<stop>\", nodes, default=len(nodes) + 1, start=start_index)\n        old_start_index = start_index\n        start_index = stop_index + 1\n\n        src_node, src_backr = nodes[old_start_index], backreferences[old_start_index]\n\n        if src_node == \"<stop>\":\n            continue\n\n        trg_nodes_edges = nodes[old_start_index:stop_index]\n        trg_nodes_edges_backr = backreferences[old_start_index:stop_index]\n        trg_nodes_edges_indices = list(range(old_start_index, stop_index))\n\n        if isinstance(src_node, str):\n            if src_node in (\"<s>\", \"</s>\", \"<stop>\"):\n                continue\n            elif (\"/\" in src_node) or (\":\" in src_node) or (\"(\" in src_node) or (\")\" in src_node):\n                src_node = \"thing\"\n\n        if src_node is not None:\n            src_node = str(src_node)\n            src_var = src_node[0].lower()\n            if not src_var not in \"abcdefghijklmnopqrstuvwxyz\":\n                src_var = \"x\"\n            # src_var = f'{src_var}_{len(variable2index)}'\n            src_var = f\"{src_var}{len(variable2index)}\"\n            src_var_i = old_start_index\n            variable2index[src_var] = src_var_i\n            index2variable[src_var_i] = src_var\n            triple = penman.Triple(src_var, \":instance\", src_node)\n            if triple not in triples_added:\n                triples.append(triple)\n                triples_added.add(triple)\n        else:\n            if src_backr in index2variable:\n                src_var = index2variable[src_backr]\n        # more resilient logic here\n        (trg_edges, trg_nodes), (_, trg_nodes_backr), (_, trg_nodes_indices) = separate_edges_nodes(\n            trg_nodes_edges, trg_nodes_edges, trg_nodes_edges_backr, trg_nodes_edges_indices\n        )\n\n        for n, e, nb, ni in zip(trg_nodes, trg_edges, trg_nodes_backr, trg_nodes_indices):\n\n            if isinstance(n, str) and n.startswith(\":\"):\n                continue\n            if isinstance(n, str) and n.startswith(\"<\") and n.endswith(\">\"):\n                continue\n            if e == \":li\":\n                pass\n            elif len(e) < 4 or (not e.startswith(\":\")):\n                continue\n\n            # same edge more than once\n            num = cnt[src_var][e]\n            # num = 0\n            if num:\n\n                if e.startswith(\":op\") or e.startswith(\":snt\"):\n                    continue\n                # elif e.startswith(':ARG'):\n                #    continue\n                elif num > 3:\n                    continue\n\n            if n is None:\n                if nb not in index2variable:\n                    continue\n                trg_var = index2variable[nb]\n                trg = trg_var\n            elif e == \":mode\":\n                trg = n\n            elif (\n                (not isinstance(n, str))\n                or re.match(r\"^[+-]?\\d+\\.?\\d*$\", n)\n                or (n == \"-\")\n                or (n == \"+\")\n            ):\n                trg = str(n)\n            elif n.startswith('\"') and n.endswith('\"') and len(n) > 2:\n                trg = '\"' + n.replace('\"', \"\") + '\"'\n            elif (\"/\" in n) or (\":\" in n) or (\"(\" in n) or (\")\" in n) or (\"=\" in n):\n                trg = f'\"{n}\"'\n            elif n == '\"':\n                continue\n            elif (\n                (n.startswith('\"') and (not n.endswith('\"')))\n                or (not n.startswith('\"') and (n.endswith('\"')))\n                or ('\"' in n)\n            ):\n                trg = '\"' + n.replace('\"', \"\") + '\"'\n            else:\n                trg_var = n[0].lower()\n                if trg_var not in \"abcdefghijklmnopqrstuvwxyz\":\n                    trg_var = \"x\"\n                # trg_var = f'{trg_var}_{len(variable2index)}'\n                trg_var = f\"{trg_var}{len(variable2index)}\"\n                trg_var_i = ni\n                variable2index[trg_var] = trg_var_i\n                index2variable[trg_var_i] = trg_var\n                triple = penman.Triple(trg_var, \":instance\", n)\n                if triple not in triples_added:\n                    triples.append(triple)\n                    triples_added.add(triple)\n                trg = trg_var\n\n            triple = penman.Triple(src_var, e, trg)\n            if triple not in triples_added:\n                triples.append(triple)\n                triples_added.add(triple)\n\n            cnt[src_var][e] += 1\n\n    return penman.Graph(triples)\n\n\ndef build_graph(nodes, backreferences, restore_name_ops=False):\n    graph = _reconstruct_graph_from_nodes(nodes, backreferences)\n    if restore_name_ops:\n        graph = _split_name_ops(graph)\n    return graph\n\n\nclass ParsedStatus(enum.Enum):\n    OK = 0\n    FIXED = 1\n    BACKOFF = 2\n\n\ndef connect_graph_if_not_connected(graph):\n\n    try:\n        encoded = encode(graph)\n        return graph, ParsedStatus.OK\n    except:\n        pass\n\n    nxgraph = nx.MultiGraph()\n    variables = graph.variables()\n    for v1, _, v2 in graph.triples:\n        if v1 in variables and v2 in variables:\n            nxgraph.add_edge(v1, v2)\n        elif v1 in variables:\n            nxgraph.add_edge(v1, v1)\n\n    triples = graph.triples.copy()\n    new_triples = []\n    addition = f\"a{len(variables) + 1}\"\n    triples.append(penman.Triple(addition, \":instance\", \"and\"))\n    for i, conn_set in enumerate(nx.connected_components(nxgraph), start=1):\n        edge = f\":op{i}\"\n        conn_set = sorted(conn_set, key=lambda x: int(x[1:]))\n        conn_set = [c for c in conn_set if c in variables]\n        node = conn_set[0]\n        new_triples.append(penman.Triple(addition, edge, node))\n    triples = new_triples + triples\n    metadata = graph.metadata\n    graph = penman.Graph(triples)\n    graph.metadata.update(metadata)\n    encode(graph)\n\n    return graph, ParsedStatus.FIXED\n\n\ndef restore_backreferences_from_pointers(nodes):\n    new_nodes, new_backreferences = [], []\n    prev_pointer = None\n    pointer2i = {}\n    for n in nodes:\n        is_pointer = isinstance(n, str) and n.startswith(\"<pointer:\") and n.endswith(\">\")\n\n        if not is_pointer:\n            if prev_pointer is not None:\n                if prev_pointer in pointer2i:\n                    new_nodes.append(None)\n                    new_backreferences.append(pointer2i[prev_pointer])\n                    new_nodes.append(n)\n                    new_backreferences.append(-1)\n\n                else:\n                    pointer2i[prev_pointer] = len(new_nodes)\n                    new_nodes.append(n)\n                    new_backreferences.append(-1)\n            else:\n                new_nodes.append(n)\n                new_backreferences.append(-1)\n\n            prev_pointer = None\n        else:\n            prev_pointer = n\n    return new_nodes, new_backreferences\n"
  },
  {
    "path": "hanlp/components/amr/amrbart/data_interface/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2022-12-07 14:36\n"
  },
  {
    "path": "hanlp/components/amr/amrbart/data_interface/dataset.py",
    "content": "# coding:utf-8\n# MIT License\n#\n# Copyright (c) 2022 xfbai\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\n\nclass AMRParsingDataSet(object):\n\n    @staticmethod\n    def tokenize(sample: dict, tokenizer, max_src_length=400, max_tgt_length=1024, unified_input=True, amr=\"src\",\n                 text=\"tgt\"):\n        amr = sample.get(amr, None)  # AMR tokens\n        txt = sample[text]  # Text tokens\n\n        if amr is not None:\n            sample['labels'] = tokenizer.tokenize_amr(amr.split())[:max_src_length - 2] + [tokenizer.amr_eos_token_id]\n\n        raw_txt_ids = tokenizer(\n            txt, max_length=max_tgt_length, padding=False, truncation=True\n        )[\"input_ids\"]\n        if unified_input:\n            txt_ids = raw_txt_ids[:max_tgt_length - 3] + [tokenizer.amr_bos_token_id, tokenizer.mask_token_id,\n                                                          tokenizer.amr_eos_token_id]\n        else:\n            txt_ids = raw_txt_ids\n        sample['input_ids'] = txt_ids\n        return sample\n\n\nclass AMR2TextDataSet(object):\n\n    @staticmethod\n    def tokenize(sample: dict, tokenizer, max_src_length=400, max_tgt_length=1024, unified_input=True, amr=\"src\",\n                 text=\"tgt\"):\n        src = sample[amr]  # AMR tokens\n        tgt = sample.get(text, None)  # Text tokens\n        if not unified_input:\n            src_ids = [tokenizer.amr_bos_token_id] + tokenizer.tokenize_amr(src.split())[\n                                                     :max_src_length - 2] + [tokenizer.amr_eos_token_id]\n\n        else:\n            # [<s>[mask]</s><AMR>xxx</AMR>]\n            src_ids = [tokenizer.bos_token_id, tokenizer.mask_token_id, tokenizer.eos_token_id] + [\n                tokenizer.amr_bos_token_id] + tokenizer.tokenize_amr(src.split())[:max_src_length - 5] + [\n                          tokenizer.amr_eos_token_id]\n        sample[\"input_ids\"] = src_ids\n\n        if tgt is not None:\n            with tokenizer.as_target_tokenizer():\n                tgt_ids = tokenizer(\n                    tgt, max_length=max_tgt_length, padding=False, truncation=True\n                )\n                tgt_ids[\"input_ids\"] = [\n                    label[1:] for label in tgt_ids[\"input_ids\"]\n                ]\n            sample[\"labels\"] = tgt_ids[\"input_ids\"]\n        return sample\n"
  },
  {
    "path": "hanlp/components/amr/amrbart/model_interface/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2022-12-03 20:33\n"
  },
  {
    "path": "hanlp/components/amr/amrbart/model_interface/modeling_bart.py",
    "content": "# coding=utf-8\n# Copyright 2021 The Fairseq Authors and The HuggingFace Inc. team. All rights reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\" PyTorch BART model.\"\"\"\nimport copy\nimport math\nimport random\nimport warnings\nfrom typing import List, Optional, Tuple, Union\n\nimport torch\nimport torch.utils.checkpoint\nfrom torch import nn\nfrom torch.nn import BCEWithLogitsLoss, CrossEntropyLoss, MSELoss\n\nfrom transformers.activations import ACT2FN\nfrom transformers.modeling_outputs import (\n    BaseModelOutput,\n    BaseModelOutputWithPastAndCrossAttentions,\n    CausalLMOutputWithCrossAttentions,\n    Seq2SeqLMOutput,\n    Seq2SeqModelOutput,\n    Seq2SeqQuestionAnsweringModelOutput,\n    Seq2SeqSequenceClassifierOutput,\n)\nfrom transformers.modeling_utils import PreTrainedModel\nfrom transformers.utils import (\n    add_code_sample_docstrings,\n    add_end_docstrings,\n    add_start_docstrings,\n    add_start_docstrings_to_model_forward,\n    logging,\n    replace_return_docstrings,\n)\nfrom transformers.models.bart.configuration_bart import BartConfig\n\n\nlogger = logging.get_logger(__name__)\n\n_CHECKPOINT_FOR_DOC = \"facebook/bart-base\"\n_CONFIG_FOR_DOC = \"BartConfig\"\n_TOKENIZER_FOR_DOC = \"BartTokenizer\"\n\n# Base model docstring\n_EXPECTED_OUTPUT_SHAPE = [1, 8, 768]\n\n# SequenceClassification docstring\n_CHECKPOINT_FOR_SEQUENCE_CLASSIFICATION = \"valhalla/bart-large-sst2\"\n_SEQ_CLASS_EXPECTED_LOSS = 0.0\n_SEQ_CLASS_EXPECTED_OUTPUT = \"'POSITIVE'\"\n\n# QuestionAsnwering docstring\n_CHECKPOINT_FOR_QA = \"valhalla/bart-large-finetuned-squadv1\"\n_QA_EXPECTED_LOSS = 0.59\n_QA_EXPECTED_OUTPUT = \"' nice puppet'\"\n\n\nBART_PRETRAINED_MODEL_ARCHIVE_LIST = [\n    \"facebook/bart-large\",\n    # see all BART models at https://huggingface.co/models?filter=bart\n]\n\n\ndef shift_tokens_right(input_ids: torch.Tensor, pad_token_id: int, decoder_start_token_id: int):\n    \"\"\"\n    Shift input ids one token to the right.\n    \"\"\"\n    shifted_input_ids = input_ids.new_zeros(input_ids.shape)\n    shifted_input_ids[:, 1:] = input_ids[:, :-1].clone()\n    shifted_input_ids[:, 0] = decoder_start_token_id\n\n    if pad_token_id is None:\n        raise ValueError(\"self.model.config.pad_token_id has to be defined.\")\n    # replace possible -100 values in labels by `pad_token_id`\n    shifted_input_ids.masked_fill_(shifted_input_ids == -100, pad_token_id)\n\n    return shifted_input_ids\n\n\ndef _make_causal_mask(input_ids_shape: torch.Size, dtype: torch.dtype, past_key_values_length: int = 0):\n    \"\"\"\n    Make causal mask used for bi-directional self-attention.\n    \"\"\"\n    bsz, tgt_len = input_ids_shape\n    mask = torch.full((tgt_len, tgt_len), torch.tensor(torch.finfo(dtype).min))\n    mask_cond = torch.arange(mask.size(-1))\n    mask.masked_fill_(mask_cond < (mask_cond + 1).view(mask.size(-1), 1), 0)\n    mask = mask.to(dtype)\n\n    if past_key_values_length > 0:\n        mask = torch.cat([torch.zeros(tgt_len, past_key_values_length, dtype=dtype), mask], dim=-1)\n    return mask[None, None, :, :].expand(bsz, 1, tgt_len, tgt_len + past_key_values_length)\n\n\ndef _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None):\n    \"\"\"\n    Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.\n    \"\"\"\n    bsz, src_len = mask.size()\n    tgt_len = tgt_len if tgt_len is not None else src_len\n\n    expanded_mask = mask[:, None, None, :].expand(bsz, 1, tgt_len, src_len).to(dtype)\n\n    inverted_mask = 1.0 - expanded_mask\n\n    return inverted_mask.masked_fill(inverted_mask.to(torch.bool), torch.finfo(dtype).min)\n\n\nclass BartLearnedPositionalEmbedding(nn.Embedding):\n    \"\"\"\n    This module learns positional embeddings up to a fixed maximum size.\n    \"\"\"\n\n    def __init__(self, num_embeddings: int, embedding_dim: int):\n        # Bart is set up so that if padding_idx is specified then offset the embedding ids by 2\n        # and adjust num_embeddings appropriately. Other models don't have this hack\n        self.offset = 2\n        super().__init__(num_embeddings + self.offset, embedding_dim)\n\n    def forward(self, input_ids_shape: torch.Size, past_key_values_length: int = 0):\n        \"\"\"`input_ids_shape` is expected to be [bsz x seqlen].\"\"\"\n        bsz, seq_len = input_ids_shape[:2]\n        positions = torch.arange(\n            past_key_values_length, past_key_values_length + seq_len, dtype=torch.long, device=self.weight.device\n        )\n        return super().forward(positions + self.offset)\n\n\nclass BartAttention(nn.Module):\n    \"\"\"Multi-headed attention from 'Attention Is All You Need' paper\"\"\"\n\n    def __init__(\n        self,\n        embed_dim: int,\n        num_heads: int,\n        dropout: float = 0.0,\n        is_decoder: bool = False,\n        bias: bool = True,\n    ):\n        super().__init__()\n        self.embed_dim = embed_dim\n        self.num_heads = num_heads\n        self.dropout = dropout\n        self.head_dim = embed_dim // num_heads\n\n        if (self.head_dim * num_heads) != self.embed_dim:\n            raise ValueError(\n                f\"embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim}\"\n                f\" and `num_heads`: {num_heads}).\"\n            )\n        self.scaling = self.head_dim**-0.5\n        self.is_decoder = is_decoder\n\n        self.k_proj = nn.Linear(embed_dim, embed_dim, bias=bias)\n        self.v_proj = nn.Linear(embed_dim, embed_dim, bias=bias)\n        self.q_proj = nn.Linear(embed_dim, embed_dim, bias=bias)\n        self.out_proj = nn.Linear(embed_dim, embed_dim, bias=bias)\n\n    def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int):\n        return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).transpose(1, 2).contiguous()\n\n    def forward(\n        self,\n        hidden_states: torch.Tensor,\n        key_value_states: Optional[torch.Tensor] = None,\n        past_key_value: Optional[Tuple[torch.Tensor]] = None,\n        attention_mask: Optional[torch.Tensor] = None,\n        layer_head_mask: Optional[torch.Tensor] = None,\n        output_attentions: bool = False,\n    ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:\n        \"\"\"Input shape: Batch x Time x Channel\"\"\"\n\n        # if key_value_states are provided this layer is used as a cross-attention layer\n        # for the decoder\n        is_cross_attention = key_value_states is not None\n\n        bsz, tgt_len, _ = hidden_states.size()\n\n        # get query proj\n        query_states = self.q_proj(hidden_states) * self.scaling\n        # get key, value proj\n        if is_cross_attention and past_key_value is not None:\n            # reuse k,v, cross_attentions\n            key_states = past_key_value[0]\n            value_states = past_key_value[1]\n        elif is_cross_attention:\n            # cross_attentions\n            key_states = self._shape(self.k_proj(key_value_states), -1, bsz)\n            value_states = self._shape(self.v_proj(key_value_states), -1, bsz)\n        elif past_key_value is not None:\n            # reuse k, v, self_attention\n            key_states = self._shape(self.k_proj(hidden_states), -1, bsz)\n            value_states = self._shape(self.v_proj(hidden_states), -1, bsz)\n            key_states = torch.cat([past_key_value[0], key_states], dim=2)\n            value_states = torch.cat([past_key_value[1], value_states], dim=2)\n        else:\n            # self_attention\n            key_states = self._shape(self.k_proj(hidden_states), -1, bsz)\n            value_states = self._shape(self.v_proj(hidden_states), -1, bsz)\n\n        if self.is_decoder:\n            # if cross_attention save Tuple(torch.Tensor, torch.Tensor) of all cross attention key/value_states.\n            # Further calls to cross_attention layer can then reuse all cross-attention\n            # key/value_states (first \"if\" case)\n            # if uni-directional self-attention (decoder) save Tuple(torch.Tensor, torch.Tensor) of\n            # all previous decoder key/value_states. Further calls to uni-directional self-attention\n            # can concat previous decoder key/value_states to current projected key/value_states (third \"elif\" case)\n            # if encoder bi-directional self-attention `past_key_value` is always `None`\n            past_key_value = (key_states, value_states)\n\n        proj_shape = (bsz * self.num_heads, -1, self.head_dim)\n        query_states = self._shape(query_states, tgt_len, bsz).view(*proj_shape)\n        key_states = key_states.view(*proj_shape)\n        value_states = value_states.view(*proj_shape)\n\n        src_len = key_states.size(1)\n        attn_weights = torch.bmm(query_states, key_states.transpose(1, 2))\n\n        if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):\n            raise ValueError(\n                f\"Attention weights should be of size {(bsz * self.num_heads, tgt_len, src_len)}, but is\"\n                f\" {attn_weights.size()}\"\n            )\n\n        if attention_mask is not None:\n            if attention_mask.size() != (bsz, 1, tgt_len, src_len):\n                raise ValueError(\n                    f\"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is {attention_mask.size()}\"\n                )\n            attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attention_mask\n            attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)\n\n        attn_weights = nn.functional.softmax(attn_weights, dim=-1)\n\n        if layer_head_mask is not None:\n            if layer_head_mask.size() != (self.num_heads,):\n                raise ValueError(\n                    f\"Head mask for a single layer should be of size {(self.num_heads,)}, but is\"\n                    f\" {layer_head_mask.size()}\"\n                )\n            attn_weights = layer_head_mask.view(1, -1, 1, 1) * attn_weights.view(bsz, self.num_heads, tgt_len, src_len)\n            attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)\n\n        if output_attentions:\n            # this operation is a bit awkward, but it's required to\n            # make sure that attn_weights keeps its gradient.\n            # In order to do so, attn_weights have to be reshaped\n            # twice and have to be reused in the following\n            attn_weights_reshaped = attn_weights.view(bsz, self.num_heads, tgt_len, src_len)\n            attn_weights = attn_weights_reshaped.view(bsz * self.num_heads, tgt_len, src_len)\n        else:\n            attn_weights_reshaped = None\n\n        attn_probs = nn.functional.dropout(attn_weights, p=self.dropout, training=self.training)\n\n        attn_output = torch.bmm(attn_probs, value_states)\n\n        if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):\n            raise ValueError(\n                f\"`attn_output` should be of size {(bsz, self.num_heads, tgt_len, self.head_dim)}, but is\"\n                f\" {attn_output.size()}\"\n            )\n\n        attn_output = attn_output.view(bsz, self.num_heads, tgt_len, self.head_dim)\n        attn_output = attn_output.transpose(1, 2)\n\n        # Use the `embed_dim` from the config (stored in the class) rather than `hidden_state` because `attn_output` can be\n        # partitioned aross GPUs when using tensor-parallelism.\n        attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim)\n\n        attn_output = self.out_proj(attn_output)\n\n        return attn_output, attn_weights_reshaped, past_key_value\n\n\nclass BartEncoderLayer(nn.Module):\n    def __init__(self, config: BartConfig):\n        super().__init__()\n        self.embed_dim = config.d_model\n        self.self_attn = BartAttention(\n            embed_dim=self.embed_dim,\n            num_heads=config.encoder_attention_heads,\n            dropout=config.attention_dropout,\n        )\n        self.self_attn_layer_norm = nn.LayerNorm(self.embed_dim)\n        self.dropout = config.dropout\n        self.activation_fn = ACT2FN[config.activation_function]\n        self.activation_dropout = config.activation_dropout\n        self.fc1 = nn.Linear(self.embed_dim, config.encoder_ffn_dim)\n        self.fc2 = nn.Linear(config.encoder_ffn_dim, self.embed_dim)\n        self.final_layer_norm = nn.LayerNorm(self.embed_dim)\n\n    def forward(\n        self,\n        hidden_states: torch.FloatTensor,\n        attention_mask: torch.FloatTensor,\n        layer_head_mask: torch.FloatTensor,\n        output_attentions: Optional[bool] = False,\n    ) -> Tuple[torch.FloatTensor, Optional[torch.FloatTensor]]:\n        \"\"\"\n        Args:\n            hidden_states (`torch.FloatTensor`): input to the layer of shape `(seq_len, batch, embed_dim)`\n            attention_mask (`torch.FloatTensor`): attention mask of size\n                `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.\n            layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size\n                `(encoder_attention_heads,)`.\n            output_attentions (`bool`, *optional*):\n                Whether or not to return the attentions tensors of all attention layers. See `attentions` under\n                returned tensors for more detail.\n        \"\"\"\n        residual = hidden_states\n        hidden_states, attn_weights, _ = self.self_attn(\n            hidden_states=hidden_states,\n            attention_mask=attention_mask,\n            layer_head_mask=layer_head_mask,\n            output_attentions=output_attentions,\n        )\n        hidden_states = nn.functional.dropout(hidden_states, p=self.dropout, training=self.training)\n        hidden_states = residual + hidden_states\n        hidden_states = self.self_attn_layer_norm(hidden_states)\n\n        residual = hidden_states\n        hidden_states = self.activation_fn(self.fc1(hidden_states))\n        hidden_states = nn.functional.dropout(hidden_states, p=self.activation_dropout, training=self.training)\n        hidden_states = self.fc2(hidden_states)\n        hidden_states = nn.functional.dropout(hidden_states, p=self.dropout, training=self.training)\n        hidden_states = residual + hidden_states\n        hidden_states = self.final_layer_norm(hidden_states)\n\n        if hidden_states.dtype == torch.float16 and (\n            torch.isinf(hidden_states).any() or torch.isnan(hidden_states).any()\n        ):\n            clamp_value = torch.finfo(hidden_states.dtype).max - 1000\n            hidden_states = torch.clamp(hidden_states, min=-clamp_value, max=clamp_value)\n\n        outputs = (hidden_states,)\n\n        if output_attentions:\n            outputs += (attn_weights,)\n\n        return outputs\n\n\nclass BartDecoderLayer(nn.Module):\n    def __init__(self, config: BartConfig):\n        super().__init__()\n        self.embed_dim = config.d_model\n\n        self.self_attn = BartAttention(\n            embed_dim=self.embed_dim,\n            num_heads=config.decoder_attention_heads,\n            dropout=config.attention_dropout,\n            is_decoder=True,\n        )\n        self.dropout = config.dropout\n        self.activation_fn = ACT2FN[config.activation_function]\n        self.activation_dropout = config.activation_dropout\n\n        self.self_attn_layer_norm = nn.LayerNorm(self.embed_dim)\n        self.encoder_attn = BartAttention(\n            self.embed_dim,\n            config.decoder_attention_heads,\n            dropout=config.attention_dropout,\n            is_decoder=True,\n        )\n        self.encoder_attn_layer_norm = nn.LayerNorm(self.embed_dim)\n        self.fc1 = nn.Linear(self.embed_dim, config.decoder_ffn_dim)\n        self.fc2 = nn.Linear(config.decoder_ffn_dim, self.embed_dim)\n        self.final_layer_norm = nn.LayerNorm(self.embed_dim)\n\n    def forward(\n        self,\n        hidden_states: torch.Tensor,\n        attention_mask: Optional[torch.Tensor] = None,\n        encoder_hidden_states: Optional[torch.Tensor] = None,\n        encoder_attention_mask: Optional[torch.Tensor] = None,\n        layer_head_mask: Optional[torch.Tensor] = None,\n        cross_attn_layer_head_mask: Optional[torch.Tensor] = None,\n        past_key_value: Optional[Tuple[torch.Tensor]] = None,\n        output_attentions: Optional[bool] = False,\n        use_cache: Optional[bool] = True,\n    ) -> Tuple[torch.FloatTensor, Optional[Tuple[torch.FloatTensor, torch.FloatTensor]]]:\n        \"\"\"\n        Args:\n            hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)`\n            attention_mask (`torch.FloatTensor`): attention mask of size\n                `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.\n            encoder_hidden_states (`torch.FloatTensor`):\n                cross attention input to the layer of shape `(batch, seq_len, embed_dim)`\n            encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size\n                `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.\n            layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size\n                `(encoder_attention_heads,)`.\n            cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of\n                size `(decoder_attention_heads,)`.\n            past_key_value (`Tuple(torch.FloatTensor)`): cached past key and value projection states\n            output_attentions (`bool`, *optional*):\n                Whether or not to return the attentions tensors of all attention layers. See `attentions` under\n                returned tensors for more detail.\n        \"\"\"\n        residual = hidden_states\n\n        # Self Attention\n        # decoder uni-directional self-attention cached key/values tuple is at positions 1,2\n        self_attn_past_key_value = past_key_value[:2] if past_key_value is not None else None\n        # add present self-attn cache to positions 1,2 of present_key_value tuple\n        hidden_states, self_attn_weights, present_key_value = self.self_attn(\n            hidden_states=hidden_states,\n            past_key_value=self_attn_past_key_value,\n            attention_mask=attention_mask,\n            layer_head_mask=layer_head_mask,\n            output_attentions=output_attentions,\n        )\n        hidden_states = nn.functional.dropout(hidden_states, p=self.dropout, training=self.training)\n        hidden_states = residual + hidden_states\n        hidden_states = self.self_attn_layer_norm(hidden_states)\n\n        # Cross-Attention Block\n        cross_attn_present_key_value = None\n        cross_attn_weights = None\n        if encoder_hidden_states is not None:\n            residual = hidden_states\n\n            # cross_attn cached key/values tuple is at positions 3,4 of present_key_value tuple\n            cross_attn_past_key_value = past_key_value[-2:] if past_key_value is not None else None\n            hidden_states, cross_attn_weights, cross_attn_present_key_value = self.encoder_attn(\n                hidden_states=hidden_states,\n                key_value_states=encoder_hidden_states,\n                attention_mask=encoder_attention_mask,\n                layer_head_mask=cross_attn_layer_head_mask,\n                past_key_value=cross_attn_past_key_value,\n                output_attentions=output_attentions,\n            )\n            hidden_states = nn.functional.dropout(hidden_states, p=self.dropout, training=self.training)\n            hidden_states = residual + hidden_states\n            hidden_states = self.encoder_attn_layer_norm(hidden_states)\n\n            # add cross-attn to positions 3,4 of present_key_value tuple\n            present_key_value = present_key_value + cross_attn_present_key_value\n\n        # Fully Connected\n        residual = hidden_states\n        hidden_states = self.activation_fn(self.fc1(hidden_states))\n        hidden_states = nn.functional.dropout(hidden_states, p=self.activation_dropout, training=self.training)\n        hidden_states = self.fc2(hidden_states)\n        hidden_states = nn.functional.dropout(hidden_states, p=self.dropout, training=self.training)\n        hidden_states = residual + hidden_states\n        hidden_states = self.final_layer_norm(hidden_states)\n\n        outputs = (hidden_states,)\n\n        if output_attentions:\n            outputs += (self_attn_weights, cross_attn_weights)\n\n        if use_cache:\n            outputs += (present_key_value,)\n\n        return outputs\n\n\nclass BartClassificationHead(nn.Module):\n    \"\"\"Head for sentence-level classification tasks.\"\"\"\n\n    def __init__(\n        self,\n        input_dim: int,\n        inner_dim: int,\n        num_classes: int,\n        pooler_dropout: float,\n    ):\n        super().__init__()\n        self.dense = nn.Linear(input_dim, inner_dim)\n        self.dropout = nn.Dropout(p=pooler_dropout)\n        self.out_proj = nn.Linear(inner_dim, num_classes)\n\n    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:\n        hidden_states = self.dropout(hidden_states)\n        hidden_states = self.dense(hidden_states)\n        hidden_states = torch.tanh(hidden_states)\n        hidden_states = self.dropout(hidden_states)\n        hidden_states = self.out_proj(hidden_states)\n        return hidden_states\n\n\nclass BartPretrainedModel(PreTrainedModel):\n    config_class = BartConfig\n    base_model_prefix = \"model\"\n    supports_gradient_checkpointing = True\n    _keys_to_ignore_on_load_unexpected = [r\"encoder.version\", r\"decoder.version\"]\n\n    def _init_weights(self, module):\n        std = self.config.init_std\n        if isinstance(module, nn.Linear):\n            module.weight.data.normal_(mean=0.0, std=std)\n            if module.bias is not None:\n                module.bias.data.zero_()\n        elif isinstance(module, nn.Embedding):\n            module.weight.data.normal_(mean=0.0, std=std)\n            if module.padding_idx is not None:\n                module.weight.data[module.padding_idx].zero_()\n\n    def _set_gradient_checkpointing(self, module, value=False):\n        if isinstance(module, (BartDecoder, BartEncoder)):\n            module.gradient_checkpointing = value\n\n    @property\n    def dummy_inputs(self):\n        pad_token = self.config.pad_token_id\n        input_ids = torch.tensor([[0, 6, 10, 4, 2], [0, 8, 12, 2, pad_token]], device=self.device)\n        dummy_inputs = {\n            \"attention_mask\": input_ids.ne(pad_token),\n            \"input_ids\": input_ids,\n        }\n        return dummy_inputs\n\n\nclass PretrainedBartModel(BartPretrainedModel):\n    def __init_subclass__(self):\n        warnings.warn(\n            \"The class `PretrainedBartModel` has been depreciated, please use `BartPretrainedModel` instead.\",\n            FutureWarning,\n        )\n\n\nBART_START_DOCSTRING = r\"\"\"\n    This model inherits from [`PreTrainedModel`]. Check the superclass documentation for the generic methods the\n    library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads\n    etc.)\n\n    This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass.\n    Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage\n    and behavior.\n\n    Parameters:\n        config ([`BartConfig`]):\n            Model configuration class with all the parameters of the model. Initializing with a config file does not\n            load the weights associated with the model, only the configuration. Check out the\n            [`~PreTrainedModel.from_pretrained`] method to load the model weights.\n\"\"\"\n\nBART_GENERATION_EXAMPLE = r\"\"\"\n    Summarization example:\n\n    ```python\n    >>> from transformers import BartTokenizer, BartForConditionalGeneration\n\n    >>> model = BartForConditionalGeneration.from_pretrained(\"facebook/bart-large-cnn\")\n    >>> tokenizer = BartTokenizer.from_pretrained(\"facebook/bart-large-cnn\")\n\n    >>> ARTICLE_TO_SUMMARIZE = (\n    ...     \"PG&E stated it scheduled the blackouts in response to forecasts for high winds \"\n    ...     \"amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were \"\n    ...     \"scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow.\"\n    ... )\n    >>> inputs = tokenizer([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors=\"pt\")\n\n    >>> # Generate Summary\n    >>> summary_ids = model.generate(inputs[\"input_ids\"], num_beams=2, min_length=0, max_length=20)\n    >>> tokenizer.batch_decode(summary_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]\n    'PG&E scheduled the blackouts in response to forecasts for high winds amid dry conditions'\n    ```\n\n    Mask filling example:\n\n    ```python\n    >>> from transformers import BartTokenizer, BartForConditionalGeneration\n\n    >>> tokenizer = BartTokenizer.from_pretrained(\"facebook/bart-base\")\n    >>> model = BartForConditionalGeneration.from_pretrained(\"facebook/bart-base\")\n\n    >>> TXT = \"My friends are <mask> but they eat too many carbs.\"\n    >>> input_ids = tokenizer([TXT], return_tensors=\"pt\")[\"input_ids\"]\n    >>> logits = model(input_ids).logits\n\n    >>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item()\n    >>> probs = logits[0, masked_index].softmax(dim=0)\n    >>> values, predictions = probs.topk(5)\n\n    >>> tokenizer.decode(predictions).split()\n    ['not', 'good', 'healthy', 'great', 'very']\n    ```\n\"\"\"\n\nBART_INPUTS_DOCSTRING = r\"\"\"\n    Args:\n        input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):\n            Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide\n            it.\n\n            Indices can be obtained using [`BartTokenizer`]. See [`PreTrainedTokenizer.encode`] and\n            [`PreTrainedTokenizer.__call__`] for details.\n\n            [What are input IDs?](../glossary#input-ids)\n        attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):\n            Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:\n\n            - 1 for tokens that are **not masked**,\n            - 0 for tokens that are **masked**.\n\n            [What are attention masks?](../glossary#attention-mask)\n        decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*):\n            Indices of decoder input sequence tokens in the vocabulary.\n\n            Indices can be obtained using [`BartTokenizer`]. See [`PreTrainedTokenizer.encode`] and\n            [`PreTrainedTokenizer.__call__`] for details.\n\n            [What are decoder input IDs?](../glossary#decoder-input-ids)\n\n            Bart uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values`\n            is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`).\n\n            For translation and summarization training, `decoder_input_ids` should be provided. If no\n            `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right\n            for denoising pre-training following the paper.\n        decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*):\n            Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also\n            be used by default.\n\n            If you want to change padding behavior, you should read [`modeling_bart._prepare_decoder_attention_mask`]\n            and modify to your needs. See diagram 1 in [the paper](https://arxiv.org/abs/1910.13461) for more\n            information on the default strategy.\n        head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*):\n            Mask to nullify selected heads of the attention modules in the encoder. Mask values selected in `[0, 1]`:\n\n            - 1 indicates the head is **not masked**,\n            - 0 indicates the head is **masked**.\n\n        decoder_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*):\n            Mask to nullify selected heads of the attention modules in the decoder. Mask values selected in `[0, 1]`:\n\n            - 1 indicates the head is **not masked**,\n            - 0 indicates the head is **masked**.\n\n        cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*):\n            Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0,\n            1]`:\n\n            - 1 indicates the head is **not masked**,\n            - 0 indicates the head is **masked**.\n\n        encoder_outputs (`tuple(tuple(torch.FloatTensor)`, *optional*):\n            Tuple consists of (`last_hidden_state`, *optional*: `hidden_states`, *optional*: `attentions`)\n            `last_hidden_state` of shape `(batch_size, sequence_length, hidden_size)`, *optional*) is a sequence of\n            hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder.\n        past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`):\n            Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape\n            `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape\n            `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`.\n\n            Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention\n            blocks) that can be used (see `past_key_values` input) to speed up sequential decoding.\n\n            If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that\n            don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all\n            `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape\n            `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you\n            can choose to directly pass an embedded representation. This is useful if you want more control over how to\n            convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix.\n        decoder_inputs_embeds (`torch.FloatTensor` of shape `(batch_size, target_sequence_length, hidden_size)`, *optional*):\n            Optionally, instead of passing `decoder_input_ids` you can choose to directly pass an embedded\n            representation. If `past_key_values` is used, optionally only the last `decoder_inputs_embeds` have to be\n            input (see `past_key_values`). This is useful if you want more control over how to convert\n            `decoder_input_ids` indices into associated vectors than the model's internal embedding lookup matrix.\n\n            If `decoder_input_ids` and `decoder_inputs_embeds` are both unset, `decoder_inputs_embeds` takes the value\n            of `inputs_embeds`.\n        use_cache (`bool`, *optional*):\n            If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see\n            `past_key_values`).\n        output_attentions (`bool`, *optional*):\n            Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned\n            tensors for more detail.\n        output_hidden_states (`bool`, *optional*):\n            Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for\n            more detail.\n        return_dict (`bool`, *optional*):\n            Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.\n\"\"\"\n\n\nclass BartEncoder(BartPretrainedModel):\n    \"\"\"\n    Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a\n    [`BartEncoderLayer`].\n\n    Args:\n        config: BartConfig\n        embed_tokens (nn.Embedding): output embedding\n    \"\"\"\n\n    def __init__(self, config: BartConfig, embed_tokens: Optional[nn.Embedding] = None):\n        super().__init__(config)\n\n        self.dropout = config.dropout\n        self.layerdrop = config.encoder_layerdrop\n\n        embed_dim = config.d_model\n        self.padding_idx = config.pad_token_id\n        self.max_source_positions = config.max_position_embeddings\n        self.embed_scale = math.sqrt(embed_dim) if config.scale_embedding else 1.0\n\n        if embed_tokens is not None:\n            self.embed_tokens = embed_tokens\n        else:\n            self.embed_tokens = nn.Embedding(config.vocab_size, embed_dim, self.padding_idx)\n\n        self.embed_positions = BartLearnedPositionalEmbedding(\n            config.max_position_embeddings,\n            embed_dim,\n        )\n        self.layers = nn.ModuleList([BartEncoderLayer(config) for _ in range(config.encoder_layers)])\n        self.layernorm_embedding = nn.LayerNorm(embed_dim)\n\n        self.gradient_checkpointing = False\n        # Initialize weights and apply final processing\n        self.post_init()\n\n    def get_input_embeddings(self):\n        return self.embed_tokens\n\n    def set_input_embeddings(self, value):\n        self.embed_tokens = value\n\n    def forward(\n        self,\n        input_ids: torch.LongTensor = None,\n        attention_mask: Optional[torch.Tensor] = None,\n        head_mask: Optional[torch.Tensor] = None,\n        inputs_embeds: Optional[torch.FloatTensor] = None,\n        output_attentions: Optional[bool] = None,\n        output_hidden_states: Optional[bool] = None,\n        return_dict: Optional[bool] = None,\n    ) -> Union[Tuple, BaseModelOutput]:\n        r\"\"\"\n        Args:\n            input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):\n                Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you\n                provide it.\n\n                Indices can be obtained using [`BartTokenizer`]. See [`PreTrainedTokenizer.encode`] and\n                [`PreTrainedTokenizer.__call__`] for details.\n\n                [What are input IDs?](../glossary#input-ids)\n            attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):\n                Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:\n\n                - 1 for tokens that are **not masked**,\n                - 0 for tokens that are **masked**.\n\n                [What are attention masks?](../glossary#attention-mask)\n            head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*):\n                Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`:\n\n                - 1 indicates the head is **not masked**,\n                - 0 indicates the head is **masked**.\n\n            inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*):\n                Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation.\n                This is useful if you want more control over how to convert `input_ids` indices into associated vectors\n                than the model's internal embedding lookup matrix.\n            output_attentions (`bool`, *optional*):\n                Whether or not to return the attentions tensors of all attention layers. See `attentions` under\n                returned tensors for more detail.\n            output_hidden_states (`bool`, *optional*):\n                Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors\n                for more detail.\n            return_dict (`bool`, *optional*):\n                Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.\n        \"\"\"\n        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions\n        output_hidden_states = (\n            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states\n        )\n        return_dict = return_dict if return_dict is not None else self.config.use_return_dict\n\n        # retrieve input_ids and inputs_embeds\n        if input_ids is not None and inputs_embeds is not None:\n            raise ValueError(\"You cannot specify both input_ids and inputs_embeds at the same time\")\n        elif input_ids is not None:\n            input_shape = input_ids.size()\n            input_ids = input_ids.view(-1, input_shape[-1])\n        elif inputs_embeds is not None:\n            input_shape = inputs_embeds.size()[:-1]\n        else:\n            raise ValueError(\"You have to specify either input_ids or inputs_embeds\")\n\n        if inputs_embeds is None:\n            inputs_embeds = self.embed_tokens(input_ids) * self.embed_scale\n\n        embed_pos = self.embed_positions(input_shape)\n\n        hidden_states = inputs_embeds + embed_pos\n        hidden_states = self.layernorm_embedding(hidden_states)\n        hidden_states = nn.functional.dropout(hidden_states, p=self.dropout, training=self.training)\n\n        # expand attention_mask\n        if attention_mask is not None:\n            # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]\n            attention_mask = _expand_mask(attention_mask, inputs_embeds.dtype)\n\n        encoder_states = () if output_hidden_states else None\n        all_attentions = () if output_attentions else None\n\n        # check if head_mask has a correct number of layers specified if desired\n        if head_mask is not None:\n            if head_mask.size()[0] != (len(self.layers)):\n                raise ValueError(\n                    f\"The head_mask should be specified for {len(self.layers)} layers, but it is for\"\n                    f\" {head_mask.size()[0]}.\"\n                )\n\n        for idx, encoder_layer in enumerate(self.layers):\n            if output_hidden_states:\n                encoder_states = encoder_states + (hidden_states,)\n            # add LayerDrop (see https://arxiv.org/abs/1909.11556 for description)\n            dropout_probability = random.uniform(0, 1)\n            if self.training and (dropout_probability < self.layerdrop):  # skip the layer\n                layer_outputs = (None, None)\n            else:\n                if self.gradient_checkpointing and self.training:\n\n                    def create_custom_forward(module):\n                        def custom_forward(*inputs):\n                            return module(*inputs, output_attentions)\n\n                        return custom_forward\n\n                    layer_outputs = torch.utils.checkpoint.checkpoint(\n                        create_custom_forward(encoder_layer),\n                        hidden_states,\n                        attention_mask,\n                        (head_mask[idx] if head_mask is not None else None),\n                    )\n                else:\n                    layer_outputs = encoder_layer(\n                        hidden_states,\n                        attention_mask,\n                        layer_head_mask=(head_mask[idx] if head_mask is not None else None),\n                        output_attentions=output_attentions,\n                    )\n\n                hidden_states = layer_outputs[0]\n\n            if output_attentions:\n                all_attentions = all_attentions + (layer_outputs[1],)\n\n        if output_hidden_states:\n            encoder_states = encoder_states + (hidden_states,)\n\n        if not return_dict:\n            return tuple(v for v in [hidden_states, encoder_states, all_attentions] if v is not None)\n        return BaseModelOutput(\n            last_hidden_state=hidden_states, hidden_states=encoder_states, attentions=all_attentions\n        )\n\n\nclass BartDecoder(BartPretrainedModel):\n    \"\"\"\n    Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`BartDecoderLayer`]\n\n    Args:\n        config: BartConfig\n        embed_tokens (nn.Embedding): output embedding\n    \"\"\"\n\n    def __init__(self, config: BartConfig, embed_tokens: Optional[nn.Embedding] = None):\n        super().__init__(config)\n        self.dropout = config.dropout\n        self.layerdrop = config.decoder_layerdrop\n        self.padding_idx = config.pad_token_id\n        self.max_target_positions = config.max_position_embeddings\n        self.embed_scale = math.sqrt(config.d_model) if config.scale_embedding else 1.0\n\n        if embed_tokens is not None:\n            self.embed_tokens = embed_tokens\n        else:\n            self.embed_tokens = nn.Embedding(config.vocab_size, config.d_model, self.padding_idx)\n\n        self.embed_positions = BartLearnedPositionalEmbedding(\n            config.max_position_embeddings,\n            config.d_model,\n        )\n        self.layers = nn.ModuleList([BartDecoderLayer(config) for _ in range(config.decoder_layers)])\n        self.layernorm_embedding = nn.LayerNorm(config.d_model)\n\n        self.gradient_checkpointing = False\n        # Initialize weights and apply final processing\n        self.post_init()\n\n    def get_input_embeddings(self):\n        return self.embed_tokens\n\n    def set_input_embeddings(self, value):\n        self.embed_tokens = value\n\n    def _prepare_decoder_attention_mask(self, attention_mask, input_shape, inputs_embeds, past_key_values_length):\n        # create causal mask\n        # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]\n        combined_attention_mask = None\n        if input_shape[-1] > 1:\n            combined_attention_mask = _make_causal_mask(\n                input_shape, inputs_embeds.dtype, past_key_values_length=past_key_values_length\n            ).to(inputs_embeds.device)\n\n        if attention_mask is not None:\n            # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]\n            expanded_attn_mask = _expand_mask(attention_mask, inputs_embeds.dtype, tgt_len=input_shape[-1])\n            combined_attention_mask = (\n                expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask + combined_attention_mask\n            )\n\n        return combined_attention_mask\n\n    def forward(\n        self,\n        input_ids: torch.LongTensor = None,\n        attention_mask: Optional[torch.Tensor] = None,\n        encoder_hidden_states: Optional[torch.FloatTensor] = None,\n        encoder_attention_mask: Optional[torch.LongTensor] = None,\n        head_mask: Optional[torch.Tensor] = None,\n        cross_attn_head_mask: Optional[torch.Tensor] = None,\n        past_key_values: Optional[List[torch.FloatTensor]] = None,\n        inputs_embeds: Optional[torch.FloatTensor] = None,\n        use_cache: Optional[bool] = None,\n        output_attentions: Optional[bool] = None,\n        output_hidden_states: Optional[bool] = None,\n        return_dict: Optional[bool] = None,\n    ) -> Union[Tuple, BaseModelOutputWithPastAndCrossAttentions]:\n        r\"\"\"\n        Args:\n            input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):\n                Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you\n                provide it.\n\n                Indices can be obtained using [`BartTokenizer`]. See [`PreTrainedTokenizer.encode`] and\n                [`PreTrainedTokenizer.__call__`] for details.\n\n                [What are input IDs?](../glossary#input-ids)\n            attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):\n                Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:\n\n                - 1 for tokens that are **not masked**,\n                - 0 for tokens that are **masked**.\n\n                [What are attention masks?](../glossary#attention-mask)\n            encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*):\n                Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention\n                of the decoder.\n            encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*):\n                Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values\n                selected in `[0, 1]`:\n\n                - 1 for tokens that are **not masked**,\n                - 0 for tokens that are **masked**.\n\n                [What are attention masks?](../glossary#attention-mask)\n            head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*):\n                Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`:\n\n                - 1 indicates the head is **not masked**,\n                - 0 indicates the head is **masked**.\n\n            cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*):\n                Mask to nullify selected heads of the cross-attention modules in the decoder to avoid performing\n                cross-attention on hidden heads. Mask values selected in `[0, 1]`:\n\n                - 1 indicates the head is **not masked**,\n                - 0 indicates the head is **masked**.\n\n            past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`):\n                Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of\n                shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of\n                shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`.\n\n                Contains pre-computed hidden-states (key and values in the self-attention blocks and in the\n                cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding.\n\n                If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those\n                that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of\n                all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of\n                shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing\n                `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more\n                control over how to convert `input_ids` indices into associated vectors than the model's internal\n                embedding lookup matrix.\n            output_attentions (`bool`, *optional*):\n                Whether or not to return the attentions tensors of all attention layers. See `attentions` under\n                returned tensors for more detail.\n            output_hidden_states (`bool`, *optional*):\n                Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors\n                for more detail.\n            return_dict (`bool`, *optional*):\n                Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.\n        \"\"\"\n        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions\n        output_hidden_states = (\n            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states\n        )\n        use_cache = use_cache if use_cache is not None else self.config.use_cache\n        return_dict = return_dict if return_dict is not None else self.config.use_return_dict\n\n        # retrieve input_ids and inputs_embeds\n        if input_ids is not None and inputs_embeds is not None:\n            raise ValueError(\"You cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time\")\n        elif input_ids is not None:\n            input_shape = input_ids.size()\n            input_ids = input_ids.view(-1, input_shape[-1])\n        elif inputs_embeds is not None:\n            input_shape = inputs_embeds.size()[:-1]\n        else:\n            raise ValueError(\"You have to specify either decoder_input_ids or decoder_inputs_embeds\")\n\n        # past_key_values_length\n        past_key_values_length = past_key_values[0][0].shape[2] if past_key_values is not None else 0\n\n        if inputs_embeds is None:\n            inputs_embeds = self.embed_tokens(input_ids) * self.embed_scale\n\n        attention_mask = self._prepare_decoder_attention_mask(\n            attention_mask, input_shape, inputs_embeds, past_key_values_length\n        )\n\n        # expand encoder attention mask\n        if encoder_hidden_states is not None and encoder_attention_mask is not None:\n            # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len]\n            encoder_attention_mask = _expand_mask(encoder_attention_mask, inputs_embeds.dtype, tgt_len=input_shape[-1])\n\n        # embed positions\n        positions = self.embed_positions(input_shape, past_key_values_length)\n\n        hidden_states = inputs_embeds + positions\n        hidden_states = self.layernorm_embedding(hidden_states)\n\n        hidden_states = nn.functional.dropout(hidden_states, p=self.dropout, training=self.training)\n\n        # decoder layers\n        all_hidden_states = () if output_hidden_states else None\n        all_self_attns = () if output_attentions else None\n        all_cross_attentions = () if (output_attentions and encoder_hidden_states is not None) else None\n        next_decoder_cache = () if use_cache else None\n\n        # check if head_mask/cross_attn_head_mask has a correct number of layers specified if desired\n        for attn_mask, mask_name in zip([head_mask, cross_attn_head_mask], [\"head_mask\", \"cross_attn_head_mask\"]):\n            if attn_mask is not None:\n                if attn_mask.size()[0] != (len(self.layers)):\n                    raise ValueError(\n                        f\"The `{mask_name}` should be specified for {len(self.layers)} layers, but it is for\"\n                        f\" {head_mask.size()[0]}.\"\n                    )\n\n        for idx, decoder_layer in enumerate(self.layers):\n            # add LayerDrop (see https://arxiv.org/abs/1909.11556 for description)\n            if output_hidden_states:\n                all_hidden_states += (hidden_states,)\n            dropout_probability = random.uniform(0, 1)\n            if self.training and (dropout_probability < self.layerdrop):\n                continue\n\n            past_key_value = past_key_values[idx] if past_key_values is not None else None\n\n            if self.gradient_checkpointing and self.training:\n\n                if use_cache:\n                    logger.warning(\n                        \"`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...\"\n                    )\n                    use_cache = False\n\n                def create_custom_forward(module):\n                    def custom_forward(*inputs):\n                        # None for past_key_value\n                        return module(*inputs, output_attentions, use_cache)\n\n                    return custom_forward\n\n                layer_outputs = torch.utils.checkpoint.checkpoint(\n                    create_custom_forward(decoder_layer),\n                    hidden_states,\n                    attention_mask,\n                    encoder_hidden_states,\n                    encoder_attention_mask,\n                    head_mask[idx] if head_mask is not None else None,\n                    cross_attn_head_mask[idx] if cross_attn_head_mask is not None else None,\n                    None,\n                )\n            else:\n\n                layer_outputs = decoder_layer(\n                    hidden_states,\n                    attention_mask=attention_mask,\n                    encoder_hidden_states=encoder_hidden_states,\n                    encoder_attention_mask=encoder_attention_mask,\n                    layer_head_mask=(head_mask[idx] if head_mask is not None else None),\n                    cross_attn_layer_head_mask=(\n                        cross_attn_head_mask[idx] if cross_attn_head_mask is not None else None\n                    ),\n                    past_key_value=past_key_value,\n                    output_attentions=output_attentions,\n                    use_cache=use_cache,\n                )\n            hidden_states = layer_outputs[0]\n\n            if use_cache:\n                next_decoder_cache += (layer_outputs[3 if output_attentions else 1],)\n\n            if output_attentions:\n                all_self_attns += (layer_outputs[1],)\n\n                if encoder_hidden_states is not None:\n                    all_cross_attentions += (layer_outputs[2],)\n\n        # add hidden states from the last decoder layer\n        if output_hidden_states:\n            all_hidden_states += (hidden_states,)\n\n        next_cache = next_decoder_cache if use_cache else None\n        if not return_dict:\n            return tuple(\n                v\n                for v in [hidden_states, next_cache, all_hidden_states, all_self_attns, all_cross_attentions]\n                if v is not None\n            )\n        return BaseModelOutputWithPastAndCrossAttentions(\n            last_hidden_state=hidden_states,\n            past_key_values=next_cache,\n            hidden_states=all_hidden_states,\n            attentions=all_self_attns,\n            cross_attentions=all_cross_attentions,\n        )\n\n\n@add_start_docstrings(\n    \"The bare BART Model outputting raw hidden-states without any specific head on top.\",\n    BART_START_DOCSTRING,\n)\nclass BartModel(BartPretrainedModel):\n    def __init__(self, config: BartConfig):\n        super().__init__(config)\n\n        padding_idx, vocab_size = config.pad_token_id, config.vocab_size\n        self.shared = nn.Embedding(vocab_size, config.d_model, padding_idx)\n\n        self.encoder = BartEncoder(config, self.shared)\n        self.decoder = BartDecoder(config, self.shared)\n\n        # Initialize weights and apply final processing\n        self.post_init()\n\n    def get_input_embeddings(self):\n        return self.shared\n\n    def set_input_embeddings(self, value):\n        self.shared = value\n        self.encoder.embed_tokens = self.shared\n        self.decoder.embed_tokens = self.shared\n\n    def get_encoder(self):\n        return self.encoder\n\n    def get_decoder(self):\n        return self.decoder\n\n    @add_start_docstrings_to_model_forward(BART_INPUTS_DOCSTRING)\n    @add_code_sample_docstrings(\n        processor_class=_TOKENIZER_FOR_DOC,\n        checkpoint=_CHECKPOINT_FOR_DOC,\n        output_type=Seq2SeqModelOutput,\n        config_class=_CONFIG_FOR_DOC,\n        expected_output=_EXPECTED_OUTPUT_SHAPE,\n    )\n    def forward(\n        self,\n        input_ids: torch.LongTensor = None,\n        attention_mask: Optional[torch.Tensor] = None,\n        decoder_input_ids: Optional[torch.LongTensor] = None,\n        decoder_attention_mask: Optional[torch.LongTensor] = None,\n        head_mask: Optional[torch.Tensor] = None,\n        decoder_head_mask: Optional[torch.Tensor] = None,\n        cross_attn_head_mask: Optional[torch.Tensor] = None,\n        encoder_outputs: Optional[List[torch.FloatTensor]] = None,\n        past_key_values: Optional[List[torch.FloatTensor]] = None,\n        inputs_embeds: Optional[torch.FloatTensor] = None,\n        decoder_inputs_embeds: Optional[torch.FloatTensor] = None,\n        use_cache: Optional[bool] = None,\n        output_attentions: Optional[bool] = None,\n        output_hidden_states: Optional[bool] = None,\n        return_dict: Optional[bool] = None,\n    ) -> Union[Tuple, Seq2SeqModelOutput]:\n\n        # different to other models, Bart automatically creates decoder_input_ids from\n        # input_ids if no decoder_input_ids are provided\n        if decoder_input_ids is None and decoder_inputs_embeds is None:\n            if input_ids is None:\n                raise ValueError(\n                    \"If no `decoder_input_ids` or `decoder_inputs_embeds` are \"\n                    \"passed, `input_ids` cannot be `None`. Please pass either \"\n                    \"`input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.\"\n                )\n\n            decoder_input_ids = shift_tokens_right(\n                input_ids, self.config.pad_token_id, self.config.decoder_start_token_id\n            )\n\n        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions\n        output_hidden_states = (\n            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states\n        )\n        use_cache = use_cache if use_cache is not None else self.config.use_cache\n        return_dict = return_dict if return_dict is not None else self.config.use_return_dict\n\n        if encoder_outputs is None:\n            encoder_outputs = self.encoder(\n                input_ids=input_ids,\n                attention_mask=attention_mask,\n                head_mask=head_mask,\n                inputs_embeds=inputs_embeds,\n                output_attentions=output_attentions,\n                output_hidden_states=output_hidden_states,\n                return_dict=return_dict,\n            )\n        # If the user passed a tuple for encoder_outputs, we wrap it in a BaseModelOutput when return_dict=True\n        elif return_dict and not isinstance(encoder_outputs, BaseModelOutput):\n            encoder_outputs = BaseModelOutput(\n                last_hidden_state=encoder_outputs[0],\n                hidden_states=encoder_outputs[1] if len(encoder_outputs) > 1 else None,\n                attentions=encoder_outputs[2] if len(encoder_outputs) > 2 else None,\n            )\n\n        # decoder outputs consists of (dec_features, past_key_value, dec_hidden, dec_attn)\n        decoder_outputs = self.decoder(\n            input_ids=decoder_input_ids,\n            attention_mask=decoder_attention_mask,\n            encoder_hidden_states=encoder_outputs[0],\n            encoder_attention_mask=attention_mask,\n            head_mask=decoder_head_mask,\n            cross_attn_head_mask=cross_attn_head_mask,\n            past_key_values=past_key_values,\n            inputs_embeds=decoder_inputs_embeds,\n            use_cache=use_cache,\n            output_attentions=output_attentions,\n            output_hidden_states=output_hidden_states,\n            return_dict=return_dict,\n        )\n\n        if not return_dict:\n            return decoder_outputs + encoder_outputs\n\n        return Seq2SeqModelOutput(\n            last_hidden_state=decoder_outputs.last_hidden_state,\n            past_key_values=decoder_outputs.past_key_values,\n            decoder_hidden_states=decoder_outputs.hidden_states,\n            decoder_attentions=decoder_outputs.attentions,\n            cross_attentions=decoder_outputs.cross_attentions,\n            encoder_last_hidden_state=encoder_outputs.last_hidden_state,\n            encoder_hidden_states=encoder_outputs.hidden_states,\n            encoder_attentions=encoder_outputs.attentions,\n        )\n\n\n@add_start_docstrings(\n    \"The BART Model with a language modeling head. Can be used for summarization.\", BART_START_DOCSTRING\n)\nclass BartForConditionalGeneration(BartPretrainedModel):\n    base_model_prefix = \"model\"\n    _keys_to_ignore_on_load_missing = [r\"final_logits_bias\", r\"lm_head.weight\"]\n\n    def __init__(self, config: BartConfig):\n        super().__init__(config)\n        self.model = BartModel(config)\n        self.register_buffer(\"final_logits_bias\", torch.zeros((1, self.model.shared.num_embeddings)))\n        self.lm_head = nn.Linear(config.d_model, self.model.shared.num_embeddings, bias=False)\n\n        # Initialize weights and apply final processing\n        self.post_init()\n\n    def get_encoder(self):\n        return self.model.get_encoder()\n\n    def get_decoder(self):\n        return self.model.get_decoder()\n\n    def resize_token_embeddings(self, new_num_tokens: int) -> nn.Embedding:\n        new_embeddings = super().resize_token_embeddings(new_num_tokens)\n        self._resize_final_logits_bias(new_num_tokens)\n        return new_embeddings\n\n    def _resize_final_logits_bias(self, new_num_tokens: int) -> None:\n        old_num_tokens = self.final_logits_bias.shape[-1]\n        if new_num_tokens <= old_num_tokens:\n            new_bias = self.final_logits_bias[:, :new_num_tokens]\n        else:\n            extra_bias = torch.zeros((1, new_num_tokens - old_num_tokens), device=self.final_logits_bias.device)\n            new_bias = torch.cat([self.final_logits_bias, extra_bias], dim=1)\n        self.register_buffer(\"final_logits_bias\", new_bias)\n\n    def get_output_embeddings(self):\n        return self.lm_head\n\n    def set_output_embeddings(self, new_embeddings):\n        self.lm_head = new_embeddings\n\n    @add_start_docstrings_to_model_forward(BART_INPUTS_DOCSTRING)\n    @replace_return_docstrings(output_type=Seq2SeqLMOutput, config_class=_CONFIG_FOR_DOC)\n    @add_end_docstrings(BART_GENERATION_EXAMPLE)\n    def forward(\n        self,\n        input_ids: torch.LongTensor = None,\n        attention_mask: Optional[torch.Tensor] = None,\n        decoder_input_ids: Optional[torch.LongTensor] = None,\n        decoder_attention_mask: Optional[torch.LongTensor] = None,\n        head_mask: Optional[torch.Tensor] = None,\n        decoder_head_mask: Optional[torch.Tensor] = None,\n        cross_attn_head_mask: Optional[torch.Tensor] = None,\n        encoder_outputs: Optional[List[torch.FloatTensor]] = None,\n        past_key_values: Optional[List[torch.FloatTensor]] = None,\n        inputs_embeds: Optional[torch.FloatTensor] = None,\n        decoder_inputs_embeds: Optional[torch.FloatTensor] = None,\n        labels: Optional[torch.LongTensor] = None,\n        use_cache: Optional[bool] = None,\n        output_attentions: Optional[bool] = None,\n        output_hidden_states: Optional[bool] = None,\n        return_dict: Optional[bool] = None,\n    ) -> Union[Tuple, Seq2SeqLMOutput]:\n        r\"\"\"\n        labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):\n            Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,\n            config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored\n            (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.\n\n        Returns:\n        \"\"\"\n        return_dict = return_dict if return_dict is not None else self.config.use_return_dict\n\n        if labels is not None:\n            if use_cache:\n                logger.warning(\"The `use_cache` argument is changed to `False` since `labels` is provided.\")\n            use_cache = False\n            if decoder_input_ids is None and decoder_inputs_embeds is None:\n                decoder_input_ids = shift_tokens_right(\n                    labels, self.config.pad_token_id, self.config.decoder_start_token_id\n                )\n\n        outputs = self.model(\n            input_ids,\n            attention_mask=attention_mask,\n            decoder_input_ids=decoder_input_ids,\n            encoder_outputs=encoder_outputs,\n            decoder_attention_mask=decoder_attention_mask,\n            head_mask=head_mask,\n            decoder_head_mask=decoder_head_mask,\n            cross_attn_head_mask=cross_attn_head_mask,\n            past_key_values=past_key_values,\n            inputs_embeds=inputs_embeds,\n            decoder_inputs_embeds=decoder_inputs_embeds,\n            use_cache=use_cache,\n            output_attentions=output_attentions,\n            output_hidden_states=output_hidden_states,\n            return_dict=return_dict,\n        )\n        lm_logits = self.lm_head(outputs[0]) + self.final_logits_bias\n\n        masked_lm_loss = None\n        if labels is not None:\n            loss_fct = CrossEntropyLoss()\n            masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1))\n\n        if not return_dict:\n            output = (lm_logits,) + outputs[1:]\n            return ((masked_lm_loss,) + output) if masked_lm_loss is not None else output\n\n        return Seq2SeqLMOutput(\n            loss=masked_lm_loss,\n            logits=lm_logits,\n            past_key_values=outputs.past_key_values,\n            decoder_hidden_states=outputs.decoder_hidden_states,\n            decoder_attentions=outputs.decoder_attentions,\n            cross_attentions=outputs.cross_attentions,\n            encoder_last_hidden_state=outputs.encoder_last_hidden_state,\n            encoder_hidden_states=outputs.encoder_hidden_states,\n            encoder_attentions=outputs.encoder_attentions,\n        )\n\n    def prepare_inputs_for_generation(\n        self,\n        decoder_input_ids,\n        past=None,\n        attention_mask=None,\n        head_mask=None,\n        decoder_head_mask=None,\n        cross_attn_head_mask=None,\n        use_cache=None,\n        encoder_outputs=None,\n        **kwargs\n    ):\n        # cut decoder_input_ids if past is used\n        if past is not None:\n            decoder_input_ids = decoder_input_ids[:, -1:]\n\n        return {\n            \"input_ids\": None,  # encoder_outputs is defined. input_ids not needed\n            \"encoder_outputs\": encoder_outputs,\n            \"past_key_values\": past,\n            \"decoder_input_ids\": decoder_input_ids,\n            \"attention_mask\": attention_mask,\n            \"head_mask\": head_mask,\n            \"decoder_head_mask\": decoder_head_mask,\n            \"cross_attn_head_mask\": cross_attn_head_mask,\n            \"use_cache\": use_cache,  # change this to avoid caching (presumably for debugging)\n        }\n\n    def prepare_decoder_input_ids_from_labels(self, labels: torch.Tensor):\n        return shift_tokens_right(labels, self.config.pad_token_id, self.config.decoder_start_token_id)\n\n    @staticmethod\n    def _reorder_cache(past, beam_idx):\n        reordered_past = ()\n        for layer_past in past:\n            # cached cross_attention states don't have to be reordered -> they are always the same\n            reordered_past += (\n                tuple(past_state.index_select(0, beam_idx) for past_state in layer_past[:2]) + layer_past[2:],\n            )\n        return reordered_past\n\n\n@add_start_docstrings(\n    \"\"\"\n    Bart model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE\n    tasks.\n    \"\"\",\n    BART_START_DOCSTRING,\n)\nclass BartForSequenceClassification(BartPretrainedModel):\n    def __init__(self, config: BartConfig, **kwargs):\n        super().__init__(config, **kwargs)\n        self.model = BartModel(config)\n        self.classification_head = BartClassificationHead(\n            config.d_model,\n            config.d_model,\n            config.num_labels,\n            config.classifier_dropout,\n        )\n        self.model._init_weights(self.classification_head.dense)\n        self.model._init_weights(self.classification_head.out_proj)\n\n    @add_start_docstrings_to_model_forward(BART_INPUTS_DOCSTRING)\n    @add_code_sample_docstrings(\n        processor_class=_TOKENIZER_FOR_DOC,\n        checkpoint=_CHECKPOINT_FOR_SEQUENCE_CLASSIFICATION,\n        output_type=Seq2SeqSequenceClassifierOutput,\n        config_class=_CONFIG_FOR_DOC,\n        expected_output=_SEQ_CLASS_EXPECTED_OUTPUT,\n        expected_loss=_SEQ_CLASS_EXPECTED_LOSS,\n    )\n    def forward(\n        self,\n        input_ids: torch.LongTensor = None,\n        attention_mask: Optional[torch.Tensor] = None,\n        decoder_input_ids: Optional[torch.LongTensor] = None,\n        decoder_attention_mask: Optional[torch.LongTensor] = None,\n        head_mask: Optional[torch.Tensor] = None,\n        decoder_head_mask: Optional[torch.Tensor] = None,\n        cross_attn_head_mask: Optional[torch.Tensor] = None,\n        encoder_outputs: Optional[List[torch.FloatTensor]] = None,\n        inputs_embeds: Optional[torch.FloatTensor] = None,\n        decoder_inputs_embeds: Optional[torch.FloatTensor] = None,\n        labels: Optional[torch.LongTensor] = None,\n        use_cache: Optional[bool] = None,\n        output_attentions: Optional[bool] = None,\n        output_hidden_states: Optional[bool] = None,\n        return_dict: Optional[bool] = None,\n    ) -> Union[Tuple, Seq2SeqSequenceClassifierOutput]:\n        r\"\"\"\n        labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*):\n            Labels for computing the sequence classification/regression loss. Indices should be in `[0, ...,\n            config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy).\n        \"\"\"\n        return_dict = return_dict if return_dict is not None else self.config.use_return_dict\n        if labels is not None:\n            use_cache = False\n\n        if input_ids is None and inputs_embeds is not None:\n            raise NotImplementedError(\n                f\"Passing input embeddings is currently not supported for {self.__class__.__name__}\"\n            )\n\n        outputs = self.model(\n            input_ids,\n            attention_mask=attention_mask,\n            decoder_input_ids=decoder_input_ids,\n            decoder_attention_mask=decoder_attention_mask,\n            head_mask=head_mask,\n            decoder_head_mask=decoder_head_mask,\n            cross_attn_head_mask=cross_attn_head_mask,\n            encoder_outputs=encoder_outputs,\n            inputs_embeds=inputs_embeds,\n            decoder_inputs_embeds=decoder_inputs_embeds,\n            use_cache=use_cache,\n            output_attentions=output_attentions,\n            output_hidden_states=output_hidden_states,\n            return_dict=return_dict,\n        )\n        hidden_states = outputs[0]  # last hidden state\n\n        eos_mask = input_ids.eq(self.config.eos_token_id)\n\n        if len(torch.unique_consecutive(eos_mask.sum(1))) > 1:\n            raise ValueError(\"All examples must have the same number of <eos> tokens.\")\n        sentence_representation = hidden_states[eos_mask, :].view(hidden_states.size(0), -1, hidden_states.size(-1))[\n            :, -1, :\n        ]\n        logits = self.classification_head(sentence_representation)\n\n        loss = None\n        if labels is not None:\n            if self.config.problem_type is None:\n                if self.config.num_labels == 1:\n                    self.config.problem_type = \"regression\"\n                elif self.config.num_labels > 1 and (labels.dtype == torch.long or labels.dtype == torch.int):\n                    self.config.problem_type = \"single_label_classification\"\n                else:\n                    self.config.problem_type = \"multi_label_classification\"\n\n            if self.config.problem_type == \"regression\":\n                loss_fct = MSELoss()\n                if self.config.num_labels == 1:\n                    loss = loss_fct(logits.squeeze(), labels.squeeze())\n                else:\n                    loss = loss_fct(logits, labels)\n            elif self.config.problem_type == \"single_label_classification\":\n                loss_fct = CrossEntropyLoss()\n                loss = loss_fct(logits.view(-1, self.config.num_labels), labels.view(-1))\n            elif self.config.problem_type == \"multi_label_classification\":\n                loss_fct = BCEWithLogitsLoss()\n                loss = loss_fct(logits, labels)\n        if not return_dict:\n            output = (logits,) + outputs[1:]\n            return ((loss,) + output) if loss is not None else output\n\n        return Seq2SeqSequenceClassifierOutput(\n            loss=loss,\n            logits=logits,\n            past_key_values=outputs.past_key_values,\n            decoder_hidden_states=outputs.decoder_hidden_states,\n            decoder_attentions=outputs.decoder_attentions,\n            cross_attentions=outputs.cross_attentions,\n            encoder_last_hidden_state=outputs.encoder_last_hidden_state,\n            encoder_hidden_states=outputs.encoder_hidden_states,\n            encoder_attentions=outputs.encoder_attentions,\n        )\n\n\n@add_start_docstrings(\n    \"\"\"\n    BART Model with a span classification head on top for extractive question-answering tasks like SQuAD (a linear\n    layer on top of the hidden-states output to compute `span start logits` and `span end logits`).\n    \"\"\",\n    BART_START_DOCSTRING,\n)\nclass BartForQuestionAnswering(BartPretrainedModel):\n    def __init__(self, config):\n        super().__init__(config)\n\n        config.num_labels = 2\n        self.num_labels = config.num_labels\n\n        self.model = BartModel(config)\n        self.qa_outputs = nn.Linear(config.hidden_size, config.num_labels)\n\n        self.model._init_weights(self.qa_outputs)\n\n    @add_start_docstrings_to_model_forward(BART_INPUTS_DOCSTRING)\n    @add_code_sample_docstrings(\n        processor_class=_TOKENIZER_FOR_DOC,\n        checkpoint=_CHECKPOINT_FOR_QA,\n        output_type=Seq2SeqQuestionAnsweringModelOutput,\n        config_class=_CONFIG_FOR_DOC,\n        expected_loss=_QA_EXPECTED_LOSS,\n        expected_output=_QA_EXPECTED_OUTPUT,\n    )\n    def forward(\n        self,\n        input_ids: torch.Tensor = None,\n        attention_mask: Optional[torch.Tensor] = None,\n        decoder_input_ids: Optional[torch.LongTensor] = None,\n        decoder_attention_mask: Optional[torch.LongTensor] = None,\n        head_mask: Optional[torch.Tensor] = None,\n        decoder_head_mask: Optional[torch.Tensor] = None,\n        cross_attn_head_mask: Optional[torch.Tensor] = None,\n        encoder_outputs: Optional[List[torch.FloatTensor]] = None,\n        start_positions: Optional[torch.LongTensor] = None,\n        end_positions: Optional[torch.LongTensor] = None,\n        inputs_embeds: Optional[torch.FloatTensor] = None,\n        decoder_inputs_embeds: Optional[torch.FloatTensor] = None,\n        use_cache: Optional[bool] = None,\n        output_attentions: Optional[bool] = None,\n        output_hidden_states: Optional[bool] = None,\n        return_dict: Optional[bool] = None,\n    ) -> Union[Tuple, Seq2SeqQuestionAnsweringModelOutput]:\n        r\"\"\"\n        start_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*):\n            Labels for position (index) of the start of the labelled span for computing the token classification loss.\n            Positions are clamped to the length of the sequence (*sequence_length*). Position outside of the sequence\n            are not taken into account for computing the loss.\n        end_positions (`torch.LongTensor` of shape `(batch_size,)`, *optional*):\n            Labels for position (index) of the end of the labelled span for computing the token classification loss.\n            Positions are clamped to the length of the sequence (*sequence_length*). Position outside of the sequence\n            are not taken into account for computing the loss.\n        \"\"\"\n        return_dict = return_dict if return_dict is not None else self.config.use_return_dict\n        if start_positions is not None and end_positions is not None:\n            use_cache = False\n\n        outputs = self.model(\n            input_ids,\n            attention_mask=attention_mask,\n            decoder_input_ids=decoder_input_ids,\n            decoder_attention_mask=decoder_attention_mask,\n            head_mask=head_mask,\n            decoder_head_mask=decoder_head_mask,\n            cross_attn_head_mask=cross_attn_head_mask,\n            encoder_outputs=encoder_outputs,\n            inputs_embeds=inputs_embeds,\n            decoder_inputs_embeds=decoder_inputs_embeds,\n            use_cache=use_cache,\n            output_attentions=output_attentions,\n            output_hidden_states=output_hidden_states,\n            return_dict=return_dict,\n        )\n\n        sequence_output = outputs[0]\n\n        logits = self.qa_outputs(sequence_output)\n        start_logits, end_logits = logits.split(1, dim=-1)\n        start_logits = start_logits.squeeze(-1).contiguous()\n        end_logits = end_logits.squeeze(-1).contiguous()\n\n        total_loss = None\n        if start_positions is not None and end_positions is not None:\n            # If we are on multi-GPU, split add a dimension\n            if len(start_positions.size()) > 1:\n                start_positions = start_positions.squeeze(-1)\n            if len(end_positions.size()) > 1:\n                end_positions = end_positions.squeeze(-1)\n            # sometimes the start/end positions are outside our model inputs, we ignore these terms\n            ignored_index = start_logits.size(1)\n            start_positions = start_positions.clamp(0, ignored_index)\n            end_positions = end_positions.clamp(0, ignored_index)\n\n            loss_fct = CrossEntropyLoss(ignore_index=ignored_index)\n            start_loss = loss_fct(start_logits, start_positions)\n            end_loss = loss_fct(end_logits, end_positions)\n            total_loss = (start_loss + end_loss) / 2\n\n        if not return_dict:\n            output = (\n                start_logits,\n                end_logits,\n            ) + outputs[1:]\n            return ((total_loss,) + output) if total_loss is not None else output\n\n        return Seq2SeqQuestionAnsweringModelOutput(\n            loss=total_loss,\n            start_logits=start_logits,\n            end_logits=end_logits,\n            past_key_values=outputs.past_key_values,\n            decoder_hidden_states=outputs.decoder_hidden_states,\n            decoder_attentions=outputs.decoder_attentions,\n            cross_attentions=outputs.cross_attentions,\n            encoder_last_hidden_state=outputs.encoder_last_hidden_state,\n            encoder_hidden_states=outputs.encoder_hidden_states,\n            encoder_attentions=outputs.encoder_attentions,\n        )\n\n\nclass BartDecoderWrapper(BartPretrainedModel):\n    \"\"\"\n    This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is\n    used in combination with the [`EncoderDecoderModel`] framework.\n    \"\"\"\n\n    def __init__(self, config):\n        super().__init__(config)\n        self.decoder = BartDecoder(config)\n\n    def forward(self, *args, **kwargs):\n        return self.decoder(*args, **kwargs)\n\n\nclass BartForCausalLM(BartPretrainedModel):\n    def __init__(self, config):\n        config = copy.deepcopy(config)\n        config.is_decoder = True\n        config.is_encoder_decoder = False\n        super().__init__(config)\n        self.model = BartDecoderWrapper(config)\n\n        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)\n\n        # Initialize weights and apply final processing\n        self.post_init()\n\n    def get_input_embeddings(self):\n        return self.model.decoder.embed_tokens\n\n    def set_input_embeddings(self, value):\n        self.model.decoder.embed_tokens = value\n\n    def get_output_embeddings(self):\n        return self.lm_head\n\n    def set_output_embeddings(self, new_embeddings):\n        self.lm_head = new_embeddings\n\n    def set_decoder(self, decoder):\n        self.model.decoder = decoder\n\n    def get_decoder(self):\n        return self.model.decoder\n\n    @replace_return_docstrings(output_type=CausalLMOutputWithCrossAttentions, config_class=_CONFIG_FOR_DOC)\n    def forward(\n        self,\n        input_ids: torch.LongTensor = None,\n        attention_mask: Optional[torch.Tensor] = None,\n        encoder_hidden_states: Optional[torch.FloatTensor] = None,\n        encoder_attention_mask: Optional[torch.FloatTensor] = None,\n        head_mask: Optional[torch.Tensor] = None,\n        cross_attn_head_mask: Optional[torch.Tensor] = None,\n        past_key_values: Optional[List[torch.FloatTensor]] = None,\n        inputs_embeds: Optional[torch.FloatTensor] = None,\n        labels: Optional[torch.LongTensor] = None,\n        use_cache: Optional[bool] = None,\n        output_attentions: Optional[bool] = None,\n        output_hidden_states: Optional[bool] = None,\n        return_dict: Optional[bool] = None,\n    ) -> Union[Tuple, CausalLMOutputWithCrossAttentions]:\n        r\"\"\"\n        Args:\n            input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`):\n                Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you\n                provide it.\n\n                Indices can be obtained using [`BartTokenizer`]. See [`PreTrainedTokenizer.encode`] and\n                [`PreTrainedTokenizer.__call__`] for details.\n\n                [What are input IDs?](../glossary#input-ids)\n            attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):\n                Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`:\n\n                - 1 for tokens that are **not masked**,\n                - 0 for tokens that are **masked**.\n\n                [What are attention masks?](../glossary#attention-mask)\n            encoder_hidden_states  (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*):\n                Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention\n                if the model is configured as a decoder.\n            encoder_attention_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, *optional*):\n                Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used\n                in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`:\n            head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*):\n                Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`:\n\n                - 1 indicates the head is **not masked**,\n                - 0 indicates the head is **masked**.\n\n            cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*):\n                Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`:\n\n                - 1 indicates the head is **not masked**,\n                - 0 indicates the head is **masked**.\n\n            past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`):\n                Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of\n                shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of\n                shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. The two additional\n                tensors are only required when the model is used as a decoder in a Sequence to Sequence model.\n\n                Contains pre-computed hidden-states (key and values in the self-attention blocks and in the\n                cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding.\n\n                If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those\n                that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of\n                all `decoder_input_ids` of shape `(batch_size, sequence_length)`.\n            labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):\n                Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,\n                config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored\n                (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.\n            use_cache (`bool`, *optional*):\n                If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding\n                (see `past_key_values`).\n\n                - 1 for tokens that are **not masked**,\n                - 0 for tokens that are **masked**.\n            output_attentions (`bool`, *optional*):\n                Whether or not to return the attentions tensors of all attention layers. See `attentions` under\n                returned tensors for more detail.\n            output_hidden_states (`bool`, *optional*):\n                Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors\n                for more detail.\n            return_dict (`bool`, *optional*):\n                Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.\n\n        Returns:\n\n        Example:\n\n        ```python\n        >>> from transformers import BartTokenizer, BartForCausalLM\n\n        >>> tokenizer = BartTokenizer.from_pretrained(\"facebook/bart-base\")\n        >>> model = BartForCausalLM.from_pretrained(\"facebook/bart-base\", add_cross_attention=False)\n        >>> assert model.config.is_decoder, f\"{model.__class__} has to be configured as a decoder.\"\n        >>> inputs = tokenizer(\"Hello, my dog is cute\", return_tensors=\"pt\")\n        >>> outputs = model(**inputs)\n\n        >>> logits = outputs.logits\n        >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size]\n        >>> list(logits.shape) == expected_shape\n        True\n        ```\"\"\"\n\n        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions\n        output_hidden_states = (\n            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states\n        )\n        return_dict = return_dict if return_dict is not None else self.config.use_return_dict\n\n        # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)\n        outputs = self.model.decoder(\n            input_ids=input_ids,\n            attention_mask=attention_mask,\n            encoder_hidden_states=encoder_hidden_states,\n            encoder_attention_mask=encoder_attention_mask,\n            head_mask=head_mask,\n            cross_attn_head_mask=cross_attn_head_mask,\n            past_key_values=past_key_values,\n            inputs_embeds=inputs_embeds,\n            use_cache=use_cache,\n            output_attentions=output_attentions,\n            output_hidden_states=output_hidden_states,\n            return_dict=return_dict,\n        )\n\n        logits = self.lm_head(outputs[0])\n\n        loss = None\n        if labels is not None:\n            loss_fct = CrossEntropyLoss()\n            loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1))\n\n        if not return_dict:\n            output = (logits,) + outputs[1:]\n            return (loss,) + output if loss is not None else output\n\n        return CausalLMOutputWithCrossAttentions(\n            loss=loss,\n            logits=logits,\n            past_key_values=outputs.past_key_values,\n            hidden_states=outputs.hidden_states,\n            attentions=outputs.attentions,\n            cross_attentions=outputs.cross_attentions,\n        )\n\n    def prepare_inputs_for_generation(self, input_ids, past=None, attention_mask=None, use_cache=None, **kwargs):\n        # if model is used as a decoder in encoder-decoder model, the decoder attention mask is created on the fly\n        if attention_mask is None:\n            attention_mask = input_ids.new_ones(input_ids.shape)\n\n        if past:\n            input_ids = input_ids[:, -1:]\n        # first step, decoder_cached_states are empty\n        return {\n            \"input_ids\": input_ids,  # encoder_outputs is defined. input_ids not needed\n            \"attention_mask\": attention_mask,\n            \"past_key_values\": past,\n            \"use_cache\": use_cache,\n        }\n\n    @staticmethod\n    def _reorder_cache(past, beam_idx):\n        reordered_past = ()\n        for layer_past in past:\n            reordered_past += (tuple(past_state.index_select(0, beam_idx) for past_state in layer_past),)\n        return reordered_past"
  },
  {
    "path": "hanlp/components/amr/amrbart/model_interface/tokenization_bart.py",
    "content": "# coding:utf-8\n# this is a simplified version of \"https://github.com/SapienzaNLP/spring/blob/main/spring_amr/tokenization_bart.py\"\n# MIT License\n#\n# Copyright (c) 2022 xfbai\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\nimport penman\nimport regex as re\nfrom transformers import BartTokenizer\n\nfrom hanlp.components.amr.amrbart.common import postprocessing\nfrom hanlp.components.amr.amrbart.common.constant import raw_special_tokens, recategorizations\nfrom hanlp.components.amr.amrbart.common.penman_interface import encode\n\n\nclass AMRBartTokenizer(BartTokenizer):\n    INIT = 'Ġ'\n\n    def __init__(self, vocab_file, merges_file, errors=\"replace\", bos_token=\"<s>\", eos_token=\"</s>\", sep_token=\"</s>\", cls_token=\"<s>\", unk_token=\"<unk>\", pad_token=\"<pad>\", mask_token=\"<mask>\", add_prefix_space=False, **kwargs):\n        super().__init__(vocab_file, merges_file, errors, bos_token, eos_token, sep_token, cls_token, unk_token, pad_token, mask_token, add_prefix_space, **kwargs)\n        self.modified = 0\n        self.recategorizations = set(recategorizations)\n        self.patterns = re.compile(r\"\"\" ?<[a-z]+:?\\d*>| ?:[^\\s]+|'s|'t|'re|'ve|'m|'ll|'d| ?\\p{L}+| ?\\p{N}+| ?[^\\s\\p{L}\\p{N}]+|\\s+(?!\\S)|\\s+\"\"\")\n        self.remove_pars = False\n\n    @classmethod\n    def from_pretrained(cls, pretrained_model_path, *args, **kwargs):\n        inst = super().from_pretrained(pretrained_model_path, *args, **kwargs)\n        inst.init_amr_vocabulary()\n        return inst\n\n    def init_amr_vocabulary(self):\n        self.old_enc_size = old_enc_size = len(self.encoder)\n        tokens = [t for t in raw_special_tokens if t not in self.encoder]\n\n        for i, t in enumerate(tokens, start=old_enc_size):\n            self.encoder[t] = i\n\n        self.encoder = {k: i for i, (k,v) in enumerate(sorted(self.encoder.items(), key=lambda x: x[1]))}\n        self.decoder = {v: k for k, v in sorted(self.encoder.items(), key=lambda x: x[1])}\n        self.modified = len(tokens)\n\n        self.amr_bos_token = \"<AMR>\"\n        self.amr_bos_token_id = self.encoder[self.amr_bos_token]\n        self.amr_eos_token = \"</AMR>\"\n        self.amr_eos_token_id = self.encoder[self.amr_eos_token]\n        # print(f\"Added {self.modified} AMR tokens\")\n\n    def _tokenize(self, text):\n        \"\"\" Tokenize a string. Modified in order to handle sentences with recategorization pointers\"\"\"\n        bpe_tokens = []\n        for tok_span in text.lstrip().split(' '):\n            tok_span = tok_span.strip()\n            recats = tok_span.rsplit('_', 1)\n            if len(recats) == 2 and recats[0] in self.recategorizations and ('_' + recats[1]) in self.encoder:\n                bpe_tokens.extend([self.INIT + recats[0], '_' + recats[1]])\n            else:\n                for token in re.findall(self.pat, ' ' + tok_span):\n                    token = \"\".join(\n                        self.byte_encoder[b] for b in token.encode(\"utf-8\")\n                    )   # Maps all our bytes to unicode strings, avoiding controle tokens of the BPE (spaces in our case)\n                    bpe_tokens.extend(bpe_token for bpe_token in self.bpe(token).split(\" \"))\n\n        return bpe_tokens\n\n    def _tok_bpe(self, token):\n        tokk = []\n        tok = token.strip()\n        recats = tok.rsplit('_', 1)\n        if len(recats) == 2 and recats[0] in self.recategorizations and ('_' + recats[1]) in self.encoder:\n            tokk.extend([self.INIT + recats[0], '_' + recats[1]])\n        else:\n            for tok in self.patterns.findall(' ' + token):\n                tok = \"\".join(\n                    self.byte_encoder[b] for b in tok.encode(\"utf-8\"))\n                toks = self.bpe(tok).split(' ')\n                tokk.extend(toks)\n        return tokk\n\n    def tokenize_amr(self, amr_tokens):\n        bpe_tokens = []\n        for i, tokk in enumerate(amr_tokens):\n            is_in_enc = self.INIT + tokk in self.encoder\n            is_rel = tokk.startswith(':') and len(tokk) > 1\n            is_spc = tokk.startswith('<') and tokk.endswith('>')\n            is_of = tokk.startswith(':') and tokk.endswith('-of')\n            is_frame = re.match(r'.+-\\d\\d', tokk) is not None\n\n            if tokk.startswith('\"') and tokk.endswith('\"'):                 # dealing with examples like \"The_United_Kingdom_of_xxx\"\n                tokk = tokk[1:-1].replace('_', ' ')\n                bpe_toks = [self.INIT + \"<lit>\"]\n                bpe_toks += self._tok_bpe(tokk)\n                bpe_toks.append(self.INIT + \"</lit>\")\n\n            elif (is_rel or is_spc or is_frame or is_of):\n                if is_in_enc:\n                    bpe_toks = [self.INIT + tokk]\n                elif is_frame:\n                    bpe_toks = self._tok_bpe(tokk[:-3]) + [tokk[-3:]]\n                elif is_of:\n                    rel = tokk[:-3]\n                    if self.INIT + rel in self.encoder:\n                        bpe_toks = [self.INIT + rel, '-of']\n                    else:\n                        bpe_toks = [self.INIT + ':'] + self._tok_bpe(rel[1:]) + ['-of']\n                elif is_rel:\n                    bpe_toks = [self.INIT + ':'] + self._tok_bpe(tokk[1:])\n                else:\n                    print(\"tok:\", tokk)\n                    print(f\"is_rel:{is_rel}, is_spc:{is_spc}, is_frame:{is_frame}, is_of:{is_of}\")\n                    exit()\n                    raise\n            else:\n                if is_in_enc:\n                    bpe_toks = [self.INIT + tokk]\n                else:\n                    bpe_toks = self._tok_bpe(tokk)\n\n            bpe_tokens.append(bpe_toks)\n        bpe_tokens = [b for bb in bpe_tokens for b in bb]\n        bpe_token_ids = [self.encoder.get(b, self.unk_token_id) for b in bpe_tokens]\n        return bpe_token_ids\n\n    def decode_amr(self, tokens, restore_name_ops=None):\n        try:\n            nodes, backreferences = postprocessing.decode_into_node_and_backreferences(tokens, self)\n        except Exception as e:\n            # print('Decoding failure:', file=sys.stderr)\n            # print(e, file=sys.stderr)\n            return postprocessing.BACKOFF, postprocessing.ParsedStatus.BACKOFF, (None, None)\n        try:\n            graph_ = graph = self._fix_and_make_graph(nodes)\n            # if collapse_name_ops:\n            #     graph_ = graph = postprocessing._split_name_ops(graph)\n        except Exception as e:\n            # print('Building failure:', file=sys.stderr)\n            # print(nodes, file=sys.stderr)\n            # print(backreferences, file=sys.stderr)\n            # print(e, file=sys.stderr)\n            return postprocessing.BACKOFF, postprocessing.ParsedStatus.BACKOFF, (None, None)\n        try:\n            graph, status = postprocessing.connect_graph_if_not_connected(graph)\n            # if status == postprocessing.ParsedStatus.BACKOFF:\n            #     print('Reconnection 1 failure:')\n            #     print(nodes, file=sys.stderr)\n            #     print(backreferences, file=sys.stderr)\n            #     print(graph_, file=sys.stderr)\n            return graph, status, (nodes, backreferences)\n        except Exception as e:\n            # print('Reconnction 2 failure:', file=sys.stderr)\n            # print(e, file=sys.stderr)\n            # print(nodes, file=sys.stderr)\n            # print(backreferences, file=sys.stderr)\n            # print(graph_, file=sys.stderr)\n            return postprocessing.BACKOFF, postprocessing.ParsedStatus.BACKOFF, (nodes, backreferences)\n\n    def _fix_and_make_graph(self, nodes):\n\n        nodes_ = []\n        for n in nodes:\n            if isinstance(n, str):\n                if n.startswith('<') and n.endswith('>') and (not n.startswith('<pointer:')):\n                    pass\n                else:\n                    nodes_.append(n)\n            else:\n                nodes_.append(n)\n        nodes = nodes_\n\n        if True:\n            i = 0\n            nodes_ = []\n            while i < len(nodes):\n                nxt = nodes[i]\n                pst = None\n                if isinstance(nxt, str) and nxt.startswith('<pointer:'):\n                    e = nxt.find('>')\n                    if e != len(nxt) -1:\n                        pst = nxt[e+1:]\n                        nxt = nxt[:e+1]\n                    nodes_.append(nxt)\n                    if pst is not None:\n                        nodes_.append(pst)\n                else:\n                    nodes_.append(nxt)\n                i += 1\n            nodes = nodes_\n\n            i = 1\n            nodes_ = [nodes[0]]\n            while i < len(nodes):\n                nxt = nodes[i]\n                if isinstance(nxt, str) and nxt.startswith('<pointer:'):\n                    nxt = 'z' + nxt[9:-1]\n                    fol = nodes[i+1]\n                    # is not expansion\n                    if isinstance(fol, str) and (fol.startswith(':') or (fol == ')')):\n                        nodes_.append(nxt)\n                    else:\n                        if self.remove_pars:\n                            nodes_.append('(')\n                        else:\n                            if nodes_[-1] != '(':\n                                nodes_.append('(')\n                                #pass\n                        nodes_.append(nxt)\n                        nodes_.append('/')\n                else:\n                    nodes_.append(nxt)\n                i += 1\n            nodes = nodes_\n\n        i = 0\n        nodes_ = []\n        while i < (len(nodes) - 1):\n            if nodes[i] == ':':\n                nodes_.append(nodes[i] + nodes[i+1])\n                i += 2\n                last = False\n            else:\n                nodes_.append(nodes[i])\n                i += 1\n                last = True\n        if last:\n            nodes_.append(nodes[-1])\n        nodes = nodes_\n\n        i = 0\n        nodes_ = []\n        while i < (len(nodes)):\n            if i < 2:\n                nodes_.append(nodes[i])\n                i += 1\n            elif nodes_[-2] == '/' and nodes[i] == '/':\n                i += 2\n            else:\n                nodes_.append(nodes[i])\n                i += 1\n        nodes = nodes_\n\n        i = 0\n        newvars = 0\n        variables = set()\n        remap = {}\n        nodes_ = []\n        while i < (len(nodes)):\n\n            next = nodes[i]\n\n            if next == '/':\n                last = nodes_[-1]\n                if last in variables:\n                    last_remap = f\"z{newvars+1000}\"\n                    newvars += 1\n                    nodes_[-1] = last_remap\n                    remap[last] = last_remap\n                variables.add(last)\n                nodes_.append(next)\n\n            elif self._classify(next) == 'VAR' and next in remap and (i < len(nodes) - 1) and nodes[i+1] != '/':\n                next = remap[next]\n                nodes_.append(next)\n\n            else:\n                nodes_.append(next)\n\n            i += 1\n\n        nodes = nodes_\n        pieces_ = []\n        open_cnt = 0\n        closed_cnt = 0\n        if nodes[0] != '(':\n            pieces_.append('(')\n            open_cnt += 1\n        for p in nodes:\n            if p == '(':\n                open_cnt += 1\n            elif p == ')':\n                closed_cnt += 1\n            pieces_.append(p)\n            if open_cnt == closed_cnt:\n                break\n        nodes = pieces_ + [')'] * (open_cnt - closed_cnt)\n\n        pieces = []\n        for piece in nodes:\n            if not pieces:\n                pieces.append('(')\n            else:\n                piece = str(piece)\n                if piece.startswith('\"') or piece.startswith('\"') or '\"' in piece.strip('\"'):\n                    piece = '\"' + piece.replace('\"', '') + '\"'\n\n                prev = self._classify(pieces[-1])\n                next = self._classify(piece)\n\n                if next == 'CONST':\n                    quote = False\n                    for char in (',', ':', '/', '(', ')', '.', '!', '?', '\\\\', '_', '='):\n                        if char in piece:\n                            quote = True\n                            break\n                    if quote:\n                        piece = '\"' + piece.strip('\"') + '\"'\n\n                if  prev == '(':\n                    if next in ('VAR', 'I'):\n                        pieces.append(piece)\n                elif prev == ')':\n                    if next in (')', 'EDGE', 'MODE'):\n                        pieces.append(piece)\n                elif prev == 'VAR':\n                    if next in ('/', 'EDGE', 'MODE', ')'):\n                        pieces.append(piece)\n                elif prev == '/':\n                    if next in ('INST', 'I'):\n                        pieces.append(piece)\n                elif prev == 'INST':\n                    if next in (')', 'EDGE', 'MODE'):\n                        pieces.append(piece)\n                elif prev == 'I':\n                    if next in ('/', ')', 'EDGE', 'MODE'):\n                        pieces.append(piece)\n                elif prev == 'EDGE':\n                    if next in ('(', 'VAR', 'CONST', 'I'):\n                        pieces.append(piece)\n                    elif next == ')':\n                        pieces[-1] = piece\n                    elif next in ('EDGE', 'MODE'):\n                        pieces[-1] = piece\n                elif prev == 'MODE':\n                    if next == 'INST':\n                        pieces.append(piece)\n                elif prev == 'CONST':\n                    if next in (')', 'EDGE', 'MODE'):\n                        pieces.append(piece)\n\n        pieces_ = []\n        open_cnt = 0\n        closed_cnt = 0\n        if pieces[0] != '(':\n            pieces_.append('(')\n            open_cnt += 1\n        for p in pieces:\n            if p == '(':\n                open_cnt += 1\n            elif p == ')':\n                closed_cnt += 1\n            pieces_.append(p)\n            if open_cnt == closed_cnt:\n                break\n        pieces = pieces_ + [')'] * (open_cnt - closed_cnt)\n\n        linearized = re.sub(r'\\s+', ' ', ' '.join(pieces)).strip()\n\n        \"\"\"\n        line = linearized\n        # make sure parentheses match\n        # copied from https://github.com/RikVN/AMR/blob/master/restoreAMR/restore_amr.py\n        open_count = 0\n        close_count = 0\n        for i, c in enumerate(line):\n            if c == '(':\n                open_count += 1\n            elif c == ')':\n                close_count += 1\n            if open_count == close_count and open_count > 0:\n                line = line[:i].strip()\n                break\n        old_line = line\n        while True:\n            open_count = len(re.findall(r'\\(', line))\n            close_count = len(re.findall(r'\\)', line))\n            if open_count > close_count:\n                line += ')' * (open_count - close_count)\n            elif close_count > open_count:\n                for i in range(close_count - open_count):\n                    line = line.rstrip(')')\n                    line = line.rstrip(' ')\n            if old_line == line:\n                break\n            old_line = line\n        \"\"\"\n\n        graph = penman.decode(linearized + ' ')\n        triples = []\n        newvars = 2000\n        for triple in graph.triples:\n            x, rel, y = triple\n            if x is None:\n                pass\n            elif rel == ':instance' and y is None:\n                triples.append(penman.Triple(x, rel, 'thing'))\n            elif y is None:\n                var = f'z{newvars}'\n                newvars += 1\n                triples.append(penman.Triple(x, rel, var))\n                triples.append(penman.Triple(var, ':instance', 'thing'))\n            else:\n                triples.append(triple)\n        graph = penman.Graph(triples)\n        linearized = encode(graph)\n\n        def fix_text(linearized=linearized):\n            n = 0\n            def _repl1(match):\n                nonlocal n\n                out = match.group(1) + match.group(2) + str(3000 + n) + ' / ' + match.group(2) + match.group(3)\n                n += 1\n                return out\n            linearized = re.sub(r'(\\(\\s?)([a-z])([^\\/:\\)]+[:\\)])', _repl1, linearized,\n                                flags=re.IGNORECASE | re.MULTILINE)\n\n            def _repl2(match):\n                return match.group(1)\n            linearized = re.sub(r'(\\(\\s*[a-z][\\d+]\\s*\\/\\s*[^\\s\\)\\(:\\/]+\\s*)((?:/\\s*[^\\s\\)\\(:\\/]+\\s*)+)', _repl2,\n                                linearized,\n                                flags=re.IGNORECASE | re.MULTILINE)\n\n            # adds a ':' to args w/o it\n            linearized = re.sub(r'([^:])(ARG)', r'\\1 :\\2', linearized)\n\n            # removes edges with no node\n            # linearized = re.sub(r':[^\\s\\)\\(:\\/]+?\\s*\\)', ')', linearized, flags=re.MULTILINE)\n\n            return linearized\n\n        linearized = fix_text(linearized)\n        g = penman.decode(linearized)\n        return g\n\n    def _classify(self, node):\n        if not isinstance(node, str):\n            return \"CONST\"\n        elif node == 'i':\n            return \"I\"\n        elif re.match(r'^[a-z]\\d*$', node) is not None:\n            return \"VAR\"\n        elif node[0].isdigit():\n            return \"CONST\"\n        elif node.startswith('\"') and node.endswith('\"'):\n            return \"CONST\"\n        elif node in ('+', '-'):\n            return \"CONST\"\n        elif node == ':mode':\n            return 'MODE'\n        elif node.startswith(':'):\n            return \"EDGE\"\n        elif node in ['/', '(', ')']:\n            return node\n        elif node[0].isalpha():\n            for char in (',', ':', '/', '(', ')', '.', '!', '?', '\\\\'):\n                if char in node:\n                    return \"CONST\"\n            return \"INST\"\n        else:\n            return 'CONST'"
  },
  {
    "path": "hanlp/components/amr/amrbart/preprocess/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2022-12-03 20:33\n"
  },
  {
    "path": "hanlp/components/amr/amrbart/preprocess/amr_io.py",
    "content": "# coding:utf-8\n# the code is migrated from https://github.com/SapienzaNLP/spring \n# MIT License\n#\n# Copyright (c) 2022 xfbai\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\n\nimport glob\nfrom pathlib import Path\nfrom typing import List, Union, Iterable\nfrom hanlp.components.amr.amrbart.preprocess.penman_interface import load as pm_load\n\n\ndef read_raw_amr_data(\n        paths: List[Union[str, Path]], use_recategorization=False, dereify=True, remove_wiki=False,\n):\n    \"\"\" code for loading AMR from a set of files\n        - use_recategorization: use graph recategorization trick\n        - dereify: Dereify edges in g that have reifications in model.\n        - remove_wiki: remove wiki links\n    \"\"\"\n    assert paths\n    if not isinstance(paths, Iterable):\n        paths = [paths]\n\n    graphs = []\n    for path_ in paths:\n        for path in glob.glob(str(path_)):\n            path = Path(path)\n            graphs.extend(pm_load(path, dereify=dereify, remove_wiki=remove_wiki))\n\n    assert graphs\n\n    if use_recategorization:\n        for g in graphs:\n            metadata = g.metadata\n            metadata[\"snt_orig\"] = metadata[\"snt\"]\n            tokens = eval(metadata[\"tokens\"])\n            metadata[\"snt\"] = \" \".join(\n                [\n                    t\n                    for t in tokens\n                    if not ((t.startswith(\"-L\") or t.startswith(\"-R\")) and t.endswith(\"-\"))\n                ]\n            )\n\n    return graphs\n"
  },
  {
    "path": "hanlp/components/amr/amrbart/preprocess/penman_interface.py",
    "content": "# coding:utf-8\n# MIT License\n#\n# Copyright (c) 2022 xfbai\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\nfrom penman import load as load_, Graph, Triple\nfrom penman import loads as loads_\nfrom penman import encode as encode_\nfrom penman.model import Model\nfrom penman.models.noop import NoOpModel\nfrom penman.models import amr\n\nop_model = Model()\nnoop_model = NoOpModel()\namr_model = amr.model\nDEFAULT = op_model\n\n\ndef _get_model(dereify):\n    if dereify is None:\n        return DEFAULT\n\n    elif dereify:\n        return op_model\n\n    else:\n        return noop_model\n\n\ndef _remove_wiki(graph):\n    metadata = graph.metadata\n    triples = []\n    for t in graph.triples:\n        v1, rel, v2 = t\n        if rel == \":wiki\":\n            t = Triple(v1, rel, \"+\")\n        triples.append(t)\n    graph = Graph(triples)\n    graph.metadata = metadata\n    return graph\n\n\ndef load(source, dereify=None, remove_wiki=False):\n    model = _get_model(dereify)\n    out = load_(source=source, model=model)\n    if remove_wiki:\n        for i in range(len(out)):\n            out[i] = _remove_wiki(out[i])\n    return out\n\n\ndef loads(string, dereify=None, remove_wiki=False):\n    model = _get_model(dereify)\n    out = loads_(string=string, model=model)\n    if remove_wiki:\n        for i in range(len(out)):\n            out[i] = _remove_wiki(out[i])\n    return out\n\n\ndef encode(g, top=None, indent=-1, compact=False):\n    model = amr_model\n    return encode_(g=g, top=top, indent=indent, compact=compact, model=model)\n"
  },
  {
    "path": "hanlp/components/amr/amrbart/preprocess/read_and_process.py",
    "content": "# coding:utf-8\n# MIT License\n#\n# Copyright (c) 2022 xfbai\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\nimport re\nimport copy\nimport json\nimport yaml\nimport penman\nfrom tqdm import tqdm\nfrom pathlib import Path\nfrom hanlp.components.amr.amrbart.preprocess.amr_io import read_raw_amr_data\n\n\ndef _tokenize_encoded_graph(encoded):\n    linearized = re.sub(r\"(\\\".+?\\\")\", r\" \\1 \", encoded)\n    pieces = []\n    for piece in linearized.split():\n        if piece.startswith('\"') and piece.endswith('\"'):\n            pieces.append(piece)\n        else:\n            piece = piece.replace(\"(\", \" ( \")\n            piece = piece.replace(\")\", \" ) \")\n            piece = piece.replace(\":\", \" :\")\n            piece = piece.replace(\"/\", \" / \")\n            piece = piece.strip()\n            pieces.append(piece)\n    linearized = re.sub(r\"\\s+\", \" \", \" \".join(pieces)).strip()\n    return linearized.split(\" \")\n\n\ndef dfs_linearize(graph, remove_pars=False, use_pointer_tokens=True):\n    graph_ = copy.deepcopy(graph)\n    graph_.metadata = {}\n    linearized = penman.encode(graph_)\n    linearized_nodes = _tokenize_encoded_graph(linearized)\n\n    if use_pointer_tokens:\n        remap = {}\n        for i in range(1, len(linearized_nodes)):\n            nxt = linearized_nodes[i]\n            lst = linearized_nodes[i - 1]\n            if nxt == \"/\":\n                remap[lst] = f\"<pointer:{len(remap)}>\"\n        i = 1\n        linearized_nodes_ = [linearized_nodes[0]]\n        while i < (len(linearized_nodes)):\n            nxt = linearized_nodes[i]\n            lst = linearized_nodes_[-1]\n            if nxt in remap:\n                if lst == \"(\" and linearized_nodes[i + 1] == \"/\":\n                    nxt = remap[nxt]\n                    i += 1\n                elif lst.startswith(\":\"):\n                    nxt = remap[nxt]\n            linearized_nodes_.append(nxt)\n            i += 1\n        linearized_nodes = linearized_nodes_\n        if remove_pars:\n            linearized_nodes = [n for n in linearized_nodes if n != \"(\"]\n    return linearized_nodes\n\n\ndef main():\n    from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter\n    parser = ArgumentParser(\n        description=\"AMR processing script\",\n        formatter_class=ArgumentDefaultsHelpFormatter,\n    )\n    parser.add_argument('--config', type=Path, default='default.yaml',\n                        help='Use the following config for hparams.')\n    parser.add_argument('--input_file', type=str,\n                        help='The input AMR file.')\n    parser.add_argument('--output_prefix', type=str,\n                        help='The output_prefix.')\n\n    args, unknown = parser.parse_known_args()\n\n    with args.config.open() as y:\n        config = yaml.load(y, Loader=yaml.FullLoader)\n\n    remove_pars = False\n    use_pointer_tokens = True\n    graphs = read_raw_amr_data(\n        [args.input_file],\n        use_recategorization=config[\"use_recategorization\"],\n        remove_wiki=config[\"remove_wiki\"],\n        dereify=config[\"dereify\"],\n    )\n\n    line_amr, sentences = [], []\n\n    for g in tqdm(graphs):\n        lin_tokens = dfs_linearize(g)\n        sentences.append(g.metadata[\"snt\"])\n        # line_amr.append(\" \".join(lin_tokens[1:-1]))\n        line_amr.append(\" \".join(lin_tokens))\n\n    print(f\"all {len(line_amr)} AMRs processed\")\n\n    with open(args.output_prefix + \".amr\", \"w\", encoding=\"utf-8\") as fout:\n        fout.write(\"\\n\".join(line_amr) + \"\\n\")\n\n    with open(args.output_prefix + \".txt\", \"w\", encoding=\"utf-8\") as fout:\n        fout.write(\"\\n\".join(sentences) + \"\\n\")\n\n    res_out = [json.dumps({\"sent\": sent, \"amr\": lamr}) for lamr, sent in zip(line_amr, sentences)]\n\n    with open(args.output_prefix + \".jsonl\", \"w\", encoding=\"utf-8\") as fout:\n        fout.write(\"\\n\".join(res_out) + \"\\n\")\n\n\nif __name__ == '__main__':\n    main()\n"
  },
  {
    "path": "hanlp/components/amr/seq2seq/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-04-27 19:24\n"
  },
  {
    "path": "hanlp/components/amr/seq2seq/dataset/IO.py",
    "content": "import glob\nfrom typing import List, Union, Iterable\nfrom pathlib import Path\nfrom .penman import pm_load as pm_load\n\n\ndef read_raw_amr_data(\n        paths: List[Union[str, Path]],\n        use_recategorization=False,\n        dereify=True,\n        remove_wiki=False,\n):\n    assert paths\n\n    if not isinstance(paths, Iterable):\n        paths = [paths]\n\n    graphs = []\n    for path_ in paths:\n        for path in glob.glob(str(path_)):\n            path = Path(path)\n            assert path.exists(), f'{path} not exist'\n            graphs.extend(pm_load(path, dereify=dereify, remove_wiki=remove_wiki))\n\n    assert graphs, 'No graphs loaded'\n\n    if use_recategorization:\n        for g in graphs:\n            metadata = g.metadata\n            metadata['snt_orig'] = metadata['snt']\n            tokens = eval(metadata['tokens'])\n            metadata['snt'] = ' '.join(\n                [t for t in tokens if not ((t.startswith('-L') or t.startswith('-R')) and t.endswith('-'))])\n\n    return graphs\n"
  },
  {
    "path": "hanlp/components/amr/seq2seq/dataset/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-04-27 19:29\n"
  },
  {
    "path": "hanlp/components/amr/seq2seq/dataset/dataset.py",
    "content": "from collections import Counter\nfrom typing import Union, List, Callable, Tuple\nimport torch\nimport penman\nfrom penman import Graph\nfrom hanlp.common.dataset import TransformableDataset\nfrom hanlp.components.amr.seq2seq.dataset.IO import read_raw_amr_data\nfrom hanlp.components.amr.seq2seq.dataset.penman import role_is_reverted\nfrom hanlp.components.amr.seq2seq.dataset.tokenization_bart import PENMANBartTokenizer\nfrom phrasetree.tree import Tree\nimport json\n\nfrom hanlp_common.constant import BOS, EOS, ROOT\nfrom hanlp_common.io import load_pickle\n\n\nclass AMRDataset(TransformableDataset):\n\n    def __init__(self,\n                 data: Union[str, List],\n                 use_recategorization=False,\n                 remove_wiki=False,\n                 dereify=False,\n                 transform: Union[Callable, List] = None,\n                 cache=None,\n                 generate_idx=None) -> None:\n        self.dereify = dereify\n        self.remove_wiki = remove_wiki\n        self.use_recategorization = use_recategorization\n        super().__init__(data, transform, cache, generate_idx)\n\n    def load_file(self, filepath: str):\n        graphs = read_raw_amr_data([filepath], self.use_recategorization, remove_wiki=self.remove_wiki,\n                                   dereify=self.dereify)\n        for g in graphs:\n            yield {'amr': g}\n\n    def get_roles(self):\n        roles = Counter()\n        for sample in self.data:\n            g: Graph = sample['amr']\n            for s, r, t in g.triples:\n                if role_is_reverted(r):\n                    r = r[:-3]\n                roles[r] += 1\n        return roles\n\n    def get_frames(self):\n        frames = Counter()\n        for sample in self.data:\n            g: Graph = sample['amr']\n            for i in g.instances():\n                t = i.target\n                cells = t.split('-')\n                if len(cells) == 2 and len(cells[1]) == 2 and cells[1].isdigit():\n                    frames[t] += 1\n        return frames\n\n\nclass AMRPickleDataset(AMRDataset):\n\n    def load_file(self, filepath: str):\n        items = torch.load(filepath)\n        for each in items:\n            each['amr'] = penman.decode(each['amr'])\n            yield each\n\n\ndef dfs_linearize_tokenize(sample: dict, tokenizer: PENMANBartTokenizer, remove_space=False, text_key='snt') -> dict:\n    amr = sample.get('amr', None)\n    if amr:\n        l, e = tokenizer.linearize(amr)\n        sample['graph_tokens'] = e['linearized_graphs']\n        sample['graph_token_ids'] = l\n        text = amr.metadata[text_key]\n    else:\n        text = sample['text']\n    if remove_space:\n        text = ''.join(text.split())\n    sample['text'] = text\n    sample['text_token_ids'] = tokenizer.encode(text)\n    return sample\n\n\ndef dfs_linearize_levi(sample: dict, tokenizer: PENMANBartTokenizer, remove_space=False) -> dict:\n    amr = sample.get('amr', None)\n    if amr:\n        l, e = tokenizer.linearize(amr)\n        sample['graph_tokens'] = e['linearized_graphs']\n        sample['graph_token_ids'] = l\n        tok = json.loads(amr.metadata['tok'])\n        dep = json.loads(amr.metadata['dep'])\n        levi = dep_to_levi(tok, dep)\n        sample['text'] = ' '.join(levi)\n        # ids = sum(tokenizer.batch_encode_plus([' ' + x for x in levi], add_special_tokens=False).input_ids, [])\n        ids = []\n        idx = 0\n        for t in levi:\n            if t in ('(', ')'):\n                ids.append(tokenizer.convert_tokens_to_ids(tokenizer.INIT + t))\n            else:\n                if idx % 2:\n                    ids.extend(tokenizer.encode(t, add_special_tokens=False))\n                else:\n                    ids.append(tokenizer.convert_tokens_to_ids(tokenizer.INIT + t))\n                idx += 1\n        sample['text_token_ids'] = [tokenizer.bos_token_id] + ids + [tokenizer.eos_token_id]\n    return sample\n\n\ndef dfs_linearize_rgcn(sample: dict, tokenizer: PENMANBartTokenizer) -> dict:\n    amr = sample.get('amr', None)\n    if amr:\n        l, e = tokenizer.linearize(amr)\n        sample['graph_tokens'] = e['linearized_graphs']\n        sample['graph_token_ids'] = l\n        tok = sample['tok']\n        sample['text'] = [tokenizer.cls_token] + [' ' + x for x in tok]\n        arc_scores = sample['dep']['scores']['arc_scores']\n        rel_scores = sample['dep']['scores']['rel_scores']\n        dep_graph = arc_scores[:, :, None] * rel_scores\n        root = torch.zeros((1,) + dep_graph.shape[1:])\n        sample['dep_graph'] = torch.cat([root, dep_graph], dim=0)\n    return sample\n\n\ndef dfs_linearize_constituency(sample: dict, tokenizer: PENMANBartTokenizer, remove_space=False) -> dict:\n    amr = sample.get('amr', None)\n    if amr:\n        l, e = tokenizer.linearize(amr)\n        sample['graph_tokens'] = e['linearized_graphs']\n        sample['graph_token_ids'] = l\n        tree = Tree.from_list(json.loads(sample['amr'].metadata['con_list']))\n        for each in tree.subtrees(lambda x: x.height() == 2):\n            if each[0] == '(':\n                each[0] = '<LBR>'\n            elif each[0] == ')':\n                each[0] = '<RBR>'\n        text = tree.pformat(margin=10e7)\n        tokens = []\n        buffer = []\n        for c in text:\n            if c == '(' or c == ')':\n                tokens.append(''.join(buffer))\n                tokens.append(c)\n                buffer.clear()\n                continue\n            buffer.append(c)\n        if buffer:\n            tokens.append(''.join(buffer))\n        tokens = [x.strip() for x in tokens]\n        tokens = [x for x in tokens if x]\n        restore_bracket = {'<LBR>': '(', '<RBR>': ')'}\n        tokens = [restore_bracket.get(x, x) for x in tokens]\n        ids = []\n        for each in tokens:\n            pairs = each.split(' ', 1)\n            if len(pairs) == 2:\n                con, token = pairs\n                ids.append(tokenizer.convert_tokens_to_ids(tokenizer.INIT + con))\n                ids.extend(tokenizer.encode(token, add_special_tokens=False))\n            else:\n                ids.append(tokenizer.convert_tokens_to_ids(tokenizer.INIT + each))\n        if remove_space:\n            text = ''.join(text.split())\n        sample['text'] = text\n        sample['text_token_ids'] = [tokenizer.bos_token_id] + ids + [tokenizer.eos_token_id]\n    return sample\n\n\ndef dfs_linearize_tokenize_with_linguistic_structures(sample: dict, tokenizer: PENMANBartTokenizer,\n                                                      remove_space=False,\n                                                      text_key='snt') -> dict:\n    amr = sample.get('amr', None)\n    if amr:\n        l, e = tokenizer.linearize(amr)\n        sample['graph_tokens'] = e['linearized_graphs']\n        sample['graph_token_ids'] = l\n        text = amr.metadata[text_key]\n        if remove_space:\n            text = ''.join(text.split())\n        sample['text'] = text\n        tok = json.loads(amr.metadata['tok'])\n        text_token_ids = tokenizer.batch_encode_plus(tok, add_special_tokens=False).input_ids\n        sample['text_token_ids'] = [tokenizer.bos_token_id] + sum(text_token_ids, []) + [tokenizer.eos_token_id]\n        pos = amr.metadata.get('pos', None)\n        if pos:\n            flat_pos = []\n            pos = json.loads(pos)\n            for subtokens, tag in zip(text_token_ids, pos):\n                flat_pos.extend([tag] * len(subtokens))\n            sample['pos'] = [BOS] + flat_pos + [EOS]\n        ner = amr.metadata.get('ner', None)\n        if ner is not None:\n            flat_ner = []\n            ner_spans = json.loads(ner)\n            ner = ['O'] * len(text_token_ids)\n            for form, tag, start, end in ner_spans:\n                ner[start:end] = [tag] * (end - start)\n            for subtokens, tag in zip(text_token_ids, ner):\n                flat_ner.extend([tag] * len(subtokens))\n            sample['ner'] = [BOS] + flat_ner + [EOS]\n        dep = amr.metadata.get('dep', None)\n        if dep:\n            token_to_1st_subtoken = [0]\n            num_subtokens = 1  # 1 for BOS\n            for subtokens in text_token_ids:\n                token_to_1st_subtoken.append(num_subtokens)\n                num_subtokens += len(subtokens)\n            flat_arc, flat_rel = [0], [BOS]\n            dep = json.loads(dep)\n            for subtokens, (arc, rel) in zip(text_token_ids, dep):\n                flat_arc.extend([token_to_1st_subtoken[arc]] * len(subtokens))\n                flat_rel.extend([rel] * len(subtokens))\n            sample['dep_arc'] = flat_arc + [0]\n            sample['dep_rel'] = flat_rel + [EOS]\n    return sample\n\n\ndef dep_to_levi(tok: List[str], dep: List[Tuple[int, str]]):\n    root = [i for i, x in enumerate(dep) if x[0] == 0][0]\n    seq = []\n    dfs(tok, dep, root, seq)\n    return seq\n\n\ndef dfs(tok: List[str], dep: List[Tuple[int, str]], s, seq):\n    seq.append(dep[s][1])\n    seq.append(tok[s])\n    children = [i for i, x in enumerate(dep) if x[0] == s + 1]\n    if children:\n        seq.append('(')\n        for child in children:\n            dfs(tok, dep, child, seq)\n        seq.append(')')\n"
  },
  {
    "path": "hanlp/components/amr/seq2seq/dataset/linearization.py",
    "content": "import abc\nimport itertools\nfrom collections import deque, defaultdict\nimport re\nfrom typing import List, Optional, Dict, Any, Set, TypeVar\nfrom dataclasses import dataclass\nimport networkx as nx\nimport penman\n\n\n@dataclass\nclass SemanticGraph:\n    nodes_var: List[str]\n    \"\"\"\n    List of linearized nodes, with special tokens.\n    \"\"\"\n    edges: Optional[List[str]]\n    \"\"\"\n    List of linearized edges, with special tokens.\n    \"\"\"\n    backreferences: List[int]\n    \"\"\"\n    List of backpointers to handle rentrancies and cycles.\n    \"\"\"\n    var2instance: Dict[str, str]\n    \"\"\"\n    Dict from var ids to 'lemmatized' readable strings qualifying the node (collapsing the :instance edge for AMR).\n    \"\"\"\n    extra: Dict[str, Any]\n    \"\"\"\n    Holds extra stuff that might be useful, e.g. alignments, NER, EL.\n    \"\"\"\n\n    # @cached_property\n    @property\n    def variables(self) -> Set[str]:\n        \"\"\"Set of variables in this semantic graph\"\"\"\n        variables = {v for v in self.nodes_var if not v.startswith('<')}\n        return variables\n\n    @property\n    def resolved_nodes_var(self) -> List[str]:\n        return [self.nodes_var[b] for b in self.backreferences]\n\n    # @cached_property\n    @property\n    def nodes(self) -> List[str]:\n        \"\"\"Linearized nodes with varids replaced by instances\"\"\"\n        return [self.var2instance.get(node, node) for node in self.nodes_var]\n\n    @property\n    def resolved_nodes(self) -> List[str]:\n        return [self.nodes[b] for b in self.backreferences]\n\n    def src_occurrence(self, var: str) -> int:\n        pass\n\n\nclass BaseLinearizer(metaclass=abc.ABCMeta):\n\n    @abc.abstractmethod\n    def linearize(self, *args, **kwargs) -> SemanticGraph:\n        pass\n\n\nclass AMRTokens:\n    START, END = '<', '>'\n    _TEMPL = START + '{}' + END\n\n    BOS_N = _TEMPL.format('s')\n    EOS_N = _TEMPL.format('/s')\n    START_N = _TEMPL.format('start')\n    STOP_N = _TEMPL.format('stop')\n    PNTR_N = _TEMPL.format('pointer')\n\n    LIT_START = _TEMPL.format('lit')\n    LIT_END = _TEMPL.format('/lit')\n\n    BACKR_SRC_N = _TEMPL.format('backr:src:XXX')\n    BACKR_TRG_N = _TEMPL.format('backr:trg:XXX')\n\n    BOS_E = _TEMPL.format('s')\n    EOS_E = _TEMPL.format('/s')\n    START_E = _TEMPL.format('start')\n    STOP_E = _TEMPL.format('stop')\n\n    _FIXED_SPECIAL_TOKENS_N = {\n        BOS_N, EOS_N, START_N, STOP_N}\n    _FIXED_SPECIAL_TOKENS_E = {\n        BOS_E, EOS_E, START_E, STOP_E}\n    _FIXED_SPECIAL_TOKENS = _FIXED_SPECIAL_TOKENS_N | _FIXED_SPECIAL_TOKENS_E\n\n    # match and read backreferences\n    _re_BACKR_SRC_N = re.compile(BACKR_SRC_N.replace('XXX', r'([0-9]+)'))\n    _re_BACKR_TRG_N = re.compile(BACKR_TRG_N.replace('XXX', r'([0-9]+)'))\n\n    @classmethod\n    def is_node(cls, string: str) -> bool:\n        if isinstance(string, str) and string.startswith(':'):\n            return False\n        elif string in cls._FIXED_SPECIAL_TOKENS_E:\n            return False\n        return True\n\n    @classmethod\n    def read_backr(cls, string: str) -> Optional:\n        m_src = cls._re_BACKR_SRC_N.search(string)\n        if m_src is not None:\n            return m_src\n        m_trg = cls._re_BACKR_TRG_N.search(string)\n        if m_trg is not None:\n            return m_trg\n        return None\n\n\nT = TypeVar('T')\n\n\ndef index_default(\n        item: T, list_: List[T],\n        start: Optional[int] = None,\n        stop: Optional[int] = None,\n        default: Optional[int] = None\n):\n    if start is None:\n        start = 0\n    if stop is None:\n        stop = len(list_)\n    return next((i for i, x in enumerate(list_[start:stop], start=start) if x == item), default)\n\n\nclass AMRLinearizer(BaseLinearizer):\n\n    def __init__(\n            self,\n            use_pointer_tokens: bool = True,\n            collapse_name_ops: bool = False,\n    ):\n        self.collapse_name_ops = collapse_name_ops\n        self.interleave_edges = False\n        self.use_pointer_tokens = use_pointer_tokens\n\n    def _collapse_name_ops(self, amr):\n        # identify name triples\n        name_vars = {}\n        for i, (v1, rel, v2) in enumerate(amr.triples):\n            if rel == ':instance' and v2 == 'name':\n                name_vars[v1] = 1\n\n        # check if they have ops\n        name_vars_to_ops = defaultdict(list)\n        for i, (v1, rel, v2) in enumerate(amr.triples):\n            if v1 in name_vars and rel.startswith(':op'):\n                name_vars_to_ops[v1].append((i, rel, v2.strip('\"')))\n\n        triples = amr.triples.copy()\n        for nv, ops in name_vars_to_ops.items():\n            ops = sorted(ops, key=lambda x: int(x[1][3:]))\n            idx, _, lits = zip(*ops)\n            for i in idx:\n                triples[i] = None\n            lit = '\"' + '_'.join(lits) + '\"'\n            triples[min(idx)] = penman.Triple(nv, ':op1', lit)\n\n        triples = [t for t in triples if t is not None]\n        amr_ = penman.Graph(triples)\n        amr_.metadata = amr.metadata\n        return amr_\n\n    def linearize(self, amr: penman.Graph) -> SemanticGraph:\n        if self.collapse_name_ops:\n            amr = self._collapse_name_ops(amr)\n        linearized = self._linearize(amr)\n        linearized = self._interleave(linearized)\n        if self.use_pointer_tokens:\n            linearized = self._add_pointer_tokens(linearized)\n        return linearized\n\n    def _linearize(self, amr: penman.Graph) -> SemanticGraph:\n        variables = set(amr.variables())\n        variables = {'var:' + v for v in variables}\n        var2instance = {}\n\n        graph = nx.MultiDiGraph()\n\n        triples2order = {k: i for i, k in enumerate(amr.triples)}\n\n        for triple in amr.triples:\n            var, rel, instance = triple\n            order = triples2order[triple]\n            if rel != ':instance':\n                continue\n            for expansion_candidate in itertools.chain(range(order - 1, -1), range(order + 1, len(amr.triples))):\n                if var == amr.triples[expansion_candidate][2]:\n                    expansion = expansion_candidate\n                    break\n            else:\n                expansion = 0\n            var = 'var:' + var\n            var2instance[var] = instance\n            graph.add_node(var, instance=instance, order=order, expansion=expansion)\n\n        for triple in amr.edges():\n            var1, rel, var2 = triple\n            order = triples2order[triple]\n            if rel == ':instance':\n                continue\n            var1 = 'var:' + var1\n            var2 = 'var:' + var2\n            graph.add_edge(var1, var2, rel=rel, order=order)\n\n        for triple in amr.attributes():\n            var, rel, attr = triple\n            order = triples2order[triple]\n            if rel == ':instance':\n                continue\n            var = 'var:' + var\n            graph.add_edge(var, attr, rel=rel, order=order)\n\n        # nodes that are not reachable from the root (e.g. because of reification)\n        # will be present in the not_explored queue\n        # undirected_graph = graph.to_undirected()\n        # print(amr.variables())\n        not_explored = deque(sorted(variables, key=lambda x: nx.get_node_attributes(graph, 'order')[x]))\n        # (\n        #     len(nx.shortest_path(undirected_graph, 'var:' + amr.top, x)),\n        #     -graph.out_degree(x),\n        # )\n\n        first_index = {}\n        explored = set()\n        added_to_queue = set()\n        nodes_visit = [AMRTokens.BOS_N]\n        edges_visit = [AMRTokens.BOS_E]\n        backreferences = [0]\n        queue = deque()\n        queue.append('var:' + amr.top)\n\n        while queue or not_explored:\n\n            if queue:\n                node1 = queue.popleft()\n            else:\n                node1 = not_explored.popleft()\n                if node1 in added_to_queue:\n                    continue\n                if not list(graph.successors(node1)):\n                    continue\n\n            if node1 in variables:\n                if node1 in explored:\n                    continue\n                if node1 in first_index:\n                    nodes_visit.append(AMRTokens.BACKR_TRG_N)\n                    backreferences.append(first_index[node1])\n                else:\n                    backreferences.append(len(nodes_visit))\n                    first_index[node1] = len(nodes_visit)\n                    nodes_visit.append(node1)\n                edges_visit.append(AMRTokens.START_E)\n\n                successors = []\n                for node2 in graph.successors(node1):\n                    for edge_data in graph.get_edge_data(node1, node2).values():\n                        rel = edge_data['rel']\n                        order = edge_data['order']\n                        successors.append((order, rel, node2))\n                successors = sorted(successors)\n\n                for order, rel, node2 in successors:\n                    edges_visit.append(rel)\n\n                    # node2 is a variable\n                    if node2 in variables:\n                        # ... which was mentioned before\n                        if node2 in first_index:\n                            nodes_visit.append(AMRTokens.BACKR_TRG_N)\n                            backreferences.append(first_index[node2])\n\n                        # .. which is mentioned for the first time\n                        else:\n                            backreferences.append(len(nodes_visit))\n                            first_index[node2] = len(nodes_visit)\n                            nodes_visit.append(node2)\n\n                        # 1) not already in Q\n                        # 2) has children\n                        # 3) the edge right before its expansion has been encountered\n                        if (node2 not in added_to_queue) and list(graph.successors(node2)) and (\n                                nx.get_node_attributes(graph, 'expansion')[node2] <= order):\n                            queue.append(node2)\n                            added_to_queue.add(node2)\n\n                    # node2 is a constant\n                    else:\n                        backreferences.append(len(nodes_visit))\n                        nodes_visit.append(node2)\n\n                backreferences.append(len(nodes_visit))\n                nodes_visit.append(AMRTokens.STOP_N)\n                edges_visit.append(AMRTokens.STOP_E)\n                explored.add(node1)\n\n            else:\n                backreferences.append(len(nodes_visit))\n                nodes_visit.append(node1)\n                explored.add(node1)\n\n        backreferences.append(len(nodes_visit))\n        nodes_visit.append(AMRTokens.EOS_N)\n        edges_visit.append(AMRTokens.EOS_E)\n        assert len(nodes_visit) == len(edges_visit) == len(backreferences)\n        return SemanticGraph(\n            nodes_visit,\n            edges_visit,\n            backreferences,\n            var2instance,\n            extra={'graph': graph, 'amr': amr}\n        )\n\n    def _interleave(self, graph: SemanticGraph) -> SemanticGraph:\n\n        new_backreferences_map = []\n        new_nodes = []\n        new_edges = None\n        new_backreferences = []\n\n        # to isolate sublist to the stop token\n        start_i = 1\n        end_i = index_default(AMRTokens.STOP_N, graph.nodes_var, start_i, -1, -1)\n\n        def add_node(node, backr=None):\n            old_n_node = len(new_backreferences_map)\n            new_n_node = len(new_nodes)\n\n            if backr is None:\n                backr = old_n_node\n\n            new_backreferences_map.append(new_n_node)\n            new_nodes.append(node)\n            if old_n_node == backr:\n                new_backreferences.append(new_n_node)\n            else:\n                new_backreferences.append(new_backreferences_map[backr])\n\n        def add_edge(edge):\n            new_nodes.append(edge)\n            new_backreferences.append(len(new_backreferences))\n\n        add_node(AMRTokens.BOS_N)\n\n        while end_i > -1:\n\n            # src node\n            add_node(graph.nodes_var[start_i], graph.backreferences[start_i])\n\n            # edges and trg nodes, interleaved\n            nodes = graph.nodes_var[start_i + 1:end_i]\n            edges = graph.edges[start_i + 1:end_i]\n            backr = graph.backreferences[start_i + 1:end_i]\n            for n, e, b in zip(nodes, edges, backr):\n                add_edge(e)\n                add_node(n, b)\n\n            # stop\n            add_node(graph.nodes_var[end_i], graph.backreferences[end_i])\n\n            start_i = end_i + 1\n            end_i = index_default(AMRTokens.STOP_N, graph.nodes_var, start_i, -1, -1)\n\n        add_node(AMRTokens.EOS_N)\n\n        new_graph = SemanticGraph(\n            new_nodes,\n            None,\n            new_backreferences,\n            graph.var2instance,\n            extra=graph.extra,\n        )\n        return new_graph\n\n    def _add_pointer_tokens(self, graph: SemanticGraph) -> SemanticGraph:\n        new_nodes = []\n        var2pointer = {}\n        for node, backr in zip(graph.nodes_var, graph.backreferences):\n\n            if node == AMRTokens.BACKR_TRG_N:\n                node = graph.nodes_var[backr]\n                pointer = var2pointer[node]\n                new_nodes.append(pointer)\n            elif node in graph.var2instance:\n                pointer = var2pointer.setdefault(node, f\"<pointer:{len(var2pointer)}>\")\n                new_nodes.append(pointer)\n                new_nodes.append(node)\n            else:\n                new_nodes.append(node)\n\n        new_backreferences = list(range(len(new_nodes)))\n        new_graph = SemanticGraph(\n            new_nodes,\n            None,\n            new_backreferences,\n            graph.var2instance,\n            extra=graph.extra,\n        )\n        return new_graph\n\n\n"
  },
  {
    "path": "hanlp/components/amr/seq2seq/dataset/penman.py",
    "content": "from typing import List\n\nfrom penman import load as load_, Graph, Triple\nfrom penman import loads as loads_\nfrom penman import encode as encode_\nfrom penman.model import Model\nfrom penman.models.noop import NoOpModel\nfrom penman.models import amr\nimport penman\nimport logging\n\nop_model = Model()\nnoop_model = NoOpModel()\namr_model = amr.model\nDEFAULT = op_model\n\n# Mute loggers\npenman.layout.logger.setLevel(logging.CRITICAL)\npenman._parse.logger.setLevel(logging.CRITICAL)\n\n\ndef _get_model(dereify):\n    if dereify is None:\n        return DEFAULT\n    elif dereify:\n        return op_model\n    else:\n        return noop_model\n\n\ndef _remove_wiki(graph):\n    metadata = graph.metadata\n    triples = []\n    for t in graph.triples:\n        v1, rel, v2 = t\n        if rel == ':wiki':\n            t = Triple(v1, rel, '+')\n        triples.append(t)\n    graph = Graph(triples)\n    graph.metadata = metadata\n    return graph\n\n\ndef pm_load(source, dereify=None, remove_wiki=False) -> List[penman.Graph]:\n    \"\"\"\n\n    Args:\n        source:\n        dereify: Restore reverted relations\n        remove_wiki:\n\n    Returns:\n\n    \"\"\"\n    model = _get_model(dereify)\n    out = load_(source=source, model=model)\n    if remove_wiki:\n        for i in range(len(out)):\n            out[i] = _remove_wiki(out[i])\n    return out\n\n\ndef loads(string, dereify=None, remove_wiki=False):\n    model = _get_model(dereify)\n    out = loads_(string=string, model=model)\n    if remove_wiki:\n        for i in range(len(out)):\n            out[i] = _remove_wiki(out[i])\n    return out\n\n\ndef pm_encode(g, top=None, indent=-1, compact=False):\n    model = amr_model\n    return encode_(g=g, top=top, indent=indent, compact=compact, model=model)\n\n\ndef role_is_reverted(role: str):\n    if role.endswith('consist-of'):\n        return False\n    return role.endswith('-of')\n\n\nclass AMRGraph(penman.Graph):\n    def __str__(self):\n        return penman.encode(self)\n"
  },
  {
    "path": "hanlp/components/amr/seq2seq/dataset/postprocessing.py",
    "content": "from collections import defaultdict, Counter\nimport enum\nimport re\nimport networkx as nx\nimport penman\n\nfrom hanlp.components.amr.seq2seq.dataset.penman import pm_encode\n\nBACKOFF = penman.Graph([\n    penman.Triple('d2', ':instance', 'dog'),\n    penman.Triple('b1', ':instance', 'bark-01'),\n    penman.Triple('b1', ':ARG0', 'd2'), ])\n\n\ndef token_processing(tok):\n    if tok is None:\n        return None\n    elif tok.isdigit():\n        try:\n            return eval(tok)\n        except:\n            return tok\n    elif tok.startswith('\"') and (not tok.endswith('\"')):\n        return tok + '\"'\n    elif tok.endswith('\"') and (not tok.startswith('\"')):\n        return '\"' + tok\n    else:\n        return tok\n\n\ndef decode_into_node_and_backreferences(subtoken_ids, tokenizer):\n    rex_arg = re.compile(f\"^{tokenizer.INIT}(op|snt|conj|prep)\")\n    rex_spc = re.compile(r\"<(s|/s|lit|/lit|stop|unk|pad|mask)>\")\n\n    # get strings\n    subtokens = tokenizer.convert_ids_to_tokens(subtoken_ids)\n    # fix backreferences\n    subtoken_backreferences = [max(t - len(tokenizer), -1) for t in subtoken_ids]\n    # strip padding\n    no_pad = [(s, b) for s, b in zip(subtokens, subtoken_backreferences) if s != (tokenizer.INIT + '<pad>')]\n    if no_pad:\n        subtokens, subtoken_backreferences = zip(*no_pad)\n    else:\n        subtokens, subtoken_backreferences = ['<s>'], [-1]\n\n    # subword collapse\n    tokens = []\n    backreferences = []\n    subword_to_token_map = {}\n    current_token_i = 0\n    for subw_i, (subw_backr, subtok) in enumerate(zip(subtoken_backreferences, subtokens)):\n        subword_to_token_map[subw_i] = current_token_i\n\n        # if empty you cannot do anything but add a new word\n        if not tokens:\n            tokens.append(subtok.lstrip(tokenizer.INIT))\n            backreferences.append(-1)\n            current_token_i += 1\n\n        # backref can't be splitted\n        elif subw_backr > -1:\n            tokens.append(None)\n            backreferences.append(subword_to_token_map[subw_backr])\n            current_token_i += 1\n\n        # after a special token release\n        elif isinstance(tokens[-1], str) and rex_spc.match(tokens[-1]):\n            tokens.append(subtok.lstrip(tokenizer.INIT))\n            backreferences.append(-1)\n            current_token_i += 1\n\n        # after a subtoken ':' (which should be followed by the rest of the edge) ignore tokenizer.INIT\n        # TODO: this is an ugly patch due to the fact that BART tokenizer splits after ':'\n        elif (tokens[-1] == ':') and rex_arg.match(subtok):\n            tokens[-1] = tokens[-1] + subtok[1:]\n\n        # leading tokenizer.INIT\n        elif subtok.startswith(tokenizer.INIT):\n            tokens.append(subtok.lstrip(tokenizer.INIT))\n            backreferences.append(-1)\n            current_token_i += 1\n\n        # very ugly patch for some cases in which tokenizer.INIT is not in the following token to the edge\n        elif isinstance(tokens[-1], str) and tokens[-1].startswith(':') and tokens[-1][-1].isdigit() and (\n                subtok != '-of'):\n            tokens.append(subtok.lstrip(tokenizer.INIT))\n            backreferences.append(-1)\n            current_token_i += 1\n\n        # in any other case attach to the previous\n        else:\n            tokens[-1] = tokens[-1] + subtok\n\n    # strip INIT and fix byte-level\n    tokens = [tokenizer.convert_tokens_to_string(list(t)).lstrip() if isinstance(t, str) else t for t in tokens]\n    # tokens = [t.replace(tokenizer.INIT, '') if isinstance(t, str) else t for t in tokens]\n\n    # unks are substituted with thing\n    tokens = [t if t != '<unk>' else 'thing' for t in tokens]\n\n    old_tokens = tokens\n    old_backreferences = backreferences\n\n    # <lit> Barack Obama </lit> -> \"Barack Obama\"\n    tokens = []\n    backreferences = []\n    token_to_token_map = {}\n    start_search = 0\n    removed = 0\n    while True:\n        try:\n\n            lit_start = old_tokens.index('<lit>', start_search)\n            token_addition = old_tokens[start_search:lit_start]\n            for i, t in enumerate(token_addition, start=start_search):\n                token_to_token_map[i] = i - removed\n            tokens += token_addition\n\n            backreferences_addition = [token_to_token_map[b] if b > -1 else -1 for b in\n                                       old_backreferences[start_search:lit_start]]\n            backreferences += backreferences_addition\n\n            lit_end = min(lit_start + 2, len(old_tokens) - 1)\n\n            while lit_end < len(old_tokens):\n                old_tok = old_tokens[lit_end]\n\n                if isinstance(old_tok, str) and (\n                        (old_tok.startswith(':') and len(old_tok) > 3) or (old_tok == '<stop>')):\n                    res_tok = old_tokens[lit_start + 1:lit_end]\n                    for i in range(lit_start, lit_end):\n                        token_to_token_map[i] = len(tokens)\n\n                    # Remove possible wrong None\n                    res = old_tokens[lit_start + 1:lit_end]\n                    res = [str(r) for r in res if r is not None]\n                    res = '\"' + '_'.join(res) + '\"'\n\n                    removed += len(res_tok)\n                    start_search = lit_end\n                    tokens += [res, old_tok]\n                    backreferences += [-1, -1]\n                    break\n\n                elif old_tok == '</lit>':\n                    res_tok = old_tokens[lit_start + 1:lit_end]\n                    for i in range(lit_start, lit_end + 1):\n                        token_to_token_map[i] = len(tokens)\n\n                    # Remove possible wrong None\n                    res = old_tokens[lit_start + 1:lit_end]\n                    res = [str(r) for r in res if r is not None]\n                    res = '\"' + '_'.join(res) + '\"'\n\n                    removed += len(res_tok) + 1\n                    start_search = lit_end + 1\n                    tokens.append(res)\n                    backreferences.append(-1)\n                    break\n\n                else:\n                    lit_end += 1\n                    start_search = lit_end\n\n        except ValueError:\n            token_addition = old_tokens[start_search:]\n            for i, t in enumerate(token_addition, start=start_search):\n                token_to_token_map[i] = i - removed\n            backreferences_addition = [token_to_token_map[b] if b > -1 else b for b in\n                                       old_backreferences[start_search:]]\n            tokens += token_addition\n            backreferences += backreferences_addition\n            break\n\n    tokens = [token_processing(t) for t in tokens]\n\n    shift = 1\n    if len(tokens) > 1 and tokens[1] == '<s>':\n        shift = 2\n\n    tokens = tokens[shift:]\n    backreferences = [b if b == -1 else b - shift for b in backreferences[shift:]]\n\n    if tokens and tokens[-1] == '</s>':\n        tokens.pop()\n        backreferences.pop()\n\n    return tokens, backreferences\n\n\ndef decode_into_node_and_backreferences_without_space(subtoken_ids, tokenizer):\n    rex_arg = re.compile(f\"^{tokenizer.INIT}(op|snt|conj|prep)\")\n    rex_spc = re.compile(r\"<(s|/s|lit|/lit|stop|unk|pad|mask)>\")\n\n    # get strings\n    subtokens = tokenizer.convert_ids_to_tokens(subtoken_ids)\n    # fix backreferences\n    subtoken_backreferences = [max(t - len(tokenizer), -1) for t in subtoken_ids]\n    # strip padding\n    no_pad = [(s, b) for s, b in zip(subtokens, subtoken_backreferences) if s != (tokenizer.INIT + '<pad>')]\n    if no_pad:\n        subtokens, subtoken_backreferences = zip(*no_pad)\n    else:\n        subtokens, subtoken_backreferences = ['<s>'], [-1]\n\n    # subword collapse\n    tokens = []\n    backreferences = []\n    subword_to_token_map = {}\n    current_token_i = 0\n    prev_is_pointer = False\n    prev_is_rel = False\n    for subw_i, (subw_backr, subtok) in enumerate(zip(subtoken_backreferences, subtokens)):\n        subword_to_token_map[subw_i] = current_token_i\n        is_pointer = subtok.startswith('<pointer:') and subtok.endswith('>')\n        is_rel = subtok.startswith(':') and len(subtok) > 1\n        is_bracket = subtok in '()'\n\n        # if empty you cannot do anything but add a new word\n        if not tokens:\n            tokens.append(subtok)\n            backreferences.append(-1)\n            current_token_i += 1\n\n        # backref can't be splitted\n        elif subw_backr > -1:\n            tokens.append(None)\n            backreferences.append(subword_to_token_map[subw_backr])\n            current_token_i += 1\n\n        # after a special token release\n        elif isinstance(tokens[-1], str) and rex_spc.match(tokens[-1]):\n            tokens.append(subtok)\n            backreferences.append(-1)\n            current_token_i += 1\n\n        # after a subtoken ':' (which should be followed by the rest of the edge) ignore tokenizer.INIT\n        # TODO: this is an ugly patch due to the fact that BART tokenizer splits after ':'\n        elif (tokens[-1] == ':') and rex_arg.match(subtok):\n            tokens[-1] = tokens[-1] + subtok[1:]\n\n        # current or prev is a control token\n        elif (is_pointer or is_rel or prev_is_pointer or prev_is_rel or is_bracket or subtok == '</s>') \\\n                and subtok != '-of':\n            tokens.append(subtok)\n            backreferences.append(-1)\n            current_token_i += 1\n\n        # very ugly patch for some cases in which tokenizer.INIT is not in the following token to the edge\n        elif isinstance(tokens[-1], str) and tokens[-1].startswith(':') and tokens[-1][-1].isdigit() and (\n                subtok != '-of'):\n            tokens.append(subtok)\n            backreferences.append(-1)\n            current_token_i += 1\n\n        # in any other case attach to the previous\n        else:\n            tokens[-1] = tokens[-1] + subtok\n\n        prev_is_pointer = is_pointer\n        prev_is_rel = is_rel\n\n    # strip INIT and fix byte-level\n    tokens = [tokenizer.convert_tokens_to_string(list(t)).lstrip() if isinstance(t, str) else t for t in tokens]\n    # tokens = [t.replace(tokenizer.INIT, '') if isinstance(t, str) else t for t in tokens]\n\n    # unks are substituted with thing\n    tokens = [t if t != '<unk>' else 'thing' for t in tokens]\n\n    old_tokens = tokens\n    old_backreferences = backreferences\n\n    # <lit> Barack Obama </lit> -> \"Barack Obama\"\n    tokens = []\n    backreferences = []\n    token_to_token_map = {}\n    start_search = 0\n    removed = 0\n    while True:\n        try:\n\n            lit_start = old_tokens.index('<lit>', start_search)\n            token_addition = old_tokens[start_search:lit_start]\n            for i, t in enumerate(token_addition, start=start_search):\n                token_to_token_map[i] = i - removed\n            tokens += token_addition\n\n            backreferences_addition = [token_to_token_map[b] if b > -1 else -1 for b in\n                                       old_backreferences[start_search:lit_start]]\n            backreferences += backreferences_addition\n\n            lit_end = min(lit_start + 2, len(old_tokens) - 1)\n\n            while lit_end < len(old_tokens):\n                old_tok = old_tokens[lit_end]\n\n                if isinstance(old_tok, str) and (\n                        (old_tok.startswith(':') and len(old_tok) > 3) or (old_tok == '<stop>')):\n                    res_tok = old_tokens[lit_start + 1:lit_end]\n                    for i in range(lit_start, lit_end):\n                        token_to_token_map[i] = len(tokens)\n\n                    # Remove possible wrong None\n                    res = old_tokens[lit_start + 1:lit_end]\n                    res = [str(r) for r in res if r is not None]\n                    res = '\"' + '_'.join(res) + '\"'\n\n                    removed += len(res_tok)\n                    start_search = lit_end\n                    tokens += [res, old_tok]\n                    backreferences += [-1, -1]\n                    break\n\n                elif old_tok == '</lit>':\n                    res_tok = old_tokens[lit_start + 1:lit_end]\n                    for i in range(lit_start, lit_end + 1):\n                        token_to_token_map[i] = len(tokens)\n\n                    # Remove possible wrong None\n                    res = old_tokens[lit_start + 1:lit_end]\n                    res = [str(r) for r in res if r is not None]\n                    res = '\"' + '_'.join(res) + '\"'\n\n                    removed += len(res_tok) + 1\n                    start_search = lit_end + 1\n                    tokens.append(res)\n                    backreferences.append(-1)\n                    break\n\n                else:\n                    lit_end += 1\n                    start_search = lit_end\n\n        except ValueError:\n            token_addition = old_tokens[start_search:]\n            for i, t in enumerate(token_addition, start=start_search):\n                token_to_token_map[i] = i - removed\n            backreferences_addition = [token_to_token_map[b] if b > -1 else b for b in\n                                       old_backreferences[start_search:]]\n            tokens += token_addition\n            backreferences += backreferences_addition\n            break\n\n    tokens = [token_processing(t) for t in tokens]\n\n    shift = 0\n    if len(tokens) > 1 and tokens[1] == '<s>':\n        shift = 1\n\n    tokens = tokens[shift:]\n    backreferences = [b if b == -1 else b - shift for b in backreferences[shift:]]\n\n    if tokens and tokens[-1] == '</s>':\n        tokens.pop()\n        backreferences.pop()\n\n    return tokens, backreferences\n\n\ndef index_of(element, iterable, default=None, start=None, end=None):\n    if not callable(element):\n        def check(x):\n            return element == x\n    else:\n        check = element\n    if start is None:\n        start = 0\n    if end is None:\n        end = len(iterable)\n    item = start\n    while item < end:\n        if check(iterable[item]):\n            return item\n        item += 1\n    return default\n\n\ndef separate_edges_nodes(edges_nodes_slice, *other):\n    is_arg = lambda x: isinstance(x, str) and x.startswith(':')\n    start = 0\n    edges = []\n    nodes = []\n    l = len(edges_nodes_slice)\n    while start < l:\n        edge_index = index_of(\n            is_arg,\n            edges_nodes_slice,\n            start=start)\n        if edge_index is None or edge_index == (l - 1):\n            break\n        if is_arg(edges_nodes_slice[edge_index + 1]):\n            start = edge_index + 1\n            continue\n        edges.append(edge_index)\n        nodes.append(edge_index + 1)\n        start = edge_index + 2\n    ret = []\n    for oth in other:\n        edges_oth = [oth[i] for i in edges]\n        nodes_oth = [oth[i] for i in nodes]\n        ret.append((edges_oth, nodes_oth))\n    return ret\n\n\ndef _split_name_ops(graph):\n    # identify name triples\n    name_vars = {}\n    for i, (v1, rel, v2) in enumerate(graph.triples):\n        if rel == ':instance' and v2 == 'name':\n            name_vars[v1] = 1\n\n    # check if they have ops\n    name_vars_to_ops = defaultdict(list)\n    for i, (v1, rel, v2) in enumerate(graph.triples):\n        if v1 in name_vars and rel.startswith(':op'):\n            name_vars_to_ops[v1].append((i, rel, v2.strip('\"')))\n\n    triples = graph.triples.copy()\n    for nv, ops in name_vars_to_ops.items():\n        ops = sorted(ops, key=lambda x: int(x[1][3:]))\n        idx, _, lits = zip(*ops)\n        for i in idx:\n            triples[i] = None\n\n        lits = ['\"' + l + '\"' for lit in lits for l in lit.split('_')]\n\n        tt = []\n        for i, l in enumerate(lits, start=1):\n            rel = ':op' + str(i)\n            tt.append(penman.Triple(nv, rel, l))\n\n        triples[min(idx)] = tt\n\n    triples = [t if isinstance(t, list) else [t] for t in triples if t is not None]\n    triples = [t for tt in triples for t in tt]\n\n    graph_ = penman.Graph(triples)\n    graph_.metadata = graph.metadata\n    return graph_\n\n\ndef _reconstruct_graph_from_nodes(nodes, backreferences):\n    triples = []\n    triples_added = set()\n\n    variable2index = {}\n    index2variable = {}\n    start_index = 0\n\n    cnt = defaultdict(Counter)\n\n    while start_index < len(nodes):\n        stop_index = index_of('<stop>', nodes, default=len(nodes) + 1, start=start_index)\n        old_start_index = start_index\n        start_index = stop_index + 1\n\n        src_node, src_backr = nodes[old_start_index], backreferences[old_start_index]\n\n        if src_node == '<stop>':\n            continue\n\n        trg_nodes_edges = nodes[old_start_index:stop_index]\n        trg_nodes_edges_backr = backreferences[old_start_index:stop_index]\n        trg_nodes_edges_indices = list(range(old_start_index, stop_index))\n\n        if isinstance(src_node, str):\n            if src_node in ('<s>', '</s>', '<stop>'):\n                continue\n            elif ('/' in src_node) or (':' in src_node) or ('(' in src_node) or (')' in src_node):\n                src_node = 'thing'\n\n        if src_node is not None:\n            src_node = str(src_node)\n            src_var = src_node[0].lower()\n            if not src_var not in 'abcdefghijklmnopqrstuvwxyz':\n                src_var = 'x'\n            # src_var = f'{src_var}_{len(variable2index)}'\n            src_var = f'{src_var}{len(variable2index)}'\n            src_var_i = old_start_index\n            variable2index[src_var] = src_var_i\n            index2variable[src_var_i] = src_var\n            triple = penman.Triple(src_var, ':instance', src_node)\n            if triple not in triples_added:\n                triples.append(triple)\n                triples_added.add(triple)\n        else:\n            if src_backr in index2variable:\n                src_var = index2variable[src_backr]\n        # more resilient logic here\n        (trg_edges, trg_nodes), (_, trg_nodes_backr), (_, trg_nodes_indices) = \\\n            separate_edges_nodes(\n                trg_nodes_edges,\n                trg_nodes_edges,\n                trg_nodes_edges_backr,\n                trg_nodes_edges_indices)\n\n        for n, e, nb, ni in zip(trg_nodes, trg_edges, trg_nodes_backr, trg_nodes_indices):\n\n            if isinstance(n, str) and n.startswith(':'):\n                continue\n            if isinstance(n, str) and n.startswith('<') and n.endswith('>'):\n                continue\n            if e == ':li':\n                pass\n            elif len(e) < 4 or (not e.startswith(':')):\n                continue\n\n            # same edge more than once\n            num = cnt[src_var][e]\n            # num = 0\n            if num:\n\n                if e.startswith(':op') or e.startswith(':snt'):\n                    continue\n                # elif e.startswith(':ARG'):\n                #    continue\n                elif num > 3:\n                    continue\n\n            if n is None:\n                if nb not in index2variable:\n                    continue\n                trg_var = index2variable[nb]\n                trg = trg_var\n            elif e == ':mode':\n                trg = n\n            elif (not isinstance(n, str)) or re.match(r\"^[+-]?\\d+\\.?\\d*$\", n) or (n == '-') or (n == '+'):\n                trg = str(n)\n            elif (n.startswith('\"') and n.endswith('\"') and len(n) > 2):\n                trg = '\"' + n.replace('\"', '') + '\"'\n            elif ('/' in n) or (':' in n) or ('(' in n) or (')' in n) or ('=' in n):\n                trg = f'\"{n}\"'\n            elif n == '\"':\n                continue\n            elif (n.startswith('\"') and (not n.endswith('\"'))) or (not n.startswith('\"') and (n.endswith('\"'))) or (\n                    '\"' in n):\n                trg = '\"' + n.replace('\"', '') + '\"'\n            else:\n                trg_var = n[0].lower()\n                if trg_var not in 'abcdefghijklmnopqrstuvwxyz':\n                    trg_var = 'x'\n                # trg_var = f'{trg_var}_{len(variable2index)}'\n                trg_var = f'{trg_var}{len(variable2index)}'\n                trg_var_i = ni\n                variable2index[trg_var] = trg_var_i\n                index2variable[trg_var_i] = trg_var\n                triple = penman.Triple(trg_var, ':instance', n)\n                if triple not in triples_added:\n                    triples.append(triple)\n                    triples_added.add(triple)\n                trg = trg_var\n\n            triple = penman.Triple(src_var, e, trg)\n            if triple not in triples_added:\n                triples.append(triple)\n                triples_added.add(triple)\n\n            cnt[src_var][e] += 1\n\n    return penman.Graph(triples)\n\n\ndef build_graph(nodes, backreferences, restore_name_ops=False):\n    graph = _reconstruct_graph_from_nodes(nodes, backreferences)\n    if restore_name_ops:\n        graph = _split_name_ops(graph)\n    return graph\n\n\nclass ParsedStatus(enum.Enum):\n    OK = 0\n    FIXED = 1\n    BACKOFF = 2\n\n\ndef connect_graph_if_not_connected(graph):\n    try:\n        encoded = pm_encode(graph)\n        return graph, ParsedStatus.OK\n    except:\n        pass\n\n    nxgraph = nx.MultiGraph()\n    variables = graph.variables()\n    for v1, _, v2 in graph.triples:\n        if v1 in variables and v2 in variables:\n            nxgraph.add_edge(v1, v2)\n        elif v1 in variables:\n            nxgraph.add_edge(v1, v1)\n\n    triples = graph.triples.copy()\n    new_triples = []\n    addition = f'a{len(variables) + 1}'\n    triples.append(penman.Triple(addition, ':instance', 'and'))\n    for i, conn_set in enumerate(nx.connected_components(nxgraph), start=1):\n        edge = f':op{i}'\n        conn_set = sorted(conn_set, key=lambda x: int(x[1:]))\n        conn_set = [c for c in conn_set if c in variables]\n        node = conn_set[0]\n        new_triples.append(penman.Triple(addition, edge, node))\n    triples = new_triples + triples\n    metadata = graph.metadata\n    graph = penman.Graph(triples)\n    graph.metadata.update(metadata)\n    pm_encode(graph)\n\n    return graph, ParsedStatus.FIXED\n\n\ndef restore_backreferences_from_pointers(nodes):\n    new_nodes, new_backreferences = [], []\n    prev_pointer = None\n    pointer2i = {}\n    for n in nodes:\n        is_pointer = isinstance(n, str) and n.startswith('<pointer:') and n.endswith('>')\n\n        if not is_pointer:\n            if prev_pointer is not None:\n                if prev_pointer in pointer2i:\n                    new_nodes.append(None)\n                    new_backreferences.append(pointer2i[prev_pointer])\n                    new_nodes.append(n)\n                    new_backreferences.append(-1)\n\n                else:\n                    pointer2i[prev_pointer] = len(new_nodes)\n                    new_nodes.append(n)\n                    new_backreferences.append(-1)\n            else:\n                new_nodes.append(n)\n                new_backreferences.append(-1)\n\n            prev_pointer = None\n        else:\n            prev_pointer = n\n    return new_nodes, new_backreferences\n"
  },
  {
    "path": "hanlp/components/amr/seq2seq/dataset/tokenization_bart.py",
    "content": "import copy\nimport sys\nfrom typing import Set, Iterable\n\nimport penman\nimport regex as re\nimport torch\nfrom transformers import BartTokenizer\n\nfrom . import postprocessing\nfrom .linearization import AMRTokens, AMRLinearizer\nfrom .penman import pm_encode\n\n\nclass AMRBartTokenizer(BartTokenizer):\n    ADDITIONAL = [\n        AMRTokens.PNTR_N,\n        AMRTokens.STOP_N,\n        AMRTokens.LIT_START,\n        AMRTokens.LIT_END,\n        AMRTokens.BACKR_SRC_N,\n        AMRTokens.BACKR_TRG_N, ]\n\n    def __init__(self, *args, use_pointer_tokens=False, collapse_name_ops=False, INIT='Ġ', **kwargs):\n        super().__init__(*args, **kwargs)\n        self.INIT = INIT\n        self.patterns = re.compile(\n            r\"\"\" ?<[a-z]+:?\\d*>| ?:[^\\s]+|'s|'t|'re|'ve|'m|'ll|'d| ?\\p{L}+| ?\\p{N}+| ?[^\\s\\p{L}\\p{N}]+|\\s+(?!\\S)|\\s+\"\"\")\n        self.linearizer = AMRLinearizer(use_pointer_tokens=use_pointer_tokens, collapse_name_ops=collapse_name_ops)\n        self.use_pointer_tokens = use_pointer_tokens\n        self.collapse_name_ops = collapse_name_ops\n        self.recategorizations = set()\n        self.modified = 0\n\n    @classmethod\n    def from_pretrained(cls, pretrained_model_path, additional_tokens: Iterable[str] = None,\n                        recategorization_tokens: Iterable[str] = None,\n                        *args, **kwargs):\n        inst = super().from_pretrained(pretrained_model_path, *args, **kwargs)\n        inst.init_amr_vocabulary(additions=additional_tokens, recategorization_tokens=recategorization_tokens)\n        return inst\n\n    def init_amr_vocabulary(self, additions: Set[str] = None, recategorization_tokens: Iterable[str] = None):\n        for tok in self.all_special_tokens:\n            ntok = self.INIT + tok\n            i = self.encoder[tok]\n            self.decoder[i] = ntok\n            del self.encoder[tok]\n            self.encoder[ntok] = i\n\n        tokens = []\n        if additions:\n            tokens.extend(additions)\n\n        if recategorization_tokens:\n            for tok in recategorization_tokens:\n                if not tok.startswith('_'):\n                    self.recategorizations.add(tok)\n                tokens.append(tok)\n\n        if self.use_pointer_tokens:\n            for cnt in range(512):\n                tokens.append(f\"<pointer:{cnt}>\")\n\n        tokens += self.ADDITIONAL\n        tokens = [self.INIT + t if t[0] not in ('_', '-') else t for t in tokens]\n        tokens = [t for t in tokens if t not in self.encoder]\n        self.old_enc_size = old_enc_size = len(self.encoder)\n        for i, t in enumerate(tokens, start=old_enc_size):\n            self.encoder[t] = i\n\n        self.encoder = {k: i for i, (k, v) in enumerate(sorted(self.encoder.items(), key=lambda x: x[1]))}\n        self.decoder = {v: k for k, v in sorted(self.encoder.items(), key=lambda x: x[1])}\n        self.modified = len(tokens)\n\n        self.bos_token = self.INIT + self.bos_token\n        self.pad_token = self.INIT + self.pad_token\n        self.eos_token = self.INIT + self.eos_token\n        self.unk_token = self.INIT + self.unk_token\n\n    def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None):\n        output = [self.bos_token_id] + token_ids_0 + [self.eos_token_id]\n        if token_ids_1 is None:\n            return output\n        return output + [self.eos_token_id] + token_ids_1 + [self.eos_token_id]\n\n    def _tokenize(self, text):\n        \"\"\" Tokenize a string. Modified in order to handle sentences with recategorization pointers\"\"\"\n        bpe_tokens = []\n        for tok_span in text.lstrip().split(' '):\n            tok_span = tok_span.strip()\n            recats = tok_span.rsplit('_', 1)\n            if len(recats) == 2 and recats[0] in self.recategorizations and ('_' + recats[1]) in self.encoder:\n                bpe_tokens.extend([self.INIT + recats[0], '_' + recats[1]])\n            else:\n                for token in re.findall(self.pat, ' ' + tok_span):\n                    token = \"\".join(\n                        self.byte_encoder[b] for b in token.encode(\"utf-8\")\n                    )  # Maps all our bytes to unicode strings, avoiding controle tokens of the BPE (spaces in our case)\n                    bpe_tokens.extend(bpe_token for bpe_token in self.bpe(token).split(\" \"))\n\n        return bpe_tokens\n\n    def _tok_bpe(self, token, add_space=True):\n        # if add_space:\n        #     token = ' ' + token.lstrip()\n        tokk = []\n        tok = token.strip()\n        recats = tok.rsplit('_', 1)\n        if len(recats) == 2 and recats[0] in self.recategorizations and ('_' + recats[1]) in self.encoder:\n            tokk.extend([self.INIT + recats[0], '_' + recats[1]])\n        else:\n            for tok in self.patterns.findall(' ' + token):\n                tok = \"\".join(\n                    self.byte_encoder[b] for b in tok.encode(\"utf-8\"))\n                toks = self.bpe(tok).split(' ')\n                tokk.extend(toks)\n        return tokk\n\n    def _get_nodes_and_backreferences(self, graph):\n        lin = self.linearizer.linearize(graph)\n        linearized_nodes, backreferences = lin.nodes, lin.backreferences\n        return linearized_nodes, backreferences\n\n    def tokenize_amr(self, graph):\n        linearized_nodes, backreferences = self._get_nodes_and_backreferences(graph)\n\n        bpe_tokens = []\n        bpe_backreferences = []\n        counter = 0\n\n        for i, (backr, tokk) in enumerate(zip(backreferences, linearized_nodes)):\n            is_in_enc = self.INIT + tokk in self.encoder\n            is_rel = tokk.startswith(':') and len(tokk) > 1\n            is_spc = tokk.startswith('<') and tokk.endswith('>')\n            is_of = tokk.startswith(':') and tokk.endswith('-of')\n            is_frame = re.match(r'.+-\\d\\d', tokk) is not None\n\n            if tokk.startswith('\"') and tokk.endswith('\"'):\n                tokk = tokk[1:-1].replace('_', ' ')\n                bpe_toks = [self.INIT + AMRTokens.LIT_START]\n                bpe_toks += self._tok_bpe(tokk, add_space=True)\n                bpe_toks.append(self.INIT + AMRTokens.LIT_END)\n\n            elif (is_rel or is_spc or is_frame or is_of):\n                if is_in_enc:\n                    bpe_toks = [self.INIT + tokk]\n                elif is_frame:\n                    bpe_toks = self._tok_bpe(tokk[:-3], add_space=True) + [tokk[-3:]]\n                elif is_of:\n                    rel = tokk[:-3]\n                    if self.INIT + rel in self.encoder:\n                        bpe_toks = [self.INIT + rel, '-of']\n                    else:\n                        bpe_toks = [self.INIT + ':'] + self._tok_bpe(rel[1:], add_space=True) + ['-of']\n                elif is_rel:\n                    bpe_toks = [self.INIT + ':'] + self._tok_bpe(tokk[1:], add_space=True)\n                else:\n                    raise\n\n            else:\n                if is_in_enc:\n                    bpe_toks = [self.INIT + tokk]\n                else:\n                    bpe_toks = self._tok_bpe(tokk, add_space=True)\n\n            bpe_tokens.append(bpe_toks)\n\n            if i == backr:\n                bpe_backr = list(range(counter, counter + len(bpe_toks)))\n                counter += len(bpe_toks)\n                bpe_backreferences.append(bpe_backr)\n            else:\n                bpe_backreferences.append(bpe_backreferences[backr][0:1])\n                counter += 1\n        bpe_tokens = [b for bb in bpe_tokens for b in bb]\n        bpe_token_ids = [self.encoder.get(b, self.unk_token_id) for b in bpe_tokens]\n        bpe_backreferences = [b for bb in bpe_backreferences for b in bb]\n        return bpe_tokens, bpe_token_ids, bpe_backreferences\n\n    def batch_encode_sentences(self, sentences, device=torch.device('cpu')):\n        sentences = [s for s in sentences]\n        extra = {'sentences': sentences}\n        batch = super().batch_encode_plus(sentences, return_tensors='pt', pad_to_max_length=True)\n        batch = {k: v.to(device) for k, v in batch.items()}\n        return batch, extra\n\n    def linearize(self, graph):\n        shift = len(self.encoder)\n        tokens, token_ids, backreferences = self.tokenize_amr(graph)\n        extra = {'linearized_graphs': tokens, 'graphs': graph}\n        token_uni_ids = \\\n            [idx if i == b else b + shift for i, (idx, b) in enumerate(zip(token_ids, backreferences))]\n        if token_uni_ids[-1] != (self.INIT + AMRTokens.EOS_N):\n            tokens.append(self.INIT + AMRTokens.EOS_N)\n            token_ids.append(self.eos_token_id)\n            token_uni_ids.append(self.eos_token_id)\n            backreferences.append(len(backreferences))\n        return token_uni_ids, extra\n\n    def batch_encode_graphs(self, graphs, device=torch.device('cpu')):\n        linearized, extras = zip(*[self.linearize(g) for g in graphs])\n        return self.batch_encode_graphs_from_linearized(linearized, extras, device=device)\n\n    def batch_encode_graphs_from_linearized(self, linearized, extras=None, device=torch.device('cpu')):\n        if extras is not None:\n            batch_extra = {'linearized_graphs': [], 'graphs': []}\n            for extra in extras:\n                batch_extra['graphs'].append(extra['graphs'])\n                batch_extra['linearized_graphs'].append(extra['linearized_graphs'])\n        else:\n            batch_extra = {}\n        maxlen = 0\n        batch = []\n        for token_uni_ids in linearized:\n            maxlen = max(len(token_uni_ids), maxlen)\n            batch.append(token_uni_ids)\n        batch = [x + [self.pad_token_id] * (maxlen - len(x)) for x in batch]\n        batch = torch.tensor(batch).to(device)\n        batch = {'decoder_input_ids': batch[:, :-1], 'lm_labels': batch[:, 1:]}\n        return batch, batch_extra\n\n    def decode_amr(self, tokens, restore_name_ops=False):\n        try:\n            nodes, backreferences = postprocessing.decode_into_node_and_backreferences(tokens, self)\n        except Exception as e:\n            print('Decoding failure:', file=sys.stderr)\n            print(e, file=sys.stderr)\n            return postprocessing.BACKOFF, postprocessing.ParsedStatus.BACKOFF, (None, None)\n        if self.use_pointer_tokens:\n            nodes, backreferences = postprocessing.restore_backreferences_from_pointers(nodes)\n        try:\n            graph_ = graph = postprocessing.build_graph(nodes, backreferences, restore_name_ops=restore_name_ops)\n        except Exception as e:\n            print('Building failure:', file=sys.stderr)\n            print(nodes, file=sys.stderr)\n            print(backreferences, file=sys.stderr)\n            print(e, file=sys.stderr)\n            return postprocessing.BACKOFF, postprocessing.ParsedStatus.BACKOFF, (None, None)\n        try:\n            graph, status = postprocessing.connect_graph_if_not_connected(graph)\n            if status == postprocessing.ParsedStatus.BACKOFF:\n                print('Reconnection 1 failure:')\n                print(nodes, file=sys.stderr)\n                print(backreferences, file=sys.stderr)\n                print(graph_, file=sys.stderr)\n            return graph, status, (nodes, backreferences)\n        except Exception as e:\n            print('Reconnction 2 failure:', file=sys.stderr)\n            print(e, file=sys.stderr)\n            print(nodes, file=sys.stderr)\n            print(backreferences, file=sys.stderr)\n            print(graph_, file=sys.stderr)\n            return postprocessing.BACKOFF, postprocessing.ParsedStatus.BACKOFF, (nodes, backreferences)\n\n\nclass PENMANBartTokenizer(AMRBartTokenizer):\n\n    def __init__(self, *args, raw_graph=False, **kwargs):\n        super().__init__(*args, **kwargs)\n        self.linearizer = None\n        self.remove_pars = False\n        self.raw_graph = raw_graph\n\n    def _tokenize_encoded_graph(self, encoded):\n        linearized = re.sub(r\"(\\\".+?\\\")\", r' \\1 ', encoded)\n        pieces = []\n        for piece in linearized.split():\n            if piece.startswith('\"') and piece.endswith('\"'):\n                pieces.append(piece)\n            else:\n                piece = piece.replace('(', ' ( ')\n                piece = piece.replace(')', ' ) ')\n                piece = piece.replace(':', ' :')\n                piece = piece.replace('/', ' / ')\n                piece = piece.strip()\n                pieces.append(piece)\n        linearized = re.sub(r'\\s+', ' ', ' '.join(pieces)).strip()\n        linearized_nodes = [AMRTokens.BOS_N] + linearized.split(' ')\n        return linearized_nodes\n\n    def tokenize_amr(self, graph):\n        if self.raw_graph:\n            graph_ = copy.deepcopy(graph)\n            graph_.metadata = {}\n            linearized = penman.encode(graph_)\n            linearized = re.sub(r\"\\s+\", ' ', linearized)\n            bpe_tokens = [self.bos_token] + self._tokenize(linearized)[:1022]\n            bpe_token_ids = [self.encoder.get(b, self.unk_token_id) for b in bpe_tokens]\n            bpe_backreferences = list(range(len(bpe_token_ids)))\n            return bpe_tokens, bpe_token_ids, bpe_backreferences\n        else:\n            return super().tokenize_amr(graph)\n\n    def _get_nodes_and_backreferences(self, graph):\n        graph_ = copy.deepcopy(graph)\n        graph_.metadata = {}\n        linearized = penman.encode(graph_)\n        linearized_nodes = self._tokenize_encoded_graph(linearized)\n\n        if self.use_pointer_tokens:\n            remap = {}\n            for i in range(1, len(linearized_nodes)):\n                nxt = linearized_nodes[i]\n                lst = linearized_nodes[i - 1]\n                if nxt == '/':\n                    remap[lst] = f'<pointer:{len(remap)}>'\n            i = 1\n            linearized_nodes_ = [linearized_nodes[0]]\n            while i < (len(linearized_nodes)):\n                nxt = linearized_nodes[i]\n                lst = linearized_nodes_[-1]\n                if nxt in remap:\n                    if lst == '(' and linearized_nodes[i + 1] == '/':\n                        nxt = remap[nxt]\n                        i += 1\n                    elif lst.startswith(':'):\n                        nxt = remap[nxt]\n                linearized_nodes_.append(nxt)\n                i += 1\n            linearized_nodes = linearized_nodes_\n            if self.remove_pars:\n                linearized_nodes = [n for n in linearized_nodes if n != '(']\n        backreferences = list(range(len(linearized_nodes)))\n        return linearized_nodes, backreferences\n\n    def _classify(self, node):\n        if not isinstance(node, str):\n            return \"CONST\"\n        elif node == 'i':\n            return \"I\"\n        elif re.match(r'^[a-z]\\d*$', node) is not None:\n            return \"VAR\"\n        elif node[0].isdigit():\n            return \"CONST\"\n        elif node.startswith('\"') and node.endswith('\"'):\n            return \"CONST\"\n        elif node in ('+', '-'):\n            return \"CONST\"\n        elif node == ':mode':\n            return 'MODE'\n        elif node.startswith(':'):\n            return \"EDGE\"\n        elif node in ['/', '(', ')']:\n            return node\n        elif node[0].isalpha():\n            for char in (',', ':', '/', '(', ')', '.', '!', '?', '\\\\'):\n                if char in node:\n                    return \"CONST\"\n            return \"INST\"\n        else:\n            return 'CONST'\n\n    def _fix_and_make_graph(self, nodes):\n\n        nodes_ = []\n        for n in nodes:\n            if isinstance(n, str):\n                if n.startswith('<') and n.endswith('>') and (not n.startswith('<pointer:')):\n                    pass\n                else:\n                    nodes_.append(n)\n            else:\n                nodes_.append(n)\n        nodes = nodes_\n\n        if self.use_pointer_tokens:\n\n            i = 0\n            nodes_ = []\n            while i < len(nodes):\n                nxt = nodes[i]\n                pst = None\n                if isinstance(nxt, str) and nxt.startswith('<pointer:'):\n                    e = nxt.find('>')\n                    if e != len(nxt) - 1:\n                        pst = nxt[e + 1:]\n                        nxt = nxt[:e + 1]\n                    nodes_.append(nxt)\n                    if pst is not None:\n                        nodes_.append(pst)\n                else:\n                    nodes_.append(nxt)\n                i += 1\n            nodes = nodes_\n\n            i = 1\n            nodes_ = [nodes[0]]\n            while i < len(nodes):\n                nxt = nodes[i]\n                if isinstance(nxt, str) and nxt.startswith('<pointer:'):\n                    nxt = 'z' + nxt[9:-1]\n                    fol = nodes[i + 1]\n                    # is not expansion\n                    if isinstance(fol, str) and (fol.startswith(':') or (fol == ')')):\n                        nodes_.append(nxt)\n                    else:\n                        if self.remove_pars:\n                            nodes_.append('(')\n                        else:\n                            if nodes_[-1] != '(':\n                                nodes_.append('(')\n                                # pass\n                        nodes_.append(nxt)\n                        nodes_.append('/')\n                else:\n                    nodes_.append(nxt)\n                i += 1\n            nodes = nodes_\n\n        i = 0\n        nodes_ = []\n        while i < (len(nodes) - 1):\n            if nodes[i] == ':':\n                nodes_.append(nodes[i] + nodes[i + 1])\n                i += 2\n                last = False\n            else:\n                nodes_.append(nodes[i])\n                i += 1\n                last = True\n        if last:\n            nodes_.append(nodes[-1])\n        nodes = nodes_\n\n        i = 0\n        nodes_ = []\n        while i < (len(nodes)):\n            if i < 2:\n                nodes_.append(nodes[i])\n                i += 1\n            elif nodes_[-2] == '/' and nodes[i] == '/':\n                i += 2\n            else:\n                nodes_.append(nodes[i])\n                i += 1\n        nodes = nodes_\n\n        i = 0\n        newvars = 0\n        variables = set()\n        remap = {}\n        nodes_ = []\n        while i < (len(nodes)):\n\n            next = nodes[i]\n\n            if next == '/':\n                last = nodes_[-1]\n                if last in variables:\n                    last_remap = f\"x{newvars + 1000}\"\n                    newvars += 1\n                    nodes_[-1] = last_remap\n                    remap[last] = last_remap\n                variables.add(last)\n                nodes_.append(next)\n\n            elif self._classify(next) == 'VAR' and next in remap and (i < len(nodes) - 1) and nodes[i + 1] != '/':\n                next = remap[next]\n                nodes_.append(next)\n\n            else:\n                nodes_.append(next)\n\n            i += 1\n\n        nodes = nodes_\n        pieces_ = []\n        open_cnt = 0\n        closed_cnt = 0\n        if nodes[0] != '(':\n            pieces_.append('(')\n            open_cnt += 1\n        for p in nodes:\n            if p == '(':\n                open_cnt += 1\n            elif p == ')':\n                closed_cnt += 1\n            pieces_.append(p)\n            if open_cnt == closed_cnt:\n                break\n        nodes = pieces_ + [')'] * (open_cnt - closed_cnt)\n\n        pieces = []\n        for piece in nodes:\n            if not pieces:\n                pieces.append('(')\n            else:\n                piece = str(piece)\n                if piece.startswith('\"') or piece.startswith('\"') or '\"' in piece.strip('\"'):\n                    piece = '\"' + piece.replace('\"', '') + '\"'\n\n                prev = self._classify(pieces[-1])\n                next = self._classify(piece)\n\n                if next == 'CONST':\n                    quote = False\n                    for char in (',', ':', '/', '(', ')', '.', '!', '?', '\\\\', '_', '='):\n                        if char in piece:\n                            quote = True\n                            break\n                    if quote:\n                        piece = '\"' + piece.strip('\"') + '\"'\n\n                if prev == '(':\n                    if next in ('VAR', 'I'):\n                        pieces.append(piece)\n                elif prev == ')':\n                    if next in (')', 'EDGE', 'MODE'):\n                        pieces.append(piece)\n                elif prev == 'VAR':\n                    if next in ('/', 'EDGE', 'MODE', ')'):\n                        pieces.append(piece)\n                elif prev == '/':\n                    if next in ('INST', 'I'):\n                        pieces.append(piece)\n                elif prev == 'INST':\n                    if next in (')', 'EDGE', 'MODE'):\n                        pieces.append(piece)\n                elif prev == 'I':\n                    if next in ('/', ')', 'EDGE', 'MODE'):\n                        pieces.append(piece)\n                elif prev == 'EDGE':\n                    if next in ('(', 'VAR', 'CONST', 'I'):\n                        pieces.append(piece)\n                    elif next == ')':\n                        pieces[-1] = piece\n                    elif next in ('EDGE', 'MODE'):\n                        pieces[-1] = piece\n                elif prev == 'MODE':\n                    if next == 'INST':\n                        pieces.append(piece)\n                elif prev == 'CONST':\n                    if next in (')', 'EDGE', 'MODE'):\n                        pieces.append(piece)\n\n        pieces_ = []\n        open_cnt = 0\n        closed_cnt = 0\n        if pieces[0] != '(':\n            pieces_.append('(')\n            open_cnt += 1\n        for p in pieces:\n            if p == '(':\n                open_cnt += 1\n            elif p == ')':\n                closed_cnt += 1\n            pieces_.append(p)\n            if open_cnt == closed_cnt:\n                break\n        pieces = pieces_ + [')'] * (open_cnt - closed_cnt)\n\n        linearized = re.sub(r'\\s+', ' ', ' '.join(pieces)).strip()\n\n        \"\"\"\n        line = linearized\n        # make sure parentheses match\n        # copied from https://github.com/RikVN/AMR/blob/master/restoreAMR/restore_amr.py\n        open_count = 0\n        close_count = 0\n        for i, c in enumerate(line):\n            if c == '(':\n                open_count += 1\n            elif c == ')':\n                close_count += 1\n            if open_count == close_count and open_count > 0:\n                line = line[:i].strip()\n                break\n        old_line = line\n        while True:\n            open_count = len(re.findall(r'\\(', line))\n            close_count = len(re.findall(r'\\)', line))\n            if open_count > close_count:\n                line += ')' * (open_count - close_count)\n            elif close_count > open_count:\n                for i in range(close_count - open_count):\n                    line = line.rstrip(')')\n                    line = line.rstrip(' ')\n            if old_line == line:\n                break\n            old_line = line\n        \"\"\"\n\n        graph = penman.decode(linearized + ' ')\n        triples = []\n        newvars = 2000\n        for triple in graph.triples:\n            x, rel, y = triple\n            if x is None:\n                pass\n            elif rel == ':instance' and y is None:\n                triples.append(penman.Triple(x, rel, 'thing'))\n            elif y is None:\n                var = f'x{newvars}'\n                newvars += 1\n                triples.append(penman.Triple(x, rel, var))\n                triples.append(penman.Triple(var, ':instance', 'thing'))\n            else:\n                triples.append(triple)\n        graph = penman.Graph(triples)\n        linearized = pm_encode(graph)\n\n        def fix_text(linearized=linearized):\n            n = 0\n\n            def _repl1(match):\n                nonlocal n\n                out = match.group(1) + match.group(2) + str(3000 + n) + ' / ' + match.group(2) + match.group(3)\n                n += 1\n                return out\n\n            linearized = re.sub(r'(\\(\\s?)([a-z])([^\\/:\\)]+[:\\)])', _repl1, linearized,\n                                flags=re.IGNORECASE | re.MULTILINE)\n\n            def _repl2(match):\n                return match.group(1)\n\n            linearized = re.sub(r'(\\(\\s*[a-z][\\d+]\\s*\\/\\s*[^\\s\\)\\(:\\/]+\\s*)((?:/\\s*[^\\s\\)\\(:\\/]+\\s*)+)', _repl2,\n                                linearized,\n                                flags=re.IGNORECASE | re.MULTILINE)\n\n            # adds a ':' to args w/o it\n            linearized = re.sub(r'([^:])(ARG)', r'\\1 :\\2', linearized)\n\n            # removes edges with no node\n            # linearized = re.sub(r':[^\\s\\)\\(:\\/]+?\\s*\\)', ')', linearized, flags=re.MULTILINE)\n\n            return linearized\n\n        linearized = fix_text(linearized)\n\n        g = penman.decode(linearized)\n        return g\n\n    def decode_amr(self, tokens, restore_name_ops=None):\n        try:\n            if self.raw_graph:\n                nodes = self._tokenize_encoded_graph(self.decode(tokens))\n                backreferences = list(range(len(nodes)))\n            else:\n                nodes, backreferences = postprocessing.decode_into_node_and_backreferences(tokens, self)\n            nodes_ = nodes\n        except Exception as e:\n            print('Decoding failure:', file=sys.stderr)\n            print(e, file=sys.stderr)\n            return postprocessing.BACKOFF, postprocessing.ParsedStatus.BACKOFF, (None, None)\n        try:\n            graph_ = graph = self._fix_and_make_graph(nodes)\n            if self.collapse_name_ops:\n                graph_ = graph = postprocessing._split_name_ops(graph)\n        except Exception as e:\n            print('Building failure:', file=sys.stderr)\n            print(nodes, file=sys.stderr)\n            print(backreferences, file=sys.stderr)\n            print(e, file=sys.stderr)\n            return postprocessing.BACKOFF, postprocessing.ParsedStatus.BACKOFF, (None, None)\n        try:\n            graph, status = postprocessing.connect_graph_if_not_connected(graph)\n            if status == postprocessing.ParsedStatus.BACKOFF:\n                print('Reconnection 1 failure:')\n                print(nodes, file=sys.stderr)\n                print(backreferences, file=sys.stderr)\n                print(graph_, file=sys.stderr)\n            return graph, status, (nodes_, backreferences)\n        except Exception as e:\n            print('Reconnction 2 failure:', file=sys.stderr)\n            print(e, file=sys.stderr)\n            print(nodes, file=sys.stderr)\n            print(backreferences, file=sys.stderr)\n            print(graph_, file=sys.stderr)\n            return postprocessing.BACKOFF, postprocessing.ParsedStatus.BACKOFF, (nodes_, backreferences)\n"
  },
  {
    "path": "hanlp/components/amr/seq2seq/dataset/tokenization_t5.py",
    "content": "import copy\nimport sys\nfrom typing import Set, Iterable, Dict\n\nimport penman\nimport regex as re\nimport torch\nimport traceback\nfrom transformers import T5Tokenizer, T5TokenizerFast\n\nfrom . import postprocessing\nfrom .linearization import AMRTokens, AMRLinearizer\nfrom .penman import pm_encode\n\n\nclass AMRT5Tokenizer(T5TokenizerFast):\n    ADDITIONAL = [\n        AMRTokens.PNTR_N,\n        AMRTokens.STOP_N,\n        AMRTokens.LIT_START,\n        AMRTokens.LIT_END,\n        AMRTokens.BACKR_SRC_N,\n        AMRTokens.BACKR_TRG_N, ]\n\n    def __init__(self, *args, use_pointer_tokens=False, collapse_name_ops=False, INIT='', **kwargs):\n        super().__init__(*args, **kwargs)\n        self.INIT = INIT\n        self.patterns = re.compile(\n            r\"\"\" ?<[a-z]+:?\\d*>| ?:[^\\s]+|'s|'t|'re|'ve|'m|'ll|'d| ?\\p{L}+| ?\\p{N}+| ?[^\\s\\p{L}\\p{N}]+|\\s+(?!\\S)|\\s+\"\"\")\n        self.linearizer = AMRLinearizer(use_pointer_tokens=use_pointer_tokens, collapse_name_ops=collapse_name_ops)\n        self.use_pointer_tokens = use_pointer_tokens\n        self.collapse_name_ops = collapse_name_ops\n        self.recategorizations = set()\n        self.modified = 0\n\n    @classmethod\n    def from_pretrained(cls, pretrained_model_path, additional_tokens: Iterable[str] = None,\n                        recategorization_tokens: Iterable[str] = None,\n                        *args, **kwargs):\n        inst = super().from_pretrained(pretrained_model_path, *args, **kwargs)\n        inst.init_amr_vocabulary(additions=additional_tokens, recategorization_tokens=recategorization_tokens)\n        return inst\n\n    def init_amr_vocabulary(self, additions: Set[str] = None, recategorization_tokens: Iterable[str] = None):\n        # T5 has no encoder but it's not a problem for Chinese\n        # for tok in self.all_special_tokens:\n        #     ntok = self.INIT + tok\n        #     i = self.encoder[tok]\n        #     self.decoder[i] = ntok\n        #     del self.encoder[tok]\n        #     self.encoder[ntok] = i\n\n        tokens = [AMRTokens.BOS_N]\n        if additions:\n            tokens.extend(additions)\n\n        if recategorization_tokens:\n            for tok in recategorization_tokens:\n                if not tok.startswith('_'):\n                    self.recategorizations.add(tok)\n                tokens.append(tok)\n\n        if self.use_pointer_tokens:\n            for cnt in range(512):\n                tokens.append(f\"<pointer:{cnt}>\")\n\n        tokens += self.ADDITIONAL\n        tokens = [self.INIT + t if t[0] not in ('_', '-') else t for t in tokens]\n        self.old_enc_size = len(self)\n        self.add_tokens(tokens)\n        self.modified = len(tokens)\n\n    def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None):\n        output = [self.bos_token_id] + token_ids_0 + [self.eos_token_id]\n        if token_ids_1 is None:\n            return output\n        return output + [self.eos_token_id] + token_ids_1 + [self.eos_token_id]\n\n    def _tokenize(self, text):\n        \"\"\" Tokenize a string. Modified in order to handle sentences with recategorization pointers\"\"\"\n        bpe_tokens = []\n        for tok_span in text.lstrip().split(' '):\n            tok_span = tok_span.strip()\n            recats = tok_span.rsplit('_', 1)\n            if len(recats) == 2 and recats[0] in self.recategorizations and ('_' + recats[1]) in self.encoder:\n                bpe_tokens.extend([self.INIT + recats[0], '_' + recats[1]])\n            else:\n                for token in re.findall(self.pat, ' ' + tok_span):\n                    token = \"\".join(\n                        self.byte_encoder[b] for b in token.encode(\"utf-8\")\n                    )  # Maps all our bytes to unicode strings, avoiding controle tokens of the BPE (spaces in our case)\n                    bpe_tokens.extend(bpe_token for bpe_token in self.bpe(token).split(\" \"))\n\n        return bpe_tokens\n\n    def _tok_bpe(self, token, add_space=True):\n        # if add_space:\n        #     token = ' ' + token.lstrip()\n        tokk = []\n        tok = token.strip()\n        recats = tok.rsplit('_', 1)\n        if len(recats) == 2 and recats[0] in self.recategorizations and ('_' + recats[1]) in self.encoder:\n            tokk.extend([self.INIT + recats[0], '_' + recats[1]])\n        else:\n            for tok in self.patterns.findall(token):\n                tokk.extend(self.tokenize(tok))\n        return tokk\n\n    def _get_nodes_and_backreferences(self, graph):\n        lin = self.linearizer.linearize(graph)\n        linearized_nodes, backreferences = lin.nodes, lin.backreferences\n        return linearized_nodes, backreferences\n\n    def tokenize_amr(self, graph):\n        linearized_nodes, backreferences = self._get_nodes_and_backreferences(graph)\n\n        bpe_tokens = []\n        bpe_backreferences = []\n        counter = 0\n\n        encoder = self.encoder\n        for i, (backr, tokk) in enumerate(zip(backreferences, linearized_nodes)):\n            is_in_enc = self.INIT + tokk in encoder\n            is_rel = tokk.startswith(':') and len(tokk) > 1\n            is_spc = tokk.startswith('<') and tokk.endswith('>')\n            is_of = tokk.startswith(':') and tokk.endswith('-of')\n            is_frame = re.match(r'.+-\\d\\d', tokk) is not None\n\n            if tokk.startswith('\"') and tokk.endswith('\"'):\n                tokk = tokk[1:-1].replace('_', ' ')\n                bpe_toks = [self.INIT + AMRTokens.LIT_START]\n                bpe_toks += self._tok_bpe(tokk, add_space=True)\n                bpe_toks.append(self.INIT + AMRTokens.LIT_END)\n\n            elif (is_rel or is_spc or is_frame or is_of):\n                if is_in_enc:\n                    bpe_toks = [self.INIT + tokk]\n                elif is_frame:\n                    bpe_toks = self._tok_bpe(tokk[:-3], add_space=True) + [tokk[-3:]]\n                elif is_of:\n                    rel = tokk[:-3]\n                    if self.INIT + rel in encoder:\n                        bpe_toks = [self.INIT + rel, '-of']\n                    else:\n                        bpe_toks = [self.INIT + ':'] + self._tok_bpe(rel[1:], add_space=True) + ['-of']\n                elif is_rel:\n                    bpe_toks = [self.INIT + ':'] + self._tok_bpe(tokk[1:], add_space=True)\n                else:\n                    raise\n\n            else:\n                if is_in_enc:\n                    bpe_toks = [self.INIT + tokk]\n                else:\n                    bpe_toks = self._tok_bpe(tokk, add_space=True)\n\n            bpe_tokens.append(bpe_toks)\n\n            if i == backr:\n                bpe_backr = list(range(counter, counter + len(bpe_toks)))\n                counter += len(bpe_toks)\n                bpe_backreferences.append(bpe_backr)\n            else:\n                bpe_backreferences.append(bpe_backreferences[backr][0:1])\n                counter += 1\n        bpe_tokens = [b for bb in bpe_tokens for b in bb]\n        bpe_token_ids = self.convert_tokens_to_ids(bpe_tokens)\n        bpe_backreferences = [b for bb in bpe_backreferences for b in bb]\n        return bpe_tokens, bpe_token_ids, bpe_backreferences\n\n    def batch_encode_sentences(self, sentences, device=torch.device('cpu')):\n        sentences = [s for s in sentences]\n        extra = {'sentences': sentences}\n        batch = super().batch_encode_plus(sentences, return_tensors='pt', pad_to_max_length=True)\n        batch = {k: v.to(device) for k, v in batch.items()}\n        return batch, extra\n\n    def linearize(self, graph):\n        shift = len(self)\n        tokens, token_ids, backreferences = self.tokenize_amr(graph)\n        extra = {'linearized_graphs': tokens, 'graphs': graph}\n        token_uni_ids = \\\n            [idx if i == b else b + shift for i, (idx, b) in enumerate(zip(token_ids, backreferences))]\n        if token_uni_ids[-1] != (self.INIT + AMRTokens.EOS_N):\n            tokens.append(self.INIT + AMRTokens.EOS_N)\n            token_ids.append(self.eos_token_id)\n            token_uni_ids.append(self.eos_token_id)\n            backreferences.append(len(backreferences))\n        return token_uni_ids, extra\n\n    def batch_encode_graphs(self, graphs, device=torch.device('cpu')):\n        linearized, extras = zip(*[self.linearize(g) for g in graphs])\n        return self.batch_encode_graphs_from_linearized(linearized, extras, device=device)\n\n    def batch_encode_graphs_from_linearized(self, linearized, extras=None, device=torch.device('cpu')):\n        if extras is not None:\n            batch_extra = {'linearized_graphs': [], 'graphs': []}\n            for extra in extras:\n                batch_extra['graphs'].append(extra['graphs'])\n                batch_extra['linearized_graphs'].append(extra['linearized_graphs'])\n        else:\n            batch_extra = {}\n        maxlen = 0\n        batch = []\n        for token_uni_ids in linearized:\n            maxlen = max(len(token_uni_ids), maxlen)\n            batch.append(token_uni_ids)\n        batch = [x + [self.pad_token_id] * (maxlen - len(x)) for x in batch]\n        batch = torch.tensor(batch).to(device)\n        batch = {'decoder_input_ids': batch[:, :-1], 'lm_labels': batch[:, 1:]}\n        return batch, batch_extra\n\n    def decode_amr(self, tokens, restore_name_ops=False):\n        try:\n            nodes, backreferences = postprocessing.decode_into_node_and_backreferences(tokens, self)\n        except Exception as e:\n            print('Decoding failure:', file=sys.stderr)\n            traceback.print_exc()\n            return postprocessing.BACKOFF, postprocessing.ParsedStatus.BACKOFF, (None, None)\n        if self.use_pointer_tokens:\n            nodes, backreferences = postprocessing.restore_backreferences_from_pointers(nodes)\n        try:\n            graph_ = graph = postprocessing.build_graph(nodes, backreferences, restore_name_ops=restore_name_ops)\n        except Exception as e:\n            print('Building failure:', file=sys.stderr)\n            traceback.print_exc()\n            print(nodes, file=sys.stderr)\n            print(backreferences, file=sys.stderr)\n            print(e, file=sys.stderr)\n            return postprocessing.BACKOFF, postprocessing.ParsedStatus.BACKOFF, (None, None)\n        try:\n            graph, status = postprocessing.connect_graph_if_not_connected(graph)\n            if status == postprocessing.ParsedStatus.BACKOFF:\n                print('Reconnection 1 failure:')\n                print(nodes, file=sys.stderr)\n                print(backreferences, file=sys.stderr)\n                print(graph_, file=sys.stderr)\n            return graph, status, (nodes, backreferences)\n        except Exception as e:\n            print('Reconnction 2 failure:', file=sys.stderr)\n            traceback.print_exc()\n            print(nodes, file=sys.stderr)\n            print(backreferences, file=sys.stderr)\n            print(graph_, file=sys.stderr)\n            return postprocessing.BACKOFF, postprocessing.ParsedStatus.BACKOFF, (nodes, backreferences)\n\n\nclass PENMANT5Tokenizer(AMRT5Tokenizer):\n\n    def __init__(self, *args, raw_graph=False, **kwargs):\n        super().__init__(*args, **kwargs)\n        self.linearizer = None\n        self.remove_pars = False\n        self.raw_graph = raw_graph\n\n    def _tokenize_encoded_graph(self, encoded):\n        linearized = re.sub(r\"(\\\".+?\\\")\", r' \\1 ', encoded)\n        pieces = []\n        for piece in linearized.split():\n            if piece.startswith('\"') and piece.endswith('\"'):\n                pieces.append(piece)\n            else:\n                piece = piece.replace('(', ' ( ')\n                piece = piece.replace(')', ' ) ')\n                piece = piece.replace(':', ' :')\n                piece = piece.replace('/', ' / ')\n                piece = piece.strip()\n                pieces.append(piece)\n        linearized = re.sub(r'\\s+', ' ', ' '.join(pieces)).strip()\n        # T5 uses pad instead of <s>\n        # linearized_nodes = [AMRTokens.BOS_N] + linearized.split(' ')\n        linearized_nodes = [self.pad_token] + linearized.split(' ')\n        return linearized_nodes\n\n    def tokenize_amr(self, graph):\n        if self.raw_graph:\n            graph_ = copy.deepcopy(graph)\n            graph_.metadata = {}\n            linearized = penman.encode(graph_)\n            linearized = re.sub(r\"\\s+\", ' ', linearized)\n            bpe_tokens = [self.bos_token] + self._tokenize(linearized)[:1022]\n            bpe_token_ids = [self.encoder.get(b, self.unk_token_id) for b in bpe_tokens]\n            bpe_backreferences = list(range(len(bpe_token_ids)))\n            return bpe_tokens, bpe_token_ids, bpe_backreferences\n        else:\n            return super().tokenize_amr(graph)\n\n    def _get_nodes_and_backreferences(self, graph):\n        graph_ = copy.deepcopy(graph)\n        graph_.metadata = {}\n        linearized = penman.encode(graph_)\n        linearized_nodes = self._tokenize_encoded_graph(linearized)\n\n        if self.use_pointer_tokens:\n            remap = {}\n            for i in range(1, len(linearized_nodes)):\n                nxt = linearized_nodes[i]\n                lst = linearized_nodes[i - 1]\n                if nxt == '/':\n                    remap[lst] = f'<pointer:{len(remap)}>'\n            i = 1\n            linearized_nodes_ = [linearized_nodes[0]]\n            while i < (len(linearized_nodes)):\n                nxt = linearized_nodes[i]\n                lst = linearized_nodes_[-1]\n                if nxt in remap:\n                    if lst == '(' and linearized_nodes[i + 1] == '/':\n                        nxt = remap[nxt]\n                        i += 1\n                    elif lst.startswith(':'):\n                        nxt = remap[nxt]\n                linearized_nodes_.append(nxt)\n                i += 1\n            linearized_nodes = linearized_nodes_\n            if self.remove_pars:\n                linearized_nodes = [n for n in linearized_nodes if n != '(']\n        backreferences = list(range(len(linearized_nodes)))\n        return linearized_nodes, backreferences\n\n    def _classify(self, node):\n        if not isinstance(node, str):\n            return \"CONST\"\n        elif node == 'i':\n            return \"I\"\n        elif re.match(r'^[a-z]\\d*$', node) is not None:\n            return \"VAR\"\n        elif node[0].isdigit():\n            return \"CONST\"\n        elif node.startswith('\"') and node.endswith('\"'):\n            return \"CONST\"\n        elif node in ('+', '-'):\n            return \"CONST\"\n        elif node == ':mode':\n            return 'MODE'\n        elif node.startswith(':'):\n            return \"EDGE\"\n        elif node in ['/', '(', ')']:\n            return node\n        elif node[0].isalpha():\n            for char in (',', ':', '/', '(', ')', '.', '!', '?', '\\\\'):\n                if char in node:\n                    return \"CONST\"\n            return \"INST\"\n        else:\n            return 'CONST'\n\n    def _fix_and_make_graph(self, nodes):\n\n        nodes_ = []\n        for n in nodes:\n            if isinstance(n, str):\n                if n.startswith('<') and n.endswith('>') and (not n.startswith('<pointer:')):\n                    pass\n                else:\n                    nodes_.append(n)\n            else:\n                nodes_.append(n)\n        nodes = nodes_\n        if not nodes:\n            return penman.Graph()\n\n        if self.use_pointer_tokens:\n\n            i = 0\n            nodes_ = []\n            while i < len(nodes):\n                nxt = nodes[i]\n                pst = None\n                if isinstance(nxt, str) and nxt.startswith('<pointer:'):\n                    e = nxt.find('>')\n                    if e != len(nxt) - 1:\n                        pst = nxt[e + 1:]\n                        nxt = nxt[:e + 1]\n                    nodes_.append(nxt)\n                    if pst is not None:\n                        nodes_.append(pst)\n                else:\n                    nodes_.append(nxt)\n                i += 1\n            nodes = nodes_\n\n            i = 1\n            nodes_ = [nodes[0]]\n            while i < len(nodes):\n                nxt = nodes[i]\n                if isinstance(nxt, str) and nxt.startswith('<pointer:') and i + 1 < len(nodes):\n                    nxt = 'z' + nxt[9:-1]\n                    fol = nodes[i + 1]\n                    # is not expansion\n                    if isinstance(fol, str) and (fol.startswith(':') or (fol == ')')):\n                        nodes_.append(nxt)\n                    else:\n                        if self.remove_pars:\n                            nodes_.append('(')\n                        else:\n                            if nodes_[-1] != '(':\n                                nodes_.append('(')\n                                # pass\n                        nodes_.append(nxt)\n                        nodes_.append('/')\n                else:\n                    nodes_.append(nxt)\n                i += 1\n            nodes = nodes_\n\n        i = 0\n        nodes_ = []\n        while i < (len(nodes) - 1):\n            if nodes[i] == ':':\n                nodes_.append(nodes[i] + nodes[i + 1])\n                i += 2\n                last = False\n            else:\n                nodes_.append(nodes[i])\n                i += 1\n                last = True\n        if last:\n            nodes_.append(nodes[-1])\n        nodes = nodes_\n\n        i = 0\n        nodes_ = []\n        while i < (len(nodes)):\n            if i < 2:\n                nodes_.append(nodes[i])\n                i += 1\n            elif nodes_[-2] == '/' and nodes[i] == '/':\n                i += 2\n            else:\n                nodes_.append(nodes[i])\n                i += 1\n        nodes = nodes_\n\n        i = 0\n        newvars = 0\n        variables = set()\n        remap = {}\n        nodes_ = []\n        while i < (len(nodes)):\n\n            next = nodes[i]\n\n            if next == '/':\n                last = nodes_[-1]\n                if last in variables:\n                    last_remap = f\"z{newvars + 1000}\"\n                    newvars += 1\n                    nodes_[-1] = last_remap\n                    remap[last] = last_remap\n                variables.add(last)\n                nodes_.append(next)\n\n            elif self._classify(next) == 'VAR' and next in remap and (i < len(nodes) - 1) and nodes[i + 1] != '/':\n                next = remap[next]\n                nodes_.append(next)\n\n            else:\n                nodes_.append(next)\n\n            i += 1\n\n        nodes = nodes_\n        pieces_ = []\n        open_cnt = 0\n        closed_cnt = 0\n        if nodes[0] != '(':\n            pieces_.append('(')\n            open_cnt += 1\n        for p in nodes:\n            if p == '(':\n                open_cnt += 1\n            elif p == ')':\n                closed_cnt += 1\n            pieces_.append(p)\n            if open_cnt == closed_cnt:\n                break\n        nodes = pieces_ + [')'] * (open_cnt - closed_cnt)\n\n        pieces = []\n        for piece in nodes:\n            if not pieces:\n                pieces.append('(')\n            else:\n                piece = str(piece)\n                if piece.startswith('\"') or piece.startswith('\"') or '\"' in piece.strip('\"'):\n                    piece = '\"' + piece.replace('\"', '') + '\"'\n\n                prev = self._classify(pieces[-1])\n                next = self._classify(piece)\n\n                if next == 'CONST':\n                    quote = False\n                    for char in (',', ':', '/', '(', ')', '.', '!', '?', '\\\\', '_', '='):\n                        if char in piece:\n                            quote = True\n                            break\n                    if quote:\n                        piece = '\"' + piece.strip('\"') + '\"'\n\n                if prev == '(':\n                    if next in ('VAR', 'I'):\n                        pieces.append(piece)\n                elif prev == ')':\n                    if next in (')', 'EDGE', 'MODE'):\n                        pieces.append(piece)\n                elif prev == 'VAR':\n                    if next in ('/', 'EDGE', 'MODE', ')'):\n                        pieces.append(piece)\n                elif prev == '/':\n                    if next in ('INST', 'I'):\n                        pieces.append(piece)\n                elif prev == 'INST':\n                    if next in (')', 'EDGE', 'MODE'):\n                        pieces.append(piece)\n                elif prev == 'I':\n                    if next in ('/', ')', 'EDGE', 'MODE'):\n                        pieces.append(piece)\n                elif prev == 'EDGE':\n                    if next in ('(', 'VAR', 'CONST', 'I'):\n                        pieces.append(piece)\n                    elif next == ')':\n                        pieces[-1] = piece\n                    elif next in ('EDGE', 'MODE'):\n                        pieces[-1] = piece\n                elif prev == 'MODE':\n                    if next == 'INST':\n                        pieces.append(piece)\n                elif prev == 'CONST':\n                    if next in (')', 'EDGE', 'MODE'):\n                        pieces.append(piece)\n\n        pieces_ = []\n        open_cnt = 0\n        closed_cnt = 0\n        if pieces[0] != '(':\n            pieces_.append('(')\n            open_cnt += 1\n        for p in pieces:\n            if p == '(':\n                open_cnt += 1\n            elif p == ')':\n                closed_cnt += 1\n            pieces_.append(p)\n            if open_cnt == closed_cnt:\n                break\n        pieces = pieces_ + [')'] * (open_cnt - closed_cnt)\n\n        linearized = re.sub(r'\\s+', ' ', ' '.join(pieces)).strip()\n\n        \"\"\"\n        line = linearized\n        # make sure parentheses match\n        # copied from https://github.com/RikVN/AMR/blob/master/restoreAMR/restore_amr.py\n        open_count = 0\n        close_count = 0\n        for i, c in enumerate(line):\n            if c == '(':\n                open_count += 1\n            elif c == ')':\n                close_count += 1\n            if open_count == close_count and open_count > 0:\n                line = line[:i].strip()\n                break\n        old_line = line\n        while True:\n            open_count = len(re.findall(r'\\(', line))\n            close_count = len(re.findall(r'\\)', line))\n            if open_count > close_count:\n                line += ')' * (open_count - close_count)\n            elif close_count > open_count:\n                for i in range(close_count - open_count):\n                    line = line.rstrip(')')\n                    line = line.rstrip(' ')\n            if old_line == line:\n                break\n            old_line = line\n        \"\"\"\n\n        graph = penman.decode(linearized + ' ')\n        triples = []\n        newvars = 2000\n        for triple in graph.triples:\n            x, rel, y = triple\n            if x is None:\n                pass\n            elif rel == ':instance' and y is None:\n                triples.append(penman.Triple(x, rel, 'thing'))\n            elif y is None:\n                var = f'z{newvars}'\n                newvars += 1\n                triples.append(penman.Triple(x, rel, var))\n                triples.append(penman.Triple(var, ':instance', 'thing'))\n            else:\n                triples.append(triple)\n        graph = penman.Graph(triples)\n        linearized = pm_encode(graph)\n\n        def fix_text(linearized=linearized):\n            n = 0\n\n            def _repl1(match):\n                nonlocal n\n                out = match.group(1) + match.group(2) + str(3000 + n) + ' / ' + match.group(2) + match.group(3)\n                n += 1\n                return out\n\n            linearized = re.sub(r'(\\(\\s?)([a-z])([^\\/:\\)]+[:\\)])', _repl1, linearized,\n                                flags=re.IGNORECASE | re.MULTILINE)\n\n            def _repl2(match):\n                return match.group(1)\n\n            linearized = re.sub(r'(\\(\\s*[a-z][\\d+]\\s*\\/\\s*[^\\s\\)\\(:\\/]+\\s*)((?:/\\s*[^\\s\\)\\(:\\/]+\\s*)+)', _repl2,\n                                linearized,\n                                flags=re.IGNORECASE | re.MULTILINE)\n\n            # adds a ':' to args w/o it\n            linearized = re.sub(r'([^:])(ARG)', r'\\1 :\\2', linearized)\n\n            # removes edges with no node\n            # linearized = re.sub(r':[^\\s\\)\\(:\\/]+?\\s*\\)', ')', linearized, flags=re.MULTILINE)\n\n            return linearized\n\n        linearized = fix_text(linearized)\n\n        g = penman.decode(linearized)\n        return g\n\n    def decode_amr(self, tokens, restore_name_ops=None):\n        try:\n            if self.raw_graph:\n                nodes = self._tokenize_encoded_graph(self.decode(tokens))\n                backreferences = list(range(len(nodes)))\n            else:\n                nodes, backreferences = postprocessing.decode_into_node_and_backreferences_without_space(tokens, self) \\\n                    if not self.INIT else postprocessing.decode_into_node_and_backreferences(tokens, self)\n            nodes_ = nodes\n        except Exception as e:\n            print('Decoding failure:', file=sys.stderr)\n            traceback.print_exc()\n            return postprocessing.BACKOFF, postprocessing.ParsedStatus.BACKOFF, (None, None)\n        try:\n            graph_ = graph = self._fix_and_make_graph(nodes)\n            if self.collapse_name_ops:\n                graph_ = graph = postprocessing._split_name_ops(graph)\n        except Exception as e:\n            print('Building failure:', file=sys.stderr)\n            traceback.print_exc()\n            print(nodes, file=sys.stderr)\n            print(backreferences, file=sys.stderr)\n            print(e, file=sys.stderr)\n            return postprocessing.BACKOFF, postprocessing.ParsedStatus.BACKOFF, (None, None)\n        try:\n            graph, status = postprocessing.connect_graph_if_not_connected(graph)\n            if status == postprocessing.ParsedStatus.BACKOFF:\n                print('Reconnection 1 failure:')\n                print(nodes, file=sys.stderr)\n                print(backreferences, file=sys.stderr)\n                print(graph_, file=sys.stderr)\n            return graph, status, (nodes_, backreferences)\n        except Exception as e:\n            print('Reconnction 2 failure:', file=sys.stderr)\n            print(e, file=sys.stderr)\n            traceback.print_exc()\n            print(nodes, file=sys.stderr)\n            print(backreferences, file=sys.stderr)\n            print(graph_, file=sys.stderr)\n            return postprocessing.BACKOFF, postprocessing.ParsedStatus.BACKOFF, (nodes_, backreferences)\n\n    @property\n    def encoder(self) -> Dict[str, int]:\n        return self.get_vocab()\n"
  },
  {
    "path": "hanlp/components/amr/seq2seq/evaluation.py",
    "content": "from pathlib import Path\n\nimport penman\n\n\ndef write_predictions(predictions_path, tokenizer, graphs):\n    pieces = [penman.encode(g) for g in graphs]\n    text = '\\n\\n'.join(pieces)\n    if tokenizer:\n        text = text.replace(tokenizer.INIT, '')\n    Path(predictions_path).write_text(text)\n    return predictions_path\n\n\ndef compute_smatch(pred, gold):\n    from perin_parser.thirdparty.mtool import smatch\n    with Path(pred).open() as p, Path(gold).open() as g:\n        score = next(smatch.score_amr_pairs(p, g))\n    return score[2]\n\n\ndef compute_bleu(gold_sentences, pred_sentences):\n    from sacrebleu import corpus_bleu\n    return corpus_bleu(pred_sentences, [gold_sentences])\n"
  },
  {
    "path": "hanlp/components/amr/seq2seq/optim.py",
    "content": "# taken from\n\nimport math\nimport torch\nfrom torch.optim.optimizer import Optimizer\n\n\nclass RAdam(Optimizer):\n\n    def __init__(self, params, lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0, degenerated_to_sgd=True):\n        if not 0.0 <= lr:\n            raise ValueError(\"Invalid learning rate: {}\".format(lr))\n        if not 0.0 <= eps:\n            raise ValueError(\"Invalid epsilon value: {}\".format(eps))\n        if not 0.0 <= betas[0] < 1.0:\n            raise ValueError(\"Invalid beta parameter at index 0: {}\".format(betas[0]))\n        if not 0.0 <= betas[1] < 1.0:\n            raise ValueError(\"Invalid beta parameter at index 1: {}\".format(betas[1]))\n\n        self.degenerated_to_sgd = degenerated_to_sgd\n        if isinstance(params, (list, tuple)) and len(params) > 0 and isinstance(params[0], dict):\n            for param in params:\n                if 'betas' in param and (param['betas'][0] != betas[0] or param['betas'][1] != betas[1]):\n                    param['buffer'] = [[None, None, None] for _ in range(10)]\n        defaults = dict(lr=lr, betas=betas, eps=eps, weight_decay=weight_decay,\n                        buffer=[[None, None, None] for _ in range(10)])\n        super(RAdam, self).__init__(params, defaults)\n\n    def __setstate__(self, state):\n        super(RAdam, self).__setstate__(state)\n\n    def step(self, closure=None):\n\n        loss = None\n        if closure is not None:\n            loss = closure()\n\n        for group in self.param_groups:\n\n            for p in group['params']:\n                if p.grad is None:\n                    continue\n                grad = p.grad.data.float()\n                if grad.is_sparse:\n                    raise RuntimeError('RAdam does not support sparse gradients')\n\n                p_data_fp32 = p.data.float()\n\n                state = self.state[p]\n\n                if len(state) == 0:\n                    state['step'] = 0\n                    state['exp_avg'] = torch.zeros_like(p_data_fp32)\n                    state['exp_avg_sq'] = torch.zeros_like(p_data_fp32)\n                else:\n                    state['exp_avg'] = state['exp_avg'].type_as(p_data_fp32)\n                    state['exp_avg_sq'] = state['exp_avg_sq'].type_as(p_data_fp32)\n\n                exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq']\n                beta1, beta2 = group['betas']\n\n                exp_avg_sq.mul_(beta2).addcmul_(grad, grad, value=1 - beta2)\n                exp_avg.mul_(beta1).add_(grad, alpha=1 - beta1)\n\n                state['step'] += 1\n                buffered = group['buffer'][int(state['step'] % 10)]\n                if state['step'] == buffered[0]:\n                    N_sma, step_size = buffered[1], buffered[2]\n                else:\n                    buffered[0] = state['step']\n                    beta2_t = beta2 ** state['step']\n                    N_sma_max = 2 / (1 - beta2) - 1\n                    N_sma = N_sma_max - 2 * state['step'] * beta2_t / (1 - beta2_t)\n                    buffered[1] = N_sma\n\n                    # more conservative since it's an approximated value\n                    if N_sma >= 5:\n                        step_size = math.sqrt(\n                            (1 - beta2_t) * (N_sma - 4) / (N_sma_max - 4) * (N_sma - 2) / N_sma * N_sma_max / (\n                                    N_sma_max - 2)) / (1 - beta1 ** state['step'])\n                    elif self.degenerated_to_sgd:\n                        step_size = 1.0 / (1 - beta1 ** state['step'])\n                    else:\n                        step_size = -1\n                    buffered[2] = step_size\n\n                # more conservative since it's an approximated value\n                if N_sma >= 5:\n                    if group['weight_decay'] != 0:\n                        p_data_fp32.add_(p_data_fp32, alpha=-group['weight_decay'] * group['lr'])\n                    denom = exp_avg_sq.sqrt().add_(group['eps'])\n                    p_data_fp32.addcdiv_(exp_avg, denom, value=-step_size * group['lr'])\n                    p.data.copy_(p_data_fp32)\n                elif step_size > 0:\n                    if group['weight_decay'] != 0:\n                        p_data_fp32.add_(p_data_fp32, alpha=-group['weight_decay'] * group['lr'])\n                    p_data_fp32.add_(exp_avg, alpha=-step_size * group['lr'])\n                    p.data.copy_(p_data_fp32)\n\n        return loss\n"
  },
  {
    "path": "hanlp/components/amr/seq2seq/seq2seq_amr_parser.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-04-28 17:33\nimport datetime\nimport functools\nimport logging\nimport os\nfrom typing import Union, List, Callable\n\nimport torch\nfrom torch.utils.data import DataLoader\nfrom transformers import get_constant_schedule_with_warmup, T5ForConditionalGeneration\nfrom transformers.models.bart.modeling_bart import BartForConditionalGeneration\n\nfrom hanlp.common.dataset import SamplerBuilder, SortingSamplerBuilder, PadSequenceDataLoader\nfrom hanlp.common.structure import History\nfrom hanlp.common.torch_component import TorchComponent\nfrom hanlp.common.vocab import Vocab\nfrom hanlp.components.amr.seq2seq.dataset.dataset import AMRDataset, dfs_linearize_tokenize\nfrom hanlp.components.amr.seq2seq.dataset.penman import AMRGraph\nfrom hanlp.components.amr.seq2seq.dataset.tokenization_bart import PENMANBartTokenizer\nfrom hanlp.components.amr.seq2seq.dataset.tokenization_t5 import PENMANT5Tokenizer\nfrom hanlp.components.amr.seq2seq.evaluation import write_predictions, compute_smatch\nfrom hanlp.components.amr.seq2seq.optim import RAdam\nfrom hanlp.layers.transformers.pt_imports import PretrainedConfig, AutoConfig_\nfrom hanlp.layers.transformers.resource import get_model_mirror, get_tokenizer_mirror\nfrom hanlp.metrics.amr.smatch_eval import smatch_eval\nfrom hanlp.metrics.mtl import MetricDict\nfrom hanlp.utils.time_util import CountdownTimer\nfrom hanlp_common.constant import IDX\nfrom hanlp_common.util import merge_locals_kwargs, reorder\n\n\nclass Seq2seq_AMR_Parser(TorchComponent):\n    def __init__(self, **kwargs):\n        super().__init__(**kwargs)\n        self._transformer_config: PretrainedConfig = None\n        self._tokenizer: PENMANBartTokenizer = None\n        self.model: BartForConditionalGeneration = None\n\n    def build_dataloader(self, data, batch_size,\n                         gradient_accumulation=1,\n                         shuffle=False,\n                         sampler_builder: SamplerBuilder = None,\n                         device=None,\n                         logger: logging.Logger = None,\n                         **kwargs) -> DataLoader:\n        dataset = self.build_dataset(data, not shuffle)\n        if self.vocabs.mutable:\n            self.build_vocabs(dataset, logger)\n        self.finalize_dataset(dataset, logger)\n        if isinstance(data, str):\n            dataset.purge_cache()\n            timer = CountdownTimer(len(dataset))\n            max_num_tokens = 0\n            # lc = Counter()\n            for each in dataset:\n                max_num_tokens = max(max_num_tokens, len(each['text_token_ids']))\n                # lc[len(each['text_token_ids'])] += 1\n                timer.log(f'Preprocessing and caching samples (longest sequence {max_num_tokens})'\n                          f'[blink][yellow]...[/yellow][/blink]')\n            # print(lc.most_common())\n            if self.vocabs.mutable:\n                self.vocabs.lock()\n                self.vocabs.summary(logger)\n\n        if not sampler_builder:\n            sampler_builder = SortingSamplerBuilder(batch_max_tokens=500)\n        sampler = sampler_builder.build([len(x['text_token_ids']) for x in dataset], shuffle,\n                                        gradient_accumulation if dataset.cache else 1)\n        return self._create_dataloader(dataset, batch_size, device, sampler, shuffle)\n\n    def _create_dataloader(self, dataset, batch_size, device, sampler, shuffle):\n        return PadSequenceDataLoader(dataset, batch_size, shuffle, device=device, batch_sampler=sampler,\n                                     pad=self._get_pad_dict())\n\n    def _get_pad_dict(self):\n        return {'text_token_ids': self._transformer_config.pad_token_id,\n                'graph_token_ids': self._transformer_config.pad_token_id}\n\n    def finalize_dataset(self, dataset, logger: logging.Logger = None):\n        dataset.append_transform(functools.partial(dfs_linearize_tokenize, tokenizer=self._tokenizer,\n                                                   remove_space='chinese' in self.config.transformer))\n\n    def build_dataset(self, data, generate_idx):\n        dataset = AMRDataset(data, generate_idx=generate_idx)\n        return dataset\n\n    def collect_additional_tokens(self, additional_tokens, dataset):\n        pred_min = self.config.pred_min\n        frames = dataset.get_frames()\n        for token, freq in frames.items():\n            if freq >= pred_min:\n                additional_tokens.add(token)\n        for token, freq in dataset.get_roles().items():\n            additional_tokens.add(token)\n        additional_tokens.update(self.config.additional_tokens)\n\n    def build_tokenizer(self, additional_tokens) -> PENMANBartTokenizer:\n        transformer = self.config.transformer\n        if 't5-' in transformer:\n            cls = PENMANT5Tokenizer\n        elif 'bart-' in transformer:\n            cls = PENMANBartTokenizer\n        else:\n            raise NotImplemented(f'Unsupported transformer {transformer}')\n        transformer = get_tokenizer_mirror(transformer)\n        self._tokenizer = cls.from_pretrained(\n            transformer,\n            collapse_name_ops=self.config.collapse_name_ops,\n            use_pointer_tokens=self.config.use_pointer_tokens,\n            raw_graph=self.config.raw_graph,\n            additional_tokens=additional_tokens,\n            recategorization_tokens=self.config.recategorization_tokens,\n            config=self._transformer_config,\n        )\n        return self._tokenizer\n\n    def build_optimizer(self, trn, lr, epochs, gradient_accumulation, warmup_steps, weight_decay, **kwargs):\n        num_training_steps = len(trn) * epochs // gradient_accumulation\n        if isinstance(warmup_steps, float):\n            warmup_steps = int(num_training_steps * warmup_steps)\n        optimizer = RAdam(\n            self.model.parameters(),\n            lr=lr,\n            weight_decay=weight_decay)\n        scheduler = get_constant_schedule_with_warmup(\n            optimizer,\n            num_warmup_steps=warmup_steps)\n        return optimizer, scheduler\n\n    def build_criterion(self, **kwargs):\n        pass\n\n    def build_metric(self, **kwargs):\n        pass\n\n    def execute_training_loop(self, trn: DataLoader, dev: DataLoader, epochs, criterion, optimizer, metric, save_dir,\n                              logger: logging.Logger, devices, ratio_width=None, dev_data=None, eval_after=None,\n                              **kwargs):\n        best_epoch, best_metric = 0, -1\n        if isinstance(eval_after, float):\n            eval_after = int(epochs * eval_after)\n        timer = CountdownTimer(epochs)\n        history = History()\n        for epoch in range(1, epochs + 1):\n            logger.info(f\"[yellow]Epoch {epoch} / {epochs}:[/yellow]\")\n            self.fit_dataloader(trn, criterion, optimizer, metric, logger, history=history, ratio_width=ratio_width,\n                                **self.config)\n            if epoch > eval_after:\n                dev_metric = self.evaluate_dataloader(dev, criterion, logger=logger, ratio_width=ratio_width,\n                                                      output=os.path.join(save_dir, 'dev.pred.txt'),\n                                                      input=dev_data, use_fast=True)\n            timer.update()\n            report = f\"{timer.elapsed_human} / {timer.total_time_human} ETA: {timer.eta_human}\"\n            if epoch > eval_after:\n                if dev_metric > best_metric:\n                    best_epoch, best_metric = epoch, dev_metric\n                    self.save_weights(save_dir)\n                    report += ' [red](saved)[/red]'\n                else:\n                    report += f' ({epoch - best_epoch})'\n                # if epoch - best_epoch >= patience:\n                #     report += ' early stop'\n            logger.info(report)\n            # if epoch - best_epoch >= patience:\n            #     break\n        if not best_epoch:\n            self.save_weights(save_dir)\n        elif best_epoch != epoch:\n            self.load_weights(save_dir)\n        logger.info(f\"Max score of dev is {best_metric} at epoch {best_epoch}\")\n        logger.info(f\"Average time of each epoch is {timer.elapsed_average_human}\")\n        logger.info(f\"{timer.elapsed_human} elapsed\")\n        return best_metric\n\n    def fit_dataloader(self, trn: DataLoader, criterion, optimizer, metric, logger: logging.Logger,\n                       history: History = None, gradient_accumulation=1, ratio_percentage=None, **kwargs):\n        optimizer, scheduler = optimizer\n        self.model.train()\n        timer = CountdownTimer(history.num_training_steps(len(trn), gradient_accumulation=gradient_accumulation))\n        total_loss = 0\n        for batch in trn:\n            output_dict = self.feed_batch(batch)\n            loss = output_dict['loss']\n            if gradient_accumulation and gradient_accumulation > 1:\n                loss /= gradient_accumulation\n            loss.backward()\n            total_loss += loss.item()\n            if history.step(gradient_accumulation):\n                self._step(optimizer, scheduler)\n                timer.log(self.report_metrics(total_loss / (timer.current + 1)),\n                          ratio_percentage=ratio_percentage, logger=logger)\n            del loss\n            del output_dict\n        return total_loss / max(timer.total, 1)\n\n    def _step(self, optimizer, scheduler):\n        if self.config.grad_norm:\n            torch.nn.utils.clip_grad_norm_(self.model.parameters(), self.config.grad_norm)\n        optimizer.step()\n        if scheduler:\n            scheduler.step()\n        optimizer.zero_grad()\n\n    def report_metrics(self, loss):\n        return f'loss: {loss:.4f}'\n\n    def feed_batch(self, batch):\n        input_ids, labels = batch['text_token_ids'], batch.get('graph_token_ids')\n        attention_mask = input_ids.ne(self.model.config.pad_token_id).to(torch.long)\n        if labels is not None:\n            decoder_input_ids = labels[:, :-1]\n            labels = labels[:, 1:].contiguous()\n        else:\n            decoder_input_ids = None\n        return self.model(input_ids=input_ids, attention_mask=attention_mask, decoder_input_ids=decoder_input_ids,\n                          labels=labels)\n\n    @torch.no_grad()\n    def evaluate_dataloader(self, data: DataLoader, criterion: Callable, metric=None, output=False, ratio_width=None,\n                            logger=None, input=None, use_fast=False,\n                            **kwargs):\n        self.model.eval()\n        timer = CountdownTimer(len(data))\n        graphs = []\n        orders = []\n        smatch = 0\n        for idx, batch in enumerate(data):\n            graphs_per_batch = self.predict_amrs(batch)\n            graphs_per_batch = [x[0] for x in graphs_per_batch]\n            # Copy meta data from gold graph\n            for gp, gg in zip(graphs_per_batch, batch['amr']):\n                metadata = gg.metadata.copy()\n                metadata['annotator'] = f'{self.config.transformer}-amr'\n                metadata['date'] = str(datetime.datetime.now())\n                if 'save-date' in metadata:\n                    del metadata['save-date']\n                gp.metadata = metadata\n            graphs.extend(graphs_per_batch)\n            orders.extend(batch[IDX])\n            if idx == timer.total - 1:\n                graphs = reorder(graphs, orders)\n                write_predictions(output, self._tokenizer, graphs)\n                try:\n                    if use_fast:\n                        smatch = compute_smatch(output, input)\n                    else:\n                        smatch = smatch_eval(output, input, use_fast=False)\n                except:\n                    pass\n                timer.log(smatch.cstr() if isinstance(smatch, MetricDict) else f'{smatch:.2%}', ratio_percentage=False,\n                          logger=logger)\n            else:\n                timer.log(ratio_percentage=False, logger=logger)\n\n        return smatch\n\n    def predict_amrs(self, batch, beam_size=1):\n        out = self._model_generate(batch, beam_size)\n        tokens = []\n        for i1 in range(0, out.size(0), beam_size):\n            tokens_same_source = []\n            tokens.append(tokens_same_source)\n            for i2 in range(i1, i1 + beam_size):\n                tokk = out[i2].tolist()\n                tokens_same_source.append(tokk)\n        tokens = [t for tt in tokens for t in tt]\n        graphs = []\n        tokenizer = self._tokenizer\n        for i1 in range(0, len(tokens), beam_size):\n            graphs_same_source = []\n            graphs.append(graphs_same_source)\n            for i2 in range(i1, i1 + beam_size):\n                tokk = tokens[i2]\n                graph, status, (lin, backr) = tokenizer.decode_amr(tokk, restore_name_ops=False)\n                graph.status = status\n                graph.nodes = lin\n                graph.backreferences = backr\n                graph.tokens = tokk\n                graphs_same_source.append(graph)\n            graphs_same_source[:] = \\\n                tuple(zip(*sorted(enumerate(graphs_same_source), key=lambda x: (x[1].status.value, x[0]))))[1]\n\n        return graphs\n\n    def _model_generate(self, batch, beam_size):\n        input_ids = batch['text_token_ids']\n        attention_mask = input_ids.ne(self.model.config.pad_token_id).to(torch.long)\n        out = self.model.generate(\n            input_ids=input_ids,\n            attention_mask=attention_mask,\n            max_length=1024,\n            decoder_start_token_id=0,\n            num_beams=beam_size,\n            num_return_sequences=beam_size)\n        return out\n\n    def build_model(self, training=True, **kwargs) -> torch.nn.Module:\n        # noinspection PyTypeChecker\n        transformer = self.config.transformer\n        cls = self._get_model_cls(transformer)\n        transformer = get_model_mirror(self.config.transformer)\n        model: cls = cls.from_pretrained(\n            transformer,\n            config=self._transformer_config) if training else cls(self._transformer_config)\n        if not training:\n            self.build_tokenizer(self.vocabs['additional_tokens'])\n        tokenizer = self._tokenizer\n        model.resize_token_embeddings(len(tokenizer.encoder))\n        if training:\n            self._init_new_embeddings(model if cls == T5ForConditionalGeneration else model.model, tokenizer)\n        return model\n\n    def _get_model_cls(self, transformer: str):\n        if 't5-' in transformer:\n            cls = T5ForConditionalGeneration\n        elif 'bart-' in transformer:\n            cls = BartForConditionalGeneration\n        else:\n            raise NotImplemented(f'Unsupported transformer {transformer}')\n        return cls\n\n    @staticmethod\n    def _init_new_embeddings(model, tokenizer):\n        modified = 0\n        encoder = tokenizer.encoder\n        for tok, idx in encoder.items():\n            tok = tok.lstrip(tokenizer.INIT)\n\n            if idx < tokenizer.old_enc_size:\n                continue\n\n            elif tok.startswith('<pointer:') and tok.endswith('>'):\n                tok_split = ['pointer', str(tok.split(':')[1].strip('>'))]\n\n            elif tok.startswith('<'):\n                continue\n\n            elif tok.startswith(':'):\n                if tok.startswith(':op'):\n                    tok_split = ['relation', 'operator', str(int(tok[3:]))]\n\n                elif tok.startswith(':snt'):\n                    tok_split = ['relation', 'sentence', str(int(tok[4:]))]\n\n                elif tok.startswith(':ARG'):\n                    tok_split = ['relation', 'argument', str(int(tok[4:]))]\n                else:\n                    tok_split = ['relation'] + tok.lstrip(':').split('-')\n            else:\n                tok_split = tok.split('-')\n\n            tok_split_ = tok_split\n            tok_split = []\n            for s in tok_split_:\n                s_ = s + tokenizer.INIT\n                if s_ in encoder:\n                    tok_split.append(s_)\n                else:\n                    tok_split.extend(tokenizer._tok_bpe(s))\n\n            vecs = []\n            for s in tok_split:\n                idx_split = encoder.get(s, -1)\n                if idx_split > -1:\n                    vec_split = model.encoder.embed_tokens.weight.data[idx_split].clone()\n                    vecs.append(vec_split)\n\n            if vecs:\n                vec = torch.stack(vecs, 0).mean(0)\n                noise = torch.empty_like(vec)\n                noise.uniform_(-0.1, +0.1)\n                model.encoder.embed_tokens.weight.data[idx] = vec + noise\n                modified += 1\n\n    def input_is_flat(self, data):\n        return isinstance(data, str)\n\n    def predict(self, data: Union[str, List[str]], beautiful_amr_graph=True, **kwargs):\n        flat = self.input_is_flat(data)\n        if flat:\n            data = [data]\n        dataloader = self.build_dataloader([{'text': x} for x in data], **self.config, device=self.device)\n        orders = []\n        results = []\n        for batch in dataloader:\n            graphs = self.predict_amrs(batch)\n            graphs = [x[0] for x in graphs]\n            if beautiful_amr_graph:\n                graphs = [AMRGraph(x.triples, x.top, x.epidata, x.metadata) for x in graphs]\n            results.extend(graphs)\n            orders.extend(batch[IDX])\n        results = reorder(results, orders)\n        if flat:\n            results = results[0]\n        return results\n\n    def fit(self, trn_data, dev_data, save_dir, batch_size=32, epochs=30,\n            transformer='facebook/bart-base',\n            lr=5e-05,\n            grad_norm=2.5,\n            weight_decay=0.004,\n            warmup_steps=1,\n            dropout=0.25,\n            attention_dropout=0.0,\n            pred_min=5,\n            eval_after=0.5,\n            collapse_name_ops=False,\n            use_pointer_tokens=True,\n            raw_graph=False,\n            gradient_accumulation=1,\n            recategorization_tokens=(\n                    'PERSON', 'COUNTRY', 'QUANTITY', 'ORGANIZATION', 'DATE_ATTRS', 'NATIONALITY', 'LOCATION', 'ENTITY',\n                    'CITY',\n                    'MISC', 'ORDINAL_ENTITY', 'IDEOLOGY', 'RELIGION', 'STATE_OR_PROVINCE', 'URL', 'CAUSE_OF_DEATH', 'O',\n                    'TITLE', 'DATE', 'NUMBER', 'HANDLE', 'SCORE_ENTITY', 'DURATION', 'ORDINAL', 'MONEY', 'SET',\n                    'CRIMINAL_CHARGE', '_1', '_2', '_3', '_4', '_2', '_5', '_6', '_7', '_8', '_9', '_10', '_11', '_12',\n                    '_13',\n                    '_14', '_15'),\n            additional_tokens=(\n                    'date-entity', 'government-organization', 'temporal-quantity', 'amr-unknown', 'multi-sentence',\n                    'political-party', 'monetary-quantity', 'ordinal-entity', 'religious-group', 'percentage-entity',\n                    'world-region', 'url-entity', 'political-movement', 'et-cetera', 'at-least', 'mass-quantity',\n                    'have-org-role-91', 'have-rel-role-91', 'include-91', 'have-concession-91', 'have-condition-91',\n                    'be-located-at-91', 'rate-entity-91', 'instead-of-91', 'hyperlink-91', 'request-confirmation-91',\n                    'have-purpose-91', 'be-temporally-at-91', 'regardless-91', 'have-polarity-91', 'byline-91',\n                    'have-manner-91', 'have-part-91', 'have-quant-91', 'publication-91', 'be-from-91', 'have-mod-91',\n                    'have-frequency-91', 'score-on-scale-91', 'have-li-91', 'be-compared-to-91', 'be-destined-for-91',\n                    'course-91', 'have-subevent-91', 'street-address-91', 'have-extent-91', 'statistical-test-91',\n                    'have-instrument-91', 'have-name-91', 'be-polite-91', '-00', '-01', '-02', '-03', '-04', '-05',\n                    '-06',\n                    '-07', '-08', '-09', '-10', '-11', '-12', '-13', '-14', '-15', '-16', '-17', '-18', '-19', '-20',\n                    '-21',\n                    '-22', '-23', '-24', '-25', '-26', '-27', '-28', '-29', '-20', '-31', '-32', '-33', '-34', '-35',\n                    '-36',\n                    '-37', '-38', '-39', '-40', '-41', '-42', '-43', '-44', '-45', '-46', '-47', '-48', '-49', '-50',\n                    '-51',\n                    '-52', '-53', '-54', '-55', '-56', '-57', '-58', '-59', '-60', '-61', '-62', '-63', '-64', '-65',\n                    '-66',\n                    '-67', '-68', '-69', '-70', '-71', '-72', '-73', '-74', '-75', '-76', '-77', '-78', '-79', '-80',\n                    '-81',\n                    '-82', '-83', '-84', '-85', '-86', '-87', '-88', '-89', '-90', '-91', '-92', '-93', '-94', '-95',\n                    '-96',\n                    '-97', '-98', '-of'),\n            devices=None,\n            logger=None,\n            seed=None,\n            finetune: Union[bool, str] = False,\n            eval_trn=True,\n            _device_placeholder=False,\n            **kwargs):\n        \"\"\"\n\n        Args:\n            trn_data:\n            dev_data:\n            save_dir:\n            batch_size:\n            epochs:\n            transformer:\n            lr:\n            grad_norm:\n            weight_decay:\n            warmup_steps:\n            dropout:\n            attention_dropout:\n            pred_min:\n            eval_after:\n            collapse_name_ops: ``True`` to merge name ops.\n            use_pointer_tokens: ``True`` to use pointer tokens to represent variables.\n            raw_graph: ``True`` to use the raw graph as input and skip all pre/post-processing steps.\n            gradient_accumulation:\n            recategorization_tokens: Tokens used in re-categorization. They will be added to tokenizer too but do not\n            put them into ``additional_tokens``.\n            additional_tokens: Tokens to be added to the tokenizer vocab.\n            devices:\n            logger:\n            seed:\n            finetune:\n            eval_trn:\n            _device_placeholder:\n            **kwargs:\n\n        Returns:\n\n        \"\"\"\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    def on_config_ready(self, **kwargs):\n        super().on_config_ready(**kwargs)\n        config = AutoConfig_.from_pretrained(self.config.transformer)\n        config.output_past = False\n        config.no_repeat_ngram_size = 0\n        config.prefix = \" \"\n        # config.output_attentions = True\n        config.dropout = self.config.dropout\n        config.attention_dropout = self.config.attention_dropout\n        self._transformer_config = config\n\n    def evaluate(self, tst_data, save_dir=None, logger: logging.Logger = None, batch_size=None, output=True,\n                 cache=None, ret_speed=False, **kwargs):\n        return super().evaluate(tst_data, save_dir, logger, batch_size, output, cache, ret_speed, **kwargs)\n\n    def build_vocabs(self, trn: torch.utils.data.Dataset, logger: logging.Logger):\n        additional_tokens = set()\n        self.collect_additional_tokens(additional_tokens, trn)\n        additional_tokens = sorted(additional_tokens)\n        self.build_tokenizer(additional_tokens)\n        self.vocabs['additional_tokens'] = Vocab(idx_to_token=list(additional_tokens))\n"
  },
  {
    "path": "hanlp/components/classifiers/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-11-10 13:18"
  },
  {
    "path": "hanlp/components/classifiers/fasttext_classifier.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2022-09-28 13:31\nimport os\nimport sys\nfrom typing import List, Union\n\nimport fasttext\nfrom fasttext.FastText import _FastText\n\nimport hanlp\nfrom hanlp.common.component import Component\nfrom hanlp.utils.io_util import get_resource, stdout_redirected\nfrom hanlp_common.io import load_json\nfrom hanlp_common.reflection import classpath_of\nfrom hanlp_common.structure import SerializableDict\n\n\nclass FastTextClassifier(Component):\n\n    def __init__(self) -> None:\n        super().__init__()\n        self._model: _FastText = None\n        self.config = SerializableDict({\n            'classpath': classpath_of(self),\n            'hanlp_version': hanlp.__version__,\n        })\n\n    def load(self, save_dir, model_path=None, **kwargs):\n        config_path = os.path.join(save_dir, 'config.json')\n        if os.path.isfile(config_path):\n            self.config: dict = load_json(config_path)\n            model_path = self.config.get('model_path', model_path)\n        else:\n            model_path = model_path or save_dir\n            self.config['model_path'] = model_path\n        filepath = get_resource(model_path)\n        with stdout_redirected(to=os.devnull, stdout=sys.stderr):\n            self._model = fasttext.load_model(filepath)\n\n    def predict(self, text: Union[str, List[str]], topk=False, prob=False, max_len=None, **kwargs):\n        \"\"\"\n        Classify text.\n\n        Args:\n            text: A document or a list of documents.\n            topk: ``True`` or ``int`` to return the top-k labels.\n            prob: Return also probabilities.\n            max_len: Strip long document into ``max_len`` characters for faster prediction.\n            **kwargs: Not used\n\n        Returns:\n            Classification results.\n        \"\"\"\n        num_labels = len(self._model.get_labels())\n        flat = isinstance(text, str)\n        if flat:\n            text = [text]\n        if not isinstance(topk, list):\n            topk = [topk] * len(text)\n        if not isinstance(prob, list):\n            prob = [prob] * len(text)\n        if max_len:\n            text = [x[:max_len] for x in text]\n        text = [x.replace('\\n', ' ') for x in text]\n        batch_labels, batch_probs = self._model.predict(text, k=num_labels)\n        results = []\n        for labels, probs, k, p in zip(batch_labels, batch_probs, topk, prob):\n            labels = [self._strip_prefix(x) for x in labels]\n            if k is False:\n                labels = labels[0]\n            elif k is True:\n                pass\n            elif k:\n                labels = labels[:k]\n            if p:\n                probs = probs.tolist()\n                if k is False:\n                    result = labels, probs[0]\n                else:\n                    result = dict(zip(labels, probs))\n            else:\n                result = labels\n            results.append(result)\n        if flat:\n            results = results[0]\n        return results\n\n    @property\n    def labels(self):\n        return [self._strip_prefix(x) for x in self._model.get_labels()]\n\n    @staticmethod\n    def _strip_prefix(label: str):\n        return label[len('__label__'):]\n"
  },
  {
    "path": "hanlp/components/classifiers/transformer_classifier.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-08 16:31\nimport logging\nfrom abc import ABC\nfrom typing import Callable, Union\nfrom typing import List\n\nimport torch\nfrom torch import nn\nfrom torch.utils.data import DataLoader\n\nfrom hanlp_common.constant import IDX\nfrom hanlp.common.dataset import TableDataset, SortingSampler, PadSequenceDataLoader, TransformableDataset\nfrom hanlp.common.torch_component import TorchComponent\nfrom hanlp.common.vocab import Vocab\nfrom hanlp.components.distillation.schedulers import LinearTeacherAnnealingScheduler\nfrom hanlp.layers.scalar_mix import ScalarMixWithDropoutBuilder\nfrom hanlp.layers.transformers.encoder import TransformerEncoder\nfrom hanlp.layers.transformers.pt_imports import PreTrainedModel, AutoTokenizer, BertTokenizer, AutoTokenizer_\nfrom hanlp.layers.transformers.utils import transformer_sliding_window, build_optimizer_scheduler_with_transformer\nfrom hanlp.metrics.accuracy import CategoricalAccuracy\nfrom hanlp.transform.transformer_tokenizer import TransformerTextTokenizer\nfrom hanlp.utils.time_util import CountdownTimer\nfrom hanlp_common.util import merge_locals_kwargs, merge_dict, isdebugging\n\n\nclass TransformerClassificationModel(nn.Module):\n\n    def __init__(self,\n                 transformer: PreTrainedModel,\n                 num_labels: int,\n                 max_seq_length=512) -> None:\n        super().__init__()\n        self.max_seq_length = max_seq_length\n        self.transformer = transformer\n        self.dropout = nn.Dropout(transformer.config.hidden_dropout_prob)\n        self.classifier = nn.Linear(transformer.config.hidden_size, num_labels)\n\n    def forward(self, input_ids, attention_mask, token_type_ids):\n        seq_length = input_ids.size(-1)\n        if seq_length > self.max_seq_length:\n            sequence_output = transformer_sliding_window(self.transformer, input_ids,\n                                                         max_pieces=self.max_seq_length, ret_cls='max')\n        else:\n            sequence_output = self.transformer(input_ids, attention_mask, token_type_ids)[0][:, 0, :]\n        sequence_output = self.dropout(sequence_output)\n        logits = self.classifier(sequence_output)\n        return logits\n\n\nclass TransformerComponent(TorchComponent, ABC):\n    def __init__(self, **kwargs) -> None:\n        \"\"\" The base class for transorfmer based components. If offers methods to build transformer tokenizers\n        , optimizers and models.\n\n        Args:\n            **kwargs: Passed to config.\n        \"\"\"\n        super().__init__(**kwargs)\n        self.transformer_tokenizer = None\n\n    def build_optimizer(self,\n                        trn,\n                        epochs,\n                        lr,\n                        adam_epsilon,\n                        weight_decay,\n                        warmup_steps,\n                        transformer_lr=None,\n                        teacher=None,\n                        **kwargs):\n        num_training_steps = len(trn) * epochs // self.config.get('gradient_accumulation', 1)\n        if transformer_lr is None:\n            transformer_lr = lr\n        transformer = self.model.encoder.transformer\n        optimizer, scheduler = build_optimizer_scheduler_with_transformer(self.model, transformer,\n                                                                          lr, transformer_lr,\n                                                                          num_training_steps, warmup_steps,\n                                                                          weight_decay, adam_epsilon)\n        if teacher:\n            lambda_scheduler = LinearTeacherAnnealingScheduler(num_training_steps)\n            scheduler = (scheduler, lambda_scheduler)\n        return optimizer, scheduler\n\n    def fit(self, trn_data, dev_data, save_dir,\n            transformer=None,\n            lr=5e-5,\n            transformer_lr=None,\n            adam_epsilon=1e-8,\n            weight_decay=0,\n            warmup_steps=0.1,\n            batch_size=32,\n            gradient_accumulation=1,\n            grad_norm=5.0,\n            transformer_grad_norm=None,\n            average_subwords=False,\n            scalar_mix: Union[ScalarMixWithDropoutBuilder, int] = None,\n            word_dropout=None,\n            hidden_dropout=None,\n            max_seq_len=None,\n            ret_raw_hidden_states=False,\n            batch_max_tokens=None,\n            epochs=3,\n            logger=None,\n            devices: Union[float, int, List[int]] = None,\n            **kwargs):\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    def on_config_ready(self, **kwargs):\n        super().on_config_ready(**kwargs)\n        if 'albert_chinese' in self.config.transformer:\n            self.transformer_tokenizer = BertTokenizer.from_pretrained(self.config.transformer, use_fast=True)\n        else:\n            self.transformer_tokenizer = AutoTokenizer_.from_pretrained(self.config.transformer, use_fast=True)\n\n    def build_transformer(self, training=True):\n        transformer = TransformerEncoder(self.config.transformer, self.transformer_tokenizer,\n                                         self.config.average_subwords,\n                                         self.config.scalar_mix, self.config.word_dropout,\n                                         ret_raw_hidden_states=self.config.ret_raw_hidden_states,\n                                         training=training)\n        transformer_layers = self.config.get('transformer_layers', None)\n        if transformer_layers:\n            transformer.transformer.encoder.layer = transformer.transformer.encoder.layer[:transformer_layers]\n        return transformer\n\n\nclass TransformerClassifier(TransformerComponent):\n\n    def __init__(self, **kwargs) -> None:\n        \"\"\"A classifier using transformer as encoder.\n\n        Args:\n            **kwargs: Passed to config.\n        \"\"\"\n        super().__init__(**kwargs)\n        self.model: TransformerClassificationModel = None\n\n    def build_criterion(self, **kwargs):\n        criterion = nn.CrossEntropyLoss()\n        return criterion\n\n    def build_metric(self, **kwargs):\n        return CategoricalAccuracy()\n\n    def execute_training_loop(self, trn: DataLoader, dev: DataLoader, epochs, criterion, optimizer, metric, save_dir,\n                              logger: logging.Logger, devices, **kwargs):\n        best_epoch, best_metric = 0, -1\n        timer = CountdownTimer(epochs)\n        ratio_width = len(f'{len(trn)}/{len(trn)}')\n        for epoch in range(1, epochs + 1):\n            logger.info(f\"[yellow]Epoch {epoch} / {epochs}:[/yellow]\")\n            self.fit_dataloader(trn, criterion, optimizer, metric, logger)\n            if dev:\n                self.evaluate_dataloader(dev, criterion, metric, logger, ratio_width=ratio_width)\n            report = f'{timer.elapsed_human}/{timer.total_time_human}'\n            dev_score = metric.get_metric()\n            if dev_score > best_metric:\n                self.save_weights(save_dir)\n                best_metric = dev_score\n                report += ' [red]saved[/red]'\n            timer.log(report, ratio_percentage=False, newline=True, ratio=False)\n\n    @property\n    def label_vocab(self):\n        return self.vocabs[self.config.label_key]\n\n    def fit_dataloader(self, trn: DataLoader, criterion, optimizer, metric, logger: logging.Logger, **kwargs):\n        self.model.train()\n        timer = CountdownTimer(len(trn))\n        optimizer, scheduler = optimizer\n        total_loss = 0\n        metric.reset()\n        for batch in trn:\n            optimizer.zero_grad()\n            logits = self.feed_batch(batch)\n            target = batch['label_id']\n            loss = self.compute_loss(criterion, logits, target, batch)\n            loss.backward()\n            optimizer.step()\n            scheduler.step()\n            total_loss += loss.item()\n            self.update_metric(metric, logits, target)\n            timer.log(f'loss: {total_loss / (timer.current + 1):.4f} acc: {metric.get_metric():.2%}',\n                      ratio_percentage=None,\n                      logger=logger)\n            del loss\n        return total_loss / timer.total\n\n    def update_metric(self, metric, logits: torch.Tensor, target, output=None):\n        metric(logits, target)\n        if output:\n            label_ids = logits.argmax(-1)\n            return label_ids\n\n    def compute_loss(self, criterion, logits, target, batch):\n        loss = criterion(logits, target)\n        return loss\n\n    def feed_batch(self, batch) -> torch.LongTensor:\n        logits = self.model(*[batch[key] for key in ['input_ids', 'attention_mask', 'token_type_ids']])\n        return logits\n\n    # noinspection PyMethodOverriding\n    def evaluate_dataloader(self,\n                            data: DataLoader,\n                            criterion: Callable,\n                            metric,\n                            logger,\n                            ratio_width=None,\n                            filename=None,\n                            output=None,\n                            **kwargs):\n        self.model.eval()\n        timer = CountdownTimer(len(data))\n        total_loss = 0\n        metric.reset()\n        num_samples = 0\n        if output:\n            output = open(output, 'w')\n        for batch in data:\n            logits = self.feed_batch(batch)\n            target = batch['label_id']\n            loss = self.compute_loss(criterion, logits, target, batch)\n            total_loss += loss.item()\n            label_ids = self.update_metric(metric, logits, target, output)\n            if output:\n                labels = [self.vocabs[self.config.label_key].idx_to_token[i] for i in label_ids.tolist()]\n                for i, label in enumerate(labels):\n                    # text_a text_b pred gold\n                    columns = [batch[self.config.text_a_key][i]]\n                    if self.config.text_b_key:\n                        columns.append(batch[self.config.text_b_key][i])\n                    columns.append(label)\n                    columns.append(batch[self.config.label_key][i])\n                    output.write('\\t'.join(columns))\n                    output.write('\\n')\n            num_samples += len(target)\n            report = f'loss: {total_loss / (timer.current + 1):.4f} acc: {metric.get_metric():.2%}'\n            if filename:\n                report = f'{filename} {report} {num_samples / timer.elapsed:.0f} samples/sec'\n            timer.log(report, ratio_percentage=None, logger=logger, ratio_width=ratio_width)\n        if output:\n            output.close()\n        return total_loss / timer.total\n\n    # noinspection PyMethodOverriding\n    def build_model(self, transformer, training=True, **kwargs) -> torch.nn.Module:\n        # config: PretrainedConfig = AutoConfig.from_pretrained(transformer)\n        # config.num_labels = len(self.vocabs.label)\n        # config.hidden_dropout_prob = self.config.hidden_dropout_prob\n        transformer = self.build_transformer(training=training).transformer\n        model = TransformerClassificationModel(transformer, len(self.vocabs.label))\n        # truncated_normal_(model.classifier.weight, mean=0.02, std=0.05)\n        return model\n\n    # noinspection PyMethodOverriding\n    def build_dataloader(self, data, batch_size, shuffle, device, text_a_key, text_b_key,\n                         label_key,\n                         logger: logging.Logger = None,\n                         sorting=True,\n                         **kwargs) -> DataLoader:\n        if not batch_size:\n            batch_size = self.config.batch_size\n        dataset = self.build_dataset(data)\n        dataset.append_transform(self.vocabs)\n        if self.vocabs.mutable:\n            if not any([text_a_key, text_b_key]):\n                if len(dataset.headers) == 2:\n                    self.config.text_a_key = dataset.headers[0]\n                    self.config.label_key = dataset.headers[1]\n                elif len(dataset.headers) >= 3:\n                    self.config.text_a_key, self.config.text_b_key, self.config.label_key = dataset.headers[0], \\\n                                                                                            dataset.headers[1], \\\n                                                                                            dataset.headers[-1]\n                else:\n                    raise ValueError('Wrong dataset format')\n                report = {'text_a_key', 'text_b_key', 'label_key'}\n                report = dict((k, self.config[k]) for k in report)\n                report = [f'{k}={v}' for k, v in report.items() if v]\n                report = ', '.join(report)\n                logger.info(f'Guess [bold][blue]{report}[/blue][/bold] according to the headers of training dataset: '\n                            f'[blue]{dataset}[/blue]')\n            self.build_vocabs(dataset, logger)\n            dataset.purge_cache()\n        # if self.config.transform:\n        #     dataset.append_transform(self.config.transform)\n        dataset.append_transform(TransformerTextTokenizer(tokenizer=self.transformer_tokenizer,\n                                                          text_a_key=self.config.text_a_key,\n                                                          text_b_key=self.config.text_b_key,\n                                                          max_seq_length=self.config.max_seq_length,\n                                                          truncate_long_sequences=self.config.truncate_long_sequences,\n                                                          output_key=''))\n        batch_sampler = None\n        if sorting and not isdebugging():\n            if dataset.cache and len(dataset) > 1000:\n                timer = CountdownTimer(len(dataset))\n                lens = []\n                for idx, sample in enumerate(dataset):\n                    lens.append(len(sample['input_ids']))\n                    timer.log('Pre-processing and caching dataset [blink][yellow]...[/yellow][/blink]',\n                              ratio_percentage=None)\n            else:\n                lens = [len(sample['input_ids']) for sample in dataset]\n            batch_sampler = SortingSampler(lens, batch_size=batch_size, shuffle=shuffle,\n                                           batch_max_tokens=self.config.batch_max_tokens)\n        return PadSequenceDataLoader(dataset, batch_size, shuffle, batch_sampler=batch_sampler, device=device)\n\n    def build_dataset(self, data) -> TransformableDataset:\n        if isinstance(data, str):\n            dataset = TableDataset(data, cache=True)\n        elif isinstance(data, TableDataset):\n            dataset = data\n        elif isinstance(data, list):\n            dataset = TableDataset(data)\n        else:\n            raise ValueError(f'Unsupported data {data}')\n        return dataset\n\n    def predict(self, data: Union[str, List[str]], batch_size: int = None, **kwargs):\n        if not data:\n            return []\n        flat = isinstance(data, str) or isinstance(data, tuple)\n        if flat:\n            data = [data]\n        samples = []\n        for idx, d in enumerate(data):\n            sample = {IDX: idx}\n            if self.config.text_b_key:\n                sample[self.config.text_a_key] = d[0]\n                sample[self.config.text_b_key] = d[1]\n            else:\n                sample[self.config.text_a_key] = d\n            samples.append(sample)\n        dataloader = self.build_dataloader(samples,\n                                           sorting=False,\n                                           **merge_dict(self.config,\n                                                        batch_size=batch_size,\n                                                        shuffle=False,\n                                                        device=self.device,\n                                                        overwrite=True)\n                                           )\n        labels = [None] * len(data)\n        vocab = self.vocabs.label\n        for batch in dataloader:\n            logits = self.feed_batch(batch)\n            pred = logits.argmax(-1)\n            pred = pred.tolist()\n            for idx, tag in zip(batch[IDX], pred):\n                labels[idx] = vocab.idx_to_token[tag]\n        if flat:\n            return labels[0]\n        return labels\n\n    def fit(self, trn_data, dev_data, save_dir,\n            text_a_key=None,\n            text_b_key=None,\n            label_key=None,\n            transformer=None,\n            max_seq_len=512,\n            truncate_long_sequences=True,\n            # hidden_dropout_prob=0.0,\n            lr=5e-5,\n            transformer_lr=None,\n            adam_epsilon=1e-6,\n            weight_decay=0,\n            warmup_steps=0.1,\n            batch_size=32,\n            batch_max_tokens=None,\n            epochs=3,\n            logger=None,\n            # transform=None,\n            devices: Union[float, int, List[int]] = None,\n            **kwargs):\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    def build_vocabs(self, trn, logger, **kwargs):\n        self.vocabs.label = Vocab(pad_token=None, unk_token=None)\n        for each in trn:\n            pass\n        self.vocabs.lock()\n        self.vocabs.summary(logger)\n"
  },
  {
    "path": "hanlp/components/classifiers/transformer_classifier_hf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2023-02-17 17:54\nimport logging\nfrom typing import List, Union, Callable\n\nimport torch\nfrom torch.utils.data import DataLoader\nfrom transformers import AutoModelForSequenceClassification, PreTrainedTokenizer, AutoTokenizer\n\nfrom hanlp.common.dataset import TableDataset, PadSequenceDataLoader, SortingSamplerBuilder\nfrom hanlp.common.torch_component import TorchComponent\nfrom hanlp_common.constant import IDX\nfrom hanlp_common.util import split_dict, reorder\n\n\nclass TransformerClassifierHF(TorchComponent):\n    def __init__(self, **kwargs) -> None:\n        super().__init__(**kwargs)\n        self._tokenizer: PreTrainedTokenizer = None\n\n    def build_dataloader(self, data, sampler_builder=None, shuffle=False, device=None,\n                         logger: logging.Logger = None,\n                         **kwargs) -> DataLoader:\n        dataset = TableDataset(data)\n        lens = [len(sample['input_ids']) for sample in dataset]\n        if sampler_builder:\n            sampler = sampler_builder.build(lens, shuffle, 1)\n        else:\n            sampler = SortingSamplerBuilder(batch_size=32).build(lens, shuffle, 1)\n        loader = PadSequenceDataLoader(dataset=dataset,\n                                       batch_sampler=sampler,\n                                       pad={'input_ids': self._tokenizer.pad_token_id},\n                                       device=device,\n                                       vocabs=self.vocabs)\n        return loader\n\n    def build_optimizer(self, **kwargs):\n        raise NotImplementedError()\n\n    def build_criterion(self, **kwargs):\n        raise NotImplementedError()\n\n    def build_metric(self, **kwargs):\n        raise NotImplementedError()\n\n    def execute_training_loop(self, trn: DataLoader, dev: DataLoader, epochs, criterion, optimizer, metric, save_dir,\n                              logger: logging.Logger, devices, ratio_width=None, **kwargs):\n        raise NotImplementedError()\n\n    def fit_dataloader(self, trn: DataLoader, criterion, optimizer, metric, logger: logging.Logger, **kwargs):\n        raise NotImplementedError()\n\n    def evaluate_dataloader(self, data: DataLoader, criterion: Callable, metric=None, output=False, **kwargs):\n        raise NotImplementedError()\n\n    def load_vocabs(self, save_dir, filename='vocabs.json'):\n        self._tokenizer = AutoTokenizer.from_pretrained(save_dir)\n\n    def load_weights(self, save_dir, filename='model.pt', **kwargs):\n        pass\n\n    def build_model(self, training=True, save_dir=None, **kwargs) -> torch.nn.Module:\n        return AutoModelForSequenceClassification.from_pretrained(save_dir)\n\n    def predict(self, text: Union[str, List[str]], topk=False, prob=False, **kwargs):\n        \"\"\"\n        Classify text.\n\n        Args:\n            text: A document or a list of documents.\n            topk: ``True`` or ``int`` to return the top-k labels.\n            prob: Return also probabilities.\n            max_len: Strip long document into ``max_len`` characters for faster prediction.\n            **kwargs: Not used\n\n        Returns:\n            Classification results.\n        \"\"\"\n        flat = isinstance(text, str)\n        if flat:\n            text = [text]\n        if not isinstance(topk, list):\n            topk = [topk] * len(text)\n        if not isinstance(prob, list):\n            prob = [prob] * len(text)\n        # noinspection PyTypeChecker\n        dataloader = self.build_dataloader(\n            split_dict(self._tokenizer(text, max_length=self.model.config.max_position_embeddings, truncation=True,\n                                       return_token_type_ids=False, return_attention_mask=False)),\n            device=self.device)\n        results = []\n        order = []\n        id2label = self.model.config.id2label\n        for batch in dataloader:\n            logits = self.model(input_ids=batch['input_ids']).logits\n            logits, batch_labels = logits.sort(descending=True)\n            batch_labels = [[id2label[l] for l in ls] for ls in batch_labels.tolist()]\n            batch_probs = logits.softmax(dim=-1).tolist()\n            for labels, probs, i in zip(batch_labels, batch_probs, batch[IDX]):\n                k = topk[i]\n                p = prob[i]\n                if k is False:\n                    labels = labels[0]\n                elif k is True:\n                    pass\n                elif k:\n                    labels = labels[:k]\n                if p:\n                    if k is False:\n                        result = labels, probs[0]\n                    else:\n                        result = dict(zip(labels, probs))\n                else:\n                    result = labels\n                results.append(result)\n            order.extend(batch[IDX])\n        results = reorder(results, order)\n        if flat:\n            results = results[0]\n        return results\n\n    @property\n    def labels(self):\n        return [x[1] for x in sorted(self.model.config.id2label.items())]\n"
  },
  {
    "path": "hanlp/components/classifiers/transformer_classifier_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-11-10 13:19\n\nimport math\nfrom typing import Union, Tuple, Any, Iterable\nimport tensorflow as tf\nfrom hanlp.common.keras_component import KerasComponent\nfrom hanlp_common.structure import SerializableDict\nfrom hanlp.layers.transformers.loader_tf import build_transformer\nfrom hanlp.optimizers.adamw import create_optimizer\nfrom hanlp.transform.table_tf import TableTransform\nfrom hanlp.utils.log_util import logger\nfrom hanlp_common.util import merge_locals_kwargs\nfrom transformers.tokenization_utils import PreTrainedTokenizer\n\nclass TransformerTextTransform(TableTransform):\n\n    def __init__(self, config: SerializableDict = None, map_x=False, map_y=True, x_columns=None,\n                 y_column=-1, skip_header=True, delimiter='auto', multi_label=False, **kwargs) -> None:\n        super().__init__(config, map_x, map_y, x_columns, y_column, multi_label, skip_header, delimiter, **kwargs)\n        self.tokenizer: PreTrainedTokenizer = None\n\n    def inputs_to_samples(self, inputs, gold=False):\n        tokenizer = self.tokenizer\n        max_length = self.config.max_length\n        num_features = None\n        pad_token = None if self.label_vocab.mutable else tokenizer.convert_tokens_to_ids(['[PAD]'])[0]\n        for (X, Y) in super().inputs_to_samples(inputs, gold):\n            if self.label_vocab.mutable:\n                yield None, Y\n                continue\n            if isinstance(X, str):\n                X = (X,)\n            if num_features is None:\n                num_features = self.config.num_features\n            assert num_features == len(X), f'Numbers of features {num_features} ' \\\n                                           f'inconsistent with current {len(X)}={X}'\n            text_a = X[0]\n            text_b = X[1] if len(X) > 1 else None\n            tokens_a = self.tokenizer.tokenize(text_a)\n            tokens_b = self.tokenizer.tokenize(text_b) if text_b else None\n            tokens = [\"[CLS]\"] + tokens_a + [\"[SEP]\"]\n            segment_ids = [0] * len(tokens)\n            if tokens_b:\n                tokens += tokens_b\n                segment_ids += [1] * len(tokens_b)\n            token_ids = self.tokenizer.convert_tokens_to_ids(tokens)\n            attention_mask = [1] * len(token_ids)\n            diff = max_length - len(token_ids)\n            if diff < 0:\n                # logger.warning(\n                #     f'Input tokens {tokens} exceed the max sequence length of {max_length - 2}. '\n                #     f'The exceeded part will be truncated and ignored. '\n                #     f'You are recommended to split your long text into several sentences within '\n                #     f'{max_length - 2} tokens beforehand.')\n                token_ids = token_ids[:max_length]\n                attention_mask = attention_mask[:max_length]\n                segment_ids = segment_ids[:max_length]\n            elif diff > 0:\n                token_ids += [pad_token] * diff\n                attention_mask += [0] * diff\n                segment_ids += [0] * diff\n\n            assert len(token_ids) == max_length, \"Error with input length {} vs {}\".format(len(token_ids), max_length)\n            assert len(attention_mask) == max_length, \"Error with input length {} vs {}\".format(len(attention_mask),\n                                                                                                max_length)\n            assert len(segment_ids) == max_length, \"Error with input length {} vs {}\".format(len(segment_ids),\n                                                                                             max_length)\n\n            label = Y\n            yield (token_ids, attention_mask, segment_ids), label\n\n    def create_types_shapes_values(self) -> Tuple[Tuple, Tuple, Tuple]:\n        max_length = self.config.max_length\n        types = (tf.int32, tf.int32, tf.int32), tf.string\n        shapes = ([max_length], [max_length], [max_length]), [None, ] if self.config.get('multi_label', None) else []\n        values = (0, 0, 0), self.label_vocab.safe_pad_token\n        return types, shapes, values\n\n    def x_to_idx(self, x) -> Union[tf.Tensor, Tuple]:\n        logger.fatal('map_x should always be set to True')\n        exit(1)\n\n    def y_to_idx(self, y) -> tf.Tensor:\n        if self.config.get('multi_label', None):\n            # need to change index to binary vector\n            mapped = tf.map_fn(fn=lambda x: tf.cast(self.label_vocab.lookup(x), tf.int32), elems=y,\n                               fn_output_signature=tf.TensorSpec(dtype=tf.dtypes.int32, shape=[None, ]))\n            one_hots = tf.one_hot(mapped, len(self.label_vocab))\n            idx = tf.reduce_sum(one_hots, -2)\n        else:\n            idx = self.label_vocab.lookup(y)\n        return idx\n\n    def Y_to_outputs(self, Y: Union[tf.Tensor, Tuple[tf.Tensor]], gold=False, inputs=None, X=None,\n                     batch=None) -> Iterable:\n        # Prediction to be Y > 0:\n        if self.config.get('multi_label', None):\n            preds = Y\n        else:\n            preds = tf.argmax(Y, axis=-1)\n        for y in preds:\n            yield self.label_vocab.idx_to_token[y]\n\n    def input_is_single_sample(self, input: Any) -> bool:\n        return isinstance(input, (str, tuple))\n\n\nclass TransformerClassifierTF(KerasComponent):\n\n    def __init__(self, bert_text_transform=None) -> None:\n        if not bert_text_transform:\n            bert_text_transform = TransformerTextTransform()\n        super().__init__(bert_text_transform)\n        self.model: tf.keras.Model\n        self.transform: TransformerTextTransform = bert_text_transform\n\n    # noinspection PyMethodOverriding\n    def fit(self, trn_data: Any, dev_data: Any, save_dir: str, transformer: str, max_length: int = 128,\n            optimizer='adamw', warmup_steps_ratio=0.1, use_amp=False, batch_size=32,\n            epochs=3, logger=None, verbose=1, **kwargs):\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    def evaluate_output(self, tst_data, out, num_batches, metric):\n        out.write('sentence\\tpred\\tgold\\n')\n        total, correct, score = 0, 0, 0\n        for idx, batch in enumerate(tst_data):\n            outputs = self.model.predict_on_batch(batch[0])\n            outputs = tf.argmax(outputs, axis=1)\n            for X, Y_pred, Y_gold, in zip(batch[0][0], outputs, batch[1]):\n                feature = ' '.join(self.transform.tokenizer.convert_ids_to_tokens(X.numpy()))\n                feature = feature.replace(' ##', '')  # fix sub-word generated by BERT tagger\n                out.write('{}\\t{}\\t{}\\n'.format(feature,\n                                                self._y_id_to_str(Y_pred),\n                                                self._y_id_to_str(Y_gold)))\n                total += 1\n                correct += int(tf.equal(Y_pred, Y_gold).numpy())\n            score = correct / total\n            print('\\r{}/{} {}: {:.2f}'.format(idx + 1, num_batches, metric, score * 100), end='')\n        print()\n        return score\n\n    def _y_id_to_str(self, Y_pred) -> str:\n        return self.transform.label_vocab.idx_to_token[Y_pred.numpy()]\n\n    def build_loss(self, loss, **kwargs):\n        if loss:\n            assert isinstance(loss, tf.keras.losses.loss), 'Must specify loss as an instance in tf.keras.losses'\n            return loss\n        elif self.config.get('multi_label', None):\n            # Loss to be BinaryCrossentropy for multi-label:\n            loss = tf.keras.losses.BinaryCrossentropy(from_logits=True)\n        else:\n            loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n        return loss\n\n    # noinspection PyMethodOverriding\n    def build_optimizer(self, optimizer, use_amp, train_steps, warmup_steps, **kwargs):\n        if optimizer == 'adamw':\n            opt = create_optimizer(init_lr=5e-5, num_train_steps=train_steps, num_warmup_steps=warmup_steps)\n            # opt = tfa.optimizers.AdamW(learning_rate=3e-5, epsilon=1e-08, weight_decay=0.01)\n            # opt = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08)\n            self.config.optimizer = tf.keras.utils.serialize_keras_object(opt)\n            lr_config = self.config.optimizer['config']['learning_rate']['config']\n            if hasattr(lr_config['decay_schedule_fn'], 'get_config'):\n                lr_config['decay_schedule_fn'] = dict(\n                    (k, v) for k, v in lr_config['decay_schedule_fn'].config().items() if not k.startswith('_'))\n        else:\n            opt = super().build_optimizer(optimizer)\n        if use_amp:\n            # loss scaling is currently required when using mixed precision\n            opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(opt, 'dynamic')\n        return opt\n\n    # noinspection PyMethodOverriding\n    def build_model(self, transformer, max_length, **kwargs):\n        model, self.transform.tokenizer = build_transformer(transformer, max_length, len(self.transform.label_vocab),\n                                                            tagging=False)\n        return model\n\n    def build_vocab(self, trn_data, logger):\n        train_examples = super().build_vocab(trn_data, logger)\n        warmup_steps_per_epoch = math.ceil(train_examples * self.config.warmup_steps_ratio / self.config.batch_size)\n        self.config.warmup_steps = warmup_steps_per_epoch * self.config.epochs\n        return train_examples\n\n    def build_metrics(self, metrics, logger, **kwargs):\n        if self.config.get('multi_label', None):\n            metric = tf.keras.metrics.BinaryAccuracy('binary_accuracy')\n        else:\n            metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')\n        return [metric]\n"
  },
  {
    "path": "hanlp/components/classifiers/transformer_regression_hf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2023-02-17 17:54\nimport logging\nfrom typing import List, Union, Callable\n\nimport torch\nfrom torch.utils.data import DataLoader\nfrom transformers import AutoModelForSequenceClassification, PreTrainedTokenizer, AutoTokenizer\n\nfrom hanlp.common.dataset import TableDataset, PadSequenceDataLoader, SortingSamplerBuilder\nfrom hanlp.common.torch_component import TorchComponent\nfrom hanlp_common.constant import IDX\nfrom hanlp_common.util import split_dict, reorder\n\n\nclass TransformerRegressionHF(TorchComponent):\n    def __init__(self, **kwargs) -> None:\n        super().__init__(**kwargs)\n        self._tokenizer: PreTrainedTokenizer = None\n\n    def build_dataloader(self, data, sampler_builder=None, shuffle=False, device=None,\n                         logger: logging.Logger = None,\n                         **kwargs) -> DataLoader:\n        dataset = TableDataset(data)\n        lens = [len(sample['input_ids']) for sample in dataset]\n        if sampler_builder:\n            sampler = sampler_builder.build(lens, shuffle, 1)\n        else:\n            sampler = SortingSamplerBuilder(batch_size=32).build(lens, shuffle, 1)\n        loader = PadSequenceDataLoader(dataset=dataset,\n                                       batch_sampler=sampler,\n                                       pad={'input_ids': self._tokenizer.pad_token_id},\n                                       device=device,\n                                       vocabs=self.vocabs)\n        return loader\n\n    def build_optimizer(self, **kwargs):\n        raise NotImplementedError()\n\n    def build_criterion(self, **kwargs):\n        raise NotImplementedError()\n\n    def build_metric(self, **kwargs):\n        raise NotImplementedError()\n\n    def execute_training_loop(self, trn: DataLoader, dev: DataLoader, epochs, criterion, optimizer, metric, save_dir,\n                              logger: logging.Logger, devices, ratio_width=None, **kwargs):\n        raise NotImplementedError()\n\n    def fit_dataloader(self, trn: DataLoader, criterion, optimizer, metric, logger: logging.Logger, **kwargs):\n        raise NotImplementedError()\n\n    def evaluate_dataloader(self, data: DataLoader, criterion: Callable, metric=None, output=False, **kwargs):\n        raise NotImplementedError()\n\n    def load_vocabs(self, save_dir, filename='vocabs.json'):\n        self._tokenizer = AutoTokenizer.from_pretrained(save_dir)\n\n    def load_weights(self, save_dir, filename='model.pt', **kwargs):\n        pass\n\n    def build_model(self, training=True, save_dir=None, **kwargs) -> torch.nn.Module:\n        return AutoModelForSequenceClassification.from_pretrained(save_dir)\n\n    def predict(self, text: Union[str, List[str]], **kwargs):\n        \"\"\"\n        Classify text.\n\n        Args:\n            text: A document or a list of documents.\n            topk: ``True`` or ``int`` to return the top-k labels.\n            prob: Return also probabilities.\n            max_len: Strip long document into ``max_len`` characters for faster prediction.\n            **kwargs: Not used\n\n        Returns:\n            Classification results.\n        \"\"\"\n        flat = isinstance(text, str)\n        if flat:\n            text = [text]\n        # noinspection PyTypeChecker\n        dataloader = self.build_dataloader(\n            split_dict(self._tokenizer(text, max_length=self.model.config.max_position_embeddings, truncation=True,\n                                       return_token_type_ids=False, return_attention_mask=False)),\n            device=self.device)\n        results = []\n        order = []\n        for batch in dataloader:\n            logits = self.model(input_ids=batch['input_ids']).logits\n            logits = logits.squeeze(-1).clip(-1, 1)\n            logits = logits.tolist()\n            results.extend(logits)\n            order.extend(batch[IDX])\n        results = reorder(results, order)\n        if flat:\n            results = results[0]\n        return results\n"
  },
  {
    "path": "hanlp/components/distillation/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-10-17 20:29\n"
  },
  {
    "path": "hanlp/components/distillation/distillable_component.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-10-17 20:30\nfrom abc import ABC\nfrom copy import copy\n\nimport hanlp\nfrom hanlp.common.torch_component import TorchComponent\nfrom hanlp.components.distillation.losses import KnowledgeDistillationLoss\nfrom hanlp.components.distillation.schedulers import TemperatureScheduler\nfrom hanlp.utils.torch_util import cuda_devices\nfrom hanlp_common.util import merge_locals_kwargs\n\n\nclass DistillableComponent(TorchComponent, ABC):\n\n    # noinspection PyMethodMayBeStatic,PyTypeChecker\n    def build_teacher(self, teacher: str, devices) -> TorchComponent:\n        return hanlp.load(teacher, load_kwargs={'devices': devices})\n\n    def distill(self,\n                teacher: str,\n                trn_data,\n                dev_data,\n                save_dir,\n                batch_size=None,\n                epochs=None,\n                kd_criterion='kd_ce_loss',\n                temperature_scheduler='flsw',\n                devices=None,\n                logger=None,\n                seed=None,\n                **kwargs):\n        devices = devices or cuda_devices()\n        if isinstance(kd_criterion, str):\n            kd_criterion = KnowledgeDistillationLoss(kd_criterion)\n        if isinstance(temperature_scheduler, str):\n            temperature_scheduler = TemperatureScheduler.from_name(temperature_scheduler)\n        teacher = self.build_teacher(teacher, devices=devices)\n        self.vocabs = teacher.vocabs\n        config = copy(teacher.config)\n        batch_size = batch_size or config.get('batch_size', None)\n        epochs = epochs or config.get('epochs', None)\n        config.update(kwargs)\n        return super().fit(**merge_locals_kwargs(locals(),\n                                                 config,\n                                                 excludes=('self', 'kwargs', '__class__', 'config')))\n\n    @property\n    def _savable_config(self):\n        config = super(DistillableComponent, self)._savable_config\n        if 'teacher' in config:\n            config.teacher = config.teacher.load_path\n        return config\n"
  },
  {
    "path": "hanlp/components/distillation/losses.py",
    "content": "# Adopted from https://github.com/airaria/TextBrewer\n# Apache License Version 2.0\n\nimport torch\nimport torch.nn.functional as F\n\nfrom hanlp_common.configurable import AutoConfigurable\n\n\ndef kd_mse_loss(logits_S, logits_T, temperature=1):\n    '''\n    Calculate the mse loss between logits_S and logits_T\n\n    :param logits_S: Tensor of shape (batch_size, length, num_labels) or (batch_size, num_labels)\n    :param logits_T: Tensor of shape (batch_size, length, num_labels) or (batch_size, num_labels)\n    :param temperature: A float or a tensor of shape (batch_size, length) or (batch_size,)\n    '''\n    if isinstance(temperature, torch.Tensor) and temperature.dim() > 0:\n        temperature = temperature.unsqueeze(-1)\n    beta_logits_T = logits_T / temperature\n    beta_logits_S = logits_S / temperature\n    loss = F.mse_loss(beta_logits_S, beta_logits_T)\n    return loss\n\n\ndef kd_ce_loss(logits_S, logits_T, temperature=1):\n    '''\n    Calculate the cross entropy between logits_S and logits_T\n\n    :param logits_S: Tensor of shape (batch_size, length, num_labels) or (batch_size, num_labels)\n    :param logits_T: Tensor of shape (batch_size, length, num_labels) or (batch_size, num_labels)\n    :param temperature: A float or a tensor of shape (batch_size, length) or (batch_size,)\n    '''\n    if isinstance(temperature, torch.Tensor) and temperature.dim() > 0:\n        temperature = temperature.unsqueeze(-1)\n    beta_logits_T = logits_T / temperature\n    beta_logits_S = logits_S / temperature\n    p_T = F.softmax(beta_logits_T, dim=-1)\n    loss = -(p_T * F.log_softmax(beta_logits_S, dim=-1)).sum(dim=-1).mean()\n    return loss\n\n\ndef att_mse_loss(attention_S, attention_T, mask=None):\n    '''\n    * Calculates the mse loss between `attention_S` and `attention_T`.\n    * If the `inputs_mask` is given, masks the positions where ``input_mask==0``.\n\n    :param torch.Tensor logits_S: tensor of shape  (*batch_size*, *num_heads*, *length*, *length*)\n    :param torch.Tensor logits_T: tensor of shape  (*batch_size*, *num_heads*, *length*, *length*)\n    :param torch.Tensor mask: tensor of shape  (*batch_size*, *length*)\n    '''\n    if mask is None:\n        attention_S_select = torch.where(attention_S <= -1e-3, torch.zeros_like(attention_S), attention_S)\n        attention_T_select = torch.where(attention_T <= -1e-3, torch.zeros_like(attention_T), attention_T)\n        loss = F.mse_loss(attention_S_select, attention_T_select)\n    else:\n        mask = mask.to(attention_S).unsqueeze(1).expand(-1, attention_S.size(1), -1)  # (bs, num_of_heads, len)\n        valid_count = torch.pow(mask.sum(dim=2), 2).sum()\n        loss = (F.mse_loss(attention_S, attention_T, reduction='none') * mask.unsqueeze(-1) * mask.unsqueeze(\n            2)).sum() / valid_count\n    return loss\n\n\ndef att_mse_sum_loss(attention_S, attention_T, mask=None):\n    '''\n    * Calculates the mse loss between `attention_S` and `attention_T`. \n    * If the the shape is (*batch_size*, *num_heads*, *length*, *length*), sums along the `num_heads` dimension and then calcuates the mse loss between the two matrices.\n    * If the `inputs_mask` is given, masks the positions where ``input_mask==0``.\n\n    :param torch.Tensor logits_S: tensor of shape  (*batch_size*, *num_heads*, *length*, *length*) or (*batch_size*, *length*, *length*)\n    :param torch.Tensor logits_T: tensor of shape  (*batch_size*, *num_heads*, *length*, *length*) or (*batch_size*, *length*, *length*)\n    :param torch.Tensor mask:     tensor of shape  (*batch_size*, *length*)\n    '''\n    if len(attention_S.size()) == 4:\n        attention_T = attention_T.sum(dim=1)\n        attention_S = attention_S.sum(dim=1)\n    if mask is None:\n        attention_S_select = torch.where(attention_S <= -1e-3, torch.zeros_like(attention_S), attention_S)\n        attention_T_select = torch.where(attention_T <= -1e-3, torch.zeros_like(attention_T), attention_T)\n        loss = F.mse_loss(attention_S_select, attention_T_select)\n    else:\n        mask = mask.to(attention_S)\n        valid_count = torch.pow(mask.sum(dim=1), 2).sum()\n        loss = (F.mse_loss(attention_S, attention_T, reduction='none') * mask.unsqueeze(-1) * mask.unsqueeze(\n            1)).sum() / valid_count\n    return loss\n\n\ndef att_ce_loss(attention_S, attention_T, mask=None):\n    '''\n\n    * Calculates the cross-entropy loss between `attention_S` and `attention_T`, where softmax is to applied on ``dim=-1``.\n    * If the `inputs_mask` is given, masks the positions where ``input_mask==0``.\n    \n    :param torch.Tensor logits_S: tensor of shape  (*batch_size*, *num_heads*, *length*, *length*)\n    :param torch.Tensor logits_T: tensor of shape  (*batch_size*, *num_heads*, *length*, *length*)\n    :param torch.Tensor mask:     tensor of shape  (*batch_size*, *length*)\n    '''\n    probs_T = F.softmax(attention_T, dim=-1)\n    if mask is None:\n        probs_T_select = torch.where(attention_T <= -1e-3, torch.zeros_like(attention_T), probs_T)\n        loss = -((probs_T_select * F.log_softmax(attention_S, dim=-1)).sum(dim=-1)).mean()\n    else:\n        mask = mask.to(attention_S).unsqueeze(1).expand(-1, attention_S.size(1), -1)  # (bs, num_of_heads, len)\n        loss = -((probs_T * F.log_softmax(attention_S, dim=-1) * mask.unsqueeze(2)).sum(\n            dim=-1) * mask).sum() / mask.sum()\n    return loss\n\n\ndef att_ce_mean_loss(attention_S, attention_T, mask=None):\n    '''\n    * Calculates the cross-entropy loss between `attention_S` and `attention_T`, where softmax is to applied on ``dim=-1``.\n    * If the shape is (*batch_size*, *num_heads*, *length*, *length*), averages over dimension `num_heads` and then computes cross-entropy loss between the two matrics.\n    * If the `inputs_mask` is given, masks the positions where ``input_mask==0``.\n    \n    :param torch.tensor logits_S: tensor of shape  (*batch_size*, *num_heads*, *length*, *length*) or (*batch_size*, *length*, *length*)\n    :param torch.tensor logits_T: tensor of shape  (*batch_size*, *num_heads*, *length*, *length*) or (*batch_size*, *length*, *length*)\n    :param torch.tensor mask:     tensor of shape  (*batch_size*, *length*)\n    '''\n    if len(attention_S.size()) == 4:\n        attention_S = attention_S.mean(dim=1)  # (bs, len, len)\n        attention_T = attention_T.mean(dim=1)\n    probs_T = F.softmax(attention_T, dim=-1)\n    if mask is None:\n        probs_T_select = torch.where(attention_T <= -1e-3, torch.zeros_like(attention_T), probs_T)\n        loss = -((probs_T_select * F.log_softmax(attention_S, dim=-1)).sum(dim=-1)).mean()\n    else:\n        mask = mask.to(attention_S)\n        loss = -((probs_T * F.log_softmax(attention_S, dim=-1) * mask.unsqueeze(1)).sum(\n            dim=-1) * mask).sum() / mask.sum()\n    return loss\n\n\ndef hid_mse_loss(state_S, state_T, mask=None):\n    '''\n    * Calculates the mse loss between `state_S` and `state_T`, which are the hidden state of the models.\n    * If the `inputs_mask` is given, masks the positions where ``input_mask==0``.\n    * If the hidden sizes of student and teacher are different, 'proj' option is required in `inetermediate_matches` to match the dimensions.\n\n    :param torch.Tensor state_S: tensor of shape  (*batch_size*, *length*, *hidden_size*)\n    :param torch.Tensor state_T: tensor of shape  (*batch_size*, *length*, *hidden_size*)\n    :param torch.Tensor mask:    tensor of shape  (*batch_size*, *length*)\n    '''\n    if mask is None:\n        loss = F.mse_loss(state_S, state_T)\n    else:\n        mask = mask.to(state_S)\n        valid_count = mask.sum() * state_S.size(-1)\n        loss = (F.mse_loss(state_S, state_T, reduction='none') * mask.unsqueeze(-1)).sum() / valid_count\n    return loss\n\n\ndef cos_loss(state_S, state_T, mask=None):\n    '''\n    * Computes the cosine similarity loss between the inputs. This is the loss used in DistilBERT, see `DistilBERT <https://arxiv.org/abs/1910.01108>`_\n    * If the `inputs_mask` is given, masks the positions where ``input_mask==0``.\n    * If the hidden sizes of student and teacher are different, 'proj' option is required in `inetermediate_matches` to match the dimensions.\n\n    :param torch.Tensor state_S: tensor of shape  (*batch_size*, *length*, *hidden_size*)\n    :param torch.Tensor state_T: tensor of shape  (*batch_size*, *length*, *hidden_size*)\n    :param torch.Tensor mask:    tensor of shape  (*batch_size*, *length*)\n    '''\n    if mask is None:\n        state_S = state_S.view(-1, state_S.size(-1))\n        state_T = state_T.view(-1, state_T.size(-1))\n    else:\n        mask = mask.to(state_S).unsqueeze(-1).expand_as(state_S)  # (bs,len,dim)\n        state_S = torch.masked_select(state_S, mask).view(-1, mask.size(-1))  # (bs * select, dim)\n        state_T = torch.masked_select(state_T, mask).view(-1, mask.size(-1))  # (bs * select, dim)\n\n    target = state_S.new(state_S.size(0)).fill_(1)\n    loss = F.cosine_embedding_loss(state_S, state_T, target, reduction='mean')\n    return loss\n\n\ndef pkd_loss(state_S, state_T, mask=None):\n    '''\n    * Computes normalized vector mse loss at position 0 along `length` dimension. This is the loss used in BERT-PKD, see `Patient Knowledge Distillation for BERT Model Compression <https://arxiv.org/abs/1908.09355>`_.\n    * If the hidden sizes of student and teacher are different, 'proj' option is required in `inetermediate_matches` to match the dimensions.\n\n    :param torch.Tensor state_S: tensor of shape  (*batch_size*, *length*, *hidden_size*)\n    :param torch.Tensor state_T: tensor of shape  (*batch_size*, *length*, *hidden_size*)\n    :param mask: not used.\n    '''\n\n    cls_T = state_T[:, 0]  # (batch_size, hidden_dim)\n    cls_S = state_S[:, 0]  # (batch_size, hidden_dim)\n    normed_cls_T = cls_T / torch.norm(cls_T, dim=1, keepdim=True)\n    normed_cls_S = cls_S / torch.norm(cls_S, dim=1, keepdim=True)\n    loss = (normed_cls_S - normed_cls_T).pow(2).sum(dim=-1).mean()\n    return loss\n\n\ndef fsp_loss(state_S, state_T, mask=None):\n    r'''\n    * Takes in two lists of matrics `state_S` and `state_T`. Each list contains two matrices of the shape (*batch_size*, *length*, *hidden_size*). Computes the similarity matrix between the two matrices in `state_S` ( with the resulting shape (*batch_size*, *hidden_size*, *hidden_size*) ) and the ones in B ( with the resulting shape (*batch_size*, *hidden_size*, *hidden_size*) ), then computes the mse loss between the similarity matrices:\n\n    .. math::\n\n        loss = mean((S_{1}^T \\cdot S_{2} - T_{1}^T \\cdot T_{2})^2)\n\n    * It is a Variant of FSP loss in `A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning <http://openaccess.thecvf.com/content_cvpr_2017/papers/Yim_A_Gift_From_CVPR_2017_paper.pdf>`_.\n    * If the `inputs_mask` is given, masks the positions where ``input_mask==0``.\n    * If the hidden sizes of student and teacher are different, 'proj' option is required in `inetermediate_matches` to match the dimensions.\n\n    :param torch.tensor state_S: list of two tensors, each tensor is of the shape  (*batch_size*, *length*, *hidden_size*)\n    :param torch.tensor state_T: list of two tensors, each tensor is of the shape  (*batch_size*, *length*, *hidden_size*)\n    :param torch.tensor mask:    tensor of the shape  (*batch_size*, *length*)\n\n    Example in `intermediate_matches`::\n\n        intermediate_matches = [\n        {'layer_T':[0,0], 'layer_S':[0,0], 'feature':'hidden','loss': 'fsp', 'weight' : 1, 'proj':['linear',384,768]},\n        ...]\n    '''\n    if mask is None:\n        state_S_0 = state_S[0]  # (batch_size , length, hidden_dim)\n        state_S_1 = state_S[1]  # (batch_size,  length, hidden_dim)\n        state_T_0 = state_T[0]\n        state_T_1 = state_T[1]\n        gram_S = torch.bmm(state_S_0.transpose(1, 2), state_S_1) / state_S_1.size(\n            1)  # (batch_size, hidden_dim, hidden_dim)\n        gram_T = torch.bmm(state_T_0.transpose(1, 2), state_T_1) / state_T_1.size(1)\n    else:\n        mask = mask.to(state_S[0]).unsqueeze(-1)\n        lengths = mask.sum(dim=1, keepdim=True)\n        state_S_0 = state_S[0] * mask\n        state_S_1 = state_S[1] * mask\n        state_T_0 = state_T[0] * mask\n        state_T_1 = state_T[1] * mask\n        gram_S = torch.bmm(state_S_0.transpose(1, 2), state_S_1) / lengths\n        gram_T = torch.bmm(state_T_0.transpose(1, 2), state_T_1) / lengths\n    loss = F.mse_loss(gram_S, gram_T)\n    return loss\n\n\ndef mmd_loss(state_S, state_T, mask=None):\n    r'''\n    * Takes in two lists of matrices `state_S` and `state_T`. Each list contains 2 matrices of the shape (*batch_size*, *length*, *hidden_size*). `hidden_size` of matrices in `State_S` doesn't need to be the same as that of `state_T`. Computes the similarity matrix between the two matrices in `state_S` ( with the resulting shape (*batch_size*, *length*, *length*) ) and the ones in B ( with the resulting shape (*batch_size*, *length*, *length*) ), then computes the mse loss between the similarity matrices:\n    \n    .. math::\n\n            loss = mean((S_{1} \\cdot S_{2}^T - T_{1} \\cdot T_{2}^T)^2)\n\n    * It is a Variant of the NST loss in `Like What You Like: Knowledge Distill via Neuron Selectivity Transfer <https://arxiv.org/abs/1707.01219>`_\n    * If the `inputs_mask` is given, masks the positions where ``input_mask==0``.\n\n    :param torch.tensor state_S: list of two tensors, each tensor is of the shape  (*batch_size*, *length*, *hidden_size*)\n    :param torch.tensor state_T: list of two tensors, each tensor is of the shape  (*batch_size*, *length*, *hidden_size*)\n    :param torch.tensor mask:    tensor of the shape  (*batch_size*, *length*)\n\n    Example in `intermediate_matches`::\n\n        intermediate_matches = [\n        {'layer_T':[0,0], 'layer_S':[0,0], 'feature':'hidden','loss': 'nst', 'weight' : 1},\n        ...]\n    '''\n    state_S_0 = state_S[0]  # (batch_size , length, hidden_dim_S)\n    state_S_1 = state_S[1]  # (batch_size , length, hidden_dim_S)\n    state_T_0 = state_T[0]  # (batch_size , length, hidden_dim_T)\n    state_T_1 = state_T[1]  # (batch_size , length, hidden_dim_T)\n    if mask is None:\n        gram_S = torch.bmm(state_S_0, state_S_1.transpose(1, 2)) / state_S_1.size(2)  # (batch_size, length, length)\n        gram_T = torch.bmm(state_T_0, state_T_1.transpose(1, 2)) / state_T_1.size(2)\n        loss = F.mse_loss(gram_S, gram_T)\n    else:\n        mask = mask.to(state_S[0])\n        valid_count = torch.pow(mask.sum(dim=1), 2).sum()\n        gram_S = torch.bmm(state_S_0, state_S_1.transpose(1, 2)) / state_S_1.size(1)  # (batch_size, length, length)\n        gram_T = torch.bmm(state_T_0, state_T_1.transpose(1, 2)) / state_T_1.size(1)\n        loss = (F.mse_loss(gram_S, gram_T, reduction='none') * mask.unsqueeze(-1) * mask.unsqueeze(\n            1)).sum() / valid_count\n    return loss\n\n\nclass KnowledgeDistillationLoss(AutoConfigurable):\n    def __init__(self, name) -> None:\n        super().__init__()\n        self.name = name\n        import sys\n        thismodule = sys.modules[__name__]\n        self._loss = getattr(thismodule, name)\n\n    def __call__(self, *args, **kwargs):\n        return self._loss(*args, **kwargs)\n"
  },
  {
    "path": "hanlp/components/distillation/schedulers.py",
    "content": "# Adopted from https://github.com/airaria/TextBrewer\n# Apache License Version 2.0\nfrom abc import ABC, abstractmethod\n\nimport torch\n\n# x is between 0 and 1\nfrom hanlp_common.configurable import AutoConfigurable\n\n\ndef linear_growth_weight_scheduler(x):\n    return x\n\n\ndef linear_decay_weight_scheduler(x):\n    return 1 - x\n\n\ndef constant_temperature_scheduler(logits_S, logits_T, base_temperature):\n    '''\n    Remember to detach logits_S \n    '''\n    return base_temperature\n\n\ndef flsw_temperature_scheduler_builder(beta, gamma, eps=1e-4, *args):\n    '''\n    adapted from arXiv:1911.07471\n    '''\n\n    def flsw_temperature_scheduler(logits_S, logits_T, base_temperature):\n        v = logits_S.detach()\n        t = logits_T.detach()\n        with torch.no_grad():\n            v = v / (torch.norm(v, dim=-1, keepdim=True) + eps)\n            t = t / (torch.norm(t, dim=-1, keepdim=True) + eps)\n            w = torch.pow((1 - (v * t).sum(dim=-1)), gamma)\n            tau = base_temperature + (w.mean() - w) * beta\n        return tau\n\n    return flsw_temperature_scheduler\n\n\ndef cwsm_temperature_scheduler_builder(beta, *args):\n    '''\n    adapted from arXiv:1911.07471\n    '''\n\n    def cwsm_temperature_scheduler(logits_S, logits_T, base_temperature):\n        v = logits_S.detach()\n        with torch.no_grad():\n            v = torch.softmax(v, dim=-1)\n            v_max = v.max(dim=-1)[0]\n            w = 1 / (v_max + 1e-3)\n            tau = base_temperature + (w.mean() - w) * beta\n        return tau\n\n    return cwsm_temperature_scheduler\n\n\nclass LinearTeacherAnnealingScheduler(object):\n    def __init__(self, num_training_steps: int) -> None:\n        super().__init__()\n        self._num_training_steps = num_training_steps\n        self._current_training_steps = 0\n\n    def step(self):\n        self._current_training_steps += 1\n\n    def __float__(self):\n        return self._current_training_steps / self._num_training_steps\n\n\nclass TemperatureScheduler(ABC, AutoConfigurable):\n\n    def __init__(self, base_temperature) -> None:\n        super().__init__()\n        self.base_temperature = base_temperature\n\n    def __call__(self, logits_S, logits_T):\n        return self.forward(logits_S, logits_T)\n\n    @abstractmethod\n    def forward(self, logits_S, logits_T):\n        raise NotImplementedError()\n\n    @staticmethod\n    def from_name(name):\n        classes = {\n            'constant': ConstantScheduler,\n            'flsw': FlswScheduler,\n            'cwsm': CwsmScheduler,\n        }\n        assert name in classes, f'Unsupported temperature scheduler {name}. Expect one from {list(classes.keys())}.'\n        return classes[name]()\n\n\nclass FunctionalScheduler(TemperatureScheduler):\n\n    def __init__(self, scheduler_func, base_temperature) -> None:\n        super().__init__(base_temperature)\n        self._scheduler_func = scheduler_func\n\n    def forward(self, logits_S, logits_T):\n        return self._scheduler_func(logits_S, logits_T, self.base_temperature)\n\n\nclass ConstantScheduler(TemperatureScheduler):\n    def forward(self, logits_S, logits_T):\n        return self.base_temperature\n\n\nclass FlswScheduler(FunctionalScheduler):\n    def __init__(self, beta=1, gamma=1, eps=1e-4, base_temperature=8):\n        super().__init__(flsw_temperature_scheduler_builder(beta, gamma, eps), base_temperature)\n        self.beta = beta\n        self.gamma = gamma\n        self.eps = eps\n\n\nclass CwsmScheduler(FunctionalScheduler):\n    def __init__(self, beta=1, base_temperature=8):\n        super().__init__(cwsm_temperature_scheduler_builder(beta), base_temperature)\n        self.beta = beta\n"
  },
  {
    "path": "hanlp/components/eos/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-07-26 20:19"
  },
  {
    "path": "hanlp/components/eos/ngram.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-07-26 20:19\nimport logging\nfrom collections import Counter\nfrom typing import Union, List, Callable\n\nimport torch\nfrom torch import nn, optim\nfrom torch.nn import BCEWithLogitsLoss\nfrom torch.utils.data import DataLoader\n\nfrom hanlp.common.dataset import PadSequenceDataLoader\nfrom hanlp.common.torch_component import TorchComponent\nfrom hanlp.common.vocab import Vocab\nfrom hanlp.datasets.eos.eos import SentenceBoundaryDetectionDataset\nfrom hanlp.metrics.f1 import F1\nfrom hanlp.utils.time_util import CountdownTimer\nfrom hanlp_common.util import merge_locals_kwargs\n\n\nclass NgramSentenceBoundaryDetectionModel(nn.Module):\n\n    def __init__(self,\n                 char_vocab_size,\n                 embedding_size=128,\n                 rnn_type: str = 'LSTM',\n                 rnn_size=256,\n                 rnn_layers=1,\n                 rnn_bidirectional=False,\n                 dropout=0.2,\n                 **kwargs\n                 ):\n        super(NgramSentenceBoundaryDetectionModel, self).__init__()\n        self.embed = nn.Embedding(num_embeddings=char_vocab_size,\n                                  embedding_dim=embedding_size)\n        rnn_type = rnn_type.lower()\n        if rnn_type == 'lstm':\n            self.rnn = nn.LSTM(input_size=embedding_size,\n                               hidden_size=rnn_size,\n                               num_layers=rnn_layers,\n                               dropout=self.dropout if rnn_layers > 1 else 0.0,\n                               bidirectional=rnn_bidirectional,\n                               batch_first=True)\n        elif rnn_type == 'gru':\n            self.rnn = nn.GRU(input_size=self.embdding_size,\n                              hidden_size=rnn_size,\n                              num_layers=rnn_layers,\n                              dropout=self.dropout if rnn_layers > 1 else 0.0,\n                              bidirectional=rnn_bidirectional,\n                              batch_first=True)\n        else:\n            raise NotImplementedError(f\"'{rnn_type}' has to be one of [LSTM, GRU]\")\n        self.dropout = nn.Dropout(p=dropout) if dropout else None\n        self.dense = nn.Linear(in_features=rnn_size * (2 if rnn_bidirectional else 1),\n                               out_features=1)\n\n    def forward(self, x: torch.Tensor):\n        output = self.embed(x)\n        self.rnn.flatten_parameters()\n        output, _ = self.rnn(output)\n        if self.dropout:\n            output = self.dropout(output[:, -1, :])\n        output = output.squeeze(1)\n        output = self.dense(output).squeeze(-1)\n        return output\n\n\nclass NgramSentenceBoundaryDetector(TorchComponent):\n\n    def __init__(self, **kwargs) -> None:\n        \"\"\"A sentence boundary detector using ngram as features and LSTM as encoder (:cite:`Schweter:Ahmed:2019`).\n        It predicts whether a punctuation marks an ``EOS``.\n\n        .. Note::\n            This component won't work on text without the punctuations defined in its config. It's always\n            recommended to understand how it works before using it. The predefined punctuations can be listed by the\n            following codes.\n\n            >>> print(eos.config.eos_chars)\n\n        Args:\n            **kwargs: Passed to config.\n        \"\"\"\n        super().__init__(**kwargs)\n\n    def build_optimizer(self, **kwargs):\n        optimizer = optim.Adam(self.model.parameters(), lr=self.config.lr)\n        return optimizer\n\n    def build_criterion(self, **kwargs):\n        return BCEWithLogitsLoss()\n\n    def build_metric(self, **kwargs):\n        return F1()\n\n    def execute_training_loop(self,\n                              trn: DataLoader,\n                              dev: DataLoader,\n                              epochs,\n                              criterion,\n                              optimizer,\n                              metric,\n                              save_dir,\n                              logger: logging.Logger,\n                              devices,\n                              **kwargs):\n        best_epoch, best_metric = 0, -1\n        timer = CountdownTimer(epochs)\n        ratio_width = len(f'{len(trn)}/{len(trn)}')\n        for epoch in range(1, epochs + 1):\n            logger.info(f\"[yellow]Epoch {epoch} / {epochs}:[/yellow]\")\n            self.fit_dataloader(trn, criterion, optimizer, metric, logger)\n            if dev:\n                self.evaluate_dataloader(dev, criterion, metric, logger, ratio_width=ratio_width)\n            report = f'{timer.elapsed_human}/{timer.total_time_human}'\n            dev_score = metric.score\n            if dev_score > best_metric:\n                self.save_weights(save_dir)\n                best_metric = dev_score\n                report += ' [red]saved[/red]'\n            timer.log(report, ratio_percentage=False, newline=True, ratio=False)\n\n    def fit_dataloader(self,\n                       trn: DataLoader,\n                       criterion,\n                       optimizer,\n                       metric,\n                       logger: logging.Logger,\n                       **kwargs):\n        self.model.train()\n        timer = CountdownTimer(len(trn))\n        total_loss = 0\n        self.reset_metrics(metric)\n        for batch in trn:\n            optimizer.zero_grad()\n            prediction = self.feed_batch(batch)\n            loss = self.compute_loss(prediction, batch, criterion)\n            self.update_metrics(batch, prediction, metric)\n            loss.backward()\n            if self.config.grad_norm:\n                torch.nn.utils.clip_grad_norm_(self.model.parameters(), self.config.grad_norm)\n            optimizer.step()\n            total_loss += loss.item()\n            timer.log(self.report_metrics(total_loss / (timer.current + 1), metric), ratio_percentage=None,\n                      logger=logger)\n            del loss\n        return total_loss / timer.total\n\n    def compute_loss(self, prediction, batch, criterion):\n        loss = criterion(prediction, batch['label_id'])\n        return loss\n\n    # noinspection PyMethodOverriding\n    def evaluate_dataloader(self,\n                            data: DataLoader,\n                            criterion: Callable,\n                            metric,\n                            logger,\n                            ratio_width=None,\n                            output=False,\n                            **kwargs):\n        self.model.eval()\n        self.reset_metrics(metric)\n        timer = CountdownTimer(len(data))\n        total_loss = 0\n        for batch in data:\n            prediction = self.feed_batch(batch)\n            self.update_metrics(batch, prediction, metric)\n            loss = self.compute_loss(prediction, batch, criterion)\n            total_loss += loss.item()\n            timer.log(self.report_metrics(total_loss / (timer.current + 1), metric), ratio_percentage=None,\n                      logger=logger,\n                      ratio_width=ratio_width)\n            del loss\n        return total_loss / timer.total, metric\n\n    def build_model(self, training=True, **kwargs) -> torch.nn.Module:\n        model = NgramSentenceBoundaryDetectionModel(**self.config, char_vocab_size=len(self.vocabs.char))\n        return model\n\n    def build_dataloader(self, data, batch_size, shuffle, device, logger: logging.Logger, **kwargs) -> DataLoader:\n        dataset = SentenceBoundaryDetectionDataset(data, **self.config, transform=[self.vocabs])\n        if isinstance(data, str):\n            dataset.purge_cache()\n        if not self.vocabs:\n            self.build_vocabs(dataset, logger)\n        return PadSequenceDataLoader(dataset, batch_size=batch_size, shuffle=shuffle, device=device,\n                                     pad={'label_id': .0})\n\n    def predict(self, data: Union[str, List[str]], batch_size: int = None, strip=True, **kwargs):\n        \"\"\"Sentence split.\n\n        Args:\n            data: A paragraph or a list of paragraphs.\n            batch_size: Number of samples per batch.\n            strip: Strip out blank characters at the head and tail of each sentence.\n\n        Returns:\n            A list of sentences or a list of lists of sentences.\n        \"\"\"\n        if not data:\n            return []\n        self.model.eval()\n        flat = isinstance(data, str)\n        if flat:\n            data = [data]\n        samples = []\n        eos_chars = self.config.eos_chars\n        window_size = self.config.window_size\n        for doc_id_, corpus in enumerate(data):\n            corpus = list(corpus)\n            for i, c in enumerate(corpus):\n                if c in eos_chars:\n                    window = corpus[max(0, i - window_size): i + window_size + 1]\n                    samples.append({'char': window, 'offset_': i, 'doc_id_': doc_id_})\n        eos_prediction = [[] for _ in range(len(data))]\n        if samples:\n            dataloader = self.build_dataloader(samples, **self.config, device=self.device, shuffle=False, logger=None)\n            for batch in dataloader:\n                logits = self.feed_batch(batch)\n                prediction = (logits > 0).tolist()\n                for doc_id_, offset_, eos in zip(batch['doc_id_'], batch['offset_'], prediction):\n                    if eos:\n                        eos_prediction[doc_id_].append(offset_)\n        outputs = []\n        for corpus, output in zip(data, eos_prediction):\n            sents_per_document = []\n            prev_offset = 0\n            for offset in output:\n                offset += 1\n                sents_per_document.append(corpus[prev_offset:offset])\n                prev_offset = offset\n            if prev_offset != len(corpus):\n                sents_per_document.append(corpus[prev_offset:])\n            if strip:\n                sents_per_document = [x.strip() for x in sents_per_document]\n            sents_per_document = [x for x in sents_per_document if x]\n            outputs.append(sents_per_document)\n        if flat:\n            outputs = outputs[0]\n        return outputs\n\n    # noinspection PyMethodOverriding\n    def fit(self,\n            trn_data,\n            dev_data,\n            save_dir,\n            epochs=5,\n            append_after_sentence=None,\n            eos_chars=None,\n            eos_char_min_freq=200,\n            eos_char_is_punct=True,\n            char_min_freq=None,\n            window_size=5,\n            batch_size=32,\n            lr=0.001,\n            grad_norm=None,\n            loss_reduction='sum',\n            embedding_size=128,\n            rnn_type: str = 'LSTM',\n            rnn_size=256,\n            rnn_layers=1,\n            rnn_bidirectional=False,\n            dropout=0.2,\n            devices=None,\n            logger=None,\n            seed=None,\n            **kwargs\n            ):\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    def build_vocabs(self, dataset: SentenceBoundaryDetectionDataset, logger, **kwargs):\n        char_min_freq = self.config.char_min_freq\n        if char_min_freq:\n            has_cache = dataset.cache is not None\n            char_counter = Counter()\n            for each in dataset:\n                for c in each['char']:\n                    char_counter[c] += 1\n            self.vocabs.char = vocab = Vocab()\n            for c, f in char_counter.items():\n                if f >= char_min_freq:\n                    vocab.add(c)\n            if has_cache:\n                dataset.purge_cache()\n                for each in dataset:\n                    pass\n        else:\n            self.vocabs.char = Vocab()\n            for each in dataset:\n                pass\n        self.config.eos_chars = dataset.eos_chars\n        self.vocabs.lock()\n        self.vocabs.summary(logger)\n\n    def reset_metrics(self, metrics):\n        metrics.reset()\n\n    def report_metrics(self, loss, metrics):\n        return f'loss: {loss:.4f} {metrics}'\n\n    def update_metrics(self, batch: dict, prediction: torch.FloatTensor, metrics):\n        def nonzero_offsets(y):\n            return set(y.nonzero().squeeze(-1).tolist())\n\n        metrics(nonzero_offsets(prediction > 0), nonzero_offsets(batch['label_id']))\n\n    def feed_batch(self, batch):\n        prediction = self.model(batch['char_id'])\n        return prediction\n"
  },
  {
    "path": "hanlp/components/lambda_wrapper.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-31 18:36\nfrom typing import Callable, Any\n\nfrom hanlp.common.component import Component\nfrom hanlp_common.reflection import classpath_of, object_from_classpath, str_to_type\n\n\nclass LambdaComponent(Component):\n    def __init__(self, function: Callable) -> None:\n        super().__init__()\n        self.config = {}\n        self.function = function\n        self.config['function'] = classpath_of(function)\n        self.config['classpath'] = classpath_of(self)\n\n    def predict(self, data: Any, **kwargs):\n        unpack = kwargs.pop('_hanlp_unpack', None)\n        if unpack:\n            return self.function(*data, **kwargs)\n        return self.function(data, **kwargs)\n\n    @staticmethod\n    def from_config(meta: dict, **kwargs):\n        cls = str_to_type(meta['classpath'])\n        function = meta['function']\n        function = object_from_classpath(function)\n        return cls(function)\n"
  },
  {
    "path": "hanlp/components/lemmatizer.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-08 18:35\nfrom typing import List\n\nfrom hanlp.common.transform import TransformList\nfrom hanlp.components.parsers.ud.lemma_edit import gen_lemma_rule, apply_lemma_rule\nfrom hanlp.components.taggers.transformers.transformer_tagger import TransformerTagger\n\n\ndef add_lemma_rules_to_sample(sample: dict):\n    if 'tag' in sample and 'lemma' not in sample:\n        lemma_rules = [gen_lemma_rule(word, lemma)\n                       if lemma != \"_\" else \"_\"\n                       for word, lemma in zip(sample['token'], sample['tag'])]\n        sample['lemma'] = sample['tag'] = lemma_rules\n    return sample\n\n\nclass TransformerLemmatizer(TransformerTagger):\n\n    def __init__(self, **kwargs) -> None:\n        \"\"\"A transition based lemmatizer using transformer as encoder.\n\n        Args:\n            **kwargs: Predefined config.\n        \"\"\"\n        super().__init__(**kwargs)\n\n    def build_dataset(self, data, transform=None, **kwargs):\n        if not isinstance(transform, list):\n            transform = TransformList()\n        transform.append(add_lemma_rules_to_sample)\n        return super().build_dataset(data, transform, **kwargs)\n\n    def prediction_to_human(self, pred, vocab: List[str], batch, token=None):\n        if token is None:\n            token = batch['token']\n        rules = super().prediction_to_human(pred, vocab, batch)\n        for token_per_sent, rule_per_sent in zip(token, rules):\n            lemma_per_sent = [apply_lemma_rule(t, r) for t, r in zip(token_per_sent, rule_per_sent)]\n            yield lemma_per_sent\n"
  },
  {
    "path": "hanlp/components/lm/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2022-01-29 21:07\n"
  },
  {
    "path": "hanlp/components/lm/mlm.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2022-01-29 21:07\nimport logging\nimport math\nfrom typing import Callable, Union, List\n\nimport torch\nfrom hanlp_common.constant import IDX\nfrom hanlp_common.util import reorder\nfrom torch.utils.data import DataLoader\nfrom transformers import AutoModelForMaskedLM\nfrom transformers.tokenization_utils import PreTrainedTokenizer\n\nfrom hanlp.common.dataset import TransformableDataset, PadSequenceDataLoader, SortingSampler\nfrom hanlp.common.torch_component import TorchComponent\nfrom hanlp.layers.transformers.pt_imports import AutoTokenizer_\nfrom hanlp.transform.transformer_tokenizer import TransformerTextTokenizer\nfrom hanlp.utils.time_util import CountdownTimer\n\n\nclass MaskedLanguageModelDataset(TransformableDataset):\n\n    def load_file(self, filepath: str):\n        raise NotImplementedError()\n\n\nclass MaskedLanguageModel(TorchComponent):\n\n    def __init__(self, **kwargs) -> None:\n        super().__init__(**kwargs)\n        self.tokenizer: PreTrainedTokenizer = None\n\n    def build_dataloader(self, data, batch_size, shuffle=False, device=None, logger: logging.Logger = None,\n                         verbose=False, **kwargs) -> DataLoader:\n        dataset = MaskedLanguageModelDataset([{'token': x} for x in data], generate_idx=True,\n                                             transform=TransformerTextTokenizer(self.tokenizer, text_a_key='token'))\n        if verbose:\n            verbose = CountdownTimer(len(dataset))\n        lens = []\n        for each in dataset:\n            lens.append(len(each['token_input_ids']))\n            if verbose:\n                verbose.log('Preprocessing and caching samples [blink][yellow]...[/yellow][/blink]')\n        dataloader = PadSequenceDataLoader(dataset, batch_sampler=SortingSampler(lens, batch_size=batch_size),\n                                           device=device)\n        return dataloader\n\n    def build_optimizer(self, **kwargs):\n        raise NotImplementedError()\n\n    def build_criterion(self, **kwargs):\n        raise NotImplementedError()\n\n    def build_metric(self, **kwargs):\n        raise NotImplementedError()\n\n    def execute_training_loop(self, trn: DataLoader, dev: DataLoader, epochs, criterion, optimizer, metric, save_dir,\n                              logger: logging.Logger, devices, ratio_width=None, **kwargs):\n        raise NotImplementedError()\n\n    def fit_dataloader(self, trn: DataLoader, criterion, optimizer, metric, logger: logging.Logger, **kwargs):\n        raise NotImplementedError()\n\n    def evaluate_dataloader(self, data: DataLoader, criterion: Callable, metric=None, output=False, **kwargs):\n        raise NotImplementedError()\n\n    def build_model(self, training=True, transformer=None, **kwargs) -> torch.nn.Module:\n        return AutoModelForMaskedLM.from_pretrained(transformer)\n\n    def input_is_flat(self, masked_sents):\n        return isinstance(masked_sents, str)\n\n    def predict(self, masked_sents: Union[str, List[str]], batch_size=32, topk=10, **kwargs):\n        flat = self.input_is_flat(masked_sents)\n        if flat:\n            masked_sents = [masked_sents]\n        dataloader = self.build_dataloader(masked_sents, **self.config, device=self.device, batch_size=batch_size)\n        orders = []\n        results = []\n        for batch in dataloader:\n            input_ids = batch['token_input_ids']\n            outputs = self.model(input_ids=input_ids, attention_mask=batch['token_attention_mask'])\n            mask = input_ids == self.tokenizer.mask_token_id\n            if mask.any():\n                num_masks = mask.sum(dim=-1).tolist()\n                masked_logits = outputs.logits[mask]\n                masked_logits[:, self.tokenizer.all_special_ids] = -math.inf\n                probs, indices = torch.nn.functional.softmax(masked_logits, dim=-1).topk(topk)\n                br = []\n                for p, index in zip(probs.tolist(), indices.tolist()):\n                    br.append(dict(zip(self.tokenizer.convert_ids_to_tokens(index), p)))\n                offset = 0\n                for n in num_masks:\n                    results.append(br[offset:offset + n])\n                    offset += n\n            else:\n                results.extend([[]] * input_ids.size(0))\n            orders.extend(batch[IDX])\n        results = reorder(results, orders)\n        if flat:\n            results = results[0]\n        return results\n\n    def load_config(self, save_dir, filename='config.json', **kwargs):\n        self.config.transformer = save_dir\n\n    def load_vocabs(self, save_dir, filename='vocabs.json'):\n        self.tokenizer = AutoTokenizer_.from_pretrained(self.config.transformer)\n\n    def load_weights(self, save_dir, filename='model.pt', **kwargs):\n        pass\n"
  },
  {
    "path": "hanlp/components/mtl/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-20 19:54"
  },
  {
    "path": "hanlp/components/mtl/multi_task_learning.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-20 19:55\nimport functools\nimport itertools\nimport logging\nimport os\nfrom collections import defaultdict\nfrom copy import copy\nfrom itertools import chain\nfrom typing import Union, List, Callable, Dict, Optional, Any, Iterable, Tuple\n\nimport numpy as np\nimport torch\nfrom hanlp_common.constant import IDX, BOS, EOS\nfrom hanlp_common.document import Document\nfrom hanlp_common.util import merge_locals_kwargs, topological_sort, reorder, prefix_match\nfrom hanlp_common.visualization import markdown_table\nfrom toposort import toposort\nfrom torch.utils.data import DataLoader\n\nimport hanlp.utils.torch_util\nfrom hanlp.common.dataset import PadSequenceDataLoader, PrefetchDataLoader, CachedDataLoader\nfrom hanlp.common.structure import History\nfrom hanlp.common.torch_component import TorchComponent\nfrom hanlp.common.transform import FieldLength, TransformList\nfrom hanlp.components.mtl.tasks import Task\nfrom hanlp.layers.embeddings.contextual_word_embedding import ContextualWordEmbedding, ContextualWordEmbeddingModule\nfrom hanlp.layers.embeddings.embedding import Embedding\nfrom hanlp.layers.transformers.utils import pick_tensor_for_each_token\nfrom hanlp.metrics.metric import Metric\nfrom hanlp.metrics.mtl import MetricDict\nfrom hanlp.transform.transformer_tokenizer import TransformerSequenceTokenizer\nfrom hanlp.utils.time_util import CountdownTimer\nfrom hanlp.utils.torch_util import clip_grad_norm\n\n\nclass MultiTaskModel(torch.nn.Module):\n\n    def __init__(self,\n                 encoder: torch.nn.Module,\n                 scalar_mixes: torch.nn.ModuleDict,\n                 decoders: torch.nn.ModuleDict,\n                 use_raw_hidden_states: dict) -> None:\n        super().__init__()\n        self.use_raw_hidden_states = use_raw_hidden_states\n        self.encoder: ContextualWordEmbeddingModule = encoder\n        self.scalar_mixes = scalar_mixes\n        self.decoders = decoders\n\n\nclass MultiTaskDataLoader(DataLoader):\n\n    def __init__(self, training=True, tau: float = 0.8, **dataloaders) -> None:\n        # noinspection PyTypeChecker\n        super().__init__(None)\n        self.tau = tau\n        self.training = training\n        self.dataloaders: Dict[str, DataLoader] = dataloaders if dataloaders else {}\n        # self.iterators = dict((k, iter(v)) for k, v in dataloaders.items())\n\n    def __len__(self) -> int:\n        if self.dataloaders:\n            return sum(len(x) for x in self.dataloaders.values())\n        return 0\n\n    def __iter__(self):\n        if self.training:\n            sampling_weights, total_size = self.sampling_weights\n            task_names = list(self.dataloaders.keys())\n            iterators = dict((k, itertools.cycle(v)) for k, v in self.dataloaders.items())\n            for i in range(total_size):\n                task_name = np.random.choice(task_names, p=sampling_weights)\n                yield task_name, next(iterators[task_name])\n        else:\n            for task_name, dataloader in self.dataloaders.items():\n                for batch in dataloader:\n                    yield task_name, batch\n\n    @property\n    def sampling_weights(self):\n        sampling_weights = self.sizes\n        total_size = sum(sampling_weights)\n        Z = sum(pow(v, self.tau) for v in sampling_weights)\n        sampling_weights = [pow(v, self.tau) / Z for v in sampling_weights]\n        return sampling_weights, total_size\n\n    @property\n    def sizes(self):\n        return [len(v) for v in self.dataloaders.values()]\n\n\nclass MultiTaskLearning(TorchComponent):\n\n    def __init__(self, **kwargs) -> None:\n        \"\"\" A multi-task learning (MTL) framework. It shares the same encoder across multiple decoders. These decoders\n        can have dependencies on each other which will be properly handled during decoding. To integrate a component\n        into this MTL framework, a component needs to implement the :class:`~hanlp.components.mtl.tasks.Task` interface.\n\n        This framework mostly follows the architecture of :cite:`clark-etal-2019-bam` and :cite:`he-choi-2021-stem`, with additional scalar mix\n        tricks (:cite:`kondratyuk-straka-2019-75`) allowing each task to attend to any subset of layers. We also\n        experimented with knowledge distillation on single tasks, the performance gain was nonsignificant on a large\n        dataset. In the near future, we have no plan to invest more efforts in distillation, since most datasets HanLP\n        uses are relatively large, and our hardware is relatively powerful.\n\n        Args:\n            **kwargs: Arguments passed to config.\n        \"\"\"\n        super().__init__(**kwargs)\n        self.model: Optional[MultiTaskModel] = None\n        self.tasks: Dict[str, Task] = None\n        self.vocabs = None\n\n    def build_dataloader(self,\n                         data,\n                         batch_size,\n                         shuffle=False,\n                         device=None,\n                         logger: logging.Logger = None,\n                         gradient_accumulation=1,\n                         tau: float = 0.8,\n                         prune=None,\n                         prefetch=None,\n                         tasks_need_custom_eval=None,\n                         cache=False,\n                         debug=False,\n                         **kwargs) -> DataLoader:\n        # This method is only called during training or evaluation but not prediction\n        dataloader = MultiTaskDataLoader(training=shuffle, tau=tau)\n        for i, (task_name, task) in enumerate(self.tasks.items()):\n            encoder_transform, transform = self.build_transform(task)\n            training = None\n            if data == 'trn':\n                if debug:\n                    _data = task.dev\n                else:\n                    _data = task.trn\n                training = True\n            elif data == 'dev':\n                _data = task.dev\n                training = False\n            elif data == 'tst':\n                _data = task.tst\n                training = False\n            else:\n                _data = data\n            if isinstance(data, str):\n                logger.info(f'[yellow]{i + 1} / {len(self.tasks)}[/yellow] Building [blue]{data}[/blue] dataset for '\n                            f'[cyan]{task_name}[/cyan] ...')\n            # Adjust Tokenizer according to task config\n            config = copy(task.config)\n            config.pop('transform', None)\n            task_dataloader: DataLoader = task.build_dataloader(_data, transform, training, device, logger,\n                                                                tokenizer=encoder_transform.tokenizer,\n                                                                gradient_accumulation=gradient_accumulation,\n                                                                cache=isinstance(data, str), **config)\n            # if prune:\n            #     # noinspection PyTypeChecker\n            #     task_dataset: TransformDataset = task_dataloader.dataset\n            #     size_before = len(task_dataset)\n            #     task_dataset.prune(prune)\n            #     size_after = len(task_dataset)\n            #     num_pruned = size_before - size_after\n            #     logger.info(f'Pruned [yellow]{num_pruned} ({num_pruned / size_before:.1%})[/yellow] '\n            #                 f'samples out of {size_before}.')\n            if cache and data in ('trn', 'dev'):\n                task_dataloader: CachedDataLoader = CachedDataLoader(\n                    task_dataloader,\n                    f'{cache}/{os.getpid()}-{data}-{task_name.replace(\"/\", \"-\")}-cache.pt' if isinstance(cache,\n                                                                                                         str) else None\n                )\n            dataloader.dataloaders[task_name] = task_dataloader\n        if data == 'trn':\n            sampling_weights, total_size = dataloader.sampling_weights\n            headings = ['task', '#batches', '%batches', '#scaled', '%scaled', '#epoch']\n            matrix = []\n            min_epochs = []\n            for (task_name, dataset), weight in zip(dataloader.dataloaders.items(), sampling_weights):\n                epochs = len(dataset) / weight / total_size\n                matrix.append(\n                    [f'{task_name}', len(dataset), f'{len(dataset) / total_size:.2%}', int(total_size * weight),\n                     f'{weight:.2%}', f'{epochs:.2f}'])\n                min_epochs.append(epochs)\n            longest = int(torch.argmax(torch.tensor(min_epochs)))\n            table = markdown_table(headings, matrix)\n            rows = table.splitlines()\n            cells = rows[longest + 2].split('|')\n            cells[-2] = cells[-2].replace(f'{min_epochs[longest]:.2f}',\n                                          f'[bold][red]{min_epochs[longest]:.2f}[/red][/bold]')\n            rows[longest + 2] = '|'.join(cells)\n            logger.info(f'[bold][yellow]{\"Samples Distribution\": ^{len(rows[0])}}[/yellow][/bold]')\n            logger.info('\\n'.join(rows))\n        if prefetch and (data == 'trn' or not tasks_need_custom_eval):\n            dataloader = PrefetchDataLoader(dataloader, prefetch=prefetch)\n\n        return dataloader\n\n    def build_transform(self, task: Task) -> Tuple[TransformerSequenceTokenizer, TransformList]:\n        encoder: ContextualWordEmbedding = self.config.encoder\n        encoder_transform: TransformerSequenceTokenizer = task.build_tokenizer(encoder.transform())\n        length_transform = FieldLength('token', 'token_length')\n        transform = TransformList(encoder_transform, length_transform)\n        extra_transform = self.config.get('transform', None)\n        if extra_transform:\n            transform.insert(0, extra_transform)\n        return encoder_transform, transform\n\n    def build_optimizer(self,\n                        trn,\n                        epochs,\n                        adam_epsilon,\n                        weight_decay,\n                        warmup_steps,\n                        lr,\n                        encoder_lr,\n                        **kwargs):\n        model = self.model_\n        encoder = model.encoder\n        num_training_steps = len(trn) * epochs // self.config.get('gradient_accumulation', 1)\n        encoder_parameters = list(encoder.parameters())\n        parameter_groups: List[Dict[str, Any]] = []\n\n        decoders = model.decoders\n        decoder_optimizers = dict()\n        for k, task in self.tasks.items():\n            decoder: torch.nn.Module = decoders[k]\n            decoder_parameters = list(decoder.parameters())\n            if task.separate_optimizer:\n                decoder_optimizers[k] = task.build_optimizer(decoder=decoder, **kwargs)\n            else:\n                task_lr = task.lr or lr\n                parameter_groups.append({\"params\": decoder_parameters, 'lr': task_lr})\n        parameter_groups.append({\"params\": encoder_parameters, 'lr': encoder_lr})\n        no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']\n        no_decay_parameters = set()\n        for n, p in model.named_parameters():\n            if any(nd in n for nd in no_decay):\n                no_decay_parameters.add(p)\n        no_decay_by_lr = defaultdict(list)\n        for group in parameter_groups:\n            _lr = group['lr']\n            ps = group['params']\n            group['params'] = decay_parameters = []\n            group['weight_decay'] = weight_decay\n            for p in ps:\n                if p in no_decay_parameters:\n                    no_decay_by_lr[_lr].append(p)\n                else:\n                    decay_parameters.append(p)\n        for _lr, ps in no_decay_by_lr.items():\n            parameter_groups.append({\"params\": ps, 'lr': _lr, 'weight_decay': 0.0})\n        # noinspection PyTypeChecker\n        from transformers import optimization\n        encoder_optimizer = optimization.AdamW(\n            parameter_groups,\n            lr=lr,\n            weight_decay=weight_decay,\n            eps=adam_epsilon,\n        )\n        encoder_scheduler = optimization.get_linear_schedule_with_warmup(encoder_optimizer,\n                                                                         num_training_steps * warmup_steps,\n                                                                         num_training_steps)\n        return encoder_optimizer, encoder_scheduler, decoder_optimizers\n\n    def build_criterion(self, **kwargs):\n        return dict((k, v.build_criterion(decoder=self.model_.decoders[k], **kwargs)) for k, v in self.tasks.items())\n\n    def build_metric(self, **kwargs):\n        metrics = MetricDict()\n        for key, task in self.tasks.items():\n            metric = task.build_metric(**kwargs)\n            assert metric, f'Please implement `build_metric` of {type(task)} to return a metric.'\n            metrics[key] = metric\n        return metrics\n\n    def execute_training_loop(self, trn: DataLoader, dev: DataLoader, epochs, criterion, optimizer, metric, save_dir,\n                              logger: logging.Logger, devices, patience=0.5, **kwargs):\n        if isinstance(patience, float):\n            patience = int(patience * epochs)\n        best_epoch, best_metric = 0, -1\n        timer = CountdownTimer(epochs)\n        ratio_width = len(f'{len(trn)}/{len(trn)}')\n        epoch = 0\n        history = History()\n        for epoch in range(1, epochs + 1):\n            logger.info(f\"[yellow]Epoch {epoch} / {epochs}:[/yellow]\")\n            self.fit_dataloader(trn, criterion, optimizer, metric, logger, history, ratio_width=ratio_width,\n                                **self.config)\n            if dev:\n                self.evaluate_dataloader(dev, criterion, metric, logger, ratio_width=ratio_width, input='dev')\n            report = f'{timer.elapsed_human}/{timer.total_time_human}'\n            dev_score = metric.score\n            if dev_score > best_metric:\n                self.save_weights(save_dir)\n                best_metric = dev_score\n                best_epoch = epoch\n                report += ' [red]saved[/red]'\n            else:\n                report += f' ({epoch - best_epoch})'\n                if epoch - best_epoch >= patience:\n                    report += ' early stop'\n                    break\n            timer.log(report, ratio_percentage=False, newline=True, ratio=False)\n        for d in [trn, dev]:\n            self._close_dataloader(d)\n        if best_epoch != epoch:\n            logger.info(f'Restoring best model saved [red]{epoch - best_epoch}[/red] epochs ago')\n            self.load_weights(save_dir)\n        return best_metric\n\n    def _close_dataloader(self, d):\n        if isinstance(d, PrefetchDataLoader):\n            d.close()\n            if hasattr(d.dataset, 'close'):\n                self._close_dataloader(d.dataset)\n        elif isinstance(d, CachedDataLoader):\n            d.close()\n        elif isinstance(d, MultiTaskDataLoader):\n            for d in d.dataloaders.values():\n                self._close_dataloader(d)\n\n    # noinspection PyMethodOverriding\n    def fit_dataloader(self,\n                       trn: DataLoader,\n                       criterion,\n                       optimizer,\n                       metric,\n                       logger: logging.Logger,\n                       history: History,\n                       ratio_width=None,\n                       gradient_accumulation=1,\n                       encoder_grad_norm=None,\n                       decoder_grad_norm=None,\n                       patience=0.5,\n                       eval_trn=False,\n                       **kwargs):\n        self.model.train()\n        encoder_optimizer, encoder_scheduler, decoder_optimizers = optimizer\n        timer = CountdownTimer(len(trn))\n        total_loss = 0\n        self.reset_metrics(metric)\n        model = self.model_\n        encoder_parameters = model.encoder.parameters()\n        decoder_parameters = model.decoders.parameters()\n        for idx, (task_name, batch) in enumerate(trn):\n            decoder_optimizer = decoder_optimizers.get(task_name, None)\n            output_dict, _ = self.feed_batch(batch, task_name)\n            loss = self.compute_loss(batch, output_dict[task_name]['output'], criterion[task_name],\n                                     self.tasks[task_name])\n            if gradient_accumulation and gradient_accumulation > 1:\n                loss /= gradient_accumulation\n            loss.backward()\n            total_loss += float(loss.item())\n            if history.step(gradient_accumulation):\n                if self.config.get('grad_norm', None):\n                    clip_grad_norm(model, self.config.grad_norm)\n                if encoder_grad_norm:\n                    torch.nn.utils.clip_grad_norm_(encoder_parameters, encoder_grad_norm)\n                if decoder_grad_norm:\n                    torch.nn.utils.clip_grad_norm_(decoder_parameters, decoder_grad_norm)\n                encoder_optimizer.step()\n                encoder_optimizer.zero_grad()\n                encoder_scheduler.step()\n                if decoder_optimizer:\n                    if isinstance(decoder_optimizer, tuple):\n                        decoder_optimizer, decoder_scheduler = decoder_optimizer\n                    else:\n                        decoder_scheduler = None\n                    decoder_optimizer.step()\n                    decoder_optimizer.zero_grad()\n                    if decoder_scheduler:\n                        decoder_scheduler.step()\n            if eval_trn:\n                self.decode_output(output_dict, batch, task_name)\n                self.update_metrics(batch, output_dict, metric, task_name)\n            timer.log(self.report_metrics(total_loss / (timer.current + 1), metric if eval_trn else None),\n                      ratio_percentage=None,\n                      ratio_width=ratio_width,\n                      logger=logger)\n            del loss\n            del output_dict\n        return total_loss / timer.total\n\n    def report_metrics(self, loss, metrics: MetricDict):\n        return f'loss: {loss:.4f} {metrics.cstr()}' if metrics else f'loss: {loss:.4f}'\n\n    # noinspection PyMethodOverriding\n    @torch.no_grad()\n    def evaluate_dataloader(self,\n                            data: MultiTaskDataLoader,\n                            criterion,\n                            metric: MetricDict,\n                            logger,\n                            ratio_width=None,\n                            input: str = None,\n                            **kwargs):\n        self.model.eval()\n        self.reset_metrics(metric)\n        tasks_need_custom_eval = self.config.get('tasks_need_custom_eval', None)\n        tasks_need_custom_eval = tasks_need_custom_eval or {}\n        tasks_need_custom_eval = dict((k, None) for k in tasks_need_custom_eval)\n        for each in tasks_need_custom_eval:\n            tasks_need_custom_eval[each] = data.dataloaders.pop(each)\n        timer = CountdownTimer(len(data) + len(tasks_need_custom_eval))\n        total_loss = 0\n        for idx, (task_name, batch) in enumerate(data):\n            output_dict, _ = self.feed_batch(batch, task_name)\n            loss = self.compute_loss(batch, output_dict[task_name]['output'], criterion[task_name],\n                                     self.tasks[task_name])\n            total_loss += loss.item()\n            self.decode_output(output_dict, batch, task_name)\n            self.update_metrics(batch, output_dict, metric, task_name)\n            timer.log(self.report_metrics(total_loss / (timer.current + 1), metric), ratio_percentage=None,\n                      logger=logger,\n                      ratio_width=ratio_width)\n            del loss\n            del output_dict\n\n        for task_name, dataset in tasks_need_custom_eval.items():\n            task = self.tasks[task_name]\n            decoder = self.model_.decoders[task_name]\n            task.evaluate_dataloader(\n                dataset, task.build_criterion(decoder=decoder),\n                metric=metric[task_name],\n                input=task.dev if input == 'dev' else task.tst,\n                split=input,\n                decoder=decoder,\n                h=functools.partial(self._encode, task_name=task_name,\n                                    cls_is_bos=task.cls_is_bos, sep_is_eos=task.sep_is_eos)\n            )\n            data.dataloaders[task_name] = dataset\n            timer.log(self.report_metrics(total_loss / (timer.current + 1), metric), ratio_percentage=None,\n                      logger=logger,\n                      ratio_width=ratio_width)\n\n        return total_loss / timer.total, metric, data\n\n    def build_model(self, training=False, **kwargs) -> torch.nn.Module:\n        tasks = self.tasks\n        encoder: ContextualWordEmbedding = self.config.encoder\n        transformer_module = encoder.module(training=training)\n        encoder_size = transformer_module.get_output_dim()\n        scalar_mixes = torch.nn.ModuleDict()\n        decoders = torch.nn.ModuleDict()\n        use_raw_hidden_states = dict()\n        for task_name, task in tasks.items():\n            decoder = task.build_model(encoder_size, training=training, **task.config)\n            assert decoder, f'Please implement `build_model` of {type(task)} to return a decoder.'\n            decoders[task_name] = decoder\n            if task.scalar_mix:\n                scalar_mix = task.scalar_mix.build()\n                scalar_mixes[task_name] = scalar_mix\n                # Activate scalar mix starting from 0-th layer\n                encoder.scalar_mix = 0\n            use_raw_hidden_states[task_name] = task.use_raw_hidden_states\n        encoder.ret_raw_hidden_states = any(use_raw_hidden_states.values())\n        return MultiTaskModel(transformer_module, scalar_mixes, decoders, use_raw_hidden_states)\n\n    def predict(self,\n                data: Union[str, List[str]],\n                tasks: Optional[Union[str, List[str]]] = None,\n                skip_tasks: Optional[Union[str, List[str]]] = None,\n                resolved_tasks=None,\n                **kwargs) -> Document:\n        \"\"\"Predict on data.\n\n        Args:\n            data: A sentence or a list of sentences.\n            tasks: The tasks to predict.\n            skip_tasks: The tasks to skip.\n            resolved_tasks: The resolved tasks to override ``tasks`` and ``skip_tasks``.\n            **kwargs: Not used.\n\n        Returns:\n            A :class:`~hanlp_common.document.Document`.\n        \"\"\"\n        doc = Document()\n        target_tasks = resolved_tasks or self.resolve_tasks(tasks, skip_tasks)\n        if data == []:\n            for group in target_tasks:\n                for task_name in group:\n                    doc[task_name] = []\n            return doc\n        flatten_target_tasks = [self.tasks[t] for group in target_tasks for t in group]\n        cls_is_bos = any([x.cls_is_bos for x in flatten_target_tasks])\n        sep_is_eos = any([x.sep_is_eos for x in flatten_target_tasks])\n        # Now build the dataloaders and execute tasks\n        first_task_name: str = list(target_tasks[0])[0]\n        first_task: Task = self.tasks[first_task_name]\n        encoder_transform, transform = self.build_transform(first_task)\n        # Override the tokenizer config of the 1st task\n        encoder_transform.sep_is_eos = sep_is_eos\n        encoder_transform.cls_is_bos = cls_is_bos\n        average_subwords = self.model.encoder.average_subwords\n        flat = first_task.input_is_flat(data)\n        if flat:\n            data = [data]\n        device = self.device\n        samples = first_task.build_samples(data, cls_is_bos=cls_is_bos, sep_is_eos=sep_is_eos)\n        dataloader = first_task.build_dataloader(samples, transform=transform, device=device)\n        results = defaultdict(list)\n        order = []\n        for batch in dataloader:\n            order.extend(batch[IDX])\n            # Run the first task, let it make the initial batch for the successors\n            output_dict = self.predict_task(first_task, first_task_name, batch, results, run_transform=True,\n                                            cls_is_bos=cls_is_bos, sep_is_eos=sep_is_eos)\n            # Run each task group in order\n            for group_id, group in enumerate(target_tasks):\n                # We could parallelize this in the future\n                for task_name in group:\n                    if task_name == first_task_name:\n                        continue\n                    output_dict = self.predict_task(self.tasks[task_name], task_name, batch, results, output_dict,\n                                                    run_transform=True, cls_is_bos=cls_is_bos, sep_is_eos=sep_is_eos)\n                if group_id == 0:\n                    # We are kind of hard coding here. If the first task is a tokenizer,\n                    # we need to convert the hidden and mask to token level\n                    if first_task_name.startswith('tok'):\n                        spans = []\n                        tokens = []\n                        output_spans = first_task.config.get('output_spans', None)\n                        for span_per_sent, token_per_sent in zip(output_dict[first_task_name]['prediction'],\n                                                                 results[first_task_name][-len(batch[IDX]):]):\n                            if output_spans:\n                                token_per_sent = [x[0] for x in token_per_sent]\n                            if cls_is_bos:\n                                span_per_sent = [(-1, 0)] + span_per_sent\n                                token_per_sent = [BOS] + token_per_sent\n                            if sep_is_eos:\n                                span_per_sent = span_per_sent + [(span_per_sent[-1][0] + 1, span_per_sent[-1][1] + 1)]\n                                token_per_sent = token_per_sent + [EOS]\n                            # The offsets start with 0 while [CLS] is zero\n                            if average_subwords:\n                                span_per_sent = [list(range(x[0] + 1, x[1] + 1)) for x in span_per_sent]\n                            else:\n                                span_per_sent = [x[0] + 1 for x in span_per_sent]\n                            spans.append(span_per_sent)\n                            tokens.append(token_per_sent)\n                        spans = PadSequenceDataLoader.pad_data(spans, 0, torch.long, device=device)\n                        output_dict['hidden'] = pick_tensor_for_each_token(output_dict['hidden'], spans,\n                                                                           average_subwords)\n                        batch['token_token_span'] = spans\n                        batch['token'] = tokens\n                        # noinspection PyTypeChecker\n                        batch['token_length'] = torch.tensor([len(x) for x in tokens], dtype=torch.long, device=device)\n                        batch.pop('mask', None)\n        # Put results into doc in the order of tasks\n        for k in self.config.task_names:\n            v = results.get(k, None)\n            if v is None:\n                continue\n            doc[k] = reorder(v, order)\n        # Allow task to perform finalization on document\n        for group in target_tasks:\n            for task_name in group:\n                task = self.tasks[task_name]\n                task.finalize_document(doc, task_name)\n        # If no tok in doc, use raw input as tok\n        if not any(k.startswith('tok') for k in doc):\n            doc['tok'] = data\n        if flat:\n            for k, v in list(doc.items()):\n                doc[k] = v[0]\n        # If there is only one field, don't bother to wrap it\n        # if len(doc) == 1:\n        #     return list(doc.values())[0]\n        return doc\n\n    def resolve_tasks(self, tasks, skip_tasks) -> List[Iterable[str]]:\n        # Now we decide which tasks to perform and their orders\n        tasks_in_topological_order = self._tasks_in_topological_order\n        task_topological_order = self._task_topological_order\n        computation_graph = self._computation_graph\n        target_tasks = self._resolve_task_name(tasks)\n        if not target_tasks:\n            target_tasks = tasks_in_topological_order\n        else:\n            target_topological_order = defaultdict(set)\n            for task_name in target_tasks:\n                for dependency in topological_sort(computation_graph, task_name):\n                    target_topological_order[task_topological_order[dependency]].add(dependency)\n            target_tasks = [item[1] for item in sorted(target_topological_order.items())]\n        if skip_tasks:\n            skip_tasks = self._resolve_task_name(skip_tasks)\n            target_tasks = [x - skip_tasks for x in target_tasks]\n            target_tasks = [x for x in target_tasks if x]\n        assert target_tasks, f'No task to perform due to `tasks = {tasks}`.'\n        # Sort target tasks within the same group in a defined order\n        target_tasks = [sorted(x, key=lambda _x: self.config.task_names.index(_x)) for x in target_tasks]\n        return target_tasks\n\n    def predict_task(self, task: Task, output_key, batch, results, output_dict=None, run_transform=True,\n                     cls_is_bos=True, sep_is_eos=True):\n        output_dict, batch = self.feed_batch(batch, output_key, output_dict, run_transform, cls_is_bos, sep_is_eos,\n                                             results)\n        self.decode_output(output_dict, batch, output_key)\n        results[output_key].extend(task.prediction_to_result(output_dict[output_key]['prediction'], batch))\n        return output_dict\n\n    def _resolve_task_name(self, dependencies):\n        resolved_dependencies = set()\n        if isinstance(dependencies, str):\n            if dependencies in self.tasks:\n                resolved_dependencies.add(dependencies)\n            elif dependencies.endswith('*'):\n                resolved_dependencies.update(x for x in self.tasks if x.startswith(dependencies[:-1]))\n            else:\n                prefix_matched = prefix_match(dependencies, self.config.task_names)\n                assert prefix_matched, f'No prefix matching for {dependencies}. ' \\\n                                       f'Check your dependencies definition: {list(self.tasks.values())}'\n                resolved_dependencies.add(prefix_matched)\n        elif isinstance(dependencies, Iterable):\n            resolved_dependencies.update(set(chain.from_iterable(self._resolve_task_name(x) for x in dependencies)))\n        return resolved_dependencies\n\n    def fit(self,\n            encoder: Embedding,\n            tasks: Dict[str, Task],\n            save_dir,\n            epochs,\n            patience=0.5,\n            lr=1e-3,\n            encoder_lr=5e-5,\n            adam_epsilon=1e-8,\n            weight_decay=0.0,\n            warmup_steps=0.1,\n            gradient_accumulation=1,\n            grad_norm=5.0,\n            encoder_grad_norm=None,\n            decoder_grad_norm=None,\n            tau: float = 0.8,\n            transform=None,\n            # prune: Callable = None,\n            eval_trn=True,\n            prefetch=None,\n            tasks_need_custom_eval=None,\n            _device_placeholder=False,\n            cache=False,\n            devices=None,\n            logger=None,\n            seed=None,\n            **kwargs):\n        trn_data, dev_data, batch_size = 'trn', 'dev', None\n        task_names = list(tasks.keys())\n        return super().fit(**merge_locals_kwargs(locals(), kwargs, excludes=('self', 'kwargs', '__class__', 'tasks')),\n                           **tasks)\n\n    # noinspection PyAttributeOutsideInit\n    def on_config_ready(self, **kwargs):\n        self.tasks = dict((key, task) for key, task in self.config.items() if isinstance(task, Task))\n        computation_graph = dict()\n        for task_name, task in self.tasks.items():\n            dependencies = task.dependencies\n            resolved_dependencies = self._resolve_task_name(dependencies)\n            computation_graph[task_name] = resolved_dependencies\n\n        # We can cache this order\n        tasks_in_topological_order = list(toposort(computation_graph))\n        task_topological_order = dict()\n        for i, group in enumerate(tasks_in_topological_order):\n            for task_name in group:\n                task_topological_order[task_name] = i\n        self._tasks_in_topological_order = tasks_in_topological_order\n        self._task_topological_order = task_topological_order\n        self._computation_graph = computation_graph\n\n    @staticmethod\n    def reset_metrics(metrics: Dict[str, Metric]):\n        for metric in metrics.values():\n            metric.reset()\n\n    def feed_batch(self,\n                   batch: Dict[str, Any],\n                   task_name,\n                   output_dict=None,\n                   run_transform=False,\n                   cls_is_bos=False,\n                   sep_is_eos=False,\n                   results=None) -> Tuple[Dict[str, Any], Dict[str, Any]]:\n        h, output_dict = self._encode(batch, task_name, output_dict, cls_is_bos, sep_is_eos)\n        task = self.tasks[task_name]\n        if run_transform:\n            batch = task.transform_batch(batch, results=results, cls_is_bos=cls_is_bos, sep_is_eos=sep_is_eos)\n        batch['mask'] = mask = hanlp.utils.torch_util.lengths_to_mask(batch['token_length'])\n        output_dict[task_name] = {\n            'output': task.feed_batch(h,\n                                      batch=batch,\n                                      mask=mask,\n                                      decoder=self.model.decoders[task_name]),\n            'mask': mask\n        }\n        return output_dict, batch\n\n    def _encode(self, batch, task_name, output_dict=None, cls_is_bos=False, sep_is_eos=False):\n        model = self.model\n        if output_dict:\n            hidden, raw_hidden = output_dict['hidden'], output_dict['raw_hidden']\n        else:\n            hidden = model.encoder(batch)\n            if isinstance(hidden, tuple):\n                hidden, raw_hidden = hidden\n            else:\n                raw_hidden = None\n            output_dict = {'hidden': hidden, 'raw_hidden': raw_hidden}\n        hidden_states = raw_hidden if model.use_raw_hidden_states[task_name] else hidden\n        if task_name in model.scalar_mixes:\n            scalar_mix = model.scalar_mixes[task_name]\n            h = scalar_mix(hidden_states)\n        else:\n            if model.scalar_mixes:  # If any task enables scalar_mix, hidden_states will be a 4d tensor\n                hidden_states = hidden_states[-1, :, :, :]\n            h = hidden_states\n        # If the task doesn't need cls while h has cls, remove cls\n        task = self.tasks[task_name]\n        if cls_is_bos and not task.cls_is_bos:\n            h = h[:, 1:, :]\n        if sep_is_eos and not task.sep_is_eos:\n            h = h[:, :-1, :]\n        return h, output_dict\n\n    def decode_output(self, output_dict, batch, task_name=None):\n        if not task_name:\n            for task_name, task in self.tasks.items():\n                output_per_task = output_dict.get(task_name, None)\n                if output_per_task is not None:\n                    output_per_task['prediction'] = task.decode_output(\n                        output_per_task['output'],\n                        output_per_task['mask'],\n                        batch, self.model.decoders[task_name])\n        else:\n            output_per_task = output_dict[task_name]\n            output_per_task['prediction'] = self.tasks[task_name].decode_output(\n                output_per_task['output'],\n                output_per_task['mask'],\n                batch,\n                self.model.decoders[task_name])\n\n    def update_metrics(self, batch: Dict[str, Any], output_dict: Dict[str, Any], metrics: MetricDict, task_name):\n        task = self.tasks[task_name]\n        output_per_task = output_dict.get(task_name, None)\n        if output_per_task:\n            output = output_per_task['output']\n            prediction = output_per_task['prediction']\n            metric = metrics.get(task_name, None)\n            task.update_metrics(batch, output, prediction, metric)\n\n    def compute_loss(self,\n                     batch: Dict[str, Any],\n                     output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                     criterion: Callable,\n                     task: Task) -> torch.FloatTensor:\n        return task.compute_loss(batch, output, criterion)\n\n    def evaluate(self, save_dir=None, logger: logging.Logger = None, batch_size=None, output=False, **kwargs):\n        rets = super().evaluate('tst', save_dir, logger, batch_size, output, **kwargs)\n        tst = rets[-1]\n        self._close_dataloader(tst)\n        return rets\n\n    def save_vocabs(self, save_dir, filename='vocabs.json'):\n        for task_name, task in self.tasks.items():\n            task.save_vocabs(save_dir, f'{task_name}_{filename}')\n\n    def load_vocabs(self, save_dir, filename='vocabs.json'):\n        for task_name, task in self.tasks.items():\n            task.load_vocabs(save_dir, f'{task_name}_{filename}')\n\n    def parallelize(self, devices: List[Union[int, torch.device]]):\n        raise NotImplementedError('Parallelization is not implemented yet.')\n\n    def __call__(self, data, **kwargs) -> Document:\n        return super().__call__(data, **kwargs)\n\n    def __getitem__(self, task_name: str) -> Task:\n        return self.tasks[task_name]\n\n    def __delitem__(self, task_name: str):\n        \"\"\"Delete a task (and every resource it owns) from this component.\n\n        Args:\n            task_name: The name of the task to be deleted.\n\n        Examples:\n            >>> del mtl['dep']  # Delete dep from MTL\n\n        \"\"\"\n        del self.config[task_name]\n        self.config.task_names.remove(task_name)\n        del self.tasks[task_name]\n        del self.model.decoders[task_name]\n        del self._computation_graph[task_name]\n        self._task_topological_order.pop(task_name)\n        for group in self._tasks_in_topological_order:\n            group: set = group\n            group.discard(task_name)\n\n    def __repr__(self):\n        return repr(self.config)\n\n    def items(self):\n        yield from self.tasks.items()\n\n    def __setattr__(self, key: str, value):\n        if key and key.startswith('dict') and not hasattr(self, key):\n            please_read_the_doc_ok = f'This MTL component has no {key}.'\n            matched_children = []\n            for name in self.config.task_names:\n                if hasattr(self[name], key):\n                    matched_children.append(name)\n            if matched_children:\n                please_read_the_doc_ok += f' Maybe you are looking for one of its tasks: {matched_children}. ' \\\n                                          f'For example, HanLP[\"{matched_children[0]}\"].{key} = ...'\n            raise TypeError(please_read_the_doc_ok)\n        object.__setattr__(self, key, value)\n"
  },
  {
    "path": "hanlp/components/mtl/tasks/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-02 16:51\nimport logging\nimport os\nimport warnings\nfrom abc import ABC, abstractmethod\nfrom copy import copy\nfrom typing import Callable, Dict, Any, Union, Iterable, List\n\nimport torch\nfrom hanlp_common.util import merge_locals_kwargs\nfrom torch.utils.data import DataLoader\n\nfrom hanlp_common.constant import BOS, EOS\nfrom hanlp.common.dataset import SamplerBuilder, SortingSamplerBuilder, TransformableDataset, KMeansSamplerBuilder\nfrom hanlp_common.document import Document\nfrom hanlp.common.structure import ConfigTracker\nfrom hanlp.common.torch_component import TorchComponent\nfrom hanlp.layers.scalar_mix import ScalarMixWithDropoutBuilder\nfrom hanlp.metrics.metric import Metric\nfrom hanlp.metrics.mtl import MetricDict\nfrom hanlp.transform.transformer_tokenizer import TransformerSequenceTokenizer\nfrom hanlp.utils.time_util import CountdownTimer\n\n\nclass Task(ConfigTracker, TorchComponent, ABC):\n    # noinspection PyMissingConstructor\n    def __init__(self,\n                 trn: str = None,\n                 dev: str = None,\n                 tst: str = None,\n                 sampler_builder: SamplerBuilder = None,\n                 dependencies: str = None,\n                 scalar_mix: ScalarMixWithDropoutBuilder = None,\n                 use_raw_hidden_states=False,\n                 lr=None,\n                 separate_optimizer=False,\n                 cls_is_bos=False,\n                 sep_is_eos=False,\n                 **kwargs) -> None:\n        \"\"\"\n        A task in the multi-task learning framework\n\n        Args:\n            trn: Path to training set.\n            dev: Path to dev set.\n            tst: Path to test set.\n            sampler_builder: A builder which builds a sampler.\n            dependencies: Its dependencies on other tasks.\n            scalar_mix: A builder which builds a `ScalarMixWithDropout` object.\n            use_raw_hidden_states: Whether to use raw hidden states from transformer without any pooling.\n            lr: Learning rate for this task.\n            separate_optimizer: Use customized separate optimizer for this task.\n            cls_is_bos: ``True`` to treat the first token as ``BOS``.\n            sep_is_eos: ``True`` to treat the last token as ``EOS``.\n            **kwargs: Additional config.\n        \"\"\"\n        ConfigTracker.__init__(self, merge_locals_kwargs(locals(), kwargs))\n        for f, n in zip([trn, dev, tst], ['trn', 'dev', 'tst']):\n            if f and os.path.isfile(f):  # anonymize local file names\n                self.config.pop(n)\n        self.separate_optimizer = separate_optimizer\n        self.lr = lr\n        self.use_raw_hidden_states = use_raw_hidden_states\n        if sampler_builder is None:\n            sampler_builder = SortingSamplerBuilder(batch_size=32)\n        self.sampler_builder: Union[SortingSamplerBuilder, KMeansSamplerBuilder] = sampler_builder\n        self.dependencies = dependencies\n        self.tst = tst\n        self.dev = dev\n        self.trn = trn\n        self.scalar_mix = scalar_mix\n        self.cls_is_bos = cls_is_bos\n        self.sep_is_eos = sep_is_eos\n\n    @abstractmethod\n    def build_dataloader(self,\n                         data,\n                         transform: Callable = None,\n                         training=False,\n                         device=None,\n                         logger: logging.Logger = None,\n                         cache=False,\n                         gradient_accumulation=1,\n                         **kwargs) -> DataLoader:\n        \"\"\"\n        Build a dataloader for training or evaluation.\n\n        Args:\n            data: Either a path or a list of samples.\n            transform: The transform from MTL, which is usually [TransformerSequenceTokenizer, FieldLength('token')]\n            training: Whether this method is called on training set.\n            device: The device dataloader is intended to work with.\n            logger: Logger for printing message indicating progress.\n            cache: Whether the dataloader should be cached.\n            gradient_accumulation: Gradient accumulation to be passed to sampler builder.\n            **kwargs: Additional experimental arguments.\n        \"\"\"\n        pass\n\n    def build_optimizer(self, decoder: torch.nn.Module, **kwargs):\n        pass\n\n    def build_batch_wise_scheduler(self, decoder: torch.nn.Module, **kwargs):\n        pass\n\n    @abstractmethod\n    def compute_loss(self,\n                     batch: Dict[str, Any],\n                     output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                     criterion,\n                     ) -> Union[torch.FloatTensor, Dict[str, torch.FloatTensor]]:\n        pass\n\n    @abstractmethod\n    def decode_output(self,\n                      output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                      mask: torch.BoolTensor,\n                      batch: Dict[str, Any], decoder: torch.nn.Module, **kwargs) -> Union[Dict[str, Any], Any]:\n        pass\n\n    @abstractmethod\n    def update_metrics(self,\n                       batch: Dict[str, Any],\n                       output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                       prediction: Dict[str, Any],\n                       metric: Union[MetricDict, Metric]):\n        pass\n\n    # noinspection PyMethodOverriding\n    @abstractmethod\n    def build_model(self, encoder_size, training=True, **kwargs) -> torch.nn.Module:\n        pass\n\n    @abstractmethod\n    def build_metric(self, **kwargs):\n        pass\n\n    def fit_dataloader(self, trn: DataLoader, criterion, optimizer, metric, logger: logging.Logger, **kwargs):\n        pass\n\n    def evaluate_dataloader(self, data: DataLoader, criterion: Callable, output=False, **kwargs):\n        pass\n\n    def execute_training_loop(self, trn: DataLoader, dev: DataLoader, epochs, criterion, optimizer, metric, save_dir,\n                              logger: logging.Logger, devices, **kwargs):\n        pass\n\n    # noinspection PyMethodMayBeStatic\n    def compute_lens(self, data: Union[List[Dict[str, Any]], str], dataset: TransformableDataset,\n                     input_ids='token_input_ids'):\n        \"\"\"\n\n        Args:\n            data: Samples to be measured or path to dataset during training time.\n            dataset: During training time, use this dataset to measure the length of each sample inside.\n            input_ids: Field name corresponds to input ids.\n\n        Returns:\n\n            Length list of this samples\n\n        \"\"\"\n        if dataset.cache is None:\n            warnings.warn(f'Caching for the dataset is not enabled, '\n                          f'try `dataset.purge_cache()` if possible. The dataset is {dataset}.')\n        if isinstance(data, str):\n            timer = CountdownTimer(len(dataset))\n            for each in dataset:\n                timer.log('Preprocessing and caching samples [blink][yellow]...[/yellow][/blink]')\n            timer.erase()\n        return [len(x[input_ids]) for x in dataset]\n\n    def feed_batch(self,\n                   h: torch.FloatTensor,\n                   batch: Dict[str, torch.Tensor],\n                   mask: torch.BoolTensor,\n                   decoder: torch.nn.Module):\n        return decoder(h, batch=batch, mask=mask)\n\n    def input_is_flat(self, data) -> bool:\n        \"\"\"\n        Check whether the data is flat (meaning that it's only a single sample, not even batched).\n\n        Returns:\n            bool: ``True`` to indicate the input data is flat.\n        \"\"\"\n        raise NotImplementedError(\n            '`input_is_flat()` needs to be implemented for the task component to accept raw input from user.'\n        )\n\n    @abstractmethod\n    def prediction_to_result(self, prediction: Dict[str, Any], batch: Dict[str, Any]) -> List:\n        raise NotImplementedError()\n\n    # noinspection PyMethodMayBeStatic\n    def transform_batch(self,\n                        batch: Dict[str, Any],\n                        # inputs: List[List[str]],\n                        results: Dict[str, Any] = None,\n                        cls_is_bos=False,\n                        sep_is_eos=False) -> Dict[str, Any]:\n        \"\"\"\n        Let the task transform the batch before feeding the batch into its decoder. The default behavior is to\n        adjust the head and tail of tokens, according to ``cls_is_bos``, ``sep_is_eos`` passed in and the two\n        settings of the task itself.\n\n        Args:\n            batch: A batch of samples.\n            results: Predicted results from other tasks which might be useful for this task to utilize. Say a dep task\n                uses both token and pos as features, then it will need both tok and pos results to make a batch.\n            cls_is_bos: First token in this batch is BOS.\n            sep_is_eos: Last token in this batch is EOS.\n\n        Returns:\n            A batch.\n\n        \"\"\"\n        if cls_is_bos != self.cls_is_bos or sep_is_eos != self.sep_is_eos:\n            batch = copy(batch)\n            tokens = self._adjust_token(batch, cls_is_bos, sep_is_eos, 'token')\n            delta = len(tokens[0]) - len(batch['token'][0])\n            batch['token_length'] = batch['token_length'] + delta\n            batch['token'] = tokens\n            if 'token_' in batch:\n                if isinstance(batch['token_'][0], list):\n                    batch['token_'] = self._adjust_token(batch, cls_is_bos, sep_is_eos, 'token_')\n                else:\n                    batch['token_'] = tokens\n        return batch\n\n    def _adjust_token(self, batch, cls_is_bos, sep_is_eos, token_key):\n        tokens = []\n        for sent in batch[token_key]:\n            if cls_is_bos:\n                if not self.cls_is_bos:\n                    sent = sent[1:]\n            elif self.cls_is_bos:\n                sent = [BOS] + sent\n            if sep_is_eos:\n                if not self.sep_is_eos:\n                    sent = sent[:-1]\n            elif self.sep_is_eos:\n                sent = sent + [EOS]\n            tokens.append(sent)\n        return tokens\n\n    # noinspection PyMethodMayBeStatic\n    def build_samples(self, inputs, cls_is_bos=False, sep_is_eos=False):\n        \"\"\"\n        Build samples for this task. Called when this task is the first task. Default behaviour is to take inputs as\n        list of tokens and put these tokens into a dict per sample.\n\n        Args:\n            inputs: Inputs from users, usually a list of lists of tokens.\n            cls_is_bos: Insert BOS to the head of each sentence.\n            sep_is_eos: Append EOS to the tail of each sentence.\n\n        Returns:\n            List of samples.\n\n        \"\"\"\n        if cls_is_bos:\n            inputs = [[BOS] + x for x in inputs]\n        if sep_is_eos:\n            inputs = [x + [EOS] for x in inputs]\n        return [{'token': token} for token in inputs]\n\n    def build_tokenizer(self, tokenizer: TransformerSequenceTokenizer):\n        \"\"\"Build a transformer tokenizer for this task.\n\n        Args:\n            tokenizer: A tokenizer which is shared but can be adjusted to provide per-task settings.\n\n        Returns:\n            A TransformerSequenceTokenizer.\n\n        \"\"\"\n        if tokenizer.cls_is_bos != self.cls_is_bos or tokenizer.sep_is_eos != self.sep_is_eos:\n            tokenizer = copy(tokenizer)\n            tokenizer.cls_is_bos = self.cls_is_bos\n            tokenizer.sep_is_eos = self.sep_is_eos\n        return tokenizer\n\n    # noinspection PyMethodMayBeStatic\n    def finalize_document(self, doc: Document, task_name: str):\n        pass\n"
  },
  {
    "path": "hanlp/components/mtl/tasks/amr.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-12 16:05\nimport logging\nfrom typing import Dict, Any, List, Union, Iterable, Callable\n\nimport torch\nfrom stog.data.dataset_readers.amr_parsing.amr import AMRGraph\nfrom stog.data.dataset_readers.amr_parsing.node_utils import NodeUtilities\nfrom stog.data.dataset_readers.amr_parsing.postprocess.node_restore import NodeRestore\nfrom torch.utils.data import DataLoader\n\nfrom hanlp_common.constant import CLS\nfrom hanlp.common.dataset import PrefetchDataLoader, SamplerBuilder\nfrom hanlp.common.transform import VocabDict\nfrom hanlp.components.amr.amr_parser.graph_amr_decoder import GraphAbstractMeaningRepresentationDecoder\nfrom hanlp.components.amr.amr_parser.graph_parser import GraphAbstractMeaningRepresentationParser\nfrom hanlp.components.amr.amr_parser.postprocess import PostProcessor\nfrom hanlp.components.amr.amr_parser.work import parse_batch\nfrom hanlp.components.mtl.tasks import Task\nfrom hanlp.datasets.parsing.amr import batchify, get_concepts\nfrom hanlp.layers.scalar_mix import ScalarMixWithDropoutBuilder\nfrom hanlp.metrics.amr.smatch_eval import SmatchScores, get_amr_utils\nfrom hanlp.metrics.f1 import F1_\nfrom hanlp.metrics.metric import Metric\nfrom hanlp.metrics.mtl import MetricDict\nfrom hanlp.utils.io_util import get_resource\nfrom hanlp_common.util import merge_list_of_dict, merge_locals_kwargs\n\n\nclass GraphAbstractMeaningRepresentationParsing(Task, GraphAbstractMeaningRepresentationParser):\n\n    def __init__(self,\n                 trn: str = None,\n                 dev: str = None,\n                 tst: str = None,\n                 sampler_builder: SamplerBuilder = None,\n                 dependencies: str = None,\n                 scalar_mix: ScalarMixWithDropoutBuilder = None,\n                 use_raw_hidden_states=False,\n                 lr=1e-3,\n                 separate_optimizer=False,\n                 cls_is_bos=True,\n                 sep_is_eos=False,\n                 char2concept_dim=128,\n                 cnn_filters=((3, 256),),\n                 concept_char_dim=32,\n                 concept_dim=300,\n                 dropout=0.2,\n                 embed_dim=512,\n                 eval_every=20,\n                 ff_embed_dim=1024,\n                 graph_layers=2,\n                 inference_layers=4,\n                 num_heads=8,\n                 rel_dim=100,\n                 snt_layers=4,\n                 unk_rate=0.33,\n                 vocab_min_freq=5,\n                 beam_size=8,\n                 alpha=0.6,\n                 max_time_step=100,\n                 amr_version='2.0',\n                 **kwargs) -> None:\n        super().__init__(**merge_locals_kwargs(locals(), kwargs))\n        self.vocabs = VocabDict()\n        utils_dir = get_resource(get_amr_utils(amr_version))\n        self.sense_restore = NodeRestore(NodeUtilities.from_json(utils_dir))\n\n    def build_dataloader(self,\n                         data,\n                         transform: Callable = None,\n                         training=False,\n                         device=None,\n                         logger: logging.Logger = None,\n                         cache=False,\n                         gradient_accumulation=1,\n                         **kwargs) -> DataLoader:\n        if isinstance(data, list):\n            data = GraphAbstractMeaningRepresentationParser.build_samples(self, data)\n        dataset, lens = GraphAbstractMeaningRepresentationParser.build_dataset(self, data, logger=logger,\n                                                                               transform=transform, training=training)\n        if self.vocabs.mutable:\n            GraphAbstractMeaningRepresentationParser.build_vocabs(self, dataset, logger)\n        dataloader = PrefetchDataLoader(\n            DataLoader(batch_sampler=self.sampler_builder.build(lens, shuffle=training,\n                                                                gradient_accumulation=gradient_accumulation),\n                       dataset=dataset,\n                       collate_fn=merge_list_of_dict,\n                       num_workers=0), batchify=self.build_batchify(device, training),\n            prefetch=None)\n        return dataloader\n\n    def compute_loss(self,\n                     batch: Dict[str, Any],\n                     output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                     criterion) -> Union[torch.FloatTensor, Dict[str, torch.FloatTensor]]:\n        concept_loss, arc_loss, rel_loss, graph_arc_loss = output\n        concept_loss, concept_correct, concept_total = concept_loss\n        rel_loss, rel_correct, rel_total = rel_loss\n        loss = concept_loss + arc_loss + rel_loss\n        return loss\n\n    def decode_output(self,\n                      output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                      mask: torch.BoolTensor,\n                      batch: Dict[str, Any],\n                      decoder: torch.nn.Module, **kwargs) -> Union[Dict[str, Any], Any]:\n        return output\n\n    def update_metrics(self,\n                       batch: Dict[str, Any],\n                       output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                       prediction: Dict[str, Any],\n                       metric: Union[MetricDict, Metric]):\n        pass\n\n    def build_model(self, encoder_size, training=True, **kwargs) -> torch.nn.Module:\n        return GraphAbstractMeaningRepresentationDecoder(vocabs=self.vocabs, encoder_size=encoder_size, **self.config)\n\n    def build_metric(self, **kwargs):\n        return SmatchScores({'Smatch': F1_(0, 0, 0)})\n\n    def input_is_flat(self, data) -> bool:\n        return GraphAbstractMeaningRepresentationParser.input_is_flat(self, data)\n\n    def prediction_to_result(self, prediction: Dict[str, Any], batch: Dict[str, Any]) -> List:\n        pp = PostProcessor(self.vocabs['rel'])\n        for concept, relation, score in zip(prediction['concept'], prediction['relation'], prediction['score']):\n            amr = pp.to_amr(concept, relation)\n            amr_graph = AMRGraph(amr)\n            self.sense_restore.restore_graph(amr_graph)\n            yield amr_graph\n\n    def evaluate_dataloader(self,\n                            data: DataLoader,\n                            criterion: Callable,\n                            metric=None,\n                            output=False,\n                            input=None,\n                            decoder=None,\n                            h=None,\n                            split=None,\n                            **kwargs):\n        # noinspection PyTypeChecker\n        GraphAbstractMeaningRepresentationParser.evaluate_dataloader(self, data, logger=None, metric=metric,\n                                                                     input=input, model=decoder, h=lambda x: h(x)[0],\n                                                                     use_fast=True)\n\n    def feed_batch(self,\n                   h: torch.FloatTensor,\n                   batch: Dict[str, torch.Tensor],\n                   mask: torch.BoolTensor,\n                   decoder: torch.nn.Module):\n        if decoder.training:\n            return super().feed_batch(h, batch, mask, decoder)\n        beam_size = self.config.get('beam_size', 8)\n        alpha = self.config.get('alpha', 0.6)\n        max_time_step = self.config.get('max_time_step', 100)\n        res = parse_batch(decoder, batch, beam_size, alpha, max_time_step, h=h)\n        return res\n\n    def transform_batch(self, batch: Dict[str, Any], results: Dict[str, Any] = None, cls_is_bos=False,\n                        sep_is_eos=False) -> Dict[str, Any]:\n        batch = super().transform_batch(batch, results, cls_is_bos, sep_is_eos)\n        batch['lemma'] = [[CLS] + x for x in results['lem']]\n        copy_seq = merge_list_of_dict(\n            [get_concepts({'token': t[1:], 'lemma': l[1:]}, self.vocabs.predictable_concept) for t, l in\n             zip(batch['token'], batch['lemma'])])\n        copy_seq.pop('token')\n        copy_seq.pop('lemma')\n        batch.update(copy_seq)\n        ret = batchify(batch, self.vocabs, device=batch['token_input_ids'].device)\n        return ret\n"
  },
  {
    "path": "hanlp/components/mtl/tasks/constituency.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-11-29 16:52\nimport logging\nfrom typing import Dict, Any, List, Union, Iterable, Callable\n\nimport torch\nfrom phrasetree.tree import Tree\n\nfrom hanlp_common.constant import BOS, EOS\nfrom hanlp_common.document import Document\nfrom hanlp.components.parsers.biaffine.biaffine_dep import BiaffineDependencyParser\nfrom torch.utils.data import DataLoader\n\nfrom hanlp.common.dataset import SamplerBuilder, PadSequenceDataLoader\nfrom hanlp.common.transform import VocabDict\nfrom hanlp.components.mtl.tasks import Task\nfrom hanlp.components.parsers.constituency.crf_constituency_model import CRFConstituencyDecoder\nfrom hanlp.components.parsers.constituency.crf_constituency_parser import CRFConstituencyParser\nfrom hanlp.layers.scalar_mix import ScalarMixWithDropoutBuilder\nfrom hanlp.metrics.metric import Metric\nfrom hanlp.metrics.mtl import MetricDict\nfrom hanlp.utils.time_util import CountdownTimer\nfrom hanlp_common.util import merge_locals_kwargs, prefix_match\n\n\nclass CRFConstituencyParsing(Task, CRFConstituencyParser):\n    def __init__(self,\n                 trn: str = None,\n                 dev: str = None,\n                 tst: str = None,\n                 sampler_builder: SamplerBuilder = None,\n                 dependencies: str = None,\n                 scalar_mix: ScalarMixWithDropoutBuilder = None,\n                 use_raw_hidden_states=False,\n                 lr=None,\n                 separate_optimizer=False,\n                 cls_is_bos=True,\n                 sep_is_eos=True,\n                 delete=('', ':', '``', \"''\", '.', '?', '!', '-NONE-', 'TOP', ',', 'S1'),\n                 equal=(('ADVP', 'PRT'),),\n                 mbr=True,\n                 n_mlp_span=500,\n                 n_mlp_label=100,\n                 mlp_dropout=.33,\n                 no_subcategory=True,\n                 **kwargs\n                 ) -> None:\n        r\"\"\"Two-stage CRF Parsing (:cite:`ijcai2020-560`).\n\n        Args:\n            trn: Path to training set.\n            dev: Path to dev set.\n            tst: Path to test set.\n            sampler_builder: A builder which builds a sampler.\n            dependencies: Its dependencies on other tasks.\n            scalar_mix: A builder which builds a `ScalarMixWithDropout` object.\n            use_raw_hidden_states: Whether to use raw hidden states from transformer without any pooling.\n            lr: Learning rate for this task.\n            separate_optimizer: Use customized separate optimizer for this task.\n            cls_is_bos: ``True`` to treat the first token as ``BOS``.\n            sep_is_eos: ``True`` to treat the last token as ``EOS``.\n            delete: Constituencies to be deleted from training and evaluation.\n            equal: Constituencies that are regarded as equal during evaluation.\n            mbr: ``True`` to enable Minimum Bayes Risk (MBR) decoding (:cite:`smith-smith-2007-probabilistic`).\n            n_mlp_span: Number of features for span decoder.\n            n_mlp_label: Number of features for label decoder.\n            mlp_dropout: Dropout applied to MLPs.\n            no_subcategory: Strip out subcategories.\n            **kwargs: Not used.\n        \"\"\"\n        if isinstance(equal, tuple):\n            equal = dict(equal)\n        super().__init__(**merge_locals_kwargs(locals(), kwargs))\n        self.vocabs = VocabDict()\n\n    # noinspection DuplicatedCode\n    def build_dataloader(self,\n                         data,\n                         transform: Callable = None,\n                         training=False,\n                         device=None,\n                         logger: logging.Logger = None,\n                         cache=False,\n                         gradient_accumulation=1,\n                         **kwargs) -> DataLoader:\n        dataset = CRFConstituencyParsing.build_dataset(self, data, transform)\n        dataset.purge_cache()\n        if self.vocabs.mutable:\n            CRFConstituencyParsing.build_vocabs(self, dataset, logger)\n        if isinstance(data, str):\n            timer = CountdownTimer(len(dataset))\n            # noinspection PyCallByClass\n            BiaffineDependencyParser.cache_dataset(self, dataset, timer, training, logger)\n        return PadSequenceDataLoader(\n            batch_sampler=self.sampler_builder.build(self.compute_lens(data, dataset), shuffle=training,\n                                                     gradient_accumulation=gradient_accumulation),\n            device=device,\n            dataset=dataset)\n\n    def feed_batch(self,\n                   h: torch.FloatTensor,\n                   batch: Dict[str, torch.Tensor],\n                   mask: torch.BoolTensor,\n                   decoder: torch.nn.Module):\n        return {\n            'output': decoder(h),\n            'mask': CRFConstituencyParser.compute_mask(\n                self, batch, offset=1 if 'constituency' in batch or batch['token'][0][-1] == EOS else -1)\n        }\n\n    def compute_loss(self,\n                     batch: Dict[str, Any],\n                     output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                     criterion) -> Union[torch.FloatTensor, Dict[str, torch.FloatTensor]]:\n        out, mask = output['output'], output['mask']\n        loss, span_probs = CRFConstituencyParser.compute_loss(self, out, batch['chart_id'], mask, crf_decoder=criterion)\n        output['span_probs'] = span_probs\n        return loss\n\n    def decode_output(self,\n                      output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                      mask: torch.BoolTensor,\n                      batch: Dict[str, Any],\n                      decoder: torch.nn.Module, **kwargs) -> Union[Dict[str, Any], Any]:\n        out, mask = output['output'], output['mask']\n        tokens = []\n        for sent in batch['token']:\n            if sent[0] == BOS:\n                sent = sent[1:]\n            if sent[-1] == EOS:\n                sent = sent[:-1]\n            tokens.append(sent)\n        return CRFConstituencyParser.decode_output(self, out, mask, batch, output.get('span_probs', None),\n                                                   decoder=decoder, tokens=tokens)\n\n    def update_metrics(self,\n                       batch: Dict[str, Any],\n                       output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                       prediction: Dict[str, Any], metric: Union[MetricDict, Metric]):\n        return CRFConstituencyParser.update_metrics(self, metric, batch, prediction)\n\n    def build_model(self, encoder_size, training=True, **kwargs) -> torch.nn.Module:\n        return CRFConstituencyDecoder(n_labels=len(self.vocabs.chart), n_hidden=encoder_size)\n\n    def build_metric(self, **kwargs):\n        return CRFConstituencyParser.build_metric(self)\n\n    def input_is_flat(self, data) -> bool:\n        return CRFConstituencyParser.input_is_flat(self, data)\n\n    def prediction_to_result(self, prediction: List, batch: Dict[str, Any]) -> List:\n        return prediction\n\n    def finalize_document(self, doc: Document, task_name: str):\n        pos_key = prefix_match('pos', doc)\n        pos: List[List[str]] = doc.get(pos_key, None)\n        if pos:\n            for tree, pos_per_sent in zip(doc[task_name], pos):\n                tree: Tree = tree\n                offset = 0\n                for subtree in tree.subtrees(lambda t: t.height() == 2):\n                    tag = subtree.label()\n                    if tag == '_':\n                        subtree.set_label(pos_per_sent[offset])\n                    offset += 1\n\n    def build_samples(self, inputs, cls_is_bos=False, sep_is_eos=False):\n        return CRFConstituencyParser.build_samples(self, inputs)\n"
  },
  {
    "path": "hanlp/components/mtl/tasks/dep.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-13 21:39\nimport logging\nfrom typing import Dict, Any, Union, Iterable, List\n\nimport torch\nfrom torch.optim import Adam\nfrom torch.optim.lr_scheduler import ExponentialLR\nfrom torch.utils.data import DataLoader\n\nfrom hanlp.common.dataset import SamplerBuilder, PadSequenceDataLoader\nfrom hanlp.common.transform import VocabDict, TransformList\nfrom hanlp.components.mtl.tasks import Task\nfrom hanlp.components.parsers.biaffine.biaffine_dep import BiaffineDependencyParser\nfrom hanlp.components.parsers.biaffine.biaffine_model import BiaffineDecoder\nfrom hanlp.datasets.parsing.loaders.conll_dataset import append_bos\nfrom hanlp.layers.scalar_mix import ScalarMixWithDropoutBuilder\nfrom hanlp.metrics.metric import Metric\nfrom hanlp.metrics.mtl import MetricDict\nfrom hanlp.utils.time_util import CountdownTimer\nfrom hanlp_common.constant import EOS\nfrom hanlp_common.util import merge_locals_kwargs\n\n\nclass BiaffineDependencyParsing(Task, BiaffineDependencyParser):\n    def __init__(self,\n                 trn: str = None,\n                 dev: str = None,\n                 tst: str = None,\n                 sampler_builder: SamplerBuilder = None,\n                 dependencies: str = None,\n                 scalar_mix: ScalarMixWithDropoutBuilder = None,\n                 use_raw_hidden_states=False,\n                 lr=2e-3, separate_optimizer=False,\n                 cls_is_bos=True,\n                 sep_is_eos=False,\n                 punct=False,\n                 tree=False,\n                 proj=False,\n                 n_mlp_arc=500,\n                 n_mlp_rel=100,\n                 mlp_dropout=.33,\n                 mu=.9,\n                 nu=.9,\n                 epsilon=1e-12,\n                 decay=.75,\n                 decay_steps=5000,\n                 use_pos=False,\n                 max_seq_len=None,\n                 **kwargs) -> None:\n        \"\"\"Biaffine dependency parsing (:cite:`dozat:17a`).\n\n        Args:\n            trn: Path to training set.\n            dev: Path to dev set.\n            tst: Path to test set.\n            sampler_builder: A builder which builds a sampler.\n            dependencies: Its dependencies on other tasks.\n            scalar_mix: A builder which builds a `ScalarMixWithDropout` object.\n            use_raw_hidden_states: Whether to use raw hidden states from transformer without any pooling.\n            lr: Learning rate for this task.\n            separate_optimizer: Use customized separate optimizer for this task.\n            cls_is_bos: ``True`` to treat the first token as ``BOS``.\n            sep_is_eos: ``True`` to treat the last token as ``EOS``.\n            punct: ``True`` to include punctuations in evaluation.\n            tree: ``True`` to enforce tree constraint.\n            proj: ``True`` for projective parsing.\n            n_mlp_arc: Number of features for arc representation.\n            n_mlp_rel: Number of features for rel representation.\n            mlp_dropout: Dropout applied to MLPs.\n            mu: First coefficient used for computing running averages of gradient and its square in Adam.\n            nu: Second coefficient used for computing running averages of gradient and its square in Adam.\n            epsilon: Term added to the denominator to improve numerical stability\n            decay: Decay rate for exceptional lr scheduler.\n            decay_steps: Decay every ``decay_steps`` steps.\n            use_pos: Use pos feature.\n            max_seq_len: Prune samples longer than this length.\n            **kwargs: Not used.\n        \"\"\"\n        super().__init__(**merge_locals_kwargs(locals(), kwargs))\n        self.vocabs = VocabDict()\n\n    def update_metrics(self, batch: Dict[str, Any],\n                       output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                       prediction: Dict[str, Any], metric: Union[MetricDict, Metric]):\n        BiaffineDependencyParser.update_metric(self, *prediction, batch['arc'], batch['rel_id'], output[1],\n                                               batch.get('punct_mask', None), metric, batch)\n\n    def decode_output(self,\n                      output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                      mask: torch.BoolTensor,\n                      batch: Dict[str, Any],\n                      decoder, **kwargs) -> Union[Dict[str, Any], Any]:\n        (arc_scores, rel_scores), mask = output\n        return BiaffineDependencyParser.decode(self, arc_scores, rel_scores, mask, batch)\n\n    def compute_loss(self, batch: Dict[str, Any],\n                     output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any], criterion) -> \\\n            Union[torch.FloatTensor, Dict[str, torch.FloatTensor]]:\n        (arc_scores, rel_scores), mask = output\n        return BiaffineDependencyParser.compute_loss(self, arc_scores, rel_scores, batch['arc'], batch['rel_id'], mask,\n                                                     criterion,\n                                                     batch)\n\n    def build_model(self, encoder_size, training=True, **kwargs) -> torch.nn.Module:\n        return BiaffineDecoder(encoder_size, self.config.n_mlp_arc, self.config.n_mlp_rel, self.config.mlp_dropout,\n                               len(self.vocabs.rel))\n\n    def build_metric(self, **kwargs):\n        return BiaffineDependencyParser.build_metric(self, **kwargs)\n\n    def build_dataloader(self, data, transform: TransformList = None, training=False, device=None,\n                         logger: logging.Logger = None, gradient_accumulation=1, **kwargs) -> DataLoader:\n        transform.insert(0, append_bos)\n        dataset = BiaffineDependencyParser.build_dataset(self, data, transform)\n        dataset.purge_cache()\n        if self.vocabs.mutable:\n            BiaffineDependencyParser.build_vocabs(self, dataset, logger, transformer=True)\n        if isinstance(data, str):\n            timer = CountdownTimer(len(dataset))\n            BiaffineDependencyParser.cache_dataset(self, dataset, timer, training, logger)\n        max_seq_len = self.config.get('max_seq_len', None)\n        if max_seq_len and isinstance(data, str):\n            dataset.prune(lambda x: len(x['token_input_ids']) > max_seq_len, logger)\n        return PadSequenceDataLoader(\n            batch_sampler=self.sampler_builder.build(self.compute_lens(data, dataset),\n                                                     shuffle=training, gradient_accumulation=gradient_accumulation),\n            device=device,\n            dataset=dataset,\n            pad=self.get_pad_dict())\n\n    def feed_batch(self, h: torch.FloatTensor, batch: Dict[str, torch.Tensor], mask: torch.BoolTensor,\n                   decoder: torch.nn.Module):\n        logits = super().feed_batch(h, batch, mask, decoder)\n        mask = mask.clone()\n        mask[:, 0] = 0\n        return logits, mask\n\n    def build_optimizer(self, decoder: torch.nn.Module, **kwargs):\n        config = self.config\n        optimizer = Adam(decoder.parameters(),\n                         config.lr,\n                         (config.mu, config.nu),\n                         config.epsilon)\n        scheduler = ExponentialLR(optimizer, config.decay ** (1 / config.decay_steps))\n        return optimizer, scheduler\n\n    def input_is_flat(self, data) -> bool:\n        return BiaffineDependencyParser.input_is_flat(self, data, self.config.use_pos)\n\n    def prediction_to_result(self, prediction: Dict[str, Any], batch: Dict[str, Any]) -> List:\n        arcs, rels = prediction\n        arcs = arcs[:, 1:]  # Skip the ROOT\n        rels = rels[:, 1:]\n        arcs = arcs.tolist()\n        rels = rels.tolist()\n        vocab = self.vocabs['rel'].idx_to_token\n        for arcs_per_sent, rels_per_sent, tokens in zip(arcs, rels, batch['token']):\n            tokens = tokens[1:]\n            sent_len = len(tokens)\n            result = list(zip(arcs_per_sent[:sent_len], [vocab[r] for r in rels_per_sent[:sent_len]]))\n            yield result\n\n    def build_samples(self, inputs, cls_is_bos=False, sep_is_eos=False):\n        return [{'FORM': token + ([EOS] if sep_is_eos else [])} for token in inputs]\n"
  },
  {
    "path": "hanlp/components/mtl/tasks/dep_2nd.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-07 14:14\nimport logging\nfrom typing import Dict, Any, Union, Iterable, Callable, List\n\nimport torch\nfrom hanlp_common.util import merge_locals_kwargs\nfrom torch.utils.data import DataLoader\n\nimport hanlp.utils.torch_util\nfrom hanlp.common.dataset import SamplerBuilder, PadSequenceDataLoader\nfrom hanlp.common.transform import VocabDict\nfrom hanlp.components.mtl.tasks import Task\nfrom hanlp.components.parsers.biaffine.biaffine_2nd_dep import BiaffineSecondaryParser, BiaffineJointDecoder, \\\n    BiaffineSeparateDecoder\nfrom hanlp.layers.scalar_mix import ScalarMixWithDropoutBuilder\nfrom hanlp.metrics.metric import Metric\nfrom hanlp.metrics.mtl import MetricDict\n\n\nclass BiaffineSecondaryDependencyDecoder(torch.nn.Module):\n    def __init__(self, hidden_size, config) -> None:\n        super().__init__()\n        self.decoder = BiaffineJointDecoder(hidden_size, config) if config.joint \\\n            else BiaffineSeparateDecoder(hidden_size, config)\n\n    def forward(self, contextualized_embeddings: torch.FloatTensor, batch: Dict[str, torch.Tensor], mask=None):\n        if mask is None:\n            mask = hanlp.utils.torch_util.lengths_to_mask(batch['token_length'])\n        else:\n            mask = mask.clone()\n        scores = self.decoder(contextualized_embeddings, mask)\n        mask[:, 0] = 0\n        return scores, mask\n\n\nclass BiaffineSecondaryDependencyParsing(Task, BiaffineSecondaryParser):\n\n    def __init__(self, trn: str = None, dev: str = None, tst: str = None, sampler_builder: SamplerBuilder = None,\n                 dependencies: str = None, scalar_mix: ScalarMixWithDropoutBuilder = None, use_raw_hidden_states=False,\n                 lr=2e-3, separate_optimizer=False,\n                 punct=False,\n                 tree=False,\n                 apply_constraint=True,\n                 n_mlp_arc=500,\n                 n_mlp_rel=100,\n                 mlp_dropout=.33,\n                 pad_rel=None,\n                 joint=True,\n                 mu=.9,\n                 nu=.9,\n                 epsilon=1e-12,\n                 cls_is_bos=True,\n                 **kwargs) -> None:\n        super().__init__(**merge_locals_kwargs(locals(), kwargs))\n        self.vocabs = VocabDict()\n\n    def build_dataloader(self, data, transform: Callable = None, training=False, device=None,\n                         logger: logging.Logger = None, gradient_accumulation=1, **kwargs) -> DataLoader:\n        dataset = BiaffineSecondaryParser.build_dataset(self, data, transform)\n        dataset.purge_cache()\n        if self.vocabs.mutable:\n            BiaffineSecondaryParser.build_vocabs(self, dataset, logger, transformer=True)\n        return PadSequenceDataLoader(\n            batch_sampler=self.sampler_builder.build(self.compute_lens(data, dataset), shuffle=training,\n                                                     gradient_accumulation=gradient_accumulation),\n            device=device,\n            dataset=dataset,\n            pad={'arc': 0, 'arc_2nd': False})\n\n    def update_metrics(self, batch: Dict[str, Any],\n                       output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                       prediction: Dict[str, Any], metric: Union[MetricDict, Metric]):\n\n        BiaffineSecondaryParser.update_metric(self, *prediction, batch['arc'], batch['rel_id'], output[1],\n                                              batch['punct_mask'], metric, batch)\n\n    def decode_output(self, output: Dict[str, Any], batch: Dict[str, Any], decoder, **kwargs) \\\n            -> Union[Dict[str, Any], Any]:\n        return BiaffineSecondaryParser.decode(self, *output[0], output[1], batch)\n\n    def compute_loss(self, batch: Dict[str, Any],\n                     output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any], criterion) -> \\\n            Union[torch.FloatTensor, Dict[str, torch.FloatTensor]]:\n        return BiaffineSecondaryParser.compute_loss(self, *output[0], batch['arc'], batch['rel_id'], output[1],\n                                                    criterion, batch)\n\n    def build_model(self, encoder_size, training=True, **kwargs) -> torch.nn.Module:\n        return BiaffineSecondaryDependencyDecoder(encoder_size, self.config)\n\n    def build_metric(self, **kwargs):\n        return BiaffineSecondaryParser.build_metric(self, **kwargs)\n\n    def build_criterion(self, **kwargs):\n        return BiaffineSecondaryParser.build_criterion(self, **kwargs)\n\n    def build_optimizer(self, decoder: torch.nn.Module, **kwargs):\n        config = self.config\n        optimizer = torch.optim.Adam(decoder.parameters(),\n                                     config.lr,\n                                     (config.mu, config.nu),\n                                     config.epsilon)\n        return optimizer\n\n    def input_is_flat(self, data) -> bool:\n        return BiaffineSecondaryParser.input_is_flat(self, data)\n\n    def prediction_to_result(self, prediction: Dict[str, Any], batch: Dict[str, Any]) -> List:\n        outputs = []\n        return BiaffineSecondaryParser.predictions_to_human(self, prediction, outputs, batch['token'], use_pos=False)\n"
  },
  {
    "path": "hanlp/components/mtl/tasks/lem.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-09 16:37\nimport logging\nfrom typing import Dict, Any, Union, Iterable, Callable, List\n\nimport torch\nfrom hanlp.common.dataset import SamplerBuilder, PadSequenceDataLoader\nfrom hanlp.common.transform import VocabDict\nfrom hanlp.components.lemmatizer import TransformerLemmatizer\nfrom hanlp.components.mtl.tasks import Task\nfrom hanlp.layers.scalar_mix import ScalarMixWithDropoutBuilder\nfrom hanlp.metrics.metric import Metric\nfrom hanlp.metrics.mtl import MetricDict\nfrom hanlp_common.util import merge_locals_kwargs\nfrom torch.utils.data import DataLoader\n\n\nclass LinearDecoder(torch.nn.Module):\n    def __init__(self,\n                 hidden_size,\n                 num_labels) -> None:\n        super().__init__()\n        self.classifier = torch.nn.Linear(hidden_size, num_labels)\n\n    def forward(self, contextualized_embeddings: torch.FloatTensor, batch: Dict[str, torch.Tensor], mask=None):\n        return self.classifier(contextualized_embeddings)\n\n\nclass TransformerLemmatization(Task, TransformerLemmatizer):\n\n    def __init__(self,\n                 trn: str = None,\n                 dev: str = None,\n                 tst: str = None,\n                 sampler_builder: SamplerBuilder = None,\n                 dependencies: str = None,\n                 scalar_mix: ScalarMixWithDropoutBuilder = None,\n                 use_raw_hidden_states=False,\n                 lr=1e-3,\n                 separate_optimizer=False,\n                 cls_is_bos=False,\n                 sep_is_eos=False,\n                 max_seq_len=None,\n                 sent_delimiter=None,\n                 char_level=False,\n                 hard_constraint=False,\n                 token_key='token', **kwargs) -> None:\n        \"\"\" Transition based lemmatization (:cite:`kondratyuk-straka-2019-75`).\n\n        Args:\n            trn: Path to training set.\n            dev: Path to dev set.\n            tst: Path to test set.\n            sampler_builder: A builder which builds a sampler.\n            dependencies: Its dependencies on other tasks.\n            scalar_mix: A builder which builds a `ScalarMixWithDropout` object.\n            use_raw_hidden_states: Whether to use raw hidden states from transformer without any pooling.\n            lr: Learning rate for this task.\n            separate_optimizer: Use customized separate optimizer for this task.\n            cls_is_bos: ``True`` to treat the first token as ``BOS``.\n            sep_is_eos: ``True`` to treat the last token as ``EOS``.\n            max_seq_len: Sentences longer than ``max_seq_len`` will be split into shorter ones if possible.\n            sent_delimiter: Delimiter between sentences, like period or comma, which indicates a long sentence can\n                be split here.\n            char_level: Whether the sequence length is measured at char level, which is never the case for\n                lemmatization.\n            hard_constraint: Whether to enforce hard length constraint on sentences. If there is no ``sent_delimiter``\n                in a sentence, it will be split at a token anyway.\n            token_key: The key to tokens in dataset. This should always be set to ``token`` in MTL.\n            **kwargs: Not used.\n        \"\"\"\n        super().__init__(**merge_locals_kwargs(locals(), kwargs))\n        self.vocabs = VocabDict()\n\n    def build_dataloader(self,\n                         data: List[List[str]],\n                         transform: Callable = None,\n                         training=False,\n                         device=None,\n                         logger: logging.Logger = None,\n                         cache=False,\n                         gradient_accumulation=1,\n                         **kwargs) -> DataLoader:\n        args = dict((k, self.config[k]) for k in\n                    ['delimiter', 'max_seq_len', 'sent_delimiter', 'char_level', 'hard_constraint'] if k in self.config)\n        dataset = self.build_dataset(data, cache=True, transform=transform, **args)\n        dataset.append_transform(self.vocabs)\n        if self.vocabs.mutable:\n            self.build_vocabs(dataset, logger)\n        return PadSequenceDataLoader(\n            batch_sampler=self.sampler_builder.build(self.compute_lens(data, dataset),\n                                                     shuffle=training, gradient_accumulation=gradient_accumulation),\n            device=device,\n            dataset=dataset)\n\n    def compute_loss(self,\n                     batch: Dict[str, Any],\n                     output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                     criterion) -> Union[torch.FloatTensor, Dict[str, torch.FloatTensor]]:\n        return TransformerLemmatizer.compute_loss(self, criterion, output, batch['tag_id'], batch['mask'])\n\n    def decode_output(self,\n                      output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                      mask: torch.BoolTensor,\n                      batch: Dict[str, Any],\n                      decoder,\n                      **kwargs) -> Union[Dict[str, Any], Any]:\n        return TransformerLemmatizer.decode_output(self, output, mask, batch, decoder)\n\n    def update_metrics(self,\n                       batch: Dict[str, Any],\n                       output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                       prediction: Dict[str, Any],\n                       metric: Union[MetricDict, Metric]):\n        return TransformerLemmatizer.update_metrics(self, metric, output, batch['tag_id'], batch['mask'])\n\n    def build_model(self, encoder_size, training=True, **kwargs) -> torch.nn.Module:\n        return LinearDecoder(encoder_size, len(self.vocabs['tag']))\n\n    def build_metric(self, **kwargs):\n        return TransformerLemmatizer.build_metric(self, **kwargs)\n\n    def input_is_flat(self, data) -> bool:\n        return TransformerLemmatizer.input_is_flat(self, data)\n\n    def prediction_to_result(self, prediction: Dict[str, Any], batch: Dict[str, Any]) -> Union[List, Dict]:\n        return TransformerLemmatizer.prediction_to_human(self, prediction, self.vocabs['tag'].idx_to_token, batch,\n                                                         token=batch['token'])\n"
  },
  {
    "path": "hanlp/components/mtl/tasks/ner/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-03 14:34\n"
  },
  {
    "path": "hanlp/components/mtl/tasks/ner/biaffine_ner.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-05 01:49\nimport logging\nfrom copy import copy\nfrom typing import Dict, Any, Union, Iterable, List\n\nimport torch\nfrom torch.utils.data import DataLoader\n\nfrom hanlp.common.dataset import SamplerBuilder, PadSequenceDataLoader\nfrom hanlp.common.transform import VocabDict, TransformList\nfrom hanlp.components.mtl.tasks import Task\nfrom hanlp.components.ner.biaffine_ner.biaffine_ner import BiaffineNamedEntityRecognizer\nfrom hanlp.components.ner.biaffine_ner.biaffine_ner_model import BiaffineNamedEntityRecognitionDecoder\nfrom hanlp.datasets.ner.loaders.json_ner import unpack_ner\nfrom hanlp.layers.scalar_mix import ScalarMixWithDropoutBuilder\nfrom hanlp.metrics.metric import Metric\nfrom hanlp.metrics.mtl import MetricDict\nfrom hanlp_common.util import merge_locals_kwargs\n\n\nclass BiaffineNamedEntityRecognition(Task, BiaffineNamedEntityRecognizer):\n\n    def __init__(self, trn: str = None, dev: str = None, tst: str = None, sampler_builder: SamplerBuilder = None,\n                 dependencies: str = None, scalar_mix: ScalarMixWithDropoutBuilder = None, use_raw_hidden_states=False,\n                 lr=None, separate_optimizer=False,\n                 doc_level_offset=True, is_flat_ner=True, tagset=None, ret_tokens=' ',\n                 ffnn_size=150, loss_reduction='mean', **kwargs) -> None:\n        \"\"\"An implementation of Named Entity Recognition as Dependency Parsing (:cite:`yu-etal-2020-named`). It treats\n        every possible span as a candidate of entity and predicts its entity label. Non-entity spans are assigned NULL\n        label to be excluded. The label prediction is done with a biaffine layer (:cite:`dozat:17a`). As it makes no\n        assumption about the spans, it naturally supports flat NER and nested NER.\n\n        Args:\n            trn: Path to training set.\n            dev: Path to dev set.\n            tst: Path to test set.\n            sampler_builder: A builder which builds a sampler.\n            dependencies: Its dependencies on other tasks.\n            scalar_mix: A builder which builds a `ScalarMixWithDropout` object.\n            use_raw_hidden_states: Whether to use raw hidden states from transformer without any pooling.\n            lr: Learning rate for this task.\n            separate_optimizer: Use customized separate optimizer for this task.\n            doc_level_offset: ``True`` to indicate the offsets in ``jsonlines`` are of document level.\n            is_flat_ner: ``True`` for flat NER, otherwise nested NER.\n            tagset: Optional tagset to prune entities outside of this tagset from datasets.\n            ret_tokens: A delimiter between tokens in entities so that the surface form of an entity can be rebuilt.\n            ffnn_size: Feedforward size for MLPs extracting the head/tail representations.\n            loss_reduction: The loss reduction used in aggregating losses.\n            **kwargs: Not used.\n        \"\"\"\n        super().__init__(**merge_locals_kwargs(locals(), kwargs))\n        self.vocabs = VocabDict()\n\n    def update_metrics(self, batch: Dict[str, Any],\n                       output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                       prediction: Dict[str, Any], metric: Union[MetricDict, Metric]):\n        BiaffineNamedEntityRecognizer.update_metrics(self, batch, prediction, metric)\n\n    def decode_output(self,\n                      output: Dict[str, Any],\n                      mask: torch.BoolTensor,\n                      batch: Dict[str, Any],\n                      decoder,\n                      **kwargs) -> Union[Dict[str, Any], Any]:\n        return self.get_pred_ner(batch['token'], output['candidate_ner_scores'])\n\n    def compute_loss(self, batch: Dict[str, Any],\n                     output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any], criterion) -> \\\n            Union[torch.FloatTensor, Dict[str, torch.FloatTensor]]:\n        return output['loss']\n\n    def build_dataloader(self, data,\n                         transform: TransformList = None,\n                         training=False,\n                         device=None,\n                         logger: logging.Logger = None,\n                         gradient_accumulation=1,\n                         **kwargs) -> DataLoader:\n        transform = copy(transform)\n        transform.append(unpack_ner)\n        dataset = BiaffineNamedEntityRecognizer.build_dataset(self, data, self.vocabs, transform)\n        dataset.purge_cache()\n        if self.vocabs.mutable:\n            BiaffineNamedEntityRecognizer.build_vocabs(self, dataset, logger, self.vocabs)\n        return PadSequenceDataLoader(\n            batch_sampler=self.sampler_builder.build(self.compute_lens(data, dataset), shuffle=training,\n                                                     gradient_accumulation=gradient_accumulation),\n            device=device,\n            dataset=dataset)\n\n    def build_model(self, encoder_size, training=True, **kwargs) -> torch.nn.Module:\n        return BiaffineNamedEntityRecognitionDecoder(encoder_size, self.config.ffnn_size, len(self.vocabs.label),\n                                                     self.config.loss_reduction)\n\n    def build_metric(self, **kwargs):\n        return BiaffineNamedEntityRecognizer.build_metric(self, **kwargs)\n\n    def input_is_flat(self, data) -> bool:\n        return BiaffineNamedEntityRecognizer.input_is_flat(data)\n\n    def prediction_to_result(self, prediction: Dict[str, Any], batch: Dict[str, Any]) -> List:\n        results = []\n        BiaffineNamedEntityRecognizer.prediction_to_result(batch['token'], prediction, results,\n                                                           ret_tokens=self.config.get('ret_tokens', ' '))\n        return results\n"
  },
  {
    "path": "hanlp/components/mtl/tasks/ner/tag_ner.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-03 14:35\nimport logging\nfrom typing import Union, List, Dict, Any, Iterable, Callable, Set, Sequence\n\nimport torch\nfrom hanlp_trie import DictInterface\nfrom torch.utils.data import DataLoader\n\nfrom hanlp.common.dataset import SamplerBuilder, PadSequenceDataLoader\nfrom hanlp.common.transform import VocabDict\nfrom hanlp.components.mtl.tasks import Task\nfrom hanlp.components.ner.transformer_ner import TransformerNamedEntityRecognizer\nfrom hanlp.layers.crf.crf import CRF\nfrom hanlp.layers.scalar_mix import ScalarMixWithDropoutBuilder\nfrom hanlp.metrics.metric import Metric\nfrom hanlp.metrics.mtl import MetricDict\nfrom hanlp_common.util import merge_locals_kwargs\n\n\nclass LinearCRFDecoder(torch.nn.Module):\n    def __init__(self,\n                 hidden_size,\n                 num_labels,\n                 secondary_encoder=None,\n                 crf=False) -> None:\n        super().__init__()\n        self.secondary_encoder = secondary_encoder\n        self.classifier = torch.nn.Linear(hidden_size, num_labels)\n        self.crf = CRF(num_labels) if crf else None\n\n    def forward(self, contextualized_embeddings: torch.FloatTensor, batch: Dict[str, torch.Tensor], mask=None):\n        if self.secondary_encoder:\n            contextualized_embeddings = self.secondary_encoder(contextualized_embeddings, mask=mask)\n        return self.classifier(contextualized_embeddings)\n\n\nclass TaggingNamedEntityRecognition(Task, TransformerNamedEntityRecognizer):\n\n    def __init__(self,\n                 trn: str = None,\n                 dev: str = None,\n                 tst: str = None,\n                 sampler_builder: SamplerBuilder = None,\n                 dependencies: str = None,\n                 scalar_mix: ScalarMixWithDropoutBuilder = None,\n                 use_raw_hidden_states=False,\n                 lr=1e-3,\n                 separate_optimizer=False,\n                 max_seq_len=None,\n                 sent_delimiter=None,\n                 char_level=False,\n                 hard_constraint=False,\n                 tagging_scheme=None,\n                 crf=False,\n                 delimiter_in_entity=None,\n                 merge_types: List[str] = None,\n                 secondary_encoder=None,\n                 token_key='token',\n                 dict_whitelist: Union[DictInterface, Union[Dict[str, Any], Set[str]]] = None,\n                 dict_blacklist: Union[DictInterface, Union[Dict[str, Any], Set[str]]] = None,\n                 dict_tags: Union[\n                     DictInterface, Union[Dict[Union[str, Sequence[str]], Union[str, Sequence[str]]]]] = None,\n                 **kwargs) -> None:\n        r\"\"\"A simple tagger using a linear layer with an optional CRF (:cite:`lafferty2001conditional`) layer for\n        NER task. It can utilize whitelist gazetteers which is dict mapping from entity name to entity type.\n        During decoding, it performs longest-prefix-matching of these words to override the prediction from\n        underlying statistical model. It also uses a blacklist to mask out mis-predicted  entities.\n\n        .. Note:: For algorithm beginners, longest-prefix-matching is the prerequisite to understand what dictionary can\n            do and what it can't do. The tutorial in `this book <http://nlp.hankcs.com/book.php>`_ can be very helpful.\n\n        Args:\n            trn: Path to training set.\n            dev: Path to dev set.\n            tst: Path to test set.\n            sampler_builder: A builder which builds a sampler.\n            dependencies: Its dependencies on other tasks.\n            scalar_mix: A builder which builds a `ScalarMixWithDropout` object.\n            use_raw_hidden_states: Whether to use raw hidden states from transformer without any pooling.\n            lr: Learning rate for this task.\n            separate_optimizer: Use customized separate optimizer for this task.\n            max_seq_len: Sentences longer than ``max_seq_len`` will be split into shorter ones if possible.\n            sent_delimiter: Delimiter between sentences, like period or comma, which indicates a long sentence can\n                be split here.\n            char_level: Whether the sequence length is measured at char level, which is never the case for\n                lemmatization.\n            hard_constraint: Whether to enforce hard length constraint on sentences. If there is no ``sent_delimiter``\n                in a sentence, it will be split at a token anyway.\n            token_key: The key to tokens in dataset. This should always be set to ``token`` in MTL.\n            crf: ``True`` to enable CRF (:cite:`lafferty2001conditional`).\n            delimiter_in_entity: The delimiter between tokens in entity, which is used to rebuild entity by joining\n                tokens during decoding.\n            merge_types: The types of consecutive entities to be merged.\n            secondary_encoder: An optional secondary encoder to provide enhanced representation by taking the hidden\n                states from the main encoder as input.\n            token_key: The key to tokens in dataset. This should always be set to ``token`` in MTL.\n            dict_whitelist: A :class:`dict` or a :class:`~hanlp_trie.dictionary.DictInterface` of gazetteers to be\n                included into the final results.\n            dict_blacklist: A :class:`set` or a :class:`~hanlp_trie.dictionary.DictInterface` of badcases to be\n                excluded from the final results.\n            **kwargs:\n        \"\"\"\n        super().__init__(**merge_locals_kwargs(locals(), kwargs))\n        self.vocabs = VocabDict()\n        self.secondary_encoder = secondary_encoder\n        self.dict_whitelist = dict_whitelist\n        self.dict_blacklist = dict_blacklist\n        self.dict_tags = dict_tags\n\n    def build_dataloader(self,\n                         data,\n                         transform: Callable = None,\n                         training=False,\n                         device=None,\n                         logger: logging.Logger = None,\n                         cache=False,\n                         gradient_accumulation=1,\n                         **kwargs) -> DataLoader:\n        args = dict((k, self.config[k]) for k in\n                    ['delimiter', 'max_seq_len', 'sent_delimiter', 'char_level', 'hard_constraint'] if k in self.config)\n        dataset = self.build_dataset(data, cache=cache, transform=transform, **args)\n        dataset.append_transform(self.vocabs)\n        dataset.purge_cache()\n        if self.vocabs.mutable:\n            self.build_vocabs(dataset, logger)\n        return PadSequenceDataLoader(\n            batch_sampler=self.sampler_builder.build(\n                self.compute_lens(data, dataset),\n                shuffle=training, gradient_accumulation=gradient_accumulation),\n            device=device,\n            dataset=dataset)\n\n    def compute_loss(self,\n                     batch: Dict[str, Any],\n                     output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                     criterion) -> Union[torch.FloatTensor, Dict[str, torch.FloatTensor]]:\n        return TransformerNamedEntityRecognizer.compute_loss(self, criterion, output, batch['tag_id'], batch['mask'])\n\n    def decode_output(self,\n                      output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                      mask: torch.BoolTensor,\n                      batch: Dict[str, Any],\n                      decoder,\n                      **kwargs) -> Union[Dict[str, Any], Any]:\n        return TransformerNamedEntityRecognizer.decode_output(self, output, batch['mask'], batch, decoder)\n\n    def update_metrics(self,\n                       batch: Dict[str, Any],\n                       output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                       prediction: Dict[str, Any],\n                       metric: Union[MetricDict, Metric]):\n        return TransformerNamedEntityRecognizer.update_metrics(self, metric, output, batch['tag_id'], batch['mask'],\n                                                               batch, prediction)\n\n    def build_model(self, encoder_size, training=True, **kwargs) -> torch.nn.Module:\n        return LinearCRFDecoder(encoder_size, len(self.vocabs['tag']), self.secondary_encoder, self.config.crf)\n\n    def build_metric(self, **kwargs):\n        return TransformerNamedEntityRecognizer.build_metric(self, **kwargs)\n\n    def input_is_flat(self, data) -> bool:\n        return TransformerNamedEntityRecognizer.input_is_flat(self, data)\n\n    def prediction_to_result(self, prediction: Dict[str, Any], batch: Dict[str, Any]) -> Union[List, Dict]:\n        return TransformerNamedEntityRecognizer.prediction_to_human(self, prediction, self.vocabs['tag'].idx_to_token,\n                                                                    batch)\n"
  },
  {
    "path": "hanlp/components/mtl/tasks/pos.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-10-19 18:56\nimport logging\nfrom typing import Dict, Any, Union, Iterable, Callable, List, Tuple, Sequence\n\nimport torch\nfrom torch.utils.data import DataLoader\n\nfrom hanlp.common.dataset import SamplerBuilder, PadSequenceDataLoader\nfrom hanlp.common.transform import VocabDict\nfrom hanlp.components.mtl.tasks import Task\nfrom hanlp.components.taggers.transformers.transformer_tagger import TransformerTagger\nfrom hanlp.layers.crf.crf import CRF\nfrom hanlp.layers.scalar_mix import ScalarMixWithDropoutBuilder\nfrom hanlp.metrics.metric import Metric\nfrom hanlp.metrics.mtl import MetricDict\nfrom hanlp_common.util import merge_locals_kwargs\nfrom hanlp_trie import DictInterface, TrieDict\n\n\nclass LinearCRFDecoder(torch.nn.Module):\n    def __init__(self,\n                 hidden_size,\n                 num_labels,\n                 crf=False) -> None:\n        \"\"\"A linear layer with an optional CRF (:cite:`lafferty2001conditional`) layer on top of it.\n\n        Args:\n            hidden_size: Size of hidden states.\n            num_labels: Size of tag set.\n            crf: ``True`` to enable CRF (:cite:`lafferty2001conditional`).\n        \"\"\"\n        super().__init__()\n        self.classifier = torch.nn.Linear(hidden_size, num_labels)\n        self.crf = CRF(num_labels) if crf else None\n\n    def forward(self, contextualized_embeddings: torch.FloatTensor, batch: Dict[str, torch.Tensor], mask=None):\n        \"\"\"\n\n        Args:\n            contextualized_embeddings: Hidden states for contextual layer.\n            batch: A dict of a batch.\n            mask: Mask for tokens.\n\n        Returns:\n            Logits. Users are expected to call ``CRF.decode`` on these emissions during decoding and ``CRF.forward``\n            during training.\n\n        \"\"\"\n        return self.classifier(contextualized_embeddings)\n\n\nclass TransformerTagging(Task, TransformerTagger):\n\n    def __init__(self,\n                 trn: str = None,\n                 dev: str = None,\n                 tst: str = None,\n                 sampler_builder: SamplerBuilder = None,\n                 dependencies: str = None,\n                 scalar_mix: ScalarMixWithDropoutBuilder = None,\n                 use_raw_hidden_states=False,\n                 lr=1e-3,\n                 separate_optimizer=False,\n                 cls_is_bos=False,\n                 sep_is_eos=False,\n                 max_seq_len=None,\n                 sent_delimiter=None,\n                 char_level=False,\n                 hard_constraint=False,\n                 crf=False,\n                 token_key='token',\n                 dict_tags: Union[\n                     DictInterface, Union[Dict[Union[str, Sequence[str]], Union[str, Sequence[str]]]]] = None,\n                 **kwargs) -> None:\n        \"\"\"A simple tagger using a linear layer with an optional CRF (:cite:`lafferty2001conditional`) layer for\n        any tagging tasks including PoS tagging and many others. It also features with a custom dictionary ``dict_tags``\n        to perform ``longest-prefix-matching`` which replaces matched tokens with given tags.\n\n\n        .. Note:: For algorithm beginners, longest-prefix-matching is the prerequisite to understand what dictionary can\n            do and what it can't do. The tutorial in `this book <http://nlp.hankcs.com/book.php>`_ can be very helpful.\n\n        Args:\n            trn: Path to training set.\n            dev: Path to dev set.\n            tst: Path to test set.\n            sampler_builder: A builder which builds a sampler.\n            dependencies: Its dependencies on other tasks.\n            scalar_mix: A builder which builds a `ScalarMixWithDropout` object.\n            use_raw_hidden_states: Whether to use raw hidden states from transformer without any pooling.\n            lr: Learning rate for this task.\n            separate_optimizer: Use customized separate optimizer for this task.\n            cls_is_bos: ``True`` to treat the first token as ``BOS``.\n            sep_is_eos: ``True`` to treat the last token as ``EOS``.\n            max_seq_len: Sentences longer than ``max_seq_len`` will be split into shorter ones if possible.\n            sent_delimiter: Delimiter between sentences, like period or comma, which indicates a long sentence can\n                be split here.\n            char_level: Whether the sequence length is measured at char level, which is never the case for\n                lemmatization.\n            hard_constraint: Whether to enforce hard length constraint on sentences. If there is no ``sent_delimiter``\n                in a sentence, it will be split at a token anyway.\n            crf: ``True`` to enable CRF (:cite:`lafferty2001conditional`).\n            token_key: The key to tokens in dataset. This should always be set to ``token`` in MTL.\n            dict_tags: A custom dictionary to override predicted tags by performing longest-prefix-matching.\n            **kwargs: Not used.\n        \"\"\"\n        super().__init__(**merge_locals_kwargs(locals(), kwargs))\n        self.vocabs = VocabDict()\n        self.dict_tags = dict_tags\n\n    def build_dataloader(self,\n                         data,\n                         transform: Callable = None,\n                         training=False,\n                         device=None,\n                         logger: logging.Logger = None,\n                         cache=False,\n                         gradient_accumulation=1,\n                         **kwargs) -> DataLoader:\n        args = dict((k, self.config[k]) for k in\n                    ['delimiter', 'max_seq_len', 'sent_delimiter', 'char_level', 'hard_constraint'] if k in self.config)\n        dataset = self.build_dataset(data, cache=True, transform=transform, **args)\n        dataset.append_transform(self.vocabs)\n        if self.vocabs.mutable:\n            self.build_vocabs(dataset, logger)\n        return PadSequenceDataLoader(\n            batch_sampler=self.sampler_builder.build(self.compute_lens(data, dataset),\n                                                     shuffle=training, gradient_accumulation=gradient_accumulation),\n            device=device,\n            dataset=dataset)\n\n    def compute_loss(self,\n                     batch: Dict[str, Any],\n                     output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                     criterion) -> Union[torch.FloatTensor, Dict[str, torch.FloatTensor]]:\n        return TransformerTagger.compute_loss(self, criterion, output, batch['tag_id'], batch['mask'])\n\n    def decode_output(self,\n                      output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                      mask: torch.BoolTensor,\n                      batch: Dict[str, Any],\n                      decoder,\n                      **kwargs) -> Union[Dict[str, Any], Any]:\n        return TransformerTagger.decode_output(self, output, mask, batch, decoder)\n\n    def update_metrics(self,\n                       batch: Dict[str, Any],\n                       output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                       prediction: Dict[str, Any],\n                       metric: Union[MetricDict, Metric]):\n        return TransformerTagger.update_metrics(self, metric, output, batch['tag_id'], batch['mask'])\n\n    def build_model(self, encoder_size, training=True, **kwargs) -> torch.nn.Module:\n        return LinearCRFDecoder(encoder_size, len(self.vocabs['tag']), self.config.crf)\n\n    def build_metric(self, **kwargs):\n        return TransformerTagger.build_metric(self, **kwargs)\n\n    def input_is_flat(self, data) -> bool:\n        return TransformerTagger.input_is_flat(self, data)\n\n    def prediction_to_result(self, prediction: Dict[str, Any], batch: Dict[str, Any]) -> Union[List, Dict]:\n        return TransformerTagger.prediction_to_human(self, prediction, self.vocabs['tag'].idx_to_token, batch)\n"
  },
  {
    "path": "hanlp/components/mtl/tasks/sdp.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-13 21:39\nimport logging\nfrom typing import Dict, Any, Union, Iterable, List\n\nimport torch\nfrom torch.optim import Adam\nfrom torch.optim.lr_scheduler import ExponentialLR\nfrom torch.utils.data import DataLoader\n\nfrom hanlp.common.dataset import SamplerBuilder, PadSequenceDataLoader\nfrom hanlp.common.transform import VocabDict, TransformList\nfrom hanlp.components.mtl.tasks import Task\nfrom hanlp.components.parsers.biaffine.biaffine_model import BiaffineDecoder\nfrom hanlp.components.parsers.biaffine.biaffine_sdp import BiaffineSemanticDependencyParser\nfrom hanlp.layers.scalar_mix import ScalarMixWithDropoutBuilder\nfrom hanlp.metrics.metric import Metric\nfrom hanlp.metrics.mtl import MetricDict\nfrom hanlp.utils.time_util import CountdownTimer\nfrom hanlp_common.util import merge_locals_kwargs\n\n\nclass BiaffineSemanticDependencyParsing(Task, BiaffineSemanticDependencyParser):\n    def __init__(self,\n                 trn: str = None,\n                 dev: str = None,\n                 tst: str = None,\n                 sampler_builder: SamplerBuilder = None,\n                 dependencies: str = None,\n                 scalar_mix: ScalarMixWithDropoutBuilder = None,\n                 use_raw_hidden_states=False,\n                 lr=2e-3, separate_optimizer=False,\n                 punct=False,\n                 tree=True,\n                 pad_rel=None,\n                 apply_constraint=False,\n                 single_root=True,\n                 no_zero_head=None,\n                 n_mlp_arc=500,\n                 n_mlp_rel=100,\n                 mlp_dropout=.33,\n                 mu=.9,\n                 nu=.9,\n                 epsilon=1e-12,\n                 decay=.75,\n                 decay_steps=5000,\n                 cls_is_bos=True,\n                 use_pos=False,\n                 **kwargs) -> None:\n        r\"\"\"Implementation of \"Stanford's graph-based neural dependency parser at\n        the conll 2017 shared task\" (:cite:`dozat2017stanford`) and \"Establishing Strong Baselines for the New Decade\"\n        (:cite:`he-choi-2019`).\n\n        Args:\n            trn: Path to training set.\n            dev: Path to dev set.\n            tst: Path to test set.\n            sampler_builder: A builder which builds a sampler.\n            dependencies: Its dependencies on other tasks.\n            scalar_mix: A builder which builds a `ScalarMixWithDropout` object.\n            use_raw_hidden_states: Whether to use raw hidden states from transformer without any pooling.\n            lr: Learning rate for this task.\n            separate_optimizer: Use customized separate optimizer for this task.\n            punct: ``True`` to include punctuations in evaluation.\n            pad_rel: Padding token for relations.\n            apply_constraint: Enforce constraints (see following parameters).\n            single_root: Force single root.\n            no_zero_head: Every token has at least one head.\n            n_mlp_arc: Number of features for arc representation.\n            n_mlp_rel: Number of features for rel representation.\n            mlp_dropout: Dropout applied to MLPs.\n            mu: First coefficient used for computing running averages of gradient and its square in Adam.\n            nu: Second coefficient used for computing running averages of gradient and its square in Adam.\n            epsilon: Term added to the denominator to improve numerical stability\n            decay: Decay rate for exceptional lr scheduler.\n            decay_steps: Decay every ``decay_steps`` steps.\n            cls_is_bos: ``True`` to treat the first token as ``BOS``.\n            use_pos: Use pos feature.\n            **kwargs: Not used.\n        \"\"\"\n        super().__init__(**merge_locals_kwargs(locals(), kwargs))\n        self.vocabs = VocabDict()\n\n    def update_metrics(self, batch: Dict[str, Any],\n                       output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                       prediction: Dict[str, Any], metric: Union[MetricDict, Metric]):\n        BiaffineSemanticDependencyParser.update_metric(self, *prediction, batch['arc'], batch['rel_id'], output[1],\n                                                       output[-1], metric, batch)\n\n    def decode_output(self,\n                      output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                      mask: torch.BoolTensor,\n                      batch: Dict[str, Any],\n                      decoder, **kwargs) -> Union[Dict[str, Any], Any]:\n        (arc_scores, rel_scores), mask, punct_mask = output\n        return BiaffineSemanticDependencyParser.decode(self, arc_scores, rel_scores, mask, batch)\n\n    def compute_loss(self, batch: Dict[str, Any],\n                     output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any], criterion) -> \\\n            Union[torch.FloatTensor, Dict[str, torch.FloatTensor]]:\n        (arc_scores, rel_scores), mask, punct_mask = output\n        return BiaffineSemanticDependencyParser.compute_loss(self, arc_scores, rel_scores, batch['arc'],\n                                                             batch['rel_id'], mask, criterion,\n                                                             batch)\n\n    def build_model(self, encoder_size, training=True, **kwargs) -> torch.nn.Module:\n        return BiaffineDecoder(encoder_size, self.config.n_mlp_arc, self.config.n_mlp_rel, self.config.mlp_dropout,\n                               len(self.vocabs.rel))\n\n    def build_metric(self, **kwargs):\n        return BiaffineSemanticDependencyParser.build_metric(self, **kwargs)\n\n    def build_dataloader(self, data, transform: TransformList = None, training=False, device=None,\n                         logger: logging.Logger = None, gradient_accumulation=1, **kwargs) -> DataLoader:\n        dataset = BiaffineSemanticDependencyParser.build_dataset(self, data, transform)\n        dataset.purge_cache()\n        if self.vocabs.mutable:\n            BiaffineSemanticDependencyParser.build_vocabs(self, dataset, logger, transformer=True)\n        if isinstance(data, str):\n            timer = CountdownTimer(len(dataset))\n            BiaffineSemanticDependencyParser.cache_dataset(self, dataset, timer, training, logger)\n        return PadSequenceDataLoader(\n            batch_sampler=self.sampler_builder.build(self.compute_lens(data, dataset),\n                                                     shuffle=training, gradient_accumulation=gradient_accumulation),\n            device=device,\n            dataset=dataset,\n            pad=self.get_pad_dict())\n\n    def feed_batch(self, h: torch.FloatTensor, batch: Dict[str, torch.Tensor], mask: torch.BoolTensor,\n                   decoder: torch.nn.Module):\n        logits = super().feed_batch(h, batch, mask, decoder)\n        arc_scores = logits[0]\n        mask = mask.clone()\n        mask[:, 0] = 0\n        mask = self.convert_to_3d_mask(arc_scores, mask)\n        punct_mask = self.convert_to_3d_puncts(batch.get('punct_mask', None), mask)\n        return logits, mask, punct_mask\n\n    def build_optimizer(self, decoder: torch.nn.Module, **kwargs):\n        config = self.config\n        optimizer = Adam(decoder.parameters(),\n                         config.lr,\n                         (config.mu, config.nu),\n                         config.epsilon)\n        scheduler = ExponentialLR(optimizer, config.decay ** (1 / config.decay_steps))\n        return optimizer, scheduler\n\n    def input_is_flat(self, data) -> bool:\n        return BiaffineSemanticDependencyParser.input_is_flat(self, data, self.config.use_pos)\n\n    def prediction_to_result(self, prediction: Dict[str, Any], batch: Dict[str, Any]) -> List:\n        arcs, rels = prediction\n        arcs = arcs[:, 1:, :]  # Skip the ROOT\n        rels = rels[:, 1:, :]\n        arcs = arcs.tolist()\n        rels = rels.tolist()\n        vocab = self.vocabs['rel'].idx_to_token\n        for arcs_per_sent, rels_per_sent, tokens in zip(arcs, rels, batch['token']):\n            tokens = tokens[1:]\n            sent_len = len(tokens)\n            result = []\n            for a, r in zip(arcs_per_sent[:sent_len], rels_per_sent[:sent_len]):\n                heads = [i for i in range(sent_len + 1) if a[i]]\n                deprels = [vocab[r[i]] for i in range(sent_len + 1) if a[i]]\n                result.append(list(zip(heads, deprels)))\n            yield result\n\n    def build_samples(self, inputs, cls_is_bos=False, sep_is_eos=False):\n        return BiaffineSemanticDependencyParser.build_samples(self, inputs, self.config.use_pos)\n"
  },
  {
    "path": "hanlp/components/mtl/tasks/srl/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-04 16:49\n"
  },
  {
    "path": "hanlp/components/mtl/tasks/srl/bio_srl.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-04 16:50\nimport logging\nfrom typing import Dict, Any, List, Union, Iterable, Callable\n\nimport torch\nfrom torch.utils.data import DataLoader\n\nfrom hanlp.common.dataset import PadSequenceDataLoader, SamplerBuilder\nfrom hanlp.common.transform import VocabDict\nfrom hanlp.components.mtl.tasks import Task\nfrom hanlp.components.srl.span_bio.baffine_tagging import BiaffineTaggingDecoder\nfrom hanlp.components.srl.span_bio.span_bio import SpanBIOSemanticRoleLabeler\nfrom hanlp.layers.scalar_mix import ScalarMixWithDropoutBuilder\nfrom hanlp.metrics.metric import Metric\nfrom hanlp.metrics.mtl import MetricDict\nfrom hanlp_common.util import merge_locals_kwargs\nimport torch.nn.functional as F\n\n\nclass SpanBIOSemanticRoleLabeling(Task, SpanBIOSemanticRoleLabeler):\n\n    def __init__(self,\n                 trn: str = None,\n                 dev: str = None,\n                 tst: str = None,\n                 sampler_builder: SamplerBuilder = None,\n                 dependencies: str = None,\n                 scalar_mix: ScalarMixWithDropoutBuilder = None,\n                 use_raw_hidden_states=False,\n                 lr=None,\n                 separate_optimizer=False,\n                 cls_is_bos=False,\n                 sep_is_eos=False,\n                 crf=False,\n                 n_mlp_rel=300,\n                 mlp_dropout=0.2,\n                 loss_reduction='mean',\n                 doc_level_offset=True,\n                 **kwargs) -> None:\n        \"\"\"A span based Semantic Role Labeling task using BIO scheme for tagging the role of each token. Given a\n        predicate and a token, it uses biaffine (:cite:`dozat:17a`) to predict their relations as one of BIO-ROLE.\n\n        Args:\n            trn: Path to training set.\n            dev: Path to dev set.\n            tst: Path to test set.\n            sampler_builder: A builder which builds a sampler.\n            dependencies: Its dependencies on other tasks.\n            scalar_mix: A builder which builds a `ScalarMixWithDropout` object.\n            use_raw_hidden_states: Whether to use raw hidden states from transformer without any pooling.\n            lr: Learning rate for this task.\n            separate_optimizer: Use customized separate optimizer for this task.\n            cls_is_bos: ``True`` to treat the first token as ``BOS``.\n            sep_is_eos: ``True`` to treat the last token as ``EOS``.\n            crf: ``True`` to enable CRF (:cite:`lafferty2001conditional`).\n            n_mlp_rel: Output size of MLPs for representing predicate and tokens.\n            mlp_dropout: Dropout applied to MLPs.\n            loss_reduction: Loss reduction for aggregating losses.\n            doc_level_offset: ``True`` to indicate the offsets in ``jsonlines`` are of document level.\n            **kwargs: Not used.\n        \"\"\"\n        super().__init__(**merge_locals_kwargs(locals(), kwargs))\n        self.vocabs = VocabDict()\n\n    def build_dataloader(self, data, transform: Callable = None, training=False, device=None,\n                         logger: logging.Logger = None, cache=False, gradient_accumulation=1, **kwargs) -> DataLoader:\n        dataset = self.build_dataset(data, transform=[transform, self.vocabs])\n        dataset.purge_cache()\n        if self.vocabs.mutable:\n            SpanBIOSemanticRoleLabeler.build_vocabs(self, dataset, logger)\n        return PadSequenceDataLoader(\n            batch_sampler=self.sampler_builder.build(self.compute_lens(data, dataset), shuffle=training,\n                                                     gradient_accumulation=gradient_accumulation),\n            device=device,\n            dataset=dataset)\n\n    def compute_loss(self, batch: Dict[str, Any],\n                     output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any], criterion) -> \\\n            Union[torch.FloatTensor, Dict[str, torch.FloatTensor]]:\n        pred, mask = output\n        return SpanBIOSemanticRoleLabeler.compute_loss(self, criterion, pred, batch['srl_id'], mask)\n\n    def decode_output(self,\n                      output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                      mask: torch.BoolTensor,\n                      batch: Dict[str, Any],\n                      decoder: torch.nn.Module, **kwargs) -> Union[Dict[str, Any], Any]:\n        pred, mask = output\n        return SpanBIOSemanticRoleLabeler.decode_output(self, pred, mask, batch, decoder)\n\n    def update_metrics(self, batch: Dict[str, Any],\n                       output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                       prediction: Dict[str, Any], metric: Union[MetricDict, Metric]):\n        return SpanBIOSemanticRoleLabeler.update_metrics(self, metric, prediction, batch)\n\n    def build_model(self, encoder_size, training=True, **kwargs) -> torch.nn.Module:\n        return BiaffineTaggingDecoder(\n            len(self.vocabs['srl']),\n            encoder_size,\n            self.config.n_mlp_rel,\n            self.config.mlp_dropout,\n            self.config.crf,\n        )\n\n    def feed_batch(self, h: torch.FloatTensor, batch: Dict[str, torch.Tensor], mask: torch.BoolTensor,\n                   decoder: torch.nn.Module):\n        if not h.numel():  # No tokens, don't bother to run the decoder\n            return [], None\n        pred = decoder(h)\n        mask3d = self.compute_mask(mask)\n        if self.config.crf:\n            token_index = mask3d[0]\n            pred = pred.flatten(end_dim=1)[token_index]\n            pred = F.log_softmax(pred, dim=-1)\n        return pred, mask3d\n\n    def build_metric(self, **kwargs):\n        return SpanBIOSemanticRoleLabeler.build_metric(self)\n\n    def input_is_flat(self, data) -> bool:\n        return SpanBIOSemanticRoleLabeler.input_is_flat(self, data)\n\n    def prediction_to_result(self, prediction: List, batch: Dict[str, Any]) -> List:\n        yield from SpanBIOSemanticRoleLabeler.prediction_to_result(self, prediction, batch)\n"
  },
  {
    "path": "hanlp/components/mtl/tasks/srl/rank_srl.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-05 15:43\nimport logging\nfrom typing import Union, List, Dict, Any, Iterable, Callable\n\nimport torch\nfrom torch.utils.data import DataLoader\n\nfrom hanlp.common.dataset import SamplerBuilder, PadSequenceDataLoader\nfrom hanlp.common.transform import VocabDict\nfrom hanlp.components.mtl.tasks import Task\nfrom hanlp.components.srl.span_rank.span_rank import SpanRankingSemanticRoleLabeler\nfrom hanlp.components.srl.span_rank.span_ranking_srl_model import SpanRankingSRLDecoder\nfrom hanlp.layers.scalar_mix import ScalarMixWithDropoutBuilder\nfrom hanlp.metrics.metric import Metric\nfrom hanlp.metrics.mtl import MetricDict\nfrom hanlp_common.util import merge_locals_kwargs\n\n\nclass SpanRankingSemanticRoleLabeling(Task, SpanRankingSemanticRoleLabeler):\n\n    def __init__(self, trn: str = None, dev: str = None, tst: str = None, sampler_builder: SamplerBuilder = None,\n                 dependencies: str = None, scalar_mix: ScalarMixWithDropoutBuilder = None, use_raw_hidden_states=False,\n                 lr=1e-3, separate_optimizer=False,\n                 lexical_dropout=0.5,\n                 dropout=0.2,\n                 span_width_feature_size=20,\n                 ffnn_size=150,\n                 ffnn_depth=2,\n                 argument_ratio=0.8,\n                 predicate_ratio=0.4,\n                 max_arg_width=30,\n                 mlp_label_size=100,\n                 enforce_srl_constraint=False,\n                 use_gold_predicates=False,\n                 doc_level_offset=True,\n                 use_biaffine=False,\n                 loss_reduction='mean',\n                 with_argument=' ',\n                 **kwargs) -> None:\n        r\"\"\" An implementation of \"Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling\"\n        (:cite:`he-etal-2018-jointly`). It generates candidates triples of (predicate, arg_start, arg_end) and rank them.\n\n        Args:\n            trn: Path to training set.\n            dev: Path to dev set.\n            tst: Path to test set.\n            sampler_builder: A builder which builds a sampler.\n            dependencies: Its dependencies on other tasks.\n            scalar_mix: A builder which builds a `ScalarMixWithDropout` object.\n            use_raw_hidden_states: Whether to use raw hidden states from transformer without any pooling.\n            lr: Learning rate for this task.\n            separate_optimizer: Use customized separate optimizer for this task.\n            lexical_dropout: Dropout applied to hidden states of encoder.\n            dropout: Dropout used for other layers except the encoder.\n            span_width_feature_size: Span width feature size.\n            ffnn_size: Feedforward size.\n            ffnn_depth: Number of layers of feedforward MLPs.\n            argument_ratio: Ratio of candidate arguments over number of tokens.\n            predicate_ratio: Ratio of candidate predicates over number of tokens.\n            max_arg_width: Maximum argument width.\n            mlp_label_size: Feature size for label representation.\n            enforce_srl_constraint: Enforce SRL constraints (number of core ARGs etc.).\n            use_gold_predicates: Use gold predicates instead of predicting them.\n            doc_level_offset: ``True`` to indicate the offsets in ``jsonlines`` are of document level.\n            use_biaffine: ``True`` to use biaffine (:cite:`dozat:17a`) instead of lineary layer for label prediction.\n            loss_reduction: The loss reduction used in aggregating losses.\n            with_argument: The delimiter between tokens in arguments to be used for joining tokens for outputs.\n            **kwargs: Not used.\n        \"\"\"\n        super().__init__(**merge_locals_kwargs(locals(), kwargs))\n        self.vocabs = VocabDict()\n\n    def build_dataloader(self, data, transform: Callable = None, training=False, device=None,\n                         logger: logging.Logger = None, gradient_accumulation=1, **kwargs) -> DataLoader:\n        dataset = self.build_dataset(data, isinstance(data, list), logger, transform)\n        dataset.purge_cache()\n        return PadSequenceDataLoader(\n            batch_sampler=self.sampler_builder.build(self.compute_lens(data, dataset), shuffle=training,\n                                                     gradient_accumulation=gradient_accumulation),\n            device=device,\n            dataset=dataset)\n\n    def update_metrics(self, batch: Dict[str, Any],\n                       output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                       prediction: Dict[str, Any], metric: Union[MetricDict, Metric]):\n        return SpanRankingSemanticRoleLabeler.update_metrics(self, batch, {'prediction': prediction},\n                                                             tuple(metric.values()))\n\n    def decode_output(self,\n                      output: Dict[str, Any],\n                      mask: torch.BoolTensor,\n                      batch: Dict[str, Any],\n                      decoder, **kwargs) -> Union[Dict[str, Any], Any]:\n        return SpanRankingSemanticRoleLabeler.decode_output(self, output, batch)\n\n    def compute_loss(self, batch: Dict[str, Any],\n                     output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any], criterion) -> \\\n            Union[torch.FloatTensor, Dict[str, torch.FloatTensor]]:\n        return output['loss']\n\n    def build_model(self, encoder_size, training=True, **kwargs) -> torch.nn.Module:\n        return SpanRankingSRLDecoder(encoder_size, len(self.vocabs.srl_label), self.config)\n\n    def build_metric(self, **kwargs):\n        predicate_f1, end_to_end_f1 = SpanRankingSemanticRoleLabeler.build_metric(self, **kwargs)\n        return MetricDict({'predicate': predicate_f1, 'e2e': end_to_end_f1})\n\n    def build_criterion(self, **kwargs):\n        pass\n\n    def input_is_flat(self, data) -> bool:\n        return SpanRankingSemanticRoleLabeler.input_is_flat(self, data)\n\n    def prediction_to_result(self, prediction: Dict[str, Any], batch: Dict[str, Any]) -> List:\n        return SpanRankingSemanticRoleLabeler.format_dict_to_results(batch['token'], prediction, exclusive_offset=True,\n                                                                     with_predicate=True,\n                                                                     with_argument=self.config.get('with_argument',\n                                                                                                   ' '),\n                                                                     label_first=True)\n"
  },
  {
    "path": "hanlp/components/mtl/tasks/tok/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-11 16:34"
  },
  {
    "path": "hanlp/components/mtl/tasks/tok/reg_tok.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-02 16:51\nimport logging\nfrom typing import Union, List, Dict, Any, Iterable, Tuple\n\nimport torch\nfrom hanlp_common.util import merge_locals_kwargs\nfrom torch import Tensor\nfrom torch.utils.data import DataLoader\n\nimport hanlp.utils.torch_util\nfrom hanlp.common.dataset import SamplerBuilder, PadSequenceDataLoader\nfrom hanlp.common.transform import FieldLength, TransformList\nfrom hanlp.components.mtl.tasks import Task\nfrom hanlp.datasets.tokenization.loaders.txt import TextTokenizingDataset\nfrom hanlp.layers.scalar_mix import ScalarMixWithDropoutBuilder\nfrom hanlp.layers.transformers.pt_imports import PreTrainedTokenizer\nfrom hanlp.metrics.chunking.binary_chunking_f1 import BinaryChunkingF1\nfrom hanlp.transform.transformer_tokenizer import TransformerSequenceTokenizer\n\n\ndef generate_token_span_tuple(sample: dict):\n    prefix_mask = sample.get('text_prefix_mask', None)\n    if prefix_mask:\n        sample['span_tuple'] = spans = []\n        previous_prefix = 0\n        prefix_mask_ = prefix_mask[1:-1]\n        for i, mask in enumerate(prefix_mask_):\n            if i and mask:\n                spans.append((previous_prefix, i))\n                previous_prefix = i\n        spans.append((previous_prefix, len(prefix_mask_)))\n    return sample\n\n\nclass RegressionTokenizingDecoder(torch.nn.Linear):\n\n    def __init__(self, in_features: int, out_features: int = 1, bias: bool = ...) -> None:\n        super().__init__(in_features, out_features, bias)\n\n    # noinspection PyMethodOverriding\n    def forward(self, input: Tensor, **kwargs) -> Tensor:\n        return super().forward(input[:, 1:-1, :]).squeeze_(-1)\n\n\nclass RegressionTokenization(Task):\n\n    def __init__(self, trn: str = None, dev: str = None, tst: str = None, sampler_builder: SamplerBuilder = None,\n                 dependencies: str = None, scalar_mix: ScalarMixWithDropoutBuilder = None,\n                 use_raw_hidden_states=True, lr=1e-3, separate_optimizer=False, delimiter=None,\n                 max_seq_len=None, sent_delimiter=None) -> None:\n        super().__init__(**merge_locals_kwargs(locals()))\n\n    def build_criterion(self, **kwargs):\n        return torch.nn.BCEWithLogitsLoss(reduction='mean')\n\n    def build_metric(self, **kwargs):\n        return BinaryChunkingF1()\n\n    # noinspection PyMethodOverriding\n    def build_model(self, encoder_size, training=True, **kwargs) -> torch.nn.Module:\n        return RegressionTokenizingDecoder(encoder_size)\n\n    def predict(self, data: Union[str, List[str]], batch_size: int = None, **kwargs):\n        pass\n\n    def build_dataloader(self,\n                         data,\n                         transform: TransformList = None,\n                         training=False,\n                         device=None,\n                         logger: logging.Logger = None,\n                         tokenizer: PreTrainedTokenizer = None,\n                         **kwargs) -> DataLoader:\n        assert tokenizer\n        dataset = TextTokenizingDataset(data, cache=True, delimiter=self.config.sent_delimiter,\n                                        generate_idx=isinstance(data, list),\n                                        max_seq_len=self.config.max_seq_len,\n                                        sent_delimiter=self.config.sent_delimiter,\n                                        transform=[\n                                            TransformerSequenceTokenizer(tokenizer,\n                                                                         'text',\n                                                                         ret_prefix_mask=True,\n                                                                         ret_subtokens=True,\n                                                                         ),\n                                            FieldLength('text_input_ids', 'text_input_ids_length', delta=-2),\n                                            generate_token_span_tuple])\n        return PadSequenceDataLoader(\n            batch_sampler=self.sampler_builder.build(self.compute_lens(data, dataset, 'text_input_ids'),\n                                                     shuffle=training),\n            device=device,\n            dataset=dataset)\n\n    def decode_output(self,\n                      output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                      batch: Dict[str, Any], **kwargs) -> List[Tuple[int, int]]:\n        spans = BinaryChunkingF1.decode_spans(output > 0, batch['text_input_ids_length'])\n        return spans\n\n    def update_metrics(self, batch: Dict[str, Any],\n                       output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                       prediction: List[Tuple[int, int]], metric: BinaryChunkingF1):\n        metric.update(prediction, batch['span_tuple'])\n\n    def compute_loss(self, batch: Dict[str, Any],\n                     output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any], criterion):\n        mask = hanlp.utils.torch_util.lengths_to_mask(batch['text_input_ids_length'])\n        return criterion(output[mask], batch['text_prefix_mask'][:, 1:-1][mask].to(torch.float))\n"
  },
  {
    "path": "hanlp/components/mtl/tasks/tok/tag_tok.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-11 16:35\nimport logging\nfrom typing import Dict, Any, Union, Iterable, List, Set\n\nimport torch\nfrom torch.utils.data import DataLoader\n\nfrom hanlp.common.dataset import SamplerBuilder, PadSequenceDataLoader\nfrom hanlp.common.transform import VocabDict, TransformList\nfrom hanlp.components.mtl.tasks import Task\nfrom hanlp.components.tokenizers.transformer import TransformerTaggingTokenizer\nfrom hanlp.layers.crf.crf import CRF\nfrom hanlp.layers.scalar_mix import ScalarMixWithDropoutBuilder\nfrom hanlp.metrics.metric import Metric\nfrom hanlp.metrics.mtl import MetricDict\nfrom hanlp.transform.transformer_tokenizer import TransformerSequenceTokenizer\nfrom hanlp_common.util import merge_locals_kwargs\nfrom hanlp_trie import DictInterface, TrieDict\n\n\nclass LinearCRFDecoder(torch.nn.Module):\n    def __init__(self,\n                 hidden_size,\n                 num_labels,\n                 crf=False) -> None:\n        super().__init__()\n        self.classifier = torch.nn.Linear(hidden_size, num_labels)\n        self.crf = CRF(num_labels) if crf else None\n\n    def forward(self, contextualized_embeddings: torch.FloatTensor, batch: Dict[str, torch.Tensor], mask=None):\n        return self.classifier(contextualized_embeddings[:, 1:-1, :])\n\n\nclass TaggingTokenization(Task, TransformerTaggingTokenizer):\n\n    def __init__(self,\n                 trn: str = None,\n                 dev: str = None,\n                 tst: str = None,\n                 sampler_builder: SamplerBuilder = None,\n                 dependencies: str = None,\n                 scalar_mix: ScalarMixWithDropoutBuilder = None,\n                 use_raw_hidden_states=False,\n                 lr=1e-3, separate_optimizer=False,\n                 cls_is_bos=True,\n                 sep_is_eos=True,\n                 delimiter=None,\n                 max_seq_len=None, sent_delimiter=None, char_level=False, hard_constraint=False,\n                 transform=None,\n                 tagging_scheme='BMES',\n                 crf=False,\n                 token_key='token',\n                 dict_force: Union[DictInterface, Union[Dict[str, Any], Set[str]]] = None,\n                 dict_combine: Union[DictInterface, Union[Dict[str, Any], Set[str]]] = None,\n                 **kwargs) -> None:\n        \"\"\"Tokenization which casts a chunking problem into a tagging problem.\n        This task has to create batch of tokens containing both [CLS] and [SEP] since it's usually the first task\n        and later tasks might need them.\n\n        Args:\n            trn: Path to training set.\n            dev: Path to dev set.\n            tst: Path to test set.\n            sampler_builder: A builder which builds a sampler.\n            dependencies: Its dependencies on other tasks.\n            scalar_mix: A builder which builds a `ScalarMixWithDropout` object.\n            use_raw_hidden_states: Whether to use raw hidden states from transformer without any pooling.\n            lr: Learning rate for this task.\n            separate_optimizer: Use customized separate optimizer for this task.\n            cls_is_bos: ``True`` to treat the first token as ``BOS``.\n            sep_is_eos: ``True`` to treat the last token as ``EOS``.\n            delimiter: Delimiter used to split a line in the corpus.\n            max_seq_len: Sentences longer than ``max_seq_len`` will be split into shorter ones if possible.\n            sent_delimiter: Delimiter between sentences, like period or comma, which indicates a long sentence can\n                be split here.\n            char_level: Whether the sequence length is measured at char level.\n            hard_constraint: Whether to enforce hard length constraint on sentences. If there is no ``sent_delimiter``\n                in a sentence, it will be split at a token anyway.\n            transform: An optional transform to be applied to samples. Usually a character normalization transform is\n                passed in.\n            tagging_scheme: Either ``BMES`` or ``BI``.\n            crf: ``True`` to enable CRF (:cite:`lafferty2001conditional`).\n            token_key: The key to tokens in dataset. This should always be set to ``token`` in MTL.\n            **kwargs: Not used.\n        \"\"\"\n        super().__init__(**merge_locals_kwargs(locals(), kwargs, excludes=(\n            'self', 'kwargs', '__class__', 'dict_force', 'dict_combine')))  # avoid to config\n        self.transform = transform\n        self.vocabs = VocabDict()\n        self.dict_force = dict_force\n        self.dict_combine = dict_combine\n\n    def build_dataloader(self, data, transform: TransformList = None, training=False, device=None,\n                         logger: logging.Logger = None, cache=False, gradient_accumulation=1, **kwargs) -> DataLoader:\n        args = dict((k, self.config[k]) for k in\n                    ['delimiter', 'max_seq_len', 'sent_delimiter', 'char_level', 'hard_constraint'] if k in self.config)\n        # We only need those transforms before TransformerTokenizer\n        transformer_index = transform.index_by_type(TransformerSequenceTokenizer)\n        assert transformer_index is not None\n        transform = transform[:transformer_index + 1]\n        if self.transform:\n            transform.insert(0, self.transform)\n        transform.append(self.last_transform())\n        dataset = self.build_dataset(data, cache=cache, transform=transform, **args)\n        dataset.purge_cache()\n        if self.vocabs.mutable:\n            self.build_vocabs(dataset, logger)\n        return PadSequenceDataLoader(\n            batch_sampler=self.sampler_builder.build(self.compute_lens(data, dataset),\n                                                     shuffle=training, gradient_accumulation=gradient_accumulation),\n            device=device,\n            dataset=dataset)\n\n    def compute_loss(self,\n                     batch: Dict[str, Any],\n                     output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                     criterion) -> Union[torch.FloatTensor, Dict[str, torch.FloatTensor]]:\n        return TransformerTaggingTokenizer.compute_loss(self, criterion, output, batch['tag_id'], batch['mask'])\n\n    def decode_output(self, output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                      mask: torch.BoolTensor, batch: Dict[str, Any], decoder, **kwargs) -> Union[Dict[str, Any], Any]:\n        return TransformerTaggingTokenizer.decode_output(self, output, mask, batch, decoder)\n\n    def update_metrics(self, batch: Dict[str, Any],\n                       output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                       prediction: Dict[str, Any], metric: Union[MetricDict, Metric]):\n        TransformerTaggingTokenizer.update_metrics(self, metric, output, batch['tag_id'], None, batch, prediction)\n\n    def build_model(self, encoder_size, training=True, **kwargs) -> torch.nn.Module:\n        return LinearCRFDecoder(encoder_size, len(self.vocabs['tag']), self.config.crf)\n\n    def build_metric(self, **kwargs):\n        return TransformerTaggingTokenizer.build_metric(self)\n\n    def build_criterion(self, model=None, **kwargs):\n        return TransformerTaggingTokenizer.build_criterion(self, model=model, reduction='mean')\n\n    def input_is_flat(self, data) -> bool:\n        return TransformerTaggingTokenizer.input_is_flat(self, data)\n\n    def prediction_to_result(self, prediction: Dict[str, Any], batch: Dict[str, Any]) -> Union[List, Dict]:\n        return TransformerTaggingTokenizer.prediction_to_human(self, prediction, None, batch, rebuild_span=True)\n\n    def build_tokenizer(self, tokenizer: TransformerSequenceTokenizer):\n        # The transform for tokenizer needs very special settings, ensure these settings are set properly.\n        return TransformerSequenceTokenizer(\n            tokenizer.tokenizer,\n            tokenizer.input_key,\n            tokenizer.output_key,\n            tokenizer.max_seq_length,\n            tokenizer.truncate_long_sequences,\n            ret_subtokens=True,\n            ret_subtokens_group=True,\n            ret_token_span=True,\n            cls_is_bos=True,\n            sep_is_eos=True,\n            use_fast=tokenizer.tokenizer.is_fast,\n            dict_force=self.dict_force,\n            strip_cls_sep=False,\n        )\n\n    def build_samples(self, inputs, cls_is_bos=False, sep_is_eos=False):\n        return [{self.config.token_key: sent} for sent in inputs]\n\n    @property\n    def dict_force(self) -> DictInterface:\n        return TransformerTaggingTokenizer.dict_force.fget(self)\n\n    @dict_force.setter\n    def dict_force(self, dictionary: Union[DictInterface, Union[Dict[str, Any], Set[str]]]):\n        if dictionary is not None and not isinstance(dictionary, DictInterface):\n            dictionary = TrieDict(dictionary)\n        self.config.dict_force = dictionary\n\n    @property\n    def dict_combine(self) -> DictInterface:\n        return TransformerTaggingTokenizer.dict_combine.fget(self)\n\n    @dict_combine.setter\n    def dict_combine(self, dictionary: Union[DictInterface, Union[Dict[str, Any], Set[str]]]):\n        # noinspection PyArgumentList\n        TransformerTaggingTokenizer.dict_combine.fset(self, dictionary)\n\n    def transform_batch(self, batch: Dict[str, Any], results: Dict[str, Any] = None, cls_is_bos=False,\n                        sep_is_eos=False) -> Dict[str, Any]:\n        \"\"\"\n        This method is overrode to honor the zero indexed token used in custom dict. Although for a tokenizer,\n        cls_is_bos = sep_is_eos = True, its tokens don't contain [CLS] or [SEP]. This behaviour is adopted from the\n        early versions and it is better kept to avoid migration efforts.\n\n\n        Args:\n            batch: A batch of samples.\n            results: Predicted results from other tasks which might be useful for this task to utilize. Say a dep task\n                uses both token and pos as features, then it will need both tok and pos results to make a batch.\n            cls_is_bos: First token in this batch is BOS.\n            sep_is_eos: Last token in this batch is EOS.\n\n        Returns:\n            A batch.\n\n        \"\"\"\n        return batch\n"
  },
  {
    "path": "hanlp/components/mtl/tasks/ud.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-17 21:54\nimport logging\nfrom typing import Dict, Any, List, Union, Iterable, Callable\n\nimport torch\nfrom torch.utils.data import DataLoader\n\nfrom hanlp.common.dataset import SamplerBuilder, PadSequenceDataLoader\nfrom hanlp_common.document import Document\nfrom hanlp.common.transform import VocabDict, PunctuationMask\nfrom hanlp.components.mtl.tasks import Task\nfrom hanlp_common.conll import CoNLLUWord\nfrom hanlp.components.parsers.ud.ud_model import UniversalDependenciesDecoder\nfrom hanlp.components.parsers.ud.ud_parser import UniversalDependenciesParser\nfrom hanlp.components.parsers.ud.util import generate_lemma_rule, append_bos\nfrom hanlp.layers.scalar_mix import ScalarMixWithDropoutBuilder\nfrom hanlp.metrics.metric import Metric\nfrom hanlp.metrics.mtl import MetricDict\nfrom hanlp_common.util import merge_locals_kwargs\n\n\nclass UniversalDependenciesParsing(Task, UniversalDependenciesParser):\n\n    def __init__(self,\n                 trn: str = None,\n                 dev: str = None,\n                 tst: str = None,\n                 sampler_builder: SamplerBuilder = None,\n                 dependencies: str = None,\n                 scalar_mix: ScalarMixWithDropoutBuilder = None,\n                 use_raw_hidden_states=False,\n                 lr=None,\n                 separate_optimizer=False,\n                 cls_is_bos=True,\n                 sep_is_eos=False,\n                 n_mlp_arc=768,\n                 n_mlp_rel=256,\n                 mlp_dropout=.33,\n                 tree=False,\n                 proj=False,\n                 punct=False,\n                 max_seq_len=None,\n                 **kwargs) -> None:\n        r\"\"\"Universal Dependencies Parsing (lemmatization, features, PoS tagging and dependency parsing) implementation\n        of \"75 Languages, 1 Model: Parsing Universal Dependencies Universally\" (:cite:`kondratyuk-straka-2019-75`).\n\n        Args:\n            trn: Path to training set.\n            dev: Path to dev set.\n            tst: Path to test set.\n            sampler_builder: A builder which builds a sampler.\n            dependencies: Its dependencies on other tasks.\n            scalar_mix: A builder which builds a `ScalarMixWithDropout` object.\n            use_raw_hidden_states: Whether to use raw hidden states from transformer without any pooling.\n            lr: Learning rate for this task.\n            separate_optimizer: Use customized separate optimizer for this task.\n            cls_is_bos: ``True`` to treat the first token as ``BOS``.\n            sep_is_eos: ``True`` to treat the last token as ``EOS``.\n            n_mlp_arc: Number of features for arc representation.\n            n_mlp_rel: Number of features for rel representation.\n            mlp_dropout: Dropout applied to MLPs.\n            tree: ``True`` to enforce tree constraint.\n            proj: ``True`` for projective parsing.\n            punct: ``True`` to include punctuations in evaluation.\n            max_seq_len: Prune samples longer than this length. Useful for reducing GPU consumption.\n            **kwargs: Not used.\n        \"\"\"\n        super().__init__(**merge_locals_kwargs(locals(), kwargs))\n        self.vocabs = VocabDict()\n\n    def build_dataloader(self, data, transform: Callable = None, training=False, device=None,\n                         logger: logging.Logger = None, cache=False, gradient_accumulation=1, **kwargs) -> DataLoader:\n        _transform = [generate_lemma_rule, append_bos, self.vocabs, transform]\n        if isinstance(data, str) and not self.config.punct:\n            _transform.append(PunctuationMask('token', 'punct_mask'))\n        dataset = UniversalDependenciesParser.build_dataset(self, data, _transform)\n        dataset.purge_cache()\n        if self.vocabs.mutable:\n            UniversalDependenciesParser.build_vocabs(self, dataset, logger, transformer=True)\n        max_seq_len = self.config.get('max_seq_len', None)\n        if max_seq_len and isinstance(data, str):\n            dataset.prune(lambda x: len(x['token_input_ids']) > max_seq_len, logger)\n        return PadSequenceDataLoader(\n            batch_sampler=self.sampler_builder.build(self.compute_lens(data, dataset),\n                                                     shuffle=training, gradient_accumulation=gradient_accumulation),\n            device=device,\n            dataset=dataset,\n            pad={'arc': 0})\n\n    def compute_loss(self, batch: Dict[str, Any],\n                     output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any], criterion) -> \\\n            Union[torch.FloatTensor, Dict[str, torch.FloatTensor]]:\n        return output[0]['loss'] / 4  # we have 4 tasks\n\n    def decode_output(self, output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                      mask: torch.BoolTensor, batch: Dict[str, Any], decoder: torch.nn.Module, **kwargs) -> Union[\n        Dict[str, Any], Any]:\n        return UniversalDependenciesParser.decode_output(self, *output, batch)\n\n    def update_metrics(self, batch: Dict[str, Any],\n                       output: Union[torch.Tensor, Dict[str, torch.Tensor], Iterable[torch.Tensor], Any],\n                       prediction: Dict[str, Any], metric: Union[MetricDict, Metric]):\n        UniversalDependenciesParser.update_metrics(self, metric, batch, *output)\n\n    # noinspection PyMethodOverriding\n    def build_model(self,\n                    encoder_size,\n                    n_mlp_arc,\n                    n_mlp_rel,\n                    mlp_dropout,\n                    training=True,\n                    **kwargs) -> torch.nn.Module:\n        return UniversalDependenciesDecoder(\n            encoder_size,\n            n_mlp_arc,\n            n_mlp_rel,\n            mlp_dropout,\n            len(self.vocabs.rel),\n            len(self.vocabs.lemma),\n            len(self.vocabs.pos),\n            len(self.vocabs.feat),\n            0,\n            0\n        )\n\n    def build_metric(self, **kwargs):\n        return UniversalDependenciesParser.build_metric(self)\n\n    def input_is_flat(self, data) -> bool:\n        return UniversalDependenciesParser.input_is_flat(self, data)\n\n    def prediction_to_result(self, prediction: Dict[str, Any], batch: Dict[str, Any]) -> List:\n        yield from UniversalDependenciesParser.prediction_to_human(self, prediction, batch)\n\n    def feed_batch(self, h: torch.FloatTensor, batch: Dict[str, torch.Tensor], mask: torch.BoolTensor,\n                   decoder: torch.nn.Module):\n        mask = self.compute_mask(batch)\n        output_dict = decoder(h, batch, mask)\n        if decoder.training:\n            mask = mask.clone()\n        mask[:, 0] = 0\n        return output_dict, mask\n\n    def finalize_document(self, doc: Document, task_name: str):\n        lem = []\n        pos = []\n        feat = []\n        dep = []\n        for sent in doc[task_name]:\n            sent: List[CoNLLUWord] = sent\n            lem.append([x.lemma for x in sent])\n            pos.append([x.upos for x in sent])\n            feat.append([x.feats for x in sent])\n            dep.append([(x.head, x.deprel) for x in sent])\n        promoted = 0\n        if 'lem' not in doc:\n            doc['lem'] = lem\n            promoted += 1\n        if 'pos' not in doc:\n            doc['pos'] = pos\n            promoted += 1\n        if 'feat' not in doc:\n            doc['fea'] = feat\n            promoted += 1\n        if 'dep' not in doc:\n            doc['dep'] = dep\n            promoted += 1\n        if promoted == 4:\n            doc.pop(task_name)\n"
  },
  {
    "path": "hanlp/components/ner/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-07-21 17:22"
  },
  {
    "path": "hanlp/components/ner/biaffine_ner/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-07-21 18:41"
  },
  {
    "path": "hanlp/components/ner/biaffine_ner/biaffine_ner.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-07-09 18:13\nimport logging\nfrom typing import Union, List, Callable, Dict, Any\n\nfrom hanlp_common.constant import IDX\nfrom hanlp.common.structure import History\nfrom hanlp.components.ner.biaffine_ner.biaffine_ner_model import BiaffineNamedEntityRecognitionModel\nfrom hanlp.datasets.ner.loaders.json_ner import JsonNERDataset, unpack_ner\nfrom hanlp.layers.transformers.utils import build_optimizer_scheduler_with_transformer\nimport torch\nfrom torch.utils.data import DataLoader\nfrom hanlp.common.dataset import PadSequenceDataLoader\nfrom hanlp.common.torch_component import TorchComponent\nfrom hanlp.common.transform import FieldLength, TransformList\nfrom hanlp.common.vocab import Vocab\nfrom hanlp.layers.embeddings.embedding import Embedding\nfrom hanlp.metrics.f1 import F1\nfrom hanlp.utils.time_util import CountdownTimer\nfrom hanlp_common.util import merge_locals_kwargs, reorder\n\n\nclass BiaffineNamedEntityRecognizer(TorchComponent):\n\n    def __init__(self, **kwargs) -> None:\n        \"\"\"An implementation of Named Entity Recognition as Dependency Parsing (:cite:`yu-etal-2020-named`). It treats\n        every possible span as a candidate of entity and predicts its entity label. Non-entity spans are assigned NULL\n        label to be excluded. The label prediction is done with a biaffine layer (:cite:`dozat:17a`). As it makes no\n        assumption about the spans, it naturally supports flat NER and nested NER.\n\n        Args:\n            **kwargs: Predefined config.\n        \"\"\"\n        super().__init__(**kwargs)\n        self.model: BiaffineNamedEntityRecognitionModel = None\n\n    def build_optimizer(self,\n                        trn,\n                        epochs,\n                        lr,\n                        adam_epsilon,\n                        weight_decay,\n                        warmup_steps,\n                        transformer_lr,\n                        **kwargs):\n        # noinspection PyProtectedMember\n        if self.use_transformer:\n            num_training_steps = len(trn) * epochs // self.config.get('gradient_accumulation', 1)\n            optimizer, scheduler = build_optimizer_scheduler_with_transformer(self.model,\n                                                                              self._get_transformer(),\n                                                                              lr, transformer_lr,\n                                                                              num_training_steps, warmup_steps,\n                                                                              weight_decay, adam_epsilon)\n        else:\n            optimizer = torch.optim.Adam(self.model.parameters(), self.config.lr)\n            scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(\n                optimizer=optimizer,\n                mode='max',\n                factor=0.5,\n                patience=2,\n                verbose=True,\n            )\n        return optimizer, scheduler\n\n    @property\n    def use_transformer(self):\n        return 'token' not in self.vocabs\n\n    def _get_transformer(self):\n        return getattr(self.model_.embed, 'transformer', None)\n\n    def build_criterion(self, **kwargs):\n        pass\n\n    # noinspection PyProtectedMember\n    def build_metric(self, **kwargs) -> F1:\n        return F1()\n\n    def execute_training_loop(self,\n                              trn: DataLoader,\n                              dev: DataLoader,\n                              epochs,\n                              criterion,\n                              optimizer,\n                              metric,\n                              save_dir,\n                              logger: logging.Logger,\n                              devices,\n                              gradient_accumulation=1,\n                              **kwargs):\n        best_epoch, best_metric = 0, -1\n        optimizer, scheduler = optimizer\n        history = History()\n        timer = CountdownTimer(epochs)\n        ratio_width = len(f'{len(trn)}/{len(trn)}')\n        for epoch in range(1, epochs + 1):\n            logger.info(f\"[yellow]Epoch {epoch} / {epochs}:[/yellow]\")\n            self.fit_dataloader(trn, criterion, optimizer, metric, logger, history=history,\n                                gradient_accumulation=gradient_accumulation,\n                                linear_scheduler=scheduler if self._get_transformer() else None)\n            if dev:\n                self.evaluate_dataloader(dev, criterion, metric, logger, ratio_width=ratio_width)\n            report = f'{timer.elapsed_human}/{timer.total_time_human}'\n            dev_score = metric.score\n            if not self._get_transformer():\n                scheduler.step(dev_score)\n            if dev_score > best_metric:\n                self.save_weights(save_dir)\n                best_metric = dev_score\n                report += ' [red]saved[/red]'\n            timer.log(report, ratio_percentage=False, newline=True, ratio=False)\n        return best_metric\n\n    def fit_dataloader(self,\n                       trn: DataLoader,\n                       criterion,\n                       optimizer,\n                       metric,\n                       logger: logging.Logger,\n                       linear_scheduler=None,\n                       history: History = None,\n                       gradient_accumulation=1,\n                       **kwargs):\n        self.model.train()\n        timer = CountdownTimer(history.num_training_steps(len(trn), gradient_accumulation=gradient_accumulation))\n        total_loss = 0\n        self.reset_metrics(metric)\n        for batch in trn:\n            optimizer.zero_grad()\n            output_dict = self.feed_batch(batch)\n            self.update_metrics(batch, output_dict, metric)\n            loss = output_dict['loss']\n            if gradient_accumulation and gradient_accumulation > 1:\n                loss /= gradient_accumulation\n            loss.backward()\n            total_loss += loss.item()\n            if history.step(gradient_accumulation):\n                if self.config.grad_norm:\n                    torch.nn.utils.clip_grad_norm_(self.model.parameters(), self.config.grad_norm)\n                optimizer.step()\n                if linear_scheduler:\n                    linear_scheduler.step()\n                timer.log(self.report_metrics(total_loss / (timer.current + 1), metric), ratio_percentage=None,\n                          logger=logger)\n            del loss\n        return total_loss / timer.total\n\n    # noinspection PyMethodOverriding\n    @torch.no_grad()\n    def evaluate_dataloader(self,\n                            data: DataLoader,\n                            criterion: Callable,\n                            metric,\n                            logger,\n                            ratio_width=None,\n                            output=False,\n                            **kwargs):\n        self.model.eval()\n        self.reset_metrics(metric)\n        timer = CountdownTimer(len(data))\n        total_loss = 0\n        if output:\n            fp = open(output, 'w')\n        for batch in data:\n            output_dict = self.feed_batch(batch)\n            if output:\n                for sent, pred, gold in zip(batch['token'], output_dict['prediction'], batch['ner']):\n                    fp.write('Tokens\\t' + ' '.join(sent) + '\\n')\n                    fp.write('Pred\\t' + '\\t'.join(\n                        ['[' + ' '.join(sent[x:y + 1]) + f']/{label}' for x, y, label in pred]) + '\\n')\n                    fp.write('Gold\\t' + '\\t'.join(\n                        ['[' + ' '.join(sent[x:y + 1]) + f']/{label}' for x, y, label in gold]) + '\\n')\n                    fp.write('\\n')\n            self.update_metrics(batch, output_dict, metric)\n            loss = output_dict['loss']\n            total_loss += loss.item()\n            timer.log(self.report_metrics(total_loss / (timer.current + 1), metric), ratio_percentage=None,\n                      logger=logger,\n                      ratio_width=ratio_width)\n            del loss\n        if output:\n            fp.close()\n        return total_loss / timer.total, metric\n\n    def build_model(self,\n                    training=True,\n                    **kwargs) -> torch.nn.Module:\n        # noinspection PyTypeChecker\n        # embed: torch.nn.Embedding = self.config.embed.module(vocabs=self.vocabs)[0].embed\n        model = BiaffineNamedEntityRecognitionModel(self.config,\n                                                    self.config.embed.module(vocabs=self.vocabs),\n                                                    self.config.context_layer,\n                                                    len(self.vocabs.label))\n        return model\n\n    # noinspection PyMethodOverriding\n    def build_dataloader(self, data, batch_size, shuffle, device, logger: logging.Logger = None, vocabs=None,\n                         sampler_builder=None,\n                         gradient_accumulation=1,\n                         **kwargs) -> DataLoader:\n        if vocabs is None:\n            vocabs = self.vocabs\n        transform = TransformList(unpack_ner, FieldLength('token'))\n        if isinstance(self.config.embed, Embedding):\n            transform.append(self.config.embed.transform(vocabs=vocabs))\n        transform.append(self.vocabs)\n        dataset = self.build_dataset(data, vocabs, transform)\n        if vocabs.mutable:\n            self.build_vocabs(dataset, logger, vocabs)\n        if 'token' in vocabs:\n            lens = [x['token'] for x in dataset]\n        else:\n            lens = [len(x['token_input_ids']) for x in dataset]\n        if sampler_builder:\n            sampler = sampler_builder.build(lens, shuffle, gradient_accumulation)\n        else:\n            sampler = None\n        return PadSequenceDataLoader(batch_sampler=sampler,\n                                     device=device,\n                                     dataset=dataset)\n\n    def build_dataset(self, data, vocabs, transform):\n        dataset = JsonNERDataset(data, transform=transform,\n                                 doc_level_offset=self.config.get('doc_level_offset', True),\n                                 tagset=self.config.get('tagset', None))\n        dataset.append_transform(vocabs)\n        if isinstance(data, str):\n            dataset.purge_cache()  # Enable cache\n        return dataset\n\n    def predict(self, data: Union[List[str], List[List[str]]], batch_size: int = None, ret_tokens=True, **kwargs):\n        if not data:\n            return []\n        flat = self.input_is_flat(data)\n        if flat:\n            data = [data]\n        dataloader = self.build_dataloader([{'token': x} for x in data], batch_size, False, self.device)\n        predictions = []\n        orders = []\n        for batch in dataloader:\n            output_dict = self.feed_batch(batch)\n            token = batch['token']\n            prediction = output_dict['prediction']\n            self.prediction_to_result(token, prediction, predictions, ret_tokens)\n            orders.extend(batch[IDX])\n        predictions = reorder(predictions, orders)\n        if flat:\n            return predictions[0]\n        return predictions\n\n    @staticmethod\n    def prediction_to_result(token, prediction, predictions: List, ret_tokens: Union[bool, str]):\n        for tokens, ner in zip(token, prediction):\n            prediction_per_sent = []\n            for i, (b, e, l) in enumerate(ner):\n                if ret_tokens is not None:\n                    entity = tokens[b: e + 1]\n                    if isinstance(ret_tokens, str):\n                        entity = ret_tokens.join(entity)\n                    prediction_per_sent.append((entity, l, b, e + 1))\n                else:\n                    prediction_per_sent.append((b, e + 1, l))\n            predictions.append(prediction_per_sent)\n\n    @staticmethod\n    def input_is_flat(data):\n        return isinstance(data[0], str)\n\n    # noinspection PyMethodOverriding\n    def fit(self,\n            trn_data,\n            dev_data,\n            save_dir,\n            embed: Embedding,\n            context_layer,\n            sampler='sorting',\n            n_buckets=32,\n            batch_size=50,\n            lexical_dropout=0.5,\n            ffnn_size=150,\n            is_flat_ner=True,\n            doc_level_offset=True,\n            lr=1e-3,\n            transformer_lr=1e-5,\n            adam_epsilon=1e-6,\n            weight_decay=0.01,\n            warmup_steps=0.1,\n            grad_norm=5.0,\n            epochs=50,\n            loss_reduction='sum',\n            gradient_accumulation=1,\n            ret_tokens=True,\n            tagset=None,\n            sampler_builder=None,\n            devices=None,\n            logger=None,\n            seed=None,\n            **kwargs\n            ):\n        \"\"\"\n\n        Args:\n            trn_data: Path to training set.\n            dev_data: Path to dev set.\n            save_dir: The directory to save trained component.\n            embed: Embeddings to use.\n            context_layer: A contextualization layer (transformer or RNN).\n            sampler: Sampler to use.\n            n_buckets: Number of buckets to use in KMeans sampler.\n            batch_size: The number of samples in a batch.\n            lexical_dropout: Dropout applied to hidden states of context layer.\n            ffnn_size: Feedforward size for MLPs extracting the head/tail representations.\n            is_flat_ner: ``True`` for flat NER, otherwise nested NER.\n            doc_level_offset: ``True`` to indicate the offsets in ``jsonlines`` are of document level.\n            lr: Learning rate for decoder.\n            transformer_lr: Learning rate for encoder.\n            adam_epsilon: The epsilon to use in Adam.\n            weight_decay: The weight decay to use.\n            warmup_steps: The number of warmup steps.\n            grad_norm: Gradient norm for clipping.\n            epochs: The number of epochs to train.\n            loss_reduction: The loss reduction used in aggregating losses.\n            gradient_accumulation: Number of mini-batches per update step.\n            ret_tokens: A delimiter between tokens in entities so that the surface form of an entity can be rebuilt.\n            tagset: Optional tagset to prune entities outside of this tagset from datasets.\n            sampler_builder: The builder to build sampler, which will override batch_size.\n            devices: Devices this component will live on.\n            logger: Any :class:`logging.Logger` instance.\n            seed: Random seed to reproduce this training.\n            **kwargs: Not used.\n\n        Returns:\n            The best metrics on training set.\n        \"\"\"\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    def build_vocabs(self, dataset, logger, vocabs, lock=True, label_vocab_name='label', **kwargs):\n        vocabs[label_vocab_name] = label_vocab = Vocab(pad_token=None, unk_token=None)\n        # Use null to indicate no relationship\n        label_vocab.add('<null>')\n        timer = CountdownTimer(len(dataset))\n        for each in dataset:\n            timer.log('Building NER vocab [blink][yellow]...[/yellow][/blink]')\n        label_vocab.set_unk_as_safe_unk()\n        if lock:\n            vocabs.lock()\n            vocabs.summary(logger)\n\n    def reset_metrics(self, metrics):\n        metrics.reset()\n\n    def report_metrics(self, loss, metrics):\n        return f'loss: {loss:.4f} {metrics}'\n\n    def feed_batch(self, batch) -> Dict[str, Any]:\n        output_dict = self.model(batch)\n        output_dict['prediction'] = self.get_pred_ner(batch['token'], output_dict['candidate_ner_scores'])\n        return output_dict\n\n    def update_metrics(self, batch: dict, prediction: Union[Dict, List], metrics):\n        if isinstance(prediction, dict):\n            prediction = prediction['prediction']\n        assert len(prediction) == len(batch['ner'])\n        for pred, gold in zip(prediction, batch['ner']):\n            metrics(set(pred), set(gold))\n\n    def get_pred_ner(self, sentences, span_scores):\n        is_flat_ner = self.config.is_flat_ner\n        candidates = []\n        for sid, sent in enumerate(sentences):\n            for s in range(len(sent)):\n                for e in range(s, len(sent)):\n                    candidates.append((sid, s, e))\n\n        top_spans = [[] for _ in range(len(sentences))]\n        span_scores_cpu = span_scores.tolist()\n        for i, type in enumerate(torch.argmax(span_scores, dim=-1).tolist()):\n            if type > 0:\n                sid, s, e = candidates[i]\n                top_spans[sid].append((s, e, type, span_scores_cpu[i][type]))\n\n        top_spans = [sorted(top_span, reverse=True, key=lambda x: x[3]) for top_span in top_spans]\n        sent_pred_mentions = [[] for _ in range(len(sentences))]\n        for sid, top_span in enumerate(top_spans):\n            for ns, ne, t, _ in top_span:\n                for ts, te, _ in sent_pred_mentions[sid]:\n                    if ns < ts <= ne < te or ts < ns <= te < ne:\n                        # for both nested and flat ner no clash is allowed\n                        break\n                    if is_flat_ner and (ns <= ts <= te <= ne or ts <= ns <= ne <= te):\n                        # for flat ner nested mentions are not allowed\n                        break\n                else:\n                    sent_pred_mentions[sid].append((ns, ne, t))\n        pred_mentions = set((sid, s, e, t) for sid, spr in enumerate(sent_pred_mentions) for s, e, t in spr)\n        prediction = [[] for _ in range(len(sentences))]\n        idx_to_label = self.vocabs['label'].idx_to_token\n        for sid, s, e, t in sorted(pred_mentions):\n            prediction[sid].append((s, e, idx_to_label[t]))\n        return prediction\n"
  },
  {
    "path": "hanlp/components/ner/biaffine_ner/biaffine_ner_model.py",
    "content": "from typing import Dict\n\nimport torch\nimport torch.nn.functional as F\nfrom torch import nn\n\nimport hanlp.utils.torch_util\nfrom hanlp.layers.time_distributed import TimeDistributed\nfrom ...parsers.biaffine.biaffine import Biaffine\n\n\ndef initializer_1d(input_tensor, initializer):\n    assert len(input_tensor.size()) == 1\n    input_tensor = input_tensor.view(-1, 1)\n    input_tensor = initializer(input_tensor)\n    return input_tensor.view(-1)\n\n\nclass BiaffineNamedEntityRecognitionModel(nn.Module):\n\n    def __init__(self, config, embed: torch.nn.Module, context_layer: torch.nn.Module, label_space_size):\n        super(BiaffineNamedEntityRecognitionModel, self).__init__()\n        self.config = config\n        self.lexical_dropout = float(self.config.lexical_dropout)\n        self.label_space_size = label_space_size\n\n        # Initialize layers and parameters\n        self.word_embedding_dim = embed.get_output_dim()  # get the embedding dim\n        self.embed = embed\n        # Initialize context layer\n        self.context_layer = context_layer\n        context_layer_output_dim = context_layer.get_output_dim()\n\n        self.decoder = BiaffineNamedEntityRecognitionDecoder(context_layer_output_dim, config.ffnn_size,\n                                                             label_space_size, config.loss_reduction)\n\n    def forward(self,\n                batch: Dict[str, torch.Tensor]\n                ):\n        keys = 'token_length', 'begin_offset', 'end_offset', 'label_id'\n        sent_lengths, gold_starts, gold_ends, gold_labels = [batch.get(k, None) for k in keys]\n        masks = hanlp.utils.torch_util.lengths_to_mask(sent_lengths)\n        num_sentences, max_sent_length = masks.size()\n        raw_embeddings = self.embed(batch, mask=masks)\n\n        raw_embeddings = F.dropout(raw_embeddings, self.lexical_dropout, self.training)\n\n        contextualized_embeddings = self.context_layer(raw_embeddings, masks)\n        return self.decoder.decode(contextualized_embeddings, gold_starts, gold_ends, gold_labels, masks,\n                                   max_sent_length,\n                                   num_sentences, sent_lengths)\n\n\nclass BiaffineNamedEntityRecognitionDecoder(nn.Module):\n    def __init__(self, hidden_size, ffnn_size, label_space_size, loss_reduction='sum') -> None:\n        \"\"\"An implementation of the biaffine decoder in \"Named Entity Recognition as Dependency Parsing\"\n        (:cite:`yu-etal-2020-named`).\n\n        Args:\n            hidden_size: Size of hidden states.\n            ffnn_size: Feedforward size for MLPs extracting the head/tail representations.\n            label_space_size: Size of tag set.\n            loss_reduction: The loss reduction used in aggregating losses.\n        \"\"\"\n        super().__init__()\n        self.loss_reduction = loss_reduction\n\n        # MLPs\n        def new_mlp():\n            return TimeDistributed(nn.Linear(hidden_size, ffnn_size))\n\n        self.start_mlp = new_mlp()\n        self.end_mlp = new_mlp()\n        self.biaffine = Biaffine(ffnn_size, label_space_size)\n\n    def forward(self, contextualized_embeddings: torch.FloatTensor, batch: Dict[str, torch.Tensor], mask=None):\n        keys = 'token_length', 'begin_offset', 'end_offset', 'label_id'\n        sent_lengths, gold_starts, gold_ends, gold_labels = [batch.get(k, None) for k in keys]\n        if mask is None:\n            mask = hanlp.utils.torch_util.lengths_to_mask(sent_lengths)\n        num_sentences, max_sent_length = mask.size()\n        return self.decode(contextualized_embeddings, gold_starts, gold_ends, gold_labels, mask,\n                           max_sent_length,\n                           num_sentences, sent_lengths)\n\n    def get_dense_span_labels(self, span_starts, span_ends, span_labels, max_sentence_length):\n        num_sentences, max_spans_num = span_starts.size()\n\n        sentence_indices = torch.arange(0, num_sentences, device=span_starts.device).unsqueeze(1).expand(-1,\n                                                                                                         max_spans_num)\n\n        sparse_indices = torch.cat([sentence_indices.unsqueeze(2), span_starts.unsqueeze(2), span_ends.unsqueeze(2)],\n                                   dim=2)\n        rank = 3\n        dense_labels = torch.sparse.LongTensor(sparse_indices.view(num_sentences * max_spans_num, rank).t(),\n                                               span_labels.view(-1),\n                                               torch.Size([num_sentences] + [max_sentence_length] * (rank - 1))) \\\n            .to_dense()\n        return dense_labels\n\n    def decode(self, contextualized_embeddings, gold_starts, gold_ends, gold_labels, masks, max_sent_length,\n               num_sentences, sent_lengths):\n        # Apply MLPs to starts and ends, [num_sentences, max_sentences_length,emb]\n        candidate_starts_emb = self.start_mlp(contextualized_embeddings)\n        candidate_ends_emb = self.end_mlp(contextualized_embeddings)\n        candidate_ner_scores = self.biaffine(candidate_starts_emb, candidate_ends_emb).permute([0, 2, 3, 1])\n\n        \"\"\"generate candidate spans with argument pruning\"\"\"\n        # Generate masks\n        candidate_scores_mask = masks.unsqueeze(1) & masks.unsqueeze(2)\n        device = sent_lengths.device\n        sentence_ends_leq_starts = (\n            ~hanlp.utils.torch_util.lengths_to_mask(torch.arange(max_sent_length, device=device), max_sent_length)) \\\n            .unsqueeze_(0).expand(num_sentences, -1, -1)\n        candidate_scores_mask &= sentence_ends_leq_starts\n        candidate_ner_scores = candidate_ner_scores[candidate_scores_mask]\n        predict_dict = {\n            \"candidate_ner_scores\": candidate_ner_scores,\n\n        }\n        if gold_starts is not None:\n            gold_ner_labels = self.get_dense_span_labels(gold_starts, gold_ends, gold_labels, max_sent_length)\n            loss = torch.nn.functional.cross_entropy(candidate_ner_scores,\n                                                     gold_ner_labels[candidate_scores_mask],\n                                                     reduction=self.loss_reduction)\n            predict_dict['loss'] = loss\n        return predict_dict\n"
  },
  {
    "path": "hanlp/components/ner/ner_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-09-14 20:33\nfrom abc import ABC\nfrom typing import Union, Any, Tuple, Iterable\n\nimport tensorflow as tf\nfrom hanlp.components.taggers.transformers.transformer_transform_tf import TransformerTransform\n\nfrom hanlp.common.transform_tf import Transform\n\nfrom hanlp.common.keras_component import KerasComponent\nfrom hanlp.components.taggers.ngram_conv.ngram_conv_tagger import NgramConvTaggerTF\nfrom hanlp.components.taggers.rnn_tagger_tf import RNNTaggerTF\nfrom hanlp.components.taggers.transformers.transformer_tagger_tf import TransformerTaggerTF\nfrom hanlp.metrics.chunking.sequence_labeling import iobes_to_span\nfrom hanlp_common.util import merge_locals_kwargs\n\n\nclass IOBES_NamedEntityRecognizer(KerasComponent, ABC):\n\n    def predict_batch(self, batch, inputs=None):\n        for words, tags in zip(inputs, super().predict_batch(batch, inputs)):\n            yield from iobes_to_span(words, tags)\n\n\nclass IOBES_Transform(Transform):\n\n    def Y_to_outputs(self, Y: Union[tf.Tensor, Tuple[tf.Tensor]], gold=False, inputs=None, X=None,\n                     batch=None) -> Iterable:\n        for words, tags in zip(inputs, super().Y_to_outputs(Y, gold, inputs=inputs, X=X, batch=batch)):\n            yield from iobes_to_span(words, tags)\n\n\nclass RNNNamedEntityRecognizerTF(RNNTaggerTF, IOBES_NamedEntityRecognizer):\n\n    def fit(self, trn_data: str, dev_data: str = None, save_dir: str = None, embeddings=100, embedding_trainable=False,\n            rnn_input_dropout=0.2, rnn_units=100, rnn_output_dropout=0.2, epochs=20, logger=None,\n            loss: Union[tf.keras.losses.Loss, str] = None,\n            optimizer: Union[str, tf.keras.optimizers.Optimizer] = 'adam', metrics='f1', batch_size=32,\n            dev_batch_size=32, lr_decay_per_epoch=None,\n            run_eagerly=False,\n            verbose=True, **kwargs):\n        # assert kwargs.get('run_eagerly', True), 'This component can only run eagerly'\n        # kwargs['run_eagerly'] = True\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    def build_loss(self, loss, **kwargs):\n        if not loss:\n            loss = tf.keras.losses.SparseCategoricalCrossentropy(\n                reduction=tf.keras.losses.Reduction.SUM_OVER_BATCH_SIZE,\n                from_logits=True)\n        return super().build_loss(loss, **kwargs)\n\n\nclass NgramConvNamedEntityRecognizerTF(NgramConvTaggerTF, IOBES_NamedEntityRecognizer):\n\n    def fit(self, trn_data: Any, dev_data: Any, save_dir: str, word_embed: Union[str, int, dict] = 200,\n            ngram_embed: Union[str, int, dict] = 50, embedding_trainable=True, window_size=4, kernel_size=3,\n            filters=(200, 200, 200, 200, 200), dropout_embed=0.2, dropout_hidden=0.2, weight_norm=True,\n            loss: Union[tf.keras.losses.Loss, str] = None,\n            optimizer: Union[str, tf.keras.optimizers.Optimizer] = 'adam', metrics='f1', batch_size=100,\n            epochs=100, logger=None, verbose=True, **kwargs):\n        return super().fit(trn_data, dev_data, save_dir, word_embed, ngram_embed, embedding_trainable, window_size,\n                           kernel_size, filters, dropout_embed, dropout_hidden, weight_norm, loss, optimizer, metrics,\n                           batch_size, epochs, logger, verbose, **kwargs)\n\n\nclass IOBES_TransformerTransform(IOBES_Transform, TransformerTransform):\n    pass\n\n\nclass TransformerNamedEntityRecognizerTF(TransformerTaggerTF):\n\n    def __init__(self, transform: TransformerTransform = None) -> None:\n        if not transform:\n            transform = IOBES_TransformerTransform()\n        super().__init__(transform)\n\n    def fit(self, trn_data, dev_data, save_dir, transformer, optimizer='adamw', learning_rate=5e-5, weight_decay_rate=0,\n            epsilon=1e-8, clipnorm=1.0, warmup_steps_ratio=0, use_amp=False, max_seq_length=128, batch_size=32,\n            epochs=3, metrics='f1', run_eagerly=False, logger=None, verbose=True, **kwargs):\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n"
  },
  {
    "path": "hanlp/components/ner/rnn_ner.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-12 18:00\nfrom typing import Any\n\nimport torch\nfrom hanlp_common.util import merge_locals_kwargs\n\nimport hanlp.utils.span_util\nfrom hanlp.components.taggers.rnn_tagger import RNNTagger\nfrom hanlp.metrics.chunking.conlleval import SpanF1\n\n\nclass RNNNamedEntityRecognizer(RNNTagger):\n\n    def __init__(self, **kwargs) -> None:\n        \"\"\"An old-school RNN tagger using word2vec or fasttext embeddings.\n\n        Args:\n            **kwargs: Predefined config.\n        \"\"\"\n        super().__init__(**kwargs)\n\n    def build_metric(self, **kwargs):\n        return SpanF1(self.tagging_scheme)\n\n    def evaluate_dataloader(self, data, criterion, logger=None, ratio_width=None, **kwargs):\n        loss, metric = super().evaluate_dataloader(data, criterion, logger, ratio_width, **kwargs)\n        if logger:\n            logger.info(metric.result(True, False)[-1])\n        return loss, metric\n\n    def fit(self, trn_data, dev_data, save_dir, batch_size=50, epochs=100, embed=100, rnn_input=None, rnn_hidden=256,\n            drop=0.5, lr=0.001, patience=10, crf=True, optimizer='adam', token_key='token', tagging_scheme=None,\n            anneal_factor: float = 0.5, delimiter=None, anneal_patience=2, devices=None,\n            token_delimiter=None,\n            logger=None,\n            verbose=True, **kwargs):\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    def update_metrics(self, metric, logits, y, mask, batch, prediction):\n        logits = self.decode_output(logits, mask, batch)\n        if isinstance(logits, torch.Tensor):\n            logits = logits.tolist()\n        metric(self._id_to_tags(logits), batch['tag'])\n\n    def predict(self, tokens: Any, batch_size: int = None, **kwargs):\n        return super().predict(tokens, batch_size, **kwargs)\n\n    def predict_data(self, data, batch_size, **kwargs):\n        outputs = super().predict_data(data, batch_size)\n        tagging_scheme = self.tagging_scheme\n        if tagging_scheme == 'IOBES':\n            entities = [hanlp.utils.span_util.iobes_tags_to_spans(y) for y in outputs]\n        elif tagging_scheme == 'BIO':\n            entities = [hanlp.utils.span_util.bio_tags_to_spans(y) for y in outputs]\n        elif tagging_scheme == 'BIOUL':\n            entities = [hanlp.utils.span_util.bioul_tags_to_spans(y) for y in outputs]\n        else:\n            raise ValueError(f'Unrecognized tag scheme {tagging_scheme}')\n        for i, (tokens, es) in enumerate(zip(data, entities)):\n            outputs[i] = [(self.config.token_delimiter.join(tokens[b:e + 1]), t, b, e + 1) for t, (b, e) in es]\n        return outputs\n\n    def save_config(self, save_dir, filename='config.json'):\n        if self.config.token_delimiter is None:\n            self.config.token_delimiter = '' if all(\n                [len(x) == 1 for x in self.vocabs[self.config.token_key].idx_to_token[-100:]]) else ' '\n        super().save_config(save_dir, filename)\n"
  },
  {
    "path": "hanlp/components/ner/transformer_ner.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-10-07 11:08\nimport functools\nfrom typing import Union, List, Dict, Any, Set\n\nfrom hanlp_trie import DictInterface, TrieDict\n\nfrom hanlp.common.dataset import SamplerBuilder\nfrom hanlp.components.taggers.transformers.transformer_tagger import TransformerTagger\nfrom hanlp.metrics.chunking.sequence_labeling import get_entities\nfrom hanlp.metrics.f1 import F1\nfrom hanlp.datasets.ner.loaders.json_ner import prune_ner_tagset\nfrom hanlp.utils.string_util import guess_delimiter\nfrom hanlp_common.util import merge_locals_kwargs\n\n\nclass TransformerNamedEntityRecognizer(TransformerTagger):\n\n    def __init__(self, **kwargs) -> None:\n        r\"\"\"A simple tagger using transformers and a linear layer with an optional CRF\n        (:cite:`lafferty2001conditional`) layer for\n        NER task. It can utilize whitelist gazetteers which is dict mapping from entity name to entity type.\n        During decoding, it performs longest-prefix-matching of these words to override the prediction from\n        underlying statistical model. It also uses a blacklist to mask out mis-predicted  entities.\n\n        .. Note:: For algorithm beginners, longest-prefix-matching is the prerequisite to understand what dictionary can\n            do and what it can't do. The tutorial in `this book <http://nlp.hankcs.com/book.php>`_ can be very helpful.\n\n        Args:\n            **kwargs: Not used.\n        \"\"\"\n        super().__init__(**kwargs)\n\n    def build_metric(self, **kwargs):\n        return F1()\n\n    # noinspection PyMethodOverriding\n    def update_metrics(self, metric, logits, y, mask, batch, prediction):\n        for p, g in zip(prediction, self.tag_to_span(batch['tag'], batch)):\n            pred = set(p)\n            gold = set(g)\n            metric(pred, gold)\n\n    # noinspection PyMethodOverriding\n    def decode_output(self, logits, mask, batch, model=None):\n        output = super().decode_output(logits, mask, batch, model)\n        prediction = super().prediction_to_human(output, self.vocabs['tag'].idx_to_token, batch)\n        return self.tag_to_span(prediction, batch)\n\n    def tag_to_span(self, batch_tags, batch):\n        spans = []\n        sents = batch[self.config.token_key]\n        dict_whitelist = self.dict_whitelist\n        dict_blacklist = self.dict_blacklist\n        merge_types = self.config.get('merge_types', None)\n        for tags, tokens in zip(batch_tags, sents):\n            entities = get_entities(tags)\n            if dict_whitelist:\n                matches = dict_whitelist.tokenize(tokens)\n                if matches:\n                    # Fix O E-LOC O like predictions\n                    entities = get_entities(tags)\n                    for label, start, end in entities:\n                        if end - start == 1:\n                            tags[start] = 'S-' + label\n                        else:\n                            tags[start] = 'B-' + label\n                            for i in range(start + 1, end - 1):\n                                tags[i] = 'I-' + label\n                            tags[end - 1] = 'E-' + label\n                    for start, end, label in matches:\n                        if (not tags[start][0] in 'ME') and (not tags[end - 1][0] in 'BM'):\n                            if end - start == 1:\n                                tags[start] = 'S-' + label\n                            else:\n                                tags[start] = 'B-' + label\n                                for i in range(start + 1, end - 1):\n                                    tags[i] = 'I-' + label\n                                tags[end - 1] = 'E-' + label\n                    entities = get_entities(tags)\n            if merge_types and len(entities) > 1:\n                merged_entities = []\n                begin = 0\n                for i in range(1, len(entities)):\n                    if entities[begin][0] != entities[i][0] or entities[i - 1][2] != entities[i][1] \\\n                            or entities[i][0] not in merge_types:\n                        merged_entities.append((entities[begin][0], entities[begin][1], entities[i - 1][2]))\n                        begin = i\n                merged_entities.append((entities[begin][0], entities[begin][1], entities[-1][2]))\n                entities = merged_entities\n\n            if dict_blacklist:\n                pruned = []\n                delimiter_in_entity = self.config.get('delimiter_in_entity', ' ')\n                for label, start, end in entities:\n                    entity = delimiter_in_entity.join(tokens[start:end])\n                    if entity not in dict_blacklist:\n                        pruned.append((label, start, end))\n                entities = pruned\n            spans.append(entities)\n        return spans\n\n    def decorate_spans(self, spans, batch):\n        batch_ner = []\n        delimiter_in_entity = self.config.get('delimiter_in_entity', ' ')\n        for spans_per_sent, tokens in zip(spans, batch.get(f'{self.config.token_key}_', batch[self.config.token_key])):\n            ner_per_sent = []\n            for label, start, end in spans_per_sent:\n                ner_per_sent.append((delimiter_in_entity.join(tokens[start:end]), label, start, end))\n            batch_ner.append(ner_per_sent)\n        return batch_ner\n\n    def generate_prediction_filename(self, tst_data, save_dir):\n        return super().generate_prediction_filename(tst_data.replace('.tsv', '.txt'), save_dir)\n\n    def prediction_to_human(self, pred, vocab, batch):\n        return self.decorate_spans(pred, batch)\n\n    def input_is_flat(self, tokens):\n        return tokens and isinstance(tokens, list) and isinstance(tokens[0], str)\n\n    def fit(self, trn_data, dev_data, save_dir, transformer,\n            delimiter_in_entity=None,\n            merge_types: List[str] = None,\n            average_subwords=False,\n            word_dropout: float = 0.2,\n            hidden_dropout=None,\n            layer_dropout=0,\n            scalar_mix=None,\n            grad_norm=5.0,\n            lr=5e-5,\n            transformer_lr=None,\n            adam_epsilon=1e-8,\n            weight_decay=0,\n            warmup_steps=0.1,\n            crf=False,\n            secondary_encoder=None,\n            reduction='sum',\n            batch_size=32,\n            sampler_builder: SamplerBuilder = None,\n            epochs=3,\n            tagset=None,\n            token_key='token',\n            max_seq_len=None,\n            sent_delimiter=None,\n            char_level=False,\n            hard_constraint=False,\n            transform=None,\n            logger=None,\n            seed=None,\n            devices: Union[float, int, List[int]] = None,\n            **kwargs):\n        \"\"\"Fit component to training set.\n\n        Args:\n            trn_data: Training set.\n            dev_data: Development set.\n            save_dir: The directory to save trained component.\n            transformer: An identifier of a pre-trained transformer.\n            delimiter_in_entity: The delimiter between tokens in entity, which is used to rebuild entity by joining\n                tokens during decoding.\n            merge_types: The types of consecutive entities to be merged.\n            average_subwords: ``True`` to average subword representations.\n            word_dropout: Dropout rate to randomly replace a subword with MASK.\n            hidden_dropout: Dropout rate applied to hidden states.\n            layer_dropout: Randomly zero out hidden states of a transformer layer.\n            scalar_mix: Layer attention.\n            grad_norm: Gradient norm for clipping.\n            lr: Learning rate for decoder.\n            transformer_lr: Learning for encoder.\n            adam_epsilon: The epsilon to use in Adam.\n            weight_decay: The weight decay to use.\n            warmup_steps: The number of warmup steps.\n            crf: ``True`` to enable CRF (:cite:`lafferty2001conditional`).\n            secondary_encoder: An optional secondary encoder to provide enhanced representation by taking the hidden\n                states from the main encoder as input.\n            reduction: The loss reduction used in aggregating losses.\n            batch_size: The number of samples in a batch.\n            sampler_builder: The builder to build sampler, which will override batch_size.\n            epochs: The number of epochs to train.\n            tagset: Optional tagset to prune entities outside of this tagset from datasets.\n            token_key: The key to tokens in dataset.\n            max_seq_len: The maximum sequence length. Sequence longer than this will be handled by sliding\n                window.\n            sent_delimiter: Delimiter between sentences, like period or comma, which indicates a long sentence can\n                be split here.\n            char_level: Whether the sequence length is measured at char level, which is never the case for\n                lemmatization.\n            hard_constraint: Whether to enforce hard length constraint on sentences. If there is no ``sent_delimiter``\n                in a sentence, it will be split at a token anyway.\n            transform: An optional transform to be applied to samples. Usually a character normalization transform is\n                passed in.\n            devices: Devices this component will live on.\n            logger: Any :class:`logging.Logger` instance.\n            seed: Random seed to reproduce this training.\n            **kwargs: Not used.\n\n        Returns:\n            The best metrics on training set.\n        \"\"\"\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    def build_vocabs(self, trn, logger, **kwargs):\n        super().build_vocabs(trn, logger, **kwargs)\n        if self.config.get('delimiter_in_entity', None) is None:\n            # Check the first sample to guess the delimiter between tokens in a NE\n            tokens = trn[0][self.config.token_key]\n            delimiter_in_entity = guess_delimiter(tokens)\n            logger.info(f'Guess the delimiter between tokens in named entity could be [blue]\"{delimiter_in_entity}'\n                        f'\"[/blue]. If not, specify `delimiter_in_entity` in `fit()`')\n            self.config.delimiter_in_entity = delimiter_in_entity\n\n    def build_dataset(self, data, transform=None, **kwargs):\n        dataset = super().build_dataset(data, transform, **kwargs)\n        if isinstance(data, str):\n            tagset = self.config.get('tagset', None)\n            if tagset:\n                dataset.append_transform(functools.partial(prune_ner_tagset, tagset=tagset))\n        return dataset\n\n    @property\n    def dict_whitelist(self) -> DictInterface:\n        return self.config.get('dict_whitelist', None)\n\n    @dict_whitelist.setter\n    def dict_whitelist(self, dictionary: Union[DictInterface, Union[Dict[str, Any], Set[str]]]):\n        if dictionary is not None and not isinstance(dictionary, DictInterface):\n            dictionary = TrieDict(dictionary)\n        self.config.dict_whitelist = dictionary\n\n    @property\n    def dict_blacklist(self) -> DictInterface:\n        return self.config.get('dict_blacklist', None)\n\n    @dict_blacklist.setter\n    def dict_blacklist(self, dictionary: Union[DictInterface, Union[Dict[str, Any], Set[str]]]):\n        if dictionary is not None and not isinstance(dictionary, DictInterface):\n            dictionary = TrieDict(dictionary)\n        self.config.dict_blacklist = dictionary\n"
  },
  {
    "path": "hanlp/components/parsers/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-22 12:46"
  },
  {
    "path": "hanlp/components/parsers/alg.py",
    "content": "# MIT License\n#\n# Copyright (c) 2020 Yu Zhang\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\n\n\nimport torch\n\nfrom hanlp_common.conll import isprojective\n\n\ndef kmeans(x, k, max_it=32):\n    r\"\"\"\n    KMeans algorithm for clustering the sentences by length.\n\n    Args:\n        x (list[int]):\n            The list of sentence lengths.\n        k (int):\n            The number of clusters.\n            This is an approximate value. The final number of clusters can be less or equal to `k`.\n        max_it (int):\n            Maximum number of iterations.\n            If centroids does not converge after several iterations, the algorithm will be early stopped.\n\n    Returns:\n        list[float], list[list[int]]:\n            The first list contains average lengths of sentences in each cluster.\n            The second is the list of clusters holding indices of data points.\n\n    Examples:\n        >>> x = torch.randint(10,20,(10,)).tolist()\n        >>> x\n        [15, 10, 17, 11, 18, 13, 17, 19, 18, 14]\n        >>> centroids, clusters = kmeans(x, 3)\n        >>> centroids\n        [10.5, 14.0, 17.799999237060547]\n        >>> clusters\n        [[1, 3], [0, 5, 9], [2, 4, 6, 7, 8]]\n    \"\"\"\n\n    # the number of clusters must not be greater than the number of datapoints\n    x, k = torch.tensor(x, dtype=torch.float), min(len(x), k)\n    # collect unique datapoints\n    d = x.unique()\n    # initialize k centroids randomly\n    c = d[torch.randperm(len(d))[:k]]\n    # assign each datapoint to the cluster with the closest centroid\n    dists, y = torch.abs_(x.unsqueeze(-1) - c).min(-1)\n\n    for _ in range(max_it):\n        # if an empty cluster is encountered,\n        # choose the farthest datapoint from the biggest cluster and move that the empty one\n        mask = torch.arange(k).unsqueeze(-1).eq(y)\n        none = torch.where(~mask.any(-1))[0].tolist()\n        while len(none) > 0:\n            for i in none:\n                # the biggest cluster\n                b = torch.where(mask[mask.sum(-1).argmax()])[0]\n                # the datapoint farthest from the centroid of cluster b\n                f = dists[b].argmax()\n                # update the assigned cluster of f\n                y[b[f]] = i\n                # re-calculate the mask\n                mask = torch.arange(k).unsqueeze(-1).eq(y)\n            none = torch.where(~mask.any(-1))[0].tolist()\n        # update the centroids\n        c, old = (x * mask).sum(-1) / mask.sum(-1), c\n        # re-assign all datapoints to clusters\n        dists, y = torch.abs_(x.unsqueeze(-1) - c).min(-1)\n        # stop iteration early if the centroids converge\n        if c.equal(old):\n            break\n    # assign all datapoints to the new-generated clusters\n    # the empty ones are discarded\n    assigned = y.unique().tolist()\n    # get the centroids of the assigned clusters\n    centroids = c[assigned].tolist()\n    # map all values of datapoints to buckets\n    clusters = [torch.where(y.eq(i))[0].tolist() for i in assigned]\n\n    return centroids, clusters\n\n\ndef eisner(scores, mask):\n    r\"\"\"\n    First-order Eisner algorithm for projective decoding.\n\n    References:\n        - Ryan McDonald, Koby Crammer and Fernando Pereira. 2005.\n          `Online Large-Margin Training of Dependency Parsers`_.\n\n    Args:\n        scores (~torch.Tensor): ``[batch_size, seq_len, seq_len]``.\n            Scores of all dependent-head pairs.\n        mask (~torch.BoolTensor): ``[batch_size, seq_len]``.\n            The mask to avoid parsing over padding tokens.\n            The first column serving as pseudo words for roots should be ``False``.\n\n    Returns:\n        ~torch.Tensor:\n            A tensor with shape ``[batch_size, seq_len]`` for the resulting projective parse trees.\n\n    Examples:\n        >>> scores = torch.tensor([[[-13.5026, -18.3700, -13.0033, -16.6809],\n                                    [-36.5235, -28.6344, -28.4696, -31.6750],\n                                    [ -2.9084,  -7.4825,  -1.4861,  -6.8709],\n                                    [-29.4880, -27.6905, -26.1498, -27.0233]]])\n        >>> mask = torch.tensor([[False,  True,  True,  True]])\n        >>> eisner(scores, mask)\n        tensor([[0, 2, 0, 2]])\n\n    .. _Online Large-Margin Training of Dependency Parsers:\n        https://www.aclweb.org/anthology/P05-1012/\n    \"\"\"\n\n    lens = mask.sum(1)\n    batch_size, seq_len, _ = scores.shape\n    scores = scores.permute(2, 1, 0)\n    s_i = torch.full_like(scores, float('-inf'))\n    s_c = torch.full_like(scores, float('-inf'))\n    p_i = scores.new_zeros(seq_len, seq_len, batch_size).long()\n    p_c = scores.new_zeros(seq_len, seq_len, batch_size).long()\n    s_c.diagonal().fill_(0)\n\n    for w in range(1, seq_len):\n        n = seq_len - w\n        starts = p_i.new_tensor(range(n)).unsqueeze(0)\n        # ilr = C(i->r) + C(j->r+1)\n        ilr = stripe(s_c, n, w) + stripe(s_c, n, w, (w, 1))\n        # [batch_size, n, w]\n        il = ir = ilr.permute(2, 0, 1)\n        # I(j->i) = max(C(i->r) + C(j->r+1) + s(j->i)), i <= r < j\n        il_span, il_path = il.max(-1)\n        s_i.diagonal(-w).copy_(il_span + scores.diagonal(-w))\n        p_i.diagonal(-w).copy_(il_path + starts)\n        # I(i->j) = max(C(i->r) + C(j->r+1) + s(i->j)), i <= r < j\n        ir_span, ir_path = ir.max(-1)\n        s_i.diagonal(w).copy_(ir_span + scores.diagonal(w))\n        p_i.diagonal(w).copy_(ir_path + starts)\n\n        # C(j->i) = max(C(r->i) + I(j->r)), i <= r < j\n        cl = stripe(s_c, n, w, (0, 0), 0) + stripe(s_i, n, w, (w, 0))\n        cl_span, cl_path = cl.permute(2, 0, 1).max(-1)\n        s_c.diagonal(-w).copy_(cl_span)\n        p_c.diagonal(-w).copy_(cl_path + starts)\n        # C(i->j) = max(I(i->r) + C(r->j)), i < r <= j\n        cr = stripe(s_i, n, w, (0, 1)) + stripe(s_c, n, w, (1, w), 0)\n        cr_span, cr_path = cr.permute(2, 0, 1).max(-1)\n        s_c.diagonal(w).copy_(cr_span)\n        s_c[0, w][lens.ne(w)] = float('-inf')\n        p_c.diagonal(w).copy_(cr_path + starts + 1)\n\n    def backtrack(p_i, p_c, heads, i, j, complete):\n        if i == j:\n            return\n        if complete:\n            r = p_c[i, j]\n            backtrack(p_i, p_c, heads, i, r, False)\n            backtrack(p_i, p_c, heads, r, j, True)\n        else:\n            r, heads[j] = p_i[i, j], i\n            i, j = sorted((i, j))\n            backtrack(p_i, p_c, heads, i, r, True)\n            backtrack(p_i, p_c, heads, j, r + 1, True)\n\n    preds = []\n    p_c = p_c.permute(2, 0, 1).cpu()\n    p_i = p_i.permute(2, 0, 1).cpu()\n    for i, length in enumerate(lens.tolist()):\n        heads = p_c.new_zeros(length + 1, dtype=torch.long)\n        backtrack(p_i[i], p_c[i], heads, 0, length, True)\n        preds.append(heads.to(mask.device))\n\n    return pad(preds, total_length=seq_len).to(mask.device)\n\n\ndef backtrack(p_i, p_c, heads, i, j, complete):\n    if i == j:\n        return\n    if complete:\n        r = p_c[i, j]\n        backtrack(p_i, p_c, heads, i, r, False)\n        backtrack(p_i, p_c, heads, r, j, True)\n    else:\n        r, heads[j] = p_i[i, j], i\n        i, j = sorted((i, j))\n        backtrack(p_i, p_c, heads, i, r, True)\n        backtrack(p_i, p_c, heads, j, r + 1, True)\n\n\ndef stripe(x, n, w, offset=(0, 0), dim=1):\n    \"\"\"r'''Returns a diagonal stripe of the tensor.\n\n    Args:\n      x: Tensor\n      n: int\n      w: int\n      offset: tuple (Default value = (0)\n      dim: int (Default value = 1)\n      Example: \n      0): \n\n    Returns:\n\n    >>> x = torch.arange(25).view(5, 5)\n    >>> x\n    tensor([[ 0,  1,  2,  3,  4],\n            [ 5,  6,  7,  8,  9],\n            [10, 11, 12, 13, 14],\n            [15, 16, 17, 18, 19],\n            [20, 21, 22, 23, 24]])\n    >>> stripe(x, 2, 3, (1, 1))\n    tensor([[ 6,  7,  8],\n            [12, 13, 14]])\n    >>> stripe(x, 2, 3, dim=0)\n    tensor([[ 0,  5, 10],\n            [ 6, 11, 16]])\n    \"\"\"\n    x, seq_len = x.contiguous(), x.size(1)\n    stride, numel = list(x.stride()), x[0, 0].numel()\n    stride[0] = (seq_len + 1) * numel\n    stride[1] = (1 if dim == 1 else seq_len) * numel\n    return x.as_strided(size=(n, w, *x.shape[2:]),\n                        stride=stride,\n                        storage_offset=(offset[0] * seq_len + offset[1]) * numel)\n\n\ndef cky(scores, mask):\n    r\"\"\"\n    The implementation of `Cocke-Kasami-Younger`_ (CKY) algorithm to parse constituency trees.\n\n    References:\n        - Yu Zhang, Houquan Zhou and Zhenghua Li. 2020.\n          `Fast and Accurate Neural CRF Constituency Parsing`_.\n\n    Args:\n        scores (~torch.Tensor): ``[batch_size, seq_len, seq_len]``.\n            Scores of all candidate constituents.\n        mask (~torch.BoolTensor): ``[batch_size, seq_len, seq_len]``.\n            The mask to avoid parsing over padding tokens.\n            For each square matrix in a batch, the positions except upper triangular part should be masked out.\n\n    Returns:\n        Sequences of factorized predicted bracketed trees that are traversed in pre-order.\n\n    Examples:\n        >>> scores = torch.tensor([[[ 2.5659,  1.4253, -2.5272,  3.3011],\n                                    [ 1.3687, -0.5869,  1.0011,  3.3020],\n                                    [ 1.2297,  0.4862,  1.1975,  2.5387],\n                                    [-0.0511, -1.2541, -0.7577,  0.2659]]])\n        >>> mask = torch.tensor([[[False,  True,  True,  True],\n                                  [False, False,  True,  True],\n                                  [False, False, False,  True],\n                                  [False, False, False, False]]])\n        >>> cky(scores, mask)\n        [[(0, 3), (0, 1), (1, 3), (1, 2), (2, 3)]]\n\n    .. _Cocke-Kasami-Younger:\n        https://en.wikipedia.org/wiki/CYK_algorithm\n    .. _Fast and Accurate Neural CRF Constituency Parsing:\n        https://www.ijcai.org/Proceedings/2020/560/\n    \"\"\"\n\n    lens = mask[:, 0].sum(-1)\n    scores = scores.permute(1, 2, 0)\n    seq_len, seq_len, batch_size = scores.shape\n    s = scores.new_zeros(seq_len, seq_len, batch_size)\n    p = scores.new_zeros(seq_len, seq_len, batch_size).long()\n\n    for w in range(1, seq_len):\n        n = seq_len - w\n        starts = p.new_tensor(range(n)).unsqueeze(0)\n\n        if w == 1:\n            s.diagonal(w).copy_(scores.diagonal(w))\n            continue\n        # [n, w, batch_size]\n        s_span = stripe(s, n, w - 1, (0, 1)) + stripe(s, n, w - 1, (1, w), 0)\n        # [batch_size, n, w]\n        s_span = s_span.permute(2, 0, 1)\n        # [batch_size, n]\n        s_span, p_span = s_span.max(-1)\n        s.diagonal(w).copy_(s_span + scores.diagonal(w))\n        p.diagonal(w).copy_(p_span + starts + 1)\n\n    def backtrack(p, i, j):\n        if j == i + 1:\n            return [(i, j)]\n        split = p[i][j]\n        ltree = backtrack(p, i, split)\n        rtree = backtrack(p, split, j)\n        return [(i, j)] + ltree + rtree\n\n    p = p.permute(2, 0, 1).tolist()\n    trees = [backtrack(p[i], 0, length) if length else [] for i, length in enumerate(lens.tolist())]\n\n    return trees\n\n\ndef istree(sequence, proj=False, multiroot=False):\n    r\"\"\"\n    Checks if the arcs form an valid dependency tree.\n\n    Args:\n        sequence (list[int]):\n            A list of head indices.\n        proj (bool):\n            If ``True``, requires the tree to be projective. Default: ``False``.\n        multiroot (bool):\n            If ``False``, requires the tree to contain only a single root. Default: ``True``.\n\n    Returns:\n        ``True`` if the arcs form an valid tree, ``False`` otherwise.\n\n    Examples:\n        >>> istree([3, 0, 0, 3], multiroot=True)\n        True\n        >>> istree([3, 0, 0, 3], proj=True)\n        False\n    \"\"\"\n\n    if proj and not isprojective(sequence):\n        return False\n    n_roots = sum(head == 0 for head in sequence)\n    if n_roots == 0:\n        return False\n    if not multiroot and n_roots > 1:\n        return False\n    if any(i == head for i, head in enumerate(sequence, 1)):\n        return False\n    return next(tarjan(sequence), None) is None\n\n\ndef tarjan(sequence):\n    r\"\"\"\n    Tarjan algorithm for finding Strongly Connected Components (SCCs) of a graph.\n\n    Args:\n        sequence (list):\n            List of head indices.\n\n    Yields:\n        A list of indices that make up a SCC. All self-loops are ignored.\n\n    Examples:\n        >>> next(tarjan([2, 5, 0, 3, 1]))  # (1 -> 5 -> 2 -> 1) is a cycle\n        [2, 5, 1]\n    \"\"\"\n\n    sequence = [-1] + sequence\n    # record the search order, i.e., the timestep\n    dfn = [-1] * len(sequence)\n    # record the the smallest timestep in a SCC\n    low = [-1] * len(sequence)\n    # push the visited into the stack\n    stack, onstack = [], [False] * len(sequence)\n\n    def connect(i, timestep):\n        dfn[i] = low[i] = timestep[0]\n        timestep[0] += 1\n        stack.append(i)\n        onstack[i] = True\n\n        for j, head in enumerate(sequence):\n            if head != i:\n                continue\n            if dfn[j] == -1:\n                yield from connect(j, timestep)\n                low[i] = min(low[i], low[j])\n            elif onstack[j]:\n                low[i] = min(low[i], dfn[j])\n\n        # a SCC is completed\n        if low[i] == dfn[i]:\n            cycle = [stack.pop()]\n            while cycle[-1] != i:\n                onstack[cycle[-1]] = False\n                cycle.append(stack.pop())\n            onstack[i] = False\n            # ignore the self-loop\n            if len(cycle) > 1:\n                yield cycle\n\n    timestep = [0]\n    for i in range(len(sequence)):\n        if dfn[i] == -1:\n            yield from connect(i, timestep)\n\n\ndef chuliu_edmonds(s):\n    r\"\"\"\n    ChuLiu/Edmonds algorithm for non-projective decoding.\n\n    Some code is borrowed from `tdozat's implementation`_.\n    Descriptions of notations and formulas can be found in\n    `Non-projective Dependency Parsing using Spanning Tree Algorithms`_.\n\n    Notes:\n        The algorithm does not guarantee to parse a single-root tree.\n\n    References:\n        - Ryan McDonald, Fernando Pereira, Kiril Ribarov and Jan Hajic. 2005.\n          `Non-projective Dependency Parsing using Spanning Tree Algorithms`_.\n\n    Args:\n        s (~torch.Tensor): ``[seq_len, seq_len]``.\n            Scores of all dependent-head pairs.\n\n    Returns:\n        ~torch.Tensor:\n            A tensor with shape ``[seq_len]`` for the resulting non-projective parse tree.\n\n    .. _tdozat's implementation:\n        https://github.com/tdozat/Parser-v3\n    .. _Non-projective Dependency Parsing using Spanning Tree Algorithms:\n        https://www.aclweb.org/anthology/H05-1066/\n    \"\"\"\n\n    s[0, 1:] = float('-inf')\n    # prevent self-loops\n    s.diagonal()[1:].fill_(float('-inf'))\n    # select heads with highest scores\n    tree = s.argmax(-1)\n    # return the cycle finded by tarjan algorithm lazily\n    cycle = next(tarjan(tree.tolist()[1:]), None)\n    # if the tree has no cycles, then it is a MST\n    if not cycle:\n        return tree\n    # indices of cycle in the original tree\n    cycle = torch.tensor(cycle)\n    # indices of noncycle in the original tree\n    noncycle = torch.ones(len(s)).index_fill_(0, cycle, 0)\n    noncycle = torch.where(noncycle.gt(0))[0]\n\n    def contract(s):\n        # heads of cycle in original tree\n        cycle_heads = tree[cycle]\n        # scores of cycle in original tree\n        s_cycle = s[cycle, cycle_heads]\n\n        # calculate the scores of cycle's potential dependents\n        # s(c->x) = max(s(x'->x)), x in noncycle and x' in cycle\n        s_dep = s[noncycle][:, cycle]\n        # find the best cycle head for each noncycle dependent\n        deps = s_dep.argmax(1)\n        # calculate the scores of cycle's potential heads\n        # s(x->c) = max(s(x'->x) - s(a(x')->x') + s(cycle)), x in noncycle and x' in cycle\n        #                                                    a(v) is the predecessor of v in cycle\n        #                                                    s(cycle) = sum(s(a(v)->v))\n        s_head = s[cycle][:, noncycle] - s_cycle.view(-1, 1) + s_cycle.sum()\n        # find the best noncycle head for each cycle dependent\n        heads = s_head.argmax(0)\n\n        contracted = torch.cat((noncycle, torch.tensor([-1])))\n        # calculate the scores of contracted graph\n        s = s[contracted][:, contracted]\n        # set the contracted graph scores of cycle's potential dependents\n        s[:-1, -1] = s_dep[range(len(deps)), deps]\n        # set the contracted graph scores of cycle's potential heads\n        s[-1, :-1] = s_head[heads, range(len(heads))]\n\n        return s, heads, deps\n\n    # keep track of the endpoints of the edges into and out of cycle for reconstruction later\n    s, heads, deps = contract(s)\n\n    # y is the contracted tree\n    y = chuliu_edmonds(s)\n    # exclude head of cycle from y\n    y, cycle_head = y[:-1], y[-1]\n\n    # fix the subtree with no heads coming from the cycle\n    # len(y) denotes heads coming from the cycle\n    subtree = y < len(y)\n    # add the nodes to the new tree\n    tree[noncycle[subtree]] = noncycle[y[subtree]]\n    # fix the subtree with heads coming from the cycle\n    subtree = ~subtree\n    # add the nodes to the tree\n    tree[noncycle[subtree]] = cycle[deps[subtree]]\n    # fix the root of the cycle\n    cycle_root = heads[cycle_head]\n    # break the cycle and add the root of the cycle to the tree\n    tree[cycle[cycle_root]] = noncycle[cycle_head]\n\n    return tree\n\n\ndef mst(scores, mask, multiroot=False):\n    r\"\"\"\n    MST algorithm for decoding non-pojective trees.\n    This is a wrapper for ChuLiu/Edmonds algorithm.\n\n    The algorithm first runs ChuLiu/Edmonds to parse a tree and then have a check of multi-roots,\n    If ``multiroot=True`` and there indeed exist multi-roots, the algorithm seeks to find\n    best single-root trees by iterating all possible single-root trees parsed by ChuLiu/Edmonds.\n    Otherwise the resulting trees are directly taken as the final outputs.\n\n    Args:\n        scores (~torch.Tensor): ``[batch_size, seq_len, seq_len]``.\n            Scores of all dependent-head pairs.\n        mask (~torch.BoolTensor): ``[batch_size, seq_len]``.\n            The mask to avoid parsing over padding tokens.\n            The first column serving as pseudo words for roots should be ``False``.\n        muliroot (bool):\n            Ensures to parse a single-root tree If ``False``.\n\n    Returns:\n        ~torch.Tensor:\n            A tensor with shape ``[batch_size, seq_len]`` for the resulting non-projective parse trees.\n\n    Examples:\n        >>> scores = torch.tensor([[[-11.9436, -13.1464,  -6.4789, -13.8917],\n                                    [-60.6957, -60.2866, -48.6457, -63.8125],\n                                    [-38.1747, -49.9296, -45.2733, -49.5571],\n                                    [-19.7504, -23.9066,  -9.9139, -16.2088]]])\n        >>> scores[:, 0, 1:] = float('-inf')\n        >>> scores.diagonal(0, 1, 2)[1:].fill_(float('-inf'))\n        >>> mask = torch.tensor([[False,  True,  True,  True]])\n        >>> mst(scores, mask)\n        tensor([[0, 2, 0, 2]])\n    \"\"\"\n\n    batch_size, seq_len, _ = scores.shape\n    scores = scores.detach().cpu().unbind()\n\n    preds = []\n    for i, length in enumerate(mask.sum(1).tolist()):\n        s = scores[i][:length + 1, :length + 1]\n        tree = chuliu_edmonds(s)\n        roots = torch.where(tree[1:].eq(0))[0] + 1\n        if not multiroot and len(roots) > 1:\n            s_root = s[:, 0]\n            s_best = float('-inf')\n            s = s.index_fill(1, torch.tensor(0), float('-inf'))\n            for root in roots:\n                s[:, 0] = float('-inf')\n                s[root, 0] = s_root[root]\n                t = chuliu_edmonds(s)\n                s_tree = s[1:].gather(1, t[1:].unsqueeze(-1)).sum()\n                if s_tree > s_best:\n                    s_best, tree = s_tree, t\n        preds.append(tree)\n\n    return pad(preds, total_length=seq_len).to(mask.device)\n\n\ndef eisner2o(scores, mask):\n    r\"\"\"\n    Second-order Eisner algorithm for projective decoding.\n    This is an extension of the first-order one that further incorporates sibling scores into tree scoring.\n\n    References:\n        - Ryan McDonald and Fernando Pereira. 2006.\n          `Online Learning of Approximate Dependency Parsing Algorithms`_.\n\n    Args:\n        scores (~torch.Tensor, ~torch.Tensor):\n            A tuple of two tensors representing the first-order and second-order scores repectively.\n            The first (``[batch_size, seq_len, seq_len]``) holds scores of all dependent-head pairs.\n            The second (``[batch_size, seq_len, seq_len, seq_len]``) holds scores of all dependent-head-sibling triples.\n        mask (~torch.BoolTensor): ``[batch_size, seq_len]``.\n            The mask to avoid parsing over padding tokens.\n            The first column serving as pseudo words for roots should be ``False``.\n\n    Returns:\n        ~torch.Tensor:\n            A tensor with shape ``[batch_size, seq_len]`` for the resulting projective parse trees.\n\n    Examples:\n        >>> s_arc = torch.tensor([[[ -2.8092,  -7.9104,  -0.9414,  -5.4360],\n                                   [-10.3494,  -7.9298,  -3.6929,  -7.3985],\n                                   [  1.1815,  -3.8291,   2.3166,  -2.7183],\n                                   [ -3.9776,  -3.9063,  -1.6762,  -3.1861]]])\n        >>> s_sib = torch.tensor([[[[ 0.4719,  0.4154,  1.1333,  0.6946],\n                                    [ 1.1252,  1.3043,  2.1128,  1.4621],\n                                    [ 0.5974,  0.5635,  1.0115,  0.7550],\n                                    [ 1.1174,  1.3794,  2.2567,  1.4043]],\n                                   [[-2.1480, -4.1830, -2.5519, -1.8020],\n                                    [-1.2496, -1.7859, -0.0665, -0.4938],\n                                    [-2.6171, -4.0142, -2.9428, -2.2121],\n                                    [-0.5166, -1.0925,  0.5190,  0.1371]],\n                                   [[ 0.5827, -1.2499, -0.0648, -0.0497],\n                                    [ 1.4695,  0.3522,  1.5614,  1.0236],\n                                    [ 0.4647, -0.7996, -0.3801,  0.0046],\n                                    [ 1.5611,  0.3875,  1.8285,  1.0766]],\n                                   [[-1.3053, -2.9423, -1.5779, -1.2142],\n                                    [-0.1908, -0.9699,  0.3085,  0.1061],\n                                    [-1.6783, -2.8199, -1.8853, -1.5653],\n                                    [ 0.3629, -0.3488,  0.9011,  0.5674]]]])\n        >>> mask = torch.tensor([[False,  True,  True,  True]])\n        >>> eisner2o((s_arc, s_sib), mask)\n        tensor([[0, 2, 0, 2]])\n\n    .. _Online Learning of Approximate Dependency Parsing Algorithms:\n        https://www.aclweb.org/anthology/E06-1011/\n    \"\"\"\n\n    # the end position of each sentence in a batch\n    lens = mask.sum(1)\n    s_arc, s_sib = scores\n    batch_size, seq_len, _ = s_arc.shape\n    # [seq_len, seq_len, batch_size]\n    s_arc = s_arc.permute(2, 1, 0)\n    # [seq_len, seq_len, seq_len, batch_size]\n    s_sib = s_sib.permute(2, 1, 3, 0)\n    s_i = torch.full_like(s_arc, float('-inf'))\n    s_s = torch.full_like(s_arc, float('-inf'))\n    s_c = torch.full_like(s_arc, float('-inf'))\n    p_i = s_arc.new_zeros(seq_len, seq_len, batch_size).long()\n    p_s = s_arc.new_zeros(seq_len, seq_len, batch_size).long()\n    p_c = s_arc.new_zeros(seq_len, seq_len, batch_size).long()\n    s_c.diagonal().fill_(0)\n\n    for w in range(1, seq_len):\n        # n denotes the number of spans to iterate,\n        # from span (0, w) to span (n, n+w) given width w\n        n = seq_len - w\n        starts = p_i.new_tensor(range(n)).unsqueeze(0)\n        # I(j->i) = max(I(j->r) + S(j->r, i)), i < r < j |\n        #               C(j->j) + C(i->j-1))\n        #           + s(j->i)\n        # [n, w, batch_size]\n        il = stripe(s_i, n, w, (w, 1)) + stripe(s_s, n, w, (1, 0), 0)\n        il += stripe(s_sib[range(w, n + w), range(n)], n, w, (0, 1))\n        # [n, 1, batch_size]\n        il0 = stripe(s_c, n, 1, (w, w)) + stripe(s_c, n, 1, (0, w - 1))\n        # il0[0] are set to zeros since the scores of the complete spans starting from 0 are always -inf\n        il[:, -1] = il0.index_fill_(0, lens.new_tensor(0), 0).squeeze(1)\n        il_span, il_path = il.permute(2, 0, 1).max(-1)\n        s_i.diagonal(-w).copy_(il_span + s_arc.diagonal(-w))\n        p_i.diagonal(-w).copy_(il_path + starts + 1)\n        # I(i->j) = max(I(i->r) + S(i->r, j), i < r < j |\n        #               C(i->i) + C(j->i+1))\n        #           + s(i->j)\n        # [n, w, batch_size]\n        ir = stripe(s_i, n, w) + stripe(s_s, n, w, (0, w), 0)\n        ir += stripe(s_sib[range(n), range(w, n + w)], n, w)\n        ir[0] = float('-inf')\n        # [n, 1, batch_size]\n        ir0 = stripe(s_c, n, 1) + stripe(s_c, n, 1, (w, 1))\n        ir[:, 0] = ir0.squeeze(1)\n        ir_span, ir_path = ir.permute(2, 0, 1).max(-1)\n        s_i.diagonal(w).copy_(ir_span + s_arc.diagonal(w))\n        p_i.diagonal(w).copy_(ir_path + starts)\n\n        # [n, w, batch_size]\n        slr = stripe(s_c, n, w) + stripe(s_c, n, w, (w, 1))\n        slr_span, slr_path = slr.permute(2, 0, 1).max(-1)\n        # S(j, i) = max(C(i->r) + C(j->r+1)), i <= r < j\n        s_s.diagonal(-w).copy_(slr_span)\n        p_s.diagonal(-w).copy_(slr_path + starts)\n        # S(i, j) = max(C(i->r) + C(j->r+1)), i <= r < j\n        s_s.diagonal(w).copy_(slr_span)\n        p_s.diagonal(w).copy_(slr_path + starts)\n\n        # C(j->i) = max(C(r->i) + I(j->r)), i <= r < j\n        cl = stripe(s_c, n, w, (0, 0), 0) + stripe(s_i, n, w, (w, 0))\n        cl_span, cl_path = cl.permute(2, 0, 1).max(-1)\n        s_c.diagonal(-w).copy_(cl_span)\n        p_c.diagonal(-w).copy_(cl_path + starts)\n        # C(i->j) = max(I(i->r) + C(r->j)), i < r <= j\n        cr = stripe(s_i, n, w, (0, 1)) + stripe(s_c, n, w, (1, w), 0)\n        cr_span, cr_path = cr.permute(2, 0, 1).max(-1)\n        s_c.diagonal(w).copy_(cr_span)\n        # disable multi words to modify the root\n        s_c[0, w][lens.ne(w)] = float('-inf')\n        p_c.diagonal(w).copy_(cr_path + starts + 1)\n\n    def backtrack(p_i, p_s, p_c, heads, i, j, flag):\n        if i == j:\n            return\n        if flag == 'c':\n            r = p_c[i, j]\n            backtrack(p_i, p_s, p_c, heads, i, r, 'i')\n            backtrack(p_i, p_s, p_c, heads, r, j, 'c')\n        elif flag == 's':\n            r = p_s[i, j]\n            i, j = sorted((i, j))\n            backtrack(p_i, p_s, p_c, heads, i, r, 'c')\n            backtrack(p_i, p_s, p_c, heads, j, r + 1, 'c')\n        elif flag == 'i':\n            r, heads[j] = p_i[i, j], i\n            if r == i:\n                r = i + 1 if i < j else i - 1\n                backtrack(p_i, p_s, p_c, heads, j, r, 'c')\n            else:\n                backtrack(p_i, p_s, p_c, heads, i, r, 'i')\n                backtrack(p_i, p_s, p_c, heads, r, j, 's')\n\n    preds = []\n    p_i = p_i.permute(2, 0, 1).cpu()\n    p_s = p_s.permute(2, 0, 1).cpu()\n    p_c = p_c.permute(2, 0, 1).cpu()\n    for i, length in enumerate(lens.tolist()):\n        heads = p_c.new_zeros(length + 1, dtype=torch.long)\n        backtrack(p_i[i], p_s[i], p_c[i], heads, 0, length, 'c')\n        preds.append(heads.to(mask.device))\n\n    return pad(preds, total_length=seq_len).to(mask.device)\n\n\ndef pad(tensors, padding_value=0, total_length=None):\n    size = [len(tensors)] + [max(tensor.size(i) for tensor in tensors)\n                             for i in range(len(tensors[0].size()))]\n    if total_length is not None:\n        assert total_length >= size[1]\n        size[1] = total_length\n    out_tensor = tensors[0].data.new(*size).fill_(padding_value)\n    for i, tensor in enumerate(tensors):\n        out_tensor[i][[slice(0, i) for i in tensor.size()]] = tensor\n    return out_tensor\n\n\ndef decode_dep(s_arc, mask, tree=False, proj=False):\n    r\"\"\"\n    Args:\n        s_arc (~torch.Tensor): ``[batch_size, seq_len, seq_len]``.\n            Scores of all possible arcs.\n        mask (~torch.BoolTensor): ``[batch_size, seq_len]``.\n            The mask for covering the unpadded tokens.\n        tree (bool):\n            If ``True``, ensures to output well-formed trees. Default: ``False``.\n        proj (bool):\n            If ``True``, ensures to output projective trees. Default: ``False``.\n\n    Returns:\n        ~torch.Tensor, ~torch.Tensor:\n            Predicted arcs and labels of shape ``[batch_size, seq_len]``.\n    \"\"\"\n\n    lens = mask.sum(1)\n    arc_preds = s_arc.argmax(-1)\n    bad = [not istree(seq[1:i + 1], proj) for i, seq in zip(lens.tolist(), arc_preds.tolist())]\n    if tree and any(bad):\n        if proj:\n            alg = eisner\n        else:\n            alg = mst\n            s_arc.diagonal(0, 1, 2)[1:].fill_(float('-inf'))\n        arc_preds[bad] = alg(s_arc[bad], mask[bad])\n\n    return arc_preds\n"
  },
  {
    "path": "hanlp/components/parsers/alg_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-26 19:49\n# Ported from the PyTorch implementation https://github.com/zysite/biaffine-parser\nfrom typing import List\nimport numpy as np\nimport tensorflow as tf\nfrom collections import defaultdict\n\n\ndef nonzero(t: tf.Tensor) -> tf.Tensor:\n    return tf.where(t > 0)\n\n\ndef view(t: tf.Tensor, *dims) -> tf.Tensor:\n    return tf.reshape(t, dims)\n\n\ndef arange(n: int) -> tf.Tensor:\n    return tf.range(n)\n\n\ndef randperm(n: int) -> tf.Tensor:\n    return tf.random.shuffle(arange(n))\n\n\ndef tolist(t: tf.Tensor) -> List:\n    if isinstance(t, tf.Tensor):\n        t = t.numpy()\n    return t.tolist()\n\n\ndef kmeans(x, k, seed=None):\n    \"\"\"See https://github.com/zysite/biaffine-parser/blob/master/parser/utils/alg.py#L7\n\n    Args:\n      x(list): Lengths of sentences\n      k(int): \n      seed:  (Default value = None)\n\n    Returns:\n\n    \n    \"\"\"\n    x = tf.constant(x, dtype=tf.float32)\n    # count the frequency of each datapoint\n    d, indices, f = tf.unique_with_counts(x, tf.int32)\n    f = tf.cast(f, tf.float32)\n    # calculate the sum of the values of the same datapoints\n    total = d * f\n    # initialize k centroids randomly\n    c, old = tf.random.shuffle(d, seed)[:k], None\n    # assign labels to each datapoint based on centroids\n    dists = tf.abs(tf.expand_dims(d, -1) - c)\n    y = tf.argmin(dists, axis=-1, output_type=tf.int32)\n    dists = tf.gather_nd(dists, tf.transpose(tf.stack([tf.range(tf.shape(dists)[0], dtype=tf.int32), y])))\n    # make sure number of datapoints is greater than that of clusters\n    assert len(d) >= k, f\"unable to assign {len(d)} datapoints to {k} clusters\"\n\n    while old is None or not tf.reduce_all(c == old):\n        # if an empty cluster is encountered,\n        # choose the farthest datapoint from the biggest cluster\n        # and move that the empty one\n        for i in range(k):\n            if not tf.reduce_any(y == i):\n                mask = tf.cast(y == tf.expand_dims(tf.range(k, dtype=tf.int32), -1), tf.float32)\n                lens = tf.reduce_sum(mask, axis=-1)\n                biggest = view(nonzero(mask[tf.argmax(lens)]), -1)\n                farthest = tf.argmax(tf.gather(dists, biggest))\n                tf.tensor_scatter_nd_update(y, tf.expand_dims(tf.expand_dims(biggest[farthest], -1), -1), [i])\n        mask = tf.cast(y == tf.expand_dims(tf.range(k, dtype=tf.int32), -1), tf.float32)\n        # update the centroids\n        c, old = tf.cast(tf.reduce_sum(total * mask, axis=-1), tf.float32) / tf.cast(tf.reduce_sum(f * mask, axis=-1),\n                                                                                     tf.float32), c\n        # re-assign all datapoints to clusters\n        dists = tf.abs(tf.expand_dims(d, -1) - c)\n        y = tf.argmin(dists, axis=-1, output_type=tf.int32)\n        dists = tf.gather_nd(dists, tf.transpose(tf.stack([tf.range(tf.shape(dists)[0], dtype=tf.int32), y])))\n    # assign all datapoints to the new-generated clusters\n    # without considering the empty ones\n    y, (assigned, _) = tf.gather(y, indices), tf.unique(y)\n    # get the centroids of the assigned clusters\n    centroids = tf.gather(c, assigned).numpy().tolist()\n    # map all values of datapoints to buckets\n    clusters = [tf.squeeze(tf.where(y == i), axis=-1).numpy().tolist() for i in assigned]\n\n    return centroids, clusters\n\n\n# ***************************************************************\nclass Tarjan:\n    \"\"\"Computes Tarjan's algorithm for finding strongly connected components (cycles) of a graph\"\"\"\n\n    def __init__(self, prediction, tokens):\n        \"\"\"\n\n        Parameters\n        ----------\n        prediction : numpy.ndarray\n            a predicted dependency tree where prediction[dep_idx] = head_idx\n        tokens : numpy.ndarray\n            the tokens we care about (i.e. exclude _GO, _EOS, and _PAD)\n        \"\"\"\n        self._edges = defaultdict(set)\n        self._vertices = set((0,))\n        for dep, head in enumerate(prediction[tokens]):\n            self._vertices.add(dep + 1)\n            self._edges[head].add(dep + 1)\n        self._indices = {}\n        self._lowlinks = {}\n        self._onstack = defaultdict(lambda: False)\n        self._SCCs = []\n\n        index = 0\n        stack = []\n        for v in self.vertices:\n            if v not in self.indices:\n                self.strongconnect(v, index, stack)\n\n    # =============================================================\n    def strongconnect(self, v, index, stack):\n        \"\"\"\n\n        Args:\n          v: \n          index: \n          stack: \n\n        Returns:\n\n        \"\"\"\n\n        self._indices[v] = index\n        self._lowlinks[v] = index\n        index += 1\n        stack.append(v)\n        self._onstack[v] = True\n        for w in self.edges[v]:\n            if w not in self.indices:\n                self.strongconnect(w, index, stack)\n                self._lowlinks[v] = min(self._lowlinks[v], self._lowlinks[w])\n            elif self._onstack[w]:\n                self._lowlinks[v] = min(self._lowlinks[v], self._indices[w])\n\n        if self._lowlinks[v] == self._indices[v]:\n            self._SCCs.append(set())\n            while stack[-1] != v:\n                w = stack.pop()\n                self._onstack[w] = False\n                self._SCCs[-1].add(w)\n            w = stack.pop()\n            self._onstack[w] = False\n            self._SCCs[-1].add(w)\n        return\n\n    # ======================\n    @property\n    def edges(self):\n        return self._edges\n\n    @property\n    def vertices(self):\n        return self._vertices\n\n    @property\n    def indices(self):\n        return self._indices\n\n    @property\n    def SCCs(self):\n        return self._SCCs\n\n\ndef tarjan(parse_probs, length, tokens_to_keep, ensure_tree=True):\n    \"\"\"Adopted from Timothy Dozat https://github.com/tdozat/Parser/blob/master/lib/models/nn.py\n\n    Args:\n      parse_probs(NDArray): seq_len x seq_len, the probability of arcs\n      length(NDArray): sentence length including ROOT\n      tokens_to_keep(NDArray): mask matrix\n      ensure_tree:  (Default value = True)\n\n    Returns:\n\n    \n    \"\"\"\n    if ensure_tree:\n        I = np.eye(len(tokens_to_keep))\n        # block loops and pad heads\n        parse_probs = parse_probs * tokens_to_keep * (1 - I)\n        parse_preds = np.argmax(parse_probs, axis=1)\n        tokens = np.arange(1, length)\n        roots = np.where(parse_preds[tokens] == 0)[0] + 1\n        # ensure at least one root\n        if len(roots) < 1:\n            # The current root probabilities\n            root_probs = parse_probs[tokens, 0]\n            # The current head probabilities\n            old_head_probs = parse_probs[tokens, parse_preds[tokens]]\n            # Get new potential root probabilities\n            new_root_probs = root_probs / old_head_probs\n            # Select the most probable root\n            new_root = tokens[np.argmax(new_root_probs)]\n            # Make the change\n            parse_preds[new_root] = 0\n        # ensure at most one root\n        elif len(roots) > 1:\n            # The probabilities of the current heads\n            root_probs = parse_probs[roots, 0]\n            # Set the probability of depending on the root zero\n            parse_probs[roots, 0] = 0\n            # Get new potential heads and their probabilities\n            new_heads = np.argmax(parse_probs[roots][:, tokens], axis=1) + 1\n            new_head_probs = parse_probs[roots, new_heads] / root_probs\n            # Select the most probable root\n            new_root = roots[np.argmin(new_head_probs)]\n            # Make the change\n            parse_preds[roots] = new_heads\n            parse_preds[new_root] = 0\n        # remove cycles\n        tarjan = Tarjan(parse_preds, tokens)\n        for SCC in tarjan.SCCs:\n            if len(SCC) > 1:\n                dependents = set()\n                to_visit = set(SCC)\n                while len(to_visit) > 0:\n                    node = to_visit.pop()\n                    if not node in dependents:\n                        dependents.add(node)\n                        to_visit.update(tarjan.edges[node])\n                # The indices of the nodes that participate in the cycle\n                cycle = np.array(list(SCC))\n                # The probabilities of the current heads\n                old_heads = parse_preds[cycle]\n                old_head_probs = parse_probs[cycle, old_heads]\n                # Set the probability of depending on a non-head to zero\n                non_heads = np.array(list(dependents))\n                parse_probs[np.repeat(cycle, len(non_heads)), np.repeat([non_heads], len(cycle), axis=0).flatten()] = 0\n                # Get new potential heads and their probabilities\n                new_heads = np.argmax(parse_probs[cycle][:, tokens], axis=1) + 1\n                new_head_probs = parse_probs[cycle, new_heads] / old_head_probs\n                # Select the most probable change\n                change = np.argmax(new_head_probs)\n                changed_cycle = cycle[change]\n                old_head = old_heads[change]\n                new_head = new_heads[change]\n                # Make the change\n                parse_preds[changed_cycle] = new_head\n                tarjan.edges[new_head].add(changed_cycle)\n                tarjan.edges[old_head].remove(changed_cycle)\n        return parse_preds\n    else:\n        # block and pad heads\n        parse_probs = parse_probs * tokens_to_keep\n        parse_preds = np.argmax(parse_probs, axis=1)\n        return parse_preds\n\n\ndef rel_argmax(rel_probs, length, root, ensure_tree=True):\n    \"\"\"Fix the relation prediction by heuristic rules\n\n    Args:\n      rel_probs(NDArray): seq_len x rel_size\n      length: real sentence length\n      ensure_tree:  (Default value = True)\n      root: \n\n    Returns:\n\n    \n    \"\"\"\n    if ensure_tree:\n        tokens = np.arange(1, length)\n        rel_preds = np.argmax(rel_probs, axis=1)\n        roots = np.where(rel_preds[tokens] == root)[0] + 1\n        if len(roots) < 1:\n            rel_preds[1 + np.argmax(rel_probs[tokens, root])] = root\n        elif len(roots) > 1:\n            root_probs = rel_probs[roots, root]\n            rel_probs[roots, root] = 0\n            new_rel_preds = np.argmax(rel_probs[roots], axis=1)\n            new_rel_probs = rel_probs[roots, new_rel_preds] / root_probs\n            new_root = roots[np.argmin(new_rel_probs)]\n            rel_preds[roots] = new_rel_preds\n            rel_preds[new_root] = root\n        return rel_preds\n    else:\n        rel_preds = np.argmax(rel_probs, axis=1)\n        return rel_preds\n"
  },
  {
    "path": "hanlp/components/parsers/biaffine/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-08 20:43\n"
  },
  {
    "path": "hanlp/components/parsers/biaffine/biaffine.py",
    "content": "# MIT License\n#\n# Copyright (c) 2020 Yu Zhang\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\n\n\nimport torch\nimport torch.nn as nn\n\n\nclass Biaffine(nn.Module):\n    r\"\"\"\n    Biaffine layer for first-order scoring.\n\n    This function has a tensor of weights :math:`W` and bias terms if needed.\n    The score :math:`s(x, y)` of the vector pair :math:`(x, y)` is computed as :math:`x^T W y`,\n    in which :math:`x` and :math:`y` can be concatenated with bias terms.\n\n    References:\n        - Timothy Dozat and Christopher D. Manning. 2017.\n          `Deep Biaffine Attention for Neural Dependency Parsing`_.\n\n    Args:\n        n_in (int):\n            The size of the input feature.\n        n_out (int):\n            The number of output channels.\n        bias_x (bool):\n            If ``True``, adds a bias term for tensor :math:`x`. Default: ``True``.\n        bias_y (bool):\n            If ``True``, adds a bias term for tensor :math:`y`. Default: ``True``.\n\n    .. _Deep Biaffine Attention for Neural Dependency Parsing:\n        https://openreview.net/forum?id=Hk95PK9le\n    \"\"\"\n\n    def __init__(self, n_in, n_out=1, bias_x=True, bias_y=True):\n        super().__init__()\n\n        self.n_in = n_in\n        self.n_out = n_out\n        self.bias_x = bias_x\n        self.bias_y = bias_y\n        self.weight = nn.Parameter(torch.Tensor(n_out, n_in + bias_x, n_in + bias_y))\n\n        self.reset_parameters()\n\n    def __repr__(self):\n        s = f\"n_in={self.n_in}, n_out={self.n_out}\"\n        if self.bias_x:\n            s += f\", bias_x={self.bias_x}\"\n        if self.bias_y:\n            s += f\", bias_y={self.bias_y}\"\n\n        return f\"{self.__class__.__name__}({s})\"\n\n    def reset_parameters(self):\n        nn.init.zeros_(self.weight)\n\n    def forward(self, x, y):\n        r\"\"\"\n        Args:\n            x (torch.Tensor): ``[batch_size, seq_len, n_in]``.\n            y (torch.Tensor): ``[batch_size, seq_len, n_in]``.\n\n        Returns:\n            ~torch.Tensor:\n                A scoring tensor of shape ``[batch_size, n_out, seq_len, seq_len]``.\n                If ``n_out=1``, the dimension for ``n_out`` will be squeezed automatically.\n        \"\"\"\n\n        if self.bias_x:\n            x = torch.cat((x, torch.ones_like(x[..., :1])), -1)\n        if self.bias_y:\n            y = torch.cat((y, torch.ones_like(y[..., :1])), -1)\n        # [batch_size, n_out, seq_len, seq_len]\n        s = torch.einsum('bxi,oij,byj->boxy', x, self.weight, y)\n        # remove dim 1 if n_out == 1\n        s = s.squeeze(1)\n\n        return s\n"
  },
  {
    "path": "hanlp/components/parsers/biaffine/biaffine_2nd_dep.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-06 13:57\nimport functools\nfrom typing import Union, List, Any\n\nimport torch\nfrom hanlp_common.constant import UNK\nfrom hanlp.common.transform import TransformList\nfrom hanlp.common.vocab import Vocab\nfrom hanlp.components.parsers.biaffine.biaffine import Biaffine\nfrom hanlp.components.parsers.biaffine.biaffine_model import BiaffineDecoder, \\\n    EncoderWithContextualLayer\nfrom hanlp.components.parsers.biaffine.biaffine_dep import BiaffineDependencyParser\nfrom hanlp.components.parsers.biaffine.biaffine_sdp import BiaffineSemanticDependencyParser\nfrom hanlp_common.conll import CoNLLUWord, CoNLLSentence\nfrom hanlp.components.parsers.parse_alg import add_secondary_arcs_by_preds\nfrom hanlp.datasets.parsing.loaders.conll_dataset import append_bos\nfrom hanlp.datasets.parsing.semeval15 import unpack_deps_to_head_deprel, merge_head_deprel_with_2nd\nfrom hanlp.metrics.mtl import MetricDict\nfrom hanlp_common.util import merge_locals_kwargs\nfrom transformers import PreTrainedModel, PreTrainedTokenizer\n\n\nclass BiaffineSeparateDecoder(torch.nn.Module):\n\n    def __init__(self, hidden_size, config) -> None:\n        super().__init__()\n        self.biaffine_decoder = BiaffineDecoder(hidden_size,\n                                                config.n_mlp_arc,\n                                                config.n_mlp_rel,\n                                                config.mlp_dropout,\n                                                config.n_rels)\n        self.biaffine_decoder_2nd = BiaffineDecoder(hidden_size,\n                                                    config.n_mlp_arc,\n                                                    config.n_mlp_rel,\n                                                    config.mlp_dropout,\n                                                    config.n_rels_2nd)\n\n    def forward(self, x, mask):\n        return tuple(zip(self.biaffine_decoder(x, mask), self.biaffine_decoder_2nd(x, mask)))\n\n\nclass BiaffineJointDecoder(BiaffineDecoder):\n    def __init__(self, hidden_size, config) -> None:\n        super().__init__(hidden_size, config.n_mlp_arc, config.n_mlp_rel, config.mlp_dropout, config.n_rels)\n        # the Biaffine layers for secondary dep\n        self.arc_attn_2nd = Biaffine(n_in=config.n_mlp_arc,\n                                     bias_x=True,\n                                     bias_y=False)\n        self.rel_attn_2nd = Biaffine(n_in=config.n_mlp_rel,\n                                     n_out=config.n_rels,\n                                     bias_x=True,\n                                     bias_y=True)\n\n    def forward(self, x, mask=None, **kwargs: Any):\n        arc_d, arc_h, rel_d, rel_h = self.apply_mlps(x)\n        s_arc, s_rel = self.decode(arc_d, arc_h, rel_d, rel_h, mask, self.arc_attn, self.rel_attn)\n        s_arc_2nd, s_rel_2nd = self.decode(arc_d, arc_h, rel_d, rel_h, mask, self.arc_attn_2nd, self.rel_attn_2nd)\n        return (s_arc, s_arc_2nd), (s_rel, s_rel_2nd)\n\n\nclass BiaffineSecondaryModel(torch.nn.Module):\n\n    def __init__(self, config, pretrained_embed: torch.Tensor = None, transformer: PreTrainedModel = None,\n                 transformer_tokenizer: PreTrainedTokenizer = None):\n        super().__init__()\n        self.encoder = EncoderWithContextualLayer(config, pretrained_embed, transformer, transformer_tokenizer)\n        self.decoder = BiaffineJointDecoder(self.encoder.hidden_size, config) if config.joint \\\n            else BiaffineSeparateDecoder(self.encoder.hidden_size, config)\n\n    def forward(self,\n                words=None,\n                feats=None,\n                input_ids=None,\n                token_span=None,\n                mask=None, lens=None, **kwargs):\n        x, mask = self.encoder(words, feats, input_ids, token_span, mask, lens)\n        return self.decoder(x, mask)\n\n\nclass BiaffineSecondaryParser(BiaffineDependencyParser):\n\n    def __init__(self) -> None:\n        super().__init__()\n        self.model: BiaffineSecondaryModel = None\n\n    def build_dataset(self, data, bos_transform=None):\n        transform = TransformList(functools.partial(append_bos, pos_key='UPOS'),\n                                  functools.partial(unpack_deps_to_head_deprel, pad_rel=self.config.pad_rel,\n                                                    arc_key='arc_2nd',\n                                                    rel_key='rel_2nd'))\n        if self.config.joint:\n            transform.append(merge_head_deprel_with_2nd)\n        if bos_transform:\n            transform.append(bos_transform)\n        return super().build_dataset(data, transform)\n\n    def build_criterion(self, **kwargs):\n        # noinspection PyCallByClass\n        return super().build_criterion(**kwargs), (BiaffineSemanticDependencyParser.build_criterion(self, **kwargs))\n\n    def fit(self, trn_data, dev_data, save_dir, feat=None, n_embed=100, pretrained_embed=None, transformer=None,\n            average_subwords=False, word_dropout: float = 0.2, transformer_hidden_dropout=None, layer_dropout=0,\n            scalar_mix: int = None, embed_dropout=.33, n_lstm_hidden=400, n_lstm_layers=3, hidden_dropout=.33,\n            n_mlp_arc=500, n_mlp_rel=100, mlp_dropout=.33, lr=2e-3, transformer_lr=5e-5, mu=.9, nu=.9, epsilon=1e-12,\n            clip=5.0, decay=.75, decay_steps=5000, patience=100, batch_size=None, sampler_builder=None,\n            lowercase=False, epochs=50000, tree=False, punct=False, min_freq=2,\n            apply_constraint=True, joint=False, no_cycle=False, root=None,\n            logger=None,\n            verbose=True, unk=UNK, pad_rel=None, max_sequence_length=512, devices: Union[float, int, List[int]] = None,\n            transform=None, **kwargs):\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    def build_vocabs(self, dataset, logger=None, transformer=None):\n        self.vocabs['rel_2nd'] = rel_2nd = Vocab(pad_token=self.config.pad_rel, unk_token=self.config.pad_rel)\n        if self.config.joint:\n            self.vocabs['rel'] = rel_2nd\n        super().build_vocabs(dataset, logger, transformer)\n        self.config.n_rels_2nd = len(rel_2nd)\n\n    def create_model(self, pretrained_embed, transformer):\n        return BiaffineSecondaryModel(self.config, pretrained_embed, transformer, self.transformer_tokenizer)\n\n    def compute_loss(self, arc_scores, rel_scores, arcs, rels, mask, criterion, batch=None):\n        arc_scores_1st, arc_scores_2nd, rel_scores_1st, rel_scores_2nd = self.unpack_scores(arc_scores, rel_scores)\n        loss_1st = super().compute_loss(arc_scores_1st, rel_scores_1st, arcs, rels, mask, criterion[0], batch)\n        mask = self.compute_mask(arc_scores_2nd, batch, mask)\n        # noinspection PyCallByClass\n        loss_2st = BiaffineSemanticDependencyParser.compute_loss(self, arc_scores_2nd, rel_scores_2nd,\n                                                                 batch['arc_2nd'], batch['rel_2nd_id'], mask,\n                                                                 criterion[1], batch)\n        return loss_1st + loss_2st\n\n    @staticmethod\n    def compute_mask(arc_scores_2nd, batch, mask_1st):\n        mask = batch.get('mask_2nd', None)\n        if mask is None:\n            batch['mask_2nd'] = mask = BiaffineSemanticDependencyParser.convert_to_3d_mask(arc_scores_2nd, mask_1st)\n        return mask\n\n    def unpack_scores(self, arc_scores, rel_scores):\n        arc_scores_1st, arc_scores_2nd = arc_scores\n        rel_scores_1st, rel_scores_2nd = rel_scores\n        return arc_scores_1st, arc_scores_2nd, rel_scores_1st, rel_scores_2nd\n\n    def get_pad_dict(self):\n        d = super(BiaffineSecondaryParser, self).get_pad_dict()\n        d.update({'arc_2nd': False})\n        return d\n\n    def decode(self, arc_scores, rel_scores, mask, batch=None, predicting=None):\n        output_1st, output_2nd = batch.get('outputs', (None, None))\n        if output_1st is None:\n            arc_scores_1st, arc_scores_2nd, rel_scores_1st, rel_scores_2nd = self.unpack_scores(arc_scores, rel_scores)\n            output_1st = super().decode(arc_scores_1st, rel_scores_1st, mask)\n            mask = self.compute_mask(arc_scores_2nd, batch, mask)\n            # noinspection PyCallByClass\n            output_2nd = BiaffineSemanticDependencyParser.decode(self, arc_scores_2nd, rel_scores_2nd, mask, batch)\n            if self.config.get('no_cycle'):\n                assert predicting, 'No cycle constraint for evaluation is not implemented yet. If you are ' \\\n                                   'interested, welcome to submit a pull request.'\n                root_rel_idx = self.vocabs['rel'].token_to_idx.get(self.config.get('root', None), None)\n                arc_pred_1st, rel_pred_1st, arc_pred_2nd, rel_pred_2nd = *output_1st, *output_2nd\n                arc_scores_2nd = arc_scores_2nd.transpose(1, 2).cpu().detach().numpy()\n                arc_pred_2nd = arc_pred_2nd.cpu().detach().numpy()\n                rel_pred_2nd = rel_pred_2nd.cpu().detach().numpy()\n                trees = arc_pred_1st.cpu().detach().numpy()\n                graphs = []\n                for i, (arc_scores, arc_preds, rel_preds, tree, tokens) in enumerate(\n                        zip(arc_scores_2nd, arc_pred_2nd, rel_pred_2nd, trees, batch['token'])):\n                    sent_len = len(tokens)\n                    graph = add_secondary_arcs_by_preds(arc_scores, arc_preds[:sent_len, :sent_len], rel_preds,\n                                                        tree[:sent_len], root_rel_idx)\n                    graphs.append(graph[1:])  # Remove root\n                    # if not predicting:\n                    #     # Write back to torch Tensor\n                    #     for d, hr in zip(graph):\n                    #         pass\n                output_2nd = None, graphs\n\n        return tuple(zip(output_1st, output_2nd))\n\n    def update_metric(self, arc_preds, rel_preds, arcs, rels, mask, puncts, metric, batch=None):\n        super().update_metric(arc_preds[0], rel_preds[0], arcs, rels, mask, puncts, metric['1st'], batch)\n        puncts = BiaffineSemanticDependencyParser.convert_to_3d_puncts(puncts, batch['mask_2nd'])\n        # noinspection PyCallByClass\n        BiaffineSemanticDependencyParser.update_metric(self, arc_preds[1], rel_preds[1], batch['arc_2nd'],\n                                                       batch['rel_2nd_id'], batch['mask_2nd'], puncts, metric['2nd'],\n                                                       batch)\n\n    def build_metric(self, **kwargs):\n        # noinspection PyCallByClass\n        return MetricDict({'1st': super().build_metric(**kwargs),\n                           '2nd': BiaffineSemanticDependencyParser.build_metric(self, **kwargs)})\n\n    def collect_outputs_extend(self, predictions: list, arc_preds, rel_preds, lens, mask):\n        predictions.extend(rel_preds[1])\n\n    def predictions_to_human(self, predictions, outputs, data, use_pos, conll=True):\n        rel_vocab = self.vocabs['rel'].idx_to_token\n        for d, graph in zip(data, predictions):\n            sent = CoNLLSentence()\n            for idx, (cell, hrs) in enumerate(zip(d, graph)):\n                if use_pos:\n                    token, pos = cell\n                else:\n                    token, pos = cell, None\n                head = hrs[0][0]\n                deprel = rel_vocab[hrs[0][1]]\n                deps = [(h, rel_vocab[r]) for h, r in hrs[1:]]\n                sent.append(CoNLLUWord(idx + 1, token, upos=pos, head=head, deprel=deprel, deps=deps))\n            outputs.append(sent)\n"
  },
  {
    "path": "hanlp/components/parsers/biaffine/biaffine_dep.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-08 20:51\nimport os\nfrom collections import Counter\nfrom typing import Union, Any, List\n\nfrom hanlp.layers.transformers.pt_imports import PreTrainedTokenizer, AutoModel_, AutoTokenizer_\nimport torch\nfrom hanlp.utils.torch_util import lengths_to_mask\nfrom torch import nn\nfrom torch.optim import Adam\nfrom torch.optim.lr_scheduler import ExponentialLR\nfrom torch.utils.data import DataLoader\nfrom hanlp_common.constant import ROOT, UNK, IDX\nfrom hanlp.common.dataset import PadSequenceDataLoader\nfrom hanlp.common.structure import History\nfrom hanlp.common.torch_component import TorchComponent\nfrom hanlp.common.transform import LowerCase, FieldLength, PunctuationMask\nfrom hanlp.common.vocab import Vocab\nfrom hanlp.components.parsers.alg import decode_dep\nfrom hanlp.components.parsers.biaffine.biaffine_model import BiaffineDependencyModel\nfrom hanlp_common.conll import CoNLLWord, CoNLLSentence\nfrom hanlp.datasets.parsing.loaders.conll_dataset import CoNLLParsingDataset, append_bos\nfrom hanlp.layers.embeddings.util import index_word2vec_with_vocab\nfrom hanlp.layers.transformers.utils import build_optimizer_scheduler_with_transformer\nfrom hanlp.metrics.parsing.attachmentscore import AttachmentScore\nfrom hanlp.transform.transformer_tokenizer import TransformerSequenceTokenizer\nfrom hanlp.utils.time_util import CountdownTimer\nfrom hanlp_common.util import isdebugging, merge_locals_kwargs, merge_dict, reorder\n\n\nclass BiaffineDependencyParser(TorchComponent):\n    def __init__(self) -> None:\n        \"\"\"Biaffine dependency parsing (:cite:`dozat:17a`).\n        \"\"\"\n        super().__init__()\n        self.model: BiaffineDependencyModel = None\n        self.transformer_tokenizer: PreTrainedTokenizer = None\n\n    def predict(self, data: Any, batch_size=None, batch_max_tokens=None, conll=True, **kwargs):\n        if not data:\n            return []\n        use_pos = self.use_pos\n        flat = self.input_is_flat(data, use_pos)\n        if flat:\n            data = [data]\n        samples = self.build_samples(data, use_pos)\n        if not batch_max_tokens:\n            batch_max_tokens = self.config.get('batch_max_tokens', None)\n        if not batch_size:\n            batch_size = self.config.batch_size\n        dataloader = self.build_dataloader(samples,\n                                           device=self.devices[0], shuffle=False,\n                                           **merge_dict(self.config,\n                                                        batch_size=batch_size,\n                                                        batch_max_tokens=batch_max_tokens,\n                                                        overwrite=True,\n                                                        **kwargs))\n        predictions, build_data, data, order = self.before_outputs(data)\n        for batch in dataloader:\n            arc_scores, rel_scores, mask, puncts = self.feed_batch(batch)\n            self.collect_outputs(arc_scores, rel_scores, mask, batch, predictions, order, data, use_pos,\n                                 build_data)\n        outputs = self.post_outputs(predictions, data, order, use_pos, build_data, conll=conll)\n        if flat:\n            return outputs[0]\n        return outputs\n\n    def build_samples(self, data, use_pos=None):\n        samples = []\n        pos_key = 'CPOS' if 'CPOS' in self.vocabs else 'UPOS'\n        for idx, each in enumerate(data):\n            sample = {IDX: idx}\n            if use_pos:\n                token, pos = zip(*each)\n                sample.update({'FORM': list(token), pos_key: list(pos)})\n            else:\n                token = each\n                sample.update({'FORM': list(token)})\n            samples.append(sample)\n        return samples\n\n    def input_is_flat(self, data, use_pos=None):\n        if use_pos is None:\n            use_pos = 'CPOS' in self.vocabs\n        if use_pos:\n            flat = isinstance(data[0], (list, tuple)) and isinstance(data[0][0], str)\n        else:\n            flat = isinstance(data[0], str)\n        return flat\n\n    def before_outputs(self, data):\n        predictions, order = [], []\n        build_data = data is None\n        if build_data:\n            data = []\n        return predictions, build_data, data, order\n\n    def post_outputs(self, predictions, data, order, use_pos, build_data, conll=True):\n        predictions = reorder(predictions, order)\n        if build_data:\n            data = reorder(data, order)\n        outputs = []\n        self.predictions_to_human(predictions, outputs, data, use_pos, conll=conll)\n        return outputs\n\n    def predictions_to_human(self, predictions, outputs, data, use_pos, conll=True):\n        if conll:\n            for d, (arcs, rels) in zip(data, predictions):\n                sent = CoNLLSentence()\n                for idx, (cell, a, r) in enumerate(zip(d, arcs, rels)):\n                    if use_pos:\n                        token, pos = cell\n                    else:\n                        token, pos = cell, None\n                    sent.append(CoNLLWord(idx + 1, token, cpos=pos, head=a, deprel=self.vocabs['rel'][r]))\n                outputs.append(sent)\n        else:\n            for d, (arcs, rels) in zip(data, predictions):\n                sent = []\n                for idx, (a, r) in enumerate(zip(arcs, rels)):\n                    sent.append((a, self.vocabs['rel'][r]))\n                outputs.append(sent)\n\n    def collect_outputs(self, arc_scores, rel_scores, mask, batch, predictions, order, data, use_pos,\n                        build_data):\n        lens = [len(token) - 1 for token in batch['token']]\n        arc_preds, rel_preds = self.decode(arc_scores, rel_scores, mask, batch)\n        self.collect_outputs_extend(predictions, arc_preds, rel_preds, lens, mask)\n        order.extend(batch[IDX])\n        if build_data:\n            if use_pos:\n                data.extend(zip(batch['FORM'], batch['CPOS']))\n            else:\n                data.extend(batch['FORM'])\n\n    def collect_outputs_extend(self, predictions: list, arc_preds, rel_preds, lens, mask):\n        predictions.extend(zip([seq.tolist() for seq in arc_preds[mask].split(lens)],\n                               [seq.tolist() for seq in rel_preds[mask].split(lens)]))\n\n    @property\n    def use_pos(self):\n        return self.config.get('feat', None) == 'pos'\n\n    def fit(self, trn_data, dev_data, save_dir,\n            feat=None,\n            n_embed=100,\n            pretrained_embed=None,\n            transformer=None,\n            average_subwords=False,\n            word_dropout=0.2,\n            transformer_hidden_dropout=None,\n            layer_dropout=0,\n            scalar_mix: int = None,\n            embed_dropout=.33,\n            n_lstm_hidden=400,\n            n_lstm_layers=3,\n            hidden_dropout=.33,\n            n_mlp_arc=500,\n            n_mlp_rel=100,\n            mlp_dropout=.33,\n            lr=2e-3,\n            transformer_lr=5e-5,\n            mu=.9,\n            nu=.9,\n            epsilon=1e-12,\n            grad_norm=5.0,\n            decay=.75,\n            decay_steps=5000,\n            weight_decay=0,\n            warmup_steps=0.1,\n            separate_optimizer=False,\n            patience=100,\n            lowercase=False,\n            epochs=50000,\n            tree=False,\n            proj=False,\n            punct=False,\n            min_freq=2,\n            logger=None,\n            verbose=True,\n            unk=UNK,\n            max_sequence_length=512,\n            batch_size=None,\n            sampler_builder=None,\n            gradient_accumulation=1,\n            devices: Union[float, int, List[int]] = None,\n            transform=None,\n            secondary_encoder=None,\n            **kwargs):\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    def execute_training_loop(self, trn, dev, devices, epochs, logger, patience, save_dir, optimizer,\n                              gradient_accumulation, **kwargs):\n        optimizer, scheduler, transformer_optimizer, transformer_scheduler = optimizer\n        criterion = self.build_criterion()\n        best_e, best_metric = 0, self.build_metric()\n        timer = CountdownTimer(epochs)\n        history = History()\n        ratio_width = len(f'{len(trn) // gradient_accumulation}/{len(trn) // gradient_accumulation}')\n        for epoch in range(1, epochs + 1):\n            # train one epoch and update the parameters\n            logger.info(f\"[yellow]Epoch {epoch} / {epochs}:[/yellow]\")\n            self.fit_dataloader(trn, optimizer, scheduler, criterion, epoch, logger, history,\n                                transformer_optimizer, transformer_scheduler,\n                                gradient_accumulation=gradient_accumulation)\n            loss, dev_metric = self.evaluate_dataloader(dev, criterion, ratio_width=ratio_width, logger=logger)\n            timer.update()\n            # logger.info(f\"{'Dev' + ' ' * ratio_width} loss: {loss:.4f} {dev_metric}\")\n            # save the model if it is the best so far\n            report = f\"{timer.elapsed_human} / {timer.total_time_human} ETA: {timer.eta_human}\"\n            if dev_metric > best_metric:\n                best_e, best_metric = epoch, dev_metric\n                self.save_weights(save_dir)\n                report += ' ([red]saved[/red])'\n            else:\n                if patience != epochs:\n                    report += f' ({epoch - best_e}/{patience})'\n                else:\n                    report += f' ({epoch - best_e})'\n            logger.info(report)\n            if patience is not None and epoch - best_e >= patience:\n                logger.info(f'LAS has stopped improving for {patience} epochs, early stop.')\n                break\n        timer.stop()\n        if not best_e:\n            self.save_weights(save_dir)\n        elif best_e != epoch:\n            self.load_weights(save_dir)\n        logger.info(f\"Max score of dev is {best_metric.score:.2%} at epoch {best_e}\")\n        logger.info(f\"Average time of each epoch is {timer.elapsed_average_human}\")\n        logger.info(f\"{timer.elapsed_human} elapsed\")\n\n    def build_optimizer(self, epochs, trn, gradient_accumulation, **kwargs):\n        config = self.config\n        model = self.model\n        if isinstance(model, nn.DataParallel):\n            model = model.module\n        if self.config.transformer:\n            transformer = model.encoder.transformer\n            optimizer = Adam(set(model.parameters()) - set(transformer.parameters()),\n                             config.lr,\n                             (config.mu, config.nu),\n                             config.epsilon)\n            if self.config.transformer_lr:\n                num_training_steps = len(trn) * epochs // gradient_accumulation\n                if self.config.separate_optimizer:\n                    transformer_optimizer, transformer_scheduler = \\\n                        build_optimizer_scheduler_with_transformer(transformer,\n                                                                   transformer,\n                                                                   config.transformer_lr,\n                                                                   config.transformer_lr,\n                                                                   num_training_steps,\n                                                                   config.warmup_steps,\n                                                                   config.weight_decay,\n                                                                   adam_epsilon=1e-8)\n                else:\n                    optimizer, scheduler = build_optimizer_scheduler_with_transformer(model,\n                                                                                      transformer,\n                                                                                      config.lr,\n                                                                                      config.transformer_lr,\n                                                                                      num_training_steps,\n                                                                                      config.warmup_steps,\n                                                                                      config.weight_decay,\n                                                                                      adam_epsilon=1e-8)\n                    transformer_optimizer, transformer_scheduler = None, None\n            else:\n                transformer.requires_grad_(False)\n                transformer_optimizer, transformer_scheduler = None, None\n        else:\n            optimizer = Adam(model.parameters(),\n                             config.lr,\n                             (config.mu, config.nu),\n                             config.epsilon)\n            transformer_optimizer, transformer_scheduler = None, None\n        if self.config.separate_optimizer:\n            scheduler = ExponentialLR(optimizer, config.decay ** (1 / config.decay_steps))\n        # noinspection PyUnboundLocalVariable\n        return optimizer, scheduler, transformer_optimizer, transformer_scheduler\n\n    def build_transformer_tokenizer(self):\n        transformer = self.config.transformer\n        if transformer:\n            transformer_tokenizer: PreTrainedTokenizer = AutoTokenizer_.from_pretrained(transformer, use_fast=True)\n        else:\n            transformer_tokenizer = None\n        self.transformer_tokenizer = transformer_tokenizer\n        return transformer_tokenizer\n\n    # noinspection PyMethodOverriding\n    def build_dataloader(self,\n                         data,\n                         shuffle,\n                         device,\n                         training=False,\n                         logger=None,\n                         gradient_accumulation=1,\n                         sampler_builder=None,\n                         batch_size=None,\n                         **kwargs) -> DataLoader:\n        dataset = self.build_dataset(data)\n        if self.vocabs.mutable:\n            self.build_vocabs(dataset, logger, self.config.transformer)\n        transformer_tokenizer = self.transformer_tokenizer\n        if transformer_tokenizer:\n            dataset.transform.append(self.build_tokenizer_transform())\n        dataset.append_transform(FieldLength('token', 'sent_length'))\n        if isinstance(data, str):\n            dataset.purge_cache()\n        if len(dataset) > 1000 and isinstance(data, str):\n            timer = CountdownTimer(len(dataset))\n            self.cache_dataset(dataset, timer, training, logger)\n        if self.config.transformer:\n            lens = [len(sample['input_ids']) for sample in dataset]\n        else:\n            lens = [sample['sent_length'] for sample in dataset]\n        if sampler_builder:\n            sampler = sampler_builder.build(lens, shuffle, gradient_accumulation)\n        else:\n            sampler = None\n        loader = PadSequenceDataLoader(dataset=dataset,\n                                       batch_sampler=sampler,\n                                       batch_size=batch_size,\n                                       pad=self.get_pad_dict(),\n                                       device=device,\n                                       vocabs=self.vocabs)\n        return loader\n\n    def cache_dataset(self, dataset, timer, training=False, logger=None):\n        for each in dataset:\n            timer.log('Preprocessing and caching samples [blink][yellow]...[/yellow][/blink]')\n\n    def get_pad_dict(self):\n        return {'arc': 0}\n\n    def build_dataset(self, data, bos_transform=None):\n        if not bos_transform:\n            bos_transform = append_bos\n        transform = [bos_transform]\n        if self.config.get('transform', None):\n            transform.append(self.config.transform)\n        if self.config.get('lowercase', False):\n            transform.append(LowerCase('token'))\n        transform.append(self.vocabs)\n        if not self.config.punct:\n            transform.append(PunctuationMask('token', 'punct_mask'))\n        return CoNLLParsingDataset(data, transform=transform)\n\n    def build_tokenizer_transform(self):\n        return TransformerSequenceTokenizer(self.transformer_tokenizer, 'token', '',\n                                            ret_token_span=True, cls_is_bos=True,\n                                            max_seq_length=self.config.get('max_sequence_length',\n                                                                           512),\n                                            truncate_long_sequences=False)\n\n    def build_vocabs(self, dataset, logger=None, transformer=None):\n        rel_vocab = self.vocabs.get('rel', None)\n        if rel_vocab is None:\n            rel_vocab = Vocab(unk_token=None, pad_token=self.config.get('pad_rel', None))\n            self.vocabs.put(rel=rel_vocab)\n        if self.config.get('feat', None) == 'pos' or self.config.get('use_pos', False):\n            self.vocabs['pos'] = Vocab(unk_token=None, pad_token=None)\n\n        timer = CountdownTimer(len(dataset))\n        if transformer:\n            token_vocab = None\n        else:\n            token_vocab = Vocab()\n            self.vocabs.token = token_vocab\n            unk = self.config.get('unk', None)\n            if unk is not None:\n                token_vocab.unk_token = unk\n        if token_vocab and self.config.get('min_freq', None):\n            counter = Counter()\n            for sample in dataset:\n                for form in sample['token']:\n                    counter[form] += 1\n            reserved_token = [token_vocab.pad_token, token_vocab.unk_token]\n            if ROOT in token_vocab:\n                reserved_token.append(ROOT)\n            freq_words = reserved_token + [token for token, freq in counter.items() if\n                                           freq >= self.config.min_freq]\n            token_vocab.token_to_idx.clear()\n            for word in freq_words:\n                token_vocab(word)\n        else:\n            for i, sample in enumerate(dataset):\n                timer.log('vocab building [blink][yellow]...[/yellow][/blink]', ratio_percentage=True)\n        rel_vocab.set_unk_as_safe_unk()  # Some relation in dev set is OOV\n        self.vocabs.lock()\n        self.vocabs.summary(logger=logger)\n        if token_vocab:\n            self.config.n_words = len(self.vocabs['token'])\n        if 'pos' in self.vocabs:\n            self.config.n_feats = len(self.vocabs['pos'])\n            self.vocabs['pos'].set_unk_as_safe_unk()\n        self.config.n_rels = len(self.vocabs['rel'])\n        if token_vocab:\n            self.config.pad_index = self.vocabs['token'].pad_idx\n            self.config.unk_index = self.vocabs['token'].unk_idx\n\n    def build_model(self, training=True, **kwargs) -> torch.nn.Module:\n        pretrained_embed, transformer = self.build_embeddings(training=training)\n        if pretrained_embed is not None:\n            self.config.n_embed = pretrained_embed.size(-1)\n        model = self.create_model(pretrained_embed, transformer)\n        return model\n\n    def create_model(self, pretrained_embed, transformer):\n        return BiaffineDependencyModel(self.config,\n                                       pretrained_embed,\n                                       transformer,\n                                       self.transformer_tokenizer)\n\n    def build_embeddings(self, training=True):\n        pretrained_embed = None\n        if self.config.get('pretrained_embed', None):\n            pretrained_embed = index_word2vec_with_vocab(self.config.pretrained_embed, self.vocabs['token'],\n                                                         init='zeros', normalize=True)\n        transformer = self.config.transformer\n        if transformer:\n            transformer = AutoModel_.from_pretrained(transformer, training=training)\n        return pretrained_embed, transformer\n\n    # noinspection PyMethodOverriding\n    def fit_dataloader(self,\n                       trn,\n                       optimizer,\n                       scheduler,\n                       criterion,\n                       epoch,\n                       logger,\n                       history: History,\n                       transformer_optimizer=None,\n                       transformer_scheduler=None,\n                       gradient_accumulation=1,\n                       **kwargs):\n        self.model.train()\n\n        timer = CountdownTimer(history.num_training_steps(len(trn), gradient_accumulation))\n        metric = self.build_metric(training=True)\n        total_loss = 0\n        for idx, batch in enumerate(trn):\n            arc_scores, rel_scores, mask, puncts = self.feed_batch(batch)\n            arcs, rels = batch['arc'], batch['rel_id']\n            loss = self.compute_loss(arc_scores, rel_scores, arcs, rels, mask, criterion, batch)\n            if gradient_accumulation > 1:\n                loss /= gradient_accumulation\n            loss.backward()\n            total_loss += loss.item()\n            arc_preds, rel_preds = self.decode(arc_scores, rel_scores, mask, batch)\n            self.update_metric(arc_preds, rel_preds, arcs, rels, mask, puncts, metric, batch)\n            if history.step(gradient_accumulation):\n                self._step(optimizer, scheduler, transformer_optimizer, transformer_scheduler)\n                report = self._report(total_loss / (timer.current + 1), metric)\n                timer.log(report, ratio_percentage=False, logger=logger)\n            del loss\n\n    def _step(self, optimizer, scheduler, transformer_optimizer, transformer_scheduler):\n        if self.config.get('grad_norm', None):\n            nn.utils.clip_grad_norm_(self.model.parameters(),\n                                     self.config.grad_norm)\n        optimizer.step()\n        optimizer.zero_grad()\n        scheduler.step()\n        if self.config.transformer and self.config.transformer_lr and transformer_optimizer:\n            transformer_optimizer.step()\n            transformer_optimizer.zero_grad()\n            transformer_scheduler.step()\n\n    def feed_batch(self, batch):\n        words, feats, lens, puncts = batch.get('token_id', None), batch.get('pos_id', None), batch['sent_length'], \\\n                                     batch.get('punct_mask', None)\n        mask = lengths_to_mask(lens)\n        arc_scores, rel_scores = self.model(words=words, feats=feats, mask=mask, batch=batch, **batch)\n        # ignore the first token of each sentence\n        # RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation\n        if self.model.training:\n            mask = mask.clone()\n        mask[:, 0] = 0\n        return arc_scores, rel_scores, mask, puncts\n\n    def _report(self, loss, metric: AttachmentScore):\n        return f'loss: {loss:.4f} {metric}'\n\n    def compute_loss(self, arc_scores, rel_scores, arcs, rels, mask, criterion, batch=None):\n        arc_scores, arcs = arc_scores[mask], arcs[mask]\n        rel_scores, rels = rel_scores[mask], rels[mask]\n        rel_scores = rel_scores[torch.arange(len(arcs)), arcs]\n        arc_loss = criterion(arc_scores, arcs)\n        rel_loss = criterion(rel_scores, rels)\n        loss = arc_loss + rel_loss\n\n        return loss\n\n    # noinspection PyUnboundLocalVariable\n    @torch.no_grad()\n    def evaluate_dataloader(self, loader: PadSequenceDataLoader, criterion, logger=None, filename=None, output=False,\n                            ratio_width=None,\n                            metric=None,\n                            **kwargs):\n        self.model.eval()\n\n        loss = 0\n        if not metric:\n            metric = self.build_metric()\n        if output:\n            fp = open(output, 'w')\n            predictions, build_data, data, order = self.before_outputs(None)\n\n        timer = CountdownTimer(len(loader))\n        use_pos = self.use_pos\n        for batch in loader:\n            arc_scores, rel_scores, mask, puncts = self.feed_batch(batch)\n            if output:\n                self.collect_outputs(arc_scores, rel_scores, mask, batch, predictions, order, data, use_pos,\n                                     build_data)\n            arcs, rels = batch['arc'], batch['rel_id']\n            loss += self.compute_loss(arc_scores, rel_scores, arcs, rels, mask, criterion, batch).item()\n            arc_preds, rel_preds = self.decode(arc_scores, rel_scores, mask, batch)\n            self.update_metric(arc_preds, rel_preds, arcs, rels, mask, puncts, metric, batch)\n            report = self._report(loss / (timer.current + 1), metric)\n            if filename:\n                report = f'{os.path.basename(filename)} ' + report\n            timer.log(report, ratio_percentage=False, logger=logger, ratio_width=ratio_width)\n        loss /= len(loader)\n        if output:\n            outputs = self.post_outputs(predictions, data, order, use_pos, build_data)\n            for each in outputs:\n                fp.write(f'{each}\\n\\n')\n            fp.close()\n            logger.info(f'Predictions saved in [underline][yellow]{output}[/yellow][/underline]')\n\n        return loss, metric\n\n    def update_metric(self, arc_preds, rel_preds, arcs, rels, mask, puncts, metric, batch=None):\n        # ignore all punctuation if not specified\n        if not self.config.punct:\n            mask &= puncts\n        metric(arc_preds, rel_preds, arcs, rels, mask)\n\n    def decode(self, arc_scores, rel_scores, mask, batch=None):\n        tree, proj = self.config.tree, self.config.get('proj', False)\n        if tree:\n            arc_preds = decode_dep(arc_scores, mask, tree, proj)\n        else:\n            arc_preds = arc_scores.argmax(-1)\n        rel_preds = rel_scores.argmax(-1)\n        rel_preds = rel_preds.gather(-1, arc_preds.unsqueeze(-1)).squeeze(-1)\n\n        return arc_preds, rel_preds\n\n    def build_criterion(self, **kwargs):\n        criterion = nn.CrossEntropyLoss()\n        return criterion\n\n    def build_metric(self, **kwargs):\n        return AttachmentScore()\n\n    def on_config_ready(self, **kwargs):\n        self.build_transformer_tokenizer()  # We have to build tokenizer before building the dataloader and model\n        self.config.patience = min(self.config.patience, self.config.epochs)\n\n    def prediction_to_head_rel(self, arcs: torch.LongTensor, rels: torch.LongTensor, batch: dict):\n        arcs = arcs[:, 1:]  # Skip the ROOT\n        rels = rels[:, 1:]\n        arcs = arcs.tolist()\n        rels = rels.tolist()\n        vocab = self.vocabs['rel'].idx_to_token\n        for arcs_per_sent, rels_per_sent, tokens in zip(arcs, rels, batch['token']):\n            tokens = tokens[1:]\n            sent_len = len(tokens)\n            result = list(zip(arcs_per_sent[:sent_len], [vocab[r] for r in rels_per_sent[:sent_len]]))\n            yield result\n"
  },
  {
    "path": "hanlp/components/parsers/biaffine/biaffine_model.py",
    "content": "# -*- coding: utf-8 -*-\nfrom typing import Any, Tuple\n\nimport torch\nimport torch.nn as nn\nfrom torch.nn.utils.rnn import (pack_padded_sequence, pad_packed_sequence,\n                                pad_sequence)\n\nfrom hanlp.components.parsers.biaffine.biaffine import Biaffine\nfrom hanlp.components.parsers.biaffine.mlp import MLP\nfrom hanlp.components.parsers.biaffine.variationalbilstm import VariationalLSTM\nfrom hanlp.layers.dropout import IndependentDropout, SharedDropout, WordDropout\nfrom hanlp.layers.transformers.encoder import TransformerEncoder\nfrom hanlp.layers.transformers.pt_imports import PreTrainedModel, PreTrainedTokenizer\nfrom hanlp.layers.transformers.utils import transformer_encode\n\n\nclass EncoderWithContextualLayer(nn.Module):\n    def __init__(self,\n                 config,\n                 pretrained_embed: torch.Tensor = None,\n                 transformer: PreTrainedModel = None,\n                 transformer_tokenizer: PreTrainedTokenizer = None,\n                 ):\n        super(EncoderWithContextualLayer, self).__init__()\n\n        self.secondary_encoder = config.get('secondary_encoder', None)\n        self.config = config\n\n        if not transformer:\n            self.pad_index = config.pad_index\n            self.unk_index = config.unk_index\n            if config.word_dropout:\n                oov = self.unk_index\n                excludes = [self.pad_index]\n                self.word_dropout = WordDropout(p=config.word_dropout, oov_token=oov, exclude_tokens=excludes)\n            else:\n                self.word_dropout = None\n        if transformer:\n            input_size = 0\n            if self.config.transformer_lr:\n                hidden_size = transformer.config.hidden_size\n            else:\n                input_size = transformer.config.hidden_size\n                hidden_size = config.n_lstm_hidden * 2\n            if config.feat == 'pos':\n                self.feat_embed = nn.Embedding(num_embeddings=config.n_feats,\n                                               embedding_dim=config.n_embed)\n                self.embed_dropout = IndependentDropout(p=config.embed_dropout)\n                if self.config.transformer_lr:\n                    hidden_size += config.n_embed\n                else:\n                    input_size += config.n_embed\n            if not self.config.transformer_lr:\n                self.lstm = VariationalLSTM(input_size=input_size,\n                                            hidden_size=config.n_lstm_hidden,\n                                            num_layers=config.n_lstm_layers,\n                                            dropout=config.hidden_dropout, bidirectional=True)\n        else:\n            # the embedding layer\n            input_size = config.n_embed\n            self.word_embed = nn.Embedding(num_embeddings=config.n_words,\n                                           embedding_dim=config.n_embed)\n            if pretrained_embed is not None:\n                if not isinstance(pretrained_embed, torch.Tensor):\n                    pretrained_embed = torch.Tensor(pretrained_embed)\n                self.pretrained = nn.Embedding.from_pretrained(pretrained_embed)\n                nn.init.zeros_(self.word_embed.weight)\n            if config.feat == 'pos':\n                self.feat_embed = nn.Embedding(num_embeddings=config.n_feats,\n                                               embedding_dim=config.n_embed)\n                self.embed_dropout = IndependentDropout(p=config.embed_dropout)\n                input_size += config.n_embed\n\n            # the word-lstm layer\n            hidden_size = config.n_lstm_hidden * 2\n            self.lstm = VariationalLSTM(input_size=input_size,\n                                        hidden_size=config.n_lstm_hidden,\n                                        num_layers=config.n_lstm_layers,\n                                        dropout=config.hidden_dropout, bidirectional=True)\n        self.hidden_size = hidden_size\n        self.hidden_dropout = SharedDropout(p=config.hidden_dropout)\n        if transformer:\n            transformer = TransformerEncoder(transformer, transformer_tokenizer, config.average_subwords,\n                                             word_dropout=config.word_dropout,\n                                             max_sequence_length=config.max_sequence_length)\n        self.transformer = transformer\n\n    def forward(self, words, feats, input_ids, token_span, mask, lens):\n        if mask is None:\n            # get the mask and lengths of given batch\n            mask = words.ne(self.pad_index)\n        if lens is None:\n            lens = mask.sum(dim=1)\n        batch_size, seq_len = mask.shape\n        if self.config.transformer:\n            # trans_embed = self.run_transformer(input_ids, token_span=token_span)\n            trans_embed = self.transformer.forward(input_ids, token_span=token_span)\n            if hasattr(self, 'feat_embed'):\n                feat_embed = self.feat_embed(feats)\n                trans_embed, feat_embed = self.embed_dropout(trans_embed, feat_embed)\n                embed = torch.cat((trans_embed, feat_embed), dim=-1)\n            else:\n                embed = trans_embed\n            if hasattr(self, 'lstm'):\n                x = self.run_rnn(embed, lens, seq_len)\n            else:\n                x = embed\n            if self.secondary_encoder:\n                x = self.secondary_encoder(x, mask)\n            x = self.hidden_dropout(x)\n        else:\n            if self.word_dropout:\n                words = self.word_dropout(words)\n            # set the indices larger than num_embeddings to unk_index\n            ext_mask = words.ge(self.word_embed.num_embeddings)\n            ext_words = words.masked_fill(ext_mask, self.unk_index)\n\n            # get outputs from embedding layers\n            word_embed = self.word_embed(ext_words)\n            if hasattr(self, 'pretrained'):\n                word_embed += self.pretrained(words)\n            if self.config.feat == 'char':\n                feat_embed = self.feat_embed(feats[mask])\n                feat_embed = pad_sequence(feat_embed.split(lens.tolist()), True)\n            elif self.config.feat == 'bert':\n                feat_embed = self.feat_embed(*feats)\n            elif hasattr(self, 'feat_embed'):\n                feat_embed = self.feat_embed(feats)\n            else:\n                feat_embed = None\n            if feat_embed is not None:\n                word_embed, feat_embed = self.embed_dropout(word_embed, feat_embed)\n                # concatenate the word and feat representations\n                embed = torch.cat((word_embed, feat_embed), dim=-1)\n            else:\n                embed = word_embed\n\n            x = self.run_rnn(embed, lens, seq_len)\n            x = self.hidden_dropout(x)\n        return x, mask\n\n    def run_rnn(self, embed, lens, seq_len):\n        x = pack_padded_sequence(embed, lens, True, False)\n        x, _ = self.lstm(x)\n        x, _ = pad_packed_sequence(x, True, total_length=seq_len)\n        return x\n\n    def run_transformer(self, input_ids, token_span):\n        return transformer_encode(self.transformer, input_ids, None, None, token_span,\n                                  average_subwords=self.config.average_subwords)\n\n\nclass BiaffineDecoder(nn.Module):\n    def __init__(self, hidden_size, n_mlp_arc, n_mlp_rel, mlp_dropout, n_rels, arc_dropout=None,\n                 rel_dropout=None) -> None:\n        super().__init__()\n        # the MLP layers\n        self.mlp_arc_h = MLP(hidden_size,\n                             n_mlp_arc,\n                             dropout=arc_dropout or mlp_dropout)\n        self.mlp_arc_d = MLP(hidden_size,\n                             n_mlp_arc,\n                             dropout=arc_dropout or mlp_dropout)\n        self.mlp_rel_h = MLP(hidden_size,\n                             n_mlp_rel,\n                             dropout=rel_dropout or mlp_dropout)\n        self.mlp_rel_d = MLP(hidden_size,\n                             n_mlp_rel,\n                             dropout=rel_dropout or mlp_dropout)\n\n        # the Biaffine layers\n        self.arc_attn = Biaffine(n_in=n_mlp_arc,\n                                 bias_x=True,\n                                 bias_y=False)\n        self.rel_attn = Biaffine(n_in=n_mlp_rel,\n                                 n_out=n_rels,\n                                 bias_x=True,\n                                 bias_y=True)\n\n    def forward(self, x, mask=None, **kwargs: Any) -> Tuple[torch.Tensor, torch.Tensor]:\n        arc_d, arc_h, rel_d, rel_h = self.apply_mlps(x)\n\n        s_arc, s_rel = self.decode(arc_d, arc_h, rel_d, rel_h, mask, self.arc_attn, self.rel_attn)\n\n        return s_arc, s_rel\n\n    @staticmethod\n    def decode(arc_d, arc_h, rel_d, rel_h, mask, arc_attn, rel_attn):\n        # get arc and rel scores from the bilinear attention\n        # [batch_size, seq_len, seq_len]\n        s_arc = arc_attn(arc_d, arc_h)\n        # [batch_size, seq_len, seq_len, n_rels]\n        s_rel = rel_attn(rel_d, rel_h).permute(0, 2, 3, 1)\n        if mask is not None:\n            # set the scores that exceed the length of each sentence to -inf\n            s_arc.masked_fill_(~mask.unsqueeze(1), float('-inf'))\n        return s_arc, s_rel\n\n    def apply_mlps(self, x):\n        # apply MLPs to the hidden states\n        arc_d = self.mlp_arc_d(x)\n        arc_h = self.mlp_arc_h(x)\n        rel_d = self.mlp_rel_d(x)\n        rel_h = self.mlp_rel_h(x)\n        return arc_d, arc_h, rel_d, rel_h\n\n\nclass BiaffineDependencyModel(nn.Module):\n\n    def __init__(self, config, pretrained_embed: torch.Tensor = None, transformer: PreTrainedModel = None,\n                 transformer_tokenizer: PreTrainedTokenizer = None):\n        super().__init__()\n        self.encoder = EncoderWithContextualLayer(config, pretrained_embed, transformer, transformer_tokenizer)\n        self.biaffine_decoder = BiaffineDecoder(self.encoder.hidden_size,\n                                                config.n_mlp_arc,\n                                                config.n_mlp_rel,\n                                                config.mlp_dropout,\n                                                config.n_rels)\n\n    def forward(self,\n                words=None,\n                feats=None,\n                input_ids=None,\n                token_span=None,\n                mask=None, lens=None, **kwargs):\n        x, mask = self.encoder(words, feats, input_ids, token_span, mask, lens)\n        s_arc, s_rel = self.biaffine_decoder(x, mask)\n\n        return s_arc, s_rel\n"
  },
  {
    "path": "hanlp/components/parsers/biaffine/biaffine_sdp.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-07-28 15:12\nimport functools\nfrom collections import Counter\nfrom typing import Union, List\n\nimport torch\nfrom torch import nn\n\nfrom hanlp_common.constant import UNK\nfrom hanlp.common.transform import TransformList\nfrom hanlp.components.parsers.biaffine.biaffine_dep import BiaffineDependencyParser\nfrom hanlp_common.conll import CoNLLUWord, CoNLLSentence\nfrom hanlp.datasets.parsing.semeval15 import unpack_deps_to_head_deprel, append_bos_to_form_pos\nfrom hanlp.metrics.parsing.labeled_f1 import LabeledF1\nfrom hanlp_common.util import merge_locals_kwargs\n\n\nclass BiaffineSemanticDependencyParser(BiaffineDependencyParser):\n    def __init__(self) -> None:\n        r\"\"\"Implementation of \"Stanford's graph-based neural dependency parser at\n        the conll 2017 shared task\" (:cite:`dozat2017stanford`) and \"Establishing Strong Baselines for the New Decade\"\n        (:cite:`he-choi-2019`).\n        \"\"\"\n        super().__init__()\n\n    def get_pad_dict(self):\n        return {'arc': False}\n\n    def build_metric(self, **kwargs):\n        return LabeledF1()\n\n    # noinspection PyMethodOverriding\n    def build_dataset(self, data, transform=None):\n        transforms = TransformList(functools.partial(append_bos_to_form_pos, pos_key='UPOS'),\n                                   functools.partial(unpack_deps_to_head_deprel, pad_rel=self.config.pad_rel))\n        if transform:\n            transforms.append(transform)\n        return super(BiaffineSemanticDependencyParser, self).build_dataset(data, transforms)\n\n    def build_criterion(self, **kwargs):\n        return nn.BCEWithLogitsLoss(), nn.CrossEntropyLoss()\n\n    def feed_batch(self, batch):\n        arc_scores, rel_scores, mask, puncts = super().feed_batch(batch)\n        mask = self.convert_to_3d_mask(arc_scores, mask)\n        puncts = self.convert_to_3d_puncts(puncts, mask)\n        return arc_scores, rel_scores, mask, puncts\n\n    @staticmethod\n    def convert_to_3d_puncts(puncts, mask):\n        if puncts is not None:\n            puncts = puncts.unsqueeze(-1).expand_as(mask)\n        return puncts\n\n    @staticmethod\n    def convert_to_3d_mask(arc_scores, mask):\n        # 3d masks\n        mask = mask.unsqueeze(-1).expand_as(arc_scores).clone()\n        mask[:, :, 1:] = mask[:, :, 1:] & mask.transpose(1, 2)[:, :, 1:]  # Keep the 1st colum because it predicts root\n        return mask\n\n    def compute_loss(self, arc_scores, rel_scores, arcs, rels, mask: torch.BoolTensor, criterion, batch=None):\n        bce, ce = criterion\n        arc_scores, arcs = arc_scores[mask], arcs[mask]\n        rel_scores, rels = rel_scores[mask], rels[mask]\n        rel_scores, rels = rel_scores[arcs], rels[arcs]\n        arc_loss = bce(arc_scores, arcs.to(torch.float))\n        arc_loss_interpolation = self.config.get('arc_loss_interpolation', None)\n        loss = arc_loss * arc_loss_interpolation if arc_loss_interpolation else arc_loss\n        if len(rels):\n            rel_loss = ce(rel_scores, rels)\n            loss += (rel_loss * (1 - arc_loss_interpolation)) if arc_loss_interpolation else rel_loss\n        if arc_loss_interpolation:\n            loss *= 2\n        return loss\n\n    def cache_dataset(self, dataset, timer, training=False, logger=None):\n        if not self.config.apply_constraint:\n            return super(BiaffineSemanticDependencyParser, self).cache_dataset(dataset, timer, training)\n        num_roots = Counter()\n        no_zero_head = True\n        root_rels = Counter()\n        for each in dataset:\n            if training:\n                num_roots[sum([x[0] for x in each['arc']])] += 1\n                no_zero_head &= all([x != '_' for x in each['DEPS']])\n                head_is_root = [i for i in range(len(each['arc'])) if each['arc'][i][0]]\n                if head_is_root:\n                    for i in head_is_root:\n                        root_rels[each['rel'][i][0]] += 1\n            timer.log('Preprocessing and caching samples [blink][yellow]...[/yellow][/blink]')\n        if training:\n            if self.config.single_root is None:\n                self.config.single_root = len(num_roots) == 1 and num_roots.most_common()[0][0] == 1\n            if self.config.no_zero_head is None:\n                self.config.no_zero_head = no_zero_head\n            root_rel = root_rels.most_common()[0][0]\n            self.config.root_rel_id = self.vocabs['rel'].get_idx(root_rel)\n            if logger:\n                logger.info(f'Training set properties: [blue]single_root = {self.config.single_root}[/blue], '\n                            f'[blue]no_zero_head = {no_zero_head}[/blue], '\n                            f'[blue]root_rel = {root_rel}[/blue]')\n\n    def decode(self, arc_scores, rel_scores, mask, batch=None):\n        eye = torch.arange(0, arc_scores.size(1), device=arc_scores.device).view(1, 1, -1).expand(\n            arc_scores.size(0), -1, -1)\n        inf = float('inf')\n        arc_scores.scatter_(dim=1, index=eye, value=-inf)\n\n        if self.config.apply_constraint:\n            if self.config.get('single_root', False):\n                arc_scores[~mask] = -inf  # the biaffine decoder doesn't apply 3d mask for now\n                root_mask = arc_scores[:, :, 0].argmax(dim=-1).unsqueeze_(-1).expand_as(arc_scores[:, :, 0])\n                arc_scores[:, :, 0] = -inf\n                arc_scores[:, :, 0].scatter_(dim=-1, index=root_mask, value=inf)\n\n            root_rel_id = self.config.root_rel_id\n            rel_scores[:, :, 0, root_rel_id] = inf\n            rel_scores[:, :, 1:, root_rel_id] = -inf\n\n            arc_scores_T = arc_scores.transpose(-1, -2)\n            arc = ((arc_scores > 0) & (arc_scores_T < arc_scores))\n            if self.config.get('no_zero_head', False):\n                arc_scores_T[arc] = -inf  # avoid cycle between a pair of nodes\n                arc_scores_fix = arc_scores_T.argmax(dim=-2).unsqueeze_(-1).expand_as(arc_scores)\n                arc.scatter_(dim=-1, index=arc_scores_fix, value=True)\n        else:\n            arc = arc_scores > 0\n        rel = rel_scores.argmax(dim=-1)\n        return arc, rel\n\n    def collect_outputs_extend(self, predictions, arc_preds, rel_preds, lens, mask):\n        predictions.extend(zip(arc_preds.tolist(), rel_preds.tolist(), mask.tolist()))\n        # all_arcs.extend(seq.tolist() for seq in arc_preds[mask].split([x * x for x in lens]))\n        # all_rels.extend(seq.tolist() for seq in rel_preds[mask].split([x * x for x in lens]))\n\n    def predictions_to_human(self, predictions, outputs, data, use_pos, conll=True):\n        for d, (arcs, rels, masks) in zip(data, predictions):\n            sent = CoNLLSentence()\n            for idx, (cell, a, r) in enumerate(zip(d, arcs[1:], rels[1:])):\n                if use_pos:\n                    token, pos = cell\n                else:\n                    token, pos = cell, None\n                heads = [i for i in range(len(d) + 1) if a[i]]\n                deprels = [self.vocabs['rel'][r[i]] for i in range(len(d) + 1) if a[i]]\n                sent.append(\n                    CoNLLUWord(idx + 1, token, upos=pos, head=None, deprel=None, deps=list(zip(heads, deprels))))\n            outputs.append(sent)\n\n    def fit(self, trn_data, dev_data, save_dir,\n            feat=None,\n            n_embed=100,\n            pretrained_embed=None,\n            transformer=None,\n            average_subwords=False,\n            word_dropout: float = 0.2,\n            transformer_hidden_dropout=None,\n            layer_dropout=0,\n            mix_embedding: int = None,\n            embed_dropout=.33,\n            n_lstm_hidden=400,\n            n_lstm_layers=3,\n            hidden_dropout=.33,\n            n_mlp_arc=500,\n            n_mlp_rel=100,\n            mlp_dropout=.33,\n            arc_dropout=None,\n            rel_dropout=None,\n            arc_loss_interpolation=0.4,\n            lr=2e-3,\n            transformer_lr=5e-5,\n            mu=.9,\n            nu=.9,\n            epsilon=1e-12,\n            clip=5.0,\n            decay=.75,\n            decay_steps=5000,\n            weight_decay=0,\n            warmup_steps=0.1,\n            separate_optimizer=True,\n            patience=100,\n            batch_size=None,\n            sampler_builder=None,\n            lowercase=False,\n            epochs=50000,\n            apply_constraint=False,\n            single_root=None,\n            no_zero_head=None,\n            punct=False,\n            min_freq=2,\n            logger=None,\n            verbose=True,\n            unk=UNK,\n            pad_rel=None,\n            max_sequence_length=512,\n            gradient_accumulation=1,\n            devices: Union[float, int, List[int]] = None,\n            transform=None,\n            **kwargs):\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n"
  },
  {
    "path": "hanlp/components/parsers/biaffine/mlp.py",
    "content": "# MIT License\n#\n# Copyright (c) 2020 Yu Zhang\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\n\n\n\nimport torch.nn as nn\n\nfrom hanlp.layers.dropout import SharedDropout\n\n\nclass MLP(nn.Module):\n    r\"\"\"\n    Applies a linear transformation together with a non-linear activation to the incoming tensor:\n    :math:`y = \\mathrm{Activation}(x A^T + b)`\n\n    Args:\n        n_in (~torch.Tensor):\n            The size of each input feature.\n        n_out (~torch.Tensor):\n            The size of each output feature.\n        dropout (float):\n            If non-zero, introduce a :class:`SharedDropout` layer on the output with this dropout ratio. Default: 0.\n        activation (bool):\n            Whether to use activations. Default: True.\n    \"\"\"\n\n    def __init__(self, n_in, n_out, dropout=0, activation=True):\n        super().__init__()\n\n        self.n_in = n_in\n        self.n_out = n_out\n        self.linear = nn.Linear(n_in, n_out)\n        self.activation = nn.LeakyReLU(negative_slope=0.1) if activation else nn.Identity()\n        self.dropout = SharedDropout(p=dropout)\n\n        self.reset_parameters()\n\n    def __repr__(self):\n        s = f\"n_in={self.n_in}, n_out={self.n_out}\"\n        if self.dropout.p > 0:\n            s += f\", dropout={self.dropout.p}\"\n\n        return f\"{self.__class__.__name__}({s})\"\n\n    def reset_parameters(self):\n        nn.init.orthogonal_(self.linear.weight)\n        nn.init.zeros_(self.linear.bias)\n\n    def forward(self, x):\n        r\"\"\"\n        Args:\n            x (~torch.Tensor):\n                The size of each input feature is `n_in`.\n\n        Returns:\n            A tensor with the size of each output feature `n_out`.\n        \"\"\"\n\n        x = self.linear(x)\n        x = self.activation(x)\n        x = self.dropout(x)\n\n        return x\n\n"
  },
  {
    "path": "hanlp/components/parsers/biaffine/structual_attention.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-26 10:40\nfrom typing import Union, List\n\nimport torch\nimport torch.nn.functional as F\nfrom hanlp.utils.torch_util import lengths_to_mask\nfrom torch import nn\n\nfrom hanlp.common.torch_component import TorchComponent\nfrom hanlp.components.parsers.biaffine.biaffine_dep import BiaffineDependencyParser\nfrom hanlp.components.parsers.biaffine.biaffine_model import BiaffineDecoder\nfrom hanlp.layers.transformers.encoder import TransformerEncoder\nfrom hanlp.layers.transformers.pt_imports import PreTrainedModel, PreTrainedTokenizer\nfrom hanlp.metrics.accuracy import CategoricalAccuracy\nfrom hanlp.transform.transformer_tokenizer import TransformerSequenceTokenizer\nfrom hanlp_common.util import merge_locals_kwargs\n\n\nclass StructuralAttentionLayer(nn.Module):\n\n    def __init__(self, hidden_size, n_mlp_arc, n_mlp_rel, mlp_dropout, n_rels, projeciton=None) -> None:\n        super().__init__()\n        self.biaffine = BiaffineDecoder(hidden_size, n_mlp_arc, n_mlp_rel, mlp_dropout, n_rels)\n        if projeciton:\n            self.projection = nn.Linear(hidden_size, projeciton)\n            hidden_size = projeciton\n        else:\n            self.projection = None\n        self.head_WV = nn.Parameter(torch.randn(n_rels, hidden_size, hidden_size))\n        self.dense = nn.Linear(hidden_size * n_rels, hidden_size)\n        self.activation = nn.GELU()\n\n    def forward(self, x, mask):\n        s_arc, s_rel = self.biaffine(x, mask)\n        p_arc = F.softmax(s_arc, dim=-1) * mask.unsqueeze(-1)\n        p_rel = F.softmax(s_rel, -1)\n        A = p_arc.unsqueeze(-1) * p_rel\n        if self.projection:\n            x = self.projection(x)\n        Ax = torch.einsum('bijk,bih->bihk', A, x)\n        AxW = torch.einsum('bihk,khm->bihk', Ax, self.head_WV)\n        AxW = AxW.flatten(2)\n        x = self.dense(AxW)\n        x = self.activation(x)\n        return s_arc, s_rel, x\n\n\nclass StructuralAttentionModel(nn.Module):\n    def __init__(self,\n                 config,\n                 transformer: PreTrainedModel = None,\n                 transformer_tokenizer: PreTrainedTokenizer = None\n                 ) -> None:\n        super().__init__()\n        self.encoder = TransformerEncoder(transformer,\n                                          transformer_tokenizer,\n                                          config.average_subwords,\n                                          config.scalar_mix,\n                                          None,  # No word_dropout since SA is predicting masked tokens\n                                          config.transformer_hidden_dropout,\n                                          config.layer_dropout,\n                                          config.max_sequence_length)\n        hidden_size = transformer.config.hidden_size\n        self.sa = StructuralAttentionLayer(hidden_size,\n                                           config.n_mlp_arc,\n                                           config.n_mlp_rel,\n                                           config.mlp_dropout,\n                                           config.n_rels,\n                                           config.projection\n                                           )\n        if config.projection:\n            hidden_size = config.projection\n        self.mlm = nn.Linear(hidden_size, transformer_tokenizer.vocab_size)\n\n    def forward(self,\n                input_ids: torch.LongTensor,\n                attention_mask=None,\n                token_type_ids=None,\n                token_span=None,\n                mask=None,\n                batch=None,\n                **kwargs):\n        h = self.encoder(input_ids, attention_mask, token_type_ids, token_span)\n        s_arc, s_rel, h = self.sa(h, mask)\n        x = self.mlm(h)\n        return s_arc, s_rel, x\n\n\nclass MaskedTokenGenerator(object):\n\n    def __init__(self, transformer_tokenizer: PreTrainedTokenizer, mask_prob=0.15) -> None:\n        super().__init__()\n        self.mask_prob = mask_prob\n        self.transformer_tokenizer = transformer_tokenizer\n        self.oov = transformer_tokenizer.mask_token_id\n        self.pad = transformer_tokenizer.pad_token_id\n        self.cls = transformer_tokenizer.cls_token_id\n        self.sep = transformer_tokenizer.sep_token_id\n        self.excludes = [self.pad, self.cls, self.sep]\n\n    def __call__(self, tokens: torch.LongTensor, prefix_mask: torch.LongTensor):\n        padding_mask = tokens.new_ones(tokens.size(), dtype=torch.bool)\n        for pad in self.excludes:\n            padding_mask &= (tokens != pad)\n        padding_mask &= prefix_mask  # Only mask prefixes since the others won't be attended\n        # Create a uniformly random mask selecting either the original words or OOV tokens\n        dropout_mask = (tokens.new_empty(tokens.size(), dtype=torch.float).uniform_() < self.mask_prob)\n        oov_mask = dropout_mask & padding_mask\n\n        oov_fill = tokens.new_empty(tokens.size(), dtype=torch.long).fill_(self.oov)\n\n        result = torch.where(oov_mask, oov_fill, tokens)\n        return result, oov_mask\n\n\nclass StructuralAttentionParser(BiaffineDependencyParser):\n    def __init__(self) -> None:\n        super().__init__()\n        self.model: StructuralAttentionModel = None\n        self.mlm_generator: MaskedTokenGenerator = None\n\n    def build_model(self, training=True, **kwargs) -> torch.nn.Module:\n        transformer = TransformerEncoder.build_transformer(config=self.config, training=training)\n        model = StructuralAttentionModel(self.config, transformer, self.transformer_tokenizer)\n        return model\n\n    def fit(self, trn_data, dev_data, save_dir,\n            transformer=None,\n            mask_prob=0.15,\n            projection=None,\n            average_subwords=False,\n            transformer_hidden_dropout=None,\n            layer_dropout=0,\n            mix_embedding: int = None,\n            embed_dropout=.33,\n            n_mlp_arc=500,\n            n_mlp_rel=100,\n            mlp_dropout=.33,\n            lr=2e-3,\n            transformer_lr=5e-5,\n            mu=.9,\n            nu=.9,\n            epsilon=1e-12,\n            clip=5.0,\n            decay=.75,\n            decay_steps=5000,\n            patience=100,\n            sampler='kmeans',\n            n_buckets=32,\n            batch_max_tokens=5000,\n            batch_size=None,\n            epochs=50000,\n            tree=False,\n            punct=False,\n            logger=None,\n            verbose=True,\n            max_sequence_length=512,\n            devices: Union[float, int, List[int]] = None,\n            transform=None,\n            **kwargs):\n        return TorchComponent.fit(self, **merge_locals_kwargs(locals(), kwargs))\n\n    def feed_batch(self, batch):\n        if self.model.training:\n            input_ids = batch['input_ids']\n            prefix_mask = batch['prefix_mask']\n            batch['gold_input_ids'] = input_ids\n            batch['input_ids'], batch['input_ids_mask'] = self.mlm_generator(input_ids, prefix_mask)\n        words, feats, lens, puncts = batch.get('token_id', None), batch.get('pos_id', None), batch['sent_length'], \\\n                                     batch.get('punct_mask', None)\n        mask = lengths_to_mask(lens)\n        arc_scores, rel_scores, pred_input_ids = self.model(words=words, feats=feats, mask=mask, batch=batch, **batch)\n        batch['pred_input_ids'] = pred_input_ids\n        # ignore the first token of each sentence\n        # RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation\n        if self.model.training:\n            mask = mask.clone()\n        mask[:, 0] = 0\n        return arc_scores, rel_scores, mask, puncts\n\n    def on_config_ready(self, **kwargs):\n        super().on_config_ready(**kwargs)\n        self.mlm_generator = MaskedTokenGenerator(self.transformer_tokenizer, self.config.mask_prob)\n\n    def compute_loss(self, arc_scores, rel_scores, arcs, rels, mask, criterion, batch=None):\n        parse_loss = BiaffineDependencyParser.compute_loss(self, arc_scores, rel_scores, arcs, rels, mask, criterion, batch)\n        if self.model.training:\n            gold_input_ids = batch['gold_input_ids']\n            pred_input_ids = batch['pred_input_ids']\n            input_ids_mask = batch['input_ids_mask']\n            token_span = batch['token_span']\n            gold_input_ids = batch['gold_input_ids'] = gold_input_ids.gather(1, token_span[:, :, 0])\n            input_ids_mask = batch['input_ids_mask'] = input_ids_mask.gather(1, token_span[:, :, 0])\n            mlm_loss = F.cross_entropy(pred_input_ids[input_ids_mask], gold_input_ids[input_ids_mask])\n            loss = parse_loss + mlm_loss\n            return loss\n        return parse_loss\n\n    def build_tokenizer_transform(self):\n        return TransformerSequenceTokenizer(self.transformer_tokenizer, 'token', '', ret_prefix_mask=True,\n                                            ret_token_span=True, cls_is_bos=True,\n                                            max_seq_length=self.config.get('max_sequence_length',\n                                                                           512),\n                                            truncate_long_sequences=False)\n\n    def build_metric(self, training=None, **kwargs):\n        parse_metric = super().build_metric(**kwargs)\n        if training:\n            mlm_metric = CategoricalAccuracy()\n            return parse_metric, mlm_metric\n        return parse_metric\n\n    def update_metric(self, arc_scores, rel_scores, arcs, rels, mask, puncts, metric, batch=None):\n        if isinstance(metric, tuple):\n            parse_metric, mlm_metric = metric\n            super().update_metric(arc_scores, rel_scores, arcs, rels, mask, puncts, parse_metric)\n            gold_input_ids = batch['gold_input_ids']\n            input_ids_mask = batch['input_ids_mask']\n            pred_input_ids = batch['pred_input_ids']\n            pred_input_ids = pred_input_ids[input_ids_mask]\n            gold_input_ids = gold_input_ids[input_ids_mask]\n            if len(pred_input_ids):\n                mlm_metric(pred_input_ids, gold_input_ids)\n        else:\n            super().update_metric(arc_scores, rel_scores, arcs, rels, mask, puncts, metric)\n\n    def _report(self, loss, metric):\n        if isinstance(metric, tuple):\n            parse_metric, mlm_metric = metric\n            return super()._report(loss, parse_metric) + f' {mlm_metric}'\n        else:\n            return super()._report(loss, metric)\n"
  },
  {
    "path": "hanlp/components/parsers/biaffine/variationalbilstm.py",
    "content": "# MIT License\n#\n# Copyright (c) 2020 Yu Zhang\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\n\n\nimport torch\nimport torch.nn as nn\nfrom torch.nn.modules.rnn import apply_permutation\nfrom torch.nn.utils.rnn import PackedSequence, pack_padded_sequence, pad_packed_sequence\n\nfrom hanlp.common.structure import ConfigTracker\nfrom hanlp.layers.dropout import SharedDropout\n\n\nclass VariationalLSTM(nn.Module):\n    r\"\"\"\n    LSTM is an variant of the vanilla bidirectional LSTM adopted by Biaffine Parser\n    with the only difference of the dropout strategy.\n    It drops nodes in the LSTM layers (input and recurrent connections)\n    and applies the same dropout mask at every recurrent timesteps.\n\n    APIs are roughly the same as :class:`~torch.nn.LSTM` except that we only allows\n    :class:`~torch.nn.utils.rnn.PackedSequence` as input.\n\n    References:\n        - Timothy Dozat and Christopher D. Manning. 2017.\n          `Deep Biaffine Attention for Neural Dependency Parsing`_.\n\n    Args:\n        input_size (int):\n            The number of expected features in the input.\n        hidden_size (int):\n            The number of features in the hidden state `h`.\n        num_layers (int):\n            The number of recurrent layers. Default: 1.\n        bidirectional (bool):\n            If ``True``, becomes a bidirectional LSTM. Default: ``False``\n        dropout (float):\n            If non-zero, introduces a :class:`SharedDropout` layer on the outputs of each LSTM layer except the last layer.\n            Default: 0.\n\n    .. _Deep Biaffine Attention for Neural Dependency Parsing:\n        https://openreview.net/forum?id=Hk95PK9le\n    \"\"\"\n\n    def __init__(self, input_size, hidden_size, num_layers=1, bidirectional=False, dropout=0):\n        super().__init__()\n\n        self.input_size = input_size\n        self.hidden_size = hidden_size\n        self.num_layers = num_layers\n        self.bidirectional = bidirectional\n        self.dropout = dropout\n        self.num_directions = 1 + self.bidirectional\n\n        self.f_cells = nn.ModuleList()\n        if bidirectional:\n            self.b_cells = nn.ModuleList()\n        for _ in range(self.num_layers):\n            self.f_cells.append(nn.LSTMCell(input_size=input_size, hidden_size=hidden_size))\n            if bidirectional:\n                self.b_cells.append(nn.LSTMCell(input_size=input_size, hidden_size=hidden_size))\n            input_size = hidden_size * self.num_directions\n\n        self.reset_parameters()\n\n    def __repr__(self):\n        s = f\"{self.input_size}, {self.hidden_size}\"\n        if self.num_layers > 1:\n            s += f\", num_layers={self.num_layers}\"\n        if self.bidirectional:\n            s += f\", bidirectional={self.bidirectional}\"\n        if self.dropout > 0:\n            s += f\", dropout={self.dropout}\"\n\n        return f\"{self.__class__.__name__}({s})\"\n\n    def reset_parameters(self):\n        for param in self.parameters():\n            # apply orthogonal_ to weight\n            if len(param.shape) > 1:\n                nn.init.orthogonal_(param)\n            # apply zeros_ to bias\n            else:\n                nn.init.zeros_(param)\n\n    def permute_hidden(self, hx, permutation):\n        if permutation is None:\n            return hx\n        h = apply_permutation(hx[0], permutation)\n        c = apply_permutation(hx[1], permutation)\n\n        return h, c\n\n    def layer_forward(self, x, hx, cell, batch_sizes, reverse=False):\n        hx_0 = hx_i = hx\n        hx_n, output = [], []\n        steps = reversed(range(len(x))) if reverse else range(len(x))\n        if self.training:\n            hid_mask = SharedDropout.get_mask(hx_0[0], self.dropout)\n\n        for t in steps:\n            last_batch_size, batch_size = len(hx_i[0]), batch_sizes[t]\n            if last_batch_size < batch_size:\n                hx_i = [torch.cat((h, ih[last_batch_size:batch_size])) for h, ih in zip(hx_i, hx_0)]\n            else:\n                hx_n.append([h[batch_size:] for h in hx_i])\n                hx_i = [h[:batch_size] for h in hx_i]\n            hx_i = [h for h in cell(x[t], hx_i)]\n            output.append(hx_i[0])\n            if self.training:\n                hx_i[0] = hx_i[0] * hid_mask[:batch_size]\n        if reverse:\n            hx_n = hx_i\n            output.reverse()\n        else:\n            hx_n.append(hx_i)\n            hx_n = [torch.cat(h) for h in zip(*reversed(hx_n))]\n        output = torch.cat(output)\n\n        return output, hx_n\n\n    def forward(self, sequence, hx=None):\n        r\"\"\"\n        Args:\n            sequence (~torch.nn.utils.rnn.PackedSequence):\n                A packed variable length sequence.\n            hx (~torch.Tensor, ~torch.Tensor):\n                A tuple composed of two tensors `h` and `c`.\n                `h` of shape ``[num_layers*num_directions, batch_size, hidden_size]`` holds the initial hidden state\n                for each element in the batch.\n                `c` of shape ``[num_layers*num_directions, batch_size, hidden_size]`` holds the initial cell state\n                for each element in the batch.\n                If `hx` is not provided, both `h` and `c` default to zero.\n                Default: ``None``.\n\n        Returns:\n            ~torch.nn.utils.rnn.PackedSequence, (~torch.Tensor, ~torch.Tensor):\n                The first is a packed variable length sequence.\n                The second is a tuple of tensors `h` and `c`.\n                `h` of shape ``[num_layers*num_directions, batch_size, hidden_size]`` holds the hidden state for `t=seq_len`.\n                Like output, the layers can be separated using ``h.view(num_layers, num_directions, batch_size, hidden_size)``\n                and similarly for c.\n                `c` of shape ``[num_layers*num_directions, batch_size, hidden_size]`` holds the cell state for `t=seq_len`.\n        \"\"\"\n        x, batch_sizes = sequence.data, sequence.batch_sizes.tolist()\n        batch_size = batch_sizes[0]\n        h_n, c_n = [], []\n\n        if hx is None:\n            ih = x.new_zeros(self.num_layers * self.num_directions, batch_size, self.hidden_size)\n            h, c = ih, ih\n        else:\n            h, c = self.permute_hidden(hx, sequence.sorted_indices)\n        h = h.view(self.num_layers, self.num_directions, batch_size, self.hidden_size)\n        c = c.view(self.num_layers, self.num_directions, batch_size, self.hidden_size)\n\n        for i in range(self.num_layers):\n            x = torch.split(x, batch_sizes)\n            if self.training:\n                mask = SharedDropout.get_mask(x[0], self.dropout)\n                x = [i * mask[:len(i)] for i in x]\n            x_i, (h_i, c_i) = self.layer_forward(x=x,\n                                                 hx=(h[i, 0], c[i, 0]),\n                                                 cell=self.f_cells[i],\n                                                 batch_sizes=batch_sizes)\n            if self.bidirectional:\n                x_b, (h_b, c_b) = self.layer_forward(x=x,\n                                                     hx=(h[i, 1], c[i, 1]),\n                                                     cell=self.b_cells[i],\n                                                     batch_sizes=batch_sizes,\n                                                     reverse=True)\n                x_i = torch.cat((x_i, x_b), -1)\n                h_i = torch.stack((h_i, h_b))\n                c_i = torch.stack((c_i, c_b))\n            x = x_i\n            h_n.append(h_i)\n            c_n.append(h_i)\n\n        x = PackedSequence(x,\n                           sequence.batch_sizes,\n                           sequence.sorted_indices,\n                           sequence.unsorted_indices)\n        hx = torch.cat(h_n, 0), torch.cat(c_n, 0)\n        hx = self.permute_hidden(hx, sequence.unsorted_indices)\n\n        return x, hx\n\n\nclass VariationalLSTMEncoder(VariationalLSTM, ConfigTracker):\n    def __init__(self,\n                 input_size,\n                 hidden_size,\n                 num_layers=1,\n                 bidirectional=False,\n                 variational_dropout=0,\n                 word_dropout=0,\n                 ):\n        super().__init__(input_size, hidden_size, num_layers, bidirectional, variational_dropout)\n        ConfigTracker.__init__(self, locals())\n        self.lstm_dropout = SharedDropout(p=word_dropout)\n\n    # noinspection PyMethodOverriding\n    def forward(self, embed, mask):\n        batch_size, seq_len = mask.shape\n        x = pack_padded_sequence(embed, mask.sum(1), True, False)\n        x, _ = super().forward(x)\n        x, _ = pad_packed_sequence(x, True, total_length=seq_len)\n        x = self.lstm_dropout(x)\n        return x\n\n    def get_output_dim(self):\n        return self.hidden_size * self.num_directions\n"
  },
  {
    "path": "hanlp/components/parsers/biaffine_parser_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-22 12:47\nimport logging\nimport math\nimport os\nfrom typing import List\nimport numpy as np\nimport tensorflow as tf\n\nfrom hanlp.components.parsers.parse_alg import unique_root, adjust_root_score, chu_liu_edmonds\nfrom hanlp.layers.transformers.loader_tf import build_transformer\n\nfrom hanlp.common.keras_component import KerasComponent\nfrom hanlp.components.parsers.alg_tf import tarjan\nfrom hanlp.components.parsers.biaffine_tf.model import BiaffineModelTF\nfrom hanlp.transform.conll_tf import CoNLL_DEP_Transform, CoNLL_Transformer_Transform, CoNLL_SDP_Transform\nfrom hanlp.layers.embeddings.util_tf import build_embedding\nfrom hanlp.layers.transformers.tf_imports import PreTrainedTokenizer, TFAutoModel, TFPreTrainedModel, AutoTokenizer, \\\n    TFAutoModelWithLMHead, BertTokenizerFast, AlbertConfig, BertTokenizer, TFBertModel\nfrom hanlp.layers.transformers.utils_tf import build_adamw_optimizer\nfrom hanlp.metrics.parsing.labeled_f1_tf import LabeledF1TF\nfrom hanlp.metrics.parsing.labeled_score import LabeledScore\nfrom hanlp_common.util import merge_locals_kwargs\n\n\nclass BiaffineDependencyParserTF(KerasComponent):\n    def __init__(self, transform: CoNLL_DEP_Transform = None) -> None:\n        if not transform:\n            transform = CoNLL_DEP_Transform()\n        super().__init__(transform)\n        self.transform: CoNLL_DEP_Transform = transform\n        self.model: BiaffineModelTF = None\n\n    def build_model(self, pretrained_embed, n_embed, training, **kwargs) -> tf.keras.Model:\n        if training:\n            self.config.n_words = len(self.transform.form_vocab)\n        else:\n            self.config.lstm_dropout = 0.  # keras will use cuda lstm when config.lstm_dropout is 0\n        self.config.n_feats = len(self.transform.cpos_vocab)\n        self._init_config()\n        pretrained: tf.keras.layers.Embedding = build_embedding(pretrained_embed, self.transform.form_vocab,\n                                                                self.transform) if pretrained_embed else None\n        if pretrained_embed:\n            self.config.n_embed = pretrained.output_dim\n        model = BiaffineModelTF(self.config, pretrained)\n        return model\n\n    def _init_config(self):\n        self.config.n_rels = len(self.transform.rel_vocab)\n        self.config.pad_index = self.transform.form_vocab.pad_idx\n        self.config.unk_index = self.transform.form_vocab.unk_idx\n        self.config.bos_index = 2\n\n    def load_weights(self, save_dir, filename='model.h5', functional=False, **kwargs):\n        super().load_weights(save_dir, filename)\n        if functional:\n            self.model = self.model.to_functional()\n\n    def fit(self, trn_data, dev_data, save_dir,\n            n_embed=100,\n            pretrained_embed=None,\n            embed_dropout=.33,\n            n_lstm_hidden=400,\n            n_lstm_layers=3,\n            lstm_dropout=.33,\n            n_mlp_arc=500,\n            n_mlp_rel=100,\n            mlp_dropout=.33,\n            optimizer='adam',\n            lr=2e-3,\n            mu=.9,\n            nu=.9,\n            epsilon=1e-12,\n            clip=5.0,\n            decay=.75,\n            decay_steps=5000,\n            patience=100,\n            arc_loss='sparse_categorical_crossentropy',\n            rel_loss='sparse_categorical_crossentropy',\n            metrics=('UAS', 'LAS'),\n            n_buckets=32,\n            batch_size=5000,\n            epochs=50000,\n            early_stopping_patience=100,\n            tree=False,\n            punct=False,\n            min_freq=2,\n            run_eagerly=False, logger=None, verbose=True,\n            **kwargs):\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    # noinspection PyMethodOverriding\n    def train_loop(self, trn_data, dev_data, epochs, num_examples,\n                   train_steps_per_epoch, dev_steps, model, optimizer, loss, metrics,\n                   callbacks, logger: logging.Logger, arc_loss, rel_loss,\n                   **kwargs):\n        arc_loss, rel_loss = loss\n        # because we are customizing batching\n        train_steps_per_epoch = len(list(iter(trn_data)))\n        # progbar: tf.keras.callbacks.ProgbarLogger = callbacks[-1]\n        c: tf.keras.callbacks.Callback = None\n        metric = self._build_metrics()\n        for c in callbacks:\n            if not hasattr(c, 'params'):\n                c.params = dict()\n            c.params['epochs'] = epochs\n            c.params['trn_data'] = trn_data\n            c.params['metrics'] = ['loss'] + self.config.metrics\n            c.params['metrics'] = c.params['metrics'] + [f'val_{k}' for k in c.params['metrics']]\n            c.on_train_begin()\n        for epoch in range(epochs):\n            metric.reset_states()\n            for c in callbacks:\n                c.params['steps'] = train_steps_per_epoch\n                c.on_epoch_begin(epoch)\n            for idx, ((words, feats), (arcs, rels)) in enumerate(iter(trn_data)):\n                logs = {}\n                for c in callbacks:\n                    c.on_batch_begin(idx, logs)\n                mask = tf.not_equal(words, self.config.pad_index) & tf.not_equal(words, self.config.bos_index)\n                loss, arc_scores, rel_scores = self.train_batch(words, feats, arcs, rels, mask,\n                                                                optimizer, arc_loss, rel_loss)\n                self.run_metrics(arcs, rels, arc_scores, rel_scores, words, mask, metric)\n                logs['loss'] = loss\n                logs.update(metric.to_dict())\n                if epoch == epochs - 1:\n                    self.model.stop_training = True\n                for c in callbacks:\n                    c.on_batch_end(idx, logs)\n            # evaluate on dev\n            metric.reset_states()\n            logs = {}\n            for idx, ((words, feats), (arcs, rels)) in enumerate(iter(dev_data)):\n                arc_scores, rel_scores, loss, mask, arc_preds, rel_preds = self.evaluate_batch(words, feats, arcs, rels,\n                                                                                               arc_loss, rel_loss,\n                                                                                               metric)\n                logs['val_loss'] = loss\n                logs.update((f'val_{k}', v) for k, v in metric.to_dict().items())\n\n            for c in callbacks:\n                c.on_epoch_end(epoch, logs)\n            if getattr(self.model, 'stop_training', None):\n                break\n\n        for c in callbacks:\n            c.on_train_end()\n\n    def evaluate(self, input_path: str, save_dir=None, output=False, batch_size=None, logger: logging.Logger = None,\n                 callbacks: List[tf.keras.callbacks.Callback] = None, warm_up=False, verbose=True, **kwargs):\n        if batch_size is None:\n            batch_size = self.config.batch_size\n        return super().evaluate(input_path, save_dir, output, batch_size, logger, callbacks, warm_up, verbose, **kwargs)\n\n    def evaluate_batch(self, words, feats, arcs, rels, arc_loss, rel_loss, metric):\n        mask = tf.not_equal(words, self.config.pad_index) & tf.not_equal(words, self.config.bos_index)\n        arc_scores, rel_scores = self.model((words, feats))\n        loss = self.get_loss(arc_scores, rel_scores, arcs, rels, mask, arc_loss, rel_loss)\n        arc_preds, rel_preds = self.run_metrics(arcs, rels, arc_scores, rel_scores, words, mask, metric)\n        return arc_scores, rel_scores, loss, mask, arc_preds, rel_preds\n\n    def _build_metrics(self):\n        if isinstance(self.config.metrics, tuple):\n            self.config.metrics = list(self.config.metrics)\n        if self.config.metrics == ['UAS', 'LAS']:\n            metric = LabeledScore()\n        else:\n            metric = LabeledF1TF()\n        return metric\n\n    def run_metrics(self, arcs, rels, arc_scores, rel_scores, words, mask, metric):\n        arc_preds, rel_preds = self.decode(arc_scores, rel_scores, mask)\n        # ignore all punctuation if not specified\n        if not self.config.punct:\n            mask &= tf.reduce_all(tf.not_equal(tf.expand_dims(words, axis=-1), self.transform.puncts), axis=-1)\n        metric(arc_preds, rel_preds, arcs, rels, mask)\n        return arc_preds, rel_preds\n\n    def train_batch(self, words, feats, arcs, rels, mask, optimizer, arc_loss, rel_loss):\n        with tf.GradientTape() as tape:\n            arc_scores, rel_scores = self.model((words, feats), training=True)\n            loss = self.get_loss(arc_scores, rel_scores, arcs, rels, mask, arc_loss, rel_loss)\n        grads = tape.gradient(loss, self.model.trainable_variables)\n        optimizer.apply_gradients(zip(grads, self.model.trainable_variables))\n        return loss, arc_scores, rel_scores\n\n    def get_loss(self, arc_scores, rel_scores, arcs, rels, mask, arc_loss, rel_loss):\n        arc_scores, arcs = arc_scores[mask], arcs[mask]\n        rel_scores, rels = rel_scores[mask], rels[mask]\n        rel_scores = tf.gather_nd(rel_scores, tf.stack([tf.range(len(arcs), dtype=tf.int64), arcs], axis=1))\n        arc_loss = arc_loss(arcs, arc_scores)\n        rel_loss = rel_loss(rels, rel_scores)\n        loss = arc_loss + rel_loss\n\n        return loss\n\n    def build_optimizer(self, optimizer='adam', lr=2e-3, mu=.9, nu=.9, epsilon=1e-12, clip=5.0, decay=.75,\n                        decay_steps=5000, **kwargs):\n        if optimizer == 'adam':\n            scheduler = tf.keras.optimizers.schedules.ExponentialDecay(initial_learning_rate=lr,\n                                                                       decay_steps=decay_steps,\n                                                                       decay_rate=decay)\n            from hanlp.optimizers.adamw.optimization import AdamTF\n            optimizer = AdamTF(learning_rate=scheduler,\n                               beta_1=mu,\n                               beta_2=nu,\n                               epsilon=epsilon,\n                               clipnorm=clip)\n            return optimizer\n        return super().build_optimizer(optimizer, **kwargs)\n\n    # noinspection PyMethodOverriding\n    def build_loss(self, arc_loss, rel_loss, **kwargs):\n        if arc_loss == 'binary_crossentropy':\n            arc_loss = tf.losses.BinaryCrossentropy(from_logits=True)\n        else:\n            arc_loss = tf.keras.losses.SparseCategoricalCrossentropy(\n                from_logits=True) if arc_loss == 'sparse_categorical_crossentropy' else super().build_loss(arc_loss)\n        rel_loss = tf.keras.losses.SparseCategoricalCrossentropy(\n            from_logits=True) if rel_loss == 'sparse_categorical_crossentropy' else super().build_loss(rel_loss)\n        return arc_loss, rel_loss\n\n    @property\n    def sample_data(self):\n        return tf.constant([[2, 3, 4], [2, 5, 0]], dtype=tf.int64), tf.constant([[1, 2, 3], [4, 5, 0]], dtype=tf.int64)\n\n    def num_samples_in(self, dataset):\n        return sum(len(x[0][0]) for x in iter(dataset))\n\n    def build_train_dataset(self, trn_data, batch_size, num_examples):\n        trn_data = self.transform.file_to_dataset(trn_data, batch_size=batch_size,\n                                                  shuffle=True,\n                                                  repeat=None)\n        return trn_data\n\n    # noinspection PyMethodOverriding\n    def build_callbacks(self, save_dir, logger, metrics, **kwargs):\n        callbacks = super().build_callbacks(save_dir, logger, metrics=metrics, **kwargs)\n        if isinstance(metrics, tuple):\n            metrics = list(metrics)\n        callbacks.append(self.build_progbar(metrics))\n        params = {'verbose': 1, 'epochs': 1}\n        for c in callbacks:\n            c.set_params(params)\n            c.set_model(self.model)\n        return callbacks\n\n    def build_progbar(self, metrics, training=True):\n        return tf.keras.callbacks.ProgbarLogger(count_mode='steps',\n                                                stateful_metrics=metrics + [f'val_{k}' for k in metrics] if training\n                                                else [])\n\n    def decode(self, arc_scores, rel_scores, mask):\n        if self.config.tree:\n            root_rel_idx = self.transform.root_rel_idx\n            root_rel_onehot = np.eye(len(self.transform.rel_vocab))[root_rel_idx]\n            arc_preds = np.zeros_like(mask, dtype=np.int64)\n            rel_preds = np.zeros_like(mask, dtype=np.int64)\n            for arc, rel, m, arc_pred, rel_pred in zip(arc_scores, rel_scores, mask, arc_preds, rel_preds):\n                length = int(tf.math.count_nonzero(m)) + 1\n                arc = arc[:length, :length]\n                arc_probs = tf.nn.softmax(arc).numpy()\n                m = np.expand_dims(m.numpy()[:length], -1)\n                if self.config.tree == 'tarjan':\n                    heads = tarjan(arc_probs, length, m)\n                elif self.config.tree == 'mst':\n                    heads, head_probs, tokens = unique_root(arc_probs, m, length)\n                    arc = arc.numpy()\n                    adjust_root_score(arc, heads, root_rel_idx)\n                    heads = chu_liu_edmonds(arc, length)\n                else:\n                    raise ValueError(f'Unknown tree algorithm {self.config.tree}')\n                arc_pred[:length] = heads\n                root = np.where(heads[np.arange(1, length)] == 0)[0] + 1\n                rel_prob = tf.nn.softmax(rel[:length, :length, :]).numpy()\n                rel_prob = rel_prob[np.arange(length), heads]\n                rel_prob[root] = root_rel_onehot\n                rel_prob[np.arange(length) != root, np.arange(len(self.transform.rel_vocab)) == root_rel_idx] = 0\n                # rels = rel_argmax(rel_prob, length, root_rel_idx)\n                rels = np.argmax(rel_prob, axis=1)\n                rel_pred[:length] = rels\n            arc_preds = tf.constant(arc_preds)\n            rel_preds = tf.constant(rel_preds)\n        else:\n            arc_preds = tf.argmax(arc_scores, -1)\n            rel_preds = tf.argmax(rel_scores, -1)\n            rel_preds = tf.squeeze(tf.gather(rel_preds, tf.expand_dims(arc_preds, -1), batch_dims=2), axis=-1)\n\n        return arc_preds, rel_preds\n\n    def evaluate_dataset(self, tst_data, callbacks, output, num_batches, ret_scores=None, **kwargs):\n        if 'mask_p' in self.config:\n            self.config['mask_p'] = None\n        arc_loss, rel_loss = self.build_loss(**self.config)\n        callbacks = [self.build_progbar(self.config['metrics'])]\n        steps_per_epoch = len(list(iter(tst_data)))\n        metric = self._build_metrics()\n        params = {'verbose': 1, 'epochs': 1, 'metrics': ['loss'] + self.config.metrics, 'steps': steps_per_epoch}\n        for c in callbacks:\n            c.set_params(params)\n            c.on_test_begin()\n            c.on_epoch_end(0)\n        logs = {}\n        if ret_scores:\n            scores = []\n        if output:\n            ext = os.path.splitext(output)[-1]\n            output = open(output, 'w', encoding='utf-8')\n        for idx, ((words, feats), Y) in enumerate(iter(tst_data)):\n            arcs, rels = Y[0], Y[1]\n            for c in callbacks:\n                c.on_test_batch_begin(idx, logs)\n            arc_scores, rel_scores, loss, mask, arc_preds, rel_preds = self.evaluate_batch(words, feats, arcs, rels,\n                                                                                           arc_loss, rel_loss, metric)\n            if ret_scores:\n                scores.append((arc_scores.numpy(), rel_scores.numpy(), mask.numpy()))\n            if output:\n                for sent in self.transform.XY_to_inputs_outputs((words, feats, mask), (arc_preds, rel_preds),\n                                                                conll=ext, arc_scores=arc_scores,\n                                                                rel_scores=rel_scores):\n                    output.write(str(sent))\n                    output.write('\\n\\n')\n            logs['loss'] = loss\n            logs.update(metric.to_dict())\n            for c in callbacks:\n                c.on_test_batch_end(idx, logs)\n        for c in callbacks:\n            c.on_epoch_end(0)\n            c.on_test_end()\n        if output:\n            output.close()\n        loss = float(c.progbar._values['loss'][0] / c.progbar._values['loss'][1])\n        outputs = loss, metric.to_dict(), False\n        if ret_scores:\n            outputs += (scores,)\n        return outputs\n\n    def predict_batch(self, batch, inputs=None, conll=True, **kwargs):\n        ((words, feats), (arcs, rels)) = batch\n        mask = tf.not_equal(words, self.config.pad_index) & tf.not_equal(words, self.config.bos_index)\n        arc_scores, rel_scores = self.model((words, feats))\n        arc_preds, rel_preds = self.decode(arc_scores, rel_scores, mask)\n        for sent in self.transform.XY_to_inputs_outputs((words, feats, mask), (arc_preds, rel_preds), gold=False,\n                                                        inputs=inputs, conll=conll):\n            yield sent\n\n    def compile_model(self, optimizer, loss, metrics):\n        super().compile_model(optimizer, loss, metrics)\n\n\nclass BiaffineSemanticDependencyParserTF(BiaffineDependencyParserTF):\n    def __init__(self, transform: CoNLL_SDP_Transform = None) -> None:\n        if not transform:\n            transform = CoNLL_SDP_Transform()\n        # noinspection PyTypeChecker\n        super().__init__(transform)\n        self.transform: CoNLL_SDP_Transform = transform\n\n    def fit(self, trn_data, dev_data, save_dir, n_embed=100, pretrained_embed=None, embed_dropout=.33,\n            n_lstm_hidden=400, n_lstm_layers=3, lstm_dropout=.33, n_mlp_arc=500, n_mlp_rel=100, mlp_dropout=.33,\n            optimizer='adam', lr=2e-3, mu=.9, nu=.9, epsilon=1e-12, clip=5.0, decay=.75, decay_steps=5000, patience=100,\n            arc_loss='binary_crossentropy', rel_loss='sparse_categorical_crossentropy',\n            metrics=('UF', 'LF'), n_buckets=32, batch_size=5000, epochs=50000, early_stopping_patience=100,\n            tree=False, punct=False, min_freq=2, run_eagerly=False, logger=None, verbose=True, **kwargs):\n        return super().fit(trn_data, dev_data, save_dir, n_embed, pretrained_embed, embed_dropout, n_lstm_hidden,\n                           n_lstm_layers, lstm_dropout, n_mlp_arc, n_mlp_rel, mlp_dropout, optimizer, lr, mu, nu,\n                           epsilon, clip, decay, decay_steps, patience, arc_loss, rel_loss, metrics, n_buckets,\n                           batch_size, epochs, early_stopping_patience, tree, punct, min_freq, run_eagerly, logger,\n                           verbose, **kwargs)\n\n    def get_loss(self, arc_scores, rel_scores, arcs, rels, mask, arc_loss, rel_loss):\n        mask = tf.tile(tf.expand_dims(mask, -1), [1, 1, tf.shape(mask)[-1]])\n        mask &= tf.transpose(mask, [0, 2, 1])\n        arc_scores, arcs = arc_scores[mask], arcs[mask]\n        rel_scores, rels = rel_scores[mask], rels[mask]\n        rel_scores, rels = rel_scores[arcs], rels[arcs]\n        arc_loss = arc_loss(arcs, arc_scores)\n        rel_loss = rel_loss(rels, rel_scores)\n        loss = arc_loss + rel_loss\n\n        return loss\n\n    def decode(self, arc_scores, rel_scores, mask):\n        arc_preds = arc_scores > 0\n        rel_preds = tf.argmax(rel_scores, -1)\n\n        return arc_preds, rel_preds\n\n\nclass BiaffineTransformerDependencyParserTF(BiaffineDependencyParserTF, tf.keras.callbacks.Callback):\n    def __init__(self, transform: CoNLL_Transformer_Transform = None) -> None:\n        if not transform:\n            transform = CoNLL_Transformer_Transform()\n        super().__init__(transform)\n        self.transform: CoNLL_Transformer_Transform = transform\n\n    def build_model(self, transformer, training, **kwargs) -> tf.keras.Model:\n        transformer = self.build_transformer(training, transformer)\n        model = BiaffineModelTF(self.config, transformer=transformer)\n        return model\n\n    def build_transformer(self, training, transformer):\n        if training:\n            self.config.n_words = len(self.transform.form_vocab)\n        self._init_config()\n        if isinstance(transformer, str):\n            if 'albert_chinese' in transformer:\n                tokenizer = BertTokenizerFast.from_pretrained(transformer, add_special_tokens=False)\n                transformer: TFPreTrainedModel = TFAutoModel.from_pretrained(transformer, name=transformer,\n                                                                             from_pt=True)\n            elif transformer.startswith('albert') and transformer.endswith('zh'):\n                transformer, tokenizer, path = build_transformer(transformer)\n                transformer.config = AlbertConfig.from_json_file(os.path.join(path, \"albert_config.json\"))\n                tokenizer = BertTokenizer.from_pretrained(os.path.join(path, \"vocab_chinese.txt\"),\n                                                          add_special_tokens=False)\n            elif 'chinese-roberta' in transformer:\n                tokenizer = BertTokenizer.from_pretrained(transformer)\n                transformer = TFBertModel.from_pretrained(transformer, name=transformer, from_pt=True)\n            else:\n                tokenizer: PreTrainedTokenizer = AutoTokenizer.from_pretrained(transformer)\n                try:\n                    transformer: TFPreTrainedModel = TFAutoModel.from_pretrained(transformer, name=transformer)\n                except (TypeError, OSError):\n                    transformer: TFPreTrainedModel = TFAutoModel.from_pretrained(transformer, name=transformer,\n                                                                                 from_pt=True)\n        elif transformer[0] == 'AutoModelWithLMHead':\n            tokenizer: PreTrainedTokenizer = AutoTokenizer.from_pretrained(transformer[1])\n            transformer: TFAutoModelWithLMHead = TFAutoModelWithLMHead.from_pretrained(transformer[1])\n        else:\n            raise ValueError(f'Unknown identifier {transformer}')\n        self.transform.tokenizer = tokenizer\n        if self.config.get('fp16', None) or self.config.get('use_amp', None):\n            policy = tf.keras.mixed_precision.experimental.Policy('mixed_float16')\n            tf.keras.mixed_precision.experimental.set_policy(policy)\n            # tf.config.optimizer.set_experimental_options({\"auto_mixed_precision\": True})\n            transformer.set_weights([w.astype('float16') for w in transformer.get_weights()])\n        self.transform.transformer_config = transformer.config\n        return transformer\n\n    # noinspection PyMethodOverriding\n    def fit(self, trn_data, dev_data, save_dir, transformer, max_seq_length=256, transformer_dropout=.33,\n            d_positional=None,\n            n_mlp_arc=500, n_mlp_rel=100, mlp_dropout=.33,\n            optimizer='adamw',\n            learning_rate=5e-5,\n            learning_rate_transformer=None,\n            weight_decay_rate=0,\n            epsilon=1e-8,\n            clipnorm=None,\n            fp16=False,\n            warmup_steps_ratio=0,\n            arc_loss='sparse_categorical_crossentropy', rel_loss='sparse_categorical_crossentropy',\n            metrics=('UAS', 'LAS'),\n            batch_size=3000,\n            samples_per_batch=150,\n            max_samples_per_batch=None,\n            epochs=100,\n            tree=False, punct=False, token_mapping=None, run_eagerly=False, logger=None, verbose=True, **kwargs):\n        self.set_params({})\n        return KerasComponent.fit(self, **merge_locals_kwargs(locals(), kwargs))\n\n    @property\n    def sample_data(self):\n        dataset = self.transform.inputs_to_dataset(\n            [[('Hello', 'NN'), ('world', 'NN')], [('HanLP', 'NN'), ('is', 'NN'), ('good', 'NN')]] if self.config.get(\n                'use_pos', None) else\n            [['Hello', 'world'], ['HanLP', 'is', 'good']])\n        return next(iter(dataset))[0]\n\n    # noinspection PyMethodOverriding\n    def build_optimizer(self, optimizer, learning_rate, epsilon, weight_decay_rate, clipnorm, fp16, train_steps,\n                        **kwargs):\n        if optimizer == 'adamw':\n            epochs = self.config['epochs']\n            learning_rate_transformer = kwargs.get('learning_rate_transformer', None)\n            train_steps = math.ceil(self.config.train_examples * epochs / self.config.samples_per_batch)\n            warmup_steps = math.ceil(train_steps * self.config['warmup_steps_ratio'])\n            if learning_rate_transformer is not None:\n                if learning_rate_transformer > 0:\n                    self.params['optimizer_transformer'] = build_adamw_optimizer(self.config, learning_rate_transformer,\n                                                                                 epsilon,\n                                                                                 clipnorm, train_steps, fp16,\n                                                                                 math.ceil(warmup_steps),\n                                                                                 weight_decay_rate)\n                else:\n                    self.model.transformer.trainable = False\n                return super().build_optimizer(lr=learning_rate)  # use a normal adam for biaffine\n            else:\n                return build_adamw_optimizer(self.config, learning_rate, epsilon, clipnorm, train_steps, fp16,\n                                             math.ceil(warmup_steps), weight_decay_rate)\n        return super().build_optimizer(optimizer, **kwargs)\n\n    def build_vocab(self, trn_data, logger):\n        self.config.train_examples = train_examples = super().build_vocab(trn_data, logger)\n        return train_examples\n\n    def build_callbacks(self, save_dir, logger, metrics, **kwargs):\n        callbacks = super().build_callbacks(save_dir, logger, metrics=metrics, **kwargs)\n        callbacks.append(self)\n        if not self.params:\n            self.set_params({})\n        return callbacks\n\n    def on_train_begin(self):\n        self.params['accum_grads'] = [tf.Variable(tf.zeros_like(tv.read_value()), trainable=False) for tv in\n                                      self.model.trainable_variables]\n        self.params['trained_samples'] = 0\n        self.params['transformer_variable_names'] = {x.name for x in self.model.transformer.trainable_variables}\n\n    def train_batch(self, words, feats, arcs, rels, mask, optimizer, arc_loss, rel_loss):\n        with tf.GradientTape() as tape:\n            arc_scores, rel_scores = self.model((words, feats), training=True)\n            loss = self.get_loss(arc_scores, rel_scores, arcs, rels, mask, arc_loss, rel_loss)\n        grads = tape.gradient(loss, self.model.trainable_variables)\n        accum_grads = self.params['accum_grads']\n        for i, grad in enumerate(grads):\n            if grad is not None:\n                accum_grads[i].assign_add(grad)\n        self.params['trained_samples'] += tf.shape(words)[0]\n        if self.params['trained_samples'] >= self.config.samples_per_batch:\n            self._apply_grads(accum_grads)\n        return loss, arc_scores, rel_scores\n\n    def _apply_grads(self, accum_grads):\n        optimizer_transformer = self.params.get('optimizer_transformer', None)\n        if optimizer_transformer:\n            transformer = self.params['transformer_variable_names']\n            trainable_variables = self.model.trainable_variables\n            optimizer_transformer.apply_gradients(\n                (g, w) for g, w in zip(accum_grads, trainable_variables) if w.name in transformer)\n            self.model.optimizer.apply_gradients(\n                (g, w) for g, w in zip(accum_grads, trainable_variables) if w.name not in transformer)\n        else:\n            self.model.optimizer.apply_gradients(zip(accum_grads, self.model.trainable_variables))\n        for tv in accum_grads:\n            tv.assign(tf.zeros_like(tv))\n        # print('Apply grads after', self.params['trained_samples'], 'samples')\n        self.params['trained_samples'] = 0\n\n    def on_epoch_end(self, epoch, logs=None):\n        if self.params['trained_samples']:\n            self._apply_grads(self.params['accum_grads'])\n\n\nclass BiaffineTransformerSemanticDependencyParser(BiaffineTransformerDependencyParserTF):\n\n    def __init__(self, transform: CoNLL_Transformer_Transform = None) -> None:\n        if not transform:\n            transform = CoNLL_Transformer_Transform(graph=True)\n        super().__init__(transform)\n\n    def get_loss(self, arc_scores, rel_scores, arcs, rels, mask, arc_loss, rel_loss):\n        return BiaffineSemanticDependencyParserTF.get_loss(self, arc_scores, rel_scores, arcs, rels, mask, arc_loss,\n                                                           rel_loss)\n\n    def fit(self, trn_data, dev_data, save_dir, transformer, max_seq_length=256, transformer_dropout=.33,\n            d_positional=None, n_mlp_arc=500, n_mlp_rel=100, mlp_dropout=.33, optimizer='adamw', learning_rate=5e-5,\n            learning_rate_transformer=None, weight_decay_rate=0, epsilon=1e-8, clipnorm=None, fp16=False,\n            warmup_steps_ratio=0, arc_loss='binary_crossentropy',\n            rel_loss='sparse_categorical_crossentropy', metrics=('UF', 'LF'), batch_size=3000, samples_per_batch=150,\n            max_samples_per_batch=None, epochs=100, tree=False, punct=False, token_mapping=None, enhanced_only=False,\n            run_eagerly=False,\n            logger=None, verbose=True, **kwargs):\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    def decode(self, arc_scores, rel_scores, mask):\n        return BiaffineSemanticDependencyParserTF.decode(self, arc_scores, rel_scores, mask)\n"
  },
  {
    "path": "hanlp/components/parsers/biaffine_tf/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-26 23:03"
  },
  {
    "path": "hanlp/components/parsers/biaffine_tf/alg.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-26 19:49\n# Ported from the PyTorch implementation https://github.com/zysite/biaffine-parser\nfrom typing import List\nimport numpy as np\nimport tensorflow as tf\nfrom collections import defaultdict\n\n\ndef nonzero(t: tf.Tensor) -> tf.Tensor:\n    return tf.where(t > 0)\n\n\ndef view(t: tf.Tensor, *dims) -> tf.Tensor:\n    return tf.reshape(t, dims)\n\n\ndef arange(n: int) -> tf.Tensor:\n    return tf.range(n)\n\n\ndef randperm(n: int) -> tf.Tensor:\n    return tf.random.shuffle(arange(n))\n\n\ndef tolist(t: tf.Tensor) -> List:\n    if isinstance(t, tf.Tensor):\n        t = t.numpy()\n    return t.tolist()\n\n\ndef kmeans(x, k, seed=None):\n    \"\"\"See https://github.com/zysite/biaffine-parser/blob/master/parser/utils/alg.py#L7\n\n    Args:\n      x(list): Lengths of sentences\n      k(int): \n      seed:  (Default value = None)\n\n    Returns:\n\n    \n    \"\"\"\n    x = tf.constant(x, dtype=tf.float32)\n    # count the frequency of each datapoint\n    d, indices, f = tf.unique_with_counts(x, tf.int32)\n    f = tf.cast(f, tf.float32)\n    # calculate the sum of the values of the same datapoints\n    total = d * f\n    # initialize k centroids randomly\n    c, old = tf.random.shuffle(d, seed)[:k], None\n    # assign labels to each datapoint based on centroids\n    dists = tf.abs(tf.expand_dims(d, -1) - c)\n    y = tf.argmin(dists, axis=-1, output_type=tf.int32)\n    dists = tf.gather_nd(dists, tf.transpose(tf.stack([tf.range(tf.shape(dists)[0], dtype=tf.int32), y])))\n    # make sure number of datapoints is greater than that of clusters\n    assert len(d) >= k, f\"unable to assign {len(d)} datapoints to {k} clusters\"\n\n    while old is None or not tf.reduce_all(c == old):\n        # if an empty cluster is encountered,\n        # choose the farthest datapoint from the biggest cluster\n        # and move that the empty one\n        for i in range(k):\n            if not tf.reduce_any(y == i):\n                mask = tf.cast(y == tf.expand_dims(tf.range(k, dtype=tf.int32), -1), tf.float32)\n                lens = tf.reduce_sum(mask, axis=-1)\n                biggest = view(nonzero(mask[tf.argmax(lens)]), -1)\n                farthest = tf.argmax(tf.gather(dists, biggest))\n                tf.tensor_scatter_nd_update(y, tf.expand_dims(tf.expand_dims(biggest[farthest], -1), -1), [i])\n        mask = tf.cast(y == tf.expand_dims(tf.range(k, dtype=tf.int32), -1), tf.float32)\n        # update the centroids\n        c, old = tf.cast(tf.reduce_sum(total * mask, axis=-1), tf.float32) / tf.cast(tf.reduce_sum(f * mask, axis=-1),\n                                                                                     tf.float32), c\n        # re-assign all datapoints to clusters\n        dists = tf.abs(tf.expand_dims(d, -1) - c)\n        y = tf.argmin(dists, axis=-1, output_type=tf.int32)\n        dists = tf.gather_nd(dists, tf.transpose(tf.stack([tf.range(tf.shape(dists)[0], dtype=tf.int32), y])))\n    # assign all datapoints to the new-generated clusters\n    # without considering the empty ones\n    y, (assigned, _) = tf.gather(y, indices), tf.unique(y)\n    # get the centroids of the assigned clusters\n    centroids = tf.gather(c, assigned).numpy().tolist()\n    # map all values of datapoints to buckets\n    clusters = [tf.squeeze(tf.where(y == i), axis=-1).numpy().tolist() for i in assigned]\n\n    return centroids, clusters\n\n\n# ***************************************************************\nclass Tarjan:\n    \"\"\"Computes Tarjan's algorithm for finding strongly connected components (cycles) of a graph\"\"\"\n\n    def __init__(self, prediction, tokens):\n        \"\"\"\n\n        Parameters\n        ----------\n        prediction : numpy.ndarray\n            a predicted dependency tree where prediction[dep_idx] = head_idx\n        tokens : numpy.ndarray\n            the tokens we care about (i.e. exclude _GO, _EOS, and _PAD)\n        \"\"\"\n        self._edges = defaultdict(set)\n        self._vertices = set((0,))\n        for dep, head in enumerate(prediction[tokens]):\n            self._vertices.add(dep + 1)\n            self._edges[head].add(dep + 1)\n        self._indices = {}\n        self._lowlinks = {}\n        self._onstack = defaultdict(lambda: False)\n        self._SCCs = []\n\n        index = 0\n        stack = []\n        for v in self.vertices:\n            if v not in self.indices:\n                self.strongconnect(v, index, stack)\n\n    # =============================================================\n    def strongconnect(self, v, index, stack):\n        \"\"\"\n\n        Args:\n          v: \n          index: \n          stack: \n\n        Returns:\n\n        \"\"\"\n\n        self._indices[v] = index\n        self._lowlinks[v] = index\n        index += 1\n        stack.append(v)\n        self._onstack[v] = True\n        for w in self.edges[v]:\n            if w not in self.indices:\n                self.strongconnect(w, index, stack)\n                self._lowlinks[v] = min(self._lowlinks[v], self._lowlinks[w])\n            elif self._onstack[w]:\n                self._lowlinks[v] = min(self._lowlinks[v], self._indices[w])\n\n        if self._lowlinks[v] == self._indices[v]:\n            self._SCCs.append(set())\n            while stack[-1] != v:\n                w = stack.pop()\n                self._onstack[w] = False\n                self._SCCs[-1].add(w)\n            w = stack.pop()\n            self._onstack[w] = False\n            self._SCCs[-1].add(w)\n        return\n\n    # ======================\n    @property\n    def edges(self):\n        return self._edges\n\n    @property\n    def vertices(self):\n        return self._vertices\n\n    @property\n    def indices(self):\n        return self._indices\n\n    @property\n    def SCCs(self):\n        return self._SCCs\n\n\ndef tarjan(parse_probs, length, tokens_to_keep, ensure_tree=True):\n    \"\"\"Adopted from Timothy Dozat https://github.com/tdozat/Parser/blob/master/lib/models/nn.py\n\n    Args:\n      parse_probs(NDArray): seq_len x seq_len, the probability of arcs\n      length(NDArray): sentence length including ROOT\n      tokens_to_keep(NDArray): mask matrix\n      ensure_tree:  (Default value = True)\n\n    Returns:\n\n    \n    \"\"\"\n    if ensure_tree:\n        I = np.eye(len(tokens_to_keep))\n        # block loops and pad heads\n        parse_probs = parse_probs * tokens_to_keep * (1 - I)\n        parse_preds = np.argmax(parse_probs, axis=1)\n        tokens = np.arange(1, length)\n        roots = np.where(parse_preds[tokens] == 0)[0] + 1\n        # ensure at least one root\n        if len(roots) < 1:\n            # The current root probabilities\n            root_probs = parse_probs[tokens, 0]\n            # The current head probabilities\n            old_head_probs = parse_probs[tokens, parse_preds[tokens]]\n            # Get new potential root probabilities\n            new_root_probs = root_probs / old_head_probs\n            # Select the most probable root\n            new_root = tokens[np.argmax(new_root_probs)]\n            # Make the change\n            parse_preds[new_root] = 0\n        # ensure at most one root\n        elif len(roots) > 1:\n            # The probabilities of the current heads\n            root_probs = parse_probs[roots, 0]\n            # Set the probability of depending on the root zero\n            parse_probs[roots, 0] = 0\n            # Get new potential heads and their probabilities\n            new_heads = np.argmax(parse_probs[roots][:, tokens], axis=1) + 1\n            new_head_probs = parse_probs[roots, new_heads] / root_probs\n            # Select the most probable root\n            new_root = roots[np.argmin(new_head_probs)]\n            # Make the change\n            parse_preds[roots] = new_heads\n            parse_preds[new_root] = 0\n        # remove cycles\n        tarjan = Tarjan(parse_preds, tokens)\n        for SCC in tarjan.SCCs:\n            if len(SCC) > 1:\n                dependents = set()\n                to_visit = set(SCC)\n                while len(to_visit) > 0:\n                    node = to_visit.pop()\n                    if not node in dependents:\n                        dependents.add(node)\n                        to_visit.update(tarjan.edges[node])\n                # The indices of the nodes that participate in the cycle\n                cycle = np.array(list(SCC))\n                # The probabilities of the current heads\n                old_heads = parse_preds[cycle]\n                old_head_probs = parse_probs[cycle, old_heads]\n                # Set the probability of depending on a non-head to zero\n                non_heads = np.array(list(dependents))\n                parse_probs[np.repeat(cycle, len(non_heads)), np.repeat([non_heads], len(cycle), axis=0).flatten()] = 0\n                # Get new potential heads and their probabilities\n                new_heads = np.argmax(parse_probs[cycle][:, tokens], axis=1) + 1\n                new_head_probs = parse_probs[cycle, new_heads] / old_head_probs\n                # Select the most probable change\n                change = np.argmax(new_head_probs)\n                changed_cycle = cycle[change]\n                old_head = old_heads[change]\n                new_head = new_heads[change]\n                # Make the change\n                parse_preds[changed_cycle] = new_head\n                tarjan.edges[new_head].add(changed_cycle)\n                tarjan.edges[old_head].remove(changed_cycle)\n        return parse_preds\n    else:\n        # block and pad heads\n        parse_probs = parse_probs * tokens_to_keep\n        parse_preds = np.argmax(parse_probs, axis=1)\n        return parse_preds\n\n\ndef rel_argmax(rel_probs, length, root, ensure_tree=True):\n    \"\"\"Fix the relation prediction by heuristic rules\n\n    Args:\n      rel_probs(NDArray): seq_len x rel_size\n      length: real sentence length\n      ensure_tree:  (Default value = True)\n      root: \n\n    Returns:\n\n    \n    \"\"\"\n    if ensure_tree:\n        tokens = np.arange(1, length)\n        rel_preds = np.argmax(rel_probs, axis=1)\n        roots = np.where(rel_preds[tokens] == root)[0] + 1\n        if len(roots) < 1:\n            rel_preds[1 + np.argmax(rel_probs[tokens, root])] = root\n        elif len(roots) > 1:\n            root_probs = rel_probs[roots, root]\n            rel_probs[roots, root] = 0\n            new_rel_preds = np.argmax(rel_probs[roots], axis=1)\n            new_rel_probs = rel_probs[roots, new_rel_preds] / root_probs\n            new_root = roots[np.argmin(new_rel_probs)]\n            rel_preds[roots] = new_rel_preds\n            rel_preds[new_root] = root\n        return rel_preds\n    else:\n        rel_preds = np.argmax(rel_probs, axis=1)\n        return rel_preds\n"
  },
  {
    "path": "hanlp/components/parsers/biaffine_tf/layers.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-26 23:05\n# Ported from the PyTorch implementation https://github.com/zysite/biaffine-parser\nimport tensorflow as tf\nfrom hanlp.utils.tf_util import tf_bernoulli\n\n\nclass Biaffine(tf.keras.layers.Layer):\n    def __init__(self, n_in, n_out=1, bias_x=True, bias_y=True, trainable=True, name=None, dtype=None, dynamic=False,\n                 **kwargs):\n        super().__init__(trainable, name, dtype, dynamic, **kwargs)\n        self.n_in = n_in\n        self.n_out = n_out\n        self.bias_x = bias_x\n        self.bias_y = bias_y\n        self.weight = None\n\n    def build(self, input_shape):\n        self.weight = self.add_weight(name='kernel',\n                                      shape=(self.n_out,\n                                             self.n_in + self.bias_x,\n                                             self.n_in + self.bias_y),\n                                      initializer='zero')\n\n    def extra_repr(self):\n        s = f\"n_in={self.n_in}, n_out={self.n_out}\"\n        if self.bias_x:\n            s += f\", bias_x={self.bias_x}\"\n        if self.bias_y:\n            s += f\", bias_y={self.bias_y}\"\n\n        return s\n\n    # noinspection PyMethodOverriding\n    def call(self, x, y, **kwargs):\n        if self.bias_x:\n            x = tf.concat((x, tf.ones_like(x[..., :1])), -1)\n        if self.bias_y:\n            y = tf.concat((y, tf.ones_like(y[..., :1])), -1)\n        # [batch_size, n_out, seq_len, seq_len]\n        s = tf.einsum('bxi,oij,byj->boxy', x, self.weight, y)\n        # remove dim 1 if n_out == 1\n        if self.n_out == 1:\n            s = tf.squeeze(s, axis=1)\n\n        return s\n\n\nclass MLP(tf.keras.layers.Layer):\n    def __init__(self, n_hidden, dropout=0, trainable=True, name=None, dtype=None, dynamic=False, **kwargs):\n        super().__init__(trainable, name, dtype, dynamic, **kwargs)\n        self.linear = tf.keras.layers.Dense(n_hidden, kernel_initializer='orthogonal')\n        self.activation = tf.keras.layers.LeakyReLU(0.1)\n        self.dropout = SharedDropout(p=dropout)\n\n    def call(self, x, **kwargs):\n        x = self.linear(x)\n        x = self.activation(x)\n        x = self.dropout(x)\n\n        return x\n\n\nclass SharedDropout(tf.keras.layers.Layer):\n\n    def __init__(self, p=0.5, batch_first=True, trainable=True, name=None, dtype=None, dynamic=False, **kwargs):\n        \"\"\"Dropout on timesteps with bernoulli distribution\"\"\"\n        super().__init__(trainable, name, dtype, dynamic, **kwargs)\n        self.p = p\n        self.batch_first = batch_first\n\n    def extra_repr(self):\n        s = f\"p={self.p}\"\n        if self.batch_first:\n            s += f\", batch_first={self.batch_first}\"\n\n        return s\n\n    def call(self, x, training=None, **kwargs):\n        if training and self.p > 0:\n            if self.batch_first:\n                mask = self.get_mask(x[:, 0], self.p)\n            else:\n                mask = self.get_mask(x[0], self.p)\n            x *= tf.expand_dims(mask, axis=1) if self.batch_first else mask\n\n        return x\n\n    @staticmethod\n    def get_mask(x, p):\n        mask = tf_bernoulli(tf.shape(x), 1 - p, x.dtype)\n        mask = mask / (1 - p)\n\n        return mask\n\n\nclass IndependentDropout(tf.keras.layers.Layer):\n\n    def __init__(self, p=0.5, trainable=True, name=None, dtype=None, dynamic=False, **kwargs):\n        \"\"\"Dropout on the first two dimensions\"\"\"\n        super().__init__(trainable, name, dtype, dynamic, **kwargs)\n        self.p = p\n\n    def extra_repr(self):\n        return f\"p={self.p}\"\n\n    def call(self, inputs, training=None, **kwargs):\n        if training and self.p > 0:\n            masks = [tf_bernoulli(tf.shape(x)[:2], 1 - self.p)\n                     for x in inputs]\n            total = sum(masks)\n            scale = len(inputs) / tf.reduce_max(tf.ones_like(total))\n            masks = [mask * scale for mask in masks]\n            inputs = [item * tf.expand_dims(mask, axis=-1)\n                      for item, mask in zip(inputs, masks)]\n\n        return inputs\n\n\n"
  },
  {
    "path": "hanlp/components/parsers/biaffine_tf/model.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-26 23:04\nimport tensorflow as tf\nfrom hanlp.layers.transformers.tf_imports import TFPreTrainedModel\nfrom hanlp.components.parsers.biaffine_tf.layers import IndependentDropout, SharedDropout, Biaffine, MLP\n\n\nclass BiaffineModelTF(tf.keras.Model):\n\n    def __init__(self, config, embed=None, transformer: TFPreTrainedModel = None):\n        \"\"\"An implementation of T. Dozat and C. D. Manning, “Deep Biaffine Attention for Neural Dependency Parsing.,” ICLR, 2017.\n            Although I have my MXNet implementation, I found zysite's PyTorch implementation is cleaner so I port it to TensorFlow\n\n        Args:\n          config: param embed:\n\n        Returns:\n\n        \"\"\"\n        super(BiaffineModelTF, self).__init__()\n        assert not (embed and transformer), 'Either pre-trained word embed and transformer is supported, but not both'\n        normal = tf.keras.initializers.RandomNormal(stddev=1.)\n        if not transformer:\n            # the embedding layer\n            self.word_embed = tf.keras.layers.Embedding(input_dim=config.n_words,\n                                                        output_dim=config.n_embed,\n                                                        embeddings_initializer=tf.keras.initializers.zeros() if embed\n                                                        else normal,\n                                                        name='word_embed')\n            self.feat_embed = tf.keras.layers.Embedding(input_dim=config.n_feats,\n                                                        output_dim=config.n_embed,\n                                                        embeddings_initializer=tf.keras.initializers.zeros() if embed\n                                                        else normal,\n                                                        name='feat_embed')\n            self.embed_dropout = IndependentDropout(p=config.embed_dropout, name='embed_dropout')\n\n            # the word-lstm layer\n            self.lstm = tf.keras.models.Sequential(name='lstm')\n            for _ in range(config.n_lstm_layers):\n                self.lstm.add(tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(\n                    units=config.n_lstm_hidden,\n                    dropout=config.lstm_dropout,\n                    recurrent_dropout=config.lstm_dropout,\n                    return_sequences=True,\n                    kernel_initializer='orthogonal',\n                    unit_forget_bias=False,  # turns out to hinder performance\n                )))\n            self.lstm_dropout = SharedDropout(p=config.lstm_dropout, name='lstm_dropout')\n        else:\n            self.transformer = transformer\n            transformer_dropout = config.get('transformer_dropout', None)\n            if transformer_dropout:\n                self.transformer_dropout = SharedDropout(p=config.transformer_dropout, name='transformer_dropout')\n            d_positional = config.get('d_positional', None)\n            if d_positional:\n                max_seq_length = config.get('max_seq_length', 256)\n                self.position_table = self.add_weight(shape=(max_seq_length, d_positional),\n                                                      initializer='random_normal',\n                                                      trainable=True)\n        # the MLP layers\n        self.mlp_arc_h = MLP(n_hidden=config.n_mlp_arc,\n                             dropout=config.mlp_dropout, name='mlp_arc_h')\n        self.mlp_arc_d = MLP(n_hidden=config.n_mlp_arc,\n                             dropout=config.mlp_dropout, name='mlp_arc_d')\n        self.mlp_rel_h = MLP(n_hidden=config.n_mlp_rel,\n                             dropout=config.mlp_dropout, name='mlp_rel_h')\n        self.mlp_rel_d = MLP(n_hidden=config.n_mlp_rel,\n                             dropout=config.mlp_dropout, name='mlp_rel_d')\n\n        # the Biaffine layers\n        self.arc_attn = Biaffine(n_in=config.n_mlp_arc,\n                                 bias_x=True,\n                                 bias_y=False, name='arc_attn')\n        self.rel_attn = Biaffine(n_in=config.n_mlp_rel,\n                                 n_out=config.n_rels,\n                                 bias_x=True,\n                                 bias_y=True, name='rel_attn')\n        if embed is not None:\n            self.pretrained = embed\n        self.pad_index = tf.constant(config.pad_index, dtype=tf.int64)\n        self.unk_index = tf.constant(config.unk_index, dtype=tf.int64)\n\n    # noinspection PyMethodOverriding\n    def call(self, inputs, mask_inf=True, **kwargs):\n        # batch_size, seq_len = words.shape\n        # get the mask and lengths of given batch\n        # mask = words.ne(self.pad_index)\n        if hasattr(self, 'lstm'):\n            words, feats = inputs\n            mask = tf.not_equal(words, self.pad_index)\n            # set the indices larger than num_embeddings to unk_index\n            # ext_mask = words.ge(self.word_embed.num_embeddings)\n            ext_mask = tf.greater_equal(words, self.word_embed.input_dim)\n            ext_words = tf.where(ext_mask, self.unk_index, words)\n\n            # get outputs from embedding layers\n            word_embed = self.word_embed(ext_words)\n            if hasattr(self, 'pretrained'):\n                word_embed += self.pretrained(words)\n            feat_embed = self.feat_embed(feats)\n            word_embed, feat_embed = self.embed_dropout([word_embed, feat_embed])\n            # concatenate the word and feat representations\n            embed = tf.concat((word_embed, feat_embed), axis=-1)\n\n            x = self.lstm(embed, mask=mask)\n            x = self.lstm_dropout(x)\n        else:\n            words, (input_ids, input_mask, prefix_offset) = inputs\n            mask = tf.not_equal(words, self.pad_index)\n            x = self.run_transformer(input_ids, input_mask, prefix_offset)\n\n        # apply MLPs to the BiLSTM output states\n        arc_h = self.mlp_arc_h(x)\n        arc_d = self.mlp_arc_d(x)\n        rel_h = self.mlp_rel_h(x)\n        rel_d = self.mlp_rel_d(x)\n\n        # get arc and rel scores from the bilinear attention\n        # [batch_size, seq_len, seq_len]\n        s_arc = self.arc_attn(arc_d, arc_h)\n        # [batch_size, seq_len, seq_len, n_rels]\n        s_rel = tf.transpose(self.rel_attn(rel_d, rel_h), [0, 2, 3, 1])\n        # set the scores that exceed the length of each sentence to -inf\n        if mask_inf:\n            s_arc = tf.where(tf.expand_dims(mask, 1), s_arc, float('-inf'))\n\n        return s_arc, s_rel\n\n    def run_transformer(self, input_ids, input_mask, prefix_offset):\n        if isinstance(self.transformer, TFPreTrainedModel):\n            sequence_output = self.transformer([input_ids, input_mask])\n            sequence_output = sequence_output[0]\n        else:\n            sequence_output = self.transformer([input_ids, tf.zeros_like(input_ids)], mask=input_mask)\n        x = tf.gather(sequence_output, prefix_offset, batch_dims=1)\n        if hasattr(self, 'transformer_dropout'):\n            x = self.transformer_dropout(x)\n        if hasattr(self, 'position_table'):\n            batch_size, seq_length = tf.shape(x)[:2]\n            timing_signal = tf.broadcast_to(self.position_table[:seq_length],\n                                            [batch_size, seq_length, self.position_table.shape[-1]])\n            x = tf.concat([x, timing_signal], axis=-1)\n        return x\n\n    def to_functional(self):\n        words = tf.keras.Input(shape=[None], dtype=tf.int64, name='words')\n        feats = tf.keras.Input(shape=[None], dtype=tf.int64, name='feats')\n        s_arc, s_rel = self.call([words, feats], mask_inf=False)\n        return tf.keras.Model(inputs=[words, feats], outputs=[s_arc, s_rel])\n"
  },
  {
    "path": "hanlp/components/parsers/chu_liu_edmonds.py",
    "content": "# Adopted from https://github.com/allenai/allennlp under Apache Licence 2.0.\n# Changed the packaging.\n\nfrom typing import List, Set, Tuple, Dict\nimport numpy\n\n\ndef decode_mst(\n        energy: numpy.ndarray, length: int, has_labels: bool = True\n) -> Tuple[numpy.ndarray, numpy.ndarray]:\n    \"\"\"Note: Counter to typical intuition, this function decodes the _maximum_\n    spanning tree.\n    \n    Decode the optimal MST tree with the Chu-Liu-Edmonds algorithm for\n    maximum spanning arborescences on graphs.\n    \n    Adopted from https://github.com/allenai/allennlp/blob/master/allennlp/nn/chu_liu_edmonds.py\n    which is licensed under the Apache License 2.0\n    \n    # Parameters\n    \n    energy : `numpy.ndarray`, required.\n        A tensor with shape (num_labels, timesteps, timesteps)\n        containing the energy of each edge. If has_labels is `False`,\n        the tensor should have shape (timesteps, timesteps) instead.\n    length : `int`, required.\n        The length of this sequence, as the energy may have come\n        from a padded batch.\n    has_labels : `bool`, optional, (default = True)\n        Whether the graph has labels or not.\n\n    Args:\n      energy: numpy.ndarray: \n      length: int: \n      has_labels: bool:  (Default value = True)\n\n    Returns:\n\n    \"\"\"\n    if has_labels and energy.ndim != 3:\n        raise ValueError(\"The dimension of the energy array is not equal to 3.\")\n    elif not has_labels and energy.ndim != 2:\n        raise ValueError(\"The dimension of the energy array is not equal to 2.\")\n    input_shape = energy.shape\n    max_length = input_shape[-1]\n\n    # Our energy matrix might have been batched -\n    # here we clip it to contain only non padded tokens.\n    if has_labels:\n        energy = energy[:, :length, :length]\n        # get best label for each edge.\n        label_id_matrix = energy.argmax(axis=0)\n        energy = energy.max(axis=0)\n    else:\n        energy = energy[:length, :length]\n        label_id_matrix = None\n    # get original score matrix\n    original_score_matrix = energy\n    # initialize score matrix to original score matrix\n    score_matrix = numpy.array(original_score_matrix, copy=True)\n\n    old_input = numpy.zeros([length, length], dtype=numpy.int32)\n    old_output = numpy.zeros([length, length], dtype=numpy.int32)\n    current_nodes = [True for _ in range(length)]\n    representatives: List[Set[int]] = []\n\n    for node1 in range(length):\n        original_score_matrix[node1, node1] = 0.0\n        score_matrix[node1, node1] = 0.0\n        representatives.append({node1})\n\n        for node2 in range(node1 + 1, length):\n            old_input[node1, node2] = node1\n            old_output[node1, node2] = node2\n\n            old_input[node2, node1] = node2\n            old_output[node2, node1] = node1\n\n    final_edges: Dict[int, int] = {}\n\n    # The main algorithm operates inplace.\n    chu_liu_edmonds(\n        length, score_matrix, current_nodes, final_edges, old_input, old_output, representatives\n    )\n\n    heads = numpy.zeros([max_length], numpy.int32)\n    if has_labels:\n        head_type = numpy.ones([max_length], numpy.int32)\n    else:\n        head_type = None\n\n    for child, parent in final_edges.items():\n        heads[child] = parent\n        if has_labels:\n            head_type[child] = label_id_matrix[parent, child]\n\n    return heads, head_type\n\n\ndef chu_liu_edmonds(\n        length: int,\n        score_matrix: numpy.ndarray,\n        current_nodes: List[bool],\n        final_edges: Dict[int, int],\n        old_input: numpy.ndarray,\n        old_output: numpy.ndarray,\n        representatives: List[Set[int]],\n):\n    \"\"\"Applies the chu-liu-edmonds algorithm recursively\n    to a graph with edge weights defined by score_matrix.\n    \n    Note that this function operates in place, so variables\n    will be modified.\n    \n    # Parameters\n    \n    length : `int`, required.\n        The number of nodes.\n    score_matrix : `numpy.ndarray`, required.\n        The score matrix representing the scores for pairs\n        of nodes.\n    current_nodes : `List[bool]`, required.\n        The nodes which are representatives in the graph.\n        A representative at it's most basic represents a node,\n        but as the algorithm progresses, individual nodes will\n        represent collapsed cycles in the graph.\n    final_edges : `Dict[int, int]`, required.\n        An empty dictionary which will be populated with the\n        nodes which are connected in the maximum spanning tree.\n    old_input : `numpy.ndarray`, required.\n    old_output : `numpy.ndarray`, required.\n    representatives : `List[Set[int]]`, required.\n        A list containing the nodes that a particular node\n        is representing at this iteration in the graph.\n    \n    # Returns\n    \n    Nothing - all variables are modified in place.\n\n    Args:\n      length: int: \n      score_matrix: numpy.ndarray: \n      current_nodes: List[bool]: \n      final_edges: Dict[int: \n      int]: \n      old_input: numpy.ndarray: \n      old_output: numpy.ndarray: \n      representatives: List[Set[int]]: \n\n    Returns:\n\n    \"\"\"\n    # Set the initial graph to be the greedy best one.\n    parents = [-1]\n    for node1 in range(1, length):\n        parents.append(0)\n        if current_nodes[node1]:\n            max_score = score_matrix[0, node1]\n            for node2 in range(1, length):\n                if node2 == node1 or not current_nodes[node2]:\n                    continue\n\n                new_score = score_matrix[node2, node1]\n                if new_score > max_score:\n                    max_score = new_score\n                    parents[node1] = node2\n\n    # Check if this solution has a cycle.\n    has_cycle, cycle = _find_cycle(parents, length, current_nodes)\n    # If there are no cycles, find all edges and return.\n    if not has_cycle:\n        final_edges[0] = -1\n        for node in range(1, length):\n            if not current_nodes[node]:\n                continue\n\n            parent = old_input[parents[node], node]\n            child = old_output[parents[node], node]\n            final_edges[child] = parent\n        return\n\n    # Otherwise, we have a cycle so we need to remove an edge.\n    # From here until the recursive call is the contraction stage of the algorithm.\n    cycle_weight = 0.0\n    # Find the weight of the cycle.\n    index = 0\n    for node in cycle:\n        index += 1\n        cycle_weight += score_matrix[parents[node], node]\n\n    # For each node in the graph, find the maximum weight incoming\n    # and outgoing edge into the cycle.\n    cycle_representative = cycle[0]\n    for node in range(length):\n        if not current_nodes[node] or node in cycle:\n            continue\n\n        in_edge_weight = float(\"-inf\")\n        in_edge = -1\n        out_edge_weight = float(\"-inf\")\n        out_edge = -1\n\n        for node_in_cycle in cycle:\n            if score_matrix[node_in_cycle, node] > in_edge_weight:\n                in_edge_weight = score_matrix[node_in_cycle, node]\n                in_edge = node_in_cycle\n\n            # Add the new edge score to the cycle weight\n            # and subtract the edge we're considering removing.\n            score = (\n                    cycle_weight\n                    + score_matrix[node, node_in_cycle]\n                    - score_matrix[parents[node_in_cycle], node_in_cycle]\n            )\n\n            if score > out_edge_weight:\n                out_edge_weight = score\n                out_edge = node_in_cycle\n\n        score_matrix[cycle_representative, node] = in_edge_weight\n        old_input[cycle_representative, node] = old_input[in_edge, node]\n        old_output[cycle_representative, node] = old_output[in_edge, node]\n\n        score_matrix[node, cycle_representative] = out_edge_weight\n        old_output[node, cycle_representative] = old_output[node, out_edge]\n        old_input[node, cycle_representative] = old_input[node, out_edge]\n\n    # For the next recursive iteration, we want to consider the cycle as a\n    # single node. Here we collapse the cycle into the first node in the\n    # cycle (first node is arbitrary), set all the other nodes not be\n    # considered in the next iteration. We also keep track of which\n    # representatives we are considering this iteration because we need\n    # them below to check if we're done.\n    considered_representatives: List[Set[int]] = []\n    for i, node_in_cycle in enumerate(cycle):\n        considered_representatives.append(set())\n        if i > 0:\n            # We need to consider at least one\n            # node in the cycle, arbitrarily choose\n            # the first.\n            current_nodes[node_in_cycle] = False\n\n        for node in representatives[node_in_cycle]:\n            considered_representatives[i].add(node)\n            if i > 0:\n                representatives[cycle_representative].add(node)\n\n    chu_liu_edmonds(\n        length, score_matrix, current_nodes, final_edges, old_input, old_output, representatives\n    )\n\n    # Expansion stage.\n    # check each node in cycle, if one of its representatives\n    # is a key in the final_edges, it is the one we need.\n    found = False\n    key_node = -1\n    for i, node in enumerate(cycle):\n        for cycle_rep in considered_representatives[i]:\n            if cycle_rep in final_edges:\n                key_node = node\n                found = True\n                break\n        if found:\n            break\n\n    previous = parents[key_node]\n    while previous != key_node:\n        child = old_output[parents[previous], previous]\n        parent = old_input[parents[previous], previous]\n        final_edges[child] = parent\n        previous = parents[previous]\n\n\ndef _find_cycle(\n        parents: List[int], length: int, current_nodes: List[bool]\n) -> Tuple[bool, List[int]]:\n    added = [False for _ in range(length)]\n    added[0] = True\n    cycle = set()\n    has_cycle = False\n    for i in range(1, length):\n        if has_cycle:\n            break\n        # don't redo nodes we've already\n        # visited or aren't considering.\n        if added[i] or not current_nodes[i]:\n            continue\n        # Initialize a new possible cycle.\n        this_cycle = set()\n        this_cycle.add(i)\n        added[i] = True\n        has_cycle = True\n        next_node = i\n        while parents[next_node] not in this_cycle:\n            next_node = parents[next_node]\n            # If we see a node we've already processed,\n            # we can stop, because the node we are\n            # processing would have been in that cycle.\n            if added[next_node]:\n                has_cycle = False\n                break\n            added[next_node] = True\n            this_cycle.add(next_node)\n\n        if has_cycle:\n            original = next_node\n            cycle.add(original)\n            next_node = parents[original]\n            while next_node != original:\n                cycle.add(next_node)\n                next_node = parents[next_node]\n            break\n\n    return has_cycle, list(cycle)\n"
  },
  {
    "path": "hanlp/components/parsers/conll.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-26 15:37\nfrom typing import Union\n\nfrom hanlp.utils.io_util import get_resource, TimingFileIterator\nfrom hanlp.utils.log_util import logger\n\n\ndef collapse_enhanced_empty_nodes(sent: list):\n    collapsed = []\n    for cells in sent:\n        if isinstance(cells[0], float):\n            id = cells[0]\n            head, deprel = cells[8].split(':', 1)\n            for x in sent:\n                arrows = [s.split(':', 1) for s in x[8].split('|')]\n                arrows = [(head, f'{head}:{deprel}>{r}') if h == str(id) else (h, r) for h, r in arrows]\n                arrows = sorted(arrows)\n                x[8] = '|'.join(f'{h}:{r}' for h, r in arrows)\n            sent[head][7] += f'>{cells[7]}'\n        else:\n            collapsed.append(cells)\n    return collapsed\n\n\ndef read_conll(filepath: Union[str, TimingFileIterator], underline_to_none=False, enhanced_collapse_empty_nodes=False):\n    sent = []\n    if isinstance(filepath, str):\n        filepath: str = get_resource(filepath)\n        if filepath.endswith('.conllu') and enhanced_collapse_empty_nodes is None:\n            enhanced_collapse_empty_nodes = True\n        src = open(filepath, encoding='utf-8')\n    else:\n        src = filepath\n    for idx, line in enumerate(src):\n        if line.startswith('#'):\n            continue\n        line = line.strip()\n        cells = line.split('\\t')\n        if line and cells:\n            if enhanced_collapse_empty_nodes and '.' in cells[0]:\n                cells[0] = float(cells[0])\n                cells[6] = None\n            else:\n                if '-' in cells[0] or '.' in cells[0]:\n                    # sent[-1][1] += cells[1]\n                    continue\n                cells[0] = int(cells[0])\n                if cells[6] != '_':\n                    try:\n                        cells[6] = int(cells[6])\n                    except ValueError:\n                        cells[6] = 0\n                        logger.exception(f'Wrong CoNLL format {filepath}:{idx + 1}\\n{line}')\n            if underline_to_none:\n                for i, x in enumerate(cells):\n                    if x == '_':\n                        cells[i] = None\n            sent.append(cells)\n        else:\n            if enhanced_collapse_empty_nodes:\n                sent = collapse_enhanced_empty_nodes(sent)\n            yield sent\n            sent = []\n\n    if sent:\n        if enhanced_collapse_empty_nodes:\n            sent = collapse_enhanced_empty_nodes(sent)\n        yield sent\n\n    src.close()\n\n"
  },
  {
    "path": "hanlp/components/parsers/constituency/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-11-28 19:26\n"
  },
  {
    "path": "hanlp/components/parsers/constituency/crf_constituency_model.py",
    "content": "# -*- coding:utf-8 -*-\n# Adopted from https://github.com/yzhangcs/parser\n# MIT License\n#\n# Copyright (c) 2020 Yu Zhang\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\nimport torch\nfrom torch import nn\nfrom hanlp.components.parsers.constituency.treecrf import CRFConstituency\nfrom hanlp.components.parsers.alg import cky\nfrom hanlp.components.parsers.biaffine.biaffine import Biaffine\nfrom hanlp.components.parsers.biaffine.mlp import MLP\n\n\nclass CRFConstituencyDecoder(nn.Module):\n    r\"\"\"\n    The implementation of CRF Constituency Parser,\n    also called FANCY (abbr. of Fast and Accurate Neural Crf constituencY) Parser.\n\n    References:\n        - Yu Zhang, Houquan Zhou and Zhenghua Li. 2020.\n          `Fast and Accurate Neural CRF Constituency Parsing`_.\n\n    Args:\n        n_words (int):\n            The size of the word vocabulary.\n        n_feats (int):\n            The size of the feat vocabulary.\n        n_labels (int):\n            The number of labels.\n        feat (str):\n            Specifies which type of additional feature to use: ``'char'`` | ``'bert'`` | ``'tag'``.\n            ``'char'``: Character-level representations extracted by CharLSTM.\n            ``'bert'``: BERT representations, other pretrained langugae models like XLNet are also feasible.\n            ``'tag'``: POS tag embeddings.\n            Default: 'char'.\n        n_embed (int):\n            The size of word embeddings. Default: 100.\n        n_feat_embed (int):\n            The size of feature representations. Default: 100.\n        n_char_embed (int):\n            The size of character embeddings serving as inputs of CharLSTM, required if ``feat='char'``. Default: 50.\n        bert (str):\n            Specifies which kind of language model to use, e.g., ``'bert-base-cased'`` and ``'xlnet-base-cased'``.\n            This is required if ``feat='bert'``. The full list can be found in `transformers`.\n            Default: ``None``.\n        n_bert_layers (int):\n            Specifies how many last layers to use. Required if ``feat='bert'``.\n            The final outputs would be the weight sum of the hidden states of these layers.\n            Default: 4.\n        mix_dropout (float):\n            The dropout ratio of BERT layers. Required if ``feat='bert'``. Default: .0.\n        embed_dropout (float):\n            The dropout ratio of input embeddings. Default: .33.\n        n_hidden (int):\n            The size of LSTM hidden states. Default: 400.\n        n_lstm_layers (int):\n            The number of LSTM layers. Default: 3.\n        lstm_dropout (float):\n            The dropout ratio of LSTM. Default: .33.\n        n_mlp_span (int):\n            Span MLP size. Default: 500.\n        n_mlp_label  (int):\n            Label MLP size. Default: 100.\n        mlp_dropout (float):\n            The dropout ratio of MLP layers. Default: .33.\n        feat_pad_index (int):\n            The index of the padding token in the feat vocabulary. Default: 0.\n        pad_index (int):\n            The index of the padding token in the word vocabulary. Default: 0.\n        unk_index (int):\n            The index of the unknown token in the word vocabulary. Default: 1.\n\n    .. _Fast and Accurate Neural CRF Constituency Parsing:\n        https://www.ijcai.org/Proceedings/2020/560/\n    .. _transformers:\n        https://github.com/huggingface/transformers\n    \"\"\"\n\n    def __init__(self,\n                 n_labels,\n                 n_hidden=400,\n                 n_mlp_span=500,\n                 n_mlp_label=100,\n                 mlp_dropout=.33,\n                 **kwargs\n                 ):\n        super().__init__()\n\n        # the MLP layers\n        self.mlp_span_l = MLP(n_in=n_hidden, n_out=n_mlp_span, dropout=mlp_dropout)\n        self.mlp_span_r = MLP(n_in=n_hidden, n_out=n_mlp_span, dropout=mlp_dropout)\n        self.mlp_label_l = MLP(n_in=n_hidden, n_out=n_mlp_label, dropout=mlp_dropout)\n        self.mlp_label_r = MLP(n_in=n_hidden, n_out=n_mlp_label, dropout=mlp_dropout)\n\n        # the Biaffine layers\n        self.span_attn = Biaffine(n_in=n_mlp_span, bias_x=True, bias_y=False)\n        self.label_attn = Biaffine(n_in=n_mlp_label, n_out=n_labels, bias_x=True, bias_y=True)\n        self.crf = CRFConstituency()\n        self.criterion = nn.CrossEntropyLoss()\n\n    def forward(self, x, **kwargs):\n        r\"\"\"\n        Args:\n            x (~torch.FloatTensor): ``[batch_size, seq_len, hidden_dim]``.\n                Hidden states from encoder.\n\n        Returns:\n            ~torch.Tensor, ~torch.Tensor:\n                The first tensor of shape ``[batch_size, seq_len, seq_len]`` holds scores of all possible spans.\n                The second of shape ``[batch_size, seq_len, seq_len, n_labels]`` holds\n                scores of all possible labels on each span.\n        \"\"\"\n\n        x_f, x_b = x.chunk(2, -1)\n        x = torch.cat((x_f[:, :-1], x_b[:, 1:]), -1)\n        # apply MLPs to the BiLSTM output states\n        span_l = self.mlp_span_l(x)\n        span_r = self.mlp_span_r(x)\n        label_l = self.mlp_label_l(x)\n        label_r = self.mlp_label_r(x)\n\n        # [batch_size, seq_len, seq_len]\n        s_span = self.span_attn(span_l, span_r)\n        # [batch_size, seq_len, seq_len, n_labels]\n        s_label = self.label_attn(label_l, label_r).permute(0, 2, 3, 1)\n\n        return s_span, s_label\n\n    def loss(self, s_span, s_label, charts, mask, mbr=True):\n        r\"\"\"\n        Args:\n            s_span (~torch.Tensor): ``[batch_size, seq_len, seq_len]``.\n                Scores of all spans\n            s_label (~torch.Tensor): ``[batch_size, seq_len, seq_len, n_labels]``.\n                Scores of all labels on each span.\n            charts (~torch.LongTensor): ``[batch_size, seq_len, seq_len]``.\n                The tensor of gold-standard labels, in which positions without labels are filled with -1.\n            mask (~torch.BoolTensor): ``[batch_size, seq_len, seq_len]``.\n                The mask for covering the unpadded tokens in each chart.\n            mbr (bool):\n                If ``True``, returns marginals for MBR decoding. Default: ``True``.\n\n        Returns:\n            ~torch.Tensor, ~torch.Tensor:\n                The training loss and\n                original span scores of shape ``[batch_size, seq_len, seq_len]`` if ``mbr=False``, or marginals otherwise.\n        \"\"\"\n\n        span_mask = charts.ge(0) & mask\n        span_loss, span_probs = self.crf(s_span, mask, span_mask, mbr)\n        label_loss = self.criterion(s_label[span_mask], charts[span_mask])\n        loss = span_loss + label_loss\n\n        return loss, span_probs\n\n    def decode(self, s_span, s_label, mask):\n        r\"\"\"\n        Args:\n            s_span (~torch.Tensor): ``[batch_size, seq_len, seq_len]``.\n                Scores of all spans.\n            s_label (~torch.Tensor): ``[batch_size, seq_len, seq_len, n_labels]``.\n                Scores of all labels on each span.\n            mask (~torch.BoolTensor): ``[batch_size, seq_len, seq_len]``.\n                The mask for covering the unpadded tokens in each chart.\n\n        Returns:\n            list[list[tuple]]:\n                Sequences of factorized labeled trees traversed in pre-order.\n        \"\"\"\n\n        span_preds = cky(s_span, mask)\n        label_preds = s_label.argmax(-1).tolist()\n        return [[(i, j, labels[i][j]) for i, j in spans] for spans, labels in zip(span_preds, label_preds)]\n\n\nclass CRFConstituencyModel(nn.Module):\n\n    def __init__(self, encoder, decoder: CRFConstituencyDecoder) -> None:\n        super().__init__()\n        self.encoder = encoder\n        self.decoder = decoder\n\n    def forward(self, batch):\n        r\"\"\"\n        Args:\n            batch (~dict):\n                Batch of input data.\n\n        Returns:\n            ~torch.Tensor, ~torch.Tensor:\n                The first tensor of shape ``[batch_size, seq_len, seq_len]`` holds scores of all possible spans.\n                The second of shape ``[batch_size, seq_len, seq_len, n_labels]`` holds\n                scores of all possible labels on each span.\n        \"\"\"\n        x = self.encoder(batch)\n        return self.decoder(x)\n"
  },
  {
    "path": "hanlp/components/parsers/constituency/crf_constituency_parser.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-11-28 21:24\nimport logging\nfrom typing import Union, List\n\nimport torch\nfrom phrasetree.tree import Tree\nfrom torch.utils.data import DataLoader\n\nfrom hanlp_common.constant import BOS, EOS, IDX\nfrom hanlp.common.dataset import TransformableDataset, SamplerBuilder, PadSequenceDataLoader\nfrom hanlp.common.structure import History\nfrom hanlp.common.torch_component import TorchComponent\nfrom hanlp.common.transform import FieldLength, TransformList\nfrom hanlp.common.vocab import VocabWithNone\nfrom hanlp.components.classifiers.transformer_classifier import TransformerComponent\nfrom hanlp.datasets.parsing.loaders.constituency_dataset import ConstituencyDataset, unpack_tree_to_features, \\\n    build_tree, factorize, remove_subcategory\nfrom hanlp.components.parsers.constituency.crf_constituency_model import CRFConstituencyDecoder, CRFConstituencyModel\nfrom hanlp.metrics.parsing.span import SpanMetric\nfrom hanlp.utils.time_util import CountdownTimer\nfrom hanlp.utils.torch_util import clip_grad_norm\nfrom hanlp_common.util import merge_locals_kwargs, merge_dict, reorder\n\n\nclass CRFConstituencyParser(TorchComponent):\n    def __init__(self, **kwargs) -> None:\n        \"\"\"Two-stage CRF Parsing (:cite:`ijcai2020-560`).\n\n        Args:\n            **kwargs: Predefined config.\n        \"\"\"\n        super().__init__(**kwargs)\n        self.model: CRFConstituencyModel = self.model\n\n    def build_optimizer(self, trn, **kwargs):\n        # noinspection PyCallByClass,PyTypeChecker\n        return TransformerComponent.build_optimizer(self, trn, **kwargs)\n\n    def build_criterion(self, decoder=None, **kwargs):\n        return decoder\n\n    def build_metric(self, **kwargs):\n        return SpanMetric()\n\n    def execute_training_loop(self, trn: DataLoader, dev: DataLoader, epochs, criterion, optimizer, metric, save_dir,\n                              logger: logging.Logger, devices, ratio_width=None, patience=0.5, eval_trn=True, **kwargs):\n        if isinstance(patience, float):\n            patience = int(patience * epochs)\n        best_epoch, best_metric = 0, -1\n        timer = CountdownTimer(epochs)\n        history = History()\n        for epoch in range(1, epochs + 1):\n            logger.info(f\"[yellow]Epoch {epoch} / {epochs}:[/yellow]\")\n            self.fit_dataloader(trn, criterion, optimizer, metric, logger, history=history, ratio_width=ratio_width,\n                                eval_trn=eval_trn, **self.config)\n            loss, dev_metric = self.evaluate_dataloader(dev, criterion, logger=logger, ratio_width=ratio_width)\n            timer.update()\n            report = f\"{timer.elapsed_human} / {timer.total_time_human} ETA: {timer.eta_human}\"\n            if dev_metric > best_metric:\n                best_epoch, best_metric = epoch, dev_metric\n                self.save_weights(save_dir)\n                report += ' [red](saved)[/red]'\n            else:\n                report += f' ({epoch - best_epoch})'\n                if epoch - best_epoch >= patience:\n                    report += ' early stop'\n            logger.info(report)\n            if epoch - best_epoch >= patience:\n                break\n        if not best_epoch:\n            self.save_weights(save_dir)\n        elif best_epoch != epoch:\n            self.load_weights(save_dir)\n        logger.info(f\"Max score of dev is {best_metric} at epoch {best_epoch}\")\n        logger.info(f\"Average time of each epoch is {timer.elapsed_average_human}\")\n        logger.info(f\"{timer.elapsed_human} elapsed\")\n\n    # noinspection PyMethodOverriding\n    def fit_dataloader(self,\n                       trn: DataLoader,\n                       criterion,\n                       optimizer,\n                       metric: SpanMetric,\n                       logger: logging.Logger,\n                       history: History,\n                       gradient_accumulation=1,\n                       grad_norm=None,\n                       ratio_width=None,\n                       eval_trn=True,\n                       **kwargs):\n        optimizer, scheduler = optimizer\n        metric.reset()\n        self.model.train()\n        timer = CountdownTimer(history.num_training_steps(len(trn), gradient_accumulation=gradient_accumulation))\n        total_loss = 0\n        for idx, batch in enumerate(trn):\n            out, mask = self.feed_batch(batch)\n            y = batch['chart_id']\n            loss, span_probs = self.compute_loss(out, y, mask)\n            if gradient_accumulation and gradient_accumulation > 1:\n                loss /= gradient_accumulation\n            loss.backward()\n            total_loss += loss.item()\n            if eval_trn:\n                prediction = self.decode_output(out, mask, batch, span_probs)\n                self.update_metrics(metric, batch, prediction)\n            if history.step(gradient_accumulation):\n                self._step(optimizer, scheduler, grad_norm)\n                report = f'loss: {total_loss / (idx + 1):.4f} {metric}' if eval_trn \\\n                    else f'loss: {total_loss / (idx + 1):.4f}'\n                timer.log(report, logger=logger, ratio_percentage=False, ratio_width=ratio_width)\n            del loss\n            del out\n            del mask\n\n    def decode_output(self, out, mask, batch, span_probs=None, decoder=None, tokens=None):\n        s_span, s_label = out\n        if not decoder:\n            decoder = self.model.decoder\n        if mask.any().item():\n            if span_probs is None:\n                if self.config.mbr:\n                    s_span = decoder.crf(s_span, mask, mbr=True)\n            else:\n                s_span = span_probs\n            chart_preds = decoder.decode(s_span, s_label, mask)\n        else:\n            chart_preds = [[]] * len(tokens)\n        idx_to_token = self.vocabs.chart.idx_to_token\n        if tokens is None:\n            tokens = batch.get('token_', None)  # Use the original tokens if any\n            if tokens is None:\n                tokens = batch['token']\n            tokens = [x[1:-1] for x in tokens]\n        trees = [build_tree(token, [(i, j, idx_to_token[label]) for i, j, label in chart]) for token, chart in\n                 zip(tokens, chart_preds)]\n        # probs = [prob[:i - 1, 1:i].cpu() for i, prob in zip(lens, s_span.unbind())]\n        return trees\n\n    def update_metrics(self, metric, batch, prediction):\n        # Add pre-terminals (pos tags) back to prediction for safe factorization (deletion based on pos)\n        for pred, gold in zip(prediction, batch['constituency']):\n            pred: Tree = pred\n            gold: Tree = gold\n            for p, g in zip(pred.subtrees(lambda t: t.height() == 2), gold.pos()):\n                token, pos = g\n                p: Tree = p\n                assert p.label() == '_'\n                p.set_label(pos)\n        metric([factorize(tree, self.config.delete, self.config.equal) for tree in prediction],\n               [factorize(tree, self.config.delete, self.config.equal) for tree in batch['constituency']])\n        return metric\n\n    def feed_batch(self, batch: dict):\n        mask = self.compute_mask(batch)\n        s_span, s_label = self.model(batch)\n        return (s_span, s_label), mask\n\n    def compute_mask(self, batch, offset=1):\n        lens = batch['token_length'] - offset\n        seq_len = lens.max()\n        mask = lens.new_tensor(range(seq_len)) < lens.view(-1, 1, 1)\n        mask = mask & mask.new_ones(seq_len, seq_len).triu_(1)\n        return mask\n\n    def compute_loss(self, out, y, mask, crf_decoder=None):\n        if not crf_decoder:\n            crf_decoder = self.model.decoder\n        loss, span_probs = crf_decoder.loss(out[0], out[1], y, mask, self.config.mbr)\n        if loss < 0:  # wired negative loss\n            loss *= 0\n        return loss, span_probs\n\n    def _step(self, optimizer, scheduler, grad_norm):\n        clip_grad_norm(self.model, grad_norm)\n        optimizer.step()\n        scheduler.step()\n        optimizer.zero_grad()\n\n    @torch.no_grad()\n    def evaluate_dataloader(self, data, criterion, logger=None, ratio_width=None, metric=None, output=None, **kwargs):\n        self.model.eval()\n        total_loss = 0\n        if not metric:\n            metric = self.build_metric()\n        else:\n            metric.reset()\n        timer = CountdownTimer(len(data))\n        for idx, batch in enumerate(data):\n            out, mask = self.feed_batch(batch)\n            y = batch['chart_id']\n            loss, span_probs = self.compute_loss(out, y, mask)\n            total_loss += loss.item()\n            prediction = self.decode_output(out, mask, batch, span_probs)\n            self.update_metrics(metric, batch, prediction)\n            timer.log(f'loss: {total_loss / (idx + 1):.4f} {metric}', ratio_percentage=False, logger=logger,\n                      ratio_width=ratio_width)\n        total_loss /= len(data)\n        if output:\n            output.close()\n        return total_loss, metric\n\n    # noinspection PyMethodOverriding\n    def build_model(self, encoder, training=True, **kwargs) -> torch.nn.Module:\n        decoder = CRFConstituencyDecoder(n_labels=len(self.vocabs.chart), n_hidden=encoder.get_output_dim(), **kwargs)\n        encoder = encoder.module(vocabs=self.vocabs, training=training)\n        return CRFConstituencyModel(encoder, decoder)\n\n    def build_dataloader(self,\n                         data,\n                         batch_size,\n                         sampler_builder: SamplerBuilder = None,\n                         gradient_accumulation=1,\n                         shuffle=False,\n                         device=None,\n                         logger: logging.Logger = None,\n                         **kwargs) -> DataLoader:\n        if isinstance(data, TransformableDataset):\n            dataset = data\n        else:\n            transform = self.config.encoder.transform()\n            if self.config.get('transform', None):\n                transform = TransformList(self.config.transform, transform)\n            dataset = self.build_dataset(data, transform, logger)\n        if self.vocabs.mutable:\n            # noinspection PyTypeChecker\n            self.build_vocabs(dataset, logger)\n        lens = [len(x['token_input_ids']) for x in dataset]\n        if sampler_builder:\n            sampler = sampler_builder.build(lens, shuffle, gradient_accumulation)\n        else:\n            sampler = None\n        return PadSequenceDataLoader(dataset, batch_size, shuffle, device=device, batch_sampler=sampler)\n\n    def predict(self, data: Union[str, List[str]], **kwargs):\n        if not data:\n            return []\n        flat = self.input_is_flat(data)\n        if flat:\n            data = [data]\n        samples = self.build_samples(data)\n        dataloader = self.build_dataloader(samples, device=self.device, **kwargs)\n        outputs = []\n        orders = []\n        for idx, batch in enumerate(dataloader):\n            out, mask = self.feed_batch(batch)\n            prediction = self.decode_output(out, mask, batch, span_probs=None)\n            # prediction = [x[0] for x in prediction]\n            outputs.extend(prediction)\n            orders.extend(batch[IDX])\n        outputs = reorder(outputs, orders)\n        if flat:\n            return outputs[0]\n        return outputs\n\n    def input_is_flat(self, data):\n        return isinstance(data[0], str)\n\n    def build_samples(self, data):\n        return [{'token': [BOS] + token + [EOS]} for token in data]\n\n    # noinspection PyMethodOverriding\n    def fit(self,\n            trn_data,\n            dev_data,\n            save_dir,\n            encoder,\n            lr=5e-5,\n            transformer_lr=None,\n            adam_epsilon=1e-8,\n            weight_decay=0,\n            warmup_steps=0.1,\n            grad_norm=1.0,\n            n_mlp_span=500,\n            n_mlp_label=100,\n            mlp_dropout=.33,\n            batch_size=None,\n            batch_max_tokens=5000,\n            gradient_accumulation=1,\n            epochs=30,\n            patience=0.5,\n            mbr=True,\n            sampler_builder=None,\n            delete=('', ':', '``', \"''\", '.', '?', '!', '-NONE-', 'TOP', ',', 'S1'),\n            equal=(('ADVP', 'PRT'),),\n            no_subcategory=True,\n            eval_trn=True,\n            transform=None,\n            devices=None,\n            logger=None,\n            seed=None,\n            **kwargs):\n        if isinstance(equal, tuple):\n            equal = dict(equal)\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    def build_dataset(self, data, transform, logger=None):\n        _transform = [\n            unpack_tree_to_features,\n            self.vocabs,\n            FieldLength('token'),\n            transform\n        ]\n        if self.config.get('no_subcategory', True):\n            _transform.insert(0, remove_subcategory)\n        dataset = ConstituencyDataset(data,\n                                      transform=_transform,\n                                      cache=isinstance(data, str))\n        return dataset\n\n    def build_vocabs(self, trn, logger, **kwargs):\n        self.vocabs.chart = VocabWithNone(pad_token=None, unk_token=None)\n        timer = CountdownTimer(len(trn))\n        max_seq_len = 0\n        for each in trn:\n            max_seq_len = max(max_seq_len, len(each['token_input_ids']))\n            timer.log(f'Building vocab [blink][yellow]...[/yellow][/blink] (longest sequence: {max_seq_len})')\n        self.vocabs.chart.set_unk_as_safe_unk()\n        self.vocabs.lock()\n        self.vocabs.summary(logger)\n"
  },
  {
    "path": "hanlp/components/parsers/constituency/treecrf.py",
    "content": "# -*- coding:utf-8 -*-\n# Adopted from https://github.com/yzhangcs/parser\n# MIT License\n#\n# Copyright (c) 2020 Yu Zhang\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\n\nimport torch\nimport torch.autograd as autograd\nimport torch.nn as nn\n\nfrom hanlp.components.parsers.alg import stripe, istree, eisner, mst, eisner2o\n\n\nclass CRFConstituency(nn.Module):\n    r\"\"\"\n    TreeCRF for calculating partition functions and marginals in :math:`O(n^3)` for constituency trees.\n\n    References:\n        - Yu Zhang, houquan Zhou and Zhenghua Li. 2020.\n          `Fast and Accurate Neural CRF Constituency Parsing`_.\n\n    .. _Fast and Accurate Neural CRF Constituency Parsing:\n        https://www.ijcai.org/Proceedings/2020/560/\n    \"\"\"\n\n    @torch.enable_grad()\n    def forward(self, scores, mask, target=None, mbr=False):\n        r\"\"\"\n        Args:\n            scores (~torch.Tensor): ``[batch_size, seq_len, seq_len]``.\n                Scores of all possible constituents.\n            mask (~torch.BoolTensor): ``[batch_size, seq_len, seq_len]``.\n                The mask to avoid parsing over padding tokens.\n                For each square matrix in a batch, the positions except upper triangular part should be masked out.\n            target (~torch.BoolTensor): ``[batch_size, seq_len, seq_len]``.\n                The tensor of gold-standard constituents. ``True`` if a constituent exists. Default: ``None``.\n            mbr (bool):\n                If ``True``, marginals will be returned to perform minimum Bayes-risk (MBR) decoding. Default: ``False``.\n\n        Returns:\n            ~torch.Tensor, ~torch.Tensor:\n                The first is the training loss averaged by the number of tokens, which won't be returned if ``target=None``.\n                The second is a tensor of shape ``[batch_size, seq_len, seq_len]``, in which are marginals if ``mbr=True``,\n                or original scores otherwise.\n        \"\"\"\n\n        training = scores.requires_grad\n        # always enable the gradient computation of scores in order for the computation of marginals\n        logZ = self.inside(scores.requires_grad_(), mask)\n        # marginals are used for decoding, and can be computed by combining the inside pass and autograd mechanism\n        probs = scores\n        if mbr:\n            probs, = autograd.grad(logZ, scores, retain_graph=training)\n        if target is None:\n            return probs\n        loss = (logZ - scores[mask & target].sum()) / mask[:, 0].sum()\n\n        return loss, probs\n\n    def inside(self, scores, mask):\n        lens = mask[:, 0].sum(-1)\n        batch_size, seq_len, _ = scores.shape\n        # [seq_len, seq_len, batch_size]\n        scores, mask = scores.permute(1, 2, 0), mask.permute(1, 2, 0)\n        s = torch.full_like(scores, float('-inf'))\n\n        for w in range(1, seq_len):\n            # n denotes the number of spans to iterate,\n            # from span (0, w) to span (n, n+w) given width w\n            n = seq_len - w\n\n            if w == 1:\n                s.diagonal(w).copy_(scores.diagonal(w))\n                continue\n            # [n, w, batch_size]\n            s_s = stripe(s, n, w - 1, (0, 1)) + stripe(s, n, w - 1, (1, w), 0)\n            # [batch_size, n, w]\n            s_s = s_s.permute(2, 0, 1)\n            if s_s.requires_grad:\n                s_s.register_hook(lambda x: x.masked_fill_(torch.isnan(x), 0))\n            s_s = s_s.logsumexp(-1)\n            s.diagonal(w).copy_(s_s + scores.diagonal(w))\n\n        return s[0].gather(0, lens.unsqueeze(0)).sum()\n\n\nclass CRF2oDependency(nn.Module):\n    r\"\"\"\n    Second-order TreeCRF for calculating partition functions and marginals in :math:`O(n^3)` for projective dependency trees.\n\n    References:\n        - Yu Zhang, Zhenghua Li and Min Zhang. 2020.\n          `Efficient Second-Order TreeCRF for Neural Dependency Parsing`_.\n\n    .. _Efficient Second-Order TreeCRF for Neural Dependency Parsing:\n        https://www.aclweb.org/anthology/2020.acl-main.302/\n    \"\"\"\n\n    def __init__(self):\n        super().__init__()\n        self.criterion = nn.CrossEntropyLoss()\n\n    @torch.enable_grad()\n    def forward(self, scores, mask, target=None, mbr=True, partial=False):\n        r\"\"\"\n        Args:\n            scores (~torch.Tensor, ~torch.Tensor):\n                Tuple of two tensors `s_arc` and `s_sib`.\n                `s_arc` (``[batch_size, seq_len, seq_len]``) holds Scores of all possible dependent-head pairs.\n                `s_sib` (``[batch_size, seq_len, seq_len, seq_len]``) holds the scores of dependent-head-sibling triples.\n            mask (~torch.BoolTensor): ``[batch_size, seq_len]``.\n                The mask to avoid aggregation on padding tokens.\n                The first column serving as pseudo words for roots should be ``False``.\n            target (~torch.LongTensor): ``[batch_size, seq_len]``.\n                Tensors of gold-standard dependent-head pairs and dependent-head-sibling triples.\n                If partially annotated, the unannotated positions should be filled with -1.\n                Default: ``None``.\n            mbr (bool):\n                If ``True``, marginals will be returned to perform minimum Bayes-risk (MBR) decoding. Default: ``False``.\n            partial (bool):\n                ``True`` indicates that the trees are partially annotated. Default: ``False``.\n\n        Returns:\n            ~torch.Tensor, ~torch.Tensor:\n                The first is the training loss averaged by the number of tokens, which won't be returned if ``target=None``.\n                The second is a tensor of shape ``[batch_size, seq_len, seq_len]``, in which are marginals if ``mbr=True``,\n                or original scores otherwise.\n        \"\"\"\n\n        s_arc, s_sib = scores\n        training = s_arc.requires_grad\n        batch_size, seq_len, _ = s_arc.shape\n        # always enable the gradient computation of scores in order for the computation of marginals\n        logZ = self.inside((s.requires_grad_() for s in scores), mask)\n        # marginals are used for decoding, and can be computed by combining the inside pass and autograd mechanism\n        probs = s_arc\n        if mbr:\n            probs, = autograd.grad(logZ, s_arc, retain_graph=training)\n\n        if target is None:\n            return probs\n        arcs, sibs = target\n        # the second inside process is needed if use partial annotation\n        if partial:\n            score = self.inside(scores, mask, arcs)\n        else:\n            arc_seq, sib_seq = arcs[mask], sibs[mask]\n            arc_mask, sib_mask = mask, sib_seq.gt(0)\n            sib_seq = sib_seq[sib_mask]\n            s_sib = s_sib[mask][torch.arange(len(arc_seq)), arc_seq]\n            s_arc = s_arc[arc_mask].gather(-1, arc_seq.unsqueeze(-1))\n            s_sib = s_sib[sib_mask].gather(-1, sib_seq.unsqueeze(-1))\n            score = s_arc.sum() + s_sib.sum()\n        loss = (logZ - score) / mask.sum()\n\n        return loss, probs\n\n    def inside(self, scores, mask, cands=None):\n        # the end position of each sentence in a batch\n        lens = mask.sum(1)\n        s_arc, s_sib = scores\n        batch_size, seq_len, _ = s_arc.shape\n        # [seq_len, seq_len, batch_size]\n        s_arc = s_arc.permute(2, 1, 0)\n        # [seq_len, seq_len, seq_len, batch_size]\n        s_sib = s_sib.permute(2, 1, 3, 0)\n        s_i = torch.full_like(s_arc, float('-inf'))\n        s_s = torch.full_like(s_arc, float('-inf'))\n        s_c = torch.full_like(s_arc, float('-inf'))\n        s_c.diagonal().fill_(0)\n\n        # set the scores of arcs excluded by cands to -inf\n        if cands is not None:\n            mask = mask.index_fill(1, lens.new_tensor(0), 1)\n            mask = (mask.unsqueeze(1) & mask.unsqueeze(-1)).permute(2, 1, 0)\n            cands = cands.unsqueeze(-1).index_fill(1, lens.new_tensor(0), -1)\n            cands = cands.eq(lens.new_tensor(range(seq_len))) | cands.lt(0)\n            cands = cands.permute(2, 1, 0) & mask\n            s_arc = s_arc.masked_fill(~cands, float('-inf'))\n\n        for w in range(1, seq_len):\n            # n denotes the number of spans to iterate,\n            # from span (0, w) to span (n, n+w) given width w\n            n = seq_len - w\n            # I(j->i) = logsum(exp(I(j->r) + S(j->r, i)) +, i < r < j\n            #                  exp(C(j->j) + C(i->j-1)))\n            #           + s(j->i)\n            # [n, w, batch_size]\n            il = stripe(s_i, n, w, (w, 1)) + stripe(s_s, n, w, (1, 0), 0)\n            il += stripe(s_sib[range(w, n + w), range(n)], n, w, (0, 1))\n            # [n, 1, batch_size]\n            il0 = stripe(s_c, n, 1, (w, w)) + stripe(s_c, n, 1, (0, w - 1))\n            # il0[0] are set to zeros since the scores of the complete spans starting from 0 are always -inf\n            il[:, -1] = il0.index_fill_(0, lens.new_tensor(0), 0).squeeze(1)\n            if il.requires_grad:\n                il.register_hook(lambda x: x.masked_fill_(torch.isnan(x), 0))\n            il = il.permute(2, 0, 1).logsumexp(-1)\n            s_i.diagonal(-w).copy_(il + s_arc.diagonal(-w))\n            # I(i->j) = logsum(exp(I(i->r) + S(i->r, j)) +, i < r < j\n            #                  exp(C(i->i) + C(j->i+1)))\n            #           + s(i->j)\n            # [n, w, batch_size]\n            ir = stripe(s_i, n, w) + stripe(s_s, n, w, (0, w), 0)\n            ir += stripe(s_sib[range(n), range(w, n + w)], n, w)\n            ir[0] = float('-inf')\n            # [n, 1, batch_size]\n            ir0 = stripe(s_c, n, 1) + stripe(s_c, n, 1, (w, 1))\n            ir[:, 0] = ir0.squeeze(1)\n            if ir.requires_grad:\n                ir.register_hook(lambda x: x.masked_fill_(torch.isnan(x), 0))\n            ir = ir.permute(2, 0, 1).logsumexp(-1)\n            s_i.diagonal(w).copy_(ir + s_arc.diagonal(w))\n\n            # [n, w, batch_size]\n            slr = stripe(s_c, n, w) + stripe(s_c, n, w, (w, 1))\n            if slr.requires_grad:\n                slr.register_hook(lambda x: x.masked_fill_(torch.isnan(x), 0))\n            slr = slr.permute(2, 0, 1).logsumexp(-1)\n            # S(j, i) = logsumexp(C(i->r) + C(j->r+1)), i <= r < j\n            s_s.diagonal(-w).copy_(slr)\n            # S(i, j) = logsumexp(C(i->r) + C(j->r+1)), i <= r < j\n            s_s.diagonal(w).copy_(slr)\n\n            # C(j->i) = logsumexp(C(r->i) + I(j->r)), i <= r < j\n            cl = stripe(s_c, n, w, (0, 0), 0) + stripe(s_i, n, w, (w, 0))\n            cl.register_hook(lambda x: x.masked_fill_(torch.isnan(x), 0))\n            s_c.diagonal(-w).copy_(cl.permute(2, 0, 1).logsumexp(-1))\n            # C(i->j) = logsumexp(I(i->r) + C(r->j)), i < r <= j\n            cr = stripe(s_i, n, w, (0, 1)) + stripe(s_c, n, w, (1, w), 0)\n            cr.register_hook(lambda x: x.masked_fill_(torch.isnan(x), 0))\n            s_c.diagonal(w).copy_(cr.permute(2, 0, 1).logsumexp(-1))\n            # disable multi words to modify the root\n            s_c[0, w][lens.ne(w)] = float('-inf')\n\n        return s_c[0].gather(0, lens.unsqueeze(0)).sum()\n\n    def loss(self, s_arc, s_sib, s_rel, arcs, sibs, rels, mask, mbr=True, partial=False):\n        r\"\"\"\n        Args:\n            s_arc (~torch.Tensor): ``[batch_size, seq_len, seq_len]``.\n                Scores of all possible arcs.\n            s_sib (~torch.Tensor): ``[batch_size, seq_len, seq_len, seq_len]``.\n                Scores of all possible dependent-head-sibling triples.\n            s_rel (~torch.Tensor): ``[batch_size, seq_len, seq_len, n_labels]``.\n                Scores of all possible labels on each arc.\n            arcs (~torch.LongTensor): ``[batch_size, seq_len]``.\n                The tensor of gold-standard arcs.\n            sibs (~torch.LongTensor): ``[batch_size, seq_len]``.\n                The tensor of gold-standard siblings.\n            rels (~torch.LongTensor): ``[batch_size, seq_len]``.\n                The tensor of gold-standard labels.\n            mask (~torch.BoolTensor): ``[batch_size, seq_len]``.\n                The mask for covering the unpadded tokens.\n            mbr (bool):\n                If ``True``, returns marginals for MBR decoding. Default: ``True``.\n            partial (bool):\n                ``True`` denotes the trees are partially annotated. Default: ``False``.\n\n        Returns:\n            ~torch.Tensor, ~torch.Tensor:\n                The training loss and\n                original arc scores of shape ``[batch_size, seq_len, seq_len]`` if ``mbr=False``, or marginals otherwise.\n        \"\"\"\n\n        scores, target = (s_arc, s_sib), (arcs, sibs)\n        arc_loss, arc_probs = self.forward(scores, mask, target, mbr, partial)\n        # -1 denotes un-annotated arcs\n        if partial:\n            mask = mask & arcs.ge(0)\n        s_rel, rels = s_rel[mask], rels[mask]\n        s_rel = s_rel[torch.arange(len(rels)), arcs[mask]]\n        rel_loss = self.criterion(s_rel, rels)\n        loss = arc_loss + rel_loss\n        return loss, arc_probs\n\n    # def decode(self, s_arc, s_rel, mask, tree=False, proj=False, alg=None):\n    #     r\"\"\"\n    #     Args:\n    #         s_arc (~torch.Tensor): ``[batch_size, seq_len, seq_len]``.\n    #             Scores of all possible arcs.\n    #         s_rel (~torch.Tensor): ``[batch_size, seq_len, seq_len, n_labels]``.\n    #             Scores of all possible labels on each arc.\n    #         mask (~torch.BoolTensor): ``[batch_size, seq_len]``.\n    #             The mask for covering the unpadded tokens.\n    #         tree (bool):\n    #             If ``True``, ensures to output well-formed trees. Default: ``False``.\n    #         proj (bool):\n    #             If ``True``, ensures to output projective trees. Default: ``False``.\n    # \n    #     Returns:\n    #         ~torch.Tensor, ~torch.Tensor:\n    #             Predicted arcs and labels of shape ``[batch_size, seq_len]``.\n    #     \"\"\"\n    # \n    #     lens = mask.sum(1)\n    #     arc_preds = s_arc.argmax(-1)\n    #     if tree and not alg:\n    #         bad = [not istree(seq[1:i + 1], proj)\n    #                for i, seq in zip(lens.tolist(), arc_preds.tolist())]\n    #         if any(bad):\n    #             alg = eisner if proj else mst\n    #             arc_preds[bad] = alg(s_arc[bad], mask[bad])\n    #     rel_preds = s_rel.argmax(-1).gather(-1, arc_preds.unsqueeze(-1)).squeeze(-1)\n    # \n    #     return arc_preds, rel_preds\n    def decode(self, s_arc, s_sib, s_rel, mask, tree=False, mbr=True, proj=False):\n        r\"\"\"\n        Args:\n            s_arc (~torch.Tensor): ``[batch_size, seq_len, seq_len]``.\n                Scores of all possible arcs.\n            s_sib (~torch.Tensor): ``[batch_size, seq_len, seq_len, seq_len]``.\n                Scores of all possible dependent-head-sibling triples.\n            s_rel (~torch.Tensor): ``[batch_size, seq_len, seq_len, n_labels]``.\n                Scores of all possible labels on each arc.\n            mask (~torch.BoolTensor): ``[batch_size, seq_len]``.\n                The mask for covering the unpadded tokens.\n            tree (bool):\n                If ``True``, ensures to output well-formed trees. Default: ``False``.\n            mbr (bool):\n                If ``True``, performs MBR decoding. Default: ``True``.\n            proj (bool):\n                If ``True``, ensures to output projective trees. Default: ``False``.\n\n        Returns:\n            ~torch.Tensor, ~torch.Tensor:\n                Predicted arcs and labels of shape ``[batch_size, seq_len]``.\n        \"\"\"\n\n        lens = mask.sum(1)\n        arc_preds = s_arc.argmax(-1)\n        if tree:\n            bad = [not istree(seq[1:i + 1], proj)\n                   for i, seq in zip(lens.tolist(), arc_preds.tolist())]\n            if any(bad):\n                if proj and not mbr:\n                    arc_preds = eisner2o((s_arc, s_sib), mask)\n                else:\n                    alg = eisner if proj else mst\n                    arc_preds[bad] = alg(s_arc[bad], mask[bad])\n        rel_preds = s_rel.argmax(-1).gather(-1, arc_preds.unsqueeze(-1)).squeeze(-1)\n\n        return arc_preds, rel_preds\n"
  },
  {
    "path": "hanlp/components/parsers/parse_alg.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-04-02 23:20\nfrom collections import defaultdict\n\nimport hanlp.utils.span_util\nfrom hanlp.components.parsers.chu_liu_edmonds import decode_mst\nimport numpy as np\n\n\nclass Tarjan:\n    \"\"\"Computes Tarjan's algorithm for finding strongly connected components (cycles) of a graph\"\"\"\n\n    def __init__(self, prediction, tokens):\n        \"\"\"\n\n        Parameters\n        ----------\n        prediction : numpy.ndarray\n            a predicted dependency tree where prediction[dep_idx] = head_idx\n        tokens : numpy.ndarray\n            the tokens we care about (i.e. exclude _GO, _EOS, and _PAD)\n        \"\"\"\n        self._edges = defaultdict(set)\n        self._vertices = set((0,))\n        for dep, head in enumerate(prediction[tokens]):\n            self._vertices.add(dep + 1)\n            self._edges[head].add(dep + 1)\n        self._indices = {}\n        self._lowlinks = {}\n        self._onstack = defaultdict(lambda: False)\n        self._SCCs = []\n\n        index = 0\n        stack = []\n        for v in self.vertices:\n            if v not in self.indices:\n                self.strongconnect(v, index, stack)\n\n    # =============================================================\n    def strongconnect(self, v, index, stack):\n        \"\"\"\n\n        Args:\n          v: \n          index: \n          stack: \n\n        Returns:\n\n        \"\"\"\n\n        self._indices[v] = index\n        self._lowlinks[v] = index\n        index += 1\n        stack.append(v)\n        self._onstack[v] = True\n        for w in self.edges[v]:\n            if w not in self.indices:\n                self.strongconnect(w, index, stack)\n                self._lowlinks[v] = min(self._lowlinks[v], self._lowlinks[w])\n            elif self._onstack[w]:\n                self._lowlinks[v] = min(self._lowlinks[v], self._indices[w])\n\n        if self._lowlinks[v] == self._indices[v]:\n            self._SCCs.append(set())\n            while stack[-1] != v:\n                w = stack.pop()\n                self._onstack[w] = False\n                self._SCCs[-1].add(w)\n            w = stack.pop()\n            self._onstack[w] = False\n            self._SCCs[-1].add(w)\n        return\n\n    # ======================\n    @property\n    def edges(self):\n        return self._edges\n\n    @property\n    def vertices(self):\n        return self._vertices\n\n    @property\n    def indices(self):\n        return self._indices\n\n    @property\n    def SCCs(self):\n        return self._SCCs\n\n\nclass UnionFind(object):\n\n    def __init__(self, n) -> None:\n        super().__init__()\n        self.parent = [x for x in range(n)]\n        self.height = [0] * n\n\n    def find(self, x):\n        if self.parent[x] == x:\n            return x\n        self.parent[x] = self.find(self.parent[x])\n        return self.parent[x]\n\n    def unite(self, x, y):\n        x = self.find(x)\n        y = self.find(y)\n        if x == y:\n            return\n        if self.height[x] < self.height[y]:\n            self.parent[x] = y\n        else:\n            self.parent[y] = x\n            if self.height[x] == self.height[y]:\n                self.height[x] += 1\n\n    def same(self, x, y):\n        return self.find(x) == self.find(y)\n\n\ndef tarjan(parse_probs, length, tokens_to_keep, ensure_tree=True):\n    \"\"\"Adopted from Timothy Dozat https://github.com/tdozat/Parser/blob/master/lib/models/nn.py\n\n    Args:\n      parse_probs(NDArray): seq_len x seq_len, the probability of arcs\n      length(NDArray): sentence length including ROOT\n      tokens_to_keep(NDArray): mask matrix\n      ensure_tree:  (Default value = True)\n\n    Returns:\n\n    \n    \"\"\"\n    if ensure_tree:\n        parse_preds, parse_probs, tokens = unique_root(parse_probs, tokens_to_keep, length)\n        # remove cycles\n        tarjan = Tarjan(parse_preds, tokens)\n        for SCC in tarjan.SCCs:\n            if len(SCC) > 1:\n                dependents = set()\n                to_visit = set(SCC)\n                while len(to_visit) > 0:\n                    node = to_visit.pop()\n                    if not node in dependents:\n                        dependents.add(node)\n                        to_visit.update(tarjan.edges[node])\n                # The indices of the nodes that participate in the cycle\n                cycle = np.array(list(SCC))\n                # The probabilities of the current heads\n                old_heads = parse_preds[cycle]\n                old_head_probs = parse_probs[cycle, old_heads]\n                # Set the probability of depending on a non-head to zero\n                non_heads = np.array(list(dependents))\n                parse_probs[np.repeat(cycle, len(non_heads)), np.repeat([non_heads], len(cycle), axis=0).flatten()] = 0\n                # Get new potential heads and their probabilities\n                new_heads = np.argmax(parse_probs[cycle][:, tokens], axis=1) + 1\n                new_head_probs = parse_probs[cycle, new_heads] / old_head_probs\n                # Select the most probable change\n                change = np.argmax(new_head_probs)\n                changed_cycle = cycle[change]\n                old_head = old_heads[change]\n                new_head = new_heads[change]\n                # Make the change\n                parse_preds[changed_cycle] = new_head\n                tarjan.edges[new_head].add(changed_cycle)\n                tarjan.edges[old_head].remove(changed_cycle)\n        return parse_preds\n    else:\n        # block and pad heads\n        parse_probs = parse_probs * tokens_to_keep\n        parse_preds = np.argmax(parse_probs, axis=1)\n        return parse_preds\n\n\ndef chu_liu_edmonds(parse_probs, length):\n    tree = decode_mst(hanlp.utils.span_util.T, length, False)[0]\n    tree[0] = 0\n    return tree\n\n\ndef unique_root(parse_probs, tokens_to_keep: np.ndarray, length):\n    I = np.eye(len(tokens_to_keep))\n    # block loops and pad heads\n    if tokens_to_keep.ndim == 1:\n        tokens_to_keep = np.expand_dims(tokens_to_keep, -1)\n    parse_probs = parse_probs * tokens_to_keep * (1 - I)\n    parse_preds = np.argmax(parse_probs, axis=1)\n    tokens = np.arange(1, length)\n    roots = np.where(parse_preds[tokens] == 0)[0] + 1\n    # ensure at least one root\n    if len(roots) < 1:\n        # The current root probabilities\n        root_probs = parse_probs[tokens, 0]\n        # The current head probabilities\n        old_head_probs = parse_probs[tokens, parse_preds[tokens]]\n        # Get new potential root probabilities\n        new_root_probs = root_probs / old_head_probs\n        # Select the most probable root\n        new_root = tokens[np.argmax(new_root_probs)]\n        # Make the change\n        parse_preds[new_root] = 0\n    # ensure at most one root\n    elif len(roots) > 1:\n        # The probabilities of the current heads\n        root_probs = parse_probs[roots, 0]\n        # Set the probability of depending on the root zero\n        parse_probs[roots, 0] = 0\n        # Get new potential heads and their probabilities\n        new_heads = np.argmax(parse_probs[roots][:, tokens], axis=1) + 1\n        new_head_probs = parse_probs[roots, new_heads] / root_probs\n        # Select the most probable root\n        new_root = roots[np.argmin(new_head_probs)]\n        # Make the change\n        parse_preds[roots] = new_heads\n        parse_preds[new_root] = 0\n    return parse_preds, parse_probs, tokens\n\n\ndef dfs(graph, start, end):\n    fringe = [(start, [])]\n    while fringe:\n        state, path = fringe.pop()\n        if path and state == end:\n            yield path\n            continue\n        for next_state in graph[state]:\n            if next_state in path:\n                continue\n            fringe.append((next_state, path + [next_state]))\n\n\ndef mst_then_greedy(arc_scores, rel_scores, mask, root_rel_idx, rel_idx=None):\n    from scipy.special import softmax\n    from scipy.special import expit as sigmoid\n    length = sum(mask) + 1\n    mask = mask[:length]\n    arc_scores = arc_scores[:length, :length]\n    arc_pred = arc_scores > 0\n    arc_probs = sigmoid(arc_scores)\n    rel_scores = rel_scores[:length, :length, :]\n    rel_probs = softmax(rel_scores, -1)\n    if not any(arc_pred[:, 0][1:]):  # no root\n        root = np.argmax(rel_probs[1:, 0, root_rel_idx]) + 1\n        arc_probs[root, 0] = 1\n    parse_preds, parse_probs, tokens = unique_root(arc_probs, mask, length)\n    root = adjust_root_score(arc_scores, parse_preds, root_rel_idx, rel_scores)\n    tree = chu_liu_edmonds(arc_scores, length)\n    if rel_idx is not None:  # Unknown DEPREL label: 'ref'\n        rel_scores[np.arange(len(tree)), tree, rel_idx] = -float('inf')\n    return tree, add_secondary_arcs_by_scores(arc_scores, rel_scores, tree, root_rel_idx)\n\n\ndef adjust_root_score(arc_scores, parse_preds, root_rel_idx, rel_scores=None):\n    root = np.where(parse_preds[1:] == 0)[0] + 1\n    arc_scores[:, 0] = min(np.min(arc_scores), -1000)\n    arc_scores[root, 0] = max(np.max(arc_scores), 1000)\n    if rel_scores is not None:\n        rel_scores[:, :, root_rel_idx] = -float('inf')\n        rel_scores[root, 0, root_rel_idx] = float('inf')\n    return root\n\n\ndef add_secondary_arcs_by_scores(arc_scores, rel_scores, tree, root_rel_idx, arc_preds=None):\n    if not isinstance(tree, np.ndarray):\n        tree = np.array(tree)\n    if arc_preds is None:\n        arc_preds = arc_scores > 0\n    rel_pred = np.argmax(rel_scores, axis=-1)\n\n    return add_secondary_arcs_by_preds(arc_scores, arc_preds, rel_pred, tree, root_rel_idx)\n\n\ndef add_secondary_arcs_by_preds(arc_scores, arc_preds, rel_preds, tree, root_rel_idx=None):\n    dh = np.argwhere(arc_preds)\n    sdh = sorted([(arc_scores[x[0], x[1]], list(x)) for x in dh], reverse=True)\n    graph = [[] for _ in range(len(tree))]\n    for d, h in enumerate(tree):\n        if d:\n            graph[h].append(d)\n    for s, (d, h) in sdh:\n        if not d or not h or d in graph[h]:\n            continue\n        try:\n            path = next(dfs(graph, d, h))\n        except StopIteration:\n            # no path from d to h\n            graph[h].append(d)\n    parse_graph = [[] for _ in range(len(tree))]\n    num_root = 0\n    for h in range(len(tree)):\n        for d in graph[h]:\n            rel = rel_preds[d, h]\n            if h == 0 and root_rel_idx is not None:\n                rel = root_rel_idx\n                assert num_root == 0\n                num_root += 1\n            parse_graph[d].append((h, rel))\n        parse_graph[d] = sorted(parse_graph[d])\n    return parse_graph\n\n\ndef adjust_root_score_then_add_secondary_arcs(arc_scores, rel_scores, tree, root_rel_idx):\n    if len(arc_scores) != tree:\n        arc_scores = arc_scores[:len(tree), :len(tree)]\n        rel_scores = rel_scores[:len(tree), :len(tree), :]\n    parse_preds = arc_scores > 0\n    # adjust_root_score(arc_scores, parse_preds, rel_scores)\n    parse_preds[:, 0] = False  # set heads to False\n    rel_scores[:, :, root_rel_idx] = -float('inf')\n    return add_secondary_arcs_by_scores(arc_scores, rel_scores, tree, root_rel_idx, parse_preds)\n"
  },
  {
    "path": "hanlp/components/parsers/ud/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-14 20:34\n"
  },
  {
    "path": "hanlp/components/parsers/ud/lemma_edit.py",
    "content": "\"\"\"\nUtilities for processing lemmas\n\nAdopted from UDPipe Future\nhttps://github.com/CoNLL-UD-2018/UDPipe-Future\n\"\"\"\n\n\ndef min_edit_script(source, target, allow_copy=False):\n    \"\"\"Finds the minimum edit script to transform the source to the target\n\n    Args:\n      source: \n      target: \n      allow_copy:  (Default value = False)\n\n    Returns:\n\n    \"\"\"\n    a = [[(len(source) + len(target) + 1, None)] * (len(target) + 1) for _ in range(len(source) + 1)]\n    for i in range(0, len(source) + 1):\n        for j in range(0, len(target) + 1):\n            if i == 0 and j == 0:\n                a[i][j] = (0, \"\")\n            else:\n                if allow_copy and i and j and source[i - 1] == target[j - 1] and a[i - 1][j - 1][0] < a[i][j][0]:\n                    a[i][j] = (a[i - 1][j - 1][0], a[i - 1][j - 1][1] + \"→\")\n                if i and a[i - 1][j][0] < a[i][j][0]:\n                    a[i][j] = (a[i - 1][j][0] + 1, a[i - 1][j][1] + \"-\")\n                if j and a[i][j - 1][0] < a[i][j][0]:\n                    a[i][j] = (a[i][j - 1][0] + 1, a[i][j - 1][1] + \"+\" + target[j - 1])\n    return a[-1][-1][1]\n\n\ndef gen_lemma_rule(form, lemma, allow_copy=False):\n    \"\"\"Generates a lemma rule to transform the source to the target\n\n    Args:\n      form: \n      lemma: \n      allow_copy:  (Default value = False)\n\n    Returns:\n\n    \"\"\"\n    form = form.lower()\n\n    previous_case = -1\n    lemma_casing = \"\"\n    for i, c in enumerate(lemma):\n        case = \"↑\" if c.lower() != c else \"↓\"\n        if case != previous_case:\n            lemma_casing += \"{}{}{}\".format(\"¦\" if lemma_casing else \"\", case,\n                                            i if i <= len(lemma) // 2 else i - len(lemma))\n        previous_case = case\n    lemma = lemma.lower()\n\n    best, best_form, best_lemma = 0, 0, 0\n    for l in range(len(lemma)):\n        for f in range(len(form)):\n            cpl = 0\n            while f + cpl < len(form) and l + cpl < len(lemma) and form[f + cpl] == lemma[l + cpl]: cpl += 1\n            if cpl > best:\n                best = cpl\n                best_form = f\n                best_lemma = l\n\n    rule = lemma_casing + \";\"\n    if not best:\n        rule += \"a\" + lemma\n    else:\n        rule += \"d{}¦{}\".format(\n            min_edit_script(form[:best_form], lemma[:best_lemma], allow_copy),\n            min_edit_script(form[best_form + best:], lemma[best_lemma + best:], allow_copy),\n        )\n    return rule\n\n\ndef apply_lemma_rule(form, lemma_rule):\n    \"\"\"Applies the lemma rule to the form to generate the lemma\n\n    Args:\n      form: \n      lemma_rule: \n\n    Returns:\n\n    \"\"\"\n    cells = lemma_rule.split(\";\", 1)\n    if len(cells) == 1:  # Some predicted lemma rules are _, which might be due to partial annotation\n        return form.lower()\n    casing, rule = cells\n    if rule.startswith(\"a\"):\n        lemma = rule[1:]\n    else:\n        form = form.lower()\n        rules, rule_sources = rule[1:].split(\"¦\"), []\n        assert len(rules) == 2\n        for rule in rules:\n            source, i = 0, 0\n            while i < len(rule):\n                if rule[i] == \"→\" or rule[i] == \"-\":\n                    source += 1\n                else:\n                    assert rule[i] == \"+\"\n                    i += 1\n                i += 1\n            rule_sources.append(source)\n\n        try:\n            lemma, form_offset = \"\", 0\n            for i in range(2):\n                j, offset = 0, (0 if i == 0 else len(form) - rule_sources[1])\n                while j < len(rules[i]):\n                    if rules[i][j] == \"→\":\n                        lemma += form[offset]\n                        offset += 1\n                    elif rules[i][j] == \"-\":\n                        offset += 1\n                    else:\n                        assert (rules[i][j] == \"+\")\n                        lemma += rules[i][j + 1]\n                        j += 1\n                    j += 1\n                if i == 0:\n                    lemma += form[rule_sources[0]: len(form) - rule_sources[1]]\n        except:\n            lemma = form\n\n    for rule in casing.split(\"¦\"):\n        if rule == \"↓0\": continue  # The lemma is lowercased initially\n        case, offset = rule[0], int(rule[1:])\n        lemma = lemma[:offset] + (lemma[offset:].upper() if case == \"↑\" else lemma[offset:].lower())\n\n    return lemma\n"
  },
  {
    "path": "hanlp/components/parsers/ud/tag_decoder.py",
    "content": "# This file is modified from udify, which is licensed under the MIT license:\n# MIT License\n#\n# Copyright (c) 2019 Dan Kondratyuk\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\n\n\"\"\"\nDecodes sequences of tags, e.g., POS tags, given a list of contextualized word embeddings\n\"\"\"\n\nfrom typing import Dict\n\nimport numpy\nimport torch\nimport torch.nn.functional as F\nfrom torch.nn.modules.adaptive import AdaptiveLogSoftmaxWithLoss\nfrom torch.nn.modules.linear import Linear\n\nfrom hanlp.components.parsers.ud.lemma_edit import apply_lemma_rule\nfrom hanlp.components.parsers.ud.udify_util import sequence_cross_entropy, sequence_cross_entropy_with_logits\n\n\nclass TagDecoder(torch.nn.Module):\n    \"\"\"A basic sequence tagger that decodes from inputs of word embeddings\"\"\"\n\n    def __init__(self,\n                 input_dim,\n                 num_classes,\n                 label_smoothing: float = 0.03,\n                 adaptive: bool = False) -> None:\n        super(TagDecoder, self).__init__()\n\n        self.label_smoothing = label_smoothing\n        self.num_classes = num_classes\n        self.adaptive = adaptive\n\n        if self.adaptive:\n            adaptive_cutoffs = [round(self.num_classes / 15), 3 * round(self.num_classes / 15)]\n            self.task_output = AdaptiveLogSoftmaxWithLoss(input_dim,\n                                                          self.num_classes,\n                                                          cutoffs=adaptive_cutoffs,\n                                                          div_value=4.0)\n        else:\n            self.task_output = Linear(self.output_dim, self.num_classes)\n\n    def forward(self,\n                encoded_text: torch.FloatTensor,\n                mask: torch.LongTensor,\n                gold_tags: torch.LongTensor,\n                ) -> Dict[str, torch.Tensor]:\n        hidden = encoded_text\n\n        batch_size, sequence_length, _ = hidden.size()\n        output_dim = [batch_size, sequence_length, self.num_classes]\n\n        loss_fn = self._adaptive_loss if self.adaptive else self._loss\n\n        output_dict = loss_fn(hidden, mask, gold_tags, output_dim)\n\n        return output_dict\n\n    def _adaptive_loss(self, hidden, mask, gold_tags, output_dim):\n        logits = hidden\n        reshaped_log_probs = logits.reshape(-1, logits.size(2))\n\n        class_probabilities = self.task_output.log_prob(reshaped_log_probs).view(output_dim)\n\n        output_dict = {\"logits\": logits, \"class_probabilities\": class_probabilities}\n\n        if gold_tags is not None:\n            output_dict[\"loss\"] = sequence_cross_entropy(class_probabilities,\n                                                         gold_tags,\n                                                         mask,\n                                                         label_smoothing=self.label_smoothing)\n\n        return output_dict\n\n    def _loss(self, hidden, mask, gold_tags, output_dim):\n        logits = self.task_output(hidden)\n        reshaped_log_probs = logits.view(-1, self.num_classes)\n        class_probabilities = F.softmax(reshaped_log_probs, dim=-1).view(output_dim)\n\n        output_dict = {\"logits\": logits, \"class_probabilities\": class_probabilities}\n\n        if gold_tags is not None:\n            output_dict[\"loss\"] = sequence_cross_entropy_with_logits(logits,\n                                                                     gold_tags,\n                                                                     mask,\n                                                                     label_smoothing=self.label_smoothing)\n        return output_dict\n\n    def decode(self, output_dict: Dict[str, torch.Tensor]) -> Dict[str, torch.Tensor]:\n        all_words = output_dict[\"words\"]\n\n        all_predictions = output_dict[\"class_probabilities\"][self.task].cpu().data.numpy()\n        if all_predictions.ndim == 3:\n            predictions_list = [all_predictions[i] for i in range(all_predictions.shape[0])]\n        else:\n            predictions_list = [all_predictions]\n        all_tags = []\n        for predictions, words in zip(predictions_list, all_words):\n            argmax_indices = numpy.argmax(predictions, axis=-1)\n            tags = [self.vocab.get_token_from_index(x, namespace=self.task)\n                    for x in argmax_indices]\n\n            if self.task == \"lemmas\":\n                def decode_lemma(word, rule):\n                    if rule == \"_\":\n                        return \"_\"\n                    if rule == \"@@UNKNOWN@@\":\n                        return word\n                    return apply_lemma_rule(word, rule)\n\n                tags = [decode_lemma(word, rule) for word, rule in zip(words, tags)]\n\n            all_tags.append(tags)\n        output_dict[self.task] = all_tags\n\n        return output_dict\n"
  },
  {
    "path": "hanlp/components/parsers/ud/ud_model.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-15 14:21\n\nfrom typing import Dict, Any\n\nimport torch\n\nfrom hanlp.components.parsers.biaffine.biaffine_dep import BiaffineDependencyParser\nfrom hanlp.components.parsers.biaffine.biaffine_model import BiaffineDecoder\nfrom hanlp.components.parsers.ud.tag_decoder import TagDecoder\nfrom hanlp.layers.embeddings.contextual_word_embedding import ContextualWordEmbeddingModule\nfrom hanlp.layers.scalar_mix import ScalarMixWithDropout\n\n\nclass UniversalDependenciesModel(torch.nn.Module):\n    def __init__(self,\n                 encoder: ContextualWordEmbeddingModule,\n                 n_mlp_arc,\n                 n_mlp_rel,\n                 mlp_dropout,\n                 num_rels,\n                 num_lemmas,\n                 num_upos,\n                 num_feats,\n                 mix_embedding: int = 13,\n                 layer_dropout: int = 0.0):\n        super().__init__()\n        self.encoder = encoder\n        self.decoder = UniversalDependenciesDecoder(\n            encoder.get_output_dim(),\n            n_mlp_arc,\n            n_mlp_rel,\n            mlp_dropout,\n            num_rels,\n            num_lemmas,\n            num_upos,\n            num_feats,\n            mix_embedding,\n            layer_dropout\n        )\n\n    def forward(self,\n                batch: Dict[str, torch.Tensor],\n                mask,\n                ):\n        hidden = self.encoder(batch)\n        return self.decoder(hidden, batch=batch, mask=mask)\n\n\nclass UniversalDependenciesDecoder(torch.nn.Module):\n    def __init__(self,\n                 hidden_size,\n                 n_mlp_arc,\n                 n_mlp_rel,\n                 mlp_dropout,\n                 num_rels,\n                 num_lemmas,\n                 num_upos,\n                 num_feats,\n                 mix_embedding: int = 13,\n                 layer_dropout: int = 0.0,\n                 ) -> None:\n        super(UniversalDependenciesDecoder, self).__init__()\n\n        # decoders\n        self.decoders = torch.nn.ModuleDict({\n            'lemmas': TagDecoder(hidden_size, num_lemmas, label_smoothing=0.03, adaptive=True),\n            'upos': TagDecoder(hidden_size, num_upos, label_smoothing=0.03, adaptive=True),\n            'deps': BiaffineDecoder(hidden_size, n_mlp_arc, n_mlp_rel, mlp_dropout, num_rels),\n            'feats': TagDecoder(hidden_size, num_feats, label_smoothing=0.03, adaptive=True),\n        })\n        self.gold_keys = {\n            'lemmas': 'lemma_id',\n            'upos': 'pos_id',\n            'feats': 'feat_id',\n        }\n\n        if mix_embedding:\n            self.scalar_mix = torch.nn.ModuleDict({\n                task: ScalarMixWithDropout((1, mix_embedding),\n                                           do_layer_norm=False,\n                                           dropout=layer_dropout)\n                for task in self.decoders\n            })\n        else:\n            self.scalar_mix = None\n\n    def forward(self,\n                hidden,\n                batch: Dict[str, torch.Tensor],\n                mask) -> Dict[str, Any]:\n        mask_without_root = mask.clone()\n        mask_without_root[:, 0] = False\n\n        logits = {}\n        class_probabilities = {}\n        output_dict = {\"logits\": logits,\n                       \"class_probabilities\": class_probabilities}\n        loss = 0\n\n        arc = batch.get('arc', None)\n        # Run through each of the tasks on the shared encoder and save predictions\n        for task in self.decoders:\n            if self.scalar_mix:\n                decoder_input = self.scalar_mix[task](hidden, mask)\n            else:\n                decoder_input = hidden\n\n            if task == \"deps\":\n                s_arc, s_rel = self.decoders[task](decoder_input, mask)\n                pred_output = {'class_probabilities': {'s_arc': s_arc, 's_rel': s_rel}}\n                if arc is not None:\n                    # noinspection PyTypeChecker\n                    pred_output['loss'] = BiaffineDependencyParser.compute_loss(None, s_arc, s_rel, arc,\n                                                                                batch['rel_id'],\n                                                                                mask_without_root,\n                                                                                torch.nn.functional.cross_entropy)\n            else:\n                pred_output = self.decoders[task](decoder_input, mask_without_root,\n                                                  batch.get(self.gold_keys[task], None))\n            if 'logits' in pred_output:\n                logits[task] = pred_output[\"logits\"]\n            if 'class_probabilities' in pred_output:\n                class_probabilities[task] = pred_output[\"class_probabilities\"]\n            if 'loss' in pred_output:\n                # Keep track of the loss if we have the gold tags available\n                loss += pred_output[\"loss\"]\n\n        if arc is not None:\n            output_dict[\"loss\"] = loss\n\n        return output_dict\n\n    def decode(self, output_dict: Dict[str, torch.Tensor]) -> Dict[str, torch.Tensor]:\n        for task in self.tasks:\n            self.decoders[task].decode(output_dict)\n\n        return output_dict\n"
  },
  {
    "path": "hanlp/components/parsers/ud/ud_parser.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-14 20:34\nimport logging\nfrom copy import deepcopy\nfrom typing import Union, List, Callable\n\nimport torch\nfrom torch.utils.data import DataLoader\n\nfrom hanlp_common.constant import IDX\nfrom hanlp.common.dataset import PadSequenceDataLoader, SortingSamplerBuilder\nfrom hanlp.common.structure import History\nfrom hanlp.common.torch_component import TorchComponent\nfrom hanlp.common.transform import FieldLength, PunctuationMask\nfrom hanlp.common.vocab import Vocab\nfrom hanlp.components.classifiers.transformer_classifier import TransformerComponent\nfrom hanlp.components.parsers.biaffine.biaffine_dep import BiaffineDependencyParser\nfrom hanlp_common.conll import CoNLLUWord, CoNLLSentence\nfrom hanlp.components.parsers.ud.ud_model import UniversalDependenciesModel\nfrom hanlp.components.parsers.ud.util import generate_lemma_rule, append_bos, sample_form_missing\nfrom hanlp.components.parsers.ud.lemma_edit import apply_lemma_rule\nfrom hanlp.datasets.parsing.loaders.conll_dataset import CoNLLParsingDataset\nfrom hanlp.layers.embeddings.contextual_word_embedding import ContextualWordEmbedding\nfrom hanlp.metrics.accuracy import CategoricalAccuracy\nfrom hanlp.metrics.metric import Metric\nfrom hanlp.metrics.mtl import MetricDict\nfrom hanlp.metrics.parsing.attachmentscore import AttachmentScore\nfrom hanlp.utils.time_util import CountdownTimer\nfrom hanlp.utils.torch_util import clip_grad_norm, lengths_to_mask\nfrom hanlp_common.util import merge_locals_kwargs, merge_dict, reorder\n\n\nclass UniversalDependenciesParser(TorchComponent):\n\n    def __init__(self, **kwargs) -> None:\n        \"\"\"Universal Dependencies Parsing (lemmatization, features, PoS tagging and dependency parsing) implementation\n        of \"75 Languages, 1 Model: Parsing Universal Dependencies Universally\" (:cite:`kondratyuk-straka-2019-75`).\n\n        Args:\n            **kwargs: Predefined config.\n        \"\"\"\n        super().__init__(**kwargs)\n        self.model: UniversalDependenciesModel = self.model\n\n    def build_dataloader(self,\n                         data,\n                         batch_size,\n                         shuffle=False,\n                         device=None,\n                         logger: logging.Logger = None,\n                         sampler_builder=None,\n                         gradient_accumulation=1,\n                         transformer: ContextualWordEmbedding = None,\n                         **kwargs) -> DataLoader:\n        transform = [generate_lemma_rule, append_bos, self.vocabs, transformer.transform(), FieldLength('token')]\n        if not self.config.punct:\n            transform.append(PunctuationMask('token', 'punct_mask'))\n        dataset = self.build_dataset(data, transform)\n        if self.vocabs.mutable:\n            # noinspection PyTypeChecker\n            self.build_vocabs(dataset, logger)\n        lens = [len(x['token_input_ids']) for x in dataset]\n        if sampler_builder:\n            sampler = sampler_builder.build(lens, shuffle, gradient_accumulation)\n        else:\n            sampler = SortingSamplerBuilder(batch_size).build(lens, shuffle, gradient_accumulation)\n        return PadSequenceDataLoader(dataset, batch_size, shuffle, device=device, batch_sampler=sampler,\n                                     pad={'arc': 0}, )\n\n    def build_vocabs(self, trn, logger, **kwargs):\n        self.vocabs.pos = Vocab(unk_token=None, pad_token=None)\n        self.vocabs.rel = Vocab(unk_token=None, pad_token=None)\n        self.vocabs.lemma = Vocab(unk_token=None, pad_token=None)\n        self.vocabs.feat = Vocab(unk_token=None, pad_token=None)\n        timer = CountdownTimer(len(trn))\n        max_seq_len = 0\n        for each in trn:\n            max_seq_len = max(max_seq_len, len(each['token']))\n            timer.log(f'Building vocab [blink][yellow]...[/yellow][/blink] (longest sequence: {max_seq_len})')\n        for v in self.vocabs.values():\n            v.set_unk_as_safe_unk()\n        self.vocabs.lock()\n        self.vocabs.summary(logger)\n\n    def build_dataset(self, data, transform):\n        dataset = CoNLLParsingDataset(data, transform=transform, prune=sample_form_missing, cache=isinstance(data, str))\n        return dataset\n\n    def build_optimizer(self, trn, **kwargs):\n        # noinspection PyCallByClass,PyTypeChecker\n        return TransformerComponent.build_optimizer(self, trn, **kwargs)\n\n    def build_criterion(self, **kwargs):\n        pass\n\n    def build_metric(self, **kwargs):\n        return MetricDict({\n            'lemmas': CategoricalAccuracy(),\n            'upos': CategoricalAccuracy(),\n            'deps': AttachmentScore(),\n            'feats': CategoricalAccuracy(),\n        })\n\n    def evaluate_dataloader(self,\n                            data: DataLoader,\n                            criterion: Callable,\n                            metric: MetricDict = None,\n                            output=False,\n                            logger=None,\n                            ratio_width=None,\n                            **kwargs):\n\n        metric.reset()\n        self.model.eval()\n        timer = CountdownTimer(len(data))\n        total_loss = 0\n        for idx, batch in enumerate(data):\n            out, mask = self.feed_batch(batch)\n            loss = out['loss']\n            total_loss += loss.item()\n            self.decode_output(out, mask, batch)\n            self.update_metrics(metric, batch, out, mask)\n            report = f'loss: {total_loss / (idx + 1):.4f} {metric.cstr()}'\n            timer.log(report, logger=logger, ratio_percentage=False, ratio_width=ratio_width)\n            del loss\n            del out\n            del mask\n        return total_loss / len(data), metric\n\n    # noinspection PyMethodOverriding\n    def build_model(self,\n                    transformer: ContextualWordEmbedding,\n                    n_mlp_arc,\n                    n_mlp_rel,\n                    mlp_dropout,\n                    mix_embedding,\n                    layer_dropout,\n                    training=True,\n                    **kwargs) -> torch.nn.Module:\n        assert bool(transformer.scalar_mix) == bool(mix_embedding), 'transformer.scalar_mix has to be 1 ' \\\n                                                                    'when mix_embedding is non-zero.'\n        # noinspection PyTypeChecker\n        return UniversalDependenciesModel(transformer.module(training=training),\n                                          n_mlp_arc,\n                                          n_mlp_rel,\n                                          mlp_dropout,\n                                          len(self.vocabs.rel),\n                                          len(self.vocabs.lemma),\n                                          len(self.vocabs.pos),\n                                          len(self.vocabs.feat),\n                                          mix_embedding,\n                                          layer_dropout)\n\n    def predict(self, data: Union[List[str], List[List[str]]], batch_size: int = None, **kwargs):\n        if not data:\n            return []\n        flat = self.input_is_flat(data)\n        if flat:\n            data = [data]\n        samples = self.build_samples(data)\n        if not batch_size:\n            batch_size = self.config.batch_size\n        dataloader = self.build_dataloader(samples,\n                                           device=self.devices[0], shuffle=False,\n                                           **merge_dict(self.config,\n                                                        batch_size=batch_size,\n                                                        overwrite=True,\n                                                        **kwargs))\n        order = []\n        outputs = []\n        for batch in dataloader:\n            out, mask = self.feed_batch(batch)\n            self.decode_output(out, mask, batch)\n            outputs.extend(self.prediction_to_human(out, batch))\n            order.extend(batch[IDX])\n        outputs = reorder(outputs, order)\n        if flat:\n            return outputs[0]\n        return outputs\n\n    def build_samples(self, data: List[List[str]]):\n        return [{'FORM': x} for x in data]\n\n    def fit(self,\n            trn_data,\n            dev_data,\n            save_dir,\n            transformer: ContextualWordEmbedding,\n            sampler_builder=None,\n            mix_embedding: int = 13,\n            layer_dropout: int = 0.1,\n            n_mlp_arc=768,\n            n_mlp_rel=256,\n            mlp_dropout=.33,\n            lr=1e-3,\n            transformer_lr=2.5e-5,\n            patience=0.1,\n            batch_size=32,\n            epochs=30,\n            gradient_accumulation=1,\n            adam_epsilon=1e-8,\n            weight_decay=0,\n            warmup_steps=0.1,\n            grad_norm=1.0,\n            tree=False,\n            proj=False,\n            punct=False,\n            logger=None,\n            verbose=True,\n            devices: Union[float, int, List[int]] = None, **kwargs):\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    def execute_training_loop(self, trn: DataLoader, dev: DataLoader, epochs, criterion, optimizer, metric, save_dir,\n                              logger: logging.Logger, devices, ratio_width=None, patience=0.5, eval_trn=True, **kwargs):\n        if isinstance(patience, float):\n            patience = int(patience * epochs)\n        best_epoch, best_metric = 0, -1\n        timer = CountdownTimer(epochs)\n        history = History()\n        for epoch in range(1, epochs + 1):\n            logger.info(f\"[yellow]Epoch {epoch} / {epochs}:[/yellow]\")\n            self.fit_dataloader(trn, criterion, optimizer, metric, logger, history=history, ratio_width=ratio_width,\n                                eval_trn=eval_trn, **self.config)\n            loss, dev_metric = self.evaluate_dataloader(dev, criterion, metric, logger=logger, ratio_width=ratio_width)\n            timer.update()\n            report = f\"{timer.elapsed_human} / {timer.total_time_human} ETA: {timer.eta_human}\"\n            if dev_metric > best_metric:\n                best_epoch, best_metric = epoch, deepcopy(dev_metric)\n                self.save_weights(save_dir)\n                report += ' [red](saved)[/red]'\n            else:\n                report += f' ({epoch - best_epoch})'\n                if epoch - best_epoch >= patience:\n                    report += ' early stop'\n            logger.info(report)\n            if epoch - best_epoch >= patience:\n                break\n        if not best_epoch:\n            self.save_weights(save_dir)\n        elif best_epoch != epoch:\n            self.load_weights(save_dir)\n        logger.info(f\"Max score of dev is {best_metric.cstr()} at epoch {best_epoch}\")\n        logger.info(f\"Average time of each epoch is {timer.elapsed_average_human}\")\n        logger.info(f\"{timer.elapsed_human} elapsed\")\n\n    # noinspection PyMethodOverriding\n    def fit_dataloader(self,\n                       trn: DataLoader,\n                       criterion,\n                       optimizer,\n                       metric: MetricDict,\n                       logger: logging.Logger,\n                       history: History,\n                       gradient_accumulation=1,\n                       grad_norm=None,\n                       ratio_width=None,\n                       eval_trn=True,\n                       **kwargs):\n        optimizer, scheduler = optimizer\n        metric.reset()\n        self.model.train()\n        timer = CountdownTimer(history.num_training_steps(len(trn), gradient_accumulation=gradient_accumulation))\n        total_loss = 0\n        for idx, batch in enumerate(trn):\n            out, mask = self.feed_batch(batch)\n            loss = out['loss']\n            if gradient_accumulation and gradient_accumulation > 1:\n                loss /= gradient_accumulation\n            loss.backward()\n            total_loss += loss.item()\n            if eval_trn:\n                self.decode_output(out, mask, batch)\n                self.update_metrics(metric, batch, out, mask)\n            if history.step(gradient_accumulation):\n                self._step(optimizer, scheduler, grad_norm)\n                report = f'loss: {total_loss / (idx + 1):.4f} {metric.cstr()}' if eval_trn \\\n                    else f'loss: {total_loss / (idx + 1):.4f}'\n                timer.log(report, logger=logger, ratio_percentage=False, ratio_width=ratio_width)\n            del loss\n            del out\n            del mask\n\n    def decode_output(self, outputs, mask, batch):\n        arc_scores, rel_scores = outputs['class_probabilities']['deps']['s_arc'], \\\n                                 outputs['class_probabilities']['deps']['s_rel']\n        arc_preds, rel_preds = BiaffineDependencyParser.decode(self, arc_scores, rel_scores, mask, batch)\n        outputs['arc_preds'], outputs['rel_preds'] = arc_preds, rel_preds\n        return outputs\n\n    def update_metrics(self, metrics, batch, outputs, mask):\n        arc_preds, rel_preds, puncts = outputs['arc_preds'], outputs['rel_preds'], batch.get('punct_mask', None)\n        BiaffineDependencyParser.update_metric(self, arc_preds, rel_preds, batch['arc'], batch['rel_id'], mask, puncts,\n                                               metrics['deps'], batch)\n        for task, key in zip(['lemmas', 'upos', 'feats'], ['lemma_id', 'pos_id', 'feat_id']):\n            metric: Metric = metrics[task]\n            pred = outputs['class_probabilities'][task]\n            gold = batch[key]\n            metric(pred.detach(), gold, mask=mask)\n        return metrics\n\n    def feed_batch(self, batch: dict):\n        mask = self.compute_mask(batch)\n        output_dict = self.model(batch, mask)\n        if self.model.training:\n            mask = mask.clone()\n        mask[:, 0] = 0\n        return output_dict, mask\n\n    def compute_mask(self, batch):\n        lens = batch['token_length']\n        mask = lengths_to_mask(lens)\n        return mask\n\n    def _step(self, optimizer, scheduler, grad_norm):\n        clip_grad_norm(self.model, grad_norm)\n        optimizer.step()\n        scheduler.step()\n        optimizer.zero_grad()\n\n    def input_is_flat(self, data):\n        # noinspection PyCallByClass,PyTypeChecker\n        return BiaffineDependencyParser.input_is_flat(self, data, False)\n\n    def prediction_to_human(self, outputs: dict, batch):\n        arcs, rels = outputs['arc_preds'], outputs['rel_preds']\n        upos = outputs['class_probabilities']['upos'][:, 1:, :].argmax(-1).tolist()\n        feats = outputs['class_probabilities']['feats'][:, 1:, :].argmax(-1).tolist()\n        lemmas = outputs['class_probabilities']['lemmas'][:, 1:, :].argmax(-1).tolist()\n        lem_vocab = self.vocabs['lemma'].idx_to_token\n        pos_vocab = self.vocabs['pos'].idx_to_token\n        feat_vocab = self.vocabs['feat'].idx_to_token\n        # noinspection PyCallByClass,PyTypeChecker\n        for tree, form, lemma, pos, feat in zip(BiaffineDependencyParser.prediction_to_head_rel(\n                self, arcs, rels, batch), batch['token'], lemmas, upos, feats):\n            form = form[1:]\n            assert len(form) == len(tree)\n            lemma = [apply_lemma_rule(t, lem_vocab[r]) for t, r in zip(form, lemma)]\n            pos = [pos_vocab[x] for x in pos]\n            feat = [feat_vocab[x] for x in feat]\n            yield CoNLLSentence(\n                [CoNLLUWord(id=i + 1, form=fo, lemma=l, upos=p, feats=fe, head=a, deprel=r) for\n                 i, (fo, (a, r), l, p, fe) in enumerate(zip(form, tree, lemma, pos, feat))])\n\n    def __call__(self, data, batch_size=None, **kwargs) -> Union[CoNLLSentence, List[CoNLLSentence]]:\n        return super().__call__(data, batch_size, **kwargs)\n"
  },
  {
    "path": "hanlp/components/parsers/ud/udify_util.py",
    "content": "# This file is modified from udify and allennlp, which are licensed under the MIT license:\n# MIT License\n#\n# Copyright (c) 2019 Dan Kondratyuk and allennlp\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\n\nimport os\nfrom typing import List, Dict, Tuple, Union\n\nimport numpy\nimport torch\n\n\ndef get_ud_treebank_files(dataset_dir: str, treebanks: List[str] = None) -> Dict[str, Tuple[str, str, str]]:\n    \"\"\"Retrieves all treebank data paths in the given directory.\n    Adopted from https://github.com/Hyperparticle/udify\n    MIT Licence\n\n    Args:\n      dataset_dir: \n      treebanks: \n      dataset_dir: str: \n      treebanks: List[str]:  (Default value = None)\n\n    Returns:\n\n    \n    \"\"\"\n    datasets = {}\n    treebanks = os.listdir(dataset_dir) if not treebanks else treebanks\n    for treebank in treebanks:\n        treebank_path = os.path.join(dataset_dir, treebank)\n        conllu_files = [file for file in sorted(os.listdir(treebank_path)) if file.endswith(\".conllu\")]\n\n        train_file = [file for file in conllu_files if file.endswith(\"train.conllu\")]\n        dev_file = [file for file in conllu_files if file.endswith(\"dev.conllu\")]\n        test_file = [file for file in conllu_files if file.endswith(\"test.conllu\")]\n\n        train_file = os.path.join(treebank_path, train_file[0]) if train_file else None\n        dev_file = os.path.join(treebank_path, dev_file[0]) if dev_file else None\n        test_file = os.path.join(treebank_path, test_file[0]) if test_file else None\n\n        datasets[treebank] = (train_file, dev_file, test_file)\n    return datasets\n\n\ndef sequence_cross_entropy(log_probs: torch.FloatTensor,\n                           targets: torch.LongTensor,\n                           weights: torch.FloatTensor,\n                           average: str = \"batch\",\n                           label_smoothing: float = None) -> torch.FloatTensor:\n    if average not in {None, \"token\", \"batch\"}:\n        raise ValueError(\"Got average f{average}, expected one of \"\n                         \"None, 'token', or 'batch'\")\n    # shape : (batch * sequence_length, num_classes)\n    log_probs_flat = log_probs.view(-1, log_probs.size(2))\n    # shape : (batch * max_len, 1)\n    targets_flat = targets.view(-1, 1).long()\n\n    if label_smoothing is not None and label_smoothing > 0.0:\n        num_classes = log_probs.size(-1)\n        smoothing_value = label_smoothing / num_classes\n        # Fill all the correct indices with 1 - smoothing value.\n        one_hot_targets = torch.zeros_like(log_probs_flat).scatter_(-1, targets_flat, 1.0 - label_smoothing)\n        smoothed_targets = one_hot_targets + smoothing_value\n        negative_log_likelihood_flat = - log_probs_flat * smoothed_targets\n        negative_log_likelihood_flat = negative_log_likelihood_flat.sum(-1, keepdim=True)\n    else:\n        # Contribution to the negative log likelihood only comes from the exact indices\n        # of the targets, as the target distributions are one-hot. Here we use torch.gather\n        # to extract the indices of the num_classes dimension which contribute to the loss.\n        # shape : (batch * sequence_length, 1)\n        negative_log_likelihood_flat = - torch.gather(log_probs_flat, dim=1, index=targets_flat)\n    # shape : (batch, sequence_length)\n    negative_log_likelihood = negative_log_likelihood_flat.view(*targets.size())\n    # shape : (batch, sequence_length)\n    negative_log_likelihood = negative_log_likelihood * weights.float()\n\n    if average == \"batch\":\n        # shape : (batch_size,)\n        per_batch_loss = negative_log_likelihood.sum(1) / (weights.sum(1).float() + 1e-13)\n        num_non_empty_sequences = ((weights.sum(1) > 0).float().sum() + 1e-13)\n        return per_batch_loss.sum() / num_non_empty_sequences\n    elif average == \"token\":\n        return negative_log_likelihood.sum() / (weights.sum().float() + 1e-13)\n    else:\n        # shape : (batch_size,)\n        per_batch_loss = negative_log_likelihood.sum(1) / (weights.sum(1).float() + 1e-13)\n        return per_batch_loss\n\n\ndef sequence_cross_entropy_with_logits(\n        logits: torch.FloatTensor,\n        targets: torch.LongTensor,\n        weights: Union[torch.FloatTensor, torch.BoolTensor],\n        average: str = \"batch\",\n        label_smoothing: float = None,\n        gamma: float = None,\n        alpha: Union[float, List[float], torch.FloatTensor] = None,\n) -> torch.FloatTensor:\n    \"\"\"Computes the cross entropy loss of a sequence, weighted with respect to\n    some user provided weights. Note that the weighting here is not the same as\n    in the `torch.nn.CrossEntropyLoss()` criterion, which is weighting\n    classes; here we are weighting the loss contribution from particular elements\n    in the sequence. This allows loss computations for models which use padding.\n    \n    # Parameters\n    \n    logits : `torch.FloatTensor`, required.\n        A `torch.FloatTensor` of size (batch_size, sequence_length, num_classes)\n        which contains the unnormalized probability for each class.\n    targets : `torch.LongTensor`, required.\n        A `torch.LongTensor` of size (batch, sequence_length) which contains the\n        index of the true class for each corresponding step.\n    weights : `Union[torch.FloatTensor, torch.BoolTensor]`, required.\n        A `torch.FloatTensor` of size (batch, sequence_length)\n    average: `str`, optional (default = `\"batch\"`)\n        If \"batch\", average the loss across the batches. If \"token\", average\n        the loss across each item in the input. If `None`, return a vector\n        of losses per batch element.\n    label_smoothing : `float`, optional (default = `None`)\n        Whether or not to apply label smoothing to the cross-entropy loss.\n        For example, with a label smoothing value of 0.2, a 4 class classification\n        target would look like `[0.05, 0.05, 0.85, 0.05]` if the 3rd class was\n        the correct label.\n    gamma : `float`, optional (default = `None`)\n        Focal loss[*] focusing parameter `gamma` to reduces the relative loss for\n        well-classified examples and put more focus on hard. The greater value\n        `gamma` is, the more focus on hard examples.\n    alpha : `Union[float, List[float]]`, optional (default = `None`)\n        Focal loss[*] weighting factor `alpha` to balance between classes. Can be\n        used independently with `gamma`. If a single `float` is provided, it\n        is assumed binary case using `alpha` and `1 - alpha` for positive and\n        negative respectively. If a list of `float` is provided, with the same\n        length as the number of classes, the weights will match the classes.\n        [*] T. Lin, P. Goyal, R. Girshick, K. He and P. Dollár, \"Focal Loss for\n        Dense Object Detection,\" 2017 IEEE International Conference on Computer\n        Vision (ICCV), Venice, 2017, pp. 2999-3007.\n    \n    # Returns\n    \n    `torch.FloatTensor`\n        A torch.FloatTensor representing the cross entropy loss.\n        If `average==\"batch\"` or `average==\"token\"`, the returned loss is a scalar.\n        If `average is None`, the returned loss is a vector of shape (batch_size,).\n\n    Args:\n      logits: torch.FloatTensor: \n      targets: torch.LongTensor: \n      weights: Union[torch.FloatTensor: \n      torch.BoolTensor]: \n      average: str:  (Default value = \"batch\")\n      label_smoothing: float:  (Default value = None)\n      gamma: float:  (Default value = None)\n      alpha: Union[float: \n      List[float]: \n      torch.FloatTensor]:  (Default value = None)\n\n    Returns:\n\n    \"\"\"\n    if average not in {None, \"token\", \"batch\"}:\n        raise ValueError(\"Got average f{average}, expected one of None, 'token', or 'batch'\")\n\n    # make sure weights are float\n    weights = weights.to(logits.dtype)\n    # sum all dim except batch\n    non_batch_dims = tuple(range(1, len(weights.shape)))\n    # shape : (batch_size,)\n    weights_batch_sum = weights.sum(dim=non_batch_dims)\n    # shape : (batch * sequence_length, num_classes)\n    logits_flat = logits.view(-1, logits.size(-1))\n    # shape : (batch * sequence_length, num_classes)\n    log_probs_flat = torch.nn.functional.log_softmax(logits_flat, dim=-1)\n    # shape : (batch * max_len, 1)\n    targets_flat = targets.view(-1, 1).long()\n    # focal loss coefficient\n    if gamma:\n        # shape : (batch * sequence_length, num_classes)\n        probs_flat = log_probs_flat.exp()\n        # shape : (batch * sequence_length,)\n        probs_flat = torch.gather(probs_flat, dim=1, index=targets_flat)\n        # shape : (batch * sequence_length,)\n        focal_factor = (1.0 - probs_flat) ** gamma\n        # shape : (batch, sequence_length)\n        focal_factor = focal_factor.view(*targets.size())\n        weights = weights * focal_factor\n\n    if alpha is not None:\n        # shape : () / (num_classes,)\n        if isinstance(alpha, (float, int)):\n\n            # shape : (2,)\n            alpha_factor = torch.tensor(\n                [1.0 - float(alpha), float(alpha)], dtype=weights.dtype, device=weights.device\n            )\n\n        elif isinstance(alpha, (list, numpy.ndarray, torch.Tensor)):\n\n            # shape : (c,)\n            alpha_factor = torch.tensor(alpha, dtype=weights.dtype, device=weights.device)\n\n            if not alpha_factor.size():\n                # shape : (1,)\n                alpha_factor = alpha_factor.view(1)\n                # shape : (2,)\n                alpha_factor = torch.cat([1 - alpha_factor, alpha_factor])\n        else:\n            raise TypeError(\n                (\"alpha must be float, list of float, or torch.FloatTensor, {} provided.\").format(\n                    type(alpha)\n                )\n            )\n        # shape : (batch, max_len)\n        alpha_factor = torch.gather(alpha_factor, dim=0, index=targets_flat.view(-1)).view(\n            *targets.size()\n        )\n        weights = weights * alpha_factor\n\n    if label_smoothing is not None and label_smoothing > 0.0:\n        num_classes = logits.size(-1)\n        smoothing_value = label_smoothing / num_classes\n        # Fill all the correct indices with 1 - smoothing value.\n        one_hot_targets = torch.zeros_like(log_probs_flat).scatter_(\n            -1, targets_flat, 1.0 - label_smoothing\n        )\n        smoothed_targets = one_hot_targets + smoothing_value\n        negative_log_likelihood_flat = -log_probs_flat * smoothed_targets\n        negative_log_likelihood_flat = negative_log_likelihood_flat.sum(-1, keepdim=True)\n    else:\n        # Contribution to the negative log likelihood only comes from the exact indices\n        # of the targets, as the target distributions are one-hot. Here we use torch.gather\n        # to extract the indices of the num_classes dimension which contribute to the loss.\n        # shape : (batch * sequence_length, 1)\n        negative_log_likelihood_flat = -torch.gather(log_probs_flat, dim=1, index=targets_flat)\n    # shape : (batch, sequence_length)\n    negative_log_likelihood = negative_log_likelihood_flat.view(*targets.size())\n    # shape : (batch, sequence_length)\n    negative_log_likelihood = negative_log_likelihood * weights\n\n    if average == \"batch\":\n        # shape : (batch_size,)\n        per_batch_loss = negative_log_likelihood.sum(non_batch_dims) / (\n                weights_batch_sum + tiny_value_of_dtype(negative_log_likelihood.dtype)\n        )\n        num_non_empty_sequences = (weights_batch_sum > 0).sum() + tiny_value_of_dtype(\n            negative_log_likelihood.dtype\n        )\n        return per_batch_loss.sum() / num_non_empty_sequences\n    elif average == \"token\":\n        return negative_log_likelihood.sum() / (\n                weights_batch_sum.sum() + tiny_value_of_dtype(negative_log_likelihood.dtype)\n        )\n    else:\n        # shape : (batch_size,)\n        per_batch_loss = negative_log_likelihood.sum(non_batch_dims) / (\n                weights_batch_sum + tiny_value_of_dtype(negative_log_likelihood.dtype)\n        )\n        return per_batch_loss\n\n\ndef tiny_value_of_dtype(dtype: torch.dtype):\n    \"\"\"Returns a moderately tiny value for a given PyTorch data type that is used to avoid numerical\n    issues such as division by zero.\n    This is different from `info_value_of_dtype(dtype).tiny` because it causes some NaN bugs.\n    Only supports floating point dtypes.\n\n    Args:\n      dtype: torch.dtype: \n\n    Returns:\n\n    \"\"\"\n    if not dtype.is_floating_point:\n        raise TypeError(\"Only supports floating point dtypes.\")\n    if dtype == torch.float or dtype == torch.double:\n        return 1e-13\n    elif dtype == torch.half:\n        return 1e-4\n    else:\n        raise TypeError(\"Does not support dtype \" + str(dtype))\n\n\ndef combine_initial_dims_to_1d_or_2d(tensor: torch.Tensor) -> torch.Tensor:\n    \"\"\"Given a (possibly higher order) tensor of ids with shape\n    (d1, ..., dn, sequence_length)\n\n    Args:\n      tensor: torch.Tensor: \n\n    Returns:\n      If original tensor is 1-d or 2-d, return it as is.\n\n    \"\"\"\n    if tensor.dim() <= 2:\n        return tensor\n    else:\n        return tensor.view(-1, tensor.size(-1))\n\n\ndef uncombine_initial_dims(tensor: torch.Tensor, original_size: torch.Size) -> torch.Tensor:\n    \"\"\"Given a tensor of embeddings with shape\n    (d1 * ... * dn, sequence_length, embedding_dim)\n    and the original shape\n    (d1, ..., dn, sequence_length),\n\n    Args:\n      tensor: torch.Tensor: \n      original_size: torch.Size: \n\n    Returns:\n      (d1, ..., dn, sequence_length, embedding_dim).\n      If original size is 1-d or 2-d, return it as is.\n\n    \"\"\"\n    if len(original_size) <= 2:\n        return tensor\n    else:\n        view_args = list(original_size) + [tensor.size(-1)]\n        return tensor.view(*view_args)\n\n\ndef get_range_vector(size: int, device: int) -> torch.Tensor:\n    \"\"\"Returns a range vector with the desired size, starting at 0. The CUDA implementation\n    is meant to avoid copy data from CPU to GPU.\n\n    Args:\n      size: int: \n      device: int: \n\n    Returns:\n\n    \"\"\"\n    if device > -1:\n        return torch.cuda.LongTensor(size, device=device).fill_(1).cumsum(0) - 1\n    else:\n        return torch.arange(0, size, dtype=torch.long)\n\n\ndef get_device_of(tensor: torch.Tensor) -> int:\n    \"\"\"Returns the device of the tensor.\n\n    Args:\n      tensor: torch.Tensor: \n\n    Returns:\n\n    \"\"\"\n    if not tensor.is_cuda:\n        return -1\n    else:\n        return tensor.get_device()\n"
  },
  {
    "path": "hanlp/components/parsers/ud/util.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-14 20:44\nfrom hanlp_common.constant import ROOT\nfrom hanlp.components.parsers.ud.lemma_edit import gen_lemma_rule\n\n\ndef generate_lemma_rule(sample: dict):\n    if 'LEMMA' in sample:\n        sample['lemma'] = [gen_lemma_rule(word, lemma) if lemma != \"_\" else \"_\" for word, lemma in\n                           zip(sample['FORM'], sample['LEMMA'])]\n    return sample\n\n\ndef append_bos(sample: dict):\n    if 'FORM' in sample:\n        sample['token'] = [ROOT] + sample['FORM']\n    if 'UPOS' in sample:\n        sample['pos'] = sample['UPOS'][:1] + sample['UPOS']\n        sample['arc'] = [0] + sample['HEAD']\n        sample['rel'] = sample['DEPREL'][:1] + sample['DEPREL']\n        sample['lemma'] = sample['lemma'][:1] + sample['lemma']\n        sample['feat'] = sample['FEATS'][:1] + sample['FEATS']\n    return sample\n\n\ndef sample_form_missing(sample: dict):\n    return all(t == '_' for t in sample['FORM'])\n"
  },
  {
    "path": "hanlp/components/pipeline.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-31 00:22\nimport types\nfrom typing import Callable, Union, Iterable, Any\nfrom hanlp.components.lambda_wrapper import LambdaComponent\nfrom hanlp.common.component import Component\nfrom hanlp_common.document import Document\nfrom hanlp.utils.component_util import load_from_meta\nfrom hanlp_common.io import save_json, load_json\nfrom hanlp_common.reflection import str_to_type, classpath_of\nimport hanlp\n\n\nclass Pipe(Component):\n\n    def __init__(self, component: Component, input_key: str = None, output_key: str = None, **kwargs) -> None:\n        super().__init__()\n        if not hasattr(self, 'config'):\n            self.config = {'classpath': classpath_of(self)}\n        self.output_key = output_key\n        self.input_key = input_key\n        self.component = component\n        self.kwargs = kwargs\n        self.config.update({\n            'component': component.config,\n            'input_key': self.input_key,\n            'output_key': self.output_key,\n            'kwargs': self.kwargs\n        })\n\n    # noinspection PyShadowingBuiltins\n    def predict(self, doc: Document, **kwargs) -> Document:\n\n        unpack = False\n        if self.input_key:\n            if isinstance(self.input_key, (tuple, list)):\n                if isinstance(self.component, LambdaComponent):  # assume functions take multiple arguments\n                    input = [doc[key] for key in self.input_key]\n                    unpack = True\n                else:\n                    input = list(list(zip(*sent)) for sent in zip(*[doc[key] for key in self.input_key]))\n            else:\n                input = doc[self.input_key]\n        else:\n            input = doc\n\n        if self.kwargs:\n            kwargs.update(self.kwargs)\n        if unpack:\n            kwargs['_hanlp_unpack'] = True\n        output = self.component(input, **kwargs)\n        if isinstance(output, types.GeneratorType):\n            output = list(output)\n        if self.output_key:\n            if not isinstance(doc, Document):\n                doc = Document()\n            if isinstance(self.output_key, tuple):\n                for key, value in zip(self.output_key, output):\n                    doc[key] = value\n            else:\n                doc[self.output_key] = output\n            return doc\n        return output\n\n    def __repr__(self):\n        name = self.component.function.__name__ if isinstance(self.component, LambdaComponent) \\\n            else self.component.__class__.__name__\n        return f'{self.input_key}->{name}->{self.output_key}'\n\n    @staticmethod\n    def from_config(meta: dict, **kwargs):\n        cls = str_to_type(meta['classpath'])\n        component = load_from_meta(meta['component'])\n        return cls(component, meta['input_key'], meta['output_key'], **meta['kwargs'])\n\n\nclass Pipeline(Component, list):\n    def __init__(self, *pipes: Pipe) -> None:\n        super().__init__()\n        if not hasattr(self, 'config'):\n            self.config = {'classpath': classpath_of(self)}\n        if pipes:\n            self.extend(pipes)\n\n    def append(self, component: Callable, input_key: Union[str, Iterable[str]] = None,\n               output_key: Union[str, Iterable[str]] = None, **kwargs):\n        \"\"\"\n        Append a pipe to the tail of this pipeline.\n\n        Args:\n            component: A callable function.\n            input_key: The input key indicating which fields will be inputted to the pipe. ``None``: inherit from\n                previous pipe; ``*``: use all the outputs from previous pipes wrapped in a\n                :class:`~hanlp_common.document.Document`.\n            output_key: The output key indicating where to store the outputs\n            **kwargs: Extra arguments passed to the ``Pipe`` constructor.\n\n        Returns:\n\n            Pipeline: A pipeline.\n        \"\"\"\n        self.insert(len(self), component, input_key, output_key, **kwargs)\n        return self\n\n    def insert(self, index: int, component: Callable, input_key: Union[str, Iterable[str]] = None,\n               output_key: Union[str, Iterable[str]] = None,\n               **kwargs):\n        \"\"\"\n\n        Args:\n            index: The index of the new pipe.\n            input_key: The input key indicating which fields will be inputted to the pipe. ``None``: inherit from\n                previous pipe; ``*``: use all the outputs from previous pipes wrapped in a\n                :class:`~hanlp_common.document.Document`.\n            output_key: The output key indicating where to store the outputs\n            **kwargs: Extra arguments passed to the ``Pipe`` constructor.\n\n        Returns:\n\n            Pipeline: A pipeline.\n        \"\"\"\n        if input_key == '*':\n            input_key = None\n        elif not input_key and len(self) and index:\n            input_key = self[index - 1].output_key\n        if not isinstance(component, Component):\n            component = LambdaComponent(component)\n        super().insert(index, Pipe(component, input_key, output_key, **kwargs))\n        return self\n\n    def __call__(self, doc: Union[Document, Any] = None, **kwargs) -> Document:\n        \"\"\"Run the pipeline as a function.\n\n        Args:\n            doc: A :class:`~hanlp_common.document.Document` or other data types.\n            **kwargs: If `doc` is set to None then create a :class:`~hanlp_common.document.Document` as the\n                input to the first pipe using all the parameters in ``kwargs``.\n\n        Returns:\n            A :class:`~hanlp_common.document.Document`.\n        \"\"\"\n        if doc is None:\n            doc = Document(**kwargs)\n        for component in self:\n            doc = component(doc)\n        return doc\n    \n    def copy(self):\n        return self.__copy__()\n    \n    def __copy__(self):\n        config = self.meta\n        return Pipeline.from_config(config)\n    \n    @property\n    def meta(self):\n        return {\n            'classpath': classpath_of(self),\n            'hanlp_version': hanlp.version.__version__,\n            'pipes': [pipe.config for pipe in self]\n        }    \n    @meta.setter\n    def meta(self, value):\n        pass\n\n    def save(self, filepath):\n        save_json(self.meta, filepath)\n\n    def load(self, filepath):\n        meta = load_json(filepath)\n        self.clear()\n        self.extend(Pipeline.from_config(meta))\n\n    @staticmethod\n    def from_config(meta: Union[dict, str], **kwargs):\n        if isinstance(meta, str):\n            meta = load_json(meta)\n        return Pipeline(*[load_from_meta(pipe) for pipe in meta['pipes']])\n"
  },
  {
    "path": "hanlp/components/rnn_language_model_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-04 17:28\nfrom typing import List, Union\n\nimport tensorflow as tf\n\nfrom hanlp.common.keras_component import KerasComponent\nfrom hanlp.transform.text_tf import TextTransform\n\n\nclass RNNLanguageModel(KerasComponent):\n\n    def __init__(self, transform: TextTransform = None) -> None:\n        if not transform:\n            transform = TextTransform()\n        super().__init__(transform)\n        self.transform: TextTransform = transform\n\n    def fit(self, trn_data, dev_data, save_dir,\n            forward=True,\n            embedding=100,\n            rnn_input_dropout=0.1,\n            rnn_units: int = 1024,\n            rnn_output_dropout=0.1,\n            seq_len: int = 250,\n            optimizer='sgd',\n            learning_rate=20,\n            anneal_factor: float = 0.25,\n            anneal_patience: int = 10,\n            clipnorm=0.25,\n            batch_size: int = 100, epochs=1000, run_eagerly=False, logger=None, verbose=True,\n            **kwargs):\n        return super().fit(**dict((k, v) for k, v in locals().items() if k not in ('self', 'kwargs')))\n\n    def build_model(self, embedding, rnn_input_dropout, rnn_units, rnn_output_dropout, batch_size, seq_len, training,\n                    **kwargs) -> tf.keras.Model:\n        model = tf.keras.Sequential()\n        extra_args = {}\n        if training:\n            extra_args['batch_input_shape'] = [batch_size, seq_len]\n        embedding = tf.keras.layers.Embedding(input_dim=len(self.transform.vocab), output_dim=embedding,\n                                              trainable=True, mask_zero=True, **extra_args)\n        model.add(embedding)\n        if rnn_input_dropout:\n            model.add(tf.keras.layers.Dropout(rnn_input_dropout, name='rnn_input_dropout'))\n        model.add(tf.keras.layers.LSTM(units=rnn_units, return_sequences=True, stateful=training, name='encoder'))\n        if rnn_output_dropout:\n            model.add(tf.keras.layers.Dropout(rnn_output_dropout, name='rnn_output_dropout'))\n        model.add(tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(len(self.transform.vocab)), name='decoder'))\n        return model\n\n    # noinspection PyMethodOverriding\n    def build_optimizer(self, optimizer, learning_rate, clipnorm, **kwargs):\n        if optimizer == 'sgd':\n            optimizer = tf.keras.optimizers.SGD(learning_rate=learning_rate, clipnorm=clipnorm)\n        return super().build_optimizer(optimizer, **kwargs)\n\n    def build_train_dataset(self, trn_data, batch_size):\n        trn_data = self.transform.file_to_dataset(trn_data, batch_size=batch_size, shuffle=False, repeat=-1)\n        return trn_data\n\n    def build_valid_dataset(self, dev_data, batch_size):\n        dev_data = self.transform.file_to_dataset(dev_data, batch_size=batch_size, shuffle=False, drop_remainder=True)\n        return dev_data\n\n    def generate_text(self, text: Union[str, List[str]] = '\\n', num_steps=50):\n        char_mode = False\n        if isinstance(text, str):\n            text = list(text)\n            char_mode = True\n        forward = self.config['forward']\n        # A slow implementation. Might better to let LSTM return states.\n        # But anyway, this interface is for fun so let's take it easy\n        for step in range(num_steps):\n            output = self.predict(text)\n            first_or_last_token = output[-1]\n            if forward:\n                text += first_or_last_token\n            else:\n                text = [first_or_last_token] + text\n        if char_mode:\n            text = ''.join(text)\n        return text\n"
  },
  {
    "path": "hanlp/components/srl/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-22 20:50"
  },
  {
    "path": "hanlp/components/srl/span_bio/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-04 13:59\n"
  },
  {
    "path": "hanlp/components/srl/span_bio/baffine_tagging.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-04 13:59\nimport math\n\nimport torch\nfrom torch import nn\n\nfrom hanlp.components.parsers.biaffine.biaffine import Biaffine\nfrom hanlp.components.parsers.biaffine.mlp import MLP\nfrom hanlp.layers.crf.crf import CRF\n\n\nclass BiaffineTaggingDecoder(nn.Module):\n\n    def __init__(self,\n                 n_rels,\n                 hidden_size,\n                 n_mlp_rel=300,\n                 mlp_dropout=0.2,\n                 crf=False) -> None:\n        super().__init__()\n        self.mlp_rel_h = MLP(n_in=hidden_size,\n                             n_out=n_mlp_rel,\n                             dropout=mlp_dropout)\n        self.mlp_rel_d = MLP(n_in=hidden_size,\n                             n_out=n_mlp_rel,\n                             dropout=mlp_dropout)\n        self.rel_attn = Biaffine(n_in=n_mlp_rel,\n                                 n_out=n_rels,\n                                 bias_x=True,\n                                 bias_y=True)\n        bias = 1 / math.sqrt(self.rel_attn.weight.size(1))\n        nn.init.uniform_(self.rel_attn.weight, -bias, bias)\n        self.crf = CRF(n_rels) if crf else None\n\n    # noinspection PyUnusedLocal\n    def forward(self, x: torch.Tensor, **kwargs):\n        rel_h = self.mlp_rel_h(x)\n        rel_d = self.mlp_rel_d(x)\n\n        # get arc and rel scores from the bilinear attention\n        # [batch_size, seq_len, seq_len, n_rels]\n        s_rel = self.rel_attn(rel_d, rel_h).permute(0, 2, 3, 1)\n        return s_rel\n\n\nclass SpanBIOSemanticRoleLabelingModel(nn.Module):\n\n    def __init__(self,\n                 embed,\n                 encoder,\n                 num_labels: int,\n                 n_mlp_rel,\n                 mlp_dropout,\n                 crf=False,\n                 ) -> None:\n        super().__init__()\n        self.embed = embed\n        self.encoder = encoder\n        hidden_size = encoder.get_output_dim() if encoder else embed.get_output_dim()\n        self.decoder = BiaffineTaggingDecoder(\n            num_labels,\n            hidden_size,\n            n_mlp_rel,\n            mlp_dropout,\n            crf,\n        )\n\n    def forward(self, batch, mask):\n        x = self.embed(batch)\n        if self.encoder:\n            x = self.encoder(x, mask=mask)\n        x = self.decoder(x)\n        return x\n"
  },
  {
    "path": "hanlp/components/srl/span_bio/span_bio.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-22 20:54\nimport logging\nfrom copy import copy\nfrom typing import Union, List, Callable, Dict, Any\nfrom bisect import bisect\nimport torch\nimport torch.nn.functional as F\nfrom torch import nn\nfrom torch.utils.data import DataLoader\n\nfrom hanlp_common.constant import IDX, PRED\nfrom hanlp.common.dataset import PadSequenceDataLoader, SamplerBuilder, TransformableDataset\nfrom hanlp.common.structure import History\nfrom hanlp.common.torch_component import TorchComponent\nfrom hanlp.common.transform import FieldLength\nfrom hanlp.common.vocab import Vocab\nfrom hanlp.components.srl.span_bio.baffine_tagging import SpanBIOSemanticRoleLabelingModel\nfrom hanlp.datasets.srl.loaders.conll2012 import CoNLL2012SRLBIODataset\nfrom hanlp.layers.crf.crf import CRF\nfrom hanlp.layers.embeddings.contextual_word_embedding import find_transformer\nfrom hanlp.layers.embeddings.embedding import Embedding\nfrom hanlp.layers.transformers.utils import build_optimizer_scheduler_with_transformer\nfrom hanlp.metrics.chunking.sequence_labeling import get_entities\nfrom hanlp.metrics.f1 import F1\nfrom hanlp.utils.string_util import guess_delimiter\nfrom hanlp.utils.time_util import CountdownTimer\nfrom hanlp.utils.torch_util import clip_grad_norm, lengths_to_mask\nfrom hanlp_common.util import merge_locals_kwargs, reorder\n\n\nclass SpanBIOSemanticRoleLabeler(TorchComponent):\n\n    def __init__(self, **kwargs) -> None:\n        \"\"\"A span based Semantic Role Labeling task using BIO scheme for tagging the role of each token. Given a\n        predicate and a token, it uses biaffine (:cite:`dozat:17a`) to predict their relations as one of BIO-ROLE.\n\n        Args:\n            **kwargs: Predefined config.\n        \"\"\"\n        super().__init__(**kwargs)\n        self.model: SpanBIOSemanticRoleLabelingModel = None\n\n    def build_optimizer(self,\n                        trn,\n                        epochs,\n                        lr,\n                        adam_epsilon,\n                        weight_decay,\n                        warmup_steps,\n                        transformer_lr=None,\n                        gradient_accumulation=1,\n                        **kwargs):\n        num_training_steps = len(trn) * epochs // gradient_accumulation\n        if transformer_lr is None:\n            transformer_lr = lr\n        transformer = find_transformer(self.model.embed)\n        optimizer, scheduler = build_optimizer_scheduler_with_transformer(self.model, transformer,\n                                                                          lr, transformer_lr,\n                                                                          num_training_steps, warmup_steps,\n                                                                          weight_decay, adam_epsilon)\n        return optimizer, scheduler\n\n    def build_criterion(self, decoder=None, **kwargs):\n        if self.config.crf:\n            if not decoder:\n                decoder = self.model.decoder\n            if isinstance(decoder, torch.nn.DataParallel):\n                decoder = decoder.module\n            return decoder.crf\n        else:\n            return nn.CrossEntropyLoss(reduction=self.config.loss_reduction)\n\n    def build_metric(self, **kwargs):\n        return F1()\n\n    def execute_training_loop(self,\n                              trn: DataLoader,\n                              dev: DataLoader,\n                              epochs,\n                              criterion,\n                              optimizer,\n                              metric,\n                              save_dir,\n                              logger: logging.Logger,\n                              devices,\n                              ratio_width=None,\n                              patience=0.5,\n                              **kwargs):\n        if isinstance(patience, float):\n            patience = int(patience * epochs)\n        best_epoch, best_metric = 0, -1\n        timer = CountdownTimer(epochs)\n        history = History()\n        for epoch in range(1, epochs + 1):\n            logger.info(f\"[yellow]Epoch {epoch} / {epochs}:[/yellow]\")\n            self.fit_dataloader(trn, criterion, optimizer, metric, logger, history=history, ratio_width=ratio_width,\n                                **self.config)\n            loss, dev_metric = self.evaluate_dataloader(dev, criterion, metric, logger=logger, ratio_width=ratio_width)\n            timer.update()\n            report = f\"{timer.elapsed_human} / {timer.total_time_human} ETA: {timer.eta_human}\"\n            if dev_metric > best_metric:\n                best_epoch, best_metric = epoch, copy(dev_metric)\n                self.save_weights(save_dir)\n                report += ' [red](saved)[/red]'\n            else:\n                report += f' ({epoch - best_epoch})'\n                if epoch - best_epoch >= patience:\n                    report += ' early stop'\n            logger.info(report)\n            if epoch - best_epoch >= patience:\n                break\n        if not best_epoch:\n            self.save_weights(save_dir)\n        elif best_epoch != epoch:\n            self.load_weights(save_dir)\n        logger.info(f\"Max score of dev is {best_metric} at epoch {best_epoch}\")\n        logger.info(f\"Average time of each epoch is {timer.elapsed_average_human}\")\n        logger.info(f\"{timer.elapsed_human} elapsed\")\n\n    # noinspection PyMethodOverriding\n    def fit_dataloader(self,\n                       trn: DataLoader,\n                       criterion,\n                       optimizer,\n                       metric,\n                       logger: logging.Logger,\n                       history: History,\n                       gradient_accumulation=1,\n                       grad_norm=None,\n                       ratio_width=None,\n                       eval_trn=False,\n                       **kwargs):\n        optimizer, scheduler = optimizer\n        self.model.train()\n        timer = CountdownTimer(history.num_training_steps(len(trn), gradient_accumulation=gradient_accumulation))\n        total_loss = 0\n        for idx, batch in enumerate(trn):\n            pred, mask = self.feed_batch(batch)\n            loss = self.compute_loss(criterion, pred, batch['srl_id'], mask)\n            if gradient_accumulation and gradient_accumulation > 1:\n                loss /= gradient_accumulation\n            loss.backward()\n            total_loss += loss.item()\n            if eval_trn:\n                prediction = self.decode_output(pred, mask, batch)\n                self.update_metrics(metric, prediction, batch)\n            if history.step(gradient_accumulation):\n                self._step(optimizer, scheduler, grad_norm)\n                report = f'loss: {total_loss / (idx + 1):.4f} {metric}' if eval_trn else f'loss: {total_loss / (idx + 1):.4f}'\n                timer.log(report, logger=logger, ratio_percentage=False, ratio_width=ratio_width)\n            del loss\n            del pred\n            del mask\n\n    def naive_decode(self, pred, mask, batch, decoder=None):\n        vocab = self.vocabs['srl'].idx_to_token\n        results = []\n        for sent, matrix in zip(batch['token'], pred.argmax(-1).tolist()):\n            results.append([])\n            for token, tags_per_token in zip(sent, matrix):\n                tags_per_token = [vocab[x] for x in tags_per_token][:len(sent)]\n                srl_per_token = get_entities(tags_per_token)\n                results[-1].append(srl_per_token)\n        return results\n\n    def decode_output(self, pred, mask, batch, decoder=None):\n        # naive = self.naive_decode(pred, mask, batch, decoder)\n        vocab = self.vocabs['srl'].idx_to_token\n        if mask is not None:\n            if self.config.crf:\n                if not decoder:\n                    decoder = self.model.decoder\n                crf: CRF = decoder.crf\n                token_index, mask = mask\n                pred = crf.decode(pred, mask)\n                pred = sum(pred, [])\n            else:\n                pred = pred[mask].argmax(-1)\n                pred = pred.tolist()\n        pred = [vocab[x] for x in pred]\n        results = []\n        offset = 0\n        for sent in batch['token']:\n            results.append([])\n            for token in sent:\n                tags_per_token = pred[offset:offset + len(sent)]\n                srl_per_token = get_entities(tags_per_token)\n                results[-1].append(srl_per_token)\n                offset += len(sent)\n        assert offset == len(pred)\n        # assert results == naive\n        return results\n\n    def update_metrics(self, metric, prediction, batch):\n        for p, g in zip(prediction, batch['srl_set']):\n            srl = set()\n            for i, args in enumerate(p):\n                srl.update((i, start, end, label) for (label, start, end) in args)\n            metric(srl, g)\n        return metric\n\n    def feed_batch(self, batch: dict):\n        lens = batch['token_length']\n        mask2d = lengths_to_mask(lens)\n        pred = self.model(batch, mask=mask2d)\n        mask3d = self.compute_mask(mask2d)\n        if self.config.crf:\n            token_index = mask3d[0]\n            pred = pred.flatten(end_dim=1)[token_index]\n            pred = F.log_softmax(pred, dim=-1)\n        return pred, mask3d\n\n    def compute_mask(self, mask2d):\n        mask3d = mask2d.unsqueeze_(-1).expand(-1, -1, mask2d.size(1))\n        mask3d = mask3d & mask3d.transpose(1, 2)\n        if self.config.crf:\n            mask3d = mask3d.flatten(end_dim=1)\n            token_index = mask3d[:, 0]\n            mask3d = mask3d[token_index]\n            return token_index, mask3d\n        else:\n            return mask3d\n\n    def _step(self, optimizer, scheduler, grad_norm):\n        clip_grad_norm(self.model, grad_norm)\n        optimizer.step()\n        scheduler.step()\n        optimizer.zero_grad()\n\n    # noinspection PyMethodOverriding\n    def build_model(self, embed: Embedding, encoder, training, **kwargs) -> torch.nn.Module:\n        # noinspection PyCallByClass\n        model = SpanBIOSemanticRoleLabelingModel(\n            embed.module(training=training, vocabs=self.vocabs),\n            encoder,\n            len(self.vocabs.srl),\n            self.config.n_mlp_rel,\n            self.config.mlp_dropout,\n            self.config.crf,\n        )\n        return model\n\n    # noinspection PyMethodOverriding\n    def build_dataloader(self, data, batch_size,\n                         sampler_builder: SamplerBuilder = None,\n                         gradient_accumulation=1,\n                         shuffle=False, device=None, logger: logging.Logger = None,\n                         transform=None,\n                         **kwargs) -> DataLoader:\n        if isinstance(data, TransformableDataset):\n            dataset = data\n        else:\n            transforms = [self.config.embed.transform(vocabs=self.vocabs), self.vocabs, FieldLength('token')]\n            if transform:\n                transforms.insert(0, transform)\n            dataset = self.build_dataset(data, transforms)\n        if self.vocabs.mutable:\n            # noinspection PyTypeChecker\n            self.build_vocabs(dataset, logger)\n        lens = [len(x['token_input_ids']) for x in dataset]\n        if sampler_builder:\n            sampler = sampler_builder.build(lens, shuffle, gradient_accumulation)\n        else:\n            sampler = None\n        return PadSequenceDataLoader(dataset, batch_size, shuffle, device=device, batch_sampler=sampler)\n\n    def build_dataset(self, data, transform):\n        dataset = CoNLL2012SRLBIODataset(data,\n                                         transform=transform,\n                                         doc_level_offset=self.config.get('doc_level_offset', True),\n                                         cache=isinstance(data, str))\n        return dataset\n\n    def build_vocabs(self, dataset, logger, **kwargs):\n        self.vocabs.srl = Vocab(pad_token=None, unk_token=None)\n        timer = CountdownTimer(len(dataset))\n        max_seq_len = 0\n        for sample in dataset:\n            max_seq_len = max(max_seq_len, len(sample['token_input_ids']))\n            timer.log(f'Building vocab [blink][yellow]...[/yellow][/blink] (longest sequence: {max_seq_len})')\n        self.vocabs['srl'].set_unk_as_safe_unk()  # C-ARGM-FRQ appears only in test set\n        self.vocabs.lock()\n        self.vocabs.summary(logger)\n        if self.config.get('delimiter') is None:\n            tokens = dataset[0]['token']\n            self.config.delimiter = guess_delimiter(tokens)\n            logger.info(f'Guess the delimiter between tokens could be [blue]\"{self.config.delimiter}\"[/blue]. '\n                        f'If not, specify `delimiter` in `fit()`')\n\n    def predict(self, data: Union[str, List[str]], batch_size: int = None, **kwargs):\n        if not data:\n            return []\n        flat = self.input_is_flat(data)\n        if flat:\n            data = [data]\n        dataloader = self.build_dataloader(self.build_samples(data), batch_size, device=self.device, **kwargs)\n        results = []\n        order = []\n        for batch in dataloader:\n            pred, mask = self.feed_batch(batch)\n            prediction = self.decode_output(pred, mask, batch)\n            results.extend(self.prediction_to_result(prediction, batch))\n            order.extend(batch[IDX])\n        results = reorder(results, order)\n        if flat:\n            return results[0]\n        return results\n\n    def build_samples(self, data):\n        return [{'token': token} for token in data]\n\n    # noinspection PyMethodOverriding\n    def fit(self,\n            trn_data,\n            dev_data,\n            save_dir,\n            embed,\n            encoder=None,\n            lr=1e-3,\n            transformer_lr=1e-4,\n            adam_epsilon=1e-8,\n            warmup_steps=0.1,\n            weight_decay=0,\n            crf=False,\n            n_mlp_rel=300,\n            mlp_dropout=0.2,\n            batch_size=32,\n            gradient_accumulation=1,\n            grad_norm=1,\n            loss_reduction='mean',\n            epochs=30,\n            delimiter=None,\n            doc_level_offset=True,\n            eval_trn=False,\n            logger=None,\n            devices: Union[float, int, List[int]] = None,\n            transform=None,\n            **kwargs):\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    def compute_loss(self, criterion, pred, srl, mask):\n        if self.config.crf:\n            token_index, mask = mask\n            criterion: CRF = criterion\n            loss = -criterion.forward(pred, srl.flatten(end_dim=1)[token_index], mask,\n                                      reduction=self.config.loss_reduction)\n        else:\n            loss = criterion(pred[mask], srl[mask])\n        return loss\n\n    # noinspection PyMethodOverriding\n    @torch.no_grad()\n    def evaluate_dataloader(self, data: DataLoader, criterion: Callable, metric, logger, ratio_width=None,\n                            filename=None, **kwargs):\n        self.model.eval()\n        timer = CountdownTimer(len(data))\n        total_loss = 0\n        metric.reset()\n        for idx, batch in enumerate(data):\n            pred, mask = self.feed_batch(batch)\n            loss = self.compute_loss(criterion, pred, batch['srl_id'], mask)\n            total_loss += loss.item()\n            prediction = self.decode_output(pred, mask, batch)\n            self.update_metrics(metric, prediction, batch)\n            report = f'loss: {total_loss / (idx + 1):.4f} {metric}'\n            timer.log(report, logger=logger, ratio_percentage=False, ratio_width=ratio_width)\n        return total_loss / timer.total, metric\n\n    def input_is_flat(self, data) -> bool:\n        return isinstance(data[0], str)\n\n    def prediction_to_result(self, prediction: List, batch: Dict[str, Any], delimiter=None) -> List:\n        if delimiter is None:\n            delimiter = self.config.delimiter\n        for matrix, tokens in zip(prediction, batch['token']):\n            result = []\n            for i, arguments in enumerate(matrix):\n                if arguments:\n                    pas = [(delimiter.join(tokens[x[1]:x[2]]),) + x for x in arguments]\n                    pas.insert(bisect([a[1] for a in arguments], i), (tokens[i], PRED, i, i + 1))\n                    result.append(pas)\n            yield result\n"
  },
  {
    "path": "hanlp/components/srl/span_rank/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-07-19 22:22"
  },
  {
    "path": "hanlp/components/srl/span_rank/highway_variational_lstm.py",
    "content": "# Adopted from https://github.com/KiroSummer/A_Syntax-aware_MTL_Framework_for_Chinese_SRL\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.nn.init as init\nfrom torch.autograd import Variable\n\nfrom .layer import DropoutLayer, HighwayLSTMCell, VariationalLSTMCell\n\n\ndef initializer_1d(input_tensor, initializer):\n    assert len(input_tensor.size()) == 1\n    input_tensor = input_tensor.view(-1, 1)\n    input_tensor = initializer(input_tensor)\n    return input_tensor.view(-1)\n\n\nclass HighwayBiLSTM(nn.Module):\n    \"\"\"A module that runs multiple steps of HighwayBiLSTM.\"\"\"\n\n    def __init__(self, input_size, hidden_size, num_layers=1, batch_first=False, bidirectional=False, dropout_in=0,\n                 dropout_out=0):\n        super(HighwayBiLSTM, self).__init__()\n        self.input_size = input_size\n        self.hidden_size = hidden_size\n        self.num_layers = num_layers\n        self.batch_first = batch_first\n        self.bidirectional = bidirectional\n        self.dropout_in = dropout_in\n        self.dropout_out = dropout_out\n        self.num_directions = 2 if bidirectional else 1\n\n        self.fcells, self.f_dropout, self.f_hidden_dropout = [], [], []\n        self.bcells, self.b_dropout, self.b_hidden_dropout = [], [], []\n        for layer in range(num_layers):\n            layer_input_size = input_size if layer == 0 else hidden_size\n            self.fcells.append(HighwayLSTMCell(input_size=layer_input_size, hidden_size=hidden_size))\n            self.f_dropout.append(DropoutLayer(hidden_size, self.dropout_out))\n            self.f_hidden_dropout.append(DropoutLayer(hidden_size, self.dropout_out))\n            if self.bidirectional:\n                self.bcells.append(HighwayLSTMCell(input_size=hidden_size, hidden_size=hidden_size))\n                self.b_dropout.append(DropoutLayer(hidden_size, self.dropout_out))\n                self.b_hidden_dropout.append(DropoutLayer(hidden_size, self.dropout_out))\n        self.fcells, self.bcells = nn.ModuleList(self.fcells), nn.ModuleList(self.bcells)\n        self.f_dropout, self.b_dropout = nn.ModuleList(self.f_dropout), nn.ModuleList(self.b_dropout)\n\n    def reset_dropout_layer(self, batch_size):\n        for layer in range(self.num_layers):\n            self.f_dropout[layer].reset_dropout_mask(batch_size)\n            if self.bidirectional:\n                self.b_dropout[layer].reset_dropout_mask(batch_size)\n\n    @staticmethod\n    def _forward_rnn(cell, gate, input, masks, initial, drop_masks=None, hidden_drop=None):\n        max_time = input.size(0)\n        output = []\n        hx = initial\n        for time in range(max_time):\n            h_next, c_next = cell(input[time], mask=masks[time], hx=hx, dropout=drop_masks)\n            hx = (h_next, c_next)\n            output.append(h_next)\n        output = torch.stack(output, 0)\n        return output, hx\n\n    @staticmethod\n    def _forward_brnn(cell, gate, input, masks, initial, drop_masks=None, hidden_drop=None):\n        max_time = input.size(0)\n        output = []\n        hx = initial\n        for time in reversed(list(range(max_time))):\n            h_next, c_next = cell(input[time], mask=masks[time], hx=hx, dropout=drop_masks)\n            hx = (h_next, c_next)\n            output.append(h_next)\n        output.reverse()\n        output = torch.stack(output, 0)\n        return output, hx\n\n    def forward(self, input, masks, initial=None):\n        if self.batch_first:\n            input = input.transpose(0, 1)  # transpose: return the transpose matrix\n            masks = torch.unsqueeze(masks.transpose(0, 1), dim=2)\n        max_time, batch_size, _ = input.size()\n\n        self.reset_dropout_layer(batch_size)  # reset the dropout each batch forward\n\n        masks = masks.expand(-1, -1, self.hidden_size)  # expand: -1 means not expand that dimension\n        if initial is None:\n            initial = Variable(input.data.new(batch_size, self.hidden_size).zero_())\n            initial = (initial, initial)  # h0, c0\n\n        h_n, c_n = [], []\n        for layer in range(self.num_layers):\n            # hidden_mask, hidden_drop = None, None\n            hidden_mask, hidden_drop = self.f_dropout[layer], self.f_hidden_dropout[layer]\n            layer_output, (layer_h_n, layer_c_n) = HighwayBiLSTM._forward_rnn(cell=self.fcells[layer], \\\n                                                                              gate=None, input=input, masks=masks,\n                                                                              initial=initial, \\\n                                                                              drop_masks=hidden_mask,\n                                                                              hidden_drop=hidden_drop)\n            h_n.append(layer_h_n)\n            c_n.append(layer_c_n)\n            if self.bidirectional:\n                hidden_mask, hidden_drop = self.b_dropout[layer], self.b_hidden_dropout[layer]\n                blayer_output, (blayer_h_n, blayer_c_n) = HighwayBiLSTM._forward_brnn(cell=self.bcells[layer], \\\n                                                                                      gate=None, input=layer_output,\n                                                                                      masks=masks, initial=initial, \\\n                                                                                      drop_masks=hidden_mask,\n                                                                                      hidden_drop=hidden_drop)\n                h_n.append(blayer_h_n)\n                c_n.append(blayer_c_n)\n\n            input = blayer_output if self.bidirectional else layer_output\n\n        h_n, c_n = torch.stack(h_n, 0), torch.stack(c_n, 0)\n        if self.batch_first:\n            input = input.transpose(1, 0)  # transpose: return the transpose matrix\n        return input, (h_n, c_n)\n\n\nclass StackedHighwayBiLSTM(nn.Module):\n    \"\"\"A module that runs multiple steps of HighwayBiLSTM.\"\"\"\n\n    def __init__(self, input_size, hidden_size, num_layers=1, batch_first=False, \\\n                 bidirectional=False, dropout_in=0, dropout_out=0):\n        super(StackedHighwayBiLSTM, self).__init__()\n        self.input_size = input_size\n        self.hidden_size = hidden_size\n        self.num_layers = num_layers\n        self.batch_first = batch_first\n        self.bidirectional = bidirectional\n        self.dropout_in = dropout_in\n        self.dropout_out = dropout_out\n        self.num_directions = 2 if bidirectional else 1\n\n        self.fcells, self.f_dropout, self.f_hidden_dropout = [], [], []\n        self.bcells, self.b_dropout, self.b_hidden_dropout = [], [], []\n        self.f_initial, self.b_initial = [], []\n        for layer in range(num_layers):\n            layer_input_size = input_size if layer == 0 else 2 * hidden_size if self.bidirectional else hidden_size\n            self.fcells.append(VariationalLSTMCell(input_size=layer_input_size, hidden_size=hidden_size))\n            self.f_dropout.append(DropoutLayer(hidden_size, self.dropout_out))\n            self.f_hidden_dropout.append(DropoutLayer(hidden_size, self.dropout_out))\n            self.f_initial.append(nn.Parameter(torch.Tensor(2, self.hidden_size)))\n            assert self.bidirectional is True\n            self.bcells.append(VariationalLSTMCell(input_size=layer_input_size, hidden_size=hidden_size))\n            self.b_dropout.append(DropoutLayer(hidden_size, self.dropout_out))\n            self.b_hidden_dropout.append(DropoutLayer(hidden_size, self.dropout_out))\n            self.b_initial.append(nn.Parameter(torch.Tensor(2, self.hidden_size)))\n        self.lstm_project_layer = nn.ModuleList([nn.Linear(2 * self.hidden_size, 2 * self.hidden_size)\n                                                 for _ in range(num_layers - 1)])\n        self.fcells, self.bcells = nn.ModuleList(self.fcells), nn.ModuleList(self.bcells)\n        self.f_dropout, self.b_dropout = nn.ModuleList(self.f_dropout), nn.ModuleList(self.b_dropout)\n        self.f_hidden_dropout, self.b_hidden_dropout = \\\n            nn.ModuleList(self.f_hidden_dropout), nn.ModuleList(self.b_hidden_dropout)\n        self.f_initial, self.b_initial = nn.ParameterList(self.f_initial), nn.ParameterList(self.b_initial)\n        self.reset_parameters()\n\n    def reset_parameters(self):\n        for layer_initial in [self.f_initial, self.b_initial]:\n            for initial in layer_initial:\n                init.xavier_uniform_(initial)\n        for layer in self.lstm_project_layer:\n            init.xavier_uniform_(layer.weight)\n            initializer_1d(layer.bias, init.xavier_uniform_)\n\n    def reset_dropout_layer(self, batch_size):\n        for layer in range(self.num_layers):\n            self.f_dropout[layer].reset_dropout_mask(batch_size)\n            self.f_hidden_dropout[layer].reset_dropout_mask(batch_size)\n            if self.bidirectional:\n                self.b_dropout[layer].reset_dropout_mask(batch_size)\n                self.b_hidden_dropout[layer].reset_dropout_mask(batch_size)\n\n    def reset_state(self, batch_size):\n        f_states, b_states = [], []\n        for f_layer_initial, b_layer_initial in zip(self.f_initial, self.b_initial):\n            f_states.append([f_layer_initial[0].expand(batch_size, -1), f_layer_initial[1].expand(batch_size, -1)])\n            b_states.append([b_layer_initial[0].expand(batch_size, -1), b_layer_initial[1].expand(batch_size, -1)])\n        return f_states, b_states\n\n    @staticmethod\n    def _forward_rnn(cell, gate, input, masks, initial, drop_masks=None, hidden_drop=None):\n        max_time = input.size(0)\n        output = []\n        hx = initial\n        for time in range(max_time):\n            h_next, c_next = cell(input[time], mask=masks[time], hx=hx, dropout=drop_masks)\n            hx = (h_next, c_next)\n            output.append(h_next)\n        output = torch.stack(output, 0)\n        return output, hx\n\n    @staticmethod\n    def _forward_brnn(cell, gate, input, masks, initial, drop_masks=None, hidden_drop=None):\n        max_time = input.size(0)\n        output = []\n        hx = initial\n        for time in reversed(list(range(max_time))):\n            h_next, c_next = cell(input[time], mask=masks[time], hx=hx, dropout=drop_masks)\n            hx = (h_next, c_next)\n            output.append(h_next)\n        output.reverse()\n        output = torch.stack(output, 0)\n        return output, hx\n\n    def forward(self, input, masks, initial=None):\n        if self.batch_first:\n            input = input.transpose(0, 1)  # transpose: return the transpose matrix\n            masks = torch.unsqueeze(masks.transpose(0, 1), dim=2)\n        max_time, batch_size, _ = input.size()\n\n        self.reset_dropout_layer(batch_size)  # reset the dropout each batch forward\n        f_states, b_states = self.reset_state(batch_size)\n\n        masks = masks.expand(-1, -1, self.hidden_size)  # expand: -1 means not expand that dimension\n\n        h_n, c_n = [], []\n        outputs = []\n        for layer in range(self.num_layers):\n            hidden_mask, hidden_drop = self.f_dropout[layer], self.f_hidden_dropout[layer]\n            layer_output, (layer_h_n, layer_c_n) = \\\n                StackedHighwayBiLSTM._forward_rnn(cell=self.fcells[layer],\n                                                  gate=None, input=input, masks=masks, initial=f_states[layer],\n                                                  drop_masks=hidden_mask, hidden_drop=hidden_drop)\n            h_n.append(layer_h_n)\n            c_n.append(layer_c_n)\n            assert self.bidirectional is True\n            hidden_mask, hidden_drop = self.b_dropout[layer], self.b_hidden_dropout[layer]\n            blayer_output, (blayer_h_n, blayer_c_n) = \\\n                StackedHighwayBiLSTM._forward_brnn(cell=self.bcells[layer],\n                                                   gate=None, input=input, masks=masks, initial=b_states[layer],\n                                                   drop_masks=hidden_mask, hidden_drop=hidden_drop)\n            h_n.append(blayer_h_n)\n            c_n.append(blayer_c_n)\n\n            output = torch.cat([layer_output, blayer_output], 2) if self.bidirectional else layer_output\n            output = F.dropout(output, self.dropout_out, self.training)\n            if layer > 0:  # Highway\n                highway_gates = torch.sigmoid(self.lstm_project_layer[layer - 1].forward(output))\n                output = highway_gates * output + (1 - highway_gates) * input\n            if self.batch_first:\n                outputs.append(output.transpose(1, 0))\n            else:\n                outputs.append(output)\n            input = output\n\n        h_n, c_n = torch.stack(h_n, 0), torch.stack(c_n, 0)\n        if self.batch_first:\n            output = output.transpose(1, 0)  # transpose: return the transpose matrix\n        return output, (h_n, c_n), outputs\n"
  },
  {
    "path": "hanlp/components/srl/span_rank/inference_utils.py",
    "content": "# Adopted from https://github.com/KiroSummer/A_Syntax-aware_MTL_Framework_for_Chinese_SRL\n\n# Inference functions for the SRL model.\nimport numpy as np\n\n\ndef decode_spans(span_starts, span_ends, span_scores, labels_inv):\n    \"\"\"\n\n    Args:\n      span_starts: [num_candidates,]\n      span_scores: [num_candidates, num_labels]\n      span_ends: \n      labels_inv: \n\n    Returns:\n\n    \n    \"\"\"\n    pred_spans = []\n    span_labels = np.argmax(span_scores, axis=1)  # [num_candidates]\n    spans_list = list(zip(span_starts, span_ends, span_labels, span_scores))\n    spans_list = sorted(spans_list, key=lambda x: x[3][x[2]], reverse=True)\n    predicted_spans = {}\n    for start, end, label, _ in spans_list:\n        # Skip invalid span.\n        if label == 0 or (start, end) in predicted_spans:\n            continue\n        pred_spans.append((start, end, labels_inv[label]))\n        predicted_spans[(start, end)] = label\n    return pred_spans\n\n\ndef greedy_decode(predict_dict, srl_labels_inv):\n    \"\"\"Greedy decoding for SRL predicate-argument structures.\n\n    Args:\n      predict_dict: Dictionary of name to numpy arrays.\n      srl_labels_inv: SRL label id to string name.\n      suppress_overlap: Whether to greedily suppress overlapping arguments for the same predicate.\n\n    Returns:\n\n    \n    \"\"\"\n    arg_starts = predict_dict[\"arg_starts\"]\n    arg_ends = predict_dict[\"arg_ends\"]\n    predicates = predict_dict[\"predicates\"]\n    arg_labels = predict_dict[\"arg_labels\"]\n    scores = predict_dict[\"srl_scores\"]\n\n    num_suppressed_args = 0\n\n    # Map from predicates to a list of labeled spans.\n    pred_to_args = {}\n    if len(arg_ends) > 0 and len(predicates) > 0:\n        max_len = max(np.max(arg_ends), np.max(predicates)) + 1\n    else:\n        max_len = 1\n\n    for j, pred_id in enumerate(predicates):\n        args_list = []\n        for i, (arg_start, arg_end) in enumerate(zip(arg_starts, arg_ends)):\n            # If label is not null.\n            if arg_labels[i][j] == 0:\n                continue\n            label = srl_labels_inv[arg_labels[i][j]]\n            # if label not in [\"V\", \"C-V\"]:\n            args_list.append((arg_start, arg_end, label, scores[i][j][arg_labels[i][j]]))\n\n        # Sort arguments by highest score first.\n        args_list = sorted(args_list, key=lambda x: x[3], reverse=True)\n        new_args_list = []\n\n        flags = [False for _ in range(max_len)]\n        # Predicate will not overlap with arguments either.\n        flags[pred_id] = True\n\n        for (arg_start, arg_end, label, score) in args_list:\n            # If none of the tokens has been covered:\n            if not max(flags[arg_start:arg_end + 1]):\n                new_args_list.append((arg_start, arg_end, label))\n                for k in range(arg_start, arg_end + 1):\n                    flags[k] = True\n\n        # Only add predicate if it has any argument.\n        if new_args_list:\n            pred_to_args[pred_id] = new_args_list\n\n        num_suppressed_args += len(args_list) - len(new_args_list)\n\n    return pred_to_args, num_suppressed_args\n\n\n_CORE_ARGS = {\"ARG0\": 1, \"ARG1\": 2, \"ARG2\": 4, \"ARG3\": 8, \"ARG4\": 16, \"ARG5\": 32, \"ARGA\": 64,\n              \"A0\": 1, \"A1\": 2, \"A2\": 4, \"A3\": 8, \"A4\": 16, \"A5\": 32, \"AA\": 64}\n\n\ndef get_predicted_clusters(top_span_starts, top_span_ends, predicted_antecedents):\n    mention_to_predicted = {}\n    predicted_clusters = []\n    for i, predicted_index in enumerate(predicted_antecedents):\n        if predicted_index < 0:\n            continue\n        assert i > predicted_index\n        predicted_antecedent = (int(top_span_starts[predicted_index]), int(top_span_ends[predicted_index]))\n        if predicted_antecedent in mention_to_predicted:\n            predicted_cluster = mention_to_predicted[predicted_antecedent]\n        else:\n            predicted_cluster = len(predicted_clusters)\n            predicted_clusters.append([predicted_antecedent])\n            mention_to_predicted[predicted_antecedent] = predicted_cluster\n\n        mention = (int(top_span_starts[i]), int(top_span_ends[i]))\n        predicted_clusters[predicted_cluster].append(mention)\n        mention_to_predicted[mention] = predicted_cluster\n\n    predicted_clusters = [tuple(pc) for pc in predicted_clusters]\n    mention_to_predicted = {m: predicted_clusters[i] for m, i in list(mention_to_predicted.items())}\n\n    return predicted_clusters, mention_to_predicted\n\n\ndef _decode_non_overlapping_spans(starts, ends, scores, max_len, labels_inv, pred_id):\n    labels = np.argmax(scores, axis=1)\n    spans = []\n    for i, (start, end, label) in enumerate(zip(starts, ends, labels)):\n        if label <= 0:\n            continue\n        label_str = labels_inv[label]\n        if pred_id is not None and label_str == \"V\":\n            continue\n        spans.append((start, end, label_str, scores[i][label]))\n    spans = sorted(spans, key=lambda x: x[3], reverse=True)\n    flags = np.zeros([max_len], dtype=bool)\n    if pred_id is not None:\n        flags[pred_id] = True\n    new_spans = []\n    for start, end, label_str, score in spans:\n        if not max(flags[start:end + 1]):\n            new_spans.append((start, end, label_str))  # , score))\n            for k in range(start, end + 1):\n                flags[k] = True\n    return new_spans\n\n\ndef _dp_decode_non_overlapping_spans(starts, ends, scores, max_len, labels_inv, pred_id, u_constraint=False):\n    num_roles = scores.shape[1]  # [num_arg, num_roles]\n    labels = np.argmax(scores, axis=1).astype(np.int64)\n    spans = list(zip(starts, ends, list(range(len(starts)))))\n    spans = sorted(spans, key=lambda x: (x[0], x[1]))  # sort according to the span start index\n\n    if u_constraint:\n        f = np.zeros([max_len + 1, 128], dtype=float) - 0.1\n    else:  # This one\n        f = np.zeros([max_len + 1, 1], dtype=float) - 0.1\n\n    f[0, 0] = 0\n    states = {0: set([0])}  # A dictionary from id to list of binary core-arg states.\n    pointers = {}  # A dictionary from states to (arg_id, role, prev_t, prev_rs)\n    best_state = [(0, 0)]\n\n    def _update_state(t0, rs0, t1, rs1, delta, arg_id, role):\n        if f[t0][rs0] + delta > f[t1][rs1]:\n            f[t1][rs1] = f[t0][rs0] + delta\n            if t1 not in states:\n                states[t1] = set()\n            states[t1].update([rs1])\n            pointers[(t1, rs1)] = (arg_id, role, t0, rs0)  # the pointers store\n            if f[t1][rs1] > f[best_state[0][0]][best_state[0][1]]:\n                best_state[0] = (t1, rs1)\n\n    for start, end, i in spans:  # [arg_start, arg_end, arg_span_id]\n        assert scores[i][0] == 0  # dummy score\n        # The extra dummy score should be same for all states, so we can safely skip arguments overlap\n        # with the predicate.\n        if pred_id is not None and start <= pred_id and pred_id <= end:  # skip the span contains the predicate\n            continue\n        r0 = labels[i]  # Locally best role assignment.\n        # Strictly better to incorporate a dummy span if it has the highest local score.\n        if r0 == 0:  # labels_inv[r0] == \"O\"\n            continue\n        r0_str = labels_inv[r0]\n        # Enumerate explored states.\n        t_states = [t for t in list(states.keys()) if t <= start]  # collect the state which is before the current span\n        for t in t_states:  # for each state\n            role_states = states[t]\n            # Update states if best role is not a core arg.\n            if not u_constraint or r0_str not in _CORE_ARGS:  # True; this one\n                for rs in role_states:  # the set type in the value in the state dict\n                    _update_state(t, rs, end + 1, rs, scores[i][r0], i, r0)  # update the state\n            else:\n                for rs in role_states:\n                    for r in range(1, num_roles):\n                        if scores[i][r] > 0:\n                            r_str = labels_inv[r]\n                            core_state = _CORE_ARGS.get(r_str, 0)\n                            # print start, end, i, r_str, core_state, rs\n                            if core_state & rs == 0:\n                                _update_state(t, rs, end + 1, rs | core_state, scores[i][r], i, r)\n    # Backtrack to decode.\n    new_spans = []\n    t, rs = best_state[0]\n    while (t, rs) in pointers:\n        i, r, t0, rs0 = pointers[(t, rs)]\n        new_spans.append((int(starts[i]), int(ends[i]), labels_inv[r]))\n        t = t0\n        rs = rs0\n    return new_spans[::-1]\n\n\ndef srl_decode(sentence_lengths, predict_dict, srl_labels_inv, config):  # decode the predictions.\n    # Decode sentence-level tasks.\n    num_sentences = len(sentence_lengths)\n    predictions = [{} for _ in range(num_sentences)]\n    # Sentence-level predictions.\n    for i in range(num_sentences):  # for each sentences\n        # if predict_dict[\"No_arg\"] is True:\n        #     predictions[\"srl\"][i][predict_dict[\"predicates\"][i]] = []\n        #     continue\n        predict_dict_num_args_ = predict_dict[\"num_args\"].cpu().numpy()\n        predict_dict_num_preds_ = predict_dict[\"num_preds\"].cpu().numpy()\n        predict_dict_predicates_ = predict_dict[\"predicates\"].cpu().numpy()\n        predict_dict_arg_starts_ = predict_dict[\"arg_starts\"].cpu().numpy()\n        predict_dict_arg_ends_ = predict_dict[\"arg_ends\"].cpu().numpy()\n        predict_dict_srl_scores_ = predict_dict[\"srl_scores\"].detach().cpu().numpy()\n        num_args = predict_dict_num_args_[i]  # the number of the candidate argument spans\n        num_preds = predict_dict_num_preds_[i]  # the number of the candidate predicates\n        # for each predicate id, exec the decode process\n        for j, pred_id in enumerate(predict_dict_predicates_[i][:num_preds]):\n            # sorted arg_starts and arg_ends and srl_scores ? should be??? enforce_srl_constraint = False\n            arg_spans = _dp_decode_non_overlapping_spans(\n                predict_dict_arg_starts_[i][:num_args],\n                predict_dict_arg_ends_[i][:num_args],\n                predict_dict_srl_scores_[i, :num_args, j, :],\n                sentence_lengths[i], srl_labels_inv, pred_id, config.enforce_srl_constraint)\n            # To avoid warnings in the eval script.\n            if config.use_gold_predicates:  # false\n                arg_spans.append((pred_id, pred_id, \"V\"))\n            if arg_spans:\n                predictions[i][int(pred_id)] = sorted(arg_spans, key=lambda x: (x[0], x[1]))\n\n    return predictions\n"
  },
  {
    "path": "hanlp/components/srl/span_rank/layer.py",
    "content": "# Adopted from https://github.com/KiroSummer/A_Syntax-aware_MTL_Framework_for_Chinese_SRL\n\nimport torch\nimport torch.nn as nn\nfrom torch.autograd import Variable\nimport numpy as np\nimport torch.nn.functional as F\n\nfrom hanlp.components.srl.span_rank.util import block_orth_normal_initializer\n\n\ndef get_tensor_np(t):\n    return t.data.cpu().numpy()\n\n\ndef orthonormal_initializer(output_size, input_size):\n    \"\"\"adopted from Timothy Dozat https://github.com/tdozat/Parser/blob/master/lib/linalg.py\n\n    Args:\n      output_size: \n      input_size: \n\n    Returns:\n\n    \n    \"\"\"\n    print((output_size, input_size))\n    I = np.eye(output_size)\n    lr = .1\n    eps = .05 / (output_size + input_size)\n    success = False\n    tries = 0\n    while not success and tries < 10:\n        Q = np.random.randn(input_size, output_size) / np.sqrt(output_size)\n        for i in range(100):\n            QTQmI = Q.T.dot(Q) - I\n            loss = np.sum(QTQmI ** 2 / 2)\n            Q2 = Q ** 2\n            Q -= lr * Q.dot(QTQmI) / (\n                    np.abs(Q2 + Q2.sum(axis=0, keepdims=True) + Q2.sum(axis=1, keepdims=True) - 1) + eps)\n            if np.max(Q) > 1e6 or loss > 1e6 or not np.isfinite(loss):\n                tries += 1\n                lr /= 2\n                break\n        success = True\n    if success:\n        print(('Orthogonal pretrainer loss: %.2e' % loss))\n    else:\n        print('Orthogonal pretrainer failed, using non-orthogonal random matrix')\n        Q = np.random.randn(input_size, output_size) / np.sqrt(output_size)\n    return np.transpose(Q.astype(np.float32))\n\n\nclass LayerNorm(nn.Module):\n    def __init__(self, features, eps=1e-8):\n        super(LayerNorm, self).__init__()\n        self.gamma = nn.Parameter(torch.ones(features))\n        self.beta = nn.Parameter(torch.zeros(features))\n        self.eps = eps\n\n    def forward(self, x):\n        mean = x.mean(-1, keepdim=True)\n        std = x.std(-1, keepdim=True)\n        return self.gamma * (x - mean) / (std + self.eps) + self.beta\n\n\nclass DropoutLayer3D(nn.Module):\n    def __init__(self, input_size, dropout_rate=0.0):\n        super(DropoutLayer3D, self).__init__()\n        self.dropout_rate = dropout_rate\n        self.input_size = input_size\n        self.drop_mask = torch.FloatTensor(self.input_size).fill_(1 - self.dropout_rate)\n        self.drop_mask = Variable(torch.bernoulli(self.drop_mask), requires_grad=False)\n        if torch.cuda.is_available():\n            self.drop_mask = self.drop_mask.cuda()\n\n    def reset_dropout_mask(self, batch_size, length):\n        self.drop_mask = torch.FloatTensor(batch_size, length, self.input_size).fill_(1 - self.dropout_rate)\n        self.drop_mask = Variable(torch.bernoulli(self.drop_mask), requires_grad=False)\n        if torch.cuda.is_available():\n            self.drop_mask = self.drop_mask.cuda()\n\n    def forward(self, x):\n        if self.training:\n            return torch.mul(x, self.drop_mask)\n        else:  # eval\n            return x * (1.0 - self.dropout_rate)\n\n\nclass DropoutLayer(nn.Module):\n    def __init__(self, input_size, dropout_rate=0.0):\n        super(DropoutLayer, self).__init__()\n        self.dropout_rate = dropout_rate\n        self.input_size = input_size\n        self.drop_mask = torch.Tensor(self.input_size).fill_(1 - self.dropout_rate)\n        self.drop_mask = torch.bernoulli(self.drop_mask)\n\n    def reset_dropout_mask(self, batch_size):\n        self.drop_mask = torch.Tensor(batch_size, self.input_size).fill_(1 - self.dropout_rate)\n        self.drop_mask = torch.bernoulli(self.drop_mask)\n\n    def forward(self, x):\n        if self.training:\n            return torch.mul(x, self.drop_mask.to(x.device))\n        else:  # eval\n            return x * (1.0 - self.dropout_rate)\n\n\nclass NonLinear(nn.Module):\n    def __init__(self, input_size, hidden_size, activation=None):\n        super(NonLinear, self).__init__()\n        self.input_size = input_size\n        self.hidden_size = hidden_size\n        self.linear = nn.Linear(in_features=input_size, out_features=hidden_size)\n        if activation is None:\n            self._activate = lambda x: x\n        else:\n            if not callable(activation):\n                raise ValueError(\"activation must be callable: type={}\".format(type(activation)))\n            self._activate = activation\n\n        self.reset_parameters()\n\n    def forward(self, x):\n        y = self.linear(x)\n        return self._activate(y)\n\n    def reset_parameters(self):\n        nn.init.xavier_uniform_(self.linear.weight)\n        nn.init.zeros_(self.linear.bias)\n\n\nclass Biaffine(nn.Module):\n    def __init__(self, in1_features, in2_features, out_features,\n                 bias=(True, True)):\n        super(Biaffine, self).__init__()\n        self.in1_features = in1_features\n        self.in2_features = in2_features\n        self.out_features = out_features\n        self.bias = bias\n        self.linear_input_size = in1_features + int(bias[0])\n        self.linear_output_size = out_features * (in2_features + int(bias[1]))\n        self.linear = nn.Linear(in_features=self.linear_input_size,\n                                out_features=self.linear_output_size,\n                                bias=False)\n\n        self.reset_parameters()\n\n    def reset_parameters(self):\n        torch.nn.init.xavier_uniform_(self.linear.weight)\n\n    def forward(self, input1, input2):\n        batch_size, len1, dim1 = input1.size()\n        batch_size, len2, dim2 = input2.size()\n        if self.bias[0]:\n            ones = input1.data.new(batch_size, len1, 1).zero_().fill_(1)  # this kind of implementation is too tedious\n            input1 = torch.cat((input1, Variable(ones)), dim=2)\n            dim1 += 1\n        if self.bias[1]:\n            ones = input2.data.new(batch_size, len2, 1).zero_().fill_(1)\n            input2 = torch.cat((input2, Variable(ones)), dim=2)\n            dim2 += 1\n\n        affine = self.linear(input1)\n\n        affine = affine.view(batch_size, len1 * self.out_features, dim2)\n        input2 = torch.transpose(input2, 1, 2)\n        # torch.bmm: Performs a batch matrix-matrix product of matrices stored in batch1 and batch2.\n        biaffine = torch.transpose(torch.bmm(affine, input2), 1, 2)\n        # view: Returns a new tensor with the same data as the self tensor but of a different size.\n        biaffine = biaffine.contiguous().view(batch_size, len2, len1, self.out_features)\n\n        return biaffine\n\n    def __repr__(self):\n        return self.__class__.__name__ + ' (' \\\n               + 'in1_features=' + str(self.in1_features) \\\n               + ', in2_features=' + str(self.in2_features) \\\n               + ', out_features=' + str(self.out_features) + ')'\n\n\nclass HighwayLSTMCell(nn.Module):\n    def __init__(self, input_size, hidden_size):\n        super(HighwayLSTMCell, self).__init__()\n        self.input_size = input_size\n        self.hidden_size = hidden_size\n        self.linear_ih = nn.Linear(in_features=input_size,\n                                   out_features=6 * hidden_size)\n        self.linear_hh = nn.Linear(in_features=hidden_size,\n                                   out_features=5 * hidden_size,\n                                   bias=False)\n        self.reset_parameters()  # reset all the param in the MyLSTMCell\n\n    def reset_parameters(self):\n        weight_ih = block_orth_normal_initializer([self.input_size, ], [self.hidden_size] * 6)\n        self.linear_ih.weight.data.copy_(weight_ih)\n\n        weight_hh = block_orth_normal_initializer([self.hidden_size, ], [self.hidden_size] * 5)\n        self.linear_hh.weight.data.copy_(weight_hh)\n        # nn.init.constant(self.linear_hh.weight, 1.0)\n        # nn.init.constant(self.linear_ih.weight, 1.0)\n\n        nn.init.constant(self.linear_ih.bias, 0.0)\n\n    def forward(self, x, mask=None, hx=None, dropout=None):\n        assert mask is not None and hx is not None\n        _h, _c = hx\n        _x = self.linear_ih(x)  # compute the x\n        preact = self.linear_hh(_h) + _x[:, :self.hidden_size * 5]\n\n        i, f, o, t, j = preact.chunk(chunks=5, dim=1)\n        i, f, o, t, j = F.sigmoid(i), F.sigmoid(f + 1.0), F.sigmoid(o), F.sigmoid(t), F.tanh(j)\n        k = _x[:, self.hidden_size * 5:]\n\n        c = f * _c + i * j\n        c = mask * c + (1.0 - mask) * _c\n\n        h = t * o * F.tanh(c) + (1.0 - t) * k\n        if dropout is not None:\n            h = dropout(h)\n        h = mask * h + (1.0 - mask) * _h\n        return h, c\n\n\nclass VariationalLSTMCell(nn.Module):\n    def __init__(self, input_size, hidden_size):\n        super(VariationalLSTMCell, self).__init__()\n        self.input_size = input_size\n        self.hidden_size = hidden_size\n        self.linear = nn.Linear(in_features=input_size + self.hidden_size, out_features=3 * hidden_size)\n        self.reset_parameters()  # reset all the param in the MyLSTMCell\n\n    def reset_parameters(self):\n        weight = block_orth_normal_initializer([self.input_size + self.hidden_size, ], [self.hidden_size] * 3)\n        self.linear.weight.data.copy_(weight)\n        nn.init.constant_(self.linear.bias, 0.0)\n\n    def forward(self, x, mask=None, hx=None, dropout=None):\n        assert mask is not None and hx is not None\n        _h, _c = hx\n        _h = dropout(_h)\n        _x = self.linear(torch.cat([x, _h], 1))  # compute the x\n        i, j, o = _x.chunk(3, dim=1)\n        i = torch.sigmoid(i)\n        c = (1.0 - i) * _c + i * torch.tanh(j)\n        c = mask * c  # + (1.0 - mask) * _c\n        h = torch.tanh(c) * torch.sigmoid(o)\n        h = mask * h  # + (1.0 - mask) * _h\n\n        return h, c\n\n\nclass VariationalLSTM(nn.Module):\n    \"\"\"A module that runs multiple steps of LSTM.\"\"\"\n\n    def __init__(self, input_size, hidden_size, num_layers=1, batch_first=False, \\\n                 bidirectional=False, dropout_in=0, dropout_out=0):\n        super(VariationalLSTM, self).__init__()\n        self.input_size = input_size\n        self.hidden_size = hidden_size\n        self.num_layers = num_layers\n        self.batch_first = batch_first\n        self.bidirectional = bidirectional\n        self.dropout_in = dropout_in\n        self.dropout_out = dropout_out\n        self.num_directions = 2 if bidirectional else 1\n\n        self.fcells = []\n        self.bcells = []\n        for layer in range(num_layers):\n            layer_input_size = input_size if layer == 0 else hidden_size * self.num_directions\n            self.fcells.append(nn.LSTMCell(input_size=layer_input_size, hidden_size=hidden_size))\n            if self.bidirectional:\n                self.bcells.append(nn.LSTMCell(input_size=layer_input_size, hidden_size=hidden_size))\n\n        self._all_weights = []\n        for layer in range(num_layers):\n            layer_params = (self.fcells[layer].weight_ih, self.fcells[layer].weight_hh, \\\n                            self.fcells[layer].bias_ih, self.fcells[layer].bias_hh)\n            suffix = ''\n            param_names = ['weight_ih_l{}{}', 'weight_hh_l{}{}']\n            param_names += ['bias_ih_l{}{}', 'bias_hh_l{}{}']\n            param_names = [x.format(layer, suffix) for x in param_names]\n            for name, param in zip(param_names, layer_params):\n                setattr(self, name, param)\n            self._all_weights.append(param_names)\n\n            if self.bidirectional:\n                layer_params = (self.bcells[layer].weight_ih, self.bcells[layer].weight_hh, \\\n                                self.bcells[layer].bias_ih, self.bcells[layer].bias_hh)\n                suffix = '_reverse'\n                param_names = ['weight_ih_l{}{}', 'weight_hh_l{}{}']\n                param_names += ['bias_ih_l{}{}', 'bias_hh_l{}{}']\n                param_names = [x.format(layer, suffix) for x in param_names]\n                for name, param in zip(param_names, layer_params):\n                    setattr(self, name, param)\n                self._all_weights.append(param_names)\n\n        self.reset_parameters()\n\n    def reset_parameters(self):  # modified by kiro\n        for name, param in self.named_parameters():\n            print(name)\n            if \"weight\" in name:\n                # for i in range(4):\n                # nn.init.orthogonal(self.__getattr__(name)[self.hidden_size*i:self.hidden_size*(i+1),:])\n                nn.init.orthogonal(self.__getattr__(name))\n            if \"bias\" in name:\n                nn.init.normal(self.__getattr__(name), 0.0, 0.01)\n                # nn.init.constant(self.__getattr__(name), 1.0)  # different from zhang's 0\n\n    @staticmethod\n    def _forward_rnn(cell, input, masks, initial, drop_masks):\n        max_time = input.size(0)\n        output = []\n        hx = initial\n        for time in range(max_time):\n            h_next, c_next = cell(input=input[time], hx=hx)\n            h_next = h_next * masks[time] + initial[0] * (1 - masks[time])\n            c_next = c_next * masks[time] + initial[1] * (1 - masks[time])\n            output.append(h_next)\n            if drop_masks is not None: h_next = h_next * drop_masks\n            hx = (h_next, c_next)\n        output = torch.stack(output, 0)\n        return output, hx\n\n    @staticmethod\n    def _forward_brnn(cell, input, masks, initial, drop_masks):\n        max_time = input.size(0)\n        output = []\n        hx = initial\n        for time in reversed(list(range(max_time))):\n            h_next, c_next = cell(input=input[time], hx=hx)\n            h_next = h_next * masks[time] + initial[0] * (1 - masks[time])\n            c_next = c_next * masks[time] + initial[1] * (1 - masks[time])\n            output.append(h_next)\n            if drop_masks is not None: h_next = h_next * drop_masks\n            hx = (h_next, c_next)\n        output.reverse()\n        output = torch.stack(output, 0)\n        return output, hx\n\n    def forward(self, input, masks, initial=None):\n        if self.batch_first:\n            input = input.transpose(0, 1)  # transpose: return the transpose matrix\n            masks = torch.unsqueeze(masks.transpose(0, 1), dim=2)\n        max_time, batch_size, _ = input.size()\n        masks = masks.expand(-1, -1, self.hidden_size)  # expand: -1 means not expand that dimension\n        if initial is None:\n            initial = Variable(input.data.new(batch_size, self.hidden_size).zero_())\n            initial = (initial, initial)  # h0, c0\n        h_n = []\n        c_n = []\n\n        for layer in range(self.num_layers):\n            max_time, batch_size, input_size = input.size()\n            input_mask, hidden_mask = None, None\n            if self.training:  # when training, use the dropout\n                input_mask = input.data.new(batch_size, input_size).fill_(1 - self.dropout_in)\n                input_mask = Variable(torch.bernoulli(input_mask), requires_grad=False)\n                input_mask = input_mask / (1 - self.dropout_in)\n                # permute: exchange the dimension\n                input_mask = torch.unsqueeze(input_mask, dim=2).expand(-1, -1, max_time).permute(2, 0, 1)\n                input = input * input_mask\n\n                hidden_mask = input.data.new(batch_size, self.hidden_size).fill_(1 - self.dropout_out)\n                hidden_mask = Variable(torch.bernoulli(hidden_mask), requires_grad=False)\n                hidden_mask = hidden_mask / (1 - self.dropout_out)\n\n            layer_output, (layer_h_n, layer_c_n) = VariationalLSTM._forward_rnn(cell=self.fcells[layer], \\\n                                                                                input=input, masks=masks,\n                                                                                initial=initial,\n                                                                                drop_masks=hidden_mask)\n            if self.bidirectional:\n                blayer_output, (blayer_h_n, blayer_c_n) = VariationalLSTM._forward_brnn(cell=self.bcells[layer], \\\n                                                                                        input=input, masks=masks,\n                                                                                        initial=initial,\n                                                                                        drop_masks=hidden_mask)\n\n            h_n.append(torch.cat([layer_h_n, blayer_h_n], 1) if self.bidirectional else layer_h_n)\n            c_n.append(torch.cat([layer_c_n, blayer_c_n], 1) if self.bidirectional else layer_c_n)\n            input = torch.cat([layer_output, blayer_output], 2) if self.bidirectional else layer_output\n\n        h_n = torch.stack(h_n, 0)\n        c_n = torch.stack(c_n, 0)\n        if self.batch_first:\n            input = input.transpose(1, 0)  # transpose: return the transpose matrix\n        return input, (h_n, c_n)\n"
  },
  {
    "path": "hanlp/components/srl/span_rank/span_rank.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-07-09 18:13\nimport logging\nfrom bisect import bisect\nfrom typing import Union, List, Callable, Tuple, Dict, Any\n\nfrom hanlp_common.constant import IDX\nfrom hanlp.layers.transformers.utils import build_optimizer_scheduler_with_transformer\nimport torch\nfrom torch.utils.data import DataLoader\nfrom hanlp.common.dataset import PadSequenceDataLoader, SortingSampler\nfrom hanlp.common.torch_component import TorchComponent\nfrom hanlp.common.transform import FieldLength\nfrom hanlp.common.vocab import Vocab\nfrom hanlp.components.srl.span_rank.inference_utils import srl_decode\nfrom hanlp.components.srl.span_rank.span_ranking_srl_model import SpanRankingSRLModel\nfrom hanlp.components.srl.span_rank.srl_eval_utils import compute_srl_f1\nfrom hanlp.datasets.srl.loaders.conll2012 import CoNLL2012SRLDataset, filter_v_args, unpack_srl, \\\n    group_pa_by_p\nfrom hanlp.layers.embeddings.embedding import Embedding\nfrom hanlp.metrics.f1 import F1\nfrom hanlp_common.visualization import markdown_table\nfrom hanlp.utils.time_util import CountdownTimer\nfrom hanlp_common.util import merge_locals_kwargs, reorder\n\n\nclass SpanRankingSemanticRoleLabeler(TorchComponent):\n    def __init__(self, **kwargs) -> None:\n        \"\"\"An implementation of \"Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling\"\n        (:cite:`he-etal-2018-jointly`). It generates candidates triples of (predicate, arg_start, arg_end) and rank them.\n\n        Args:\n            **kwargs: Predefined config.\n        \"\"\"\n        super().__init__(**kwargs)\n        self.model: SpanRankingSRLModel = None\n\n    def build_optimizer(self,\n                        trn,\n                        epochs,\n                        lr,\n                        adam_epsilon,\n                        weight_decay,\n                        warmup_steps,\n                        transformer_lr,\n                        **kwargs):\n        # noinspection PyProtectedMember\n        transformer = self._get_transformer()\n        if transformer:\n            num_training_steps = len(trn) * epochs // self.config.get('gradient_accumulation', 1)\n            optimizer, scheduler = build_optimizer_scheduler_with_transformer(self.model,\n                                                                              transformer,\n                                                                              lr, transformer_lr,\n                                                                              num_training_steps, warmup_steps,\n                                                                              weight_decay, adam_epsilon)\n        else:\n            optimizer = torch.optim.Adam(self.model.parameters(), self.config.lr)\n            scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(\n                optimizer=optimizer,\n                mode='max',\n                factor=0.5,\n                patience=2,\n                verbose=True,\n            )\n        return optimizer, scheduler\n\n    def _get_transformer(self):\n        return getattr(self.model_.embed, 'transformer', None)\n\n    def build_criterion(self, **kwargs):\n        pass\n\n    # noinspection PyProtectedMember\n    def build_metric(self, **kwargs) -> Tuple[F1, F1]:\n        predicate_f1 = F1()\n        end_to_end_f1 = F1()\n        return predicate_f1, end_to_end_f1\n\n    def execute_training_loop(self,\n                              trn: DataLoader,\n                              dev: DataLoader,\n                              epochs,\n                              criterion,\n                              optimizer,\n                              metric,\n                              save_dir,\n                              logger: logging.Logger,\n                              devices,\n                              **kwargs):\n        best_epoch, best_metric = 0, -1\n        predicate, end_to_end = metric\n        optimizer, scheduler = optimizer\n        timer = CountdownTimer(epochs)\n        ratio_width = len(f'{len(trn)}/{len(trn)}')\n        for epoch in range(1, epochs + 1):\n            logger.info(f\"[yellow]Epoch {epoch} / {epochs}:[/yellow]\")\n            self.fit_dataloader(trn, criterion, optimizer, metric, logger,\n                                linear_scheduler=scheduler if self._get_transformer() else None)\n            if dev:\n                self.evaluate_dataloader(dev, criterion, metric, logger, ratio_width=ratio_width)\n            report = f'{timer.elapsed_human}/{timer.total_time_human}'\n            dev_score = end_to_end.score\n            if not self._get_transformer():\n                scheduler.step(dev_score)\n            if dev_score > best_metric:\n                self.save_weights(save_dir)\n                best_metric = dev_score\n                report += ' [red]saved[/red]'\n            timer.log(report, ratio_percentage=False, newline=True, ratio=False)\n\n    def fit_dataloader(self,\n                       trn: DataLoader,\n                       criterion,\n                       optimizer,\n                       metric,\n                       logger: logging.Logger,\n                       linear_scheduler=None,\n                       gradient_accumulation=1,\n                       **kwargs):\n        self.model.train()\n        timer = CountdownTimer(len(trn) // gradient_accumulation)\n        total_loss = 0\n        self.reset_metrics(metric)\n        for idx, batch in enumerate(trn):\n            output_dict = self.feed_batch(batch)\n            self.update_metrics(batch, output_dict, metric)\n            loss = output_dict['loss']\n            loss = loss.sum()  # For data parallel\n            if torch.isnan(loss):  # w/ gold pred, some batches do not have PAs at all, resulting in empty scores\n                loss = torch.zeros((1,), device=loss.device)\n            else:\n                loss.backward()\n            if gradient_accumulation and gradient_accumulation > 1:\n                loss /= gradient_accumulation\n            if self.config.grad_norm:\n                torch.nn.utils.clip_grad_norm_(self.model.parameters(), self.config.grad_norm)\n            if (idx + 1) % gradient_accumulation == 0:\n                self._step(optimizer, linear_scheduler)\n                timer.log(self.report_metrics(total_loss / (timer.current + 1), metric), ratio_percentage=None,\n                          logger=logger)\n            total_loss += loss.item()\n            del loss\n        if len(trn) % gradient_accumulation:\n            self._step(optimizer, linear_scheduler)\n        return total_loss / timer.total\n\n    def _step(self, optimizer, linear_scheduler):\n        optimizer.step()\n        optimizer.zero_grad()\n        if linear_scheduler:\n            linear_scheduler.step()\n\n    # noinspection PyMethodOverriding\n    @torch.no_grad()\n    def evaluate_dataloader(self,\n                            data: DataLoader,\n                            criterion: Callable,\n                            metric,\n                            logger,\n                            ratio_width=None,\n                            output=False,\n                            official=False,\n                            confusion_matrix=False,\n                            **kwargs):\n        self.model.eval()\n        self.reset_metrics(metric)\n        timer = CountdownTimer(len(data))\n        total_loss = 0\n        if official:\n            sentences = []\n            gold = []\n            pred = []\n        for batch in data:\n            output_dict = self.feed_batch(batch)\n            if official:\n                sentences += batch['token']\n                gold += batch['srl']\n                pred += output_dict['prediction']\n            self.update_metrics(batch, output_dict, metric)\n            loss = output_dict['loss']\n            total_loss += loss.item()\n            timer.log(self.report_metrics(total_loss / (timer.current + 1), metric), ratio_percentage=None,\n                      logger=logger,\n                      ratio_width=ratio_width)\n            del loss\n        if official:\n            scores = compute_srl_f1(sentences, gold, pred)\n            if logger:\n                if confusion_matrix:\n                    labels = sorted(set(y for x in scores.label_confusions.keys() for y in x))\n                    headings = ['GOLD↓PRED→'] + labels\n                    matrix = []\n                    for i, gold in enumerate(labels):\n                        row = [gold]\n                        matrix.append(row)\n                        for j, pred in enumerate(labels):\n                            row.append(scores.label_confusions.get((gold, pred), 0))\n                    matrix = markdown_table(headings, matrix)\n                    logger.info(f'{\"Confusion Matrix\": ^{len(matrix.splitlines()[0])}}')\n                    logger.info(matrix)\n                headings = ['Settings', 'Precision', 'Recall', 'F1']\n                data = []\n                for h, (p, r, f) in zip(['Unlabeled', 'Labeled', 'Official'], [\n                    [scores.unlabeled_precision, scores.unlabeled_recall, scores.unlabeled_f1],\n                    [scores.precision, scores.recall, scores.f1],\n                    [scores.conll_precision, scores.conll_recall, scores.conll_f1],\n                ]):\n                    data.append([h] + [f'{x:.2%}' for x in [p, r, f]])\n                table = markdown_table(headings, data)\n                logger.info(f'{\"Scores\": ^{len(table.splitlines()[0])}}')\n                logger.info(table)\n        else:\n            scores = metric\n        return total_loss / timer.total, scores\n\n    def build_model(self,\n                    training=True,\n                    **kwargs) -> torch.nn.Module:\n        # noinspection PyTypeChecker\n        # embed: torch.nn.Embedding = self.config.embed.module(vocabs=self.vocabs)[0].embed\n        model = SpanRankingSRLModel(self.config,\n                                    self.config.embed.module(vocabs=self.vocabs, training=training),\n                                    self.config.context_layer,\n                                    len(self.vocabs.srl_label))\n        return model\n\n    # noinspection PyMethodOverriding\n    def build_dataloader(self, data, batch_size, shuffle, device, logger: logging.Logger,\n                         generate_idx=False, transform=None, **kwargs) -> DataLoader:\n        batch_max_tokens = self.config.batch_max_tokens\n        gradient_accumulation = self.config.get('gradient_accumulation', 1)\n        if batch_size:\n            batch_size //= gradient_accumulation\n        if batch_max_tokens:\n            batch_max_tokens //= gradient_accumulation\n        dataset = self.build_dataset(data, generate_idx, logger, transform)\n\n        sampler = SortingSampler([x['token_length'] for x in dataset],\n                                 batch_size=batch_size,\n                                 batch_max_tokens=batch_max_tokens,\n                                 shuffle=shuffle)\n        return PadSequenceDataLoader(batch_sampler=sampler,\n                                     device=device,\n                                     dataset=dataset)\n\n    def build_dataset(self, data, generate_idx, logger, transform=None):\n        dataset = CoNLL2012SRLDataset(data, transform=[filter_v_args, unpack_srl, group_pa_by_p],\n                                      doc_level_offset=self.config.doc_level_offset, generate_idx=generate_idx)\n        if transform:\n            dataset.append_transform(transform)\n        if isinstance(self.config.get('embed', None), Embedding):\n            transform = self.config.embed.transform(vocabs=self.vocabs)\n            if transform:\n                dataset.append_transform(transform)\n        dataset.append_transform(self.vocabs)\n        dataset.append_transform(FieldLength('token'))\n        if isinstance(data, str):\n            dataset.purge_cache()  # Enable cache\n        if self.vocabs.mutable:\n            self.build_vocabs(dataset, logger)\n        return dataset\n\n    def predict(self, data: Union[str, List[str]], batch_size: int = None, fmt='dict', **kwargs):\n        if not data:\n            return []\n        flat = self.input_is_flat(data)\n        if flat:\n            data = [data]\n        samples = []\n        for token in data:\n            sample = dict()\n            sample['token'] = token\n            samples.append(sample)\n        batch_size = batch_size or self.config.batch_size\n        dataloader = self.build_dataloader(samples, batch_size, False, self.device, None, generate_idx=True)\n        outputs = []\n        order = []\n        for batch in dataloader:\n            output_dict = self.feed_batch(batch)\n            outputs.extend(output_dict['prediction'])\n            order.extend(batch[IDX])\n        outputs = reorder(outputs, order)\n        if fmt == 'list':\n            outputs = self.format_dict_to_results(data, outputs)\n        if flat:\n            return outputs[0]\n        return outputs\n\n    @staticmethod\n    def format_dict_to_results(data, outputs, exclusive_offset=False, with_predicate=False, with_argument=False,\n                               label_first=False):\n        results = []\n        for i in range(len(outputs)):\n            tokens = data[i]\n            output = []\n            for p, a in outputs[i].items():\n                # a: [(0, 0, 'ARG0')]\n                if with_predicate:\n                    a.insert(bisect([x[0] for x in a], p), (p, p, 'PRED'))\n                if with_argument is not False:\n                    a = [x + (tokens[x[0]:x[1] + 1],) for x in a]\n                    if isinstance(with_argument, str):\n                        a = [x[:-1] + (with_argument.join(x[-1]),) for x in a]\n                if exclusive_offset:\n                    a = [(x[0], x[1] + 1) + x[2:] for x in a]\n                if label_first:\n                    a = [tuple(reversed(x[2:])) + x[:2] for x in a]\n                output.append(a)\n            results.append(output)\n        return results\n\n    def input_is_flat(self, data):\n        return isinstance(data[0], str)\n\n    # noinspection PyMethodOverriding\n    def fit(self,\n            trn_data,\n            dev_data,\n            save_dir,\n            embed,\n            context_layer,\n            batch_size=40,\n            batch_max_tokens=700,\n            lexical_dropout=0.5,\n            dropout=0.2,\n            span_width_feature_size=20,\n            ffnn_size=150,\n            ffnn_depth=2,\n            argument_ratio=0.8,\n            predicate_ratio=0.4,\n            max_arg_width=30,\n            mlp_label_size=100,\n            enforce_srl_constraint=False,\n            use_gold_predicates=False,\n            doc_level_offset=True,\n            use_biaffine=False,\n            lr=1e-3,\n            transformer_lr=1e-5,\n            adam_epsilon=1e-6,\n            weight_decay=0.01,\n            warmup_steps=0.1,\n            grad_norm=5.0,\n            gradient_accumulation=1,\n            loss_reduction='sum',\n            transform=None,\n            devices=None,\n            logger=None,\n            seed=None,\n            **kwargs\n            ):\n\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    def build_vocabs(self, dataset, logger, **kwargs):\n        self.vocabs.srl_label = Vocab(pad_token=None, unk_token=None)\n        # Use null to indicate no relationship\n        self.vocabs.srl_label.add('<null>')\n        timer = CountdownTimer(len(dataset))\n        max_seq_len = 0\n        for each in dataset:\n            max_seq_len = max(max_seq_len, len(each['token_input_ids']))\n            timer.log(f'Building vocabs (max sequence length {max_seq_len}) [blink][yellow]...[/yellow][/blink]')\n            pass\n        timer.stop()\n        timer.erase()\n        self.vocabs['srl_label'].set_unk_as_safe_unk()\n        self.vocabs.lock()\n        self.vocabs.summary(logger)\n\n    def reset_metrics(self, metrics):\n        for each in metrics:\n            each.reset()\n\n    def report_metrics(self, loss, metrics):\n        predicate, end_to_end = metrics\n        return f'loss: {loss:.4f} predicate: {predicate.score:.2%} end_to_end: {end_to_end.score:.2%}'\n\n    def feed_batch(self, batch) -> Dict[str, Any]:\n        output_dict = self.model(batch)\n        prediction = self.decode_output(output_dict, batch, self.model.training)\n        output_dict['prediction'] = prediction\n        return output_dict\n\n    def decode_output(self, output_dict, batch, training=False):\n        idx_to_label = self.vocabs['srl_label'].idx_to_token\n        if training:\n            # Use fast decoding during training,\n            prediction = []\n            top_predicate_indices = output_dict['predicates'].tolist()\n            top_spans = torch.stack([output_dict['arg_starts'], output_dict['arg_ends']], dim=-1).tolist()\n            srl_mask = output_dict['srl_mask'].tolist()\n            srl_scores = output_dict['srl_scores']\n            pal_list = srl_scores.argmax(-1).tolist() if srl_scores.numel() else []\n            for n, (pal, predicate_indices, argument_spans) in enumerate(\n                    zip(pal_list, top_predicate_indices, top_spans)):\n                srl_per_sentence = {}\n                for p, (al, predicate_index) in enumerate(zip(pal, predicate_indices)):\n                    for a, (l, argument_span) in enumerate(zip(al, argument_spans)):\n                        if l and srl_mask[n][p][a]:\n                            args = srl_per_sentence.get(p, None)\n                            if args is None:\n                                args = srl_per_sentence[p] = []\n                            args.append((*argument_span, idx_to_label[l]))\n                prediction.append(srl_per_sentence)\n        else:\n            prediction = srl_decode(batch['token_length'], output_dict, idx_to_label, self.config)\n        return prediction\n\n    def update_metrics(self, batch: dict, output_dict: dict, metrics):\n        def unpack(y: dict):\n            return set((p, bel) for p, a in y.items() for bel in a)\n\n        predicate, end_to_end = metrics\n        for pred, gold in zip(output_dict['prediction'], batch['srl']):\n            predicate(pred.keys(), gold.keys())\n            end_to_end(unpack(pred), unpack(gold))\n"
  },
  {
    "path": "hanlp/components/srl/span_rank/span_ranking_srl_model.py",
    "content": "from typing import Dict\n\nimport hanlp.utils.torch_util\nfrom hanlp.layers.feedforward import FeedForward\nfrom hanlp.layers.time_distributed import TimeDistributed\n\nfrom .highway_variational_lstm import *\nimport torch\n\nfrom ...parsers.biaffine.biaffine import Biaffine\n\n\ndef initializer_1d(input_tensor, initializer):\n    assert len(input_tensor.size()) == 1\n    input_tensor = input_tensor.view(-1, 1)\n    input_tensor = initializer(input_tensor)\n    return input_tensor.view(-1)\n\n\nclass SpanRankingSRLDecoder(nn.Module):\n\n    def __init__(self, context_layer_output_dim, label_space_size, config) -> None:\n        super().__init__()\n        self.config = config\n        self.label_space_size = label_space_size\n        self.dropout = float(config.dropout)\n        self.use_gold_predicates = config.use_gold_predicates\n        # span width feature embedding\n        self.span_width_embedding = nn.Embedding(self.config.max_arg_width, self.config.span_width_feature_size)\n        # self.context_projective_layer = nn.Linear(2 * self.lstm_hidden_size, self.config.num_attention_heads)\n        # span scores\n        self.span_emb_size = 3 * context_layer_output_dim + self.config.span_width_feature_size\n        self.arg_unary_score_layers = nn.ModuleList([nn.Linear(self.span_emb_size, self.config.ffnn_size) if i == 0\n                                                     else nn.Linear(self.config.ffnn_size, self.config.ffnn_size) for i\n                                                     in range(self.config.ffnn_depth)])  # [,150]\n        self.arg_dropout_layers = nn.ModuleList([nn.Dropout(self.dropout) for _ in range(self.config.ffnn_depth)])\n        self.arg_unary_score_projection = nn.Linear(self.config.ffnn_size, 1)\n        # predicate scores\n        self.pred_unary_score_layers = nn.ModuleList(\n            [nn.Linear(context_layer_output_dim, self.config.ffnn_size) if i == 0\n             else nn.Linear(self.config.ffnn_size, self.config.ffnn_size) for i\n             in range(self.config.ffnn_depth)])  # [,150]\n        self.pred_dropout_layers = nn.ModuleList([nn.Dropout(self.dropout) for _ in range(self.config.ffnn_depth)])\n        self.pred_unary_score_projection = nn.Linear(self.config.ffnn_size, 1)\n        # srl scores\n        self.srl_unary_score_input_size = self.span_emb_size + context_layer_output_dim\n        self.srl_unary_score_layers = nn.ModuleList([nn.Linear(self.srl_unary_score_input_size, self.config.ffnn_size)\n                                                     if i == 0 else nn.Linear(self.config.ffnn_size,\n                                                                              self.config.ffnn_size)\n                                                     for i in range(self.config.ffnn_depth)])\n        self.srl_dropout_layers = nn.ModuleList([nn.Dropout(self.dropout) for _ in range(self.config.ffnn_depth)])\n        self.srl_unary_score_projection = nn.Linear(self.config.ffnn_size, self.label_space_size - 1)\n        if config.use_biaffine:\n            self.predicate_scale = TimeDistributed(FeedForward(context_layer_output_dim, 1, self.span_emb_size, 'ReLU'))\n            self.biaffine = Biaffine(self.span_emb_size, self.label_space_size - 1)\n        self.loss_reduction = config.loss_reduction\n        self.reset_parameters()\n\n    def reset_parameters(self):\n        init.xavier_uniform_(self.span_width_embedding.weight)\n        # init.xavier_uniform_(self.context_projective_layer.weight)\n        # initializer_1d(self.context_projective_layer.bias, init.xavier_uniform_)\n\n        for layer in self.arg_unary_score_layers:\n            init.xavier_uniform_(layer.weight)\n            initializer_1d(layer.bias, init.xavier_uniform_)\n        init.xavier_uniform_(self.arg_unary_score_projection.weight)\n        initializer_1d(self.arg_unary_score_projection.bias, init.xavier_uniform_)\n\n        for layer in self.pred_unary_score_layers:\n            init.xavier_uniform_(layer.weight)\n            initializer_1d(layer.bias, init.xavier_uniform_)\n        init.xavier_uniform_(self.pred_unary_score_projection.weight)\n        initializer_1d(self.pred_unary_score_projection.bias, init.xavier_uniform_)\n\n        for layer in self.srl_unary_score_layers:\n            init.xavier_uniform_(layer.weight)\n            initializer_1d(layer.bias, init.xavier_uniform_)\n        init.xavier_uniform_(self.srl_unary_score_projection.weight)\n        initializer_1d(self.srl_unary_score_projection.bias, init.xavier_uniform_)\n        return None\n\n    def forward(self, hidden_states, batch, mask=None):\n        gold_arg_ends, gold_arg_labels, gold_arg_starts, gold_predicates, masks, sent_lengths = SpanRankingSRLModel.unpack(\n            batch, mask=mask, training=self.training)\n        return self.decode(hidden_states, sent_lengths, masks, gold_arg_starts, gold_arg_ends, gold_arg_labels,\n                           gold_predicates)\n\n    @staticmethod\n    def get_candidate_spans(sent_lengths: torch.Tensor, max_sent_length, max_arg_width):\n        num_sentences = len(sent_lengths)\n        device = sent_lengths.device\n        candidate_starts = torch.arange(0, max_sent_length, device=device).expand(num_sentences, max_arg_width, -1)\n        candidate_width = torch.arange(0, max_arg_width, device=device).view(1, -1, 1)\n        candidate_ends = candidate_starts + candidate_width\n\n        candidate_starts = candidate_starts.contiguous().view(num_sentences, max_sent_length * max_arg_width)\n        candidate_ends = candidate_ends.contiguous().view(num_sentences, max_sent_length * max_arg_width)\n        actual_sent_lengths = sent_lengths.view(-1, 1).expand(-1, max_sent_length * max_arg_width)\n        candidate_mask = candidate_ends < actual_sent_lengths\n\n        candidate_starts = candidate_starts * candidate_mask\n        candidate_ends = candidate_ends * candidate_mask\n        return candidate_starts, candidate_ends, candidate_mask\n\n    @staticmethod\n    def exclusive_cumsum(input: torch.Tensor, exclusive=True):\n        \"\"\"\n\n        Args:\n          input: input is the sentence lengths tensor.\n          exclusive: exclude the last sentence length (Default value = True)\n          input(torch.Tensor :): \n          input: torch.Tensor: \n\n        Returns:\n\n        \n        \"\"\"\n        assert exclusive is True\n        if exclusive is True:\n            exclusive_sent_lengths = input.new_zeros(1, dtype=torch.long)\n            result = torch.cumsum(torch.cat([exclusive_sent_lengths, input], 0)[:-1], 0).view(-1, 1)\n        else:\n            result = torch.cumsum(input, 0).view(-1, 1)\n        return result\n\n    def flatten_emb(self, emb):\n        num_sentences, max_sentence_length = emb.size()[0], emb.size()[1]\n        assert len(emb.size()) == 3\n        flatted_emb = emb.contiguous().view(num_sentences * max_sentence_length, -1)\n        return flatted_emb\n\n    def flatten_emb_in_sentence(self, emb, batch_sentences_mask):\n        num_sentences, max_sentence_length = emb.size()[0], emb.size()[1]\n        flatted_emb = self.flatten_emb(emb)\n        return flatted_emb[batch_sentences_mask.reshape(num_sentences * max_sentence_length)]\n\n    def get_span_emb(self, flatted_context_emb, flatted_candidate_starts, flatted_candidate_ends,\n                     config, dropout=0.0):\n        batch_word_num = flatted_context_emb.size()[0]\n        # gather slices from embeddings according to indices\n        span_start_emb = flatted_context_emb[flatted_candidate_starts]\n        span_end_emb = flatted_context_emb[flatted_candidate_ends]\n        span_emb_feature_list = [span_start_emb, span_end_emb]  # store the span vector representations for span rep.\n\n        span_width = 1 + flatted_candidate_ends - flatted_candidate_starts  # [num_spans], generate the span width\n        max_arg_width = config.max_arg_width\n\n        # get the span width feature emb\n        span_width_index = span_width - 1\n        span_width_emb = self.span_width_embedding(span_width_index)\n        span_width_emb = F.dropout(span_width_emb, dropout, self.training)\n        span_emb_feature_list.append(span_width_emb)\n\n        \"\"\"head features\"\"\"\n        cpu_flatted_candidte_starts = flatted_candidate_starts\n        span_indices = torch.arange(0, max_arg_width, device=flatted_context_emb.device).view(1, -1) + \\\n                       cpu_flatted_candidte_starts.view(-1, 1)  # For all the i, where i in [begin, ..i, end] for span\n        # reset the position index to the batch_word_num index with index - 1\n        span_indices = torch.clamp(span_indices, max=batch_word_num - 1)\n        num_spans, spans_width = span_indices.size()[0], span_indices.size()[1]\n        flatted_span_indices = span_indices.view(-1)  # so Huge!!!, column is the span?\n        # if torch.cuda.is_available():\n        flatted_span_indices = flatted_span_indices\n        span_text_emb = flatted_context_emb.index_select(0, flatted_span_indices).view(num_spans, spans_width, -1)\n        span_indices_mask = hanlp.utils.torch_util.lengths_to_mask(span_width, max_len=max_arg_width)\n        # project context output to num head\n        # head_scores = self.context_projective_layer.forward(flatted_context_emb)\n        # get span attention\n        # span_attention = head_scores.index_select(0, flatted_span_indices).view(num_spans, spans_width)\n        # span_attention = torch.add(span_attention, expanded_span_indices_log_mask).unsqueeze(2)  # control the span len\n        # span_attention = F.softmax(span_attention, dim=1)\n        span_text_emb = span_text_emb * span_indices_mask.unsqueeze(2).expand(-1, -1, span_text_emb.size()[-1])\n        span_head_emb = torch.mean(span_text_emb, 1)\n        span_emb_feature_list.append(span_head_emb)\n\n        span_emb = torch.cat(span_emb_feature_list, 1)\n        return span_emb, None, span_text_emb, span_indices, span_indices_mask\n\n    def get_arg_unary_scores(self, span_emb):\n        \"\"\"Compute span score with FFNN(span embedding)\n\n        Args:\n          span_emb: tensor of [num_sentences, num_spans, emb_size]\n          config: param dropout:\n          num_labels: param name:\n\n        Returns:\n\n        \n        \"\"\"\n        input = span_emb\n        for i, ffnn in enumerate(self.arg_unary_score_layers):\n            input = F.relu(ffnn.forward(input))\n            input = self.arg_dropout_layers[i].forward(input)\n        output = self.arg_unary_score_projection.forward(input)\n        return output\n\n    def get_pred_unary_scores(self, span_emb):\n        input = span_emb\n        for i, ffnn in enumerate(self.pred_unary_score_layers):\n            input = F.relu(ffnn.forward(input))\n            input = self.pred_dropout_layers[i].forward(input)\n        output = self.pred_unary_score_projection.forward(input)\n        return output\n\n    def extract_spans(self, candidate_scores, candidate_starts, candidate_ends, topk, max_sentence_length,\n                      sort_spans, enforce_non_crossing):\n        \"\"\"extract the topk span indices\n\n        Args:\n          candidate_scores: param candidate_starts:\n          candidate_ends: param topk: [num_sentences]\n          max_sentence_length: param sort_spans:\n          enforce_non_crossing: return: indices [num_sentences, max_num_predictions]\n          candidate_starts: \n          topk: \n          sort_spans: \n\n        Returns:\n\n        \n        \"\"\"\n        # num_sentences = candidate_scores.size()[0]\n        # num_input_spans = candidate_scores.size()[1]\n        max_num_output_spans = int(torch.max(topk))\n        indices = [score.topk(k)[1] for score, k in zip(candidate_scores, topk)]\n        output_span_indices_tensor = [F.pad(item, [0, max_num_output_spans - item.size()[0]], value=item[-1])\n                                      for item in indices]\n        output_span_indices_tensor = torch.stack(output_span_indices_tensor)\n        return output_span_indices_tensor\n\n    def batch_index_select(self, emb, indices):\n        num_sentences = emb.size()[0]\n        max_sent_length = emb.size()[1]\n        flatten_emb = self.flatten_emb(emb)\n        offset = (torch.arange(0, num_sentences, device=emb.device) * max_sent_length).unsqueeze(1)\n        return torch.index_select(flatten_emb, 0, (indices + offset).view(-1)) \\\n            .view(indices.size()[0], indices.size()[1], emb.size(-1))\n\n    def get_batch_topk(self, candidate_starts: torch.Tensor, candidate_ends, candidate_scores, topk_ratio, text_len,\n                       max_sentence_length, sort_spans=False, enforce_non_crossing=True):\n        num_sentences = candidate_starts.size()[0]\n        max_sentence_length = candidate_starts.size()[1]\n\n        topk = torch.floor(text_len.to(torch.float) * topk_ratio).to(torch.long)\n        topk = torch.max(topk, torch.ones(num_sentences, device=candidate_starts.device, dtype=torch.long))\n\n        # this part should be implemented with C++\n        predicted_indices = self.extract_spans(candidate_scores, candidate_starts, candidate_ends, topk,\n                                               max_sentence_length, sort_spans, enforce_non_crossing)\n        predicted_starts = torch.gather(candidate_starts, 1, predicted_indices)\n        predicted_ends = torch.gather(candidate_ends, 1, predicted_indices)\n        predicted_scores = torch.gather(candidate_scores, 1, predicted_indices)\n        return predicted_starts, predicted_ends, predicted_scores, topk, predicted_indices\n\n    def get_dense_span_labels(self, span_starts, span_ends, span_labels, max_sentence_length,\n                              span_parents=None):\n        num_sentences = span_starts.size()[0]\n        max_spans_num = span_starts.size()[1]\n\n        # span_starts = span_starts + 1 - (span_labels > 0).to(torch.long)\n        span_starts[(span_labels == 0) & (span_starts < max_sentence_length - 1)] += 1  # make start > end\n        sentence_indices = torch.arange(0, num_sentences, device=span_starts.device).unsqueeze(1).expand(-1,\n                                                                                                         max_spans_num)\n\n        sparse_indices = torch.cat([sentence_indices.unsqueeze(2), span_starts.unsqueeze(2), span_ends.unsqueeze(2)],\n                                   dim=2)\n        if span_parents is not None:  # semantic span predicate offset\n            sparse_indices = torch.cat([sparse_indices, span_parents.unsqueeze(2)], 2)\n\n        rank = 3 if span_parents is None else 4\n        dense_labels = torch.sparse.LongTensor(sparse_indices.view(num_sentences * max_spans_num, rank).t(),\n                                               span_labels.view(-1),\n                                               torch.Size([num_sentences] + [max_sentence_length] * (rank - 1))) \\\n            .to_dense()\n        return dense_labels\n\n    @staticmethod\n    def gather_4d(params, indices):\n        assert len(params.size()) == 4 and len(indices) == 4\n        indices_a, indices_b, indices_c, indices_d = indices\n        result = params[indices_a, indices_b, indices_c, indices_d]\n        return result\n\n    def get_srl_labels(self,\n                       arg_starts,\n                       arg_ends,\n                       predicates,\n                       gold_predicates,\n                       gold_arg_starts,\n                       gold_arg_ends,\n                       gold_arg_labels,\n                       max_sentence_length\n                       ):\n        num_sentences = arg_starts.size()[0]\n        max_arg_num = arg_starts.size()[1]\n        max_pred_num = predicates.size()[1]\n\n        sentence_indices_2d = torch.arange(0, num_sentences, device=arg_starts.device).unsqueeze(1).unsqueeze(2).expand(\n            -1, max_arg_num, max_pred_num)\n        expanded_arg_starts = arg_starts.unsqueeze(2).expand(-1, -1, max_pred_num)\n        expanded_arg_ends = arg_ends.unsqueeze(2).expand(-1, -1, max_pred_num)\n        expanded_predicates = predicates.unsqueeze(1).expand(-1, max_arg_num, -1)\n\n        dense_srl_labels = self.get_dense_span_labels(gold_arg_starts,\n                                                      gold_arg_ends,\n                                                      gold_arg_labels,\n                                                      max_sentence_length, span_parents=gold_predicates)  # ans\n        srl_labels = self.gather_4d(dense_srl_labels,\n                                    [sentence_indices_2d, expanded_arg_starts, expanded_arg_ends, expanded_predicates])\n        return srl_labels\n\n    def get_srl_unary_scores(self, span_emb):\n        input = span_emb\n        for i, ffnn in enumerate(self.srl_unary_score_layers):\n            input = F.relu(ffnn.forward(input))\n            input = self.srl_dropout_layers[i].forward(input)\n        output = self.srl_unary_score_projection.forward(input)\n        return output\n\n    def get_srl_scores(self, arg_emb, pred_emb, arg_scores, pred_scores, num_labels, config, dropout):\n        num_sentences = arg_emb.size()[0]\n        num_args = arg_emb.size()[1]  # [batch_size, max_arg_num, arg_emb_size]\n        num_preds = pred_emb.size()[1]  # [batch_size, max_pred_num, pred_emb_size]\n\n        unsqueezed_arg_emb = arg_emb.unsqueeze(2)\n        unsqueezed_pred_emb = pred_emb.unsqueeze(1)\n        expanded_arg_emb = unsqueezed_arg_emb.expand(-1, -1, num_preds, -1)\n        expanded_pred_emb = unsqueezed_pred_emb.expand(-1, num_args, -1, -1)\n        pair_emb_list = [expanded_arg_emb, expanded_pred_emb]\n        pair_emb = torch.cat(pair_emb_list, 3)  # concatenate the argument emb and pre emb\n        pair_emb_size = pair_emb.size()[3]\n        flat_pair_emb = pair_emb.view(num_sentences * num_args * num_preds, pair_emb_size)\n        # get unary scores\n        flat_srl_scores = self.get_srl_unary_scores(flat_pair_emb)\n        srl_scores = flat_srl_scores.view(num_sentences, num_args, num_preds, flat_srl_scores.size(-1))\n        if self.config.use_biaffine:\n            srl_scores += self.biaffine(arg_emb, self.predicate_scale(pred_emb)).permute([0, 2, 3, 1])\n        unsqueezed_arg_scores, unsqueezed_pred_scores = \\\n            arg_scores.unsqueeze(2).unsqueeze(3), pred_scores.unsqueeze(1).unsqueeze(3)\n        srl_scores = srl_scores + unsqueezed_arg_scores + unsqueezed_pred_scores\n        dummy_scores = torch.zeros([num_sentences, num_args, num_preds, 1], device=arg_emb.device)\n        srl_scores = torch.cat([dummy_scores, srl_scores], 3)\n        return srl_scores\n\n    def get_srl_softmax_loss(self, srl_scores, srl_labels, num_predicted_args, num_predicted_preds):\n        srl_loss_mask = self.get_srl_loss_mask(srl_scores, num_predicted_args, num_predicted_preds)\n\n        loss = torch.nn.functional.cross_entropy(srl_scores[srl_loss_mask], srl_labels[srl_loss_mask],\n                                                 reduction=self.loss_reduction)\n        return loss, srl_loss_mask\n\n    def get_srl_loss_mask(self, srl_scores, num_predicted_args, num_predicted_preds):\n        max_num_arg = srl_scores.size()[1]\n        max_num_pred = srl_scores.size()[2]\n        # num_predicted_args, 1D tensor; max_num_arg: a int variable means the gold ans's max arg number\n        args_mask = hanlp.utils.torch_util.lengths_to_mask(num_predicted_args, max_num_arg)\n        pred_mask = hanlp.utils.torch_util.lengths_to_mask(num_predicted_preds, max_num_pred)\n        srl_loss_mask = args_mask.unsqueeze(2) & pred_mask.unsqueeze(1)\n        return srl_loss_mask\n\n    def decode(self, contextualized_embeddings, sent_lengths, masks, gold_arg_starts, gold_arg_ends, gold_arg_labels,\n               gold_predicates):\n        num_sentences, max_sent_length = masks.size()\n        device = sent_lengths.device\n        \"\"\"generate candidate spans with argument pruning\"\"\"\n        # candidate_starts [num_sentences, max_sent_length * max_arg_width]\n        candidate_starts, candidate_ends, candidate_mask = self.get_candidate_spans(\n            sent_lengths, max_sent_length, self.config.max_arg_width)\n        flatted_candidate_mask = candidate_mask.view(-1)\n        batch_word_offset = self.exclusive_cumsum(sent_lengths)  # get the word offset in a batch\n        # choose the flatted_candidate_starts with the actual existing positions, i.e. exclude the illegal starts\n        flatted_candidate_starts = candidate_starts + batch_word_offset\n        flatted_candidate_starts = flatted_candidate_starts.view(-1)[flatted_candidate_mask].to(torch.long)\n        flatted_candidate_ends = candidate_ends + batch_word_offset\n        flatted_candidate_ends = flatted_candidate_ends.view(-1)[flatted_candidate_mask].to(torch.long)\n        # flatten the lstm output according to the sentence mask, i.e. exclude the illegal (padding) lstm output\n        flatted_context_output = self.flatten_emb_in_sentence(contextualized_embeddings, masks)\n        \"\"\"generate the span embedding\"\"\"\n        candidate_span_emb, head_scores, span_head_emb, head_indices, head_indices_log_mask = self.get_span_emb(\n            flatted_context_output, flatted_candidate_starts, flatted_candidate_ends,\n            self.config, dropout=self.dropout)\n        \"\"\"Get the span ids\"\"\"\n        candidate_span_number = candidate_span_emb.size()[0]\n        max_candidate_spans_num_per_sentence = candidate_mask.size()[1]\n        sparse_indices = candidate_mask.nonzero(as_tuple=False)\n        sparse_values = torch.arange(0, candidate_span_number, device=device)\n        candidate_span_ids = torch.sparse.FloatTensor(sparse_indices.t(), sparse_values,\n                                                      torch.Size([num_sentences,\n                                                                  max_candidate_spans_num_per_sentence])).to_dense()\n        spans_log_mask = torch.log(candidate_mask.to(torch.float))\n        predict_dict = {\"candidate_starts\": candidate_starts, \"candidate_ends\": candidate_ends,\n                        \"head_scores\": head_scores}\n        \"\"\"Get unary scores and topk of candidate argument spans.\"\"\"\n        flatted_candidate_arg_scores = self.get_arg_unary_scores(candidate_span_emb)\n        candidate_arg_scores = flatted_candidate_arg_scores.index_select(0, candidate_span_ids.view(-1)) \\\n            .view(candidate_span_ids.size()[0], candidate_span_ids.size()[1])\n        candidate_arg_scores = candidate_arg_scores + spans_log_mask\n        arg_starts, arg_ends, arg_scores, num_args, top_arg_indices = \\\n            self.get_batch_topk(candidate_starts, candidate_ends, candidate_arg_scores,\n                                self.config.argument_ratio, sent_lengths, max_sent_length,\n                                sort_spans=False, enforce_non_crossing=False)\n        \"\"\"Get the candidate predicate\"\"\"\n        candidate_pred_ids = torch.arange(0, max_sent_length, device=device).unsqueeze(0).expand(num_sentences, -1)\n        candidate_pred_emb = contextualized_embeddings\n        candidate_pred_scores = self.get_pred_unary_scores(candidate_pred_emb)\n        candidate_pred_scores = candidate_pred_scores + torch.log(masks.to(torch.float).unsqueeze(2))\n        candidate_pred_scores = candidate_pred_scores.squeeze(2)\n        if self.use_gold_predicates is True:\n            predicates = gold_predicates\n            num_preds = (gold_arg_labels > 0).sum(dim=-1)\n            pred_scores = torch.zeros_like(predicates)\n            top_pred_indices = predicates\n        else:\n            predicates, _, pred_scores, num_preds, top_pred_indices = self.get_batch_topk(\n                candidate_pred_ids, candidate_pred_ids, candidate_pred_scores, self.config.predicate_ratio,\n                sent_lengths, max_sent_length,\n                sort_spans=False, enforce_non_crossing=False)\n        \"\"\"Get top arg embeddings\"\"\"\n        arg_span_indices = torch.gather(candidate_span_ids, 1, top_arg_indices)  # [num_sentences, max_num_args]\n        arg_emb = candidate_span_emb.index_select(0, arg_span_indices.view(-1)).view(\n            arg_span_indices.size()[0], arg_span_indices.size()[1], -1\n        )  # [num_sentences, max_num_args, emb]\n        \"\"\"Get top predicate embeddings\"\"\"\n        pred_emb = self.batch_index_select(candidate_pred_emb,\n                                           top_pred_indices)  # [num_sentences, max_num_preds, emb]\n        \"\"\"Get the srl scores according to the arg emb and pre emb.\"\"\"\n        srl_scores = self.get_srl_scores(arg_emb, pred_emb, arg_scores, pred_scores, self.label_space_size, self.config,\n                                         self.dropout)  # [num_sentences, max_num_args, max_num_preds, num_labels]\n        if gold_arg_labels is not None:\n            \"\"\"Get the answers according to the labels\"\"\"\n            srl_labels = self.get_srl_labels(arg_starts, arg_ends, predicates, gold_predicates, gold_arg_starts,\n                                             gold_arg_ends, gold_arg_labels, max_sent_length)\n\n            \"\"\"Compute the srl loss\"\"\"\n            srl_loss, srl_mask = self.get_srl_softmax_loss(srl_scores, srl_labels, num_args, num_preds)\n            predict_dict.update({\n                'srl_mask': srl_mask,\n                'loss': srl_loss\n            })\n        else:\n            predict_dict['srl_mask'] = self.get_srl_loss_mask(srl_scores, num_args, num_preds)\n        predict_dict.update({\n            \"candidate_arg_scores\": candidate_arg_scores,\n            \"candidate_pred_scores\": candidate_pred_scores,\n            \"predicates\": predicates,\n            \"arg_starts\": arg_starts,\n            \"arg_ends\": arg_ends,\n            \"arg_scores\": arg_scores,\n            \"pred_scores\": pred_scores,\n            \"num_args\": num_args,\n            \"num_preds\": num_preds,\n            # [num_sentences, num_args, num_preds] avoid max on empty tensor\n            # \"arg_labels\": torch.max(srl_scores, 1)[1] if srl_scores.numel() else srl_scores[:, :, :, 0],\n            \"srl_scores\": srl_scores,\n        })\n        return predict_dict\n\n\nclass SpanRankingSRLModel(nn.Module):\n\n    def __init__(self, config, embed: torch.nn.Module, context_layer: torch.nn.Module, label_space_size):\n        super(SpanRankingSRLModel, self).__init__()\n        self.config = config\n        self.dropout = float(config.dropout)\n        self.lexical_dropout = float(self.config.lexical_dropout)\n        self.label_space_size = label_space_size\n\n        # Initialize layers and parameters\n        self.word_embedding_dim = embed.get_output_dim()  # get the embedding dim\n        self.embed = embed\n        # Initialize context layer\n        self.context_layer = context_layer\n        context_layer_output_dim = context_layer.get_output_dim() if context_layer else self.word_embedding_dim\n        self.decoder = SpanRankingSRLDecoder(context_layer_output_dim, label_space_size, config)\n\n    def forward(self,\n                batch: Dict[str, torch.Tensor]\n                ):\n        gold_arg_ends, gold_arg_labels, gold_arg_starts, gold_predicates, masks, sent_lengths = \\\n            self.unpack(batch, training=self.training)\n\n        context_embeddings = self.embed(batch)\n        context_embeddings = F.dropout(context_embeddings, self.lexical_dropout, self.training)\n        if self.context_layer:\n            context_embeddings = self.context_layer(context_embeddings, masks)\n\n        return self.decoder.decode(context_embeddings, sent_lengths, masks, gold_arg_starts, gold_arg_ends,\n                                   gold_arg_labels, gold_predicates)\n\n    @staticmethod\n    def unpack(batch, mask=None, training=False):\n        keys = 'token_length', 'predicate_offset', 'argument_begin_offset', 'argument_end_offset', 'srl_label_id'\n        sent_lengths, gold_predicates, gold_arg_starts, gold_arg_ends, gold_arg_labels = [batch.get(k, None) for k in\n                                                                                          keys]\n        if mask is None:\n            mask = hanlp.utils.torch_util.lengths_to_mask(sent_lengths)\n        # elif not training:\n        #     sent_lengths = mask.sum(dim=1)\n        return gold_arg_ends, gold_arg_labels, gold_arg_starts, gold_predicates, mask, sent_lengths\n"
  },
  {
    "path": "hanlp/components/srl/span_rank/srl_eval_utils.py",
    "content": "# Evaluation util functions for PropBank SRL.\n\nimport codecs\nimport collections\nimport operator\nimport tempfile\nfrom collections import Counter\n\nfrom hanlp.metrics.srl.srlconll import official_conll_05_evaluate\n\n_SRL_CONLL_EVAL_SCRIPT = \"../run_eval.sh\"\n\n\ndef split_example_for_eval(example):\n    \"\"\"Split document-based samples into sentence-based samples for evaluation.\n\n    Args:\n      example: \n\n    Returns:\n\n    \n    \"\"\"\n    sentences = example[\"sentences\"]\n    num_words = sum(len(s) for s in sentences)\n    word_offset = 0\n    samples = []\n    # assert len(sentences) == 1\n    for i, sentence in enumerate(sentences):\n        # assert i == 0  # For CoNLL-2005, there are always document == sentence.\n        srl_rels = {}\n        ner_spans = []  # Unused.\n        for r in example[\"srl\"][i]:\n            pred_id = r[0] - word_offset\n            if pred_id not in srl_rels:\n                srl_rels[pred_id] = []\n            srl_rels[pred_id].append((r[1] - word_offset, r[2] - word_offset, r[3]))\n        samples.append((sentence, srl_rels, ner_spans))\n        word_offset += len(sentence)\n    return samples\n\n\ndef evaluate_retrieval(span_starts, span_ends, span_scores, pred_starts, pred_ends, gold_spans,\n                       text_length, evaluators, debugging=False):\n    \"\"\"Evaluation for unlabeled retrieval.\n\n    Args:\n      gold_spans: Set of tuples of (start, end).\n      span_starts: \n      span_ends: \n      span_scores: \n      pred_starts: \n      pred_ends: \n      text_length: \n      evaluators: \n      debugging: (Default value = False)\n\n    Returns:\n\n    \n    \"\"\"\n    if len(span_starts) > 0:\n        sorted_starts, sorted_ends, sorted_scores = list(zip(*sorted(\n            zip(span_starts, span_ends, span_scores),\n            key=operator.itemgetter(2), reverse=True)))\n    else:\n        sorted_starts = []\n        sorted_ends = []\n    for k, evaluator in list(evaluators.items()):\n        if k == -3:\n            predicted_spans = set(zip(span_starts, span_ends)) & gold_spans\n        else:\n            if k == -2:\n                predicted_starts = pred_starts\n                predicted_ends = pred_ends\n                if debugging:\n                    print(\"Predicted\", list(zip(sorted_starts, sorted_ends, sorted_scores))[:len(gold_spans)])\n                    print(\"Gold\", gold_spans)\n            # FIXME: scalar index error\n            elif k == 0:\n                is_predicted = span_scores > 0\n                predicted_starts = span_starts[is_predicted]\n                predicted_ends = span_ends[is_predicted]\n            else:\n                if k == -1:\n                    num_predictions = len(gold_spans)\n                else:\n                    num_predictions = (k * text_length) / 100\n                predicted_starts = sorted_starts[:num_predictions]\n                predicted_ends = sorted_ends[:num_predictions]\n            predicted_spans = set(zip(predicted_starts, predicted_ends))\n        evaluator.update(gold_set=gold_spans, predicted_set=predicted_spans)\n\n\ndef _calc_f1(total_gold, total_predicted, total_matched, message=None):\n    precision = total_matched / total_predicted if total_predicted > 0 else 0\n    recall = total_matched / total_gold if total_gold > 0 else 0\n    f1 = 2 * precision * recall / (precision + recall) if precision + recall > 0 else 0\n    if message:\n        print((\"{}: Precision: {:.2%} Recall: {:.2%} F1: {:.2%}\".format(message, precision, recall, f1)))\n    return precision, recall, f1\n\n\ndef compute_span_f1(gold_data, predictions, task_name):\n    assert len(gold_data) == len(predictions)\n    total_gold = 0\n    total_predicted = 0\n    total_matched = 0\n    total_unlabeled_matched = 0\n    label_confusions = Counter()  # Counter of (gold, pred) label pairs.\n\n    for i in range(len(gold_data)):\n        gold = gold_data[i]\n        pred = predictions[i]\n        total_gold += len(gold)\n        total_predicted += len(pred)\n        for a0 in gold:\n            for a1 in pred:\n                if a0[0] == a1[0] and a0[1] == a1[1]:\n                    total_unlabeled_matched += 1\n                    label_confusions.update([(a0[2], a1[2]), ])\n                    if a0[2] == a1[2]:\n                        total_matched += 1\n    prec, recall, f1 = _calc_f1(total_gold, total_predicted, total_matched, task_name)\n    ul_prec, ul_recall, ul_f1 = _calc_f1(total_gold, total_predicted, total_unlabeled_matched,\n                                         \"Unlabeled \" + task_name)\n    return prec, recall, f1, ul_prec, ul_recall, ul_f1, label_confusions\n\n\ndef compute_unlabeled_span_f1(gold_data, predictions, task_name):\n    assert len(gold_data) == len(predictions)\n    total_gold = 0\n    total_predicted = 0\n    total_matched = 0\n    total_unlabeled_matched = 0\n    label_confusions = Counter()  # Counter of (gold, pred) label pairs.\n\n    for i in range(len(gold_data)):\n        gold = gold_data[i]\n        pred = predictions[i]\n        total_gold += len(gold)\n        total_predicted += len(pred)\n        for a0 in gold:\n            for a1 in pred:\n                if a0[0] == a1[0] and a0[1] == a1[1]:\n                    total_unlabeled_matched += 1\n                    label_confusions.update([(a0[2], a1[2]), ])\n                    if a0[2] == a1[2]:\n                        total_matched += 1\n    prec, recall, f1 = _calc_f1(total_gold, total_predicted, total_matched, task_name)\n    ul_prec, ul_recall, ul_f1 = _calc_f1(total_gold, total_predicted, total_unlabeled_matched,\n                                         \"Unlabeled \" + task_name)\n    return prec, recall, f1, ul_prec, ul_recall, ul_f1, label_confusions\n\n\nSRLScores = collections.namedtuple('SRLScores',\n                                   ['unlabeled_precision', 'unlabeled_recall', 'unlabeled_f1', 'precision', 'recall',\n                                    'f1', 'conll_precision', 'conll_recall', 'conll_f1', 'label_confusions',\n                                    'num_sents'])\n\n\ndef compute_srl_f1(sentences, gold_srl, predictions, gold_path=None) -> SRLScores:\n    assert len(gold_srl) == len(predictions)\n    total_gold = 0\n    total_predicted = 0\n    total_matched = 0\n    total_unlabeled_matched = 0\n    num_sents = 0\n    label_confusions = Counter()\n\n    # Compute unofficial F1 of SRL relations.\n    for gold, prediction in zip(gold_srl, predictions):\n        gold_rels = 0\n        pred_rels = 0\n        matched = 0\n        for pred_id, gold_args in gold.items():\n            filtered_gold_args = [a for a in gold_args if a[2] not in [\"V\", \"C-V\"]]\n            total_gold += len(filtered_gold_args)\n            gold_rels += len(filtered_gold_args)\n            if pred_id not in prediction:\n                continue\n            for a0 in filtered_gold_args:\n                for a1 in prediction[pred_id]:\n                    if a0[0] == a1[0] and a0[1] == a1[1]:\n                        total_unlabeled_matched += 1\n                        label_confusions.update([(a0[2], a1[2]), ])\n                        if a0[2] == a1[2]:\n                            total_matched += 1\n                            matched += 1\n        for pred_id, args in prediction.items():\n            filtered_args = [a for a in args if a[2] not in [\"V\"]]  # \"C-V\"]]\n            total_predicted += len(filtered_args)\n            pred_rels += len(filtered_args)\n\n        if gold_rels == matched and pred_rels == matched:\n            num_sents += 1\n\n    precision, recall, f1 = _calc_f1(total_gold, total_predicted, total_matched,\n                                     # \"SRL (unofficial)\"\n                                     )\n    unlabeled_precision, unlabeled_recall, unlabeled_f1 = _calc_f1(total_gold, total_predicted,\n                                                                   total_unlabeled_matched,\n                                                                   # \"Unlabeled SRL (unofficial)\"\n                                                                   )\n\n    # Prepare to compute official F1.\n    if not gold_path:\n        # print(\"No gold conll_eval data provided. Recreating ...\")\n        gold_path = tempfile.NamedTemporaryFile().name\n        print_to_conll(sentences, gold_srl, gold_path, None)\n        gold_predicates = None\n    else:\n        gold_predicates = read_gold_predicates(gold_path)\n\n    temp_output = tempfile.NamedTemporaryFile().name\n    # print((\"Output temp outoput {}\".format(temp_output)))\n    print_to_conll(sentences, predictions, temp_output, gold_predicates)\n\n    # Evaluate twice with official script.\n    conll_precision, conll_recall, conll_f1 = official_conll_05_evaluate(temp_output, gold_path)\n    return SRLScores(unlabeled_precision, unlabeled_recall, unlabeled_f1, precision, recall, f1, conll_precision,\n                     conll_recall, conll_f1, label_confusions, num_sents)\n\n\ndef print_sentence_to_conll(fout, tokens, labels):\n    \"\"\"Print a labeled sentence into CoNLL format.\n\n    Args:\n      fout: \n      tokens: \n      labels: \n\n    Returns:\n\n    \n    \"\"\"\n    for label_column in labels:\n        assert len(label_column) == len(tokens)\n    for i in range(len(tokens)):\n        fout.write(tokens[i].ljust(15))\n        for label_column in labels:\n            fout.write(label_column[i].rjust(15))\n        fout.write(\"\\n\")\n    fout.write(\"\\n\")\n\n\ndef read_gold_predicates(gold_path):\n    print(\"gold path\", gold_path)\n    fin = codecs.open(gold_path, \"r\", \"utf-8\")\n    gold_predicates = [[], ]\n    for line in fin:\n        line = line.strip()\n        if not line:\n            gold_predicates.append([])\n        else:\n            info = line.split()\n            gold_predicates[-1].append(info[0])\n    fin.close()\n    return gold_predicates\n\n\ndef print_to_conll(sentences, srl_labels, output_filename, gold_predicates=None):\n    fout = codecs.open(output_filename, \"w\", \"utf-8\")\n    for sent_id, words in enumerate(sentences):\n        if gold_predicates:\n            assert len(gold_predicates[sent_id]) == len(words)\n        pred_to_args = srl_labels[sent_id]\n        props = [\"-\" for _ in words]\n        col_labels = [[\"*\" for _ in words] for _ in range(len(pred_to_args))]\n        for i, pred_id in enumerate(sorted(pred_to_args.keys())):\n            # To make sure CoNLL-eval script count matching predicates as correct.\n            if gold_predicates and gold_predicates[sent_id][pred_id] != \"-\":\n                props[pred_id] = gold_predicates[sent_id][pred_id]\n            else:\n                props[pred_id] = \"P\" + words[pred_id]\n            flags = [False for _ in words]\n            for start, end, label in pred_to_args[pred_id]:\n                if not max(flags[start:end + 1]):\n                    col_labels[i][start] = \"(\" + label + col_labels[i][start]\n                    col_labels[i][end] = col_labels[i][end] + \")\"\n                    for j in range(start, end + 1):\n                        flags[j] = True\n            # Add unpredicted verb (for predicted SRL).\n            if not flags[pred_id]:  # if the predicate id is False\n                col_labels[i][pred_id] = \"(V*)\"\n        print_sentence_to_conll(fout, props, col_labels)\n    fout.close()\n"
  },
  {
    "path": "hanlp/components/srl/span_rank/util.py",
    "content": "# Adopted from https://github.com/KiroSummer/A_Syntax-aware_MTL_Framework_for_Chinese_SRL\nimport torch\n\n\ndef block_orth_normal_initializer(input_size, output_size):\n    weight = []\n    for o in output_size:\n        for i in input_size:\n            param = torch.FloatTensor(o, i)\n            torch.nn.init.orthogonal_(param)\n            weight.append(param)\n    return torch.cat(weight)\n"
  },
  {
    "path": "hanlp/components/sts/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-05-20 17:02\n"
  },
  {
    "path": "hanlp/components/sts/transformer_sts.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-05-20 17:03\nimport logging\nfrom typing import Union, List\nimport torch\nfrom torch.utils.data import DataLoader\nfrom hanlp.common.structure import History\nfrom hanlp.layers.transformers.pt_imports import AutoConfig_, AutoTokenizer_\nfrom transformers import AutoModelForSequenceClassification\nfrom transformers.modeling_outputs import SequenceClassifierOutput\nfrom hanlp.common.dataset import SortingSamplerBuilder, PadSequenceDataLoader\nfrom hanlp.common.torch_component import TorchComponent\nfrom hanlp.datasets.sts.stsb import SemanticTextualSimilarityDataset\nfrom hanlp.layers.transformers.utils import build_optimizer_scheduler_with_transformer\nfrom hanlp.metrics.spearman_correlation import SpearmanCorrelation\nfrom hanlp.transform.transformer_tokenizer import TransformerTextTokenizer\nfrom hanlp.utils.time_util import CountdownTimer\nfrom hanlp_common.util import merge_locals_kwargs, reorder\nfrom hanlp_common.constant import IDX\n\n\nclass TransformerSemanticTextualSimilarity(TorchComponent):\n\n    def __init__(self, **kwargs) -> None:\n        \"\"\"\n        A simple Semantic Textual Similarity (STS) baseline which fine-tunes a transformer with a regression layer on\n        top of it.\n\n        Args:\n            **kwargs: Predefined config.\n        \"\"\"\n        super().__init__(**kwargs)\n        self._tokenizer = None\n\n    # noinspection PyMethodOverriding\n    def build_dataloader(self, data, batch_size, sent_a_col=None,\n                         sent_b_col=None,\n                         similarity_col=None,\n                         delimiter='auto',\n                         gradient_accumulation=1,\n                         sampler_builder=None,\n                         shuffle=False, device=None, logger: logging.Logger = None,\n                         split=None,\n                         **kwargs) -> DataLoader:\n        dataset = SemanticTextualSimilarityDataset(data,\n                                                   sent_a_col,\n                                                   sent_b_col,\n                                                   similarity_col,\n                                                   delimiter=delimiter,\n                                                   transform=self._tokenizer,\n                                                   cache=isinstance(data, str))\n        if split == 'trn':\n            scores = [x['similarity'] for x in dataset]\n            self.config.max_score = max(scores)\n            self.config.min_score = min(scores)\n        if not sampler_builder:\n            sampler_builder = SortingSamplerBuilder(batch_size=batch_size)\n        lens = [len(x['input_ids']) for x in dataset]\n        return PadSequenceDataLoader(dataset, batch_sampler=sampler_builder.build(lens, shuffle, gradient_accumulation),\n                                     device=device,\n                                     pad={'similarity': 0.0, 'input_ids': self._tokenizer.tokenizer.pad_token_id})\n\n    def build_optimizer(self, trn, epochs, gradient_accumulation=1, lr=1e-3, transformer_lr=5e-5, adam_epsilon=1e-8,\n                        weight_decay=0.0, warmup_steps=0.1, **kwargs):\n        num_training_steps = len(trn) * epochs // gradient_accumulation\n        optimizer, scheduler = build_optimizer_scheduler_with_transformer(self.model,\n                                                                          self.model.base_model,\n                                                                          lr, transformer_lr,\n                                                                          num_training_steps, warmup_steps,\n                                                                          weight_decay, adam_epsilon)\n        return optimizer, scheduler\n\n    def build_criterion(self, **kwargs):\n        pass\n\n    def build_metric(self, **kwargs):\n        return SpearmanCorrelation()\n\n    def execute_training_loop(self, trn: DataLoader, dev: DataLoader, epochs, criterion, optimizer, metric, save_dir,\n                              logger: logging.Logger, devices, ratio_width=None, gradient_accumulation=1, **kwargs):\n        best_epoch, best_metric = 0, -1\n        timer = CountdownTimer(epochs)\n        history = History()\n        for epoch in range(1, epochs + 1):\n            logger.info(f\"[yellow]Epoch {epoch} / {epochs}:[/yellow]\")\n            self.fit_dataloader(trn, criterion, optimizer, metric, logger, ratio_width=ratio_width,\n                                gradient_accumulation=gradient_accumulation, history=history, save_dir=save_dir)\n            report = f'{timer.elapsed_human}/{timer.total_time_human}'\n            self.evaluate_dataloader(dev, logger, ratio_width=ratio_width, save_dir=save_dir, metric=metric)\n            if metric > best_metric:\n                self.save_weights(save_dir)\n                best_metric = float(metric)\n                best_epoch = epoch\n                report += ' [red]saved[/red]'\n            timer.log(report, ratio_percentage=False, newline=True, ratio=False)\n        if best_epoch and best_epoch != epochs:\n            logger.info(f'Restored the best model with {best_metric} saved {epochs - best_epoch} epochs ago')\n            self.load_weights(save_dir)\n\n    def fit_dataloader(self, trn: DataLoader, criterion, optimizer, metric: SpearmanCorrelation, logger: logging.Logger,\n                       history=None, gradient_accumulation=1, **kwargs):\n        self.model.train()\n        optimizer, scheduler = optimizer\n        timer = CountdownTimer(history.num_training_steps(len(trn), gradient_accumulation=gradient_accumulation))\n        total_loss = 0\n        metric.reset()\n        for batch in trn:\n            output = self.feed_batch(batch)\n            prediction = self.decode(output)\n            metric(prediction, batch['similarity'])\n            loss = output['loss']\n            if gradient_accumulation and gradient_accumulation > 1:\n                loss /= gradient_accumulation\n            loss.backward()\n            total_loss += loss.item()\n            if history.step(gradient_accumulation):\n                if self.config.grad_norm:\n                    torch.nn.utils.clip_grad_norm_(self.model.parameters(), self.config.grad_norm)\n                optimizer.step()\n                if scheduler:\n                    scheduler.step()\n                optimizer.zero_grad()\n                timer.log(self.report_metrics(total_loss / (timer.current + 1), metric), ratio_percentage=None,\n                          logger=logger)\n            del loss\n        return total_loss / timer.total\n\n    @torch.no_grad()\n    def evaluate_dataloader(self, data: DataLoader, logger: logging.Logger, metric=None, output=False, **kwargs):\n        self.model.eval()\n        timer = CountdownTimer(len(data))\n        total_loss = 0\n        metric.reset()\n        if output:\n            predictions = []\n            orders = []\n            samples = []\n        for batch in data:\n            output_dict = self.feed_batch(batch)\n            prediction = self.decode(output_dict)\n            metric(prediction, batch['similarity'])\n            if output:\n                predictions.extend(prediction.tolist())\n                orders.extend(batch[IDX])\n                samples.extend(list(zip(batch['sent_a'], batch['sent_b'])))\n            loss = output_dict['loss']\n            total_loss += loss.item()\n            timer.log(self.report_metrics(total_loss / (timer.current + 1), metric), ratio_percentage=None,\n                      logger=logger)\n            del loss\n        if output:\n            predictions = reorder(predictions, orders)\n            samples = reorder(samples, orders)\n            with open(output, 'w') as out:\n                for s, p in zip(samples, predictions):\n                    out.write('\\t'.join(s + (str(p),)))\n                    out.write('\\n')\n        return total_loss / timer.total\n\n    # noinspection PyMethodOverriding\n    def build_model(self, transformer, training=True, **kwargs) -> torch.nn.Module:\n        config = AutoConfig_.from_pretrained(transformer, num_labels=1)\n        if training:\n            model = AutoModelForSequenceClassification.from_pretrained(transformer, config=config)\n        else:\n            model = AutoModelForSequenceClassification.from_config(config)\n        return model\n\n    def predict(self, data: Union[List[str], List[List[str]]], batch_size: int = None, **kwargs) -> Union[\n        float, List[float]]:\n        \"\"\" Predict the similarity between sentence pairs.\n\n        Args:\n            data: Sentence pairs.\n            batch_size: The number of samples in a batch.\n            **kwargs: Not used.\n\n        Returns:\n            Similarities between sentences.\n        \"\"\"\n        if not data:\n            return []\n        flat = isinstance(data[0], str)\n        if flat:\n            data = [data]\n        dataloader = self.build_dataloader([{'sent_a': x[0], 'sent_b': x[1]} for x in data],\n                                           batch_size=batch_size or self.config.batch_size,\n                                           device=self.device)\n        orders = []\n        predictions = []\n        for batch in dataloader:\n            output_dict = self.feed_batch(batch)\n            prediction = self.decode(output_dict)\n            predictions.extend(prediction.tolist())\n            orders.extend(batch[IDX])\n        predictions = reorder(predictions, orders)\n        if flat:\n            return predictions[0]\n        return predictions\n\n    # noinspection PyMethodOverriding\n    def fit(self, trn_data, dev_data, save_dir,\n            transformer,\n            sent_a_col,\n            sent_b_col,\n            similarity_col,\n            delimiter='auto',\n            batch_size=32,\n            max_seq_len=128,\n            epochs=3,\n            lr=1e-3,\n            transformer_lr=5e-5,\n            adam_epsilon=1e-8,\n            weight_decay=0.0,\n            warmup_steps=0.1,\n            gradient_accumulation=1,\n            grad_norm=1.0,\n            sampler_builder=None,\n            devices=None,\n            logger=None,\n            seed=None,\n            finetune: Union[bool, str] = False, eval_trn=True, _device_placeholder=False, **kwargs):\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    def on_config_ready(self, transformer, max_seq_len, **kwargs):\n        super().on_config_ready(**kwargs)\n        self._tokenizer = TransformerTextTokenizer(AutoTokenizer_.from_pretrained(transformer),\n                                                   text_a_key='sent_a',\n                                                   text_b_key='sent_b',\n                                                   output_key='',\n                                                   max_seq_length=max_seq_len)\n\n    def feed_batch(self, batch) -> SequenceClassifierOutput:\n        return self.model(input_ids=batch['input_ids'], attention_mask=batch['attention_mask'],\n                          token_type_ids=batch['token_type_ids'], labels=batch.get('similarity', None))\n\n    def decode(self, output: SequenceClassifierOutput):\n        return output.logits.squeeze(-1).detach().clip(self.config.min_score, self.config.max_score)\n\n    def report_metrics(self, loss, metric):\n        return f'loss: {loss:.4f} {metric}'\n"
  },
  {
    "path": "hanlp/components/taggers/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-08-28 15:39"
  },
  {
    "path": "hanlp/components/taggers/cnn_tagger_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-10-31 13:52\nfrom abc import ABC\nfrom typing import Union, Tuple, Any, List, Iterable\n\nimport tensorflow as tf\n\nfrom hanlp.components.taggers.tagger_tf import TaggerComponent\nfrom hanlp.transform.tsv_tf import TSVTaggingTransform\nfrom hanlp.common.vocab_tf import VocabTF\nfrom hanlp.layers.embeddings.util_tf import build_embedding\n\n\nclass WindowTokenTransform(TSVTaggingTransform):\n\n    def fit(self, trn_path: str, **kwargs):\n        self.word_vocab = VocabTF()\n        self.tag_vocab = VocabTF(pad_token=None, unk_token=None)\n        for ngrams, tags in self.file_to_samples(trn_path):\n            for words in ngrams:\n                self.word_vocab.update(words)\n            self.tag_vocab.update(tags)\n\n    def create_types_shapes_values(self) -> Tuple[Tuple, Tuple, Tuple]:\n        window_radius = self.config.window_radius\n        window_size = 2 * window_radius + 1\n        types = tf.string, tf.string\n        shapes = [None, window_size], [None]\n        values = self.word_vocab.pad_token, self.tag_vocab.first_token\n        return types, shapes, values\n\n    def inputs_to_samples(self, inputs, gold=False):\n        window_radius = self.config.window_radius\n        for t in inputs:\n            if gold:\n                words, tags = t\n            else:\n                words, tags = t, [self.padding_values[-1]] * len(t)\n            ngrams = []\n            for i, word in enumerate(words):\n                features = []\n                for t in range(-window_radius, window_radius + 1):\n                    index = i + t\n                    if index < 0:\n                        feature = 'bos{}'.format(index)\n                    elif index >= len(words):\n                        feature = 'eos+{}'.format(index - len(words) + 1)\n                    else:\n                        feature = words[index]\n                    features.append(feature)\n                ngrams.append(features)\n            yield ngrams, tags\n\n    def X_to_inputs(self, X: Union[tf.Tensor, Tuple[tf.Tensor]]) -> Iterable:\n        for xs in X:\n            words = []\n            for x in xs:\n                words.append(self.word_vocab.idx_to_token[int(x[len(x) // 2])])\n            yield words\n\n\nclass CNNTaggingModel(tf.keras.models.Model):\n    def __init__(self, filters, num_tags, embed, dropout, kernels, **kwargs):\n        super().__init__()\n        self.embed = embed\n        self.embed_dropout = tf.keras.layers.Dropout(rate=dropout)\n        self.conv2d = []\n        for k in kernels:\n            self.conv2d.append(\n                tf.keras.layers.Conv2D(filters=filters, kernel_size=k, data_format='channels_last', padding='same'))\n        self.conv2d_dropout = tf.keras.layers.Dropout(rate=dropout)\n        self.concat = tf.keras.layers.Concatenate()\n        self.dense = tf.keras.layers.Dense(units=num_tags)\n\n    def call(self, inputs, **kwargs):\n        # if inputs.shape_h[0] is None:\n        #     return tf.zeros_like()\n        #     print(inputs)\n        embeds = self.embed(inputs)\n        embeds = self.embed_dropout(embeds)\n        hs = [conv(embeds) for conv in self.conv2d]\n        h = self.concat(hs)\n        h = self.conv2d_dropout(h)\n        shape_h = tf.shape(h)\n        h = tf.reshape(h, [shape_h[0], shape_h[1], h.shape[2] * h.shape[3]])\n        o = self.dense(h)\n        if h.shape[0]:\n            mask = embeds._keras_mask[:, :, 0]\n            o._keras_mask = mask\n        return o\n\n\nclass CNNTaggerTF(TaggerComponent, ABC):\n    def __init__(self, transform: WindowTokenTransform = None) -> None:\n        if not transform:\n            transform = WindowTokenTransform()\n        super().__init__(transform)\n        self.model: CNNTaggingModel = self.model  # refine the type\n        self.transform: WindowTokenTransform = self.transform\n\n    def build_model(self, embedding, **kwargs) -> tf.keras.Model:\n        embed = build_embedding(embedding, self.transform.word_vocab, self.transform)\n        self.transform.map_x = embed.dtype != tf.string\n        model = CNNTaggingModel(num_tags=len(self.transform.tag_vocab),\n                                embed=embed,\n                                **kwargs)\n        # model.build((None, None, 3))\n        return model\n\n    # noinspection PyMethodOverriding\n    def fit(self, trn_data: Any, dev_data: Any, save_dir: str, embedding=200, window_radius=3,\n            kernels=(1, 2, 3, 4, 5), filters=200, dropout=0.3,\n            loss: Union[tf.keras.losses.Loss, str] = None,\n            optimizer: Union[str, tf.keras.optimizers.Optimizer] = 'adam', metrics='accuracy', batch_size=100,\n            epochs=100,\n            logger=None, verbose=True, **kwargs):\n        kwargs.update(locals())\n        for k in 'self', 'kwargs', '__class__':\n            kwargs.pop(k)\n        super().fit(**kwargs)\n\n    @property\n    def input_shape(self) -> List:\n        return [[None, None, self.config.window_radius * 2 + 1]]\n"
  },
  {
    "path": "hanlp/components/taggers/ngram_conv/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-29 22:18"
  },
  {
    "path": "hanlp/components/taggers/ngram_conv/ngram_conv_tagger.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-10-25 00:04\n\nfrom typing import Union, Optional, Tuple, Any, Iterable, List\n\nimport tensorflow as tf\n\nfrom hanlp_common.structure import SerializableDict\nfrom hanlp.components.taggers.tagger_tf import TaggerComponent\nfrom hanlp.transform.tsv_tf import TSVTaggingTransform\nfrom hanlp.transform.txt_tf import bmes_to_words, extract_ngram_features\nfrom hanlp.common.vocab_tf import VocabTF\nfrom hanlp.layers.embeddings.util_tf import build_embedding\nfrom hanlp.layers.weight_normalization import WeightNormalization\nfrom hanlp_common.util import merge_locals_kwargs\n\n\nclass NgramTransform(TSVTaggingTransform):\n\n    def __init__(self, config: SerializableDict = None, map_x=True, map_y=True, **kwargs) -> None:\n        super().__init__(config, map_x, map_y, **kwargs)\n        self.ngram_vocab: Optional[VocabTF] = None\n        self.tag_vocab: Optional[VocabTF] = None\n\n    def inputs_to_samples(self, inputs, gold=False):\n        for data in inputs:\n            if gold:\n                words, tags = data\n            else:\n                words, tags = data, [self.tag_vocab.safe_pad_token] * len(data)\n            features = [words]\n            if not tags:\n                tags = [self.tag_vocab.first_token] * len(words)\n            features.extend(extract_ngram_features(words, False, self.config.window_size))\n            yield tuple(features), tags\n\n    def x_to_idx(self, x) -> Union[tf.Tensor, Tuple]:\n        ids = [self.word_vocab.lookup(x[0]) if self.config.map_word_feature else x[0]]\n        for ngram in x[1:]:\n            ids.append(self.ngram_vocab.lookup(ngram))\n        return tuple(ids)\n\n    def y_to_idx(self, y) -> tf.Tensor:\n        return self.tag_vocab.lookup(y)\n\n    def create_types_shapes_values(self) -> Tuple[Tuple, Tuple, Tuple]:\n        window_size = self.config.window_size\n        ngram_size = window_size * (window_size + 1) // 2\n        vec_dim = 2 + ngram_size\n        shapes = tuple([[None]] * (vec_dim - 1)), [None]\n        types = tuple([tf.string] * (vec_dim - 1)), tf.string\n        word_vocab, ngram_vocab, tag_vocab = self.word_vocab, self.ngram_vocab, self.tag_vocab\n        defaults = tuple([word_vocab.pad_token] + [\n            ngram_vocab.pad_token if ngram_vocab else word_vocab.pad_token] * ngram_size), (\n                       tag_vocab.pad_token if tag_vocab.pad_token else tag_vocab.first_token)\n        return types, shapes, defaults\n\n    def fit(self, trn_path: str, **kwargs):\n        word_vocab, ngram_vocab, tag_vocab = VocabTF(), VocabTF(), VocabTF(pad_token=None, unk_token=None)\n        num_samples = 0\n        for X, Y in self.file_to_samples(trn_path, gold=True):\n            num_samples += 1\n            word_vocab.update(X[0])\n            for ngram in X[1:]:\n                ngram_vocab.update(filter(lambda x: x, ngram))\n            tag_vocab.update(Y)\n        self.word_vocab, self.ngram_vocab, self.tag_vocab = word_vocab, ngram_vocab, tag_vocab\n        if self.config.window_size:\n            vocabs = word_vocab, ngram_vocab, tag_vocab\n        else:\n            vocabs = word_vocab, None, tag_vocab\n        self.word_vocab, self.ngram_vocab, self.tag_vocab = vocabs\n        return num_samples\n\n    def X_to_inputs(self, X: Union[tf.Tensor, Tuple[tf.Tensor]]) -> Iterable:\n        yield from super().X_to_inputs(X[0])\n\n    def input_truth_output_to_str(self, input: List[str], truth: List[str], output: List[str]):\n        words = bmes_to_words(input, output)\n        return ' '.join(words)\n\n\nclass NgramConvTaggingModel(tf.keras.models.Model):\n    def __init__(self, word_embed: tf.keras.layers.Embedding, ngram_embed: tf.keras.layers.Embedding, filters,\n                 kernel_size, dropout_embed, dropout_hidden, weight_norm, num_tags, **kwargs):\n        super().__init__(**kwargs)\n        if ngram_embed is not None:\n            self.ngram_embed = ngram_embed\n        self.word_embed = word_embed\n        # self.concat = tf.keras.layers.Concatenate(axis=2)\n        self.dropout_embed = tf.keras.layers.Dropout(rate=dropout_embed)\n        self.filters_w = []\n        self.filters_v = []\n\n        def create_conv1d(filter, name):\n            conv = tf.keras.layers.Conv1D(filter, kernel_size, padding=\"same\", name=name)\n            if weight_norm:\n                conv_norm = WeightNormalization(conv, name=name + '_norm', data_init=False)\n                return conv_norm\n            return conv\n\n        for idx, filter in enumerate(filters):\n            self.filters_w.append(create_conv1d(filter, 'Conv1Dw_{}'.format(idx)))\n            self.filters_v.append(create_conv1d(filter, 'Conv1Dv_{}'.format(idx)))\n        self.dropout_hidden = tf.keras.layers.Dropout(rate=dropout_hidden)\n        self.dense = tf.keras.layers.Dense(num_tags, use_bias=False)\n\n    def call(self, inputs, **kwargs):\n        if hasattr(self, 'ngram_embed'):\n            chars, ngrams = inputs[0], inputs[1:]\n            embeds = [self.word_embed(chars)]\n            mask = embeds[0]._keras_mask\n            for ngram in ngrams:\n                embeds.append(self.ngram_embed(ngram))\n            if len(embeds) > 1:\n                embed_input = tf.concat(embeds, axis=2)\n            else:\n                embed_input = embeds[0]\n        else:\n            chars = inputs if isinstance(inputs, tf.Tensor) else inputs[0]\n            embed_input = self.word_embed(chars)\n            mask = embed_input._keras_mask\n\n        mask_float = tf.dtypes.cast(mask, tf.float32)\n        embed_input = self.dropout_embed(embed_input)\n        hidden_output = embed_input\n        for fw, fv in zip(self.filters_w.layers, self.filters_v.layers):\n            w = fw(hidden_output)\n            v = fv(hidden_output)\n            hidden_output = w * tf.nn.sigmoid(v)\n            # Mask paddings.\n            hidden_output = hidden_output * tf.expand_dims(mask_float, -1)\n            hidden_output = self.dropout_hidden(hidden_output)\n        # dirty hack\n        hidden_output._keras_mask = mask\n        logits = self.dense(hidden_output)\n        return logits\n\n\nclass NgramConvTaggerTF(TaggerComponent):\n\n    def __init__(self, transform: NgramTransform = None) -> None:\n        if not transform:\n            transform = NgramTransform()\n        super().__init__(transform)\n        self.transform: NgramTransform = transform\n\n    def build_model(self, word_embed, ngram_embed, window_size, weight_norm, filters, kernel_size, dropout_embed,\n                    dropout_hidden, **kwargs) -> tf.keras.Model:\n        word_vocab, ngram_vocab, tag_vocab = self.transform.word_vocab, self.transform.ngram_vocab, \\\n                                             self.transform.tag_vocab\n        word_embed = build_embedding(word_embed, word_vocab, self.transform)\n        if 'map_x' in self.config:\n            self.config.map_word_feature = self.config.map_x\n            del self.config.map_x\n        else:\n            self.config.map_word_feature = True\n        if window_size:\n            ngram_embed = build_embedding(ngram_embed, ngram_vocab, self.transform)\n        else:\n            ngram_embed = None\n        model = NgramConvTaggingModel(word_embed, ngram_embed, filters, kernel_size, dropout_embed, dropout_hidden,\n                                      weight_norm, len(tag_vocab))\n\n        return model\n\n    def fit(self, trn_data: Any, dev_data: Any, save_dir: str, word_embed: Union[str, int, dict] = 200,\n            ngram_embed: Union[str, int,dict] = 50, embedding_trainable=True, window_size=4, kernel_size=3,\n            filters=(200, 200, 200, 200, 200), dropout_embed=0.2, dropout_hidden=0.2, weight_norm=True,\n            loss: Union[tf.keras.losses.Loss, str] = None,\n            optimizer: Union[str, tf.keras.optimizers.Optimizer] = 'adam', metrics='accuracy', batch_size=100,\n            epochs=100,\n            logger=None, verbose=True, **kwargs):\n        assert kwargs.get('run_eagerly', True), 'NgramConvTaggingModel can only run eagerly'\n        kwargs['run_eagerly'] = True\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n"
  },
  {
    "path": "hanlp/components/taggers/pos_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-05 23:05\nfrom hanlp.components.taggers.cnn_tagger_tf import CNNTaggerTF\nfrom hanlp.components.taggers.rnn_tagger_tf import RNNTaggerTF\n\n\nclass CNNPartOfSpeechTaggerTF(CNNTaggerTF):\n    pass\n\n\nclass RNNPartOfSpeechTaggerTF(RNNTaggerTF):\n    pass\n"
  },
  {
    "path": "hanlp/components/taggers/rnn/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-19 15:41"
  },
  {
    "path": "hanlp/components/taggers/rnn/rnntaggingmodel.py",
    "content": "# MIT License\n#\n# Copyright (c) 2020 Yu Zhang\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\n\nfrom typing import Union\n\nimport torch\nimport torch.nn as nn\nfrom torch.nn.utils.rnn import pack_sequence, pad_packed_sequence\nfrom hanlp.layers.crf.crf import CRF\n\n\nclass RNNTaggingModel(nn.Module):\n\n    def __init__(self,\n                 embed: Union[nn.Embedding, int],\n                 rnn_input,\n                 rnn_hidden,\n                 n_out,\n                 drop=0.5,\n                 crf=True,\n                 crf_constraints=None):\n        super(RNNTaggingModel, self).__init__()\n\n        # the embedding layer\n        if isinstance(embed, nn.Module):\n            self.embed = embed\n            n_embed = embed.embedding_dim\n        else:\n            self.embed = None\n            n_embed = embed\n\n        if rnn_input:\n            self.embed_to_rnn = nn.Linear(n_embed, rnn_input)\n        else:\n            self.embed_to_rnn = None\n            rnn_input = n_embed\n\n        # the word-lstm layer\n        self.word_lstm = nn.LSTM(input_size=rnn_input,\n                                 hidden_size=rnn_hidden,\n                                 batch_first=True,\n                                 bidirectional=True)\n\n        # the output layer\n        self.out = nn.Linear(rnn_hidden * 2, n_out)\n        # the CRF layer\n        self.crf = CRF(n_out, crf_constraints) if crf else None\n\n        self.drop = nn.Dropout(drop)\n        # self.drop = SharedDropout(drop)\n        # self.drop = LockedDropout(drop)\n\n        self.reset_parameters()\n\n    def reset_parameters(self):\n        # init Linear\n        nn.init.xavier_uniform_(self.out.weight)\n\n    def forward(self,\n                x: torch.Tensor,\n                batch=None,\n                **kwargs):\n        # get the mask and lengths of given batch\n        mask = x.gt(0)\n        lens = mask.sum(dim=1)\n        # get outputs from embedding layers\n        if isinstance(self.embed, nn.Embedding):\n            x = self.embed(x[mask])\n        else:\n            x = self.embed(batch, mask=mask)\n            if x.dim() == 3:\n                x = x[mask]\n        x = self.drop(x)\n        if self.embed_to_rnn:\n            x = self.embed_to_rnn(x)\n        x = pack_sequence(torch.split(x, lens.tolist()), True)\n        x, _ = self.word_lstm(x)\n        x, _ = pad_packed_sequence(x, True)\n        x = self.drop(x)\n\n        return self.out(x), mask\n"
  },
  {
    "path": "hanlp/components/taggers/rnn_tagger.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-20 13:12\nimport logging\n\nimport torch\nfrom torch import nn\nfrom torch.optim.lr_scheduler import ReduceLROnPlateau\nfrom torch.utils.data import DataLoader\n\nfrom hanlp.common.dataset import PadSequenceDataLoader, SortingSampler, TransformableDataset\nfrom hanlp_common.configurable import Configurable\nfrom hanlp.common.transform import EmbeddingNamedTransform\nfrom hanlp.common.vocab import Vocab\nfrom hanlp.components.taggers.rnn.rnntaggingmodel import RNNTaggingModel\nfrom hanlp.components.taggers.tagger import Tagger\nfrom hanlp.datasets.ner.loaders.tsv import TSVTaggingDataset\nfrom hanlp.layers.embeddings.embedding import Embedding\nfrom hanlp.layers.embeddings.util import build_word2vec_with_vocab\nfrom hanlp.utils.time_util import CountdownTimer\nfrom hanlp_common.util import merge_locals_kwargs, merge_dict\n\n\nclass RNNTagger(Tagger):\n\n    def __init__(self, **kwargs) -> None:\n        \"\"\"An old-school tagger using non-contextualized embeddings and RNNs as context layer.\n\n        Args:\n            **kwargs: Predefined config.\n        \"\"\"\n        super().__init__(**kwargs)\n        self.model: RNNTaggingModel = None\n\n    # noinspection PyMethodOverriding\n    def execute_training_loop(self, trn: DataLoader, dev: DataLoader, epochs, criterion,\n                              optimizer,\n                              metric,\n                              save_dir,\n                              logger,\n                              patience,\n                              **kwargs):\n        max_e, max_metric = 0, -1\n\n        criterion = self.build_criterion()\n        timer = CountdownTimer(epochs)\n        ratio_width = len(f'{len(trn)}/{len(trn)}')\n        scheduler = self.build_scheduler(**merge_dict(self.config, optimizer=optimizer, overwrite=True))\n        if not patience:\n            patience = epochs\n        for epoch in range(1, epochs + 1):\n            logger.info(f\"[yellow]Epoch {epoch} / {epochs}:[/yellow]\")\n            self.fit_dataloader(trn, criterion, optimizer, metric, logger, ratio_width=ratio_width)\n            loss, dev_metric = self.evaluate_dataloader(dev, criterion, logger)\n            if scheduler:\n                if isinstance(scheduler, ReduceLROnPlateau):\n                    scheduler.step(dev_metric.score)\n                else:\n                    scheduler.step(epoch)\n            report_patience = f'Patience: {epoch - max_e}/{patience}'\n            # save the model if it is the best so far\n            if dev_metric > max_metric:\n                self.save_weights(save_dir)\n                max_e, max_metric = epoch, dev_metric\n                report_patience = '[red]Saved[/red] '\n            stop = epoch - max_e >= patience\n            if stop:\n                timer.stop()\n            timer.log(f'{report_patience} lr: {optimizer.param_groups[0][\"lr\"]:.4f}',\n                      ratio_percentage=False, newline=True, ratio=False)\n            if stop:\n                break\n        timer.stop()\n        if max_e != epoch:\n            self.load_weights(save_dir)\n        logger.info(f\"Max score of dev is {max_metric.score:.2%} at epoch {max_e}\")\n        logger.info(f\"{timer.elapsed_human} elapsed, average time of each epoch is {timer.elapsed_average_human}\")\n\n    def build_scheduler(self, optimizer, anneal_factor, anneal_patience, **kwargs):\n        scheduler: ReduceLROnPlateau = ReduceLROnPlateau(optimizer,\n                                                         factor=anneal_factor,\n                                                         patience=anneal_patience,\n                                                         mode='max') if anneal_factor and anneal_patience else None\n        return scheduler\n\n    def fit_dataloader(self, trn: DataLoader, criterion, optimizer, metric, logger: logging.Logger, ratio_width=None,\n                       **kwargs):\n        self.model.train()\n        timer = CountdownTimer(len(trn))\n        total_loss = 0\n        for idx, batch in enumerate(trn):\n            optimizer.zero_grad()\n            out, mask = self.feed_batch(batch)\n            y = batch['tag_id']\n            loss = self.compute_loss(criterion, out, y, mask)\n            loss.backward()\n            nn.utils.clip_grad_norm_(self.model.parameters(), 5.0)\n            optimizer.step()\n            total_loss += loss.item()\n            prediction = self.decode_output(out, mask, batch)\n            self.update_metrics(metric, out, y, mask, batch, prediction)\n            timer.log(f'loss: {loss / (idx + 1):.4f} {metric}', ratio_percentage=False, logger=logger,\n                      ratio_width=ratio_width)\n            del loss\n            del out\n            del mask\n\n    def feed_batch(self, batch):\n        x = batch[f'{self.config.token_key}_id']\n        out, mask = self.model(x, **batch, batch=batch)\n        return out, mask\n\n    # noinspection PyMethodOverriding\n    def build_model(self, rnn_input, rnn_hidden, drop, crf, **kwargs) -> torch.nn.Module:\n        vocabs = self.vocabs\n        token_embed = self._convert_embed()\n        if isinstance(token_embed, EmbeddingNamedTransform):\n            token_embed = token_embed.output_dim\n        elif isinstance(token_embed, Embedding):\n            token_embed = token_embed.module(vocabs=vocabs)\n        else:\n            token_embed = build_word2vec_with_vocab(token_embed, vocabs[self.config.token_key])\n        model = RNNTaggingModel(token_embed, rnn_input, rnn_hidden, len(vocabs['tag']), drop, crf)\n        return model\n\n    def _convert_embed(self):\n        embed = self.config['embed']\n        if isinstance(embed, dict):\n            self.config['embed'] = embed = Configurable.from_config(embed)\n        return embed\n\n    def build_dataloader(self, data, batch_size, shuffle, device, logger=None, **kwargs) -> DataLoader:\n        vocabs = self.vocabs\n        token_embed = self._convert_embed()\n        dataset = data if isinstance(data, TransformableDataset) else self.build_dataset(data, transform=[vocabs])\n        if vocabs.mutable:\n            # Before building vocabs, let embeddings submit their vocabs, some embeddings will possibly opt out as their\n            # transforms are not relevant to vocabs\n            if isinstance(token_embed, Embedding):\n                transform = token_embed.transform(vocabs=vocabs)\n                if transform:\n                    dataset.transform.insert(-1, transform)\n            self.build_vocabs(dataset, logger)\n        if isinstance(token_embed, Embedding):\n            # Vocabs built, now add all transforms to the pipeline. Be careful about redundant ones.\n            transform = token_embed.transform(vocabs=vocabs)\n            if transform and transform not in dataset.transform:\n                dataset.transform.insert(-1, transform)\n        sampler = SortingSampler([len(sample[self.config.token_key]) for sample in dataset], batch_size,\n                                 shuffle=shuffle)\n        return PadSequenceDataLoader(dataset,\n                                     device=device,\n                                     batch_sampler=sampler,\n                                     vocabs=vocabs)\n\n    def build_dataset(self, data, transform):\n        return TSVTaggingDataset(data, transform)\n\n    def build_vocabs(self, dataset, logger):\n        self.vocabs.tag = Vocab(unk_token=None, pad_token=None)\n        self.vocabs[self.config.token_key] = Vocab()\n        for each in dataset:\n            pass\n        self.vocabs.lock()\n        self.vocabs.summary(logger)\n\n    def fit(self, trn_data, dev_data, save_dir,\n            batch_size=50,\n            epochs=100,\n            embed=100,\n            rnn_input=None,\n            rnn_hidden=256,\n            drop=0.5,\n            lr=0.001,\n            patience=10,\n            crf=True,\n            optimizer='adam',\n            token_key='token',\n            tagging_scheme=None,\n            anneal_factor: float = 0.5,\n            anneal_patience=2,\n            devices=None, logger=None, verbose=True, **kwargs):\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    def _id_to_tags(self, ids):\n        batch = []\n        vocab = self.vocabs['tag'].idx_to_token\n        for b in ids:\n            batch.append([])\n            for i in b:\n                batch[-1].append(vocab[i])\n        return batch\n\n    def write_output(self, yhat, y, mask, batch, prediction, output):\n        pass\n"
  },
  {
    "path": "hanlp/components/taggers/rnn_tagger_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-09-14 20:30\nfrom typing import Union, List\n\nimport tensorflow as tf\n\nfrom hanlp.common.transform_tf import Transform\nfrom hanlp.components.taggers.tagger_tf import TaggerComponent\nfrom hanlp.transform.tsv_tf import TSVTaggingTransform\nfrom hanlp.common.vocab_tf import VocabTF\nfrom hanlp.layers.embeddings.util_tf import build_embedding, embeddings_require_string_input, \\\n    embeddings_require_char_input\nfrom hanlp_common.util import merge_locals_kwargs\n\n\nclass RNNTaggerTF(TaggerComponent):\n\n    def __init__(self, transform: Transform = None) -> None:\n        if not transform:\n            self.transform = transform = TSVTaggingTransform()\n        super().__init__(transform)\n\n    def fit(self, trn_data: str, dev_data: str = None, save_dir: str = None, embeddings=100, embedding_trainable=False,\n            rnn_input_dropout=0.2, rnn_units=100, rnn_output_dropout=0.2, epochs=20, lower=False, logger=None,\n            loss: Union[tf.keras.losses.Loss, str] = None,\n            optimizer: Union[str, tf.keras.optimizers.Optimizer] = 'adam', metrics='accuracy',\n            batch_size=32, dev_batch_size=32, lr_decay_per_epoch=None, verbose=True, **kwargs):\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    def build_model(self, embeddings, embedding_trainable, rnn_input_dropout, rnn_output_dropout, rnn_units,\n                    loss,\n                    **kwargs) -> tf.keras.Model:\n        model = tf.keras.Sequential()\n        embeddings = build_embedding(embeddings, self.transform.word_vocab, self.transform)\n        model.add(embeddings)\n        if rnn_input_dropout:\n            model.add(tf.keras.layers.Dropout(rnn_input_dropout, name='rnn_input_dropout'))\n        model.add(\n            tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(units=rnn_units, return_sequences=True), name='bilstm'))\n        if rnn_output_dropout:\n            model.add(tf.keras.layers.Dropout(rnn_output_dropout, name='rnn_output_dropout'))\n        model.add(tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(len(self.transform.tag_vocab)), name='dense'))\n        return model\n\n    def predict(self, sents: Union[List[str], List[List[str]]], batch_size=32, **kwargs) -> Union[\n        List[str], List[List[str]]]:\n        return super().predict(sents, batch_size)\n\n    def save_weights(self, save_dir, filename='model.h5'):\n        # remove the pre-trained embedding\n        embedding_layer: tf.keras.layers.Embedding = self.model.get_layer(index=0)\n        if embedding_layer.trainable:\n            super().save_weights(save_dir, filename)\n        else:\n            truncated_model = tf.keras.Sequential(layers=self.model.layers[1:])\n            truncated_model.build(input_shape=embedding_layer.output_shape)\n            truncated_model.save_weights(save_dir)\n\n    def build_loss(self, loss, **kwargs):\n        if not loss:\n            loss = tf.keras.losses.SparseCategoricalCrossentropy(reduction=tf.keras.losses.Reduction.SUM,\n                                                                 from_logits=True)\n            return loss\n        return super().build_loss(loss, **kwargs)\n\n    @property\n    def tag_vocab(self) -> VocabTF:\n        return self.transform.tag_vocab\n\n    def build_transform(self, embeddings, **kwargs):\n        if embeddings_require_string_input(embeddings):\n            self.transform.map_x = False\n            if embeddings_require_char_input(embeddings):\n                self.transform.char_vocab = VocabTF()\n        return super().build_transform(**kwargs)\n\n    @property\n    def sample_data(self):\n        if self.transform.char_vocab:\n            # You cannot build your model by calling `build` if your layers do not support float type inputs.\n            # Instead, in order to instantiate and build your model, `call` your model on real tensor data (of the\n            # correct dtype).\n            sample = tf.constant([\n                ['hello', 'world', self.transform.word_vocab.pad_token],\n                ['hello', 'this', 'world'],\n            ])\n            sample._keras_mask = tf.not_equal(sample, self.transform.word_vocab.pad_token)\n            return sample\n"
  },
  {
    "path": "hanlp/components/taggers/tagger.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-11 12:19\nimport logging\nimport warnings\nfrom abc import ABC, abstractmethod\nfrom typing import List, TextIO, Any, Union, Dict, Tuple, Sequence\n\nimport torch\nfrom torch import optim, nn\nfrom torch.utils.data import DataLoader\n\nfrom hanlp_common.constant import IDX\nfrom hanlp.common.structure import History\nfrom hanlp.components.distillation.distillable_component import DistillableComponent\nfrom hanlp.components.taggers.util import guess_tagging_scheme\nfrom hanlp.layers.crf.crf import CRF\nfrom hanlp.metrics.accuracy import CategoricalAccuracy\nfrom hanlp.utils.time_util import CountdownTimer\nfrom hanlp_common.util import reorder\nfrom hanlp_trie import DictInterface, TrieDict\nfrom hanlp_trie.dictionary import TupleTrieDict\n\n\nclass Tagger(DistillableComponent, ABC):\n    def build_optimizer(self, optimizer, lr, **kwargs):\n        if optimizer == 'adam':\n            return optim.Adam(params=self.model.parameters(), lr=lr)\n        elif optimizer == 'sgd':\n            return torch.optim.SGD(self.model.parameters(), lr=lr)\n\n    def build_criterion(self, model=None, reduction='mean', decoder=None, **kwargs):\n        if self.config.get('crf', False):\n            if not model:\n                model = decoder or self.model\n            if isinstance(model, nn.DataParallel):\n                raise ValueError('DataParallel not supported when CRF is used')\n                return self.model_from_config.module.crf\n            return model.crf\n        else:\n            return nn.CrossEntropyLoss(reduction=reduction)\n\n    def build_metric(self, **kwargs):\n        return CategoricalAccuracy()\n\n    @abstractmethod\n    def feed_batch(self, batch):\n        pass\n\n    def compute_loss(self, criterion, out, y, mask):\n        if self.config.get('crf', False):\n            criterion: CRF = criterion\n            loss = -criterion.forward(out, y, mask)\n        else:\n            loss = criterion(out[mask], y[mask])\n        return loss\n\n    def decode_output(self, logits, mask, batch, model=None):\n        if self.config.get('crf', False):\n            if model is None:\n                model = self.model\n            crf: CRF = model.crf\n            return crf.decode(logits, mask)\n        else:\n            return logits.argmax(-1)\n\n    def execute_training_loop(self, trn: DataLoader, dev: DataLoader, epochs, criterion, optimizer, metric, save_dir,\n                              logger: logging.Logger, devices, ratio_width=None, patience=5, teacher=None,\n                              kd_criterion=None, eval_trn=True,\n                              **kwargs):\n        best_epoch, best_metric = 0, -1\n        timer = CountdownTimer(epochs)\n        history = History()\n        for epoch in range(1, epochs + 1):\n            logger.info(f\"[yellow]Epoch {epoch} / {epochs}:[/yellow]\")\n            self.fit_dataloader(trn, criterion, optimizer, metric, logger, history=history, ratio_width=ratio_width,\n                                eval_trn=eval_trn, **self.config)\n            loss, dev_metric = self.evaluate_dataloader(dev, criterion, logger=logger, ratio_width=ratio_width)\n            timer.update()\n            report = f\"{timer.elapsed_human} / {timer.total_time_human} ETA: {timer.eta_human}\"\n            if dev_metric > best_metric:\n                best_epoch, best_metric = epoch, dev_metric\n                self.save_weights(save_dir)\n                report += ' [red](saved)[/red]'\n            else:\n                report += f' ({epoch - best_epoch})'\n                if epoch - best_epoch >= patience:\n                    report += ' early stop'\n            logger.info(report)\n            if epoch - best_epoch >= patience:\n                break\n        if not best_epoch:\n            self.save_weights(save_dir)\n        elif best_epoch != epoch:\n            self.load_weights(save_dir)\n        logger.info(f\"Max score of dev is {best_metric} at epoch {best_epoch}\")\n        logger.info(f\"Average time of each epoch is {timer.elapsed_average_human}\")\n        logger.info(f\"{timer.elapsed_human} elapsed\")\n        return best_metric\n\n    def id_to_tags(self, ids: torch.LongTensor, lens: List[int]):\n        batch = []\n        vocab = self.vocabs['tag'].idx_to_token\n        for b, l in zip(ids, lens):\n            batch.append([])\n            for i in b[:l]:\n                batch[-1].append(vocab[i])\n        return batch\n\n    def update_metrics(self, metric, logits, y, mask, batch=None, prediction=None):\n        metric(logits, y, mask)\n\n    @torch.no_grad()\n    def evaluate_dataloader(self, data, criterion, logger=None, ratio_width=None, metric=None, output=None, **kwargs):\n        self.model.eval()\n        if isinstance(output, str):\n            output = open(output, 'w')\n\n        loss = 0\n        if not metric:\n            metric = self.build_metric()\n        else:\n            metric.reset()\n        timer = CountdownTimer(len(data))\n        for idx, batch in enumerate(data):\n            logits, mask = self.feed_batch(batch)\n            y = batch['tag_id']\n            loss += self.compute_loss(criterion, logits, y, mask).item()\n            prediction = self.decode_output(logits, mask, batch)\n            self.update_metrics(metric, logits, y, mask, batch, prediction)\n            if output:\n                self.write_prediction(prediction, batch, output)\n            timer.log(f'loss: {loss / (idx + 1):.4f} {metric}', ratio_percentage=False, logger=logger,\n                      ratio_width=ratio_width)\n        loss /= len(data)\n        if output:\n            output.close()\n        return float(loss), metric\n\n    def write_prediction(self, prediction, batch, output: TextIO):\n        for tokens, ps, gs in zip(batch[self.config.token_key], prediction, batch['tag']):\n            output.write('\\n'.join('\\t'.join([t, p, g]) for t, p, g in zip(tokens, ps, gs)))\n            output.write('\\n')\n\n    def predict(self, tokens: Any, batch_size: int = None, **kwargs):\n        if not tokens:\n            return []\n        flat = self.input_is_flat(tokens)\n        if flat:\n            tokens = [tokens]\n        outputs = self.predict_data(tokens, batch_size, **kwargs)\n        if flat:\n            return outputs[0]\n        return outputs\n\n    def input_is_flat(self, tokens):\n        return isinstance(tokens, list) and isinstance(tokens[0], str)\n\n    def predict_data(self, data, batch_size, sampler_builder=None, **kwargs):\n        samples = self.build_samples(data, **kwargs)\n        if not batch_size:\n            batch_size = self.config.get('batch_size', 32)\n        dataloader = self.build_dataloader(samples, batch_size, False, self.device, sampler_builder=sampler_builder,\n                                           **kwargs)\n        outputs = []\n        orders = []\n        vocab = self.vocabs['tag'].idx_to_token\n        for batch in dataloader:\n            out, mask = self.feed_batch(batch)\n            pred = self.decode_output(out, mask, batch)\n            outputs.extend(self.prediction_to_human(pred, vocab, batch))\n            orders.extend(batch[IDX])\n        outputs = reorder(outputs, orders)\n        return outputs\n\n    def build_samples(self, data: List[str], **kwargs):\n        return [{self.config.token_key: sent} for sent in data]\n\n    def prediction_to_human(self, pred_ids, vocab: List[str], batch):\n        if isinstance(pred_ids, torch.Tensor):\n            pred_ids = pred_ids.tolist()\n        sents = batch.get(f'{self.config.token_key}_')\n        if not sents:\n            sents = batch[self.config.token_key]\n        dict_tags: DictInterface = self.dict_tags\n        for each, sent in zip(pred_ids, sents):\n            tags = [vocab[id] for id in each[:len(sent)]]\n            if dict_tags:\n                for begin, end, label in dict_tags.tokenize(sent):\n                    tags[begin:end] = label\n            yield tags\n\n    @property\n    def tagging_scheme(self):\n        tagging_scheme = self.config.tagging_scheme\n        if not tagging_scheme:\n            self.config.tagging_scheme = tagging_scheme = guess_tagging_scheme(self.vocabs.tag.idx_to_token)\n            if tagging_scheme == 'BIO':\n                warnings.warn(f'The tag scheme for {self.vocabs.tag.idx_to_token} might be IOB1 or IOB2 '\n                              f'but we are using IOB2 by default. Please set tagging_scheme=\"IOB1\" or tagging_scheme=\"BIO\" '\n                              f'to get rid of this warning.')\n        return tagging_scheme\n\n    @property\n    def dict_tags(self) -> DictInterface:\n        r\"\"\" A custom dictionary to override predicted tags by performing longest-prefix-matching.\n\n        Examples:\n            >>> pos.dict_tags = {'HanLP': 'state-of-the-art-tool'} # Force 'HanLP' to be 'state-of-the-art-tool'\n            >>> tagger(\"HanLP为生产环境带来次世代最先进的多语种NLP技术。\")\n                # HanLP/state-of-the-art-tool 为/P 生产/NN 环境/NN 带来/VV 次世代/NN 最/AD 先进/VA 的/DEC 多语种/NN NLP/NR 技术/NN 。/PU\n            >>> pos.dict_tags = {('的', '希望'): ('补语成分', '名词'), '希望': '动词'} # Conditional matching\n            >>> tagger(\"我的希望是希望张晚霞的背影被晚霞映红。\")\n                # 我/PN 的/补语成分 希望/名词 是/VC 希望/动词 张晚霞/NR 的/DEG 背影/NN 被/LB 晚霞/NN 映红/VV 。/PU\n        \"\"\"\n        return self.config.get('dict_tags', None)\n\n    @dict_tags.setter\n    def dict_tags(self,\n                  dictionary: Union[DictInterface, Union[Dict[Union[str, Sequence[str]], Union[str, Sequence[str]]]]]):\n        if dictionary is not None and not isinstance(dictionary, DictInterface):\n            assert isinstance(dictionary, dict), f'Expected dictionary to be `dict` but got {type(dictionary)}.'\n            _d = dict()\n            for k, v in dictionary.items():\n                if isinstance(k, str):\n                    k = (k,)\n                if isinstance(v, str):\n                    v = (v,) * len(k)\n                _d[k] = v\n            dictionary = TupleTrieDict(_d)\n        self.config.dict_tags = dictionary\n"
  },
  {
    "path": "hanlp/components/taggers/tagger_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-10-25 21:49\nimport logging\nfrom abc import ABC\n\nimport tensorflow as tf\n\nfrom hanlp.common.keras_component import KerasComponent\nfrom hanlp.layers.crf.crf_layer_tf import CRF, CRFLoss, CRFWrapper\nfrom hanlp.metrics.chunking.iobes_tf import IOBES_F1_TF\n\n\nclass TaggerComponent(KerasComponent, ABC):\n\n    def build_metrics(self, metrics, logger: logging.Logger, **kwargs):\n        if metrics == 'f1':\n            assert hasattr(self.transform, 'tag_vocab'), 'Name your tag vocab tag_vocab in your transform ' \\\n                                                         'or override build_metrics'\n            if not self.config.get('run_eagerly', None):\n                logger.debug('ChunkingF1 runs only under eager mode, '\n                             'set run_eagerly=True to remove this warning')\n            self.config.run_eagerly = True\n            return IOBES_F1_TF(self.transform.tag_vocab)\n        return super().build_metrics(metrics, logger, **kwargs)\n\n    def build_loss(self, loss, **kwargs):\n        assert self.model is not None, 'should create model before build loss'\n        if loss == 'crf':\n            if isinstance(self.model, tf.keras.models.Sequential):\n                crf = CRF(len(self.transform.tag_vocab))\n                self.model.add(crf)\n                loss = CRFLoss(crf, self.model.dtype)\n            else:\n                self.model = CRFWrapper(self.model, len(self.transform.tag_vocab))\n                loss = CRFLoss(self.model.crf, self.model.dtype)\n            return loss\n        return super().build_loss(loss, **kwargs)\n"
  },
  {
    "path": "hanlp/components/taggers/transformers/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-29 13:57"
  },
  {
    "path": "hanlp/components/taggers/transformers/metrics_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-30 16:33\nimport tensorflow as tf\n\n\nclass Accuracy(tf.keras.metrics.SparseCategoricalAccuracy):\n\n    def __init__(self, name='sparse_categorical_accuracy', dtype=None, mask_value=0):\n        super().__init__(name, dtype)\n        self.mask_value = mask_value\n\n    def update_state(self, y_true, y_pred, sample_weight=None):\n        sample_weight = tf.not_equal(y_true, self.mask_value)\n        return super().update_state(y_true, y_pred, sample_weight)\n"
  },
  {
    "path": "hanlp/components/taggers/transformers/transformer_tagger.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-15 20:55\nimport logging\nfrom typing import Union, List\n\nimport torch\nfrom torch import nn\nfrom torch.utils.data import DataLoader\n\nfrom hanlp.common.dataset import PadSequenceDataLoader, SamplerBuilder, TransformableDataset\nfrom hanlp.common.structure import History\nfrom hanlp.common.transform import FieldLength, TransformList\nfrom hanlp.common.vocab import Vocab\nfrom hanlp.components.classifiers.transformer_classifier import TransformerComponent\nfrom hanlp.components.taggers.tagger import Tagger\nfrom hanlp.datasets.ner.loaders.tsv import TSVTaggingDataset\nfrom hanlp.layers.crf.crf import CRF\nfrom hanlp.layers.embeddings.embedding import EmbeddingDim, Embedding\nfrom hanlp.layers.transformers.encoder import TransformerEncoder\nfrom hanlp.transform.transformer_tokenizer import TransformerSequenceTokenizer\nfrom hanlp.utils.time_util import CountdownTimer\nfrom hanlp.utils.torch_util import clip_grad_norm, lengths_to_mask, filter_state_dict_safely\nfrom hanlp_common.util import merge_locals_kwargs\n\n\n# noinspection PyAbstractClass\nclass TransformerTaggingModel(nn.Module):\n    def __init__(self,\n                 encoder: TransformerEncoder,\n                 num_labels,\n                 crf=False,\n                 secondary_encoder=None,\n                 extra_embeddings: EmbeddingDim = None) -> None:\n        \"\"\"\n        A shallow tagging model use transformer as decoder.\n        Args:\n            encoder: A pretrained transformer.\n            num_labels: Size of tagset.\n            crf: True to enable CRF.\n            extra_embeddings: Extra embeddings which will be concatenated to the encoder outputs.\n        \"\"\"\n        super().__init__()\n        self.encoder = encoder\n        self.secondary_encoder = secondary_encoder\n        self.extra_embeddings = extra_embeddings\n        # noinspection PyUnresolvedReferences\n        feature_size = encoder.transformer.config.hidden_size\n        if extra_embeddings:\n            feature_size += extra_embeddings.get_output_dim()\n        self.classifier = nn.Linear(feature_size, num_labels)\n        self.crf = CRF(num_labels) if crf else None\n\n    def forward(self, lens: torch.LongTensor, input_ids, token_span, token_type_ids=None, batch=None):\n        mask = lengths_to_mask(lens)\n        x = self.encoder(input_ids, token_span=token_span, token_type_ids=token_type_ids)\n        if self.secondary_encoder:\n            x = self.secondary_encoder(x, mask=mask)\n        if self.extra_embeddings:\n            # noinspection PyCallingNonCallable\n            embed = self.extra_embeddings(batch, mask=mask)\n            x = torch.cat([x, embed], dim=-1)\n        x = self.classifier(x)\n        return x, mask\n\n\nclass TransformerTagger(TransformerComponent, Tagger):\n\n    def __init__(self, **kwargs) -> None:\n        \"\"\"A simple tagger using a linear layer with an optional CRF (:cite:`lafferty2001conditional`) layer for\n        any tagging tasks including PoS tagging and many others.\n\n        Args:\n            **kwargs: Not used.\n        \"\"\"\n        super().__init__(**kwargs)\n        self._tokenizer_transform = None\n        self.model: TransformerTaggingModel = None\n\n    # noinspection PyMethodOverriding\n    def fit_dataloader(self,\n                       trn: DataLoader,\n                       criterion,\n                       optimizer,\n                       metric,\n                       logger: logging.Logger,\n                       history: History,\n                       gradient_accumulation=1,\n                       grad_norm=None,\n                       transformer_grad_norm=None,\n                       teacher: Tagger = None,\n                       kd_criterion=None,\n                       temperature_scheduler=None,\n                       ratio_width=None,\n                       eval_trn=True,\n                       **kwargs):\n        optimizer, scheduler = optimizer\n        if teacher:\n            scheduler, lambda_scheduler = scheduler\n        else:\n            lambda_scheduler = None\n        self.model.train()\n        timer = CountdownTimer(history.num_training_steps(len(trn), gradient_accumulation=gradient_accumulation))\n        total_loss = 0\n        for idx, batch in enumerate(trn):\n            out, mask = self.feed_batch(batch)\n            y = batch['tag_id']\n            loss = self.compute_loss(criterion, out, y, mask)\n            if gradient_accumulation and gradient_accumulation > 1:\n                loss /= gradient_accumulation\n            if teacher:\n                with torch.no_grad():\n                    out_T, _ = teacher.feed_batch(batch)\n                # noinspection PyNoneFunctionAssignment\n                kd_loss = self.compute_distill_loss(kd_criterion, out, out_T, mask, temperature_scheduler)\n                _lambda = float(lambda_scheduler)\n                loss = _lambda * loss + (1 - _lambda) * kd_loss\n            loss.backward()\n            total_loss += loss.item()\n            if eval_trn:\n                prediction = self.decode_output(out, mask, batch)\n                self.update_metrics(metric, out, y, mask, batch, prediction)\n            if history.step(gradient_accumulation):\n                self._step(optimizer, scheduler, grad_norm, transformer_grad_norm, lambda_scheduler)\n                report = f'loss: {total_loss / (idx + 1):.4f} {metric if eval_trn else \"\"}'\n                timer.log(report, logger=logger, ratio_percentage=False, ratio_width=ratio_width)\n            del loss\n            del out\n            del mask\n\n    def _step(self, optimizer, scheduler, grad_norm, transformer_grad_norm, lambda_scheduler):\n        clip_grad_norm(self.model, grad_norm, self.model.encoder.transformer, transformer_grad_norm)\n        optimizer.step()\n        scheduler.step()\n        if lambda_scheduler:\n            lambda_scheduler.step()\n        optimizer.zero_grad()\n\n    def compute_distill_loss(self, kd_criterion, out_S, out_T, mask, temperature_scheduler):\n        logits_S = out_S[mask]\n        logits_T = out_T[mask]\n        temperature = temperature_scheduler(logits_S, logits_T)\n        return kd_criterion(logits_S, logits_T, temperature)\n\n    def build_model(self, training=True, extra_embeddings: Embedding = None, finetune=False, logger=None,\n                    **kwargs) -> torch.nn.Module:\n        model = TransformerTaggingModel(\n            self.build_transformer(training=training),\n            len(self.vocabs.tag),\n            self.config.crf,\n            self.config.get('secondary_encoder', None),\n            extra_embeddings=extra_embeddings.module(self.vocabs) if extra_embeddings else None,\n        )\n        if finetune and self.model:\n            model_state = model.state_dict()\n            load_state = self.model.state_dict()\n            safe_state = filter_state_dict_safely(model_state, load_state)\n            missing_params = model_state.keys() - safe_state.keys()\n            if missing_params:\n                logger.info(f'The following parameters were missing from the checkpoint: '\n                            f'{\", \".join(sorted(missing_params))}.')\n            model.load_state_dict(safe_state, strict=False)\n            n = self.model.classifier.bias.size(0)\n            if model.classifier.bias.size(0) != n:\n                model.classifier.weight.data[:n, :] = self.model.classifier.weight.data[:n, :]\n                model.classifier.bias.data[:n] = self.model.classifier.bias.data[:n]\n        return model\n\n    # noinspection PyMethodOverriding\n    def build_dataloader(self, data, batch_size, shuffle, device, logger: logging.Logger = None,\n                         sampler_builder: SamplerBuilder = None, gradient_accumulation=1,\n                         extra_embeddings: Embedding = None, transform=None, max_seq_len=None, **kwargs) -> DataLoader:\n        if isinstance(data, TransformableDataset):\n            dataset = data\n        else:\n            args = dict((k, self.config.get(k, None)) for k in\n                        ['delimiter', 'max_seq_len', 'sent_delimiter', 'char_level', 'hard_constraint'])\n            dataset = self.build_dataset(data, **args)\n        if self.config.token_key is None:\n            self.config.token_key = next(iter(dataset[0]))\n            logger.info(\n                f'Guess [bold][blue]token_key={self.config.token_key}[/blue][/bold] according to the '\n                f'training dataset: [blue]{dataset}[/blue]')\n        if transform:\n            dataset.append_transform(transform)\n        if extra_embeddings:\n            dataset.append_transform(extra_embeddings.transform(self.vocabs))\n        dataset.append_transform(self.tokenizer_transform)\n        dataset.append_transform(self.last_transform())\n        if not isinstance(data, list):\n            dataset.purge_cache()\n        if self.vocabs.mutable:\n            self.build_vocabs(dataset, logger)\n        if isinstance(data, str) and max_seq_len:\n            token_key = self.config.token_key\n            dataset.prune(lambda x: len(x[token_key]) > max_seq_len, logger)\n        if sampler_builder is not None:\n            sampler = sampler_builder.build([len(x[f'{self.config.token_key}_input_ids']) for x in dataset], shuffle,\n                                            gradient_accumulation=gradient_accumulation if shuffle else 1)\n        else:\n            sampler = None\n        return PadSequenceDataLoader(dataset, batch_size, shuffle, device=device, batch_sampler=sampler)\n\n    def build_dataset(self, data, transform=None, **kwargs):\n        return TSVTaggingDataset(data, transform=transform, **kwargs)\n\n    def last_transform(self):\n        transforms = TransformList(self.vocabs, FieldLength(self.config.token_key))\n        return transforms\n\n    @property\n    def tokenizer_transform(self) -> TransformerSequenceTokenizer:\n        if not self._tokenizer_transform:\n            self._tokenizer_transform = TransformerSequenceTokenizer(self.transformer_tokenizer,\n                                                                     self.config.token_key,\n                                                                     ret_token_span=True)\n        return self._tokenizer_transform\n\n    def build_vocabs(self, trn, logger, **kwargs):\n        if 'tag' not in self.vocabs:\n            self.vocabs.tag = Vocab(pad_token=None, unk_token=None)\n        timer = CountdownTimer(len(trn))\n        max_seq_len = 0\n        token_key = self.config.token_key\n        for each in trn:\n            max_seq_len = max(max_seq_len, len(each[token_key]))\n            timer.log(f'Building vocab [blink][yellow]...[/yellow][/blink] (longest sequence: {max_seq_len})')\n        self.vocabs.tag.set_unk_as_safe_unk()\n        self.vocabs.lock()\n        self.vocabs.summary(logger)\n\n    # noinspection PyMethodOverriding\n    def fit(self,\n            trn_data,\n            dev_data,\n            save_dir,\n            transformer,\n            average_subwords=False,\n            word_dropout: float = 0.2,\n            hidden_dropout=None,\n            layer_dropout=0,\n            scalar_mix=None,\n            mix_embedding: int = 0,\n            grad_norm=5.0,\n            transformer_grad_norm=None,\n            lr=5e-5,\n            transformer_lr=None,\n            transformer_layers=None,\n            gradient_accumulation=1,\n            adam_epsilon=1e-6,\n            weight_decay=0,\n            warmup_steps=0.1,\n            secondary_encoder=None,\n            extra_embeddings: Embedding = None,\n            crf=False,\n            reduction='sum',\n            batch_size=32,\n            sampler_builder: SamplerBuilder = None,\n            epochs=3,\n            patience=5,\n            token_key=None,\n            max_seq_len=None, sent_delimiter=None, char_level=False, hard_constraint=False,\n            transform=None,\n            logger=None,\n            devices: Union[float, int, List[int]] = None,\n            **kwargs):\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    def feed_batch(self, batch: dict):\n        features = [batch[k] for k in self.tokenizer_transform.output_key]\n        if len(features) == 2:\n            input_ids, token_span = features\n        else:\n            input_ids, token_span = features[0], None\n        lens = batch[f'{self.config.token_key}_length']\n        x, mask = self.model(lens, input_ids, token_span, batch.get(f'{self.config.token_key}_token_type_ids'),\n                             batch=batch)\n        return x, mask\n\n    # noinspection PyMethodOverriding\n    def distill(self,\n                teacher: str,\n                trn_data,\n                dev_data,\n                save_dir,\n                transformer: str,\n                batch_size=None,\n                temperature_scheduler='flsw',\n                epochs=None,\n                devices=None,\n                logger=None,\n                seed=None,\n                **kwargs):\n        return super().distill(**merge_locals_kwargs(locals(), kwargs))\n"
  },
  {
    "path": "hanlp/components/taggers/transformers/transformer_tagger_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-29 13:55\nimport math\n\nimport tensorflow as tf\n\nfrom hanlp.common.transform_tf import Transform\nfrom hanlp.components.taggers.tagger_tf import TaggerComponent\nfrom hanlp.components.taggers.transformers.transformer_transform_tf import TransformerTransform\nfrom hanlp.layers.transformers.loader_tf import build_transformer\nfrom hanlp.layers.transformers.utils_tf import build_adamw_optimizer\nfrom hanlp.losses.sparse_categorical_crossentropy import SparseCategoricalCrossentropyOverBatchFirstDim\nfrom hanlp_common.util import merge_locals_kwargs\n\n\nclass TransformerTaggingModel(tf.keras.Model):\n    def __init__(self, transformer: tf.keras.Model, *args, **kwargs):\n        super().__init__(*args, **kwargs)\n        self.transformer = transformer\n\n    def call(self, inputs, training=None, mask=None):\n        return super().call(inputs, training, mask)\n\n\nclass TransformerTaggerTF(TaggerComponent):\n    def __init__(self, transform: TransformerTransform = None) -> None:\n        if transform is None:\n            transform = TransformerTransform()\n        super().__init__(transform)\n        self.transform: TransformerTransform = transform\n\n    def build_model(self, transformer, max_seq_length, **kwargs) -> tf.keras.Model:\n        model, tokenizer = build_transformer(transformer, max_seq_length, len(self.transform.tag_vocab), tagging=True)\n        self.transform.tokenizer = tokenizer\n        return model\n\n    def fit(self, trn_data, dev_data, save_dir,\n            transformer,\n            optimizer='adamw',\n            learning_rate=5e-5,\n            weight_decay_rate=0,\n            epsilon=1e-8,\n            clipnorm=1.0,\n            warmup_steps_ratio=0,\n            use_amp=False,\n            max_seq_length=128,\n            batch_size=32,\n            epochs=3,\n            metrics='accuracy',\n            run_eagerly=False,\n            logger=None,\n            verbose=True,\n            **kwargs):\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    # noinspection PyMethodOverriding\n    def build_optimizer(self, optimizer, learning_rate, epsilon, weight_decay_rate, clipnorm, use_amp, train_steps,\n                        warmup_steps, **kwargs):\n        if optimizer == 'adamw':\n            opt = build_adamw_optimizer(self.config, learning_rate, epsilon, clipnorm, train_steps, use_amp,\n                                        warmup_steps, weight_decay_rate)\n        else:\n            opt = super().build_optimizer(optimizer)\n        return opt\n\n    def build_vocab(self, trn_data, logger):\n        train_examples = super().build_vocab(trn_data, logger)\n        warmup_steps_per_epoch = math.ceil(train_examples * self.config.warmup_steps_ratio / self.config.batch_size)\n        self.config.warmup_steps = warmup_steps_per_epoch * self.config.epochs\n        return train_examples\n\n    def train_loop(self, trn_data, dev_data, epochs, num_examples, train_steps_per_epoch, dev_steps, model, optimizer,\n                   loss, metrics, callbacks, logger, **kwargs):\n        history = self.model.fit(trn_data, epochs=epochs, steps_per_epoch=train_steps_per_epoch,\n                                 validation_data=dev_data,\n                                 callbacks=callbacks,\n                                 validation_steps=dev_steps,\n                                 # mask out padding labels\n                                 # class_weight=dict(\n                                 #     (i, 0 if i == 0 else 1) for i in range(len(self.transform.tag_vocab)))\n                                 )  # type:tf.keras.callbacks.History\n        return history\n\n    def build_loss(self, loss, **kwargs):\n        if not loss:\n            return SparseCategoricalCrossentropyOverBatchFirstDim()\n        return super().build_loss(loss, **kwargs)\n\n    def load_transform(self, save_dir) -> Transform:\n        super().load_transform(save_dir)\n        self.transform.tokenizer = build_transformer(self.config.transformer, self.config.max_seq_length,\n                                                     len(self.transform.tag_vocab), tagging=True, tokenizer_only=True)\n        return self.transform\n"
  },
  {
    "path": "hanlp/components/taggers/transformers/transformer_transform_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-29 15:14\nfrom typing import Union, Tuple, List, Iterable\n\nimport tensorflow as tf\n\nfrom hanlp_common.structure import SerializableDict\nfrom hanlp.common.transform_tf import Transform\nfrom hanlp.common.vocab_tf import VocabTF\nfrom hanlp.layers.transformers.utils_tf import convert_examples_to_features\nfrom hanlp.transform.tsv_tf import TsvTaggingFormat\n\n\nclass TransformerTransform(TsvTaggingFormat, Transform):\n    def __init__(self,\n                 tokenizer=None,\n                 config: SerializableDict = None,\n                 map_x=False, map_y=False, **kwargs) -> None:\n        super().__init__(config, map_x, map_y, **kwargs)\n        self._tokenizer = tokenizer\n        self.tag_vocab: VocabTF = None\n        self.special_token_ids = None\n        self.pad = '[PAD]'\n        self.unk = '[UNK]'\n\n    @property\n    def max_seq_length(self):\n        # -2 for special tokens [CLS] and [SEP]\n        return self.config.get('max_seq_length', 128) - 2\n\n    @property\n    def tokenizer(self):\n        return self._tokenizer\n\n    @tokenizer.setter\n    def tokenizer(self, tokenizer):\n        self._tokenizer = tokenizer\n        vocab = tokenizer._vocab if hasattr(tokenizer, '_vocab') else tokenizer.vocab\n        if self.pad not in vocab:\n            # English albert use <pad> instead of [PAD]\n            self.pad = '<pad>'\n        if self.unk not in vocab:\n            self.unk = '<unk>'\n        self.special_token_ids = tf.constant([vocab[token] for token in [self.pad, '[CLS]', '[SEP]']],\n                                             dtype=tf.int32)\n\n    def fit(self, trn_path: str, **kwargs) -> int:\n        self.tag_vocab = VocabTF(unk_token=None)\n        num_samples = 0\n        for words, tags in self.file_to_inputs(trn_path, gold=True):\n            num_samples += 1\n            self.tag_vocab.update(tags)\n        return num_samples\n\n    def create_types_shapes_values(self) -> Tuple[Tuple, Tuple, Tuple]:\n        max_seq_length = self.config.get('max_seq_length', 128)\n        types = (tf.int32, tf.int32, tf.int32), tf.int32\n        # (input_ids, input_mask, segment_ids), label_ids\n        shapes = ([max_seq_length], [max_seq_length], [max_seq_length]), [None]\n        values = (0, 0, 0), self.tag_vocab.pad_idx\n        return types, shapes, values\n\n    def lock_vocabs(self):\n        super().lock_vocabs()\n\n    def inputs_to_samples(self, inputs, gold=False):\n        max_seq_length = self.config.get('max_seq_length', 128)\n        tokenizer = self._tokenizer\n        xlnet = False\n        roberta = False\n        pad_token = self.pad\n        cls_token = '[CLS]'\n        sep_token = '[SEP]'\n        unk_token = self.unk\n\n        pad_label_idx = self.tag_vocab.pad_idx\n        pad_token = tokenizer.convert_tokens_to_ids([pad_token])[0]\n        for sample in inputs:\n            if gold:\n                words, tags = sample\n            else:\n                words, tags = sample, [self.tag_vocab.idx_to_token[1]] * len(sample)\n\n            input_ids, input_mask, segment_ids, label_ids = convert_examples_to_features(words,\n                                                                                         max_seq_length, tokenizer,\n                                                                                         tags,\n                                                                                         self.tag_vocab.token_to_idx,\n                                                                                         cls_token_at_end=xlnet,\n                                                                                         # xlnet has a cls token at the end\n                                                                                         cls_token=cls_token,\n                                                                                         cls_token_segment_id=2 if xlnet else 0,\n                                                                                         sep_token=sep_token,\n                                                                                         sep_token_extra=roberta,\n                                                                                         # roberta uses an extra separator b/w pairs of sentences, cf. github.com/pytorch/fairseq/commit/1684e166e3da03f5b600dbb7855cb98ddfcd0805\n                                                                                         pad_on_left=xlnet,\n                                                                                         # pad on the left for xlnet\n                                                                                         pad_token_id=pad_token,\n                                                                                         pad_token_segment_id=4 if xlnet else 0,\n                                                                                         pad_token_label_id=pad_label_idx,\n                                                                                         unk_token=unk_token)\n\n            if None in input_ids:\n                print(input_ids)\n            if None in input_mask:\n                print(input_mask)\n            if None in segment_ids:\n                print(input_mask)\n            yield (input_ids, input_mask, segment_ids), label_ids\n\n    def x_to_idx(self, x) -> Union[tf.Tensor, Tuple]:\n        raise NotImplementedError('transformers has its own tagger, not need to convert idx for x')\n\n    def y_to_idx(self, y) -> tf.Tensor:\n        raise NotImplementedError('transformers has its own tagger, not need to convert idx for y')\n\n    def input_is_single_sample(self, input: Union[List[str], List[List[str]]]) -> bool:\n        return isinstance(input[0], str)\n\n    def Y_to_outputs(self, Y: Union[tf.Tensor, Tuple[tf.Tensor]], gold=False, X=None, inputs=None, batch=None,\n                     **kwargs) -> Iterable:\n        assert batch is not None, 'Need the batch to know actual length of Y'\n        label_mask = batch[1]\n        if self.tag_vocab.pad_token:\n            Y[:, :, self.tag_vocab.pad_idx] = float('-inf')\n        Y = tf.argmax(Y, axis=-1)\n        Y = Y[label_mask > 0]\n        tags = [self.tag_vocab.idx_to_token[tid] for tid in Y]\n        offset = 0\n        for words in inputs:\n            yield tags[offset:offset + len(words)]\n            offset += len(words)\n"
  },
  {
    "path": "hanlp/components/taggers/util.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-01 00:31\nfrom typing import List, Tuple\nfrom hanlp.utils.span_util import allowed_transitions\n\n\ndef guess_tagging_scheme(labels: List[str]) -> str:\n    tagset = set(y.split('-')[0] for y in labels)\n    for scheme in \"BIO\", \"BIOUL\", \"BMES\", 'IOBES':\n        if tagset == set(list(scheme)):\n            return scheme\n\n\ndef guess_allowed_transitions(labels) -> List[Tuple[int, int]]:\n    scheme = guess_tagging_scheme(labels)\n    if not scheme:\n        return None\n    if scheme == 'IOBES':\n        scheme = 'BIOUL'\n        labels = [y.replace('E-', 'L-').replace('S-', 'U-') for y in labels]\n    return allowed_transitions(scheme, dict(enumerate(labels)))\n"
  },
  {
    "path": "hanlp/components/tokenizers/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-11 02:48"
  },
  {
    "path": "hanlp/components/tokenizers/multi_criteria_cws_transformer.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-10-21 19:55\nfrom typing import List, Union\n\nfrom hanlp.common.dataset import SamplerBuilder\nfrom hanlp.components.taggers.transformers.transformer_tagger import TransformerTagger\nfrom hanlp.components.tokenizers.transformer import TransformerTaggingTokenizer\nfrom hanlp.datasets.tokenization.loaders.multi_criteria_cws.mcws_dataset import MultiCriteriaTextTokenizingDataset, append_criteria_token\nimport functools\n\nfrom hanlp.metrics.f1 import F1\nfrom hanlp.metrics.mtl import MetricDict\nfrom hanlp_common.util import merge_locals_kwargs\n\n\nclass MultiCriteriaTransformerTaggingTokenizer(TransformerTaggingTokenizer):\n    def __init__(self, **kwargs) -> None:\n        r\"\"\"Transformer based implementation of \"Effective Neural Solution for Multi-Criteria Word Segmentation\"\n        (:cite:`he2019effective`). It uses an artificial token ``[unused_i]`` instead of ``[SEP]`` in the input_ids to\n        mark the i-th segmentation criteria.\n\n        Args:\n            **kwargs: Not used.\n        \"\"\"\n        super().__init__(**kwargs)\n\n    def build_dataset(self, data, **kwargs):\n        return MultiCriteriaTextTokenizingDataset(data, **kwargs)\n\n    def on_config_ready(self, **kwargs):\n        super().on_config_ready(**kwargs)\n        # noinspection PyAttributeOutsideInit\n        if 'criteria_token_map' not in self.config:\n            unused_tokens = [f'[unused{i}]' for i in range(1, 100)]\n            ids = self.transformer_tokenizer.convert_tokens_to_ids(unused_tokens)\n            self.config.unused_tokens = dict((x, ids[i]) for i, x in enumerate(unused_tokens) if\n                                             ids[i] != self.transformer_tokenizer.unk_token_id)\n            self.config.criteria_token_map = dict()\n\n    def last_transform(self):\n        transforms = super().last_transform()\n        transforms.append(functools.partial(append_criteria_token,\n                                            criteria_tokens=self.config.unused_tokens,\n                                            criteria_token_map=self.config.criteria_token_map))\n        return transforms\n\n    def build_vocabs(self, trn, logger, **kwargs):\n        super().build_vocabs(trn, logger, **kwargs)\n        logger.info(f'criteria[{len(self.config.criteria_token_map)}] = {list(self.config.criteria_token_map)}')\n\n    def feed_batch(self, batch: dict):\n        x, mask = TransformerTagger.feed_batch(self, batch)\n        # strip [CLS], [SEP] and [unused_i]\n        return x[:, 1:-2, :], mask\n\n    def build_samples(self, data: List[str], criteria=None, **kwargs):\n        if not criteria:\n            criteria = next(iter(self.config.criteria_token_map.keys()))\n        else:\n            assert criteria in self.config.criteria_token_map, \\\n                f'Unsupported criteria {criteria}. Choose one from {list(self.config.criteria_token_map.keys())}'\n        samples = super().build_samples(data, **kwargs)\n        for sample in samples:\n            sample['criteria'] = criteria\n        return samples\n\n    def build_metric(self, **kwargs):\n        metrics = MetricDict()\n        for criteria in self.config.criteria_token_map:\n            metrics[criteria] = F1()\n        return metrics\n\n    def update_metrics(self, metric, logits, y, mask, batch, prediction):\n        for p, g, c in zip(prediction, self.tag_to_span(batch['tag']), batch['criteria']):\n            pred = set(p)\n            gold = set(g)\n            metric[c](pred, gold)\n\n    def fit(self, trn_data, dev_data, save_dir, transformer, average_subwords=False, word_dropout: float = 0.2,\n            hidden_dropout=None, layer_dropout=0, scalar_mix=None, mix_embedding: int = 0, grad_norm=5.0,\n            transformer_grad_norm=None, lr=5e-5,\n            transformer_lr=None, transformer_layers=None, gradient_accumulation=1,\n            adam_epsilon=1e-8, weight_decay=0, warmup_steps=0.1, crf=False, reduction='sum',\n            batch_size=32, sampler_builder: SamplerBuilder = None, epochs=30, patience=5, token_key=None,\n            tagging_scheme='BMES', delimiter=None,\n            max_seq_len=None, sent_delimiter=None, char_level=False, hard_constraint=False, transform=None, logger=None,\n            devices: Union[float, int, List[int]] = None, **kwargs):\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n"
  },
  {
    "path": "hanlp/components/tokenizers/tok.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-12 13:08\nfrom typing import Any, Callable\n\nfrom hanlp.components.taggers.rnn_tagger import RNNTagger\nfrom hanlp.datasets.tokenization.loaders.chunking_dataset import ChunkingDataset\nfrom hanlp.metrics.chunking.chunking_f1 import ChunkingF1\nfrom hanlp.utils.span_util import bmes_to_words\nfrom hanlp_common.util import merge_locals_kwargs\n\n\nclass RNNTokenizer(RNNTagger):\n\n    def predict(self, sentence: Any, batch_size: int = None, **kwargs):\n        flat = isinstance(sentence, str)\n        if flat:\n            sentence = [sentence]\n        for i, s in enumerate(sentence):\n            sentence[i] = list(s)\n        outputs = RNNTagger.predict(self, sentence, batch_size, **kwargs)\n        if flat:\n            return outputs[0]\n        return outputs\n\n    def predict_data(self, data, batch_size, **kwargs):\n        tags = RNNTagger.predict_data(self, data, batch_size, **kwargs)\n        words = [bmes_to_words(c, t) for c, t in zip(data, tags)]\n        return words\n\n    def build_dataset(self, data, transform=None):\n        dataset = ChunkingDataset(data)\n        if 'transform' in self.config:\n            dataset.append_transform(self.config.transform)\n        if transform:\n            dataset.append_transform(transform)\n        return dataset\n\n    def build_metric(self, **kwargs):\n        return ChunkingF1()\n\n    def update_metrics(self, metric, logits, y, mask, batch):\n        pred = self.decode_output(logits, mask, batch)\n        pred = self._id_to_tags(pred)\n        gold = batch['tag']\n        metric(pred, gold)\n\n    def fit(self, trn_data, dev_data, save_dir, batch_size=50, epochs=100, embed=100, rnn_input=None, rnn_hidden=256,\n            drop=0.5, lr=0.001, patience=10, crf=True, optimizer='adam', token_key='char', tagging_scheme=None,\n            anneal_factor: float = 0.5, anneal_patience=2, devices=None, logger=None,\n            verbose=True, transform: Callable = None, **kwargs):\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n\n"
  },
  {
    "path": "hanlp/components/tokenizers/tok_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-10-27 14:30\nimport logging\nfrom typing import Union, Any, List, Tuple, Iterable\n\nimport tensorflow as tf\n\nfrom hanlp.common.keras_component import KerasComponent\nfrom hanlp.components.taggers.ngram_conv.ngram_conv_tagger import NgramTransform, NgramConvTaggerTF\nfrom hanlp.components.taggers.rnn_tagger_tf import RNNTaggerTF\nfrom hanlp.components.taggers.transformers.transformer_tagger_tf import TransformerTaggerTF\nfrom hanlp.components.taggers.transformers.transformer_transform_tf import TransformerTransform\nfrom hanlp.losses.sparse_categorical_crossentropy import SparseCategoricalCrossentropyOverBatchFirstDim\nfrom hanlp.metrics.chunking.bmes_tf import BMES_F1_TF\nfrom hanlp.transform.tsv_tf import TSVTaggingTransform\nfrom hanlp.transform.txt_tf import TxtFormat, TxtBMESFormat, extract_ngram_features_and_tags, bmes_to_words\nfrom hanlp_common.util import merge_locals_kwargs\n\n\nclass BMESTokenizerTF(KerasComponent):\n\n    def build_metrics(self, metrics, logger: logging.Logger, **kwargs):\n        if metrics == 'f1':\n            self.config.run_eagerly = True\n            return BMES_F1_TF(self.transform.tag_vocab)\n        return super().build_metrics(metrics, logger, **kwargs)\n\n\nclass NgramConvTokenizerTransform(TxtFormat, NgramTransform):\n\n    def inputs_to_samples(self, inputs, gold=False):\n        if self.input_is_single_sample(inputs):\n            inputs = [inputs]\n        for sent in inputs:\n            # bigram_only = false\n            yield extract_ngram_features_and_tags(sent, False, self.config.window_size, gold)\n\n    def input_is_single_sample(self, input: Union[List[str], List[List[str]]]) -> bool:\n        if not input:\n            return True\n        return isinstance(input, str)\n\n    def Y_to_outputs(self, Y: Union[tf.Tensor, Tuple[tf.Tensor]], gold=False, inputs=None, X=None,\n                     **kwargs) -> Iterable:\n        yield from TxtBMESFormat.Y_to_tokens(self, self.tag_vocab, Y, gold, inputs)\n\n\nclass NgramConvTokenizerTF(BMESTokenizerTF, NgramConvTaggerTF):\n\n    def __init__(self) -> None:\n        super().__init__(NgramConvTokenizerTransform())\n\n    def fit(self, trn_data: Any, dev_data: Any, save_dir: str, word_embed: Union[str, int, dict] = 200,\n            ngram_embed: Union[str, int, dict] = 50, embedding_trainable=True, window_size=4, kernel_size=3,\n            filters=(200, 200, 200, 200, 200), dropout_embed=0.2, dropout_hidden=0.2, weight_norm=True,\n            loss: Union[tf.keras.losses.Loss, str] = None,\n            optimizer: Union[str, tf.keras.optimizers.Optimizer] = 'adam', metrics='f1', batch_size=100,\n            epochs=100, logger=None, verbose=True, **kwargs):\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    def evaluate_output_to_file(self, batch, outputs, out):\n        for x, y_pred in zip(self.transform.X_to_inputs(batch[0]),\n                             self.transform.Y_to_outputs(outputs, gold=False)):\n            out.write(self.transform.input_truth_output_to_str(x, None, y_pred))\n            out.write('\\n')\n\n    def build_loss(self, loss, **kwargs):\n        if loss is None:\n            return SparseCategoricalCrossentropyOverBatchFirstDim()\n        return super().build_loss(loss, **kwargs)\n\n\nclass TransformerTokenizerTransform(TxtBMESFormat, TransformerTransform):\n\n    def inputs_to_samples(self, inputs, gold=False):\n        yield from TransformerTransform.inputs_to_samples(self, TxtBMESFormat.inputs_to_samples(self, inputs, gold),\n                                                          True)\n\n    def Y_to_tokens(self, tag_vocab, Y, gold, inputs):\n        if not gold:\n            Y = tf.argmax(Y, axis=2)\n        for text, ys in zip(inputs, Y):\n            tags = [tag_vocab.idx_to_token[int(y)] for y in ys[1:len(text) + 1]]\n            yield bmes_to_words(list(text), tags)\n\n\nclass TransformerTokenizerTF(BMESTokenizerTF, TransformerTaggerTF):\n    def __init__(self, transform: TransformerTokenizerTransform = None) -> None:\n        if transform is None:\n            transform = TransformerTokenizerTransform()\n        super().__init__(transform)\n\n\nclass RNNTokenizerTransform(TxtBMESFormat, TSVTaggingTransform):\n    pass\n\n\nclass RNNTokenizerTF(BMESTokenizerTF, RNNTaggerTF):\n    def __init__(self, transform: RNNTokenizerTransform = None) -> None:\n        if not transform:\n            transform = RNNTokenizerTransform()\n        super().__init__(transform)\n\n    def fit(self, trn_data: str, dev_data: str = None, save_dir: str = None, embeddings=100, embedding_trainable=False,\n            rnn_input_dropout=0.2, rnn_units=100, rnn_output_dropout=0.2, epochs=20, lower=False, max_seq_len=50,\n            logger=None, loss: Union[tf.keras.losses.Loss, str] = None,\n            optimizer: Union[str, tf.keras.optimizers.Optimizer] = 'adam', metrics='f1', batch_size=32,\n            dev_batch_size=32, lr_decay_per_epoch=None, verbose=True, **kwargs):\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n"
  },
  {
    "path": "hanlp/components/tokenizers/transformer.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-11 02:48\nimport functools\nfrom typing import TextIO, Union, List, Dict, Any, Set\n\nimport torch\nfrom hanlp.common.dataset import SamplerBuilder\nfrom hanlp.common.transform import TransformList\nfrom hanlp.components.taggers.transformers.transformer_tagger import TransformerTagger\nfrom hanlp.datasets.tokenization.loaders.txt import TextTokenizingDataset, generate_tags_for_subtokens\nfrom hanlp.metrics.f1 import F1\nfrom hanlp.transform.transformer_tokenizer import TransformerSequenceTokenizer\nfrom hanlp.utils.span_util import bmes_to_spans\nfrom hanlp.utils.string_util import possible_tokenization\nfrom hanlp_common.util import merge_locals_kwargs\nfrom hanlp_trie import DictInterface, TrieDict\nfrom hanlp_trie.dictionary import TupleTrieDict\n\n\nclass TransformerTaggingTokenizer(TransformerTagger):\n\n    def __init__(self, **kwargs) -> None:\n        \"\"\" A tokenizer using transformer tagger for span prediction. It features with 2 high performance dictionaries\n        to handle edge cases in real application.\n\n        - ``dict_force``: High priority dictionary performs longest-prefix-matching on input text which takes higher\n          priority over model predictions.\n        - ``dict_combine``: Low priority dictionary performs longest-prefix-matching on model predictions then\n          combines them.\n\n        .. Note:: For algorithm beginners, longest-prefix-matching is the prerequisite to understand what dictionary can\n            do and what it can't do. The tutorial in `this book <http://nlp.hankcs.com/book.php>`_ can be very helpful.\n\n        It also supports outputting the span of each token by setting ``config.output_spans = True``.\n\n        Args:\n            **kwargs: Predefined config.\n        \"\"\"\n        super().__init__(**kwargs)\n\n    @property\n    def dict_force(self) -> DictInterface:\n        r\"\"\" The high priority dictionary which perform longest-prefix-matching on inputs to split them into two subsets:\n\n        1. spans containing no keywords, which are then fed into tokenizer for further tokenization.\n        2. keywords, which will be outputed without furthur tokenization.\n\n        .. Caution::\n            Longest-prefix-matching **NEVER** guarantee the presence of any keywords. Abuse of\n            ``dict_force`` can lead to low quality results. For more details, refer to\n            `this book <http://nlp.hankcs.com/book.php>`_.\n\n        Examples:\n            >>> tok.dict_force = {'和服', '服务行业'} # Force '和服' and '服务行业' by longest-prefix-matching\n            >>> tok(\"商品和服务行业\")\n                ['商品', '和服', '务行业']\n            >>> tok.dict_force = {'和服务': ['和', '服务']} # Force '和服务' to be tokenized as ['和', '服务']\n            >>> tok(\"商品和服务行业\")\n                ['商品', '和', '服务', '行业']\n        \"\"\"\n        return self.config.get('dict_force', None)\n\n    @dict_force.setter\n    def dict_force(self, dictionary: Union[DictInterface, Union[Dict[str, Any], Set[str]]]):\n        if dictionary is not None and not isinstance(dictionary, DictInterface):\n            dictionary = TrieDict(dictionary)\n        self.config.dict_force = dictionary\n        self.tokenizer_transform.dict = dictionary\n\n    @property\n    def dict_combine(self) -> DictInterface:\n        \"\"\" The low priority dictionary which perform longest-prefix-matching on model predictions and combing them.\n\n        Examples:\n            >>> tok.dict_combine = {'和服', '服务行业'}\n            >>> tok(\"商品和服务行业\") # '和服' is not in the original results ['商品', '和', '服务']. '服务', '行业' are combined to '服务行业'\n                ['商品', '和', '服务行业']\n\n        \"\"\"\n        return self.config.get('dict_combine', None)\n\n    @dict_combine.setter\n    def dict_combine(self, dictionary: Union[DictInterface, Union[Dict[str, Any], Set[str]]]):\n        if dictionary is not None and not isinstance(dictionary, DictInterface):\n            if all(isinstance(k, str) for k in dictionary):\n                dictionary = TrieDict(dictionary)\n            else:\n                _d = set()\n                for k in dictionary:\n                    if isinstance(k, str):\n                        _d.update(possible_tokenization(k))\n                    else:\n                        _d.add(k)\n                dictionary = TupleTrieDict(_d)\n        self.config.dict_combine = dictionary\n\n    def build_metric(self, **kwargs):\n        return F1()\n\n    # noinspection PyMethodOverriding\n    def update_metrics(self, metric, logits, y, mask, batch, prediction):\n        for p, g in zip(prediction, self.tag_to_span(batch['tag'], batch)):\n            pred = set(p)\n            gold = set(g)\n            metric(pred, gold)\n\n    def decode_output(self, logits, mask, batch, model=None):\n        output = super().decode_output(logits, mask, batch, model)\n        if isinstance(output, torch.Tensor):\n            output = output.tolist()\n        prediction = self.id_to_tags(output, [len(x) for x in batch['token']])\n        return self.tag_to_span(prediction, batch)\n\n    def tag_to_span(self, batch_tags, batch: dict):\n        spans = []\n        if 'custom_words' in batch:\n            if self.config.tagging_scheme == 'BMES':\n                S = 'S'\n                M = 'M'\n                E = 'E'\n            else:\n                S = 'B'\n                M = 'I'\n                E = 'I'\n            for tags, custom_words in zip(batch_tags, batch['custom_words']):\n                # [batch['raw_token'][0][x[0]:x[1]] for x in subwords]\n                if custom_words:\n                    for start, end, label in custom_words:\n                        if end - start == 1:\n                            tags[start] = S\n                        else:\n                            tags[start] = 'B'\n                            tags[end - 1] = E\n                            for i in range(start + 1, end - 1):\n                                tags[i] = M\n                        if end < len(tags):\n                            tags[end] = 'B'\n        if 'token_subtoken_offsets_group' not in batch:  # only check prediction on raw text for now\n            # Check cases that a single char gets split into multiple subtokens, e.g., ‥ -> . + .\n            for tags, subtoken_offsets in zip(batch_tags, batch['token_subtoken_offsets']):\n                offset = -1  # BERT produces 'ᄒ', '##ᅡ', '##ᆫ' for '한' and they share the same span\n                prev_tag = None\n                for i, (tag, (b, e)) in enumerate(zip(tags, subtoken_offsets)):\n                    if b < offset:\n                        if prev_tag == 'S':\n                            tags[i - 1] = 'B'\n                        elif prev_tag == 'E':\n                            tags[i - 1] = 'M'\n                        tags[i] = tag = 'M'\n                    offset = e\n                    prev_tag = tag\n        for tags in batch_tags:\n            spans.append(bmes_to_spans(tags))\n        return spans\n\n    def write_prediction(self, prediction, batch, output: TextIO):\n        batch_tokens = self.spans_to_tokens(prediction, batch)\n        for tokens in batch_tokens:\n            output.write(' '.join(tokens))\n            output.write('\\n')\n\n    @property\n    def tokenizer_transform(self):\n        if not self._tokenizer_transform:\n            self._tokenizer_transform = TransformerSequenceTokenizer(self.transformer_tokenizer,\n                                                                     self.config.token_key,\n                                                                     ret_subtokens=True,\n                                                                     ret_subtokens_group=True,\n                                                                     ret_token_span=False,\n                                                                     dict_force=self.dict_force)\n        return self._tokenizer_transform\n\n    def spans_to_tokens(self, spans, batch, rebuild_span=False):\n        batch_tokens = []\n        dict_combine = self.dict_combine\n        raw_text = batch.get('token_', None)  # Use raw text to rebuild the token according to its offset\n        for b, (spans_per_sent, sub_tokens) in enumerate(zip(spans, batch[self.config.token_key])):\n            if raw_text:  # This will restore iPhone X as a whole\n                text = raw_text[b]\n                offsets = batch['token_subtoken_offsets'][b]\n                tokens = [text[offsets[b][0]:offsets[e - 1][-1]] for b, e in spans_per_sent]\n            else:  # This will merge iPhone X into iPhoneX\n                tokens = [''.join(sub_tokens[span[0]:span[1]]) for span in spans_per_sent]\n            if dict_combine:\n                buffer = []\n                offset = 0\n                delta = 0\n                for start, end, label in dict_combine.tokenize(tokens):\n                    if offset < start:\n                        buffer.extend(tokens[offset:start])\n                    if raw_text:\n                        # noinspection PyUnboundLocalVariable\n                        combined = text[offsets[spans_per_sent[start - delta][0]][0]:\n                                        offsets[spans_per_sent[end - delta - 1][1] - 1][1]]\n                    else:\n                        combined = ''.join(tokens[start:end])\n                    buffer.append(combined)\n                    offset = end\n                    if rebuild_span:\n                        start -= delta\n                        end -= delta\n                        combined_span = (spans_per_sent[start][0], spans_per_sent[end - 1][1])\n                        del spans_per_sent[start:end]\n                        delta += end - start - 1\n                        spans_per_sent.insert(start, combined_span)\n                if offset < len(tokens):\n                    buffer.extend(tokens[offset:])\n                tokens = buffer\n            batch_tokens.append(tokens)\n        return batch_tokens\n\n    def generate_prediction_filename(self, tst_data, save_dir):\n        return super().generate_prediction_filename(tst_data.replace('.tsv', '.txt'), save_dir)\n\n    def prediction_to_human(self, pred, vocab, batch, rebuild_span=False):\n        output_spans = self.config.get('output_spans', None)\n        tokens = self.spans_to_tokens(pred, batch, rebuild_span or output_spans)\n        if output_spans:\n            subtoken_spans = batch['token_subtoken_offsets']\n            results = []\n            for toks, offs, subs in zip(tokens, pred, subtoken_spans):\n                r = []\n                results.append(r)\n                for t, (b, e) in zip(toks, offs):\n                    r.append([t, subs[b][0], subs[e - 1][-1]])\n            return results\n        return tokens\n\n    def input_is_flat(self, tokens):\n        return isinstance(tokens, str)\n\n    def build_dataset(self, data, **kwargs):\n        return TextTokenizingDataset(data, **kwargs)\n\n    def last_transform(self):\n        return TransformList(functools.partial(generate_tags_for_subtokens, tagging_scheme=self.config.tagging_scheme),\n                             super().last_transform())\n\n    def fit(self, trn_data, dev_data, save_dir, transformer, average_subwords=False, word_dropout: float = 0.2,\n            hidden_dropout=None, layer_dropout=0, scalar_mix=None, grad_norm=5.0,\n            transformer_grad_norm=None, lr=5e-5, eval_trn=True,\n            transformer_lr=None, transformer_layers=None, gradient_accumulation=1,\n            adam_epsilon=1e-8, weight_decay=0, warmup_steps=0.1, crf=False, reduction='sum',\n            batch_size=32, sampler_builder: SamplerBuilder = None, epochs=30, patience=5, token_key=None,\n            tagging_scheme='BMES', delimiter=None,\n            max_seq_len=None, sent_delimiter=None, char_level=False, hard_constraint=False, transform=None, logger=None,\n            devices: Union[float, int, List[int]] = None, **kwargs):\n        \"\"\"\n\n        Args:\n            trn_data: Training set.\n            dev_data: Development set.\n            save_dir: The directory to save trained component.\n            transformer: An identifier of a pre-trained transformer.\n            average_subwords: ``True`` to average subword representations.\n            word_dropout: Dropout rate to randomly replace a subword with MASK.\n            hidden_dropout: Dropout rate applied to hidden states.\n            layer_dropout: Randomly zero out hidden states of a transformer layer.\n            scalar_mix: Layer attention.\n            grad_norm: Gradient norm for clipping.\n            transformer_grad_norm: Gradient norm for clipping transformer gradient.\n            lr: Learning rate for decoder.\n            transformer_lr: Learning for encoder.\n            transformer_layers: The number of bottom layers to use.\n            gradient_accumulation: Number of batches per update.\n            adam_epsilon: The epsilon to use in Adam.\n            weight_decay: The weight decay to use.\n            warmup_steps: The number of warmup steps.\n            crf: ``True`` to enable CRF (:cite:`lafferty2001conditional`).\n            reduction: The loss reduction used in aggregating losses.\n            batch_size: The number of samples in a batch.\n            sampler_builder: The builder to build sampler, which will override batch_size.\n            epochs: The number of epochs to train.\n            patience: The number of patience epochs before early stopping.\n            token_key: The key to tokens in dataset.\n            tagging_scheme: Either ``BMES`` or ``BI``.\n            delimiter: Delimiter between tokens used to split a line in the corpus.\n            max_seq_len: Sentences longer than ``max_seq_len`` will be split into shorter ones if possible.\n            sent_delimiter: Delimiter between sentences, like period or comma, which indicates a long sentence can\n                be split here.\n            char_level: Whether the sequence length is measured at char level.\n            hard_constraint: Whether to enforce hard length constraint on sentences. If there is no ``sent_delimiter``\n                in a sentence, it will be split at a token anyway.\n            transform: An optional transform to be applied to samples. Usually a character normalization transform is\n                passed in.\n            devices: Devices this component will live on.\n            logger: Any :class:`logging.Logger` instance.\n            seed: Random seed to reproduce this training.\n            **kwargs: Not used.\n\n        Returns:\n            Best metrics on dev set.\n        \"\"\"\n        return super().fit(**merge_locals_kwargs(locals(), kwargs))\n\n    def feed_batch(self, batch: dict):\n        x, mask = super().feed_batch(batch)\n        return x[:, 1:-1, :], mask\n"
  },
  {
    "path": "hanlp/datasets/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-06-13 18:15\n"
  },
  {
    "path": "hanlp/datasets/classification/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-11-10 11:49"
  },
  {
    "path": "hanlp/datasets/classification/sentiment.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-30 21:03\n_ERNIE_TASK_DATA = 'https://ernie.bj.bcebos.com/task_data_zh.tgz#'\n\nCHNSENTICORP_ERNIE_TRAIN = _ERNIE_TASK_DATA + 'chnsenticorp/train.tsv'\nCHNSENTICORP_ERNIE_DEV = _ERNIE_TASK_DATA + 'chnsenticorp/dev.tsv'\nCHNSENTICORP_ERNIE_TEST = _ERNIE_TASK_DATA + 'chnsenticorp/test.tsv'\n"
  },
  {
    "path": "hanlp/datasets/coref/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-07-04 13:39"
  },
  {
    "path": "hanlp/datasets/coref/loaders/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-12-28 19:03\n"
  },
  {
    "path": "hanlp/datasets/coref/loaders/conll12coref.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-07-04 15:33\nimport collections\nimport os\nfrom typing import Union, List, Callable, DefaultDict, Tuple, Optional, Iterator\n\nfrom hanlp.datasets.srl.loaders.ontonotes_loader import Ontonotes as _Ontonotes, OntonotesSentence, \\\n    make_coref_instance\n\nfrom hanlp.common.dataset import TransformableDataset\nfrom hanlp.utils.io_util import TimingFileIterator\n\n\nclass Ontonotes(_Ontonotes):\n    def dataset_document_iterator(self, file_path: str) -> Iterator[List[OntonotesSentence]]:\n        \"\"\"An iterator over CONLL formatted files which yields documents, regardless\n        of the number of document annotations in a particular file. This is useful\n        for conll data which has been preprocessed, such as the preprocessing which\n        takes place for the 2012 CONLL Coreference Resolution task.\n\n        Args:\n          file_path: str: \n\n        Returns:\n\n        \"\"\"\n        open_file = TimingFileIterator(file_path)\n        conll_rows = []\n        document: List[OntonotesSentence] = []\n        for line in open_file:\n            open_file.log(f'Loading {os.path.basename(file_path)}')\n            line = line.strip()\n            if line != \"\" and not line.startswith(\"#\"):\n                # Non-empty line. Collect the annotation.\n                conll_rows.append(line)\n            else:\n                if conll_rows:\n                    document.append(self._conll_rows_to_sentence(conll_rows))\n                    conll_rows = []\n            if line.startswith(\"#end document\"):\n                yield document\n                document = []\n        open_file.erase()\n        if document:\n            # Collect any stragglers or files which might not\n            # have the '#end document' format for the end of the file.\n            yield document\n\n\nclass CONLL12CorefDataset(TransformableDataset):\n\n    def __init__(self, data: Union[str, List], transform: Union[Callable, List] = None, cache=None,\n                 max_span_width=10, max_sentences=None, remove_singleton_clusters=False) -> None:\n        self.remove_singleton_clusters = remove_singleton_clusters\n        self.max_sentences = max_sentences\n        self.max_span_width = max_span_width\n        super().__init__(data, transform, cache)\n\n    def load_file(self, filepath: str):\n        ontonotes_reader = Ontonotes()\n        for sentences in ontonotes_reader.dataset_document_iterator(filepath):\n            clusters: DefaultDict[int, List[Tuple[int, int]]] = collections.defaultdict(list)\n\n            total_tokens = 0\n            for sentence in sentences:\n                for typed_span in sentence.coref_spans:\n                    # Coref annotations are on a _per sentence_\n                    # basis, so we need to adjust them to be relative\n                    # to the length of the document.\n                    span_id, (start, end) = typed_span\n                    clusters[span_id].append((start + total_tokens, end + total_tokens))\n                total_tokens += len(sentence.words)\n\n            yield self.text_to_instance([s.words for s in sentences], list(clusters.values()))\n\n    def text_to_instance(\n            self,  # type: ignore\n            sentences: List[List[str]],\n            gold_clusters: Optional[List[List[Tuple[int, int]]]] = None,\n    ) -> dict:\n        return make_coref_instance(\n            sentences,\n            self.max_span_width,\n            gold_clusters,\n            self.max_sentences,\n            self.remove_singleton_clusters,\n        )\n"
  },
  {
    "path": "hanlp/datasets/eos/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-07-26 18:11"
  },
  {
    "path": "hanlp/datasets/eos/eos.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-07-26 18:12\nimport itertools\nfrom collections import Counter\nfrom typing import Union, List, Callable\n\nfrom hanlp.common.dataset import TransformableDataset\nfrom hanlp.utils.io_util import TimingFileIterator\nfrom hanlp.utils.log_util import cprint\nfrom hanlp.utils.string_util import ispunct\n\n\nclass SentenceBoundaryDetectionDataset(TransformableDataset):\n\n    def __init__(self,\n                 data: Union[str, List],\n                 transform: Union[Callable, List] = None,\n                 cache=None,\n                 append_after_sentence=None,\n                 eos_chars=None,\n                 eos_char_min_freq=200,\n                 eos_char_is_punct=True,\n                 window_size=5,\n                 **kwargs,\n                 ) -> None:\n        \"\"\"Dataset for sentence boundary detection (eos).\n\n        Args:\n            data: The local or remote path to a dataset, or a list of samples where each sample is a dict.\n            transform: Predefined transform(s).\n            cache: ``True`` to enable caching, so that transforms won't be called twice.\n            append_after_sentence: A :class:`str` to insert at the tail of each sentence. For example, English always\n                have a space between sentences.\n            eos_chars: Punctuations at the tail of sentences. If ``None``, then it will built from training samples.\n            eos_char_min_freq: Minimal frequency to keep a eos char.\n            eos_char_is_punct: Limit eos chars to punctuations.\n            window_size: Window size to extract ngram features.\n            kwargs: Not used.\n        \"\"\"\n        self.eos_char_is_punct = eos_char_is_punct\n        self.append_after_sentence = append_after_sentence\n        self.window_size = window_size\n        self.eos_chars = eos_chars\n        self.eos_char_min_freq = eos_char_min_freq\n        super().__init__(data, transform, cache)\n\n    def load_file(self, filepath: str):\n        \"\"\"Load eos corpus.\n\n        Args:\n            filepath: Path to the corpus.\n\n        .. highlight:: bash\n        .. code-block:: bash\n\n            $ head -n 2 ctb8.txt\n            中国经济简讯\n            新华社北京十月二十九日电中国经济简讯\n\n        \"\"\"\n        f = TimingFileIterator(filepath)\n        sents = []\n        eos_offsets = []\n        offset = 0\n        for line in f:\n            if not line.strip():\n                continue\n            line = line.rstrip('\\n')\n            eos_offsets.append(offset + len(line.rstrip()) - 1)\n            offset += len(line)\n            if self.append_after_sentence:\n                line += self.append_after_sentence\n                offset += len(self.append_after_sentence)\n            f.log(line)\n            sents.append(line)\n        f.erase()\n        corpus = list(itertools.chain.from_iterable(sents))\n\n        if self.eos_chars:\n            if not isinstance(self.eos_chars, set):\n                self.eos_chars = set(self.eos_chars)\n        else:\n            eos_chars = Counter()\n            for i in eos_offsets:\n                eos_chars[corpus[i]] += 1\n            self.eos_chars = set(k for (k, v) in eos_chars.most_common() if\n                                 v >= self.eos_char_min_freq and (not self.eos_char_is_punct or ispunct(k)))\n            cprint(f'eos_chars = [yellow]{self.eos_chars}[/yellow]')\n\n        eos_index = 0\n        eos_offsets = [i for i in eos_offsets if corpus[i] in self.eos_chars]\n        window_size = self.window_size\n        for i, c in enumerate(corpus):\n            if c in self.eos_chars:\n                window = corpus[i - window_size: i + window_size + 1]\n                label_id = 1. if eos_offsets[eos_index] == i else 0.\n                if label_id > 0:\n                    eos_index += 1\n                yield {'char': window, 'label_id': label_id}\n        assert eos_index == len(eos_offsets), f'{eos_index} != {len(eos_offsets)}'\n"
  },
  {
    "path": "hanlp/datasets/eos/loaders/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-12-28 19:03\n"
  },
  {
    "path": "hanlp/datasets/eos/loaders/nn_eos.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-24 22:51\n_SETIMES2_EN_HR_SENTENCES_HOME = 'https://schweter.eu/cloud/nn_eos/SETIMES2.en-hr.sentences.tar.xz'\nSETIMES2_EN_HR_HR_SENTENCES_TRAIN = _SETIMES2_EN_HR_SENTENCES_HOME + '#SETIMES2.en-hr.hr.sentences.train'\n'''Training set of SETimes corpus.'''\nSETIMES2_EN_HR_HR_SENTENCES_DEV = _SETIMES2_EN_HR_SENTENCES_HOME + '#SETIMES2.en-hr.hr.sentences.dev'\n'''Dev set of SETimes corpus.'''\nSETIMES2_EN_HR_HR_SENTENCES_TEST = _SETIMES2_EN_HR_SENTENCES_HOME + '#SETIMES2.en-hr.hr.sentences.test'\n'''Test set of SETimes corpus.'''\n_EUROPARL_V7_DE_EN_EN_SENTENCES_HOME = 'http://schweter.eu/cloud/nn_eos/europarl-v7.de-en.en.sentences.tar.xz'\nEUROPARL_V7_DE_EN_EN_SENTENCES_TRAIN = _EUROPARL_V7_DE_EN_EN_SENTENCES_HOME + '#europarl-v7.de-en.en.sentences.train'\n'''Training set of Europarl corpus (:cite:`koehn2005europarl`).'''\nEUROPARL_V7_DE_EN_EN_SENTENCES_DEV = _EUROPARL_V7_DE_EN_EN_SENTENCES_HOME + '#europarl-v7.de-en.en.sentences.dev'\n'''Dev set of Europarl corpus (:cite:`koehn2005europarl`).'''\nEUROPARL_V7_DE_EN_EN_SENTENCES_TEST = _EUROPARL_V7_DE_EN_EN_SENTENCES_HOME + '#europarl-v7.de-en.en.sentences.test'\n'''Test set of Europarl corpus (:cite:`koehn2005europarl`).'''\n"
  },
  {
    "path": "hanlp/datasets/lm/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-05 21:41\n\n_PTB_HOME = 'http://www.fit.vutbr.cz/~imikolov/rnnlm/simple-examples.tgz#'\nPTB_TOKEN_TRAIN = _PTB_HOME + 'data/ptb.train.txt'\nPTB_TOKEN_DEV = _PTB_HOME + 'data/ptb.valid.txt'\nPTB_TOKEN_TEST = _PTB_HOME + 'data/ptb.test.txt'\n\nPTB_CHAR_TRAIN = _PTB_HOME + 'data/ptb.char.train.txt'\nPTB_CHAR_DEV = _PTB_HOME + 'data/ptb.char.valid.txt'\nPTB_CHAR_TEST = _PTB_HOME + 'data/ptb.char.test.txt'\n"
  },
  {
    "path": "hanlp/datasets/lm/loaders/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-12-28 19:04\n"
  },
  {
    "path": "hanlp/datasets/lm/loaders/lm_dataset.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-05 21:42\nimport os\nfrom typing import Union, Callable, List\n\nimport hanlp_common.io\nimport torch\n\nfrom hanlp.common.dataset import TransformSequentialDataset\nfrom hanlp.common.transform import ToChar, WhitespaceTokenizer, AppendEOS, FieldToIndex\nfrom hanlp.common.vocab import Vocab\nfrom hanlp.utils.io_util import file_cache, get_resource, TimingFileIterator\n\n\nclass LanguageModelDataset(TransformSequentialDataset):\n\n    def __init__(self,\n                 data: str,\n                 batch_size,\n                 seq_len,\n                 tokenizer='char',\n                 eos='\\n',\n                 strip=True,\n                 vocab=None,\n                 cache=False,\n                 transform: Union[Callable, List] = None) -> None:\n        self.cache = cache\n        self.eos = eos\n        self.strip = strip\n        super().__init__(transform)\n        if isinstance(tokenizer, str):\n            available_tokenizers = {\n                'char': ToChar('text', 'token'),\n                'whitespace': WhitespaceTokenizer('text', 'token')\n            }\n            assert tokenizer in available_tokenizers, f'{tokenizer} not supported, available options: {available_tokenizers.keys()} '\n            self.append_transform(available_tokenizers[tokenizer])\n\n        if vocab is None:\n            vocab = Vocab()\n            self.training = True\n        else:\n            self.training = vocab.mutable\n        self.append_transform(AppendEOS('token', eos=eos))\n        self.append_transform(FieldToIndex('token', vocab))\n        self.batch_size = batch_size\n        data = get_resource(data)\n        self.data = data\n        self.num_tokens = None\n        self.load_file(data)\n        self._fp = None\n        if isinstance(seq_len, int):\n            self.seq_len = lambda: seq_len\n        else:\n            self.seq_len = seq_len\n\n    @property\n    def vocab(self):\n        return self.transform[-1].vocab\n\n    @property\n    def vocab_path(self):\n        return os.path.splitext(self.data)[0] + '.vocab.json'\n\n    def load_file(self, filepath):\n        cache, valid = file_cache(filepath, not self.cache)\n        if not valid or (self.vocab.mutable and not os.path.isfile(self.vocab_path)):\n            with open(cache, 'wb') as out:\n                tokens, lines = 0, 0\n                f = TimingFileIterator(filepath)\n                for line in f:\n                    if self.strip:\n                        line = line.strip()\n                        if not line:\n                            continue\n                    sample = {'text': line}\n                    sample = self.transform_sample(sample, inplace=True)\n                    for id in sample['token_id']:\n                        out.write((id).to_bytes(4, 'little'))\n                    tokens += len(sample['token_id'])\n                    lines += 1\n                    f.log(f'{tokens // 1000000}M tokens, {lines // 1000000}M lines\\n'\n                          f'{sample[\"token\"][:10]}')\n                f.erase()\n                if self.vocab.mutable:\n                    self.vocab.lock()\n                    hanlp_common.io.save_json(self.vocab_path)\n                self.num_tokens = tokens\n        else:\n            self.num_tokens = int(os.path.getsize(self.filecache) / 4)\n            if self.vocab.mutable:\n                hanlp_common.io.load_json(self.vocab_path)\n\n    def __iter__(self):\n        batch_size = self.batch_size\n        max_seq_len = self.max_seq_len\n        i = 0\n        safety = 2 if self.training else 1\n        with open(self.filecache, 'rb') as fp:\n            while i < max_seq_len - safety:\n                seq_len = self.seq_len()\n                seq_len = min(seq_len, max_seq_len - 1 - i)\n                data = []\n                for j in range(batch_size):\n                    data.append(self._read_chunk(fp, max_seq_len * j + i, seq_len + 1))\n                data = torch.LongTensor(data)\n                data.transpose_(0, 1)\n                data, targets = data[:seq_len, :], data[1:, :]\n                yield data, targets.contiguous().view(-1)\n                i += seq_len\n\n    def estimate_num_batches(self, seq_len=None):\n        if not seq_len:\n            seq_len = self.seq_len()\n        return self.max_seq_len // seq_len\n\n    @property\n    def max_seq_len(self):\n        max_seq_len = self.num_tokens // self.batch_size\n        return max_seq_len\n\n    @staticmethod\n    def _read_chunk(fp, offset, length):\n        data = []\n        fp.seek(offset * 4)\n        for i in range(length):\n            id = int.from_bytes(fp.read(4), 'little')\n            data.append(id)\n        return data\n\n    def _debug_load_cache(self):\n        with open(self.filecache, 'rb') as src:\n            ids = []\n            for i in range(self.num_tokens):\n                id = int.from_bytes(src.read(4), 'little')\n                ids.append(id)\n            return torch.LongTensor(ids)\n\n    @property\n    def filecache(self):\n        return file_cache(self.data)[0]\n"
  },
  {
    "path": "hanlp/datasets/lu/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-12-28 19:08\n"
  },
  {
    "path": "hanlp/datasets/lu/glue.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-11-10 11:47\nfrom hanlp.common.dataset import TableDataset\n\nSTANFORD_SENTIMENT_TREEBANK_2_TRAIN = 'http://file.hankcs.com/corpus/SST2.zip#train.tsv'\nSTANFORD_SENTIMENT_TREEBANK_2_DEV = 'http://file.hankcs.com/corpus/SST2.zip#dev.tsv'\nSTANFORD_SENTIMENT_TREEBANK_2_TEST = 'http://file.hankcs.com/corpus/SST2.zip#test.tsv'\n\nMICROSOFT_RESEARCH_PARAPHRASE_CORPUS_TRAIN = 'http://file.hankcs.com/corpus/mrpc.zip#train.tsv'\nMICROSOFT_RESEARCH_PARAPHRASE_CORPUS_DEV = 'http://file.hankcs.com/corpus/mrpc.zip#dev.tsv'\nMICROSOFT_RESEARCH_PARAPHRASE_CORPUS_TEST = 'http://file.hankcs.com/corpus/mrpc.zip#test.tsv'\n\n\nclass SST2Dataset(TableDataset):\n    pass\n\n\ndef main():\n    dataset = SST2Dataset(STANFORD_SENTIMENT_TREEBANK_2_TEST)\n    print(dataset)\n\n\nif __name__ == '__main__':\n    main()\n"
  },
  {
    "path": "hanlp/datasets/ner/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-06 15:32"
  },
  {
    "path": "hanlp/datasets/ner/conll03.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-06 15:31\n\n\nCONLL03_EN_TRAIN = 'https://file.hankcs.com/corpus/conll03_en_iobes.zip#eng.train.tsv'\n'''Training set of CoNLL03 (:cite:`tjong-kim-sang-de-meulder-2003-introduction`)'''\nCONLL03_EN_DEV = 'https://file.hankcs.com/corpus/conll03_en_iobes.zip#eng.dev.tsv'\n'''Dev set of CoNLL03 (:cite:`tjong-kim-sang-de-meulder-2003-introduction`)'''\nCONLL03_EN_TEST = 'https://file.hankcs.com/corpus/conll03_en_iobes.zip#eng.test.tsv'\n'''Test set of CoNLL03 (:cite:`tjong-kim-sang-de-meulder-2003-introduction`)'''\n"
  },
  {
    "path": "hanlp/datasets/ner/loaders/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-12-28 19:04\n"
  },
  {
    "path": "hanlp/datasets/ner/loaders/json_ner.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-07-21 16:26\nimport json\nimport os\nfrom typing import Union, List, Callable, Dict\n\nfrom hanlp_common.constant import NULL\n\nimport hanlp.utils.span_util\nfrom hanlp.common.dataset import TransformableDataset\nfrom hanlp.utils.io_util import TimingFileIterator, read_tsv_as_sents\n\n\nclass JsonNERDataset(TransformableDataset):\n\n    def __init__(self, data: Union[str, List], transform: Union[Callable, List] = None, cache=None,\n                 generate_idx=None, doc_level_offset=True, tagset=None) -> None:\n        \"\"\"A dataset for ``.jsonlines`` format NER corpora.\n\n        Args:\n            data: The local or remote path to a dataset, or a list of samples where each sample is a dict.\n            transform: Predefined transform(s).\n            cache: ``True`` to enable caching, so that transforms won't be called twice.\n            generate_idx: Create a :const:`~hanlp_common.constants.IDX` field for each sample to store its order in dataset. Useful for prediction when\n                samples are re-ordered by a sampler.\n            doc_level_offset: ``True`` to indicate the offsets in ``jsonlines`` are of document level.\n            tagset: Optional tagset to prune entities outside of this tagset from datasets.\n        \"\"\"\n        self.tagset = tagset\n        self.doc_level_offset = doc_level_offset\n        super().__init__(data, transform, cache, generate_idx)\n\n    def load_file(self, filepath: str):\n        \"\"\"Load ``.jsonlines`` NER corpus. Samples of this corpus can be found using the following scripts.\n\n        .. highlight:: python\n        .. code-block:: python\n\n            import json\n            from hanlp_common.document import Document\n            from hanlp.datasets.srl.ontonotes5.chinese import ONTONOTES5_CONLL12_CHINESE_DEV\n            from hanlp.utils.io_util import get_resource\n\n            with open(get_resource(ONTONOTES5_CONLL12_CHINESE_DEV)) as src:\n                for line in src:\n                    doc = json.loads(line)\n                    print(Document(doc))\n                    break\n\n        Args:\n            filepath: ``.jsonlines`` NER corpus.\n        \"\"\"\n        filename = os.path.basename(filepath)\n        reader = TimingFileIterator(filepath)\n        num_docs, num_sentences = 0, 0\n        for line in reader:\n            line = line.strip()\n            if not line:\n                continue\n            doc = json.loads(line)\n            num_docs += 1\n            num_tokens_in_doc = 0\n            for sentence, ner in zip(doc['sentences'], doc['ner']):\n                if self.doc_level_offset:\n                    ner = [(x[0] - num_tokens_in_doc, x[1] - num_tokens_in_doc, x[2]) for x in ner]\n                else:\n                    ner = [(x[0], x[1], x[2]) for x in ner]\n                if self.tagset:\n                    ner = [x for x in ner if x[2] in self.tagset]\n                    if isinstance(self.tagset, dict):\n                        ner = [(x[0], x[1], self.tagset[x[2]]) for x in ner]\n                deduplicated_srl = []\n                be_set = set()\n                for b, e, l in ner:\n                    be = (b, e)\n                    if be in be_set:\n                        continue\n                    be_set.add(be)\n                    deduplicated_srl.append((b, e, l))\n                yield {\n                    'token': sentence,\n                    'ner': deduplicated_srl\n                }\n                num_sentences += 1\n                num_tokens_in_doc += len(sentence)\n            reader.log(\n                f'{filename} {num_docs} documents, {num_sentences} sentences [blink][yellow]...[/yellow][/blink]')\n        reader.erase()\n\n\ndef convert_conll03_to_json(file_path):\n    dataset = []\n    num_docs = [0]\n\n    def new_doc():\n        doc_key = num_docs[0]\n        num_docs[0] += 1\n        return {\n            'doc_key': doc_key,\n            'sentences': [],\n            'ner': [],\n        }\n\n    doc = new_doc()\n    offset = 0\n    for cells in read_tsv_as_sents(file_path):\n        if cells[0][0] == '-DOCSTART-' and doc['ner']:\n            dataset.append(doc)\n            doc = new_doc()\n            offset = 0\n        sentence = [x[0] for x in cells]\n        ner = [x[-1] for x in cells]\n        ner = hanlp.utils.span_util.iobes_tags_to_spans(ner)\n        adjusted_ner = []\n        for label, (span_start, span_end) in ner:\n            adjusted_ner.append([span_start + offset, span_end + offset, label])\n        doc['sentences'].append(sentence)\n        doc['ner'].append(adjusted_ner)\n        offset += len(sentence)\n    if doc['ner']:\n        dataset.append(doc)\n    output_path = os.path.splitext(file_path)[0] + '.json'\n    with open(output_path, 'w') as out:\n        for each in dataset:\n            json.dump(each, out)\n            out.write('\\n')\n\n\ndef unpack_ner(sample: dict) -> dict:\n    ner: list = sample.get('ner', None)\n    if ner is not None:\n        if ner:\n            sample['begin_offset'], sample['end_offset'], sample['label'] = zip(*ner)\n        else:\n            # It's necessary to create a null label when there is no NER in the sentence for the sake of padding.\n            sample['begin_offset'], sample['end_offset'], sample['label'] = [0], [0], [NULL]\n    return sample\n\n\ndef prune_ner_tagset(sample: dict, tagset: Union[set, Dict[str, str]]):\n    if 'tag' in sample:\n        pruned_tag = []\n        for tag in sample['tag']:\n            cells = tag.split('-', 1)\n            if len(cells) == 2:\n                role, ner_type = cells\n                if ner_type in tagset:\n                    if isinstance(tagset, dict):\n                        tag = role + '-' + tagset[ner_type]\n                else:\n                    tag = 'O'\n            pruned_tag.append(tag)\n        sample['tag'] = pruned_tag\n    return sample\n"
  },
  {
    "path": "hanlp/datasets/ner/loaders/tsv.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-24 23:09\nfrom typing import Union, List, Callable\n\nfrom hanlp.common.dataset import TransformableDataset\nfrom hanlp.utils.io_util import get_resource, generate_words_tags_from_tsv\nfrom hanlp.utils.string_util import split_long_sentence_into\n\n\nclass TSVTaggingDataset(TransformableDataset):\n\n    def __init__(self,\n                 data: Union[str, List],\n                 transform: Union[Callable, List] = None,\n                 cache=None,\n                 generate_idx=None,\n                 max_seq_len=None,\n                 sent_delimiter=None,\n                 char_level=False,\n                 hard_constraint=False,\n                 **kwargs\n                 ) -> None:\n        \"\"\"\n\n        Args:\n            data: The local or remote path to a dataset, or a list of samples where each sample is a dict.\n            transform: Predefined transform(s).\n            cache: ``True`` to enable caching, so that transforms won't be called twice.\n            generate_idx: Create a :const:`~hanlp_common.constants.IDX` field for each sample to store its order in dataset. Useful for prediction when\n                samples are re-ordered by a sampler.\n            max_seq_len: Sentences longer than ``max_seq_len`` will be split into shorter ones if possible.\n            sent_delimiter: Delimiter between sentences, like period or comma, which indicates a long sentence can\n                be split here.\n            char_level: Whether the sequence length is measured at char level, which is never the case for\n                lemmatization.\n            hard_constraint: Whether to enforce hard length constraint on sentences. If there is no ``sent_delimiter``\n                in a sentence, it will be split at a token anyway.\n            kwargs: Not used.\n        \"\"\"\n        self.char_level = char_level\n        self.hard_constraint = hard_constraint\n        self.sent_delimiter = sent_delimiter\n        self.max_seq_len = max_seq_len\n        super().__init__(data, transform, cache, generate_idx)\n\n    def load_file(self, filepath):\n        \"\"\"Load a ``.tsv`` file. A ``.tsv`` file for tagging is defined as a tab separated text file, where non-empty\n        lines have two columns for token and tag respectively, empty lines mark the end of sentences.\n\n        Args:\n            filepath: Path to a ``.tsv`` tagging file.\n\n        .. highlight:: bash\n        .. code-block:: bash\n\n            $ head eng.train.tsv\n            -DOCSTART-      O\n\n            EU      S-ORG\n            rejects O\n            German  S-MISC\n            call    O\n            to      O\n            boycott O\n            British S-MISC\n            lamb    O\n\n        \"\"\"\n        filepath = get_resource(filepath)\n        # idx = 0\n        for words, tags in generate_words_tags_from_tsv(filepath, lower=False):\n            # idx += 1\n            # if idx % 1000 == 0:\n            #     print(f'\\rRead instances {idx // 1000}k', end='')\n            if self.max_seq_len:\n                start = 0\n                for short_sents in split_long_sentence_into(words, self.max_seq_len, self.sent_delimiter,\n                                                            char_level=self.char_level,\n                                                            hard_constraint=self.hard_constraint):\n                    end = start + len(short_sents)\n                    yield {'token': short_sents, 'tag': tags[start:end]}\n                    start = end\n            else:\n                yield {'token': words, 'tag': tags}\n        # print('\\r', end='')\n"
  },
  {
    "path": "hanlp/datasets/ner/msra.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 23:13\n\n_MSRA_NER_HOME = 'http://file.hankcs.com/corpus/msra_ner.zip'\n_MSRA_NER_TOKEN_LEVEL_HOME = 'http://file.hankcs.com/corpus/msra_ner_token_level.zip'\n\nMSRA_NER_CHAR_LEVEL_TRAIN = f'{_MSRA_NER_HOME}#train.tsv'\n'''Training set of MSRA (:cite:`levow-2006-third`) in character level.'''\nMSRA_NER_CHAR_LEVEL_DEV = f'{_MSRA_NER_HOME}#dev.tsv'\n'''Dev set of MSRA (:cite:`levow-2006-third`) in character level.'''\nMSRA_NER_CHAR_LEVEL_TEST = f'{_MSRA_NER_HOME}#test.tsv'\n'''Test set of MSRA (:cite:`levow-2006-third`) in character level.'''\n\nMSRA_NER_TOKEN_LEVEL_IOBES_TRAIN = f'{_MSRA_NER_TOKEN_LEVEL_HOME}#word_level.train.tsv'\n'''Training set of MSRA (:cite:`levow-2006-third`) in token level.'''\nMSRA_NER_TOKEN_LEVEL_IOBES_DEV = f'{_MSRA_NER_TOKEN_LEVEL_HOME}#word_level.dev.tsv'\n'''Dev set of MSRA (:cite:`levow-2006-third`) in token level.'''\nMSRA_NER_TOKEN_LEVEL_IOBES_TEST = f'{_MSRA_NER_TOKEN_LEVEL_HOME}#word_level.test.tsv'\n'''Test set of MSRA (:cite:`levow-2006-third`) in token level.'''\n\nMSRA_NER_TOKEN_LEVEL_SHORT_IOBES_TRAIN = f'{_MSRA_NER_TOKEN_LEVEL_HOME}#word_level.train.short.tsv'\n'''Training set of shorten (<= 128 tokens) MSRA (:cite:`levow-2006-third`) in token level.'''\nMSRA_NER_TOKEN_LEVEL_SHORT_IOBES_DEV = f'{_MSRA_NER_TOKEN_LEVEL_HOME}#word_level.dev.short.tsv'\n'''Dev set of shorten (<= 128 tokens) MSRA (:cite:`levow-2006-third`) in token level.'''\nMSRA_NER_TOKEN_LEVEL_SHORT_IOBES_TEST = f'{_MSRA_NER_TOKEN_LEVEL_HOME}#word_level.test.short.tsv'\n'''Test set of shorten (<= 128 tokens) MSRA (:cite:`levow-2006-third`) in token level.'''\n\nMSRA_NER_TOKEN_LEVEL_SHORT_JSON_TRAIN = f'{_MSRA_NER_TOKEN_LEVEL_HOME}#word_level.train.short.jsonlines'\n'''Training set of shorten (<= 128 tokens) MSRA (:cite:`levow-2006-third`) in token level and jsonlines format.'''\nMSRA_NER_TOKEN_LEVEL_SHORT_JSON_DEV = f'{_MSRA_NER_TOKEN_LEVEL_HOME}#word_level.dev.short.jsonlines'\n'''Dev set of shorten (<= 128 tokens) MSRA (:cite:`levow-2006-third`) in token level and jsonlines format.'''\nMSRA_NER_TOKEN_LEVEL_SHORT_JSON_TEST = f'{_MSRA_NER_TOKEN_LEVEL_HOME}#word_level.test.short.jsonlines'\n'''Test set of shorten (<= 128 tokens) MSRA (:cite:`levow-2006-third`) in token level and jsonlines format.'''\n"
  },
  {
    "path": "hanlp/datasets/ner/resume.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-08 12:10\nfrom hanlp.common.dataset import TransformableDataset\n\nfrom hanlp.utils.io_util import get_resource, generate_words_tags_from_tsv\n\n_RESUME_NER_HOME = 'https://github.com/jiesutd/LatticeLSTM/archive/master.zip#'\n\nRESUME_NER_TRAIN = _RESUME_NER_HOME + 'ResumeNER/train.char.bmes'\n'''Training set of Resume in char level.'''\nRESUME_NER_DEV = _RESUME_NER_HOME + 'ResumeNER/dev.char.bmes'\n'''Dev set of Resume in char level.'''\nRESUME_NER_TEST = _RESUME_NER_HOME + 'ResumeNER/test.char.bmes'\n'''Test set of Resume in char level.'''\n\n"
  },
  {
    "path": "hanlp/datasets/ner/weibo.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-03 23:33\nfrom hanlp.common.dataset import TransformableDataset\n\nfrom hanlp.utils.io_util import get_resource, generate_words_tags_from_tsv\n\n_WEIBO_NER_HOME = 'https://github.com/hltcoe/golden-horse/archive/master.zip#data/'\n\nWEIBO_NER_TRAIN = _WEIBO_NER_HOME + 'weiboNER_2nd_conll.train'\n'''Training set of Weibo in char level.'''\nWEIBO_NER_DEV = _WEIBO_NER_HOME + 'weiboNER_2nd_conll.dev'\n'''Dev set of Weibo in char level.'''\nWEIBO_NER_TEST = _WEIBO_NER_HOME + 'weiboNER_2nd_conll.test'\n'''Test set of Weibo in char level.'''\n"
  },
  {
    "path": "hanlp/datasets/parsing/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 00:51\n"
  },
  {
    "path": "hanlp/datasets/parsing/amr.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-18 17:47\nfrom collections import defaultdict\nfrom copy import copy\nfrom typing import List\n\nimport numpy as np\nimport torch\n\n\nfrom hanlp_common.constant import CLS\nfrom hanlp.common.dataset import TransformableDataset, PadSequenceDataLoader\nfrom hanlp.common.transform import VocabDict\nfrom hanlp.common.vocab import VocabWithFrequency\nfrom hanlp.components.amr.amr_parser.amrio import AMRIO\nfrom hanlp.components.amr.amr_parser.data import END, DUM, list_to_tensor, lists_of_string_to_tensor, NIL, REL\nfrom hanlp.components.amr.amr_parser.transformer import SelfAttentionMask\nfrom hanlp.transform.transformer_tokenizer import TransformerSequenceTokenizer\nfrom hanlp_common.util import merge_list_of_dict\n\n\nclass AbstractMeaningRepresentationDataset(TransformableDataset):\n    def load_file(self, filepath: str):\n        for tok, lem, pos, ner, amr in AMRIO.read(filepath):\n            yield {'token': tok, 'lemma': lem, 'pos': pos, 'ner': ner, 'amr': amr}\n\n\ndef generate_oracle(sample: dict):\n    amr = sample.get('amr', None)\n    if amr:\n        concept, edge, _ = amr.root_centered_sort()\n        sample['concept'] = concept\n        sample['edge'] = edge\n    return sample\n\n\ndef chars_for_tok(sample: dict, max_string_len=20):\n    token = sample['token']\n    chars = []\n    for each in token:\n        each = each[:max_string_len]\n        chars.append([CLS] + list(each) + [END])\n    sample['word_char'] = chars\n    return sample\n\n\ndef append_bos(sample: dict):\n    for key in ['token', 'lemma', 'pos', 'ner']:\n        if key in sample:\n            sample[key] = [CLS] + sample[key]\n    return sample\n\n\ndef get_concepts(sample: dict, vocab: VocabWithFrequency = None, rel_vocab: VocabWithFrequency = None):\n    lem, tok = sample['lemma'], sample['token']\n    cp_seq, mp_seq = [], []\n    new_tokens = set()\n    for le, to in zip(lem, tok):\n        cp_seq.append(le + '_')\n        mp_seq.append(le)\n\n    for cp, mp in zip(cp_seq, mp_seq):\n        if vocab.get_idx(cp) == vocab.unk_idx:\n            new_tokens.add(cp)\n        if vocab.get_idx(mp) == vocab.unk_idx:\n            new_tokens.add(mp)\n    nxt = len(vocab)\n    token2idx, idx2token = dict(), dict()\n    if rel_vocab:\n        new_tokens = rel_vocab.idx_to_token + sorted(new_tokens)\n    else:\n        new_tokens = sorted(new_tokens)\n    for x in new_tokens:\n        token2idx[x] = nxt\n        idx2token[nxt] = x\n        nxt += 1\n    for k, v in zip(['cp_seq', 'mp_seq', 'token2idx', 'idx2token'], [cp_seq, mp_seq, token2idx, idx2token]):\n        sample[k] = v\n    return sample\n\n\ndef batchify(data, vocabs: VocabDict, unk_rate=0., device=None, squeeze=False,\n             tokenizer: TransformerSequenceTokenizer = None, shuffle_sibling=True,\n             levi_graph=False, extra_arc=False, bart=False):\n    rel_vocab: VocabWithFrequency = vocabs.rel\n    _tok = list_to_tensor(data['token'], vocabs['token'], unk_rate=unk_rate) if 'token' in vocabs else None\n    _lem = list_to_tensor(data['lemma'], vocabs['lemma'], unk_rate=unk_rate)\n    _pos = list_to_tensor(data['pos'], vocabs['pos'], unk_rate=unk_rate) if 'pos' in vocabs else None\n    _ner = list_to_tensor(data['ner'], vocabs['ner'], unk_rate=unk_rate) if 'ner' in vocabs else None\n    _word_char = lists_of_string_to_tensor(data['token'], vocabs['word_char']) if 'word_char' in vocabs else None\n\n    local_token2idx = data['token2idx']\n    local_idx2token = data['idx2token']\n    _cp_seq = list_to_tensor(data['cp_seq'], vocabs['predictable_concept'], local_token2idx)\n    _mp_seq = list_to_tensor(data['mp_seq'], vocabs['predictable_concept'], local_token2idx)\n\n    ret = copy(data)\n    if 'amr' in data:\n        concept, edge = [], []\n        for amr in data['amr']:\n            if levi_graph == 'kahn':\n                concept_i, edge_i = amr.to_levi(rel_vocab.get_frequency, shuffle=shuffle_sibling)\n            else:\n                concept_i, edge_i, _ = amr.root_centered_sort(rel_vocab.get_frequency, shuffle=shuffle_sibling)\n            concept.append(concept_i)\n            edge.append(edge_i)\n        if levi_graph is True:\n            concept_with_rel, edge_with_rel = levi_amr(concept, edge, extra_arc=extra_arc)\n            concept = concept_with_rel\n            edge = edge_with_rel\n\n        augmented_concept = [[DUM] + x + [END] for x in concept]\n\n        _concept_in = list_to_tensor(augmented_concept, vocabs.get('concept_and_rel', vocabs['concept']),\n                                     unk_rate=unk_rate)[:-1]\n        _concept_char_in = lists_of_string_to_tensor(augmented_concept, vocabs['concept_char'])[:-1]\n        _concept_out = list_to_tensor(augmented_concept, vocabs['predictable_concept'], local_token2idx)[1:]\n\n        out_conc_len, bsz = _concept_out.shape\n        _rel = np.full((1 + out_conc_len, bsz, out_conc_len), rel_vocab.pad_idx)\n        # v: [<dummy>, concept_0, ..., concept_l, ..., concept_{n-1}, <end>] u: [<dummy>, concept_0, ..., concept_l, ..., concept_{n-1}]\n\n        for bidx, (x, y) in enumerate(zip(edge, concept)):\n            for l, _ in enumerate(y):\n                if l > 0:\n                    # l=1 => pos=l+1=2\n                    _rel[l + 1, bidx, 1:l + 1] = rel_vocab.get_idx(NIL)\n            for v, u, r in x:\n                if levi_graph:\n                    r = 1\n                else:\n                    r = rel_vocab.get_idx(r)\n                assert v > u, 'Invalid typological order'\n                _rel[v + 1, bidx, u + 1] = r\n        ret.update(\n            {'concept_in': _concept_in, 'concept_char_in': _concept_char_in, 'concept_out': _concept_out, 'rel': _rel})\n    else:\n        augmented_concept = None\n\n    token_length = ret.get('token_length', None)\n    if token_length is not None and not isinstance(token_length, torch.Tensor):\n        ret['token_length'] = torch.tensor(token_length, dtype=torch.long, device=device if (\n                    isinstance(device, torch.device) or device >= 0) else 'cpu:0')\n    ret.update({'lem': _lem, 'tok': _tok, 'pos': _pos, 'ner': _ner, 'word_char': _word_char,\n                'copy_seq': np.stack([_cp_seq, _mp_seq], -1), 'local_token2idx': local_token2idx,\n                'local_idx2token': local_idx2token})\n    if squeeze:\n        token_field = make_batch_for_squeeze(data, augmented_concept, tokenizer, device, ret)\n    else:\n        token_field = 'token'\n    subtoken_to_tensor(token_field, ret)\n    if bart:\n        make_batch_for_bart(augmented_concept, ret, tokenizer, device)\n    move_dict_to_device(ret, device)\n\n    return ret\n\n\ndef make_batch_for_bart(augmented_concept, ret, tokenizer, device, training=True):\n    token_field = 'concept'\n    tokenizer = TransformerSequenceTokenizer(tokenizer.tokenizer, token_field, cls_is_bos=True, sep_is_eos=None)\n    encodings = [tokenizer({token_field: x[:-1] if training else x}) for x in augmented_concept]\n    ret.update(merge_list_of_dict(encodings))\n    decoder_mask = []\n    max_seq_len = len(max(ret['concept_input_ids'], key=len))\n    last_concept_offset = []\n    for spans, concepts in zip(ret['concept_token_span'], augmented_concept):\n        mask = ~SelfAttentionMask.get_mask(max_seq_len, device, ret_parameter=False)\n        for group in spans:\n            for i in range(len(group)):\n                for j in range(i + 1, len(group)):\n                    mask[group[i], group[j]] = True\n        decoder_mask.append(mask)\n        last_concept_offset.append(len(concepts) - 1)\n    ret['decoder_mask'] = torch.stack(decoder_mask)\n    if not training:\n        ret['last_concept_offset'] = torch.tensor(last_concept_offset, device=device, dtype=torch.long)\n    subtoken_to_tensor(token_field, ret)\n\n\ndef levi_amr(concept, edge, extra_arc=False):\n    concept_with_rel = []\n    edge_with_rel = []\n    for bidx, (edge_i, concept_i) in enumerate(zip(edge, concept)):\n        concept_i, edge_i = linearize(concept_i, edge_i, NIL, prefix=REL, extra_arc=extra_arc)\n        # This is a undirectional graph, so we can safely reverse edge\n        edge_i = [tuple(reversed(sorted(x[:2]))) + x[2:] for x in edge_i]\n        concept_with_rel.append(concept_i)\n        edge_with_rel.append(edge_i)\n    return concept_with_rel, edge_with_rel\n\n\ndef move_dict_to_device(ret, device):\n    if device == -1:\n        device = 'cpu:0'\n    for k, v in ret.items():\n        if isinstance(v, np.ndarray):\n            ret[k] = torch.tensor(v, device=device).contiguous()\n        elif isinstance(v, torch.Tensor):\n            ret[k] = v.to(device).contiguous()\n\n\ndef subtoken_to_tensor(token_field, ret):\n    token_input_ids = PadSequenceDataLoader.pad_data(ret[f'{token_field}_input_ids'], 0, torch.long)\n    token_token_span = PadSequenceDataLoader.pad_data(ret[f'{token_field}_token_span'], 0, torch.long)\n    ret.update({f'{token_field}_token_span': token_token_span, f'{token_field}_input_ids': token_input_ids})\n\n\ndef make_batch_for_squeeze(data, augmented_concept, tokenizer, device, ret):\n    token_field = 'token_and_concept'\n    attention_mask = []\n    token_and_concept = [t + [tokenizer.sep_token] + c for t, c in zip(data['token'], augmented_concept)]\n    encodings = [tokenizer({token_field: x}) for x in token_and_concept]\n    ret.update(merge_list_of_dict(encodings))\n    max_input_len = len(max(ret[f'{token_field}_input_ids'], key=len))\n    concept_mask = []\n    token_mask = []\n    token_type_ids = []\n    snt_len = []\n    last_concept_offset = []\n    for tokens, concepts, input_ids, spans in zip(data['token'], augmented_concept,\n                                                  ret['token_and_concept_input_ids'],\n                                                  ret['token_and_concept_token_span']):\n        raw_sent_len = len(tokens) + 1  # for [SEP]\n        raw_concept_len = len(concepts)\n        if concepts[-1] == END:\n            concept_mask.append([False] * raw_sent_len + [True] * (raw_concept_len - 1) + [False])  # skip END concept\n        else:\n            concept_mask.append([False] * raw_sent_len + [True] * raw_concept_len)\n        token_mask.append([False] + [True] * (raw_sent_len - 2) + [False] * (raw_concept_len + 1))\n        assert len(concept_mask) == len(token_mask)\n        snt_len.append(raw_sent_len - 2)  # skip [CLS] and [SEP]\n        sent_len = input_ids.index(tokenizer.tokenizer.sep_token_id) + 1\n        concept_len = len(input_ids) - sent_len\n        mask = torch.zeros((max_input_len, max_input_len), dtype=torch.bool)\n        mask[:sent_len + concept_len, :sent_len] = True\n        bottom_right = ~SelfAttentionMask.get_mask(concept_len, device, ret_parameter=False)\n        mask[sent_len:sent_len + concept_len, sent_len:sent_len + concept_len] = bottom_right\n        for group in spans:\n            if group[0] >= sent_len:\n                for i in range(len(group)):\n                    for j in range(i + 1, len(group)):\n                        mask[group[i], group[j]] = True\n        attention_mask.append(mask)\n        _token_type_ids = [0] * sent_len + [1] * concept_len\n        token_type_ids.append(_token_type_ids)\n        assert len(input_ids) == len(_token_type_ids)\n        last_concept_offset.append(raw_concept_len - 1)\n    ret['attention_mask'] = torch.stack(attention_mask)\n    ret['concept_mask'] = PadSequenceDataLoader.pad_data(concept_mask, 0, torch.bool)\n    ret['token_mask'] = PadSequenceDataLoader.pad_data(token_mask, 0, torch.bool)\n    ret['token_type_ids'] = PadSequenceDataLoader.pad_data(token_type_ids, 0, torch.long)\n    ret['snt_len'] = PadSequenceDataLoader.pad_data(snt_len, 0, torch.long)\n    ret['last_concept_offset'] = PadSequenceDataLoader.pad_data(last_concept_offset, 0, torch.long)\n    return token_field\n\n\ndef linearize(concept: List, edge: List, label='', prefix=REL, extra_arc=False):\n    vur = defaultdict(dict)\n    for v, u, r in edge:\n        vur[v][u] = r\n    concept_with_rel = []\n    edge_with_rel = []\n    reorder = dict()\n    for v, c in enumerate(concept):\n        reorder[v] = len(concept_with_rel)\n        concept_with_rel.append(c)\n        ur = vur[v]\n        for u, r in ur.items():\n            if u < v:\n                concept_with_rel.append(prefix + r)\n    for k, v in reorder.items():\n        assert concept[k] == concept_with_rel[v]\n    for v, c in enumerate(concept):\n        ur = vur[v]\n        for i, (u, r) in enumerate(ur.items()):\n            if u < v:\n                _v = reorder[v]\n                _u = reorder[u]\n                _m = _v + i + 1\n                edge_with_rel.append((_v, _m, label))\n                edge_with_rel.append((_m, _u, label))\n                if extra_arc:\n                    edge_with_rel.append((_v, _u, label))\n    return concept_with_rel, edge_with_rel\n\n\ndef unlinearize(concept: List, edge: List, prefix=REL, extra_arc=False):\n    real_concept, reorder = separate_concept_rel(concept, prefix)\n    if extra_arc:\n        edge = [x for x in edge if concept[x[0]].startswith(REL) or concept[x[1]].startswith(REL)]\n    real_edge = []\n    for f, b in zip(edge[::2], edge[1::2]):\n        if b[1] not in reorder:\n            continue\n        u = reorder[b[1]]\n        if f[0] not in reorder:\n            continue\n        v = reorder[f[0]]\n        r = concept[f[1]][len(prefix):]\n        real_edge.append((v, u, r))\n    return real_concept, real_edge\n\n\ndef separate_concept_rel(concept, prefix=REL):\n    reorder = dict()\n    real_concept = []\n    for i, c in enumerate(concept):\n        if not c.startswith(prefix):\n            reorder[i] = len(real_concept)\n            real_concept.append(c)\n    return real_concept, reorder\n\n\ndef remove_unconnected_components(concept: List, edge: List):\n    from scipy.sparse import csr_matrix\n    from scipy.sparse.csgraph._traversal import connected_components\n    row = np.array([x[0] for x in edge], dtype=np.int)\n    col = np.array([x[1] for x in edge], dtype=np.int)\n    data = np.ones(len(row), dtype=np.int)\n    graph = csr_matrix((data, (row, col)), shape=(len(concept), len(concept)))\n    n_components, labels = connected_components(csgraph=graph, directed=True, return_labels=True)\n    if n_components > 1:\n        unique, counts = np.unique(labels, return_counts=True)\n        largest_component = max(zip(counts, unique))[-1]\n        connected_nodes = set(np.where(labels == largest_component)[0])\n        reorder = dict()\n        good_concept = []\n        good_edge = []\n        for i, c in enumerate(concept):\n            if i in connected_nodes:\n                reorder[i] = len(good_concept)\n                good_concept.append(c)\n        for v, u, r in edge:\n            if v in connected_nodes and u in connected_nodes:\n                good_edge.append((reorder[v], reorder[u], r))\n        concept, edge = good_concept, good_edge\n    return concept, edge\n\n\ndef largest_connected_component(triples: List):\n    node_to_id = dict()\n    concept = []\n    edge = []\n    for u, r, v in triples:\n        if u not in node_to_id:\n            node_to_id[u] = len(node_to_id)\n            concept.append(u)\n        if v not in node_to_id:\n            node_to_id[v] = len(node_to_id)\n            concept.append(v)\n        edge.append((node_to_id[u], node_to_id[v], r))\n    concept, edge = remove_unconnected_components(concept, edge)\n    return concept, edge\n\n\ndef to_triples(concept: List, edge: List):\n    return [(concept[u], r, concept[v]) for u, v, r in edge]\n\n\ndef reverse_edge_for_levi_bfs(concept, edge):\n    for v, u, r in edge:\n        if r == '_reverse_':\n            for x in v, u:\n                if concept[x].startswith(REL) and not concept[x].endswith('_reverse_'):\n                    concept[x] += '_reverse_'\n\n\ndef un_kahn(concept, edge):\n    # (['want', 'rel=ARG1', 'rel=ARG0', 'believe', 'rel=ARG1', 'rel=ARG0', 'boy', 'girl'],\n    # [(0, 1, 0.9999417066574097), (0, 2, 0.9999995231628418), (1, 3, 0.9999992847442627), (3, 4, 1.0), (3, 5, 0.9999996423721313), (2, 6, 0.9996106624603271), (4, 6, 0.9999767541885376), (5, 7, 0.9999860525131226)])\n    real_concept, reorder = separate_concept_rel(concept)\n    tri_edge = dict()\n    for m, (a, b, p1) in enumerate(edge):\n        if concept[a].startswith(REL):\n            continue\n        for n, (c, d, p2) in enumerate(edge[m + 1:]):\n            if b == c:\n                key = (a, d)\n                _, p = tri_edge.get(key, (None, 0))\n                if p1 * p2 > p:\n                    tri_edge[key] = (b, p1 * p2)\n    real_edge = []\n    for (a, d), (r, p) in tri_edge.items():\n        u = reorder[a]\n        r = concept[r][len(REL):]\n        v = reorder[d]\n        real_edge.append((v, u, r))\n    return real_concept, real_edge\n"
  },
  {
    "path": "hanlp/datasets/parsing/ctb5.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 18:44\nfrom hanlp_common.constant import HANLP_URL\n\n_CTB_HOME = HANLP_URL + 'embeddings/SUDA-LA-CIP_20200109_021624.zip#'\n\n_CTB5_DEP_HOME = _CTB_HOME + 'BPNN/data/ctb5/'\n\nCTB5_DEP_TRAIN = _CTB5_DEP_HOME + 'train.conll'\n'''Training set for ctb5 dependency parsing.'''\nCTB5_DEP_DEV = _CTB5_DEP_HOME + 'dev.conll'\n'''Dev set for ctb5 dependency parsing.'''\nCTB5_DEP_TEST = _CTB5_DEP_HOME + 'test.conll'\n'''Test set for ctb5 dependency parsing.'''\n\nCIP_W2V_100_CN = _CTB_HOME + 'BPNN/data/embed.txt'\n"
  },
  {
    "path": "hanlp/datasets/parsing/ctb7.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 18:44\nfrom hanlp.datasets.parsing.ctb5 import _CTB_HOME\n\n_CTB7_HOME = _CTB_HOME + 'BPNN/data/ctb7/'\n\nCTB7_DEP_TRAIN = _CTB7_HOME + 'train.conll'\n'''Training set for ctb7 dependency parsing.'''\nCTB7_DEP_DEV = _CTB7_HOME + 'dev.conll'\n'''Dev set for ctb7 dependency parsing.'''\nCTB7_DEP_TEST = _CTB7_HOME + 'test.conll'\n'''Test set for ctb7 dependency parsing.'''\n"
  },
  {
    "path": "hanlp/datasets/parsing/ctb8.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-10-14 20:54\n\nfrom hanlp.datasets.parsing.loaders._ctb_utils import make_ctb\n\n_CTB8_HOME = 'https://wakespace.lib.wfu.edu/bitstream/handle/10339/39379/LDC2013T21.tgz#data/'\n\nCTB8_CWS_TRAIN = _CTB8_HOME + 'tasks/cws/train.txt'\n'''Training set for ctb8 Chinese word segmentation.'''\nCTB8_CWS_DEV = _CTB8_HOME + 'tasks/cws/dev.txt'\n'''Dev set for ctb8 Chinese word segmentation.'''\nCTB8_CWS_TEST = _CTB8_HOME + 'tasks/cws/test.txt'\n'''Test set for ctb8 Chinese word segmentation.'''\n\nCTB8_POS_TRAIN = _CTB8_HOME + 'tasks/pos/train.tsv'\n'''Training set for ctb8 PoS tagging.'''\nCTB8_POS_DEV = _CTB8_HOME + 'tasks/pos/dev.tsv'\n'''Dev set for ctb8 PoS tagging.'''\nCTB8_POS_TEST = _CTB8_HOME + 'tasks/pos/test.tsv'\n'''Test set for ctb8 PoS tagging.'''\n\nCTB8_BRACKET_LINE_TRAIN = _CTB8_HOME + 'tasks/par/train.txt'\n'''Training set for ctb8 constituency parsing with empty categories.'''\nCTB8_BRACKET_LINE_DEV = _CTB8_HOME + 'tasks/par/dev.txt'\n'''Dev set for ctb8 constituency parsing with empty categories.'''\nCTB8_BRACKET_LINE_TEST = _CTB8_HOME + 'tasks/par/test.txt'\n'''Test set for ctb8 constituency parsing with empty categories.'''\n\nCTB8_BRACKET_LINE_NOEC_TRAIN = _CTB8_HOME + 'tasks/par/train.noempty.txt'\n'''Training set for ctb8 constituency parsing without empty categories.'''\nCTB8_BRACKET_LINE_NOEC_DEV = _CTB8_HOME + 'tasks/par/dev.noempty.txt'\n'''Dev set for ctb8 constituency parsing without empty categories.'''\nCTB8_BRACKET_LINE_NOEC_TEST = _CTB8_HOME + 'tasks/par/test.noempty.txt'\n'''Test set for ctb8 constituency parsing without empty categories.'''\n\nCTB8_SD330_TRAIN = _CTB8_HOME + 'tasks/dep/train.conllx'\n'''Training set for ctb8 in Stanford Dependencies 3.3.0 standard.'''\nCTB8_SD330_DEV = _CTB8_HOME + 'tasks/dep/dev.conllx'\n'''Dev set for ctb8 in Stanford Dependencies 3.3.0 standard.'''\nCTB8_SD330_TEST = _CTB8_HOME + 'tasks/dep/test.conllx'\n'''Test set for ctb8 in Stanford Dependencies 3.3.0 standard.'''\n\nmake_ctb(_CTB8_HOME)\n"
  },
  {
    "path": "hanlp/datasets/parsing/ctb9.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-10-14 20:54\nfrom urllib.error import HTTPError\n\nfrom hanlp.datasets.parsing.loaders._ctb_utils import make_ctb\nfrom hanlp.utils.io_util import get_resource, path_from_url\n\n_CTB9_HOME = 'https://catalog.ldc.upenn.edu/LDC2016T13/ctb9.0_LDC2016T13.tgz#data/'\n\nCTB9_CWS_TRAIN = _CTB9_HOME + 'tasks/cws/train.txt'\n'''Training set for ctb9 Chinese word segmentation.'''\nCTB9_CWS_DEV = _CTB9_HOME + 'tasks/cws/dev.txt'\n'''Dev set for ctb9 Chinese word segmentation.'''\nCTB9_CWS_TEST = _CTB9_HOME + 'tasks/cws/test.txt'\n'''Test set for ctb9 Chinese word segmentation.'''\n\nCTB9_POS_TRAIN = _CTB9_HOME + 'tasks/pos/train.tsv'\n'''Training set for ctb9 PoS tagging.'''\nCTB9_POS_DEV = _CTB9_HOME + 'tasks/pos/dev.tsv'\n'''Dev set for ctb9 PoS tagging.'''\nCTB9_POS_TEST = _CTB9_HOME + 'tasks/pos/test.tsv'\n'''Test set for ctb9 PoS tagging.'''\n\nCTB9_BRACKET_LINE_TRAIN = _CTB9_HOME + 'tasks/par/train.txt'\n'''Training set for ctb9 constituency parsing with empty categories.'''\nCTB9_BRACKET_LINE_DEV = _CTB9_HOME + 'tasks/par/dev.txt'\n'''Dev set for ctb9 constituency parsing with empty categories.'''\nCTB9_BRACKET_LINE_TEST = _CTB9_HOME + 'tasks/par/test.txt'\n'''Test set for ctb9 constituency parsing with empty categories.'''\n\nCTB9_BRACKET_LINE_NOEC_TRAIN = _CTB9_HOME + 'tasks/par/train.noempty.txt'\n'''Training set for ctb9 constituency parsing without empty categories.'''\nCTB9_BRACKET_LINE_NOEC_DEV = _CTB9_HOME + 'tasks/par/dev.noempty.txt'\n'''Dev set for ctb9 constituency parsing without empty categories.'''\nCTB9_BRACKET_LINE_NOEC_TEST = _CTB9_HOME + 'tasks/par/test.noempty.txt'\n'''Test set for ctb9 constituency parsing without empty categories.'''\n\nCTB9_SD330_TRAIN = _CTB9_HOME + 'tasks/dep/train.conllx'\n'''Training set for ctb9 in Stanford Dependencies 3.3.0 standard.'''\nCTB9_SD330_DEV = _CTB9_HOME + 'tasks/dep/dev.conllx'\n'''Dev set for ctb9 in Stanford Dependencies 3.3.0 standard.'''\nCTB9_SD330_TEST = _CTB9_HOME + 'tasks/dep/test.conllx'\n'''Test set for ctb9 in Stanford Dependencies 3.3.0 standard.'''\n\ntry:\n    get_resource(_CTB9_HOME)\nexcept HTTPError:\n    raise FileNotFoundError(\n        'Chinese Treebank 9.0 is a copyright dataset owned by LDC which we cannot re-distribute. '\n        f'Please apply for a licence from LDC (https://catalog.ldc.upenn.edu/LDC2016T13) '\n        f'then download it to {path_from_url(_CTB9_HOME)}'\n    )\n\nmake_ctb(_CTB9_HOME)\n"
  },
  {
    "path": "hanlp/datasets/parsing/loaders/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-12-28 19:04\n"
  },
  {
    "path": "hanlp/datasets/parsing/loaders/_ctb_utils.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-11-25 16:14\nimport os\nimport shutil\nimport sys\nfrom collections import defaultdict\nfrom os import listdir\nfrom os.path import join, isfile\nfrom typing import List\n\nfrom phrasetree.tree import Tree\n\nfrom hanlp.components.parsers.conll import read_conll\nfrom hanlp.utils.io_util import get_resource, get_exitcode_stdout_stderr, read_tsv_as_sents, run_cmd, pushd\nfrom hanlp.utils.log_util import cprint\nfrom hanlp.utils.time_util import CountdownTimer\n\n\n# See Shao et al., 2017\n# CTB9_ACADEMIA_SPLITS = {\n#     'train': '''\n# 0044-0143, 0170-0270, 0400-0899,\n# 1001-1017, 1019, 1021-1035, 1037-\n# 1043, 1045-1059, 1062-1071, 1073-\n# 1117, 1120-1131, 1133-1140, 1143-\n# 1147, 1149-1151, 2000-2915, 4051-\n# 4099, 4112-4180, 4198-4368, 5000-\n# 5446, 6000-6560, 7000-7013\n#     ''',\n#     'dev': '''\n# 0301-0326, 2916-3030, 4100-4106,\n# 4181-4189, 4369-4390, 5447-5492,\n# 6561-6630, 7013-7014\n#     ''',\n#     'test': '''\n# 0001-0043, 0144-0169, 0271-0301,\n# 0900-0931, 1018, 1020, 1036, 1044,\n# 1060, 1061, 1072, 1118, 1119, 1132,\n# 1141, 1142, 1148, 3031-3145, 4107-\n# 4111, 4190-4197, 4391-4411, 5493-\n# 5558, 6631-6700, 7015-7017\n#     '''\n# }\n#\n#\n# def _make_splits(splits: Dict[str, str]):\n#     total = set()\n#     for part, text in list(splits.items()):\n#         if not isinstance(text, str):\n#             continue\n#         lines = text.replace('\\n', '').split()\n#         cids = set()\n#         for line in lines:\n#             for each in line.split(','):\n#                 each = each.strip()\n#                 if not each:\n#                     continue\n#                 if '-' in each:\n#                     start, end = each.split('-')\n#                     start, end = map(lambda x: int(x), [start, end])\n#                     cids.update(range(start, end + 1))\n#                     # cids.update(map(lambda x: f'{x:04d}', range(start, end)))\n#                 else:\n#                     cids.add(int(each))\n#         cids = set(f'{x:04d}' for x in cids)\n#         assert len(cids & total) == 0, f'Overlap found in {part}'\n#         splits[part] = cids\n#\n#     return splits\n#\n#\n# _make_splits(CTB9_ACADEMIA_SPLITS)\n\n\ndef convert_to_dependency(src, dst, language='zh', version='3.3.0', conllx=True, ud=False):\n    cprint(f'Converting {os.path.basename(src)} to {os.path.basename(dst)} using Stanford Parser Version {version}. '\n           f'It might take a while [blink][yellow]...[/yellow][/blink]')\n    if version == '3.3.0':\n        sp_home = 'https://nlp.stanford.edu/software/stanford-parser-full-2013-11-12.zip'\n    elif version == '4.2.0':\n        sp_home = 'https://nlp.stanford.edu/software/stanford-parser-4.2.0.zip'\n    else:\n        raise ValueError(f'Unsupported version {version}')\n    sp_home = get_resource(sp_home)\n    # jar_path = get_resource(f'{sp_home}#stanford-parser.jar')\n    if ud:\n        jclass = 'edu.stanford.nlp.trees.international.pennchinese.UniversalChineseGrammaticalStructure' if language == 'zh' \\\n            else 'edu.stanford.nlp.trees.ud.UniversalDependenciesConverter'\n    else:\n        jclass = 'edu.stanford.nlp.trees.international.pennchinese.ChineseGrammaticalStructure' if language == 'zh' \\\n            else 'edu.stanford.nlp.trees.EnglishGrammaticalStructure'\n    cmd = f'java -cp {sp_home}/* {jclass} ' \\\n          f'-treeFile {src}'\n    if conllx:\n        cmd += ' -conllx'\n    if not ud:\n        cmd += f' -basic -keepPunct'\n    code, out, err = get_exitcode_stdout_stderr(cmd)\n    with open(dst, 'w') as f:\n        f.write(out)\n    if code:\n        raise RuntimeError(f'Conversion failed with code {code} for {src}. The err message is:\\n {err}\\n'\n                           f'Do you have java installed? Do you have enough memory?')\n\n\ndef clean_ctb_bracketed(ctb_root, out_root):\n    os.makedirs(out_root, exist_ok=True)\n    ctb_root = join(ctb_root, 'bracketed')\n    chtbs = _list_treebank_root(ctb_root)\n    timer = CountdownTimer(len(chtbs))\n    for f in chtbs:\n        with open(join(ctb_root, f), encoding='utf-8') as src, open(join(out_root, f + '.txt'), 'w',\n                                                                    encoding='utf-8') as out:\n            for line in src:\n                if not line.strip().startswith('<'):\n                    out.write(line)\n        timer.log('Cleaning up CTB [blink][yellow]...[/yellow][/blink]', erase=False)\n\n\ndef _list_treebank_root(ctb_root):\n    chtbs = [f for f in listdir(ctb_root) if isfile(join(ctb_root, f)) and f.startswith('chtb')]\n    return sorted(chtbs)\n\n\ndef list_treebank(ctb_home):\n    ctb_home = get_resource(ctb_home)\n    cleaned_root = join(ctb_home, 'cleaned_bracket')\n    return _list_treebank_root(cleaned_root)\n\n\ndef load_bracketed_trees(chtbs) -> List[Tree]:\n    trees = []\n    for f in chtbs:\n        with open(f, encoding='utf-8') as src:\n            content = src.read()\n            trees = [x for x in content.split('\\n\\n') if x.strip()]\n            for tree in trees:\n                tree = Tree.fromstring(tree)\n                trees.append(tree)\n    return trees\n\n\ndef split_str_to_trees(text: str):\n    trees = []\n    buffer = []\n    for line in text.split('\\n'):\n        if not line.strip():\n            continue\n        if line.startswith('('):\n            if buffer:\n                trees.append('\\n'.join(buffer).strip())\n                buffer = []\n        buffer.append(line)\n    if buffer:\n        trees.append('\\n'.join(buffer).strip())\n    return trees\n\n\ndef make_ctb_tasks(chtbs, out_root, part):\n    for task in ['cws', 'pos', 'par', 'dep']:\n        os.makedirs(join(out_root, task), exist_ok=True)\n    timer = CountdownTimer(len(chtbs))\n    par_path = join(out_root, 'par', f'{part}.txt')\n    with open(join(out_root, 'cws', f'{part}.txt'), 'w', encoding='utf-8') as cws, \\\n            open(join(out_root, 'pos', f'{part}.tsv'), 'w', encoding='utf-8') as pos, \\\n            open(par_path, 'w', encoding='utf-8') as par:\n        for f in chtbs:\n            with open(f, encoding='utf-8') as src:\n                content = src.read()\n                trees = split_str_to_trees(content)\n                for tree in trees:\n                    try:\n                        tree = Tree.fromstring(tree)\n                    except ValueError:\n                        print(tree)\n                        exit(1)\n                    words = []\n                    for word, tag in tree.pos():\n                        if tag == '-NONE-' or not tag:\n                            continue\n                        tag = tag.split('-')[0]\n                        if tag == 'X':  # 铜_NN 30_CD ｘ_X 25_CD ｘ_X 14_CD cm_NT 1999_NT\n                            tag = 'FW'\n                        pos.write('{}\\t{}\\n'.format(word, tag))\n                        words.append(word)\n                    cws.write(' '.join(words))\n                    par.write(tree.pformat(margin=sys.maxsize))\n                    for fp in cws, pos, par:\n                        fp.write('\\n')\n            timer.log(f'Preprocesing the [blue]{part}[/blue] set of CTB [blink][yellow]...[/yellow][/blink]',\n                      erase=False)\n    remove_all_ec(par_path)\n    dep_path = join(out_root, 'dep', f'{part}.conllx')\n    convert_to_dependency(par_path, dep_path)\n    sents = list(read_conll(dep_path))\n    with open(dep_path, 'w') as out:\n        for sent in sents:\n            for i, cells in enumerate(sent):\n                tag = cells[3]\n                tag = tag.split('-')[0]  # NT-SHORT ---> NT\n                if tag == 'X':  # 铜_NN 30_CD ｘ_X 25_CD ｘ_X 14_CD cm_NT 1999_NT\n                    tag = 'FW'\n                cells[3] = cells[4] = tag\n                out.write('\\t'.join(str(x) for x in cells))\n                out.write('\\n')\n            out.write('\\n')\n\n\ndef reverse_splits(splits):\n    cid_domain = dict()\n    for domain, cids in splits.items():\n        for each in cids:\n            cid_domain[each] = domain\n    return cid_domain\n\n\ndef split_chtb(chtbs: List[str], splits=None):\n    train, dev, test = [], [], []\n    unused = []\n    for each in chtbs:\n        name, domain, ext = each.split('.', 2)\n        _, cid = name.split('_')\n        if splits:\n            if cid in splits['train']:\n                bin = train\n            elif cid in splits['dev']:\n                bin = dev\n            elif cid in splits['test']:\n                bin = test\n            else:\n                bin = unused\n                # raise IOError(f'{name} not in any splits')\n        else:\n            bin = train\n            if name.endswith('8'):\n                bin = dev\n            elif name.endswith('9'):\n                bin = test\n        bin.append(each)\n    return train, dev, test\n\n\ndef id_of_chtb(each: str):\n    return int(each.split('.')[0].split('_')[-1])\n\n\ndef make_ctb(ctb_home):\n    ctb_home = get_resource(ctb_home)\n    cleaned_root = join(ctb_home, 'cleaned_bracket')\n    if not os.path.isdir(cleaned_root):\n        clean_ctb_bracketed(ctb_home, cleaned_root)\n    tasks_root = join(ctb_home, 'tasks')\n    if not os.path.isdir(tasks_root):\n        try:\n            chtbs = _list_treebank_root(cleaned_root)\n            print(f'For the {len(chtbs)} files in CTB, we apply the following splits:')\n            train, dev, test = split_chtb(chtbs)\n            for part, name in zip([train, dev, test], ['train', 'dev', 'test']):\n                print(f'{name} = {[id_of_chtb(x) for x in part]}')\n            cprint('[yellow]Each file id ending with 8/9 is put into '\n                   'dev/test respectively, the rest are put into train. '\n                   'Our splits ensure files are evenly split across each genre, which is recommended '\n                   'for production systems.[/yellow]')\n            for part, name in zip([train, dev, test], ['train', 'dev', 'test']):\n                make_ctb_tasks([join(cleaned_root, x) for x in part], tasks_root, name)\n            cprint('Done pre-processing CTB. Enjoy your research with [blue]HanLP[/blue]!')\n        except Exception as e:\n            shutil.rmtree(tasks_root, ignore_errors=True)\n            raise e\n\n\ndef load_domains(ctb_home):\n    \"\"\"\n    Load file ids from a Chinese treebank grouped by domains.\n\n    Args:\n        ctb_home: Root path to CTB.\n\n    Returns:\n        A dict of sets, each represents a domain.\n    \"\"\"\n    ctb_home = get_resource(ctb_home)\n    ctb_root = join(ctb_home, 'bracketed')\n    chtbs = _list_treebank_root(ctb_root)\n    domains = defaultdict(set)\n    for each in chtbs:\n        name, domain = each.split('.')\n        _, fid = name.split('_')\n        domains[domain].add(fid)\n    return domains\n\n\ndef ctb_pos_to_text_format(path, delimiter='_'):\n    \"\"\"\n    Convert ctb pos tagging corpus from tsv format to text format, where each word is followed by\n    its pos tag.\n    Args:\n        path: File to be converted.\n        delimiter: Delimiter between word and tag.\n    \"\"\"\n    path = get_resource(path)\n    name, ext = os.path.splitext(path)\n    with open(f'{name}.txt', 'w', encoding='utf-8') as out:\n        for sent in read_tsv_as_sents(path):\n            out.write(' '.join([delimiter.join(x) for x in sent]))\n            out.write('\\n')\n\n\ndef remove_all_ec(path):\n    \"\"\"\n    Remove empty categories for all trees in this file and save them into a \"noempty\" file.\n\n    Args:\n        path: File path.\n    \"\"\"\n    script = get_resource('https://file.hankcs.com/bin/remove_ec.zip')\n    with pushd(script):\n        run_cmd(f'java -cp elit-ddr-0.0.5-SNAPSHOT.jar:elit-sdk-0.0.5-SNAPSHOT.jar:hanlp-1.7.8.jar:'\n                f'fastutil-8.1.1.jar:. demo.RemoveEmptyCategoriesTreebank {path}')\n"
  },
  {
    "path": "hanlp/datasets/parsing/loaders/conll_dataset.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-08 16:10\nfrom typing import Union, List, Callable, Dict\n\nfrom hanlp_common.constant import ROOT, EOS, BOS\nfrom hanlp.common.dataset import TransformableDataset\nfrom hanlp.components.parsers.conll import read_conll\nfrom hanlp.utils.io_util import TimingFileIterator\n\n\nclass CoNLLParsingDataset(TransformableDataset):\n\n    def __init__(self,\n                 data: Union[str, List],\n                 transform: Union[Callable, List] = None,\n                 cache=None,\n                 generate_idx=None,\n                 prune: Callable[[Dict[str, List[str]]], bool] = None) -> None:\n        \"\"\"General class for CoNLL style dependency parsing datasets.\n\n        Args:\n            data: The local or remote path to a dataset, or a list of samples where each sample is a dict.\n            transform: Predefined transform(s).\n            cache: ``True`` to enable caching, so that transforms won't be called twice.\n            generate_idx: Create a :const:`~hanlp_common.constants.IDX` field for each sample to store its order in dataset. Useful for prediction when\n                samples are re-ordered by a sampler.\n            prune: A filter to prune unwanted samples.\n        \"\"\"\n        self._prune = prune\n        super().__init__(data, transform, cache, generate_idx)\n\n    def load_file(self, filepath):\n        \"\"\"Both ``.conllx`` and ``.conllu`` are supported. Their descriptions can be found in\n        :class:`hanlp_common.conll.CoNLLWord` and :class:`hanlp_common.conll.CoNLLUWord` respectively.\n\n        Args:\n            filepath: ``.conllx`` or ``.conllu`` file path.\n        \"\"\"\n        if filepath.endswith('.conllu'):\n            # See https://universaldependencies.org/format.html\n            field_names = ['ID', 'FORM', 'LEMMA', 'UPOS', 'XPOS',\n                           'FEATS', 'HEAD', 'DEPREL', 'DEPS', 'MISC']\n        else:\n            field_names = ['ID', 'FORM', 'LEMMA', 'CPOS', 'POS',\n                           'FEATS', 'HEAD', 'DEPREL', 'PHEAD', 'PDEPREL']\n        fp = TimingFileIterator(filepath)\n        for idx, sent in enumerate(read_conll(fp)):\n            sample = {}\n            for i, field in enumerate(field_names):\n                sample[field] = [cell[i] for cell in sent]\n            if not self._prune or not self._prune(sample):\n                yield sample\n            fp.log(f'{idx + 1} samples [blink][yellow]...[/yellow][/blink]')\n\n    def __len__(self) -> int:\n        return len(self.data)\n\n\ndef append_bos(sample: dict, pos_key='CPOS', bos=ROOT) -> dict:\n    \"\"\"\n\n    Args:\n        sample:\n        pos_key:\n        bos: A special token inserted to the head of tokens.\n\n    Returns:\n\n    \"\"\"\n    sample['token'] = [bos] + sample['FORM']\n    if pos_key in sample:\n        sample['pos'] = [ROOT] + sample[pos_key]\n    if 'HEAD' in sample:\n        sample['arc'] = [0] + sample['HEAD']\n        sample['rel'] = sample['DEPREL'][:1] + sample['DEPREL']\n    return sample\n\n\ndef append_bos_eos(sample: dict) -> dict:\n    sample['token'] = [BOS] + sample['FORM'] + [EOS]\n    if 'CPOS' in sample:\n        sample['pos'] = [BOS] + sample['CPOS'] + [EOS]\n    if 'HEAD' in sample:\n        sample['arc'] = [0] + sample['HEAD'] + [0]\n        sample['rel'] = sample['DEPREL'][:1] + sample['DEPREL'] + sample['DEPREL'][:1]\n    return sample\n\n\ndef get_sibs(sample: dict) -> dict:\n    heads = sample.get('arc', None)\n    if heads:\n        sibs = [-1] * len(heads)\n        for i in range(1, len(heads)):\n            hi = heads[i]\n            for j in range(i + 1, len(heads)):\n                hj = heads[j]\n                di, dj = hi - i, hj - j\n                if hi >= 0 and hj >= 0 and hi == hj and di * dj > 0:\n                    if abs(di) > abs(dj):\n                        sibs[i] = j\n                    else:\n                        sibs[j] = i\n                    break\n        sample['sib_id'] = [0] + sibs[1:]\n    return sample\n"
  },
  {
    "path": "hanlp/datasets/parsing/loaders/constituency_dataset.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-11-28 19:27\nfrom typing import List\n\nfrom phrasetree.tree import Tree\n\nfrom hanlp_common.constant import EOS, BOS\nfrom hanlp.common.dataset import TransformableDataset\n\n\nclass ConstituencyDataset(TransformableDataset):\n    def load_file(self, filepath: str):\n        with open(filepath) as src:\n            for line in src:\n                line = line.strip()\n                if not line:\n                    continue\n                yield {'constituency': Tree.fromstring(line)}\n\n\ndef unpack_tree_to_features(sample: dict):\n    tree = sample.get('constituency', None)\n    if tree:\n        words, tags = zip(*tree.pos())\n        chart = [[None] * (len(words) + 1) for _ in range(len(words) + 1)]\n        for i, j, label in factorize(binarize(tree)[0]):\n            # if no_subcategory:\n            #     label = label.split('-')[0]\n            chart[i][j] = label\n        sample['token'] = [BOS] + list(words) + [EOS]\n        sample['chart'] = chart\n    return sample\n\n\ndef append_bos_eos(sample: dict):\n    if '_con_token' not in sample:\n        sample['_con_token'] = sample['token']\n        sample['token'] = [BOS] + sample['token'] + [EOS]\n    return sample\n\n\ndef remove_subcategory(sample: dict):\n    tree: Tree = sample.get('constituency', None)\n    if tree:\n        for subtree in tree.subtrees():\n            label = subtree.label()\n            subtree.set_label(label.split('-')[0])\n    return sample\n\n\ndef binarize(tree: Tree):\n    r\"\"\"\n    Conducts binarization over the tree.\n\n    First, the tree is transformed to satisfy `Chomsky Normal Form (CNF)`_.\n    Here we call :meth:`~tree.Tree.chomsky_normal_form` to conduct left-binarization.\n    Second, all unary productions in the tree are collapsed.\n\n    Args:\n        tree (tree.Tree):\n            The tree to be binarized.\n\n    Returns:\n        The binarized tree.\n\n    Examples:\n        >>> tree = Tree.fromstring('''\n                                        (TOP\n                                          (S\n                                            (NP (_ She))\n                                            (VP (_ enjoys) (S (VP (_ playing) (NP (_ tennis)))))\n                                            (_ .)))\n                                        ''')\n        >>> print(Tree.binarize(tree))\n        (TOP\n          (S\n            (S|<>\n              (NP (_ She))\n              (VP\n                (VP|<> (_ enjoys))\n                (S+VP (VP|<> (_ playing)) (NP (_ tennis)))))\n            (S|<> (_ .))))\n\n    .. _Chomsky Normal Form (CNF):\n        https://en.wikipedia.org/wiki/Chomsky_normal_form\n    \"\"\"\n\n    tree: Tree = tree.copy(True)\n    nodes = [tree]\n    while nodes:\n        node = nodes.pop()\n        if isinstance(node, Tree):\n            nodes.extend([child for child in node])\n            if len(node) > 1:\n                for i, child in enumerate(node):\n                    if not isinstance(child[0], Tree):\n                        node[i] = Tree(f\"{node.label()}|<>\", [child])\n    tree.chomsky_normal_form('left', 0, 0)\n    tree.collapse_unary()\n\n    return tree\n\n\ndef factorize(tree, delete_labels=None, equal_labels=None):\n    r\"\"\"\n    Factorizes the tree into a sequence.\n    The tree is traversed in pre-order.\n\n    Args:\n        tree (tree.Tree):\n            The tree to be factorized.\n        delete_labels (set[str]):\n            A set of labels to be ignored. This is used for evaluation.\n            If it is a pre-terminal label, delete the word along with the brackets.\n            If it is a non-terminal label, just delete the brackets (don't delete childrens).\n            In `EVALB`_, the default set is:\n            {'TOP', 'S1', '-NONE-', ',', ':', '``', \"''\", '.', '?', '!', ''}\n            Default: ``None``.\n        equal_labels (dict[str, str]):\n            The key-val pairs in the dict are considered equivalent (non-directional). This is used for evaluation.\n            The default dict defined in `EVALB`_ is: {'ADVP': 'PRT'}\n            Default: ``None``.\n\n    Returns:\n        The sequence of the factorized tree.\n\n    Examples:\n        >>> tree = Tree.fromstring('' (TOP\n                                          (S\n                                            (NP (_ She))\n                                            (VP (_ enjoys) (S (VP (_ playing) (NP (_ tennis)))))\n                                            (_ .)))\n                                    '')\n        >>> Tree.factorize(tree)\n        [(0, 5, 'TOP'), (0, 5, 'S'), (0, 1, 'NP'), (1, 4, 'VP'), (2, 4, 'S'), (2, 4, 'VP'), (3, 4, 'NP')]\n        >>> Tree.factorize(tree, delete_labels={'TOP', 'S1', '-NONE-', ',', ':', '``', \"''\", '.', '?', '!', ''})\n        [(0, 5, 'S'), (0, 1, 'NP'), (1, 4, 'VP'), (2, 4, 'S'), (2, 4, 'VP'), (3, 4, 'NP')]\n\n    .. _EVALB:\n        https://nlp.cs.nyu.edu/evalb/\n    \"\"\"\n\n    def track(tree, i):\n        label = tree.label()\n        if delete_labels is not None and label in delete_labels:\n            label = None\n        if equal_labels is not None:\n            label = equal_labels.get(label, label)\n        if len(tree) == 1 and not isinstance(tree[0], Tree):\n            return (i + 1 if label is not None else i), []\n        j, spans = i, []\n        for child in tree:\n            if isinstance(child, Tree):\n                j, s = track(child, j)\n                spans += s\n        if label is not None and j > i:\n            spans = [(i, j, label)] + spans\n        return j, spans\n\n    return track(tree, 0)[1]\n\n\ndef build_tree(tokens: List[str], sequence):\n    r\"\"\"\n    Builds a constituency tree from the sequence. The sequence is generated in pre-order.\n    During building the tree, the sequence is de-binarized to the original format (i.e.,\n    the suffixes ``|<>`` are ignored, the collapsed labels are recovered).\n\n    Args:\n        tokens :\n            All tokens in a sentence.\n        sequence (list[tuple]):\n            A list of tuples used for generating a tree.\n            Each tuple consits of the indices of left/right span boundaries and label of the span.\n\n    Returns:\n        A result constituency tree.\n\n    Examples:\n        >>> tree = Tree.totree(['She', 'enjoys', 'playing', 'tennis', '.'], 'TOP')\n        >>> sequence = [(0, 5, 'S'), (0, 4, 'S|<>'), (0, 1, 'NP'), (1, 4, 'VP'), (1, 2, 'VP|<>'),\n                        (2, 4, 'S+VP'), (2, 3, 'VP|<>'), (3, 4, 'NP'), (4, 5, 'S|<>')]\n        >>> print(Tree.build_tree(root, sequence))\n        (TOP\n          (S\n            (NP (_ She))\n            (VP (_ enjoys) (S (VP (_ playing) (NP (_ tennis)))))\n            (_ .)))\n    \"\"\"\n    if not tokens:  # User passed in [], which is the tokenized result of ''\n        return Tree('TOP', [])\n    tree = Tree('TOP', [Tree('_', [t]) for t in tokens])\n    root = tree.label()\n    leaves = [subtree for subtree in tree.subtrees() if not isinstance(subtree[0], Tree)]\n\n    def track(node):\n        i, j, label = next(node)\n        if j == i + 1:\n            children = [leaves[i]]\n        else:\n            children = track(node) + track(node)\n        if label.endswith('|<>'):\n            return children\n        labels = label.split('+')\n        tree = Tree(labels[-1], children)\n        for label in reversed(labels[:-1]):\n            tree = Tree(label, [tree])\n        return [tree]\n\n    return Tree(root, track(iter(sequence)))\n"
  },
  {
    "path": "hanlp/datasets/parsing/pmt1.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2022-02-15 04:14\nimport os.path\n\nfrom hanlp.utils.io_util import get_resource\nfrom hanlp.utils.log_util import cprint\nfrom hanlp_common.conll import CoNLLSentence, CoNLLWord\n\n_HOME = 'https://github.com/qiulikun/PKUMultiviewTreebank/archive/refs/heads/master.zip'\nPTM_V1_RAW = _HOME + '#199801_dependency_treebank_2014pos.txt'\nPTM_V1_TRAIN = _HOME + '#train.conllx'\n'The training set of PKU Multi-view Chinese Treebank (PMT) 1.0 (:cite:`qiu-etal-2014-multi`).'\nPTM_V1_DEV = _HOME + '#dev.conllx'\n'The dev set of PKU Multi-view Chinese Treebank (PMT) 1.0 (:cite:`qiu-etal-2014-multi`).'\nPTM_V1_TEST = _HOME + '#test.conllx'\n'The test set of PKU Multi-view Chinese Treebank (PMT) 1.0 (:cite:`qiu-etal-2014-multi`).'\n\n\ndef _make_ptm():\n    raw = get_resource(PTM_V1_RAW)\n    home = os.path.dirname(raw)\n    done = True\n    for part in ['train', 'dev', 'test']:\n        if not os.path.isfile(os.path.join(home, f'{part}.conllx')):\n            done = False\n            break\n    if done:\n        return\n    sents = []\n    with open(raw) as src:\n        buffer = []\n        for line in src:\n            line = line.strip()\n            if line:\n                buffer.append(line)\n            else:\n                if buffer:\n                    tok, pos, rel, arc = [x.split() for x in buffer]\n                    sent = CoNLLSentence()\n                    for i, (t, p, r, a) in enumerate(zip(tok, pos, rel, arc)):\n                        sent.append(CoNLLWord(i + 1, form=t, cpos=p, head=a, deprel=r))\n                    sents.append(sent)\n                    buffer.clear()\n\n    prev_offset = 0\n    # Sentences 12001-13000 and 13001-14463 are used as the development and test set, respectively. The remaining\n    # sentences are used as training data.\n    for part, offset in zip(['train', 'dev', 'test'], [12000, 13000, 14463]):\n        with open(os.path.join(home, f'{part}.conllx'), 'w') as out:\n            portion = sents[prev_offset:offset]\n            cprint(f'[yellow]{len(portion)}[/yellow] sentences [cyan][{prev_offset + 1}:{offset})[/cyan] in {part}')\n            for sent in portion:\n                out.write(str(sent) + '\\n\\n')\n        prev_offset = offset\n\n\n_make_ptm()\n"
  },
  {
    "path": "hanlp/datasets/parsing/ptb.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-02-17 15:46\n\n_PTB_HOME = 'https://github.com/KhalilMrini/LAL-Parser/archive/master.zip#data/'\n\nPTB_TRAIN = _PTB_HOME + '02-21.10way.clean'\n'''Training set of PTB without empty categories. PoS tags are automatically predicted using 10-fold \njackknifing (:cite:`collins-koo-2005-discriminative`).'''\nPTB_DEV = _PTB_HOME + '22.auto.clean'\n'''Dev set of PTB without empty categories. PoS tags are automatically predicted using 10-fold \njackknifing (:cite:`collins-koo-2005-discriminative`).'''\nPTB_TEST = _PTB_HOME + '23.auto.clean'\n'''Test set of PTB without empty categories. PoS tags are automatically predicted using 10-fold \njackknifing (:cite:`collins-koo-2005-discriminative`).'''\n\nPTB_SD330_TRAIN = _PTB_HOME + 'ptb_train_3.3.0.sd.clean'\n'''Training set of PTB in Stanford Dependencies 3.3.0 format. PoS tags are automatically predicted using 10-fold \njackknifing (:cite:`collins-koo-2005-discriminative`).'''\nPTB_SD330_DEV = _PTB_HOME + 'ptb_dev_3.3.0.sd.clean'\n'''Dev set of PTB in Stanford Dependencies 3.3.0 format. PoS tags are automatically predicted using 10-fold \njackknifing (:cite:`collins-koo-2005-discriminative`).'''\nPTB_SD330_TEST = _PTB_HOME + 'ptb_test_3.3.0.sd.clean'\n'''Test set of PTB in Stanford Dependencies 3.3.0 format. PoS tags are automatically predicted using 10-fold \njackknifing (:cite:`collins-koo-2005-discriminative`).'''\n\nPTB_TOKEN_MAPPING = {\n    \"-LRB-\": \"(\",\n    \"-RRB-\": \")\",\n    \"-LCB-\": \"{\",\n    \"-RCB-\": \"}\",\n    \"-LSB-\": \"[\",\n    \"-RSB-\": \"]\",\n    \"``\": '\"',\n    \"''\": '\"',\n    \"`\": \"'\",\n    '«': '\"',\n    '»': '\"',\n    '‘': \"'\",\n    '’': \"'\",\n    '“': '\"',\n    '”': '\"',\n    '„': '\"',\n    '‹': \"'\",\n    '›': \"'\",\n    \"\\u2013\": \"--\",  # en dash\n    \"\\u2014\": \"--\",  # em dash\n}\n"
  },
  {
    "path": "hanlp/datasets/parsing/semeval15.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-07-28 14:40\n# from hanlp.datasets.parsing.conll_dataset import CoNLLParsingDataset\n#\n#\n# class SemEval15Dataset(CoNLLParsingDataset):\n#     def load_file(self, filepath: str):\n#         pass\nimport warnings\n\nfrom hanlp_common.constant import ROOT, PAD\nfrom hanlp_common.conll import CoNLLSentence\n\n\ndef unpack_deps_to_head_deprel(sample: dict, pad_rel=None, arc_key='arc', rel_key='rel'):\n    if 'DEPS' in sample:\n        deps = ['_'] + sample['DEPS']\n        sample[arc_key] = arc = []\n        sample[rel_key] = rel = []\n        for each in deps:\n            arc_per_token = [False] * len(deps)\n            rel_per_token = [None] * len(deps)\n            if each != '_':\n                for ar in each.split('|'):\n                    a, r = ar.split(':')\n                    a = int(a)\n                    arc_per_token[a] = True\n                    rel_per_token[a] = r\n                    if not pad_rel:\n                        pad_rel = r\n            arc.append(arc_per_token)\n            rel.append(rel_per_token)\n        if not pad_rel:\n            pad_rel = PAD\n        for i in range(len(rel)):\n            rel[i] = [r if r else pad_rel for r in rel[i]]\n    return sample\n\n\ndef append_bos_to_form_pos(sample, pos_key='CPOS'):\n    sample['token'] = [ROOT] + sample['FORM']\n    if pos_key in sample:\n        sample['pos'] = [ROOT] + sample[pos_key]\n    return sample\n\n\ndef merge_head_deprel_with_2nd(sample: dict):\n    if 'arc' in sample:\n        arc_2nd = sample['arc_2nd']\n        rel_2nd = sample['rel_2nd']\n        for i, (arc, rel) in enumerate(zip(sample['arc'], sample['rel'])):\n            if i:\n                if arc_2nd[i][arc] and rel_2nd[i][arc] != rel:\n                    sample_str = CoNLLSentence.from_dict(sample, conllu=True).to_markdown()\n                    warnings.warn(f'The main dependency conflicts with 2nd dependency at ID={i}, ' \\\n                                  'which means joint mode might not be suitable. ' \\\n                                  f'The sample is\\n{sample_str}')\n                arc_2nd[i][arc] = True\n                rel_2nd[i][arc] = rel\n    return sample\n"
  },
  {
    "path": "hanlp/datasets/parsing/semeval16.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 00:51\nfrom hanlp_common.conll import CoNLLSentence\nimport os\n\nfrom hanlp.utils.io_util import get_resource, merge_files\nfrom hanlp_common.io import eprint\n\n_SEMEVAL2016_HOME = 'https://github.com/HIT-SCIR/SemEval-2016/archive/master.zip'\n\nSEMEVAL2016_NEWS_TRAIN = _SEMEVAL2016_HOME + '#train/news.train.conll'\nSEMEVAL2016_NEWS_DEV = _SEMEVAL2016_HOME + '#validation/news.valid.conll'\nSEMEVAL2016_NEWS_TEST = _SEMEVAL2016_HOME + '#test/news.test.conll'\n\nSEMEVAL2016_NEWS_TRAIN_CONLLU = _SEMEVAL2016_HOME + '#train/news.train.conllu'\nSEMEVAL2016_NEWS_DEV_CONLLU = _SEMEVAL2016_HOME + '#validation/news.valid.conllu'\nSEMEVAL2016_NEWS_TEST_CONLLU = _SEMEVAL2016_HOME + '#test/news.test.conllu'\n\nSEMEVAL2016_TEXT_TRAIN = _SEMEVAL2016_HOME + '#train/text.train.conll'\nSEMEVAL2016_TEXT_DEV = _SEMEVAL2016_HOME + '#validation/text.valid.conll'\nSEMEVAL2016_TEXT_TEST = _SEMEVAL2016_HOME + '#test/text.test.conll'\n\nSEMEVAL2016_TEXT_TRAIN_CONLLU = _SEMEVAL2016_HOME + '#train/text.train.conllu'\nSEMEVAL2016_TEXT_DEV_CONLLU = _SEMEVAL2016_HOME + '#validation/text.valid.conllu'\nSEMEVAL2016_TEXT_TEST_CONLLU = _SEMEVAL2016_HOME + '#test/text.test.conllu'\n\nSEMEVAL2016_FULL_TRAIN_CONLLU = _SEMEVAL2016_HOME + '#train/full.train.conllu'\nSEMEVAL2016_FULL_DEV_CONLLU = _SEMEVAL2016_HOME + '#validation/full.valid.conllu'\nSEMEVAL2016_FULL_TEST_CONLLU = _SEMEVAL2016_HOME + '#test/full.test.conllu'\n\n\ndef convert_conll_to_conllu(path):\n    sents = CoNLLSentence.from_file(path, conllu=True)\n    with open(os.path.splitext(path)[0] + '.conllu', 'w') as out:\n        for sent in sents:\n            for word in sent:\n                if not word.deps:\n                    word.deps = [(word.head, word.deprel)]\n                    word.head = None\n                    word.deprel = None\n            out.write(str(sent))\n            out.write('\\n\\n')\n\n\nfor file in [SEMEVAL2016_NEWS_TRAIN, SEMEVAL2016_NEWS_DEV, SEMEVAL2016_NEWS_TEST,\n             SEMEVAL2016_TEXT_TRAIN, SEMEVAL2016_TEXT_DEV, SEMEVAL2016_TEXT_TEST]:\n    file = get_resource(file)\n    conllu = os.path.splitext(file)[0] + '.conllu'\n    if not os.path.isfile(conllu):\n        eprint(f'Converting {os.path.basename(file)} to {os.path.basename(conllu)} ...')\n        convert_conll_to_conllu(file)\n\nfor group, part in zip([[SEMEVAL2016_NEWS_TRAIN_CONLLU, SEMEVAL2016_TEXT_TRAIN_CONLLU],\n                        [SEMEVAL2016_NEWS_DEV_CONLLU, SEMEVAL2016_TEXT_DEV_CONLLU],\n                        [SEMEVAL2016_NEWS_TEST_CONLLU, SEMEVAL2016_TEXT_TEST_CONLLU]],\n                       ['train', 'valid', 'test']):\n    root = get_resource(_SEMEVAL2016_HOME)\n    dst = f'{root}/train/full.{part}.conllu'\n    if not os.path.isfile(dst):\n        group = [get_resource(x) for x in group]\n        eprint(f'Concatenating {os.path.basename(group[0])} and {os.path.basename(group[1])} '\n               f'into full dataset {os.path.basename(dst)} ...')\n        merge_files(group, dst)\n"
  },
  {
    "path": "hanlp/datasets/parsing/ud/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-07 21:45\nimport os\nimport shutil\n\nfrom hanlp.components.parsers.ud.udify_util import get_ud_treebank_files\nfrom hanlp.utils.io_util import get_resource\nfrom hanlp.utils.log_util import flash\n\n\ndef concat_treebanks(home, version):\n    ud_home = get_resource(home)\n    treebanks = get_ud_treebank_files(ud_home)\n    output_dir = os.path.abspath(os.path.join(ud_home, os.path.pardir, os.path.pardir, f'ud-multilingual-v{version}'))\n    if os.path.isdir(output_dir):\n        return output_dir\n    os.makedirs(output_dir)\n    train, dev, test = list(zip(*[treebanks[k] for k in treebanks]))\n\n    for treebank, name in zip([train, dev, test], [\"train.conllu\", \"dev.conllu\", \"test.conllu\"]):\n        flash(f'Concatenating {len(train)} treebanks into {name} [blink][yellow]...[/yellow][/blink]')\n        with open(os.path.join(output_dir, name), 'w') as write:\n            for t in treebank:\n                if not t:\n                    continue\n                with open(t, 'r') as read:\n                    shutil.copyfileobj(read, write)\n        flash('')\n    return output_dir\n"
  },
  {
    "path": "hanlp/datasets/parsing/ud/ud210.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-07 21:03\nimport glob\nimport os\n\nfrom hanlp.utils.io_util import uncompress, get_resource\n\n_UD_210_URL = \"https://lindat.mff.cuni.cz/repository/xmlui/handle/11234/1-4758/allzip\"\n_UD_210_HOME = _UD_210_URL + '#ud-treebanks-v2.10/'\n_path = get_resource(_UD_210_URL)\nif os.path.isfile(_path):\n    os.rename(_path, _path + '.zip')\n    uncompress(_path + '.zip')\n    uncompress(os.path.join(_path, 'ud-treebanks-v2.10.tgz'))\n\n\n# noinspection PyShadowingNames\ndef _list_dir(path, home):\n    prefix = home.lstrip('_').replace('_HOME', '')\n\n    path = get_resource(path)\n    with open('ud210.py', 'a') as out:\n        for f in sorted(glob.glob(path + '/ud-treebanks-v2.10/UD_*')):\n            basename = os.path.basename(f)\n            name = basename[len('UD_'):]\n            name = name.upper().replace('-', '_')\n            for split in 'train', 'dev', 'test':\n                sp = glob.glob(f + f'/*{split}.conllu')\n                if not sp:\n                    continue\n                sp = os.path.basename(sp[0])\n                out.write(f'{prefix}_{name}_{split.upper()} = {home} + \"{basename}/{sp}\"\\n')\n                out.write(f'\"{prefix} {split} set of {name}.\"\\n')\n\n\ndef main():\n    _list_dir(_UD_210_URL, '_UD_210_HOME')\n    pass\n\n\nif __name__ == '__main__':\n    main()\nUD_210_AFRIKAANS_AFRIBOOMS_TRAIN = _UD_210_HOME + \"UD_Afrikaans-AfriBooms/af_afribooms-ud-train.conllu\"\n\"UD_210 train set of AFRIKAANS_AFRIBOOMS.\"\nUD_210_AFRIKAANS_AFRIBOOMS_DEV = _UD_210_HOME + \"UD_Afrikaans-AfriBooms/af_afribooms-ud-dev.conllu\"\n\"UD_210 dev set of AFRIKAANS_AFRIBOOMS.\"\nUD_210_AFRIKAANS_AFRIBOOMS_TEST = _UD_210_HOME + \"UD_Afrikaans-AfriBooms/af_afribooms-ud-test.conllu\"\n\"UD_210 test set of AFRIKAANS_AFRIBOOMS.\"\nUD_210_AKKADIAN_PISANDUB_TEST = _UD_210_HOME + \"UD_Akkadian-PISANDUB/akk_pisandub-ud-test.conllu\"\n\"UD_210 test set of AKKADIAN_PISANDUB.\"\nUD_210_AKKADIAN_RIAO_TEST = _UD_210_HOME + \"UD_Akkadian-RIAO/akk_riao-ud-test.conllu\"\n\"UD_210 test set of AKKADIAN_RIAO.\"\nUD_210_AKUNTSU_TUDET_TEST = _UD_210_HOME + \"UD_Akuntsu-TuDeT/aqz_tudet-ud-test.conllu\"\n\"UD_210 test set of AKUNTSU_TUDET.\"\nUD_210_ALBANIAN_TSA_TEST = _UD_210_HOME + \"UD_Albanian-TSA/sq_tsa-ud-test.conllu\"\n\"UD_210 test set of ALBANIAN_TSA.\"\nUD_210_AMHARIC_ATT_TEST = _UD_210_HOME + \"UD_Amharic-ATT/am_att-ud-test.conllu\"\n\"UD_210 test set of AMHARIC_ATT.\"\nUD_210_ANCIENT_GREEK_PROIEL_TRAIN = _UD_210_HOME + \"UD_Ancient_Greek-PROIEL/grc_proiel-ud-train.conllu\"\n\"UD_210 train set of ANCIENT_GREEK_PROIEL.\"\nUD_210_ANCIENT_GREEK_PROIEL_DEV = _UD_210_HOME + \"UD_Ancient_Greek-PROIEL/grc_proiel-ud-dev.conllu\"\n\"UD_210 dev set of ANCIENT_GREEK_PROIEL.\"\nUD_210_ANCIENT_GREEK_PROIEL_TEST = _UD_210_HOME + \"UD_Ancient_Greek-PROIEL/grc_proiel-ud-test.conllu\"\n\"UD_210 test set of ANCIENT_GREEK_PROIEL.\"\nUD_210_ANCIENT_GREEK_PERSEUS_TRAIN = _UD_210_HOME + \"UD_Ancient_Greek-Perseus/grc_perseus-ud-train.conllu\"\n\"UD_210 train set of ANCIENT_GREEK_PERSEUS.\"\nUD_210_ANCIENT_GREEK_PERSEUS_DEV = _UD_210_HOME + \"UD_Ancient_Greek-Perseus/grc_perseus-ud-dev.conllu\"\n\"UD_210 dev set of ANCIENT_GREEK_PERSEUS.\"\nUD_210_ANCIENT_GREEK_PERSEUS_TEST = _UD_210_HOME + \"UD_Ancient_Greek-Perseus/grc_perseus-ud-test.conllu\"\n\"UD_210 test set of ANCIENT_GREEK_PERSEUS.\"\nUD_210_ANCIENT_HEBREW_PTNK_TRAIN = _UD_210_HOME + \"UD_Ancient_Hebrew-PTNK/hbo_ptnk-ud-train.conllu\"\n\"UD_210 train set of ANCIENT_HEBREW_PTNK.\"\nUD_210_ANCIENT_HEBREW_PTNK_DEV = _UD_210_HOME + \"UD_Ancient_Hebrew-PTNK/hbo_ptnk-ud-dev.conllu\"\n\"UD_210 dev set of ANCIENT_HEBREW_PTNK.\"\nUD_210_ANCIENT_HEBREW_PTNK_TEST = _UD_210_HOME + \"UD_Ancient_Hebrew-PTNK/hbo_ptnk-ud-test.conllu\"\n\"UD_210 test set of ANCIENT_HEBREW_PTNK.\"\nUD_210_APURINA_UFPA_TEST = _UD_210_HOME + \"UD_Apurina-UFPA/apu_ufpa-ud-test.conllu\"\n\"UD_210 test set of APURINA_UFPA.\"\nUD_210_ARABIC_NYUAD_TRAIN = _UD_210_HOME + \"UD_Arabic-NYUAD/ar_nyuad-ud-train.conllu\"\n\"UD_210 train set of ARABIC_NYUAD.\"\nUD_210_ARABIC_NYUAD_DEV = _UD_210_HOME + \"UD_Arabic-NYUAD/ar_nyuad-ud-dev.conllu\"\n\"UD_210 dev set of ARABIC_NYUAD.\"\nUD_210_ARABIC_NYUAD_TEST = _UD_210_HOME + \"UD_Arabic-NYUAD/ar_nyuad-ud-test.conllu\"\n\"UD_210 test set of ARABIC_NYUAD.\"\nUD_210_ARABIC_PADT_TRAIN = _UD_210_HOME + \"UD_Arabic-PADT/ar_padt-ud-train.conllu\"\n\"UD_210 train set of ARABIC_PADT.\"\nUD_210_ARABIC_PADT_DEV = _UD_210_HOME + \"UD_Arabic-PADT/ar_padt-ud-dev.conllu\"\n\"UD_210 dev set of ARABIC_PADT.\"\nUD_210_ARABIC_PADT_TEST = _UD_210_HOME + \"UD_Arabic-PADT/ar_padt-ud-test.conllu\"\n\"UD_210 test set of ARABIC_PADT.\"\nUD_210_ARABIC_PUD_TEST = _UD_210_HOME + \"UD_Arabic-PUD/ar_pud-ud-test.conllu\"\n\"UD_210 test set of ARABIC_PUD.\"\nUD_210_ARMENIAN_ARMTDP_TRAIN = _UD_210_HOME + \"UD_Armenian-ArmTDP/hy_armtdp-ud-train.conllu\"\n\"UD_210 train set of ARMENIAN_ARMTDP.\"\nUD_210_ARMENIAN_ARMTDP_DEV = _UD_210_HOME + \"UD_Armenian-ArmTDP/hy_armtdp-ud-dev.conllu\"\n\"UD_210 dev set of ARMENIAN_ARMTDP.\"\nUD_210_ARMENIAN_ARMTDP_TEST = _UD_210_HOME + \"UD_Armenian-ArmTDP/hy_armtdp-ud-test.conllu\"\n\"UD_210 test set of ARMENIAN_ARMTDP.\"\nUD_210_ARMENIAN_BSUT_TRAIN = _UD_210_HOME + \"UD_Armenian-BSUT/hy_bsut-ud-train.conllu\"\n\"UD_210 train set of ARMENIAN_BSUT.\"\nUD_210_ARMENIAN_BSUT_DEV = _UD_210_HOME + \"UD_Armenian-BSUT/hy_bsut-ud-dev.conllu\"\n\"UD_210 dev set of ARMENIAN_BSUT.\"\nUD_210_ARMENIAN_BSUT_TEST = _UD_210_HOME + \"UD_Armenian-BSUT/hy_bsut-ud-test.conllu\"\n\"UD_210 test set of ARMENIAN_BSUT.\"\nUD_210_ASSYRIAN_AS_TEST = _UD_210_HOME + \"UD_Assyrian-AS/aii_as-ud-test.conllu\"\n\"UD_210 test set of ASSYRIAN_AS.\"\nUD_210_BAMBARA_CRB_TEST = _UD_210_HOME + \"UD_Bambara-CRB/bm_crb-ud-test.conllu\"\n\"UD_210 test set of BAMBARA_CRB.\"\nUD_210_BASQUE_BDT_TRAIN = _UD_210_HOME + \"UD_Basque-BDT/eu_bdt-ud-train.conllu\"\n\"UD_210 train set of BASQUE_BDT.\"\nUD_210_BASQUE_BDT_DEV = _UD_210_HOME + \"UD_Basque-BDT/eu_bdt-ud-dev.conllu\"\n\"UD_210 dev set of BASQUE_BDT.\"\nUD_210_BASQUE_BDT_TEST = _UD_210_HOME + \"UD_Basque-BDT/eu_bdt-ud-test.conllu\"\n\"UD_210 test set of BASQUE_BDT.\"\nUD_210_BEJA_NSC_TEST = _UD_210_HOME + \"UD_Beja-NSC/bej_nsc-ud-test.conllu\"\n\"UD_210 test set of BEJA_NSC.\"\nUD_210_BELARUSIAN_HSE_TRAIN = _UD_210_HOME + \"UD_Belarusian-HSE/be_hse-ud-train.conllu\"\n\"UD_210 train set of BELARUSIAN_HSE.\"\nUD_210_BELARUSIAN_HSE_DEV = _UD_210_HOME + \"UD_Belarusian-HSE/be_hse-ud-dev.conllu\"\n\"UD_210 dev set of BELARUSIAN_HSE.\"\nUD_210_BELARUSIAN_HSE_TEST = _UD_210_HOME + \"UD_Belarusian-HSE/be_hse-ud-test.conllu\"\n\"UD_210 test set of BELARUSIAN_HSE.\"\nUD_210_BENGALI_BRU_TEST = _UD_210_HOME + \"UD_Bengali-BRU/bn_bru-ud-test.conllu\"\n\"UD_210 test set of BENGALI_BRU.\"\nUD_210_BHOJPURI_BHTB_TEST = _UD_210_HOME + \"UD_Bhojpuri-BHTB/bho_bhtb-ud-test.conllu\"\n\"UD_210 test set of BHOJPURI_BHTB.\"\nUD_210_BRETON_KEB_TEST = _UD_210_HOME + \"UD_Breton-KEB/br_keb-ud-test.conllu\"\n\"UD_210 test set of BRETON_KEB.\"\nUD_210_BULGARIAN_BTB_TRAIN = _UD_210_HOME + \"UD_Bulgarian-BTB/bg_btb-ud-train.conllu\"\n\"UD_210 train set of BULGARIAN_BTB.\"\nUD_210_BULGARIAN_BTB_DEV = _UD_210_HOME + \"UD_Bulgarian-BTB/bg_btb-ud-dev.conllu\"\n\"UD_210 dev set of BULGARIAN_BTB.\"\nUD_210_BULGARIAN_BTB_TEST = _UD_210_HOME + \"UD_Bulgarian-BTB/bg_btb-ud-test.conllu\"\n\"UD_210 test set of BULGARIAN_BTB.\"\nUD_210_BURYAT_BDT_TRAIN = _UD_210_HOME + \"UD_Buryat-BDT/bxr_bdt-ud-train.conllu\"\n\"UD_210 train set of BURYAT_BDT.\"\nUD_210_BURYAT_BDT_TEST = _UD_210_HOME + \"UD_Buryat-BDT/bxr_bdt-ud-test.conllu\"\n\"UD_210 test set of BURYAT_BDT.\"\nUD_210_CANTONESE_HK_TEST = _UD_210_HOME + \"UD_Cantonese-HK/yue_hk-ud-test.conllu\"\n\"UD_210 test set of CANTONESE_HK.\"\nUD_210_CATALAN_ANCORA_TRAIN = _UD_210_HOME + \"UD_Catalan-AnCora/ca_ancora-ud-train.conllu\"\n\"UD_210 train set of CATALAN_ANCORA.\"\nUD_210_CATALAN_ANCORA_DEV = _UD_210_HOME + \"UD_Catalan-AnCora/ca_ancora-ud-dev.conllu\"\n\"UD_210 dev set of CATALAN_ANCORA.\"\nUD_210_CATALAN_ANCORA_TEST = _UD_210_HOME + \"UD_Catalan-AnCora/ca_ancora-ud-test.conllu\"\n\"UD_210 test set of CATALAN_ANCORA.\"\nUD_210_CEBUANO_GJA_TEST = _UD_210_HOME + \"UD_Cebuano-GJA/ceb_gja-ud-test.conllu\"\n\"UD_210 test set of CEBUANO_GJA.\"\nUD_210_CHINESE_CFL_TEST = _UD_210_HOME + \"UD_Chinese-CFL/zh_cfl-ud-test.conllu\"\n\"UD_210 test set of CHINESE_CFL.\"\nUD_210_CHINESE_GSD_TRAIN = _UD_210_HOME + \"UD_Chinese-GSD/zh_gsd-ud-train.conllu\"\n\"UD_210 train set of CHINESE_GSD.\"\nUD_210_CHINESE_GSD_DEV = _UD_210_HOME + \"UD_Chinese-GSD/zh_gsd-ud-dev.conllu\"\n\"UD_210 dev set of CHINESE_GSD.\"\nUD_210_CHINESE_GSD_TEST = _UD_210_HOME + \"UD_Chinese-GSD/zh_gsd-ud-test.conllu\"\n\"UD_210 test set of CHINESE_GSD.\"\nUD_210_CHINESE_GSDSIMP_TRAIN = _UD_210_HOME + \"UD_Chinese-GSDSimp/zh_gsdsimp-ud-train.conllu\"\n\"UD_210 train set of CHINESE_GSDSIMP.\"\nUD_210_CHINESE_GSDSIMP_DEV = _UD_210_HOME + \"UD_Chinese-GSDSimp/zh_gsdsimp-ud-dev.conllu\"\n\"UD_210 dev set of CHINESE_GSDSIMP.\"\nUD_210_CHINESE_GSDSIMP_TEST = _UD_210_HOME + \"UD_Chinese-GSDSimp/zh_gsdsimp-ud-test.conllu\"\n\"UD_210 test set of CHINESE_GSDSIMP.\"\nUD_210_CHINESE_HK_TEST = _UD_210_HOME + \"UD_Chinese-HK/zh_hk-ud-test.conllu\"\n\"UD_210 test set of CHINESE_HK.\"\nUD_210_CHINESE_PUD_TEST = _UD_210_HOME + \"UD_Chinese-PUD/zh_pud-ud-test.conllu\"\n\"UD_210 test set of CHINESE_PUD.\"\nUD_210_CHUKCHI_HSE_TEST = _UD_210_HOME + \"UD_Chukchi-HSE/ckt_hse-ud-test.conllu\"\n\"UD_210 test set of CHUKCHI_HSE.\"\nUD_210_CLASSICAL_CHINESE_KYOTO_TRAIN = _UD_210_HOME + \"UD_Classical_Chinese-Kyoto/lzh_kyoto-ud-train.conllu\"\n\"UD_210 train set of CLASSICAL_CHINESE_KYOTO.\"\nUD_210_CLASSICAL_CHINESE_KYOTO_DEV = _UD_210_HOME + \"UD_Classical_Chinese-Kyoto/lzh_kyoto-ud-dev.conllu\"\n\"UD_210 dev set of CLASSICAL_CHINESE_KYOTO.\"\nUD_210_CLASSICAL_CHINESE_KYOTO_TEST = _UD_210_HOME + \"UD_Classical_Chinese-Kyoto/lzh_kyoto-ud-test.conllu\"\n\"UD_210 test set of CLASSICAL_CHINESE_KYOTO.\"\nUD_210_COPTIC_SCRIPTORIUM_TRAIN = _UD_210_HOME + \"UD_Coptic-Scriptorium/cop_scriptorium-ud-train.conllu\"\n\"UD_210 train set of COPTIC_SCRIPTORIUM.\"\nUD_210_COPTIC_SCRIPTORIUM_DEV = _UD_210_HOME + \"UD_Coptic-Scriptorium/cop_scriptorium-ud-dev.conllu\"\n\"UD_210 dev set of COPTIC_SCRIPTORIUM.\"\nUD_210_COPTIC_SCRIPTORIUM_TEST = _UD_210_HOME + \"UD_Coptic-Scriptorium/cop_scriptorium-ud-test.conllu\"\n\"UD_210 test set of COPTIC_SCRIPTORIUM.\"\nUD_210_CROATIAN_SET_TRAIN = _UD_210_HOME + \"UD_Croatian-SET/hr_set-ud-train.conllu\"\n\"UD_210 train set of CROATIAN_SET.\"\nUD_210_CROATIAN_SET_DEV = _UD_210_HOME + \"UD_Croatian-SET/hr_set-ud-dev.conllu\"\n\"UD_210 dev set of CROATIAN_SET.\"\nUD_210_CROATIAN_SET_TEST = _UD_210_HOME + \"UD_Croatian-SET/hr_set-ud-test.conllu\"\n\"UD_210 test set of CROATIAN_SET.\"\nUD_210_CZECH_CAC_TRAIN = _UD_210_HOME + \"UD_Czech-CAC/cs_cac-ud-train.conllu\"\n\"UD_210 train set of CZECH_CAC.\"\nUD_210_CZECH_CAC_DEV = _UD_210_HOME + \"UD_Czech-CAC/cs_cac-ud-dev.conllu\"\n\"UD_210 dev set of CZECH_CAC.\"\nUD_210_CZECH_CAC_TEST = _UD_210_HOME + \"UD_Czech-CAC/cs_cac-ud-test.conllu\"\n\"UD_210 test set of CZECH_CAC.\"\nUD_210_CZECH_CLTT_TRAIN = _UD_210_HOME + \"UD_Czech-CLTT/cs_cltt-ud-train.conllu\"\n\"UD_210 train set of CZECH_CLTT.\"\nUD_210_CZECH_CLTT_DEV = _UD_210_HOME + \"UD_Czech-CLTT/cs_cltt-ud-dev.conllu\"\n\"UD_210 dev set of CZECH_CLTT.\"\nUD_210_CZECH_CLTT_TEST = _UD_210_HOME + \"UD_Czech-CLTT/cs_cltt-ud-test.conllu\"\n\"UD_210 test set of CZECH_CLTT.\"\nUD_210_CZECH_FICTREE_TRAIN = _UD_210_HOME + \"UD_Czech-FicTree/cs_fictree-ud-train.conllu\"\n\"UD_210 train set of CZECH_FICTREE.\"\nUD_210_CZECH_FICTREE_DEV = _UD_210_HOME + \"UD_Czech-FicTree/cs_fictree-ud-dev.conllu\"\n\"UD_210 dev set of CZECH_FICTREE.\"\nUD_210_CZECH_FICTREE_TEST = _UD_210_HOME + \"UD_Czech-FicTree/cs_fictree-ud-test.conllu\"\n\"UD_210 test set of CZECH_FICTREE.\"\nUD_210_CZECH_PDT_TRAIN = _UD_210_HOME + \"UD_Czech-PDT/cs_pdt-ud-train.conllu\"\n\"UD_210 train set of CZECH_PDT.\"\nUD_210_CZECH_PDT_DEV = _UD_210_HOME + \"UD_Czech-PDT/cs_pdt-ud-dev.conllu\"\n\"UD_210 dev set of CZECH_PDT.\"\nUD_210_CZECH_PDT_TEST = _UD_210_HOME + \"UD_Czech-PDT/cs_pdt-ud-test.conllu\"\n\"UD_210 test set of CZECH_PDT.\"\nUD_210_CZECH_PUD_TEST = _UD_210_HOME + \"UD_Czech-PUD/cs_pud-ud-test.conllu\"\n\"UD_210 test set of CZECH_PUD.\"\nUD_210_DANISH_DDT_TRAIN = _UD_210_HOME + \"UD_Danish-DDT/da_ddt-ud-train.conllu\"\n\"UD_210 train set of DANISH_DDT.\"\nUD_210_DANISH_DDT_DEV = _UD_210_HOME + \"UD_Danish-DDT/da_ddt-ud-dev.conllu\"\n\"UD_210 dev set of DANISH_DDT.\"\nUD_210_DANISH_DDT_TEST = _UD_210_HOME + \"UD_Danish-DDT/da_ddt-ud-test.conllu\"\n\"UD_210 test set of DANISH_DDT.\"\nUD_210_DUTCH_ALPINO_TRAIN = _UD_210_HOME + \"UD_Dutch-Alpino/nl_alpino-ud-train.conllu\"\n\"UD_210 train set of DUTCH_ALPINO.\"\nUD_210_DUTCH_ALPINO_DEV = _UD_210_HOME + \"UD_Dutch-Alpino/nl_alpino-ud-dev.conllu\"\n\"UD_210 dev set of DUTCH_ALPINO.\"\nUD_210_DUTCH_ALPINO_TEST = _UD_210_HOME + \"UD_Dutch-Alpino/nl_alpino-ud-test.conllu\"\n\"UD_210 test set of DUTCH_ALPINO.\"\nUD_210_DUTCH_LASSYSMALL_TRAIN = _UD_210_HOME + \"UD_Dutch-LassySmall/nl_lassysmall-ud-train.conllu\"\n\"UD_210 train set of DUTCH_LASSYSMALL.\"\nUD_210_DUTCH_LASSYSMALL_DEV = _UD_210_HOME + \"UD_Dutch-LassySmall/nl_lassysmall-ud-dev.conllu\"\n\"UD_210 dev set of DUTCH_LASSYSMALL.\"\nUD_210_DUTCH_LASSYSMALL_TEST = _UD_210_HOME + \"UD_Dutch-LassySmall/nl_lassysmall-ud-test.conllu\"\n\"UD_210 test set of DUTCH_LASSYSMALL.\"\nUD_210_ENGLISH_ATIS_TRAIN = _UD_210_HOME + \"UD_English-Atis/en_atis-ud-train.conllu\"\n\"UD_210 train set of ENGLISH_ATIS.\"\nUD_210_ENGLISH_ATIS_DEV = _UD_210_HOME + \"UD_English-Atis/en_atis-ud-dev.conllu\"\n\"UD_210 dev set of ENGLISH_ATIS.\"\nUD_210_ENGLISH_ATIS_TEST = _UD_210_HOME + \"UD_English-Atis/en_atis-ud-test.conllu\"\n\"UD_210 test set of ENGLISH_ATIS.\"\nUD_210_ENGLISH_ESL_TRAIN = _UD_210_HOME + \"UD_English-ESL/en_esl-ud-train.conllu\"\n\"UD_210 train set of ENGLISH_ESL.\"\nUD_210_ENGLISH_ESL_DEV = _UD_210_HOME + \"UD_English-ESL/en_esl-ud-dev.conllu\"\n\"UD_210 dev set of ENGLISH_ESL.\"\nUD_210_ENGLISH_ESL_TEST = _UD_210_HOME + \"UD_English-ESL/en_esl-ud-test.conllu\"\n\"UD_210 test set of ENGLISH_ESL.\"\nUD_210_ENGLISH_EWT_TRAIN = _UD_210_HOME + \"UD_English-EWT/en_ewt-ud-train.conllu\"\n\"UD_210 train set of ENGLISH_EWT.\"\nUD_210_ENGLISH_EWT_DEV = _UD_210_HOME + \"UD_English-EWT/en_ewt-ud-dev.conllu\"\n\"UD_210 dev set of ENGLISH_EWT.\"\nUD_210_ENGLISH_EWT_TEST = _UD_210_HOME + \"UD_English-EWT/en_ewt-ud-test.conllu\"\n\"UD_210 test set of ENGLISH_EWT.\"\nUD_210_ENGLISH_GUM_TRAIN = _UD_210_HOME + \"UD_English-GUM/en_gum-ud-train.conllu\"\n\"UD_210 train set of ENGLISH_GUM.\"\nUD_210_ENGLISH_GUM_DEV = _UD_210_HOME + \"UD_English-GUM/en_gum-ud-dev.conllu\"\n\"UD_210 dev set of ENGLISH_GUM.\"\nUD_210_ENGLISH_GUM_TEST = _UD_210_HOME + \"UD_English-GUM/en_gum-ud-test.conllu\"\n\"UD_210 test set of ENGLISH_GUM.\"\nUD_210_ENGLISH_GUMREDDIT_TRAIN = _UD_210_HOME + \"UD_English-GUMReddit/en_gumreddit-ud-train.conllu\"\n\"UD_210 train set of ENGLISH_GUMREDDIT.\"\nUD_210_ENGLISH_GUMREDDIT_DEV = _UD_210_HOME + \"UD_English-GUMReddit/en_gumreddit-ud-dev.conllu\"\n\"UD_210 dev set of ENGLISH_GUMREDDIT.\"\nUD_210_ENGLISH_GUMREDDIT_TEST = _UD_210_HOME + \"UD_English-GUMReddit/en_gumreddit-ud-test.conllu\"\n\"UD_210 test set of ENGLISH_GUMREDDIT.\"\nUD_210_ENGLISH_LINES_TRAIN = _UD_210_HOME + \"UD_English-LinES/en_lines-ud-train.conllu\"\n\"UD_210 train set of ENGLISH_LINES.\"\nUD_210_ENGLISH_LINES_DEV = _UD_210_HOME + \"UD_English-LinES/en_lines-ud-dev.conllu\"\n\"UD_210 dev set of ENGLISH_LINES.\"\nUD_210_ENGLISH_LINES_TEST = _UD_210_HOME + \"UD_English-LinES/en_lines-ud-test.conllu\"\n\"UD_210 test set of ENGLISH_LINES.\"\nUD_210_ENGLISH_PUD_TEST = _UD_210_HOME + \"UD_English-PUD/en_pud-ud-test.conllu\"\n\"UD_210 test set of ENGLISH_PUD.\"\nUD_210_ENGLISH_PARTUT_TRAIN = _UD_210_HOME + \"UD_English-ParTUT/en_partut-ud-train.conllu\"\n\"UD_210 train set of ENGLISH_PARTUT.\"\nUD_210_ENGLISH_PARTUT_DEV = _UD_210_HOME + \"UD_English-ParTUT/en_partut-ud-dev.conllu\"\n\"UD_210 dev set of ENGLISH_PARTUT.\"\nUD_210_ENGLISH_PARTUT_TEST = _UD_210_HOME + \"UD_English-ParTUT/en_partut-ud-test.conllu\"\n\"UD_210 test set of ENGLISH_PARTUT.\"\nUD_210_ENGLISH_PRONOUNS_TEST = _UD_210_HOME + \"UD_English-Pronouns/en_pronouns-ud-test.conllu\"\n\"UD_210 test set of ENGLISH_PRONOUNS.\"\nUD_210_ERZYA_JR_TEST = _UD_210_HOME + \"UD_Erzya-JR/myv_jr-ud-test.conllu\"\n\"UD_210 test set of ERZYA_JR.\"\nUD_210_ESTONIAN_EDT_TRAIN = _UD_210_HOME + \"UD_Estonian-EDT/et_edt-ud-train.conllu\"\n\"UD_210 train set of ESTONIAN_EDT.\"\nUD_210_ESTONIAN_EDT_DEV = _UD_210_HOME + \"UD_Estonian-EDT/et_edt-ud-dev.conllu\"\n\"UD_210 dev set of ESTONIAN_EDT.\"\nUD_210_ESTONIAN_EDT_TEST = _UD_210_HOME + \"UD_Estonian-EDT/et_edt-ud-test.conllu\"\n\"UD_210 test set of ESTONIAN_EDT.\"\nUD_210_ESTONIAN_EWT_TRAIN = _UD_210_HOME + \"UD_Estonian-EWT/et_ewt-ud-train.conllu\"\n\"UD_210 train set of ESTONIAN_EWT.\"\nUD_210_ESTONIAN_EWT_DEV = _UD_210_HOME + \"UD_Estonian-EWT/et_ewt-ud-dev.conllu\"\n\"UD_210 dev set of ESTONIAN_EWT.\"\nUD_210_ESTONIAN_EWT_TEST = _UD_210_HOME + \"UD_Estonian-EWT/et_ewt-ud-test.conllu\"\n\"UD_210 test set of ESTONIAN_EWT.\"\nUD_210_FAROESE_FARPAHC_TRAIN = _UD_210_HOME + \"UD_Faroese-FarPaHC/fo_farpahc-ud-train.conllu\"\n\"UD_210 train set of FAROESE_FARPAHC.\"\nUD_210_FAROESE_FARPAHC_DEV = _UD_210_HOME + \"UD_Faroese-FarPaHC/fo_farpahc-ud-dev.conllu\"\n\"UD_210 dev set of FAROESE_FARPAHC.\"\nUD_210_FAROESE_FARPAHC_TEST = _UD_210_HOME + \"UD_Faroese-FarPaHC/fo_farpahc-ud-test.conllu\"\n\"UD_210 test set of FAROESE_FARPAHC.\"\nUD_210_FAROESE_OFT_TEST = _UD_210_HOME + \"UD_Faroese-OFT/fo_oft-ud-test.conllu\"\n\"UD_210 test set of FAROESE_OFT.\"\nUD_210_FINNISH_FTB_TRAIN = _UD_210_HOME + \"UD_Finnish-FTB/fi_ftb-ud-train.conllu\"\n\"UD_210 train set of FINNISH_FTB.\"\nUD_210_FINNISH_FTB_DEV = _UD_210_HOME + \"UD_Finnish-FTB/fi_ftb-ud-dev.conllu\"\n\"UD_210 dev set of FINNISH_FTB.\"\nUD_210_FINNISH_FTB_TEST = _UD_210_HOME + \"UD_Finnish-FTB/fi_ftb-ud-test.conllu\"\n\"UD_210 test set of FINNISH_FTB.\"\nUD_210_FINNISH_OOD_TEST = _UD_210_HOME + \"UD_Finnish-OOD/fi_ood-ud-test.conllu\"\n\"UD_210 test set of FINNISH_OOD.\"\nUD_210_FINNISH_PUD_TEST = _UD_210_HOME + \"UD_Finnish-PUD/fi_pud-ud-test.conllu\"\n\"UD_210 test set of FINNISH_PUD.\"\nUD_210_FINNISH_TDT_TRAIN = _UD_210_HOME + \"UD_Finnish-TDT/fi_tdt-ud-train.conllu\"\n\"UD_210 train set of FINNISH_TDT.\"\nUD_210_FINNISH_TDT_DEV = _UD_210_HOME + \"UD_Finnish-TDT/fi_tdt-ud-dev.conllu\"\n\"UD_210 dev set of FINNISH_TDT.\"\nUD_210_FINNISH_TDT_TEST = _UD_210_HOME + \"UD_Finnish-TDT/fi_tdt-ud-test.conllu\"\n\"UD_210 test set of FINNISH_TDT.\"\nUD_210_FRENCH_FQB_TEST = _UD_210_HOME + \"UD_French-FQB/fr_fqb-ud-test.conllu\"\n\"UD_210 test set of FRENCH_FQB.\"\nUD_210_FRENCH_FTB_TRAIN = _UD_210_HOME + \"UD_French-FTB/fr_ftb-ud-train.conllu\"\n\"UD_210 train set of FRENCH_FTB.\"\nUD_210_FRENCH_FTB_DEV = _UD_210_HOME + \"UD_French-FTB/fr_ftb-ud-dev.conllu\"\n\"UD_210 dev set of FRENCH_FTB.\"\nUD_210_FRENCH_FTB_TEST = _UD_210_HOME + \"UD_French-FTB/fr_ftb-ud-test.conllu\"\n\"UD_210 test set of FRENCH_FTB.\"\nUD_210_FRENCH_GSD_TRAIN = _UD_210_HOME + \"UD_French-GSD/fr_gsd-ud-train.conllu\"\n\"UD_210 train set of FRENCH_GSD.\"\nUD_210_FRENCH_GSD_DEV = _UD_210_HOME + \"UD_French-GSD/fr_gsd-ud-dev.conllu\"\n\"UD_210 dev set of FRENCH_GSD.\"\nUD_210_FRENCH_GSD_TEST = _UD_210_HOME + \"UD_French-GSD/fr_gsd-ud-test.conllu\"\n\"UD_210 test set of FRENCH_GSD.\"\nUD_210_FRENCH_PUD_TEST = _UD_210_HOME + \"UD_French-PUD/fr_pud-ud-test.conllu\"\n\"UD_210 test set of FRENCH_PUD.\"\nUD_210_FRENCH_PARTUT_TRAIN = _UD_210_HOME + \"UD_French-ParTUT/fr_partut-ud-train.conllu\"\n\"UD_210 train set of FRENCH_PARTUT.\"\nUD_210_FRENCH_PARTUT_DEV = _UD_210_HOME + \"UD_French-ParTUT/fr_partut-ud-dev.conllu\"\n\"UD_210 dev set of FRENCH_PARTUT.\"\nUD_210_FRENCH_PARTUT_TEST = _UD_210_HOME + \"UD_French-ParTUT/fr_partut-ud-test.conllu\"\n\"UD_210 test set of FRENCH_PARTUT.\"\nUD_210_FRENCH_PARISSTORIES_TRAIN = _UD_210_HOME + \"UD_French-ParisStories/fr_parisstories-ud-train.conllu\"\n\"UD_210 train set of FRENCH_PARISSTORIES.\"\nUD_210_FRENCH_PARISSTORIES_TEST = _UD_210_HOME + \"UD_French-ParisStories/fr_parisstories-ud-test.conllu\"\n\"UD_210 test set of FRENCH_PARISSTORIES.\"\nUD_210_FRENCH_RHAPSODIE_TRAIN = _UD_210_HOME + \"UD_French-Rhapsodie/fr_rhapsodie-ud-train.conllu\"\n\"UD_210 train set of FRENCH_RHAPSODIE.\"\nUD_210_FRENCH_RHAPSODIE_DEV = _UD_210_HOME + \"UD_French-Rhapsodie/fr_rhapsodie-ud-dev.conllu\"\n\"UD_210 dev set of FRENCH_RHAPSODIE.\"\nUD_210_FRENCH_RHAPSODIE_TEST = _UD_210_HOME + \"UD_French-Rhapsodie/fr_rhapsodie-ud-test.conllu\"\n\"UD_210 test set of FRENCH_RHAPSODIE.\"\nUD_210_FRENCH_SEQUOIA_TRAIN = _UD_210_HOME + \"UD_French-Sequoia/fr_sequoia-ud-train.conllu\"\n\"UD_210 train set of FRENCH_SEQUOIA.\"\nUD_210_FRENCH_SEQUOIA_DEV = _UD_210_HOME + \"UD_French-Sequoia/fr_sequoia-ud-dev.conllu\"\n\"UD_210 dev set of FRENCH_SEQUOIA.\"\nUD_210_FRENCH_SEQUOIA_TEST = _UD_210_HOME + \"UD_French-Sequoia/fr_sequoia-ud-test.conllu\"\n\"UD_210 test set of FRENCH_SEQUOIA.\"\nUD_210_FRISIAN_DUTCH_FAME_TEST = _UD_210_HOME + \"UD_Frisian_Dutch-Fame/qfn_fame-ud-test.conllu\"\n\"UD_210 test set of FRISIAN_DUTCH_FAME.\"\nUD_210_GALICIAN_CTG_TRAIN = _UD_210_HOME + \"UD_Galician-CTG/gl_ctg-ud-train.conllu\"\n\"UD_210 train set of GALICIAN_CTG.\"\nUD_210_GALICIAN_CTG_DEV = _UD_210_HOME + \"UD_Galician-CTG/gl_ctg-ud-dev.conllu\"\n\"UD_210 dev set of GALICIAN_CTG.\"\nUD_210_GALICIAN_CTG_TEST = _UD_210_HOME + \"UD_Galician-CTG/gl_ctg-ud-test.conllu\"\n\"UD_210 test set of GALICIAN_CTG.\"\nUD_210_GALICIAN_TREEGAL_TRAIN = _UD_210_HOME + \"UD_Galician-TreeGal/gl_treegal-ud-train.conllu\"\n\"UD_210 train set of GALICIAN_TREEGAL.\"\nUD_210_GALICIAN_TREEGAL_TEST = _UD_210_HOME + \"UD_Galician-TreeGal/gl_treegal-ud-test.conllu\"\n\"UD_210 test set of GALICIAN_TREEGAL.\"\nUD_210_GERMAN_GSD_TRAIN = _UD_210_HOME + \"UD_German-GSD/de_gsd-ud-train.conllu\"\n\"UD_210 train set of GERMAN_GSD.\"\nUD_210_GERMAN_GSD_DEV = _UD_210_HOME + \"UD_German-GSD/de_gsd-ud-dev.conllu\"\n\"UD_210 dev set of GERMAN_GSD.\"\nUD_210_GERMAN_GSD_TEST = _UD_210_HOME + \"UD_German-GSD/de_gsd-ud-test.conllu\"\n\"UD_210 test set of GERMAN_GSD.\"\nUD_210_GERMAN_HDT_TRAIN = _UD_210_HOME + \"UD_German-HDT/de_hdt-ud-train.conllu\"\n\"UD_210 train set of GERMAN_HDT.\"\nUD_210_GERMAN_HDT_DEV = _UD_210_HOME + \"UD_German-HDT/de_hdt-ud-dev.conllu\"\n\"UD_210 dev set of GERMAN_HDT.\"\nUD_210_GERMAN_HDT_TEST = _UD_210_HOME + \"UD_German-HDT/de_hdt-ud-test.conllu\"\n\"UD_210 test set of GERMAN_HDT.\"\nUD_210_GERMAN_LIT_TEST = _UD_210_HOME + \"UD_German-LIT/de_lit-ud-test.conllu\"\n\"UD_210 test set of GERMAN_LIT.\"\nUD_210_GERMAN_PUD_TEST = _UD_210_HOME + \"UD_German-PUD/de_pud-ud-test.conllu\"\n\"UD_210 test set of GERMAN_PUD.\"\nUD_210_GOTHIC_PROIEL_TRAIN = _UD_210_HOME + \"UD_Gothic-PROIEL/got_proiel-ud-train.conllu\"\n\"UD_210 train set of GOTHIC_PROIEL.\"\nUD_210_GOTHIC_PROIEL_DEV = _UD_210_HOME + \"UD_Gothic-PROIEL/got_proiel-ud-dev.conllu\"\n\"UD_210 dev set of GOTHIC_PROIEL.\"\nUD_210_GOTHIC_PROIEL_TEST = _UD_210_HOME + \"UD_Gothic-PROIEL/got_proiel-ud-test.conllu\"\n\"UD_210 test set of GOTHIC_PROIEL.\"\nUD_210_GREEK_GDT_TRAIN = _UD_210_HOME + \"UD_Greek-GDT/el_gdt-ud-train.conllu\"\n\"UD_210 train set of GREEK_GDT.\"\nUD_210_GREEK_GDT_DEV = _UD_210_HOME + \"UD_Greek-GDT/el_gdt-ud-dev.conllu\"\n\"UD_210 dev set of GREEK_GDT.\"\nUD_210_GREEK_GDT_TEST = _UD_210_HOME + \"UD_Greek-GDT/el_gdt-ud-test.conllu\"\n\"UD_210 test set of GREEK_GDT.\"\nUD_210_GUAJAJARA_TUDET_TEST = _UD_210_HOME + \"UD_Guajajara-TuDeT/gub_tudet-ud-test.conllu\"\n\"UD_210 test set of GUAJAJARA_TUDET.\"\nUD_210_GUARANI_OLDTUDET_TEST = _UD_210_HOME + \"UD_Guarani-OldTuDeT/gn_oldtudet-ud-test.conllu\"\n\"UD_210 test set of GUARANI_OLDTUDET.\"\nUD_210_HEBREW_HTB_TRAIN = _UD_210_HOME + \"UD_Hebrew-HTB/he_htb-ud-train.conllu\"\n\"UD_210 train set of HEBREW_HTB.\"\nUD_210_HEBREW_HTB_DEV = _UD_210_HOME + \"UD_Hebrew-HTB/he_htb-ud-dev.conllu\"\n\"UD_210 dev set of HEBREW_HTB.\"\nUD_210_HEBREW_HTB_TEST = _UD_210_HOME + \"UD_Hebrew-HTB/he_htb-ud-test.conllu\"\n\"UD_210 test set of HEBREW_HTB.\"\nUD_210_HEBREW_IAHLTWIKI_TRAIN = _UD_210_HOME + \"UD_Hebrew-IAHLTwiki/he_iahltwiki-ud-train.conllu\"\n\"UD_210 train set of HEBREW_IAHLTWIKI.\"\nUD_210_HEBREW_IAHLTWIKI_DEV = _UD_210_HOME + \"UD_Hebrew-IAHLTwiki/he_iahltwiki-ud-dev.conllu\"\n\"UD_210 dev set of HEBREW_IAHLTWIKI.\"\nUD_210_HEBREW_IAHLTWIKI_TEST = _UD_210_HOME + \"UD_Hebrew-IAHLTwiki/he_iahltwiki-ud-test.conllu\"\n\"UD_210 test set of HEBREW_IAHLTWIKI.\"\nUD_210_HINDI_HDTB_TRAIN = _UD_210_HOME + \"UD_Hindi-HDTB/hi_hdtb-ud-train.conllu\"\n\"UD_210 train set of HINDI_HDTB.\"\nUD_210_HINDI_HDTB_DEV = _UD_210_HOME + \"UD_Hindi-HDTB/hi_hdtb-ud-dev.conllu\"\n\"UD_210 dev set of HINDI_HDTB.\"\nUD_210_HINDI_HDTB_TEST = _UD_210_HOME + \"UD_Hindi-HDTB/hi_hdtb-ud-test.conllu\"\n\"UD_210 test set of HINDI_HDTB.\"\nUD_210_HINDI_PUD_TEST = _UD_210_HOME + \"UD_Hindi-PUD/hi_pud-ud-test.conllu\"\n\"UD_210 test set of HINDI_PUD.\"\nUD_210_HINDI_ENGLISH_HIENCS_TRAIN = _UD_210_HOME + \"UD_Hindi_English-HIENCS/qhe_hiencs-ud-train.conllu\"\n\"UD_210 train set of HINDI_ENGLISH_HIENCS.\"\nUD_210_HINDI_ENGLISH_HIENCS_DEV = _UD_210_HOME + \"UD_Hindi_English-HIENCS/qhe_hiencs-ud-dev.conllu\"\n\"UD_210 dev set of HINDI_ENGLISH_HIENCS.\"\nUD_210_HINDI_ENGLISH_HIENCS_TEST = _UD_210_HOME + \"UD_Hindi_English-HIENCS/qhe_hiencs-ud-test.conllu\"\n\"UD_210 test set of HINDI_ENGLISH_HIENCS.\"\nUD_210_HITTITE_HITTB_TEST = _UD_210_HOME + \"UD_Hittite-HitTB/hit_hittb-ud-test.conllu\"\n\"UD_210 test set of HITTITE_HITTB.\"\nUD_210_HUNGARIAN_SZEGED_TRAIN = _UD_210_HOME + \"UD_Hungarian-Szeged/hu_szeged-ud-train.conllu\"\n\"UD_210 train set of HUNGARIAN_SZEGED.\"\nUD_210_HUNGARIAN_SZEGED_DEV = _UD_210_HOME + \"UD_Hungarian-Szeged/hu_szeged-ud-dev.conllu\"\n\"UD_210 dev set of HUNGARIAN_SZEGED.\"\nUD_210_HUNGARIAN_SZEGED_TEST = _UD_210_HOME + \"UD_Hungarian-Szeged/hu_szeged-ud-test.conllu\"\n\"UD_210 test set of HUNGARIAN_SZEGED.\"\nUD_210_ICELANDIC_ICEPAHC_TRAIN = _UD_210_HOME + \"UD_Icelandic-IcePaHC/is_icepahc-ud-train.conllu\"\n\"UD_210 train set of ICELANDIC_ICEPAHC.\"\nUD_210_ICELANDIC_ICEPAHC_DEV = _UD_210_HOME + \"UD_Icelandic-IcePaHC/is_icepahc-ud-dev.conllu\"\n\"UD_210 dev set of ICELANDIC_ICEPAHC.\"\nUD_210_ICELANDIC_ICEPAHC_TEST = _UD_210_HOME + \"UD_Icelandic-IcePaHC/is_icepahc-ud-test.conllu\"\n\"UD_210 test set of ICELANDIC_ICEPAHC.\"\nUD_210_ICELANDIC_MODERN_TRAIN = _UD_210_HOME + \"UD_Icelandic-Modern/is_modern-ud-train.conllu\"\n\"UD_210 train set of ICELANDIC_MODERN.\"\nUD_210_ICELANDIC_MODERN_DEV = _UD_210_HOME + \"UD_Icelandic-Modern/is_modern-ud-dev.conllu\"\n\"UD_210 dev set of ICELANDIC_MODERN.\"\nUD_210_ICELANDIC_MODERN_TEST = _UD_210_HOME + \"UD_Icelandic-Modern/is_modern-ud-test.conllu\"\n\"UD_210 test set of ICELANDIC_MODERN.\"\nUD_210_ICELANDIC_PUD_TEST = _UD_210_HOME + \"UD_Icelandic-PUD/is_pud-ud-test.conllu\"\n\"UD_210 test set of ICELANDIC_PUD.\"\nUD_210_INDONESIAN_CSUI_TRAIN = _UD_210_HOME + \"UD_Indonesian-CSUI/id_csui-ud-train.conllu\"\n\"UD_210 train set of INDONESIAN_CSUI.\"\nUD_210_INDONESIAN_CSUI_TEST = _UD_210_HOME + \"UD_Indonesian-CSUI/id_csui-ud-test.conllu\"\n\"UD_210 test set of INDONESIAN_CSUI.\"\nUD_210_INDONESIAN_GSD_TRAIN = _UD_210_HOME + \"UD_Indonesian-GSD/id_gsd-ud-train.conllu\"\n\"UD_210 train set of INDONESIAN_GSD.\"\nUD_210_INDONESIAN_GSD_DEV = _UD_210_HOME + \"UD_Indonesian-GSD/id_gsd-ud-dev.conllu\"\n\"UD_210 dev set of INDONESIAN_GSD.\"\nUD_210_INDONESIAN_GSD_TEST = _UD_210_HOME + \"UD_Indonesian-GSD/id_gsd-ud-test.conllu\"\n\"UD_210 test set of INDONESIAN_GSD.\"\nUD_210_INDONESIAN_PUD_TEST = _UD_210_HOME + \"UD_Indonesian-PUD/id_pud-ud-test.conllu\"\n\"UD_210 test set of INDONESIAN_PUD.\"\nUD_210_IRISH_IDT_TRAIN = _UD_210_HOME + \"UD_Irish-IDT/ga_idt-ud-train.conllu\"\n\"UD_210 train set of IRISH_IDT.\"\nUD_210_IRISH_IDT_DEV = _UD_210_HOME + \"UD_Irish-IDT/ga_idt-ud-dev.conllu\"\n\"UD_210 dev set of IRISH_IDT.\"\nUD_210_IRISH_IDT_TEST = _UD_210_HOME + \"UD_Irish-IDT/ga_idt-ud-test.conllu\"\n\"UD_210 test set of IRISH_IDT.\"\nUD_210_IRISH_TWITTIRISH_TEST = _UD_210_HOME + \"UD_Irish-TwittIrish/ga_twittirish-ud-test.conllu\"\n\"UD_210 test set of IRISH_TWITTIRISH.\"\nUD_210_ITALIAN_ISDT_TRAIN = _UD_210_HOME + \"UD_Italian-ISDT/it_isdt-ud-train.conllu\"\n\"UD_210 train set of ITALIAN_ISDT.\"\nUD_210_ITALIAN_ISDT_DEV = _UD_210_HOME + \"UD_Italian-ISDT/it_isdt-ud-dev.conllu\"\n\"UD_210 dev set of ITALIAN_ISDT.\"\nUD_210_ITALIAN_ISDT_TEST = _UD_210_HOME + \"UD_Italian-ISDT/it_isdt-ud-test.conllu\"\n\"UD_210 test set of ITALIAN_ISDT.\"\nUD_210_ITALIAN_MARKIT_TRAIN = _UD_210_HOME + \"UD_Italian-MarkIT/it_markit-ud-train.conllu\"\n\"UD_210 train set of ITALIAN_MARKIT.\"\nUD_210_ITALIAN_MARKIT_DEV = _UD_210_HOME + \"UD_Italian-MarkIT/it_markit-ud-dev.conllu\"\n\"UD_210 dev set of ITALIAN_MARKIT.\"\nUD_210_ITALIAN_MARKIT_TEST = _UD_210_HOME + \"UD_Italian-MarkIT/it_markit-ud-test.conllu\"\n\"UD_210 test set of ITALIAN_MARKIT.\"\nUD_210_ITALIAN_PUD_TEST = _UD_210_HOME + \"UD_Italian-PUD/it_pud-ud-test.conllu\"\n\"UD_210 test set of ITALIAN_PUD.\"\nUD_210_ITALIAN_PARTUT_TRAIN = _UD_210_HOME + \"UD_Italian-ParTUT/it_partut-ud-train.conllu\"\n\"UD_210 train set of ITALIAN_PARTUT.\"\nUD_210_ITALIAN_PARTUT_DEV = _UD_210_HOME + \"UD_Italian-ParTUT/it_partut-ud-dev.conllu\"\n\"UD_210 dev set of ITALIAN_PARTUT.\"\nUD_210_ITALIAN_PARTUT_TEST = _UD_210_HOME + \"UD_Italian-ParTUT/it_partut-ud-test.conllu\"\n\"UD_210 test set of ITALIAN_PARTUT.\"\nUD_210_ITALIAN_POSTWITA_TRAIN = _UD_210_HOME + \"UD_Italian-PoSTWITA/it_postwita-ud-train.conllu\"\n\"UD_210 train set of ITALIAN_POSTWITA.\"\nUD_210_ITALIAN_POSTWITA_DEV = _UD_210_HOME + \"UD_Italian-PoSTWITA/it_postwita-ud-dev.conllu\"\n\"UD_210 dev set of ITALIAN_POSTWITA.\"\nUD_210_ITALIAN_POSTWITA_TEST = _UD_210_HOME + \"UD_Italian-PoSTWITA/it_postwita-ud-test.conllu\"\n\"UD_210 test set of ITALIAN_POSTWITA.\"\nUD_210_ITALIAN_TWITTIRO_TRAIN = _UD_210_HOME + \"UD_Italian-TWITTIRO/it_twittiro-ud-train.conllu\"\n\"UD_210 train set of ITALIAN_TWITTIRO.\"\nUD_210_ITALIAN_TWITTIRO_DEV = _UD_210_HOME + \"UD_Italian-TWITTIRO/it_twittiro-ud-dev.conllu\"\n\"UD_210 dev set of ITALIAN_TWITTIRO.\"\nUD_210_ITALIAN_TWITTIRO_TEST = _UD_210_HOME + \"UD_Italian-TWITTIRO/it_twittiro-ud-test.conllu\"\n\"UD_210 test set of ITALIAN_TWITTIRO.\"\nUD_210_ITALIAN_VIT_TRAIN = _UD_210_HOME + \"UD_Italian-VIT/it_vit-ud-train.conllu\"\n\"UD_210 train set of ITALIAN_VIT.\"\nUD_210_ITALIAN_VIT_DEV = _UD_210_HOME + \"UD_Italian-VIT/it_vit-ud-dev.conllu\"\n\"UD_210 dev set of ITALIAN_VIT.\"\nUD_210_ITALIAN_VIT_TEST = _UD_210_HOME + \"UD_Italian-VIT/it_vit-ud-test.conllu\"\n\"UD_210 test set of ITALIAN_VIT.\"\nUD_210_ITALIAN_VALICO_TEST = _UD_210_HOME + \"UD_Italian-Valico/it_valico-ud-test.conllu\"\n\"UD_210 test set of ITALIAN_VALICO.\"\nUD_210_JAPANESE_BCCWJ_TRAIN = _UD_210_HOME + \"UD_Japanese-BCCWJ/ja_bccwj-ud-train.conllu\"\n\"UD_210 train set of JAPANESE_BCCWJ.\"\nUD_210_JAPANESE_BCCWJ_DEV = _UD_210_HOME + \"UD_Japanese-BCCWJ/ja_bccwj-ud-dev.conllu\"\n\"UD_210 dev set of JAPANESE_BCCWJ.\"\nUD_210_JAPANESE_BCCWJ_TEST = _UD_210_HOME + \"UD_Japanese-BCCWJ/ja_bccwj-ud-test.conllu\"\n\"UD_210 test set of JAPANESE_BCCWJ.\"\nUD_210_JAPANESE_BCCWJLUW_TRAIN = _UD_210_HOME + \"UD_Japanese-BCCWJLUW/ja_bccwjluw-ud-train.conllu\"\n\"UD_210 train set of JAPANESE_BCCWJLUW.\"\nUD_210_JAPANESE_BCCWJLUW_DEV = _UD_210_HOME + \"UD_Japanese-BCCWJLUW/ja_bccwjluw-ud-dev.conllu\"\n\"UD_210 dev set of JAPANESE_BCCWJLUW.\"\nUD_210_JAPANESE_BCCWJLUW_TEST = _UD_210_HOME + \"UD_Japanese-BCCWJLUW/ja_bccwjluw-ud-test.conllu\"\n\"UD_210 test set of JAPANESE_BCCWJLUW.\"\nUD_210_JAPANESE_GSD_TRAIN = _UD_210_HOME + \"UD_Japanese-GSD/ja_gsd-ud-train.conllu\"\n\"UD_210 train set of JAPANESE_GSD.\"\nUD_210_JAPANESE_GSD_DEV = _UD_210_HOME + \"UD_Japanese-GSD/ja_gsd-ud-dev.conllu\"\n\"UD_210 dev set of JAPANESE_GSD.\"\nUD_210_JAPANESE_GSD_TEST = _UD_210_HOME + \"UD_Japanese-GSD/ja_gsd-ud-test.conllu\"\n\"UD_210 test set of JAPANESE_GSD.\"\nUD_210_JAPANESE_GSDLUW_TRAIN = _UD_210_HOME + \"UD_Japanese-GSDLUW/ja_gsdluw-ud-train.conllu\"\n\"UD_210 train set of JAPANESE_GSDLUW.\"\nUD_210_JAPANESE_GSDLUW_DEV = _UD_210_HOME + \"UD_Japanese-GSDLUW/ja_gsdluw-ud-dev.conllu\"\n\"UD_210 dev set of JAPANESE_GSDLUW.\"\nUD_210_JAPANESE_GSDLUW_TEST = _UD_210_HOME + \"UD_Japanese-GSDLUW/ja_gsdluw-ud-test.conllu\"\n\"UD_210 test set of JAPANESE_GSDLUW.\"\nUD_210_JAPANESE_MODERN_TEST = _UD_210_HOME + \"UD_Japanese-Modern/ja_modern-ud-test.conllu\"\n\"UD_210 test set of JAPANESE_MODERN.\"\nUD_210_JAPANESE_PUD_TEST = _UD_210_HOME + \"UD_Japanese-PUD/ja_pud-ud-test.conllu\"\n\"UD_210 test set of JAPANESE_PUD.\"\nUD_210_JAPANESE_PUDLUW_TEST = _UD_210_HOME + \"UD_Japanese-PUDLUW/ja_pudluw-ud-test.conllu\"\n\"UD_210 test set of JAPANESE_PUDLUW.\"\nUD_210_JAVANESE_CSUI_TEST = _UD_210_HOME + \"UD_Javanese-CSUI/jv_csui-ud-test.conllu\"\n\"UD_210 test set of JAVANESE_CSUI.\"\nUD_210_KAAPOR_TUDET_TEST = _UD_210_HOME + \"UD_Kaapor-TuDeT/urb_tudet-ud-test.conllu\"\n\"UD_210 test set of KAAPOR_TUDET.\"\nUD_210_KANGRI_KDTB_TEST = _UD_210_HOME + \"UD_Kangri-KDTB/xnr_kdtb-ud-test.conllu\"\n\"UD_210 test set of KANGRI_KDTB.\"\nUD_210_KARELIAN_KKPP_TEST = _UD_210_HOME + \"UD_Karelian-KKPP/krl_kkpp-ud-test.conllu\"\n\"UD_210 test set of KARELIAN_KKPP.\"\nUD_210_KARO_TUDET_TEST = _UD_210_HOME + \"UD_Karo-TuDeT/arr_tudet-ud-test.conllu\"\n\"UD_210 test set of KARO_TUDET.\"\nUD_210_KAZAKH_KTB_TRAIN = _UD_210_HOME + \"UD_Kazakh-KTB/kk_ktb-ud-train.conllu\"\n\"UD_210 train set of KAZAKH_KTB.\"\nUD_210_KAZAKH_KTB_TEST = _UD_210_HOME + \"UD_Kazakh-KTB/kk_ktb-ud-test.conllu\"\n\"UD_210 test set of KAZAKH_KTB.\"\nUD_210_KHUNSARI_AHA_TEST = _UD_210_HOME + \"UD_Khunsari-AHA/kfm_aha-ud-test.conllu\"\n\"UD_210 test set of KHUNSARI_AHA.\"\nUD_210_KICHE_IU_TEST = _UD_210_HOME + \"UD_Kiche-IU/quc_iu-ud-test.conllu\"\n\"UD_210 test set of KICHE_IU.\"\nUD_210_KOMI_PERMYAK_UH_TEST = _UD_210_HOME + \"UD_Komi_Permyak-UH/koi_uh-ud-test.conllu\"\n\"UD_210 test set of KOMI_PERMYAK_UH.\"\nUD_210_KOMI_ZYRIAN_IKDP_TEST = _UD_210_HOME + \"UD_Komi_Zyrian-IKDP/kpv_ikdp-ud-test.conllu\"\n\"UD_210 test set of KOMI_ZYRIAN_IKDP.\"\nUD_210_KOMI_ZYRIAN_LATTICE_TEST = _UD_210_HOME + \"UD_Komi_Zyrian-Lattice/kpv_lattice-ud-test.conllu\"\n\"UD_210 test set of KOMI_ZYRIAN_LATTICE.\"\nUD_210_KOREAN_GSD_TRAIN = _UD_210_HOME + \"UD_Korean-GSD/ko_gsd-ud-train.conllu\"\n\"UD_210 train set of KOREAN_GSD.\"\nUD_210_KOREAN_GSD_DEV = _UD_210_HOME + \"UD_Korean-GSD/ko_gsd-ud-dev.conllu\"\n\"UD_210 dev set of KOREAN_GSD.\"\nUD_210_KOREAN_GSD_TEST = _UD_210_HOME + \"UD_Korean-GSD/ko_gsd-ud-test.conllu\"\n\"UD_210 test set of KOREAN_GSD.\"\nUD_210_KOREAN_KAIST_TRAIN = _UD_210_HOME + \"UD_Korean-Kaist/ko_kaist-ud-train.conllu\"\n\"UD_210 train set of KOREAN_KAIST.\"\nUD_210_KOREAN_KAIST_DEV = _UD_210_HOME + \"UD_Korean-Kaist/ko_kaist-ud-dev.conllu\"\n\"UD_210 dev set of KOREAN_KAIST.\"\nUD_210_KOREAN_KAIST_TEST = _UD_210_HOME + \"UD_Korean-Kaist/ko_kaist-ud-test.conllu\"\n\"UD_210 test set of KOREAN_KAIST.\"\nUD_210_KOREAN_PUD_TEST = _UD_210_HOME + \"UD_Korean-PUD/ko_pud-ud-test.conllu\"\n\"UD_210 test set of KOREAN_PUD.\"\nUD_210_KURMANJI_MG_TRAIN = _UD_210_HOME + \"UD_Kurmanji-MG/kmr_mg-ud-train.conllu\"\n\"UD_210 train set of KURMANJI_MG.\"\nUD_210_KURMANJI_MG_TEST = _UD_210_HOME + \"UD_Kurmanji-MG/kmr_mg-ud-test.conllu\"\n\"UD_210 test set of KURMANJI_MG.\"\nUD_210_LATIN_ITTB_TRAIN = _UD_210_HOME + \"UD_Latin-ITTB/la_ittb-ud-train.conllu\"\n\"UD_210 train set of LATIN_ITTB.\"\nUD_210_LATIN_ITTB_DEV = _UD_210_HOME + \"UD_Latin-ITTB/la_ittb-ud-dev.conllu\"\n\"UD_210 dev set of LATIN_ITTB.\"\nUD_210_LATIN_ITTB_TEST = _UD_210_HOME + \"UD_Latin-ITTB/la_ittb-ud-test.conllu\"\n\"UD_210 test set of LATIN_ITTB.\"\nUD_210_LATIN_LLCT_TRAIN = _UD_210_HOME + \"UD_Latin-LLCT/la_llct-ud-train.conllu\"\n\"UD_210 train set of LATIN_LLCT.\"\nUD_210_LATIN_LLCT_DEV = _UD_210_HOME + \"UD_Latin-LLCT/la_llct-ud-dev.conllu\"\n\"UD_210 dev set of LATIN_LLCT.\"\nUD_210_LATIN_LLCT_TEST = _UD_210_HOME + \"UD_Latin-LLCT/la_llct-ud-test.conllu\"\n\"UD_210 test set of LATIN_LLCT.\"\nUD_210_LATIN_PROIEL_TRAIN = _UD_210_HOME + \"UD_Latin-PROIEL/la_proiel-ud-train.conllu\"\n\"UD_210 train set of LATIN_PROIEL.\"\nUD_210_LATIN_PROIEL_DEV = _UD_210_HOME + \"UD_Latin-PROIEL/la_proiel-ud-dev.conllu\"\n\"UD_210 dev set of LATIN_PROIEL.\"\nUD_210_LATIN_PROIEL_TEST = _UD_210_HOME + \"UD_Latin-PROIEL/la_proiel-ud-test.conllu\"\n\"UD_210 test set of LATIN_PROIEL.\"\nUD_210_LATIN_PERSEUS_TRAIN = _UD_210_HOME + \"UD_Latin-Perseus/la_perseus-ud-train.conllu\"\n\"UD_210 train set of LATIN_PERSEUS.\"\nUD_210_LATIN_PERSEUS_TEST = _UD_210_HOME + \"UD_Latin-Perseus/la_perseus-ud-test.conllu\"\n\"UD_210 test set of LATIN_PERSEUS.\"\nUD_210_LATIN_UDANTE_TRAIN = _UD_210_HOME + \"UD_Latin-UDante/la_udante-ud-train.conllu\"\n\"UD_210 train set of LATIN_UDANTE.\"\nUD_210_LATIN_UDANTE_DEV = _UD_210_HOME + \"UD_Latin-UDante/la_udante-ud-dev.conllu\"\n\"UD_210 dev set of LATIN_UDANTE.\"\nUD_210_LATIN_UDANTE_TEST = _UD_210_HOME + \"UD_Latin-UDante/la_udante-ud-test.conllu\"\n\"UD_210 test set of LATIN_UDANTE.\"\nUD_210_LATVIAN_LVTB_TRAIN = _UD_210_HOME + \"UD_Latvian-LVTB/lv_lvtb-ud-train.conllu\"\n\"UD_210 train set of LATVIAN_LVTB.\"\nUD_210_LATVIAN_LVTB_DEV = _UD_210_HOME + \"UD_Latvian-LVTB/lv_lvtb-ud-dev.conllu\"\n\"UD_210 dev set of LATVIAN_LVTB.\"\nUD_210_LATVIAN_LVTB_TEST = _UD_210_HOME + \"UD_Latvian-LVTB/lv_lvtb-ud-test.conllu\"\n\"UD_210 test set of LATVIAN_LVTB.\"\nUD_210_LIGURIAN_GLT_TRAIN = _UD_210_HOME + \"UD_Ligurian-GLT/lij_glt-ud-train.conllu\"\n\"UD_210 train set of LIGURIAN_GLT.\"\nUD_210_LIGURIAN_GLT_TEST = _UD_210_HOME + \"UD_Ligurian-GLT/lij_glt-ud-test.conllu\"\n\"UD_210 test set of LIGURIAN_GLT.\"\nUD_210_LITHUANIAN_ALKSNIS_TRAIN = _UD_210_HOME + \"UD_Lithuanian-ALKSNIS/lt_alksnis-ud-train.conllu\"\n\"UD_210 train set of LITHUANIAN_ALKSNIS.\"\nUD_210_LITHUANIAN_ALKSNIS_DEV = _UD_210_HOME + \"UD_Lithuanian-ALKSNIS/lt_alksnis-ud-dev.conllu\"\n\"UD_210 dev set of LITHUANIAN_ALKSNIS.\"\nUD_210_LITHUANIAN_ALKSNIS_TEST = _UD_210_HOME + \"UD_Lithuanian-ALKSNIS/lt_alksnis-ud-test.conllu\"\n\"UD_210 test set of LITHUANIAN_ALKSNIS.\"\nUD_210_LITHUANIAN_HSE_TRAIN = _UD_210_HOME + \"UD_Lithuanian-HSE/lt_hse-ud-train.conllu\"\n\"UD_210 train set of LITHUANIAN_HSE.\"\nUD_210_LITHUANIAN_HSE_DEV = _UD_210_HOME + \"UD_Lithuanian-HSE/lt_hse-ud-dev.conllu\"\n\"UD_210 dev set of LITHUANIAN_HSE.\"\nUD_210_LITHUANIAN_HSE_TEST = _UD_210_HOME + \"UD_Lithuanian-HSE/lt_hse-ud-test.conllu\"\n\"UD_210 test set of LITHUANIAN_HSE.\"\nUD_210_LIVVI_KKPP_TRAIN = _UD_210_HOME + \"UD_Livvi-KKPP/olo_kkpp-ud-train.conllu\"\n\"UD_210 train set of LIVVI_KKPP.\"\nUD_210_LIVVI_KKPP_TEST = _UD_210_HOME + \"UD_Livvi-KKPP/olo_kkpp-ud-test.conllu\"\n\"UD_210 test set of LIVVI_KKPP.\"\nUD_210_LOW_SAXON_LSDC_TEST = _UD_210_HOME + \"UD_Low_Saxon-LSDC/nds_lsdc-ud-test.conllu\"\n\"UD_210 test set of LOW_SAXON_LSDC.\"\nUD_210_MADI_JARAWARA_TEST = _UD_210_HOME + \"UD_Madi-Jarawara/jaa_jarawara-ud-test.conllu\"\n\"UD_210 test set of MADI_JARAWARA.\"\nUD_210_MAKURAP_TUDET_TEST = _UD_210_HOME + \"UD_Makurap-TuDeT/mpu_tudet-ud-test.conllu\"\n\"UD_210 test set of MAKURAP_TUDET.\"\nUD_210_MALTESE_MUDT_TRAIN = _UD_210_HOME + \"UD_Maltese-MUDT/mt_mudt-ud-train.conllu\"\n\"UD_210 train set of MALTESE_MUDT.\"\nUD_210_MALTESE_MUDT_DEV = _UD_210_HOME + \"UD_Maltese-MUDT/mt_mudt-ud-dev.conllu\"\n\"UD_210 dev set of MALTESE_MUDT.\"\nUD_210_MALTESE_MUDT_TEST = _UD_210_HOME + \"UD_Maltese-MUDT/mt_mudt-ud-test.conllu\"\n\"UD_210 test set of MALTESE_MUDT.\"\nUD_210_MANX_CADHAN_TEST = _UD_210_HOME + \"UD_Manx-Cadhan/gv_cadhan-ud-test.conllu\"\n\"UD_210 test set of MANX_CADHAN.\"\nUD_210_MARATHI_UFAL_TRAIN = _UD_210_HOME + \"UD_Marathi-UFAL/mr_ufal-ud-train.conllu\"\n\"UD_210 train set of MARATHI_UFAL.\"\nUD_210_MARATHI_UFAL_DEV = _UD_210_HOME + \"UD_Marathi-UFAL/mr_ufal-ud-dev.conllu\"\n\"UD_210 dev set of MARATHI_UFAL.\"\nUD_210_MARATHI_UFAL_TEST = _UD_210_HOME + \"UD_Marathi-UFAL/mr_ufal-ud-test.conllu\"\n\"UD_210 test set of MARATHI_UFAL.\"\nUD_210_MBYA_GUARANI_DOOLEY_TEST = _UD_210_HOME + \"UD_Mbya_Guarani-Dooley/gun_dooley-ud-test.conllu\"\n\"UD_210 test set of MBYA_GUARANI_DOOLEY.\"\nUD_210_MBYA_GUARANI_THOMAS_TEST = _UD_210_HOME + \"UD_Mbya_Guarani-Thomas/gun_thomas-ud-test.conllu\"\n\"UD_210 test set of MBYA_GUARANI_THOMAS.\"\nUD_210_MOKSHA_JR_TEST = _UD_210_HOME + \"UD_Moksha-JR/mdf_jr-ud-test.conllu\"\n\"UD_210 test set of MOKSHA_JR.\"\nUD_210_MUNDURUKU_TUDET_TEST = _UD_210_HOME + \"UD_Munduruku-TuDeT/myu_tudet-ud-test.conllu\"\n\"UD_210 test set of MUNDURUKU_TUDET.\"\nUD_210_NAIJA_NSC_TRAIN = _UD_210_HOME + \"UD_Naija-NSC/pcm_nsc-ud-train.conllu\"\n\"UD_210 train set of NAIJA_NSC.\"\nUD_210_NAIJA_NSC_DEV = _UD_210_HOME + \"UD_Naija-NSC/pcm_nsc-ud-dev.conllu\"\n\"UD_210 dev set of NAIJA_NSC.\"\nUD_210_NAIJA_NSC_TEST = _UD_210_HOME + \"UD_Naija-NSC/pcm_nsc-ud-test.conllu\"\n\"UD_210 test set of NAIJA_NSC.\"\nUD_210_NAYINI_AHA_TEST = _UD_210_HOME + \"UD_Nayini-AHA/nyq_aha-ud-test.conllu\"\n\"UD_210 test set of NAYINI_AHA.\"\nUD_210_NEAPOLITAN_RB_TEST = _UD_210_HOME + \"UD_Neapolitan-RB/nap_rb-ud-test.conllu\"\n\"UD_210 test set of NEAPOLITAN_RB.\"\nUD_210_NORTH_SAMI_GIELLA_TRAIN = _UD_210_HOME + \"UD_North_Sami-Giella/sme_giella-ud-train.conllu\"\n\"UD_210 train set of NORTH_SAMI_GIELLA.\"\nUD_210_NORTH_SAMI_GIELLA_TEST = _UD_210_HOME + \"UD_North_Sami-Giella/sme_giella-ud-test.conllu\"\n\"UD_210 test set of NORTH_SAMI_GIELLA.\"\nUD_210_NORWEGIAN_BOKMAAL_TRAIN = _UD_210_HOME + \"UD_Norwegian-Bokmaal/no_bokmaal-ud-train.conllu\"\n\"UD_210 train set of NORWEGIAN_BOKMAAL.\"\nUD_210_NORWEGIAN_BOKMAAL_DEV = _UD_210_HOME + \"UD_Norwegian-Bokmaal/no_bokmaal-ud-dev.conllu\"\n\"UD_210 dev set of NORWEGIAN_BOKMAAL.\"\nUD_210_NORWEGIAN_BOKMAAL_TEST = _UD_210_HOME + \"UD_Norwegian-Bokmaal/no_bokmaal-ud-test.conllu\"\n\"UD_210 test set of NORWEGIAN_BOKMAAL.\"\nUD_210_NORWEGIAN_NYNORSK_TRAIN = _UD_210_HOME + \"UD_Norwegian-Nynorsk/no_nynorsk-ud-train.conllu\"\n\"UD_210 train set of NORWEGIAN_NYNORSK.\"\nUD_210_NORWEGIAN_NYNORSK_DEV = _UD_210_HOME + \"UD_Norwegian-Nynorsk/no_nynorsk-ud-dev.conllu\"\n\"UD_210 dev set of NORWEGIAN_NYNORSK.\"\nUD_210_NORWEGIAN_NYNORSK_TEST = _UD_210_HOME + \"UD_Norwegian-Nynorsk/no_nynorsk-ud-test.conllu\"\n\"UD_210 test set of NORWEGIAN_NYNORSK.\"\nUD_210_NORWEGIAN_NYNORSKLIA_TRAIN = _UD_210_HOME + \"UD_Norwegian-NynorskLIA/no_nynorsklia-ud-train.conllu\"\n\"UD_210 train set of NORWEGIAN_NYNORSKLIA.\"\nUD_210_NORWEGIAN_NYNORSKLIA_DEV = _UD_210_HOME + \"UD_Norwegian-NynorskLIA/no_nynorsklia-ud-dev.conllu\"\n\"UD_210 dev set of NORWEGIAN_NYNORSKLIA.\"\nUD_210_NORWEGIAN_NYNORSKLIA_TEST = _UD_210_HOME + \"UD_Norwegian-NynorskLIA/no_nynorsklia-ud-test.conllu\"\n\"UD_210 test set of NORWEGIAN_NYNORSKLIA.\"\nUD_210_OLD_CHURCH_SLAVONIC_PROIEL_TRAIN = _UD_210_HOME + \"UD_Old_Church_Slavonic-PROIEL/cu_proiel-ud-train.conllu\"\n\"UD_210 train set of OLD_CHURCH_SLAVONIC_PROIEL.\"\nUD_210_OLD_CHURCH_SLAVONIC_PROIEL_DEV = _UD_210_HOME + \"UD_Old_Church_Slavonic-PROIEL/cu_proiel-ud-dev.conllu\"\n\"UD_210 dev set of OLD_CHURCH_SLAVONIC_PROIEL.\"\nUD_210_OLD_CHURCH_SLAVONIC_PROIEL_TEST = _UD_210_HOME + \"UD_Old_Church_Slavonic-PROIEL/cu_proiel-ud-test.conllu\"\n\"UD_210 test set of OLD_CHURCH_SLAVONIC_PROIEL.\"\nUD_210_OLD_EAST_SLAVIC_BIRCHBARK_TRAIN = _UD_210_HOME + \"UD_Old_East_Slavic-Birchbark/orv_birchbark-ud-train.conllu\"\n\"UD_210 train set of OLD_EAST_SLAVIC_BIRCHBARK.\"\nUD_210_OLD_EAST_SLAVIC_BIRCHBARK_DEV = _UD_210_HOME + \"UD_Old_East_Slavic-Birchbark/orv_birchbark-ud-dev.conllu\"\n\"UD_210 dev set of OLD_EAST_SLAVIC_BIRCHBARK.\"\nUD_210_OLD_EAST_SLAVIC_BIRCHBARK_TEST = _UD_210_HOME + \"UD_Old_East_Slavic-Birchbark/orv_birchbark-ud-test.conllu\"\n\"UD_210 test set of OLD_EAST_SLAVIC_BIRCHBARK.\"\nUD_210_OLD_EAST_SLAVIC_RNC_TRAIN = _UD_210_HOME + \"UD_Old_East_Slavic-RNC/orv_rnc-ud-train.conllu\"\n\"UD_210 train set of OLD_EAST_SLAVIC_RNC.\"\nUD_210_OLD_EAST_SLAVIC_RNC_TEST = _UD_210_HOME + \"UD_Old_East_Slavic-RNC/orv_rnc-ud-test.conllu\"\n\"UD_210 test set of OLD_EAST_SLAVIC_RNC.\"\nUD_210_OLD_EAST_SLAVIC_TOROT_TRAIN = _UD_210_HOME + \"UD_Old_East_Slavic-TOROT/orv_torot-ud-train.conllu\"\n\"UD_210 train set of OLD_EAST_SLAVIC_TOROT.\"\nUD_210_OLD_EAST_SLAVIC_TOROT_DEV = _UD_210_HOME + \"UD_Old_East_Slavic-TOROT/orv_torot-ud-dev.conllu\"\n\"UD_210 dev set of OLD_EAST_SLAVIC_TOROT.\"\nUD_210_OLD_EAST_SLAVIC_TOROT_TEST = _UD_210_HOME + \"UD_Old_East_Slavic-TOROT/orv_torot-ud-test.conllu\"\n\"UD_210 test set of OLD_EAST_SLAVIC_TOROT.\"\nUD_210_OLD_FRENCH_SRCMF_TRAIN = _UD_210_HOME + \"UD_Old_French-SRCMF/fro_srcmf-ud-train.conllu\"\n\"UD_210 train set of OLD_FRENCH_SRCMF.\"\nUD_210_OLD_FRENCH_SRCMF_DEV = _UD_210_HOME + \"UD_Old_French-SRCMF/fro_srcmf-ud-dev.conllu\"\n\"UD_210 dev set of OLD_FRENCH_SRCMF.\"\nUD_210_OLD_FRENCH_SRCMF_TEST = _UD_210_HOME + \"UD_Old_French-SRCMF/fro_srcmf-ud-test.conllu\"\n\"UD_210 test set of OLD_FRENCH_SRCMF.\"\nUD_210_OLD_TURKISH_TONQQ_TEST = _UD_210_HOME + \"UD_Old_Turkish-Tonqq/otk_tonqq-ud-test.conllu\"\n\"UD_210 test set of OLD_TURKISH_TONQQ.\"\nUD_210_PERSIAN_PERDT_TRAIN = _UD_210_HOME + \"UD_Persian-PerDT/fa_perdt-ud-train.conllu\"\n\"UD_210 train set of PERSIAN_PERDT.\"\nUD_210_PERSIAN_PERDT_DEV = _UD_210_HOME + \"UD_Persian-PerDT/fa_perdt-ud-dev.conllu\"\n\"UD_210 dev set of PERSIAN_PERDT.\"\nUD_210_PERSIAN_PERDT_TEST = _UD_210_HOME + \"UD_Persian-PerDT/fa_perdt-ud-test.conllu\"\n\"UD_210 test set of PERSIAN_PERDT.\"\nUD_210_PERSIAN_SERAJI_TRAIN = _UD_210_HOME + \"UD_Persian-Seraji/fa_seraji-ud-train.conllu\"\n\"UD_210 train set of PERSIAN_SERAJI.\"\nUD_210_PERSIAN_SERAJI_DEV = _UD_210_HOME + \"UD_Persian-Seraji/fa_seraji-ud-dev.conllu\"\n\"UD_210 dev set of PERSIAN_SERAJI.\"\nUD_210_PERSIAN_SERAJI_TEST = _UD_210_HOME + \"UD_Persian-Seraji/fa_seraji-ud-test.conllu\"\n\"UD_210 test set of PERSIAN_SERAJI.\"\nUD_210_POLISH_LFG_TRAIN = _UD_210_HOME + \"UD_Polish-LFG/pl_lfg-ud-train.conllu\"\n\"UD_210 train set of POLISH_LFG.\"\nUD_210_POLISH_LFG_DEV = _UD_210_HOME + \"UD_Polish-LFG/pl_lfg-ud-dev.conllu\"\n\"UD_210 dev set of POLISH_LFG.\"\nUD_210_POLISH_LFG_TEST = _UD_210_HOME + \"UD_Polish-LFG/pl_lfg-ud-test.conllu\"\n\"UD_210 test set of POLISH_LFG.\"\nUD_210_POLISH_PDB_TRAIN = _UD_210_HOME + \"UD_Polish-PDB/pl_pdb-ud-train.conllu\"\n\"UD_210 train set of POLISH_PDB.\"\nUD_210_POLISH_PDB_DEV = _UD_210_HOME + \"UD_Polish-PDB/pl_pdb-ud-dev.conllu\"\n\"UD_210 dev set of POLISH_PDB.\"\nUD_210_POLISH_PDB_TEST = _UD_210_HOME + \"UD_Polish-PDB/pl_pdb-ud-test.conllu\"\n\"UD_210 test set of POLISH_PDB.\"\nUD_210_POLISH_PUD_TEST = _UD_210_HOME + \"UD_Polish-PUD/pl_pud-ud-test.conllu\"\n\"UD_210 test set of POLISH_PUD.\"\nUD_210_POMAK_PHILOTIS_TRAIN = _UD_210_HOME + \"UD_Pomak-Philotis/qpm_philotis-ud-train.conllu\"\n\"UD_210 train set of POMAK_PHILOTIS.\"\nUD_210_POMAK_PHILOTIS_DEV = _UD_210_HOME + \"UD_Pomak-Philotis/qpm_philotis-ud-dev.conllu\"\n\"UD_210 dev set of POMAK_PHILOTIS.\"\nUD_210_POMAK_PHILOTIS_TEST = _UD_210_HOME + \"UD_Pomak-Philotis/qpm_philotis-ud-test.conllu\"\n\"UD_210 test set of POMAK_PHILOTIS.\"\nUD_210_PORTUGUESE_BOSQUE_TRAIN = _UD_210_HOME + \"UD_Portuguese-Bosque/pt_bosque-ud-train.conllu\"\n\"UD_210 train set of PORTUGUESE_BOSQUE.\"\nUD_210_PORTUGUESE_BOSQUE_DEV = _UD_210_HOME + \"UD_Portuguese-Bosque/pt_bosque-ud-dev.conllu\"\n\"UD_210 dev set of PORTUGUESE_BOSQUE.\"\nUD_210_PORTUGUESE_BOSQUE_TEST = _UD_210_HOME + \"UD_Portuguese-Bosque/pt_bosque-ud-test.conllu\"\n\"UD_210 test set of PORTUGUESE_BOSQUE.\"\nUD_210_PORTUGUESE_GSD_TRAIN = _UD_210_HOME + \"UD_Portuguese-GSD/pt_gsd-ud-train.conllu\"\n\"UD_210 train set of PORTUGUESE_GSD.\"\nUD_210_PORTUGUESE_GSD_DEV = _UD_210_HOME + \"UD_Portuguese-GSD/pt_gsd-ud-dev.conllu\"\n\"UD_210 dev set of PORTUGUESE_GSD.\"\nUD_210_PORTUGUESE_GSD_TEST = _UD_210_HOME + \"UD_Portuguese-GSD/pt_gsd-ud-test.conllu\"\n\"UD_210 test set of PORTUGUESE_GSD.\"\nUD_210_PORTUGUESE_PUD_TEST = _UD_210_HOME + \"UD_Portuguese-PUD/pt_pud-ud-test.conllu\"\n\"UD_210 test set of PORTUGUESE_PUD.\"\nUD_210_ROMANIAN_ART_TEST = _UD_210_HOME + \"UD_Romanian-ArT/ro_art-ud-test.conllu\"\n\"UD_210 test set of ROMANIAN_ART.\"\nUD_210_ROMANIAN_NONSTANDARD_TRAIN = _UD_210_HOME + \"UD_Romanian-Nonstandard/ro_nonstandard-ud-train.conllu\"\n\"UD_210 train set of ROMANIAN_NONSTANDARD.\"\nUD_210_ROMANIAN_NONSTANDARD_DEV = _UD_210_HOME + \"UD_Romanian-Nonstandard/ro_nonstandard-ud-dev.conllu\"\n\"UD_210 dev set of ROMANIAN_NONSTANDARD.\"\nUD_210_ROMANIAN_NONSTANDARD_TEST = _UD_210_HOME + \"UD_Romanian-Nonstandard/ro_nonstandard-ud-test.conllu\"\n\"UD_210 test set of ROMANIAN_NONSTANDARD.\"\nUD_210_ROMANIAN_RRT_TRAIN = _UD_210_HOME + \"UD_Romanian-RRT/ro_rrt-ud-train.conllu\"\n\"UD_210 train set of ROMANIAN_RRT.\"\nUD_210_ROMANIAN_RRT_DEV = _UD_210_HOME + \"UD_Romanian-RRT/ro_rrt-ud-dev.conllu\"\n\"UD_210 dev set of ROMANIAN_RRT.\"\nUD_210_ROMANIAN_RRT_TEST = _UD_210_HOME + \"UD_Romanian-RRT/ro_rrt-ud-test.conllu\"\n\"UD_210 test set of ROMANIAN_RRT.\"\nUD_210_ROMANIAN_SIMONERO_TRAIN = _UD_210_HOME + \"UD_Romanian-SiMoNERo/ro_simonero-ud-train.conllu\"\n\"UD_210 train set of ROMANIAN_SIMONERO.\"\nUD_210_ROMANIAN_SIMONERO_DEV = _UD_210_HOME + \"UD_Romanian-SiMoNERo/ro_simonero-ud-dev.conllu\"\n\"UD_210 dev set of ROMANIAN_SIMONERO.\"\nUD_210_ROMANIAN_SIMONERO_TEST = _UD_210_HOME + \"UD_Romanian-SiMoNERo/ro_simonero-ud-test.conllu\"\n\"UD_210 test set of ROMANIAN_SIMONERO.\"\nUD_210_RUSSIAN_GSD_TRAIN = _UD_210_HOME + \"UD_Russian-GSD/ru_gsd-ud-train.conllu\"\n\"UD_210 train set of RUSSIAN_GSD.\"\nUD_210_RUSSIAN_GSD_DEV = _UD_210_HOME + \"UD_Russian-GSD/ru_gsd-ud-dev.conllu\"\n\"UD_210 dev set of RUSSIAN_GSD.\"\nUD_210_RUSSIAN_GSD_TEST = _UD_210_HOME + \"UD_Russian-GSD/ru_gsd-ud-test.conllu\"\n\"UD_210 test set of RUSSIAN_GSD.\"\nUD_210_RUSSIAN_PUD_TEST = _UD_210_HOME + \"UD_Russian-PUD/ru_pud-ud-test.conllu\"\n\"UD_210 test set of RUSSIAN_PUD.\"\nUD_210_RUSSIAN_SYNTAGRUS_TRAIN = _UD_210_HOME + \"UD_Russian-SynTagRus/ru_syntagrus-ud-train.conllu\"\n\"UD_210 train set of RUSSIAN_SYNTAGRUS.\"\nUD_210_RUSSIAN_SYNTAGRUS_DEV = _UD_210_HOME + \"UD_Russian-SynTagRus/ru_syntagrus-ud-dev.conllu\"\n\"UD_210 dev set of RUSSIAN_SYNTAGRUS.\"\nUD_210_RUSSIAN_SYNTAGRUS_TEST = _UD_210_HOME + \"UD_Russian-SynTagRus/ru_syntagrus-ud-test.conllu\"\n\"UD_210 test set of RUSSIAN_SYNTAGRUS.\"\nUD_210_RUSSIAN_TAIGA_TRAIN = _UD_210_HOME + \"UD_Russian-Taiga/ru_taiga-ud-train.conllu\"\n\"UD_210 train set of RUSSIAN_TAIGA.\"\nUD_210_RUSSIAN_TAIGA_DEV = _UD_210_HOME + \"UD_Russian-Taiga/ru_taiga-ud-dev.conllu\"\n\"UD_210 dev set of RUSSIAN_TAIGA.\"\nUD_210_RUSSIAN_TAIGA_TEST = _UD_210_HOME + \"UD_Russian-Taiga/ru_taiga-ud-test.conllu\"\n\"UD_210 test set of RUSSIAN_TAIGA.\"\nUD_210_SANSKRIT_UFAL_TEST = _UD_210_HOME + \"UD_Sanskrit-UFAL/sa_ufal-ud-test.conllu\"\n\"UD_210 test set of SANSKRIT_UFAL.\"\nUD_210_SANSKRIT_VEDIC_TRAIN = _UD_210_HOME + \"UD_Sanskrit-Vedic/sa_vedic-ud-train.conllu\"\n\"UD_210 train set of SANSKRIT_VEDIC.\"\nUD_210_SANSKRIT_VEDIC_TEST = _UD_210_HOME + \"UD_Sanskrit-Vedic/sa_vedic-ud-test.conllu\"\n\"UD_210 test set of SANSKRIT_VEDIC.\"\nUD_210_SCOTTISH_GAELIC_ARCOSG_TRAIN = _UD_210_HOME + \"UD_Scottish_Gaelic-ARCOSG/gd_arcosg-ud-train.conllu\"\n\"UD_210 train set of SCOTTISH_GAELIC_ARCOSG.\"\nUD_210_SCOTTISH_GAELIC_ARCOSG_DEV = _UD_210_HOME + \"UD_Scottish_Gaelic-ARCOSG/gd_arcosg-ud-dev.conllu\"\n\"UD_210 dev set of SCOTTISH_GAELIC_ARCOSG.\"\nUD_210_SCOTTISH_GAELIC_ARCOSG_TEST = _UD_210_HOME + \"UD_Scottish_Gaelic-ARCOSG/gd_arcosg-ud-test.conllu\"\n\"UD_210 test set of SCOTTISH_GAELIC_ARCOSG.\"\nUD_210_SERBIAN_SET_TRAIN = _UD_210_HOME + \"UD_Serbian-SET/sr_set-ud-train.conllu\"\n\"UD_210 train set of SERBIAN_SET.\"\nUD_210_SERBIAN_SET_DEV = _UD_210_HOME + \"UD_Serbian-SET/sr_set-ud-dev.conllu\"\n\"UD_210 dev set of SERBIAN_SET.\"\nUD_210_SERBIAN_SET_TEST = _UD_210_HOME + \"UD_Serbian-SET/sr_set-ud-test.conllu\"\n\"UD_210 test set of SERBIAN_SET.\"\nUD_210_SKOLT_SAMI_GIELLAGAS_TEST = _UD_210_HOME + \"UD_Skolt_Sami-Giellagas/sms_giellagas-ud-test.conllu\"\n\"UD_210 test set of SKOLT_SAMI_GIELLAGAS.\"\nUD_210_SLOVAK_SNK_TRAIN = _UD_210_HOME + \"UD_Slovak-SNK/sk_snk-ud-train.conllu\"\n\"UD_210 train set of SLOVAK_SNK.\"\nUD_210_SLOVAK_SNK_DEV = _UD_210_HOME + \"UD_Slovak-SNK/sk_snk-ud-dev.conllu\"\n\"UD_210 dev set of SLOVAK_SNK.\"\nUD_210_SLOVAK_SNK_TEST = _UD_210_HOME + \"UD_Slovak-SNK/sk_snk-ud-test.conllu\"\n\"UD_210 test set of SLOVAK_SNK.\"\nUD_210_SLOVENIAN_SSJ_TRAIN = _UD_210_HOME + \"UD_Slovenian-SSJ/sl_ssj-ud-train.conllu\"\n\"UD_210 train set of SLOVENIAN_SSJ.\"\nUD_210_SLOVENIAN_SSJ_DEV = _UD_210_HOME + \"UD_Slovenian-SSJ/sl_ssj-ud-dev.conllu\"\n\"UD_210 dev set of SLOVENIAN_SSJ.\"\nUD_210_SLOVENIAN_SSJ_TEST = _UD_210_HOME + \"UD_Slovenian-SSJ/sl_ssj-ud-test.conllu\"\n\"UD_210 test set of SLOVENIAN_SSJ.\"\nUD_210_SLOVENIAN_SST_TRAIN = _UD_210_HOME + \"UD_Slovenian-SST/sl_sst-ud-train.conllu\"\n\"UD_210 train set of SLOVENIAN_SST.\"\nUD_210_SLOVENIAN_SST_TEST = _UD_210_HOME + \"UD_Slovenian-SST/sl_sst-ud-test.conllu\"\n\"UD_210 test set of SLOVENIAN_SST.\"\nUD_210_SOI_AHA_TEST = _UD_210_HOME + \"UD_Soi-AHA/soj_aha-ud-test.conllu\"\n\"UD_210 test set of SOI_AHA.\"\nUD_210_SOUTH_LEVANTINE_ARABIC_MADAR_TEST = _UD_210_HOME + \"UD_South_Levantine_Arabic-MADAR/ajp_madar-ud-test.conllu\"\n\"UD_210 test set of SOUTH_LEVANTINE_ARABIC_MADAR.\"\nUD_210_SPANISH_ANCORA_TRAIN = _UD_210_HOME + \"UD_Spanish-AnCora/es_ancora-ud-train.conllu\"\n\"UD_210 train set of SPANISH_ANCORA.\"\nUD_210_SPANISH_ANCORA_DEV = _UD_210_HOME + \"UD_Spanish-AnCora/es_ancora-ud-dev.conllu\"\n\"UD_210 dev set of SPANISH_ANCORA.\"\nUD_210_SPANISH_ANCORA_TEST = _UD_210_HOME + \"UD_Spanish-AnCora/es_ancora-ud-test.conllu\"\n\"UD_210 test set of SPANISH_ANCORA.\"\nUD_210_SPANISH_GSD_TRAIN = _UD_210_HOME + \"UD_Spanish-GSD/es_gsd-ud-train.conllu\"\n\"UD_210 train set of SPANISH_GSD.\"\nUD_210_SPANISH_GSD_DEV = _UD_210_HOME + \"UD_Spanish-GSD/es_gsd-ud-dev.conllu\"\n\"UD_210 dev set of SPANISH_GSD.\"\nUD_210_SPANISH_GSD_TEST = _UD_210_HOME + \"UD_Spanish-GSD/es_gsd-ud-test.conllu\"\n\"UD_210 test set of SPANISH_GSD.\"\nUD_210_SPANISH_PUD_TEST = _UD_210_HOME + \"UD_Spanish-PUD/es_pud-ud-test.conllu\"\n\"UD_210 test set of SPANISH_PUD.\"\nUD_210_SWEDISH_LINES_TRAIN = _UD_210_HOME + \"UD_Swedish-LinES/sv_lines-ud-train.conllu\"\n\"UD_210 train set of SWEDISH_LINES.\"\nUD_210_SWEDISH_LINES_DEV = _UD_210_HOME + \"UD_Swedish-LinES/sv_lines-ud-dev.conllu\"\n\"UD_210 dev set of SWEDISH_LINES.\"\nUD_210_SWEDISH_LINES_TEST = _UD_210_HOME + \"UD_Swedish-LinES/sv_lines-ud-test.conllu\"\n\"UD_210 test set of SWEDISH_LINES.\"\nUD_210_SWEDISH_PUD_TEST = _UD_210_HOME + \"UD_Swedish-PUD/sv_pud-ud-test.conllu\"\n\"UD_210 test set of SWEDISH_PUD.\"\nUD_210_SWEDISH_TALBANKEN_TRAIN = _UD_210_HOME + \"UD_Swedish-Talbanken/sv_talbanken-ud-train.conllu\"\n\"UD_210 train set of SWEDISH_TALBANKEN.\"\nUD_210_SWEDISH_TALBANKEN_DEV = _UD_210_HOME + \"UD_Swedish-Talbanken/sv_talbanken-ud-dev.conllu\"\n\"UD_210 dev set of SWEDISH_TALBANKEN.\"\nUD_210_SWEDISH_TALBANKEN_TEST = _UD_210_HOME + \"UD_Swedish-Talbanken/sv_talbanken-ud-test.conllu\"\n\"UD_210 test set of SWEDISH_TALBANKEN.\"\nUD_210_SWEDISH_SIGN_LANGUAGE_SSLC_TRAIN = _UD_210_HOME + \"UD_Swedish_Sign_Language-SSLC/swl_sslc-ud-train.conllu\"\n\"UD_210 train set of SWEDISH_SIGN_LANGUAGE_SSLC.\"\nUD_210_SWEDISH_SIGN_LANGUAGE_SSLC_DEV = _UD_210_HOME + \"UD_Swedish_Sign_Language-SSLC/swl_sslc-ud-dev.conllu\"\n\"UD_210 dev set of SWEDISH_SIGN_LANGUAGE_SSLC.\"\nUD_210_SWEDISH_SIGN_LANGUAGE_SSLC_TEST = _UD_210_HOME + \"UD_Swedish_Sign_Language-SSLC/swl_sslc-ud-test.conllu\"\n\"UD_210 test set of SWEDISH_SIGN_LANGUAGE_SSLC.\"\nUD_210_SWISS_GERMAN_UZH_TEST = _UD_210_HOME + \"UD_Swiss_German-UZH/gsw_uzh-ud-test.conllu\"\n\"UD_210 test set of SWISS_GERMAN_UZH.\"\nUD_210_TAGALOG_TRG_TEST = _UD_210_HOME + \"UD_Tagalog-TRG/tl_trg-ud-test.conllu\"\n\"UD_210 test set of TAGALOG_TRG.\"\nUD_210_TAGALOG_UGNAYAN_TEST = _UD_210_HOME + \"UD_Tagalog-Ugnayan/tl_ugnayan-ud-test.conllu\"\n\"UD_210 test set of TAGALOG_UGNAYAN.\"\nUD_210_TAMIL_MWTT_TEST = _UD_210_HOME + \"UD_Tamil-MWTT/ta_mwtt-ud-test.conllu\"\n\"UD_210 test set of TAMIL_MWTT.\"\nUD_210_TAMIL_TTB_TRAIN = _UD_210_HOME + \"UD_Tamil-TTB/ta_ttb-ud-train.conllu\"\n\"UD_210 train set of TAMIL_TTB.\"\nUD_210_TAMIL_TTB_DEV = _UD_210_HOME + \"UD_Tamil-TTB/ta_ttb-ud-dev.conllu\"\n\"UD_210 dev set of TAMIL_TTB.\"\nUD_210_TAMIL_TTB_TEST = _UD_210_HOME + \"UD_Tamil-TTB/ta_ttb-ud-test.conllu\"\n\"UD_210 test set of TAMIL_TTB.\"\nUD_210_TATAR_NMCTT_TEST = _UD_210_HOME + \"UD_Tatar-NMCTT/tt_nmctt-ud-test.conllu\"\n\"UD_210 test set of TATAR_NMCTT.\"\nUD_210_TEKO_TUDET_TEST = _UD_210_HOME + \"UD_Teko-TuDeT/eme_tudet-ud-test.conllu\"\n\"UD_210 test set of TEKO_TUDET.\"\nUD_210_TELUGU_MTG_TRAIN = _UD_210_HOME + \"UD_Telugu-MTG/te_mtg-ud-train.conllu\"\n\"UD_210 train set of TELUGU_MTG.\"\nUD_210_TELUGU_MTG_DEV = _UD_210_HOME + \"UD_Telugu-MTG/te_mtg-ud-dev.conllu\"\n\"UD_210 dev set of TELUGU_MTG.\"\nUD_210_TELUGU_MTG_TEST = _UD_210_HOME + \"UD_Telugu-MTG/te_mtg-ud-test.conllu\"\n\"UD_210 test set of TELUGU_MTG.\"\nUD_210_THAI_PUD_TEST = _UD_210_HOME + \"UD_Thai-PUD/th_pud-ud-test.conllu\"\n\"UD_210 test set of THAI_PUD.\"\nUD_210_TUPINAMBA_TUDET_TEST = _UD_210_HOME + \"UD_Tupinamba-TuDeT/tpn_tudet-ud-test.conllu\"\n\"UD_210 test set of TUPINAMBA_TUDET.\"\nUD_210_TURKISH_ATIS_TRAIN = _UD_210_HOME + \"UD_Turkish-Atis/tr_atis-ud-train.conllu\"\n\"UD_210 train set of TURKISH_ATIS.\"\nUD_210_TURKISH_ATIS_DEV = _UD_210_HOME + \"UD_Turkish-Atis/tr_atis-ud-dev.conllu\"\n\"UD_210 dev set of TURKISH_ATIS.\"\nUD_210_TURKISH_ATIS_TEST = _UD_210_HOME + \"UD_Turkish-Atis/tr_atis-ud-test.conllu\"\n\"UD_210 test set of TURKISH_ATIS.\"\nUD_210_TURKISH_BOUN_TRAIN = _UD_210_HOME + \"UD_Turkish-BOUN/tr_boun-ud-train.conllu\"\n\"UD_210 train set of TURKISH_BOUN.\"\nUD_210_TURKISH_BOUN_DEV = _UD_210_HOME + \"UD_Turkish-BOUN/tr_boun-ud-dev.conllu\"\n\"UD_210 dev set of TURKISH_BOUN.\"\nUD_210_TURKISH_BOUN_TEST = _UD_210_HOME + \"UD_Turkish-BOUN/tr_boun-ud-test.conllu\"\n\"UD_210 test set of TURKISH_BOUN.\"\nUD_210_TURKISH_FRAMENET_TRAIN = _UD_210_HOME + \"UD_Turkish-FrameNet/tr_framenet-ud-train.conllu\"\n\"UD_210 train set of TURKISH_FRAMENET.\"\nUD_210_TURKISH_FRAMENET_DEV = _UD_210_HOME + \"UD_Turkish-FrameNet/tr_framenet-ud-dev.conllu\"\n\"UD_210 dev set of TURKISH_FRAMENET.\"\nUD_210_TURKISH_FRAMENET_TEST = _UD_210_HOME + \"UD_Turkish-FrameNet/tr_framenet-ud-test.conllu\"\n\"UD_210 test set of TURKISH_FRAMENET.\"\nUD_210_TURKISH_GB_TEST = _UD_210_HOME + \"UD_Turkish-GB/tr_gb-ud-test.conllu\"\n\"UD_210 test set of TURKISH_GB.\"\nUD_210_TURKISH_IMST_TRAIN = _UD_210_HOME + \"UD_Turkish-IMST/tr_imst-ud-train.conllu\"\n\"UD_210 train set of TURKISH_IMST.\"\nUD_210_TURKISH_IMST_DEV = _UD_210_HOME + \"UD_Turkish-IMST/tr_imst-ud-dev.conllu\"\n\"UD_210 dev set of TURKISH_IMST.\"\nUD_210_TURKISH_IMST_TEST = _UD_210_HOME + \"UD_Turkish-IMST/tr_imst-ud-test.conllu\"\n\"UD_210 test set of TURKISH_IMST.\"\nUD_210_TURKISH_KENET_TRAIN = _UD_210_HOME + \"UD_Turkish-Kenet/tr_kenet-ud-train.conllu\"\n\"UD_210 train set of TURKISH_KENET.\"\nUD_210_TURKISH_KENET_DEV = _UD_210_HOME + \"UD_Turkish-Kenet/tr_kenet-ud-dev.conllu\"\n\"UD_210 dev set of TURKISH_KENET.\"\nUD_210_TURKISH_KENET_TEST = _UD_210_HOME + \"UD_Turkish-Kenet/tr_kenet-ud-test.conllu\"\n\"UD_210 test set of TURKISH_KENET.\"\nUD_210_TURKISH_PUD_TEST = _UD_210_HOME + \"UD_Turkish-PUD/tr_pud-ud-test.conllu\"\n\"UD_210 test set of TURKISH_PUD.\"\nUD_210_TURKISH_PENN_TRAIN = _UD_210_HOME + \"UD_Turkish-Penn/tr_penn-ud-train.conllu\"\n\"UD_210 train set of TURKISH_PENN.\"\nUD_210_TURKISH_PENN_DEV = _UD_210_HOME + \"UD_Turkish-Penn/tr_penn-ud-dev.conllu\"\n\"UD_210 dev set of TURKISH_PENN.\"\nUD_210_TURKISH_PENN_TEST = _UD_210_HOME + \"UD_Turkish-Penn/tr_penn-ud-test.conllu\"\n\"UD_210 test set of TURKISH_PENN.\"\nUD_210_TURKISH_TOURISM_TRAIN = _UD_210_HOME + \"UD_Turkish-Tourism/tr_tourism-ud-train.conllu\"\n\"UD_210 train set of TURKISH_TOURISM.\"\nUD_210_TURKISH_TOURISM_DEV = _UD_210_HOME + \"UD_Turkish-Tourism/tr_tourism-ud-dev.conllu\"\n\"UD_210 dev set of TURKISH_TOURISM.\"\nUD_210_TURKISH_TOURISM_TEST = _UD_210_HOME + \"UD_Turkish-Tourism/tr_tourism-ud-test.conllu\"\n\"UD_210 test set of TURKISH_TOURISM.\"\nUD_210_TURKISH_GERMAN_SAGT_TRAIN = _UD_210_HOME + \"UD_Turkish_German-SAGT/qtd_sagt-ud-train.conllu\"\n\"UD_210 train set of TURKISH_GERMAN_SAGT.\"\nUD_210_TURKISH_GERMAN_SAGT_DEV = _UD_210_HOME + \"UD_Turkish_German-SAGT/qtd_sagt-ud-dev.conllu\"\n\"UD_210 dev set of TURKISH_GERMAN_SAGT.\"\nUD_210_TURKISH_GERMAN_SAGT_TEST = _UD_210_HOME + \"UD_Turkish_German-SAGT/qtd_sagt-ud-test.conllu\"\n\"UD_210 test set of TURKISH_GERMAN_SAGT.\"\nUD_210_UKRAINIAN_IU_TRAIN = _UD_210_HOME + \"UD_Ukrainian-IU/uk_iu-ud-train.conllu\"\n\"UD_210 train set of UKRAINIAN_IU.\"\nUD_210_UKRAINIAN_IU_DEV = _UD_210_HOME + \"UD_Ukrainian-IU/uk_iu-ud-dev.conllu\"\n\"UD_210 dev set of UKRAINIAN_IU.\"\nUD_210_UKRAINIAN_IU_TEST = _UD_210_HOME + \"UD_Ukrainian-IU/uk_iu-ud-test.conllu\"\n\"UD_210 test set of UKRAINIAN_IU.\"\nUD_210_UMBRIAN_IKUVINA_TEST = _UD_210_HOME + \"UD_Umbrian-IKUVINA/xum_ikuvina-ud-test.conllu\"\n\"UD_210 test set of UMBRIAN_IKUVINA.\"\nUD_210_UPPER_SORBIAN_UFAL_TRAIN = _UD_210_HOME + \"UD_Upper_Sorbian-UFAL/hsb_ufal-ud-train.conllu\"\n\"UD_210 train set of UPPER_SORBIAN_UFAL.\"\nUD_210_UPPER_SORBIAN_UFAL_TEST = _UD_210_HOME + \"UD_Upper_Sorbian-UFAL/hsb_ufal-ud-test.conllu\"\n\"UD_210 test set of UPPER_SORBIAN_UFAL.\"\nUD_210_URDU_UDTB_TRAIN = _UD_210_HOME + \"UD_Urdu-UDTB/ur_udtb-ud-train.conllu\"\n\"UD_210 train set of URDU_UDTB.\"\nUD_210_URDU_UDTB_DEV = _UD_210_HOME + \"UD_Urdu-UDTB/ur_udtb-ud-dev.conllu\"\n\"UD_210 dev set of URDU_UDTB.\"\nUD_210_URDU_UDTB_TEST = _UD_210_HOME + \"UD_Urdu-UDTB/ur_udtb-ud-test.conllu\"\n\"UD_210 test set of URDU_UDTB.\"\nUD_210_UYGHUR_UDT_TRAIN = _UD_210_HOME + \"UD_Uyghur-UDT/ug_udt-ud-train.conllu\"\n\"UD_210 train set of UYGHUR_UDT.\"\nUD_210_UYGHUR_UDT_DEV = _UD_210_HOME + \"UD_Uyghur-UDT/ug_udt-ud-dev.conllu\"\n\"UD_210 dev set of UYGHUR_UDT.\"\nUD_210_UYGHUR_UDT_TEST = _UD_210_HOME + \"UD_Uyghur-UDT/ug_udt-ud-test.conllu\"\n\"UD_210 test set of UYGHUR_UDT.\"\nUD_210_VIETNAMESE_VTB_TRAIN = _UD_210_HOME + \"UD_Vietnamese-VTB/vi_vtb-ud-train.conllu\"\n\"UD_210 train set of VIETNAMESE_VTB.\"\nUD_210_VIETNAMESE_VTB_DEV = _UD_210_HOME + \"UD_Vietnamese-VTB/vi_vtb-ud-dev.conllu\"\n\"UD_210 dev set of VIETNAMESE_VTB.\"\nUD_210_VIETNAMESE_VTB_TEST = _UD_210_HOME + \"UD_Vietnamese-VTB/vi_vtb-ud-test.conllu\"\n\"UD_210 test set of VIETNAMESE_VTB.\"\nUD_210_WARLPIRI_UFAL_TEST = _UD_210_HOME + \"UD_Warlpiri-UFAL/wbp_ufal-ud-test.conllu\"\n\"UD_210 test set of WARLPIRI_UFAL.\"\nUD_210_WELSH_CCG_TRAIN = _UD_210_HOME + \"UD_Welsh-CCG/cy_ccg-ud-train.conllu\"\n\"UD_210 train set of WELSH_CCG.\"\nUD_210_WELSH_CCG_DEV = _UD_210_HOME + \"UD_Welsh-CCG/cy_ccg-ud-dev.conllu\"\n\"UD_210 dev set of WELSH_CCG.\"\nUD_210_WELSH_CCG_TEST = _UD_210_HOME + \"UD_Welsh-CCG/cy_ccg-ud-test.conllu\"\n\"UD_210 test set of WELSH_CCG.\"\nUD_210_WESTERN_ARMENIAN_ARMTDP_TRAIN = _UD_210_HOME + \"UD_Western_Armenian-ArmTDP/hyw_armtdp-ud-train.conllu\"\n\"UD_210 train set of WESTERN_ARMENIAN_ARMTDP.\"\nUD_210_WESTERN_ARMENIAN_ARMTDP_DEV = _UD_210_HOME + \"UD_Western_Armenian-ArmTDP/hyw_armtdp-ud-dev.conllu\"\n\"UD_210 dev set of WESTERN_ARMENIAN_ARMTDP.\"\nUD_210_WESTERN_ARMENIAN_ARMTDP_TEST = _UD_210_HOME + \"UD_Western_Armenian-ArmTDP/hyw_armtdp-ud-test.conllu\"\n\"UD_210 test set of WESTERN_ARMENIAN_ARMTDP.\"\nUD_210_WOLOF_WTB_TRAIN = _UD_210_HOME + \"UD_Wolof-WTB/wo_wtb-ud-train.conllu\"\n\"UD_210 train set of WOLOF_WTB.\"\nUD_210_WOLOF_WTB_DEV = _UD_210_HOME + \"UD_Wolof-WTB/wo_wtb-ud-dev.conllu\"\n\"UD_210 dev set of WOLOF_WTB.\"\nUD_210_WOLOF_WTB_TEST = _UD_210_HOME + \"UD_Wolof-WTB/wo_wtb-ud-test.conllu\"\n\"UD_210 test set of WOLOF_WTB.\"\nUD_210_XIBE_XDT_TEST = _UD_210_HOME + \"UD_Xibe-XDT/sjo_xdt-ud-test.conllu\"\n\"UD_210 test set of XIBE_XDT.\"\nUD_210_YAKUT_YKTDT_TEST = _UD_210_HOME + \"UD_Yakut-YKTDT/sah_yktdt-ud-test.conllu\"\n\"UD_210 test set of YAKUT_YKTDT.\"\nUD_210_YORUBA_YTB_TEST = _UD_210_HOME + \"UD_Yoruba-YTB/yo_ytb-ud-test.conllu\"\n\"UD_210 test set of YORUBA_YTB.\"\nUD_210_YUPIK_SLI_TEST = _UD_210_HOME + \"UD_Yupik-SLI/ess_sli-ud-test.conllu\"\n\"UD_210 test set of YUPIK_SLI.\"\n"
  },
  {
    "path": "hanlp/datasets/parsing/ud/ud210m.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-21 20:39\nimport os\n\nfrom hanlp.datasets.parsing.ud import concat_treebanks\nfrom hanlp.datasets.parsing.ud.ud210 import _UD_210_HOME\n\n_UD_210_MULTILINGUAL_HOME = concat_treebanks(_UD_210_HOME, '2.10')\nUD_210_MULTILINGUAL_TRAIN = os.path.join(_UD_210_MULTILINGUAL_HOME, 'train.conllu')\n\"Training set of multilingual UD_210 obtained by concatenating all training sets.\"\nUD_210_MULTILINGUAL_DEV = os.path.join(_UD_210_MULTILINGUAL_HOME, 'dev.conllu')\n\"Dev set of multilingual UD_210 obtained by concatenating all dev sets.\"\nUD_210_MULTILINGUAL_TEST = os.path.join(_UD_210_MULTILINGUAL_HOME, 'test.conllu')\n\"Test set of multilingual UD_210 obtained by concatenating all test sets.\"\n"
  },
  {
    "path": "hanlp/datasets/parsing/ud/ud23.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-21 20:26\n\n_UD_23_HOME = \"https://lindat.mff.cuni.cz/repository/xmlui/bitstream/handle/11234/1-2895/ud-treebanks-v2.3.tgz?sequence=1&isAllowed=y\"\n_UD_24_HOME = \"https://lindat.mff.cuni.cz/repository/xmlui/bitstream/handle/11234/1-2988/ud-treebanks-v2.4.tgz?sequence=4&isAllowed=y\"\n\n\ndef _list_dir(path, home):\n    prefix = home.lstrip('_').replace('_HOME', '')\n\n    from hanlp.utils.io_util import get_resource\n    import glob\n    import os\n    path = get_resource(path)\n    with open('ud23.py', 'a') as out:\n        for f in sorted(glob.glob(path + '/UD_*')):\n            basename = os.path.basename(f)\n            name = basename[len('UD_'):]\n            name = name.upper().replace('-', '_')\n            for split in 'train', 'dev', 'test':\n                sp = glob.glob(f + f'/*{split}.conllu')\n                if not sp:\n                    continue\n                sp = os.path.basename(sp[0])\n                out.write(f'{prefix}_{name}_{split.upper()} = {home} + \"#{basename}/{sp}\"\\n')\n\n\ndef main():\n    _list_dir(_UD_23_HOME, '_UD_23_HOME')\n    pass\n\n\nif __name__ == '__main__':\n    main()\n\nUD_23_AFRIKAANS_AFRIBOOMS_TRAIN = _UD_23_HOME + \"#UD_Afrikaans-AfriBooms/af_afribooms-ud-train.conllu\"\nUD_23_AFRIKAANS_AFRIBOOMS_DEV = _UD_23_HOME + \"#UD_Afrikaans-AfriBooms/af_afribooms-ud-dev.conllu\"\nUD_23_AFRIKAANS_AFRIBOOMS_TEST = _UD_23_HOME + \"#UD_Afrikaans-AfriBooms/af_afribooms-ud-test.conllu\"\nUD_23_AKKADIAN_PISANDUB_TEST = _UD_23_HOME + \"#UD_Akkadian-PISANDUB/akk_pisandub-ud-test.conllu\"\nUD_23_AMHARIC_ATT_TEST = _UD_23_HOME + \"#UD_Amharic-ATT/am_att-ud-test.conllu\"\nUD_23_ANCIENT_GREEK_PROIEL_TRAIN = _UD_23_HOME + \"#UD_Ancient_Greek-PROIEL/grc_proiel-ud-train.conllu\"\nUD_23_ANCIENT_GREEK_PROIEL_DEV = _UD_23_HOME + \"#UD_Ancient_Greek-PROIEL/grc_proiel-ud-dev.conllu\"\nUD_23_ANCIENT_GREEK_PROIEL_TEST = _UD_23_HOME + \"#UD_Ancient_Greek-PROIEL/grc_proiel-ud-test.conllu\"\nUD_23_ANCIENT_GREEK_PERSEUS_TRAIN = _UD_23_HOME + \"#UD_Ancient_Greek-Perseus/grc_perseus-ud-train.conllu\"\nUD_23_ANCIENT_GREEK_PERSEUS_DEV = _UD_23_HOME + \"#UD_Ancient_Greek-Perseus/grc_perseus-ud-dev.conllu\"\nUD_23_ANCIENT_GREEK_PERSEUS_TEST = _UD_23_HOME + \"#UD_Ancient_Greek-Perseus/grc_perseus-ud-test.conllu\"\nUD_23_ARABIC_NYUAD_TRAIN = _UD_23_HOME + \"#UD_Arabic-NYUAD/ar_nyuad-ud-train.conllu\"\nUD_23_ARABIC_NYUAD_DEV = _UD_23_HOME + \"#UD_Arabic-NYUAD/ar_nyuad-ud-dev.conllu\"\nUD_23_ARABIC_NYUAD_TEST = _UD_23_HOME + \"#UD_Arabic-NYUAD/ar_nyuad-ud-test.conllu\"\nUD_23_ARABIC_PADT_TRAIN = _UD_23_HOME + \"#UD_Arabic-PADT/ar_padt-ud-train.conllu\"\nUD_23_ARABIC_PADT_DEV = _UD_23_HOME + \"#UD_Arabic-PADT/ar_padt-ud-dev.conllu\"\nUD_23_ARABIC_PADT_TEST = _UD_23_HOME + \"#UD_Arabic-PADT/ar_padt-ud-test.conllu\"\nUD_23_ARABIC_PUD_TEST = _UD_23_HOME + \"#UD_Arabic-PUD/ar_pud-ud-test.conllu\"\nUD_23_ARMENIAN_ARMTDP_TRAIN = _UD_23_HOME + \"#UD_Armenian-ArmTDP/hy_armtdp-ud-train.conllu\"\nUD_23_ARMENIAN_ARMTDP_TEST = _UD_23_HOME + \"#UD_Armenian-ArmTDP/hy_armtdp-ud-test.conllu\"\nUD_23_BAMBARA_CRB_TEST = _UD_23_HOME + \"#UD_Bambara-CRB/bm_crb-ud-test.conllu\"\nUD_23_BASQUE_BDT_TRAIN = _UD_23_HOME + \"#UD_Basque-BDT/eu_bdt-ud-train.conllu\"\nUD_23_BASQUE_BDT_DEV = _UD_23_HOME + \"#UD_Basque-BDT/eu_bdt-ud-dev.conllu\"\nUD_23_BASQUE_BDT_TEST = _UD_23_HOME + \"#UD_Basque-BDT/eu_bdt-ud-test.conllu\"\nUD_23_BELARUSIAN_HSE_TRAIN = _UD_23_HOME + \"#UD_Belarusian-HSE/be_hse-ud-train.conllu\"\nUD_23_BELARUSIAN_HSE_DEV = _UD_23_HOME + \"#UD_Belarusian-HSE/be_hse-ud-dev.conllu\"\nUD_23_BELARUSIAN_HSE_TEST = _UD_23_HOME + \"#UD_Belarusian-HSE/be_hse-ud-test.conllu\"\nUD_23_BRETON_KEB_TEST = _UD_23_HOME + \"#UD_Breton-KEB/br_keb-ud-test.conllu\"\nUD_23_BULGARIAN_BTB_TRAIN = _UD_23_HOME + \"#UD_Bulgarian-BTB/bg_btb-ud-train.conllu\"\nUD_23_BULGARIAN_BTB_DEV = _UD_23_HOME + \"#UD_Bulgarian-BTB/bg_btb-ud-dev.conllu\"\nUD_23_BULGARIAN_BTB_TEST = _UD_23_HOME + \"#UD_Bulgarian-BTB/bg_btb-ud-test.conllu\"\nUD_23_BURYAT_BDT_TRAIN = _UD_23_HOME + \"#UD_Buryat-BDT/bxr_bdt-ud-train.conllu\"\nUD_23_BURYAT_BDT_TEST = _UD_23_HOME + \"#UD_Buryat-BDT/bxr_bdt-ud-test.conllu\"\nUD_23_CANTONESE_HK_TEST = _UD_23_HOME + \"#UD_Cantonese-HK/yue_hk-ud-test.conllu\"\nUD_23_CATALAN_ANCORA_TRAIN = _UD_23_HOME + \"#UD_Catalan-AnCora/ca_ancora-ud-train.conllu\"\nUD_23_CATALAN_ANCORA_DEV = _UD_23_HOME + \"#UD_Catalan-AnCora/ca_ancora-ud-dev.conllu\"\nUD_23_CATALAN_ANCORA_TEST = _UD_23_HOME + \"#UD_Catalan-AnCora/ca_ancora-ud-test.conllu\"\nUD_23_CHINESE_CFL_TEST = _UD_23_HOME + \"#UD_Chinese-CFL/zh_cfl-ud-test.conllu\"\nUD_23_CHINESE_GSD_TRAIN = _UD_23_HOME + \"#UD_Chinese-GSD/zh_gsd-ud-train.conllu\"\nUD_23_CHINESE_GSD_DEV = _UD_23_HOME + \"#UD_Chinese-GSD/zh_gsd-ud-dev.conllu\"\nUD_23_CHINESE_GSD_TEST = _UD_23_HOME + \"#UD_Chinese-GSD/zh_gsd-ud-test.conllu\"\nUD_23_CHINESE_HK_TEST = _UD_23_HOME + \"#UD_Chinese-HK/zh_hk-ud-test.conllu\"\nUD_23_CHINESE_PUD_TEST = _UD_23_HOME + \"#UD_Chinese-PUD/zh_pud-ud-test.conllu\"\nUD_23_COPTIC_SCRIPTORIUM_TRAIN = _UD_23_HOME + \"#UD_Coptic-Scriptorium/cop_scriptorium-ud-train.conllu\"\nUD_23_COPTIC_SCRIPTORIUM_DEV = _UD_23_HOME + \"#UD_Coptic-Scriptorium/cop_scriptorium-ud-dev.conllu\"\nUD_23_COPTIC_SCRIPTORIUM_TEST = _UD_23_HOME + \"#UD_Coptic-Scriptorium/cop_scriptorium-ud-test.conllu\"\nUD_23_CROATIAN_SET_TRAIN = _UD_23_HOME + \"#UD_Croatian-SET/hr_set-ud-train.conllu\"\nUD_23_CROATIAN_SET_DEV = _UD_23_HOME + \"#UD_Croatian-SET/hr_set-ud-dev.conllu\"\nUD_23_CROATIAN_SET_TEST = _UD_23_HOME + \"#UD_Croatian-SET/hr_set-ud-test.conllu\"\nUD_23_CZECH_CAC_TRAIN = _UD_23_HOME + \"#UD_Czech-CAC/cs_cac-ud-train.conllu\"\nUD_23_CZECH_CAC_DEV = _UD_23_HOME + \"#UD_Czech-CAC/cs_cac-ud-dev.conllu\"\nUD_23_CZECH_CAC_TEST = _UD_23_HOME + \"#UD_Czech-CAC/cs_cac-ud-test.conllu\"\nUD_23_CZECH_CLTT_TRAIN = _UD_23_HOME + \"#UD_Czech-CLTT/cs_cltt-ud-train.conllu\"\nUD_23_CZECH_CLTT_DEV = _UD_23_HOME + \"#UD_Czech-CLTT/cs_cltt-ud-dev.conllu\"\nUD_23_CZECH_CLTT_TEST = _UD_23_HOME + \"#UD_Czech-CLTT/cs_cltt-ud-test.conllu\"\nUD_23_CZECH_FICTREE_TRAIN = _UD_23_HOME + \"#UD_Czech-FicTree/cs_fictree-ud-train.conllu\"\nUD_23_CZECH_FICTREE_DEV = _UD_23_HOME + \"#UD_Czech-FicTree/cs_fictree-ud-dev.conllu\"\nUD_23_CZECH_FICTREE_TEST = _UD_23_HOME + \"#UD_Czech-FicTree/cs_fictree-ud-test.conllu\"\nUD_23_CZECH_PDT_TRAIN = _UD_23_HOME + \"#UD_Czech-PDT/cs_pdt-ud-train.conllu\"\nUD_23_CZECH_PDT_DEV = _UD_23_HOME + \"#UD_Czech-PDT/cs_pdt-ud-dev.conllu\"\nUD_23_CZECH_PDT_TEST = _UD_23_HOME + \"#UD_Czech-PDT/cs_pdt-ud-test.conllu\"\nUD_23_CZECH_PUD_TEST = _UD_23_HOME + \"#UD_Czech-PUD/cs_pud-ud-test.conllu\"\nUD_23_DANISH_DDT_TRAIN = _UD_23_HOME + \"#UD_Danish-DDT/da_ddt-ud-train.conllu\"\nUD_23_DANISH_DDT_DEV = _UD_23_HOME + \"#UD_Danish-DDT/da_ddt-ud-dev.conllu\"\nUD_23_DANISH_DDT_TEST = _UD_23_HOME + \"#UD_Danish-DDT/da_ddt-ud-test.conllu\"\nUD_23_DUTCH_ALPINO_TRAIN = _UD_23_HOME + \"#UD_Dutch-Alpino/nl_alpino-ud-train.conllu\"\nUD_23_DUTCH_ALPINO_DEV = _UD_23_HOME + \"#UD_Dutch-Alpino/nl_alpino-ud-dev.conllu\"\nUD_23_DUTCH_ALPINO_TEST = _UD_23_HOME + \"#UD_Dutch-Alpino/nl_alpino-ud-test.conllu\"\nUD_23_DUTCH_LASSYSMALL_TRAIN = _UD_23_HOME + \"#UD_Dutch-LassySmall/nl_lassysmall-ud-train.conllu\"\nUD_23_DUTCH_LASSYSMALL_DEV = _UD_23_HOME + \"#UD_Dutch-LassySmall/nl_lassysmall-ud-dev.conllu\"\nUD_23_DUTCH_LASSYSMALL_TEST = _UD_23_HOME + \"#UD_Dutch-LassySmall/nl_lassysmall-ud-test.conllu\"\nUD_23_ENGLISH_ESL_TRAIN = _UD_23_HOME + \"#UD_English-ESL/en_esl-ud-train.conllu\"\nUD_23_ENGLISH_ESL_DEV = _UD_23_HOME + \"#UD_English-ESL/en_esl-ud-dev.conllu\"\nUD_23_ENGLISH_ESL_TEST = _UD_23_HOME + \"#UD_English-ESL/en_esl-ud-test.conllu\"\nUD_23_ENGLISH_EWT_TRAIN = _UD_23_HOME + \"#UD_English-EWT/en_ewt-ud-train.conllu\"\nUD_23_ENGLISH_EWT_DEV = _UD_23_HOME + \"#UD_English-EWT/en_ewt-ud-dev.conllu\"\nUD_23_ENGLISH_EWT_TEST = _UD_23_HOME + \"#UD_English-EWT/en_ewt-ud-test.conllu\"\nUD_23_ENGLISH_GUM_TRAIN = _UD_23_HOME + \"#UD_English-GUM/en_gum-ud-train.conllu\"\nUD_23_ENGLISH_GUM_DEV = _UD_23_HOME + \"#UD_English-GUM/en_gum-ud-dev.conllu\"\nUD_23_ENGLISH_GUM_TEST = _UD_23_HOME + \"#UD_English-GUM/en_gum-ud-test.conllu\"\nUD_23_ENGLISH_LINES_TRAIN = _UD_23_HOME + \"#UD_English-LinES/en_lines-ud-train.conllu\"\nUD_23_ENGLISH_LINES_DEV = _UD_23_HOME + \"#UD_English-LinES/en_lines-ud-dev.conllu\"\nUD_23_ENGLISH_LINES_TEST = _UD_23_HOME + \"#UD_English-LinES/en_lines-ud-test.conllu\"\nUD_23_ENGLISH_PUD_TEST = _UD_23_HOME + \"#UD_English-PUD/en_pud-ud-test.conllu\"\nUD_23_ENGLISH_PARTUT_TRAIN = _UD_23_HOME + \"#UD_English-ParTUT/en_partut-ud-train.conllu\"\nUD_23_ENGLISH_PARTUT_DEV = _UD_23_HOME + \"#UD_English-ParTUT/en_partut-ud-dev.conllu\"\nUD_23_ENGLISH_PARTUT_TEST = _UD_23_HOME + \"#UD_English-ParTUT/en_partut-ud-test.conllu\"\nUD_23_ERZYA_JR_TEST = _UD_23_HOME + \"#UD_Erzya-JR/myv_jr-ud-test.conllu\"\nUD_23_ESTONIAN_EDT_TRAIN = _UD_23_HOME + \"#UD_Estonian-EDT/et_edt-ud-train.conllu\"\nUD_23_ESTONIAN_EDT_DEV = _UD_23_HOME + \"#UD_Estonian-EDT/et_edt-ud-dev.conllu\"\nUD_23_ESTONIAN_EDT_TEST = _UD_23_HOME + \"#UD_Estonian-EDT/et_edt-ud-test.conllu\"\nUD_23_FAROESE_OFT_TEST = _UD_23_HOME + \"#UD_Faroese-OFT/fo_oft-ud-test.conllu\"\nUD_23_FINNISH_FTB_TRAIN = _UD_23_HOME + \"#UD_Finnish-FTB/fi_ftb-ud-train.conllu\"\nUD_23_FINNISH_FTB_DEV = _UD_23_HOME + \"#UD_Finnish-FTB/fi_ftb-ud-dev.conllu\"\nUD_23_FINNISH_FTB_TEST = _UD_23_HOME + \"#UD_Finnish-FTB/fi_ftb-ud-test.conllu\"\nUD_23_FINNISH_PUD_TEST = _UD_23_HOME + \"#UD_Finnish-PUD/fi_pud-ud-test.conllu\"\nUD_23_FINNISH_TDT_TRAIN = _UD_23_HOME + \"#UD_Finnish-TDT/fi_tdt-ud-train.conllu\"\nUD_23_FINNISH_TDT_DEV = _UD_23_HOME + \"#UD_Finnish-TDT/fi_tdt-ud-dev.conllu\"\nUD_23_FINNISH_TDT_TEST = _UD_23_HOME + \"#UD_Finnish-TDT/fi_tdt-ud-test.conllu\"\nUD_23_FRENCH_FTB_TRAIN = _UD_23_HOME + \"#UD_French-FTB/fr_ftb-ud-train.conllu\"\nUD_23_FRENCH_FTB_DEV = _UD_23_HOME + \"#UD_French-FTB/fr_ftb-ud-dev.conllu\"\nUD_23_FRENCH_FTB_TEST = _UD_23_HOME + \"#UD_French-FTB/fr_ftb-ud-test.conllu\"\nUD_23_FRENCH_GSD_TRAIN = _UD_23_HOME + \"#UD_French-GSD/fr_gsd-ud-train.conllu\"\nUD_23_FRENCH_GSD_DEV = _UD_23_HOME + \"#UD_French-GSD/fr_gsd-ud-dev.conllu\"\nUD_23_FRENCH_GSD_TEST = _UD_23_HOME + \"#UD_French-GSD/fr_gsd-ud-test.conllu\"\nUD_23_FRENCH_PUD_TEST = _UD_23_HOME + \"#UD_French-PUD/fr_pud-ud-test.conllu\"\nUD_23_FRENCH_PARTUT_TRAIN = _UD_23_HOME + \"#UD_French-ParTUT/fr_partut-ud-train.conllu\"\nUD_23_FRENCH_PARTUT_DEV = _UD_23_HOME + \"#UD_French-ParTUT/fr_partut-ud-dev.conllu\"\nUD_23_FRENCH_PARTUT_TEST = _UD_23_HOME + \"#UD_French-ParTUT/fr_partut-ud-test.conllu\"\nUD_23_FRENCH_SEQUOIA_TRAIN = _UD_23_HOME + \"#UD_French-Sequoia/fr_sequoia-ud-train.conllu\"\nUD_23_FRENCH_SEQUOIA_DEV = _UD_23_HOME + \"#UD_French-Sequoia/fr_sequoia-ud-dev.conllu\"\nUD_23_FRENCH_SEQUOIA_TEST = _UD_23_HOME + \"#UD_French-Sequoia/fr_sequoia-ud-test.conllu\"\nUD_23_FRENCH_SPOKEN_TRAIN = _UD_23_HOME + \"#UD_French-Spoken/fr_spoken-ud-train.conllu\"\nUD_23_FRENCH_SPOKEN_DEV = _UD_23_HOME + \"#UD_French-Spoken/fr_spoken-ud-dev.conllu\"\nUD_23_FRENCH_SPOKEN_TEST = _UD_23_HOME + \"#UD_French-Spoken/fr_spoken-ud-test.conllu\"\nUD_23_GALICIAN_CTG_TRAIN = _UD_23_HOME + \"#UD_Galician-CTG/gl_ctg-ud-train.conllu\"\nUD_23_GALICIAN_CTG_DEV = _UD_23_HOME + \"#UD_Galician-CTG/gl_ctg-ud-dev.conllu\"\nUD_23_GALICIAN_CTG_TEST = _UD_23_HOME + \"#UD_Galician-CTG/gl_ctg-ud-test.conllu\"\nUD_23_GALICIAN_TREEGAL_TRAIN = _UD_23_HOME + \"#UD_Galician-TreeGal/gl_treegal-ud-train.conllu\"\nUD_23_GALICIAN_TREEGAL_TEST = _UD_23_HOME + \"#UD_Galician-TreeGal/gl_treegal-ud-test.conllu\"\nUD_23_GERMAN_GSD_TRAIN = _UD_23_HOME + \"#UD_German-GSD/de_gsd-ud-train.conllu\"\nUD_23_GERMAN_GSD_DEV = _UD_23_HOME + \"#UD_German-GSD/de_gsd-ud-dev.conllu\"\nUD_23_GERMAN_GSD_TEST = _UD_23_HOME + \"#UD_German-GSD/de_gsd-ud-test.conllu\"\nUD_23_GERMAN_PUD_TEST = _UD_23_HOME + \"#UD_German-PUD/de_pud-ud-test.conllu\"\nUD_23_GOTHIC_PROIEL_TRAIN = _UD_23_HOME + \"#UD_Gothic-PROIEL/got_proiel-ud-train.conllu\"\nUD_23_GOTHIC_PROIEL_DEV = _UD_23_HOME + \"#UD_Gothic-PROIEL/got_proiel-ud-dev.conllu\"\nUD_23_GOTHIC_PROIEL_TEST = _UD_23_HOME + \"#UD_Gothic-PROIEL/got_proiel-ud-test.conllu\"\nUD_23_GREEK_GDT_TRAIN = _UD_23_HOME + \"#UD_Greek-GDT/el_gdt-ud-train.conllu\"\nUD_23_GREEK_GDT_DEV = _UD_23_HOME + \"#UD_Greek-GDT/el_gdt-ud-dev.conllu\"\nUD_23_GREEK_GDT_TEST = _UD_23_HOME + \"#UD_Greek-GDT/el_gdt-ud-test.conllu\"\nUD_23_HEBREW_HTB_TRAIN = _UD_23_HOME + \"#UD_Hebrew-HTB/he_htb-ud-train.conllu\"\nUD_23_HEBREW_HTB_DEV = _UD_23_HOME + \"#UD_Hebrew-HTB/he_htb-ud-dev.conllu\"\nUD_23_HEBREW_HTB_TEST = _UD_23_HOME + \"#UD_Hebrew-HTB/he_htb-ud-test.conllu\"\nUD_23_HINDI_HDTB_TRAIN = _UD_23_HOME + \"#UD_Hindi-HDTB/hi_hdtb-ud-train.conllu\"\nUD_23_HINDI_HDTB_DEV = _UD_23_HOME + \"#UD_Hindi-HDTB/hi_hdtb-ud-dev.conllu\"\nUD_23_HINDI_HDTB_TEST = _UD_23_HOME + \"#UD_Hindi-HDTB/hi_hdtb-ud-test.conllu\"\nUD_23_HINDI_PUD_TEST = _UD_23_HOME + \"#UD_Hindi-PUD/hi_pud-ud-test.conllu\"\nUD_23_HINDI_ENGLISH_HIENCS_TRAIN = _UD_23_HOME + \"#UD_Hindi_English-HIENCS/qhe_hiencs-ud-train.conllu\"\nUD_23_HINDI_ENGLISH_HIENCS_DEV = _UD_23_HOME + \"#UD_Hindi_English-HIENCS/qhe_hiencs-ud-dev.conllu\"\nUD_23_HINDI_ENGLISH_HIENCS_TEST = _UD_23_HOME + \"#UD_Hindi_English-HIENCS/qhe_hiencs-ud-test.conllu\"\nUD_23_HUNGARIAN_SZEGED_TRAIN = _UD_23_HOME + \"#UD_Hungarian-Szeged/hu_szeged-ud-train.conllu\"\nUD_23_HUNGARIAN_SZEGED_DEV = _UD_23_HOME + \"#UD_Hungarian-Szeged/hu_szeged-ud-dev.conllu\"\nUD_23_HUNGARIAN_SZEGED_TEST = _UD_23_HOME + \"#UD_Hungarian-Szeged/hu_szeged-ud-test.conllu\"\nUD_23_INDONESIAN_GSD_TRAIN = _UD_23_HOME + \"#UD_Indonesian-GSD/id_gsd-ud-train.conllu\"\nUD_23_INDONESIAN_GSD_DEV = _UD_23_HOME + \"#UD_Indonesian-GSD/id_gsd-ud-dev.conllu\"\nUD_23_INDONESIAN_GSD_TEST = _UD_23_HOME + \"#UD_Indonesian-GSD/id_gsd-ud-test.conllu\"\nUD_23_INDONESIAN_PUD_TEST = _UD_23_HOME + \"#UD_Indonesian-PUD/id_pud-ud-test.conllu\"\nUD_23_IRISH_IDT_TRAIN = _UD_23_HOME + \"#UD_Irish-IDT/ga_idt-ud-train.conllu\"\nUD_23_IRISH_IDT_TEST = _UD_23_HOME + \"#UD_Irish-IDT/ga_idt-ud-test.conllu\"\nUD_23_ITALIAN_ISDT_TRAIN = _UD_23_HOME + \"#UD_Italian-ISDT/it_isdt-ud-train.conllu\"\nUD_23_ITALIAN_ISDT_DEV = _UD_23_HOME + \"#UD_Italian-ISDT/it_isdt-ud-dev.conllu\"\nUD_23_ITALIAN_ISDT_TEST = _UD_23_HOME + \"#UD_Italian-ISDT/it_isdt-ud-test.conllu\"\nUD_23_ITALIAN_PUD_TEST = _UD_23_HOME + \"#UD_Italian-PUD/it_pud-ud-test.conllu\"\nUD_23_ITALIAN_PARTUT_TRAIN = _UD_23_HOME + \"#UD_Italian-ParTUT/it_partut-ud-train.conllu\"\nUD_23_ITALIAN_PARTUT_DEV = _UD_23_HOME + \"#UD_Italian-ParTUT/it_partut-ud-dev.conllu\"\nUD_23_ITALIAN_PARTUT_TEST = _UD_23_HOME + \"#UD_Italian-ParTUT/it_partut-ud-test.conllu\"\nUD_23_ITALIAN_POSTWITA_TRAIN = _UD_23_HOME + \"#UD_Italian-PoSTWITA/it_postwita-ud-train.conllu\"\nUD_23_ITALIAN_POSTWITA_DEV = _UD_23_HOME + \"#UD_Italian-PoSTWITA/it_postwita-ud-dev.conllu\"\nUD_23_ITALIAN_POSTWITA_TEST = _UD_23_HOME + \"#UD_Italian-PoSTWITA/it_postwita-ud-test.conllu\"\nUD_23_JAPANESE_BCCWJ_TRAIN = _UD_23_HOME + \"#UD_Japanese-BCCWJ/ja_bccwj-ud-train.conllu\"\nUD_23_JAPANESE_BCCWJ_DEV = _UD_23_HOME + \"#UD_Japanese-BCCWJ/ja_bccwj-ud-dev.conllu\"\nUD_23_JAPANESE_BCCWJ_TEST = _UD_23_HOME + \"#UD_Japanese-BCCWJ/ja_bccwj-ud-test.conllu\"\nUD_23_JAPANESE_GSD_TRAIN = _UD_23_HOME + \"#UD_Japanese-GSD/ja_gsd-ud-train.conllu\"\nUD_23_JAPANESE_GSD_DEV = _UD_23_HOME + \"#UD_Japanese-GSD/ja_gsd-ud-dev.conllu\"\nUD_23_JAPANESE_GSD_TEST = _UD_23_HOME + \"#UD_Japanese-GSD/ja_gsd-ud-test.conllu\"\nUD_23_JAPANESE_MODERN_TEST = _UD_23_HOME + \"#UD_Japanese-Modern/ja_modern-ud-test.conllu\"\nUD_23_JAPANESE_PUD_TEST = _UD_23_HOME + \"#UD_Japanese-PUD/ja_pud-ud-test.conllu\"\nUD_23_KAZAKH_KTB_TRAIN = _UD_23_HOME + \"#UD_Kazakh-KTB/kk_ktb-ud-train.conllu\"\nUD_23_KAZAKH_KTB_TEST = _UD_23_HOME + \"#UD_Kazakh-KTB/kk_ktb-ud-test.conllu\"\nUD_23_KOMI_ZYRIAN_IKDP_TEST = _UD_23_HOME + \"#UD_Komi_Zyrian-IKDP/kpv_ikdp-ud-test.conllu\"\nUD_23_KOMI_ZYRIAN_LATTICE_TEST = _UD_23_HOME + \"#UD_Komi_Zyrian-Lattice/kpv_lattice-ud-test.conllu\"\nUD_23_KOREAN_GSD_TRAIN = _UD_23_HOME + \"#UD_Korean-GSD/ko_gsd-ud-train.conllu\"\nUD_23_KOREAN_GSD_DEV = _UD_23_HOME + \"#UD_Korean-GSD/ko_gsd-ud-dev.conllu\"\nUD_23_KOREAN_GSD_TEST = _UD_23_HOME + \"#UD_Korean-GSD/ko_gsd-ud-test.conllu\"\nUD_23_KOREAN_KAIST_TRAIN = _UD_23_HOME + \"#UD_Korean-Kaist/ko_kaist-ud-train.conllu\"\nUD_23_KOREAN_KAIST_DEV = _UD_23_HOME + \"#UD_Korean-Kaist/ko_kaist-ud-dev.conllu\"\nUD_23_KOREAN_KAIST_TEST = _UD_23_HOME + \"#UD_Korean-Kaist/ko_kaist-ud-test.conllu\"\nUD_23_KOREAN_PUD_TEST = _UD_23_HOME + \"#UD_Korean-PUD/ko_pud-ud-test.conllu\"\nUD_23_KURMANJI_MG_TRAIN = _UD_23_HOME + \"#UD_Kurmanji-MG/kmr_mg-ud-train.conllu\"\nUD_23_KURMANJI_MG_TEST = _UD_23_HOME + \"#UD_Kurmanji-MG/kmr_mg-ud-test.conllu\"\nUD_23_LATIN_ITTB_TRAIN = _UD_23_HOME + \"#UD_Latin-ITTB/la_ittb-ud-train.conllu\"\nUD_23_LATIN_ITTB_DEV = _UD_23_HOME + \"#UD_Latin-ITTB/la_ittb-ud-dev.conllu\"\nUD_23_LATIN_ITTB_TEST = _UD_23_HOME + \"#UD_Latin-ITTB/la_ittb-ud-test.conllu\"\nUD_23_LATIN_PROIEL_TRAIN = _UD_23_HOME + \"#UD_Latin-PROIEL/la_proiel-ud-train.conllu\"\nUD_23_LATIN_PROIEL_DEV = _UD_23_HOME + \"#UD_Latin-PROIEL/la_proiel-ud-dev.conllu\"\nUD_23_LATIN_PROIEL_TEST = _UD_23_HOME + \"#UD_Latin-PROIEL/la_proiel-ud-test.conllu\"\nUD_23_LATIN_PERSEUS_TRAIN = _UD_23_HOME + \"#UD_Latin-Perseus/la_perseus-ud-train.conllu\"\nUD_23_LATIN_PERSEUS_TEST = _UD_23_HOME + \"#UD_Latin-Perseus/la_perseus-ud-test.conllu\"\nUD_23_LATVIAN_LVTB_TRAIN = _UD_23_HOME + \"#UD_Latvian-LVTB/lv_lvtb-ud-train.conllu\"\nUD_23_LATVIAN_LVTB_DEV = _UD_23_HOME + \"#UD_Latvian-LVTB/lv_lvtb-ud-dev.conllu\"\nUD_23_LATVIAN_LVTB_TEST = _UD_23_HOME + \"#UD_Latvian-LVTB/lv_lvtb-ud-test.conllu\"\nUD_23_LITHUANIAN_HSE_TRAIN = _UD_23_HOME + \"#UD_Lithuanian-HSE/lt_hse-ud-train.conllu\"\nUD_23_LITHUANIAN_HSE_DEV = _UD_23_HOME + \"#UD_Lithuanian-HSE/lt_hse-ud-dev.conllu\"\nUD_23_LITHUANIAN_HSE_TEST = _UD_23_HOME + \"#UD_Lithuanian-HSE/lt_hse-ud-test.conllu\"\nUD_23_MALTESE_MUDT_TRAIN = _UD_23_HOME + \"#UD_Maltese-MUDT/mt_mudt-ud-train.conllu\"\nUD_23_MALTESE_MUDT_DEV = _UD_23_HOME + \"#UD_Maltese-MUDT/mt_mudt-ud-dev.conllu\"\nUD_23_MALTESE_MUDT_TEST = _UD_23_HOME + \"#UD_Maltese-MUDT/mt_mudt-ud-test.conllu\"\nUD_23_MARATHI_UFAL_TRAIN = _UD_23_HOME + \"#UD_Marathi-UFAL/mr_ufal-ud-train.conllu\"\nUD_23_MARATHI_UFAL_DEV = _UD_23_HOME + \"#UD_Marathi-UFAL/mr_ufal-ud-dev.conllu\"\nUD_23_MARATHI_UFAL_TEST = _UD_23_HOME + \"#UD_Marathi-UFAL/mr_ufal-ud-test.conllu\"\nUD_23_NAIJA_NSC_TEST = _UD_23_HOME + \"#UD_Naija-NSC/pcm_nsc-ud-test.conllu\"\nUD_23_NORTH_SAMI_GIELLA_TRAIN = _UD_23_HOME + \"#UD_North_Sami-Giella/sme_giella-ud-train.conllu\"\nUD_23_NORTH_SAMI_GIELLA_TEST = _UD_23_HOME + \"#UD_North_Sami-Giella/sme_giella-ud-test.conllu\"\nUD_23_NORWEGIAN_BOKMAAL_TRAIN = _UD_23_HOME + \"#UD_Norwegian-Bokmaal/no_bokmaal-ud-train.conllu\"\nUD_23_NORWEGIAN_BOKMAAL_DEV = _UD_23_HOME + \"#UD_Norwegian-Bokmaal/no_bokmaal-ud-dev.conllu\"\nUD_23_NORWEGIAN_BOKMAAL_TEST = _UD_23_HOME + \"#UD_Norwegian-Bokmaal/no_bokmaal-ud-test.conllu\"\nUD_23_NORWEGIAN_NYNORSK_TRAIN = _UD_23_HOME + \"#UD_Norwegian-Nynorsk/no_nynorsk-ud-train.conllu\"\nUD_23_NORWEGIAN_NYNORSK_DEV = _UD_23_HOME + \"#UD_Norwegian-Nynorsk/no_nynorsk-ud-dev.conllu\"\nUD_23_NORWEGIAN_NYNORSK_TEST = _UD_23_HOME + \"#UD_Norwegian-Nynorsk/no_nynorsk-ud-test.conllu\"\nUD_23_NORWEGIAN_NYNORSKLIA_TRAIN = _UD_23_HOME + \"#UD_Norwegian-NynorskLIA/no_nynorsklia-ud-train.conllu\"\nUD_23_NORWEGIAN_NYNORSKLIA_TEST = _UD_23_HOME + \"#UD_Norwegian-NynorskLIA/no_nynorsklia-ud-test.conllu\"\nUD_23_OLD_CHURCH_SLAVONIC_PROIEL_TRAIN = _UD_23_HOME + \"#UD_Old_Church_Slavonic-PROIEL/cu_proiel-ud-train.conllu\"\nUD_23_OLD_CHURCH_SLAVONIC_PROIEL_DEV = _UD_23_HOME + \"#UD_Old_Church_Slavonic-PROIEL/cu_proiel-ud-dev.conllu\"\nUD_23_OLD_CHURCH_SLAVONIC_PROIEL_TEST = _UD_23_HOME + \"#UD_Old_Church_Slavonic-PROIEL/cu_proiel-ud-test.conllu\"\nUD_23_OLD_FRENCH_SRCMF_TRAIN = _UD_23_HOME + \"#UD_Old_French-SRCMF/fro_srcmf-ud-train.conllu\"\nUD_23_OLD_FRENCH_SRCMF_DEV = _UD_23_HOME + \"#UD_Old_French-SRCMF/fro_srcmf-ud-dev.conllu\"\nUD_23_OLD_FRENCH_SRCMF_TEST = _UD_23_HOME + \"#UD_Old_French-SRCMF/fro_srcmf-ud-test.conllu\"\nUD_23_PERSIAN_SERAJI_TRAIN = _UD_23_HOME + \"#UD_Persian-Seraji/fa_seraji-ud-train.conllu\"\nUD_23_PERSIAN_SERAJI_DEV = _UD_23_HOME + \"#UD_Persian-Seraji/fa_seraji-ud-dev.conllu\"\nUD_23_PERSIAN_SERAJI_TEST = _UD_23_HOME + \"#UD_Persian-Seraji/fa_seraji-ud-test.conllu\"\nUD_23_POLISH_LFG_TRAIN = _UD_23_HOME + \"#UD_Polish-LFG/pl_lfg-ud-train.conllu\"\nUD_23_POLISH_LFG_DEV = _UD_23_HOME + \"#UD_Polish-LFG/pl_lfg-ud-dev.conllu\"\nUD_23_POLISH_LFG_TEST = _UD_23_HOME + \"#UD_Polish-LFG/pl_lfg-ud-test.conllu\"\nUD_23_POLISH_SZ_TRAIN = _UD_23_HOME + \"#UD_Polish-SZ/pl_sz-ud-train.conllu\"\nUD_23_POLISH_SZ_DEV = _UD_23_HOME + \"#UD_Polish-SZ/pl_sz-ud-dev.conllu\"\nUD_23_POLISH_SZ_TEST = _UD_23_HOME + \"#UD_Polish-SZ/pl_sz-ud-test.conllu\"\nUD_23_PORTUGUESE_BOSQUE_TRAIN = _UD_23_HOME + \"#UD_Portuguese-Bosque/pt_bosque-ud-train.conllu\"\nUD_23_PORTUGUESE_BOSQUE_DEV = _UD_23_HOME + \"#UD_Portuguese-Bosque/pt_bosque-ud-dev.conllu\"\nUD_23_PORTUGUESE_BOSQUE_TEST = _UD_23_HOME + \"#UD_Portuguese-Bosque/pt_bosque-ud-test.conllu\"\nUD_23_PORTUGUESE_GSD_TRAIN = _UD_23_HOME + \"#UD_Portuguese-GSD/pt_gsd-ud-train.conllu\"\nUD_23_PORTUGUESE_GSD_DEV = _UD_23_HOME + \"#UD_Portuguese-GSD/pt_gsd-ud-dev.conllu\"\nUD_23_PORTUGUESE_GSD_TEST = _UD_23_HOME + \"#UD_Portuguese-GSD/pt_gsd-ud-test.conllu\"\nUD_23_PORTUGUESE_PUD_TEST = _UD_23_HOME + \"#UD_Portuguese-PUD/pt_pud-ud-test.conllu\"\nUD_23_ROMANIAN_NONSTANDARD_TRAIN = _UD_23_HOME + \"#UD_Romanian-Nonstandard/ro_nonstandard-ud-train.conllu\"\nUD_23_ROMANIAN_NONSTANDARD_DEV = _UD_23_HOME + \"#UD_Romanian-Nonstandard/ro_nonstandard-ud-dev.conllu\"\nUD_23_ROMANIAN_NONSTANDARD_TEST = _UD_23_HOME + \"#UD_Romanian-Nonstandard/ro_nonstandard-ud-test.conllu\"\nUD_23_ROMANIAN_RRT_TRAIN = _UD_23_HOME + \"#UD_Romanian-RRT/ro_rrt-ud-train.conllu\"\nUD_23_ROMANIAN_RRT_DEV = _UD_23_HOME + \"#UD_Romanian-RRT/ro_rrt-ud-dev.conllu\"\nUD_23_ROMANIAN_RRT_TEST = _UD_23_HOME + \"#UD_Romanian-RRT/ro_rrt-ud-test.conllu\"\nUD_23_RUSSIAN_GSD_TRAIN = _UD_23_HOME + \"#UD_Russian-GSD/ru_gsd-ud-train.conllu\"\nUD_23_RUSSIAN_GSD_DEV = _UD_23_HOME + \"#UD_Russian-GSD/ru_gsd-ud-dev.conllu\"\nUD_23_RUSSIAN_GSD_TEST = _UD_23_HOME + \"#UD_Russian-GSD/ru_gsd-ud-test.conllu\"\nUD_23_RUSSIAN_PUD_TEST = _UD_23_HOME + \"#UD_Russian-PUD/ru_pud-ud-test.conllu\"\nUD_23_RUSSIAN_SYNTAGRUS_TRAIN = _UD_23_HOME + \"#UD_Russian-SynTagRus/ru_syntagrus-ud-train.conllu\"\nUD_23_RUSSIAN_SYNTAGRUS_DEV = _UD_23_HOME + \"#UD_Russian-SynTagRus/ru_syntagrus-ud-dev.conllu\"\nUD_23_RUSSIAN_SYNTAGRUS_TEST = _UD_23_HOME + \"#UD_Russian-SynTagRus/ru_syntagrus-ud-test.conllu\"\nUD_23_RUSSIAN_TAIGA_TRAIN = _UD_23_HOME + \"#UD_Russian-Taiga/ru_taiga-ud-train.conllu\"\nUD_23_RUSSIAN_TAIGA_TEST = _UD_23_HOME + \"#UD_Russian-Taiga/ru_taiga-ud-test.conllu\"\nUD_23_SANSKRIT_UFAL_TEST = _UD_23_HOME + \"#UD_Sanskrit-UFAL/sa_ufal-ud-test.conllu\"\nUD_23_SERBIAN_SET_TRAIN = _UD_23_HOME + \"#UD_Serbian-SET/sr_set-ud-train.conllu\"\nUD_23_SERBIAN_SET_DEV = _UD_23_HOME + \"#UD_Serbian-SET/sr_set-ud-dev.conllu\"\nUD_23_SERBIAN_SET_TEST = _UD_23_HOME + \"#UD_Serbian-SET/sr_set-ud-test.conllu\"\nUD_23_SLOVAK_SNK_TRAIN = _UD_23_HOME + \"#UD_Slovak-SNK/sk_snk-ud-train.conllu\"\nUD_23_SLOVAK_SNK_DEV = _UD_23_HOME + \"#UD_Slovak-SNK/sk_snk-ud-dev.conllu\"\nUD_23_SLOVAK_SNK_TEST = _UD_23_HOME + \"#UD_Slovak-SNK/sk_snk-ud-test.conllu\"\nUD_23_SLOVENIAN_SSJ_TRAIN = _UD_23_HOME + \"#UD_Slovenian-SSJ/sl_ssj-ud-train.conllu\"\nUD_23_SLOVENIAN_SSJ_DEV = _UD_23_HOME + \"#UD_Slovenian-SSJ/sl_ssj-ud-dev.conllu\"\nUD_23_SLOVENIAN_SSJ_TEST = _UD_23_HOME + \"#UD_Slovenian-SSJ/sl_ssj-ud-test.conllu\"\nUD_23_SLOVENIAN_SST_TRAIN = _UD_23_HOME + \"#UD_Slovenian-SST/sl_sst-ud-train.conllu\"\nUD_23_SLOVENIAN_SST_TEST = _UD_23_HOME + \"#UD_Slovenian-SST/sl_sst-ud-test.conllu\"\nUD_23_SPANISH_ANCORA_TRAIN = _UD_23_HOME + \"#UD_Spanish-AnCora/es_ancora-ud-train.conllu\"\nUD_23_SPANISH_ANCORA_DEV = _UD_23_HOME + \"#UD_Spanish-AnCora/es_ancora-ud-dev.conllu\"\nUD_23_SPANISH_ANCORA_TEST = _UD_23_HOME + \"#UD_Spanish-AnCora/es_ancora-ud-test.conllu\"\nUD_23_SPANISH_GSD_TRAIN = _UD_23_HOME + \"#UD_Spanish-GSD/es_gsd-ud-train.conllu\"\nUD_23_SPANISH_GSD_DEV = _UD_23_HOME + \"#UD_Spanish-GSD/es_gsd-ud-dev.conllu\"\nUD_23_SPANISH_GSD_TEST = _UD_23_HOME + \"#UD_Spanish-GSD/es_gsd-ud-test.conllu\"\nUD_23_SPANISH_PUD_TEST = _UD_23_HOME + \"#UD_Spanish-PUD/es_pud-ud-test.conllu\"\nUD_23_SWEDISH_LINES_TRAIN = _UD_23_HOME + \"#UD_Swedish-LinES/sv_lines-ud-train.conllu\"\nUD_23_SWEDISH_LINES_DEV = _UD_23_HOME + \"#UD_Swedish-LinES/sv_lines-ud-dev.conllu\"\nUD_23_SWEDISH_LINES_TEST = _UD_23_HOME + \"#UD_Swedish-LinES/sv_lines-ud-test.conllu\"\nUD_23_SWEDISH_PUD_TEST = _UD_23_HOME + \"#UD_Swedish-PUD/sv_pud-ud-test.conllu\"\nUD_23_SWEDISH_TALBANKEN_TRAIN = _UD_23_HOME + \"#UD_Swedish-Talbanken/sv_talbanken-ud-train.conllu\"\nUD_23_SWEDISH_TALBANKEN_DEV = _UD_23_HOME + \"#UD_Swedish-Talbanken/sv_talbanken-ud-dev.conllu\"\nUD_23_SWEDISH_TALBANKEN_TEST = _UD_23_HOME + \"#UD_Swedish-Talbanken/sv_talbanken-ud-test.conllu\"\nUD_23_SWEDISH_SIGN_LANGUAGE_SSLC_TRAIN = _UD_23_HOME + \"#UD_Swedish_Sign_Language-SSLC/swl_sslc-ud-train.conllu\"\nUD_23_SWEDISH_SIGN_LANGUAGE_SSLC_DEV = _UD_23_HOME + \"#UD_Swedish_Sign_Language-SSLC/swl_sslc-ud-dev.conllu\"\nUD_23_SWEDISH_SIGN_LANGUAGE_SSLC_TEST = _UD_23_HOME + \"#UD_Swedish_Sign_Language-SSLC/swl_sslc-ud-test.conllu\"\nUD_23_TAGALOG_TRG_TEST = _UD_23_HOME + \"#UD_Tagalog-TRG/tl_trg-ud-test.conllu\"\nUD_23_TAMIL_TTB_TRAIN = _UD_23_HOME + \"#UD_Tamil-TTB/ta_ttb-ud-train.conllu\"\nUD_23_TAMIL_TTB_DEV = _UD_23_HOME + \"#UD_Tamil-TTB/ta_ttb-ud-dev.conllu\"\nUD_23_TAMIL_TTB_TEST = _UD_23_HOME + \"#UD_Tamil-TTB/ta_ttb-ud-test.conllu\"\nUD_23_TELUGU_MTG_TRAIN = _UD_23_HOME + \"#UD_Telugu-MTG/te_mtg-ud-train.conllu\"\nUD_23_TELUGU_MTG_DEV = _UD_23_HOME + \"#UD_Telugu-MTG/te_mtg-ud-dev.conllu\"\nUD_23_TELUGU_MTG_TEST = _UD_23_HOME + \"#UD_Telugu-MTG/te_mtg-ud-test.conllu\"\nUD_23_THAI_PUD_TEST = _UD_23_HOME + \"#UD_Thai-PUD/th_pud-ud-test.conllu\"\nUD_23_TURKISH_IMST_TRAIN = _UD_23_HOME + \"#UD_Turkish-IMST/tr_imst-ud-train.conllu\"\nUD_23_TURKISH_IMST_DEV = _UD_23_HOME + \"#UD_Turkish-IMST/tr_imst-ud-dev.conllu\"\nUD_23_TURKISH_IMST_TEST = _UD_23_HOME + \"#UD_Turkish-IMST/tr_imst-ud-test.conllu\"\nUD_23_TURKISH_PUD_TEST = _UD_23_HOME + \"#UD_Turkish-PUD/tr_pud-ud-test.conllu\"\nUD_23_UKRAINIAN_IU_TRAIN = _UD_23_HOME + \"#UD_Ukrainian-IU/uk_iu-ud-train.conllu\"\nUD_23_UKRAINIAN_IU_DEV = _UD_23_HOME + \"#UD_Ukrainian-IU/uk_iu-ud-dev.conllu\"\nUD_23_UKRAINIAN_IU_TEST = _UD_23_HOME + \"#UD_Ukrainian-IU/uk_iu-ud-test.conllu\"\nUD_23_UPPER_SORBIAN_UFAL_TRAIN = _UD_23_HOME + \"#UD_Upper_Sorbian-UFAL/hsb_ufal-ud-train.conllu\"\nUD_23_UPPER_SORBIAN_UFAL_TEST = _UD_23_HOME + \"#UD_Upper_Sorbian-UFAL/hsb_ufal-ud-test.conllu\"\nUD_23_URDU_UDTB_TRAIN = _UD_23_HOME + \"#UD_Urdu-UDTB/ur_udtb-ud-train.conllu\"\nUD_23_URDU_UDTB_DEV = _UD_23_HOME + \"#UD_Urdu-UDTB/ur_udtb-ud-dev.conllu\"\nUD_23_URDU_UDTB_TEST = _UD_23_HOME + \"#UD_Urdu-UDTB/ur_udtb-ud-test.conllu\"\nUD_23_UYGHUR_UDT_TRAIN = _UD_23_HOME + \"#UD_Uyghur-UDT/ug_udt-ud-train.conllu\"\nUD_23_UYGHUR_UDT_DEV = _UD_23_HOME + \"#UD_Uyghur-UDT/ug_udt-ud-dev.conllu\"\nUD_23_UYGHUR_UDT_TEST = _UD_23_HOME + \"#UD_Uyghur-UDT/ug_udt-ud-test.conllu\"\nUD_23_VIETNAMESE_VTB_TRAIN = _UD_23_HOME + \"#UD_Vietnamese-VTB/vi_vtb-ud-train.conllu\"\nUD_23_VIETNAMESE_VTB_DEV = _UD_23_HOME + \"#UD_Vietnamese-VTB/vi_vtb-ud-dev.conllu\"\nUD_23_VIETNAMESE_VTB_TEST = _UD_23_HOME + \"#UD_Vietnamese-VTB/vi_vtb-ud-test.conllu\"\nUD_23_WARLPIRI_UFAL_TEST = _UD_23_HOME + \"#UD_Warlpiri-UFAL/wbp_ufal-ud-test.conllu\"\nUD_23_YORUBA_YTB_TEST = _UD_23_HOME + \"#UD_Yoruba-YTB/yo_ytb-ud-test.conllu\"\n"
  },
  {
    "path": "hanlp/datasets/parsing/ud/ud23m.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-21 20:39\nimport os\n\nfrom hanlp.datasets.parsing.ud import concat_treebanks\nfrom .ud23 import _UD_23_HOME\n\n_UD_23_MULTILINGUAL_HOME = concat_treebanks(_UD_23_HOME, '2.3')\nUD_23_MULTILINGUAL_TRAIN = os.path.join(_UD_23_MULTILINGUAL_HOME, 'train.conllu')\nUD_23_MULTILINGUAL_DEV = os.path.join(_UD_23_MULTILINGUAL_HOME, 'dev.conllu')\nUD_23_MULTILINGUAL_TEST = os.path.join(_UD_23_MULTILINGUAL_HOME, 'test.conllu')\n"
  },
  {
    "path": "hanlp/datasets/parsing/ud/ud27.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-07 21:03\nimport glob\nimport os\n\nfrom hanlp.utils.io_util import uncompress, get_resource\n\n_UD_27_URL = \"https://lindat.mff.cuni.cz/repository/xmlui/handle/11234/1-3424/allzip\"\n_UD_27_HOME = _UD_27_URL + '#ud-treebanks-v2.7/'\n_path = get_resource(_UD_27_URL)\nif os.path.isfile(_path):\n    os.rename(_path, _path + '.zip')\n    uncompress(_path + '.zip')\n    uncompress(os.path.join(_path, 'ud-treebanks-v2.7.tgz'))\n\n\n# noinspection PyShadowingNames\ndef _list_dir(path, home):\n    prefix = home.lstrip('_').replace('_HOME', '')\n\n    path = get_resource(path)\n    with open('ud27.py', 'a') as out:\n        for f in sorted(glob.glob(path + '/ud-treebanks-v2.7/UD_*')):\n            basename = os.path.basename(f)\n            name = basename[len('UD_'):]\n            name = name.upper().replace('-', '_')\n            for split in 'train', 'dev', 'test':\n                sp = glob.glob(f + f'/*{split}.conllu')\n                if not sp:\n                    continue\n                sp = os.path.basename(sp[0])\n                out.write(f'{prefix}_{name}_{split.upper()} = {home} + \"{basename}/{sp}\"\\n')\n                out.write(f'\"{prefix} {split} set of {name}.\"\\n')\n\n\ndef main():\n    _list_dir(_UD_27_URL, '_UD_27_HOME')\n    pass\n\n\nif __name__ == '__main__':\n    main()\nUD_27_AFRIKAANS_AFRIBOOMS_TRAIN = _UD_27_HOME + \"UD_Afrikaans-AfriBooms/af_afribooms-ud-train.conllu\"\n\"UD_27 train set of AFRIKAANS_AFRIBOOMS.\"\nUD_27_AFRIKAANS_AFRIBOOMS_DEV = _UD_27_HOME + \"UD_Afrikaans-AfriBooms/af_afribooms-ud-dev.conllu\"\n\"UD_27 dev set of AFRIKAANS_AFRIBOOMS.\"\nUD_27_AFRIKAANS_AFRIBOOMS_TEST = _UD_27_HOME + \"UD_Afrikaans-AfriBooms/af_afribooms-ud-test.conllu\"\n\"UD_27 test set of AFRIKAANS_AFRIBOOMS.\"\nUD_27_AKKADIAN_PISANDUB_TEST = _UD_27_HOME + \"UD_Akkadian-PISANDUB/akk_pisandub-ud-test.conllu\"\n\"UD_27 test set of AKKADIAN_PISANDUB.\"\nUD_27_AKKADIAN_RIAO_TEST = _UD_27_HOME + \"UD_Akkadian-RIAO/akk_riao-ud-test.conllu\"\n\"UD_27 test set of AKKADIAN_RIAO.\"\nUD_27_AKUNTSU_TUDET_TEST = _UD_27_HOME + \"UD_Akuntsu-TuDeT/aqz_tudet-ud-test.conllu\"\n\"UD_27 test set of AKUNTSU_TUDET.\"\nUD_27_ALBANIAN_TSA_TEST = _UD_27_HOME + \"UD_Albanian-TSA/sq_tsa-ud-test.conllu\"\n\"UD_27 test set of ALBANIAN_TSA.\"\nUD_27_AMHARIC_ATT_TEST = _UD_27_HOME + \"UD_Amharic-ATT/am_att-ud-test.conllu\"\n\"UD_27 test set of AMHARIC_ATT.\"\nUD_27_ANCIENT_GREEK_PROIEL_TRAIN = _UD_27_HOME + \"UD_Ancient_Greek-PROIEL/grc_proiel-ud-train.conllu\"\n\"UD_27 train set of ANCIENT_GREEK_PROIEL.\"\nUD_27_ANCIENT_GREEK_PROIEL_DEV = _UD_27_HOME + \"UD_Ancient_Greek-PROIEL/grc_proiel-ud-dev.conllu\"\n\"UD_27 dev set of ANCIENT_GREEK_PROIEL.\"\nUD_27_ANCIENT_GREEK_PROIEL_TEST = _UD_27_HOME + \"UD_Ancient_Greek-PROIEL/grc_proiel-ud-test.conllu\"\n\"UD_27 test set of ANCIENT_GREEK_PROIEL.\"\nUD_27_ANCIENT_GREEK_PERSEUS_TRAIN = _UD_27_HOME + \"UD_Ancient_Greek-Perseus/grc_perseus-ud-train.conllu\"\n\"UD_27 train set of ANCIENT_GREEK_PERSEUS.\"\nUD_27_ANCIENT_GREEK_PERSEUS_DEV = _UD_27_HOME + \"UD_Ancient_Greek-Perseus/grc_perseus-ud-dev.conllu\"\n\"UD_27 dev set of ANCIENT_GREEK_PERSEUS.\"\nUD_27_ANCIENT_GREEK_PERSEUS_TEST = _UD_27_HOME + \"UD_Ancient_Greek-Perseus/grc_perseus-ud-test.conllu\"\n\"UD_27 test set of ANCIENT_GREEK_PERSEUS.\"\nUD_27_APURINA_UFPA_TEST = _UD_27_HOME + \"UD_Apurina-UFPA/apu_ufpa-ud-test.conllu\"\n\"UD_27 test set of APURINA_UFPA.\"\nUD_27_ARABIC_NYUAD_TRAIN = _UD_27_HOME + \"UD_Arabic-NYUAD/ar_nyuad-ud-train.conllu\"\n\"UD_27 train set of ARABIC_NYUAD.\"\nUD_27_ARABIC_NYUAD_DEV = _UD_27_HOME + \"UD_Arabic-NYUAD/ar_nyuad-ud-dev.conllu\"\n\"UD_27 dev set of ARABIC_NYUAD.\"\nUD_27_ARABIC_NYUAD_TEST = _UD_27_HOME + \"UD_Arabic-NYUAD/ar_nyuad-ud-test.conllu\"\n\"UD_27 test set of ARABIC_NYUAD.\"\nUD_27_ARABIC_PADT_TRAIN = _UD_27_HOME + \"UD_Arabic-PADT/ar_padt-ud-train.conllu\"\n\"UD_27 train set of ARABIC_PADT.\"\nUD_27_ARABIC_PADT_DEV = _UD_27_HOME + \"UD_Arabic-PADT/ar_padt-ud-dev.conllu\"\n\"UD_27 dev set of ARABIC_PADT.\"\nUD_27_ARABIC_PADT_TEST = _UD_27_HOME + \"UD_Arabic-PADT/ar_padt-ud-test.conllu\"\n\"UD_27 test set of ARABIC_PADT.\"\nUD_27_ARABIC_PUD_TEST = _UD_27_HOME + \"UD_Arabic-PUD/ar_pud-ud-test.conllu\"\n\"UD_27 test set of ARABIC_PUD.\"\nUD_27_ARMENIAN_ARMTDP_TRAIN = _UD_27_HOME + \"UD_Armenian-ArmTDP/hy_armtdp-ud-train.conllu\"\n\"UD_27 train set of ARMENIAN_ARMTDP.\"\nUD_27_ARMENIAN_ARMTDP_DEV = _UD_27_HOME + \"UD_Armenian-ArmTDP/hy_armtdp-ud-dev.conllu\"\n\"UD_27 dev set of ARMENIAN_ARMTDP.\"\nUD_27_ARMENIAN_ARMTDP_TEST = _UD_27_HOME + \"UD_Armenian-ArmTDP/hy_armtdp-ud-test.conllu\"\n\"UD_27 test set of ARMENIAN_ARMTDP.\"\nUD_27_ASSYRIAN_AS_TEST = _UD_27_HOME + \"UD_Assyrian-AS/aii_as-ud-test.conllu\"\n\"UD_27 test set of ASSYRIAN_AS.\"\nUD_27_BAMBARA_CRB_TEST = _UD_27_HOME + \"UD_Bambara-CRB/bm_crb-ud-test.conllu\"\n\"UD_27 test set of BAMBARA_CRB.\"\nUD_27_BASQUE_BDT_TRAIN = _UD_27_HOME + \"UD_Basque-BDT/eu_bdt-ud-train.conllu\"\n\"UD_27 train set of BASQUE_BDT.\"\nUD_27_BASQUE_BDT_DEV = _UD_27_HOME + \"UD_Basque-BDT/eu_bdt-ud-dev.conllu\"\n\"UD_27 dev set of BASQUE_BDT.\"\nUD_27_BASQUE_BDT_TEST = _UD_27_HOME + \"UD_Basque-BDT/eu_bdt-ud-test.conllu\"\n\"UD_27 test set of BASQUE_BDT.\"\nUD_27_BELARUSIAN_HSE_TRAIN = _UD_27_HOME + \"UD_Belarusian-HSE/be_hse-ud-train.conllu\"\n\"UD_27 train set of BELARUSIAN_HSE.\"\nUD_27_BELARUSIAN_HSE_DEV = _UD_27_HOME + \"UD_Belarusian-HSE/be_hse-ud-dev.conllu\"\n\"UD_27 dev set of BELARUSIAN_HSE.\"\nUD_27_BELARUSIAN_HSE_TEST = _UD_27_HOME + \"UD_Belarusian-HSE/be_hse-ud-test.conllu\"\n\"UD_27 test set of BELARUSIAN_HSE.\"\nUD_27_BHOJPURI_BHTB_TEST = _UD_27_HOME + \"UD_Bhojpuri-BHTB/bho_bhtb-ud-test.conllu\"\n\"UD_27 test set of BHOJPURI_BHTB.\"\nUD_27_BRETON_KEB_TEST = _UD_27_HOME + \"UD_Breton-KEB/br_keb-ud-test.conllu\"\n\"UD_27 test set of BRETON_KEB.\"\nUD_27_BULGARIAN_BTB_TRAIN = _UD_27_HOME + \"UD_Bulgarian-BTB/bg_btb-ud-train.conllu\"\n\"UD_27 train set of BULGARIAN_BTB.\"\nUD_27_BULGARIAN_BTB_DEV = _UD_27_HOME + \"UD_Bulgarian-BTB/bg_btb-ud-dev.conllu\"\n\"UD_27 dev set of BULGARIAN_BTB.\"\nUD_27_BULGARIAN_BTB_TEST = _UD_27_HOME + \"UD_Bulgarian-BTB/bg_btb-ud-test.conllu\"\n\"UD_27 test set of BULGARIAN_BTB.\"\nUD_27_BURYAT_BDT_TRAIN = _UD_27_HOME + \"UD_Buryat-BDT/bxr_bdt-ud-train.conllu\"\n\"UD_27 train set of BURYAT_BDT.\"\nUD_27_BURYAT_BDT_TEST = _UD_27_HOME + \"UD_Buryat-BDT/bxr_bdt-ud-test.conllu\"\n\"UD_27 test set of BURYAT_BDT.\"\nUD_27_CANTONESE_HK_TEST = _UD_27_HOME + \"UD_Cantonese-HK/yue_hk-ud-test.conllu\"\n\"UD_27 test set of CANTONESE_HK.\"\nUD_27_CATALAN_ANCORA_TRAIN = _UD_27_HOME + \"UD_Catalan-AnCora/ca_ancora-ud-train.conllu\"\n\"UD_27 train set of CATALAN_ANCORA.\"\nUD_27_CATALAN_ANCORA_DEV = _UD_27_HOME + \"UD_Catalan-AnCora/ca_ancora-ud-dev.conllu\"\n\"UD_27 dev set of CATALAN_ANCORA.\"\nUD_27_CATALAN_ANCORA_TEST = _UD_27_HOME + \"UD_Catalan-AnCora/ca_ancora-ud-test.conllu\"\n\"UD_27 test set of CATALAN_ANCORA.\"\nUD_27_CHINESE_CFL_TEST = _UD_27_HOME + \"UD_Chinese-CFL/zh_cfl-ud-test.conllu\"\n\"UD_27 test set of CHINESE_CFL.\"\nUD_27_CHINESE_GSD_TRAIN = _UD_27_HOME + \"UD_Chinese-GSD/zh_gsd-ud-train.conllu\"\n\"UD_27 train set of CHINESE_GSD.\"\nUD_27_CHINESE_GSD_DEV = _UD_27_HOME + \"UD_Chinese-GSD/zh_gsd-ud-dev.conllu\"\n\"UD_27 dev set of CHINESE_GSD.\"\nUD_27_CHINESE_GSD_TEST = _UD_27_HOME + \"UD_Chinese-GSD/zh_gsd-ud-test.conllu\"\n\"UD_27 test set of CHINESE_GSD.\"\nUD_27_CHINESE_GSDSIMP_TRAIN = _UD_27_HOME + \"UD_Chinese-GSDSimp/zh_gsdsimp-ud-train.conllu\"\n\"UD_27 train set of CHINESE_GSDSIMP.\"\nUD_27_CHINESE_GSDSIMP_DEV = _UD_27_HOME + \"UD_Chinese-GSDSimp/zh_gsdsimp-ud-dev.conllu\"\n\"UD_27 dev set of CHINESE_GSDSIMP.\"\nUD_27_CHINESE_GSDSIMP_TEST = _UD_27_HOME + \"UD_Chinese-GSDSimp/zh_gsdsimp-ud-test.conllu\"\n\"UD_27 test set of CHINESE_GSDSIMP.\"\nUD_27_CHINESE_HK_TEST = _UD_27_HOME + \"UD_Chinese-HK/zh_hk-ud-test.conllu\"\n\"UD_27 test set of CHINESE_HK.\"\nUD_27_CHINESE_PUD_TEST = _UD_27_HOME + \"UD_Chinese-PUD/zh_pud-ud-test.conllu\"\n\"UD_27 test set of CHINESE_PUD.\"\nUD_27_CHUKCHI_HSE_TEST = _UD_27_HOME + \"UD_Chukchi-HSE/ckt_hse-ud-test.conllu\"\n\"UD_27 test set of CHUKCHI_HSE.\"\nUD_27_CLASSICAL_CHINESE_KYOTO_TRAIN = _UD_27_HOME + \"UD_Classical_Chinese-Kyoto/lzh_kyoto-ud-train.conllu\"\n\"UD_27 train set of CLASSICAL_CHINESE_KYOTO.\"\nUD_27_CLASSICAL_CHINESE_KYOTO_DEV = _UD_27_HOME + \"UD_Classical_Chinese-Kyoto/lzh_kyoto-ud-dev.conllu\"\n\"UD_27 dev set of CLASSICAL_CHINESE_KYOTO.\"\nUD_27_CLASSICAL_CHINESE_KYOTO_TEST = _UD_27_HOME + \"UD_Classical_Chinese-Kyoto/lzh_kyoto-ud-test.conllu\"\n\"UD_27 test set of CLASSICAL_CHINESE_KYOTO.\"\nUD_27_COPTIC_SCRIPTORIUM_TRAIN = _UD_27_HOME + \"UD_Coptic-Scriptorium/cop_scriptorium-ud-train.conllu\"\n\"UD_27 train set of COPTIC_SCRIPTORIUM.\"\nUD_27_COPTIC_SCRIPTORIUM_DEV = _UD_27_HOME + \"UD_Coptic-Scriptorium/cop_scriptorium-ud-dev.conllu\"\n\"UD_27 dev set of COPTIC_SCRIPTORIUM.\"\nUD_27_COPTIC_SCRIPTORIUM_TEST = _UD_27_HOME + \"UD_Coptic-Scriptorium/cop_scriptorium-ud-test.conllu\"\n\"UD_27 test set of COPTIC_SCRIPTORIUM.\"\nUD_27_CROATIAN_SET_TRAIN = _UD_27_HOME + \"UD_Croatian-SET/hr_set-ud-train.conllu\"\n\"UD_27 train set of CROATIAN_SET.\"\nUD_27_CROATIAN_SET_DEV = _UD_27_HOME + \"UD_Croatian-SET/hr_set-ud-dev.conllu\"\n\"UD_27 dev set of CROATIAN_SET.\"\nUD_27_CROATIAN_SET_TEST = _UD_27_HOME + \"UD_Croatian-SET/hr_set-ud-test.conllu\"\n\"UD_27 test set of CROATIAN_SET.\"\nUD_27_CZECH_CAC_TRAIN = _UD_27_HOME + \"UD_Czech-CAC/cs_cac-ud-train.conllu\"\n\"UD_27 train set of CZECH_CAC.\"\nUD_27_CZECH_CAC_DEV = _UD_27_HOME + \"UD_Czech-CAC/cs_cac-ud-dev.conllu\"\n\"UD_27 dev set of CZECH_CAC.\"\nUD_27_CZECH_CAC_TEST = _UD_27_HOME + \"UD_Czech-CAC/cs_cac-ud-test.conllu\"\n\"UD_27 test set of CZECH_CAC.\"\nUD_27_CZECH_CLTT_TRAIN = _UD_27_HOME + \"UD_Czech-CLTT/cs_cltt-ud-train.conllu\"\n\"UD_27 train set of CZECH_CLTT.\"\nUD_27_CZECH_CLTT_DEV = _UD_27_HOME + \"UD_Czech-CLTT/cs_cltt-ud-dev.conllu\"\n\"UD_27 dev set of CZECH_CLTT.\"\nUD_27_CZECH_CLTT_TEST = _UD_27_HOME + \"UD_Czech-CLTT/cs_cltt-ud-test.conllu\"\n\"UD_27 test set of CZECH_CLTT.\"\nUD_27_CZECH_FICTREE_TRAIN = _UD_27_HOME + \"UD_Czech-FicTree/cs_fictree-ud-train.conllu\"\n\"UD_27 train set of CZECH_FICTREE.\"\nUD_27_CZECH_FICTREE_DEV = _UD_27_HOME + \"UD_Czech-FicTree/cs_fictree-ud-dev.conllu\"\n\"UD_27 dev set of CZECH_FICTREE.\"\nUD_27_CZECH_FICTREE_TEST = _UD_27_HOME + \"UD_Czech-FicTree/cs_fictree-ud-test.conllu\"\n\"UD_27 test set of CZECH_FICTREE.\"\nUD_27_CZECH_PDT_TRAIN = _UD_27_HOME + \"UD_Czech-PDT/cs_pdt-ud-train.conllu\"\n\"UD_27 train set of CZECH_PDT.\"\nUD_27_CZECH_PDT_DEV = _UD_27_HOME + \"UD_Czech-PDT/cs_pdt-ud-dev.conllu\"\n\"UD_27 dev set of CZECH_PDT.\"\nUD_27_CZECH_PDT_TEST = _UD_27_HOME + \"UD_Czech-PDT/cs_pdt-ud-test.conllu\"\n\"UD_27 test set of CZECH_PDT.\"\nUD_27_CZECH_PUD_TEST = _UD_27_HOME + \"UD_Czech-PUD/cs_pud-ud-test.conllu\"\n\"UD_27 test set of CZECH_PUD.\"\nUD_27_DANISH_DDT_TRAIN = _UD_27_HOME + \"UD_Danish-DDT/da_ddt-ud-train.conllu\"\n\"UD_27 train set of DANISH_DDT.\"\nUD_27_DANISH_DDT_DEV = _UD_27_HOME + \"UD_Danish-DDT/da_ddt-ud-dev.conllu\"\n\"UD_27 dev set of DANISH_DDT.\"\nUD_27_DANISH_DDT_TEST = _UD_27_HOME + \"UD_Danish-DDT/da_ddt-ud-test.conllu\"\n\"UD_27 test set of DANISH_DDT.\"\nUD_27_DUTCH_ALPINO_TRAIN = _UD_27_HOME + \"UD_Dutch-Alpino/nl_alpino-ud-train.conllu\"\n\"UD_27 train set of DUTCH_ALPINO.\"\nUD_27_DUTCH_ALPINO_DEV = _UD_27_HOME + \"UD_Dutch-Alpino/nl_alpino-ud-dev.conllu\"\n\"UD_27 dev set of DUTCH_ALPINO.\"\nUD_27_DUTCH_ALPINO_TEST = _UD_27_HOME + \"UD_Dutch-Alpino/nl_alpino-ud-test.conllu\"\n\"UD_27 test set of DUTCH_ALPINO.\"\nUD_27_DUTCH_LASSYSMALL_TRAIN = _UD_27_HOME + \"UD_Dutch-LassySmall/nl_lassysmall-ud-train.conllu\"\n\"UD_27 train set of DUTCH_LASSYSMALL.\"\nUD_27_DUTCH_LASSYSMALL_DEV = _UD_27_HOME + \"UD_Dutch-LassySmall/nl_lassysmall-ud-dev.conllu\"\n\"UD_27 dev set of DUTCH_LASSYSMALL.\"\nUD_27_DUTCH_LASSYSMALL_TEST = _UD_27_HOME + \"UD_Dutch-LassySmall/nl_lassysmall-ud-test.conllu\"\n\"UD_27 test set of DUTCH_LASSYSMALL.\"\nUD_27_ENGLISH_ESL_TRAIN = _UD_27_HOME + \"UD_English-ESL/en_esl-ud-train.conllu\"\n\"UD_27 train set of ENGLISH_ESL.\"\nUD_27_ENGLISH_ESL_DEV = _UD_27_HOME + \"UD_English-ESL/en_esl-ud-dev.conllu\"\n\"UD_27 dev set of ENGLISH_ESL.\"\nUD_27_ENGLISH_ESL_TEST = _UD_27_HOME + \"UD_English-ESL/en_esl-ud-test.conllu\"\n\"UD_27 test set of ENGLISH_ESL.\"\nUD_27_ENGLISH_EWT_TRAIN = _UD_27_HOME + \"UD_English-EWT/en_ewt-ud-train.conllu\"\n\"UD_27 train set of ENGLISH_EWT.\"\nUD_27_ENGLISH_EWT_DEV = _UD_27_HOME + \"UD_English-EWT/en_ewt-ud-dev.conllu\"\n\"UD_27 dev set of ENGLISH_EWT.\"\nUD_27_ENGLISH_EWT_TEST = _UD_27_HOME + \"UD_English-EWT/en_ewt-ud-test.conllu\"\n\"UD_27 test set of ENGLISH_EWT.\"\nUD_27_ENGLISH_GUM_TRAIN = _UD_27_HOME + \"UD_English-GUM/en_gum-ud-train.conllu\"\n\"UD_27 train set of ENGLISH_GUM.\"\nUD_27_ENGLISH_GUM_DEV = _UD_27_HOME + \"UD_English-GUM/en_gum-ud-dev.conllu\"\n\"UD_27 dev set of ENGLISH_GUM.\"\nUD_27_ENGLISH_GUM_TEST = _UD_27_HOME + \"UD_English-GUM/en_gum-ud-test.conllu\"\n\"UD_27 test set of ENGLISH_GUM.\"\nUD_27_ENGLISH_GUMREDDIT_TRAIN = _UD_27_HOME + \"UD_English-GUMReddit/en_gumreddit-ud-train.conllu\"\n\"UD_27 train set of ENGLISH_GUMREDDIT.\"\nUD_27_ENGLISH_GUMREDDIT_DEV = _UD_27_HOME + \"UD_English-GUMReddit/en_gumreddit-ud-dev.conllu\"\n\"UD_27 dev set of ENGLISH_GUMREDDIT.\"\nUD_27_ENGLISH_GUMREDDIT_TEST = _UD_27_HOME + \"UD_English-GUMReddit/en_gumreddit-ud-test.conllu\"\n\"UD_27 test set of ENGLISH_GUMREDDIT.\"\nUD_27_ENGLISH_LINES_TRAIN = _UD_27_HOME + \"UD_English-LinES/en_lines-ud-train.conllu\"\n\"UD_27 train set of ENGLISH_LINES.\"\nUD_27_ENGLISH_LINES_DEV = _UD_27_HOME + \"UD_English-LinES/en_lines-ud-dev.conllu\"\n\"UD_27 dev set of ENGLISH_LINES.\"\nUD_27_ENGLISH_LINES_TEST = _UD_27_HOME + \"UD_English-LinES/en_lines-ud-test.conllu\"\n\"UD_27 test set of ENGLISH_LINES.\"\nUD_27_ENGLISH_PUD_TEST = _UD_27_HOME + \"UD_English-PUD/en_pud-ud-test.conllu\"\n\"UD_27 test set of ENGLISH_PUD.\"\nUD_27_ENGLISH_PARTUT_TRAIN = _UD_27_HOME + \"UD_English-ParTUT/en_partut-ud-train.conllu\"\n\"UD_27 train set of ENGLISH_PARTUT.\"\nUD_27_ENGLISH_PARTUT_DEV = _UD_27_HOME + \"UD_English-ParTUT/en_partut-ud-dev.conllu\"\n\"UD_27 dev set of ENGLISH_PARTUT.\"\nUD_27_ENGLISH_PARTUT_TEST = _UD_27_HOME + \"UD_English-ParTUT/en_partut-ud-test.conllu\"\n\"UD_27 test set of ENGLISH_PARTUT.\"\nUD_27_ENGLISH_PRONOUNS_TEST = _UD_27_HOME + \"UD_English-Pronouns/en_pronouns-ud-test.conllu\"\n\"UD_27 test set of ENGLISH_PRONOUNS.\"\nUD_27_ERZYA_JR_TEST = _UD_27_HOME + \"UD_Erzya-JR/myv_jr-ud-test.conllu\"\n\"UD_27 test set of ERZYA_JR.\"\nUD_27_ESTONIAN_EDT_TRAIN = _UD_27_HOME + \"UD_Estonian-EDT/et_edt-ud-train.conllu\"\n\"UD_27 train set of ESTONIAN_EDT.\"\nUD_27_ESTONIAN_EDT_DEV = _UD_27_HOME + \"UD_Estonian-EDT/et_edt-ud-dev.conllu\"\n\"UD_27 dev set of ESTONIAN_EDT.\"\nUD_27_ESTONIAN_EDT_TEST = _UD_27_HOME + \"UD_Estonian-EDT/et_edt-ud-test.conllu\"\n\"UD_27 test set of ESTONIAN_EDT.\"\nUD_27_ESTONIAN_EWT_TRAIN = _UD_27_HOME + \"UD_Estonian-EWT/et_ewt-ud-train.conllu\"\n\"UD_27 train set of ESTONIAN_EWT.\"\nUD_27_ESTONIAN_EWT_DEV = _UD_27_HOME + \"UD_Estonian-EWT/et_ewt-ud-dev.conllu\"\n\"UD_27 dev set of ESTONIAN_EWT.\"\nUD_27_ESTONIAN_EWT_TEST = _UD_27_HOME + \"UD_Estonian-EWT/et_ewt-ud-test.conllu\"\n\"UD_27 test set of ESTONIAN_EWT.\"\nUD_27_FAROESE_FARPAHC_TRAIN = _UD_27_HOME + \"UD_Faroese-FarPaHC/fo_farpahc-ud-train.conllu\"\n\"UD_27 train set of FAROESE_FARPAHC.\"\nUD_27_FAROESE_FARPAHC_DEV = _UD_27_HOME + \"UD_Faroese-FarPaHC/fo_farpahc-ud-dev.conllu\"\n\"UD_27 dev set of FAROESE_FARPAHC.\"\nUD_27_FAROESE_FARPAHC_TEST = _UD_27_HOME + \"UD_Faroese-FarPaHC/fo_farpahc-ud-test.conllu\"\n\"UD_27 test set of FAROESE_FARPAHC.\"\nUD_27_FAROESE_OFT_TEST = _UD_27_HOME + \"UD_Faroese-OFT/fo_oft-ud-test.conllu\"\n\"UD_27 test set of FAROESE_OFT.\"\nUD_27_FINNISH_FTB_TRAIN = _UD_27_HOME + \"UD_Finnish-FTB/fi_ftb-ud-train.conllu\"\n\"UD_27 train set of FINNISH_FTB.\"\nUD_27_FINNISH_FTB_DEV = _UD_27_HOME + \"UD_Finnish-FTB/fi_ftb-ud-dev.conllu\"\n\"UD_27 dev set of FINNISH_FTB.\"\nUD_27_FINNISH_FTB_TEST = _UD_27_HOME + \"UD_Finnish-FTB/fi_ftb-ud-test.conllu\"\n\"UD_27 test set of FINNISH_FTB.\"\nUD_27_FINNISH_OOD_TEST = _UD_27_HOME + \"UD_Finnish-OOD/fi_ood-ud-test.conllu\"\n\"UD_27 test set of FINNISH_OOD.\"\nUD_27_FINNISH_PUD_TEST = _UD_27_HOME + \"UD_Finnish-PUD/fi_pud-ud-test.conllu\"\n\"UD_27 test set of FINNISH_PUD.\"\nUD_27_FINNISH_TDT_TRAIN = _UD_27_HOME + \"UD_Finnish-TDT/fi_tdt-ud-train.conllu\"\n\"UD_27 train set of FINNISH_TDT.\"\nUD_27_FINNISH_TDT_DEV = _UD_27_HOME + \"UD_Finnish-TDT/fi_tdt-ud-dev.conllu\"\n\"UD_27 dev set of FINNISH_TDT.\"\nUD_27_FINNISH_TDT_TEST = _UD_27_HOME + \"UD_Finnish-TDT/fi_tdt-ud-test.conllu\"\n\"UD_27 test set of FINNISH_TDT.\"\nUD_27_FRENCH_FQB_TEST = _UD_27_HOME + \"UD_French-FQB/fr_fqb-ud-test.conllu\"\n\"UD_27 test set of FRENCH_FQB.\"\nUD_27_FRENCH_FTB_TRAIN = _UD_27_HOME + \"UD_French-FTB/fr_ftb-ud-train.conllu\"\n\"UD_27 train set of FRENCH_FTB.\"\nUD_27_FRENCH_FTB_DEV = _UD_27_HOME + \"UD_French-FTB/fr_ftb-ud-dev.conllu\"\n\"UD_27 dev set of FRENCH_FTB.\"\nUD_27_FRENCH_FTB_TEST = _UD_27_HOME + \"UD_French-FTB/fr_ftb-ud-test.conllu\"\n\"UD_27 test set of FRENCH_FTB.\"\nUD_27_FRENCH_GSD_TRAIN = _UD_27_HOME + \"UD_French-GSD/fr_gsd-ud-train.conllu\"\n\"UD_27 train set of FRENCH_GSD.\"\nUD_27_FRENCH_GSD_DEV = _UD_27_HOME + \"UD_French-GSD/fr_gsd-ud-dev.conllu\"\n\"UD_27 dev set of FRENCH_GSD.\"\nUD_27_FRENCH_GSD_TEST = _UD_27_HOME + \"UD_French-GSD/fr_gsd-ud-test.conllu\"\n\"UD_27 test set of FRENCH_GSD.\"\nUD_27_FRENCH_PUD_TEST = _UD_27_HOME + \"UD_French-PUD/fr_pud-ud-test.conllu\"\n\"UD_27 test set of FRENCH_PUD.\"\nUD_27_FRENCH_PARTUT_TRAIN = _UD_27_HOME + \"UD_French-ParTUT/fr_partut-ud-train.conllu\"\n\"UD_27 train set of FRENCH_PARTUT.\"\nUD_27_FRENCH_PARTUT_DEV = _UD_27_HOME + \"UD_French-ParTUT/fr_partut-ud-dev.conllu\"\n\"UD_27 dev set of FRENCH_PARTUT.\"\nUD_27_FRENCH_PARTUT_TEST = _UD_27_HOME + \"UD_French-ParTUT/fr_partut-ud-test.conllu\"\n\"UD_27 test set of FRENCH_PARTUT.\"\nUD_27_FRENCH_SEQUOIA_TRAIN = _UD_27_HOME + \"UD_French-Sequoia/fr_sequoia-ud-train.conllu\"\n\"UD_27 train set of FRENCH_SEQUOIA.\"\nUD_27_FRENCH_SEQUOIA_DEV = _UD_27_HOME + \"UD_French-Sequoia/fr_sequoia-ud-dev.conllu\"\n\"UD_27 dev set of FRENCH_SEQUOIA.\"\nUD_27_FRENCH_SEQUOIA_TEST = _UD_27_HOME + \"UD_French-Sequoia/fr_sequoia-ud-test.conllu\"\n\"UD_27 test set of FRENCH_SEQUOIA.\"\nUD_27_FRENCH_SPOKEN_TRAIN = _UD_27_HOME + \"UD_French-Spoken/fr_spoken-ud-train.conllu\"\n\"UD_27 train set of FRENCH_SPOKEN.\"\nUD_27_FRENCH_SPOKEN_DEV = _UD_27_HOME + \"UD_French-Spoken/fr_spoken-ud-dev.conllu\"\n\"UD_27 dev set of FRENCH_SPOKEN.\"\nUD_27_FRENCH_SPOKEN_TEST = _UD_27_HOME + \"UD_French-Spoken/fr_spoken-ud-test.conllu\"\n\"UD_27 test set of FRENCH_SPOKEN.\"\nUD_27_GALICIAN_CTG_TRAIN = _UD_27_HOME + \"UD_Galician-CTG/gl_ctg-ud-train.conllu\"\n\"UD_27 train set of GALICIAN_CTG.\"\nUD_27_GALICIAN_CTG_DEV = _UD_27_HOME + \"UD_Galician-CTG/gl_ctg-ud-dev.conllu\"\n\"UD_27 dev set of GALICIAN_CTG.\"\nUD_27_GALICIAN_CTG_TEST = _UD_27_HOME + \"UD_Galician-CTG/gl_ctg-ud-test.conllu\"\n\"UD_27 test set of GALICIAN_CTG.\"\nUD_27_GALICIAN_TREEGAL_TRAIN = _UD_27_HOME + \"UD_Galician-TreeGal/gl_treegal-ud-train.conllu\"\n\"UD_27 train set of GALICIAN_TREEGAL.\"\nUD_27_GALICIAN_TREEGAL_TEST = _UD_27_HOME + \"UD_Galician-TreeGal/gl_treegal-ud-test.conllu\"\n\"UD_27 test set of GALICIAN_TREEGAL.\"\nUD_27_GERMAN_GSD_TRAIN = _UD_27_HOME + \"UD_German-GSD/de_gsd-ud-train.conllu\"\n\"UD_27 train set of GERMAN_GSD.\"\nUD_27_GERMAN_GSD_DEV = _UD_27_HOME + \"UD_German-GSD/de_gsd-ud-dev.conllu\"\n\"UD_27 dev set of GERMAN_GSD.\"\nUD_27_GERMAN_GSD_TEST = _UD_27_HOME + \"UD_German-GSD/de_gsd-ud-test.conllu\"\n\"UD_27 test set of GERMAN_GSD.\"\nUD_27_GERMAN_HDT_TRAIN = _UD_27_HOME + \"UD_German-HDT/de_hdt-ud-train.conllu\"\n\"UD_27 train set of GERMAN_HDT.\"\nUD_27_GERMAN_HDT_DEV = _UD_27_HOME + \"UD_German-HDT/de_hdt-ud-dev.conllu\"\n\"UD_27 dev set of GERMAN_HDT.\"\nUD_27_GERMAN_HDT_TEST = _UD_27_HOME + \"UD_German-HDT/de_hdt-ud-test.conllu\"\n\"UD_27 test set of GERMAN_HDT.\"\nUD_27_GERMAN_LIT_TEST = _UD_27_HOME + \"UD_German-LIT/de_lit-ud-test.conllu\"\n\"UD_27 test set of GERMAN_LIT.\"\nUD_27_GERMAN_PUD_TEST = _UD_27_HOME + \"UD_German-PUD/de_pud-ud-test.conllu\"\n\"UD_27 test set of GERMAN_PUD.\"\nUD_27_GOTHIC_PROIEL_TRAIN = _UD_27_HOME + \"UD_Gothic-PROIEL/got_proiel-ud-train.conllu\"\n\"UD_27 train set of GOTHIC_PROIEL.\"\nUD_27_GOTHIC_PROIEL_DEV = _UD_27_HOME + \"UD_Gothic-PROIEL/got_proiel-ud-dev.conllu\"\n\"UD_27 dev set of GOTHIC_PROIEL.\"\nUD_27_GOTHIC_PROIEL_TEST = _UD_27_HOME + \"UD_Gothic-PROIEL/got_proiel-ud-test.conllu\"\n\"UD_27 test set of GOTHIC_PROIEL.\"\nUD_27_GREEK_GDT_TRAIN = _UD_27_HOME + \"UD_Greek-GDT/el_gdt-ud-train.conllu\"\n\"UD_27 train set of GREEK_GDT.\"\nUD_27_GREEK_GDT_DEV = _UD_27_HOME + \"UD_Greek-GDT/el_gdt-ud-dev.conllu\"\n\"UD_27 dev set of GREEK_GDT.\"\nUD_27_GREEK_GDT_TEST = _UD_27_HOME + \"UD_Greek-GDT/el_gdt-ud-test.conllu\"\n\"UD_27 test set of GREEK_GDT.\"\nUD_27_HEBREW_HTB_TRAIN = _UD_27_HOME + \"UD_Hebrew-HTB/he_htb-ud-train.conllu\"\n\"UD_27 train set of HEBREW_HTB.\"\nUD_27_HEBREW_HTB_DEV = _UD_27_HOME + \"UD_Hebrew-HTB/he_htb-ud-dev.conllu\"\n\"UD_27 dev set of HEBREW_HTB.\"\nUD_27_HEBREW_HTB_TEST = _UD_27_HOME + \"UD_Hebrew-HTB/he_htb-ud-test.conllu\"\n\"UD_27 test set of HEBREW_HTB.\"\nUD_27_HINDI_HDTB_TRAIN = _UD_27_HOME + \"UD_Hindi-HDTB/hi_hdtb-ud-train.conllu\"\n\"UD_27 train set of HINDI_HDTB.\"\nUD_27_HINDI_HDTB_DEV = _UD_27_HOME + \"UD_Hindi-HDTB/hi_hdtb-ud-dev.conllu\"\n\"UD_27 dev set of HINDI_HDTB.\"\nUD_27_HINDI_HDTB_TEST = _UD_27_HOME + \"UD_Hindi-HDTB/hi_hdtb-ud-test.conllu\"\n\"UD_27 test set of HINDI_HDTB.\"\nUD_27_HINDI_PUD_TEST = _UD_27_HOME + \"UD_Hindi-PUD/hi_pud-ud-test.conllu\"\n\"UD_27 test set of HINDI_PUD.\"\nUD_27_HINDI_ENGLISH_HIENCS_TRAIN = _UD_27_HOME + \"UD_Hindi_English-HIENCS/qhe_hiencs-ud-train.conllu\"\n\"UD_27 train set of HINDI_ENGLISH_HIENCS.\"\nUD_27_HINDI_ENGLISH_HIENCS_DEV = _UD_27_HOME + \"UD_Hindi_English-HIENCS/qhe_hiencs-ud-dev.conllu\"\n\"UD_27 dev set of HINDI_ENGLISH_HIENCS.\"\nUD_27_HINDI_ENGLISH_HIENCS_TEST = _UD_27_HOME + \"UD_Hindi_English-HIENCS/qhe_hiencs-ud-test.conllu\"\n\"UD_27 test set of HINDI_ENGLISH_HIENCS.\"\nUD_27_HUNGARIAN_SZEGED_TRAIN = _UD_27_HOME + \"UD_Hungarian-Szeged/hu_szeged-ud-train.conllu\"\n\"UD_27 train set of HUNGARIAN_SZEGED.\"\nUD_27_HUNGARIAN_SZEGED_DEV = _UD_27_HOME + \"UD_Hungarian-Szeged/hu_szeged-ud-dev.conllu\"\n\"UD_27 dev set of HUNGARIAN_SZEGED.\"\nUD_27_HUNGARIAN_SZEGED_TEST = _UD_27_HOME + \"UD_Hungarian-Szeged/hu_szeged-ud-test.conllu\"\n\"UD_27 test set of HUNGARIAN_SZEGED.\"\nUD_27_ICELANDIC_ICEPAHC_TRAIN = _UD_27_HOME + \"UD_Icelandic-IcePaHC/is_icepahc-ud-train.conllu\"\n\"UD_27 train set of ICELANDIC_ICEPAHC.\"\nUD_27_ICELANDIC_ICEPAHC_DEV = _UD_27_HOME + \"UD_Icelandic-IcePaHC/is_icepahc-ud-dev.conllu\"\n\"UD_27 dev set of ICELANDIC_ICEPAHC.\"\nUD_27_ICELANDIC_ICEPAHC_TEST = _UD_27_HOME + \"UD_Icelandic-IcePaHC/is_icepahc-ud-test.conllu\"\n\"UD_27 test set of ICELANDIC_ICEPAHC.\"\nUD_27_ICELANDIC_PUD_TEST = _UD_27_HOME + \"UD_Icelandic-PUD/is_pud-ud-test.conllu\"\n\"UD_27 test set of ICELANDIC_PUD.\"\nUD_27_INDONESIAN_CSUI_TRAIN = _UD_27_HOME + \"UD_Indonesian-CSUI/id_csui-ud-train.conllu\"\n\"UD_27 train set of INDONESIAN_CSUI.\"\nUD_27_INDONESIAN_CSUI_TEST = _UD_27_HOME + \"UD_Indonesian-CSUI/id_csui-ud-test.conllu\"\n\"UD_27 test set of INDONESIAN_CSUI.\"\nUD_27_INDONESIAN_GSD_TRAIN = _UD_27_HOME + \"UD_Indonesian-GSD/id_gsd-ud-train.conllu\"\n\"UD_27 train set of INDONESIAN_GSD.\"\nUD_27_INDONESIAN_GSD_DEV = _UD_27_HOME + \"UD_Indonesian-GSD/id_gsd-ud-dev.conllu\"\n\"UD_27 dev set of INDONESIAN_GSD.\"\nUD_27_INDONESIAN_GSD_TEST = _UD_27_HOME + \"UD_Indonesian-GSD/id_gsd-ud-test.conllu\"\n\"UD_27 test set of INDONESIAN_GSD.\"\nUD_27_INDONESIAN_PUD_TEST = _UD_27_HOME + \"UD_Indonesian-PUD/id_pud-ud-test.conllu\"\n\"UD_27 test set of INDONESIAN_PUD.\"\nUD_27_IRISH_IDT_TRAIN = _UD_27_HOME + \"UD_Irish-IDT/ga_idt-ud-train.conllu\"\n\"UD_27 train set of IRISH_IDT.\"\nUD_27_IRISH_IDT_DEV = _UD_27_HOME + \"UD_Irish-IDT/ga_idt-ud-dev.conllu\"\n\"UD_27 dev set of IRISH_IDT.\"\nUD_27_IRISH_IDT_TEST = _UD_27_HOME + \"UD_Irish-IDT/ga_idt-ud-test.conllu\"\n\"UD_27 test set of IRISH_IDT.\"\nUD_27_ITALIAN_ISDT_TRAIN = _UD_27_HOME + \"UD_Italian-ISDT/it_isdt-ud-train.conllu\"\n\"UD_27 train set of ITALIAN_ISDT.\"\nUD_27_ITALIAN_ISDT_DEV = _UD_27_HOME + \"UD_Italian-ISDT/it_isdt-ud-dev.conllu\"\n\"UD_27 dev set of ITALIAN_ISDT.\"\nUD_27_ITALIAN_ISDT_TEST = _UD_27_HOME + \"UD_Italian-ISDT/it_isdt-ud-test.conllu\"\n\"UD_27 test set of ITALIAN_ISDT.\"\nUD_27_ITALIAN_PUD_TEST = _UD_27_HOME + \"UD_Italian-PUD/it_pud-ud-test.conllu\"\n\"UD_27 test set of ITALIAN_PUD.\"\nUD_27_ITALIAN_PARTUT_TRAIN = _UD_27_HOME + \"UD_Italian-ParTUT/it_partut-ud-train.conllu\"\n\"UD_27 train set of ITALIAN_PARTUT.\"\nUD_27_ITALIAN_PARTUT_DEV = _UD_27_HOME + \"UD_Italian-ParTUT/it_partut-ud-dev.conllu\"\n\"UD_27 dev set of ITALIAN_PARTUT.\"\nUD_27_ITALIAN_PARTUT_TEST = _UD_27_HOME + \"UD_Italian-ParTUT/it_partut-ud-test.conllu\"\n\"UD_27 test set of ITALIAN_PARTUT.\"\nUD_27_ITALIAN_POSTWITA_TRAIN = _UD_27_HOME + \"UD_Italian-PoSTWITA/it_postwita-ud-train.conllu\"\n\"UD_27 train set of ITALIAN_POSTWITA.\"\nUD_27_ITALIAN_POSTWITA_DEV = _UD_27_HOME + \"UD_Italian-PoSTWITA/it_postwita-ud-dev.conllu\"\n\"UD_27 dev set of ITALIAN_POSTWITA.\"\nUD_27_ITALIAN_POSTWITA_TEST = _UD_27_HOME + \"UD_Italian-PoSTWITA/it_postwita-ud-test.conllu\"\n\"UD_27 test set of ITALIAN_POSTWITA.\"\nUD_27_ITALIAN_TWITTIRO_TRAIN = _UD_27_HOME + \"UD_Italian-TWITTIRO/it_twittiro-ud-train.conllu\"\n\"UD_27 train set of ITALIAN_TWITTIRO.\"\nUD_27_ITALIAN_TWITTIRO_DEV = _UD_27_HOME + \"UD_Italian-TWITTIRO/it_twittiro-ud-dev.conllu\"\n\"UD_27 dev set of ITALIAN_TWITTIRO.\"\nUD_27_ITALIAN_TWITTIRO_TEST = _UD_27_HOME + \"UD_Italian-TWITTIRO/it_twittiro-ud-test.conllu\"\n\"UD_27 test set of ITALIAN_TWITTIRO.\"\nUD_27_ITALIAN_VIT_TRAIN = _UD_27_HOME + \"UD_Italian-VIT/it_vit-ud-train.conllu\"\n\"UD_27 train set of ITALIAN_VIT.\"\nUD_27_ITALIAN_VIT_DEV = _UD_27_HOME + \"UD_Italian-VIT/it_vit-ud-dev.conllu\"\n\"UD_27 dev set of ITALIAN_VIT.\"\nUD_27_ITALIAN_VIT_TEST = _UD_27_HOME + \"UD_Italian-VIT/it_vit-ud-test.conllu\"\n\"UD_27 test set of ITALIAN_VIT.\"\nUD_27_JAPANESE_BCCWJ_TRAIN = _UD_27_HOME + \"UD_Japanese-BCCWJ/ja_bccwj-ud-train.conllu\"\n\"UD_27 train set of JAPANESE_BCCWJ.\"\nUD_27_JAPANESE_BCCWJ_DEV = _UD_27_HOME + \"UD_Japanese-BCCWJ/ja_bccwj-ud-dev.conllu\"\n\"UD_27 dev set of JAPANESE_BCCWJ.\"\nUD_27_JAPANESE_BCCWJ_TEST = _UD_27_HOME + \"UD_Japanese-BCCWJ/ja_bccwj-ud-test.conllu\"\n\"UD_27 test set of JAPANESE_BCCWJ.\"\nUD_27_JAPANESE_GSD_TRAIN = _UD_27_HOME + \"UD_Japanese-GSD/ja_gsd-ud-train.conllu\"\n\"UD_27 train set of JAPANESE_GSD.\"\nUD_27_JAPANESE_GSD_DEV = _UD_27_HOME + \"UD_Japanese-GSD/ja_gsd-ud-dev.conllu\"\n\"UD_27 dev set of JAPANESE_GSD.\"\nUD_27_JAPANESE_GSD_TEST = _UD_27_HOME + \"UD_Japanese-GSD/ja_gsd-ud-test.conllu\"\n\"UD_27 test set of JAPANESE_GSD.\"\nUD_27_JAPANESE_MODERN_TEST = _UD_27_HOME + \"UD_Japanese-Modern/ja_modern-ud-test.conllu\"\n\"UD_27 test set of JAPANESE_MODERN.\"\nUD_27_JAPANESE_PUD_TEST = _UD_27_HOME + \"UD_Japanese-PUD/ja_pud-ud-test.conllu\"\n\"UD_27 test set of JAPANESE_PUD.\"\nUD_27_KARELIAN_KKPP_TEST = _UD_27_HOME + \"UD_Karelian-KKPP/krl_kkpp-ud-test.conllu\"\n\"UD_27 test set of KARELIAN_KKPP.\"\nUD_27_KAZAKH_KTB_TRAIN = _UD_27_HOME + \"UD_Kazakh-KTB/kk_ktb-ud-train.conllu\"\n\"UD_27 train set of KAZAKH_KTB.\"\nUD_27_KAZAKH_KTB_TEST = _UD_27_HOME + \"UD_Kazakh-KTB/kk_ktb-ud-test.conllu\"\n\"UD_27 test set of KAZAKH_KTB.\"\nUD_27_KHUNSARI_AHA_TEST = _UD_27_HOME + \"UD_Khunsari-AHA/kfm_aha-ud-test.conllu\"\n\"UD_27 test set of KHUNSARI_AHA.\"\nUD_27_KOMI_PERMYAK_UH_TEST = _UD_27_HOME + \"UD_Komi_Permyak-UH/koi_uh-ud-test.conllu\"\n\"UD_27 test set of KOMI_PERMYAK_UH.\"\nUD_27_KOMI_ZYRIAN_IKDP_TEST = _UD_27_HOME + \"UD_Komi_Zyrian-IKDP/kpv_ikdp-ud-test.conllu\"\n\"UD_27 test set of KOMI_ZYRIAN_IKDP.\"\nUD_27_KOMI_ZYRIAN_LATTICE_TEST = _UD_27_HOME + \"UD_Komi_Zyrian-Lattice/kpv_lattice-ud-test.conllu\"\n\"UD_27 test set of KOMI_ZYRIAN_LATTICE.\"\nUD_27_KOREAN_GSD_TRAIN = _UD_27_HOME + \"UD_Korean-GSD/ko_gsd-ud-train.conllu\"\n\"UD_27 train set of KOREAN_GSD.\"\nUD_27_KOREAN_GSD_DEV = _UD_27_HOME + \"UD_Korean-GSD/ko_gsd-ud-dev.conllu\"\n\"UD_27 dev set of KOREAN_GSD.\"\nUD_27_KOREAN_GSD_TEST = _UD_27_HOME + \"UD_Korean-GSD/ko_gsd-ud-test.conllu\"\n\"UD_27 test set of KOREAN_GSD.\"\nUD_27_KOREAN_KAIST_TRAIN = _UD_27_HOME + \"UD_Korean-Kaist/ko_kaist-ud-train.conllu\"\n\"UD_27 train set of KOREAN_KAIST.\"\nUD_27_KOREAN_KAIST_DEV = _UD_27_HOME + \"UD_Korean-Kaist/ko_kaist-ud-dev.conllu\"\n\"UD_27 dev set of KOREAN_KAIST.\"\nUD_27_KOREAN_KAIST_TEST = _UD_27_HOME + \"UD_Korean-Kaist/ko_kaist-ud-test.conllu\"\n\"UD_27 test set of KOREAN_KAIST.\"\nUD_27_KOREAN_PUD_TEST = _UD_27_HOME + \"UD_Korean-PUD/ko_pud-ud-test.conllu\"\n\"UD_27 test set of KOREAN_PUD.\"\nUD_27_KURMANJI_MG_TRAIN = _UD_27_HOME + \"UD_Kurmanji-MG/kmr_mg-ud-train.conllu\"\n\"UD_27 train set of KURMANJI_MG.\"\nUD_27_KURMANJI_MG_TEST = _UD_27_HOME + \"UD_Kurmanji-MG/kmr_mg-ud-test.conllu\"\n\"UD_27 test set of KURMANJI_MG.\"\nUD_27_LATIN_ITTB_TRAIN = _UD_27_HOME + \"UD_Latin-ITTB/la_ittb-ud-train.conllu\"\n\"UD_27 train set of LATIN_ITTB.\"\nUD_27_LATIN_ITTB_DEV = _UD_27_HOME + \"UD_Latin-ITTB/la_ittb-ud-dev.conllu\"\n\"UD_27 dev set of LATIN_ITTB.\"\nUD_27_LATIN_ITTB_TEST = _UD_27_HOME + \"UD_Latin-ITTB/la_ittb-ud-test.conllu\"\n\"UD_27 test set of LATIN_ITTB.\"\nUD_27_LATIN_LLCT_TRAIN = _UD_27_HOME + \"UD_Latin-LLCT/la_llct-ud-train.conllu\"\n\"UD_27 train set of LATIN_LLCT.\"\nUD_27_LATIN_LLCT_DEV = _UD_27_HOME + \"UD_Latin-LLCT/la_llct-ud-dev.conllu\"\n\"UD_27 dev set of LATIN_LLCT.\"\nUD_27_LATIN_LLCT_TEST = _UD_27_HOME + \"UD_Latin-LLCT/la_llct-ud-test.conllu\"\n\"UD_27 test set of LATIN_LLCT.\"\nUD_27_LATIN_PROIEL_TRAIN = _UD_27_HOME + \"UD_Latin-PROIEL/la_proiel-ud-train.conllu\"\n\"UD_27 train set of LATIN_PROIEL.\"\nUD_27_LATIN_PROIEL_DEV = _UD_27_HOME + \"UD_Latin-PROIEL/la_proiel-ud-dev.conllu\"\n\"UD_27 dev set of LATIN_PROIEL.\"\nUD_27_LATIN_PROIEL_TEST = _UD_27_HOME + \"UD_Latin-PROIEL/la_proiel-ud-test.conllu\"\n\"UD_27 test set of LATIN_PROIEL.\"\nUD_27_LATIN_PERSEUS_TRAIN = _UD_27_HOME + \"UD_Latin-Perseus/la_perseus-ud-train.conllu\"\n\"UD_27 train set of LATIN_PERSEUS.\"\nUD_27_LATIN_PERSEUS_TEST = _UD_27_HOME + \"UD_Latin-Perseus/la_perseus-ud-test.conllu\"\n\"UD_27 test set of LATIN_PERSEUS.\"\nUD_27_LATVIAN_LVTB_TRAIN = _UD_27_HOME + \"UD_Latvian-LVTB/lv_lvtb-ud-train.conllu\"\n\"UD_27 train set of LATVIAN_LVTB.\"\nUD_27_LATVIAN_LVTB_DEV = _UD_27_HOME + \"UD_Latvian-LVTB/lv_lvtb-ud-dev.conllu\"\n\"UD_27 dev set of LATVIAN_LVTB.\"\nUD_27_LATVIAN_LVTB_TEST = _UD_27_HOME + \"UD_Latvian-LVTB/lv_lvtb-ud-test.conllu\"\n\"UD_27 test set of LATVIAN_LVTB.\"\nUD_27_LITHUANIAN_ALKSNIS_TRAIN = _UD_27_HOME + \"UD_Lithuanian-ALKSNIS/lt_alksnis-ud-train.conllu\"\n\"UD_27 train set of LITHUANIAN_ALKSNIS.\"\nUD_27_LITHUANIAN_ALKSNIS_DEV = _UD_27_HOME + \"UD_Lithuanian-ALKSNIS/lt_alksnis-ud-dev.conllu\"\n\"UD_27 dev set of LITHUANIAN_ALKSNIS.\"\nUD_27_LITHUANIAN_ALKSNIS_TEST = _UD_27_HOME + \"UD_Lithuanian-ALKSNIS/lt_alksnis-ud-test.conllu\"\n\"UD_27 test set of LITHUANIAN_ALKSNIS.\"\nUD_27_LITHUANIAN_HSE_TRAIN = _UD_27_HOME + \"UD_Lithuanian-HSE/lt_hse-ud-train.conllu\"\n\"UD_27 train set of LITHUANIAN_HSE.\"\nUD_27_LITHUANIAN_HSE_DEV = _UD_27_HOME + \"UD_Lithuanian-HSE/lt_hse-ud-dev.conllu\"\n\"UD_27 dev set of LITHUANIAN_HSE.\"\nUD_27_LITHUANIAN_HSE_TEST = _UD_27_HOME + \"UD_Lithuanian-HSE/lt_hse-ud-test.conllu\"\n\"UD_27 test set of LITHUANIAN_HSE.\"\nUD_27_LIVVI_KKPP_TRAIN = _UD_27_HOME + \"UD_Livvi-KKPP/olo_kkpp-ud-train.conllu\"\n\"UD_27 train set of LIVVI_KKPP.\"\nUD_27_LIVVI_KKPP_TEST = _UD_27_HOME + \"UD_Livvi-KKPP/olo_kkpp-ud-test.conllu\"\n\"UD_27 test set of LIVVI_KKPP.\"\nUD_27_MALTESE_MUDT_TRAIN = _UD_27_HOME + \"UD_Maltese-MUDT/mt_mudt-ud-train.conllu\"\n\"UD_27 train set of MALTESE_MUDT.\"\nUD_27_MALTESE_MUDT_DEV = _UD_27_HOME + \"UD_Maltese-MUDT/mt_mudt-ud-dev.conllu\"\n\"UD_27 dev set of MALTESE_MUDT.\"\nUD_27_MALTESE_MUDT_TEST = _UD_27_HOME + \"UD_Maltese-MUDT/mt_mudt-ud-test.conllu\"\n\"UD_27 test set of MALTESE_MUDT.\"\nUD_27_MANX_CADHAN_TEST = _UD_27_HOME + \"UD_Manx-Cadhan/gv_cadhan-ud-test.conllu\"\n\"UD_27 test set of MANX_CADHAN.\"\nUD_27_MARATHI_UFAL_TRAIN = _UD_27_HOME + \"UD_Marathi-UFAL/mr_ufal-ud-train.conllu\"\n\"UD_27 train set of MARATHI_UFAL.\"\nUD_27_MARATHI_UFAL_DEV = _UD_27_HOME + \"UD_Marathi-UFAL/mr_ufal-ud-dev.conllu\"\n\"UD_27 dev set of MARATHI_UFAL.\"\nUD_27_MARATHI_UFAL_TEST = _UD_27_HOME + \"UD_Marathi-UFAL/mr_ufal-ud-test.conllu\"\n\"UD_27 test set of MARATHI_UFAL.\"\nUD_27_MBYA_GUARANI_DOOLEY_TEST = _UD_27_HOME + \"UD_Mbya_Guarani-Dooley/gun_dooley-ud-test.conllu\"\n\"UD_27 test set of MBYA_GUARANI_DOOLEY.\"\nUD_27_MBYA_GUARANI_THOMAS_TEST = _UD_27_HOME + \"UD_Mbya_Guarani-Thomas/gun_thomas-ud-test.conllu\"\n\"UD_27 test set of MBYA_GUARANI_THOMAS.\"\nUD_27_MOKSHA_JR_TEST = _UD_27_HOME + \"UD_Moksha-JR/mdf_jr-ud-test.conllu\"\n\"UD_27 test set of MOKSHA_JR.\"\nUD_27_MUNDURUKU_TUDET_TEST = _UD_27_HOME + \"UD_Munduruku-TuDeT/myu_tudet-ud-test.conllu\"\n\"UD_27 test set of MUNDURUKU_TUDET.\"\nUD_27_NAIJA_NSC_TRAIN = _UD_27_HOME + \"UD_Naija-NSC/pcm_nsc-ud-train.conllu\"\n\"UD_27 train set of NAIJA_NSC.\"\nUD_27_NAIJA_NSC_DEV = _UD_27_HOME + \"UD_Naija-NSC/pcm_nsc-ud-dev.conllu\"\n\"UD_27 dev set of NAIJA_NSC.\"\nUD_27_NAIJA_NSC_TEST = _UD_27_HOME + \"UD_Naija-NSC/pcm_nsc-ud-test.conllu\"\n\"UD_27 test set of NAIJA_NSC.\"\nUD_27_NAYINI_AHA_TEST = _UD_27_HOME + \"UD_Nayini-AHA/nyq_aha-ud-test.conllu\"\n\"UD_27 test set of NAYINI_AHA.\"\nUD_27_NORTH_SAMI_GIELLA_TRAIN = _UD_27_HOME + \"UD_North_Sami-Giella/sme_giella-ud-train.conllu\"\n\"UD_27 train set of NORTH_SAMI_GIELLA.\"\nUD_27_NORTH_SAMI_GIELLA_TEST = _UD_27_HOME + \"UD_North_Sami-Giella/sme_giella-ud-test.conllu\"\n\"UD_27 test set of NORTH_SAMI_GIELLA.\"\nUD_27_NORWEGIAN_BOKMAAL_TRAIN = _UD_27_HOME + \"UD_Norwegian-Bokmaal/no_bokmaal-ud-train.conllu\"\n\"UD_27 train set of NORWEGIAN_BOKMAAL.\"\nUD_27_NORWEGIAN_BOKMAAL_DEV = _UD_27_HOME + \"UD_Norwegian-Bokmaal/no_bokmaal-ud-dev.conllu\"\n\"UD_27 dev set of NORWEGIAN_BOKMAAL.\"\nUD_27_NORWEGIAN_BOKMAAL_TEST = _UD_27_HOME + \"UD_Norwegian-Bokmaal/no_bokmaal-ud-test.conllu\"\n\"UD_27 test set of NORWEGIAN_BOKMAAL.\"\nUD_27_NORWEGIAN_NYNORSK_TRAIN = _UD_27_HOME + \"UD_Norwegian-Nynorsk/no_nynorsk-ud-train.conllu\"\n\"UD_27 train set of NORWEGIAN_NYNORSK.\"\nUD_27_NORWEGIAN_NYNORSK_DEV = _UD_27_HOME + \"UD_Norwegian-Nynorsk/no_nynorsk-ud-dev.conllu\"\n\"UD_27 dev set of NORWEGIAN_NYNORSK.\"\nUD_27_NORWEGIAN_NYNORSK_TEST = _UD_27_HOME + \"UD_Norwegian-Nynorsk/no_nynorsk-ud-test.conllu\"\n\"UD_27 test set of NORWEGIAN_NYNORSK.\"\nUD_27_NORWEGIAN_NYNORSKLIA_TRAIN = _UD_27_HOME + \"UD_Norwegian-NynorskLIA/no_nynorsklia-ud-train.conllu\"\n\"UD_27 train set of NORWEGIAN_NYNORSKLIA.\"\nUD_27_NORWEGIAN_NYNORSKLIA_DEV = _UD_27_HOME + \"UD_Norwegian-NynorskLIA/no_nynorsklia-ud-dev.conllu\"\n\"UD_27 dev set of NORWEGIAN_NYNORSKLIA.\"\nUD_27_NORWEGIAN_NYNORSKLIA_TEST = _UD_27_HOME + \"UD_Norwegian-NynorskLIA/no_nynorsklia-ud-test.conllu\"\n\"UD_27 test set of NORWEGIAN_NYNORSKLIA.\"\nUD_27_OLD_CHURCH_SLAVONIC_PROIEL_TRAIN = _UD_27_HOME + \"UD_Old_Church_Slavonic-PROIEL/cu_proiel-ud-train.conllu\"\n\"UD_27 train set of OLD_CHURCH_SLAVONIC_PROIEL.\"\nUD_27_OLD_CHURCH_SLAVONIC_PROIEL_DEV = _UD_27_HOME + \"UD_Old_Church_Slavonic-PROIEL/cu_proiel-ud-dev.conllu\"\n\"UD_27 dev set of OLD_CHURCH_SLAVONIC_PROIEL.\"\nUD_27_OLD_CHURCH_SLAVONIC_PROIEL_TEST = _UD_27_HOME + \"UD_Old_Church_Slavonic-PROIEL/cu_proiel-ud-test.conllu\"\n\"UD_27 test set of OLD_CHURCH_SLAVONIC_PROIEL.\"\nUD_27_OLD_FRENCH_SRCMF_TRAIN = _UD_27_HOME + \"UD_Old_French-SRCMF/fro_srcmf-ud-train.conllu\"\n\"UD_27 train set of OLD_FRENCH_SRCMF.\"\nUD_27_OLD_FRENCH_SRCMF_DEV = _UD_27_HOME + \"UD_Old_French-SRCMF/fro_srcmf-ud-dev.conllu\"\n\"UD_27 dev set of OLD_FRENCH_SRCMF.\"\nUD_27_OLD_FRENCH_SRCMF_TEST = _UD_27_HOME + \"UD_Old_French-SRCMF/fro_srcmf-ud-test.conllu\"\n\"UD_27 test set of OLD_FRENCH_SRCMF.\"\nUD_27_OLD_RUSSIAN_RNC_TRAIN = _UD_27_HOME + \"UD_Old_Russian-RNC/orv_rnc-ud-train.conllu\"\n\"UD_27 train set of OLD_RUSSIAN_RNC.\"\nUD_27_OLD_RUSSIAN_RNC_TEST = _UD_27_HOME + \"UD_Old_Russian-RNC/orv_rnc-ud-test.conllu\"\n\"UD_27 test set of OLD_RUSSIAN_RNC.\"\nUD_27_OLD_RUSSIAN_TOROT_TRAIN = _UD_27_HOME + \"UD_Old_Russian-TOROT/orv_torot-ud-train.conllu\"\n\"UD_27 train set of OLD_RUSSIAN_TOROT.\"\nUD_27_OLD_RUSSIAN_TOROT_DEV = _UD_27_HOME + \"UD_Old_Russian-TOROT/orv_torot-ud-dev.conllu\"\n\"UD_27 dev set of OLD_RUSSIAN_TOROT.\"\nUD_27_OLD_RUSSIAN_TOROT_TEST = _UD_27_HOME + \"UD_Old_Russian-TOROT/orv_torot-ud-test.conllu\"\n\"UD_27 test set of OLD_RUSSIAN_TOROT.\"\nUD_27_OLD_TURKISH_TONQQ_TEST = _UD_27_HOME + \"UD_Old_Turkish-Tonqq/otk_tonqq-ud-test.conllu\"\n\"UD_27 test set of OLD_TURKISH_TONQQ.\"\nUD_27_PERSIAN_PERDT_TRAIN = _UD_27_HOME + \"UD_Persian-PerDT/fa_perdt-ud-train.conllu\"\n\"UD_27 train set of PERSIAN_PERDT.\"\nUD_27_PERSIAN_PERDT_DEV = _UD_27_HOME + \"UD_Persian-PerDT/fa_perdt-ud-dev.conllu\"\n\"UD_27 dev set of PERSIAN_PERDT.\"\nUD_27_PERSIAN_PERDT_TEST = _UD_27_HOME + \"UD_Persian-PerDT/fa_perdt-ud-test.conllu\"\n\"UD_27 test set of PERSIAN_PERDT.\"\nUD_27_PERSIAN_SERAJI_TRAIN = _UD_27_HOME + \"UD_Persian-Seraji/fa_seraji-ud-train.conllu\"\n\"UD_27 train set of PERSIAN_SERAJI.\"\nUD_27_PERSIAN_SERAJI_DEV = _UD_27_HOME + \"UD_Persian-Seraji/fa_seraji-ud-dev.conllu\"\n\"UD_27 dev set of PERSIAN_SERAJI.\"\nUD_27_PERSIAN_SERAJI_TEST = _UD_27_HOME + \"UD_Persian-Seraji/fa_seraji-ud-test.conllu\"\n\"UD_27 test set of PERSIAN_SERAJI.\"\nUD_27_POLISH_LFG_TRAIN = _UD_27_HOME + \"UD_Polish-LFG/pl_lfg-ud-train.conllu\"\n\"UD_27 train set of POLISH_LFG.\"\nUD_27_POLISH_LFG_DEV = _UD_27_HOME + \"UD_Polish-LFG/pl_lfg-ud-dev.conllu\"\n\"UD_27 dev set of POLISH_LFG.\"\nUD_27_POLISH_LFG_TEST = _UD_27_HOME + \"UD_Polish-LFG/pl_lfg-ud-test.conllu\"\n\"UD_27 test set of POLISH_LFG.\"\nUD_27_POLISH_PDB_TRAIN = _UD_27_HOME + \"UD_Polish-PDB/pl_pdb-ud-train.conllu\"\n\"UD_27 train set of POLISH_PDB.\"\nUD_27_POLISH_PDB_DEV = _UD_27_HOME + \"UD_Polish-PDB/pl_pdb-ud-dev.conllu\"\n\"UD_27 dev set of POLISH_PDB.\"\nUD_27_POLISH_PDB_TEST = _UD_27_HOME + \"UD_Polish-PDB/pl_pdb-ud-test.conllu\"\n\"UD_27 test set of POLISH_PDB.\"\nUD_27_POLISH_PUD_TEST = _UD_27_HOME + \"UD_Polish-PUD/pl_pud-ud-test.conllu\"\n\"UD_27 test set of POLISH_PUD.\"\nUD_27_PORTUGUESE_BOSQUE_TRAIN = _UD_27_HOME + \"UD_Portuguese-Bosque/pt_bosque-ud-train.conllu\"\n\"UD_27 train set of PORTUGUESE_BOSQUE.\"\nUD_27_PORTUGUESE_BOSQUE_DEV = _UD_27_HOME + \"UD_Portuguese-Bosque/pt_bosque-ud-dev.conllu\"\n\"UD_27 dev set of PORTUGUESE_BOSQUE.\"\nUD_27_PORTUGUESE_BOSQUE_TEST = _UD_27_HOME + \"UD_Portuguese-Bosque/pt_bosque-ud-test.conllu\"\n\"UD_27 test set of PORTUGUESE_BOSQUE.\"\nUD_27_PORTUGUESE_GSD_TRAIN = _UD_27_HOME + \"UD_Portuguese-GSD/pt_gsd-ud-train.conllu\"\n\"UD_27 train set of PORTUGUESE_GSD.\"\nUD_27_PORTUGUESE_GSD_DEV = _UD_27_HOME + \"UD_Portuguese-GSD/pt_gsd-ud-dev.conllu\"\n\"UD_27 dev set of PORTUGUESE_GSD.\"\nUD_27_PORTUGUESE_GSD_TEST = _UD_27_HOME + \"UD_Portuguese-GSD/pt_gsd-ud-test.conllu\"\n\"UD_27 test set of PORTUGUESE_GSD.\"\nUD_27_PORTUGUESE_PUD_TEST = _UD_27_HOME + \"UD_Portuguese-PUD/pt_pud-ud-test.conllu\"\n\"UD_27 test set of PORTUGUESE_PUD.\"\nUD_27_ROMANIAN_NONSTANDARD_TRAIN = _UD_27_HOME + \"UD_Romanian-Nonstandard/ro_nonstandard-ud-train.conllu\"\n\"UD_27 train set of ROMANIAN_NONSTANDARD.\"\nUD_27_ROMANIAN_NONSTANDARD_DEV = _UD_27_HOME + \"UD_Romanian-Nonstandard/ro_nonstandard-ud-dev.conllu\"\n\"UD_27 dev set of ROMANIAN_NONSTANDARD.\"\nUD_27_ROMANIAN_NONSTANDARD_TEST = _UD_27_HOME + \"UD_Romanian-Nonstandard/ro_nonstandard-ud-test.conllu\"\n\"UD_27 test set of ROMANIAN_NONSTANDARD.\"\nUD_27_ROMANIAN_RRT_TRAIN = _UD_27_HOME + \"UD_Romanian-RRT/ro_rrt-ud-train.conllu\"\n\"UD_27 train set of ROMANIAN_RRT.\"\nUD_27_ROMANIAN_RRT_DEV = _UD_27_HOME + \"UD_Romanian-RRT/ro_rrt-ud-dev.conllu\"\n\"UD_27 dev set of ROMANIAN_RRT.\"\nUD_27_ROMANIAN_RRT_TEST = _UD_27_HOME + \"UD_Romanian-RRT/ro_rrt-ud-test.conllu\"\n\"UD_27 test set of ROMANIAN_RRT.\"\nUD_27_ROMANIAN_SIMONERO_TRAIN = _UD_27_HOME + \"UD_Romanian-SiMoNERo/ro_simonero-ud-train.conllu\"\n\"UD_27 train set of ROMANIAN_SIMONERO.\"\nUD_27_ROMANIAN_SIMONERO_DEV = _UD_27_HOME + \"UD_Romanian-SiMoNERo/ro_simonero-ud-dev.conllu\"\n\"UD_27 dev set of ROMANIAN_SIMONERO.\"\nUD_27_ROMANIAN_SIMONERO_TEST = _UD_27_HOME + \"UD_Romanian-SiMoNERo/ro_simonero-ud-test.conllu\"\n\"UD_27 test set of ROMANIAN_SIMONERO.\"\nUD_27_RUSSIAN_GSD_TRAIN = _UD_27_HOME + \"UD_Russian-GSD/ru_gsd-ud-train.conllu\"\n\"UD_27 train set of RUSSIAN_GSD.\"\nUD_27_RUSSIAN_GSD_DEV = _UD_27_HOME + \"UD_Russian-GSD/ru_gsd-ud-dev.conllu\"\n\"UD_27 dev set of RUSSIAN_GSD.\"\nUD_27_RUSSIAN_GSD_TEST = _UD_27_HOME + \"UD_Russian-GSD/ru_gsd-ud-test.conllu\"\n\"UD_27 test set of RUSSIAN_GSD.\"\nUD_27_RUSSIAN_PUD_TEST = _UD_27_HOME + \"UD_Russian-PUD/ru_pud-ud-test.conllu\"\n\"UD_27 test set of RUSSIAN_PUD.\"\nUD_27_RUSSIAN_SYNTAGRUS_TRAIN = _UD_27_HOME + \"UD_Russian-SynTagRus/ru_syntagrus-ud-train.conllu\"\n\"UD_27 train set of RUSSIAN_SYNTAGRUS.\"\nUD_27_RUSSIAN_SYNTAGRUS_DEV = _UD_27_HOME + \"UD_Russian-SynTagRus/ru_syntagrus-ud-dev.conllu\"\n\"UD_27 dev set of RUSSIAN_SYNTAGRUS.\"\nUD_27_RUSSIAN_SYNTAGRUS_TEST = _UD_27_HOME + \"UD_Russian-SynTagRus/ru_syntagrus-ud-test.conllu\"\n\"UD_27 test set of RUSSIAN_SYNTAGRUS.\"\nUD_27_RUSSIAN_TAIGA_TRAIN = _UD_27_HOME + \"UD_Russian-Taiga/ru_taiga-ud-train.conllu\"\n\"UD_27 train set of RUSSIAN_TAIGA.\"\nUD_27_RUSSIAN_TAIGA_DEV = _UD_27_HOME + \"UD_Russian-Taiga/ru_taiga-ud-dev.conllu\"\n\"UD_27 dev set of RUSSIAN_TAIGA.\"\nUD_27_RUSSIAN_TAIGA_TEST = _UD_27_HOME + \"UD_Russian-Taiga/ru_taiga-ud-test.conllu\"\n\"UD_27 test set of RUSSIAN_TAIGA.\"\nUD_27_SANSKRIT_UFAL_TEST = _UD_27_HOME + \"UD_Sanskrit-UFAL/sa_ufal-ud-test.conllu\"\n\"UD_27 test set of SANSKRIT_UFAL.\"\nUD_27_SANSKRIT_VEDIC_TRAIN = _UD_27_HOME + \"UD_Sanskrit-Vedic/sa_vedic-ud-train.conllu\"\n\"UD_27 train set of SANSKRIT_VEDIC.\"\nUD_27_SANSKRIT_VEDIC_TEST = _UD_27_HOME + \"UD_Sanskrit-Vedic/sa_vedic-ud-test.conllu\"\n\"UD_27 test set of SANSKRIT_VEDIC.\"\nUD_27_SCOTTISH_GAELIC_ARCOSG_TRAIN = _UD_27_HOME + \"UD_Scottish_Gaelic-ARCOSG/gd_arcosg-ud-train.conllu\"\n\"UD_27 train set of SCOTTISH_GAELIC_ARCOSG.\"\nUD_27_SCOTTISH_GAELIC_ARCOSG_DEV = _UD_27_HOME + \"UD_Scottish_Gaelic-ARCOSG/gd_arcosg-ud-dev.conllu\"\n\"UD_27 dev set of SCOTTISH_GAELIC_ARCOSG.\"\nUD_27_SCOTTISH_GAELIC_ARCOSG_TEST = _UD_27_HOME + \"UD_Scottish_Gaelic-ARCOSG/gd_arcosg-ud-test.conllu\"\n\"UD_27 test set of SCOTTISH_GAELIC_ARCOSG.\"\nUD_27_SERBIAN_SET_TRAIN = _UD_27_HOME + \"UD_Serbian-SET/sr_set-ud-train.conllu\"\n\"UD_27 train set of SERBIAN_SET.\"\nUD_27_SERBIAN_SET_DEV = _UD_27_HOME + \"UD_Serbian-SET/sr_set-ud-dev.conllu\"\n\"UD_27 dev set of SERBIAN_SET.\"\nUD_27_SERBIAN_SET_TEST = _UD_27_HOME + \"UD_Serbian-SET/sr_set-ud-test.conllu\"\n\"UD_27 test set of SERBIAN_SET.\"\nUD_27_SKOLT_SAMI_GIELLAGAS_TEST = _UD_27_HOME + \"UD_Skolt_Sami-Giellagas/sms_giellagas-ud-test.conllu\"\n\"UD_27 test set of SKOLT_SAMI_GIELLAGAS.\"\nUD_27_SLOVAK_SNK_TRAIN = _UD_27_HOME + \"UD_Slovak-SNK/sk_snk-ud-train.conllu\"\n\"UD_27 train set of SLOVAK_SNK.\"\nUD_27_SLOVAK_SNK_DEV = _UD_27_HOME + \"UD_Slovak-SNK/sk_snk-ud-dev.conllu\"\n\"UD_27 dev set of SLOVAK_SNK.\"\nUD_27_SLOVAK_SNK_TEST = _UD_27_HOME + \"UD_Slovak-SNK/sk_snk-ud-test.conllu\"\n\"UD_27 test set of SLOVAK_SNK.\"\nUD_27_SLOVENIAN_SSJ_TRAIN = _UD_27_HOME + \"UD_Slovenian-SSJ/sl_ssj-ud-train.conllu\"\n\"UD_27 train set of SLOVENIAN_SSJ.\"\nUD_27_SLOVENIAN_SSJ_DEV = _UD_27_HOME + \"UD_Slovenian-SSJ/sl_ssj-ud-dev.conllu\"\n\"UD_27 dev set of SLOVENIAN_SSJ.\"\nUD_27_SLOVENIAN_SSJ_TEST = _UD_27_HOME + \"UD_Slovenian-SSJ/sl_ssj-ud-test.conllu\"\n\"UD_27 test set of SLOVENIAN_SSJ.\"\nUD_27_SLOVENIAN_SST_TRAIN = _UD_27_HOME + \"UD_Slovenian-SST/sl_sst-ud-train.conllu\"\n\"UD_27 train set of SLOVENIAN_SST.\"\nUD_27_SLOVENIAN_SST_TEST = _UD_27_HOME + \"UD_Slovenian-SST/sl_sst-ud-test.conllu\"\n\"UD_27 test set of SLOVENIAN_SST.\"\nUD_27_SOI_AHA_TEST = _UD_27_HOME + \"UD_Soi-AHA/soj_aha-ud-test.conllu\"\n\"UD_27 test set of SOI_AHA.\"\nUD_27_SOUTH_LEVANTINE_ARABIC_MADAR_TEST = _UD_27_HOME + \"UD_South_Levantine_Arabic-MADAR/ajp_madar-ud-test.conllu\"\n\"UD_27 test set of SOUTH_LEVANTINE_ARABIC_MADAR.\"\nUD_27_SPANISH_ANCORA_TRAIN = _UD_27_HOME + \"UD_Spanish-AnCora/es_ancora-ud-train.conllu\"\n\"UD_27 train set of SPANISH_ANCORA.\"\nUD_27_SPANISH_ANCORA_DEV = _UD_27_HOME + \"UD_Spanish-AnCora/es_ancora-ud-dev.conllu\"\n\"UD_27 dev set of SPANISH_ANCORA.\"\nUD_27_SPANISH_ANCORA_TEST = _UD_27_HOME + \"UD_Spanish-AnCora/es_ancora-ud-test.conllu\"\n\"UD_27 test set of SPANISH_ANCORA.\"\nUD_27_SPANISH_GSD_TRAIN = _UD_27_HOME + \"UD_Spanish-GSD/es_gsd-ud-train.conllu\"\n\"UD_27 train set of SPANISH_GSD.\"\nUD_27_SPANISH_GSD_DEV = _UD_27_HOME + \"UD_Spanish-GSD/es_gsd-ud-dev.conllu\"\n\"UD_27 dev set of SPANISH_GSD.\"\nUD_27_SPANISH_GSD_TEST = _UD_27_HOME + \"UD_Spanish-GSD/es_gsd-ud-test.conllu\"\n\"UD_27 test set of SPANISH_GSD.\"\nUD_27_SPANISH_PUD_TEST = _UD_27_HOME + \"UD_Spanish-PUD/es_pud-ud-test.conllu\"\n\"UD_27 test set of SPANISH_PUD.\"\nUD_27_SWEDISH_LINES_TRAIN = _UD_27_HOME + \"UD_Swedish-LinES/sv_lines-ud-train.conllu\"\n\"UD_27 train set of SWEDISH_LINES.\"\nUD_27_SWEDISH_LINES_DEV = _UD_27_HOME + \"UD_Swedish-LinES/sv_lines-ud-dev.conllu\"\n\"UD_27 dev set of SWEDISH_LINES.\"\nUD_27_SWEDISH_LINES_TEST = _UD_27_HOME + \"UD_Swedish-LinES/sv_lines-ud-test.conllu\"\n\"UD_27 test set of SWEDISH_LINES.\"\nUD_27_SWEDISH_PUD_TEST = _UD_27_HOME + \"UD_Swedish-PUD/sv_pud-ud-test.conllu\"\n\"UD_27 test set of SWEDISH_PUD.\"\nUD_27_SWEDISH_TALBANKEN_TRAIN = _UD_27_HOME + \"UD_Swedish-Talbanken/sv_talbanken-ud-train.conllu\"\n\"UD_27 train set of SWEDISH_TALBANKEN.\"\nUD_27_SWEDISH_TALBANKEN_DEV = _UD_27_HOME + \"UD_Swedish-Talbanken/sv_talbanken-ud-dev.conllu\"\n\"UD_27 dev set of SWEDISH_TALBANKEN.\"\nUD_27_SWEDISH_TALBANKEN_TEST = _UD_27_HOME + \"UD_Swedish-Talbanken/sv_talbanken-ud-test.conllu\"\n\"UD_27 test set of SWEDISH_TALBANKEN.\"\nUD_27_SWEDISH_SIGN_LANGUAGE_SSLC_TRAIN = _UD_27_HOME + \"UD_Swedish_Sign_Language-SSLC/swl_sslc-ud-train.conllu\"\n\"UD_27 train set of SWEDISH_SIGN_LANGUAGE_SSLC.\"\nUD_27_SWEDISH_SIGN_LANGUAGE_SSLC_DEV = _UD_27_HOME + \"UD_Swedish_Sign_Language-SSLC/swl_sslc-ud-dev.conllu\"\n\"UD_27 dev set of SWEDISH_SIGN_LANGUAGE_SSLC.\"\nUD_27_SWEDISH_SIGN_LANGUAGE_SSLC_TEST = _UD_27_HOME + \"UD_Swedish_Sign_Language-SSLC/swl_sslc-ud-test.conllu\"\n\"UD_27 test set of SWEDISH_SIGN_LANGUAGE_SSLC.\"\nUD_27_SWISS_GERMAN_UZH_TEST = _UD_27_HOME + \"UD_Swiss_German-UZH/gsw_uzh-ud-test.conllu\"\n\"UD_27 test set of SWISS_GERMAN_UZH.\"\nUD_27_TAGALOG_TRG_TEST = _UD_27_HOME + \"UD_Tagalog-TRG/tl_trg-ud-test.conllu\"\n\"UD_27 test set of TAGALOG_TRG.\"\nUD_27_TAGALOG_UGNAYAN_TEST = _UD_27_HOME + \"UD_Tagalog-Ugnayan/tl_ugnayan-ud-test.conllu\"\n\"UD_27 test set of TAGALOG_UGNAYAN.\"\nUD_27_TAMIL_MWTT_TEST = _UD_27_HOME + \"UD_Tamil-MWTT/ta_mwtt-ud-test.conllu\"\n\"UD_27 test set of TAMIL_MWTT.\"\nUD_27_TAMIL_TTB_TRAIN = _UD_27_HOME + \"UD_Tamil-TTB/ta_ttb-ud-train.conllu\"\n\"UD_27 train set of TAMIL_TTB.\"\nUD_27_TAMIL_TTB_DEV = _UD_27_HOME + \"UD_Tamil-TTB/ta_ttb-ud-dev.conllu\"\n\"UD_27 dev set of TAMIL_TTB.\"\nUD_27_TAMIL_TTB_TEST = _UD_27_HOME + \"UD_Tamil-TTB/ta_ttb-ud-test.conllu\"\n\"UD_27 test set of TAMIL_TTB.\"\nUD_27_TELUGU_MTG_TRAIN = _UD_27_HOME + \"UD_Telugu-MTG/te_mtg-ud-train.conllu\"\n\"UD_27 train set of TELUGU_MTG.\"\nUD_27_TELUGU_MTG_DEV = _UD_27_HOME + \"UD_Telugu-MTG/te_mtg-ud-dev.conllu\"\n\"UD_27 dev set of TELUGU_MTG.\"\nUD_27_TELUGU_MTG_TEST = _UD_27_HOME + \"UD_Telugu-MTG/te_mtg-ud-test.conllu\"\n\"UD_27 test set of TELUGU_MTG.\"\nUD_27_THAI_PUD_TEST = _UD_27_HOME + \"UD_Thai-PUD/th_pud-ud-test.conllu\"\n\"UD_27 test set of THAI_PUD.\"\nUD_27_TUPINAMBA_TUDET_TEST = _UD_27_HOME + \"UD_Tupinamba-TuDeT/tpn_tudet-ud-test.conllu\"\n\"UD_27 test set of TUPINAMBA_TUDET.\"\nUD_27_TURKISH_BOUN_TRAIN = _UD_27_HOME + \"UD_Turkish-BOUN/tr_boun-ud-train.conllu\"\n\"UD_27 train set of TURKISH_BOUN.\"\nUD_27_TURKISH_BOUN_DEV = _UD_27_HOME + \"UD_Turkish-BOUN/tr_boun-ud-dev.conllu\"\n\"UD_27 dev set of TURKISH_BOUN.\"\nUD_27_TURKISH_BOUN_TEST = _UD_27_HOME + \"UD_Turkish-BOUN/tr_boun-ud-test.conllu\"\n\"UD_27 test set of TURKISH_BOUN.\"\nUD_27_TURKISH_GB_TEST = _UD_27_HOME + \"UD_Turkish-GB/tr_gb-ud-test.conllu\"\n\"UD_27 test set of TURKISH_GB.\"\nUD_27_TURKISH_IMST_TRAIN = _UD_27_HOME + \"UD_Turkish-IMST/tr_imst-ud-train.conllu\"\n\"UD_27 train set of TURKISH_IMST.\"\nUD_27_TURKISH_IMST_DEV = _UD_27_HOME + \"UD_Turkish-IMST/tr_imst-ud-dev.conllu\"\n\"UD_27 dev set of TURKISH_IMST.\"\nUD_27_TURKISH_IMST_TEST = _UD_27_HOME + \"UD_Turkish-IMST/tr_imst-ud-test.conllu\"\n\"UD_27 test set of TURKISH_IMST.\"\nUD_27_TURKISH_PUD_TEST = _UD_27_HOME + \"UD_Turkish-PUD/tr_pud-ud-test.conllu\"\n\"UD_27 test set of TURKISH_PUD.\"\nUD_27_TURKISH_GERMAN_SAGT_TRAIN = _UD_27_HOME + \"UD_Turkish_German-SAGT/qtd_sagt-ud-train.conllu\"\n\"UD_27 train set of TURKISH_GERMAN_SAGT.\"\nUD_27_TURKISH_GERMAN_SAGT_DEV = _UD_27_HOME + \"UD_Turkish_German-SAGT/qtd_sagt-ud-dev.conllu\"\n\"UD_27 dev set of TURKISH_GERMAN_SAGT.\"\nUD_27_TURKISH_GERMAN_SAGT_TEST = _UD_27_HOME + \"UD_Turkish_German-SAGT/qtd_sagt-ud-test.conllu\"\n\"UD_27 test set of TURKISH_GERMAN_SAGT.\"\nUD_27_UKRAINIAN_IU_TRAIN = _UD_27_HOME + \"UD_Ukrainian-IU/uk_iu-ud-train.conllu\"\n\"UD_27 train set of UKRAINIAN_IU.\"\nUD_27_UKRAINIAN_IU_DEV = _UD_27_HOME + \"UD_Ukrainian-IU/uk_iu-ud-dev.conllu\"\n\"UD_27 dev set of UKRAINIAN_IU.\"\nUD_27_UKRAINIAN_IU_TEST = _UD_27_HOME + \"UD_Ukrainian-IU/uk_iu-ud-test.conllu\"\n\"UD_27 test set of UKRAINIAN_IU.\"\nUD_27_UPPER_SORBIAN_UFAL_TRAIN = _UD_27_HOME + \"UD_Upper_Sorbian-UFAL/hsb_ufal-ud-train.conllu\"\n\"UD_27 train set of UPPER_SORBIAN_UFAL.\"\nUD_27_UPPER_SORBIAN_UFAL_TEST = _UD_27_HOME + \"UD_Upper_Sorbian-UFAL/hsb_ufal-ud-test.conllu\"\n\"UD_27 test set of UPPER_SORBIAN_UFAL.\"\nUD_27_URDU_UDTB_TRAIN = _UD_27_HOME + \"UD_Urdu-UDTB/ur_udtb-ud-train.conllu\"\n\"UD_27 train set of URDU_UDTB.\"\nUD_27_URDU_UDTB_DEV = _UD_27_HOME + \"UD_Urdu-UDTB/ur_udtb-ud-dev.conllu\"\n\"UD_27 dev set of URDU_UDTB.\"\nUD_27_URDU_UDTB_TEST = _UD_27_HOME + \"UD_Urdu-UDTB/ur_udtb-ud-test.conllu\"\n\"UD_27 test set of URDU_UDTB.\"\nUD_27_UYGHUR_UDT_TRAIN = _UD_27_HOME + \"UD_Uyghur-UDT/ug_udt-ud-train.conllu\"\n\"UD_27 train set of UYGHUR_UDT.\"\nUD_27_UYGHUR_UDT_DEV = _UD_27_HOME + \"UD_Uyghur-UDT/ug_udt-ud-dev.conllu\"\n\"UD_27 dev set of UYGHUR_UDT.\"\nUD_27_UYGHUR_UDT_TEST = _UD_27_HOME + \"UD_Uyghur-UDT/ug_udt-ud-test.conllu\"\n\"UD_27 test set of UYGHUR_UDT.\"\nUD_27_VIETNAMESE_VTB_TRAIN = _UD_27_HOME + \"UD_Vietnamese-VTB/vi_vtb-ud-train.conllu\"\n\"UD_27 train set of VIETNAMESE_VTB.\"\nUD_27_VIETNAMESE_VTB_DEV = _UD_27_HOME + \"UD_Vietnamese-VTB/vi_vtb-ud-dev.conllu\"\n\"UD_27 dev set of VIETNAMESE_VTB.\"\nUD_27_VIETNAMESE_VTB_TEST = _UD_27_HOME + \"UD_Vietnamese-VTB/vi_vtb-ud-test.conllu\"\n\"UD_27 test set of VIETNAMESE_VTB.\"\nUD_27_WARLPIRI_UFAL_TEST = _UD_27_HOME + \"UD_Warlpiri-UFAL/wbp_ufal-ud-test.conllu\"\n\"UD_27 test set of WARLPIRI_UFAL.\"\nUD_27_WELSH_CCG_TRAIN = _UD_27_HOME + \"UD_Welsh-CCG/cy_ccg-ud-train.conllu\"\n\"UD_27 train set of WELSH_CCG.\"\nUD_27_WELSH_CCG_TEST = _UD_27_HOME + \"UD_Welsh-CCG/cy_ccg-ud-test.conllu\"\n\"UD_27 test set of WELSH_CCG.\"\nUD_27_WOLOF_WTB_TRAIN = _UD_27_HOME + \"UD_Wolof-WTB/wo_wtb-ud-train.conllu\"\n\"UD_27 train set of WOLOF_WTB.\"\nUD_27_WOLOF_WTB_DEV = _UD_27_HOME + \"UD_Wolof-WTB/wo_wtb-ud-dev.conllu\"\n\"UD_27 dev set of WOLOF_WTB.\"\nUD_27_WOLOF_WTB_TEST = _UD_27_HOME + \"UD_Wolof-WTB/wo_wtb-ud-test.conllu\"\n\"UD_27 test set of WOLOF_WTB.\"\nUD_27_YORUBA_YTB_TEST = _UD_27_HOME + \"UD_Yoruba-YTB/yo_ytb-ud-test.conllu\"\n\"UD_27 test set of YORUBA_YTB.\"\n"
  },
  {
    "path": "hanlp/datasets/parsing/ud/ud27m.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-21 20:39\nimport os\n\nfrom hanlp.datasets.parsing.ud import concat_treebanks\nfrom hanlp.datasets.parsing.ud.ud27 import _UD_27_HOME\n\n_UD_27_MULTILINGUAL_HOME = concat_treebanks(_UD_27_HOME, '2.7')\nUD_27_MULTILINGUAL_TRAIN = os.path.join(_UD_27_MULTILINGUAL_HOME, 'train.conllu')\n\"Training set of multilingual UD_27 obtained by concatenating all training sets.\"\nUD_27_MULTILINGUAL_DEV = os.path.join(_UD_27_MULTILINGUAL_HOME, 'dev.conllu')\n\"Dev set of multilingual UD_27 obtained by concatenating all dev sets.\"\nUD_27_MULTILINGUAL_TEST = os.path.join(_UD_27_MULTILINGUAL_HOME, 'test.conllu')\n\"Test set of multilingual UD_27 obtained by concatenating all test sets.\"\n"
  },
  {
    "path": "hanlp/datasets/pos/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 22:50"
  },
  {
    "path": "hanlp/datasets/pos/ctb5.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 22:51\n\n_CTB5_POS_HOME = 'http://file.hankcs.com/corpus/ctb5.1-pos.zip'\n\nCTB5_POS_TRAIN = f'{_CTB5_POS_HOME}#train.tsv'\n'''PoS training set for CTB5.'''\nCTB5_POS_DEV = f'{_CTB5_POS_HOME}#dev.tsv'\n'''PoS dev set for CTB5.'''\nCTB5_POS_TEST = f'{_CTB5_POS_HOME}#test.tsv'\n'''PoS test set for CTB5.'''\n"
  },
  {
    "path": "hanlp/datasets/qa/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-03-20 19:17"
  },
  {
    "path": "hanlp/datasets/qa/hotpotqa.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-03-20 19:46\nfrom enum import Enum, auto\n\nimport torch\nimport ujson\nfrom torch.nn.utils.rnn import pad_sequence\n\nfrom hanlp.common.dataset import TransformableDataset\nfrom hanlp_common.util import merge_list_of_dict\n\nHOTPOT_QA_TRAIN = 'http://curtis.ml.cmu.edu/datasets/hotpot/hotpot_train_v1.1.json'\nHOTPOT_QA_DISTRACTOR_DEV = 'http://curtis.ml.cmu.edu/datasets/hotpot/hotpot_dev_distractor_v1.json'\nHOTPOT_QA_FULLWIKI_DEV = 'http://curtis.ml.cmu.edu/datasets/hotpot/hotpot_dev_fullwiki_v1.json'\n\n\nclass HotpotQADataset(TransformableDataset):\n\n    def load_file(self, filepath):\n        with open(filepath) as fd:\n            return ujson.load(fd)\n\n\nclass BuildGraph(object):\n\n    def __init__(self, dst='graph') -> None:\n        super().__init__()\n        self.dst = dst\n\n    def __call__(self, sample: dict):\n        sample[self.dst] = build_graph(sample)\n        return sample\n\n\ndef hotpotqa_collate_fn(samples):\n    batch = merge_list_of_dict(samples)\n    max_seq_len = len(max([x['graph'] for x in samples], key=len))\n    arc = torch.zeros([len(samples), max_seq_len, max_seq_len])\n    token_offset = torch.zeros([len(samples), max_seq_len], dtype=torch.long)\n    src_mask = torch.zeros([len(samples), max_seq_len], dtype=torch.bool)\n    sp_candidate_mask = torch.zeros([len(samples), max_seq_len], dtype=torch.bool)\n    sp_label = torch.zeros([len(samples), max_seq_len], dtype=torch.float)\n    # sp = torch.zeros([len(samples), max_seq_len], dtype=torch.bool)\n    tokens = []\n    offset = 0\n    for i, sample in enumerate(samples):\n        graph = sample['graph']\n        for j, u in enumerate(graph):\n            u: Vertex = u\n            for v in u.to:\n                v: Vertex = v\n                arc[i, v.id, u.id] = 1\n                arc[i, u.id, v.id] = 1\n            # record each vertex's token offset\n            token_offset[i, u.id] = offset\n            src_mask[i, u.id] = True\n            sp_candidate_mask[i, u.id] = u.is_sp_root_candidate()\n            sp_label[i, u.id] = u.is_sp_root()\n            offset += 1\n        tokens.extend(sample['token_id'])\n    seq_lengths = torch.LongTensor(list(map(len, tokens)))\n    tokens = [torch.LongTensor(x) for x in tokens]\n    tokens = pad_sequence(tokens, batch_first=True)\n    batch['adj'] = arc\n    batch['tokens'] = tokens\n    batch['src_mask'] = src_mask\n    batch['seq_lengths'] = seq_lengths\n    batch['token_offset'] = token_offset\n    batch['sp_candidate_mask'] = sp_candidate_mask\n    batch['sp_label'] = sp_label\n    return batch\n\n\ndef flat_sentence(sample: dict) -> dict:\n    sample['token'] = token = []\n    for sent in sample['parsed_sentences']:\n        token.append(['bos'] + [x.lower() for x in sent[0]])\n    return sample\n\n\ndef create_sp_label(sample: dict) -> dict:\n    sample['sp_label'] = sp_label = []\n\n    def label(title_, index_):\n        for t, i in sample['supporting_facts']:\n            if t == title_ and i == index_:\n                return 1\n        return 0\n\n    for context in sample['context']:\n        title, sents = context\n        for idx, sent in enumerate(sents):\n            sp_label.append(label(title, idx))\n    assert len(sample['supporting_facts']) == sum(sp_label)\n    return sample\n\n\nclass Type(Enum):\n    Q_ROOT = auto()\n    Q_WORD = auto()\n    SP_ROOT = auto()\n    SP_WORD = auto()\n    NON_SP_ROOT = auto()\n    NON_SP_WORD = auto()\n    DOCUMENT_TITLE = auto()\n\n\nclass Vertex(object):\n\n    def __init__(self, id, type: Type, text=None) -> None:\n        super().__init__()\n        self.id = id\n        self.type = type\n        if not text:\n            text = str(type).split('.')[-1]\n        self.text = text\n        self.to = []\n        self.rel = []\n\n    def connect(self, to, rel):\n        self.to.append(to)\n        self.rel.append(rel)\n\n    def __str__(self) -> str:\n        return f'{self.text} {self.id}'\n\n    def __hash__(self) -> int:\n        return self.id\n\n    def is_word(self):\n        return self.type in {Type.SP_WORD, Type.Q_WORD, Type.NON_SP_WORD}\n\n    def is_question(self):\n        return self.type in {Type.Q_ROOT, Type.Q_WORD}\n\n    def is_sp(self):\n        return self.type in {Type.SP_ROOT, Type.SP_WORD}\n\n    def is_sp_root(self):\n        return self.type in {Type.SP_ROOT}\n\n    def is_sp_root_candidate(self):\n        return self.type in {Type.SP_ROOT, Type.NON_SP_ROOT}\n\n\ndef build_graph(each: dict, debug=False):\n    raw_sents = []\n    raw_sents.append(each['question'])\n    sp_idx = set()\n    sp_sents = {}\n    for sp in each['supporting_facts']:\n        title, offset = sp\n        ids = sp_sents.get(title, None)\n        if ids is None:\n            sp_sents[title] = ids = set()\n        ids.add(offset)\n    idx = 1\n    for document in each['context']:\n        title, sents = document\n        raw_sents += sents\n        for i, s in enumerate(sents):\n            if title in sp_sents and i in sp_sents[title]:\n                sp_idx.add(idx)\n            idx += 1\n    assert idx == len(raw_sents)\n    parsed_sents = each['parsed_sentences']\n    assert len(raw_sents) == len(parsed_sents)\n    graph = []\n    for idx, (raw, sent) in enumerate(zip(raw_sents, parsed_sents)):\n        if debug:\n            if idx > 1 and idx not in sp_idx:\n                continue\n        offset = len(graph)\n        if idx == 0:\n            if debug:\n                print(f'Question: {raw}')\n            graph.append(Vertex(len(graph), Type.Q_ROOT))\n        else:\n            if debug:\n                if idx in sp_idx:\n                    print(f'Supporting Fact: {raw}')\n            graph.append(Vertex(len(graph), Type.SP_ROOT if idx in sp_idx else Type.NON_SP_ROOT))\n        tokens, heads, deprels = sent\n        for t, h, d in zip(tokens, heads, deprels):\n            graph.append(\n                Vertex(len(graph), (Type.SP_WORD if idx in sp_idx else Type.NON_SP_WORD) if idx else Type.Q_WORD, t))\n        for i, (h, d) in enumerate(zip(heads, deprels)):\n            graph[offset + h].connect(graph[offset + i + 1], d)\n    q_root = graph[0]\n    for u in graph:\n        if u.type == Type.SP_ROOT or u.type == Type.NON_SP_ROOT:\n            q_root.connect(u, 'supporting fact?')\n    return graph\n"
  },
  {
    "path": "hanlp/datasets/srl/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-22 19:15\n\n\n"
  },
  {
    "path": "hanlp/datasets/srl/loaders/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-12-28 19:05\n"
  },
  {
    "path": "hanlp/datasets/srl/loaders/conll2012.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-22 19:15\nimport glob\nimport json\nimport os\nfrom typing import Union, List, Callable\n\nfrom hanlp.utils.span_util import enumerate_spans\n\nfrom hanlp.common.dataset import TransformableDataset\nfrom hanlp.common.transform import NamedTransform\nfrom hanlp.utils.io_util import read_tsv_as_sents, get_resource, TimingFileIterator\nfrom hanlp.utils.time_util import CountdownTimer\n\n\nclass CoNLL2012BIOSRLDataset(TransformableDataset):\n    def load_file(self, filepath: str):\n        filepath = get_resource(filepath)\n        if os.path.isfile(filepath):\n            files = [filepath]\n        else:\n            assert os.path.isdir(filepath), f'{filepath} has to be a directory of CoNLL 2012'\n            files = sorted(glob.glob(f'{filepath}/**/*gold_conll', recursive=True))\n        timer = CountdownTimer(len(files))\n        for fid, f in enumerate(files):\n            timer.log(f'files loading[blink][yellow]...[/yellow][/blink]')\n            # 0:DOCUMENT 1:PART 2:INDEX 3:WORD 4:POS 5:PARSE 6:LEMMA 7:FRAME 8:SENSE 9:SPEAKER 10:NE 11-N:ARGS N:COREF\n            for sent in read_tsv_as_sents(f, ignore_prefix='#'):\n                sense = [cell[7] for cell in sent]\n                props = [cell[11:-1] for cell in sent]\n                props = map(lambda p: p, zip(*props))\n                prd_bio_labels = [self._make_bio_labels(prop) for prop in props]\n                prd_bio_labels = [self._remove_B_V(x) for x in prd_bio_labels]\n                prd_indices = [i for i, x in enumerate(sense) if x != '-']\n                token = [x[3] for x in sent]\n                srl = [None for x in token]\n                for idx, labels in zip(prd_indices, prd_bio_labels):\n                    srl[idx] = labels\n                srl = [x if x else ['O'] * len(token) for x in srl]\n                yield {'token': token, 'srl': srl}\n\n    @staticmethod\n    def _make_bio_labels(prop):\n        \"\"\"Copied from https://github.com/hiroki13/span-based-srl/blob/2c8b677c4e00b6c607e09ef4f9fe3d54961e4f2e/src/utils/sent.py#L42\n\n        Args:\n          prop: 1D: n_words; elem=bracket label\n\n        Returns:\n          1D: n_words; elem=BIO label\n\n        \"\"\"\n        labels = []\n        prev = None\n        for arg in prop:\n            if arg.startswith('('):\n                if arg.endswith(')'):\n                    prev = arg.split(\"*\")[0][1:]\n                    label = 'B-' + prev\n                    prev = None\n                else:\n                    prev = arg[1:-1]\n                    label = 'B-' + prev\n            else:\n                if prev:\n                    label = 'I-' + prev\n                    if arg.endswith(')'):\n                        prev = None\n                else:\n                    label = 'O'\n            labels.append(label)\n        return labels\n\n    @staticmethod\n    def _remove_B_V(labels):\n        return ['O' if x == 'B-V' else x for x in labels]\n\n\nclass CoNLL2012SRLDataset(TransformableDataset):\n\n    def __init__(self,\n                 data: Union[str, List],\n                 transform: Union[Callable, List] = None,\n                 cache=None,\n                 doc_level_offset=True,\n                 generate_idx=None) -> None:\n        self.doc_level_offset = doc_level_offset\n        super().__init__(data, transform, cache, generate_idx=generate_idx)\n\n    def load_file(self, filepath: str):\n        \"\"\"Load ``.jsonlines`` CoNLL12-style corpus. Samples of this corpus can be found using the following scripts.\n\n        .. highlight:: python\n        .. code-block:: python\n\n            import json\n            from hanlp_common.document import Document\n            from hanlp.datasets.srl.ontonotes5.chinese import ONTONOTES5_CONLL12_CHINESE_DEV\n            from hanlp.utils.io_util import get_resource\n\n            with open(get_resource(ONTONOTES5_CONLL12_CHINESE_DEV)) as src:\n                for line in src:\n                    doc = json.loads(line)\n                    print(Document(doc))\n                    break\n\n        Args:\n            filepath: ``.jsonlines`` CoNLL12 corpus.\n        \"\"\"\n        filename = os.path.basename(filepath)\n        reader = TimingFileIterator(filepath)\n        num_docs, num_sentences = 0, 0\n        for line in reader:\n            doc = json.loads(line)\n            num_docs += 1\n            num_tokens_in_doc = 0\n            for sid, (sentence, srl) in enumerate(zip(doc['sentences'], doc['srl'])):\n                if self.doc_level_offset:\n                    srl = [(x[0] - num_tokens_in_doc, x[1] - num_tokens_in_doc, x[2] - num_tokens_in_doc, x[3]) for x in\n                           srl]\n                else:\n                    srl = [(x[0], x[1], x[2], x[3]) for x in srl]\n                for x in srl:\n                    if any([o < 0 for o in x[:3]]):\n                        raise ValueError(f'Negative offset occurred, maybe doc_level_offset=False')\n                    if any([o >= len(sentence) for o in x[:3]]):\n                        raise ValueError('Offset exceeds sentence length, maybe doc_level_offset=True')\n                deduplicated_srl = set()\n                pa_set = set()\n                for p, b, e, l in srl:\n                    pa = (p, b, e)\n                    if pa in pa_set:\n                        continue\n                    pa_set.add(pa)\n                    deduplicated_srl.add((p, b, e, l))\n                yield self.build_sample(sentence, deduplicated_srl, doc, sid)\n                num_sentences += 1\n                num_tokens_in_doc += len(sentence)\n            reader.log(\n                f'{filename} {num_docs} documents, {num_sentences} sentences [blink][yellow]...[/yellow][/blink]')\n        reader.erase()\n\n    # noinspection PyMethodMayBeStatic\n    def build_sample(self, sentence, deduplicated_srl, doc, sid):\n        return {\n            'token': sentence,\n            'srl': deduplicated_srl\n        }\n\n\ndef group_pa_by_p(sample: dict) -> dict:\n    if 'srl' in sample:\n        srl: list = sample['srl']\n        grouped_srl = group_pa_by_p_(srl)\n        sample['srl'] = grouped_srl\n    return sample\n\n\ndef group_pa_by_p_(srl):\n    grouped_srl = {}\n    for p, b, e, l in srl:\n        bel = grouped_srl.get(p, None)\n        if not bel:\n            bel = grouped_srl[p] = set()\n        bel.add((b, e, l))\n    return grouped_srl\n\n\ndef filter_v_args(sample: dict) -> dict:\n    if 'srl' in sample:\n        sample['srl'] = [t for t in sample['srl'] if t[-1] not in [\"V\", \"C-V\"]]\n    return sample\n\n\ndef unpack_srl(sample: dict) -> dict:\n    if 'srl' in sample:\n        srl = sample['srl']\n        predicate_offset = [x[0] for x in srl]\n        argument_begin_offset = [x[1] for x in srl]\n        argument_end_offset = [x[2] for x in srl]\n        srl_label = [x[-1] for x in srl]\n        sample.update({\n            'predicate_offset': predicate_offset,\n            'argument_begin_offset': argument_begin_offset,\n            'argument_end_offset': argument_end_offset,\n            'srl_label': srl_label,  # We can obtain mask by srl_label > 0\n            # 'srl_mask': len(srl_label),\n        })\n    return sample\n\n\nclass SpanCandidatesGenerator(NamedTransform):\n\n    def __init__(self, src: str, dst: str = None, max_span_width=None) -> None:\n        if not dst:\n            dst = f'{src}_span'\n        super().__init__(src, dst)\n        self.max_span_width = max_span_width\n\n    def __call__(self, sample: dict) -> dict:\n        sample[self.dst] = list(enumerate_spans(sample[self.src], max_span_width=self.max_span_width))\n        return sample\n\n\nclass CoNLL2012SRLBIODataset(CoNLL2012SRLDataset):\n    def build_sample(self, tokens, deduplicated_srl, doc, sid):\n        # Convert srl to exclusive format\n        deduplicated_srl = set((x[0], x[1], x[2] + 1, x[3]) for x in deduplicated_srl if x[3] != 'V')\n        labels = [['O'] * len(tokens) for _ in range(len(tokens))]\n        srl = group_pa_by_p_(deduplicated_srl)\n        for p, args in sorted(srl.items()):\n            labels_per_p = labels[p]\n            for start, end, label in args:\n                assert end > start\n                assert label != 'V'  # We don't predict predicate\n                labels_per_p[start] = 'B-' + label\n                for j in range(start + 1, end):\n                    labels_per_p[j] = 'I-' + label\n        sample = {\n            'token': tokens,\n            'srl': labels,\n            'srl_set': deduplicated_srl,\n        }\n        if 'pos' in doc:\n            sample['pos'] = doc['pos'][sid]\n        return sample\n"
  },
  {
    "path": "hanlp/datasets/srl/loaders/ontonotes_loader.py",
    "content": "from typing import DefaultDict, List, Optional, Iterator, Set, Tuple, Dict\nfrom collections import defaultdict\nimport codecs\nimport os\nimport logging\n\nfrom hanlp.utils.span_util import TypedSpan, enumerate_spans\nfrom phrasetree.tree import Tree\n\nlogger = logging.getLogger(__name__)\n\n\nclass OntonotesSentence:\n    \"\"\"\n    A class representing the annotations available for a single CONLL formatted sentence.\n\n    # Parameters\n\n    document_id : `str`\n        This is a variation on the document filename\n    sentence_id : `int`\n        The integer ID of the sentence within a document.\n    words : `List[str]`\n        This is the tokens as segmented/tokenized in the Treebank.\n    pos_tags : `List[str]`\n        This is the Penn-Treebank-style part of speech. When parse information is missing,\n        all parts of speech except the one for which there is some sense or proposition\n        annotation are marked with a XX tag. The verb is marked with just a VERB tag.\n    parse_tree : `nltk.Tree`\n        An nltk Tree representing the parse. It includes POS tags as pre-terminal nodes.\n        When the parse information is missing, the parse will be `None`.\n    predicate_lemmas : `List[Optional[str]]`\n        The predicate lemma of the words for which we have semantic role\n        information or word sense information. All other indices are `None`.\n    predicate_framenet_ids : `List[Optional[int]]`\n        The PropBank frameset ID of the lemmas in `predicate_lemmas`, or `None`.\n    word_senses : `List[Optional[float]]`\n        The word senses for the words in the sentence, or `None`. These are floats\n        because the word sense can have values after the decimal, like `1.1`.\n    speakers : `List[Optional[str]]`\n        The speaker information for the words in the sentence, if present, or `None`\n        This is the speaker or author name where available. Mostly in Broadcast Conversation\n        and Web Log data. When not available the rows are marked with an \"-\".\n    named_entities : `List[str]`\n        The BIO tags for named entities in the sentence.\n    srl_frames : `List[Tuple[str, List[str]]]`\n        A dictionary keyed by the verb in the sentence for the given\n        Propbank frame labels, in a BIO format.\n    coref_spans : `Set[TypedSpan]`\n        The spans for entity mentions involved in coreference resolution within the sentence.\n        Each element is a tuple composed of (cluster_id, (start_index, end_index)). Indices\n        are `inclusive`.\n    \"\"\"\n\n    def __init__(\n        self,\n        document_id: str,\n        sentence_id: int,\n        words: List[str],\n        pos_tags: List[str],\n        parse_tree: Optional[Tree],\n        predicate_lemmas: List[Optional[str]],\n        predicate_framenet_ids: List[Optional[str]],\n        word_senses: List[Optional[float]],\n        speakers: List[Optional[str]],\n        named_entities: List[str],\n        srl_frames: List[Tuple[str, List[str]]],\n        coref_spans: Set[TypedSpan],\n    ) -> None:\n\n        self.document_id = document_id\n        self.sentence_id = sentence_id\n        self.words = words\n        self.pos_tags = pos_tags\n        self.parse_tree = parse_tree\n        self.predicate_lemmas = predicate_lemmas\n        self.predicate_framenet_ids = predicate_framenet_ids\n        self.word_senses = word_senses\n        self.speakers = speakers\n        self.named_entities = named_entities\n        self.srl_frames = srl_frames\n        self.coref_spans = coref_spans\n\n\nclass Ontonotes:\n    \"\"\"\n    This `DatasetReader` is designed to read in the English OntoNotes v5.0 data\n    in the format used by the CoNLL 2011/2012 shared tasks. In order to use this\n    Reader, you must follow the instructions provided [here (v12 release):]\n    (https://cemantix.org/data/ontonotes.html), which will allow you to download\n    the CoNLL style annotations for the  OntoNotes v5.0 release -- LDC2013T19.tgz\n    obtained from LDC.\n\n    Once you have run the scripts on the extracted data, you will have a folder\n    structured as follows:\n\n    ```\n    conll-formatted-ontonotes-5.0/\n     ── data\n       ├── development\n           └── data\n               └── english\n                   └── annotations\n                       ├── bc\n                       ├── bn\n                       ├── mz\n                       ├── nw\n                       ├── pt\n                       ├── tc\n                       └── wb\n       ├── test\n           └── data\n               └── english\n                   └── annotations\n                       ├── bc\n                       ├── bn\n                       ├── mz\n                       ├── nw\n                       ├── pt\n                       ├── tc\n                       └── wb\n       └── train\n           └── data\n               └── english\n                   └── annotations\n                       ├── bc\n                       ├── bn\n                       ├── mz\n                       ├── nw\n                       ├── pt\n                       ├── tc\n                       └── wb\n    ```\n\n    The file path provided to this class can then be any of the train, test or development\n    directories(or the top level data directory, if you are not utilizing the splits).\n\n    The data has the following format, ordered by column.\n\n    1.  Document ID : `str`\n        This is a variation on the document filename\n    2.  Part number : `int`\n        Some files are divided into multiple parts numbered as 000, 001, 002, ... etc.\n    3.  Word number : `int`\n        This is the word index of the word in that sentence.\n    4.  Word : `str`\n        This is the token as segmented/tokenized in the Treebank. Initially the `*_skel` file\n        contain the placeholder [WORD] which gets replaced by the actual token from the\n        Treebank which is part of the OntoNotes release.\n    5.  POS Tag : `str`\n        This is the Penn Treebank style part of speech. When parse information is missing,\n        all part of speeches except the one for which there is some sense or proposition\n        annotation are marked with a XX tag. The verb is marked with just a VERB tag.\n    6.  Parse bit : `str`\n        This is the bracketed structure broken before the first open parenthesis in the parse,\n        and the word/part-of-speech leaf replaced with a `*`. When the parse information is\n        missing, the first word of a sentence is tagged as `(TOP*` and the last word is tagged\n        as `*)` and all intermediate words are tagged with a `*`.\n    7.  Predicate lemma : `str`\n        The predicate lemma is mentioned for the rows for which we have semantic role\n        information or word sense information. All other rows are marked with a \"-\".\n    8.  Predicate Frameset ID : `int`\n        The PropBank frameset ID of the predicate in Column 7.\n    9.  Word sense : `float`\n        This is the word sense of the word in Column 3.\n    10. Speaker/Author : `str`\n        This is the speaker or author name where available. Mostly in Broadcast Conversation\n        and Web Log data. When not available the rows are marked with an \"-\".\n    11. Named Entities : `str`\n        These columns identifies the spans representing various named entities. For documents\n        which do not have named entity annotation, each line is represented with an `*`.\n    12. Predicate Arguments : `str`\n        There is one column each of predicate argument structure information for the predicate\n        mentioned in Column 7. If there are no predicates tagged in a sentence this is a\n        single column with all rows marked with an `*`.\n    -1. Co-reference : `str`\n        Co-reference chain information encoded in a parenthesis structure. For documents that do\n         not have co-reference annotations, each line is represented with a \"-\".\n    \"\"\"\n\n    def dataset_iterator(self, file_path: str) -> Iterator[OntonotesSentence]:\n        \"\"\"\n        An iterator over the entire dataset, yielding all sentences processed.\n        \"\"\"\n        for conll_file in self.dataset_path_iterator(file_path):\n            yield from self.sentence_iterator(conll_file)\n\n    @staticmethod\n    def dataset_path_iterator(file_path: str) -> Iterator[str]:\n        \"\"\"\n        An iterator returning file_paths in a directory\n        containing CONLL-formatted files.\n        \"\"\"\n        logger.info(\"Reading CONLL sentences from dataset files at: %s\", file_path)\n        for root, _, files in list(os.walk(file_path)):\n            for data_file in files:\n                # These are a relic of the dataset pre-processing. Every\n                # file will be duplicated - one file called filename.gold_skel\n                # and one generated from the preprocessing called filename.gold_conll.\n                if not data_file.endswith(\"gold_conll\"):\n                    continue\n\n                yield os.path.join(root, data_file)\n\n    def dataset_document_iterator(self, file_path: str) -> Iterator[List[OntonotesSentence]]:\n        \"\"\"\n        An iterator over CONLL formatted files which yields documents, regardless\n        of the number of document annotations in a particular file. This is useful\n        for conll data which has been preprocessed, such as the preprocessing which\n        takes place for the 2012 CONLL Coreference Resolution task.\n        \"\"\"\n        with codecs.open(file_path, \"r\", encoding=\"utf8\") as open_file:\n            conll_rows = []\n            document: List[OntonotesSentence] = []\n            for line in open_file:\n                line = line.strip()\n                if line != \"\" and not line.startswith(\"#\"):\n                    # Non-empty line. Collect the annotation.\n                    conll_rows.append(line)\n                else:\n                    if conll_rows:\n                        document.append(self._conll_rows_to_sentence(conll_rows))\n                        conll_rows = []\n                if line.startswith(\"#end document\"):\n                    yield document\n                    document = []\n            if document:\n                # Collect any stragglers or files which might not\n                # have the '#end document' format for the end of the file.\n                yield document\n\n    def sentence_iterator(self, file_path: str) -> Iterator[OntonotesSentence]:\n        \"\"\"\n        An iterator over the sentences in an individual CONLL formatted file.\n        \"\"\"\n        for document in self.dataset_document_iterator(file_path):\n            for sentence in document:\n                yield sentence\n\n    def _conll_rows_to_sentence(self, conll_rows: List[str]) -> OntonotesSentence:\n        document_id: str = None\n        sentence_id: int = None\n        # The words in the sentence.\n        sentence: List[str] = []\n        # The pos tags of the words in the sentence.\n        pos_tags: List[str] = []\n        # the pieces of the parse tree.\n        parse_pieces: List[str] = []\n        # The lemmatised form of the words in the sentence which\n        # have SRL or word sense information.\n        predicate_lemmas: List[str] = []\n        # The FrameNet ID of the predicate.\n        predicate_framenet_ids: List[str] = []\n        # The sense of the word, if available.\n        word_senses: List[float] = []\n        # The current speaker, if available.\n        speakers: List[str] = []\n\n        verbal_predicates: List[str] = []\n        span_labels: List[List[str]] = []\n        current_span_labels: List[str] = []\n\n        # Cluster id -> List of (start_index, end_index) spans.\n        clusters: DefaultDict[int, List[Tuple[int, int]]] = defaultdict(list)\n        # Cluster id -> List of start_indices which are open for this id.\n        coref_stacks: DefaultDict[int, List[int]] = defaultdict(list)\n\n        for index, row in enumerate(conll_rows):\n            conll_components = row.split()\n\n            document_id = conll_components[0]\n            sentence_id = int(conll_components[1])\n            word = conll_components[3]\n            pos_tag = conll_components[4]\n            parse_piece = conll_components[5]\n\n            # Replace brackets in text and pos tags\n            # with a different token for parse trees.\n            if pos_tag != \"XX\" and word != \"XX\":\n                if word == \"(\":\n                    parse_word = \"-LRB-\"\n                elif word == \")\":\n                    parse_word = \"-RRB-\"\n                else:\n                    parse_word = word\n                if pos_tag == \"(\":\n                    pos_tag = \"-LRB-\"\n                if pos_tag == \")\":\n                    pos_tag = \"-RRB-\"\n                (left_brackets, right_hand_side) = parse_piece.split(\"*\")\n                # only keep ')' if there are nested brackets with nothing in them.\n                right_brackets = right_hand_side.count(\")\") * \")\"\n                parse_piece = f\"{left_brackets} ({pos_tag} {parse_word}) {right_brackets}\"\n            else:\n                # There are some bad annotations in the CONLL data.\n                # They contain no information, so to make this explicit,\n                # we just set the parse piece to be None which will result\n                # in the overall parse tree being None.\n                parse_piece = None\n\n            lemmatised_word = conll_components[6]\n            framenet_id = conll_components[7]\n            word_sense = conll_components[8]\n            speaker = conll_components[9]\n\n            if not span_labels:\n                # If this is the first word in the sentence, create\n                # empty lists to collect the NER and SRL BIO labels.\n                # We can't do this upfront, because we don't know how many\n                # components we are collecting, as a sentence can have\n                # variable numbers of SRL frames.\n                span_labels = [[] for _ in conll_components[10:-1]]\n                # Create variables representing the current label for each label\n                # sequence we are collecting.\n                current_span_labels = [None for _ in conll_components[10:-1]]\n\n            self._process_span_annotations_for_word(\n                conll_components[10:-1], span_labels, current_span_labels\n            )\n\n            # If any annotation marks this word as a verb predicate,\n            # we need to record its index. This also has the side effect\n            # of ordering the verbal predicates by their location in the\n            # sentence, automatically aligning them with the annotations.\n            word_is_verbal_predicate = any(\"(V\" in x for x in conll_components[11:-1])\n            if word_is_verbal_predicate:\n                verbal_predicates.append(word)\n\n            self._process_coref_span_annotations_for_word(\n                conll_components[-1], index, clusters, coref_stacks\n            )\n\n            sentence.append(word)\n            pos_tags.append(pos_tag)\n            parse_pieces.append(parse_piece)\n            predicate_lemmas.append(lemmatised_word if lemmatised_word != \"-\" else None)\n            predicate_framenet_ids.append(framenet_id if framenet_id != \"-\" else None)\n            word_senses.append(float(word_sense) if word_sense != \"-\" else None)\n            speakers.append(speaker if speaker != \"-\" else None)\n\n        named_entities = span_labels[0]\n        srl_frames = [\n            (predicate, labels) for predicate, labels in zip(verbal_predicates, span_labels[1:])\n        ]\n\n        if all(parse_pieces):\n            parse_tree = Tree.fromstring(\"\".join(parse_pieces))\n        else:\n            parse_tree = None\n        coref_span_tuples: Set[TypedSpan] = {\n            (cluster_id, span) for cluster_id, span_list in clusters.items() for span in span_list\n        }\n        return OntonotesSentence(\n            document_id,\n            sentence_id,\n            sentence,\n            pos_tags,\n            parse_tree,\n            predicate_lemmas,\n            predicate_framenet_ids,\n            word_senses,\n            speakers,\n            named_entities,\n            srl_frames,\n            coref_span_tuples,\n        )\n\n    @staticmethod\n    def _process_coref_span_annotations_for_word(\n        label: str,\n        word_index: int,\n        clusters: DefaultDict[int, List[Tuple[int, int]]],\n        coref_stacks: DefaultDict[int, List[int]],\n    ) -> None:\n        \"\"\"\n        For a given coref label, add it to a currently open span(s), complete a span(s) or\n        ignore it, if it is outside of all spans. This method mutates the clusters and coref_stacks\n        dictionaries.\n\n        # Parameters\n\n        label : `str`\n            The coref label for this word.\n        word_index : `int`\n            The word index into the sentence.\n        clusters : `DefaultDict[int, List[Tuple[int, int]]]`\n            A dictionary mapping cluster ids to lists of inclusive spans into the\n            sentence.\n        coref_stacks : `DefaultDict[int, List[int]]`\n            Stacks for each cluster id to hold the start indices of active spans (spans\n            which we are inside of when processing a given word). Spans with the same id\n            can be nested, which is why we collect these opening spans on a stack, e.g:\n\n            [Greg, the baker who referred to [himself]_ID1 as 'the bread man']_ID1\n        \"\"\"\n        if label != \"-\":\n            for segment in label.split(\"|\"):\n                # The conll representation of coref spans allows spans to\n                # overlap. If spans end or begin at the same word, they are\n                # separated by a \"|\".\n                if segment[0] == \"(\":\n                    # The span begins at this word.\n                    if segment[-1] == \")\":\n                        # The span begins and ends at this word (single word span).\n                        cluster_id = int(segment[1:-1])\n                        clusters[cluster_id].append((word_index, word_index))\n                    else:\n                        # The span is starting, so we record the index of the word.\n                        cluster_id = int(segment[1:])\n                        coref_stacks[cluster_id].append(word_index)\n                else:\n                    # The span for this id is ending, but didn't start at this word.\n                    # Retrieve the start index from the document state and\n                    # add the span to the clusters for this id.\n                    cluster_id = int(segment[:-1])\n                    start = coref_stacks[cluster_id].pop()\n                    clusters[cluster_id].append((start, word_index))\n\n    @staticmethod\n    def _process_span_annotations_for_word(\n        annotations: List[str],\n        span_labels: List[List[str]],\n        current_span_labels: List[Optional[str]],\n    ) -> None:\n        \"\"\"\n        Given a sequence of different label types for a single word and the current\n        span label we are inside, compute the BIO tag for each label and append to a list.\n\n        # Parameters\n\n        annotations : `List[str]`\n            A list of labels to compute BIO tags for.\n        span_labels : `List[List[str]]`\n            A list of lists, one for each annotation, to incrementally collect\n            the BIO tags for a sequence.\n        current_span_labels : `List[Optional[str]]`\n            The currently open span per annotation type, or `None` if there is no open span.\n        \"\"\"\n        for annotation_index, annotation in enumerate(annotations):\n            # strip all bracketing information to\n            # get the actual propbank label.\n            label = annotation.strip(\"()*\")\n\n            if \"(\" in annotation:\n                # Entering into a span for a particular semantic role label.\n                # We append the label and set the current span for this annotation.\n                bio_label = \"B-\" + label\n                span_labels[annotation_index].append(bio_label)\n                current_span_labels[annotation_index] = label\n            elif current_span_labels[annotation_index] is not None:\n                # If there's no '(' token, but the current_span_label is not None,\n                # then we are inside a span.\n                bio_label = \"I-\" + current_span_labels[annotation_index]\n                span_labels[annotation_index].append(bio_label)\n            else:\n                # We're outside a span.\n                span_labels[annotation_index].append(\"O\")\n            # Exiting a span, so we reset the current span label for this annotation.\n            if \")\" in annotation:\n                current_span_labels[annotation_index] = None\n\n\ndef make_coref_instance(\n        sentences: List[List[str]],\n        max_span_width: int,\n        gold_clusters: Optional[List[List[Tuple[int, int]]]] = None,\n        max_sentences: int = None,\n        remove_singleton_clusters: bool = True,\n) -> dict:\n    \"\"\"\n    # Parameters\n\n    sentences : `List[List[str]]`, required.\n        A list of lists representing the tokenised words and sentences in the document.\n    token_indexers : `Dict[str, TokenIndexer]`\n        This is used to index the words in the document.  See :class:`TokenIndexer`.\n    max_span_width : `int`, required.\n        The maximum width of candidate spans to consider.\n    gold_clusters : `Optional[List[List[Tuple[int, int]]]]`, optional (default = None)\n        A list of all clusters in the document, represented as word spans with absolute indices\n        in the entire document. Each cluster contains some number of spans, which can be nested\n        and overlap. If there are exact matches between clusters, they will be resolved\n        using `_canonicalize_clusters`.\n    wordpiece_modeling_tokenizer: `PretrainedTransformerTokenizer`, optional (default = None)\n        If not None, this dataset reader does subword tokenization using the supplied tokenizer\n        and distribute the labels to the resulting wordpieces. All the modeling will be based on\n        wordpieces. If this is set to `False` (default), the user is expected to use\n        `PretrainedTransformerMismatchedIndexer` and `PretrainedTransformerMismatchedEmbedder`,\n        and the modeling will be on the word-level.\n    max_sentences: int, optional (default = None)\n        The maximum number of sentences in each document to keep. By default keeps all sentences.\n    remove_singleton_clusters : `bool`, optional (default = True)\n        Some datasets contain clusters that are singletons (i.e. no coreferents). This option allows\n        the removal of them.\n\n    # Returns\n\n    An `Instance` containing the following `Fields`:\n        text : `TextField`\n            The text of the full document.\n        spans : `ListField[SpanField]`\n            A ListField containing the spans represented as `SpanFields`\n            with respect to the document text.\n        span_labels : `SequenceLabelField`, optional\n            The id of the cluster which each possible span belongs to, or -1 if it does\n                not belong to a cluster. As these labels have variable length (it depends on\n                how many spans we are considering), we represent this a as a `SequenceLabelField`\n                with respect to the spans `ListField`.\n    \"\"\"\n    if max_sentences is not None and len(sentences) > max_sentences:\n        sentences = sentences[:max_sentences]\n        total_length = sum(len(sentence) for sentence in sentences)\n\n        if gold_clusters is not None:\n            new_gold_clusters = []\n\n            for cluster in gold_clusters:\n                new_cluster = []\n                for mention in cluster:\n                    if mention[1] < total_length:\n                        new_cluster.append(mention)\n                if new_cluster:\n                    new_gold_clusters.append(new_cluster)\n\n            gold_clusters = new_gold_clusters\n\n    flattened_sentences = [_normalize_word(word) for sentence in sentences for word in sentence]\n    flat_sentences_tokens = [word for word in flattened_sentences]\n\n    text_field = flat_sentences_tokens\n\n    cluster_dict = {}\n    if gold_clusters is not None:\n        gold_clusters = _canonicalize_clusters(gold_clusters)\n        if remove_singleton_clusters:\n            gold_clusters = [cluster for cluster in gold_clusters if len(cluster) > 1]\n\n        for cluster_id, cluster in enumerate(gold_clusters):\n            for mention in cluster:\n                cluster_dict[tuple(mention)] = cluster_id\n\n    spans: List = []\n    span_labels: Optional[List[int]] = [] if gold_clusters is not None else None\n\n    sentence_offset = 0\n    for sentence in sentences:\n        for start, end in enumerate_spans(\n                sentence, offset=sentence_offset, max_span_width=max_span_width\n        ):\n\n            if span_labels is not None:\n                if (start, end) in cluster_dict:\n                    span_labels.append(cluster_dict[(start, end)])\n                else:\n                    span_labels.append(-1)\n\n            spans.append((start, end))\n        sentence_offset += len(sentence)\n\n    span_field = spans\n\n    # metadata: Dict[str, Any] = {\"original_text\": flattened_sentences}\n    # if gold_clusters is not None:\n    #     metadata[\"clusters\"] = gold_clusters\n    # metadata_field = MetadataField(metadata)\n\n    fields: Dict[str, List] = {\n        \"text\": text_field,\n        \"spans\": span_field,\n        'clusters': gold_clusters,\n        # \"metadata\": metadata_field,\n    }\n    if span_labels is not None:\n        fields[\"span_labels\"] = span_labels\n\n    return fields\n\n\ndef _normalize_word(word):\n    if word in (\"/.\", \"/?\"):\n        return word[1:]\n    else:\n        return word\n\n\ndef _canonicalize_clusters(clusters: List[List[Tuple[int, int]]]) -> List[List[Tuple[int, int]]]:\n    \"\"\"\n    The data might include 2 annotated spans which are identical,\n    but have different ids. This checks all clusters for spans which are\n    identical, and if it finds any, merges the clusters containing the\n    identical spans.\n    \"\"\"\n    merged_clusters: List[Set[Tuple[int, int]]] = []\n    for cluster in clusters:\n        cluster_with_overlapping_mention = None\n        for mention in cluster:\n            # Look at clusters we have already processed to\n            # see if they contain a mention in the current\n            # cluster for comparison.\n            for cluster2 in merged_clusters:\n                if mention in cluster2:\n                    # first cluster in merged clusters\n                    # which contains this mention.\n                    cluster_with_overlapping_mention = cluster2\n                    break\n            # Already encountered overlap - no need to keep looking.\n            if cluster_with_overlapping_mention is not None:\n                break\n        if cluster_with_overlapping_mention is not None:\n            # Merge cluster we are currently processing into\n            # the cluster in the processed list.\n            cluster_with_overlapping_mention.update(cluster)\n        else:\n            merged_clusters.append(set(cluster))\n    return [list(c) for c in merged_clusters]"
  },
  {
    "path": "hanlp/datasets/srl/ontonotes5/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-11-26 16:07\nONTONOTES5_HOME = 'https://catalog.ldc.upenn.edu/LDC2013T19/LDC2013T19.tgz#/ontonotes-release-5.0/data/'\nCONLL12_HOME = ONTONOTES5_HOME + '../conll-2012/'\n"
  },
  {
    "path": "hanlp/datasets/srl/ontonotes5/_utils.py",
    "content": "#!/usr/bin/env python\nimport codecs\nimport collections\nimport glob\nimport json\nimport os\nimport re\nimport sys\nfrom pprint import pprint\nfrom typing import List, Dict, Union\n\nfrom hanlp_common.io import eprint, save_json\n\nfrom hanlp.common.transform import NormalizeToken\nfrom hanlp.datasets.parsing.loaders._ctb_utils import remove_all_ec, convert_to_dependency\nfrom hanlp.datasets.parsing.ptb import PTB_TOKEN_MAPPING\nfrom hanlp.utils.io_util import merge_files, get_resource, pushd, run_cmd, read_tsv_as_sents, replace_ext, \\\n    get_exitcode_stdout_stderr\nfrom hanlp.utils.log_util import flash\n\nBEGIN_DOCUMENT_REGEX = re.compile(r\"#begin document \\((.*)\\); part (\\d+)\")\n\n\ndef flatten(l):\n    return [item for sublist in l for item in sublist]\n\n\ndef get_doc_key(doc_id, part):\n    return \"{}_{}\".format(doc_id, int(part))\n\n\nclass DocumentState(object):\n    def __init__(self):\n        self.doc_key = None\n        self.text = []\n        self.text_speakers = []\n        self.speakers = []\n        self.sentences = []\n        self.pos = []\n        self.lemma = []\n        self.pos_buffer = []\n        self.lemma_buffer = []\n        self.constituents = []  # {}\n        self.const_stack = []\n        self.const_buffer = []\n        self.ner = []\n        self.ner_stack = []\n        self.ner_buffer = []\n        self.srl = []\n        self.argument_stacks = []\n        self.argument_buffers = []\n        self.predicate_buffer = []\n        self.clusters = collections.defaultdict(list)\n        self.coref_stacks = collections.defaultdict(list)\n\n    def assert_empty(self):\n        assert self.doc_key is None\n        assert len(self.text) == 0\n        assert len(self.text_speakers) == 0\n        assert len(self.speakers) == 0\n        assert len(self.sentences) == 0\n        assert len(self.srl) == 0\n        assert len(self.predicate_buffer) == 0\n        assert len(self.argument_buffers) == 0\n        assert len(self.argument_stacks) == 0\n        assert len(self.constituents) == 0\n        assert len(self.const_stack) == 0\n        assert len(self.const_buffer) == 0\n        assert len(self.ner) == 0\n        assert len(self.lemma_buffer) == 0\n        assert len(self.pos_buffer) == 0\n        assert len(self.ner_stack) == 0\n        assert len(self.ner_buffer) == 0\n        assert len(self.coref_stacks) == 0\n        assert len(self.clusters) == 0\n\n    def assert_finalizable(self):\n        assert self.doc_key is not None\n        assert len(self.text) == 0\n        assert len(self.text_speakers) == 0\n        assert len(self.speakers) > 0\n        assert len(self.sentences) > 0\n        assert len(self.constituents) > 0\n        assert len(self.const_stack) == 0\n        assert len(self.ner_stack) == 0\n        assert len(self.predicate_buffer) == 0\n        assert all(len(s) == 0 for s in list(self.coref_stacks.values()))\n\n    def finalize_sentence(self):\n        self.sentences.append(tuple(self.text))\n        del self.text[:]\n        self.lemma.append(tuple(self.lemma_buffer))\n        del self.lemma_buffer[:]\n        self.pos.append(tuple(self.pos_buffer))\n        del self.pos_buffer[:]\n        self.speakers.append(tuple(self.text_speakers))\n        del self.text_speakers[:]\n\n        assert len(self.predicate_buffer) == len(self.argument_buffers)\n        self.srl.append([])\n        for pred, args in zip(self.predicate_buffer, self.argument_buffers):\n            for start, end, label in args:\n                self.srl[-1].append((pred, start, end, label))\n        self.predicate_buffer = []\n        self.argument_buffers = []\n        self.argument_stacks = []\n        self.constituents.append([c for c in self.const_buffer])\n        self.const_buffer = []\n        self.ner.append([c for c in self.ner_buffer])\n        self.ner_buffer = []\n\n    def finalize(self):\n        merged_clusters = []\n        for c1 in list(self.clusters.values()):\n            existing = None\n            for m in c1:\n                for c2 in merged_clusters:\n                    if m in c2:\n                        existing = c2\n                        break\n                if existing is not None:\n                    break\n            if existing is not None:\n                print(\"Merging clusters (shouldn't happen very often.)\")\n                existing.update(c1)\n            else:\n                merged_clusters.append(set(c1))\n        merged_clusters = [list(c) for c in merged_clusters]\n        all_mentions = flatten(merged_clusters)\n        assert len(all_mentions) == len(set(all_mentions))\n        assert len(self.sentences) == len(self.srl)\n        assert len(self.sentences) == len(self.constituents)\n        assert len(self.sentences) == len(self.ner)\n        return {\n            \"doc_key\": self.doc_key,\n            \"sentences\": self.sentences,\n            \"lemma\": self.lemma,\n            \"pos\": self.pos,\n            \"speakers\": self.speakers,\n            \"srl\": self.srl,\n            \"constituents\": self.constituents,\n            \"ner\": self.ner,\n            \"clusters\": merged_clusters\n        }\n\n\ndef filter_data(input_json_file, output_json_file, doc_ids_file=None, annotation=None):\n    \"\"\"Filter OntoNotes5 data based on CoNLL2012 (coref) doc ids.\n    https://github.com/bcmi220/unisrl/blob/master/scripts/filter_conll2012_data.py\n\n    Args:\n      input_json_file: All documents.\n      output_json_file:\n      doc_ids_file:\n\n    Returns:\n\n    \"\"\"\n    assert doc_ids_file or annotation\n    doc_count = 0\n    sentence_count = 0\n    srl_count = 0\n    ner_count = 0\n    cluster_count = 0\n    word_count = 0\n    missing_count = 0\n    doc_ids = []\n    doc_ids_to_keys = collections.defaultdict(list)\n    filtered_examples = {}\n    ontonotes_root = os.path.abspath(os.path.join(os.path.dirname(input_json_file), *['..'] * 2))\n    language = os.path.basename(input_json_file).split('.')[1]\n\n    if doc_ids_file:\n        with open(doc_ids_file, \"r\") as f:\n            for line in f:\n                doc_id = line.strip().split(\"annotations/\")[1]\n                doc_ids.append(doc_id)\n                doc_ids_to_keys[doc_id] = []\n            f.close()\n\n    with codecs.open(input_json_file, \"r\", \"utf8\") as f:\n        for jsonline in f:\n            example = json.loads(jsonline)\n            doc_key = example[\"doc_key\"]\n            dk_prefix = \"_\".join(doc_key.split(\"_\")[:-1])\n            if doc_ids_file and dk_prefix not in doc_ids_to_keys:\n                continue\n            if annotation and not os.path.isfile(\n                    os.path.join(ontonotes_root, 'data/files/data', language, 'annotations', dk_prefix) + annotation):\n                print(os.path.join(ontonotes_root, 'data/files/data', language, 'annotations', dk_prefix) + annotation)\n                missing_count += 1\n                continue\n            doc_ids_to_keys[dk_prefix].append(doc_key)\n            filtered_examples[doc_key] = example\n\n            sentences = example[\"sentences\"]\n            word_count += sum([len(s) for s in sentences])\n            sentence_count += len(sentences)\n            srl_count += sum([len(srl) for srl in example[\"srl\"]])\n            ner_count += sum([len(ner) for ner in example[\"ner\"]])\n            coref = example[\"clusters\"]\n            cluster_count += len(coref)\n            doc_count += 1\n        f.close()\n\n    print((\"Documents: {}\\nSentences: {}\\nWords: {}\\nNER: {}, PAS: {}, Clusters: {}, No annotations: {}\".format(\n        doc_count, sentence_count, word_count, ner_count, srl_count, cluster_count, missing_count)))\n\n    if doc_ids_file:\n        with codecs.open(output_json_file, \"w\", \"utf8\") as f:\n            for doc_id in doc_ids:  # Arrange the files in order of id files\n                for key in doc_ids_to_keys[doc_id]:\n                    f.write(json.dumps(filtered_examples[key], ensure_ascii=False))\n                    f.write(\"\\n\")\n            f.close()\n    else:\n        with codecs.open(output_json_file, \"w\", \"utf8\") as f:\n            for doc in filtered_examples.values():\n                f.write(json.dumps(doc, ensure_ascii=False))\n                f.write(\"\\n\")\n            f.close()\n\n\ndef normalize_word(word, language):\n    if language == \"arabic\":\n        word = word[:word.find(\"#\")]\n    if word == \"/.\" or word == \"/?\":\n        return word[1:]\n    else:\n        return word\n\n\ndef handle_bit(word_index, bit, stack, spans, label_set):\n    asterisk_idx = bit.find(\"*\")\n    if asterisk_idx >= 0:\n        open_parens = bit[:asterisk_idx]\n        close_parens = bit[asterisk_idx + 1:]\n    else:\n        open_parens = bit[:-1]\n        close_parens = bit[-1]\n\n    current_idx = open_parens.find(\"(\")\n    while current_idx >= 0:\n        next_idx = open_parens.find(\"(\", current_idx + 1)\n        if next_idx >= 0:\n            label = open_parens[current_idx + 1:next_idx]\n        else:\n            label = open_parens[current_idx + 1:]\n        label_set.add(label)\n        stack.append((word_index, label))\n        current_idx = next_idx\n\n    for c in close_parens:\n        try:\n            assert c == \")\"\n        except AssertionError:\n            print(word_index, bit, spans, stack)\n            continue\n        open_index, label = stack.pop()\n        spans.append((open_index, word_index, label))\n        ''' current_span = (open_index, word_index)\n        if current_span in spans:\n          spans[current_span] += \"_\" + label\n        else:\n          spans[current_span] = label\n        spans[current_span] = label '''\n\n\ndef handle_line(line, document_state: DocumentState, language, labels, stats):\n    begin_document_match = re.match(BEGIN_DOCUMENT_REGEX, line)\n    if begin_document_match:\n        document_state.assert_empty()\n        document_state.doc_key = get_doc_key(begin_document_match.group(1), begin_document_match.group(2))\n        return None\n    elif line.startswith(\"#end document\"):\n        document_state.assert_finalizable()\n        finalized_state = document_state.finalize()\n        stats[\"num_clusters\"] += len(finalized_state[\"clusters\"])\n        stats[\"num_mentions\"] += sum(len(c) for c in finalized_state[\"clusters\"])\n        # labels[\"{}_const_labels\".format(language)].update(l for _, _, l in finalized_state[\"constituents\"])\n        # labels[\"ner\"].update(l for _, _, l in finalized_state[\"ner\"])\n        return finalized_state\n    else:\n        row = line.split()\n        # Starting a new sentence.\n        if len(row) == 0:\n            stats[\"max_sent_len_{}\".format(language)] = max(len(document_state.text),\n                                                            stats[\"max_sent_len_{}\".format(language)])\n            stats[\"num_sents_{}\".format(language)] += 1\n            document_state.finalize_sentence()\n            return None\n        assert len(row) >= 12\n\n        doc_key = get_doc_key(row[0], row[1])\n        word = normalize_word(row[3], language)\n        pos = row[4]\n        parse = row[5]\n        lemma = row[6]\n        predicate_sense = row[7]\n        speaker = row[9]\n        ner = row[10]\n        args = row[11:-1]\n        coref = row[-1]\n\n        word_index = len(document_state.text) + sum(len(s) for s in document_state.sentences)\n        document_state.text.append(word)\n        document_state.text_speakers.append(speaker)\n        document_state.pos_buffer.append(pos)\n        document_state.lemma_buffer.append(lemma)\n\n        handle_bit(word_index, parse, document_state.const_stack, document_state.const_buffer, labels[\"categories\"])\n        handle_bit(word_index, ner, document_state.ner_stack, document_state.ner_buffer, labels[\"ner\"])\n\n        if len(document_state.argument_stacks) < len(args):\n            document_state.argument_stacks = [[] for _ in args]\n            document_state.argument_buffers = [[] for _ in args]\n\n        for i, arg in enumerate(args):\n            handle_bit(word_index, arg, document_state.argument_stacks[i], document_state.argument_buffers[i],\n                       labels[\"srl\"])\n        if predicate_sense != \"-\":\n            document_state.predicate_buffer.append(word_index)\n        if coref != \"-\":\n            for segment in coref.split(\"|\"):\n                if segment[0] == \"(\":\n                    if segment[-1] == \")\":\n                        cluster_id = int(segment[1:-1])\n                        document_state.clusters[cluster_id].append((word_index, word_index))\n                    else:\n                        cluster_id = int(segment[1:])\n                        document_state.coref_stacks[cluster_id].append(word_index)\n                else:\n                    cluster_id = int(segment[:-1])\n                    start = document_state.coref_stacks[cluster_id].pop()\n                    document_state.clusters[cluster_id].append((start, word_index))\n        return None\n\n\ndef ontonotes_document_generator(input_path, language, labels, stats):\n    with open(input_path, \"r\") as input_file:\n        document_state = DocumentState()\n        for line in input_file.readlines():\n            document = handle_line(line, document_state, language, labels, stats)\n            if document is not None:\n                yield document\n                document_state = DocumentState()\n\n\ndef convert_to_jsonlines(input_path, output_path, language, labels=None, stats=None):\n    if labels is None:\n        labels = collections.defaultdict(set)\n    if stats is None:\n        stats = collections.defaultdict(int)\n    count = 0\n    with open(output_path, \"w\") as output_file:\n        for document in ontonotes_document_generator(input_path, language, labels, stats):\n            output_file.write(json.dumps(document, ensure_ascii=False))\n            output_file.write(\"\\n\")\n            count += 1\n\n    return labels, stats\n\n\ndef make_ontonotes_jsonlines(conll12_ontonotes_path, output_path, languages=None):\n    if languages is None:\n        languages = ['english', 'chinese', 'arabic']\n    for language in languages:\n        make_ontonotes_language_jsonlines(conll12_ontonotes_path, output_path, language)\n\n\ndef make_ontonotes_language_jsonlines(conll12_ontonotes_path, output_path=None, language='english'):\n    conll12_ontonotes_path = get_resource(conll12_ontonotes_path)\n    if output_path is None:\n        output_path = os.path.dirname(conll12_ontonotes_path)\n    for split in ['train', 'development', 'test']:\n        pattern = f'{conll12_ontonotes_path}/data/{split}/data/{language}/annotations/*/*/*/*gold_conll'\n        files = sorted(glob.glob(pattern, recursive=True))\n        assert files, f'No gold_conll files found in {pattern}'\n        version = os.path.basename(files[0]).split('.')[-1].split('_')[0]\n        if version.startswith('v'):\n            assert all([version in os.path.basename(f) for f in files])\n        else:\n            version = 'v5'\n        lang_dir = f'{output_path}/{language}'\n        if split == 'conll-2012-test':\n            split = 'test'\n        full_file = f'{lang_dir}/{split}.{language}.{version}_gold_conll'\n        os.makedirs(lang_dir, exist_ok=True)\n        print(f'Merging {len(files)} files to {full_file}')\n        merge_files(files, full_file)\n        v5_json_file = full_file.replace(f'.{version}_gold_conll', f'.{version}.jsonlines')\n        print(f'Converting CoNLL file {full_file} to json file {v5_json_file}')\n        labels, stats = convert_to_jsonlines(full_file, v5_json_file, language)\n        print('Labels:')\n        pprint(labels)\n        print('Statistics:')\n        pprint(stats)\n        conll12_json_file = f'{lang_dir}/{split}.{language}.conll12.jsonlines'\n        print(f'Applying CoNLL 12 official splits on {v5_json_file} to {conll12_json_file}')\n        id_file = get_resource(f'https://file.hankcs.com/research/emnlp2021/conll.cemantix.org.zip#2012/download/ids/'\n                               f'{language}/coref/{split}.id')\n        filter_data(v5_json_file, conll12_json_file, id_file)\n\n\ndef ensure_python_points_to_python2():\n    exitcode, out, version = get_exitcode_stdout_stderr('python --version')\n    if not version:\n        version = out\n    if not version.startswith('Python 2'):\n        raise EnvironmentError(f'Your python command needs to be Python2, not {version.strip()}. Try:\\n\\n\\t'\n                               'ln -sf \"$(which python2)\" \"$(which python)\"')\n\n\ndef make_gold_conll(ontonotes_path, language):\n    ensure_python_points_to_python2()\n    ontonotes_path = os.path.abspath(get_resource(ontonotes_path))\n    to_conll = get_resource(\n        'https://gist.githubusercontent.com/hankcs/46b9137016c769e4b6137104daf43a92/raw/66369de6c24b5ec47696ae307591f0d72c6f3f02/ontonotes_to_conll.sh')\n    to_conll = os.path.abspath(to_conll)\n    # shutil.rmtree(os.path.join(ontonotes_path, 'conll-2012'), ignore_errors=True)\n    with pushd(ontonotes_path):\n        try:\n            flash(f'Converting [blue]{language}[/blue] to CoNLL format, '\n                  f'this might take half an hour [blink][yellow]...[/yellow][/blink]')\n            run_cmd(f'bash {to_conll} {ontonotes_path} {language}')\n            flash('')\n        except RuntimeError as e:\n            flash(f'[red]Failed[/red] to convert {language} of {ontonotes_path} to CoNLL. See exceptions for detail')\n            raise e\n\n\ndef convert_jsonlines_to_IOBES(json_file, output_file=None, doc_level_offset=True, normalize_token=False):\n    json_file = get_resource(json_file)\n    if not output_file:\n        output_file = os.path.splitext(json_file)[0] + '.ner.tsv'\n    if normalize_token:\n        transform = NormalizeToken(PTB_TOKEN_MAPPING, 'token')\n    with open(json_file) as src, open(output_file, 'w', encoding='utf-8') as out:\n        for line in src:\n            doc = json.loads(line)\n            offset = 0\n            for sent, ner in zip(doc['sentences'], doc['ner']):\n                if normalize_token:\n                    sent = transform({'token': sent})['token']\n                tags = ['O'] * len(sent)\n                for start, end, label in ner:\n                    if doc_level_offset:\n                        start -= offset\n                        end -= offset\n                    if start == end:\n                        tags[start] = 'S-' + label\n                    else:\n                        tags[start] = 'B-' + label\n                        for i in range(start + 1, end + 1):\n                            tags[i] = 'I-' + label\n                        tags[end] = 'E-' + label\n                offset += len(sent)\n                for token, tag in zip(sent, tags):\n                    out.write(f'{token}\\t{tag}\\n')\n                out.write('\\n')\n\n\ndef make_ner_tsv_if_necessary(json_file):\n    json_file = get_resource(json_file)\n    output_file = os.path.splitext(json_file)[0] + '.ner.tsv'\n    if not os.path.isfile(output_file):\n        convert_jsonlines_to_IOBES(json_file, output_file)\n    return output_file\n\n\ndef batch_make_ner_tsv_if_necessary(json_files):\n    for each in json_files:\n        make_ner_tsv_if_necessary(each)\n\n\ndef make_pos_tsv_if_necessary(json_file):\n    json_file = get_resource(json_file)\n    output_file = os.path.splitext(json_file)[0] + '.pos.tsv'\n    if not os.path.isfile(output_file):\n        make_pos_tsv(json_file, output_file)\n    return output_file\n\n\ndef make_pos_tsv(json_file, output_file):\n    with open(json_file) as src, open(output_file, 'w', encoding='utf-8') as out:\n        for line in src:\n            doc = json.loads(line)\n            for sent, pos in zip(doc['sentences'], doc['pos']):\n                for token, tag in zip(sent, pos):\n                    out.write(f'{token}\\t{tag}\\n')\n                out.write('\\n')\n\n\ndef batch_make_pos_tsv_if_necessary(json_files):\n    for each in json_files:\n        make_pos_tsv_if_necessary(each)\n\n\ndef make_con_txt(conll_file, output_file):\n    with open(output_file, 'w') as out:\n        for sent in read_tsv_as_sents(conll_file):\n            tree = []\n            pos_per_sent = []\n            for cell in sent:\n                if cell[0] == '#begin' or cell[0] == '#end':\n                    continue\n                if len(cell) < 8:\n                    print(cell)\n                filename, sentence_id, token_id, word, POS, parse, framefile, roleset, *_ = cell\n                parse = parse.replace('*', f'({POS} {word})')\n                tree.append(parse)\n                pos_per_sent.append(POS)\n            bracketed = ' '.join(tree)\n            out.write(bracketed)\n            out.write('\\n')\n\n\ndef make_con_txt_if_necessary(json_file):\n    json_file = get_resource(json_file)\n    output_file = os.path.splitext(json_file)[0] + '.con.txt'\n    if not os.path.isfile(output_file):\n        make_con_txt(json_file, output_file)\n    return output_file\n\n\ndef batch_make_con_txt_if_necessary(json_files):\n    for each in json_files:\n        make_con_txt_if_necessary(each)\n\n\ndef batch_remove_empty_category_if_necessary(json_files):\n    for each in json_files:\n        src = get_resource(each)\n        dst = replace_ext(src, '.noempty.txt')\n        if not os.path.isfile(dst):\n            remove_all_ec(src)\n\n\ndef make_dep_conllx(con_txt_file, output_file, language='en'):\n    con_txt_file = get_resource(con_txt_file)\n    convert_to_dependency(con_txt_file, output_file, language=language)\n\n\ndef make_dep_conllx_if_necessary(con_txt_file: str, language='en'):\n    con_txt_file = get_resource(con_txt_file)\n    output_file = con_txt_file.replace('.con.txt', '.dep.conllx', 1)\n    if os.path.isfile(output_file):\n        return\n    make_dep_conllx(con_txt_file, output_file, language)\n\n\ndef batch_make_dep_conllx_if_necessary(con_txt_files, language='en'):\n    for each in con_txt_files:\n        make_dep_conllx_if_necessary(each, language)\n\n\ndef make_ner_json_if_necessary(json_file):\n    json_file = get_resource(json_file)\n    output_file = os.path.splitext(json_file)[0] + '.ner.jsonlines'\n    if not os.path.isfile(output_file):\n        make_ner_json(json_file, output_file)\n    return output_file\n\n\ndef batch_make_ner_json_if_necessary(json_files):\n    for each in json_files:\n        make_ner_json_if_necessary(each)\n\n\ndef make_ner_json(json_file, output_file):\n    filter_data(json_file, output_file, doc_ids_file=None, annotation='.name')\n\n\ndef make_srl_json_if_necessary(json_file):\n    json_file = get_resource(json_file)\n    output_file = os.path.splitext(json_file)[0] + '.srl.jsonlines'\n    if not os.path.isfile(output_file):\n        make_srl_json(json_file, output_file)\n    return output_file\n\n\ndef make_coref_json_if_necessary(json_file):\n    json_file = get_resource(json_file)\n    output_file = os.path.splitext(json_file)[0] + '.coref.jsonlines'\n    if not os.path.isfile(output_file):\n        make_coref_json(json_file, output_file)\n    return output_file\n\n\ndef batch_make_srl_json_if_necessary(json_files):\n    for each in json_files:\n        make_srl_json_if_necessary(each)\n\n\ndef make_srl_json(json_file, output_file):\n    filter_data(json_file, output_file, doc_ids_file=None, annotation='.prop')\n\n\ndef batch_make_coref_json_if_necessary(json_files):\n    for each in json_files:\n        make_coref_json_if_necessary(each)\n\n\ndef make_coref_json(json_file, output_file):\n    filter_data(json_file, output_file, doc_ids_file=None, annotation='.coref')\n\n\ndef load_raw_text(onf_file) -> List[str]:\n    with open(onf_file) as src:\n        sents = []\n        expect_sent = False\n        expect_sent_line = False\n        sent_parts = []\n        for line in src:\n            line = line.strip()\n            if line == 'Plain sentence:':\n                expect_sent_line = True\n            elif expect_sent_line:\n                expect_sent_line = False\n                expect_sent = True\n                continue\n            elif expect_sent:\n                if not line:\n                    sents.append(' '.join(sent_parts))\n                    expect_sent = False\n                    sent_parts = []\n                else:\n                    sent_parts.append(line)\n\n        return sents\n\n\ndef batch_load_raw_text(root: str) -> Dict[str, List[str]]:\n    onf_files = sorted(glob.glob(os.path.join(root, '**/*.onf'), recursive=True))\n    sents = dict()\n    for path in onf_files:\n        filename = path.split('annotations/')[1][:-len('.onf')]\n        sents[filename] = load_raw_text(path)\n    return sents\n\n\ndef make_raw_text_if_necessary(home: str):\n    home = get_resource(home)\n    jsonpath = os.path.join(home, 'text.jsonlines')\n    if os.path.isfile(jsonpath):\n        return\n    sents = batch_load_raw_text(home)\n    save_json(sents, jsonpath)\n\n\nclass RestoreToken(NormalizeToken):\n    def __init__(self, src: str, mapper: Union[str, dict] = None, dst: str = None) -> None:\n        if not mapper:\n            mapper = {\n                '/-': '-',\n                '/.': '.',\n            }\n        super().__init__(mapper, src, dst)\n\n    def __call__(self, sample: dict) -> dict:\n        src = sample[self.src]\n        src = [[self.convert(y) for y in x] for x in src]\n        sample[self.dst] = src\n        return sample\n\n\ndef main():\n    if len(sys.argv) != 3:\n        eprint('2 arguments required: ontonotes_path output_path')\n        exit(1)\n    ontonotes_path = sys.argv[1]\n    output_path = sys.argv[2]\n    make_ontonotes_jsonlines(ontonotes_path, output_path)\n\n\nif __name__ == \"__main__\":\n    main()\n"
  },
  {
    "path": "hanlp/datasets/srl/ontonotes5/chinese.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-11-26 16:07\nimport os\nfrom urllib.error import HTTPError\nimport shutil\n\nfrom hanlp.datasets.srl.ontonotes5 import ONTONOTES5_HOME, CONLL12_HOME\nfrom hanlp.datasets.srl.ontonotes5._utils import make_gold_conll, make_ontonotes_language_jsonlines, \\\n    batch_make_ner_tsv_if_necessary, batch_make_pos_tsv_if_necessary, batch_make_con_txt_if_necessary, \\\n    batch_make_dep_conllx_if_necessary\nfrom hanlp.utils.io_util import get_resource, path_from_url\nfrom hanlp.utils.log_util import cprint, flash\n\n_ONTONOTES5_CHINESE_HOME = ONTONOTES5_HOME + 'files/data/chinese/'\n_ONTONOTES5_CONLL12_CHINESE_HOME = CONLL12_HOME + 'chinese/'\nONTONOTES5_CONLL12_CHINESE_TRAIN = _ONTONOTES5_CONLL12_CHINESE_HOME + 'train.chinese.conll12.jsonlines'\n'''Training set of OntoNotes5 used in CoNLL12 (:cite:`pradhan-etal-2012-conll`).'''\nONTONOTES5_CONLL12_CHINESE_DEV = _ONTONOTES5_CONLL12_CHINESE_HOME + 'development.chinese.conll12.jsonlines'\n'''Dev set of OntoNotes5 used in CoNLL12 (:cite:`pradhan-etal-2012-conll`).'''\nONTONOTES5_CONLL12_CHINESE_TEST = _ONTONOTES5_CONLL12_CHINESE_HOME + 'test.chinese.conll12.jsonlines'\n'''Test set of OntoNotes5 used in CoNLL12 (:cite:`pradhan-etal-2012-conll`).'''\n\nONTONOTES5_CONLL12_NER_CHINESE_TRAIN = _ONTONOTES5_CONLL12_CHINESE_HOME + 'train.chinese.conll12.ner.tsv'\n'''Training set of OntoNotes5 used in CoNLL12 (:cite:`pradhan-etal-2012-conll`).'''\nONTONOTES5_CONLL12_NER_CHINESE_DEV = _ONTONOTES5_CONLL12_CHINESE_HOME + 'development.chinese.conll12.ner.tsv'\n'''Dev set of OntoNotes5 used in CoNLL12 (:cite:`pradhan-etal-2012-conll`).'''\nONTONOTES5_CONLL12_NER_CHINESE_TEST = _ONTONOTES5_CONLL12_CHINESE_HOME + 'test.chinese.conll12.ner.tsv'\n'''Test set of OntoNotes5 used in CoNLL12 (:cite:`pradhan-etal-2012-conll`).'''\n\nONTONOTES5_CHINESE_TRAIN = _ONTONOTES5_CONLL12_CHINESE_HOME + 'train.chinese.v4.jsonlines'\nONTONOTES5_CHINESE_DEV = _ONTONOTES5_CONLL12_CHINESE_HOME + 'development.chinese.v4.jsonlines'\nONTONOTES5_CHINESE_TEST = _ONTONOTES5_CONLL12_CHINESE_HOME + 'test.chinese.v4.jsonlines'\n\nONTONOTES5_CONLL_CHINESE_TRAIN = _ONTONOTES5_CONLL12_CHINESE_HOME + 'train.chinese.v4_gold_conll'\nONTONOTES5_CONLL_CHINESE_DEV = _ONTONOTES5_CONLL12_CHINESE_HOME + 'development.chinese.v4_gold_conll'\nONTONOTES5_CONLL_CHINESE_TEST = _ONTONOTES5_CONLL12_CHINESE_HOME + 'test.chinese.v4_gold_conll'\n\nONTONOTES5_POS_CHINESE_TRAIN = _ONTONOTES5_CONLL12_CHINESE_HOME + 'train.chinese.v4.pos.tsv'\nONTONOTES5_POS_CHINESE_DEV = _ONTONOTES5_CONLL12_CHINESE_HOME + 'development.chinese.v4.pos.tsv'\nONTONOTES5_POS_CHINESE_TEST = _ONTONOTES5_CONLL12_CHINESE_HOME + 'test.chinese.v4.pos.tsv'\n\nONTONOTES5_CON_CHINESE_TRAIN = _ONTONOTES5_CONLL12_CHINESE_HOME + 'train.chinese.con.txt'\nONTONOTES5_CON_CHINESE_DEV = _ONTONOTES5_CONLL12_CHINESE_HOME + 'development.chinese.con.txt'\nONTONOTES5_CON_CHINESE_TEST = _ONTONOTES5_CONLL12_CHINESE_HOME + 'test.chinese.con.txt'\n\nONTONOTES5_DEP_CHINESE_TRAIN = _ONTONOTES5_CONLL12_CHINESE_HOME + 'train.chinese.dep.conllx'\nONTONOTES5_DEP_CHINESE_DEV = _ONTONOTES5_CONLL12_CHINESE_HOME + 'development.chinese.dep.conllx'\nONTONOTES5_DEP_CHINESE_TEST = _ONTONOTES5_CONLL12_CHINESE_HOME + 'test.chinese.dep.conllx'\n\n# ONTONOTES5_CON_CHINESE_NOEC_TRAIN = _ONTONOTES5_CONLL12_CHINESE_HOME + 'train.chinese.con.noempty.txt'\n# ONTONOTES5_CON_CHINESE_NOEC_DEV = _ONTONOTES5_CONLL12_CHINESE_HOME + 'development.chinese.con.noempty.txt'\n# ONTONOTES5_CON_CHINESE_NOEC_TEST = _ONTONOTES5_CONLL12_CHINESE_HOME + 'test.chinese.con.noempty.txt'\n\n\nONTONOTES5_NER_CHINESE_TRAIN = _ONTONOTES5_CONLL12_CHINESE_HOME + 'train.chinese.v4.ner.tsv'\nONTONOTES5_NER_CHINESE_DEV = _ONTONOTES5_CONLL12_CHINESE_HOME + 'development.chinese.v4.ner.tsv'\nONTONOTES5_NER_CHINESE_TEST = _ONTONOTES5_CONLL12_CHINESE_HOME + 'test.chinese.v4.ner.tsv'\n\ntry:\n    get_resource(ONTONOTES5_HOME, verbose=False)\nexcept HTTPError:\n    intended_file_path = path_from_url(ONTONOTES5_HOME)\n    cprint('Ontonotes 5.0 is a [red][bold]copyright[/bold][/red] dataset owned by LDC which we cannot re-distribute. '\n           f'Please apply for a licence from LDC (https://catalog.ldc.upenn.edu/LDC2016T13) '\n           f'then download it to {intended_file_path}')\n    cprint('Luckily, an [red]unofficial[/red] Chinese version is provided on GitHub '\n           'which will be used for demonstration purpose.')\n    unofficial_chinese = get_resource('https://github.com/GuocaiL/Coref_Resolution/archive/master.zip#data/')\n    intended_home, _ = os.path.splitext(intended_file_path)\n    intended_home = os.path.join(os.path.dirname(intended_home), 'ontonotes-release-5.0')\n    intended_chinese = f'{intended_home}/data/files/data/chinese/'\n    # print(os.path.dirname(intended_chinese))\n    # print(unofficial_chinese)\n    # print(intended_chinese)\n    for folder in ['annotations', 'metadata']:\n        flash(f'Copying {unofficial_chinese}{folder} to {intended_chinese}{folder} [blink][yellow]...[/yellow][/blink]')\n        shutil.copytree(f'{unofficial_chinese}{folder}', f'{intended_chinese}{folder}')\n    flash('')\n\ntry:\n    get_resource(ONTONOTES5_CONLL12_CHINESE_TRAIN, verbose=False)\nexcept HTTPError:\n    make_gold_conll(ONTONOTES5_HOME + '..', 'chinese')\n    make_ontonotes_language_jsonlines(CONLL12_HOME + 'v4', language='chinese')\n\nbatch_make_ner_tsv_if_necessary(\n    [ONTONOTES5_CONLL12_CHINESE_TRAIN, ONTONOTES5_CONLL12_CHINESE_DEV, ONTONOTES5_CONLL12_CHINESE_TEST])\n\nbatch_make_ner_tsv_if_necessary(\n    [ONTONOTES5_CONLL12_CHINESE_TRAIN, ONTONOTES5_CONLL12_CHINESE_DEV, ONTONOTES5_CONLL12_CHINESE_TEST])\n\nbatch_make_ner_tsv_if_necessary(\n    [ONTONOTES5_CHINESE_TRAIN, ONTONOTES5_CHINESE_DEV, ONTONOTES5_CHINESE_TEST])\n\nbatch_make_pos_tsv_if_necessary(\n    [ONTONOTES5_CHINESE_TRAIN, ONTONOTES5_CHINESE_DEV, ONTONOTES5_CHINESE_TEST])\n\nbatch_make_con_txt_if_necessary(\n    [ONTONOTES5_CONLL_CHINESE_TRAIN, ONTONOTES5_CONLL_CHINESE_DEV, ONTONOTES5_CONLL_CHINESE_TEST])\n\nbatch_make_dep_conllx_if_necessary(\n    [ONTONOTES5_CON_CHINESE_TRAIN, ONTONOTES5_CON_CHINESE_DEV, ONTONOTES5_CON_CHINESE_TEST], language='zh')\n\n# batch_remove_empty_category_if_necessary(\n#     [ONTONOTES5_CON_CHINESE_TRAIN, ONTONOTES5_CON_CHINESE_DEV, ONTONOTES5_CON_CHINESE_TEST])\n"
  },
  {
    "path": "hanlp/datasets/srl/ontonotes5/english.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-25 18:48\n\n\nfrom urllib.error import HTTPError\n\nfrom hanlp.datasets.srl.ontonotes5 import ONTONOTES5_HOME, CONLL12_HOME\nfrom hanlp.datasets.srl.ontonotes5._utils import make_gold_conll, make_ontonotes_language_jsonlines, \\\n    batch_make_ner_tsv_if_necessary, batch_make_pos_tsv_if_necessary, batch_make_con_txt_if_necessary, \\\n    batch_make_dep_conllx_if_necessary\nfrom hanlp.utils.io_util import get_resource, path_from_url\nfrom hanlp.utils.log_util import cprint\n\n_ONTONOTES5_ENGLISH_HOME = ONTONOTES5_HOME + 'files/data/english/'\n_ONTONOTES5_CONLL12_ENGLISH_HOME = CONLL12_HOME + 'english/'\n\nONTONOTES5_CONLL12_ENGLISH_TRAIN = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'train.english.conll12.jsonlines'\n'''Training set of English OntoNotes5 used in CoNLL12 (:cite:`pradhan-etal-2012-conll`).'''\nONTONOTES5_CONLL12_ENGLISH_DEV = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'development.english.conll12.jsonlines'\n'''Dev set of English OntoNotes5 used in CoNLL12 (:cite:`pradhan-etal-2012-conll`).'''\nONTONOTES5_CONLL12_ENGLISH_TEST = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'test.english.conll12.jsonlines'\n'''Test set of English OntoNotes5 used in CoNLL12 (:cite:`pradhan-etal-2012-conll`).'''\n\nONTONOTES5_ENGLISH_TRAIN = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'train.english.v4.jsonlines'\nONTONOTES5_ENGLISH_DEV = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'development.english.v4.jsonlines'\nONTONOTES5_ENGLISH_TEST = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'test.english.v4.jsonlines'\n\nONTONOTES5_CONLL_ENGLISH_TRAIN = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'train.english.v4_gold_conll'\nONTONOTES5_CONLL_ENGLISH_DEV = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'development.english.v4_gold_conll'\nONTONOTES5_CONLL_ENGLISH_TEST = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'test.english.v4_gold_conll'\n\nONTONOTES5_POS_ENGLISH_TRAIN = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'train.english.v4.pos.tsv'\nONTONOTES5_POS_ENGLISH_DEV = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'development.english.v4.pos.tsv'\nONTONOTES5_POS_ENGLISH_TEST = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'test.english.v4.pos.tsv'\n\nONTONOTES5_CON_ENGLISH_TRAIN = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'train.english.con.txt'\nONTONOTES5_CON_ENGLISH_DEV = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'development.english.con.txt'\nONTONOTES5_CON_ENGLISH_TEST = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'test.english.con.txt'\n\nONTONOTES5_DEP_ENGLISH_TRAIN = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'train.english.dep.conllx'\nONTONOTES5_DEP_ENGLISH_DEV = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'development.english.dep.conllx'\nONTONOTES5_DEP_ENGLISH_TEST = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'test.english.dep.conllx'\n\n# ONTONOTES5_CON_ENGLISH_NOEC_TRAIN = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'train.english.con.noempty.txt'\n# ONTONOTES5_CON_ENGLISH_NOEC_DEV = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'development.english.con.noempty.txt'\n# ONTONOTES5_CON_ENGLISH_NOEC_TEST = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'test.english.con.noempty.txt'\n\nONTONOTES5_CONLL12_NER_ENGLISH_TRAIN = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'train.english.conll12.ner.tsv'\n'''Training set of English OntoNotes5 used in CoNLL12 (:cite:`pradhan-etal-2012-conll`).'''\nONTONOTES5_CONLL12_NER_ENGLISH_DEV = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'development.english.conll12.ner.tsv'\n'''Dev set of English OntoNotes5 used in CoNLL12 (:cite:`pradhan-etal-2012-conll`).'''\nONTONOTES5_CONLL12_NER_ENGLISH_TEST = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'test.english.conll12.ner.tsv'\n'''Test set of English OntoNotes5 used in CoNLL12 (:cite:`pradhan-etal-2012-conll`).'''\n\nONTONOTES5_NER_ENGLISH_TRAIN = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'train.english.v4.ner.tsv'\nONTONOTES5_NER_ENGLISH_DEV = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'development.english.v4.ner.tsv'\nONTONOTES5_NER_ENGLISH_TEST = _ONTONOTES5_CONLL12_ENGLISH_HOME + 'test.english.v4.ner.tsv'\n\ntry:\n    get_resource(ONTONOTES5_HOME, verbose=False)\nexcept HTTPError:\n    intended_file_path = path_from_url(ONTONOTES5_HOME)\n    cprint('Ontonotes 5.0 is a [red][bold]copyright[/bold][/red] dataset owned by LDC which we cannot re-distribute. '\n           f'Please apply for a licence from LDC (https://catalog.ldc.upenn.edu/LDC2016T13) '\n           f'then download it to {intended_file_path}')\n    exit(1)\n\ntry:\n    get_resource(ONTONOTES5_CONLL12_ENGLISH_TRAIN, verbose=False)\nexcept HTTPError:\n    make_gold_conll(ONTONOTES5_HOME + '..', 'english')\n    make_ontonotes_language_jsonlines(CONLL12_HOME + 'v4', language='english')\n\nbatch_make_ner_tsv_if_necessary(\n    [ONTONOTES5_CONLL12_ENGLISH_TRAIN, ONTONOTES5_CONLL12_ENGLISH_DEV, ONTONOTES5_CONLL12_ENGLISH_TEST])\n\nbatch_make_ner_tsv_if_necessary(\n    [ONTONOTES5_ENGLISH_TRAIN, ONTONOTES5_ENGLISH_DEV, ONTONOTES5_ENGLISH_TEST])\n\nbatch_make_pos_tsv_if_necessary(\n    [ONTONOTES5_ENGLISH_TRAIN, ONTONOTES5_ENGLISH_DEV, ONTONOTES5_ENGLISH_TEST])\n\nbatch_make_con_txt_if_necessary(\n    [ONTONOTES5_CONLL_ENGLISH_TRAIN, ONTONOTES5_CONLL_ENGLISH_DEV, ONTONOTES5_CONLL_ENGLISH_TEST])\n\nbatch_make_dep_conllx_if_necessary(\n    [ONTONOTES5_CON_ENGLISH_TRAIN, ONTONOTES5_CON_ENGLISH_DEV, ONTONOTES5_CON_ENGLISH_TEST])\n\n# batch_remove_empty_category_if_necessary(\n#     [ONTONOTES5_CON_ENGLISH_TRAIN, ONTONOTES5_CON_ENGLISH_DEV, ONTONOTES5_CON_ENGLISH_TEST])\n"
  },
  {
    "path": "hanlp/datasets/sts/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-05-20 16:25\n"
  },
  {
    "path": "hanlp/datasets/sts/stsb.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-05-20 16:25\nfrom typing import Union, List, Callable\n\nfrom hanlp.common.dataset import TransformableDataset\nfrom hanlp.utils.io_util import read_cells\n\nSTS_B_TRAIN = 'http://ixa2.si.ehu.es/stswiki/images/4/48/Stsbenchmark.tar.gz#sts-train.csv'\nSTS_B_DEV = 'http://ixa2.si.ehu.es/stswiki/images/4/48/Stsbenchmark.tar.gz#sts-dev.csv'\nSTS_B_TEST = 'http://ixa2.si.ehu.es/stswiki/images/4/48/Stsbenchmark.tar.gz#sts-test.csv'\n\n\nclass SemanticTextualSimilarityDataset(TransformableDataset):\n    def __init__(self,\n                 data: Union[str, List],\n                 sent_a_col,\n                 sent_b_col,\n                 similarity_col,\n                 delimiter='auto',\n                 transform: Union[Callable, List] = None,\n                 cache=None,\n                 generate_idx=None) -> None:\n        self.delimiter = delimiter\n        self.similarity_col = similarity_col\n        self.sent_b_col = sent_b_col\n        self.sent_a_col = sent_a_col\n        super().__init__(data, transform, cache, generate_idx)\n\n    def load_file(self, filepath: str):\n        for i, cells in enumerate(read_cells(filepath, strip=True, delimiter=self.delimiter)):\n            yield {\n                'sent_a': cells[self.sent_a_col],\n                'sent_b': cells[self.sent_b_col],\n                'similarity': float(cells[self.similarity_col])\n            }\n"
  },
  {
    "path": "hanlp/datasets/tokenization/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-01 12:33"
  },
  {
    "path": "hanlp/datasets/tokenization/ctb6.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 22:19\n\n_CTB6_CWS_HOME = 'http://file.hankcs.com/corpus/ctb6_cws.zip'\n\nCTB6_CWS_TRAIN = _CTB6_CWS_HOME + '#train.txt'\n'''CTB6 training set.'''\nCTB6_CWS_DEV = _CTB6_CWS_HOME + '#dev.txt'\n'''CTB6 dev set.'''\nCTB6_CWS_TEST = _CTB6_CWS_HOME + '#test.txt'\n'''CTB6 test set.'''\n"
  },
  {
    "path": "hanlp/datasets/tokenization/loaders/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-12-28 19:06\n"
  },
  {
    "path": "hanlp/datasets/tokenization/loaders/chunking_dataset.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-03 18:50\nfrom typing import Union, List, Callable\n\nfrom hanlp.common.dataset import TransformableDataset\nfrom hanlp.utils.io_util import get_resource\nfrom hanlp.utils.span_util import bmes_of\nfrom hanlp.utils.string_util import ispunct\n\n\nclass ChunkingDataset(TransformableDataset):\n\n    def __init__(self, data: Union[str, List], transform: Union[Callable, List] = None, cache=None,\n                 generate_idx=None, max_seq_len=None, sent_delimiter=None) -> None:\n        if not sent_delimiter:\n            sent_delimiter = lambda x: ispunct(x)\n        elif isinstance(sent_delimiter, str):\n            sent_delimiter = set(list(sent_delimiter))\n            sent_delimiter = lambda x: x in sent_delimiter\n        self.sent_delimiter = sent_delimiter\n        self.max_seq_len = max_seq_len\n        super().__init__(data, transform, cache, generate_idx)\n\n    def load_file(self, filepath):\n        max_seq_len = self.max_seq_len\n        delimiter = self.sent_delimiter\n        for chars, tags in self._generate_chars_tags(filepath, delimiter, max_seq_len):\n            yield {'char': chars, 'tag': tags}\n\n    @staticmethod\n    def _generate_chars_tags(filepath, delimiter, max_seq_len):\n        filepath = get_resource(filepath)\n        with open(filepath, encoding='utf8') as src:\n            for text in src:\n                chars, tags = bmes_of(text, True)\n                if max_seq_len and delimiter and len(chars) > max_seq_len:\n                    short_chars, short_tags = [], []\n                    for idx, (char, tag) in enumerate(zip(chars, tags)):\n                        short_chars.append(char)\n                        short_tags.append(tag)\n                        if len(short_chars) >= max_seq_len and delimiter(char):\n                            yield short_chars, short_tags\n                            short_chars, short_tags = [], []\n                    if short_chars:\n                        yield short_chars, short_tags\n                else:\n                    yield chars, tags\n"
  },
  {
    "path": "hanlp/datasets/tokenization/loaders/multi_criteria_cws/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-11 20:35\n\n_HOME = 'https://github.com/hankcs/multi-criteria-cws/archive/naive-mix.zip#data/raw/'\n\nCNC_TRAIN_ALL = _HOME + 'cnc/train-all.txt'\nCNC_TRAIN = _HOME + 'cnc/train.txt'\nCNC_DEV = _HOME + 'cnc/dev.txt'\nCNC_TEST = _HOME + 'cnc/test.txt'\n\nCTB_TRAIN_ALL = _HOME + 'ctb/train-all.txt'\nCTB_TRAIN = _HOME + 'ctb/train.txt'\nCTB_DEV = _HOME + 'ctb/dev.txt'\nCTB_TEST = _HOME + 'ctb/test.txt'\n\nSXU_TRAIN_ALL = _HOME + 'sxu/train-all.txt'\nSXU_TRAIN = _HOME + 'sxu/train.txt'\nSXU_DEV = _HOME + 'sxu/dev.txt'\nSXU_TEST = _HOME + 'sxu/test.txt'\n\nUDC_TRAIN_ALL = _HOME + 'udc/train-all.txt'\nUDC_TRAIN = _HOME + 'udc/train.txt'\nUDC_DEV = _HOME + 'udc/dev.txt'\nUDC_TEST = _HOME + 'udc/test.txt'\n\nWTB_TRAIN_ALL = _HOME + 'wtb/train-all.txt'\nWTB_TRAIN = _HOME + 'wtb/train.txt'\nWTB_DEV = _HOME + 'wtb/dev.txt'\nWTB_TEST = _HOME + 'wtb/test.txt'\n\nZX_TRAIN_ALL = _HOME + 'zx/train-all.txt'\nZX_TRAIN = _HOME + 'zx/train.txt'\nZX_DEV = _HOME + 'zx/dev.txt'\nZX_TEST = _HOME + 'zx/test.txt'\n"
  },
  {
    "path": "hanlp/datasets/tokenization/loaders/multi_criteria_cws/mcws_dataset.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-10-21 19:11\nimport os\nfrom typing import Union, List, Callable, Dict, Iterable\n\nfrom hanlp.datasets.tokenization.loaders.txt import TextTokenizingDataset\nfrom hanlp.utils.io_util import get_resource\n\n\nclass MultiCriteriaTextTokenizingDataset(TextTokenizingDataset):\n    def __init__(self,\n                 data: Union[str, List],\n                 transform: Union[Callable, List] = None,\n                 cache=None,\n                 generate_idx=None,\n                 delimiter=None,\n                 max_seq_len=None,\n                 sent_delimiter=None,\n                 char_level=False,\n                 hard_constraint=False) -> None:\n        super().__init__(data, transform, cache, generate_idx, delimiter, max_seq_len, sent_delimiter, char_level,\n                         hard_constraint)\n\n    def should_load_file(self, data) -> bool:\n        return isinstance(data, (tuple, dict))\n\n    def load_file(self, filepath: Union[Iterable[str], Dict[str, str]]):\n        \"\"\"Load multi-criteria corpora specified in filepath.\n\n        Args:\n            filepath: A list of files where filename is its criterion. Or a dict of filename-criterion pairs.\n\n        .. highlight:: bash\n        .. code-block:: bash\n\n            $ tree -L 2 .\n            .\n            ├── cnc\n            │   ├── dev.txt\n            │   ├── test.txt\n            │   ├── train-all.txt\n            │   └── train.txt\n            ├── ctb\n            │   ├── dev.txt\n            │   ├── test.txt\n            │   ├── train-all.txt\n            │   └── train.txt\n            ├── sxu\n            │   ├── dev.txt\n            │   ├── test.txt\n            │   ├── train-all.txt\n            │   └── train.txt\n            ├── udc\n            │   ├── dev.txt\n            │   ├── test.txt\n            │   ├── train-all.txt\n            │   └── train.txt\n            ├── wtb\n            │   ├── dev.txt\n            │   ├── test.txt\n            │   ├── train-all.txt\n            │   └── train.txt\n            └── zx\n                ├── dev.txt\n                ├── test.txt\n                ├── train-all.txt\n                └── train.txt\n\n            $ head -n 2 ctb/dev.txt\n            上海 浦东 开发 与 法制 建设 同步\n            新华社 上海 二月 十日 电 （ 记者 谢金虎 、 张持坚 ）\n\n        \"\"\"\n        for eachpath in (filepath.items() if isinstance(filepath, dict) else filepath):\n            if isinstance(eachpath, tuple):\n                criteria, eachpath = eachpath\n                eachpath = get_resource(eachpath)\n            else:\n                eachpath = get_resource(eachpath)\n                criteria = os.path.basename(os.path.dirname(eachpath))\n            for sample in super().load_file(eachpath):\n                sample['criteria'] = criteria\n                yield sample\n\n\ndef append_criteria_token(sample: dict, criteria_tokens: Dict[str, int], criteria_token_map: dict) -> dict:\n    criteria = sample['criteria']\n    token = criteria_token_map.get(criteria, None)\n    if not token:\n        unused_tokens = list(criteria_tokens.keys())\n        size = len(criteria_token_map)\n        assert size + 1 < len(unused_tokens), f'No unused token available for criteria {criteria}. ' \\\n                                              f'Current criteria_token_map = {criteria_token_map}'\n        token = criteria_token_map[criteria] = unused_tokens[size]\n    sample['token_token_type_ids'] = [0] * len(sample['token_input_ids']) + [1]\n    sample['token_input_ids'] = sample['token_input_ids'] + [criteria_tokens[token]]\n    return sample\n"
  },
  {
    "path": "hanlp/datasets/tokenization/loaders/txt.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-01 12:35\nfrom typing import Union, List, Callable\n\nfrom hanlp.common.dataset import TransformableDataset\nfrom hanlp.utils.io_util import TimingFileIterator\nfrom hanlp.utils.span_util import words_to_bmes, words_to_bi\nfrom hanlp.utils.string_util import split_long_sentence_into\n\n\nclass TextTokenizingDataset(TransformableDataset):\n    def __init__(self,\n                 data: Union[str, List],\n                 transform: Union[Callable, List] = None,\n                 cache=None,\n                 generate_idx=None,\n                 delimiter=None,\n                 max_seq_len=None,\n                 sent_delimiter=None,\n                 char_level=False,\n                 hard_constraint=False,\n                 ) -> None:\n        \"\"\"A dataset for tagging tokenization tasks.\n\n        Args:\n            data: The local or remote path to a dataset, or a list of samples where each sample is a dict.\n            transform: Predefined transform(s).\n            cache: ``True`` to enable caching, so that transforms won't be called twice.\n            generate_idx: Create a :const:`~hanlp_common.constants.IDX` field for each sample to store its order in dataset. Useful for prediction when\n                samples are re-ordered by a sampler.\n            delimiter: Delimiter between tokens used to split a line in the corpus.\n            max_seq_len: Sentences longer than ``max_seq_len`` will be split into shorter ones if possible.\n            sent_delimiter: Delimiter between sentences, like period or comma, which indicates a long sentence can\n                be split here.\n            char_level: Whether the sequence length is measured at char level.\n            hard_constraint: Whether to enforce hard length constraint on sentences. If there is no ``sent_delimiter``\n                in a sentence, it will be split at a token anyway.\n        \"\"\"\n        self.hard_constraint = hard_constraint\n        self.char_level = char_level\n        self.sent_delimiter = sent_delimiter\n        self.max_seq_len = max_seq_len\n        self.delimiter = delimiter\n        super().__init__(data, transform, cache, generate_idx)\n\n    def load_file(self, filepath: str):\n        \"\"\"Load tokenized corpus. The format is one sentence per line, where each line consisits of tokens seperated\n        by a delimiter (usually space).\n\n        .. highlight:: bash\n        .. code-block:: bash\n\n            $ head train.txt\n            上海 浦东 开发 与 法制 建设 同步\n            新华社 上海 二月 十日 电 （ 记者 谢金虎 、 张持坚 ）\n\n        Args:\n            filepath: The path to the corpus.\n        \"\"\"\n        f = TimingFileIterator(filepath)\n        # longest_sent = 0\n        for line in f:\n            line = line.rstrip('\\n')\n            tokens = line.split(self.delimiter)\n            if not tokens:\n                continue\n            if self.max_seq_len and sum(len(t) for t in tokens) > self.max_seq_len:\n                # debug = []\n                for short_sents in split_long_sentence_into(tokens, self.max_seq_len, self.sent_delimiter,\n                                                            char_level=self.char_level,\n                                                            hard_constraint=self.hard_constraint):\n                    # debug.extend(short_sents)\n                    # longest_sent = max(longest_sent, len(''.join(short_sents)))\n                    yield {'token': short_sents}\n                # assert debug == tokens\n            else:\n                # longest_sent = max(longest_sent, len(''.join(tokens)))\n                yield {'token': tokens}\n            f.log(line[:20])\n        f.erase()\n        # print(f'Longest sent: {longest_sent} in {filepath}')\n\n\ndef generate_tags_for_subtokens(sample: dict, tagging_scheme='BMES'):\n    \"\"\"\n    Create a sequence of x for tokenization task. Each x is an atomic subtoken that will be tagged with BMES or BI tags.\n\n    Args:\n        sample: During prediction, it is a dict with 'token' being the input text, 'token_subtoken_offsets' being\n         incremental offsets per each subtoken. During training, it is a dict with 'token' being a sequence of tokens,\n         'token_subtoken_offsets' being non-incremental offsets per each subtoken, 'token_subtoken_offsets_group' being\n         subtoken offsets grouped by each token.\n        tagging_scheme:\n\n    Returns:\n\n    \"\"\"\n    # We could use token_token_span but we don't want token_token_span in the batch\n    subtokens_group = sample.get('token_subtoken_offsets_group', None)\n    sample['raw_token'] = sample['token']\n    tokens = sample.get('token_') or sample['token']\n\n    if subtokens_group:\n        sample['token'] = subtokens_group_to_subtokens(tokens, subtokens_group)\n        if tagging_scheme == 'BMES':\n            sample['tag'] = words_to_bmes(subtokens_group)\n        elif tagging_scheme == 'BI':\n            sample['tag'] = words_to_bi(subtokens_group)\n        else:\n            raise NotImplementedError(f'Unsupported tagging scheme {tagging_scheme}.')\n    else:\n        sample['token'] = subtoken_offsets_to_subtokens(tokens, sample['token_subtoken_offsets'])\n    return sample\n\n\ndef subtoken_offsets_to_subtokens(text, token_subtoken_offsets):\n    results = []\n    for b, e in token_subtoken_offsets:\n        results.append(text[b:e])\n    return results\n\n\ndef subtokens_group_to_subtokens(tokens, subtoken_offsets_group):\n    results = []\n    for subtoken_offsets, token in zip(subtoken_offsets_group, tokens):\n        for b, e in subtoken_offsets:\n            results.append(token[b:e])\n    return results\n"
  },
  {
    "path": "hanlp/datasets/tokenization/sighan2005/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-21 15:42\nimport os\n\nfrom hanlp.utils.io_util import get_resource, split_file\nfrom hanlp.utils.log_util import logger\n\nSIGHAN2005 = 'http://sighan.cs.uchicago.edu/bakeoff2005/data/icwb2-data.zip'\n\n\ndef make(train):\n    root = get_resource(SIGHAN2005)\n    train = os.path.join(root, train.split('#')[-1])\n    if not os.path.isfile(train):\n        full = train.replace('_90.txt', '.utf8')\n        logger.info(f'Splitting {full} into training set and valid set with 9:1 proportion')\n        valid = train.replace('90.txt', '10.txt')\n        split_file(full, train=0.9, dev=0.1, test=0, names={'train': train, 'dev': valid})\n        assert os.path.isfile(train), f'Failed to make {train}'\n        assert os.path.isfile(valid), f'Failed to make {valid}'\n        logger.info(f'Successfully made {train} {valid}')\n"
  },
  {
    "path": "hanlp/datasets/tokenization/sighan2005/as_.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-21 15:42\nfrom hanlp.datasets.tokenization.sighan2005 import SIGHAN2005, make\n\nSIGHAN2005_AS_DICT = SIGHAN2005 + \"#\" + \"gold/as_training_words.utf8\"\n'''Dictionary built on trainings set.'''\nSIGHAN2005_AS_TRAIN_ALL = SIGHAN2005 + \"#\" + \"training/as_training.utf8\"\n'''Full training set.'''\nSIGHAN2005_AS_TRAIN = SIGHAN2005 + \"#\" + \"training/as_training_90.txt\"\n'''Training set (first 90% of the full official training set).'''\nSIGHAN2005_AS_DEV = SIGHAN2005 + \"#\" + \"training/as_training_10.txt\"\n'''Dev set (last 10% of full official training set).'''\nSIGHAN2005_AS_TEST_INPUT = SIGHAN2005 + \"#\" + \"testing/as_testing.utf8\"\n'''Test input.'''\nSIGHAN2005_AS_TEST = SIGHAN2005 + \"#\" + \"gold/as_testing_gold.utf8\"\n'''Test set.'''\n\nmake(SIGHAN2005_AS_TRAIN)\n"
  },
  {
    "path": "hanlp/datasets/tokenization/sighan2005/cityu.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-21 15:42\nfrom hanlp.datasets.tokenization.sighan2005 import SIGHAN2005, make\n\nSIGHAN2005_CITYU_DICT = SIGHAN2005 + \"#\" + \"gold/cityu_training_words.utf8\"\n'''Dictionary built on trainings set.'''\nSIGHAN2005_CITYU_TRAIN_ALL = SIGHAN2005 + \"#\" + \"training/cityu_training.utf8\"\n'''Full training set.'''\nSIGHAN2005_CITYU_TRAIN = SIGHAN2005 + \"#\" + \"training/cityu_training_90.txt\"\n'''Training set (first 90% of the full official training set).'''\nSIGHAN2005_CITYU_DEV = SIGHAN2005 + \"#\" + \"training/cityu_training_10.txt\"\n'''Dev set (last 10% of full official training set).'''\nSIGHAN2005_CITYU_TEST_INPUT = SIGHAN2005 + \"#\" + \"testing/cityu_test.utf8\"\n'''Test input.'''\nSIGHAN2005_CITYU_TEST = SIGHAN2005 + \"#\" + \"gold/cityu_test_gold.utf8\"\n'''Test set.'''\n\nmake(SIGHAN2005_CITYU_TRAIN)\n"
  },
  {
    "path": "hanlp/datasets/tokenization/sighan2005/msr.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-21 15:42\nfrom hanlp.datasets.tokenization.sighan2005 import SIGHAN2005, make\n\nSIGHAN2005_MSR_DICT = SIGHAN2005 + \"#\" + \"gold/msr_training_words.utf8\"\n'''Dictionary built on trainings set.'''\nSIGHAN2005_MSR_TRAIN_ALL = SIGHAN2005 + \"#\" + \"training/msr_training.utf8\"\n'''Full training set.'''\nSIGHAN2005_MSR_TRAIN = SIGHAN2005 + \"#\" + \"training/msr_training_90.txt\"\n'''Training set (first 90% of the full official training set).'''\nSIGHAN2005_MSR_DEV = SIGHAN2005 + \"#\" + \"training/msr_training_10.txt\"\n'''Dev set (last 10% of full official training set).'''\nSIGHAN2005_MSR_TEST_INPUT = SIGHAN2005 + \"#\" + \"testing/msr_test.utf8\"\n'''Test input.'''\nSIGHAN2005_MSR_TEST = SIGHAN2005 + \"#\" + \"gold/msr_test_gold.utf8\"\n'''Test set.'''\n\nmake(SIGHAN2005_MSR_TRAIN)\n"
  },
  {
    "path": "hanlp/datasets/tokenization/sighan2005/pku.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-21 15:42\nfrom hanlp.datasets.tokenization.sighan2005 import SIGHAN2005, make\n\nSIGHAN2005_PKU_DICT = SIGHAN2005 + \"#\" + \"gold/pku_training_words.utf8\"\n'''Dictionary built on trainings set.'''\nSIGHAN2005_PKU_TRAIN_ALL = SIGHAN2005 + \"#\" + \"training/pku_training.utf8\"\n'''Full training set.'''\nSIGHAN2005_PKU_TRAIN = SIGHAN2005 + \"#\" + \"training/pku_training_90.txt\"\n'''Training set (first 90% of the full official training set).'''\nSIGHAN2005_PKU_DEV = SIGHAN2005 + \"#\" + \"training/pku_training_10.txt\"\n'''Dev set (last 10% of full official training set).'''\nSIGHAN2005_PKU_TEST_INPUT = SIGHAN2005 + \"#\" + \"testing/pku_test.utf8\"\n'''Test input.'''\nSIGHAN2005_PKU_TEST = SIGHAN2005 + \"#\" + \"gold/pku_test_gold.utf8\"\n'''Test set.'''\n\nmake(SIGHAN2005_PKU_TRAIN)\n"
  },
  {
    "path": "hanlp/layers/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-10-26 00:50"
  },
  {
    "path": "hanlp/layers/cnn_encoder.py",
    "content": "from typing import Optional, Tuple\n\nimport torch\nfrom torch.nn import Conv1d, Linear\n\n\nclass CnnEncoder(torch.nn.Module):\n    \"\"\"\n    A `CnnEncoder` is a combination of multiple convolution layers and max pooling layers.  As a\n    [`Seq2VecEncoder`](./seq2vec_encoder.md), the input to this module is of shape `(batch_size, num_tokens,\n    input_dim)`, and the output is of shape `(batch_size, output_dim)`.\n\n    The CNN has one convolution layer for each ngram filter size. Each convolution operation gives\n    out a vector of size num_filters. The number of times a convolution layer will be used\n    is `num_tokens - ngram_size + 1`. The corresponding maxpooling layer aggregates all these\n    outputs from the convolution layer and outputs the max.\n\n    This operation is repeated for every ngram size passed, and consequently the dimensionality of\n    the output after maxpooling is `len(ngram_filter_sizes) * num_filters`.  This then gets\n    (optionally) projected down to a lower dimensional output, specified by `output_dim`.\n\n    We then use a fully connected layer to project in back to the desired output_dim.  For more\n    details, refer to \"A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural\n    Networks for Sentence Classification\", Zhang and Wallace 2016, particularly Figure 1.\n\n    Registered as a `Seq2VecEncoder` with name \"cnn\".\n\n    # Parameters\n\n    embedding_dim : `int`, required\n        This is the input dimension to the encoder.  We need this because we can't do shape\n        inference in pytorch, and we need to know what size filters to construct in the CNN.\n    num_filters : `int`, required\n        This is the output dim for each convolutional layer, which is the number of \"filters\"\n        learned by that layer.\n    ngram_filter_sizes : `Tuple[int]`, optional (default=`(2, 3, 4, 5)`)\n        This specifies both the number of convolutional layers we will create and their sizes.  The\n        default of `(2, 3, 4, 5)` will have four convolutional layers, corresponding to encoding\n        ngrams of size 2 to 5 with some number of filters.\n    conv_layer_activation : `Activation`, optional (default=`torch.nn.ReLU`)\n        Activation to use after the convolution layers.\n    output_dim : `Optional[int]`, optional (default=`None`)\n        After doing convolutions and pooling, we'll project the collected features into a vector of\n        this size.  If this value is `None`, we will just return the result of the max pooling,\n        giving an output of shape `len(ngram_filter_sizes) * num_filters`.\n    \"\"\"\n\n    def __init__(\n        self,\n        embedding_dim: int,\n        num_filters: int,\n        ngram_filter_sizes: Tuple[int, ...] = (2, 3, 4, 5),\n        conv_layer_activation: str = 'ReLU',\n        output_dim: Optional[int] = None,\n    ) -> None:\n        super().__init__()\n        self._embedding_dim = embedding_dim\n        self._num_filters = num_filters\n        self._ngram_filter_sizes = ngram_filter_sizes\n        self._activation = getattr(torch.nn, conv_layer_activation)()\n        self._output_dim = output_dim\n\n        self._convolution_layers = [\n            Conv1d(\n                in_channels=self._embedding_dim,\n                out_channels=self._num_filters,\n                kernel_size=ngram_size,\n            )\n            for ngram_size in self._ngram_filter_sizes\n        ]\n        for i, conv_layer in enumerate(self._convolution_layers):\n            self.add_module(\"conv_layer_%d\" % i, conv_layer)\n\n        maxpool_output_dim = self._num_filters * len(self._ngram_filter_sizes)\n        if self._output_dim:\n            self.projection_layer = Linear(maxpool_output_dim, self._output_dim)\n        else:\n            self.projection_layer = None\n            self._output_dim = maxpool_output_dim\n\n    def get_input_dim(self) -> int:\n        return self._embedding_dim\n\n    def get_output_dim(self) -> int:\n        return self._output_dim\n\n    def forward(self, tokens: torch.Tensor, mask: torch.BoolTensor):\n        if mask is not None:\n            tokens = tokens * mask.unsqueeze(-1)\n\n        # Our input is expected to have shape `(batch_size, num_tokens, embedding_dim)`.  The\n        # convolution layers expect input of shape `(batch_size, in_channels, sequence_length)`,\n        # where the conv layer `in_channels` is our `embedding_dim`.  We thus need to transpose the\n        # tensor first.\n        tokens = torch.transpose(tokens, 1, 2)\n        # Each convolution layer returns output of size `(batch_size, num_filters, pool_length)`,\n        # where `pool_length = num_tokens - ngram_size + 1`.  We then do an activation function,\n        # then do max pooling over each filter for the whole input sequence.  Because our max\n        # pooling is simple, we just use `torch.max`.  The resultant tensor of has shape\n        # `(batch_size, num_conv_layers * num_filters)`, which then gets projected using the\n        # projection layer, if requested.\n\n        filter_outputs = []\n        for i in range(len(self._convolution_layers)):\n            convolution_layer = getattr(self, \"conv_layer_{}\".format(i))\n            filter_outputs.append(self._activation(convolution_layer(tokens)).max(dim=2)[0])\n\n        # Now we have a list of `num_conv_layers` tensors of shape `(batch_size, num_filters)`.\n        # Concatenating them gives us a tensor of shape `(batch_size, num_filters * num_conv_layers)`.\n        maxpool_output = (\n            torch.cat(filter_outputs, dim=1) if len(filter_outputs) > 1 else filter_outputs[0]\n        )\n\n        if self.projection_layer:\n            result = self.projection_layer(maxpool_output)\n        else:\n            result = maxpool_output\n        return result\n"
  },
  {
    "path": "hanlp/layers/crf/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-18 22:55"
  },
  {
    "path": "hanlp/layers/crf/crf.py",
    "content": "# Copied from https://github.com/kmkurn/pytorch-crf\n# Copyright 2017 Kemal Kurniawan <kemal@kkurniawan.com>\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy of\n# this software and associated documentation files (the \"Software\"), to deal in\n# the Software without restriction, including without limitation the rights to\n# use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies\n# of the Software, and to permit persons to whom the Software is furnished to do\n# so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,\n# INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A\n# PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT\n# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION\n# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE\n# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n__version__ = '0.7.2'\n\nfrom typing import List, Optional\n\nimport torch\nimport torch.nn as nn\n\n\nclass CRF(nn.Module):\n    \"\"\"Conditional random field.\n\n    This module implements a conditional random field [LMP01]_. The forward computation\n    of this class computes the log likelihood of the given sequence of tags and\n    emission score tensor. This class also has `~CRF.decode` method which finds\n    the best tag sequence given an emission score tensor using `Viterbi algorithm`_.\n\n    Args:\n        num_tags: Number of tags.\n        batch_first: Whether the first dimension corresponds to the size of a minibatch.\n\n    Attributes:\n        start_transitions (`~torch.nn.Parameter`): Start transition score tensor of size\n            ``(num_tags,)``.\n        end_transitions (`~torch.nn.Parameter`): End transition score tensor of size\n            ``(num_tags,)``.\n        transitions (`~torch.nn.Parameter`): Transition score tensor of size\n            ``(num_tags, num_tags)``.\n\n\n    .. [LMP01] Lafferty, J., McCallum, A., Pereira, F. (2001).\n       \"Conditional random fields: Probabilistic models for segmenting and\n       labeling sequence data\". *Proc. 18th International Conf. on Machine\n       Learning*. Morgan Kaufmann. pp. 282–289.\n\n    .. _Viterbi algorithm: https://en.wikipedia.org/wiki/Viterbi_algorithm\n    \"\"\"\n\n    def __init__(self, num_tags: int, batch_first: bool = True) -> None:\n        if num_tags <= 0:\n            raise ValueError(f'invalid number of tags: {num_tags}')\n        super().__init__()\n        self.num_tags = num_tags\n        self.batch_first = batch_first\n        self.start_transitions = nn.Parameter(torch.empty(num_tags))\n        self.end_transitions = nn.Parameter(torch.empty(num_tags))\n        self.transitions = nn.Parameter(torch.empty(num_tags, num_tags))\n\n        self.reset_parameters()\n\n    def reset_parameters(self) -> None:\n        \"\"\"Initialize the transition parameters.\n\n        The parameters will be initialized randomly from a uniform distribution\n        between -0.1 and 0.1.\n        \"\"\"\n        nn.init.uniform_(self.start_transitions, -0.1, 0.1)\n        nn.init.uniform_(self.end_transitions, -0.1, 0.1)\n        nn.init.uniform_(self.transitions, -0.1, 0.1)\n\n    def __repr__(self) -> str:\n        return f'{self.__class__.__name__}(num_tags={self.num_tags})'\n\n    def forward(\n            self,\n            emissions: torch.Tensor,\n            tags: torch.LongTensor,\n            mask: Optional[torch.ByteTensor] = None,\n            reduction: str = 'sum',\n    ) -> torch.Tensor:\n        \"\"\"Compute the conditional log likelihood of a sequence of tags given emission scores.\n\n        Args:\n            emissions (`~torch.Tensor`): Emission score tensor of size\n                ``(seq_length, batch_size, num_tags)`` if ``batch_first`` is ``False``,\n                ``(batch_size, seq_length, num_tags)`` otherwise.\n            tags (`~torch.LongTensor`): Sequence of tags tensor of size\n                ``(seq_length, batch_size)`` if ``batch_first`` is ``False``,\n                ``(batch_size, seq_length)`` otherwise.\n            mask (`~torch.ByteTensor`): Mask tensor of size ``(seq_length, batch_size)``\n                if ``batch_first`` is ``False``, ``(batch_size, seq_length)`` otherwise.\n            reduction: Specifies  the reduction to apply to the output:\n                ``none|sum|mean|token_mean``. ``none``: no reduction will be applied.\n                ``sum``: the output will be summed over batches. ``mean``: the output will be\n                averaged over batches. ``token_mean``: the output will be averaged over tokens.\n\n        Returns:\n            `~torch.Tensor`: The log likelihood. This will have size ``(batch_size,)`` if\n            reduction is ``none``, ``()`` otherwise.\n        \"\"\"\n        self._validate(emissions, tags=tags, mask=mask)\n        if reduction not in ('none', 'sum', 'mean', 'token_mean'):\n            raise ValueError(f'invalid reduction: {reduction}')\n        if mask is None:\n            mask = torch.ones_like(tags, dtype=torch.uint8)\n\n        if self.batch_first:\n            emissions = emissions.transpose(0, 1)\n            tags = tags.transpose(0, 1)\n            mask = mask.transpose(0, 1)\n\n        # shape: (batch_size,)\n        numerator = self._compute_score(emissions, tags, mask)\n        # shape: (batch_size,)\n        denominator = self._compute_normalizer(emissions, mask)\n        # shape: (batch_size,)\n        llh = numerator - denominator\n\n        if reduction == 'none':\n            return llh\n        if reduction == 'sum':\n            return llh.sum()\n        if reduction == 'mean':\n            return llh.mean()\n        assert reduction == 'token_mean'\n        return llh.sum() / mask.type_as(emissions).sum()\n\n    def decode(self, emissions: torch.Tensor,\n               mask: Optional[torch.ByteTensor] = None) -> List[List[int]]:\n        \"\"\"Find the most likely tag sequence using Viterbi algorithm.\n\n        Args:\n            emissions (`~torch.Tensor`): Emission score tensor of size\n                ``(seq_length, batch_size, num_tags)`` if ``batch_first`` is ``False``,\n                ``(batch_size, seq_length, num_tags)`` otherwise.\n            mask (`~torch.ByteTensor`): Mask tensor of size ``(seq_length, batch_size)``\n                if ``batch_first`` is ``False``, ``(batch_size, seq_length)`` otherwise.\n\n        Returns:\n            List of list containing the best tag sequence for each batch.\n        \"\"\"\n        self._validate(emissions, mask=mask)\n        if mask is None:\n            mask = emissions.new_ones(emissions.shape[:2], dtype=torch.uint8)\n\n        if self.batch_first:\n            emissions = emissions.transpose(0, 1)\n            mask = mask.transpose(0, 1)\n\n        return self._viterbi_decode(emissions, mask)\n\n    def _validate(\n            self,\n            emissions: torch.Tensor,\n            tags: Optional[torch.LongTensor] = None,\n            mask: Optional[torch.ByteTensor] = None) -> None:\n        if emissions.dim() != 3:\n            raise ValueError(f'emissions must have dimension of 3, got {emissions.dim()}')\n        if emissions.size(2) != self.num_tags:\n            raise ValueError(\n                f'expected last dimension of emissions is {self.num_tags}, '\n                f'got {emissions.size(2)}')\n\n        if tags is not None:\n            if emissions.shape[:2] != tags.shape:\n                raise ValueError(\n                    'the first two dimensions of emissions and tags must match, '\n                    f'got {tuple(emissions.shape[:2])} and {tuple(tags.shape)}')\n\n        if mask is not None:\n            if emissions.shape[:2] != mask.shape:\n                raise ValueError(\n                    'the first two dimensions of emissions and mask must match, '\n                    f'got {tuple(emissions.shape[:2])} and {tuple(mask.shape)}')\n            no_empty_seq = not self.batch_first and mask[0].all()\n            no_empty_seq_bf = self.batch_first and mask[:, 0].all()\n            if not no_empty_seq and not no_empty_seq_bf:\n                raise ValueError('mask of the first timestep must all be on')\n\n    def _compute_score(\n            self, emissions: torch.Tensor, tags: torch.LongTensor,\n            mask: torch.ByteTensor) -> torch.Tensor:\n        # emissions: (seq_length, batch_size, num_tags)\n        # tags: (seq_length, batch_size)\n        # mask: (seq_length, batch_size)\n        assert emissions.dim() == 3 and tags.dim() == 2\n        assert emissions.shape[:2] == tags.shape\n        assert emissions.size(2) == self.num_tags\n        assert mask.shape == tags.shape\n        assert mask[0].all()\n\n        seq_length, batch_size = tags.shape\n        mask = mask.type_as(emissions)\n\n        # Start transition score and first emission\n        # shape: (batch_size,)\n        score = self.start_transitions[tags[0]]\n        score += emissions[0, torch.arange(batch_size), tags[0]]\n\n        for i in range(1, seq_length):\n            # Transition score to next tag, only added if next timestep is valid (mask == 1)\n            # shape: (batch_size,)\n            score += self.transitions[tags[i - 1], tags[i]] * mask[i]\n\n            # Emission score for next tag, only added if next timestep is valid (mask == 1)\n            # shape: (batch_size,)\n            score += emissions[i, torch.arange(batch_size), tags[i]] * mask[i]\n\n        # End transition score\n        # shape: (batch_size,)\n        seq_ends = mask.long().sum(dim=0) - 1\n        # shape: (batch_size,)\n        last_tags = tags[seq_ends, torch.arange(batch_size)]\n        # shape: (batch_size,)\n        score += self.end_transitions[last_tags]\n\n        return score\n\n    def _compute_normalizer(\n            self, emissions: torch.Tensor, mask: torch.ByteTensor) -> torch.Tensor:\n        # emissions: (seq_length, batch_size, num_tags)\n        # mask: (seq_length, batch_size)\n        assert emissions.dim() == 3 and mask.dim() == 2\n        assert emissions.shape[:2] == mask.shape\n        assert emissions.size(2) == self.num_tags\n        assert mask[0].all()\n\n        seq_length = emissions.size(0)\n\n        # Start transition score and first emission; score has size of\n        # (batch_size, num_tags) where for each batch, the j-th column stores\n        # the score that the first timestep has tag j\n        # shape: (batch_size, num_tags)\n        score = self.start_transitions + emissions[0]\n\n        for i in range(1, seq_length):\n            # Broadcast score for every possible next tag\n            # shape: (batch_size, num_tags, 1)\n            broadcast_score = score.unsqueeze(2)\n\n            # Broadcast emission score for every possible current tag\n            # shape: (batch_size, 1, num_tags)\n            broadcast_emissions = emissions[i].unsqueeze(1)\n\n            # Compute the score tensor of size (batch_size, num_tags, num_tags) where\n            # for each sample, entry at row i and column j stores the sum of scores of all\n            # possible tag sequences so far that end with transitioning from tag i to tag j\n            # and emitting\n            # shape: (batch_size, num_tags, num_tags)\n            next_score = broadcast_score + self.transitions + broadcast_emissions\n\n            # Sum over all possible current tags, but we're in score space, so a sum\n            # becomes a log-sum-exp: for each sample, entry i stores the sum of scores of\n            # all possible tag sequences so far, that end in tag i\n            # shape: (batch_size, num_tags)\n            next_score = torch.logsumexp(next_score, dim=1)\n\n            # Set score to the next score if this timestep is valid (mask == 1)\n            # shape: (batch_size, num_tags)\n            score = torch.where(mask[i].unsqueeze(1), next_score, score)\n\n        # End transition score\n        # shape: (batch_size, num_tags)\n        score += self.end_transitions\n\n        # Sum (log-sum-exp) over all possible tags\n        # shape: (batch_size,)\n        return torch.logsumexp(score, dim=1)\n\n    def _viterbi_decode(self, emissions: torch.FloatTensor,\n                        mask: torch.ByteTensor) -> List[List[int]]:\n        # emissions: (seq_length, batch_size, num_tags)\n        # mask: (seq_length, batch_size)\n        assert emissions.dim() == 3 and mask.dim() == 2\n        assert emissions.shape[:2] == mask.shape\n        assert emissions.size(2) == self.num_tags\n        assert mask[0].all()\n\n        seq_length, batch_size = mask.shape\n\n        # Start transition and first emission\n        # shape: (batch_size, num_tags)\n        score = self.start_transitions + emissions[0]\n        history = []\n\n        # score is a tensor of size (batch_size, num_tags) where for every batch,\n        # value at column j stores the score of the best tag sequence so far that ends\n        # with tag j\n        # history saves where the best tags candidate transitioned from; this is used\n        # when we trace back the best tag sequence\n\n        # Viterbi algorithm recursive case: we compute the score of the best tag sequence\n        # for every possible next tag\n        for i in range(1, seq_length):\n            # Broadcast viterbi score for every possible next tag\n            # shape: (batch_size, num_tags, 1)\n            broadcast_score = score.unsqueeze(2)\n\n            # Broadcast emission score for every possible current tag\n            # shape: (batch_size, 1, num_tags)\n            broadcast_emission = emissions[i].unsqueeze(1)\n\n            # Compute the score tensor of size (batch_size, num_tags, num_tags) where\n            # for each sample, entry at row i and column j stores the score of the best\n            # tag sequence so far that ends with transitioning from tag i to tag j and emitting\n            # shape: (batch_size, num_tags, num_tags)\n            next_score = broadcast_score + self.transitions + broadcast_emission\n\n            # Find the maximum score over all possible current tag\n            # shape: (batch_size, num_tags)\n            next_score, indices = next_score.max(dim=1)\n\n            # Set score to the next score if this timestep is valid (mask == 1)\n            # and save the index that produces the next score\n            # shape: (batch_size, num_tags)\n            score = torch.where(mask[i].unsqueeze(1), next_score, score)\n            history.append(indices)\n\n        # End transition score\n        # shape: (batch_size, num_tags)\n        score += self.end_transitions\n\n        # Now, compute the best path for each sample\n\n        # shape: (batch_size,)\n        seq_ends = mask.long().sum(dim=0) - 1\n        best_tags_list = []\n\n        for idx in range(batch_size):\n            # Find the tag which maximizes the score at the last timestep; this is our best tag\n            # for the last timestep\n            _, best_last_tag = score[idx].max(dim=0)\n            best_tags = [best_last_tag.item()]\n\n            # We trace back where the best last tag comes from, append that to our best tag\n            # sequence, and trace it back again, and so on\n            for hist in reversed(history[:seq_ends[idx]]):\n                best_last_tag = hist[idx][best_tags[-1]]\n                best_tags.append(best_last_tag.item())\n\n            # Reverse the order because we start from the last timestep\n            best_tags.reverse()\n            best_tags_list.append(best_tags)\n\n        return best_tags_list\n"
  },
  {
    "path": "hanlp/layers/crf/crf_layer_tf.py",
    "content": "# ******************************************************************************\n# Copyright 2017-2018 Intel Corporation\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ******************************************************************************\nimport tensorflow as tf\n\nfrom hanlp.layers.crf.crf_tf import crf_decode, crf_log_likelihood\n\n\nclass CRF(tf.keras.layers.Layer):\n    \"\"\"Conditional Random Field layer (tf.keras)\n    `CRF` can be used as the last layer in a network (as a classifier). Input shape (features)\n    must be equal to the number of classes the CRF can predict (a linear layer is recommended).\n    \n    Note: the loss and accuracy functions of networks using `CRF` must\n    use the provided loss and accuracy functions (denoted as loss and viterbi_accuracy)\n    as the classification of sequences are used with the layers internal weights.\n    \n    Copyright: this is a modified version of\n    https://github.com/NervanaSystems/nlp-architect/blob/master/nlp_architect/nn/tensorflow/python/keras/layers/crf.py\n\n    Args:\n      num_labels(int): the number of labels to tag each temporal input.\n    Input shape:\n      num_labels(int): the number of labels to tag each temporal input.\n    Input shape:\n    nD tensor with shape `(batch_size, sentence length, num_classes)`.\n    Output shape:\n      nD tensor with shape: `(batch_size, sentence length, num_classes)`.\n\n    Returns:\n\n    \"\"\"\n\n    def __init__(self, num_classes, **kwargs):\n        self.transitions = None\n        super(CRF, self).__init__(**kwargs)\n        # num of output labels\n        self.output_dim = int(num_classes)\n        self.input_spec = tf.keras.layers.InputSpec(min_ndim=3)\n        self.supports_masking = False\n        sequence_lengths = None\n\n    def get_config(self):\n        config = {\n            'output_dim': self.output_dim,\n            'supports_masking': self.supports_masking,\n            'transitions': tf.keras.backend.eval(self.transitions)\n        }\n        base_config = super(CRF, self).get_config()\n        return dict(list(base_config.items()) + list(config.items()))\n\n    def build(self, input_shape):\n        assert len(input_shape) == 3\n        f_shape = tf.TensorShape(input_shape)\n        input_spec = tf.keras.layers.InputSpec(min_ndim=3, axes={-1: f_shape[-1]})\n\n        if f_shape[-1] is None:\n            raise ValueError('The last dimension of the inputs to `CRF` '\n                             'should be defined. Found `None`.')\n        if f_shape[-1] != self.output_dim:\n            raise ValueError('The last dimension of the input shape must be equal to output'\n                             ' shape. Use a linear layer if needed.')\n        self.input_spec = input_spec\n        self.transitions = self.add_weight(name='transitions',\n                                           shape=[self.output_dim, self.output_dim],\n                                           initializer='glorot_uniform',\n                                           trainable=True)\n        self.built = True\n\n    def compute_mask(self, inputs, mask=None):\n        # Just pass the received mask from previous layer, to the next layer or\n        # manipulate it if this layer changes the shape of the input\n        return mask\n\n    # pylint: disable=arguments-differ\n    def call(self, inputs, sequence_lengths=None, mask=None, training=None, **kwargs):\n        sequences = tf.convert_to_tensor(inputs, dtype=self.dtype)\n        if sequence_lengths is not None:\n            assert len(sequence_lengths.shape) == 2\n            assert tf.convert_to_tensor(sequence_lengths).dtype == 'int32'\n            seq_len_shape = tf.convert_to_tensor(sequence_lengths).get_shape().as_list()\n            assert seq_len_shape[1] == 1\n            sequence_lengths = tf.keras.backend.flatten(sequence_lengths)\n        else:\n            sequence_lengths = tf.math.count_nonzero(mask, axis=1)\n\n        viterbi_sequence, _ = crf_decode(sequences, self.transitions,\n                                         sequence_lengths)\n        output = tf.keras.backend.one_hot(viterbi_sequence, self.output_dim)\n        return tf.keras.backend.in_train_phase(sequences, output)\n\n    # def loss(self, y_true, y_pred):\n    #     y_pred = tf.convert_to_tensor(y_pred, dtype=self.dtype)\n    #     log_likelihood, self.transitions = \\\n    #         crf_log_likelihood(y_pred,\n    #                            tf.cast(y_true, dtype=tf.int32),\n    #                            sequence_lengths,\n    #                            transition_params=self.transitions)\n    #     return tf.reduce_mean(-log_likelihood)\n\n    def compute_output_shape(self, input_shape):\n        tf.TensorShape(input_shape).assert_has_rank(3)\n        return input_shape[:2] + (self.output_dim,)\n\n    @property\n    def viterbi_accuracy(self):\n        def accuracy(y_true, y_pred):\n            shape = tf.shape(y_pred)\n            sequence_lengths = tf.ones(shape[0], dtype=tf.int32) * (shape[1])\n            viterbi_sequence, _ = crf_decode(y_pred, self.transitions,\n                                             sequence_lengths)\n            output = tf.keras.backend.one_hot(viterbi_sequence, self.output_dim)\n            return tf.keras.metrics.categorical_accuracy(y_true, output)\n\n        accuracy.func_name = 'viterbi_accuracy'\n        return accuracy\n\n\nclass CRFLoss(object):\n\n    def __init__(self, crf: CRF, dtype) -> None:\n        super().__init__()\n        self.crf = crf\n        self.dtype = dtype\n        self.__name__ = type(self).__name__\n\n    def __call__(self, y_true, y_pred, sample_weight=None, **kwargs):\n        assert sample_weight is not None, 'your model has to support masking'\n        if len(y_true.shape) == 3:\n            y_true = tf.argmax(y_true, axis=-1)\n        sequence_lengths = tf.math.count_nonzero(sample_weight, axis=1)\n        y_pred = tf.convert_to_tensor(y_pred, dtype=self.dtype)\n        log_likelihood, self.crf.transitions = \\\n            crf_log_likelihood(y_pred,\n                               tf.cast(y_true, dtype=tf.int32),\n                               sequence_lengths,\n                               transition_params=self.crf.transitions)\n        return tf.reduce_mean(-log_likelihood)\n\n\nclass CRFWrapper(tf.keras.Model):\n    def __init__(self, model: tf.keras.Model, num_classes=None, *args, **kwargs):\n        super().__init__(*args, **kwargs)\n        self.model = model\n        self.crf = CRF(model.output.shape[-1] if not num_classes else num_classes)\n\n    def call(self, inputs, training=None, mask=None):\n        output = self.model(inputs, training=training, mask=mask)\n        viterbi_output = self.crf(output)\n        return viterbi_output\n\n    def compute_output_shape(self, input_shape):\n        return self.model.compute_output_shape(input_shape)\n"
  },
  {
    "path": "hanlp/layers/crf/crf_tf.py",
    "content": "# Copyright 2019 The TensorFlow Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport numpy as np\nimport tensorflow as tf\n\n# TODO: Wrap functions in @tf.function once\n# https://github.com/tensorflow/tensorflow/issues/29075 is resolved\n\n\ndef crf_sequence_score(inputs, tag_indices, sequence_lengths,\n                       transition_params):\n    \"\"\"Computes the unnormalized score for a tag sequence.\n\n    Args:\n      inputs: A [batch_size, max_seq_len, num_tags] tensor of unary potentials\n    to use as input to the CRF layer.\n      tag_indices: A [batch_size, max_seq_len] matrix of tag indices for which\n    we compute the unnormalized score.\n      sequence_lengths: A [batch_size] vector of true sequence lengths.\n      transition_params: \n\n    Returns:\n      sequence_scores: A [batch_size] vector of unnormalized sequence scores.\n\n    \"\"\"\n    tag_indices = tf.cast(tag_indices, dtype=tf.int32)\n    sequence_lengths = tf.cast(sequence_lengths, dtype=tf.int32)\n\n    # If max_seq_len is 1, we skip the score calculation and simply gather the\n    # unary potentials of the single tag.\n    def _single_seq_fn():\n        batch_size = tf.shape(inputs, out_type=tag_indices.dtype)[0]\n\n        example_inds = tf.reshape(\n            tf.range(batch_size, dtype=tag_indices.dtype), [-1, 1])\n        sequence_scores = tf.gather_nd(\n            tf.squeeze(inputs, [1]),\n            tf.concat([example_inds, tag_indices], axis=1))\n        sequence_scores = tf.where(\n            tf.less_equal(sequence_lengths, 0), tf.zeros_like(sequence_scores),\n            sequence_scores)\n        return sequence_scores\n\n    def _multi_seq_fn():\n        # Compute the scores of the given tag sequence.\n        unary_scores = crf_unary_score(tag_indices, sequence_lengths, inputs)\n        binary_scores = crf_binary_score(tag_indices, sequence_lengths,\n                                         transition_params)\n        sequence_scores = unary_scores + binary_scores\n        return sequence_scores\n\n    if inputs.shape[1] == 1:\n        return _single_seq_fn()\n    else:\n        return _multi_seq_fn()\n\n\ndef crf_multitag_sequence_score(inputs, tag_bitmap, sequence_lengths,\n                                transition_params):\n    \"\"\"Computes the unnormalized score of all tag sequences matching\n    tag_bitmap.\n    \n    tag_bitmap enables more than one tag to be considered correct at each time\n    step. This is useful when an observed output at a given time step is\n    consistent with more than one tag, and thus the log likelihood of that\n    observation must take into account all possible consistent tags.\n    \n    Using one-hot vectors in tag_bitmap gives results identical to\n    crf_sequence_score.\n\n    Args:\n      inputs: A [batch_size, max_seq_len, num_tags] tensor of unary potentials\n    to use as input to the CRF layer.\n      tag_bitmap: A [batch_size, max_seq_len, num_tags] boolean tensor\n    representing all active tags at each index for which to calculate the\n    unnormalized score.\n      sequence_lengths: A [batch_size] vector of true sequence lengths.\n      transition_params: \n\n    Returns:\n      sequence_scores: A [batch_size] vector of unnormalized sequence scores.\n\n    \"\"\"\n    tag_bitmap = tf.cast(tag_bitmap, dtype=tf.bool)\n    sequence_lengths = tf.cast(sequence_lengths, dtype=tf.int32)\n    filtered_inputs = tf.where(tag_bitmap, inputs,\n                               tf.fill(tf.shape(inputs), float(\"-inf\")))\n\n    # If max_seq_len is 1, we skip the score calculation and simply gather the\n    # unary potentials of all active tags.\n    def _single_seq_fn():\n        return tf.reduce_logsumexp(\n            filtered_inputs, axis=[1, 2], keepdims=False)\n\n    def _multi_seq_fn():\n        # Compute the logsumexp of all scores of sequences matching the given tags.\n        return crf_log_norm(\n            inputs=filtered_inputs,\n            sequence_lengths=sequence_lengths,\n            transition_params=transition_params)\n\n    if inputs.shape[1] == 1:\n        return _single_seq_fn()\n    else:\n        return _multi_seq_fn()\n\n\ndef crf_log_norm(inputs, sequence_lengths, transition_params):\n    \"\"\"Computes the normalization for a CRF.\n\n    Args:\n      inputs: A [batch_size, max_seq_len, num_tags] tensor of unary potentials\n    to use as input to the CRF layer.\n      sequence_lengths: A [batch_size] vector of true sequence lengths.\n      transition_params: \n\n    Returns:\n      log_norm: A [batch_size] vector of normalizers for a CRF.\n\n    \"\"\"\n    sequence_lengths = tf.cast(sequence_lengths, dtype=tf.int32)\n    # Split up the first and rest of the inputs in preparation for the forward\n    # algorithm.\n    first_input = tf.slice(inputs, [0, 0, 0], [-1, 1, -1])\n    first_input = tf.squeeze(first_input, [1])\n\n    # If max_seq_len is 1, we skip the algorithm and simply reduce_logsumexp over\n    # the \"initial state\" (the unary potentials).\n    def _single_seq_fn():\n        log_norm = tf.reduce_logsumexp(first_input, [1])\n        # Mask `log_norm` of the sequences with length <= zero.\n        log_norm = tf.where(\n            tf.less_equal(sequence_lengths, 0), tf.zeros_like(log_norm),\n            log_norm)\n        return log_norm\n\n    def _multi_seq_fn():\n        \"\"\"Forward computation of alpha values.\"\"\"\n        rest_of_input = tf.slice(inputs, [0, 1, 0], [-1, -1, -1])\n        # Compute the alpha values in the forward algorithm in order to get the\n        # partition function.\n\n        alphas = crf_forward(rest_of_input, first_input, transition_params,\n                             sequence_lengths)\n        log_norm = tf.reduce_logsumexp(alphas, [1])\n        # Mask `log_norm` of the sequences with length <= zero.\n        log_norm = tf.where(\n            tf.less_equal(sequence_lengths, 0), tf.zeros_like(log_norm),\n            log_norm)\n        return log_norm\n\n    if inputs.shape[1] == 1:\n        return _single_seq_fn()\n    else:\n        return _multi_seq_fn()\n\n\ndef crf_log_likelihood(inputs,\n                       tag_indices,\n                       sequence_lengths,\n                       transition_params=None):\n    \"\"\"Computes the log-likelihood of tag sequences in a CRF.\n\n    Args:\n      inputs: A [batch_size, max_seq_len, num_tags] tensor of unary potentials\n    to use as input to the CRF layer.\n      tag_indices: A [batch_size, max_seq_len] matrix of tag indices for which\n    we compute the log-likelihood.\n      sequence_lengths: A [batch_size] vector of true sequence lengths.\n      transition_params: A [num_tags, num_tags] transition matrix, (Default value = None)\n\n    Returns:\n      log_likelihood: A [batch_size] `Tensor` containing the log-likelihood of\n      each example, given the sequence of tag indices.\n      transition_params: A [num_tags, num_tags] transition matrix. This is\n      either provided by the caller or created in this function.\n\n    \"\"\"\n    num_tags = inputs.shape[2]\n\n    # cast type to handle different types\n    tag_indices = tf.cast(tag_indices, dtype=tf.int32)\n    sequence_lengths = tf.cast(sequence_lengths, dtype=tf.int32)\n\n    if transition_params is None:\n        initializer = tf.keras.initializers.GlorotUniform()\n        transition_params = tf.Variable(\n            initializer([num_tags, num_tags]), \"transitions\")\n\n    sequence_scores = crf_sequence_score(inputs, tag_indices, sequence_lengths,\n                                         transition_params)\n    log_norm = crf_log_norm(inputs, sequence_lengths, transition_params)\n\n    # Normalize the scores to get the log-likelihood per example.\n    log_likelihood = sequence_scores - log_norm\n    return log_likelihood, transition_params\n\n\ndef crf_unary_score(tag_indices, sequence_lengths, inputs):\n    \"\"\"Computes the unary scores of tag sequences.\n\n    Args:\n      tag_indices: A [batch_size, max_seq_len] matrix of tag indices.\n      sequence_lengths: A [batch_size] vector of true sequence lengths.\n      inputs: \n\n    Returns:\n      unary_scores: A [batch_size] vector of unary scores.\n\n    \"\"\"\n    assert len(tag_indices.shape) == 2, 'tag_indices: A [batch_size, max_seq_len] matrix of tag indices.'\n    tag_indices = tf.cast(tag_indices, dtype=tf.int32)\n    sequence_lengths = tf.cast(sequence_lengths, dtype=tf.int32)\n\n    batch_size = tf.shape(inputs)[0]\n    max_seq_len = tf.shape(inputs)[1]\n    num_tags = tf.shape(inputs)[2]\n\n    flattened_inputs = tf.reshape(inputs, [-1])\n\n    offsets = tf.expand_dims(tf.range(batch_size) * max_seq_len * num_tags, 1)\n    offsets += tf.expand_dims(tf.range(max_seq_len) * num_tags, 0)\n    # Use int32 or int64 based on tag_indices' dtype.\n    if tag_indices.dtype == tf.int64:\n        offsets = tf.cast(offsets, tf.int64)\n    flattened_tag_indices = tf.reshape(offsets + tag_indices, [-1])\n\n    unary_scores = tf.reshape(\n        tf.gather(flattened_inputs, flattened_tag_indices),\n        [batch_size, max_seq_len])\n\n    masks = tf.sequence_mask(\n        sequence_lengths, maxlen=tf.shape(tag_indices)[1], dtype=tf.float32)\n\n    unary_scores = tf.reduce_sum(unary_scores * masks, 1)\n    return unary_scores\n\n\ndef crf_binary_score(tag_indices, sequence_lengths, transition_params):\n    \"\"\"Computes the binary scores of tag sequences.\n\n    Args:\n      tag_indices: A [batch_size, max_seq_len] matrix of tag indices.\n      sequence_lengths: A [batch_size] vector of true sequence lengths.\n      transition_params: \n\n    Returns:\n      binary_scores: A [batch_size] vector of binary scores.\n\n    \"\"\"\n    tag_indices = tf.cast(tag_indices, dtype=tf.int32)\n    sequence_lengths = tf.cast(sequence_lengths, dtype=tf.int32)\n\n    num_tags = tf.shape(transition_params)[0]\n    num_transitions = tf.shape(tag_indices)[1] - 1\n\n    # Truncate by one on each side of the sequence to get the start and end\n    # indices of each transition.\n    start_tag_indices = tf.slice(tag_indices, [0, 0], [-1, num_transitions])\n    end_tag_indices = tf.slice(tag_indices, [0, 1], [-1, num_transitions])\n\n    # Encode the indices in a flattened representation.\n    flattened_transition_indices = start_tag_indices * \\\n        num_tags + end_tag_indices\n    flattened_transition_params = tf.reshape(transition_params, [-1])\n\n    # Get the binary scores based on the flattened representation.\n    binary_scores = tf.gather(flattened_transition_params,\n                              flattened_transition_indices)\n\n    masks = tf.sequence_mask(\n        sequence_lengths, maxlen=tf.shape(tag_indices)[1], dtype=tf.float32)\n    truncated_masks = tf.slice(masks, [0, 1], [-1, -1])\n    binary_scores = tf.reduce_sum(binary_scores * truncated_masks, 1)\n    return binary_scores\n\n\ndef crf_forward(inputs, state, transition_params, sequence_lengths):\n    \"\"\"Computes the alpha values in a linear-chain CRF.\n    \n    See http://www.cs.columbia.edu/~mcollins/fb.pdf for reference.\n\n    Args:\n      inputs: A [batch_size, num_tags] matrix of unary potentials.\n      state: A [batch_size, num_tags] matrix containing the previous alpha\n    values.\n      transition_params: A [num_tags, num_tags] matrix of binary potentials.\n    This matrix is expanded into a [1, num_tags, num_tags] in preparation\n    for the broadcast summation occurring within the cell.\n      sequence_lengths: A [batch_size] vector of true sequence lengths.\n\n    Returns:\n      new_alphas: A [batch_size, num_tags] matrix containing the\n      new alpha values.\n\n    \"\"\"\n    sequence_lengths = tf.cast(sequence_lengths, dtype=tf.int32)\n\n    sequence_lengths = tf.maximum(\n        tf.constant(0, dtype=sequence_lengths.dtype), sequence_lengths - 2)\n    inputs = tf.transpose(inputs, [1, 0, 2])\n    transition_params = tf.expand_dims(transition_params, 0)\n\n    def _scan_fn(state, inputs):\n        state = tf.expand_dims(state, 2)\n        transition_scores = state + transition_params\n        new_alphas = inputs + tf.reduce_logsumexp(transition_scores, [1])\n        return new_alphas\n\n    all_alphas = tf.transpose(tf.scan(_scan_fn, inputs, state), [1, 0, 2])\n    idxs = tf.stack(\n        [tf.range(tf.shape(sequence_lengths)[0]), sequence_lengths], axis=1)\n    return tf.gather_nd(all_alphas, idxs)\n\n\ndef viterbi_decode(score, transition_params):\n    \"\"\"Decode the highest scoring sequence of tags outside of TensorFlow.\n    \n    This should only be used at test time.\n\n    Args:\n      score: A [seq_len, num_tags] matrix of unary potentials.\n      transition_params: A [num_tags, num_tags] matrix of binary potentials.\n\n    Returns:\n      viterbi: A [seq_len] list of integers containing the highest scoring tag\n      indices.\n      viterbi_score: A float containing the score for the Viterbi sequence.\n\n    \"\"\"\n    trellis = np.zeros_like(score)\n    backpointers = np.zeros_like(score, dtype=np.int32)\n    trellis[0] = score[0]\n\n    for t in range(1, score.shape[0]):\n        v = np.expand_dims(trellis[t - 1], 1) + transition_params\n        trellis[t] = score[t] + np.max(v, 0)\n        backpointers[t] = np.argmax(v, 0)\n\n    viterbi = [np.argmax(trellis[-1])]\n    for bp in reversed(backpointers[1:]):\n        viterbi.append(bp[viterbi[-1]])\n    viterbi.reverse()\n\n    viterbi_score = np.max(trellis[-1])\n    return viterbi, viterbi_score\n\n\nclass CrfDecodeForwardRnnCell(tf.keras.layers.AbstractRNNCell):\n    \"\"\"Computes the forward decoding in a linear-chain CRF.\"\"\"\n\n    def __init__(self, transition_params, **kwargs):\n        \"\"\"Initialize the CrfDecodeForwardRnnCell.\n\n        Args:\n          transition_params: A [num_tags, num_tags] matrix of binary\n            potentials. This matrix is expanded into a\n            [1, num_tags, num_tags] in preparation for the broadcast\n            summation occurring within the cell.\n        \"\"\"\n        super(CrfDecodeForwardRnnCell, self).__init__(**kwargs)\n        self._transition_params = tf.expand_dims(transition_params, 0)\n        self._num_tags = transition_params.shape[0]\n\n    @property\n    def state_size(self):\n        return self._num_tags\n\n    @property\n    def output_size(self):\n        return self._num_tags\n\n    def build(self, input_shape):\n        super(CrfDecodeForwardRnnCell, self).build(input_shape)\n\n    def call(self, inputs, state):\n        \"\"\"Build the CrfDecodeForwardRnnCell.\n\n        Args:\n          inputs: A [batch_size, num_tags] matrix of unary potentials.\n          state: A [batch_size, num_tags] matrix containing the previous step's\n        score values.\n\n        Returns:\n          backpointers: A [batch_size, num_tags] matrix of backpointers.\n          new_state: A [batch_size, num_tags] matrix of new score values.\n\n        \"\"\"\n        state = tf.expand_dims(state[0], 2)\n        transition_scores = state + self._transition_params\n        new_state = inputs + tf.reduce_max(transition_scores, [1])\n        backpointers = tf.argmax(transition_scores, 1)\n        backpointers = tf.cast(backpointers, dtype=tf.int32)\n        return backpointers, new_state\n\n\ndef crf_decode_forward(inputs, state, transition_params, sequence_lengths):\n    \"\"\"Computes forward decoding in a linear-chain CRF.\n\n    Args:\n      inputs: A [batch_size, num_tags] matrix of unary potentials.\n      state: A [batch_size, num_tags] matrix containing the previous step's\n    score values.\n      transition_params: A [num_tags, num_tags] matrix of binary potentials.\n      sequence_lengths: A [batch_size] vector of true sequence lengths.\n\n    Returns:\n      backpointers: A [batch_size, num_tags] matrix of backpointers.\n      new_state: A [batch_size, num_tags] matrix of new score values.\n\n    \"\"\"\n    sequence_lengths = tf.cast(sequence_lengths, dtype=tf.int32)\n    mask = tf.sequence_mask(sequence_lengths, tf.shape(inputs)[1])\n    crf_fwd_cell = CrfDecodeForwardRnnCell(transition_params)\n    crf_fwd_layer = tf.keras.layers.RNN(\n        crf_fwd_cell, return_sequences=True, return_state=True)\n    return crf_fwd_layer(inputs, state, mask=mask)\n\n\ndef crf_decode_backward(inputs, state):\n    \"\"\"Computes backward decoding in a linear-chain CRF.\n\n    Args:\n      inputs: A [batch_size, num_tags] matrix of\n    backpointer of next step (in time order).\n      state: A [batch_size, 1] matrix of tag index of next step.\n\n    Returns:\n      new_tags: A [batch_size, num_tags]\n      tensor containing the new tag indices.\n\n    \"\"\"\n    inputs = tf.transpose(inputs, [1, 0, 2])\n\n    def _scan_fn(state, inputs):\n        state = tf.squeeze(state, axis=[1])\n        idxs = tf.stack([tf.range(tf.shape(inputs)[0]), state], axis=1)\n        new_tags = tf.expand_dims(tf.gather_nd(inputs, idxs), axis=-1)\n        return new_tags\n\n    return tf.transpose(tf.scan(_scan_fn, inputs, state), [1, 0, 2])\n\n\ndef crf_decode(potentials, transition_params, sequence_length):\n    \"\"\"Decode the highest scoring sequence of tags in TensorFlow.\n    \n    This is a function for tensor.\n\n    Args:\n      potentials: A [batch_size, max_seq_len, num_tags] tensor of\n    unary potentials.\n      transition_params: A [num_tags, num_tags] matrix of\n    binary potentials.\n      sequence_length: A [batch_size] vector of true sequence lengths.\n\n    Returns:\n      decode_tags: A [batch_size, max_seq_len] matrix, with dtype `tf.int32`.\n      Contains the highest scoring tag indices.\n      best_score: A [batch_size] vector, containing the score of `decode_tags`.\n\n    \"\"\"\n    sequence_length = tf.cast(sequence_length, dtype=tf.int32)\n\n    # If max_seq_len is 1, we skip the algorithm and simply return the argmax tag\n    # and the max activation.\n    def _single_seq_fn():\n        squeezed_potentials = tf.squeeze(potentials, [1])\n        decode_tags = tf.expand_dims(tf.argmax(squeezed_potentials, axis=1), 1)\n        best_score = tf.reduce_max(squeezed_potentials, axis=1)\n        return tf.cast(decode_tags, dtype=tf.int32), best_score\n\n    def _multi_seq_fn():\n        \"\"\"Decoding of highest scoring sequence.\"\"\"\n        # Computes forward decoding. Get last score and backpointers.\n        initial_state = tf.slice(potentials, [0, 0, 0], [-1, 1, -1])\n        initial_state = tf.squeeze(initial_state, axis=[1])\n        inputs = tf.slice(potentials, [0, 1, 0], [-1, -1, -1])\n\n        sequence_length_less_one = tf.maximum(\n            tf.constant(0, dtype=sequence_length.dtype), sequence_length - 1)\n\n        backpointers, last_score = crf_decode_forward(\n            inputs, initial_state, transition_params, sequence_length_less_one)\n\n        backpointers = tf.reverse_sequence(\n            backpointers, sequence_length_less_one, seq_axis=1)\n\n        initial_state = tf.cast(tf.argmax(last_score, axis=1), dtype=tf.int32)\n        initial_state = tf.expand_dims(initial_state, axis=-1)\n\n        decode_tags = crf_decode_backward(backpointers, initial_state)\n        decode_tags = tf.squeeze(decode_tags, axis=[2])\n        decode_tags = tf.concat([initial_state, decode_tags], axis=1)\n        decode_tags = tf.reverse_sequence(\n            decode_tags, sequence_length, seq_axis=1)\n\n        best_score = tf.reduce_max(last_score, axis=1)\n        return decode_tags, best_score\n\n    if potentials.shape[1] == 1:\n        return _single_seq_fn()\n    else:\n        return _multi_seq_fn()\n"
  },
  {
    "path": "hanlp/layers/dropout.py",
    "content": "# -*- coding:utf-8 -*-\n# Date: 2020-06-05 17:47\nfrom typing import List\n\nimport torch\nimport torch.nn as nn\n\n\nclass WordDropout(nn.Module):\n    def __init__(self, p: float, oov_token: int, exclude_tokens: List[int] = None) -> None:\n        super().__init__()\n        self.oov_token = oov_token\n        self.p = p\n        if not exclude_tokens:\n            exclude_tokens = [0]\n        self.exclude = exclude_tokens\n\n    @staticmethod\n    def token_dropout(tokens: torch.LongTensor,\n                      oov_token: int,\n                      exclude_tokens: List[int],\n                      p: float = 0.2,\n                      training: float = True) -> torch.LongTensor:\n        \"\"\"During training, randomly replaces some of the non-padding tokens to a mask token with probability ``p``\n        \n        Adopted from https://github.com/Hyperparticle/udify\n\n        Args:\n          tokens: The current batch of padded sentences with word ids\n          oov_token: The mask token\n          exclude_tokens: The tokens for padding the input batch\n          p: The probability a word gets mapped to the unknown token\n          training: Applies the dropout if set to ``True``\n          tokens: torch.LongTensor: \n          oov_token: int: \n          exclude_tokens: List[int]: \n          p: float:  (Default value = 0.2)\n          training: float:  (Default value = True)\n\n        Returns:\n          A copy of the input batch with token dropout applied\n\n        \"\"\"\n        if training and p > 0:\n            # This creates a mask that only considers unpadded tokens for mapping to oov\n            padding_mask = tokens.new_ones(tokens.size(), dtype=torch.bool)\n            for pad in exclude_tokens:\n                padding_mask &= (tokens != pad)\n\n            # Create a uniformly random mask selecting either the original words or OOV tokens\n            dropout_mask = (tokens.new_empty(tokens.size(), dtype=torch.float).uniform_() < p)\n            oov_mask = dropout_mask & padding_mask\n\n            oov_fill = tokens.new_empty(tokens.size(), dtype=torch.long).fill_(oov_token)\n\n            result = torch.where(oov_mask, oov_fill, tokens)\n\n            return result\n        else:\n            return tokens\n\n    def forward(self, tokens: torch.LongTensor) -> torch.LongTensor:\n        return self.token_dropout(tokens, self.oov_token, self.exclude, self.p, self.training)\n\n\nclass SharedDropout(nn.Module):\n\n    def __init__(self, p=0.5, batch_first=True):\n        super(SharedDropout, self).__init__()\n\n        self.p = p\n        self.batch_first = batch_first\n\n    def extra_repr(self):\n        s = f\"p={self.p}\"\n        if self.batch_first:\n            s += f\", batch_first={self.batch_first}\"\n\n        return s\n\n    def forward(self, x):\n        if self.training:\n            if self.batch_first:\n                mask = self.get_mask(x[:, 0], self.p)\n            else:\n                mask = self.get_mask(x[0], self.p)\n            x *= mask.unsqueeze(1) if self.batch_first else mask\n\n        return x\n\n    @staticmethod\n    def get_mask(x, p):\n        mask = x.new_empty(x.shape).bernoulli_(1 - p)\n        mask = mask / (1 - p)\n\n        return mask\n\n\nclass IndependentDropout(nn.Module):\n\n    def __init__(self, p=0.5):\n        r\"\"\"\n        For :math:`N` tensors, they use different dropout masks respectively.\n        When :math:`N-M` of them are dropped, the remaining :math:`M` ones are scaled by a factor of :math:`N/M` to compensate,\n        and when all of them are dropped together, zeros are returned.\n        Copied from https://github.com/yzhangcs/parser/master/supar/modules/dropout.py.\n\n        Args:\n            p (float):\n                The probability of an element to be zeroed. Default: 0.5.\n\n        Examples:\n            >>> x, y = torch.ones(1, 3, 5), torch.ones(1, 3, 5)\n            >>> x, y = IndependentDropout()(x, y)\n            >>> x\n            tensor([[[1., 1., 1., 1., 1.],\n                     [0., 0., 0., 0., 0.],\n                     [2., 2., 2., 2., 2.]]])\n            >>> y\n            tensor([[[1., 1., 1., 1., 1.],\n                     [2., 2., 2., 2., 2.],\n                     [0., 0., 0., 0., 0.]]])\n        \"\"\"\n        super(IndependentDropout, self).__init__()\n        self.p = p\n\n    def extra_repr(self):\n        return f\"p={self.p}\"\n\n    def forward(self, *items):\n        if self.training:\n            masks = [x.new_empty(x.shape[:2]).bernoulli_(1 - self.p)\n                     for x in items]\n            total = sum(masks)\n            scale = len(items) / total.max(torch.ones_like(total))\n            masks = [mask * scale for mask in masks]\n            items = [item * mask.unsqueeze(dim=-1)\n                     for item, mask in zip(items, masks)]\n\n        return items\n\n\nclass LockedDropout(nn.Module):\n    def __init__(self, dropout_rate=0.5):\n        super(LockedDropout, self).__init__()\n        self.dropout_rate = dropout_rate\n\n    def forward(self, x):\n        if not self.training or not self.dropout_rate:\n            return x\n\n        if x.dim() == 3:\n            mask = x.new(x.size(0), 1, x.size(2)).bernoulli_(1 - self.dropout_rate) / (1 - self.dropout_rate)\n            mask = mask.expand_as(x)\n        elif x.dim() == 2:\n            mask = torch.empty_like(x).bernoulli_(1 - self.dropout_rate) / (1 - self.dropout_rate)\n        else:\n            raise ValueError(f'Unsupported dim: {x.dim()}. Only 2d (T,C) or 3d (B,T,C) is supported')\n        return mask * x\n"
  },
  {
    "path": "hanlp/layers/embeddings/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-08-24 21:48\n"
  },
  {
    "path": "hanlp/layers/embeddings/char_cnn.py",
    "content": "# Adopted from https://github.com/allenai/allennlp under Apache Licence 2.0.\n# Changed the packaging and created a subclass CharCNNEmbedding\n\nfrom typing import Union, Tuple, Optional, Callable\nimport torch\nfrom torch import nn\nfrom hanlp.layers.cnn_encoder import CnnEncoder\nfrom hanlp.layers.time_distributed import TimeDistributed\nfrom hanlp_common.configurable import AutoConfigurable\nfrom hanlp.common.transform import VocabDict, ToChar\nfrom hanlp.common.vocab import Vocab\nfrom hanlp.layers.embeddings.embedding import EmbeddingDim, Embedding\n\n\nclass CharCNN(nn.Module):\n    def __init__(self,\n                 field: str,\n                 embed: Union[int, Embedding], num_filters: int,\n                 ngram_filter_sizes: Tuple[int, ...] = (2, 3, 4, 5),\n                 conv_layer_activation: str = 'ReLU',\n                 output_dim: Optional[int] = None,\n                 vocab_size=None) -> None:\n        \"\"\"A `CnnEncoder` is a combination of multiple convolution layers and max pooling layers.\n        The input to this module is of shape `(batch_size, num_tokens,\n        input_dim)`, and the output is of shape `(batch_size, output_dim)`.\n\n        The CNN has one convolution layer for each ngram filter size. Each convolution operation gives\n        out a vector of size num_filters. The number of times a convolution layer will be used\n        is `num_tokens - ngram_size + 1`. The corresponding maxpooling layer aggregates all these\n        outputs from the convolution layer and outputs the max.\n\n        This operation is repeated for every ngram size passed, and consequently the dimensionality of\n        the output after maxpooling is `len(ngram_filter_sizes) * num_filters`.  This then gets\n        (optionally) projected down to a lower dimensional output, specified by `output_dim`.\n\n        We then use a fully connected layer to project in back to the desired output_dim.  For more\n        details, refer to \"A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural\n        Networks for Sentence Classification\", Zhang and Wallace 2016, particularly Figure 1.\n\n        See allennlp.modules.seq2vec_encoders.cnn_encoder.CnnEncoder, Apache 2.0\n\n        Args:\n            field: The field in samples this encoder will work on.\n            embed: An ``Embedding`` object or the feature size to create an ``Embedding`` object.\n            num_filters: This is the output dim for each convolutional layer, which is the number of \"filters\"\n                learned by that layer.\n            ngram_filter_sizes: This specifies both the number of convolutional layers we will create and their sizes.  The\n                default of `(2, 3, 4, 5)` will have four convolutional layers, corresponding to encoding\n                ngrams of size 2 to 5 with some number of filters.\n            conv_layer_activation: `Activation`, optional (default=`torch.nn.ReLU`)\n                Activation to use after the convolution layers.\n            output_dim: After doing convolutions and pooling, we'll project the collected features into a vector of\n                this size.  If this value is `None`, we will just return the result of the max pooling,\n                giving an output of shape `len(ngram_filter_sizes) * num_filters`.\n            vocab_size: The size of character vocab.\n\n        Returns:\n            A tensor of shape `(batch_size, output_dim)`.\n        \"\"\"\n        super().__init__()\n        EmbeddingDim.__init__(self)\n        # the embedding layer\n        if isinstance(embed, int):\n            embed = nn.Embedding(num_embeddings=vocab_size,\n                                 embedding_dim=embed)\n        else:\n            raise ValueError(f'Unrecognized type for {embed}')\n        self.field = field\n        self.embed = TimeDistributed(embed)\n        self.encoder = TimeDistributed(\n            CnnEncoder(embed.embedding_dim, num_filters, ngram_filter_sizes, conv_layer_activation, output_dim))\n        self.embedding_dim = output_dim or num_filters * len(ngram_filter_sizes)\n\n    def forward(self, batch: dict, **kwargs):\n        tokens: torch.Tensor = batch[f'{self.field}_char_id']\n        mask = tokens.ge(0)\n        x = self.embed(tokens)\n        return self.encoder(x, mask)\n\n    def get_output_dim(self) -> int:\n        return self.embedding_dim\n\n\nclass CharCNNEmbedding(Embedding, AutoConfigurable):\n    def __init__(self,\n                 field,\n                 embed: Union[int, Embedding],\n                 num_filters: int,\n                 ngram_filter_sizes: Tuple[int, ...] = (2, 3, 4, 5),\n                 conv_layer_activation: str = 'ReLU',\n                 output_dim: Optional[int] = None,\n                 min_word_length=None\n                 ) -> None:\n        \"\"\"\n\n        Args:\n            field: The character field in samples this encoder will work on.\n            embed: An ``Embedding`` object or the feature size to create an ``Embedding`` object.\n            num_filters: This is the output dim for each convolutional layer, which is the number of \"filters\"\n                learned by that layer.\n            ngram_filter_sizes: This specifies both the number of convolutional layers we will create and their sizes.  The\n                default of `(2, 3, 4, 5)` will have four convolutional layers, corresponding to encoding\n                ngrams of size 2 to 5 with some number of filters.\n            conv_layer_activation: `Activation`, optional (default=`torch.nn.ReLU`)\n                Activation to use after the convolution layers.\n            output_dim: After doing convolutions and pooling, we'll project the collected features into a vector of\n                this size.  If this value is `None`, we will just return the result of the max pooling,\n                giving an output of shape `len(ngram_filter_sizes) * num_filters`.\n            min_word_length: For ngram filter with max size, the input (chars) is required to have at least max size\n                chars.\n        \"\"\"\n        super().__init__()\n        if min_word_length is None:\n            min_word_length = max(ngram_filter_sizes)\n        self.min_word_length = min_word_length\n        self.output_dim = output_dim\n        self.conv_layer_activation = conv_layer_activation\n        self.ngram_filter_sizes = ngram_filter_sizes\n        self.num_filters = num_filters\n        self.embed = embed\n        self.field = field\n\n    def transform(self, vocabs: VocabDict, **kwargs) -> Optional[Callable]:\n        if isinstance(self.embed, Embedding):\n            self.embed.transform(vocabs=vocabs)\n        vocab_name = self.vocab_name\n        if vocab_name not in vocabs:\n            vocabs[vocab_name] = Vocab()\n        return ToChar(self.field, vocab_name, min_word_length=self.min_word_length,\n                      pad=vocabs[vocab_name].safe_pad_token)\n\n    @property\n    def vocab_name(self):\n        vocab_name = f'{self.field}_char'\n        return vocab_name\n\n    def module(self, vocabs: VocabDict, **kwargs) -> Optional[nn.Module]:\n        embed = self.embed\n        if isinstance(embed, Embedding):\n            embed = embed.module(vocabs=vocabs)\n        return CharCNN(self.field,\n                       embed,\n                       self.num_filters,\n                       self.ngram_filter_sizes,\n                       self.conv_layer_activation,\n                       self.output_dim,\n                       vocab_size=len(vocabs[self.vocab_name]))\n"
  },
  {
    "path": "hanlp/layers/embeddings/char_cnn_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-20 21:15\nfrom functools import reduce\n\nimport tensorflow as tf\n\nfrom hanlp.common.vocab_tf import VocabTF\nfrom hanlp.utils.tf_util import hanlp_register\n\n\n@hanlp_register\nclass CharCNNEmbeddingTF(tf.keras.layers.Layer):\n    def __init__(self, word_vocab: VocabTF, char_vocab: VocabTF,\n                 char_embedding=100,\n                 kernel_size=3,\n                 filters=50,\n                 dropout=0.5,\n                 trainable=True, name=None, dtype=None, dynamic=False,\n                 **kwargs):\n        super().__init__(trainable, name, dtype, dynamic, **kwargs)\n        self.char_embedding = char_embedding\n        self.filters = filters\n        self.kernel_size = kernel_size\n        self.char_vocab = char_vocab\n        self.word_vocab = word_vocab\n        self.embedding = tf.keras.layers.Embedding(input_dim=len(self.char_vocab), output_dim=char_embedding,\n                                                   trainable=True, mask_zero=True)\n        self.dropout = tf.keras.layers.Dropout(dropout)\n        self.cnn = tf.keras.layers.Conv1D(filters, kernel_size, padding='same')\n\n    def call(self, inputs: tf.Tensor, **kwargs):\n        mask = tf.not_equal(inputs, self.word_vocab.pad_token)\n        inputs = tf.ragged.boolean_mask(inputs, mask)\n        chars = tf.strings.unicode_split(inputs, input_encoding='UTF-8')\n        chars = chars.to_tensor(default_value=self.char_vocab.pad_token)\n        chars = self.char_vocab.lookup(chars)\n        embed = self.embedding(chars)\n        weights = embed._keras_mask\n        embed = self.dropout(embed)\n        features = masked_conv1d_and_max(embed, weights, self.cnn)\n        features._keras_mask = mask\n        return features\n\n    def compute_output_shape(self, input_shape):\n        return super().compute_output_shape(input_shape)\n\n    def get_config(self):\n        config = {\n            'char_embedding': self.char_embedding,\n            'kernel_size': self.kernel_size,\n            'filters': self.filters,\n            'dropout': self.dropout.rate,\n        }\n        base_config = super(CharCNNEmbeddingTF, self).get_config()\n        return dict(list(base_config.items()) + list(config.items()))\n\n\ndef masked_conv1d_and_max(t, weights, conv1d):\n    \"\"\"Applies 1d convolution and a masked max-pooling\n    \n    https://github.com/guillaumegenthial/tf_ner/blob/master/models/chars_conv_lstm_crf/masked_conv.py\n\n    Args:\n      t(tf.Tensor): A tensor with at least 3 dimensions [d1, d2, ..., dn-1, dn]\n      weights(tf.Tensor of tf.bool): A Tensor of shape [d1, d2, dn-1]\n      filters(int): number of filters\n      kernel_size(int): kernel size for the temporal convolution\n      conv1d: \n\n    Returns:\n\n    \n    \"\"\"\n    # Get shape and parameters\n    shape = tf.shape(t)\n    ndims = t.shape.ndims\n    dim1 = reduce(lambda x, y: x * y, [shape[i] for i in range(ndims - 2)])\n    dim2 = shape[-2]\n    dim3 = t.shape[-1]\n\n    # Reshape weights\n    weights = tf.reshape(weights, shape=[dim1, dim2, 1])\n    weights = tf.cast(weights, tf.float32)\n\n    # Reshape input and apply weights\n    flat_shape = [dim1, dim2, dim3]\n    t = tf.reshape(t, shape=flat_shape)\n    t *= weights\n\n    # Apply convolution\n    t_conv = conv1d(t)\n    t_conv *= weights\n\n    # Reduce max -- set to zero if all padded\n    t_conv += (1. - weights) * tf.reduce_min(t_conv, axis=-2, keepdims=True)\n    t_max = tf.reduce_max(t_conv, axis=-2)\n\n    # Reshape the output\n    final_shape = [shape[i] for i in range(ndims - 2)] + [conv1d.filters]\n    t_max = tf.reshape(t_max, shape=final_shape)\n\n    return t_max\n"
  },
  {
    "path": "hanlp/layers/embeddings/char_rnn.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-02 23:49\nfrom typing import Optional, Callable, Union\n\nimport torch\nimport torch.nn as nn\nfrom torch.nn.utils.rnn import pack_padded_sequence\n\nfrom hanlp_common.configurable import AutoConfigurable\nfrom hanlp.common.transform import VocabDict, ToChar\nfrom hanlp.common.vocab import Vocab\nfrom hanlp.layers.embeddings.embedding import Embedding, EmbeddingDim\n\n\nclass CharRNN(nn.Module, EmbeddingDim):\n    def __init__(self,\n                 field,\n                 vocab_size,\n                 embed: Union[int, nn.Embedding],\n                 hidden_size):\n        \"\"\"Character level RNN embedding module.\n\n        Args:\n            field: The field in samples this encoder will work on.\n            vocab_size: The size of character vocab.\n            embed: An ``Embedding`` object or the feature size to create an ``Embedding`` object.\n            hidden_size: The hidden size of RNNs.\n        \"\"\"\n        super(CharRNN, self).__init__()\n        self.field = field\n        # the embedding layer\n        if isinstance(embed, int):\n            self.embed = nn.Embedding(num_embeddings=vocab_size,\n                                      embedding_dim=embed)\n        elif isinstance(embed, nn.Module):\n            self.embed = embed\n            embed = embed.embedding_dim\n        else:\n            raise ValueError(f'Unrecognized type for {embed}')\n        # the lstm layer\n        self.lstm = nn.LSTM(input_size=embed,\n                            hidden_size=hidden_size,\n                            batch_first=True,\n                            bidirectional=True)\n\n    def forward(self, batch, mask, **kwargs):\n        x = batch[f'{self.field}_char_id']\n        # [batch_size, seq_len, fix_len]\n        mask = x.ne(0)\n        # [batch_size, seq_len]\n        lens = mask.sum(-1)\n        char_mask = lens.gt(0)\n\n        # [n, fix_len, n_embed]\n        x = self.embed(batch) if isinstance(self.embed, EmbeddingDim) else self.embed(x[char_mask])\n        x = pack_padded_sequence(x[char_mask], lens[char_mask].cpu(), True, False)\n        x, (h, _) = self.lstm(x)\n        # [n, fix_len, n_out]\n        h = torch.cat(torch.unbind(h), -1)\n        # [batch_size, seq_len, n_out]\n        embed = h.new_zeros(*lens.shape, h.size(-1))\n        embed = embed.masked_scatter_(char_mask.unsqueeze(-1), h)\n\n        return embed\n\n    @property\n    def embedding_dim(self) -> int:\n        return self.lstm.hidden_size * 2\n\n\nclass CharRNNEmbedding(Embedding, AutoConfigurable):\n    def __init__(self,\n                 field,\n                 embed,\n                 hidden_size,\n                 max_word_length=None) -> None:\n        \"\"\"Character level RNN embedding module builder.\n\n        Args:\n            field: The field in samples this encoder will work on.\n            embed: An ``Embedding`` object or the feature size to create an ``Embedding`` object.\n            hidden_size: The hidden size of RNNs.\n            max_word_length: Character sequence longer than ``max_word_length`` will be truncated.\n        \"\"\"\n        super().__init__()\n        self.field = field\n        self.hidden_size = hidden_size\n        self.embed = embed\n        self.max_word_length = max_word_length\n\n    def transform(self, vocabs: VocabDict, **kwargs) -> Optional[Callable]:\n        if isinstance(self.embed, Embedding):\n            self.embed.transform(vocabs=vocabs)\n        vocab_name = self.vocab_name\n        if vocab_name not in vocabs:\n            vocabs[vocab_name] = Vocab()\n        return ToChar(self.field, vocab_name, max_word_length=self.max_word_length)\n\n    @property\n    def vocab_name(self):\n        vocab_name = f'{self.field}_char'\n        return vocab_name\n\n    def module(self, vocabs: VocabDict, **kwargs) -> Optional[nn.Module]:\n        embed = self.embed\n        if isinstance(self.embed, Embedding):\n            embed = self.embed.module(vocabs=vocabs)\n        return CharRNN(self.field, len(vocabs[self.vocab_name]), embed, self.hidden_size)\n"
  },
  {
    "path": "hanlp/layers/embeddings/char_rnn_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-20 17:02\nimport tensorflow as tf\n\nfrom hanlp.common.vocab_tf import VocabTF\nfrom hanlp.utils.tf_util import hanlp_register\n\n\n@hanlp_register\nclass CharRNNEmbeddingTF(tf.keras.layers.Layer):\n    def __init__(self, word_vocab: VocabTF, char_vocab: VocabTF,\n                 char_embedding=100,\n                 char_rnn_units=25,\n                 dropout=0.5,\n                 trainable=True, name=None, dtype=None, dynamic=False,\n                 **kwargs):\n        super().__init__(trainable, name, dtype, dynamic, **kwargs)\n        self.char_embedding = char_embedding\n        self.char_rnn_units = char_rnn_units\n        self.char_vocab = char_vocab\n        self.word_vocab = word_vocab\n        self.embedding = tf.keras.layers.Embedding(input_dim=len(self.char_vocab), output_dim=char_embedding,\n                                                   trainable=True, mask_zero=True)\n        self.dropout = tf.keras.layers.Dropout(dropout)\n        self.rnn = tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(units=char_rnn_units,\n                                                                      return_state=True), name='bilstm')\n\n    def call(self, inputs: tf.Tensor, **kwargs):\n        mask = tf.not_equal(inputs, self.word_vocab.pad_token)\n        inputs = tf.ragged.boolean_mask(inputs, mask)\n        chars = tf.strings.unicode_split(inputs, input_encoding='UTF-8')\n        chars = chars.to_tensor(default_value=self.char_vocab.pad_token)\n        chars = self.char_vocab.lookup(chars)\n        embed = self.embedding(chars)\n        char_mask = embed._keras_mask\n        embed = self.dropout(embed)\n        embed_shape = tf.shape(embed)\n        embed = tf.reshape(embed, [-1, embed_shape[2], embed_shape[3]])\n        char_mask = tf.reshape(char_mask, [-1, embed_shape[2]])\n        all_zeros = tf.reduce_sum(tf.cast(char_mask, tf.int32), axis=1) == 0\n        char_mask_shape = tf.shape(char_mask)\n        hole = tf.zeros(shape=(char_mask_shape[0], char_mask_shape[1] - 1), dtype=tf.bool)\n        all_zeros = tf.expand_dims(all_zeros, -1)\n        non_all_zeros = tf.concat([all_zeros, hole], axis=1)\n        char_mask = tf.logical_or(char_mask, non_all_zeros)\n        output, h_fw, c_fw, h_bw, c_bw = self.rnn(embed, mask=char_mask)\n        hidden = tf.concat([h_fw, h_bw], axis=-1)\n        # hidden = output\n        hidden = tf.reshape(hidden, [embed_shape[0], embed_shape[1], -1])\n        hidden._keras_mask = mask\n        return hidden\n\n    def get_config(self):\n        config = {\n            'char_embedding': self.char_embedding,\n            'char_rnn_units': self.char_rnn_units,\n            'dropout': self.dropout.rate,\n        }\n        base_config = super(CharRNNEmbeddingTF, self).get_config()\n        return dict(list(base_config.items()) + list(config.items()))\n"
  },
  {
    "path": "hanlp/layers/embeddings/concat_embedding.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-20 17:08\nimport tensorflow as tf\n\nfrom hanlp.utils.tf_util import hanlp_register, copy_mask\n\n\n@hanlp_register\nclass ConcatEmbedding(tf.keras.layers.Layer):\n    def __init__(self, *embeddings, trainable=True, name=None, dtype=None, dynamic=False, **kwargs):\n        self.embeddings = []\n        for embed in embeddings:\n            embed: tf.keras.layers.Layer = tf.keras.utils.deserialize_keras_object(embed) if isinstance(embed,\n                                                                                                        dict) else embed\n            self.embeddings.append(embed)\n            if embed.trainable:\n                trainable = True\n            if embed.dynamic:\n                dynamic = True\n            if embed.supports_masking:\n                self.supports_masking = True\n\n        super().__init__(trainable, name, dtype, dynamic, **kwargs)\n\n    def build(self, input_shape):\n        for embed in self.embeddings:\n            embed.build(input_shape)\n        super().build(input_shape)\n\n    def compute_mask(self, inputs, mask=None):\n        for embed in self.embeddings:\n            mask = embed.compute_mask(inputs, mask)\n            if mask is not None:\n                return mask\n        return mask\n\n    def call(self, inputs, **kwargs):\n        embeds = [embed.call(inputs) for embed in self.embeddings]\n        feature = tf.concat(embeds, axis=-1)\n\n        for embed in embeds:\n            mask = copy_mask(embed, feature)\n            if mask is not None:\n                break\n        return feature\n\n    def get_config(self):\n        config = {\n            'embeddings': [embed.get_config() for embed in self.embeddings],\n        }\n        base_config = super(ConcatEmbedding, self).get_config()\n        return dict(list(base_config.items()) + list(config.items()))\n\n    def compute_output_shape(self, input_shape):\n        dim = 0\n        for embed in self.embeddings:\n            dim += embed.compute_output_shape(input_shape)[-1]\n\n        return input_shape + dim\n"
  },
  {
    "path": "hanlp/layers/embeddings/contextual_string_embedding.py",
    "content": "# Most codes of this file is adopted from flair, which is licenced under:\n#\n# The MIT License (MIT)\n#\n# Flair is licensed under the following MIT License (MIT) Copyright © 2018 Zalando SE, https://tech.zalando.com\n# Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n# The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n# THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n\nimport os\nfrom typing import List, Dict, Callable\n\nimport torch\nimport torch.nn as nn\nfrom torch.nn.utils.rnn import pack_padded_sequence, pad_packed_sequence\n\nfrom hanlp_common.configurable import Configurable\nfrom hanlp.common.transform import TransformList, FieldToIndex\nfrom hanlp.common.vocab import Vocab\nfrom hanlp.layers.embeddings.embedding import Embedding, EmbeddingDim\nfrom hanlp.utils.io_util import get_resource\nfrom hanlp.utils.torch_util import pad_lists, batched_index_select\nfrom tests import cdroot\n\n\nclass RNNLanguageModel(nn.Module):\n    \"\"\"Container module with an encoder, a recurrent module, and a decoder.\"\"\"\n\n    def __init__(self,\n                 n_tokens,\n                 is_forward_lm: bool,\n                 hidden_size: int,\n                 embedding_size: int = 100):\n        super(RNNLanguageModel, self).__init__()\n\n        self.is_forward_lm: bool = is_forward_lm\n        self.n_tokens = n_tokens\n        self.hidden_size = hidden_size\n        self.embedding_size = embedding_size\n\n        self.encoder = nn.Embedding(n_tokens, embedding_size)\n        self.rnn = nn.LSTM(embedding_size, hidden_size, batch_first=True)\n\n    def forward(self, ids: torch.LongTensor, lens: torch.LongTensor):\n        emb = self.encoder(ids)\n        x = pack_padded_sequence(emb, lens, True, False)\n        x, _ = self.rnn(x)\n        x, _ = pad_packed_sequence(x, True)\n        return x\n\n    @classmethod\n    def load_language_model(cls, model_file):\n        model_file = get_resource(model_file)\n        state = torch.load(model_file)\n        model = RNNLanguageModel(state['n_tokens'],\n                                 state['is_forward_lm'],\n                                 state['hidden_size'],\n                                 state['embedding_size'])\n        model.load_state_dict(state['state_dict'], strict=False)\n        return model\n\n    def save(self, file):\n        model_state = {\n            'state_dict': self.state_dict(),\n            'n_tokens': self.n_tokens,\n            'is_forward_lm': self.is_forward_lm,\n            'hidden_size': self.hidden_size,\n            'embedding_size': self.embedding_size,\n        }\n        torch.save(model_state, file, pickle_protocol=4)\n\n\nclass ContextualStringEmbeddingModule(nn.Module, EmbeddingDim):\n\n    def __init__(self, field: str, path: str, trainable=False) -> None:\n        super().__init__()\n        self.field = field\n        path = get_resource(path)\n        f = os.path.join(path, 'forward.pt')\n        b = os.path.join(path, 'backward.pt')\n        self.f: RNNLanguageModel = RNNLanguageModel.load_language_model(f)\n        self.b: RNNLanguageModel = RNNLanguageModel.load_language_model(b)\n        if not trainable:\n            for p in self.parameters():\n                p.requires_grad_(False)\n\n    def __call__(self, batch: dict, **kwargs):\n        args = ['f_char_id', 'f_offset', 'b_char_id', 'b_offset']\n        keys = [f'{self.field}_{key}' for key in args]\n        args = [batch[key] for key in keys]\n        return super().__call__(*args, **kwargs)\n\n    @property\n    def embedding_dim(self):\n        return self.f.rnn.hidden_size + self.b.rnn.hidden_size\n\n    def run_lm(self, lm, ids: torch.Tensor, offsets: torch.LongTensor):\n        lens = offsets.max(-1)[0] + 1\n        rnn_output = lm(ids, lens)\n        return batched_index_select(rnn_output, offsets)\n\n    def forward(self,\n                f_chars_id: torch.Tensor,\n                f_offset: torch.LongTensor,\n                b_chars_id: torch.Tensor,\n                b_offset: torch.LongTensor, **kwargs):\n        f = self.run_lm(self.f, f_chars_id, f_offset)\n        b = self.run_lm(self.b, b_chars_id, b_offset)\n        return torch.cat([f, b], dim=-1)\n\n    def embed(self, sents: List[List[str]], vocab: Dict[str, int]):\n        f_chars, f_offsets = [], []\n        b_chars, b_offsets = [], []\n\n        transform = ContextualStringEmbeddingTransform('token')\n        for tokens in sents:\n            sample = transform({'token': tokens})\n            for each, name in zip([f_chars, b_chars, f_offsets, b_offsets],\n                                  'f_chars, b_chars, f_offsets, b_offsets'.split(', ')):\n                each.append(sample[f'token_{name}'])\n        f_ids = []\n        for cs in f_chars:\n            f_ids.append([vocab[c] for c in cs])\n        f_ids = pad_lists(f_ids)\n        f_offsets = pad_lists(f_offsets)\n\n        b_ids = []\n        for cs in b_chars:\n            b_ids.append([vocab[c] for c in cs])\n        b_ids = pad_lists(b_ids)\n        b_offsets = pad_lists(b_offsets)\n        return self.forward(f_ids, f_offsets, b_ids, b_offsets)\n\n\nclass ContextualStringEmbeddingTransform(Configurable):\n\n    def __init__(self, src: str) -> None:\n        self.src = src\n\n    def __call__(self, sample: dict):\n        tokens = sample[self.src]\n        f_o = []\n        b_o = []\n        sentence_text = ' '.join(tokens)\n        end_marker = ' '\n        extra_offset = 1\n        # f\n        input_text = '\\n' + sentence_text + end_marker\n        f_chars = list(input_text)\n        # b\n        sentence_text = sentence_text[::-1]\n        input_text = '\\n' + sentence_text + end_marker\n        b_chars = list(input_text)\n        offset_forward: int = extra_offset\n        offset_backward: int = len(sentence_text) + extra_offset\n        for token in tokens:\n            offset_forward += len(token)\n\n            f_o.append(offset_forward)\n            b_o.append(offset_backward)\n\n            # This language model is tokenized\n            offset_forward += 1\n            offset_backward -= 1\n\n            offset_backward -= len(token)\n        sample[f'{self.src}_f_char'] = f_chars\n        sample[f'{self.src}_b_char'] = b_chars\n        sample[f'{self.src}_f_offset'] = f_o\n        sample[f'{self.src}_b_offset'] = b_o\n        return sample\n\n\nclass ContextualStringEmbedding(Embedding):\n    def __init__(self, field, path, trainable=False) -> None:\n        super().__init__()\n        self.trainable = trainable\n        self.path = path\n        self.field = field\n\n    def transform(self, **kwargs) -> Callable:\n        vocab = Vocab()\n        vocab.load(os.path.join(get_resource(self.path), 'vocab.json'))\n        return TransformList(ContextualStringEmbeddingTransform(self.field),\n                             FieldToIndex(f'{self.field}_f_char', vocab),\n                             FieldToIndex(f'{self.field}_b_char', vocab))\n\n    def module(self, **kwargs) -> nn.Module:\n        return ContextualStringEmbeddingModule(self.field, self.path, self.trainable)\n\n\ndef main():\n    # _validate()\n    flair = ContextualStringEmbedding('token', 'FASTTEXT_DEBUG_EMBEDDING_EN')\n    print(flair.config)\n\n\ndef _validate():\n    cdroot()\n    flair = ContextualStringEmbeddingModule('token', 'FLAIR_LM_WMT11_EN')\n    vocab = torch.load('/home/hhe43/flair/item2idx.pt')\n    vocab = dict((x.decode(), y) for x, y in vocab.items())\n    # vocab = Vocab(token_to_idx=vocab, pad_token='<unk>')\n    # vocab.lock()\n    # vocab.summary()\n    # vocab.save('vocab.json')\n    tokens = 'I love Berlin .'.split()\n    sent = ' '.join(tokens)\n    embed = flair.embed([tokens, tokens], vocab)\n    gold = torch.load('/home/hhe43/flair/gold.pt')\n    print(torch.allclose(embed[1, :, :2048], gold, atol=1e-6))\n    # print(torch.all(torch.eq(embed[1, :, :], gold)))\n\n\nif __name__ == '__main__':\n    main()\n"
  },
  {
    "path": "hanlp/layers/embeddings/contextual_string_embedding_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-19 03:24\nfrom typing import List\n\nimport tensorflow as tf\nimport numpy as np\nfrom hanlp.components.rnn_language_model_tf import RNNLanguageModel\nfrom hanlp_common.constant import PAD\nfrom hanlp.utils.io_util import get_resource\nfrom hanlp.utils.tf_util import copy_mask, hanlp_register, str_tensor_2d_to_list\nfrom hanlp_common.util import infer_space_after\n\n\n@hanlp_register\nclass ContextualStringEmbeddingTF(tf.keras.layers.Layer):\n\n    def __init__(self, forward_model_path=None, backward_model_path=None, max_word_len=10,\n                 trainable=False, name=None, dtype=None,\n                 dynamic=True, **kwargs):\n        assert dynamic, 'ContextualStringEmbedding works only in eager mode'\n        super().__init__(trainable, name, dtype, dynamic, **kwargs)\n        assert any([forward_model_path, backward_model_path]), 'At least one model is required'\n        self.forward_model_path = forward_model_path\n        self.backward_model_path = backward_model_path\n        self.forward_model = self._load_lm(forward_model_path) if forward_model_path else None\n        self.backward_model = self._load_lm(backward_model_path) if backward_model_path else None\n        if trainable:\n            self._fw = self.forward_model.model\n            self._bw = self.backward_model.model\n            for m in self._fw, self._bw:\n                m.trainable = True\n        self.supports_masking = True\n        self.max_word_len = max_word_len\n\n    def call(self, inputs, **kwargs):\n        str_inputs = str_tensor_2d_to_list(inputs)\n        outputs = self.embed(str_inputs)\n        copy_mask(inputs, outputs)\n        return outputs\n\n    def _load_lm(self, filepath):\n        filepath = get_resource(filepath)\n        lm = RNNLanguageModel()\n        lm.load(filepath)\n        model: tf.keras.Sequential = lm.model\n        for idx, layer in enumerate(model.layers):\n            if isinstance(layer, tf.keras.layers.LSTM):\n                lm.model = tf.keras.Sequential(model.layers[:idx + 1])  # discard dense layer\n                return lm\n\n    def embed(self, texts: List[List[str]]):\n        \"\"\"Embedding sentences (list of words) with contextualized string embedding\n\n        Args:\n          texts: List of words, not chars\n          texts: List[List[str]]: \n\n        Returns:\n\n        \n        \"\"\"\n        fw = None\n        if self.forward_model:\n            fw = self._run_rnn(texts, model=self.forward_model)\n        bw = None\n        if self.backward_model:\n            bw = self._run_rnn(texts, model=self.backward_model)\n        if not all(x is not None for x in [fw, bw]):\n            return fw if fw is not None else bw\n        else:\n            return tf.concat([fw, bw], axis=-1)\n\n    def _run_rnn(self, texts, model):\n        embeddings = []\n        inputs = []\n        offsets = []\n        tokenizer = model.transform.tokenize_func()\n        backward = not model.config['forward']\n        for sent in texts:\n            raw, off = self._get_raw_string(sent, tokenizer)\n            inputs.append(raw)\n            offsets.append(off)\n        outputs = model.model_from_config.predict(model.transform.inputs_to_dataset(inputs))\n        if backward:\n            outputs = tf.reverse(outputs, axis=[1])\n        maxlen = len(max(texts, key=len))\n        for hidden, off, sent in zip(outputs, offsets, texts):\n            embed = []\n            for (start, end), word in zip(off, sent):\n                embed.append(hidden[end - 1, :])\n            if len(embed) < maxlen:\n                embed += [np.zeros_like(embed[-1])] * (maxlen - len(embed))\n            embeddings.append(np.stack(embed))\n        return tf.stack(embeddings)\n\n    def _get_raw_string(self, sent: List[str], tokenizer):\n        raw_string = []\n        offsets = []\n        whitespace_after = infer_space_after(sent)\n        start = 0\n        for word, space in zip(sent, whitespace_after):\n            chars = tokenizer(word)\n            chars = chars[:self.max_word_len]\n            if space:\n                chars += [' ']\n            end = start + len(chars)\n            offsets.append((start, end))\n            start = end\n            raw_string += chars\n        return raw_string, offsets\n\n    def get_config(self):\n        config = {\n            'forward_model_path': self.forward_model_path,\n            'backward_model_path': self.backward_model_path,\n            'max_word_len': self.max_word_len,\n        }\n        base_config = super(ContextualStringEmbeddingTF, self).get_config()\n        return dict(list(base_config.items()) + list(config.items()))\n\n    @property\n    def output_dim(self):\n        dim = 0\n        for model in self.forward_model, self.backward_model:\n            if model:\n                dim += model.config['rnn_units']\n        return dim\n\n    def compute_output_shape(self, input_shape):\n        return input_shape + self.output_dim\n\n    def compute_mask(self, inputs, mask=None):\n\n        return tf.not_equal(inputs, PAD)\n"
  },
  {
    "path": "hanlp/layers/embeddings/contextual_word_embedding.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-07-05 13:50\nfrom typing import Optional, Union, List, Any, Dict, Tuple\n\nimport torch\nfrom torch import nn\n\nfrom hanlp_common.configurable import AutoConfigurable\nfrom hanlp.layers.embeddings.embedding import Embedding\nfrom hanlp.layers.scalar_mix import ScalarMixWithDropoutBuilder\nfrom hanlp.layers.transformers.encoder import TransformerEncoder\nfrom hanlp.layers.transformers.pt_imports import PreTrainedTokenizer, AutoConfig_, AutoTokenizer_\nfrom hanlp.transform.transformer_tokenizer import TransformerSequenceTokenizer\n\n\nclass ContextualWordEmbeddingModule(TransformerEncoder):\n    def __init__(self,\n                 field: str,\n                 transformer: str,\n                 transformer_tokenizer: PreTrainedTokenizer,\n                 average_subwords=False,\n                 scalar_mix: Union[ScalarMixWithDropoutBuilder, int] = None,\n                 word_dropout=None,\n                 max_sequence_length=None,\n                 ret_raw_hidden_states=False,\n                 transformer_args: Dict[str, Any] = None,\n                 trainable=True,\n                 training=True) -> None:\n        \"\"\"A contextualized word embedding module.\n\n        Args:\n            field: The field to work on. Usually some token fields.\n            transformer:  An identifier of a ``PreTrainedModel``.\n            transformer_tokenizer:\n            average_subwords: ``True`` to average subword representations.\n            scalar_mix: Layer attention.\n            word_dropout: Dropout rate of randomly replacing a subword with MASK.\n            max_sequence_length: The maximum sequence length. Sequence longer than this will be handled by sliding\n                window.\n            ret_raw_hidden_states: ``True`` to return hidden states of each layer.\n            transformer_args: Extra arguments passed to the transformer.\n            trainable: ``False`` to use static embeddings.\n            training: ``False`` to skip loading weights from pre-trained transformers.\n        \"\"\"\n        super().__init__(transformer, transformer_tokenizer, average_subwords, scalar_mix, word_dropout,\n                         max_sequence_length, ret_raw_hidden_states, transformer_args, trainable,\n                         training)\n        self.field = field\n\n    # noinspection PyMethodOverriding\n    # noinspection PyTypeChecker\n    def forward(self, batch: dict, mask=None, **kwargs):\n        input_ids: torch.LongTensor = batch[f'{self.field}_input_ids']\n        token_span: torch.LongTensor = batch.get(f'{self.field}_token_span', None)\n        # input_device = input_ids.device\n        # this_device = self.get_device()\n        # if input_device != this_device:\n        #     input_ids = input_ids.to(this_device)\n        #     token_span = token_span.to(this_device)\n        # We might want to apply mask here\n        output: Union[torch.Tensor, List[torch.Tensor]] = super().forward(input_ids, token_span=token_span, **kwargs)\n        # if input_device != this_device:\n        #     if isinstance(output, torch.Tensor):\n        #         output = output.to(input_device)\n        #     else:\n        #         output = [x.to(input_device) for x in output]\n        return output\n\n    def get_output_dim(self):\n        return self.transformer.config.hidden_size\n\n    def get_device(self):\n        device: torch.device = next(self.parameters()).device\n        return device\n\n\nclass ContextualWordEmbedding(Embedding, AutoConfigurable):\n    def __init__(self, field: str,\n                 transformer: str,\n                 average_subwords=False,\n                 scalar_mix: Union[ScalarMixWithDropoutBuilder, int] = None,\n                 word_dropout: Optional[Union[float, Tuple[float, str]]] = None,\n                 max_sequence_length=None,\n                 truncate_long_sequences=False,\n                 cls_is_bos=False,\n                 sep_is_eos=False,\n                 ret_token_span=True,\n                 ret_subtokens=False,\n                 ret_subtokens_group=False,\n                 ret_prefix_mask=False,\n                 ret_raw_hidden_states=False,\n                 transformer_args: Dict[str, Any] = None,\n                 use_fast=True,\n                 do_basic_tokenize=True,\n                 trainable=True) -> None:\n        \"\"\"A contextual word embedding builder which builds a\n        :class:`~hanlp.layers.embeddings.contextual_word_embedding.ContextualWordEmbeddingModule` and a\n        :class:`~hanlp.transform.transformer_tokenizer.TransformerSequenceTokenizer`.\n\n        Args:\n            field: The field to work on. Usually some token fields.\n            transformer:  An identifier of a ``PreTrainedModel``.\n            average_subwords: ``True`` to average subword representations.\n            scalar_mix: Layer attention.\n            word_dropout: Dropout rate of randomly replacing a subword with MASK.\n            max_sequence_length: The maximum sequence length. Sequence longer than this will be handled by sliding\n                window.\n            truncate_long_sequences: ``True`` to return hidden states of each layer.\n            cls_is_bos: ``True`` means the first token of input is treated as [CLS] no matter what its surface form is.\n                        ``False`` (default) means the first token is not [CLS], it will have its own embedding other than\n                        the embedding of [CLS].\n            sep_is_eos: ``True`` means the last token of input is [SEP].\n                        ``False`` means it's not but [SEP] will be appended,\n                        ``None`` means it dependents on `input[-1] == [EOS]`.\n            ret_token_span: ``True`` to return span of each token measured by subtoken offsets.\n            ret_subtokens: ``True`` to return list of subtokens belonging to each token.\n            ret_subtokens_group: ``True`` to return list of offsets of subtokens belonging to each token.\n            ret_prefix_mask: ``True`` to generate a mask where each non-zero element corresponds to a prefix of a token.\n            ret_raw_hidden_states: ``True`` to return hidden states of each layer.\n            transformer_args: Extra arguments passed to the transformer.\n            use_fast: Whether or not to try to load the fast version of the tokenizer.\n            do_basic_tokenize: Whether to do basic tokenization before wordpiece.\n            trainable: ``False`` to use static embeddings.\n        \"\"\"\n        super().__init__()\n        self.truncate_long_sequences = truncate_long_sequences\n        self.transformer_args = transformer_args\n        self.trainable = trainable\n        self.ret_subtokens_group = ret_subtokens_group\n        self.ret_subtokens = ret_subtokens\n        self.ret_raw_hidden_states = ret_raw_hidden_states\n        self.sep_is_eos = sep_is_eos\n        self.cls_is_bos = cls_is_bos\n        self.max_sequence_length = max_sequence_length\n        self.word_dropout = word_dropout\n        self.scalar_mix = scalar_mix\n        self.average_subwords = average_subwords\n        self.transformer = transformer\n        self.field = field\n        self._transformer_tokenizer = AutoTokenizer_.from_pretrained(self.transformer,\n                                                                     use_fast=use_fast,\n                                                                     do_basic_tokenize=do_basic_tokenize)\n        self._tokenizer_transform = TransformerSequenceTokenizer(self._transformer_tokenizer,\n                                                                 field,\n                                                                 truncate_long_sequences=truncate_long_sequences,\n                                                                 ret_prefix_mask=ret_prefix_mask,\n                                                                 ret_token_span=ret_token_span,\n                                                                 cls_is_bos=cls_is_bos,\n                                                                 sep_is_eos=sep_is_eos,\n                                                                 ret_subtokens=ret_subtokens,\n                                                                 ret_subtokens_group=ret_subtokens_group,\n                                                                 max_seq_length=self.max_sequence_length\n                                                                 )\n\n    def transform(self, **kwargs) -> TransformerSequenceTokenizer:\n        return self._tokenizer_transform\n\n    def module(self, training=True, **kwargs) -> Optional[nn.Module]:\n        return ContextualWordEmbeddingModule(self.field,\n                                             self.transformer,\n                                             self._transformer_tokenizer,\n                                             self.average_subwords,\n                                             self.scalar_mix,\n                                             self.word_dropout,\n                                             self.max_sequence_length,\n                                             self.ret_raw_hidden_states,\n                                             self.transformer_args,\n                                             self.trainable,\n                                             training=training)\n\n    def get_output_dim(self):\n        config = AutoConfig_.from_pretrained(self.transformer)\n        return config.hidden_size\n\n    def get_tokenizer(self):\n        return self._transformer_tokenizer\n\n\ndef find_transformer(embed: nn.Module):\n    if isinstance(embed, ContextualWordEmbeddingModule):\n        return embed\n    if isinstance(embed, nn.ModuleList):\n        for child in embed:\n            found = find_transformer(child)\n            if found:\n                return found\n"
  },
  {
    "path": "hanlp/layers/embeddings/embedding.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-02 13:04\nfrom abc import ABC, abstractmethod\nfrom typing import Callable, List, Optional, Iterable\n\nimport torch\nfrom torch import nn\nfrom torch.nn import Module\n\nfrom hanlp_common.configurable import AutoConfigurable\nfrom hanlp.common.transform import TransformList\nfrom hanlp.layers.dropout import IndependentDropout\n\n\nclass EmbeddingDim(ABC):\n    @property\n    @abstractmethod\n    def embedding_dim(self) -> int:\n        return -1\n\n    def get_output_dim(self) -> int:\n        return self.embedding_dim\n\n\nclass Embedding(AutoConfigurable, ABC):\n\n    def __init__(self) -> None:\n        \"\"\"\n        Base class for embedding builders.\n        \"\"\"\n        super().__init__()\n\n    def transform(self, **kwargs) -> Optional[Callable]:\n        \"\"\"Build a transform function for this embedding.\n\n        Args:\n            **kwargs: Containing vocabs, training etc. Not finalized for now.\n\n        Returns:\n            A transform function.\n        \"\"\"\n        return None\n\n    def module(self, **kwargs) -> Optional[nn.Module]:\n        \"\"\"Build a module for this embedding.\n\n        Args:\n            **kwargs: Containing vocabs, training etc. Not finalized for now.\n\n        Returns:\n            A module.\n        \"\"\"\n        return None\n\n\nclass ConcatModuleList(nn.ModuleList, EmbeddingDim):\n\n    def __init__(self, *modules: Optional[Iterable[Module]], dropout=None) -> None:\n        \"\"\"A ``nn.ModuleList`` to bundle several embeddings modules.\n\n        Args:\n            *modules: Embedding layers.\n            dropout: Dropout applied on the concatenated embedding.\n        \"\"\"\n        super().__init__(*modules)\n        if dropout:\n            dropout = IndependentDropout(p=dropout)\n        self.dropout = dropout\n\n    @property\n    def embedding_dim(self) -> int:\n        return sum(embed.embedding_dim for embed in self)\n\n    def get_output_dim(self) -> int:\n        return sum(embed.get_output_dim() for embed in self)\n\n    # noinspection PyMethodOverriding\n    def forward(self, batch: dict, **kwargs):\n        embeds = [embed(batch, **kwargs) for embed in self.embeddings]\n        if self.dropout:\n            embeds = self.dropout(*embeds)\n        return torch.cat(embeds, -1)\n\n    @property\n    def embeddings(self):\n        embeddings = [x for x in self]\n        if self.dropout:\n            embeddings.remove(self.dropout)\n        return embeddings\n\n\nclass EmbeddingList(Embedding):\n    def __init__(self, *embeddings_, embeddings: dict = None, dropout=None) -> None:\n        \"\"\"An embedding builder to bundle several embedding builders.\n\n        Args:\n            *embeddings_: A list of embedding builders.\n            embeddings: Deserialization for a dict of embedding builders.\n            dropout: Dropout applied on the concatenated embedding.\n        \"\"\"\n        # noinspection PyTypeChecker\n        self.dropout = dropout\n        self._embeddings: List[Embedding] = list(embeddings_)\n        if embeddings:\n            for each in embeddings:\n                if isinstance(each, dict):\n                    each = AutoConfigurable.from_config(each)\n                self._embeddings.append(each)\n        self.embeddings = [e.config for e in self._embeddings]\n\n    def transform(self, **kwargs):\n        transforms = [e.transform(**kwargs) for e in self._embeddings]\n        transforms = [t for t in transforms if t]\n        return TransformList(*transforms)\n\n    def module(self, **kwargs):\n        modules = [e.module(**kwargs) for e in self._embeddings]\n        modules = [m for m in modules if m]\n        return ConcatModuleList(modules, dropout=self.dropout)\n\n    def to_list(self):\n        return self._embeddings\n\n\ndef find_embedding_by_class(embed: Embedding, cls):\n    if isinstance(embed, cls):\n        return embed\n    if isinstance(embed, EmbeddingList):\n        for child in embed.to_list():\n            found = find_embedding_by_class(child, cls)\n            if found:\n                return found\n"
  },
  {
    "path": "hanlp/layers/embeddings/fast_text.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-27 15:06\nimport logging\nimport os\nimport sys\nfrom typing import Optional, Callable\n\nimport fasttext\nimport torch\nfrom torch import nn\nfrom torch.nn.utils.rnn import pad_sequence\n\nfrom hanlp_common.configurable import AutoConfigurable\nfrom torch.utils.data import DataLoader\n\nfrom hanlp.common.dataset import PadSequenceDataLoader, TransformableDataset\nfrom hanlp.common.torch_component import TorchComponent\nfrom hanlp.common.transform import EmbeddingNamedTransform\nfrom hanlp.common.vocab import Vocab\nfrom hanlp.layers.embeddings.embedding import Embedding\nfrom hanlp.utils.io_util import get_resource, stdout_redirected\nfrom hanlp.utils.log_util import flash\n\n\nclass FastTextTransform(EmbeddingNamedTransform):\n    def __init__(self, filepath: str, src, dst=None, **kwargs) -> None:\n        if not dst:\n            dst = src + '_fasttext'\n        self.filepath = filepath\n        flash(f'Loading fasttext model {filepath} [blink][yellow]...[/yellow][/blink]')\n        filepath = get_resource(filepath)\n        with stdout_redirected(to=os.devnull, stdout=sys.stderr):\n            self._model = fasttext.load_model(filepath)\n        flash('')\n        output_dim = self._model['king'].size\n        super().__init__(output_dim, src, dst)\n\n    def __call__(self, sample: dict):\n        word = sample[self.src]\n        if isinstance(word, str):\n            vector = self.embed(word)\n        else:\n            vector = torch.stack([self.embed(each) for each in word])\n        sample[self.dst] = vector\n        return sample\n\n    def embed(self, word: str):\n        return torch.tensor(self._model[word])\n\n\nclass SelectFromBatchModule(torch.nn.Module):\n    def __init__(self, key) -> None:\n        super().__init__()\n        self.key = key\n\n    def __call__(self, batch: dict, mask=None, **kwargs):\n        return batch[self.key]\n\n\nclass FastTextEmbeddingModule(SelectFromBatchModule):\n\n    def __init__(self, key, embedding_dim: int) -> None:\n        \"\"\"An embedding layer for fastText (:cite:`bojanowski2017enriching`).\n\n        Args:\n            key: Field name.\n            embedding_dim: Size of this embedding layer\n        \"\"\"\n        super().__init__(key)\n        self.embedding_dim = embedding_dim\n\n    def __call__(self, batch: dict, mask=None, **kwargs):\n        outputs = super().__call__(batch, **kwargs)\n        outputs = pad_sequence(outputs, True, 0)\n        if mask is not None:\n            outputs = outputs.to(mask.device)\n        return outputs\n\n    def __repr__(self):\n        s = self.__class__.__name__ + '('\n        s += f'key={self.key}, embedding_dim={self.embedding_dim}'\n        s += ')'\n        return s\n\n    def get_output_dim(self):\n        return self.embedding_dim\n\n\nclass FastTextEmbedding(Embedding, AutoConfigurable):\n    def __init__(self, src: str, filepath: str) -> None:\n        \"\"\"An embedding layer builder for fastText (:cite:`bojanowski2017enriching`).\n\n        Args:\n            src: Field name.\n            filepath: Filepath to pretrained fastText embeddings.\n        \"\"\"\n        super().__init__()\n        self.src = src\n        self.filepath = filepath\n        self._fasttext = FastTextTransform(self.filepath, self.src)\n\n    def transform(self, **kwargs) -> Optional[Callable]:\n        return self._fasttext\n\n    def module(self, **kwargs) -> Optional[nn.Module]:\n        return FastTextEmbeddingModule(self._fasttext.dst, self._fasttext.output_dim)\n\n\nclass FastTextDataset(TransformableDataset):\n\n    def load_file(self, filepath: str):\n        raise NotImplementedError('Not supported.')\n\n\nclass FastTextEmbeddingComponent(TorchComponent):\n    def __init__(self, **kwargs) -> None:\n        \"\"\" Toy example of Word2VecEmbedding. It simply returns the embedding of a given word\n\n        Args:\n            **kwargs:\n        \"\"\"\n        super().__init__(**kwargs)\n\n    def build_dataloader(self, data, shuffle=False, device=None, logger: logging.Logger = None,\n                         **kwargs) -> DataLoader:\n        embed: FastTextEmbedding = self.config.embed\n        dataset = FastTextDataset([{'token': data}], transform=embed.transform())\n        return PadSequenceDataLoader(dataset, device=device)\n\n    def build_optimizer(self, **kwargs):\n        raise NotImplementedError('Not supported.')\n\n    def build_criterion(self, **kwargs):\n        raise NotImplementedError('Not supported.')\n\n    def build_metric(self, **kwargs):\n        raise NotImplementedError('Not supported.')\n\n    def execute_training_loop(self, trn: DataLoader, dev: DataLoader, epochs, criterion, optimizer, metric, save_dir,\n                              logger: logging.Logger, devices, ratio_width=None, **kwargs):\n        raise NotImplementedError('Not supported.')\n\n    def fit_dataloader(self, trn: DataLoader, criterion, optimizer, metric, logger: logging.Logger, **kwargs):\n        raise NotImplementedError('Not supported.')\n\n    def evaluate_dataloader(self, data: DataLoader, criterion: Callable, metric=None, output=False, **kwargs):\n        raise NotImplementedError('Not supported.')\n\n    def load_vocabs(self, save_dir, filename='vocabs.json'):\n        pass\n\n    def load_weights(self, save_dir, filename='model.pt', **kwargs):\n        pass\n\n    def build_model(self, training=True, **kwargs) -> torch.nn.Module:\n        embed: FastTextEmbedding = self.config.embed\n        return embed.module()\n\n    def predict(self, data: str, **kwargs):\n        dataloader = self.build_dataloader(data, device=self.device)\n        for batch in dataloader:  # It's a toy so doesn't really do batching\n            return self.model(batch)[0]\n\n    @property\n    def devices(self):\n        return [torch.device('cpu')]\n"
  },
  {
    "path": "hanlp/layers/embeddings/fast_text_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-10-29 13:14\nimport os\nimport sys\n\nimport numpy as np\nimport tensorflow as tf\nfrom tensorflow.python.keras.utils import tf_utils\n\nfrom hanlp_common.constant import PAD\nfrom hanlp.utils.io_util import get_resource, stdout_redirected\nfrom hanlp.utils.log_util import logger\nfrom hanlp.utils.tf_util import hanlp_register\n\n\n@hanlp_register\nclass FastTextEmbeddingTF(tf.keras.layers.Embedding):\n\n    def __init__(self, filepath: str, padding=PAD, name=None, **kwargs):\n        import fasttext\n        self.padding = padding.encode('utf-8')\n        self.filepath = filepath\n        filepath = get_resource(filepath)\n        assert os.path.isfile(filepath), f'Resolved path {filepath} is not a file'\n        logger.debug('Loading fasttext model from [{}].'.format(filepath))\n        # fasttext print a blank line here\n        with stdout_redirected(to=os.devnull, stdout=sys.stderr):\n            self.model = fasttext.load_model(filepath)\n        kwargs.pop('input_dim', None)\n        kwargs.pop('output_dim', None)\n        kwargs.pop('mask_zero', None)\n        if not name:\n            name = os.path.splitext(os.path.basename(filepath))[0]\n        super().__init__(input_dim=len(self.model.words), output_dim=self.model['king'].size,\n                         mask_zero=padding is not None, trainable=False, dtype=tf.string, name=name, **kwargs)\n        embed_fn = np.frompyfunc(self.embed, 1, 1)\n        # vf = np.vectorize(self.embed, otypes=[np.ndarray])\n        self._embed_np = embed_fn\n\n    def embed(self, word):\n        return self.model[word]\n\n    def embed_np(self, words: np.ndarray):\n        output = self._embed_np(words)\n        if self.mask_zero:\n            mask = words != self.padding\n            output *= mask\n            output = np.stack(output.reshape(-1)).reshape(list(words.shape) + [self.output_dim])\n            return output, tf.constant(mask)\n        else:\n            output = np.stack(output.reshape(-1)).reshape(list(words.shape) + [self.output_dim])\n            return output\n\n    @tf_utils.shape_type_conversion\n    def build(self, input_shape):\n        self.built = True\n\n    @tf_utils.shape_type_conversion\n    def compute_output_shape(self, input_shape):\n        return input_shape + (self.output_dim,)\n\n    def call(self, inputs: tf.Tensor):\n        if isinstance(inputs, list):\n            inputs = inputs[0]\n        if not hasattr(inputs, 'numpy'):  # placeholder tensor\n            inputs = tf.expand_dims(inputs, axis=-1)\n            inputs = tf.tile(inputs, [1] * (len(inputs.shape) - 1) + [self.output_dim])\n            inputs = tf.zeros_like(inputs, dtype=tf.float32)\n            return inputs\n            # seq_len = inputs.shape[-1]\n            # if not seq_len:\n            #     seq_len = 1\n            # return tf.zeros([1, seq_len, self.output_dim])\n        if self.mask_zero:\n            outputs, masks = self.embed_np(inputs.numpy())\n            outputs = tf.constant(outputs)\n            outputs._keras_mask = masks\n        else:\n            outputs = self.embed_np(inputs.numpy())\n            outputs = tf.constant(outputs)\n        return outputs\n\n    def compute_mask(self, inputs, mask=None):\n        if not self.mask_zero:\n            return None\n        return tf.not_equal(inputs, self.padding)\n\n    def get_config(self):\n        config = {\n            'filepath': self.filepath,\n            'padding': self.padding.decode('utf-8')\n        }\n        base_config = super(FastTextEmbeddingTF, self).get_config()\n        for junk in 'embeddings_initializer' \\\n                , 'batch_input_shape' \\\n                , 'embeddings_regularizer' \\\n                , 'embeddings_constraint' \\\n                , 'activity_regularizer' \\\n                , 'trainable' \\\n                , 'input_length' \\\n                :\n            base_config.pop(junk)\n        return dict(list(base_config.items()) + list(config.items()))\n"
  },
  {
    "path": "hanlp/layers/embeddings/util.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-09 15:45\nfrom typing import Union\n\nimport torch\nfrom torch import nn\n\nfrom hanlp.common.vocab import Vocab\nfrom hanlp.utils.init_util import embedding_uniform\nfrom hanlp.utils.torch_util import load_word2vec, load_word2vec_as_vocab_tensor\n\n\ndef index_word2vec_with_vocab(filepath: str,\n                              vocab: Vocab,\n                              extend_vocab=True,\n                              unk=None,\n                              lowercase=False,\n                              init='uniform',\n                              normalize=None) -> torch.Tensor:\n    \"\"\"\n\n    Args:\n        filepath: The path to pretrained embedding.\n        vocab: The vocabulary from training set.\n        extend_vocab: Unlock vocabulary of training set to add those tokens in pretrained embedding file.\n        unk: UNK token.\n        lowercase: Convert words in pretrained embeddings into lowercase.\n        init: Indicate which initialization to use for oov tokens.\n        normalize: ``True`` or a method to normalize the embedding matrix.\n\n    Returns:\n        An embedding matrix.\n\n    \"\"\"\n    pret_vocab, pret_matrix = load_word2vec_as_vocab_tensor(filepath)\n    if unk and unk in pret_vocab:\n        pret_vocab[vocab.safe_unk_token] = pret_vocab.pop(unk)\n    if extend_vocab:\n        vocab.unlock()\n        for word in pret_vocab:\n            vocab.get_idx(word.lower() if lowercase else word)\n    vocab.lock()\n    ids = []\n\n    unk_id_offset = 0\n    for word, idx in vocab.token_to_idx.items():\n        word_id = pret_vocab.get(word, None)\n        # Retry lower case\n        if word_id is None:\n            word_id = pret_vocab.get(word.lower(), None)\n        if word_id is None:\n            word_id = len(pret_vocab) + unk_id_offset\n            unk_id_offset += 1\n        ids.append(word_id)\n    if unk_id_offset:\n        unk_embeds = torch.zeros(unk_id_offset, pret_matrix.size(1))\n        if init and init != 'zeros':\n            if init == 'uniform':\n                init = embedding_uniform\n            else:\n                raise ValueError(f'Unsupported init {init}')\n            unk_embeds = init(unk_embeds)\n        pret_matrix = torch.cat([pret_matrix, unk_embeds])\n    ids = torch.LongTensor(ids)\n    embedding = pret_matrix.index_select(0, ids)\n    if normalize == 'norm':\n        embedding /= (torch.norm(embedding, dim=1, keepdim=True) + 1e-12)\n    elif normalize == 'l2':\n        embedding = torch.nn.functional.normalize(embedding, p=2, dim=1)\n    elif normalize == 'std':\n        embedding /= torch.std(embedding)\n    else:\n        raise ValueError(f'Unsupported normalization method {normalize}')\n    return embedding\n\n\ndef build_word2vec_with_vocab(embed: Union[str, int],\n                              vocab: Vocab,\n                              extend_vocab=True,\n                              unk=None,\n                              lowercase=False,\n                              trainable=False,\n                              init='zeros',\n                              normalize=None) -> nn.Embedding:\n    \"\"\"Build word2vec embedding and a vocab.\n\n    Args:\n        embed:\n        vocab: The vocabulary from training set.\n        extend_vocab: Unlock vocabulary of training set to add those tokens in pretrained embedding file.\n        unk: UNK token.\n        lowercase: Convert words in pretrained embeddings into lowercase.\n        trainable: ``False`` to use static embeddings.\n        init: Indicate which initialization to use for oov tokens.\n        normalize: ``True`` or a method to normalize the embedding matrix.\n\n    Returns:\n        An embedding matrix.\n\n    \"\"\"\n    if isinstance(embed, str):\n        embed = index_word2vec_with_vocab(embed, vocab, extend_vocab, unk, lowercase, init, normalize)\n        embed = nn.Embedding.from_pretrained(embed, freeze=not trainable, padding_idx=vocab.pad_idx)\n        return embed\n    elif isinstance(embed, int):\n        embed = nn.Embedding(len(vocab), embed, padding_idx=vocab.pad_idx)\n        return embed\n    else:\n        raise ValueError(f'Unsupported parameter type: {embed}')\n"
  },
  {
    "path": "hanlp/layers/embeddings/util_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-09 15:46\nfrom typing import Union\n\nimport tensorflow as tf\n\nfrom hanlp.common.transform_tf import Transform\nfrom hanlp.common.vocab_tf import VocabTF\nfrom hanlp.layers.embeddings.char_cnn_tf import CharCNNEmbeddingTF\nfrom hanlp.layers.embeddings.char_rnn_tf import CharRNNEmbeddingTF\nfrom hanlp.layers.embeddings.concat_embedding import ConcatEmbedding\nfrom hanlp.layers.embeddings.contextual_string_embedding_tf import ContextualStringEmbeddingTF\nfrom hanlp.layers.embeddings.fast_text_tf import FastTextEmbeddingTF\nfrom hanlp.layers.embeddings.word2vec_tf import Word2VecEmbeddingTF, StringWord2VecEmbeddingTF, Word2VecEmbeddingV1\n\n_upgrade = tf.keras.utils.get_custom_objects()\nfor k, v in list(_upgrade.items()):\n    if k.startswith('HanLP>') and k.endswith('TF'):\n        _upgrade[k[:-2]] = v\n\n\ndef build_embedding(embeddings: Union[str, int, dict], word_vocab: VocabTF, transform: Transform):\n    if not embeddings:\n        return None\n    config = transform.config\n    if isinstance(embeddings, int):\n        embeddings = tf.keras.layers.Embedding(input_dim=len(word_vocab), output_dim=embeddings,\n                                               trainable=True, mask_zero=True)\n        config.embedding_trainable = True\n    elif isinstance(embeddings, dict):\n        # Upgrade to 2.1\n        embed_name = embeddings['class_name'].split('>')[-1]\n        if embeddings['class_name'].startswith('HanLP>') and not embeddings['class_name'].endswith('TF'):\n            embed_name += 'TF'\n        # Embeddings need vocab\n        if embed_name in (Word2VecEmbeddingTF.__name__, StringWord2VecEmbeddingTF.__name__):\n            # Vocab won't present in the dict\n            embeddings['config']['vocab'] = word_vocab\n        elif embed_name in (CharRNNEmbeddingTF.__name__, CharCNNEmbeddingTF.__name__):\n            embeddings['config']['word_vocab'] = word_vocab\n            embeddings['config']['char_vocab'] = transform.char_vocab\n            transform.map_x = False\n        layer: tf.keras.layers.Embedding = tf.keras.utils.deserialize_keras_object(embeddings)\n        # Embedding specific configuration\n        if layer.__class__.__name__ in ('FastTextEmbedding', 'FastTextEmbeddingTF'):\n            config.run_eagerly = True  # fasttext can only run in eager mode\n            config.embedding_trainable = False\n            transform.map_x = False  # fasttext accept string instead of int\n        return layer\n    elif isinstance(embeddings, list):\n        if embeddings_require_string_input(embeddings):\n            # those embeddings require string as input\n            transform.map_x = False\n            # use the string version of Word2VecEmbedding instead\n            for embed in embeddings:\n                if embed['class_name'].split('>')[-1] == Word2VecEmbeddingTF.__name__:\n                    embed['class_name'] = 'HanLP>' + StringWord2VecEmbeddingTF.__name__\n        return ConcatEmbedding(*[build_embedding(embed, word_vocab, transform) for embed in embeddings])\n    else:\n        assert isinstance(embeddings, str), 'embedding should be str or int or dict'\n        # word_vocab.unlock()\n        embeddings = Word2VecEmbeddingV1(path=embeddings, vocab=word_vocab,\n                                         trainable=config.get('embedding_trainable', False))\n        embeddings = embeddings.array_ks\n    return embeddings\n\n\ndef any_embedding_in(embeddings, *cls):\n    names = set(x.__name__ for x in cls)\n    names.update(list(x[:-2] for x in names if x.endswith('TF')))\n    for embed in embeddings:\n        if isinstance(embed, dict) and embed['class_name'].split('>')[-1] in names:\n            return True\n    return False\n\n\ndef embeddings_require_string_input(embeddings):\n    if not isinstance(embeddings, list):\n        embeddings = [embeddings]\n    return any_embedding_in(embeddings, CharRNNEmbeddingTF, CharCNNEmbeddingTF, FastTextEmbeddingTF,\n                            ContextualStringEmbeddingTF)\n\n\ndef embeddings_require_char_input(embeddings):\n    if not isinstance(embeddings, list):\n        embeddings = [embeddings]\n    return any_embedding_in(embeddings, CharRNNEmbeddingTF, CharCNNEmbeddingTF, ContextualStringEmbeddingTF)\n"
  },
  {
    "path": "hanlp/layers/embeddings/word2vec.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-09 13:38\nimport logging\nimport math\nimport os.path\nfrom typing import Optional, Callable, Union, List, Dict\n\nimport torch\nfrom hanlp_common.configurable import AutoConfigurable\nfrom hanlp_common.constant import HANLP_VERBOSE\nfrom hanlp_trie.trie import Trie\nfrom torch import nn\nfrom torch.utils.data import DataLoader\n\nfrom hanlp.common.dataset import TransformableDataset, PadSequenceDataLoader\nfrom hanlp.common.torch_component import TorchComponent\nfrom hanlp.common.transform import VocabDict\nfrom hanlp.common.vocab import Vocab\nfrom hanlp.layers.dropout import WordDropout\nfrom hanlp.layers.embeddings.embedding import Embedding, EmbeddingDim\nfrom hanlp.layers.embeddings.util import build_word2vec_with_vocab\nfrom hanlp.utils.log_util import flash\nfrom hanlp.utils.torch_util import load_word2vec_as_vocab_tensor\n\n\nclass Word2VecEmbeddingModule(nn.Module, EmbeddingDim):\n    def __init__(self, field: str, embed: nn.Embedding, word_dropout: WordDropout = None, cpu=False,\n                 second_channel=False, num_tokens_in_trn=None, unk_idx=1) -> None:\n        \"\"\"A word2vec style embedding module which maps a token to its embedding through looking up a pre-defined table.\n\n        Args:\n            field: The field to work on. Usually some token fields.\n            embed: An ``Embedding`` layer.\n            word_dropout: The probability of randomly replacing a token with ``UNK``.\n            cpu: Reside on CPU instead of GPU.\n            second_channel: A trainable second channel for each token, which will be added to pretrained embeddings.\n            num_tokens_in_trn: The number of tokens in training set.\n            unk_idx: The index of ``UNK``.\n        \"\"\"\n        super().__init__()\n        self.cpu = cpu\n        self.field = field\n        self.embed = embed\n        self.word_dropout = word_dropout\n        self.num_tokens_in_trn = num_tokens_in_trn\n        self.unk_idx = unk_idx\n        if second_channel:\n            n_words, n_embed = embed.weight.size()\n            if num_tokens_in_trn:\n                n_words = num_tokens_in_trn\n            second_channel = nn.Embedding(num_embeddings=n_words,\n                                          embedding_dim=n_embed)\n            nn.init.zeros_(second_channel.weight)\n        self.second_channel = second_channel\n\n    def forward(self, batch: dict, **kwargs):\n        x: torch.Tensor = batch[f'{self.field}_id']\n        if self.cpu:\n            device = x.device\n            x = x.cpu()\n        if self.word_dropout:\n            x = self.word_dropout(x)\n        if self.second_channel:\n            ext_mask = x.ge(self.second_channel.num_embeddings)\n            ext_words = x.masked_fill(ext_mask, self.unk_idx)\n            x = self.embed(x) + self.second_channel(ext_words)\n        else:\n            x = self.embed(x)\n        if self.cpu:\n            # noinspection PyUnboundLocalVariable\n            x = x.to(device)\n        return x\n\n    @property\n    def embedding_dim(self) -> int:\n        return self.embed.embedding_dim\n\n    # noinspection PyMethodOverriding\n    # def to(self, device, **kwargs):\n    #     print(self.cpu)\n    #     exit(1)\n    #     if self.cpu:\n    #         return super(Word2VecEmbeddingModule, self).to(-1, **kwargs)\n    #     return super(Word2VecEmbeddingModule, self).to(device, **kwargs)\n\n    def _apply(self, fn):\n\n        if not self.cpu:  # This might block all fn not limiting to moving between devices.\n            return super(Word2VecEmbeddingModule, self)._apply(fn)\n\n\nclass Word2VecEmbedding(Embedding, AutoConfigurable):\n    def __init__(self,\n                 field,\n                 embed: Union[int, str],\n                 extend_vocab=True,\n                 pad=None,\n                 unk=None,\n                 lowercase=False,\n                 trainable=False,\n                 second_channel=False,\n                 word_dropout: float = 0,\n                 normalize=False,\n                 cpu=False,\n                 init='zeros') -> None:\n        \"\"\"A word2vec style embedding builder which maps a token to its embedding through looking up a pre-defined\n        table.\n\n        Args:\n            field: The field to work on. Usually some token fields.\n            embed: A path to pre-trained embedding file or an integer defining the size of randomly initialized\n                embedding.\n            extend_vocab: Unlock vocabulary of training set to add those tokens in pre-trained embedding file.\n            pad: The padding token.\n            unk: The unknown token.\n            lowercase: Convert words in pretrained embeddings into lowercase.\n            trainable: ``False`` to use static embeddings.\n            second_channel: A trainable second channel for each token, which will be added to pretrained embeddings.\n            word_dropout: The probability of randomly replacing a token with ``UNK``.\n            normalize: ``l2`` or ``std`` to normalize the embedding matrix.\n            cpu: Reside on CPU instead of GPU.\n            init: Indicate which initialization to use for oov tokens.\n        \"\"\"\n        super().__init__()\n        self.pad = pad\n        self.second_channel = second_channel\n        self.cpu = cpu\n        self.normalize = normalize\n        self.word_dropout = word_dropout\n        self.init = init\n        self.lowercase = lowercase\n        self.unk = unk\n        self.extend_vocab = extend_vocab\n        self.trainable = trainable\n        self.embed = embed\n        self.field = field\n\n    def module(self, vocabs: VocabDict, **kwargs) -> Optional[nn.Module]:\n        vocab = vocabs[self.field]\n        num_tokens_in_trn = len(vocab)\n        embed = build_word2vec_with_vocab(self.embed,\n                                          vocab,\n                                          self.extend_vocab,\n                                          self.unk,\n                                          self.lowercase,\n                                          self.trainable,\n                                          normalize=self.normalize)\n        if self.word_dropout:\n            assert vocab.unk_token, f'unk_token of vocab {self.field} has to be set in order to ' \\\n                                    f'make use of word_dropout'\n            padding = []\n            if vocab.pad_token:\n                padding.append(vocab.pad_idx)\n            word_dropout = WordDropout(self.word_dropout, vocab.unk_idx, exclude_tokens=padding)\n        else:\n            word_dropout = None\n        return Word2VecEmbeddingModule(self.field, embed, word_dropout=word_dropout, cpu=self.cpu,\n                                       second_channel=self.second_channel, num_tokens_in_trn=num_tokens_in_trn,\n                                       unk_idx=vocab.unk_idx)\n\n    def transform(self, vocabs: VocabDict = None, **kwargs) -> Optional[Callable]:\n        assert vocabs is not None\n        if self.field not in vocabs:\n            vocabs[self.field] = Vocab(pad_token=self.pad, unk_token=self.unk)\n        return super().transform(**kwargs)\n\n\nclass Word2VecDataset(TransformableDataset):\n\n    def load_file(self, filepath: str):\n        raise NotImplementedError('Not supported.')\n\n\nclass Word2VecEmbeddingComponent(TorchComponent):\n\n    def __init__(self, **kwargs) -> None:\n        \"\"\" Toy example of Word2VecEmbedding. It simply returns the embedding of a given word\n\n        Args:\n            **kwargs:\n        \"\"\"\n        super().__init__(**kwargs)\n        self._tokenizer: Trie = None\n\n    def build_dataloader(self, data: List[str], shuffle=False, device=None, logger: logging.Logger = None,\n                         doc2vec=False, batch_size=32, **kwargs) -> DataLoader:\n        dataset = Word2VecDataset([{'token': x} for x in data], transform=self._tokenize if doc2vec else self.vocabs)\n        return PadSequenceDataLoader(dataset, device=device, batch_size=batch_size)\n\n    def build_optimizer(self, **kwargs):\n        raise NotImplementedError('Not supported.')\n\n    def build_criterion(self, **kwargs):\n        raise NotImplementedError('Not supported.')\n\n    def build_metric(self, **kwargs):\n        raise NotImplementedError('Not supported.')\n\n    def execute_training_loop(self, trn: DataLoader, dev: DataLoader, epochs, criterion, optimizer, metric, save_dir,\n                              logger: logging.Logger, devices, ratio_width=None, **kwargs):\n        raise NotImplementedError('Not supported.')\n\n    def fit_dataloader(self, trn: DataLoader, criterion, optimizer, metric, logger: logging.Logger, **kwargs):\n        raise NotImplementedError('Not supported.')\n\n    def evaluate_dataloader(self, data: DataLoader, criterion: Callable, metric=None, output=False, **kwargs):\n        raise NotImplementedError('Not supported.')\n\n    def load_vocabs(self, save_dir, filename='vocabs.json'):\n        self.vocabs['token'] = Vocab()\n\n    def load_weights(self, save_dir, filename='model.pt', **kwargs):\n        pass\n\n    def build_model(self, training=True, **kwargs) -> torch.nn.Module:\n        self._tokenizer = None\n        embed: Word2VecEmbedding = self.config.embed\n        model = embed.module(self.vocabs)\n        return model\n\n    def predict(self, word: str, doc2vec=False, **kwargs):\n        dataloader = self.build_dataloader([word], device=self.device, doc2vec=doc2vec)\n        for batch in dataloader:  # It's a toy so doesn't really do batching\n            embeddings = self.model(batch)[0]\n            if doc2vec:\n                embeddings = embeddings[0].mean(dim=0)\n            return embeddings\n\n    @torch.no_grad()\n    def most_similar(self, words: Union[str, List[str]], topk=10, doc2vec=False, similarity_less_than=None,\n                     batch_size=32) -> Union[Dict[str, float], List[Dict[str, float]]]:\n        \"\"\"Find the `topk` most similar words of a given word or phrase.\n\n        Args:\n            words: A word or phrase or multiple words/phrases.\n            topk: Number of top similar words.\n            doc2vec: Enable doc2vec model for processing OOV and phrases.\n            similarity_less_than: Only return words with a similarity less than this value.\n            batch_size: Number of words or phrases per batch.\n\n        Returns:\n            Similar words and similarities stored in a dict.\n        \"\"\"\n        flat = isinstance(words, str)\n        if flat:\n            words = [words]\n        dataloader = self.build_dataloader(words, device=self.device, doc2vec=doc2vec, batch_size=batch_size)\n        results = []\n        vocab = self.vocabs['token']\n        for batch in dataloader:\n            embeddings = self.model(batch)\n            token_id = batch['token_id']\n            if doc2vec:\n                lens = token_id.count_nonzero(dim=1)\n                embeddings = embeddings.sum(1)\n                embeddings = embeddings / lens.unsqueeze(1)\n                block_word_id = batch['block_word_id']\n                token_is_unk = (lens == 1) & (token_id[:, 0] == vocab.unk_idx)\n            else:\n                block_word_id = token_id\n                token_is_unk = token_id == vocab.unk_idx\n            similarities = torch.nn.functional.cosine_similarity(embeddings.unsqueeze(1), self.model.embed.weight,\n                                                                 dim=-1)\n            if similarity_less_than is not None:\n                similarities[similarities > similarity_less_than] = -math.inf\n            similarities[torch.arange(similarities.size(0), device=self.device), block_word_id] = -math.inf\n            scores, indices = similarities.topk(topk)\n\n            for sc, idx, unk in zip(scores.tolist(), indices.tolist(), token_is_unk.tolist()):\n                results.append(dict() if unk else dict(zip([vocab.idx_to_token[i] for i in idx], sc)))\n        if flat:\n            results = results[0]\n        return results\n\n    def _tokenize(self, sample: dict) -> dict:\n        tokens = sample['token']\n        ids = [idx for b, e, idx in self.tokenizer.parse_longest(tokens)]\n        vocab = self.vocabs['token']\n        if not ids:\n            ids = [vocab.unk_idx]\n        sample['token_id'] = ids\n        sample['block_word_id'] = ids[0] if len(ids) == 1 else vocab.pad_idx\n        return sample\n\n    @property\n    def tokenizer(self):\n        if not self._tokenizer:\n            if HANLP_VERBOSE:\n                flash('Building Trie-based tokenizer for Doc2Vec [blink][yellow]...[/yellow][/blink]')\n            self._tokenizer = Trie(self.vocabs['token'].token_to_idx)\n            if HANLP_VERBOSE:\n                flash('')\n        return self._tokenizer\n\n    def load_config(self, save_dir, filename='config.json', **kwargs):\n        if os.path.isfile(save_dir):\n            self.config.update({'classpath': 'hanlp.layers.embeddings.word2vec.Word2VecEmbeddingComponent',\n                                'embed': Word2VecEmbedding(field='token', embed=save_dir, normalize='l2')})\n            return\n        super().load_config(save_dir, filename, **kwargs)\n\n\nclass GazetterTransform(object):\n    def __init__(self, field, words: dict) -> None:\n        super().__init__()\n        self.field = field\n        self.trie = Trie()\n        for word, idx in words.items():\n            self.trie[word] = idx\n\n    def __call__(self, sample: dict) -> dict:\n        tokens = sample[self.field]\n        lexicons = self.trie.parse(tokens)\n        skips_l2r = [[] for _ in range(len(tokens))]\n        skips_r2l = [[] for _ in range(len(tokens))]\n        for w, i, s, e in lexicons:\n            e = e - 1\n            skips_l2r[e].append((s, w, i))\n            skips_r2l[s].append((e, w, i))\n        for direction, value in zip(['skips_l2r', 'skips_r2l'], [skips_l2r, skips_r2l]):\n            sample[f'{self.field}_{direction}_offset'] = [list(map(lambda x: x[0], p)) for p in value]\n            sample[f'{self.field}_{direction}_id'] = [list(map(lambda x: x[-1], p)) for p in value]\n            sample[f'{self.field}_{direction}_count'] = list(map(len, value))\n        return sample\n\n\nclass GazetteerEmbedding(Embedding, AutoConfigurable):\n    def __init__(self, embed: str, field='char', trainable=False) -> None:\n        self.trainable = trainable\n        self.embed = embed\n        self.field = field\n        vocab, matrix = load_word2vec_as_vocab_tensor(self.embed)\n        ids = []\n        _vocab = {}\n        for word, idx in vocab.items():\n            if len(word) > 1:\n                ids.append(idx)\n                _vocab[word] = len(_vocab)\n        ids = torch.tensor(ids)\n        _matrix = matrix.index_select(0, ids)\n        self._vocab = _vocab\n        self._matrix = _matrix\n\n    def transform(self, **kwargs) -> Optional[Callable]:\n        return GazetterTransform(self.field, self._vocab)\n\n    def module(self, **kwargs) -> Optional[nn.Module]:\n        embed = nn.Embedding.from_pretrained(self._matrix, freeze=not self.trainable)\n        return embed\n\n    @staticmethod\n    def _remove_short_tokens(word2vec):\n        word2vec = dict((w, v) for w, v in word2vec.items() if len(w) > 1)\n        return word2vec\n"
  },
  {
    "path": "hanlp/layers/embeddings/word2vec_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-08-24 21:49\nimport os\nfrom typing import Tuple, Union, List\n\nimport numpy as np\nimport tensorflow as tf\nfrom tensorflow.python.ops import math_ops\n\nfrom hanlp.common.vocab_tf import VocabTF\nfrom hanlp.utils.io_util import get_resource\nfrom hanlp.utils.torch_util import load_word2vec\nfrom hanlp.utils.tf_util import hanlp_register\nfrom hanlp_common.util import DummyContext\n\n\nclass Word2VecEmbeddingV1(tf.keras.layers.Layer):\n    def __init__(self, path: str = None, vocab: VocabTF = None, normalize: bool = False, load_all=True, mask_zero=True,\n                 trainable=False, name=None, dtype=None, dynamic=False, **kwargs):\n        super().__init__(trainable, name, dtype, dynamic, **kwargs)\n        if load_all and vocab and vocab.locked:\n            vocab.unlock()\n        self.vocab, self.array_np = self._load(path, vocab, normalize)\n        self.vocab.lock()\n        self.array_ks = tf.keras.layers.Embedding(input_dim=len(self.vocab), output_dim=self.dim, trainable=trainable,\n                                                  embeddings_initializer=tf.keras.initializers.Constant(self.array_np),\n                                                  mask_zero=mask_zero)\n        self.mask_zero = mask_zero\n        self.supports_masking = mask_zero\n\n    def compute_mask(self, inputs, mask=None):\n        if not self.mask_zero:\n            return None\n\n        return math_ops.not_equal(inputs, self.vocab.pad_idx)\n\n    def call(self, inputs, **kwargs):\n        return self.array_ks(inputs, **kwargs)\n\n    def compute_output_shape(self, input_shape):\n        return input_shape[0], self.dim\n\n    @staticmethod\n    def _load(path, vocab, normalize=False) -> Tuple[VocabTF, Union[np.ndarray, None]]:\n        if not vocab:\n            vocab = VocabTF()\n        if not path:\n            return vocab, None\n        assert vocab.unk_idx is not None\n\n        word2vec, dim = load_word2vec(path)\n        for word in word2vec:\n            vocab.get_idx(word)\n\n        pret_embs = np.zeros(shape=(len(vocab), dim), dtype=np.float32)\n        state = np.random.get_state()\n        np.random.seed(0)\n        bias = np.random.uniform(low=-0.001, high=0.001, size=dim).astype(dtype=np.float32)\n        scale = np.sqrt(3.0 / dim)\n        for word, idx in vocab.token_to_idx.items():\n            vec = word2vec.get(word, None)\n            if vec is None:\n                vec = word2vec.get(word.lower(), None)\n                # if vec is not None:\n                #     vec += bias\n            if vec is None:\n                # vec = np.random.uniform(-scale, scale, [dim])\n                vec = np.zeros([dim], dtype=np.float32)\n            pret_embs[idx] = vec\n        # noinspection PyTypeChecker\n        np.random.set_state(state)\n        return vocab, pret_embs\n\n    @property\n    def size(self):\n        if self.array_np is not None:\n            return self.array_np.shape[0]\n\n    @property\n    def dim(self):\n        if self.array_np is not None:\n            return self.array_np.shape[1]\n\n    @property\n    def shape(self):\n        if self.array_np is None:\n            return None\n        return self.array_np.shape\n\n    def get_vector(self, word: str) -> np.ndarray:\n        assert self.array_np is not None\n        return self.array_np[self.vocab.get_idx_without_add(word)]\n\n    def __getitem__(self, word: Union[str, List, tf.Tensor]) -> np.ndarray:\n        if isinstance(word, str):\n            return self.get_vector(word)\n        elif isinstance(word, list):\n            vectors = np.zeros(shape=(len(word), self.dim))\n            for idx, token in enumerate(word):\n                vectors[idx] = self.get_vector(token)\n            return vectors\n        elif isinstance(word, tf.Tensor):\n            if word.dtype == tf.string:\n                word_ids = self.vocab.token_to_idx_table.lookup(word)\n                return tf.nn.embedding_lookup(self.array_tf, word_ids)\n            elif word.dtype == tf.int32 or word.dtype == tf.int64:\n                return tf.nn.embedding_lookup(self.array_tf, word)\n\n\n@hanlp_register\nclass Word2VecEmbeddingTF(tf.keras.layers.Embedding):\n\n    def __init__(self, filepath: str = None, vocab: VocabTF = None, expand_vocab=True, lowercase=True,\n                 input_dim=None, output_dim=None, unk=None, normalize=False,\n                 embeddings_initializer='VarianceScaling',\n                 embeddings_regularizer=None,\n                 activity_regularizer=None, embeddings_constraint=None, mask_zero=True, input_length=None,\n                 name=None, cpu=True, **kwargs):\n        filepath = get_resource(filepath)\n        word2vec, _output_dim = load_word2vec(filepath)\n        if output_dim:\n            assert output_dim == _output_dim, f'output_dim = {output_dim} does not match {filepath}'\n        output_dim = _output_dim\n        # if the `unk` token exists in the pretrained,\n        # then replace it with a self-defined one, usually the one in word vocab\n        if unk and unk in word2vec:\n            word2vec[vocab.safe_unk_token] = word2vec.pop(unk)\n        if vocab is None:\n            vocab = VocabTF()\n            vocab.update(word2vec.keys())\n        if expand_vocab and vocab.mutable:\n            for word in word2vec:\n                vocab.get_idx(word.lower() if lowercase else word)\n        if input_dim:\n            assert input_dim == len(vocab), f'input_dim = {input_dim} does not match {filepath}'\n        input_dim = len(vocab)\n        # init matrix\n        self._embeddings_initializer = embeddings_initializer\n        embeddings_initializer = tf.keras.initializers.get(embeddings_initializer)\n        with tf.device('cpu:0') if cpu else DummyContext():\n            pret_embs = embeddings_initializer(shape=[input_dim, output_dim]).numpy()\n        # insert to pret_embs\n        for word, idx in vocab.token_to_idx.items():\n            vec = word2vec.get(word, None)\n            # Retry lower case\n            if vec is None and lowercase:\n                vec = word2vec.get(word.lower(), None)\n            if vec is not None:\n                pret_embs[idx] = vec\n        if normalize:\n            pret_embs /= np.std(pret_embs)\n        if not name:\n            name = os.path.splitext(os.path.basename(filepath))[0]\n        super().__init__(input_dim, output_dim, tf.keras.initializers.Constant(pret_embs), embeddings_regularizer,\n                         activity_regularizer, embeddings_constraint, mask_zero, input_length, name=name, **kwargs)\n        self.filepath = filepath\n        self.expand_vocab = expand_vocab\n        self.lowercase = lowercase\n\n    def get_config(self):\n        config = {\n            'filepath': self.filepath,\n            'expand_vocab': self.expand_vocab,\n            'lowercase': self.lowercase,\n        }\n        base_config = super(Word2VecEmbeddingTF, self).get_config()\n        base_config['embeddings_initializer'] = self._embeddings_initializer\n        return dict(list(base_config.items()) + list(config.items()))\n\n\n@hanlp_register\nclass StringWord2VecEmbeddingTF(Word2VecEmbeddingTF):\n\n    def __init__(self, filepath: str = None, vocab: VocabTF = None, expand_vocab=True, lowercase=False, input_dim=None,\n                 output_dim=None, unk=None, normalize=False, embeddings_initializer='VarianceScaling',\n                 embeddings_regularizer=None, activity_regularizer=None, embeddings_constraint=None, mask_zero=True,\n                 input_length=None, name=None, **kwargs):\n        if vocab is None:\n            vocab = VocabTF()\n        self.vocab = vocab\n        super().__init__(filepath, vocab, expand_vocab, lowercase, input_dim, output_dim, unk, normalize,\n                         embeddings_initializer, embeddings_regularizer, activity_regularizer, embeddings_constraint,\n                         mask_zero, input_length, name, **kwargs)\n\n    def call(self, inputs):\n        assert inputs.dtype == tf.string, \\\n            f'Expect tf.string but got tf.{inputs.dtype.name}. {inputs}' \\\n            f'Please pass tf.{inputs.dtype.name} in.'\n        inputs = self.vocab.lookup(inputs)\n        # inputs._keras_mask = tf.not_equal(inputs, self.vocab.pad_idx)\n        return super().call(inputs)\n\n    def compute_mask(self, inputs, mask=None):\n        if not self.mask_zero:\n            return None\n        return tf.not_equal(inputs, self.vocab.pad_token)\n"
  },
  {
    "path": "hanlp/layers/feed_forward.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-07-06 14:37\nfrom typing import Union, List\n\nfrom hanlp.layers import feedforward\n\nfrom hanlp.common.structure import ConfigTracker\n\n\nclass FeedForward(feedforward.FeedForward, ConfigTracker):\n    def __init__(self, input_dim: int, num_layers: int, hidden_dims: Union[int, List[int]],\n                 activations: Union[str, List[str]], dropout: Union[float, List[float]] = 0.0) -> None:\n        super().__init__(input_dim, num_layers, hidden_dims, activations, dropout)\n        ConfigTracker.__init__(self, locals())\n"
  },
  {
    "path": "hanlp/layers/feedforward.py",
    "content": "\"\"\"\nA feed-forward neural network.\n\"\"\"\nfrom typing import List, Union\n\nimport torch\nfrom hanlp.utils.torch_util import activation_from_name\n\n\nclass FeedForward(torch.nn.Module):\n    \"\"\"\n    This `Module` is a feed-forward neural network, just a sequence of `Linear` layers with\n    activation functions in between.\n\n    # Parameters\n\n    input_dim : `int`, required\n        The dimensionality of the input.  We assume the input has shape `(batch_size, input_dim)`.\n    num_layers : `int`, required\n        The number of `Linear` layers to apply to the input.\n    hidden_dims : `Union[int, List[int]]`, required\n        The output dimension of each of the `Linear` layers.  If this is a single `int`, we use\n        it for all `Linear` layers.  If it is a `List[int]`, `len(hidden_dims)` must be\n        `num_layers`.\n    activations : `Union[Activation, List[Activation]]`, required\n        The activation function to use after each `Linear` layer.  If this is a single function,\n        we use it after all `Linear` layers.  If it is a `List[Activation]`,\n        `len(activations)` must be `num_layers`. Activation must have torch.nn.Module type.\n    dropout : `Union[float, List[float]]`, optional (default = `0.0`)\n        If given, we will apply this amount of dropout after each layer.  Semantics of `float`\n        versus `List[float]` is the same as with other parameters.\n\n    # Examples\n\n    ```python\n    FeedForward(124, 2, [64, 32], torch.nn.ReLU(), 0.2)\n    #> FeedForward(\n    #>   (_activations): ModuleList(\n    #>     (0): ReLU()\n    #>     (1): ReLU()\n    #>   )\n    #>   (_linear_layers): ModuleList(\n    #>     (0): Linear(in_features=124, out_features=64, bias=True)\n    #>     (1): Linear(in_features=64, out_features=32, bias=True)\n    #>   )\n    #>   (_dropout): ModuleList(\n    #>     (0): Dropout(p=0.2, inplace=False)\n    #>     (1): Dropout(p=0.2, inplace=False)\n    #>   )\n    #> )\n    ```\n    \"\"\"\n\n    def __init__(\n            self,\n            input_dim: int,\n            num_layers: int,\n            hidden_dims: Union[int, List[int]],\n            activations: Union[str, List[str]],\n            dropout: Union[float, List[float]] = 0.0,\n    ) -> None:\n\n        super().__init__()\n        if not isinstance(hidden_dims, list):\n            hidden_dims = [hidden_dims] * num_layers  # type: ignore\n        if not isinstance(activations, list):\n            activations = [activations] * num_layers  # type: ignore\n        activations = [activation_from_name(a)() for a in activations]\n        if not isinstance(dropout, list):\n            dropout = [dropout] * num_layers  # type: ignore\n        if len(hidden_dims) != num_layers:\n            raise ValueError(\n                \"len(hidden_dims) (%d) != num_layers (%d)\" % (len(hidden_dims), num_layers)\n            )\n        if len(activations) != num_layers:\n            raise ValueError(\n                \"len(activations) (%d) != num_layers (%d)\" % (len(activations), num_layers)\n            )\n        if len(dropout) != num_layers:\n            raise ValueError(\n                \"len(dropout) (%d) != num_layers (%d)\" % (len(dropout), num_layers)\n            )\n        self._activations = torch.nn.ModuleList(activations)\n        input_dims = [input_dim] + hidden_dims[:-1]\n        linear_layers = []\n        for layer_input_dim, layer_output_dim in zip(input_dims, hidden_dims):\n            linear_layers.append(torch.nn.Linear(layer_input_dim, layer_output_dim))\n        self._linear_layers = torch.nn.ModuleList(linear_layers)\n        dropout_layers = [torch.nn.Dropout(p=value) for value in dropout]\n        self._dropout = torch.nn.ModuleList(dropout_layers)\n        self._output_dim = hidden_dims[-1]\n        self.input_dim = input_dim\n\n    def get_output_dim(self):\n        return self._output_dim\n\n    def get_input_dim(self):\n        return self.input_dim\n\n    def forward(self, inputs: torch.Tensor) -> torch.Tensor:\n\n        output = inputs\n        for layer, activation, dropout in zip(\n                self._linear_layers, self._activations, self._dropout\n        ):\n            output = dropout(activation(layer(output)))\n        return output\n"
  },
  {
    "path": "hanlp/layers/scalar_mix.py",
    "content": "# This file is modified from udify, which is licensed under the MIT license:\n# MIT License\n#\n# Copyright (c) 2019 Dan Kondratyuk\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\n\"\"\"\nThe dot-product \"Layer Attention\" that is applied to the layers of BERT, along with layer dropout to reduce overfitting\n\"\"\"\n\nfrom typing import List, Tuple\n\nimport torch\nfrom torch.nn import ParameterList, Parameter\n\nfrom hanlp.common.structure import ConfigTracker\n\n\nclass ScalarMixWithDropout(torch.nn.Module):\n    \"\"\"Computes a parameterised scalar mixture of N tensors, ``mixture = gamma * sum(s_k * tensor_k)``\n    where ``s = softmax(w)``, with ``w`` and ``gamma`` scalar parameters.\n    \n    If ``do_layer_norm=True`` then apply layer normalization to each tensor before weighting.\n    \n    If ``dropout > 0``, then for each scalar weight, adjust its softmax weight mass to 0 with\n    the dropout probability (i.e., setting the unnormalized weight to -inf). This effectively\n    should redistribute dropped probability mass to all other weights.\n\n    Args:\n\n    Returns:\n\n    \"\"\"\n\n    def __init__(self,\n                 mixture_range: Tuple[int, int],\n                 do_layer_norm: bool = False,\n                 initial_scalar_parameters: List[float] = None,\n                 trainable: bool = True,\n                 dropout: float = None,\n                 dropout_value: float = -1e20,\n                 **kwargs) -> None:\n        super(ScalarMixWithDropout, self).__init__()\n        self.mixture_range = mixture_range\n        mixture_size = mixture_range[1] - mixture_range[0]\n        self.mixture_size = mixture_size\n        self.do_layer_norm = do_layer_norm\n        self.dropout = dropout\n\n        if initial_scalar_parameters is None:\n            initial_scalar_parameters = [0.0] * mixture_size\n        elif len(initial_scalar_parameters) != mixture_size:\n            raise ValueError(\"Length of initial_scalar_parameters {} differs \"\n                             \"from mixture_size {}\".format(\n                initial_scalar_parameters, mixture_size))\n\n        # self.scalar_parameters = ParameterList(\n        #     [Parameter(torch.FloatTensor([initial_scalar_parameters[i]]),\n        #                requires_grad=trainable) for i\n        #      in range(mixture_size)])\n        self.scalar_parameters = Parameter(torch.FloatTensor(initial_scalar_parameters), requires_grad=True)\n        self.gamma = Parameter(torch.FloatTensor([1.0]), requires_grad=trainable)\n\n        if self.dropout:\n            dropout_mask = torch.zeros(len(self.scalar_parameters))\n            dropout_fill = torch.empty(len(self.scalar_parameters)).fill_(dropout_value)\n            self.register_buffer(\"dropout_mask\", dropout_mask)\n            self.register_buffer(\"dropout_fill\", dropout_fill)\n\n    def forward(self, tensors: List[torch.Tensor],  # pylint: disable=arguments-differ\n                mask: torch.Tensor = None) -> torch.Tensor:\n        \"\"\"Compute a weighted average of the ``tensors``.  The input tensors an be any shape\n        with at least two dimensions, but must all be the same shape.\n        \n        When ``do_layer_norm=True``, the ``mask`` is required input.  If the ``tensors`` are\n        dimensioned  ``(dim_0, ..., dim_{n-1}, dim_n)``, then the ``mask`` is dimensioned\n        ``(dim_0, ..., dim_{n-1})``, as in the typical case with ``tensors`` of shape\n        ``(batch_size, timesteps, dim)`` and ``mask`` of shape ``(batch_size, timesteps)``.\n        \n        When ``do_layer_norm=False`` the ``mask`` is ignored.\n\n        Args:\n          tensors: List[torch.Tensor]: \n          # pylint: disable:  (Default value = arguments-differmask: torch.Tensor = None)\n\n        Returns:\n\n        \"\"\"\n        if len(tensors) != self.mixture_size:\n            tensors = tensors[self.mixture_range[0]:self.mixture_range[1]]\n        if len(tensors) != self.mixture_size:\n            raise ValueError(\"{} tensors were passed, but the module was initialized to \"\n                             \"mix {} tensors.\".format(len(tensors), self.mixture_size))\n\n        def _do_layer_norm(tensor, broadcast_mask, num_elements_not_masked):\n            tensor_masked = tensor * broadcast_mask\n            mean = torch.sum(tensor_masked) / num_elements_not_masked\n            variance = torch.sum(((tensor_masked - mean) * broadcast_mask) ** 2) / num_elements_not_masked\n            return (tensor - mean) / torch.sqrt(variance + 1E-12)\n\n        weights = self.scalar_parameters\n\n        if self.dropout:\n            weights = torch.where(self.dropout_mask.uniform_() > self.dropout, weights, self.dropout_fill)\n\n        normed_weights = torch.nn.functional.softmax(weights, dim=0)\n\n        if not self.do_layer_norm:\n            return self.gamma * torch.einsum('i,ijkl->jkl', normed_weights, tensors)\n            # pieces = []\n            # for weight, tensor in zip(normed_weights, tensors):\n            #     pieces.append(weight * tensor)\n            # return self.gamma * sum(pieces)\n        else:\n            normed_weights = torch.split(normed_weights, split_size_or_sections=1)\n            mask_float = mask.float()\n            broadcast_mask = mask_float.unsqueeze(-1)\n            input_dim = tensors[0].size(-1)\n            num_elements_not_masked = torch.sum(mask_float) * input_dim\n\n            pieces = []\n            for weight, tensor in zip(normed_weights, tensors):\n                pieces.append(weight * _do_layer_norm(tensor,\n                                                      broadcast_mask, num_elements_not_masked))\n            return self.gamma * sum(pieces)\n\n\nclass ScalarMixWithDropoutBuilder(ConfigTracker, ScalarMixWithDropout):\n\n    def __init__(self,\n                 mixture_range: Tuple[int, int],\n                 do_layer_norm: bool = False,\n                 initial_scalar_parameters: List[float] = None,\n                 trainable: bool = True,\n                 dropout: float = None,\n                 dropout_value: float = -1e20) -> None:\n        super().__init__(locals())\n\n    def build(self):\n        return ScalarMixWithDropout(**self.config)\n"
  },
  {
    "path": "hanlp/layers/time_distributed.py",
    "content": "\"\"\"\nA wrapper that unrolls the second (time) dimension of a tensor\ninto the first (batch) dimension, applies some other `Module`,\nand then rolls the time dimension back up.\n\"\"\"\n\nfrom typing import List\n\n\nimport torch\n\n\nclass TimeDistributed(torch.nn.Module):\n    \"\"\"\n    Given an input shaped like `(batch_size, time_steps, [rest])` and a `Module` that takes\n    inputs like `(batch_size, [rest])`, `TimeDistributed` reshapes the input to be\n    `(batch_size * time_steps, [rest])`, applies the contained `Module`, then reshapes it back.\n\n    Note that while the above gives shapes with `batch_size` first, this `Module` also works if\n    `batch_size` is second - we always just combine the first two dimensions, then split them.\n\n    It also reshapes keyword arguments unless they are not tensors or their name is specified in\n    the optional `pass_through` iterable.\n    \"\"\"\n\n    def __init__(self, module):\n        super().__init__()\n        self._module = module\n\n\n    def forward(self, *inputs, pass_through: List[str] = None, **kwargs):\n\n        pass_through = pass_through or []\n\n        reshaped_inputs = [self._reshape_tensor(input_tensor) for input_tensor in inputs]\n\n        # Need some input to then get the batch_size and time_steps.\n        some_input = None\n        if inputs:\n            some_input = inputs[-1]\n\n        reshaped_kwargs = {}\n        for key, value in kwargs.items():\n            if isinstance(value, torch.Tensor) and key not in pass_through:\n                if some_input is None:\n                    some_input = value\n\n                value = self._reshape_tensor(value)\n\n            reshaped_kwargs[key] = value\n\n        reshaped_outputs = self._module(*reshaped_inputs, **reshaped_kwargs)\n\n        if some_input is None:\n            raise RuntimeError(\"No input tensor to time-distribute\")\n\n        # Now get the output back into the right shape.\n        # (batch_size, time_steps, **output_size)\n        new_size = some_input.size()[:2] + reshaped_outputs.size()[1:]\n        outputs = reshaped_outputs.contiguous().view(new_size)\n\n        return outputs\n\n    @staticmethod\n    def _reshape_tensor(input_tensor):\n        input_size = input_tensor.size()\n        if len(input_size) <= 2:\n            raise RuntimeError(f\"No dimension to distribute: {input_size}\")\n        # Squash batch_size and time_steps into a single axis; result has shape\n        # (batch_size * time_steps, **input_size).\n        squashed_shape = [-1] + list(input_size[2:])\n        return input_tensor.contiguous().view(*squashed_shape)\n"
  },
  {
    "path": "hanlp/layers/transformers/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-29 15:17\n# mute transformers\nimport logging\n\nlogging.getLogger('transformers.file_utils').setLevel(logging.ERROR)\nlogging.getLogger('transformers.filelock').setLevel(logging.ERROR)\nlogging.getLogger('transformers.tokenization_utils').setLevel(logging.ERROR)\nlogging.getLogger('transformers.configuration_utils').setLevel(logging.ERROR)\nlogging.getLogger('transformers.modeling_tf_utils').setLevel(logging.ERROR)\nlogging.getLogger('transformers.modeling_utils').setLevel(logging.ERROR)\nlogging.getLogger('transformers.tokenization_utils_base').setLevel(logging.ERROR)\n"
  },
  {
    "path": "hanlp/layers/transformers/encoder.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-22 21:06\nimport warnings\nfrom typing import Union, Dict, Any, Sequence, Tuple, Optional\n\nimport torch\nfrom torch import nn\nfrom hanlp.layers.dropout import WordDropout\nfrom hanlp.layers.scalar_mix import ScalarMixWithDropout, ScalarMixWithDropoutBuilder\nfrom hanlp.layers.transformers.resource import get_tokenizer_mirror\nfrom hanlp.layers.transformers.pt_imports import PreTrainedModel, PreTrainedTokenizer, AutoTokenizer, AutoModel_, \\\n    BertTokenizer, AutoTokenizer_\nfrom hanlp.layers.transformers.utils import transformer_encode\n\n\n# noinspection PyAbstractClass\nclass TransformerEncoder(nn.Module):\n    def __init__(self,\n                 transformer: Union[PreTrainedModel, str],\n                 transformer_tokenizer: PreTrainedTokenizer,\n                 average_subwords=False,\n                 scalar_mix: Union[ScalarMixWithDropoutBuilder, int] = None,\n                 word_dropout=None,\n                 max_sequence_length=None,\n                 ret_raw_hidden_states=False,\n                 transformer_args: Dict[str, Any] = None,\n                 trainable=Union[bool, Optional[Tuple[int, int]]],\n                 training=True) -> None:\n        \"\"\"A pre-trained transformer encoder.\n\n        Args:\n            transformer: A ``PreTrainedModel`` or an identifier of a ``PreTrainedModel``.\n            transformer_tokenizer: A ``PreTrainedTokenizer``.\n            average_subwords: ``True`` to average subword representations.\n            scalar_mix: Layer attention.\n            word_dropout: Dropout rate of randomly replacing a subword with MASK.\n            max_sequence_length: The maximum sequence length. Sequence longer than this will be handled by sliding\n                window. If ``None``, then the ``max_position_embeddings`` of the transformer will be used.\n            ret_raw_hidden_states: ``True`` to return hidden states of each layer.\n            transformer_args: Extra arguments passed to the transformer.\n            trainable: ``False`` to use static embeddings.\n            training: ``False`` to skip loading weights from pre-trained transformers.\n        \"\"\"\n        super().__init__()\n        self.ret_raw_hidden_states = ret_raw_hidden_states\n        self.average_subwords = average_subwords\n        if word_dropout:\n            oov = transformer_tokenizer.mask_token_id\n            if isinstance(word_dropout, Sequence):\n                word_dropout, replacement = word_dropout\n                if replacement == 'unk':\n                    # Electra English has to use unk\n                    oov = transformer_tokenizer.unk_token_id\n                elif replacement == 'mask':\n                    # UDify uses [MASK]\n                    oov = transformer_tokenizer.mask_token_id\n                else:\n                    oov = replacement\n            pad = transformer_tokenizer.pad_token_id\n            cls = transformer_tokenizer.cls_token_id\n            sep = transformer_tokenizer.sep_token_id\n            excludes = [pad, cls, sep]\n            self.word_dropout = WordDropout(p=word_dropout, oov_token=oov, exclude_tokens=excludes)\n        else:\n            self.word_dropout = None\n        if isinstance(transformer, str):\n            output_hidden_states = scalar_mix is not None\n            if transformer_args is None:\n                transformer_args = dict()\n            transformer_args['output_hidden_states'] = output_hidden_states\n            transformer = AutoModel_.from_pretrained(transformer, training=training or not trainable,\n                                                     **transformer_args)\n            if max_sequence_length is None:\n                max_sequence_length = transformer.config.max_position_embeddings\n        self.max_sequence_length = max_sequence_length\n        if hasattr(transformer, 'encoder') and hasattr(transformer, 'decoder'):\n            # For seq2seq model, use its encoder\n            transformer = transformer.encoder\n        self.transformer = transformer\n        if not trainable:\n            transformer.requires_grad_(False)\n        elif isinstance(trainable, tuple):\n            layers = []\n            if hasattr(transformer, 'embeddings'):\n                layers.append(transformer.embeddings)\n            layers.extend(transformer.encoder.layer)\n            for i, layer in enumerate(layers):\n                if i < trainable[0] or i >= trainable[1]:\n                    layer.requires_grad_(False)\n\n        if isinstance(scalar_mix, ScalarMixWithDropoutBuilder):\n            self.scalar_mix: ScalarMixWithDropout = scalar_mix.build()\n        else:\n            self.scalar_mix = None\n\n    def forward(self, input_ids: torch.LongTensor, attention_mask=None, token_type_ids=None, token_span=None, **kwargs):\n        if self.word_dropout:\n            input_ids = self.word_dropout(input_ids)\n\n        x = transformer_encode(self.transformer,\n                               input_ids,\n                               attention_mask,\n                               token_type_ids,\n                               token_span,\n                               layer_range=self.scalar_mix.mixture_range if self.scalar_mix else 0,\n                               max_sequence_length=self.max_sequence_length,\n                               average_subwords=self.average_subwords,\n                               ret_raw_hidden_states=self.ret_raw_hidden_states)\n        if self.ret_raw_hidden_states:\n            x, raw_hidden_states = x\n        if self.scalar_mix:\n            x = self.scalar_mix(x)\n        if self.ret_raw_hidden_states:\n            # noinspection PyUnboundLocalVariable\n            return x, raw_hidden_states\n        return x\n\n    @staticmethod\n    def build_transformer(config, training=True) -> PreTrainedModel:\n        kwargs = {}\n        if config.scalar_mix and config.scalar_mix > 0:\n            kwargs['output_hidden_states'] = True\n        transformer = AutoModel_.from_pretrained(config.transformer, training=training, **kwargs)\n        return transformer\n\n    @staticmethod\n    def build_transformer_tokenizer(config_or_str, use_fast=True, do_basic_tokenize=True) -> PreTrainedTokenizer:\n        return AutoTokenizer_.from_pretrained(config_or_str, use_fast, do_basic_tokenize)\n"
  },
  {
    "path": "hanlp/layers/transformers/loader_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-01-04 06:05\nimport tensorflow as tf\nfrom transformers import TFAutoModel\n\nfrom hanlp.layers.transformers.pt_imports import AutoTokenizer_, AutoModel_\n\n\ndef build_transformer(transformer, max_seq_length, num_labels, tagging=True, tokenizer_only=False):\n    tokenizer = AutoTokenizer_.from_pretrained(transformer)\n    if tokenizer_only:\n        return tokenizer\n    l_bert = TFAutoModel.from_pretrained(transformer)\n    l_input_ids = tf.keras.layers.Input(shape=(max_seq_length,), dtype='int32', name=\"input_ids\")\n    l_mask_ids = tf.keras.layers.Input(shape=(max_seq_length,), dtype='int32', name=\"mask_ids\")\n    l_token_type_ids = tf.keras.layers.Input(shape=(max_seq_length,), dtype='int32', name=\"token_type_ids\")\n    output = l_bert(input_ids=l_input_ids, token_type_ids=l_token_type_ids, attention_mask=l_mask_ids).last_hidden_state\n    if not tagging:\n        output = tf.keras.layers.Lambda(lambda seq: seq[:, 0, :])(output)\n    logits = tf.keras.layers.Dense(num_labels)(output)\n    model = tf.keras.Model(inputs=[l_input_ids, l_mask_ids, l_token_type_ids], outputs=logits)\n    model.build(input_shape=(None, max_seq_length))\n    return model, tokenizer\n"
  },
  {
    "path": "hanlp/layers/transformers/pt_imports.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-09 11:25\nimport os\nimport warnings\n\nfrom hanlp.layers.transformers.resource import get_tokenizer_mirror, get_model_mirror\n\nif os.environ.get('TOKENIZERS_PARALLELISM', None) is None:\n    os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\nfrom transformers import BertTokenizer, BertConfig, PretrainedConfig, AutoConfig, AutoTokenizer, PreTrainedTokenizer, \\\n    BertTokenizerFast, AlbertConfig, BertModel, AutoModel, PreTrainedModel, AutoModelForSequenceClassification, \\\n    AutoModelForTokenClassification, BartModel\n\n\nclass AutoModel_(AutoModel):\n    @classmethod\n    def from_pretrained(cls, pretrained_model_name_or_path, *model_args, training=True, **kwargs):\n        pretrained_model_name_or_path = get_model_mirror(pretrained_model_name_or_path)\n        if training:\n            return super().from_pretrained(pretrained_model_name_or_path, *model_args, **kwargs)\n        else:\n            if isinstance(pretrained_model_name_or_path, str):\n                pretrained_model_name_or_path = get_tokenizer_mirror(pretrained_model_name_or_path)\n                return super().from_config(AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs))\n            else:\n                assert not kwargs\n                return super().from_config(pretrained_model_name_or_path)\n\n\nclass AutoConfig_(AutoConfig):\n    @classmethod\n    def from_pretrained(cls, pretrained_model_name_or_path, **kwargs):\n        pretrained_model_name_or_path = get_tokenizer_mirror(pretrained_model_name_or_path)\n        return super().from_pretrained(pretrained_model_name_or_path, **kwargs)\n\n\nclass AutoTokenizer_(AutoTokenizer):\n\n    @classmethod\n    def from_pretrained(cls, pretrained_model_name_or_path, use_fast=True,\n                        do_basic_tokenize=True) -> PreTrainedTokenizer:\n        if isinstance(pretrained_model_name_or_path, str):\n            transformer = pretrained_model_name_or_path\n        else:\n            transformer = pretrained_model_name_or_path.transformer\n        additional_config = dict()\n        if transformer.startswith('voidful/albert_chinese_') or transformer.startswith('uer/albert'):\n            cls = BertTokenizer\n        elif transformer == 'cl-tohoku/bert-base-japanese-char':\n            # Since it's char level model, it's OK to use char level tok instead of fugashi\n            # from hanlp.utils.lang.ja.bert_tok import BertJapaneseTokenizerFast\n            # cls = BertJapaneseTokenizerFast\n            from transformers import BertJapaneseTokenizer\n            cls = BertJapaneseTokenizer\n            # from transformers import BertTokenizerFast\n            # cls = BertTokenizerFast\n            additional_config['word_tokenizer_type'] = 'basic'\n        elif transformer == \"Langboat/mengzi-bert-base\":\n            cls = BertTokenizerFast if use_fast else BertTokenizer\n        else:\n            cls = AutoTokenizer\n        if use_fast and not do_basic_tokenize:\n            warnings.warn('`do_basic_tokenize=False` might not work when `use_fast=True`')\n        tokenizer = cls.from_pretrained(get_tokenizer_mirror(transformer), use_fast=use_fast,\n                                        do_basic_tokenize=do_basic_tokenize,\n                                        **additional_config)\n        tokenizer.name_or_path = transformer\n        return tokenizer\n"
  },
  {
    "path": "hanlp/layers/transformers/relative_transformer.py",
    "content": "# A modified version of the implementation from the following paper:\n# TENER: Adapting Transformer Encoder for Named Entity Recognition\n# Hang Yan, Bocao Deng, Xiaonan Li, Xipeng Qiu\n\nimport math\nimport torch\nimport torch.nn.functional as F\nfrom torch import Tensor, nn\n\nfrom hanlp.common.structure import ConfigTracker\n\n\nclass RelativeSinusoidalPositionalEmbedding(nn.Module):\n    \"\"\"This module produces sinusoidal positional embeddings of any length.\n    Padding symbols are ignored.\n\n    Args:\n        embedding_dim: embedding size of each position\n        padding_idx:\n    Returns:\n\n    \"\"\"\n\n    def __init__(self, embedding_dim, padding_idx, init_size=1024):\n        super().__init__()\n        self.embedding_dim = embedding_dim\n        self.padding_idx = padding_idx\n        assert init_size % 2 == 0\n        weights = self.get_embedding(\n            init_size + 1,\n            embedding_dim,\n            padding_idx,\n        )\n        self.register_buffer('weights', weights)\n\n    def get_embedding(self, num_embeddings, embedding_dim, padding_idx=None):\n        \"\"\"Build sinusoidal embeddings.\n        This matches the implementation in tensor2tensor, but differs slightly\n        from the description in Section 3.5 of \"Attention Is All You Need\".\n\n        Args:\n          num_embeddings:\n          embedding_dim:\n          padding_idx:  (Default value = None)\n\n        Returns:\n\n        \"\"\"\n        half_dim = embedding_dim // 2\n        emb = math.log(10000) / (half_dim - 1)\n        emb = torch.exp(torch.arange(half_dim, dtype=torch.float) * -emb)\n        emb = torch.arange(-num_embeddings // 2, num_embeddings // 2, dtype=torch.float).unsqueeze(1) * emb.unsqueeze(0)\n        emb = torch.cat([torch.sin(emb), torch.cos(emb)], dim=1).view(num_embeddings, -1)\n        if embedding_dim % 2 == 1:\n            # zero pad\n            emb = torch.cat([emb, torch.zeros(num_embeddings, 1)], dim=1)\n        if padding_idx is not None:\n            emb[padding_idx, :] = 0\n        self.origin_shift = num_embeddings // 2 + 1\n        return emb\n\n    def forward(self, inputs: Tensor):\n        \"\"\"Input is expected to be of size [bsz x seqlen].\n\n        Args:\n          inputs: Tensor:\n\n        Returns:\n\n        \"\"\"\n        bsz, seq_len = inputs.size()\n        max_pos = self.padding_idx + seq_len\n        if max_pos >= self.origin_shift:\n            # recompute/expand embeddings if needed\n            weights = self.get_embedding(\n                max_pos * 2,\n                self.embedding_dim,\n                self.padding_idx,\n            )\n            weights = weights.to(self.weights.device)\n            del self.weights\n            self.origin_shift = weights.size(0) // 2\n            self.register_buffer('weights', weights)\n\n        positions = torch.arange(-seq_len, seq_len).to(inputs.device).long() + self.origin_shift  # 2*seq_len\n        embed = self.weights.index_select(0, positions.long()).detach()\n        return embed\n\n\nclass RelativeMultiHeadAttn(nn.Module):\n    def __init__(self, in_features, num_heads, dropout, r_w_bias=None, r_r_bias=None, init_seq_length=1024,\n                 k_as_x=True):\n        \"\"\"\n        Args:\n            in_features:\n            num_heads:\n            dropout:\n            r_w_bias: n_head x head_dim or None\n            r_r_bias: n_head x head_dim or None\n            init_seq_length:\n            k_as_x:\n        \"\"\"\n        super().__init__()\n        self.k_as_x = k_as_x\n        if k_as_x:\n            self.qv_linear = nn.Linear(in_features, in_features * 2, bias=False)\n        else:\n            self.qkv_linear = nn.Linear(in_features, in_features * 3, bias=False)\n        self.n_head = num_heads\n        self.head_dim = in_features // num_heads\n        self.dropout_layer = nn.Dropout(dropout)\n        self.pos_embed = RelativeSinusoidalPositionalEmbedding(self.head_dim, 0, init_seq_length)\n        if r_r_bias is None or r_w_bias is None:  # Biases are not shared\n            self.r_r_bias = nn.Parameter(nn.init.xavier_normal_(torch.zeros(num_heads, in_features // num_heads)))\n            self.r_w_bias = nn.Parameter(nn.init.xavier_normal_(torch.zeros(num_heads, in_features // num_heads)))\n        else:\n            self.r_r_bias = r_r_bias  # r_r_bias就是v\n            self.r_w_bias = r_w_bias  # r_w_bias就是u\n\n    def forward(self, x, mask):\n        \"\"\"\n\n        Args:\n          x: batch_size x max_len x d_model\n          mask: batch_size x max_len\n\n        Returns:\n\n        \"\"\"\n\n        batch_size, max_len, d_model = x.size()\n        pos_embed = self.pos_embed(mask)  # l x head_dim\n\n        if self.k_as_x:\n            qv = self.qv_linear(x)  # batch_size x max_len x d_model2\n            q, v = torch.chunk(qv, chunks=2, dim=-1)\n            k = x.view(batch_size, max_len, self.n_head, -1).transpose(1, 2)\n        else:\n            qkv = self.qkv_linear(x)  # batch_size x max_len x d_model3\n            q, k, v = torch.chunk(qkv, chunks=3, dim=-1)\n            k = k.view(batch_size, max_len, self.n_head, -1).transpose(1, 2)\n\n        q = q.view(batch_size, max_len, self.n_head, -1).transpose(1, 2)\n        v = v.view(batch_size, max_len, self.n_head, -1).transpose(1, 2)  # b x n x l x d\n\n        rw_head_q = q + self.r_r_bias[:, None]\n        AC = torch.einsum('bnqd,bnkd->bnqk', [rw_head_q, k])  # b x n x l x d, n是head\n\n        D_ = torch.einsum('nd,ld->nl', self.r_w_bias, pos_embed)[None, :, None]  # head x 2max_len, 每个head对位置的bias\n        B_ = torch.einsum('bnqd,ld->bnql', q, pos_embed)  # bsz x head  x max_len x 2max_len，每个query对每个shift的偏移\n        E_ = torch.einsum('bnqd,ld->bnql', k, pos_embed)  # bsz x head x max_len x 2max_len, key对relative的bias\n        BD = B_ + D_  # bsz x head x max_len x 2max_len, 要转换为bsz x head x max_len x max_len\n        if self.k_as_x:\n            BD = self._shift(BD)\n            attn = AC + BD\n        else:\n            BDE = self._shift(BD) + self._transpose_shift(E_)\n            attn = AC + BDE\n\n        attn = attn.masked_fill(mask[:, None, None, :].eq(0), float('-inf'))\n\n        attn = F.softmax(attn, dim=-1)\n        attn = self.dropout_layer(attn)\n        v = torch.matmul(attn, v).transpose(1, 2).reshape(batch_size, max_len, d_model)  # b x n x l x d\n\n        return v\n\n    def _shift(self, BD):\n        \"\"\"类似\n        -3 -2 -1 0 1 2\n        -3 -2 -1 0 1 2\n        -3 -2 -1 0 1 2\n        转换为\n        0   1  2\n        -1  0  1\n        -2 -1  0\n\n        Args:\n          BD: batch_size x n_head x max_len x 2max_len\n\n        Returns:\n          batch_size x n_head x max_len x max_len\n\n        \"\"\"\n        bsz, n_head, max_len, _ = BD.size()\n        zero_pad = BD.new_zeros(bsz, n_head, max_len, 1)\n        BD = torch.cat([BD, zero_pad], dim=-1).view(bsz, n_head, -1, max_len)  # bsz x n_head x (2max_len+1) x max_len\n        BD = BD.narrow(dim=2, start=0, length=2 * max_len) \\\n            .view(bsz, n_head, max_len, -1)  # bsz x n_head x 2max_len x max_len\n        BD = BD.narrow(dim=-1, start=max_len, length=max_len)\n        return BD\n\n    def _transpose_shift(self, E):\n        \"\"\"类似\n          -3   -2   -1   0   1   2\n         -30  -20  -10  00  10  20\n        -300 -200 -100 000 100 200\n\n        转换为\n          0  -10   -200\n          1   00   -100\n          2   10    000\n\n        Args:\n          E: batch_size x n_head x max_len x 2max_len\n\n        Returns:\n          batch_size x n_head x max_len x max_len\n\n        \"\"\"\n        bsz, n_head, max_len, _ = E.size()\n        zero_pad = E.new_zeros(bsz, n_head, max_len, 1)\n        # bsz x n_head x -1 x (max_len+1)\n        E = torch.cat([E, zero_pad], dim=-1).view(bsz, n_head, -1, max_len)\n        indice = (torch.arange(max_len) * 2 + 1).to(E.device)\n        E = E.index_select(index=indice, dim=-2).transpose(-1, -2)  # bsz x n_head x max_len x max_len\n\n        return E\n\n\nclass RelativeTransformerLayer(nn.Module):\n    def __init__(self,\n                 in_features,\n                 num_heads=4,\n                 feedforward_dim=256,\n                 dropout=0.2,\n                 dropout_attn=None,\n                 after_norm=True,\n                 k_as_x=True,\n                 init_seq_length=1024):\n        super().__init__()\n        if dropout_attn is None:\n            dropout_attn = dropout\n        self.after_norm = after_norm\n        self.norm1 = nn.LayerNorm(in_features)\n        self.norm2 = nn.LayerNorm(in_features)\n        self.self_attn = RelativeMultiHeadAttn(in_features,\n                                               num_heads,\n                                               dropout=dropout_attn,\n                                               init_seq_length=init_seq_length,\n                                               k_as_x=k_as_x)\n        self.ffn = nn.Sequential(nn.Linear(in_features, feedforward_dim),\n                                 nn.LeakyReLU(),\n                                 nn.Dropout(dropout, inplace=True),\n                                 nn.Linear(feedforward_dim, in_features),\n                                 nn.Dropout(dropout, inplace=True))\n\n    def forward(self, x, mask):\n        \"\"\"\n\n        Args:\n          x: batch_size x max_len x hidden_size\n          mask: batch_size x max_len, 为0的地方为pad\n\n        Returns:\n          batch_size x max_len x hidden_size\n\n        \"\"\"\n        residual = x\n        if not self.after_norm:\n            x = self.norm1(x)\n\n        x = self.self_attn(x, mask)\n        x = x + residual\n        if self.after_norm:\n            x = self.norm1(x)\n        residual = x\n        if not self.after_norm:\n            x = self.norm2(x)\n        x = self.ffn(x)\n        x = residual + x\n        if self.after_norm:\n            x = self.norm2(x)\n        return x\n\n\nclass RelativeTransformer(nn.Module):\n    def __init__(self,\n                 in_features,\n                 num_layers,\n                 feedforward_dim,\n                 num_heads,\n                 dropout,\n                 dropout_attn=None,\n                 after_norm=True,\n                 init_seq_length=1024,\n                 k_as_x=True):\n        super().__init__()\n        self.layers = nn.ModuleList([\n            RelativeTransformerLayer(in_features, feedforward_dim, num_heads, dropout, dropout_attn, after_norm,\n                                     init_seq_length=init_seq_length, k_as_x=k_as_x)\n            for _ in range(num_layers)\n        ])\n\n    def forward(self, x: Tensor, mask: Tensor):\n        \"\"\"\n\n        Args:\n          x: batch_size x max_len\n          mask: batch_size x max_len. 有value的地方为1\n          x: Tensor: \n          mask: Tensor: \n\n        Returns:\n\n        \"\"\"\n        if not x.numel():\n            return x\n        for layer in self.layers:\n            x = layer(x, mask)\n        return x\n\n\nclass RelativeTransformerEncoder(RelativeTransformer, ConfigTracker):\n    def __init__(self,\n                 in_features,\n                 num_layers=2,\n                 num_heads=4,\n                 feedforward_dim=256,\n                 dropout=0.1,\n                 dropout_attn=0.1,\n                 after_norm=True,\n                 k_as_x=True,\n                 ):\n        super().__init__(in_features, num_layers, num_heads, feedforward_dim, dropout, dropout_attn, after_norm)\n        ConfigTracker.__init__(self, locals())\n\n    def get_output_dim(self):\n        return self.config['in_features']\n"
  },
  {
    "path": "hanlp/layers/transformers/resource.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-05-20 12:43\nfrom hanlp.utils.io_util import get_resource\nfrom hanlp_common.constant import HANLP_URL\n\ntokenizer_mirrors = {\n    'hfl/chinese-electra-180g-base-discriminator': HANLP_URL + 'transformers/electra_zh_base_20210706_125233.zip',\n    'hfl/chinese-electra-180g-small-discriminator': HANLP_URL + 'transformers/electra_zh_small_20210706_125427.zip',\n    'xlm-roberta-base': HANLP_URL + 'transformers/xlm-roberta-base_20210706_125502.zip',\n    'cl-tohoku/bert-base-japanese-char': HANLP_URL + 'transformers/bert-base-japanese-char_20210602_215445.zip',\n    'bart5-chinese-small': HANLP_URL + 'transformers/bart5-chinese-small_tok_20210723_180743.zip',\n    'ernie-gram': HANLP_URL + 'transformers/ernie-gram_20220207_103518.zip',\n    'xlm-roberta-base-no-space': HANLP_URL + 'transformers/xlm-roberta-base-no-space-tokenizer_20220610_204241.zip',\n    'mMiniLMv2L6-no-space': HANLP_URL + 'transformers/mMiniLMv2L6-no-space-tokenizer_20220616_094859.zip',\n    'mMiniLMv2L12-no-space': HANLP_URL + 'transformers/mMiniLMv2L12-no-space-tokenizer_20220616_095900.zip',\n}\n\nmodel_mirrors = {\n    'bart5-chinese-small': HANLP_URL + 'transformers/bart5-chinese-small_20210723_203923.zip',\n    'xlm-roberta-base-no-space': HANLP_URL + 'transformers/xlm-roberta-base-no-space_20220610_203944.zip',\n    'mMiniLMv2L6-no-space': HANLP_URL + 'transformers/mMiniLMv2L6-no-space_20220616_094949.zip',\n    'mMiniLMv2L12-no-space': HANLP_URL + 'transformers/mMiniLMv2L12-no-space_20220616_095924.zip',\n}\n\n\ndef get_tokenizer_mirror(transformer: str) -> str:\n    m = tokenizer_mirrors.get(transformer, None)\n    if m:\n        return get_resource(m)\n    return transformer\n\n\ndef get_model_mirror(transformer: str) -> str:\n    m = model_mirrors.get(transformer, None)\n    if m:\n        return get_resource(m)\n    return transformer\n"
  },
  {
    "path": "hanlp/layers/transformers/tf_imports.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-08 21:57\nfrom transformers import BertTokenizer, BertConfig, PretrainedConfig, TFAutoModel, \\\n    AutoConfig, AutoTokenizer, PreTrainedTokenizer, TFPreTrainedModel, TFAlbertModel, TFAutoModelWithLMHead, \\\n    BertTokenizerFast, TFAlbertForMaskedLM, AlbertConfig, TFBertModel\n"
  },
  {
    "path": "hanlp/layers/transformers/utils.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-15 21:22\nfrom collections import defaultdict\nfrom typing import Tuple, Union\n\nimport torch\nfrom torch.nn import functional as F\n\nfrom hanlp.components.parsers.ud import udify_util as util\nfrom hanlp.layers.transformers.pt_imports import PreTrainedModel\n\n\ndef transformer_encode(transformer: PreTrainedModel,\n                       input_ids,\n                       attention_mask=None,\n                       token_type_ids=None,\n                       token_span=None,\n                       layer_range: Union[int, Tuple[int, int]] = 0,\n                       max_sequence_length=None,\n                       average_subwords=False,\n                       ret_raw_hidden_states=False):\n    \"\"\"Run transformer and pool its outputs.\n\n    Args:\n        transformer: A transformer model.\n        input_ids: Indices of subwords.\n        attention_mask: Mask for these subwords.\n        token_type_ids: Type ids for each subword.\n        token_span: The spans of tokens.\n        layer_range: The range of layers to use. Note that the 0-th layer means embedding layer, so the last 3 layers\n                    of a 12-layer BERT will be (10, 13).\n        max_sequence_length: The maximum sequence length. Sequence longer than this will be handled by sliding\n                    window.\n         average_subwords: ``True`` to average subword representations.\n        ret_raw_hidden_states: ``True`` to return hidden states of each layer.\n\n    Returns:\n        Pooled outputs.\n\n    \"\"\"\n    if max_sequence_length and input_ids.size(-1) > max_sequence_length:\n        # TODO: split token type ids in transformer_sliding_window if token type ids are not always 1\n        outputs = transformer_sliding_window(transformer, input_ids, max_pieces=max_sequence_length)\n    else:\n        if attention_mask is None:\n            attention_mask = input_ids.ne(0)\n        if transformer.config.output_hidden_states:\n            outputs = transformer(input_ids, attention_mask, token_type_ids)[-1]\n        else:\n            outputs = transformer(input_ids, attention_mask, token_type_ids)[0]\n    if transformer.config.output_hidden_states:\n        if isinstance(layer_range, int):\n            outputs = outputs[layer_range:]\n        else:\n            outputs = outputs[layer_range[0], layer_range[1]]\n        # Slow pick\n        # hs = []\n        # for h in outputs:\n        #     hs.append(pick_tensor_for_each_token(h, token_span, average_subwords))\n        # Fast pick\n        if not isinstance(outputs, torch.Tensor):\n            x = torch.stack(outputs)\n        else:\n            x = outputs\n        L, B, T, F = x.size()\n        x = x.flatten(end_dim=1)\n        # tile token_span as x\n        if token_span is not None:\n            token_span = token_span.repeat(L, 1, 1)\n        hs = pick_tensor_for_each_token(x, token_span, average_subwords).view(L, B, -1, F)\n        if ret_raw_hidden_states:\n            return hs, outputs\n        return hs\n    else:\n        if ret_raw_hidden_states:\n            return pick_tensor_for_each_token(outputs, token_span, average_subwords), outputs\n        return pick_tensor_for_each_token(outputs, token_span, average_subwords)\n\n\ndef pick_tensor_for_each_token(h, token_span, average_subwords):\n    if token_span is None:\n        return h\n    if average_subwords and token_span.size(-1) > 1:\n        batch_size = h.size(0)\n        h_span = h.gather(1, token_span.view(batch_size, -1).unsqueeze(-1).expand(-1, -1, h.shape[-1]))\n        h_span = h_span.view(batch_size, *token_span.shape[1:], -1)\n        n_sub_tokens = token_span.ne(0)\n        n_sub_tokens[:, 0, 0] = True\n        h_span = (h_span * n_sub_tokens.unsqueeze(-1)).sum(2)\n        n_sub_tokens = n_sub_tokens.sum(-1).unsqueeze(-1)\n        zero_mask = n_sub_tokens == 0\n        if torch.any(zero_mask):\n            n_sub_tokens[zero_mask] = 1  # avoid dividing by zero\n        embed = h_span / n_sub_tokens\n    else:\n        embed = h.gather(1, token_span[:, :, 0].unsqueeze(-1).expand(-1, -1, h.size(-1)))\n    return embed\n\n\ndef transformer_sliding_window(transformer: PreTrainedModel,\n                               input_ids: torch.LongTensor,\n                               input_mask=None,\n                               offsets: torch.LongTensor = None,\n                               token_type_ids: torch.LongTensor = None,\n                               max_pieces=512,\n                               start_tokens: int = 1,\n                               end_tokens: int = 1,\n                               ret_cls=None,\n                               ) -> torch.Tensor:\n    \"\"\"\n\n    Args:\n      transformer:\n      input_ids: torch.LongTensor: \n      input_mask:  (Default value = None)\n      offsets: torch.LongTensor:  (Default value = None)\n      token_type_ids: torch.LongTensor:  (Default value = None)\n      max_pieces:  (Default value = 512)\n      start_tokens: int:  (Default value = 1)\n      end_tokens: int:  (Default value = 1)\n      ret_cls:  (Default value = None)\n\n    Returns:\n\n    \n    \"\"\"\n    # pylint: disable=arguments-differ\n    batch_size, full_seq_len = input_ids.size(0), input_ids.size(-1)\n    initial_dims = list(input_ids.shape[:-1])\n\n    # The embedder may receive an input tensor that has a sequence length longer than can\n    # be fit. In that case, we should expect the wordpiece indexer to create padded windows\n    # of length `max_pieces` for us, and have them concatenated into one long sequence.\n    # E.g., \"[CLS] I went to the [SEP] [CLS] to the store to [SEP] ...\"\n    # We can then split the sequence into sub-sequences of that length, and concatenate them\n    # along the batch dimension so we effectively have one huge batch of partial sentences.\n    # This can then be fed into BERT without any sentence length issues. Keep in mind\n    # that the memory consumption can dramatically increase for large batches with extremely\n    # long sentences.\n    needs_split = full_seq_len > max_pieces\n    if needs_split:\n        input_ids = split_to_sliding_window(input_ids, max_pieces)\n\n    # if token_type_ids is None:\n    #     token_type_ids = torch.zeros_like(input_ids)\n    if input_mask is None:\n        input_mask = (input_ids != 0).long()\n\n    # input_ids may have extra dimensions, so we reshape down to 2-d\n    # before calling the BERT model and then reshape back at the end.\n    outputs = transformer(input_ids=util.combine_initial_dims_to_1d_or_2d(input_ids),\n                          # token_type_ids=util.combine_initial_dims_to_1d_or_2d(token_type_ids),\n                          attention_mask=util.combine_initial_dims_to_1d_or_2d(input_mask)).to_tuple()\n    if len(outputs) == 3:\n        all_encoder_layers = outputs.hidden_states\n        all_encoder_layers = torch.stack(all_encoder_layers)\n    elif len(outputs) == 2:\n        all_encoder_layers, _ = outputs[:2]\n    else:\n        all_encoder_layers = outputs[0]\n\n    if needs_split:\n        if ret_cls is not None:\n            cls_mask = input_ids[:, 0] == input_ids[0][0]\n            cls_hidden = all_encoder_layers[:, 0, :]\n            if ret_cls == 'max':\n                cls_hidden[~cls_mask] = -1e20\n            else:\n                cls_hidden[~cls_mask] = 0\n            cls_mask = cls_mask.view(-1, batch_size).transpose(0, 1)\n            cls_hidden = cls_hidden.reshape(cls_mask.size(1), batch_size, -1).transpose(0, 1)\n            if ret_cls == 'max':\n                cls_hidden = cls_hidden.max(1)[0]\n            elif ret_cls == 'raw':\n                return cls_hidden, cls_mask\n            else:\n                cls_hidden = torch.sum(cls_hidden, dim=1)\n                cls_hidden /= torch.sum(cls_mask, dim=1, keepdim=True)\n            return cls_hidden\n        else:\n            recombined_embeddings, select_indices = restore_from_sliding_window(all_encoder_layers, batch_size,\n                                                                                max_pieces, full_seq_len, start_tokens,\n                                                                                end_tokens)\n\n            initial_dims.append(len(select_indices))\n    else:\n        recombined_embeddings = all_encoder_layers\n\n    # Recombine the outputs of all layers\n    # (layers, batch_size * d1 * ... * dn, sequence_length, embedding_dim)\n    # recombined = torch.cat(combined, dim=2)\n    # input_mask = (recombined_embeddings != 0).long()\n\n    # At this point, mix is (batch_size * d1 * ... * dn, sequence_length, embedding_dim)\n\n    if offsets is None:\n        # Resize to (batch_size, d1, ..., dn, sequence_length, embedding_dim)\n        dims = initial_dims if needs_split else input_ids.size()\n        layers = util.uncombine_initial_dims(recombined_embeddings, dims)\n    else:\n        # offsets is (batch_size, d1, ..., dn, orig_sequence_length)\n        offsets2d = util.combine_initial_dims_to_1d_or_2d(offsets)\n        # now offsets is (batch_size * d1 * ... * dn, orig_sequence_length)\n        range_vector = util.get_range_vector(offsets2d.size(0),\n                                             device=util.get_device_of(recombined_embeddings)).unsqueeze(1)\n        # selected embeddings is also (batch_size * d1 * ... * dn, orig_sequence_length)\n        selected_embeddings = recombined_embeddings[:, range_vector, offsets2d]\n\n        layers = util.uncombine_initial_dims(selected_embeddings, offsets.size())\n\n    return layers\n\n\ndef split_to_sliding_window(input_ids, max_pieces):\n    # Split the flattened list by the window size, `max_pieces`\n    split_input_ids = list(input_ids.split(max_pieces, dim=-1))\n    # We want all sequences to be the same length, so pad the last sequence\n    last_window_size = split_input_ids[-1].size(-1)\n    padding_amount = max_pieces - last_window_size\n    split_input_ids[-1] = F.pad(split_input_ids[-1], pad=[0, padding_amount], value=0)\n    # Now combine the sequences along the batch dimension\n    input_ids = torch.cat(split_input_ids, dim=0)\n    return input_ids\n\n\ndef restore_from_sliding_window(all_encoder_layers, batch_size, max_pieces, full_seq_len, start_tokens, end_tokens):\n    # First, unpack the output embeddings into one long sequence again\n    unpacked_embeddings = torch.split(all_encoder_layers, batch_size, dim=-3)\n    unpacked_embeddings = torch.cat(unpacked_embeddings, dim=-2)\n    # Next, select indices of the sequence such that it will result in embeddings representing the original\n    # sentence. To capture maximal context, the indices will be the middle part of each embedded window\n    # sub-sequence (plus any leftover start and final edge windows), e.g.,\n    #  0     1 2    3  4   5    6    7     8     9   10   11   12    13 14  15\n    # \"[CLS] I went to the very fine [SEP] [CLS] the very fine store to eat [SEP]\"\n    # with max_pieces = 8 should produce max context indices [2, 3, 4, 10, 11, 12] with additional start\n    # and final windows with indices [0, 1] and [14, 15] respectively.\n    # Find the stride as half the max pieces, ignoring the special start and end tokens\n    # Calculate an offset to extract the centermost embeddings of each window\n    stride = (max_pieces - start_tokens - end_tokens) // 2\n    stride_offset = stride // 2 + start_tokens\n    first_window = list(range(stride_offset))\n    max_context_windows = [i for i in range(full_seq_len)\n                           if stride_offset - 1 < i % max_pieces < stride_offset + stride]\n    final_window_start = max_context_windows[-1] + 1\n    final_window = list(range(final_window_start, full_seq_len))\n    select_indices = first_window + max_context_windows + final_window\n    select_indices = torch.LongTensor(select_indices).to(unpacked_embeddings.device)\n    recombined_embeddings = unpacked_embeddings.index_select(-2, select_indices)\n    return recombined_embeddings, select_indices\n\n\ndef build_optimizer_for_pretrained(model: torch.nn.Module,\n                                   pretrained: torch.nn.Module,\n                                   lr=1e-5,\n                                   weight_decay=0.01,\n                                   eps=1e-8,\n                                   transformer_lr=None,\n                                   transformer_weight_decay=None,\n                                   no_decay=('bias', 'LayerNorm.bias', 'LayerNorm.weight'),\n                                   **kwargs):\n    if transformer_lr is None:\n        transformer_lr = lr\n    if transformer_weight_decay is None:\n        transformer_weight_decay = weight_decay\n    params = defaultdict(lambda: defaultdict(list))\n    pretrained = set(pretrained.parameters())\n    if isinstance(no_decay, tuple):\n        def no_decay_fn(name):\n            return any(nd in name for nd in no_decay)\n    else:\n        assert callable(no_decay), 'no_decay has to be callable or a tuple of str'\n        no_decay_fn = no_decay\n    for n, p in model.named_parameters():\n        is_pretrained = 'pretrained' if p in pretrained else 'non_pretrained'\n        is_no_decay = 'no_decay' if no_decay_fn(n) else 'decay'\n        params[is_pretrained][is_no_decay].append(p)\n\n    grouped_parameters = [\n        {'params': params['pretrained']['decay'], 'weight_decay': transformer_weight_decay, 'lr': transformer_lr},\n        {'params': params['pretrained']['no_decay'], 'weight_decay': 0.0, 'lr': transformer_lr},\n        {'params': params['non_pretrained']['decay'], 'weight_decay': weight_decay, 'lr': lr},\n        {'params': params['non_pretrained']['no_decay'], 'weight_decay': 0.0, 'lr': lr},\n    ]\n\n    from transformers import optimization\n    return optimization.AdamW(\n        grouped_parameters,\n        lr=lr,\n        weight_decay=weight_decay,\n        eps=eps,\n        no_deprecation_warning=True,  # For backwards compatability\n        **kwargs)\n\n\ndef build_optimizer_scheduler_with_transformer(model: torch.nn.Module,\n                                               transformer: torch.nn.Module,\n                                               lr: float,\n                                               transformer_lr: float,\n                                               num_training_steps: int,\n                                               warmup_steps: Union[float, int],\n                                               weight_decay: float,\n                                               adam_epsilon: float,\n                                               no_decay=('bias', 'LayerNorm.bias', 'LayerNorm.weight')):\n    optimizer = build_optimizer_for_pretrained(model,\n                                               transformer,\n                                               lr,\n                                               weight_decay,\n                                               eps=adam_epsilon,\n                                               transformer_lr=transformer_lr,\n                                               no_decay=no_decay)\n    if isinstance(warmup_steps, float):\n        assert 0 < warmup_steps < 1, 'warmup_steps has to fall in range (0, 1) when it is float.'\n        warmup_steps = num_training_steps * warmup_steps\n    from transformers import optimization\n    scheduler = optimization.get_linear_schedule_with_warmup(optimizer, warmup_steps, num_training_steps)\n    return optimizer, scheduler\n\n\ndef get_optimizers(\n        model: torch.nn.Module,\n        num_training_steps: int,\n        learning_rate=5e-5,\n        adam_epsilon=1e-8,\n        weight_decay=0.0,\n        warmup_steps=0.1,\n) -> Tuple[torch.optim.Optimizer, torch.optim.lr_scheduler.LambdaLR]:\n    \"\"\"\n    Modified from https://github.com/huggingface/transformers/blob/7b75aa9fa55bee577e2c7403301ed31103125a35/src/transformers/trainer.py#L232\n    Setup the optimizer and the learning rate scheduler.\n\n    We provide a reasonable default that works well.\n    \"\"\"\n    if isinstance(warmup_steps, float):\n        assert 0 < warmup_steps < 1\n        warmup_steps = int(num_training_steps * warmup_steps)\n    # Prepare optimizer and schedule (linear warmup and decay)\n    no_decay = [\"bias\", \"LayerNorm.weight\"]\n    optimizer_grouped_parameters = [\n        {\n            \"params\": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],\n            \"weight_decay\": weight_decay,\n        },\n        {\n            \"params\": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],\n            \"weight_decay\": 0.0,\n        },\n    ]\n    from transformers import AdamW, get_linear_schedule_with_warmup\n    optimizer = AdamW(optimizer_grouped_parameters, lr=learning_rate, eps=adam_epsilon)\n    scheduler = get_linear_schedule_with_warmup(\n        optimizer, num_warmup_steps=warmup_steps, num_training_steps=num_training_steps\n    )\n    return optimizer, scheduler\n\n\ndef collect_decay_params(model, weight_decay):\n    no_decay = [\"bias\", \"LayerNorm.weight\"]\n    optimizer_grouped_parameters = [\n        {\n            \"params\": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],\n            \"weight_decay\": weight_decay,\n        },\n        {\n            \"params\": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],\n            \"weight_decay\": 0.0,\n        },\n    ]\n    return optimizer_grouped_parameters\n"
  },
  {
    "path": "hanlp/layers/transformers/utils_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-29 15:32\nimport tensorflow as tf\nfrom hanlp.optimizers.adamw import create_optimizer\nfrom hanlp.utils.log_util import logger\n\n\ndef config_is(config, model='bert'):\n    return model in type(config).__name__.lower()\n\n\ndef convert_examples_to_features(\n        words,\n        max_seq_length,\n        tokenizer,\n        labels=None,\n        label_map=None,\n        cls_token_at_end=False,\n        cls_token=\"[CLS]\",\n        cls_token_segment_id=1,\n        sep_token=\"[SEP]\",\n        sep_token_extra=False,\n        pad_on_left=False,\n        pad_token_id=0,\n        pad_token_segment_id=0,\n        pad_token_label_id=0,\n        sequence_a_segment_id=0,\n        mask_padding_with_zero=True,\n        unk_token='[UNK]',\n        do_padding=True\n):\n    \"\"\"Loads a data file into a list of `InputBatch`s\n        `cls_token_at_end` define the location of the CLS token:\n            - False (Default, BERT/XLM pattern): [CLS] + A + [SEP] + B + [SEP]\n            - True (XLNet/GPT pattern): A + [SEP] + B + [SEP] + [CLS]\n        `cls_token_segment_id` define the segment id associated to the CLS token (0 for BERT, 2 for XLNet)\n\n    Args:\n      words: \n      max_seq_length: \n      tokenizer: \n      labels:  (Default value = None)\n      label_map:  (Default value = None)\n      cls_token_at_end:  (Default value = False)\n      cls_token:  (Default value = \"[CLS]\")\n      cls_token_segment_id:  (Default value = 1)\n      sep_token:  (Default value = \"[SEP]\")\n      sep_token_extra:  (Default value = False)\n      pad_on_left:  (Default value = False)\n      pad_token_id:  (Default value = 0)\n      pad_token_segment_id:  (Default value = 0)\n      pad_token_label_id:  (Default value = 0)\n      sequence_a_segment_id:  (Default value = 0)\n      mask_padding_with_zero:  (Default value = True)\n      unk_token:  (Default value = '[UNK]')\n      do_padding:  (Default value = True)\n\n    Returns:\n\n    \"\"\"\n    args = locals()\n    if not labels:\n        labels = words\n        pad_token_label_id = False\n\n    tokens = []\n    label_ids = []\n    for word, label in zip(words, labels):\n        word_tokens = tokenizer.tokenize(word)\n        if not word_tokens:\n            # some wired chars cause the tagger to return empty list\n            word_tokens = [unk_token] * len(word)\n        tokens.extend(word_tokens)\n        # Use the real label id for the first token of the word, and padding ids for the remaining tokens\n        label_ids.extend([label_map[label] if label_map else True] + [pad_token_label_id] * (len(word_tokens) - 1))\n\n    # Account for [CLS] and [SEP] with \"- 2\" and with \"- 3\" for RoBERTa.\n    special_tokens_count = 3 if sep_token_extra else 2\n    if len(tokens) > max_seq_length - special_tokens_count:\n        logger.warning(\n            f'Input tokens {words} exceed the max sequence length of {max_seq_length - special_tokens_count}. '\n            f'The exceeded part will be truncated and ignored. '\n            f'You are recommended to split your long text into several sentences within '\n            f'{max_seq_length - special_tokens_count} tokens beforehand.')\n        tokens = tokens[: (max_seq_length - special_tokens_count)]\n        label_ids = label_ids[: (max_seq_length - special_tokens_count)]\n\n    # The convention in BERT is:\n    # (a) For sequence pairs:\n    #  tokens:   [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]\n    #  token_type_ids:   0   0  0    0    0     0       0   0   1  1  1  1   1   1\n    # (b) For single sequences:\n    #  tokens:   [CLS] the dog is hairy . [SEP]\n    #  token_type_ids:   0   0   0   0  0     0   0\n    #\n    # Where \"token_type_ids\" are used to indicate whether this is the first\n    # sequence or the second sequence. The embedding vectors for `type=0` and\n    # `type=1` were learned during pre-training and are added to the wordpiece\n    # embedding vector (and position vector). This is not *strictly* necessary\n    # since the [SEP] token unambiguously separates the sequences, but it makes\n    # it easier for the model to learn the concept of sequences.\n    #\n    # For classification tasks, the first vector (corresponding to [CLS]) is\n    # used as as the \"sentence vector\". Note that this only makes sense because\n    # the entire model is fine-tuned.\n    tokens += [sep_token]\n    label_ids += [pad_token_label_id]\n    if sep_token_extra:\n        # roberta uses an extra separator b/w pairs of sentences\n        tokens += [sep_token]\n        label_ids += [pad_token_label_id]\n    segment_ids = [sequence_a_segment_id] * len(tokens)\n\n    if cls_token_at_end:\n        tokens += [cls_token]\n        label_ids += [pad_token_label_id]\n        segment_ids += [cls_token_segment_id]\n    else:\n        tokens = [cls_token] + tokens\n        label_ids = [pad_token_label_id] + label_ids\n        segment_ids = [cls_token_segment_id] + segment_ids\n\n    input_ids = tokenizer.convert_tokens_to_ids(tokens)\n\n    # The mask has 1 for real tokens and 0 for padding tokens. Only real\n    # tokens are attended to.\n    input_mask = [1 if mask_padding_with_zero else 0] * len(input_ids)\n\n    if do_padding:\n        # Zero-pad up to the sequence length.\n        padding_length = max_seq_length - len(input_ids)\n        if pad_on_left:\n            input_ids = ([pad_token_id] * padding_length) + input_ids\n            input_mask = ([0 if mask_padding_with_zero else 1] * padding_length) + input_mask\n            segment_ids = ([pad_token_segment_id] * padding_length) + segment_ids\n            label_ids = ([pad_token_label_id] * padding_length) + label_ids\n        else:\n            input_ids += [pad_token_id] * padding_length\n            input_mask += [0 if mask_padding_with_zero else 1] * padding_length\n            segment_ids += [pad_token_segment_id] * padding_length\n            label_ids += [pad_token_label_id] * padding_length\n\n        assert len(input_ids) == max_seq_length\n        assert len(input_mask) == max_seq_length\n        assert len(segment_ids) == max_seq_length\n        assert len(label_ids) == max_seq_length, f'failed for:\\n {args}'\n    else:\n        assert len(set(len(x) for x in [input_ids, input_mask, segment_ids, label_ids])) == 1\n    return input_ids, input_mask, segment_ids, label_ids\n\n\ndef build_adamw_optimizer(config, learning_rate, epsilon, clipnorm, train_steps, use_amp, warmup_steps,\n                          weight_decay_rate):\n    opt = create_optimizer(init_lr=learning_rate,\n                           epsilon=epsilon,\n                           weight_decay_rate=weight_decay_rate,\n                           clipnorm=clipnorm,\n                           num_train_steps=train_steps, num_warmup_steps=warmup_steps)\n    # opt = tfa.optimizers.AdamW(learning_rate=3e-5, epsilon=1e-08, weight_decay=0.01)\n    # opt = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08)\n    config.optimizer = tf.keras.utils.serialize_keras_object(opt)\n    lr_config = config.optimizer['config']['learning_rate']['config']\n    if 'decay_schedule_fn' in lr_config:\n        lr_config['decay_schedule_fn'] = dict(\n            (k, v) for k, v in lr_config['decay_schedule_fn'].items() if not k.startswith('_'))\n    if use_amp:\n        # loss scaling is currently required when using mixed precision\n        opt = tf.keras.mixed_precision.experimental.LossScaleOptimizer(opt, 'dynamic')\n    return opt\n\n\ndef adjust_tokens_for_transformers(sentence):\n    \"\"\"Adjust tokens for BERT\n    See https://github.com/DoodleJZ/HPSG-Neural-Parser/blob/master/src_joint/Zparser.py#L1204\n\n    Args:\n      sentence: \n\n    Returns:\n\n    \n    \"\"\"\n    cleaned_words = []\n    for word in sentence:\n        # word = BERT_TOKEN_MAPPING.get(word, word)\n        if word == \"n't\" and cleaned_words:\n            cleaned_words[-1] = cleaned_words[-1] + \"n\"\n            word = \"'t\"\n        cleaned_words.append(word)\n    return cleaned_words\n"
  },
  {
    "path": "hanlp/layers/weight_normalization.py",
    "content": "# Copyright 2019 The TensorFlow Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# =============================================================================\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport tensorflow as tf\n\nfrom hanlp.utils.tf_util import hanlp_register\n\n\n@hanlp_register\nclass WeightNormalization(tf.keras.layers.Wrapper):\n    \"\"\"This wrapper reparameterizes a layer by decoupling the weight's\n    magnitude and direction.\n    \n    This speeds up convergence by improving the\n    conditioning of the optimization problem.\n    Weight Normalization: A Simple Reparameterization to Accelerate\n    Training of Deep Neural Networks: https://arxiv.org/abs/1602.07868\n    Tim Salimans, Diederik P. Kingma (2016)\n    WeightNormalization wrapper works for keras and tf layers.\n    ```python\n      net = WeightNormalization(\n          tf.keras.layers.Conv2D(2, 2, activation='relu'),\n          input_shape=(32, 32, 3),\n          data_init=True)(x)\n      net = WeightNormalization(\n          tf.keras.layers.Conv2D(16, 5, activation='relu'),\n          data_init=True)(net)\n      net = WeightNormalization(\n          tf.keras.layers.Dense(120, activation='relu'),\n          data_init=True)(net)\n      net = WeightNormalization(\n          tf.keras.layers.Dense(n_classes),\n          data_init=True)(net)\n    ```\n\n    Args:\n      layer: a layer instance\n      data_init: If\n\n    Returns:\n\n    Raises:\n      ValueError: If not initialized with a\n      ValueError: If\n      NotImplementedError: If\n\n    \"\"\"\n\n    def __init__(self, layer, data_init=True, **kwargs):\n        super(WeightNormalization, self).__init__(layer, **kwargs)\n        self.data_init = data_init\n        self._track_trackable(layer, name='layer')\n        self._init_critical_section = tf.CriticalSection(name='init_mutex')\n        self.is_rnn = isinstance(self.layer, tf.keras.layers.RNN)\n\n    def build(self, input_shape):\n        \"\"\"Build `Layer`\n\n        Args:\n          input_shape: \n\n        Returns:\n\n        \"\"\"\n        input_shape = tf.TensorShape(input_shape)\n        self.input_spec = tf.keras.layers.InputSpec(\n            shape=[None] + input_shape[1:])\n\n        if not self.layer.built:\n            self.layer.build(input_shape)\n\n        kernel_layer = self.layer.cell if self.is_rnn else self.layer\n\n        if not hasattr(kernel_layer, 'kernel'):\n            raise ValueError('`WeightNormalization` must wrap a layer that'\n                             ' contains a `kernel` for weights')\n\n        # The kernel's filter or unit dimension is -1\n        self.layer_depth = int(kernel_layer.kernel.shape[-1])\n        self.kernel_norm_axes = list(range(kernel_layer.kernel.shape.rank - 1))\n\n        self.g = self.add_weight(\n            name='g',\n            shape=(self.layer_depth,),\n            initializer='ones',\n            dtype=kernel_layer.kernel.dtype,\n            trainable=True)\n        self.v = kernel_layer.kernel\n\n        self._initialized = self.add_weight(\n            name='initialized',\n            shape=None,\n            initializer='zeros',\n            dtype=tf.dtypes.bool,\n            trainable=False)\n\n        if self.data_init:\n            # Used for data initialization in self._data_dep_init.\n            with tf.name_scope('data_dep_init'):\n                layer_config = tf.keras.layers.serialize(self.layer)\n                layer_config['config']['trainable'] = False\n                self._naked_clone_layer = tf.keras.layers.deserialize(\n                    layer_config)\n                self._naked_clone_layer.build(input_shape)\n                self._naked_clone_layer.set_weights(self.layer.get_weights())\n                if self.is_rnn:\n                    self._naked_clone_layer.cell.activation = None\n                else:\n                    self._naked_clone_layer.activation = None\n\n        self.built = True\n\n    def call(self, inputs):\n        \"\"\"Call `Layer`\n\n        Args:\n          inputs: \n\n        Returns:\n\n        \"\"\"\n\n        def _do_nothing():\n            return tf.identity(self.g)\n\n        def _update_weights():\n            # Ensure we read `self.g` after _update_weights.\n            with tf.control_dependencies(self._initialize_weights(inputs)):\n                return tf.identity(self.g)\n\n        g = self._init_critical_section.execute(lambda: tf.cond(\n            self._initialized, _do_nothing, _update_weights))\n\n        with tf.name_scope('compute_weights'):\n            # Replace kernel by normalized weight variable.\n            self.layer.kernel = tf.nn.l2_normalize(\n                self.v, axis=self.kernel_norm_axes) * g\n\n            # Ensure we calculate result after updating kernel.\n            update_kernel = tf.identity(self.layer.kernel)\n            with tf.control_dependencies([update_kernel]):\n                outputs = self.layer(inputs)\n                return outputs\n\n    def compute_output_shape(self, input_shape):\n        return tf.TensorShape(\n            self.layer.compute_output_shape(input_shape).as_list())\n\n    def _initialize_weights(self, inputs):\n        \"\"\"Initialize weight g.\n        \n        The initial value of g could either from the initial value in v,\n        or by the input value if self.data_init is True.\n\n        Args:\n          inputs: \n\n        Returns:\n\n        \"\"\"\n        with tf.control_dependencies([\n            tf.debugging.assert_equal(  # pylint: disable=bad-continuation\n                self._initialized,\n                False,\n                message='The layer has been initialized.')\n        ]):\n            if self.data_init:\n                assign_tensors = self._data_dep_init(inputs)\n            else:\n                assign_tensors = self._init_norm()\n            assign_tensors.append(self._initialized.assign(True))\n            return assign_tensors\n\n    def _init_norm(self):\n        \"\"\"Set the weight g with the norm of the weight vector.\"\"\"\n        with tf.name_scope('init_norm'):\n            v_flat = tf.reshape(self.v, [-1, self.layer_depth])\n            v_norm = tf.linalg.norm(v_flat, axis=0)\n            g_tensor = self.g.assign(tf.reshape(v_norm, (self.layer_depth,)))\n            return [g_tensor]\n\n    def _data_dep_init(self, inputs):\n        \"\"\"Data dependent initialization.\n\n        Args:\n          inputs: \n\n        Returns:\n\n        \"\"\"\n        with tf.name_scope('data_dep_init'):\n            # Generate data dependent init values\n            x_init = self._naked_clone_layer(inputs)\n            data_norm_axes = list(range(x_init.shape.rank - 1))\n            m_init, v_init = tf.nn.moments(x_init, data_norm_axes)\n            scale_init = 1. / tf.math.sqrt(v_init + 1e-10)\n\n            # Assign data dependent init values\n            g_tensor = self.g.assign(self.g * scale_init)\n            if hasattr(self.layer, 'bias') and self.layer.bias is not None:\n                bias_tensor = self.layer.bias.assign(-m_init * scale_init)\n                return [g_tensor, bias_tensor]\n            else:\n                return [g_tensor]\n\n    def get_config(self):\n        config = {'data_init': self.data_init}\n        base_config = super(WeightNormalization, self).get_config()\n        return dict(list(base_config.items()) + list(config.items()))\n"
  },
  {
    "path": "hanlp/losses/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-20 01:28"
  },
  {
    "path": "hanlp/losses/sparse_categorical_crossentropy.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-20 01:29\n\nimport tensorflow as tf\n\nfrom hanlp.utils.tf_util import hanlp_register\n\n\n@hanlp_register\nclass SparseCategoricalCrossentropyOverNonzeroWeights(object):\n    def __init__(self) -> None:\n        super().__init__()\n        self.__name__ = type(self).__name__\n\n    def __call__(self, y_true, y_pred, sample_weight=None, **kwargs):\n        loss = tf.keras.losses.sparse_categorical_crossentropy(y_true, y_pred, from_logits=True)\n        if sample_weight is not None:\n            loss = loss * sample_weight\n        loss = tf.reduce_sum(loss)\n        if sample_weight is not None:\n            # This is equivalent to SUM_OVER_BATCH_SIZE\n            # loss /= tf.reduce_sum(tf.ones_like(sample_weight, dtype=loss.dtype))\n            # This one is SUM_BY_NONZERO_WEIGHTS\n            loss /= tf.reduce_sum(sample_weight)\n        return loss\n\n\n@hanlp_register\nclass SparseCategoricalCrossentropyOverBatchFirstDim(object):\n\n    def __init__(self) -> None:\n        super().__init__()\n        self.__name__ = type(self).__name__\n\n    def __call__(self, y_true, y_pred, sample_weight=None, **kwargs):\n        loss = tf.keras.losses.sparse_categorical_crossentropy(y_true, y_pred, from_logits=True)\n        if sample_weight is not None:\n            loss = loss * sample_weight\n        # could use sum of sample_weight[:,0] too\n        loss = tf.reduce_sum(loss) / tf.cast(tf.shape(y_true)[0], tf.float32)\n        return loss\n\n    def get_config(self):\n        return {}\n\n\n@hanlp_register\nclass MaskedSparseCategoricalCrossentropyOverBatchFirstDim(object):\n    def __init__(self, mask_value=0) -> None:\n        super().__init__()\n        self.mask_value = mask_value\n        self.__name__ = type(self).__name__\n\n    def __call__(self, y_true, y_pred, sample_weight=None, **kwargs):\n        assert sample_weight is None, 'the mask will be computed via y_true != mask_value, ' \\\n                                      'it might conflict with sample_weight'\n        active_loss = tf.not_equal(y_true, self.mask_value)\n        active_labels = tf.boolean_mask(y_true, active_loss)\n        active_logits = tf.boolean_mask(y_pred, active_loss)\n        loss = tf.keras.losses.sparse_categorical_crossentropy(active_labels, active_logits, from_logits=True)\n        loss = tf.reduce_sum(loss) / tf.cast(tf.shape(y_true)[0], tf.float32)\n        return loss\n"
  },
  {
    "path": "hanlp/metrics/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-09-14 21:55"
  },
  {
    "path": "hanlp/metrics/accuracy.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-12 17:56\nfrom typing import Optional, Iterable\n\nimport torch\n\nfrom hanlp.metrics.metric import Metric\n\n\nclass CategoricalAccuracy(Metric):\n    \"\"\"\n    Categorical Top-K accuracy. Assumes integer labels, with\n    each item to be classified having a single correct class.\n    Tie break enables equal distribution of scores among the\n    classes with same maximum predicted scores.\n    Copied from AllenNLP and added several methods.\n    \"\"\"\n\n    def __init__(self, top_k: int = 1, tie_break: bool = False) -> None:\n        if top_k > 1 and tie_break:\n            raise ValueError(\n                \"Tie break in Categorical Accuracy can be done only for maximum (top_k = 1)\"\n            )\n        if top_k <= 0:\n            raise ValueError(\"top_k passed to Categorical Accuracy must be > 0\")\n        self._top_k = top_k\n        self._tie_break = tie_break\n        self.correct_count = 0.0\n        self.total_count = 0.0\n\n    def __call__(\n            self,\n            predictions: torch.Tensor,\n            gold_labels: torch.Tensor,\n            mask: Optional[torch.BoolTensor] = None,\n    ):\n        \"\"\"\n        # Parameters\n\n        predictions : `torch.Tensor`, required.\n            A tensor of predictions of shape (batch_size, ..., num_classes).\n        gold_labels : `torch.Tensor`, required.\n            A tensor of integer class label of shape (batch_size, ...). It must be the same\n            shape as the `predictions` tensor without the `num_classes` dimension.\n        mask : `torch.BoolTensor`, optional (default = `None`).\n            A masking tensor the same size as `gold_labels`.\n        \"\"\"\n        predictions, gold_labels, mask = self.detach_tensors(predictions, gold_labels, mask)\n\n        # Some sanity checks.\n        num_classes = predictions.size(-1)\n        if gold_labels.dim() != predictions.dim() - 1:\n            raise ValueError(\n                \"gold_labels must have dimension == predictions.size() - 1 but \"\n                \"found tensor of shape: {}\".format(predictions.size())\n            )\n        if (gold_labels >= num_classes).any():\n            raise ValueError(\n                \"A gold label passed to Categorical Accuracy contains an id >= {}, \"\n                \"the number of classes.\".format(num_classes)\n            )\n\n        predictions = predictions.view((-1, num_classes))\n        gold_labels = gold_labels.view(-1).long()\n        if not self._tie_break:\n            # Top K indexes of the predictions (or fewer, if there aren't K of them).\n            # Special case topk == 1, because it's common and .max() is much faster than .topk().\n            if self._top_k == 1:\n                top_k = predictions.max(-1)[1].unsqueeze(-1)\n            else:\n                top_k = predictions.topk(min(self._top_k, predictions.shape[-1]), -1)[1]\n\n            # This is of shape (batch_size, ..., top_k).\n            correct = top_k.eq(gold_labels.unsqueeze(-1)).float()\n        else:\n            # prediction is correct if gold label falls on any of the max scores. distribute score by tie_counts\n            max_predictions = predictions.max(-1)[0]\n            max_predictions_mask = predictions.eq(max_predictions.unsqueeze(-1))\n            # max_predictions_mask is (rows X num_classes) and gold_labels is (batch_size)\n            # ith entry in gold_labels points to index (0-num_classes) for ith row in max_predictions\n            # For each row check if index pointed by gold_label is was 1 or not (among max scored classes)\n            correct = max_predictions_mask[\n                torch.arange(gold_labels.numel(), device=gold_labels.device).long(), gold_labels\n            ].float()\n            tie_counts = max_predictions_mask.sum(-1)\n            correct /= tie_counts.float()\n            correct.unsqueeze_(-1)\n\n        if mask is not None:\n            correct *= mask.view(-1, 1)\n            self.total_count += mask.sum()\n        else:\n            self.total_count += gold_labels.numel()\n        self.correct_count += correct.sum()\n\n    @property\n    def score(self):\n        if self.total_count > 1e-12:\n            accuracy = float(self.correct_count) / float(self.total_count)\n        else:\n            accuracy = 0.0\n        return accuracy\n\n    def __repr__(self) -> str:\n        return f'Accuracy:{self.score:.2%}'\n\n    @staticmethod\n    def detach_tensors(*tensors: torch.Tensor) -> Iterable[torch.Tensor]:\n        \"\"\"\n        If you actually passed gradient-tracking Tensors to a Metric, there will be\n        a huge memory leak, because it will prevent garbage collection for the computation\n        graph. This method ensures the tensors are detached.\n        \"\"\"\n        # Check if it's actually a tensor in case something else was passed.\n        return (x.detach() if isinstance(x, torch.Tensor) else x for x in tensors)\n\n    def reset(self):\n        self.correct_count = 0.0\n        self.total_count = 0.0\n\n\nclass BooleanAccuracy(Metric):\n    \"\"\"\n    Just checks batch-equality of two tensors and computes an accuracy metric based on that.\n    That is, if your prediction has shape (batch_size, dim_1, ..., dim_n), this metric considers that\n    as a set of `batch_size` predictions and checks that each is *entirely* correct across the remaining dims.\n    This means the denominator in the accuracy computation is `batch_size`, with the caveat that predictions\n    that are totally masked are ignored (in which case the denominator is the number of predictions that have\n    at least one unmasked element).\n\n    This is similar to [`CategoricalAccuracy`](./categorical_accuracy.md), if you've already done a `.max()`\n    on your predictions.  If you have categorical output, though, you should typically just use\n    `CategoricalAccuracy`.  The reason you might want to use this instead is if you've done\n    some kind of constrained inference and don't have a prediction tensor that matches the API of\n    `CategoricalAccuracy`, which assumes a final dimension of size `num_classes`.\n    \"\"\"\n\n    def __init__(self) -> None:\n        self._correct_count = 0.0\n        self._total_count = 0.0\n\n    def __call__(\n            self,\n            predictions: torch.Tensor,\n            gold_labels: torch.Tensor,\n            mask: Optional[torch.BoolTensor] = None,\n    ):\n        \"\"\"\n        # Parameters\n\n        predictions : `torch.Tensor`, required.\n            A tensor of predictions of shape (batch_size, ...).\n        gold_labels : `torch.Tensor`, required.\n            A tensor of the same shape as `predictions`.\n        mask : `torch.BoolTensor`, optional (default = `None`).\n            A tensor of the same shape as `predictions`.\n        \"\"\"\n        predictions, gold_labels, mask = self.detach_tensors(predictions, gold_labels, mask)\n\n        # Some sanity checks.\n        if gold_labels.size() != predictions.size():\n            raise ValueError(\n                f\"gold_labels must have shape == predictions.size() but \"\n                f\"found tensor of shape: {gold_labels.size()}\"\n            )\n        if mask is not None and mask.size() != predictions.size():\n            raise ValueError(\n                f\"mask must have shape == predictions.size() but \"\n                f\"found tensor of shape: {mask.size()}\"\n            )\n\n        batch_size = predictions.size(0)\n\n        if mask is not None:\n            # We can multiply by the mask up front, because we're just checking equality below, and\n            # this way everything that's masked will be equal.\n            predictions = predictions * mask\n            gold_labels = gold_labels * mask\n\n            # We want to skip predictions that are completely masked;\n            # so we'll keep predictions that aren't.\n            keep = mask.view(batch_size, -1).max(dim=1)[0]\n        else:\n            keep = torch.ones(batch_size, device=predictions.device).bool()\n\n        predictions = predictions.view(batch_size, -1)\n        gold_labels = gold_labels.view(batch_size, -1)\n\n        # At this point, predictions is (batch_size, rest_of_dims_combined),\n        # so .eq -> .prod will be 1 if every element of the instance prediction is correct\n        # and 0 if at least one element of the instance prediction is wrong.\n        # Because of how we're handling masking, masked positions are automatically \"correct\".\n        correct = predictions.eq(gold_labels).prod(dim=1).float()\n\n        # Since masked positions are correct, we need to explicitly exclude instance predictions\n        # where the entire prediction is masked (because they look \"correct\").\n        self._correct_count += (correct * keep).sum()\n        self._total_count += keep.sum()\n\n    def get_metric(self, reset: bool = False):\n        \"\"\"\n        # Returns\n\n        The accumulated accuracy.\n        \"\"\"\n        if self._total_count > 0:\n            accuracy = float(self._correct_count) / float(self._total_count)\n        else:\n            accuracy = 0.0\n        if reset:\n            self.reset()\n        return accuracy\n\n    def reset(self):\n        self._correct_count = 0.0\n        self._total_count = 0.0\n\n    @staticmethod\n    def detach_tensors(*tensors: torch.Tensor) -> Iterable[torch.Tensor]:\n        \"\"\"\n        If you actually passed gradient-tracking Tensors to a Metric, there will be\n        a huge memory leak, because it will prevent garbage collection for the computation\n        graph. This method ensures the tensors are detached.\n        \"\"\"\n        # Check if it's actually a tensor in case something else was passed.\n        return (x.detach() if isinstance(x, torch.Tensor) else x for x in tensors)\n"
  },
  {
    "path": "hanlp/metrics/amr/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-24 12:47"
  },
  {
    "path": "hanlp/metrics/amr/smatch_eval.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-24 12:47\nimport os\nimport warnings\nfrom typing import Union\n\nfrom hanlp.metrics.f1 import F1_\nfrom hanlp.metrics.mtl import MetricDict\nfrom hanlp.utils.io_util import get_resource, run_cmd, pushd\nfrom hanlp.utils.log_util import flash\n\n_SMATCH_SCRIPT = 'https://github.com/ChunchuanLv/amr-evaluation-tool-enhanced/archive/master.zip#evaluation.sh'\n_FAST_SMATCH_SCRIPT = 'https://github.com/jcyk/AMR-gs/archive/master.zip#tools/fast_smatch/compute_smatch.sh'\n\n\nclass SmatchScores(MetricDict):\n    @property\n    def score(self):\n        return self['Smatch'].score\n\n\ndef smatch_eval(pred, gold, use_fast=False) -> Union[SmatchScores, F1_]:\n    script = get_resource(_FAST_SMATCH_SCRIPT if use_fast else _SMATCH_SCRIPT)\n    home = os.path.dirname(script)\n    pred = os.path.realpath(pred)\n    gold = os.path.realpath(gold)\n    with pushd(home):\n        flash('Running evaluation script [blink][yellow]...[/yellow][/blink]')\n        cmd = f'bash {script} {pred} {gold}'\n        text = run_cmd(cmd)\n        flash('')\n    return format_fast_scores(text) if use_fast else format_official_scores(text)\n\n\ndef post_process(pred, amr_version):\n    pred = os.path.realpath(pred)\n    utils_tar_gz = get_amr_utils(amr_version)\n    util_dir = get_resource(utils_tar_gz)\n    stog_home = get_resource('https://github.com/jcyk/AMR-gs/archive/master.zip')\n    with pushd(stog_home):\n        run_cmd(\n            f'python3 -u -m stog.data.dataset_readers.amr_parsing.postprocess.postprocess '\n            f'--amr_path {pred} --util_dir {util_dir} --v 2')\n    return pred + '.post'\n\n\ndef get_amr_utils(amr_version):\n    if amr_version == '1.0':\n        utils_tar_gz = 'https://www.cs.jhu.edu/~s.zhang/data/AMR/amr_1.0_utils.tar.gz'\n    elif amr_version == '2.0':\n        utils_tar_gz = 'https://www.cs.jhu.edu/~s.zhang/data/AMR/amr_2.0_utils.tar.gz'\n    elif amr_version == '3.0':\n        utils_tar_gz = 'https://file.hankcs.com/research/amr2020/amr_3.0_utils.tgz'\n    else:\n        raise ValueError(f'Unsupported AMR version {amr_version}')\n    return utils_tar_gz\n\n\ndef format_official_scores(text: str):\n    # Smatch -> P: 0.136, R: 0.107, F: 0.120\n    # Unlabeled -> P: 0.229, R: 0.180, F: 0.202\n    # No WSD -> P: 0.137, R: 0.108, F: 0.120\n    # Non_sense_frames -> P: 0.008, R: 0.008, F: 0.008\n    # Wikification -> P: 0.000, R: 0.000, F: 0.000\n    # Named Ent. -> P: 0.222, R: 0.092, F: 0.130\n    # Negations -> P: 0.000, R: 0.000, F: 0.000\n    # IgnoreVars -> P: 0.005, R: 0.003, F: 0.003\n    # Concepts -> P: 0.075, R: 0.036, F: 0.049\n    # Frames -> P: 0.007, R: 0.007, F: 0.007\n    # Reentrancies -> P: 0.113, R: 0.060, F: 0.079\n    # SRL -> P: 0.145, R: 0.104, F: 0.121\n    scores = SmatchScores()\n    for line in text.split('\\n'):\n        line = line.strip()\n        if not line:\n            continue\n        name, vs = line.split(' -> ')\n        try:\n            p, r, f = [float(x.split(': ')[-1]) for x in vs.split(', ')]\n        except ValueError:\n            warnings.warn(f'Failed to parse results from smatch: {line}')\n            p, r, f = float(\"nan\"), float(\"nan\"), float(\"nan\")\n        scores[name] = F1_(p, r, f)\n    return scores\n\n\ndef format_fast_scores(text: str):\n    # using fast smatch\n    # Precision: 0.137\n    # Recall: 0.108\n    # Document F-score: 0.121\n    scores = []\n    for line in text.split('\\n'):\n        line = line.strip()\n        if not line or ':' not in line:\n            continue\n        name, score = line.split(': ')\n        scores.append(float(score))\n    assert len(scores) == 3\n    return F1_(*scores)\n"
  },
  {
    "path": "hanlp/metrics/chunking/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-21 03:49"
  },
  {
    "path": "hanlp/metrics/chunking/binary_chunking_f1.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-02 14:27\nfrom collections import defaultdict\nfrom typing import List, Union\n\nimport torch\n\nfrom hanlp.metrics.f1 import F1\n\n\nclass BinaryChunkingF1(F1):\n    def __call__(self, pred_tags: torch.LongTensor, gold_tags: torch.LongTensor, lens: List[int] = None):\n        if lens is None:\n            lens = [gold_tags.size(1)] * gold_tags.size(0)\n        self.update(self.decode_spans(pred_tags, lens), self.decode_spans(gold_tags, lens))\n\n    def update(self, pred_tags, gold_tags):\n        for pred, gold in zip(pred_tags, gold_tags):\n            super().__call__(set(pred), set(gold))\n\n    @staticmethod\n    def decode_spans(pred_tags: torch.LongTensor, lens: Union[List[int], torch.LongTensor]):\n        if isinstance(lens, torch.Tensor):\n            lens = lens.tolist()\n        batch_pred = defaultdict(list)\n        for batch, offset in pred_tags.nonzero(as_tuple=False).tolist():\n            batch_pred[batch].append(offset)\n        batch_pred_spans = [[(0, l)] for l in lens]\n        for batch, offsets in batch_pred.items():\n            l = lens[batch]\n            batch_pred_spans[batch] = list(zip(offsets, offsets[1:] + [l]))\n        return batch_pred_spans\n"
  },
  {
    "path": "hanlp/metrics/chunking/bmes_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-09-14 21:55\n\nfrom hanlp.common.vocab_tf import VocabTF\nfrom hanlp.metrics.chunking.chunking_f1_tf import ChunkingF1_TF\nfrom hanlp.metrics.chunking.sequence_labeling import get_entities\n\n\nclass BMES_F1_TF(ChunkingF1_TF):\n\n    def __init__(self, tag_vocab: VocabTF, from_logits=True, suffix=False, name='f1', dtype=None, **kwargs):\n        super().__init__(tag_vocab, from_logits, name, dtype, **kwargs)\n        self.nb_correct = 0\n        self.nb_pred = 0\n        self.nb_true = 0\n        self.suffix = suffix\n\n    def update_tags(self, true_tags, pred_tags):\n        for t, p in zip(true_tags, pred_tags):\n            self.update_entities(get_entities(t, self.suffix), get_entities(p, self.suffix))\n        return self.result()\n\n    def update_entities(self, true_entities, pred_entities):\n        true_entities = set(true_entities)\n        pred_entities = set(pred_entities)\n        nb_correct = len(true_entities & pred_entities)\n        nb_pred = len(pred_entities)\n        nb_true = len(true_entities)\n        self.nb_correct += nb_correct\n        self.nb_pred += nb_pred\n        self.nb_true += nb_true\n\n    def result(self):\n        nb_correct = self.nb_correct\n        nb_pred = self.nb_pred\n        nb_true = self.nb_true\n        p = nb_correct / nb_pred if nb_pred > 0 else 0\n        r = nb_correct / nb_true if nb_true > 0 else 0\n        score = 2 * p * r / (p + r) if p + r > 0 else 0\n\n        return score\n\n    def reset_states(self):\n        self.nb_correct = 0\n        self.nb_pred = 0\n        self.nb_true = 0\n"
  },
  {
    "path": "hanlp/metrics/chunking/chunking_f1.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-11 22:14\nimport io\nfrom collections import defaultdict\nfrom typing import List, Set, Tuple, Dict\n\nfrom hanlp.metrics.chunking.conlleval import calculate_metrics, DetailedF1, metrics\nfrom hanlp.metrics.chunking.sequence_labeling import get_entities\nfrom hanlp.metrics.f1 import F1\nfrom hanlp.metrics.metric import Metric\n\n\nclass ChunkingF1(F1):\n\n    def __call__(self, pred_tags: List[List[str]], gold_tags: List[List[str]]):\n        for p, g in zip(pred_tags, gold_tags):\n            pred = set(get_entities(p))\n            gold = set(get_entities(g))\n            self.nb_pred += len(pred)\n            self.nb_true += len(gold)\n            self.nb_correct += len(pred & gold)\n\n\nclass DetailedSpanF1(Metric):\n    def __init__(self, do_confusion_matrix=False):\n        self.correct_chunk = 0  # number of correctly identified chunks\n        self.correct_unlabeled = 0\n        self.total_gold = 0  # number of chunks in corpus\n        self.total_pred = 0  # number of identified chunks\n        self.token_counter = 0  # token counter (ignores sentence breaks)\n\n        # counts by type\n        self.t_correct_chunk = defaultdict(int)\n        self.t_total_gold = defaultdict(int)\n        self.t_total_pred = defaultdict(int)\n\n        self.do_confusion_matrix = do_confusion_matrix\n        if do_confusion_matrix:\n            self.pred_labels = []\n            self.gold_labels = []\n\n    @property\n    def states(self):\n        return (self.t_correct_chunk, self.t_total_gold, self.t_total_pred)\n\n    def reset_state(self):\n        self.correct_chunk = 0  # number of correctly identified chunks\n        self.total_gold = 0  # number of chunks in corpus\n        self.total_pred = 0  # number of identified chunks\n        self.token_counter = 0  # token counter (ignores sentence breaks)\n        for state in self.states:\n            state.clear()\n        if self.do_confusion_matrix:\n            self.pred_labels = []\n            self.gold_labels = []\n\n    @property\n    def score(self):\n        overall = calculate_metrics(\n            self.correct_chunk, self.total_pred, self.total_gold\n        )\n        return overall.fscore\n\n    def __call__(self, pred: Set[Tuple[int, int, str]], gold: Set[Tuple[int, int, str]], num_tokens=None):\n        pred_chunks_unlabeled = set((b, e) for b, e, l in pred)\n        gold_chunks_unlabeled = set((b, e) for b, e, l in gold)\n        self.correct_unlabeled += len(pred_chunks_unlabeled & gold_chunks_unlabeled)\n        self.correct_chunk += len(pred & gold)\n        self.total_gold += len(gold)\n        self.total_pred += len(pred)\n        if num_tokens:\n            self.token_counter += num_tokens\n\n        def group_by_tag(collection: Set[Tuple[int, int, str]]):\n            group = defaultdict(set)\n            for b, e, l in collection:\n                group[l].add((b, e))\n            return group\n\n        pred_tags = group_by_tag(pred)\n        gold_tags = group_by_tag(gold)\n        for l in pred_tags.keys() | gold_tags.keys():\n            self.t_correct_chunk[l] += len(pred_tags[l] & gold_tags[l])\n            self.t_total_gold[l] += len(gold_tags[l])\n            self.t_total_pred[l] += len(pred_tags[l])\n\n        if self.do_confusion_matrix:\n            def group_by_span(collection: Set[Tuple[int, int, str]]):\n                group = dict()\n                for b, e, l in collection:\n                    group[(b, e)] = l\n                return group\n\n            pred_spans = group_by_span(pred)\n            gold_spans = group_by_span(gold)\n            for span in pred_spans.keys() & gold_spans.keys():\n                self.pred_labels.append(pred_spans[span])\n                self.gold_labels.append(gold_spans[span])\n\n    def reset(self):\n        self.reset_state()\n\n    def report(self) -> Tuple[DetailedF1, Dict[str, DetailedF1], str]:\n        out = io.StringIO()\n\n        c = self\n        out.write('processed %d tokens with %d phrases; ' % (c.token_counter, c.total_gold))\n        out.write('found: %d phrases; correct: %d.\\n' % (c.total_pred, c.correct_chunk))\n\n        overall = calculate_metrics(c.correct_unlabeled, c.total_pred, c.total_gold)\n        out.write('%17s: ' % 'unlabeled overall')\n        out.write('precision: %6.2f%%; ' % (100. * overall.prec))\n        out.write('recall: %6.2f%%; ' % (100. * overall.rec))\n        out.write('FB1: %6.2f\\n' % (100. * overall.fscore))\n\n        overall, by_type = metrics(self)\n        out.write('%17s: ' % 'labeled overall')\n        out.write('precision: %6.2f%%; ' % (100. * overall.prec))\n        out.write('recall: %6.2f%%; ' % (100. * overall.rec))\n        out.write('FB1: %6.2f\\n' % (100. * overall.fscore))\n\n        for i, m in sorted(by_type.items()):\n            out.write('%17s: ' % i)\n            out.write('precision: %6.2f%%; ' % (100. * m.prec))\n            out.write('recall: %6.2f%%; ' % (100. * m.rec))\n            out.write('FB1: %6.2f  %d\\n' % (100. * m.fscore, c.t_total_pred[i]))\n        text = out.getvalue()\n        out.close()\n        return overall, by_type, text\n\n    def __str__(self) -> str:\n        return self.report()[-1]\n\n    def confusion_matrix(self):\n        from sklearn.metrics import confusion_matrix\n        labels = sorted(self.gold_labels + self.pred_labels)\n        return confusion_matrix(self.gold_labels, self.pred_labels, labels=labels), labels\n"
  },
  {
    "path": "hanlp/metrics/chunking/chunking_f1_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-29 23:09\nfrom abc import ABC, abstractmethod\n\nimport tensorflow as tf\n\nfrom hanlp.common.vocab_tf import VocabTF\n\n\nclass ChunkingF1_TF(tf.keras.metrics.Metric, ABC):\n\n    def __init__(self, tag_vocab: VocabTF, from_logits=True, name='f1', dtype=None, **kwargs):\n        super().__init__(name, dtype, dynamic=True, **kwargs)\n        self.tag_vocab = tag_vocab\n        self.from_logits = from_logits\n\n    def update_the_state(self, y_true: tf.Tensor, y_pred: tf.Tensor, sample_weight: tf.Tensor = None, **kwargs):\n        if sample_weight is None:\n            if hasattr(y_pred, '_keras_mask'):\n                mask = y_pred._keras_mask\n            else:\n                mask = None\n        else:\n            mask = sample_weight\n        if self.tag_vocab.pad_idx is not None and mask is None:\n            # in this case, the model doesn't compute mask but provide a masking index, it's ok to\n            mask = y_true != self.tag_vocab.pad_idx\n        assert mask is not None, 'ChunkingF1 requires masking, check your _keras_mask or compute_mask'\n        if self.from_logits:\n            y_pred = tf.argmax(y_pred, axis=-1)\n        y_true = self.to_tags(y_true, mask)\n        y_pred = self.to_tags(y_pred, mask)\n        return self.update_tags(y_true, y_pred)\n\n    def __call__(self, y_true: tf.Tensor, y_pred: tf.Tensor, sample_weight: tf.Tensor = None, **kwargs):\n        return self.update_the_state(y_true, y_pred, sample_weight)\n\n    def update_state(self, y_true: tf.Tensor, y_pred: tf.Tensor, sample_weight: tf.Tensor = None, **kwargs):\n        return self.update_the_state(y_true, y_pred, sample_weight)\n\n    def to_tags(self, y: tf.Tensor, sample_weight: tf.Tensor):\n        batch = []\n        y = y.numpy()\n        sample_weight = sample_weight.numpy()\n        for sent, mask in zip(y, sample_weight):\n            tags = []\n            for tag, m in zip(sent, mask):\n                if not m:\n                    continue\n                tag = int(tag)\n                if self.tag_vocab.pad_idx is not None and tag == self.tag_vocab.pad_idx:\n                    # If model predicts <pad>, it will fail most metrics. So replace it with a valid one\n                    tag = 1\n                tags.append(self.tag_vocab.get_token(tag))\n            batch.append(tags)\n        return batch\n\n    @abstractmethod\n    def update_tags(self, true_tags, pred_tags):\n        pass\n\n    @abstractmethod\n    def result(self):\n        pass\n"
  },
  {
    "path": "hanlp/metrics/chunking/conlleval.py",
    "content": "#!/usr/bin/env python\n\n# Python version of the evaluation script from CoNLL'00-\n\n# Intentional differences:\n# - accept any space as delimiter by default\n# - optional file argument (default STDIN)\n# - option to set boundary (-b argument)\n# - LaTeX output (-l argument) not supported\n# - raw tags (-r argument) not supported\nimport io\nimport sys\n\nfrom collections import defaultdict, namedtuple\nfrom typing import Tuple, Union, List\n\nfrom hanlp.utils.span_util import bio_tags_to_spans\n\nfrom hanlp.metrics.metric import Metric\n\nANY_SPACE = '<SPACE>'\n\n\nclass FormatError(Exception):\n    pass\n\n\nDetailedF1 = namedtuple('Metrics', 'tp fp fn prec rec fscore')\n\n\nclass EvalCounts(object):\n    def __init__(self):\n        self.correct_chunk = 0  # number of correctly identified chunks\n        self.correct_tags = 0  # number of correct chunk tags\n        self.total_gold = 0  # number of chunks in corpus\n        self.total_pred = 0  # number of identified chunks\n        self.token_counter = 0  # token counter (ignores sentence breaks)\n\n        # counts by type\n        self.t_correct_chunk = defaultdict(int)\n        self.t_total_gold = defaultdict(int)\n        self.t_total_pred = defaultdict(int)\n\n    @property\n    def states(self):\n        return (self.t_correct_chunk, self.t_total_gold, self.t_total_pred)\n\n    def reset_state(self):\n        self.correct_chunk = 0  # number of correctly identified chunks\n        self.correct_tags = 0  # number of correct chunk tags\n        self.total_gold = 0  # number of chunks in corpus\n        self.total_pred = 0  # number of identified chunks\n        self.token_counter = 0  # token counter (ignores sentence breaks)\n        for state in self.states:\n            state.clear()\n\n\nclass SpanF1(Metric):\n\n    def __init__(self, label_encoding='IOBES') -> None:\n        super().__init__()\n        self.label_encoding = label_encoding\n        self.count = EvalCounts()\n\n    def reset(self):\n        self.count = EvalCounts()\n\n    @property\n    def score(self):\n        return self.result(False, False).fscore\n\n    def reset_state(self):\n        self.count.reset_state()\n\n    def update_state(self, true_seqs: List[str], pred_seqs: List[str]):\n        if self.label_encoding == 'IOBES':\n            count = evaluate_iobes(true_seqs, pred_seqs)\n        elif self.label_encoding in ['IOB2', 'BIO']:\n            count = evaluate_iob2(true_seqs, pred_seqs)\n        else:\n            raise ValueError(f'Unrecognized label encoding {self.label_encoding}')\n        self.count.correct_chunk += count.correct_chunk\n        self.count.correct_tags += count.correct_tags\n        self.count.total_gold += count.total_gold\n        self.count.total_pred += count.total_pred\n        self.count.token_counter += count.token_counter\n        for s, n in zip(self.count.states, count.states):\n            for k, v in n.items():\n                s[k] = s.get(k, 0) + v\n\n    def batch_update_state(self, true_seqs: List[List[str]], pred_seqs: List[List[str]]):\n        for t, p in zip(true_seqs, pred_seqs):\n            self.update_state(t, p)\n\n    def result(self, full=True, verbose=True) -> Union[Tuple[DetailedF1, dict, str], DetailedF1]:\n        if full:\n            out = io.StringIO()\n            overall, by_type = report(self.count, out)\n            text = out.getvalue()\n            if verbose:\n                print(text)\n            out.close()\n            return overall, by_type, text\n        else:\n            overall, _ = metrics(self.count)\n            return overall\n\n    # torch convention: put pred before gold\n    def __call__(self, pred_seqs: List[List[str]], true_seqs: List[List[str]]):\n        return self.batch_update_state(true_seqs, pred_seqs)\n\n    def __repr__(self) -> str:\n        result = self.result(False, False)\n        return f\"P: {result.prec:.2%} R: {result.rec:.2%} F: {result.fscore:.2%}\"\n\n\ndef parse_args(argv):\n    import argparse\n    parser = argparse.ArgumentParser(\n        description='evaluate tagging results using CoNLL criteria',\n        formatter_class=argparse.ArgumentDefaultsHelpFormatter\n    )\n    arg = parser.add_argument\n    arg('-b', '--boundary', metavar='STR', default='-X-',\n        help='sentence boundary')\n    arg('-d', '--delimiter', metavar='CHAR', default=ANY_SPACE,\n        help='character delimiting items in input')\n    arg('-o', '--otag', metavar='CHAR', default='O',\n        help='alternative outside tag')\n    arg('file', nargs='?', default=None)\n    return parser.parse_args(argv)\n\n\ndef split_tag(chunk_tag):\n    \"\"\"split chunk tag into IOBES prefix and chunk_type\n    e.g.\n    B-PER -> (B, PER)\n    O -> (O, None)\n\n    Args:\n      chunk_tag: \n\n    Returns:\n\n    \"\"\"\n    if chunk_tag == 'O':\n        return ('O', None)\n    return chunk_tag.split('-', maxsplit=1)\n\n\ndef evaluate_iobes(true_seqs, pred_seqs):\n    counts = EvalCounts()\n    in_correct = False  # currently processed chunks is correct until now\n    last_correct = 'O'  # previous chunk tag in corpus\n    last_correct_type = ''  # type of previously identified chunk tag\n    last_guessed = 'O'  # previously identified chunk tag\n    last_guessed_type = ''  # type of previous chunk tag in corpus\n\n    for true_tag, pred_tag in zip(true_seqs, pred_seqs):\n\n        guessed, guessed_type = split_tag(pred_tag)\n        correct, correct_type = split_tag(true_tag)\n\n        end_correct = end_of_chunk(last_correct, correct,\n                                   last_correct_type, correct_type)\n        end_guessed = end_of_chunk(last_guessed, guessed,\n                                   last_guessed_type, guessed_type)\n        start_correct = start_of_chunk(last_correct, correct,\n                                       last_correct_type, correct_type)\n        start_guessed = start_of_chunk(last_guessed, guessed,\n                                       last_guessed_type, guessed_type)\n\n        if in_correct:\n            if (end_correct and end_guessed and\n                    last_guessed_type == last_correct_type):\n                in_correct = False\n                counts.correct_chunk += 1\n                counts.t_correct_chunk[last_correct_type] += 1\n            elif (end_correct != end_guessed or guessed_type != correct_type):\n                in_correct = False\n\n        if start_correct and start_guessed and guessed_type == correct_type:\n            in_correct = True\n\n        if start_correct:\n            counts.total_gold += 1\n            counts.t_total_gold[correct_type] += 1\n        if start_guessed:\n            counts.total_pred += 1\n            counts.t_total_pred[guessed_type] += 1\n        if correct == guessed and guessed_type == correct_type:\n            counts.correct_tags += 1\n        counts.token_counter += 1\n\n        last_guessed = guessed\n        last_correct = correct\n        last_guessed_type = guessed_type\n        last_correct_type = correct_type\n\n    if in_correct:\n        counts.correct_chunk += 1\n        counts.t_correct_chunk[last_correct_type] += 1\n\n    return counts\n\n\ndef evaluate_iob2(true_seqs, pred_seqs):\n    counts = EvalCounts()\n    gold = set(bio_tags_to_spans(true_seqs))\n    pred = set(bio_tags_to_spans(pred_seqs))\n    counts.correct_chunk = len(gold & pred)\n    counts.total_pred = len(pred)\n    counts.total_gold = len(gold)\n    return counts\n\n\ndef uniq(iterable):\n    seen = set()\n    return [i for i in iterable if not (i in seen or seen.add(i))]\n\n\ndef calculate_metrics(correct, guessed, total):\n    tp, fp, fn = correct, guessed - correct, total - correct\n    p = 0. if tp + fp == 0 else 1. * tp / (tp + fp)\n    r = 0. if tp + fn == 0 else 1. * tp / (tp + fn)\n    f = 0. if p + r == 0 else 2 * p * r / (p + r)\n    return DetailedF1(tp, fp, fn, p, r, f)\n\n\ndef calc_metrics(tp, p, t, percent=True):\n    \"\"\"compute overall precision, recall and FB1 (default values are 0.0)\n    if percent is True, return 100 * original decimal value\n\n    Args:\n      tp: \n      p: \n      t: \n      percent:  (Default value = True)\n\n    Returns:\n\n    \"\"\"\n    precision = tp / p if p else 0\n    recall = tp / t if t else 0\n    fb1 = 2 * precision * recall / (precision + recall) if precision + recall else 0\n    if percent:\n        return 100 * precision, 100 * recall, 100 * fb1\n    else:\n        return precision, recall, fb1\n\n\ndef metrics(counts):\n    c = counts\n    overall = calculate_metrics(\n        c.correct_chunk, c.total_pred, c.total_gold\n    )\n    by_type = {}\n    for t in uniq(list(c.t_total_gold.keys()) + list(c.t_total_pred.keys())):\n        by_type[t] = calculate_metrics(\n            c.t_correct_chunk[t], c.t_total_pred[t], c.t_total_gold[t]\n        )\n    return overall, by_type\n\n\ndef report(counts, out=None):\n    if out is None:\n        out = sys.stdout\n\n    overall, by_type = metrics(counts)\n\n    c = counts\n    out.write('processed %d tokens with %d phrases; ' %\n              (c.token_counter, c.total_gold))\n    out.write('found: %d phrases; correct: %d.\\n' %\n              (c.total_pred, c.correct_chunk))\n\n    if c.token_counter > 0:\n        out.write('accuracy: %6.2f%%; ' %\n                  (100. * c.correct_tags / c.token_counter))\n        out.write('precision: %6.2f%%; ' % (100. * overall.prec))\n        out.write('recall: %6.2f%%; ' % (100. * overall.rec))\n        out.write('FB1: %6.2f\\n' % (100. * overall.fscore))\n\n    for i, m in sorted(by_type.items()):\n        out.write('%17s: ' % i)\n        out.write('precision: %6.2f%%; ' % (100. * m.prec))\n        out.write('recall: %6.2f%%; ' % (100. * m.rec))\n        out.write('FB1: %6.2f  %d\\n' % (100. * m.fscore, c.t_total_pred[i]))\n    return overall, by_type\n\n\ndef end_of_chunk(prev_tag, tag, prev_type, type_):\n    # check if a chunk ended between the previous and current word\n    # arguments: previous and current chunk tags, previous and current types\n    return ((prev_tag == \"B\" and tag == \"B\") or\n            (prev_tag == \"B\" and tag == \"O\") or\n            (prev_tag == \"I\" and tag == \"B\") or\n            (prev_tag == \"I\" and tag == \"O\") or\n\n            (prev_tag == \"E\" and tag == \"E\") or\n            (prev_tag == \"E\" and tag == \"I\") or\n            (prev_tag == \"E\" and tag == \"O\") or\n            (prev_tag == \"I\" and tag == \"O\") or\n\n            (prev_tag != \"O\" and prev_tag != \".\" and prev_type != type_) or\n            (prev_tag == \"]\" or prev_tag == \"[\"))\n\n\ndef start_of_chunk(prev_tag, tag, prev_type, type_):\n    # check if a chunk started between the previous and current word\n    # arguments: previous and current chunk tags, previous and current types\n    chunkStart = ((prev_tag == \"B\" and tag == \"B\") or\n                  (prev_tag == \"B\" and tag == \"B\") or\n                  (prev_tag == \"I\" and tag == \"B\") or\n                  (prev_tag == \"O\" and tag == \"B\") or\n                  (prev_tag == \"O\" and tag == \"I\") or\n\n                  (prev_tag == \"E\" and tag == \"E\") or\n                  (prev_tag == \"E\" and tag == \"I\") or\n                  (prev_tag == \"O\" and tag == \"E\") or\n                  (prev_tag == \"O\" and tag == \"I\") or\n\n                  (tag != \"O\" and tag != \".\" and prev_type != type_) or\n                  (tag == \"]\" or tag == \"[\"))\n    # corrected 1998-12-22: these chunks are assumed to have length 1\n\n    # print(\"startOfChunk?\", prevTag, tag, prevType, type)\n    # print(chunkStart)\n    return chunkStart\n\n\ndef main(argv):\n    args = parse_args(argv[1:])\n\n    if args.file is None:\n        counts = evaluate_iobes(sys.stdin, args)\n    else:\n        with open(args.file, encoding='utf-8') as f:\n            counts = evaluate_iobes(f, args)\n    report(counts)\n\n\nif __name__ == '__main__':\n    sys.exit(main(sys.argv))\n"
  },
  {
    "path": "hanlp/metrics/chunking/iobes_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-09-14 21:55\n\nfrom hanlp.common.vocab_tf import VocabTF\nfrom hanlp.metrics.chunking.conlleval import SpanF1\nfrom hanlp.metrics.chunking.chunking_f1_tf import ChunkingF1_TF\n\n\nclass IOBES_F1_TF(ChunkingF1_TF):\n\n    def __init__(self, tag_vocab: VocabTF, from_logits=True, name='f1', dtype=None, **kwargs):\n        super().__init__(tag_vocab, from_logits, name, dtype, **kwargs)\n        self.state = SpanF1()\n\n    def update_tags(self, true_tags, pred_tags):\n        # true_tags = list(itertools.chain.from_iterable(true_tags))\n        # pred_tags = list(itertools.chain.from_iterable(pred_tags))\n        # self.state.update_state(true_tags, pred_tags)\n        for gold, pred in zip(true_tags, pred_tags):\n            self.state.update_state(gold, pred)\n        return self.result()\n\n    def result(self):\n        return self.state.result(full=False, verbose=False).fscore\n\n    def reset_states(self):\n        self.state.reset_state()\n"
  },
  {
    "path": "hanlp/metrics/chunking/sequence_labeling.py",
    "content": "# MIT License\n#\n# Copyright (c) 2018 chakki\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\n\"\"\"Metrics to assess performance on sequence labeling task given prediction\nFunctions named as ``*_score`` return a scalar value to maximize: the higher\nthe better\n\"\"\"\n\nfrom collections import defaultdict\nimport numpy as np\n\n\ndef iobes_to_span(words, tags):\n    delimiter = ' '\n    if all([len(w) == 1 for w in words]):\n        delimiter = ''  # might be Chinese\n    entities = []\n    for tag, start, end in get_entities(tags):\n        entities.append((delimiter.join(words[start:end]), tag, start, end))\n    yield entities\n\n\ndef get_entities(seq, suffix=False):\n    \"\"\"Gets entities from sequence.\n\n    Args:\n      seq(list): sequence of labels.\n      suffix:  (Default value = False)\n\n    Returns:\n      list: list of (chunk_type, chunk_start, chunk_end).\n      Example:\n\n    >>> from seqeval.metrics.sequence_labeling import get_entities\n        >>> seq = ['B-PER', 'I-PER', 'O', 'B-LOC']\n        >>> get_entities(seq)\n        [('PER', 0, 2), ('LOC', 3, 4)]\n    \"\"\"\n    # for nested list\n    if any(isinstance(s, list) for s in seq):\n        seq = [item for sublist in seq for item in sublist + ['O']]\n\n    prev_tag = 'O'\n    prev_type = ''\n    begin_offset = 0\n    chunks = []\n    for i, chunk in enumerate(seq + ['O']):\n        if suffix:\n            tag = chunk[-1]\n            type_ = chunk[:-2]\n        else:\n            tag = chunk[0]\n            type_ = chunk[2:]\n\n        if end_of_chunk(prev_tag, tag, prev_type, type_):\n            chunks.append((prev_type, begin_offset, i))\n        if start_of_chunk(prev_tag, tag, prev_type, type_):\n            begin_offset = i\n        prev_tag = tag\n        prev_type = type_\n\n    return chunks\n\n\ndef end_of_chunk(prev_tag, tag, prev_type, type_):\n    \"\"\"Checks if a chunk ended between the previous and current word.\n\n    Args:\n      prev_tag: previous chunk tag.\n      tag: current chunk tag.\n      prev_type: previous type.\n      type_: current type.\n\n    Returns:\n      chunk_end: boolean.\n\n    \"\"\"\n    chunk_end = False\n\n    if prev_tag == 'E': chunk_end = True\n    if prev_tag == 'S': chunk_end = True\n\n    if prev_tag == 'B' and tag == 'B': chunk_end = True\n    if prev_tag == 'B' and tag == 'S': chunk_end = True\n    if prev_tag == 'B' and tag == 'O': chunk_end = True\n    if prev_tag == 'I' and tag == 'B': chunk_end = True\n    if prev_tag == 'I' and tag == 'S': chunk_end = True\n    if prev_tag == 'I' and tag == 'O': chunk_end = True\n\n    if prev_tag != 'O' and prev_tag != '.' and prev_type != type_:\n        chunk_end = True\n\n    return chunk_end\n\n\ndef start_of_chunk(prev_tag, tag, prev_type, type_):\n    \"\"\"Checks if a chunk started between the previous and current word.\n\n    Args:\n      prev_tag: previous chunk tag.\n      tag: current chunk tag.\n      prev_type: previous type.\n      type_: current type.\n\n    Returns:\n      chunk_start: boolean.\n\n    \"\"\"\n    chunk_start = False\n\n    if tag == 'B': chunk_start = True\n    if tag == 'S': chunk_start = True\n\n    if prev_tag == 'E' and tag == 'E': chunk_start = True\n    if prev_tag == 'E' and tag == 'I': chunk_start = True\n    if prev_tag == 'S' and tag == 'E': chunk_start = True\n    if prev_tag == 'S' and tag == 'I': chunk_start = True\n    if prev_tag == 'O' and tag == 'E': chunk_start = True\n    if prev_tag == 'O' and tag == 'I': chunk_start = True\n\n    if tag != 'O' and tag != '.' and prev_type != type_:\n        chunk_start = True\n\n    return chunk_start\n\n\ndef f1_score(y_true, y_pred, average='micro', suffix=False):\n    \"\"\"Compute the F1 score.\n    \n    The F1 score can be interpreted as a weighted average of the precision and\n    recall, where an F1 score reaches its best value at 1 and worst score at 0.\n    The relative contribution of precision and recall to the F1 score are\n    equal. The formula for the F1 score is::\n    \n        F1 = 2 * (precision * recall) / (precision + recall)\n\n    Args:\n      y_true: 2d array. Ground truth (correct) target values.\n      y_pred: 2d array. Estimated targets as returned by a tagger.\n      average:  (Default value = 'micro')\n      suffix:  (Default value = False)\n\n    Returns:\n      score: float.\n      Example:\n\n    >>> from seqeval.metrics import f1_score\n        >>> y_true = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]\n        >>> y_pred = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]\n        >>> f1_score(y_true, y_pred)\n        0.50\n    \"\"\"\n    true_entities = set(get_entities(y_true, suffix))\n    pred_entities = set(get_entities(y_pred, suffix))\n\n    nb_correct = len(true_entities & pred_entities)\n    nb_pred = len(pred_entities)\n    nb_true = len(true_entities)\n\n    p = nb_correct / nb_pred if nb_pred > 0 else 0\n    r = nb_correct / nb_true if nb_true > 0 else 0\n    score = 2 * p * r / (p + r) if p + r > 0 else 0\n\n    return score\n\n\ndef accuracy_score(y_true, y_pred):\n    \"\"\"Accuracy classification score.\n    \n    In multilabel classification, this function computes subset accuracy:\n    the set of labels predicted for a sample must *exactly* match the\n    corresponding set of labels in y_true.\n\n    Args:\n      y_true: 2d array. Ground truth (correct) target values.\n      y_pred: 2d array. Estimated targets as returned by a tagger.\n\n    Returns:\n      score: float.\n      Example:\n\n    >>> from seqeval.metrics import accuracy_score\n        >>> y_true = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]\n        >>> y_pred = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]\n        >>> accuracy_score(y_true, y_pred)\n        0.80\n    \"\"\"\n    if any(isinstance(s, list) for s in y_true):\n        y_true = [item for sublist in y_true for item in sublist]\n        y_pred = [item for sublist in y_pred for item in sublist]\n\n    nb_correct = sum(y_t == y_p for y_t, y_p in zip(y_true, y_pred))\n    nb_true = len(y_true)\n\n    score = nb_correct / nb_true\n\n    return score\n\n\ndef precision_score(y_true, y_pred, average='micro', suffix=False):\n    \"\"\"Compute the precision.\n    \n    The precision is the ratio ``tp / (tp + fp)`` where ``tp`` is the number of\n    true positives and ``fp`` the number of false positives. The precision is\n    intuitively the ability of the classifier not to label as positive a sample.\n    \n    The best value is 1 and the worst value is 0.\n\n    Args:\n      y_true: 2d array. Ground truth (correct) target values.\n      y_pred: 2d array. Estimated targets as returned by a tagger.\n      average:  (Default value = 'micro')\n      suffix:  (Default value = False)\n\n    Returns:\n      score: float.\n      Example:\n\n    >>> from seqeval.metrics import precision_score\n        >>> y_true = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]\n        >>> y_pred = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]\n        >>> precision_score(y_true, y_pred)\n        0.50\n    \"\"\"\n    true_entities = set(get_entities(y_true, suffix))\n    pred_entities = set(get_entities(y_pred, suffix))\n\n    nb_correct = len(true_entities & pred_entities)\n    nb_pred = len(pred_entities)\n\n    score = nb_correct / nb_pred if nb_pred > 0 else 0\n\n    return score\n\n\ndef recall_score(y_true, y_pred, average='micro', suffix=False):\n    \"\"\"Compute the recall.\n    \n    The recall is the ratio ``tp / (tp + fn)`` where ``tp`` is the number of\n    true positives and ``fn`` the number of false negatives. The recall is\n    intuitively the ability of the classifier to find all the positive samples.\n    \n    The best value is 1 and the worst value is 0.\n\n    Args:\n      y_true: 2d array. Ground truth (correct) target values.\n      y_pred: 2d array. Estimated targets as returned by a tagger.\n      average:  (Default value = 'micro')\n      suffix:  (Default value = False)\n\n    Returns:\n      score: float.\n      Example:\n\n    >>> from seqeval.metrics import recall_score\n        >>> y_true = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]\n        >>> y_pred = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]\n        >>> recall_score(y_true, y_pred)\n        0.50\n    \"\"\"\n    true_entities = set(get_entities(y_true, suffix))\n    pred_entities = set(get_entities(y_pred, suffix))\n\n    nb_correct = len(true_entities & pred_entities)\n    nb_true = len(true_entities)\n\n    score = nb_correct / nb_true if nb_true > 0 else 0\n\n    return score\n\n\ndef performance_measure(y_true, y_pred):\n    \"\"\"Compute the performance metrics: TP, FP, FN, TN\n\n    Args:\n      y_true: 2d array. Ground truth (correct) target values.\n      y_pred: 2d array. Estimated targets as returned by a tagger.\n\n    Returns:\n      performance_dict: dict\n      Example:\n\n    >>> from seqeval.metrics import performance_measure\n        >>> y_true = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'O', 'B-ORG'], ['B-PER', 'I-PER', 'O']]\n        >>> y_pred = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O', 'O'], ['B-PER', 'I-PER', 'O']]\n        >>> performance_measure(y_true, y_pred)\n        (3, 3, 1, 4)\n    \"\"\"\n    performace_dict = dict()\n    if any(isinstance(s, list) for s in y_true):\n        y_true = [item for sublist in y_true for item in sublist]\n        y_pred = [item for sublist in y_pred for item in sublist]\n    performace_dict['TP'] = sum(y_t == y_p for y_t, y_p in zip(y_true, y_pred)\n                                if ((y_t != 'O') or (y_p != 'O')))\n    performace_dict['FP'] = sum(y_t != y_p for y_t, y_p in zip(y_true, y_pred))\n    performace_dict['FN'] = sum(((y_t != 'O') and (y_p == 'O'))\n                                for y_t, y_p in zip(y_true, y_pred))\n    performace_dict['TN'] = sum((y_t == y_p == 'O')\n                                for y_t, y_p in zip(y_true, y_pred))\n\n    return performace_dict\n\n\ndef classification_report(y_true, y_pred, digits=2, suffix=False):\n    \"\"\"Build a text report showing the main classification metrics.\n\n    Args:\n      y_true: 2d array. Ground truth (correct) target values.\n      y_pred: 2d array. Estimated targets as returned by a classifier.\n      digits: int. Number of digits for formatting output floating point values. (Default value = 2)\n      suffix:  (Default value = False)\n\n    Returns:\n      report: string. Text summary of the precision, recall, F1 score for each class.\n      Examples:\n\n    >>> from seqeval.metrics import classification_report\n        >>> y_true = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]\n        >>> y_pred = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]\n        >>> print(classification_report(y_true, y_pred))\n                     precision    recall  f1-score   support\n        <BLANKLINE>\n               MISC       0.00      0.00      0.00         1\n                PER       1.00      1.00      1.00         1\n        <BLANKLINE>\n          micro avg       0.50      0.50      0.50         2\n          macro avg       0.50      0.50      0.50         2\n        <BLANKLINE>\n    \"\"\"\n    true_entities = set(get_entities(y_true, suffix))\n    pred_entities = set(get_entities(y_pred, suffix))\n\n    name_width = 0\n    d1 = defaultdict(set)\n    d2 = defaultdict(set)\n    for e in true_entities:\n        d1[e[0]].add((e[1], e[2]))\n        name_width = max(name_width, len(e[0]))\n    for e in pred_entities:\n        d2[e[0]].add((e[1], e[2]))\n\n    last_line_heading = 'macro avg'\n    width = max(name_width, len(last_line_heading), digits)\n\n    headers = [\"precision\", \"recall\", \"f1-score\", \"support\"]\n    head_fmt = u'{:>{width}s} ' + u' {:>9}' * len(headers)\n    report = head_fmt.format(u'', *headers, width=width)\n    report += u'\\n\\n'\n\n    row_fmt = u'{:>{width}s} ' + u' {:>9.{digits}f}' * 3 + u' {:>9}\\n'\n\n    ps, rs, f1s, s = [], [], [], []\n    for type_name, true_entities in d1.items():\n        pred_entities = d2[type_name]\n        nb_correct = len(true_entities & pred_entities)\n        nb_pred = len(pred_entities)\n        nb_true = len(true_entities)\n\n        p = nb_correct / nb_pred if nb_pred > 0 else 0\n        r = nb_correct / nb_true if nb_true > 0 else 0\n        f1 = 2 * p * r / (p + r) if p + r > 0 else 0\n\n        report += row_fmt.format(*[type_name, p, r, f1, nb_true], width=width, digits=digits)\n\n        ps.append(p)\n        rs.append(r)\n        f1s.append(f1)\n        s.append(nb_true)\n\n    report += u'\\n'\n\n    # compute averages\n    report += row_fmt.format('micro avg',\n                             precision_score(y_true, y_pred, suffix=suffix),\n                             recall_score(y_true, y_pred, suffix=suffix),\n                             f1_score(y_true, y_pred, suffix=suffix),\n                             np.sum(s),\n                             width=width, digits=digits)\n    report += row_fmt.format(last_line_heading,\n                             np.average(ps, weights=s),\n                             np.average(rs, weights=s),\n                             np.average(f1s, weights=s),\n                             np.sum(s),\n                             width=width, digits=digits)\n\n    return report\n"
  },
  {
    "path": "hanlp/metrics/f1.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-07-10 14:55\nfrom abc import ABC\n\nfrom hanlp.metrics.metric import Metric\n\n\nclass F1(Metric, ABC):\n    def __init__(self, nb_pred=0, nb_true=0, nb_correct=0) -> None:\n        super().__init__()\n        self.nb_correct = nb_correct\n        self.nb_pred = nb_pred\n        self.nb_true = nb_true\n\n    def __repr__(self) -> str:\n        p, r, f = self.prf\n        return f\"P: {p:.2%} R: {r:.2%} F1: {f:.2%}\"\n\n    @property\n    def prf(self):\n        nb_correct = self.nb_correct\n        nb_pred = self.nb_pred\n        nb_true = self.nb_true\n        p = nb_correct / nb_pred if nb_pred > 0 else .0\n        r = nb_correct / nb_true if nb_true > 0 else .0\n        f = 2 * p * r / (p + r) if p + r > 0 else .0\n        return p, r, f\n\n    @property\n    def score(self):\n        return self.prf[-1]\n\n    def reset(self):\n        self.nb_correct = 0\n        self.nb_pred = 0\n        self.nb_true = 0\n\n    def __call__(self, pred: set, gold: set):\n        self.nb_correct += len(pred & gold)\n        self.nb_pred += len(pred)\n        self.nb_true += len(gold)\n\n\nclass F1_(Metric):\n    def __init__(self, p, r, f) -> None:\n        super().__init__()\n        self.f = f\n        self.r = r\n        self.p = p\n\n    @property\n    def score(self):\n        return self.f\n\n    def __call__(self, pred, gold):\n        raise NotImplementedError()\n\n    def reset(self):\n        self.f = self.r = self.p = 0\n\n    def __repr__(self) -> str:\n        p, r, f = self.p, self.r, self.f\n        return f\"P: {p:.2%} R: {r:.2%} F1: {f:.2%}\"\n"
  },
  {
    "path": "hanlp/metrics/metric.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-03 11:35\nfrom abc import ABC, abstractmethod\n\n\nclass Metric(ABC):\n\n    def __lt__(self, other):\n        return self.score < other\n\n    def __le__(self, other):\n        return self.score <= other\n\n    def __eq__(self, other):\n        return self.score == other\n\n    def __ge__(self, other):\n        return self.score >= other\n\n    def __gt__(self, other):\n        return self.score > other\n\n    def __ne__(self, other):\n        return self.score != other\n\n    @property\n    @abstractmethod\n    def score(self):\n        pass\n\n    @abstractmethod\n    def __call__(self, pred, gold, mask=None):\n        pass\n\n    def __repr__(self) -> str:\n        return f'{self.score}:.4f'\n\n    def __float__(self):\n        return self.score\n\n    @abstractmethod\n    def reset(self):\n        pass\n"
  },
  {
    "path": "hanlp/metrics/mtl.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-03 00:16\nfrom hanlp.metrics.metric import Metric\n\n\nclass MetricDict(Metric, dict):\n    _COLORS = [\"magenta\", \"cyan\", \"green\", \"yellow\"]\n\n    @property\n    def score(self):\n        return sum(float(x) for x in self.values()) / len(self)\n\n    def __call__(self, pred, gold):\n        for metric in self.values():\n            metric(pred, gold)\n\n    def reset(self):\n        for metric in self.values():\n            metric.reset()\n\n    def __repr__(self) -> str:\n        return ' '.join(f'({k} {v})' for k, v in self.items())\n\n    def cstr(self, idx=None, level=0) -> str:\n        if idx is None:\n            idx = [0]\n        prefix = ''\n        for _, (k, v) in enumerate(self.items()):\n            color = self._COLORS[idx[0] % len(self._COLORS)]\n            idx[0] += 1\n            child_is_dict = isinstance(v, MetricDict)\n            _level = min(level, 2)\n            # if level != 0 and not child_is_dict:\n            #     _level = 2\n            lb = '{[('\n            rb = '}])'\n            k = f'[bold][underline]{k}[/underline][/bold]'\n            prefix += f'[{color}]{lb[_level]}{k} [/{color}]'\n            if child_is_dict:\n                prefix += v.cstr(idx, level + 1)\n            else:\n                prefix += f'[{color}]{v}[/{color}]'\n            prefix += f'[{color}]{rb[_level]}[/{color}]'\n        return prefix\n"
  },
  {
    "path": "hanlp/metrics/parsing/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-27 00:48"
  },
  {
    "path": "hanlp/metrics/parsing/attachmentscore.py",
    "content": "# MIT License\n#\n# Copyright (c) 2020 Yu Zhang\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\n\nfrom hanlp.metrics.metric import Metric\n\n\nclass AttachmentScore(Metric):\n\n    def __init__(self, eps=1e-12):\n        super(AttachmentScore, self).__init__()\n\n        self.eps = eps\n        self.total = 0.0\n        self.correct_arcs = 0.0\n        self.correct_rels = 0.0\n\n    def __repr__(self):\n        return f\"UAS: {self.uas:.2%} LAS: {self.las:.2%}\"\n\n    # noinspection PyMethodOverriding\n    def __call__(self, arc_preds, rel_preds, arc_golds, rel_golds, mask):\n        arc_mask = arc_preds.eq(arc_golds)[mask]\n        rel_mask = rel_preds.eq(rel_golds)[mask] & arc_mask\n\n        self.total += len(arc_mask)\n        self.correct_arcs += arc_mask.sum().item()\n        self.correct_rels += rel_mask.sum().item()\n\n    def __lt__(self, other):\n        return self.score < other\n\n    def __le__(self, other):\n        return self.score <= other\n\n    def __ge__(self, other):\n        return self.score >= other\n\n    def __gt__(self, other):\n        return self.score > other\n\n    @property\n    def score(self):\n        return self.las\n\n    @property\n    def uas(self):\n        return self.correct_arcs / (self.total + self.eps)\n\n    @property\n    def las(self):\n        return self.correct_rels / (self.total + self.eps)\n\n    def reset(self):\n        self.total = 0.0\n        self.correct_arcs = 0.0\n        self.correct_rels = 0.0\n"
  },
  {
    "path": "hanlp/metrics/parsing/conllx_eval.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-03-08 22:35\nimport tempfile\n\nfrom hanlp.utils.io_util import get_resource, get_exitcode_stdout_stderr\n\nCONLLX_EVAL = get_resource(\n    'https://github.com/elikip/bist-parser/archive/master.zip' + '#bmstparser/src/utils/eval.pl')\n\n\ndef evaluate(gold_file, pred_file):\n    \"\"\"Evaluate using official CoNLL-X evaluation script (Yuval Krymolowski)\n\n    Args:\n      gold_file(str): The gold conllx file\n      pred_file(str): The pred conllx file\n\n    Returns:\n\n    \n    \"\"\"\n    gold_file = get_resource(gold_file)\n    fixed_pred_file = tempfile.NamedTemporaryFile().name\n    copy_cols(gold_file, pred_file, fixed_pred_file, keep_comments=False)\n    if gold_file.endswith('.conllu'):\n        fixed_gold_file = tempfile.NamedTemporaryFile().name\n        copy_cols(gold_file, gold_file, fixed_gold_file, keep_comments=False)\n        gold_file = fixed_gold_file\n\n    exitcode, out, err = get_exitcode_stdout_stderr(f'perl {CONLLX_EVAL} -q -b -g {gold_file} -s {fixed_pred_file}')\n    if exitcode:\n        raise RuntimeError(f'eval.pl exited with error code {exitcode} and error message {err} and output {out}.')\n    lines = out.split('\\n')[-4:]\n    las = int(lines[0].split()[3]) / int(lines[0].split()[5])\n    uas = int(lines[1].split()[3]) / int(lines[1].split()[5])\n    return uas, las\n\n\ndef copy_cols(gold_file, pred_file, copied_pred_file, keep_comments=True):\n    \"\"\"Copy the first 6 columns from gold file to pred file\n\n    Args:\n      gold_file: \n      pred_file: \n      copied_pred_file: \n      keep_comments:  (Default value = True)\n\n    Returns:\n\n    \n    \"\"\"\n    with open(copied_pred_file, 'w') as to_out, open(pred_file) as pred_file, open(gold_file) as gold_file:\n        for idx, (p, g) in enumerate(zip(pred_file, gold_file)):\n            while p.startswith('#'):\n                p = next(pred_file)\n            if not g.strip():\n                if p.strip():\n                    raise ValueError(\n                        f'Prediction file {pred_file.name} does not end a sentence at line {idx + 1}\\n{p.strip()}')\n                to_out.write('\\n')\n                continue\n            while g.startswith('#') or '-' in g.split('\\t')[0]:\n                if keep_comments or g.startswith('-'):\n                    to_out.write(g)\n                g = next(gold_file)\n            to_out.write('\\t'.join(str(x) for x in g.split('\\t')[:6] + p.split('\\t')[6:]))\n"
  },
  {
    "path": "hanlp/metrics/parsing/labeled_f1.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-27 21:42\n\nfrom hanlp.metrics.metric import Metric\n\n\nclass LabeledF1(Metric):\n\n    def __init__(self):\n        super(LabeledF1, self).__init__()\n\n        self.sum_gold_arcs_wo_punc = 0.0\n        self.sum_pred_arcs_wo_punc = 0.0\n        self.correct_arcs_wo_punc = 0.0\n        self.correct_rels_wo_punc = 0.0\n\n    def __repr__(self):\n        return f\"UF: {self.uf:4.2%} LF: {self.lf:4.2%}\"\n\n    def __call__(self, arc_preds, rel_preds, arc_golds, rel_golds, mask):\n        mask_gold = mask & arc_golds\n        mask_pred = mask & arc_preds\n\n        correct_mask = mask_gold & mask_pred\n        correct_arcs_wo_punc = (arc_preds == arc_golds)[correct_mask]\n        correct_rels_wo_punc = (rel_preds == rel_golds)[correct_mask] & correct_arcs_wo_punc\n\n        self.sum_gold_arcs_wo_punc += float(mask_gold.sum())\n        self.sum_pred_arcs_wo_punc += float(mask_pred.sum())\n        self.correct_arcs_wo_punc += float(correct_arcs_wo_punc.sum())\n        self.correct_rels_wo_punc += float(correct_rels_wo_punc.sum())\n\n    def __lt__(self, other):\n        return self.score < other\n\n    def __le__(self, other):\n        return self.score <= other\n\n    def __ge__(self, other):\n        return self.score >= other\n\n    def __gt__(self, other):\n        return self.score > other\n\n    @property\n    def score(self):\n        return self.las\n\n    @property\n    def uas(self):\n        return self.uf\n\n    @property\n    def las(self):\n        return self.lf\n\n    @property\n    def ur(self):\n        if not self.sum_gold_arcs_wo_punc:\n            return .0\n        return self.correct_arcs_wo_punc / self.sum_gold_arcs_wo_punc\n\n    @property\n    def up(self):\n        if not self.sum_pred_arcs_wo_punc:\n            return .0\n        return self.correct_arcs_wo_punc / self.sum_pred_arcs_wo_punc\n\n    @property\n    def lr(self):\n        if not self.sum_gold_arcs_wo_punc:\n            return .0\n        return self.correct_rels_wo_punc / self.sum_gold_arcs_wo_punc\n\n    @property\n    def lp(self):\n        if not self.sum_pred_arcs_wo_punc:\n            return .0\n        return self.correct_rels_wo_punc / self.sum_pred_arcs_wo_punc\n\n    @property\n    def uf(self):\n        rp = self.ur + self.up\n        if not rp:\n            return .0\n        return 2 * self.ur * self.up / rp\n\n    @property\n    def lf(self):\n        rp = self.lr + self.lp\n        if not rp:\n            return .0\n        return 2 * self.lr * self.lp / rp\n\n    def reset(self):\n        self.sum_gold_arcs_wo_punc = 0.0\n        self.sum_pred_arcs_wo_punc = 0.0\n        self.correct_arcs_wo_punc = 0.0\n        self.correct_rels_wo_punc = 0.0\n\n    def to_dict(self) -> dict:\n        return {'UF': self.uf, 'LF': self.lf}\n"
  },
  {
    "path": "hanlp/metrics/parsing/labeled_f1_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-27 21:42\nimport tensorflow as tf\n\n\nclass LabeledF1TF(object):\n\n    def __init__(self):\n        super(LabeledF1TF, self).__init__()\n\n        self.sum_gold_arcs_wo_punc = 0.0\n        self.sum_pred_arcs_wo_punc = 0.0\n        self.correct_arcs_wo_punc = 0.0\n        self.correct_rels_wo_punc = 0.0\n\n    def __repr__(self):\n        return f\"UF: {self.uf:6.2%} LF: {self.lf:6.2%}\"\n\n    def __call__(self, arc_preds, rel_preds, arc_golds, rel_golds, mask):\n        mask = mask.unsqueeze(-1).expand_as(arc_preds)\n        mask = mask & mask.transpose(1, 2)\n\n        mask_gold = mask & arc_golds\n        mask_pred = mask & arc_preds\n        correct_arcs_wo_punc = (arc_preds == arc_golds)[mask_gold & mask_pred]\n        correct_rels_wo_punc = (rel_preds == rel_golds)[mask_gold & mask_pred] & correct_arcs_wo_punc\n\n        self.sum_gold_arcs_wo_punc += float(tf.math.count_nonzero(mask_gold))\n        self.sum_pred_arcs_wo_punc += float(tf.math.count_nonzero(mask_pred))\n        self.correct_arcs_wo_punc += float(tf.math.count_nonzero(correct_arcs_wo_punc))\n        self.correct_rels_wo_punc += float(tf.math.count_nonzero(correct_rels_wo_punc))\n\n    def __lt__(self, other):\n        return self.score < other\n\n    def __le__(self, other):\n        return self.score <= other\n\n    def __ge__(self, other):\n        return self.score >= other\n\n    def __gt__(self, other):\n        return self.score > other\n\n    @property\n    def score(self):\n        return self.las\n\n    @property\n    def uas(self):\n        return self.uf\n\n    @property\n    def las(self):\n        return self.lf\n\n    @property\n    def ur(self):\n        if not self.sum_gold_arcs_wo_punc:\n            return 0\n        return self.correct_arcs_wo_punc / self.sum_gold_arcs_wo_punc\n\n    @property\n    def up(self):\n        if not self.sum_pred_arcs_wo_punc:\n            return 0\n        return self.correct_arcs_wo_punc / self.sum_pred_arcs_wo_punc\n\n    @property\n    def lr(self):\n        if not self.sum_gold_arcs_wo_punc:\n            return 0\n        return self.correct_rels_wo_punc / self.sum_gold_arcs_wo_punc\n\n    @property\n    def lp(self):\n        if not self.sum_pred_arcs_wo_punc:\n            return 0\n        return self.correct_rels_wo_punc / self.sum_pred_arcs_wo_punc\n\n    @property\n    def uf(self):\n        rp = self.ur + self.up\n        if not rp:\n            return 0\n        return 2 * self.ur * self.up / rp\n\n    @property\n    def lf(self):\n        rp = self.lr + self.lp\n        if not rp:\n            return 0\n        return 2 * self.lr * self.lp / rp\n\n    def reset_states(self):\n        self.sum_gold_arcs_wo_punc = 0.0\n        self.sum_pred_arcs_wo_punc = 0.0\n        self.correct_arcs_wo_punc = 0.0\n        self.correct_rels_wo_punc = 0.0\n\n    def to_dict(self) -> dict:\n        return {'UF': self.uf, 'LF': self.lf}\n"
  },
  {
    "path": "hanlp/metrics/parsing/labeled_score.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-27 00:49\n\nimport tensorflow as tf\n\n\nclass LabeledScore(object):\n\n    def __init__(self, eps=1e-5):\n        super(LabeledScore, self).__init__()\n\n        self.eps = eps\n        self.total = 0.0\n        self.correct_arcs = 0.0\n        self.correct_rels = 0.0\n\n    def __repr__(self):\n        return f\"UAS: {self.uas:6.2%} LAS: {self.las:6.2%}\"\n\n    def __call__(self, arc_preds, rel_preds, arc_golds, rel_golds, mask):\n        arc_mask = (arc_preds == arc_golds)[mask]\n        rel_mask = (rel_preds == rel_golds)[mask] & arc_mask\n\n        self.total += len(arc_mask)\n        self.correct_arcs += int(tf.math.count_nonzero(arc_mask))\n        self.correct_rels += int(tf.math.count_nonzero(rel_mask))\n\n    def __lt__(self, other):\n        return self.score < other\n\n    def __le__(self, other):\n        return self.score <= other\n\n    def __ge__(self, other):\n        return self.score >= other\n\n    def __gt__(self, other):\n        return self.score > other\n\n    @property\n    def score(self):\n        return self.las\n\n    @property\n    def uas(self):\n        return self.correct_arcs / (self.total + self.eps)\n\n    @property\n    def las(self):\n        return self.correct_rels / (self.total + self.eps)\n\n    def reset_states(self):\n        self.total = 0.0\n        self.correct_arcs = 0.0\n        self.correct_rels = 0.0\n\n    def to_dict(self) -> dict:\n        return {'UAS': self.uas, 'LAS': self.las}\n"
  },
  {
    "path": "hanlp/metrics/parsing/semdep_eval.py",
    "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n\n# Copyright 2017 Timothy Dozat\n# \n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n# \n#     http://www.apache.org/licenses/LICENSE-2.0\n# \n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport codecs\nimport sys\nfrom collections import namedtuple\n\n\n# ===============================================================\ndef sdp_eval(gold_files, sys_files, labeled=False):\n    \"\"\"Modified from https://github.com/tdozat/Parser-v3/blob/2ff4061373e8aac8c962537a6220e1d5b196abf6/scripts/semdep_eval.py\n    Dozat claimed \"I tested it against the official eval script and it reported identical LF1\".\n\n    Args:\n      gold_files: \n      sys_files: \n      labeled:  (Default value = False)\n\n    Returns:\n\n    \n    \"\"\"\n\n    correct = 0\n    predicted = 0\n    actual = 0\n    n_tokens = 0\n    n_sequences = 0\n    current_seq_correct = False\n    n_correct_sequences = 0\n    current_sent = 0\n    if isinstance(gold_files, str):\n        gold_files = [gold_files]\n    if isinstance(sys_files, str):\n        sys_files = [sys_files]\n\n    for gold_file, sys_file in zip(gold_files, sys_files):\n        with codecs.open(gold_file, encoding='utf-8') as gf, \\\n                codecs.open(sys_file, encoding='utf-8') as sf:\n            gold_line = gf.readline()\n            gold_i = 1\n            sys_i = 0\n            while gold_line:\n                while gold_line.startswith('#'):\n                    current_sent += 1\n                    gold_i += 1\n                    n_sequences += 1\n                    n_correct_sequences += current_seq_correct\n                    current_seq_correct = True\n                    gold_line = gf.readline()\n                if gold_line.rstrip() != '':\n                    sys_line = sf.readline()\n                    sys_i += 1\n                    while sys_line.startswith('#') or sys_line.rstrip() == '' or sys_line.split('\\t')[0] == '0':\n                        sys_line = sf.readline()\n                        sys_i += 1\n\n                    gold_line = gold_line.rstrip().split('\\t')\n                    sys_line = sys_line.rstrip().split('\\t')\n                    # assert sys_line[1] == gold_line[1], 'Files are misaligned at lines {}, {}'.format(gold_i, sys_i)\n\n                    # Compute the gold edges\n                    gold_node = gold_line[8]\n                    if gold_node != '_':\n                        gold_node = gold_node.split('|')\n                        if labeled:\n                            gold_edges = set(tuple(gold_edge.split(':', 1)) for gold_edge in gold_node)\n                        else:\n                            gold_edges = set(gold_edge.split(':', 1)[0] for gold_edge in gold_node)\n                    else:\n                        gold_edges = set()\n\n                    # Compute the sys edges\n                    sys_node = sys_line[8]\n                    if sys_node != '_':\n                        sys_node = sys_node.split('|')\n                        if labeled:\n                            sys_edges = set(tuple(sys_edge.split(':', 1)) for sys_edge in sys_node)\n                        else:\n                            sys_edges = set(sys_edge.split(':', 1)[0] for sys_edge in sys_node)\n                    else:\n                        sys_edges = set()\n\n                    correct_edges = gold_edges & sys_edges\n                    if len(correct_edges) != len(gold_edges):\n                        current_seq_correct = False\n                    correct += len(correct_edges)\n                    predicted += len(sys_edges)\n                    actual += len(gold_edges)\n                    n_tokens += 1\n                    # current_fp += len(sys_edges) - len(gold_edges & sys_edges)\n                gold_line = gf.readline()\n                gold_i += 1\n    # print(correct, predicted - correct, actual - correct)\n    Accuracy = namedtuple('Accuracy', ['precision', 'recall', 'F1', 'seq_acc'])\n    precision = correct / (predicted + 1e-12)\n    recall = correct / (actual + 1e-12)\n    F1 = 2 * precision * recall / (precision + recall + 1e-12)\n    seq_acc = n_correct_sequences / n_sequences\n    return Accuracy(precision, recall, F1, seq_acc)\n\n\n# ===============================================================\ndef main():\n    \"\"\" \"\"\"\n\n    files = sys.argv[1:]\n    n_files = len(files)\n    assert (n_files % 2) == 0\n    gold_files, sys_files = files[:n_files // 2], files[n_files // 2:]\n    UAS = sdp_eval(gold_files, sys_files, labeled=False)\n    LAS = sdp_eval(gold_files, sys_files, labeled=True)\n    # print(UAS.F1, UAS.seq_acc)\n    print('UAS={:0.1f}'.format(UAS.F1 * 100))\n    print('LAS={:0.1f}'.format(LAS.F1 * 100))\n\n\nif __name__ == '__main__':\n    main()\n"
  },
  {
    "path": "hanlp/metrics/parsing/span.py",
    "content": "# MIT License\n#\n# Copyright (c) 2020 Yu Zhang\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\n\nfrom collections import Counter\n\nfrom hanlp.metrics.metric import Metric\n\n\nclass SpanMetric(Metric):\n\n    def __init__(self, eps=1e-12):\n        super().__init__()\n        self.reset(eps)\n\n    # noinspection PyAttributeOutsideInit\n    def reset(self, eps=1e-12):\n        self.n = 0.0\n        self.n_ucm = 0.0\n        self.n_lcm = 0.0\n        self.utp = 0.0\n        self.ltp = 0.0\n        self.pred = 0.0\n        self.gold = 0.0\n        self.eps = eps\n\n    def __call__(self, preds, golds):\n        for pred, gold in zip(preds, golds):\n            upred = Counter([(i, j) for i, j, label in pred])\n            ugold = Counter([(i, j) for i, j, label in gold])\n            utp = list((upred & ugold).elements())\n            lpred = Counter(pred)\n            lgold = Counter(gold)\n            ltp = list((lpred & lgold).elements())\n            self.n += 1\n            self.n_ucm += len(utp) == len(pred) == len(gold)\n            self.n_lcm += len(ltp) == len(pred) == len(gold)\n            self.utp += len(utp)\n            self.ltp += len(ltp)\n            self.pred += len(pred)\n            self.gold += len(gold)\n        return self\n\n    def __repr__(self):\n        s = f\"UCM: {self.ucm:.2%} LCM: {self.lcm:.2%} \"\n        s += f\"UP: {self.up:.2%} UR: {self.ur:.2%} UF: {self.uf:.2%} \"\n        s += f\"LP: {self.lp:.2%} LR: {self.lr:.2%} LF: {self.lf:.2%}\"\n\n        return s\n\n    @property\n    def score(self):\n        return self.lf\n\n    @property\n    def ucm(self):\n        return self.n_ucm / (self.n + self.eps)\n\n    @property\n    def lcm(self):\n        return self.n_lcm / (self.n + self.eps)\n\n    @property\n    def up(self):\n        return self.utp / (self.pred + self.eps)\n\n    @property\n    def ur(self):\n        return self.utp / (self.gold + self.eps)\n\n    @property\n    def uf(self):\n        return 2 * self.utp / (self.pred + self.gold + self.eps)\n\n    @property\n    def lp(self):\n        return self.ltp / (self.pred + self.eps)\n\n    @property\n    def lr(self):\n        return self.ltp / (self.gold + self.eps)\n\n    @property\n    def lf(self):\n        return 2 * self.ltp / (self.pred + self.gold + self.eps)\n"
  },
  {
    "path": "hanlp/metrics/spearman_correlation.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-05-23 16:12\nimport torch\n\nfrom hanlp.metrics.metric import Metric\n\n\ndef _get_ranks(x: torch.Tensor) -> torch.Tensor:\n    argsort = x.argsort()\n    ranks = torch.zeros_like(argsort, device=x.device)\n    ranks[argsort] = torch.arange(len(x), device=x.device)\n    return ranks\n\n\ndef spearman_correlation(x: torch.Tensor, y: torch.Tensor):\n    \"\"\"Compute correlation between 2 1-D vectors. Adopted from\n    https://discuss.pytorch.org/t/spearmans-correlation/91931/5\n\n    Args:\n        x: Shape (N, )\n        y: Shape (N, )\n\n    \"\"\"\n    x_rank = _get_ranks(x)\n    y_rank = _get_ranks(y)\n\n    n = x.size(0)\n    upper = 6 * torch.sum((x_rank - y_rank).pow(2))\n    down = n * (n ** 2 - 1.0)\n    return 1.0 - (upper / down)\n\n\nclass SpearmanCorrelation(Metric):\n    \"\"\"\n    This `Metric` calculates the sample Spearman correlation coefficient (r)\n    between two tensors. Each element in the two tensors is assumed to be\n    a different observation of the variable (i.e., the input tensors are\n    implicitly flattened into vectors and the correlation is calculated\n    between the vectors).\n\n    <https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient>\n    \"\"\"\n\n    @property\n    def score(self):\n        return spearman_correlation(self.total_predictions, self.total_gold_labels).item()\n\n    def __init__(self) -> None:\n        super().__init__()\n        self.total_predictions = torch.zeros(0)\n        self.total_gold_labels = torch.zeros(0)\n\n    def __call__(\n            self,\n            predictions: torch.Tensor,\n            gold_labels: torch.Tensor,\n            mask=None\n    ):\n        \"\"\"\n        # Parameters\n\n        predictions : `torch.Tensor`, required.\n            A tensor of predictions of shape (batch_size, ...).\n        gold_labels : `torch.Tensor`, required.\n            A tensor of the same shape as `predictions`.\n        \"\"\"\n        if mask is not None:\n            raise NotImplemented('mask not supported in SpearmanCorrelation for now.')\n        # Flatten predictions, gold_labels, and mask. We calculate the Spearman correlation between\n        # the vectors, since each element in the predictions and gold_labels tensor is assumed\n        # to be a separate observation.\n        predictions = predictions.reshape(-1)\n        gold_labels = gold_labels.reshape(-1)\n\n        self.total_predictions = self.total_predictions.to(predictions.device)\n        self.total_gold_labels = self.total_gold_labels.to(gold_labels.device)\n        self.total_predictions = torch.cat((self.total_predictions, predictions), 0)\n        self.total_gold_labels = torch.cat((self.total_gold_labels, gold_labels), 0)\n\n    def reset(self):\n        self.total_predictions = torch.zeros(0)\n        self.total_gold_labels = torch.zeros(0)\n\n    def __str__(self) -> str:\n        return f'spearman: {self.score * 100:.2f}'\n"
  },
  {
    "path": "hanlp/metrics/srl/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-07-16 18:44"
  },
  {
    "path": "hanlp/metrics/srl/srlconll.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-07-16 18:44\nimport os\n\nfrom hanlp.utils.io_util import get_resource, get_exitcode_stdout_stderr, run_cmd\n\n\ndef official_conll_05_evaluate(pred_path, gold_path):\n    script_root = get_resource('http://www.lsi.upc.edu/~srlconll/srlconll-1.1.tgz')\n    lib_path = f'{script_root}/lib'\n    if lib_path not in os.environ.get(\"PERL5LIB\", \"\"):\n        os.environ['PERL5LIB'] = f'{lib_path}:{os.environ.get(\"PERL5LIB\", \"\")}'\n    bin_path = f'{script_root}/bin'\n    if bin_path not in os.environ.get('PATH', ''):\n        os.environ['PATH'] = f'{bin_path}:{os.environ.get(\"PATH\", \"\")}'\n    eval_info_gold_pred = run_cmd(f'perl {script_root}/bin/srl-eval.pl {gold_path} {pred_path}')\n    eval_info_pred_gold = run_cmd(f'perl {script_root}/bin/srl-eval.pl {pred_path} {gold_path}')\n    conll_recall = float(eval_info_gold_pred.strip().split(\"\\n\")[6].strip().split()[5]) / 100\n    conll_precision = float(eval_info_pred_gold.strip().split(\"\\n\")[6].strip().split()[5]) / 100\n    if conll_recall + conll_precision > 0:\n        conll_f1 = 2 * conll_recall * conll_precision / (conll_recall + conll_precision)\n    else:\n        conll_f1 = 0\n    return conll_precision, conll_recall, conll_f1\n\n\ndef run_perl(script, src, dst=None):\n    os.environ['PERL5LIB'] = f''\n    exitcode, out, err = get_exitcode_stdout_stderr(\n        f'perl -I{os.path.expanduser(\"~/.local/lib/perl5\")} {script} {src}')\n    if exitcode:\n        # cpanm -l ~/.local namespace::autoclean\n        # cpanm -l ~/.local Moose\n        # cpanm -l ~/.local MooseX::SemiAffordanceAccessor module\n        raise RuntimeError(err)\n    with open(dst, 'w') as ofile:\n        ofile.write(out)\n    return dst\n"
  },
  {
    "path": "hanlp/optimizers/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-11-11 18:44"
  },
  {
    "path": "hanlp/optimizers/adamw/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-11-11 18:44\nimport tensorflow as tf\nfrom hanlp.optimizers.adamw.optimization import WarmUp, AdamWeightDecay\n\n\n# from hanlp.optimization.adamw.optimizers_v2 import AdamW\n# from hanlp.optimization.adamw.utils import get_weight_decays\n\n\n# def create_optimizer(model, init_lr, num_train_steps, num_warmup_steps):\n#     \"\"\"Creates an optimizer with learning rate schedule.\"\"\"\n#     wd_dict = get_weight_decays(model)\n#\n#     # Implements linear decay of the learning rate.\n#     learning_rate_fn = tf.keras.optimizers.schedules.PolynomialDecay(\n#         initial_learning_rate=init_lr,\n#         decay_steps=num_train_steps,\n#         end_learning_rate=0.0)\n#     if num_warmup_steps:\n#         learning_rate_fn = WarmUp(initial_learning_rate=init_lr,\n#                                   decay_schedule_fn=learning_rate_fn,\n#                                   warmup_steps=num_warmup_steps)\n#     optimizer = AdamW(\n#         learning_rate=learning_rate_fn,\n#         weight_decay_rate=0.01,\n#         beta_1=0.9,\n#         beta_2=0.999,\n#         epsilon=1e-6,\n#         exclude_from_weight_decay=['layer_norm', 'bias'])\n#     return optimizer\n\n\ndef create_optimizer(init_lr, num_train_steps, num_warmup_steps, weight_decay_rate=0.01, epsilon=1e-6, clipnorm=None):\n    \"\"\"Creates an optimizer with learning rate schedule.\n\n    Args:\n      init_lr: \n      num_train_steps: \n      num_warmup_steps: \n      weight_decay_rate:  (Default value = 0.01)\n      epsilon:  (Default value = 1e-6)\n      clipnorm:  (Default value = None)\n\n    Returns:\n\n    \"\"\"\n    # Implements linear decay of the learning rate.\n    learning_rate_fn = tf.keras.optimizers.schedules.PolynomialDecay(\n        initial_learning_rate=init_lr,\n        decay_steps=num_train_steps,\n        end_learning_rate=0.0)\n    if num_warmup_steps:\n        learning_rate_fn = WarmUp(initial_learning_rate=init_lr,\n                                  decay_schedule_fn=learning_rate_fn,\n                                  warmup_steps=num_warmup_steps)\n    additional_args = {}\n    if clipnorm:\n        additional_args['clipnorm'] = clipnorm\n    optimizer = AdamWeightDecay(\n        learning_rate=learning_rate_fn,\n        weight_decay_rate=weight_decay_rate,\n        beta_1=0.9,\n        beta_2=0.999,\n        epsilon=epsilon,\n        exclude_from_weight_decay=['LayerNorm', 'bias'],\n        **additional_args\n    )\n    # {'LayerNorm/gamma:0', 'LayerNorm/beta:0'}\n    return optimizer\n"
  },
  {
    "path": "hanlp/optimizers/adamw/optimization.py",
    "content": "# Copyright 2019 The TensorFlow Authors. All Rights Reserved.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n#     http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# ==============================================================================\n\"\"\"Functions and classes related to optimization (weight updates).\"\"\"\n\nfrom __future__ import absolute_import\nfrom __future__ import division\nfrom __future__ import print_function\n\nimport re\n\nimport tensorflow as tf\n\n\nclass WarmUp(tf.keras.optimizers.schedules.LearningRateSchedule):\n    \"\"\"Applys a warmup schedule on a given learning rate decay schedule.\"\"\"\n\n    def __init__(\n            self,\n            initial_learning_rate,\n            decay_schedule_fn,\n            warmup_steps,\n            power=1.0,\n            name=None):\n        super(WarmUp, self).__init__()\n        self.initial_learning_rate = initial_learning_rate\n        self.warmup_steps = warmup_steps\n        self.power = power\n        self.decay_schedule_fn = decay_schedule_fn\n        self.name = name\n\n    def __call__(self, step):\n        with tf.name_scope(self.name or 'WarmUp') as name:\n            # Implements polynomial warmup. i.e., if global_step < warmup_steps, the\n            # learning rate will be `global_step/num_warmup_steps * init_lr`.\n            global_step_float = tf.cast(step, tf.float32)\n            warmup_steps_float = tf.cast(self.warmup_steps, tf.float32)\n            warmup_percent_done = global_step_float / warmup_steps_float\n            warmup_learning_rate = (\n                    self.initial_learning_rate *\n                    tf.math.pow(warmup_percent_done, self.power))\n            return tf.cond(global_step_float < warmup_steps_float,\n                           lambda: warmup_learning_rate,\n                           lambda: self.decay_schedule_fn(step),\n                           name=name)\n\n    def get_config(self):\n        return {\n            'initial_learning_rate': self.initial_learning_rate,\n            'decay_schedule_fn': self.decay_schedule_fn,\n            'warmup_steps': self.warmup_steps,\n            'power': self.power,\n            'name': self.name\n        }\n\n\ndef create_optimizer(init_lr, num_train_steps, num_warmup_steps):\n    \"\"\"Creates an optimizer with learning rate schedule.\n\n    Args:\n      init_lr: \n      num_train_steps: \n      num_warmup_steps: \n\n    Returns:\n\n    \"\"\"\n    # Implements linear decay of the learning rate.\n    learning_rate_fn = tf.keras.optimizers.schedules.PolynomialDecay(\n        initial_learning_rate=init_lr,\n        decay_steps=num_train_steps,\n        end_learning_rate=0.0)\n    if num_warmup_steps:\n        learning_rate_fn = WarmUp(initial_learning_rate=init_lr,\n                                  decay_schedule_fn=learning_rate_fn,\n                                  warmup_steps=num_warmup_steps)\n    optimizer = AdamWeightDecay(\n        learning_rate=learning_rate_fn,\n        weight_decay_rate=0.01,\n        beta_1=0.9,\n        beta_2=0.999,\n        epsilon=1e-6,\n        exclude_from_weight_decay=['layer_norm', 'bias'])\n    return optimizer\n\n\ntry:\n    AdamTF = tf.keras.optimizers.legacy.Adam  # avoid slowdown when using v2.11+ Keras optimizers on M1/M2 Macs\nexcept:\n    AdamTF = tf.keras.optimizers.Adam\n\n\nclass AdamWeightDecay(AdamTF):\n    \"\"\"Adam enables L2 weight decay and clip_by_global_norm on gradients.\n    \n      Just adding the square of the weights to the loss function is *not* the\n      correct way of using L2 regularization/weight decay with Adam, since that will\n      interact with the m and v parameters in strange ways.\n    \n      Instead we want to decay the weights in a manner that doesn't interact with\n      the m/v parameters. This is equivalent to adding the square of the weights to\n      the loss with plain (non-momentum) SGD.\n\n    Args:\n\n    Returns:\n\n    \"\"\"\n\n    def __init__(self,\n                 learning_rate=0.001,\n                 beta_1=0.9,\n                 beta_2=0.999,\n                 epsilon=1e-7,\n                 amsgrad=False,\n                 weight_decay_rate=0.0,\n                 include_in_weight_decay=None,\n                 exclude_from_weight_decay=None,\n                 name='AdamWeightDecay',\n                 **kwargs):\n        super(AdamWeightDecay, self).__init__(\n            learning_rate, beta_1, beta_2, epsilon, amsgrad, name, **kwargs)\n        self.weight_decay_rate = weight_decay_rate\n        self._include_in_weight_decay = include_in_weight_decay\n        self._exclude_from_weight_decay = exclude_from_weight_decay\n\n    @classmethod\n    def from_config(cls, config):\n        \"\"\"Creates an optimizer from its config with WarmUp custom object.\n\n        Args:\n          config:\n\n        Returns:\n\n        \"\"\"\n        custom_objects = {'WarmUp': WarmUp}\n        return super(AdamWeightDecay, cls).from_config(\n            config, custom_objects=custom_objects)\n\n    def _prepare_local(self, var_device, var_dtype, apply_state):\n        super(AdamWeightDecay, self)._prepare_local(var_device, var_dtype,\n                                                    apply_state)\n        apply_state['weight_decay_rate'] = tf.constant(\n            self.weight_decay_rate, name='adam_weight_decay_rate')\n\n    def _decay_weights_op(self, var, learning_rate, apply_state):\n        do_decay = self._do_use_weight_decay(var.name)\n        if do_decay:\n            return var.assign_sub(\n                learning_rate * var *\n                apply_state['weight_decay_rate'],\n                use_locking=self._use_locking)\n        return tf.no_op()\n\n    def apply_gradients(self, grads_and_vars, name=None):\n        grads, tvars = list(zip(*grads_and_vars))\n        (grads, _) = tf.clip_by_global_norm(grads, clip_norm=1.0)\n        return super(AdamWeightDecay, self).apply_gradients(zip(grads, tvars))\n\n    def _get_lr(self, var_device, var_dtype, apply_state):\n        \"\"\"Retrieves the learning rate with the given state.\n\n        Args:\n          var_device:\n          var_dtype:\n          apply_state:\n\n        Returns:\n\n        \"\"\"\n        if apply_state is None:\n            return self._decayed_lr_t[var_dtype], {}\n\n        apply_state = apply_state or {}\n        coefficients = apply_state.get((var_device, var_dtype))\n        if coefficients is None:\n            coefficients = self._fallback_apply_state(var_device, var_dtype)\n            apply_state[(var_device, var_dtype)] = coefficients\n\n        return coefficients['lr_t'], dict(apply_state=apply_state)\n\n    def _resource_apply_dense(self, grad, var, apply_state=None):\n        lr_t, kwargs = self._get_lr(var.device, var.dtype.base_dtype, apply_state)\n        decay = self._decay_weights_op(var, lr_t, apply_state)\n        with tf.control_dependencies([decay]):\n            return super(AdamWeightDecay, self)._resource_apply_dense(\n                grad, var, **kwargs)\n\n    def _resource_apply_sparse(self, grad, var, indices, apply_state=None):\n        lr_t, kwargs = self._get_lr(var.device, var.dtype.base_dtype, apply_state)\n        decay = self._decay_weights_op(var, lr_t, apply_state)\n        with tf.control_dependencies([decay]):\n            return super(AdamWeightDecay, self)._resource_apply_sparse(\n                grad, var, indices, **kwargs)\n\n    def get_config(self):\n        config = super(AdamWeightDecay, self).get_config()\n        config.update({\n            'weight_decay_rate': self.weight_decay_rate,\n        })\n        return config\n\n    def _do_use_weight_decay(self, param_name):\n        \"\"\"Whether to use L2 weight decay for `param_name`.\n\n        Args:\n          param_name:\n\n        Returns:\n\n        \"\"\"\n        if self.weight_decay_rate == 0:\n            return False\n\n        if self._include_in_weight_decay:\n            for r in self._include_in_weight_decay:\n                if re.search(r, param_name) is not None:\n                    return True\n\n        if self._exclude_from_weight_decay:\n            for r in self._exclude_from_weight_decay:\n                if re.search(r, param_name) is not None:\n                    return False\n        return True\n\n    def apply_gradients(self, grads_and_vars, name=None, **kwargs):\n        grads, tvars = list(zip(*grads_and_vars))\n        return super(AdamWeightDecay, self).apply_gradients(zip(grads, tvars), name=name, **kwargs)\n"
  },
  {
    "path": "hanlp/pretrained/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 19:10\nfrom hanlp.pretrained import tok\nfrom hanlp.pretrained import dep\nfrom hanlp.pretrained import sdp\nfrom hanlp.pretrained import glove\nfrom hanlp.pretrained import pos\nfrom hanlp.pretrained import rnnlm\nfrom hanlp.pretrained import word2vec\nfrom hanlp.pretrained import ner\nfrom hanlp.pretrained import classifiers\nfrom hanlp.pretrained import fasttext\nfrom hanlp.pretrained import mtl\nfrom hanlp.pretrained import eos\nfrom hanlp.pretrained import sts\nfrom hanlp.pretrained import constituency\nfrom hanlp.pretrained import amr\nfrom hanlp.pretrained import amr2text\nfrom hanlp.pretrained import srl\n\n# Will be filled up during runtime\nALL = {}\n"
  },
  {
    "path": "hanlp/pretrained/amr.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2022-01-25 11:47\nfrom hanlp_common.constant import HANLP_URL\n\nAMR3_SEQ2SEQ_BART_LARGE = HANLP_URL + 'amr/amr3_seq2seq_bart_large_83.30_20220125_114450.zip'\n'''A seq2seq (:cite:`bevilacqua-etal-2021-one`) BART (:cite:`lewis-etal-2020-bart`) large parser trained on Abstract \nMeaning Representation 3.0 (:cite:`knight2014abstract`). Its performance is\n\n =================== ========= ========= ========= \n  Metric              P         R         F1       \n =================== ========= ========= ========= \n  Smatch              84.00     82.60     83.30    \n  Unlabeled           86.40     84.90     85.70    \n  No WSD              84.50     83.10     83.80    \n  Non_sense_frames    91.90     91.30     91.60    \n  Wikification        81.70     80.80     81.20    \n  Named Ent.          89.20     87.00     88.10    \n  Negations           71.70     70.90     71.30    \n  IgnoreVars          73.80     73.10     73.50    \n  Concepts            90.70     89.60     90.10    \n  Frames              88.50     87.90     88.20    \n  Reentrancies        70.40     71.80     71.10    \n  SRL                 79.00     79.60     79.30    \n =================== ========= ========= ========= \n    \nNote this parser does NOT perform wikification.\n'''\n\nAMR3_GRAPH_PRETRAIN_PARSER = HANLP_URL + 'amr/amr3_graph_pretrain_parser_20221207_153759.zip'\n'''A seq2seq (:cite:`bevilacqua-etal-2021-one`) BART (:cite:`lewis-etal-2020-bart`) large parser trained on Abstract \nMeaning Representation 3.0 (:cite:`knight2014abstract`) with graph pre-training (:cite:`bai-etal-2022-graph`). \nIts performance is ``84.3`` according to their official repository. Using ``amr-evaluation-enhanced``, the performance is\nslightly lower:\n\n =================== ========= ========= ========= \n  Metric              P         R         F1       \n =================== ========= ========= ========= \n  Smatch             84.4       83.6        84.0       \n  Unlabeled          86.7       85.8        86.2       \n  No WSD             84.9       84.1        84.5       \n  Non_sense_frames   91.8       91.6        91.7       \n  Wikification       83.6       81.7        82.6       \n  Named Ent.         89.3       87.4        88.4       \n  Negations          71.6       72.2        71.9       \n  IgnoreVars         74.6       74.2        74.4       \n  Concepts           90.7       90.0        90.3       \n  Frames             88.8       88.5        88.7       \n  Reentrancies       72.1       72.9        72.5       \n  SRL                80.1       80.7        80.4      \n =================== ========= ========= ========= \n    \nNote this parser does NOT perform wikification.\n'''\n\nMRP2020_AMR_ENG_ZHO_XLM_BASE = 'http://download.hanlp.com/amr/extra/amr-eng-zho-xlm-roberta-base_20220412_223756.zip'\n'''A wrapper for the Permutation-invariant Semantic Parser (:cite:`samuel-straka-2020-ufal`) trained on MRP2020 English \nand Chinese AMR corpus. It was ranked the top in the MRP2020 competition, while this release is a base version. \nSee the original paper for the detailed performance. Note this model requires tokens and lemmas (for English) to be \nprovided as inputs. \n'''\n\nMRP2020_AMR_ZHO_MENGZI_BASE = 'http://download.hanlp.com/amr/extra/amr-zho-mengzi-base_20220415_101941.zip'\n'''A Chinese Permutation-invariant Semantic Parser (:cite:`samuel-straka-2020-ufal`) trained on MRP2020  \nChinese AMR corpus using Mengzi BERT base (:cite:`zhang2021mengzi`). Its performance on dev set is \n``{amr-zho [tops F1: 85.43%][anchors F1: 93.41%][labels F1: 87.68%][properties F1: 82.02%][edges F1: 73.17%]\n[attributes F1: 0.00%][all F1: 84.11%]}``. Test set performance is unknown since the test set is not released to the \npublic. \n'''\n\n# Will be filled up during runtime\nALL = {}\n"
  },
  {
    "path": "hanlp/pretrained/amr2text.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2022-12-07 15:19\nfrom hanlp_common.constant import HANLP_URL\n\nAMR3_GRAPH_PRETRAIN_GENERATION = HANLP_URL + 'amr2text/amr3_graph_pretrain_generation_20221207_153535.zip'\n'''A seq2seq (:cite:`bevilacqua-etal-2021-one`) BART (:cite:`lewis-etal-2020-bart`) large AMR2Text generator trained on \nAbstract Meaning Representation 3.0 (:cite:`knight2014abstract`) with graph pre-training (:cite:`bai-etal-2022-graph`). \nIts Sacre-BLEU is ``50.38`` according to their official repository.\n'''\n\n# Will be filled up during runtime\nALL = {}\n"
  },
  {
    "path": "hanlp/pretrained/classifiers.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-01-01 03:51\nfrom hanlp_common.constant import HANLP_URL\n\nCHNSENTICORP_BERT_BASE_ZH = HANLP_URL + 'classification/chnsenticorp_bert_base_20211228_163210.zip'\nSST2_ALBERT_BASE_EN = HANLP_URL + 'classification/sst2_albert_base_20211228_164917.zip'\n\nLID_176_FASTTEXT_BASE = 'https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin'\n'''\n126MB FastText model for language identification trained on data from Wikipedia, Tatoeba and SETimes.\n'''\nLID_176_FASTTEXT_SMALL = 'https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.ftz'\n'''\n917kB FastText model for language identification trained on data from Wikipedia, Tatoeba and SETimes.\n'''\n\nALL = {}\n"
  },
  {
    "path": "hanlp/pretrained/constituency.py",
    "content": "# -*- coding:utf-8 -*-\n# Author=hankcs\n# Date=2022-01-18 10:34\nfrom hanlp_common.constant import HANLP_URL\n\nCTB9_CON_ELECTRA_SMALL = HANLP_URL + 'constituency/ctb9_con_electra_small_20220215_230116.zip'\n'Electra (:cite:`clark2020electra`) small tree CRF model (:cite:`ijcai2020-560`) trained on CTB9 with major categories. ' \\\n'Its performance is UCM=39.06% LCM=34.99% UP=90.05% UR=90.01% UF=90.03% LP=87.02% LR=86.98% LF=87.00%.'\n\nCTB9_CON_FULL_TAG_ELECTRA_SMALL = HANLP_URL + 'constituency/ctb9_full_tag_con_electra_small_20220118_103119.zip'\n'Electra (:cite:`clark2020electra`) small tree CRF model (:cite:`ijcai2020-560`) trained on CTB9 with full subcategories. ' \\\n'Its performance is UCM=38.29% LCM=28.95% UP=90.16% UR=90.13% UF=90.15% LP=83.46% LR=83.43% LF=83.45%.'\n\nCTB9_CON_FULL_TAG_ERNIE_GRAM = 'http://download.hanlp.com/constituency/extra/ctb9_full_tag_con_ernie_20220331_121430.zip'\n'ERNIE-GRAM (:cite:`xiao-etal-2021-ernie`) base tree CRF model (:cite:`ijcai2020-560`) trained on CTB9 with full subcategories. ' \\\n'Its performance is UCM=42.04% LCM=31.72% UP=91.33% UR=91.53% UF=91.43% LP=85.31% LR=85.49% LF=85.40%.'\n\n# Will be filled up during runtime\nALL = {}\n"
  },
  {
    "path": "hanlp/pretrained/dep.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-29 02:55\nfrom hanlp_common.constant import HANLP_URL\n\nCTB5_BIAFFINE_DEP_ZH = HANLP_URL + 'dep/biaffine_ctb5_20191229_025833.zip'\n'Biaffine LSTM model (:cite:`dozat:17a`) trained on CTB5.'\nCTB7_BIAFFINE_DEP_ZH = HANLP_URL + 'dep/biaffine_ctb7_20200109_022431.zip'\n'Biaffine LSTM model (:cite:`dozat:17a`) trained on CTB7.'\nCTB9_DEP_ELECTRA_SMALL = HANLP_URL + 'dep/ctb9_dep_electra_small_20220216_100306.zip'\n'Electra small encoder (:cite:`clark2020electra`) with Biaffine decoder (:cite:`dozat:17a`) trained on CTB9-SD330. ' \\\n'Performance is UAS=87.68% LAS=83.54%.'\nPMT1_DEP_ELECTRA_SMALL = HANLP_URL + 'dep/pmt_dep_electra_small_20220218_134518.zip'\n'Electra small encoder (:cite:`clark2020electra`) with Biaffine decoder (:cite:`dozat:17a`) trained on PKU ' \\\n'Multi-view Chinese Treebank (PMT) 1.0 (:cite:`qiu-etal-2014-multi`). Performance is UAS=91.21% LAS=88.65%.'\nCTB9_UDC_ELECTRA_SMALL = HANLP_URL + 'dep/udc_dep_electra_small_20220218_095452.zip'\n'Electra small encoder (:cite:`clark2020electra`) with Biaffine decoder (:cite:`dozat:17a`) trained on CTB9-UD420. ' \\\n'Performance is UAS=85.92% LAS=81.13% .'\n\nPTB_BIAFFINE_DEP_EN = HANLP_URL + 'dep/ptb_dep_biaffine_20200101_174624.zip'\n'Biaffine LSTM model (:cite:`dozat:17a`) trained on PTB.'\n\nALL = {}\n"
  },
  {
    "path": "hanlp/pretrained/eos.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-22 13:22\nfrom hanlp_common.constant import HANLP_URL\n\nUD_CTB_EOS_MUL = HANLP_URL + 'eos/eos_ud_ctb_mul_20201222_133543.zip'\n'EOS model (:cite:`Schweter:Ahmed:2019`) trained on concatenated UD2.3 and CTB9.'\n\n# Will be filled up during runtime\nALL = {}\n"
  },
  {
    "path": "hanlp/pretrained/fasttext.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-30 18:57\nFASTTEXT_DEBUG_EMBEDDING_EN = 'https://elit-models.s3-us-west-2.amazonaws.com/fasttext.debug.bin.zip'\nFASTTEXT_CC_300_EN = 'https://dl.fbaipublicfiles.com/fasttext/vectors-crawl/cc.en.300.bin.gz'\n'FastText (:cite:`bojanowski2017enriching`) embeddings trained on Common Crawl.'\nFASTTEXT_WIKI_NYT_AMAZON_FRIENDS_200_EN \\\n    = 'https://elit-models.s3-us-west-2.amazonaws.com/fasttext-200-wikipedia-nytimes-amazon-friends-20191107.bin'\n'FastText (:cite:`bojanowski2017enriching`) embeddings trained on wikipedia, nytimes and friends.'\n\nFASTTEXT_WIKI_300_ZH = 'https://dl.fbaipublicfiles.com/fasttext/vectors-wiki/wiki.zh.zip#wiki.zh.bin'\n'FastText (:cite:`bojanowski2017enriching`) embeddings trained on Chinese Wikipedia.'\nFASTTEXT_WIKI_300_ZH_CLASSICAL = 'https://dl.fbaipublicfiles.com/fasttext/vectors-wiki/wiki.zh_classical.zip#wiki.zh_classical.bin'\n'FastText (:cite:`bojanowski2017enriching`) embeddings trained on traditional Chinese wikipedia.'\n\nALL = {}\n"
  },
  {
    "path": "hanlp/pretrained/glove.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-08-27 20:42\n\n_GLOVE_6B_ROOT = 'http://downloads.cs.stanford.edu/nlp/data/glove.6B.zip'\n\nGLOVE_6B_50D = _GLOVE_6B_ROOT + '#' + 'glove.6B.50d.txt'\n'Global Vectors for Word Representation (:cite:`pennington-etal-2014-glove`) 50d trained on 6B tokens.'\nGLOVE_6B_100D = _GLOVE_6B_ROOT + '#' + 'glove.6B.100d.txt'\n'Global Vectors for Word Representation (:cite:`pennington-etal-2014-glove`) 100d trained on 6B tokens.'\nGLOVE_6B_200D = _GLOVE_6B_ROOT + '#' + 'glove.6B.200d.txt'\n'Global Vectors for Word Representation (:cite:`pennington-etal-2014-glove`) 200d trained on 6B tokens.'\nGLOVE_6B_300D = _GLOVE_6B_ROOT + '#' + 'glove.6B.300d.txt'\n'Global Vectors for Word Representation (:cite:`pennington-etal-2014-glove`) 300d trained on 6B tokens.'\n\nGLOVE_840B_300D = 'http://nlp.stanford.edu/data/glove.840B.300d.zip'\n'Global Vectors for Word Representation (:cite:`pennington-etal-2014-glove`) 300d trained on 840B tokens.'\n\nALL = {}\n"
  },
  {
    "path": "hanlp/pretrained/mtl.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-22 13:16\nfrom hanlp_common.constant import HANLP_URL\n\nOPEN_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH = HANLP_URL + 'mtl/open_tok_pos_ner_srl_dep_sdp_con_electra_small_20201223_035557.zip'\n\"Electra (:cite:`clark2020electra`) small version of joint tok, pos, ner, srl, dep, sdp and con model trained on open-source Chinese corpus.\"\nOPEN_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH = HANLP_URL + 'mtl/open_tok_pos_ner_srl_dep_sdp_con_electra_base_20201223_201906.zip'\n\"Electra (:cite:`clark2020electra`) base version of joint tok, pos, ner, srl, dep, sdp and con model trained on open-source Chinese corpus.\"\nCLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH = HANLP_URL + 'mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_small_20210111_124159.zip'\n\"Electra (:cite:`clark2020electra`) small version of joint tok, pos, ner, srl, dep (SD Standard), sdp and con model trained on close-source Chinese corpus.\"\nCLOSE_TOK_POS_NER_SRL_UDEP_SDP_CON_ELECTRA_SMALL_ZH = HANLP_URL + 'mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_small_20220626_175100.zip'\n'''\nElectra (:cite:`clark2020electra`) small version of joint tok, pos, ner, srl, dep (UD Standard), sdp and con model trained on close-source Chinese corpus.\nPerformance: ``{con UCM: 39.33% LCM: 35.69% UP: 90.24% UR: 90.28% UF: 90.26% LP: 87.55% LR: 87.59% LF: 87.57%}{dep UAS: 86.80% LAS: 82.82%}{ner/msra P: 95.45% R: 96.65% F1: 96.05%}{ner/ontonotes P: 75.98% R: 79.09% F1: 77.50%}{ner/pku P: 95.77% R: 96.75% F1: 96.26%}{pos/863 Accuracy:94.83%}{pos/ctb Accuracy:96.57%}{pos/pku Accuracy:97.54%}{sdp UF: 85.55% LF: 73.67%}{srl P: 75.71% R: 74.25% F1: 74.97%}{tok/coarse P: 97.77% R: 97.70% F1: 97.74%}{tok/fine P: 97.44% R: 97.32% F1: 97.38%}``.\n'''\nCLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH = HANLP_URL + 'mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_base_20210111_124519.zip'\n\"Electra (:cite:`clark2020electra`) base version of joint tok, pos, ner, srl, dep, sdp and con model trained on close-source Chinese corpus.\"\nCLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ERNIE_GRAM_ZH = HANLP_URL + 'mtl/close_tok_pos_ner_srl_dep_sdp_con_ernie_gram_base_aug_20210904_145403.zip'\n\"ERNIE (:cite:`xiao-etal-2021-ernie`) base version of joint tok, pos, ner, srl, dep, sdp and con model trained on close-source Chinese corpus.\"\n\nUD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_MMINILMV2L6 = HANLP_URL + 'mtl/ud_ontonotes_tok_pos_lem_fea_ner_srl_dep_sdp_con_mMiniLMv2L6_no_space_20220731_161526.zip'\n'''\nmMiniLMv2 (:cite:`wang-etal-2021-minilmv2`) L6xH384 small version of joint tok, pos, lem, fea, ner, srl, dep, sdp and con model trained on UD 2.10 and OntoNotes5 corpora.\nThe following 130 languages are supported: ``Afrikaans, Akkadian, Akuntsu, Albanian, Amharic, AncientGreek (to 1453), Ancient Hebrew, Apurinã, Arabic, Armenian, AssyrianNeo-Aramaic, Bambara, Basque, Beja, Belarusian, Bengali, Bhojpuri, Breton, Bulgarian, Catalan, Cebuano, Central Siberian Yupik, Chinese, Chukot, ChurchSlavic, Coptic, Croatian, Czech, Danish, Dutch, Emerillon, English, Erzya, Estonian, Faroese, Finnish, French, Galician, German, Gothic, Guajajára, Guarani, Hebrew, Hindi, Hittite, Hungarian, Icelandic, Indonesian, Irish, Italian, Japanese, Javanese, K\\'iche\\', Kangri, Karelian, Karo(Brazil), Kazakh, Khunsari, Komi-Permyak, Komi-Zyrian, Korean, Latin, Latvian, Ligurian, LiteraryChinese, Lithuanian, Livvi, LowGerman, Madi, Makuráp, Maltese, Manx, Marathi, MbyáGuaraní, Modern Greek (1453-), Moksha, Mundurukú, Nayini, Neapolitan, Nigerian Pidgin, NorthernKurdish, Northern Sami, Norwegian, OldFrench (842-ca. 1400), OldRussian, Old Turkish, Persian, Polish, Portuguese, Romanian, Russia Buriat, Russian, Sanskrit, ScottishGaelic, Serbian, SkoltSami, Slovak, Slovenian, Soi, South Levantine Arabic, Spanish, Swedish, SwedishSign Language, SwissGerman, Tagalog, Tamil, Tatar, Telugu, Thai, Tupinambá, Turkish, Uighur, Ukrainian, Umbrian, UpperSorbian, Urdu, Urubú-Kaapor, Vietnamese, Warlpiri, Welsh, Western Armenian, WesternFrisian, Wolof, Xibe, Yakut, Yoruba, YueChinese``.\nPerformance: ``{con UCM: 15.48% LCM: 11.45% UP: 68.92% UR: 66.88% UF: 67.88% LP: 61.19% LR: 59.38% LF: 60.27%}{ner P: 76.06% R: 77.83% F1: 76.93%}{sdp/dm UF: 91.84% LF: 91.00%}{sdp/pas UF: 95.46% LF: 93.90%}{sdp/psd UF: 91.94% LF: 81.26%}{srl [predicate P: 91.71% R: 74.51% F1: 82.22%][e2e P: 77.48% R: 55.28% F1: 64.52%]}{tok P: 93.17% R: 93.53% F1: 93.35%}{ud [lemmas Accuracy:81.74%][upos Accuracy:85.94%][deps UAS: 80.60% LAS: 71.21%][feats Accuracy:77.17%]}``.\n'''\nUD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_MMINILMV2L12 = HANLP_URL + 'mtl/ud_ontonotes_tok_pos_lem_fea_ner_srl_dep_sdp_con_mMiniLMv2L12_no_space_20220807_133143.zip'\n'''\nmMiniLMv2 (:cite:`wang-etal-2021-minilmv2`) L6xH384 base version of joint tok, pos, lem, fea, ner, srl, dep, sdp and con model trained on UD 2.10 and OntoNotes5 corpora.\nThe following 130 languages are supported: ``Afrikaans, Akkadian, Akuntsu, Albanian, Amharic, AncientGreek (to 1453), Ancient Hebrew, Apurinã, Arabic, Armenian, AssyrianNeo-Aramaic, Bambara, Basque, Beja, Belarusian, Bengali, Bhojpuri, Breton, Bulgarian, Catalan, Cebuano, Central Siberian Yupik, Chinese, Chukot, ChurchSlavic, Coptic, Croatian, Czech, Danish, Dutch, Emerillon, English, Erzya, Estonian, Faroese, Finnish, French, Galician, German, Gothic, Guajajára, Guarani, Hebrew, Hindi, Hittite, Hungarian, Icelandic, Indonesian, Irish, Italian, Japanese, Javanese, K\\'iche\\', Kangri, Karelian, Karo(Brazil), Kazakh, Khunsari, Komi-Permyak, Komi-Zyrian, Korean, Latin, Latvian, Ligurian, LiteraryChinese, Lithuanian, Livvi, LowGerman, Madi, Makuráp, Maltese, Manx, Marathi, MbyáGuaraní, Modern Greek (1453-), Moksha, Mundurukú, Nayini, Neapolitan, Nigerian Pidgin, NorthernKurdish, Northern Sami, Norwegian, OldFrench (842-ca. 1400), OldRussian, Old Turkish, Persian, Polish, Portuguese, Romanian, Russia Buriat, Russian, Sanskrit, ScottishGaelic, Serbian, SkoltSami, Slovak, Slovenian, Soi, South Levantine Arabic, Spanish, Swedish, SwedishSign Language, SwissGerman, Tagalog, Tamil, Tatar, Telugu, Thai, Tupinambá, Turkish, Uighur, Ukrainian, Umbrian, UpperSorbian, Urdu, Urubú-Kaapor, Vietnamese, Warlpiri, Welsh, Western Armenian, WesternFrisian, Wolof, Xibe, Yakut, Yoruba, YueChinese``.\nPerformance: ``{con UCM: 17.32% LCM: 13.28% UP: 70.53% UR: 68.73% UF: 69.62% LP: 63.03% LR: 61.42% LF: 62.22%}{ner P: 76.91% R: 78.72% F1: 77.80%}{sdp/dm UF: 92.78% LF: 92.02%}{sdp/pas UF: 96.43% LF: 95.02%}{sdp/psd UF: 92.75% LF: 81.86%}{srl [predicate P: 91.82% R: 77.57% F1: 84.10%][e2e P: 78.33% R: 59.14% F1: 67.40%]}{tok P: 93.69% R: 94.34% F1: 94.02%}{ud [lemmas Accuracy:82.48%][upos Accuracy:87.09%][deps UAS: 82.41% LAS: 73.69%][feats Accuracy:78.58%]}``.\n'''\nUD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_XLMR_BASE = HANLP_URL + 'mtl/ud_ontonotes_tok_pos_lem_fea_ner_srl_dep_sdp_con_xlm_base_20220608_003435.zip'\n'''\nXLM-R (:cite:`conneau-etal-2020-unsupervised`) base version of joint tok, pos, lem, fea, ner, srl, dep, sdp and con model trained on UD 2.10 and OntoNotes5 corpora.\nThe following 130 languages are supported: ``Afrikaans, Akkadian, Akuntsu, Albanian, Amharic, AncientGreek (to 1453), Ancient Hebrew, Apurinã, Arabic, Armenian, AssyrianNeo-Aramaic, Bambara, Basque, Beja, Belarusian, Bengali, Bhojpuri, Breton, Bulgarian, Catalan, Cebuano, Central Siberian Yupik, Chinese, Chukot, ChurchSlavic, Coptic, Croatian, Czech, Danish, Dutch, Emerillon, English, Erzya, Estonian, Faroese, Finnish, French, Galician, German, Gothic, Guajajára, Guarani, Hebrew, Hindi, Hittite, Hungarian, Icelandic, Indonesian, Irish, Italian, Japanese, Javanese, K\\'iche\\', Kangri, Karelian, Karo(Brazil), Kazakh, Khunsari, Komi-Permyak, Komi-Zyrian, Korean, Latin, Latvian, Ligurian, LiteraryChinese, Lithuanian, Livvi, LowGerman, Madi, Makuráp, Maltese, Manx, Marathi, MbyáGuaraní, Modern Greek (1453-), Moksha, Mundurukú, Nayini, Neapolitan, Nigerian Pidgin, NorthernKurdish, Northern Sami, Norwegian, OldFrench (842-ca. 1400), OldRussian, Old Turkish, Persian, Polish, Portuguese, Romanian, Russia Buriat, Russian, Sanskrit, ScottishGaelic, Serbian, SkoltSami, Slovak, Slovenian, Soi, South Levantine Arabic, Spanish, Swedish, SwedishSign Language, SwissGerman, Tagalog, Tamil, Tatar, Telugu, Thai, Tupinambá, Turkish, Uighur, Ukrainian, Umbrian, UpperSorbian, Urdu, Urubú-Kaapor, Vietnamese, Warlpiri, Welsh, Western Armenian, WesternFrisian, Wolof, Xibe, Yakut, Yoruba, YueChinese``.\nPerformance: ``{con UCM: 20.31% LCM: 16.82% UP: 77.50% UR: 76.63% UF: 77.06% LP: 71.25% LR: 70.46% LF: 70.85%}{ner P: 79.93% R: 80.76% F1: 80.34%}{sdp/dm UF: 93.71% LF: 93.00%}{sdp/pas UF: 97.63% LF: 96.37%}{sdp/psd UF: 93.08% LF: 80.95%}{srl [predicate P: 90.95% R: 84.25% F1: 87.47%][e2e P: 78.89% R: 67.32% F1: 72.65%]}{tok P: 98.50% R: 98.70% F1: 98.60%}{ud [lemmas Accuracy:85.95%][upos Accuracy:89.95%][deps UAS: 85.78% LAS: 78.51%][feats Accuracy:82.18%]}``.\n'''\n\nNPCMJ_UD_KYOTO_TOK_POS_CON_BERT_BASE_CHAR_JA = HANLP_URL + 'mtl/npcmj_ud_kyoto_tok_pos_ner_dep_con_srl_bert_base_char_ja_20210914_133742.zip'\n'BERT (:cite:`devlin-etal-2019-bert`) base char encoder trained on NPCMJ/UD/Kyoto corpora with decoders including tok, pos, ner, dep, con, srl.'\n\n# Will be filled up during runtime\nALL = {}\n"
  },
  {
    "path": "hanlp/pretrained/ner.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-30 20:07\nfrom hanlp_common.constant import HANLP_URL\n\nMSRA_NER_BERT_BASE_ZH = HANLP_URL + 'ner/ner_bert_base_msra_20211227_114712.zip'\n'BERT model (:cite:`devlin-etal-2019-bert`) trained on MSRA with 3 entity types.'\nMSRA_NER_ALBERT_BASE_ZH = HANLP_URL + 'ner/msra_ner_albert_base_20211228_173323.zip'\n'ALBERT model (:cite:`Lan2020ALBERT:`) trained on MSRA with 3 entity types.'\nMSRA_NER_ELECTRA_SMALL_ZH = HANLP_URL + 'ner/msra_ner_electra_small_20220215_205503.zip'\n'Electra small model (:cite:`clark2020electra`) trained on MSRA with 26 entity types. F1 = `95.16`'\nCONLL03_NER_BERT_BASE_CASED_EN = HANLP_URL + 'ner/ner_conll03_bert_base_cased_en_20211227_121443.zip'\n'BERT model (:cite:`devlin-etal-2019-bert`) trained on CoNLL03.'\n\nALL = {}\n"
  },
  {
    "path": "hanlp/pretrained/pos.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-29 01:57\nfrom hanlp_common.constant import HANLP_URL\n\nCTB5_POS_RNN = HANLP_URL + 'pos/ctb5_pos_rnn_20200113_235925.zip'\n'An old school BiLSTM tagging model trained on CTB5.'\nCTB5_POS_RNN_FASTTEXT_ZH = HANLP_URL + 'pos/ctb5_pos_rnn_fasttext_20191230_202639.zip'\n'An old school BiLSTM tagging model with FastText (:cite:`bojanowski2017enriching`) embeddings trained on CTB5.'\nCTB9_POS_ALBERT_BASE = HANLP_URL + 'pos/ctb9_albert_base_20211228_163935.zip'\n'ALBERT model (:cite:`Lan2020ALBERT:`) trained on CTB9 (:cite:`https://doi.org/10.35111/gvd0-xk91`). This is a TF component.'\nCTB9_POS_ELECTRA_SMALL_TF = HANLP_URL + 'pos/pos_ctb_electra_small_20211227_121341.zip'\n'Electra small model (:cite:`clark2020electra`) trained on CTB9 (:cite:`https://doi.org/10.35111/gvd0-xk91`). Accuracy = `96.75`. This is a TF component.'\nCTB9_POS_ELECTRA_SMALL = HANLP_URL + 'pos/pos_ctb_electra_small_20220215_111944.zip'\n'Electra small model (:cite:`clark2020electra`) trained on CTB9 (:cite:`https://doi.org/10.35111/gvd0-xk91`). Accuracy = `96.26`.'\nCTB9_POS_RADICAL_ELECTRA_SMALL = HANLP_URL + 'pos/pos_ctb_radical_electra_small_20220215_111932.zip'\n'Electra small model (:cite:`clark2020electra`) with radical embeddings (:cite:`he2018dual`) trained on CTB9 (:cite:`https://doi.org/10.35111/gvd0-xk91`). Accuracy = `96.14`.'\nC863_POS_ELECTRA_SMALL = HANLP_URL + 'pos/pos_863_electra_small_20220217_101958.zip'\n'Electra small model (:cite:`clark2020electra`) trained on Chinese 863 corpus. Accuracy = `95.19`.'\nPKU_POS_ELECTRA_SMALL = HANLP_URL + 'pos/pos_pku_electra_small_20220217_142436.zip'\n'Electra small model (:cite:`clark2020electra`) trained on Chinese PKU corpus. Accuracy = `97.55`.'\nPTB_POS_RNN_FASTTEXT_EN = HANLP_URL + 'pos/ptb_pos_rnn_fasttext_20220418_101708.zip'\n'An old school BiLSTM tagging model with FastText (:cite:`bojanowski2017enriching`) embeddings trained on PTB.'\n\nALL = {}\n"
  },
  {
    "path": "hanlp/pretrained/rnnlm.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-19 03:47\nfrom hanlp_common.constant import HANLP_URL\n\nFLAIR_LM_FW_WMT11_EN_TF = HANLP_URL + 'lm/flair_lm_wmt11_en_20200211_091932.zip#flair_lm_fw_wmt11_en'\n'The forward LSTM of Contextual String Embedding (:cite:`akbik-etal-2018-contextual`).'\nFLAIR_LM_BW_WMT11_EN_TF = HANLP_URL + 'lm/flair_lm_wmt11_en_20200211_091932.zip#flair_lm_bw_wmt11_en'\n'The backward LSTM of Contextual String Embedding (:cite:`akbik-etal-2018-contextual`).'\nFLAIR_LM_WMT11_EN = HANLP_URL + 'lm/flair_lm_wmt11_en_20200601_205350.zip'\n'The BiLSTM of Contextual String Embedding (:cite:`akbik-etal-2018-contextual`).'\n\nALL = {}\n"
  },
  {
    "path": "hanlp/pretrained/sdp.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-31 23:54\nfrom hanlp_common.constant import HANLP_URL\n\nSEMEVAL16_NEWS_BIAFFINE_ZH = HANLP_URL + 'sdp/semeval16-news-biaffine_20191231_235407.zip'\n'Biaffine SDP (:cite:`he-choi-2019`) trained on SemEval16 news data.'\nSEMEVAL16_TEXT_BIAFFINE_ZH = HANLP_URL + 'sdp/semeval16-text-biaffine_20200101_002257.zip'\n'Biaffine SDP (:cite:`he-choi-2019`) trained on SemEval16 text data.'\n\nSEMEVAL16_ALL_ELECTRA_SMALL_ZH = HANLP_URL + 'sdp/semeval16_sdp_electra_small_20220719_171433.zip'\n'Biaffine SDP (:cite:`he-choi-2019`) trained on SemEval16 text and news data. Performance: ``UF: 83.03% LF: 72.58%``'\n\nSEMEVAL15_PAS_BIAFFINE_EN = HANLP_URL + 'sdp/semeval15_biaffine_pas_20200103_152405.zip'\n'Biaffine SDP (:cite:`he-choi-2019`) trained on SemEval15 PAS data.'\nSEMEVAL15_PSD_BIAFFINE_EN = HANLP_URL + 'sdp/semeval15_biaffine_psd_20200106_123009.zip'\n'Biaffine SDP (:cite:`he-choi-2019`) trained on SemEval15 PSD data.'\nSEMEVAL15_DM_BIAFFINE_EN = HANLP_URL + 'sdp/semeval15_biaffine_dm_20200106_122808.zip'\n'Biaffine SDP (:cite:`he-choi-2019`) trained on SemEval15 DM data.'\n\nALL = {}\n"
  },
  {
    "path": "hanlp/pretrained/srl.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-08-07 19:07\nfrom hanlp_common.constant import HANLP_URL\n\nCPB3_SRL_ELECTRA_SMALL = HANLP_URL + 'srl/cpb3_electra_small_crf_has_transform_20220218_135910.zip'\n'Electra small model (:cite:`clark2020electra`) trained on CPB3. P=75.87% R=76.24% F1=76.05%.'\n\nALL = {}\n"
  },
  {
    "path": "hanlp/pretrained/sts.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-05-24 12:51\nfrom hanlp_common.constant import HANLP_URL\n\nSTS_ELECTRA_BASE_ZH = HANLP_URL + 'sts/sts_electra_base_zh_20210530_200109.zip'\n'A naive regression model trained on concatenated STS corpora.'\n\n# Will be filled up during runtime\nALL = {}\n"
  },
  {
    "path": "hanlp/pretrained/tok.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 21:12\nfrom hanlp_common.constant import HANLP_URL\n\nSIGHAN2005_PKU_CONVSEG = HANLP_URL + 'tok/sighan2005-pku-convseg_20200110_153722.zip'\n'Conv model (:cite:`wang-xu-2017-convolutional`) trained on sighan2005 pku dataset.'\nSIGHAN2005_MSR_CONVSEG = HANLP_URL + 'tok/convseg-msr-nocrf-noembed_20200110_153524.zip'\n'Conv model (:cite:`wang-xu-2017-convolutional`) trained on sighan2005 msr dataset.'\nCTB6_CONVSEG = HANLP_URL + 'tok/ctb6_convseg_nowe_nocrf_20200110_004046.zip'\n'Conv model (:cite:`wang-xu-2017-convolutional`) trained on CTB6 dataset.'\nPKU_NAME_MERGED_SIX_MONTHS_CONVSEG = HANLP_URL + 'tok/pku98_6m_conv_ngram_20200110_134736.zip'\n'Conv model (:cite:`wang-xu-2017-convolutional`) trained on pku98 six months dataset with familiy name and given name merged into one unit.'\nLARGE_ALBERT_BASE = HANLP_URL + 'tok/large_corpus_cws_albert_base_20211228_160926.zip'\n'ALBERT model (:cite:`Lan2020ALBERT:`) trained on the largest CWS dataset in the world.'\nSIGHAN2005_PKU_BERT_BASE_ZH = HANLP_URL + 'tok/sighan2005_pku_bert_base_zh_20201231_141130.zip'\n'BERT model (:cite:`devlin-etal-2019-bert`) trained on sighan2005 pku dataset.'\nCOARSE_ELECTRA_SMALL_ZH = HANLP_URL + 'tok/coarse_electra_small_20220616_012050.zip'\n'Electra (:cite:`clark2020electra`) small model trained on coarse-grained CWS corpora. Its performance is ``P: 98.34% R: 98.38% F1: 98.36%`` which is ' \\\n'much higher than that of MTL model '\nFINE_ELECTRA_SMALL_ZH = HANLP_URL + 'tok/fine_electra_small_20220615_231803.zip'\n'Electra (:cite:`clark2020electra`) small model trained on fine-grained CWS corpora. Its performance is ``P: 98.14% R: 98.07% F1: 98.11%`` which is ' \\\n'much higher than that of MTL model '\nCTB9_TOK_ELECTRA_SMALL = HANLP_URL + 'tok/ctb9_electra_small_20220215_205427.zip'\n'Electra (:cite:`clark2020electra`) small model trained on CTB9. Its performance is P=97.15% R=97.36% F1=97.26% which is ' \\\n'much higher than that of MTL model '\nCTB9_TOK_ELECTRA_BASE = 'http://download.hanlp.com/tok/extra/ctb9_tok_electra_base_20220426_111949.zip'\n'Electra (:cite:`clark2020electra`) base model trained on CTB9. Its performance is ``P: 97.62% R: 97.67% F1: 97.65%`` ' \\\n'which is much higher than that of MTL model '\nCTB9_TOK_ELECTRA_BASE_CRF = 'http://download.hanlp.com/tok/extra/ctb9_tok_electra_base_crf_20220426_161255.zip'\n'Electra (:cite:`clark2020electra`) base model trained on CTB9. Its performance is ``P: 97.68% R: 97.71% F1: 97.69%`` ' \\\n'which is much higher than that of MTL model '\nMSR_TOK_ELECTRA_BASE_CRF = 'http://download.hanlp.com/tok/extra/msra_crf_electra_base_20220507_113936.zip'\n'Electra (:cite:`clark2020electra`) base model trained on MSR CWS dataset. Its performance is ``P: 98.71% R: 98.64% F1: 98.68%`` ' \\\n'which is much higher than that of MTL model '\n\nUD_TOK_MMINILMV2L6 = HANLP_URL + 'tok/ud_tok_mMiniLMv2L6_no_space_mul_20220619_091824.zip'\n'''\nmMiniLMv2 (:cite:`wang-etal-2021-minilmv2`) L6xH384 based tokenizer trained on UD 2.10.\nThe following 130 languages are supported: ``Afrikaans, Akkadian, Akuntsu, Albanian, Amharic, AncientGreek (to 1453), Ancient Hebrew, Apurinã, Arabic, Armenian, AssyrianNeo-Aramaic, Bambara, Basque, Beja, Belarusian, Bengali, Bhojpuri, Breton, Bulgarian, Catalan, Cebuano, Central Siberian Yupik, Chinese, Chukot, ChurchSlavic, Coptic, Croatian, Czech, Danish, Dutch, Emerillon, English, Erzya, Estonian, Faroese, Finnish, French, Galician, German, Gothic, Guajajára, Guarani, Hebrew, Hindi, Hittite, Hungarian, Icelandic, Indonesian, Irish, Italian, Japanese, Javanese, K\\'iche\\', Kangri, Karelian, Karo(Brazil), Kazakh, Khunsari, Komi-Permyak, Komi-Zyrian, Korean, Latin, Latvian, Ligurian, LiteraryChinese, Lithuanian, Livvi, LowGerman, Madi, Makuráp, Maltese, Manx, Marathi, MbyáGuaraní, Modern Greek (1453-), Moksha, Mundurukú, Nayini, Neapolitan, Nigerian Pidgin, NorthernKurdish, Northern Sami, Norwegian, OldFrench (842-ca. 1400), OldRussian, Old Turkish, Persian, Polish, Portuguese, Romanian, Russia Buriat, Russian, Sanskrit, ScottishGaelic, Serbian, SkoltSami, Slovak, Slovenian, Soi, South Levantine Arabic, Spanish, Swedish, SwedishSign Language, SwissGerman, Tagalog, Tamil, Tatar, Telugu, Thai, Tupinambá, Turkish, Uighur, Ukrainian, Umbrian, UpperSorbian, Urdu, Urubú-Kaapor, Vietnamese, Warlpiri, Welsh, Western Armenian, WesternFrisian, Wolof, Xibe, Yakut, Yoruba, YueChinese``.\nPerformance: ``P: 94.99% R: 94.74% F1: 94.86%``.\n'''\nUD_TOK_MMINILMV2L12 = HANLP_URL + 'tok/ud_tok_mMiniLMv2L12_no_space_mul_20220619_091159.zip'\n'''\nmMiniLMv2 (:cite:`wang-etal-2021-minilmv2`) L12xH384 based tokenizer trained on UD 2.10.\nThe following 130 languages are supported: ``Afrikaans, Akkadian, Akuntsu, Albanian, Amharic, AncientGreek (to 1453), Ancient Hebrew, Apurinã, Arabic, Armenian, AssyrianNeo-Aramaic, Bambara, Basque, Beja, Belarusian, Bengali, Bhojpuri, Breton, Bulgarian, Catalan, Cebuano, Central Siberian Yupik, Chinese, Chukot, ChurchSlavic, Coptic, Croatian, Czech, Danish, Dutch, Emerillon, English, Erzya, Estonian, Faroese, Finnish, French, Galician, German, Gothic, Guajajára, Guarani, Hebrew, Hindi, Hittite, Hungarian, Icelandic, Indonesian, Irish, Italian, Japanese, Javanese, K\\'iche\\', Kangri, Karelian, Karo(Brazil), Kazakh, Khunsari, Komi-Permyak, Komi-Zyrian, Korean, Latin, Latvian, Ligurian, LiteraryChinese, Lithuanian, Livvi, LowGerman, Madi, Makuráp, Maltese, Manx, Marathi, MbyáGuaraní, Modern Greek (1453-), Moksha, Mundurukú, Nayini, Neapolitan, Nigerian Pidgin, NorthernKurdish, Northern Sami, Norwegian, OldFrench (842-ca. 1400), OldRussian, Old Turkish, Persian, Polish, Portuguese, Romanian, Russia Buriat, Russian, Sanskrit, ScottishGaelic, Serbian, SkoltSami, Slovak, Slovenian, Soi, South Levantine Arabic, Spanish, Swedish, SwedishSign Language, SwissGerman, Tagalog, Tamil, Tatar, Telugu, Thai, Tupinambá, Turkish, Uighur, Ukrainian, Umbrian, UpperSorbian, Urdu, Urubú-Kaapor, Vietnamese, Warlpiri, Welsh, Western Armenian, WesternFrisian, Wolof, Xibe, Yakut, Yoruba, YueChinese``.\nPerformance: ``P: 95.41% R: 95.25% F1: 95.33%``.\n'''\n\n# Will be filled up during runtime\nALL = {}\n"
  },
  {
    "path": "hanlp/pretrained/word2vec.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-21 18:25\nfrom hanlp_common.constant import HANLP_URL\n\nCONVSEG_W2V_NEWS_TENSITE = HANLP_URL + 'embeddings/convseg_embeddings.zip'\nCONVSEG_W2V_NEWS_TENSITE_WORD_PKU = CONVSEG_W2V_NEWS_TENSITE + '#news_tensite.pku.words.w2v50'\nCONVSEG_W2V_NEWS_TENSITE_WORD_MSR = CONVSEG_W2V_NEWS_TENSITE + '#news_tensite.msr.words.w2v50'\nCONVSEG_W2V_NEWS_TENSITE_CHAR = CONVSEG_W2V_NEWS_TENSITE + '#news_tensite.w2v200'\n\nSEMEVAL16_EMBEDDINGS_CN = HANLP_URL + 'embeddings/semeval16_embeddings.zip'\nSEMEVAL16_EMBEDDINGS_300_NEWS_CN = SEMEVAL16_EMBEDDINGS_CN + '#news.fasttext.300.txt'\nSEMEVAL16_EMBEDDINGS_300_TEXT_CN = SEMEVAL16_EMBEDDINGS_CN + '#text.fasttext.300.txt'\n\nCTB5_FASTTEXT_300_CN = HANLP_URL + 'embeddings/ctb.fasttext.300.txt.zip'\n\nTENCENT_AILAB_EMBEDDING_SMALL_200 = 'https://ai.tencent.com/ailab/nlp/en/data/tencent-ailab-embedding-zh-d200-v0.2.0-s.tar.gz#tencent-ailab-embedding-zh-d200-v0.2.0-s.txt'\n'Chinese word embeddings (:cite:`NIPS2013_9aa42b31`) with small vocabulary size and 200 dimension provided by Tencent AI lab.'\nTENCENT_AILAB_EMBEDDING_LARGE_200 = 'https://ai.tencent.com/ailab/nlp/en/data/tencent-ailab-embedding-zh-d200-v0.2.0.tar.gz#tencent-ailab-embedding-zh-d200-v0.2.0.txt'\n'Chinese word embeddings (:cite:`NIPS2013_9aa42b31`) with large vocabulary size and 200 dimension provided by Tencent AI lab.'\nTENCENT_AILAB_EMBEDDING_SMALL_100 = 'https://ai.tencent.com/ailab/nlp/en/data/tencent-ailab-embedding-zh-d100-v0.2.0-s.tar.gz#tencent-ailab-embedding-zh-d100-v0.2.0-s.txt'\n'Chinese word embeddings (:cite:`NIPS2013_9aa42b31`) with small vocabulary size and 100 dimension provided by Tencent AI lab.'\nTENCENT_AILAB_EMBEDDING_LARGE_100 = 'https://ai.tencent.com/ailab/nlp/en/data/tencent-ailab-embedding-zh-d100-v0.2.0.tar.gz#tencent-ailab-embedding-zh-d100-v0.2.0.txt'\n'Chinese word embeddings (:cite:`NIPS2013_9aa42b31`) with large vocabulary size and 100 dimension provided by Tencent AI lab.'\n\nMERGE_SGNS_BIGRAM_CHAR_300_ZH = 'http://download.hanlp.com/embeddings/extra/merge_sgns_bigram_char300_20220130_214613.txt.zip'\n'Chinese word embeddings trained with context features (word, ngram, character, and more) using Skip-Gram with Negative Sampling (SGNS) (:cite:`li-etal-2018-analogical`).'\n\nRADICAL_CHAR_EMBEDDING_100 = HANLP_URL + 'embeddings/radical_char_vec_20191229_013849.zip#character.vec.txt'\n'Chinese character embedding enhanced with rich radical information (:cite:`he2018dual`).'\n\n_SUBWORD_ENCODING_CWS = 'http://download.hanlp.com/embeddings/extra/subword_encoding_cws_20200524_190636.zip'\nSUBWORD_ENCODING_CWS_ZH_WIKI_BPE_50 = _SUBWORD_ENCODING_CWS + '#zh.wiki.bpe.vs200000.d50.w2v.txt'\nSUBWORD_ENCODING_CWS_GIGAWORD_UNI = _SUBWORD_ENCODING_CWS + '#gigaword_chn.all.a2b.uni.ite50.vec'\nSUBWORD_ENCODING_CWS_GIGAWORD_BI = _SUBWORD_ENCODING_CWS + '#gigaword_chn.all.a2b.bi.ite50.vec'\nSUBWORD_ENCODING_CWS_CTB_GAZETTEER_50 = _SUBWORD_ENCODING_CWS + '#ctb.50d.vec'\n\nALL = {}\n"
  },
  {
    "path": "hanlp/transform/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-29 22:24"
  },
  {
    "path": "hanlp/transform/conll_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-08 15:30\nfrom abc import abstractmethod\nfrom collections import Counter\nfrom typing import Union, Tuple, Iterable, Any, Generator\n\nimport numpy as np\nimport tensorflow as tf\nfrom transformers import PreTrainedTokenizer, PretrainedConfig\n\nfrom hanlp_common.constant import ROOT\nfrom hanlp_common.structure import SerializableDict\nfrom hanlp.common.transform_tf import Transform\nfrom hanlp.common.vocab_tf import VocabTF\nfrom hanlp.components.parsers.alg_tf import tolist, kmeans, randperm, arange\nfrom hanlp.components.parsers.conll import read_conll\nfrom hanlp_common.conll import CoNLLWord, CoNLLUWord, CoNLLSentence\nfrom hanlp.layers.transformers.utils_tf import config_is, adjust_tokens_for_transformers, convert_examples_to_features\nfrom hanlp.utils.log_util import logger\nfrom hanlp.utils.string_util import ispunct\nfrom hanlp_common.util import merge_locals_kwargs\n\n\nclass CoNLLTransform(Transform):\n\n    def __init__(self, config: SerializableDict = None, map_x=True, map_y=True, lower=True, n_buckets=32, min_freq=2,\n                 use_pos=True, **kwargs) -> None:\n        super().__init__(**merge_locals_kwargs(locals(), kwargs))\n        self.form_vocab: VocabTF = None\n        if use_pos:\n            self.cpos_vocab: VocabTF = None\n        self.rel_vocab: VocabTF = None\n        self.puncts: tf.Tensor = None\n\n    @property\n    def use_pos(self):\n        return self.config.get('use_pos', True)\n\n    def x_to_idx(self, x) -> Union[tf.Tensor, Tuple]:\n        form, cpos = x\n        return self.form_vocab.token_to_idx_table.lookup(form), self.cpos_vocab.token_to_idx_table.lookup(cpos)\n\n    def y_to_idx(self, y):\n        head, rel = y\n        return head, self.rel_vocab.token_to_idx_table.lookup(rel)\n\n    def X_to_inputs(self, X: Union[tf.Tensor, Tuple[tf.Tensor]]) -> Iterable:\n        if len(X) == 2:\n            form_batch, cposes_batch = X\n            mask = tf.not_equal(form_batch, 0)\n        elif len(X) == 3:\n            form_batch, cposes_batch, mask = X\n        else:\n            raise ValueError(f'Expect X to be 2 or 3 elements but got {repr(X)}')\n        sents = []\n\n        for form_sent, cposes_sent, length in zip(form_batch, cposes_batch,\n                                                  tf.math.count_nonzero(mask, axis=-1)):\n            forms = tolist(form_sent)[1:length + 1]\n            cposes = tolist(cposes_sent)[1:length + 1]\n            sents.append([(self.form_vocab.idx_to_token[f],\n                           self.cpos_vocab.idx_to_token[c]) for f, c in zip(forms, cposes)])\n\n        return sents\n\n    def lock_vocabs(self):\n        super().lock_vocabs()\n        self.puncts = tf.constant([i for s, i in self.form_vocab.token_to_idx.items()\n                                   if ispunct(s)], dtype=tf.int64)\n\n    def file_to_inputs(self, filepath: str, gold=True):\n        assert gold, 'only support gold file for now'\n        use_pos = self.use_pos\n        conllu = filepath.endswith('.conllu')\n        for sent in read_conll(filepath):\n            for i, cell in enumerate(sent):\n                form = cell[1]\n                cpos = cell[3]\n                head = cell[6]\n                deprel = cell[7]\n                # if conllu:\n                #     deps = cell[8]\n                #     deps = [x.split(':', 1) for x in deps.split('|')]\n                #     heads = [int(x[0]) for x in deps if '_' not in x[0] and '.' not in x[0]]\n                #     rels = [x[1] for x in deps if '_' not in x[0] and '.' not in x[0]]\n                #     if head in heads:\n                #         offset = heads.index(head)\n                #         if not self.rel_vocab or rels[offset] in self.rel_vocab:\n                #             deprel = rels[offset]\n                sent[i] = [form, cpos, head, deprel] if use_pos else [form, head, deprel]\n            yield sent\n\n    @property\n    def bos(self):\n        if self.form_vocab.idx_to_token is None:\n            return ROOT\n        return self.form_vocab.idx_to_token[2]\n\n    def input_is_single_sample(self, input: Any) -> bool:\n        if self.use_pos:\n            return isinstance(input[0][0], str) if len(input[0]) else False\n        else:\n            return isinstance(input[0], str) if len(input[0]) else False\n\n    @abstractmethod\n    def batched_inputs_to_batches(self, corpus, indices, shuffle):\n        pass\n\n    def len_of_sent(self, sent):\n        return 1 + len(sent)  # take ROOT into account\n\n    def samples_to_dataset(self, samples: Generator, map_x=None, map_y=None, batch_size=5000, shuffle=None, repeat=None,\n                           drop_remainder=False, prefetch=1, cache=True) -> tf.data.Dataset:\n        if shuffle:\n            def generator():\n                # custom bucketing, load corpus into memory\n                corpus = list(x for x in (samples() if callable(samples) else samples))\n                lengths = [self.len_of_sent(i) for i in corpus]\n                if len(corpus) < 32:\n                    n_buckets = 1\n                else:\n                    n_buckets = min(self.config.n_buckets, len(corpus))\n                buckets = dict(zip(*kmeans(lengths, n_buckets)))\n                sizes, buckets = zip(*[\n                    (size, bucket) for size, bucket in buckets.items()\n                ])\n                # the number of chunks in each bucket, which is clipped by\n                # range [1, len(bucket)]\n                chunks = [min(len(bucket), max(round(size * len(bucket) / batch_size), 1)) for size, bucket in\n                          zip(sizes, buckets)]\n                range_fn = randperm if shuffle else arange\n                max_samples_per_batch = self.config.get('max_samples_per_batch', None)\n                for i in tolist(range_fn(len(buckets))):\n                    split_sizes = [(len(buckets[i]) - j - 1) // chunks[i] + 1\n                                   for j in range(chunks[i])]  # how many sentences in each batch\n                    for batch_indices in tf.split(range_fn(len(buckets[i])), split_sizes):\n                        indices = [buckets[i][j] for j in tolist(batch_indices)]\n                        if max_samples_per_batch:\n                            for j in range(0, len(indices), max_samples_per_batch):\n                                yield from self.batched_inputs_to_batches(corpus, indices[j:j + max_samples_per_batch],\n                                                                          shuffle)\n                        else:\n                            yield from self.batched_inputs_to_batches(corpus, indices, shuffle)\n\n        else:\n            def generator():\n                # custom bucketing, load corpus into memory\n                corpus = list(x for x in (samples() if callable(samples) else samples))\n                n_tokens = 0\n                batch = []\n                for idx, sent in enumerate(corpus):\n                    sent_len = self.len_of_sent(sent)\n                    if n_tokens + sent_len > batch_size and batch:\n                        yield from self.batched_inputs_to_batches(corpus, batch, shuffle)\n                        n_tokens = 0\n                        batch = []\n                    n_tokens += sent_len\n                    batch.append(idx)\n                if batch:\n                    yield from self.batched_inputs_to_batches(corpus, batch, shuffle)\n\n        # next(generator())\n        return Transform.samples_to_dataset(self, generator, False, False, 0, False, repeat, drop_remainder, prefetch,\n                                            cache)\n\n\nclass CoNLL_DEP_Transform(CoNLLTransform):\n\n    def __init__(self, config: SerializableDict = None, map_x=True, map_y=True, lower=True, n_buckets=32,\n                 min_freq=2, **kwargs) -> None:\n        super().__init__(config, map_x, map_y, lower, n_buckets, min_freq, **kwargs)\n\n    def batched_inputs_to_batches(self, corpus, indices, shuffle):\n        \"\"\"Convert batched inputs to batches of samples\n\n        Args:\n          corpus(list): A list of inputs\n          indices(list): A list of indices, each list belongs to a batch\n          shuffle:\n\n        Returns:\n\n\n        \"\"\"\n        raw_batch = [[], [], [], []]\n        for idx in indices:\n            for b in raw_batch:\n                b.append([])\n            for cells in corpus[idx]:\n                for b, c, v in zip(raw_batch, cells,\n                                   [self.form_vocab, self.cpos_vocab, None, self.rel_vocab]):\n                    b[-1].append(v.get_idx_without_add(c) if v else c)\n        batch = []\n        for b, v in zip(raw_batch, [self.form_vocab, self.cpos_vocab, None, self.rel_vocab]):\n            b = tf.keras.preprocessing.sequence.pad_sequences(b, padding='post',\n                                                              value=v.safe_pad_token_idx if v else 0,\n                                                              dtype='int64')\n            batch.append(b)\n        assert len(batch) == 4\n        yield (batch[0], batch[1]), (batch[2], batch[3])\n\n    def create_types_shapes_values(self) -> Tuple[Tuple, Tuple, Tuple]:\n        types = (tf.int64, tf.int64), (tf.int64, tf.int64)\n        shapes = ([None, None], [None, None]), ([None, None], [None, None])\n        values = (self.form_vocab.safe_pad_token_idx, self.cpos_vocab.safe_pad_token_idx), (\n            0, self.rel_vocab.safe_pad_token_idx)\n        return types, shapes, values\n\n    def inputs_to_samples(self, inputs, gold=False):\n        token_mapping: dict = self.config.get('token_mapping', None)\n        use_pos = self.config.get('use_pos', True)\n        for sent in inputs:\n            sample = []\n            for i, cell in enumerate(sent):\n                if isinstance(cell, tuple):\n                    cell = list(cell)\n                elif isinstance(cell, str):\n                    cell = [cell]\n                if token_mapping:\n                    cell[0] = token_mapping.get(cell[0], cell[0])\n                if self.config['lower']:\n                    cell[0] = cell[0].lower()\n                if not gold:\n                    cell += [0, self.rel_vocab.safe_pad_token]\n                sample.append(cell)\n            # insert root word with arbitrary fields, anyway it will be masked\n            # form, cpos, head, deprel = sample[0]\n            sample.insert(0, [self.bos, self.bos, 0, self.bos] if use_pos else [self.bos, 0, self.bos])\n            yield sample\n\n    def XY_to_inputs_outputs(self, X: Union[tf.Tensor, Tuple[tf.Tensor]], Y: Union[tf.Tensor, Tuple[tf.Tensor]],\n                             gold=False, inputs=None, conll=True, arc_scores=None, rel_scores=None) -> Iterable:\n        (words, feats, mask), (arc_preds, rel_preds) = X, Y\n        if inputs is None:\n            inputs = self.X_to_inputs(X)\n        ys = self.Y_to_outputs((arc_preds, rel_preds, mask), inputs=inputs)\n        sents = []\n        for x, y in zip(inputs, ys):\n            sent = CoNLLSentence()\n            for idx, (cell, (head, deprel)) in enumerate(zip(x, y)):\n                if self.use_pos and not self.config.get('joint_pos', None):\n                    form, cpos = cell\n                else:\n                    form, cpos = cell, None\n                if conll:\n                    sent.append(\n                        CoNLLWord(id=idx + 1, form=form, cpos=cpos, head=head, deprel=deprel) if conll == '.conll'\n                        else CoNLLUWord(id=idx + 1, form=form, upos=cpos, head=head, deprel=deprel))\n                else:\n                    sent.append([head, deprel])\n            sents.append(sent)\n        return sents\n\n    def fit(self, trn_path: str, **kwargs) -> int:\n        use_pos = self.config.use_pos\n        self.form_vocab = VocabTF()\n        self.form_vocab.add(ROOT)  # make root the 2ed elements while 0th is pad, 1st is unk\n        if self.use_pos:\n            self.cpos_vocab = VocabTF(pad_token=None, unk_token=None)\n        self.rel_vocab = VocabTF(pad_token=None, unk_token=None)\n        num_samples = 0\n        counter = Counter()\n        for sent in self.file_to_samples(trn_path, gold=True):\n            num_samples += 1\n            for idx, cell in enumerate(sent):\n                if use_pos:\n                    form, cpos, head, deprel = cell\n                else:\n                    form, head, deprel = cell\n                if idx == 0:\n                    root = form\n                else:\n                    counter[form] += 1\n                if use_pos:\n                    self.cpos_vocab.add(cpos)\n                self.rel_vocab.add(deprel)\n\n        for token in [token for token, freq in counter.items() if freq >= self.config.min_freq]:\n            self.form_vocab.add(token)\n        return num_samples\n\n    @property\n    def root_rel_idx(self):\n        root_rel_idx = self.config.get('root_rel_idx', None)\n        if root_rel_idx is None:\n            for idx, rel in enumerate(self.rel_vocab.idx_to_token):\n                if 'root' in rel.lower() and rel != self.bos:\n                    self.config['root_rel_idx'] = root_rel_idx = idx\n                    break\n        return root_rel_idx\n\n    def Y_to_outputs(self, Y: Union[tf.Tensor, Tuple[tf.Tensor]], gold=False, inputs=None, X=None) -> Iterable:\n        arc_preds, rel_preds, mask = Y\n        sents = []\n\n        for arc_sent, rel_sent, length in zip(arc_preds, rel_preds,\n                                              tf.math.count_nonzero(mask, axis=-1)):\n            arcs = tolist(arc_sent)[1:length + 1]\n            rels = tolist(rel_sent)[1:length + 1]\n            sents.append([(a, self.rel_vocab.idx_to_token[r]) for a, r in zip(arcs, rels)])\n\n        return sents\n\n\nclass CoNLL_Transformer_Transform(CoNLL_DEP_Transform):\n\n    def __init__(self, config: SerializableDict = None, map_x=True, map_y=True,\n                 lower=True, n_buckets=32, min_freq=0, max_seq_length=256, use_pos=False,\n                 mask_p=None, graph=False, topk=None,\n                 **kwargs) -> None:\n        super().__init__(**merge_locals_kwargs(locals(), kwargs))\n        self.tokenizer: PreTrainedTokenizer = None\n        self.transformer_config: PretrainedConfig = None\n        if graph:\n            self.orphan_relation = ROOT\n\n    def lock_vocabs(self):\n        super().lock_vocabs()\n        if self.graph:\n            CoNLL_SDP_Transform._find_orphan_relation(self)\n\n    def fit(self, trn_path: str, **kwargs) -> int:\n        if self.config.get('joint_pos', None):\n            self.config.use_pos = True\n        if self.graph:\n            # noinspection PyCallByClass\n            num = CoNLL_SDP_Transform.fit(self, trn_path, **kwargs)\n        else:\n            num = super().fit(trn_path, **kwargs)\n        if self.config.get('topk', None):\n            counter = Counter()\n            for sent in self.file_to_samples(trn_path, gold=True):\n                for idx, cell in enumerate(sent):\n                    form, head, deprel = cell\n                    counter[form] += 1\n            self.topk_vocab = VocabTF()\n            for k, v in counter.most_common(self.config.topk):\n                self.topk_vocab.add(k)\n        return num\n\n    def inputs_to_samples(self, inputs, gold=False):\n        if self.graph:\n            yield from CoNLL_SDP_Transform.inputs_to_samples(self, inputs, gold)\n        else:\n            yield from super().inputs_to_samples(inputs, gold)\n\n    def file_to_inputs(self, filepath: str, gold=True):\n        if self.graph:\n            yield from CoNLL_SDP_Transform.file_to_inputs(self, filepath, gold)\n        else:\n            yield from super().file_to_inputs(filepath, gold)\n\n    @property\n    def mask_p(self) -> float:\n        return self.config.get('mask_p', None)\n\n    @property\n    def graph(self):\n        return self.config.get('graph', None)\n\n    def create_types_shapes_values(self) -> Tuple[Tuple, Tuple, Tuple]:\n        mask_p = self.mask_p\n        types = (tf.int64, (tf.int64, tf.int64, tf.int64)), (tf.bool if self.graph else tf.int64, tf.int64, tf.int64) if mask_p else (\n            tf.bool if self.graph else tf.int64, tf.int64)\n        if self.graph:\n            shapes = ([None, None], ([None, None], [None, None], [None, None])), (\n                [None, None, None], [None, None, None], [None, None]) if mask_p else (\n                [None, None, None], [None, None, None])\n        else:\n            shapes = ([None, None], ([None, None], [None, None], [None, None])), (\n                [None, None], [None, None], [None, None]) if mask_p else ([None, None], [None, None])\n\n        values = (self.form_vocab.safe_pad_token_idx, (0, 0, 0)), \\\n                 (0, self.rel_vocab.safe_pad_token_idx, 0) if mask_p else (0, self.rel_vocab.safe_pad_token_idx)\n        types_shapes_values = types, shapes, values\n        if self.use_pos:\n            types_shapes_values = [((shapes[0][0], shapes[0][1] + (shapes[0][0],)), shapes[1]) for shapes in\n                                   types_shapes_values]\n        return types_shapes_values\n\n    def X_to_inputs(self, X: Union[tf.Tensor, Tuple[tf.Tensor]]) -> Iterable:\n        form_batch, feat, prefix_mask = X\n        sents = []\n\n        for form_sent, length in zip(form_batch, tf.math.count_nonzero(prefix_mask, axis=-1)):\n            forms = tolist(form_sent)[1:length + 1]\n            sents.append([self.form_vocab.idx_to_token[f] for f in forms])\n\n        return sents\n\n    def batched_inputs_to_batches(self, corpus, indices, shuffle):\n        use_pos = self.use_pos\n        if use_pos:\n            raw_batch = [[], [], [], []]\n        else:\n            raw_batch = [[], [], []]\n        if self.graph:\n            max_len = len(max([corpus[i] for i in indices], key=len))\n            for idx in indices:\n                arc = np.zeros((max_len, max_len), dtype=np.bool)\n                rel = np.zeros((max_len, max_len), dtype=np.int64)\n                for b in raw_batch[:2 if use_pos else 1]:\n                    b.append([])\n                for m, cells in enumerate(corpus[idx]):\n                    if use_pos:\n                        for b, c, v in zip(raw_batch, cells, [None, self.cpos_vocab]):\n                            b[-1].append(v.get_idx_without_add(c) if v else c)\n                    else:\n                        for b, c, v in zip(raw_batch, cells, [None]):\n                            b[-1].append(c)\n                    for n, r in zip(cells[-2], cells[-1]):\n                        arc[m, n] = True\n                        rid = self.rel_vocab.get_idx_without_add(r)\n                        if rid is None:\n                            logger.warning(f'Relation OOV: {r} not exists in train')\n                            continue\n                        rel[m, n] = rid\n                raw_batch[-2].append(arc)\n                raw_batch[-1].append(rel)\n        else:\n            for idx in indices:\n                for s in raw_batch:\n                    s.append([])\n                for cells in corpus[idx]:\n                    if use_pos:\n                        for s, c, v in zip(raw_batch, cells, [None, self.cpos_vocab, None, self.rel_vocab]):\n                            s[-1].append(v.get_idx_without_add(c) if v else c)\n                    else:\n                        for s, c, v in zip(raw_batch, cells, [None, None, self.rel_vocab]):\n                            s[-1].append(v.get_idx_without_add(c) if v else c)\n\n        # Transformer tokenizing\n        config = self.transformer_config\n        tokenizer = self.tokenizer\n        xlnet = config_is(config, 'xlnet')\n        roberta = config_is(config, 'roberta')\n        pad_token = tokenizer.pad_token\n        pad_token_id = tokenizer.convert_tokens_to_ids([pad_token])[0]\n        cls_token = tokenizer.cls_token\n        sep_token = tokenizer.sep_token\n        max_seq_length = self.config.max_seq_length\n        batch_forms = []\n        batch_input_ids = []\n        batch_input_mask = []\n        batch_prefix_offset = []\n        mask_p = self.mask_p\n        if mask_p:\n            batch_masked_offsets = []\n            mask_token_id = tokenizer.mask_token_id\n        for sent_idx, sent in enumerate(raw_batch[0]):\n            batch_forms.append([self.form_vocab.get_idx_without_add(token) for token in sent])\n            sent = adjust_tokens_for_transformers(sent)\n            sent = sent[1:]  # remove <root> use [CLS] instead\n            pad_label_idx = self.form_vocab.pad_idx\n            input_ids, input_mask, segment_ids, prefix_mask = \\\n                convert_examples_to_features(sent,\n                                             max_seq_length,\n                                             tokenizer,\n                                             cls_token_at_end=xlnet,\n                                             # xlnet has a cls token at the end\n                                             cls_token=cls_token,\n                                             cls_token_segment_id=2 if xlnet else 0,\n                                             sep_token=sep_token,\n                                             sep_token_extra=roberta,\n                                             # roberta uses an extra separator b/w pairs of sentences, cf. github.com/pytorch/fairseq/commit/1684e166e3da03f5b600dbb7855cb98ddfcd0805\n                                             pad_on_left=xlnet,\n                                             # pad on the left for xlnet\n                                             pad_token_id=pad_token_id,\n                                             pad_token_segment_id=4 if xlnet else 0,\n                                             pad_token_label_id=pad_label_idx,\n                                             do_padding=False)\n            num_masks = sum(prefix_mask)\n            # assert len(sent) == num_masks  # each token has a True subtoken\n            if num_masks < len(sent):  # long sent gets truncated, +1 for root\n                batch_forms[-1] = batch_forms[-1][:num_masks + 1]  # form\n                raw_batch[-1][sent_idx] = raw_batch[-1][sent_idx][:num_masks + 1]  # head\n                raw_batch[-2][sent_idx] = raw_batch[-2][sent_idx][:num_masks + 1]  # rel\n                raw_batch[-3][sent_idx] = raw_batch[-3][sent_idx][:num_masks + 1]  # pos\n            prefix_mask[0] = True  # <root> is now [CLS]\n            prefix_offset = [idx for idx, m in enumerate(prefix_mask) if m]\n            batch_input_ids.append(input_ids)\n            batch_input_mask.append(input_mask)\n            batch_prefix_offset.append(prefix_offset)\n            if mask_p:\n                if shuffle:\n                    size = int(np.ceil(mask_p * len(prefix_offset[1:])))  # never mask [CLS]\n                    mask_offsets = np.random.choice(np.arange(1, len(prefix_offset)), size, replace=False)\n                    for offset in sorted(mask_offsets):\n                        assert 0 < offset < len(input_ids)\n                        # mask_word = raw_batch[0][sent_idx][offset]\n                        # mask_prefix = tokenizer.convert_ids_to_tokens([input_ids[prefix_offset[offset]]])[0]\n                        # assert mask_word.startswith(mask_prefix) or mask_prefix.startswith(\n                        #     mask_word) or mask_prefix == \"'\", \\\n                        #     f'word {mask_word} prefix {mask_prefix} not match'  # could vs couldn\n                        # mask_offsets.append(input_ids[offset]) # subword token\n                        # mask_offsets.append(offset)  # form token\n                        input_ids[prefix_offset[offset]] = mask_token_id  # mask prefix\n                        # whole word masking, mask the rest of the word\n                        for i in range(prefix_offset[offset] + 1, len(input_ids) - 1):\n                            if prefix_mask[i]:\n                                break\n                            input_ids[i] = mask_token_id\n\n                    batch_masked_offsets.append(sorted(mask_offsets))\n                else:\n                    batch_masked_offsets.append([0])  # No masking in prediction\n\n        batch_forms = tf.keras.preprocessing.sequence.pad_sequences(batch_forms, padding='post',\n                                                                    value=self.form_vocab.safe_pad_token_idx,\n                                                                    dtype='int64')\n        batch_input_ids = tf.keras.preprocessing.sequence.pad_sequences(batch_input_ids, padding='post',\n                                                                        value=pad_token_id,\n                                                                        dtype='int64')\n        batch_input_mask = tf.keras.preprocessing.sequence.pad_sequences(batch_input_mask, padding='post',\n                                                                         value=0,\n                                                                         dtype='int64')\n        batch_prefix_offset = tf.keras.preprocessing.sequence.pad_sequences(batch_prefix_offset, padding='post',\n                                                                            value=0,\n                                                                            dtype='int64')\n        batch_heads = tf.keras.preprocessing.sequence.pad_sequences(raw_batch[-2], padding='post',\n                                                                    value=0,\n                                                                    dtype='int64')\n        batch_rels = tf.keras.preprocessing.sequence.pad_sequences(raw_batch[-1], padding='post',\n                                                                   value=self.rel_vocab.safe_pad_token_idx,\n                                                                   dtype='int64')\n        if mask_p:\n            batch_masked_offsets = tf.keras.preprocessing.sequence.pad_sequences(batch_masked_offsets, padding='post',\n                                                                                 value=pad_token_id,\n                                                                                 dtype='int64')\n        feats = (tf.constant(batch_input_ids, dtype='int64'), tf.constant(batch_input_mask, dtype='int64'),\n                 tf.constant(batch_prefix_offset))\n        if use_pos:\n            batch_pos = tf.keras.preprocessing.sequence.pad_sequences(raw_batch[1], padding='post',\n                                                                      value=self.cpos_vocab.safe_pad_token_idx,\n                                                                      dtype='int64')\n            feats += (batch_pos,)\n        yield (batch_forms, feats), \\\n              (batch_heads, batch_rels, batch_masked_offsets) if mask_p else (batch_heads, batch_rels)\n\n    def len_of_sent(self, sent):\n        # Transformer tokenizing\n        config = self.transformer_config\n        tokenizer = self.tokenizer\n        xlnet = config_is(config, 'xlnet')\n        roberta = config_is(config, 'roberta')\n        pad_token = tokenizer.pad_token\n        pad_token_id = tokenizer.convert_tokens_to_ids([pad_token])[0]\n        cls_token = tokenizer.cls_token\n        sep_token = tokenizer.sep_token\n        max_seq_length = self.config.max_seq_length\n        sent = sent[1:]  # remove <root> use [CLS] instead\n        pad_label_idx = self.form_vocab.pad_idx\n        sent = [x[0] for x in sent]\n        sent = adjust_tokens_for_transformers(sent)\n        input_ids, input_mask, segment_ids, prefix_mask = \\\n            convert_examples_to_features(sent,\n                                         max_seq_length,\n                                         tokenizer,\n                                         cls_token_at_end=xlnet,\n                                         # xlnet has a cls token at the end\n                                         cls_token=cls_token,\n                                         cls_token_segment_id=2 if xlnet else 0,\n                                         sep_token=sep_token,\n                                         sep_token_extra=roberta,\n                                         # roberta uses an extra separator b/w pairs of sentences, cf. github.com/pytorch/fairseq/commit/1684e166e3da03f5b600dbb7855cb98ddfcd0805\n                                         pad_on_left=xlnet,\n                                         # pad on the left for xlnet\n                                         pad_token_id=pad_token_id,\n                                         pad_token_segment_id=4 if xlnet else 0,\n                                         pad_token_label_id=pad_label_idx,\n                                         do_padding=False)\n        return len(input_ids)\n\n    def samples_to_dataset(self, samples: Generator, map_x=None, map_y=None, batch_size=5000, shuffle=None, repeat=None,\n                           drop_remainder=False, prefetch=1, cache=True) -> tf.data.Dataset:\n        if shuffle:\n            return CoNLL_DEP_Transform.samples_to_dataset(self, samples, map_x, map_y, batch_size, shuffle, repeat,\n                                                          drop_remainder, prefetch, cache)\n\n        def generator():\n            # custom bucketing, load corpus into memory\n            corpus = list(x for x in (samples() if callable(samples) else samples))\n            n_tokens = 0\n            batch = []\n            for idx, sent in enumerate(corpus):\n                sent_len = self.len_of_sent(sent)\n                if n_tokens + sent_len > batch_size and batch:\n                    yield from self.batched_inputs_to_batches(corpus, batch, shuffle)\n                    n_tokens = 0\n                    batch = []\n                n_tokens += sent_len\n                batch.append(idx)\n            if batch:\n                yield from self.batched_inputs_to_batches(corpus, batch, shuffle)\n\n        # debug for transformer\n        # next(generator())\n        return Transform.samples_to_dataset(self, generator, False, False, 0, False, repeat, drop_remainder, prefetch,\n                                            cache)\n\n    def Y_to_outputs(self, Y: Union[tf.Tensor, Tuple[tf.Tensor]], gold=False, inputs=None, X=None) -> Iterable:\n        if self.graph:\n            ys = CoNLL_SDP_Transform.Y_to_outputs(self, Y, gold, inputs, X)\n            ys = [[([t[0] for t in l], [t[1] for t in l]) for l in y] for y in ys]\n            return ys\n        return super().Y_to_outputs(Y, gold, inputs, X)\n\n\nclass CoNLL_SDP_Transform(CoNLLTransform):\n\n    def __init__(self, config: SerializableDict = None, map_x=True, map_y=True, lower=True, n_buckets=32, min_freq=2,\n                 use_pos=True, **kwargs) -> None:\n        super().__init__(config, map_x, map_y, lower, n_buckets, min_freq, use_pos, **kwargs)\n        self.orphan_relation = ROOT\n\n    def lock_vocabs(self):\n        super().lock_vocabs()\n        # heuristic to find the orphan relation\n        self._find_orphan_relation()\n\n    def _find_orphan_relation(self):\n        for rel in self.rel_vocab.idx_to_token:\n            if 'root' in rel.lower():\n                self.orphan_relation = rel\n                break\n\n    def file_to_inputs(self, filepath: str, gold=True):\n        assert gold, 'only support gold file for now'\n        use_pos = self.use_pos\n        conllu = filepath.endswith('.conllu')\n        enhanced_only = self.config.get('enhanced_only', None)\n        for i, sent in enumerate(read_conll(filepath)):\n            parsed_sent = []\n            if conllu:\n                for cell in sent:\n                    ID = cell[0]\n                    form = cell[1]\n                    cpos = cell[3]\n                    head = cell[6]\n                    deprel = cell[7]\n                    deps = cell[8]\n                    deps = [x.split(':', 1) for x in deps.split('|')]\n                    heads = [int(x[0]) for x in deps if x[0].isdigit()]\n                    rels = [x[1] for x in deps if x[0].isdigit()]\n                    if enhanced_only:\n                        if head in heads:\n                            offset = heads.index(head)\n                            heads.pop(offset)\n                            rels.pop(offset)\n                    else:\n                        if head not in heads:\n                            heads.append(head)\n                            rels.append(deprel)\n                    parsed_sent.append([form, cpos, heads, rels] if use_pos else [form, heads, rels])\n            else:\n                prev_cells = None\n                heads = []\n                rels = []\n                for j, cell in enumerate(sent):\n                    ID = cell[0]\n                    form = cell[1]\n                    cpos = cell[3]\n                    head = cell[6]\n                    deprel = cell[7]\n                    if prev_cells and ID != prev_cells[0]:  # found end of token\n                        parsed_sent.append(\n                            [prev_cells[1], prev_cells[2], heads, rels] if use_pos else [prev_cells[1], heads, rels])\n                        heads = []\n                        rels = []\n                    heads.append(head)\n                    rels.append(deprel)\n                    prev_cells = [ID, form, cpos, head, deprel] if use_pos else [ID, form, head, deprel]\n                parsed_sent.append(\n                    [prev_cells[1], prev_cells[2], heads, rels] if use_pos else [prev_cells[1], heads, rels])\n            yield parsed_sent\n\n    def fit(self, trn_path: str, **kwargs) -> int:\n        self.form_vocab = VocabTF()\n        self.form_vocab.add(ROOT)  # make root the 2ed elements while 0th is pad, 1st is unk\n        if self.use_pos:\n            self.cpos_vocab = VocabTF(pad_token=None, unk_token=None)\n        self.rel_vocab = VocabTF(pad_token=None, unk_token=None)\n        num_samples = 0\n        counter = Counter()\n        for sent in self.file_to_samples(trn_path, gold=True):\n            num_samples += 1\n            for idx, cell in enumerate(sent):\n                if len(cell) == 4:\n                    form, cpos, head, deprel = cell\n                elif len(cell) == 3:\n                    if self.use_pos:\n                        form, cpos = cell[0]\n                    else:\n                        form = cell[0]\n                    head, deprel = cell[1:]\n                else:\n                    raise ValueError('Unknown data arrangement')\n                if idx == 0:\n                    root = form\n                else:\n                    counter[form] += 1\n                if self.use_pos:\n                    self.cpos_vocab.add(cpos)\n                self.rel_vocab.update(deprel)\n\n        for token in [token for token, freq in counter.items() if freq >= self.config.min_freq]:\n            self.form_vocab.add(token)\n        return num_samples\n\n    def inputs_to_samples(self, inputs, gold=False):\n        use_pos = self.use_pos\n        for sent in inputs:\n            sample = []\n            for i, cell in enumerate(sent):\n                if isinstance(cell, tuple):\n                    cell = list(cell)\n                elif isinstance(cell, str):\n                    cell = [cell]\n                if self.config['lower']:\n                    cell[0] = cell[0].lower()\n                if not gold:\n                    cell += [[0], [self.rel_vocab.safe_pad_token]]\n                sample.append(cell)\n            # insert root word with arbitrary fields, anyway it will be masked\n            if use_pos:\n                form, cpos, head, deprel = sample[0]\n                sample.insert(0, [self.bos, self.bos, [0], deprel])\n            else:\n                form, head, deprel = sample[0]\n                sample.insert(0, [self.bos, [0], deprel])\n            yield sample\n\n    def batched_inputs_to_batches(self, corpus, indices, shuffle):\n        use_pos = self.use_pos\n        raw_batch = [[], [], [], []] if use_pos else [[], [], []]\n        max_len = len(max([corpus[i] for i in indices], key=len))\n        for idx in indices:\n            arc = np.zeros((max_len, max_len), dtype=bool)\n            rel = np.zeros((max_len, max_len), dtype=np.int64)\n            for b in raw_batch[:2]:\n                b.append([])\n            for m, cells in enumerate(corpus[idx]):\n                if use_pos:\n                    for b, c, v in zip(raw_batch, cells,\n                                       [self.form_vocab, self.cpos_vocab]):\n                        b[-1].append(v.get_idx_without_add(c))\n                else:\n                    for b, c, v in zip(raw_batch, cells,\n                                       [self.form_vocab]):\n                        b[-1].append(v.get_idx_without_add(c))\n                for n, r in zip(cells[-2], cells[-1]):\n                    arc[m, n] = True\n                    rid = self.rel_vocab.get_idx_without_add(r)\n                    if rid is None:\n                        logger.warning(f'Relation OOV: {r} not exists in train')\n                        continue\n                    rel[m, n] = rid\n            raw_batch[-2].append(arc)\n            raw_batch[-1].append(rel)\n        batch = []\n        for b, v in zip(raw_batch, [self.form_vocab, self.cpos_vocab]):\n            b = tf.keras.preprocessing.sequence.pad_sequences(b, padding='post',\n                                                              value=v.safe_pad_token_idx,\n                                                              dtype='int64')\n            batch.append(b)\n        batch += raw_batch[2:]\n        assert len(batch) == 4\n        yield (batch[0], batch[1]), (batch[2], batch[3])\n\n    def create_types_shapes_values(self) -> Tuple[Tuple, Tuple, Tuple]:\n        types = (tf.int64, tf.int64), (tf.bool, tf.int64)\n        shapes = ([None, None], [None, None]), ([None, None, None], [None, None, None])\n        values = (self.form_vocab.safe_pad_token_idx, self.cpos_vocab.safe_pad_token_idx), (\n            False, self.rel_vocab.safe_pad_token_idx)\n        return types, shapes, values\n\n    def Y_to_outputs(self, Y: Union[tf.Tensor, Tuple[tf.Tensor]], gold=False, inputs=None, X=None) -> Iterable:\n        arc_preds, rel_preds, mask = Y\n        sents = []\n\n        for arc_sent, rel_sent, length in zip(arc_preds, rel_preds,\n                                              tf.math.count_nonzero(mask, axis=-1)):\n            sent = []\n            for arc, rel in zip(tolist(arc_sent[1:, 1:]), tolist(rel_sent[1:, 1:])):\n                ar = []\n                for idx, (a, r) in enumerate(zip(arc, rel)):\n                    if a:\n                        ar.append((idx + 1, self.rel_vocab.idx_to_token[r]))\n                if not ar:\n                    # orphan\n                    ar.append((0, self.orphan_relation))\n                sent.append(ar)\n            sents.append(sent)\n\n        return sents\n\n    def XY_to_inputs_outputs(self, X: Union[tf.Tensor, Tuple[tf.Tensor]], Y: Union[tf.Tensor, Tuple[tf.Tensor]],\n                             gold=False, inputs=None, conll=True) -> Iterable:\n        (words, feats, mask), (arc_preds, rel_preds) = X, Y\n        xs = inputs\n        ys = self.Y_to_outputs((arc_preds, rel_preds, mask))\n        sents = []\n        for x, y in zip(xs, ys):\n            sent = CoNLLSentence()\n            for idx, ((form, cpos), pred) in enumerate(zip(x, y)):\n                head = [p[0] for p in pred]\n                deprel = [p[1] for p in pred]\n                if conll:\n                    sent.append(CoNLLWord(id=idx + 1, form=form, cpos=cpos, head=head, deprel=deprel))\n                else:\n                    sent.append([head, deprel])\n            sents.append(sent)\n        return sents\n"
  },
  {
    "path": "hanlp/transform/glue_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-08 16:34\nfrom hanlp_common.structure import SerializableDict\nfrom hanlp.datasets.glu.glue import STANFORD_SENTIMENT_TREEBANK_2_TRAIN, MICROSOFT_RESEARCH_PARAPHRASE_CORPUS_DEV\nfrom hanlp.transform.table_tf import TableTransform\n\n\nclass StanfordSentimentTreebank2Transorm(TableTransform):\n    pass\n\n\nclass MicrosoftResearchParaphraseCorpus(TableTransform):\n\n    def __init__(self, config: SerializableDict = None, map_x=False, map_y=True, x_columns=(3, 4),\n                 y_column=0, skip_header=True, delimiter='auto', **kwargs) -> None:\n        super().__init__(config, map_x, map_y, x_columns, y_column, skip_header, delimiter, **kwargs)\n\n\ndef main():\n    # _test_sst2()\n    _test_mrpc()\n\n\ndef _test_sst2():\n    transform = StanfordSentimentTreebank2Transorm()\n    transform.fit(STANFORD_SENTIMENT_TREEBANK_2_TRAIN)\n    transform.lock_vocabs()\n    transform.label_vocab.summary()\n    transform.build_config()\n    dataset = transform.file_to_dataset(STANFORD_SENTIMENT_TREEBANK_2_TRAIN)\n    for batch in dataset.take(1):\n        print(batch)\n\n\ndef _test_mrpc():\n    transform = MicrosoftResearchParaphraseCorpus()\n    transform.fit(MICROSOFT_RESEARCH_PARAPHRASE_CORPUS_DEV)\n    transform.lock_vocabs()\n    transform.label_vocab.summary()\n    transform.build_config()\n    dataset = transform.file_to_dataset(MICROSOFT_RESEARCH_PARAPHRASE_CORPUS_DEV)\n    for batch in dataset.take(1):\n        print(batch)"
  },
  {
    "path": "hanlp/transform/table_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-11-10 21:00\nfrom abc import ABC\nfrom typing import Tuple, Union\nimport numpy as np\nimport tensorflow as tf\n\nfrom hanlp_common.structure import SerializableDict\nfrom hanlp.common.transform_tf import Transform\nfrom hanlp_common.constant import PAD\nfrom hanlp.common.vocab_tf import create_label_vocab\nfrom hanlp.utils.io_util import read_cells\nfrom hanlp.utils.log_util import logger\n\n\nclass TableTransform(Transform, ABC):\n    def __init__(self, config: SerializableDict = None, map_x=False, map_y=True, x_columns=None,\n                 y_column=-1, multi_label=False,\n                 skip_header=True, delimiter='auto', **kwargs) -> None:\n        super().__init__(config, map_x, map_y, x_columns=x_columns, y_column=y_column, multi_label=multi_label,\n                         skip_header=skip_header,\n                         delimiter=delimiter, **kwargs)\n        self.label_vocab = create_label_vocab()\n\n    def file_to_inputs(self, filepath: str, gold=True):\n        x_columns = self.config.x_columns\n        y_column = self.config.y_column\n        num_features = self.config.get('num_features', None)\n        for cells in read_cells(filepath, skip_header=self.config.skip_header, delimiter=self.config.delimiter):\n            #multi-label: Dataset in .tsv format: x_columns: at most 2 columns being a sentence pair while in most\n            # cases just one column being the doc content. y_column being the single label, which shall be modified\n            # to load a list of labels.\n            if x_columns:\n                inputs = tuple(c for i, c in enumerate(cells) if i in x_columns), cells[y_column]\n            else:\n                if y_column != -1:\n                    cells[-1], cells[y_column] = cells[y_column], cells[-1]\n                inputs = tuple(cells[:-1]), cells[-1]\n            if num_features is None:\n                num_features = len(inputs[0])\n                self.config.num_features = num_features\n            # multi-label support\n            if self.config.get('multi_label', None):\n                assert type(inputs[1]) is str, 'Y value has to be string'\n                if inputs[1][0] == '[':\n                    # multi-label is in literal form of a list\n                    labels = eval(inputs[1])\n                else:\n                    labels = inputs[1].strip().split(',')\n                inputs = inputs[0], labels\n            else:\n                assert num_features == len(inputs[0]), f'Numbers of columns {num_features} ' \\\n                                                       f'inconsistent with current {len(inputs[0])}'\n            yield inputs\n\n    def inputs_to_samples(self, inputs, gold=False):\n        pad = self.label_vocab.safe_pad_token\n        for cells in inputs:\n            if gold:\n                yield cells\n            else:\n                yield cells, pad\n\n    def y_to_idx(self, y) -> tf.Tensor:\n        return self.label_vocab.lookup(y)\n\n    def fit(self, trn_path: str, **kwargs):\n        samples = 0\n        for t in self.file_to_samples(trn_path, gold=True):\n            if self.config.get('multi_label', None):\n                for l in t[1]:\n                    self.label_vocab.add(l)\n            else:\n                self.label_vocab.add(t[1])  # the second one regardless of t is pair or triple\n            samples += 1\n        return samples\n\n    def create_types_shapes_values(self) -> Tuple[Tuple, Tuple, Tuple]:\n        num_features = self.config.num_features\n        # It's crucial to use tuple instead of list for all the three\n        types = tuple([tf.string] * num_features), tf.string\n        shapes = tuple([[]] * num_features), []\n        values = tuple([PAD] * num_features), self.label_vocab.safe_pad_token\n        return types, shapes, values\n\n    def x_to_idx(self, x) -> Union[tf.Tensor, Tuple]:\n        logger.warning('TableTransform can not map x to idx. Please override x_to_idx')\n        return x\n"
  },
  {
    "path": "hanlp/transform/tacred_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-03-14 17:06\nfrom typing import Union, Tuple\n\nimport tensorflow as tf\n\nfrom hanlp_common.structure import SerializableDict\nfrom hanlp.common.transform_tf import Transform\nfrom hanlp.common.vocab_tf import VocabTF\nfrom hanlp_common.io import load_json\nfrom hanlp_common.util import merge_locals_kwargs\n\n\ndef get_positions(start_idx, end_idx, length):\n    \"\"\"Get subj/obj position sequence.\n\n    Args:\n      start_idx: \n      end_idx: \n      length: \n\n    Returns:\n\n    \"\"\"\n    return list(range(-start_idx, 0)) + [0] * (end_idx - start_idx + 1) + \\\n           list(range(1, length - end_idx))\n\n\nclass TACREDTransform(Transform):\n    def __init__(self, config: SerializableDict = None, map_x=True, map_y=True, lower=False, **kwargs) -> None:\n        super().__init__(**merge_locals_kwargs(locals(), kwargs))\n        self.token_vocab = VocabTF()\n        self.pos_vocab = VocabTF(pad_token=None, unk_token=None)\n        self.ner_vocab = VocabTF(pad_token=None)\n        self.deprel_vocab = VocabTF(pad_token=None, unk_token=None)\n        self.rel_vocab = VocabTF(pad_token=None, unk_token=None)\n\n    def fit(self, trn_path: str, **kwargs) -> int:\n        count = 0\n        for (tokens, pos, ner, head, deprel, subj_positions, obj_positions, subj_type,\n             obj_type), relation in self.file_to_samples(\n            trn_path, gold=True):\n            count += 1\n            self.token_vocab.update(tokens)\n            self.pos_vocab.update(pos)\n            self.ner_vocab.update(ner)\n            self.deprel_vocab.update(deprel)\n            self.rel_vocab.add(relation)\n        return count\n\n    def file_to_inputs(self, filepath: str, gold=True):\n        data = load_json(filepath)\n        for d in data:\n            tokens = list(d['token'])\n            ss, se = d['subj_start'], d['subj_end']\n            os, oe = d['obj_start'], d['obj_end']\n            pos = d['stanford_pos']\n            ner = d['stanford_ner']\n            deprel = d['stanford_deprel']\n            head = [int(x) for x in d['stanford_head']]\n            assert any([x == 0 for x in head])\n            relation = d['relation']\n            yield (tokens, pos, ner, head, deprel, ss, se, os, oe), relation\n\n    def inputs_to_samples(self, inputs, gold=False):\n        for input in inputs:\n            if gold:\n                (tokens, pos, ner, head, deprel, ss, se, os, oe), relation = input\n            else:\n                tokens, pos, ner, head, deprel, ss, se, os, oe = input\n                relation = self.rel_vocab.safe_pad_token\n            l = len(tokens)\n            subj_positions = get_positions(ss, se, l)\n            obj_positions = get_positions(os, oe, l)\n            subj_type = ner[ss]\n            obj_type = ner[os]\n            # anonymize tokens\n            tokens[ss:se + 1] = ['SUBJ-' + subj_type] * (se - ss + 1)\n            tokens[os:oe + 1] = ['OBJ-' + obj_type] * (oe - os + 1)\n            # min head is 0, but root is not included in tokens, so take 1 off from each head\n            head = [h - 1 for h in head]\n            yield (tokens, pos, ner, head, deprel, subj_positions, obj_positions, subj_type, obj_type), relation\n\n    def create_types_shapes_values(self) -> Tuple[Tuple, Tuple, Tuple]:\n        # (tokens, pos, ner, head, deprel, subj_positions, obj_positions, subj_type, obj_type), relation\n        types = (tf.string, tf.string, tf.string, tf.int32, tf.string, tf.int32, tf.int32, tf.string,\n                 tf.string), tf.string\n        shapes = ([None], [None], [None], [None], [None], [None], [None], [], []), []\n        pads = (self.token_vocab.safe_pad_token, self.pos_vocab.safe_pad_token, self.ner_vocab.safe_pad_token, 0,\n                self.deprel_vocab.safe_pad_token,\n                0, 0, self.ner_vocab.safe_pad_token, self.ner_vocab.safe_pad_token), self.rel_vocab.safe_pad_token\n        return types, shapes, pads\n\n    def x_to_idx(self, x) -> Union[tf.Tensor, Tuple]:\n        tokens, pos, ner, head, deprel, subj_positions, obj_positions, subj_type, obj_type = x\n        tokens = self.token_vocab.lookup(tokens)\n        pos = self.pos_vocab.lookup(pos)\n        ner = self.ner_vocab.lookup(ner)\n        deprel = self.deprel_vocab.lookup(deprel)\n        subj_type = self.ner_vocab.lookup(subj_type)\n        obj_type = self.ner_vocab.lookup(obj_type)\n        return tokens, pos, ner, head, deprel, subj_positions, obj_positions, subj_type, obj_type\n\n    def y_to_idx(self, y) -> tf.Tensor:\n        return self.rel_vocab.lookup(y)\n"
  },
  {
    "path": "hanlp/transform/text_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-04 11:46\nfrom typing import Union, Tuple, Iterable, Any\n\nimport tensorflow as tf\n\nfrom hanlp_common.structure import SerializableDict\nfrom hanlp.common.transform_tf import Transform\nfrom hanlp.common.vocab_tf import VocabTF\nfrom hanlp.metrics.chunking.sequence_labeling import get_entities\nfrom hanlp.utils.file_read_backwards import FileReadBackwards\nfrom hanlp.utils.io_util import read_tsv_as_sents\n\n\nclass TextTransform(Transform):\n\n    def __init__(self,\n                 forward=True,\n                 seq_len=10,\n                 tokenizer='char',\n                 config: SerializableDict = None, map_x=True, map_y=True, **kwargs) -> None:\n        super().__init__(config, map_x, map_y, seq_len=seq_len, tokenizer=tokenizer, forward=forward, **kwargs)\n        self.vocab: VocabTF = None\n\n    def tokenize_func(self):\n        if self.config.tokenizer == 'char':\n            return list\n        elif self.config.tokenizer == 'whitespace':\n            return lambda x: x.split()\n        else:\n            return lambda x: x.split(self.config.tokenizer)\n\n    def fit(self, trn_path: str, **kwargs) -> int:\n        self.vocab = VocabTF()\n        num_samples = 0\n        for x, y in self.file_to_inputs(trn_path):\n            self.vocab.update(x)\n            num_samples += 1\n        return num_samples\n\n    def create_types_shapes_values(self) -> Tuple[Tuple, Tuple, Tuple]:\n        types = tf.string, tf.string\n        shapes = [None], [None]\n        defaults = self.vocab.pad_token, self.vocab.pad_token\n        return types, shapes, defaults\n\n    def file_to_inputs(self, filepath: str, gold=True):\n        forward = self.config.forward\n        seq_len = self.config.seq_len\n        buffer = []\n        tokenizer = self.tokenize_func()\n        with open(filepath, encoding='utf-8') if forward else FileReadBackwards(filepath, encoding=\"utf-8\") as src:\n            for line in src:\n                tokens = tokenizer(line)\n                buffer += tokens\n                while len(buffer) > seq_len:\n                    yield buffer[:seq_len], buffer[1:1 + seq_len]\n                    buffer.pop(0)\n\n    def inputs_to_samples(self, inputs, gold=False):\n        forward = self.config.forward\n        for t in inputs:\n            if gold:\n                x, y = t\n            else:\n                x, y = t, t\n            if not forward:\n                x = list(reversed(x))\n                y = list(reversed(y))\n            yield x, y\n\n    def x_to_idx(self, x) -> Union[tf.Tensor, Tuple]:\n        return self.vocab.lookup(x)\n\n    def y_to_idx(self, y) -> tf.Tensor:\n        return self.x_to_idx(y)\n\n    def Y_to_outputs(self, Y: Union[tf.Tensor, Tuple[tf.Tensor]], gold=False, inputs=None, **kwargs) -> Iterable:\n        pred = tf.argmax(Y, axis=-1)\n        for ys, ms in zip(pred, inputs):\n            ret = []\n            for y in ys:\n                ret.append(self.vocab.idx_to_token[int(y)])\n            yield ret\n\n    def input_is_single_sample(self, input: Any) -> bool:\n        return isinstance(input[0], str)\n\n\ndef bmes_to_flat(inpath, outpath):\n    with open(outpath, 'w', encoding='utf-8') as out:\n        for sent in read_tsv_as_sents(inpath):\n            chunks = get_entities([cells[1] for cells in sent])\n            chars = [cells[0] for cells in sent]\n            words = []\n            for tag, start, end in chunks:\n                word = ''.join(chars[start: end])\n                words.append(word)\n            out.write(' '.join(f'{word}/{tag}' for word, (tag, _, _) in zip(words, chunks)))\n            out.write('\\n')"
  },
  {
    "path": "hanlp/transform/transformer_tokenizer.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-03 16:23\nimport warnings\nfrom typing import Union, Optional\n\nfrom hanlp_common.constant import BOS, EOS\nfrom hanlp_common.structure import SerializableDict\nfrom hanlp.layers.transformers.pt_imports import PreTrainedTokenizer, PretrainedConfig, AutoTokenizer_\nfrom hanlp_trie import DictInterface\n\n\nclass TransformerTokenizer(object):\n\n    def __init__(self, max_seq_length=512, truncate_long_sequences=True) -> None:\n        self.truncate_long_sequences = truncate_long_sequences\n        self.max_seq_length = max_seq_length\n\n    def sliding_window(self, flat_wordpiece_ids, same_tail=True):\n        if same_tail:\n            start_piece_ids, flat_wordpiece_ids, end_piece_ids = flat_wordpiece_ids[:1], \\\n                                                                 flat_wordpiece_ids[1:-1], flat_wordpiece_ids[-1:]\n        else:\n            start_piece_ids, flat_wordpiece_ids, end_piece_ids = flat_wordpiece_ids[:1], \\\n                                                                 flat_wordpiece_ids[1:], []\n        window_length = self.max_seq_length - len(start_piece_ids) - len(end_piece_ids)\n        stride = window_length // 2\n        wordpiece_windows = [start_piece_ids + flat_wordpiece_ids[i:i + window_length] + end_piece_ids\n                             for i in range(0, len(flat_wordpiece_ids), stride)]\n\n        # Check for overlap in the last window. Throw it away if it is redundant.\n        last_window = wordpiece_windows[-1][1:]\n        penultimate_window = wordpiece_windows[-2]\n        if last_window == penultimate_window[-len(last_window):]:\n            wordpiece_windows = wordpiece_windows[:-1]\n\n        wordpiece_ids = [wordpiece for sequence in wordpiece_windows for wordpiece in sequence]\n        return wordpiece_ids\n\n\nclass TransformerTextTokenizer(TransformerTokenizer):\n    _KEY = ['input_ids', 'attention_mask', 'token_type_ids']\n\n    def __init__(self,\n                 tokenizer: Union[PreTrainedTokenizer, str],\n                 text_a_key: str,\n                 text_b_key: str = None,\n                 output_key=None,\n                 max_seq_length=512, truncate_long_sequences=True) -> None:\n        super().__init__(max_seq_length, truncate_long_sequences)\n        self.text_b = text_b_key\n        self.text_a = text_a_key\n        if output_key is None:\n            output_key = self.text_a\n            if text_b_key:\n                output_key += '_' + text_b_key\n        if output_key == '':\n            output_key = self._KEY\n        else:\n            output_key = [f'{output_key}_{key}' for key in self._KEY]\n        self.output_key = output_key\n        if isinstance(tokenizer, str):\n            tokenizer = AutoTokenizer_.from_pretrained(tokenizer)\n        self.tokenizer = tokenizer\n\n    def __call__(self, sample: dict):\n        text_a = sample[self.text_a]\n        text_b = sample[self.text_b] if self.text_b else None\n        max_seq_length = self.max_seq_length if self.truncate_long_sequences else None\n        encoding = self.tokenizer.encode_plus(text_a, text_b, max_length=max_seq_length)\n        results = dict((k, encoding.data.get(k, None)) for k in self._KEY)\n        if not self.truncate_long_sequences and len(results['input_ids']) > self.max_seq_length:\n            # TODO: other fields should be properly handled too\n            results['input_ids'] = self.sliding_window(results['input_ids'])\n        if not results['token_type_ids']:\n            results['token_type_ids'] = encoding[0].type_ids\n        for k, v in zip(self.output_key, [results[_] for _ in self._KEY]):\n            sample[k] = v\n        return sample\n\n\nclass TransformerSequenceTokenizer(TransformerTokenizer):\n\n    def __init__(self,\n                 tokenizer: Union[PreTrainedTokenizer, str],\n                 input_key,\n                 output_key=None,\n                 max_seq_length=512,\n                 truncate_long_sequences=False,\n                 config: PretrainedConfig = None,\n                 cls_token_at_end=False,\n                 cls_token_segment_id=0,\n                 pad_token_segment_id=0,\n                 pad_on_left=False,\n                 do_padding=False,\n                 sep_token_extra=False,\n                 ret_mask_and_type=False,\n                 ret_prefix_mask=False,\n                 ret_token_span=True,\n                 ret_subtokens=False,\n                 ret_subtokens_group=False,\n                 cls_is_bos=False,\n                 sep_is_eos=False,\n                 do_basic_tokenize=True,\n                 use_fast=True,\n                 dict_force=None,\n                 strip_cls_sep=True,\n                 check_space_before=None,\n                 ) -> None:\n        \"\"\"A transformer tokenizer for token-level tasks. It honors the boundary of tokens and tokenize each token into\n        several subtokens then merge them. The information about each subtoken belongs to which token are kept and\n        returned as a new field in the sample. It also provides out-of-box sliding window trick on long sequences.\n\n        Args:\n            tokenizer: The identifier of a pre-trained tokenizer or a ``PreTrainedTokenizer``.\n            input_key: The token key in samples.\n            output_key: The output keys to store results.\n                max_seq_length: Sentences longer than ``max_seq_len`` will be split into shorter ones if possible.\n            truncate_long_sequences: ``True`` to truncate exceeded parts of long sequences. ``False`` to  enable\n                sliding window.\n            config: The ``PretrainedConfig`` to determine the model structure of the transformer, so that special\n                tokenization can be applied.\n            cls_token_at_end: ``True`` to put ``[CLS]`` at the end of input tokens.\n            cls_token_segment_id: The id of ``[CLS]``.\n            pad_token_segment_id: The id of ``[SEP]``.\n            pad_on_left: ``True`` to put ``[PAD]`` at the left side of input tokens.\n            do_padding: ``True`` to pad sequence to the left.\n            sep_token_extra: ``True`` to have two ``[SEP]``.\n            ret_mask_and_type: ``True`` to return masks and type ids.\n            ret_prefix_mask: ``True`` to generate a mask where each non-zero element corresponds to a prefix of a token.\n            ret_token_span: ``True`` to return span of each token measured by subtoken offsets.\n            ret_subtokens: ``True`` to return list of subtokens belonging to each token for tokenization purpose.\n                When enabled, the prefix mask for each subtoken is set to True as each subtoken is a token unit in\n                tokenization task. Similarity, the token span for each token will be a continuous integer sequence.\n            ret_subtokens_group: ``True`` to return list of offsets of subtokens belonging to each token.\n            cls_is_bos: ``True`` means the first token of input is treated as [CLS] no matter what its surface form is.\n                        ``False`` (default) means the first token is not [CLS], it will have its own embedding other than\n                        the embedding of [CLS].\n            sep_is_eos: ``True`` means the last token of input is [SEP].\n                        ``False`` means it's not but [SEP] will be appended,\n                        ``None`` means it dependents on `input[-1] == [EOS]`.\n            do_basic_tokenize: Whether to do basic tokenization before wordpiece.\n            use_fast: Whether or not to try to load the fast version of the tokenizer.\n            dict_force: A dictionary doing longest-prefix-match on input text so that the head and tail of each keyword\n                won't be concatenated to other tokens by transformer tokenizers.\n            strip_cls_sep: ``True`` to strip [CLS] and [SEP] off the input tokens.\n            check_space_before: ``True`` to detect the space before each token to handle underline in sentence piece\n                tokenization.\n\n        Examples:\n\n        .. highlight:: python\n        .. code-block:: python\n\n            transform = TransformerSequenceTokenizer('bert-base-uncased', 'token')\n            sample = {'token': 'HanLP good'.split()}\n            print(transform(sample))\n\n        \"\"\"\n        super().__init__(max_seq_length, truncate_long_sequences)\n        tokenizer_name = tokenizer if isinstance(tokenizer, str) else tokenizer.name_or_path\n        if check_space_before is None:\n            # These tokenizer is BPE-based which appends a space before each token and tokenizes loving into\n            # ['▁lo', 'ving'], tokenize 商品 into ['▁', '商品']. For the later case, the prefix '▁' has to be removed\n            # as there is no space between some languages like Chinese\n            check_space_before = tokenizer_name in ('xlm-roberta-base', 'xlm-roberta-large', 'google/mt5-small',\n                                                    'google/mt5-base', 'xlm-roberta-base-no-space',\n                                                    'mMiniLMv2L6-no-space', 'mMiniLMv2L12-no-space')\n        self.check_space_before = check_space_before\n        self.ret_subtokens_group = ret_subtokens_group\n        self.ret_subtokens = ret_subtokens\n        self.sep_is_eos = sep_is_eos\n        self.ret_prefix_mask = ret_prefix_mask\n        self.ret_mask_and_type = ret_mask_and_type\n        self.cls_is_bos = cls_is_bos\n        self.ret_token_span = ret_token_span\n        if not output_key or isinstance(output_key, str):\n            suffixes = ['input_ids']\n            if ret_mask_and_type:\n                suffixes += 'attention_mask', 'token_type_ids'\n            if ret_prefix_mask:\n                suffixes += ['prefix_mask']\n            if ret_token_span:\n                suffixes.append('token_span')\n            if output_key is None:\n                output_key = [f'{input_key}_{key}' for key in suffixes]\n            elif output_key == '':\n                output_key = suffixes\n            else:\n                output_key = [f'{output_key}_{key}' for key in suffixes]\n\n        self.input_key = input_key\n        self.output_key = output_key\n        if config:\n            xlnet = config_is(config, 'xlnet')\n            pad_token_segment_id = 4 if xlnet else 0\n            cls_token_segment_id = 2 if xlnet else 0\n            cls_token_at_end = xlnet\n            pad_on_left = xlnet\n        if isinstance(tokenizer, str):\n            tokenizer = AutoTokenizer_.from_pretrained(tokenizer, use_fast=use_fast,\n                                                       do_basic_tokenize=do_basic_tokenize)\n        if use_fast:\n            # Dirty fix upstream bug: https://github.com/hankcs/HanLP/issues/1602\n            if hasattr(tokenizer, '_tokenizer') and hasattr(tokenizer._tokenizer, 'no_truncation'):\n                _t = tokenizer._tokenizer\n                _t.no_truncation()\n                _t.no_padding()\n                _t.no_truncation = _t.no_padding = lambda: None\n        pad_token = tokenizer.pad_token\n        self.pad_token_id = tokenizer.convert_tokens_to_ids([pad_token])[0]\n        self.pad_token_segment_id = pad_token_segment_id\n        if tokenizer_name in ('google/mt5-small', 'google/mt5-base'):\n            # mt5 doesn't have cls or sep, but we can use something similar\n            self.has_cls = False\n            self.cls_token = '▁'\n            self.cls_token_id = tokenizer.convert_tokens_to_ids(self.cls_token)\n            self.sep_token = tokenizer.eos_token\n            self.sep_token_id = tokenizer.eos_token_id\n        else:\n            self.has_cls = True\n            self.cls_token = tokenizer.cls_token\n            self.sep_token = tokenizer.sep_token\n            self.cls_token_segment_id = cls_token_segment_id\n            self.cls_token_id = tokenizer.cls_token_id\n            self.sep_token_id = tokenizer.sep_token_id\n\n        self.sep_token_extra = sep_token_extra\n        self.cls_token_at_end = cls_token_at_end\n        self.tokenizer = tokenizer\n        self.pad_on_left = pad_on_left\n        self.do_padding = do_padding\n        if self.ret_token_span or not self.truncate_long_sequences:\n            assert not self.cls_token_at_end\n            assert not self.pad_on_left\n        # if self.ret_subtokens:\n        #     if not use_fast:\n        #         raise NotImplementedError(\n        #             'ret_subtokens is not available when using Python tokenizers. '\n        #             'To use this feature, set use_fast = True.')\n        self.dict: Optional[DictInterface] = dict_force  # For tokenization of raw text\n        self.strip_cls_sep = strip_cls_sep\n\n    def __call__(self, sample: dict):\n        input_tokens = sample[self.input_key]\n        input_is_str = isinstance(input_tokens, str)\n        tokenizer = self.tokenizer\n        ret_token_span = self.ret_token_span\n        if input_is_str:  # This happens in a tokenizer component where the raw sentence is fed.\n\n            # noinspection PyShadowingNames\n            def tokenize_str(input_str, add_special_tokens=True):\n                if tokenizer.is_fast:\n                    encoding = tokenizer.encode_plus(input_str,\n                                                     return_offsets_mapping=True,\n                                                     add_special_tokens=add_special_tokens).encodings[0]\n                    subtoken_offsets = encoding.offsets\n                    input_tokens = encoding.tokens\n                    input_ids = encoding.ids\n\n                    # Fill up missing non-blank characters swallowed by HF tokenizer\n                    offset = 0\n                    fixed_offsets = []\n                    fixed_tokens = []\n                    fixed_ids = []\n                    for token, id, (b, e) in zip(input_tokens, input_ids, subtoken_offsets):\n                        if b > offset:\n                            missing_token = input_str[offset: b]\n                            if not missing_token.isspace():  # In the future, we may want space back\n                                fixed_tokens.append(missing_token)\n                                fixed_ids.append(tokenizer.unk_token_id)\n                                fixed_offsets.append((offset, b))\n                        if e == offset:  # LI™ -> LIT + M\n                            if fixed_offsets and fixed_offsets[-1][0] < b:\n                                fixed_offsets[-1] = (fixed_offsets[-1][0], b)\n\n                        fixed_tokens.append(token)\n                        fixed_ids.append(id)\n                        fixed_offsets.append((b, e))\n                        offset = e\n                    subtoken_offsets = fixed_offsets\n                    input_tokens = fixed_tokens\n                    input_ids = fixed_ids\n\n                    if add_special_tokens:\n                        subtoken_offsets = subtoken_offsets[1 if self.has_cls else 0:-1]\n\n                    # Edge case that the input_str is swallowed in whole\n                    if input_str and not subtoken_offsets and not input_str.isspace():\n                        __index = 1 if add_special_tokens and self.has_cls else 0\n                        input_tokens.insert(__index, input_str)\n                        input_ids.insert(__index, tokenizer.unk_token_id)\n                        subtoken_offsets.append((0, len(input_str)))\n\n                    if not self.has_cls:\n                        input_tokens = [self.cls_token] + input_tokens\n                        input_ids = [self.cls_token_id] + input_ids\n                else:\n                    input_tokens = tokenizer.tokenize(input_str)\n                    subtoken_offsets = []\n                    _o = 0\n                    for each in input_tokens:\n                        subtoken_offsets.append((_o, _o + len(each)))\n                        _o += len(each)\n                    if add_special_tokens:\n                        input_tokens = [self.cls_token] + input_tokens + [self.sep_token]\n                    input_ids = tokenizer.convert_tokens_to_ids(input_tokens)\n                if self.check_space_before:\n                    non_blank_offsets = [i for i in range(len(input_tokens)) if input_tokens[i] != '▁']\n                    if add_special_tokens and not self.has_cls:\n                        non_blank_offsets.insert(0, 0)\n                    input_tokens = [input_tokens[i] for i in non_blank_offsets]\n                    input_ids = [input_ids[i] for i in non_blank_offsets]\n                    if add_special_tokens:\n                        non_blank_offsets = non_blank_offsets[1:-1]\n                        subtoken_offsets = [subtoken_offsets[i - 1] for i in non_blank_offsets]\n                    else:\n                        subtoken_offsets = [subtoken_offsets[i] for i in non_blank_offsets]\n                    # MT5 generates tokens like ▁of, which is bad for the tokenizer. So we want to remove the prefix.\n                    for i, token in enumerate(input_tokens[1:-1] if add_special_tokens else input_tokens):\n                        if input_str[subtoken_offsets[i][0]] == ' ':\n                            subtoken_offsets[i] = (subtoken_offsets[i][0] + 1, subtoken_offsets[i][1])\n                # The following block will tokenize each empty string (space) into an unk token\n                # if add_special_tokens:\n                #     if len(input_tokens) == 2:  # bos and eos, meaning that the text contains only some spaces\n                #         input_tokens.insert(1, input_str)\n                #         input_ids.insert(1, tokenizer.unk_token_id)\n                #         subtoken_offsets.append((0, len(input_str)))\n                # else:\n                #     if not input_ids:  # This chunk might be some control chars getting removed by tokenizer\n                #         input_tokens = [input_str]\n                #         input_ids = [tokenizer.unk_token_id]\n                #         subtoken_offsets = [(0, len(input_str))]\n                return input_tokens, input_ids, subtoken_offsets\n\n            if self.dict:\n                chunks = self.dict.split(sample.get(f'{self.input_key}_', input_tokens))  # Match original text directly\n                _input_tokens, _input_ids, _subtoken_offsets = [self.cls_token], [self.cls_token_id], []\n                _offset = 0\n                custom_words = sample['custom_words'] = []\n                char_offset = 0\n                for chunk in chunks:\n                    if isinstance(chunk, str):  # Use transformed text as it's what models are trained on\n                        chunk = input_tokens[char_offset:char_offset + len(chunk)]\n                        tokens, ids, offsets = tokenize_str(chunk, add_special_tokens=False)\n                        char_offset += len(chunk)\n                    else:\n                        begin, end, label = chunk\n                        _offset = begin\n                        # chunk offset is on char level, at this moment, there is no concept of tokens, just subtokens\n                        if isinstance(label, list):\n                            tokens, ids, offsets, delta = [], [], [], 0\n                            for token in label:\n                                _tokens, _ids, _offsets = tokenize_str(token, add_special_tokens=False)\n                                tokens.extend(_tokens)\n                                # track the subword offset of this chunk, -1 for [CLS]\n                                custom_words.append(\n                                    (len(_input_ids) + len(ids) - 1, len(_input_ids) + len(ids) - 1 + len(_ids), token))\n                                ids.extend(_ids)\n                                offsets.extend((x[0] + delta, x[1] + delta) for x in _offsets)\n                                delta = offsets[-1][-1]\n                        else:\n                            tokens, ids, offsets = tokenize_str(input_tokens[begin:end], add_special_tokens=False)\n                            # offsets = [(offsets[0][0], offsets[-1][-1])]\n                            custom_words.append((len(_input_ids) - 1, len(_input_ids) + len(ids) - 1, label))\n                        char_offset = end\n                    _input_tokens.extend(tokens)\n                    _input_ids.extend(ids)\n                    _subtoken_offsets.extend((x[0] + _offset, x[1] + _offset) for x in offsets)\n                    _offset = _subtoken_offsets[-1][-1]\n                subtoken_offsets = _subtoken_offsets\n                input_tokens = _input_tokens + [self.sep_token]\n                input_ids = _input_ids + [self.sep_token_id]\n            else:\n                input_tokens, input_ids, subtoken_offsets = tokenize_str(input_tokens, add_special_tokens=True)\n\n            if self.ret_subtokens:\n                sample[f'{self.input_key}_subtoken_offsets'] = subtoken_offsets\n\n        cls_is_bos = self.cls_is_bos\n        if cls_is_bos is None:\n            cls_is_bos = input_tokens[0] == BOS\n        sep_is_eos = self.sep_is_eos\n        if sep_is_eos is None:\n            sep_is_eos = input_tokens[-1] == EOS\n        if self.strip_cls_sep:\n            if cls_is_bos:\n                input_tokens = input_tokens[1:]\n            if sep_is_eos:\n                input_tokens = input_tokens[:-1]\n        if not self.ret_mask_and_type:  # only need input_ids and token_span, use a light version\n            if input_is_str:\n                prefix_mask = self._init_prefix_mask(input_ids)\n            else:\n                if input_tokens:\n                    return_offsets_mapping = tokenizer.is_fast and self.ret_subtokens\n                    encodings = tokenizer.batch_encode_plus(\n                        input_tokens,\n                        return_offsets_mapping=return_offsets_mapping,  # Many tokenizers do not offer fast version\n                        add_special_tokens=False\n                    )\n                    subtoken_ids_per_token = encodings.data['input_ids']\n                    if return_offsets_mapping:\n                        offsets_mapping = [encoding.offsets for encoding in encodings.encodings]\n                    else:\n                        offsets_mapping = []\n                        for token, subtoken_ids in zip(input_tokens, subtoken_ids_per_token):\n                            if len(subtoken_ids) > len(token):  # … --> ...\n                                del subtoken_ids[len(token):]\n                            if not subtoken_ids:\n                                subtoken_ids = [tokenizer.unk_token_id]\n                            # Since non-fast tok generates no mapping, we have to guess\n                            char_per_subtoken = max(len(token) // len(subtoken_ids), 1)\n                            bes = [(b, b + char_per_subtoken) for b in range(0, len(token), char_per_subtoken)]\n                            if not bes:  # the token is an empty string\n                                bes = [(0, 0)]\n                            if len(bes) != len(subtoken_ids):\n                                bes[len(subtoken_ids) - 1] = (bes[len(subtoken_ids) - 1][0], len(token))\n                                del bes[len(subtoken_ids):]\n                            offsets_mapping.append(bes)\n                else:\n                    encodings = SerializableDict()\n                    subtoken_ids_per_token = []\n                    encodings.data = {'input_ids': subtoken_ids_per_token}\n                if self.check_space_before:\n                    # noinspection PyUnboundLocalVariable\n                    for token, subtokens, mapping, encoding in zip(input_tokens, subtoken_ids_per_token,\n                                                                   offsets_mapping, encodings.encodings):\n                        # Remove ▁ generated by spm for 2 reasons:\n                        # 1. During decoding, mostly no ▁ will be created unless blanks are placed between tokens (which\n                        # is true for English but in English it will likely be concatenated to the token following it)\n                        # 2. For T5, '▁' is used as CLS\n                        if len(subtokens) > 1 and encoding.tokens[0] == '▁':\n                            subtokens.pop(0)\n                            if mapping:\n                                mapping.pop(0)\n                # Some tokens get stripped out\n                subtoken_ids_per_token = [ids if ids else [tokenizer.unk_token_id] for ids in subtoken_ids_per_token]\n                input_ids = sum(subtoken_ids_per_token, [self.cls_token_id])\n                if self.sep_is_eos is None:\n                    # None means to check whether sep is at the tail or between tokens\n                    if sep_is_eos:\n                        input_ids += [self.sep_token_id]\n                    elif self.sep_token_id not in input_ids:\n                        input_ids += [self.sep_token_id]\n                else:\n                    input_ids += [self.sep_token_id]\n                # else self.sep_is_eos == False means sep is between tokens and don't bother to check\n\n                if self.ret_subtokens:\n                    prefix_mask = self._init_prefix_mask(input_ids)\n                    # if self.check_space_before:\n                    #     if offsets_mapping[0] and not input_tokens[0].startswith(' '):\n                    #         prefix_mask[1] = False\n                else:\n                    prefix_mask = [False] * len(input_ids)\n                    offset = 1\n                    for _subtokens in subtoken_ids_per_token:\n                        prefix_mask[offset] = True\n                        offset += len(_subtokens)\n                if self.ret_subtokens:\n                    subtoken_offsets = []\n                    for token, offsets in zip(input_tokens, offsets_mapping):\n                        if offsets:\n                            subtoken_offsets.append(offsets)\n                        else:\n                            subtoken_offsets.append([(0, len(token))])\n                    if self.ret_subtokens_group:\n                        sample[f'{self.input_key}_subtoken_offsets_group'] = subtoken_offsets\n                    else:\n                        sample[f'{self.input_key}_subtoken_offsets'] = sum(subtoken_offsets, [])\n        else:\n            input_ids, attention_mask, token_type_ids, prefix_mask = \\\n                convert_examples_to_features(input_tokens,\n                                             None,\n                                             tokenizer,\n                                             cls_token_at_end=self.cls_token_at_end,\n                                             # xlnet has a cls token at the end\n                                             cls_token=tokenizer.cls_token,\n                                             cls_token_segment_id=self.cls_token_segment_id,\n                                             sep_token=self.sep_token,\n                                             sep_token_extra=self.sep_token_extra,\n                                             # roberta uses an extra separator b/w pairs of sentences, cf. github.com/pytorch/fairseq/commit/1684e166e3da03f5b600dbb7855cb98ddfcd0805\n                                             pad_on_left=self.pad_on_left,\n                                             # pad on the left for xlnet\n                                             pad_token_id=self.pad_token_id,\n                                             pad_token_segment_id=self.pad_token_segment_id,\n                                             pad_token_label_id=0,\n                                             do_padding=self.do_padding)\n        if len(input_ids) > self.max_seq_length:\n            if self.truncate_long_sequences:\n                # raise SequenceTooLong(\n                #     f'Input tokens {input_tokens} exceed the max sequence length of {self.max_seq_length - 2}. '\n                #     f'For sequence tasks, truncate_long_sequences = True is not supported.'\n                #     f'You are recommended to split your long text into several sentences within '\n                #     f'{self.max_seq_length - 2} tokens beforehand. '\n                #     f'Or simply set truncate_long_sequences = False to enable sliding window.')\n                input_ids = input_ids[:self.max_seq_length]\n                prefix_mask = prefix_mask[:self.max_seq_length]\n                warnings.warn(\n                    f'Input tokens {input_tokens} exceed the max sequence length of {self.max_seq_length - 2}. '\n                    f'The exceeded part will be truncated and ignored. '\n                    f'You are recommended to split your long text into several sentences within '\n                    f'{self.max_seq_length - 2} tokens beforehand.'\n                    f'Or simply set truncate_long_sequences = False to enable sliding window.'\n                )\n            else:\n                input_ids = self.sliding_window(input_ids, input_ids[-1] == self.sep_token_id)\n        if prefix_mask:\n            if cls_is_bos:\n                prefix_mask[0] = True\n            if sep_is_eos:\n                prefix_mask[-1] = True\n        outputs = [input_ids]\n        if self.ret_mask_and_type:\n            # noinspection PyUnboundLocalVariable\n            outputs += [attention_mask, token_type_ids]\n        if self.ret_prefix_mask:\n            outputs += [prefix_mask]\n        if ret_token_span and prefix_mask:\n            if cls_is_bos:\n                token_span = [[0]]\n            else:\n                token_span = []\n            offset = 1\n            span = []\n            for mask in prefix_mask[1:len(prefix_mask) if sep_is_eos is None else -1]:  # skip [CLS] and [SEP]\n                if mask and span:\n                    token_span.append(span)\n                    span = []\n                span.append(offset)\n                offset += 1\n            if span:\n                token_span.append(span)\n            if sep_is_eos:\n                assert offset == len(prefix_mask) - 1\n                token_span.append([offset])\n            outputs.append(token_span)\n        for k, v in zip(self.output_key, outputs):\n            sample[k] = v\n        return sample\n\n    def _init_prefix_mask(self, input_ids):\n        prefix_mask = [True] * len(input_ids)\n        if not self.cls_is_bos:\n            prefix_mask[0] = False\n        if not self.sep_is_eos:\n            prefix_mask[-1] = False\n        return prefix_mask\n\n\ndef config_is(config, model='bert'):\n    return model in type(config).__name__.lower()\n\n\ndef convert_examples_to_features(\n        words,\n        max_seq_length: Optional[int],\n        tokenizer,\n        labels=None,\n        label_map=None,\n        cls_token_at_end=False,\n        cls_token=\"[CLS]\",\n        cls_token_segment_id=1,\n        sep_token=\"[SEP]\",\n        sep_token_extra=False,\n        pad_on_left=False,\n        pad_token_id=0,\n        pad_token_segment_id=0,\n        pad_token_label_id=0,\n        sequence_a_segment_id=0,\n        mask_padding_with_zero=True,\n        unk_token='[UNK]',\n        do_padding=True\n):\n    \"\"\"Loads a data file into a list of `InputBatch`s\n        `cls_token_at_end` define the location of the CLS token:\n            - False (Default, BERT/XLM pattern): [CLS] + A + [SEP] + B + [SEP]\n            - True (XLNet/GPT pattern): A + [SEP] + B + [SEP] + [CLS]\n        `cls_token_segment_id` define the segment id associated to the CLS token (0 for BERT, 2 for XLNet)\n\n    Args:\n      words: \n      max_seq_length: \n      tokenizer: \n      labels:  (Default value = None)\n      label_map:  (Default value = None)\n      cls_token_at_end:  (Default value = False)\n      cls_token:  (Default value = \"[CLS]\")\n      cls_token_segment_id:  (Default value = 1)\n      sep_token:  (Default value = \"[SEP]\")\n      sep_token_extra:  (Default value = False)\n      pad_on_left:  (Default value = False)\n      pad_token_id:  (Default value = 0)\n      pad_token_segment_id:  (Default value = 0)\n      pad_token_label_id:  (Default value = 0)\n      sequence_a_segment_id:  (Default value = 0)\n      mask_padding_with_zero:  (Default value = True)\n      unk_token:  (Default value = '[UNK]')\n      do_padding:  (Default value = True)\n\n    Returns:\n\n    \"\"\"\n    args = locals()\n    if not labels:\n        labels = words\n        pad_token_label_id = False\n\n    tokens = []\n    label_ids = []\n    for word, label in zip(words, labels):\n        word_tokens = tokenizer.tokenize(word)\n        if not word_tokens:\n            # some wired chars cause the tagger to return empty list\n            word_tokens = [unk_token] * len(word)\n        tokens.extend(word_tokens)\n        # Use the real label id for the first token of the word, and padding ids for the remaining tokens\n        label_ids.extend([label_map[label] if label_map else True] + [pad_token_label_id] * (len(word_tokens) - 1))\n\n    # Account for [CLS] and [SEP] with \"- 2\" and with \"- 3\" for RoBERTa.\n    special_tokens_count = 3 if sep_token_extra else 2\n    if max_seq_length and len(tokens) > max_seq_length - special_tokens_count:\n        warnings.warn(\n            f'Input tokens {words} exceed the max sequence length of {max_seq_length - special_tokens_count}. '\n            f'The exceeded part will be truncated and ignored. '\n            f'You are recommended to split your long text into several sentences within '\n            f'{max_seq_length - special_tokens_count} tokens beforehand.')\n        tokens = tokens[: (max_seq_length - special_tokens_count)]\n        label_ids = label_ids[: (max_seq_length - special_tokens_count)]\n\n    # The convention in BERT is:\n    # (a) For sequence pairs:\n    #  tokens:   [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]\n    #  token_type_ids:   0   0  0    0    0     0       0   0   1  1  1  1   1   1\n    # (b) For single sequences:\n    #  tokens:   [CLS] the dog is hairy . [SEP]\n    #  token_type_ids:   0   0   0   0  0     0   0\n    #\n    # Where \"token_type_ids\" are used to indicate whether this is the first\n    # sequence or the second sequence. The embedding vectors for `type=0` and\n    # `type=1` were learned during pre-training and are added to the wordpiece\n    # embedding vector (and position vector). This is not *strictly* necessary\n    # since the [SEP] token unambiguously separates the sequences, but it makes\n    # it easier for the model to learn the concept of sequences.\n    #\n    # For classification tasks, the first vector (corresponding to [CLS]) is\n    # used as as the \"sentence vector\". Note that this only makes sense because\n    # the entire model is fine-tuned.\n    tokens += [sep_token]\n    label_ids += [pad_token_label_id]\n    if sep_token_extra:\n        # roberta uses an extra separator b/w pairs of sentences\n        tokens += [sep_token]\n        label_ids += [pad_token_label_id]\n    segment_ids = [sequence_a_segment_id] * len(tokens)\n\n    if cls_token_at_end:\n        tokens += [cls_token]\n        label_ids += [pad_token_label_id]\n        segment_ids += [cls_token_segment_id]\n    else:\n        tokens = [cls_token] + tokens\n        label_ids = [pad_token_label_id] + label_ids\n        segment_ids = [cls_token_segment_id] + segment_ids\n\n    input_ids = tokenizer.convert_tokens_to_ids(tokens)\n\n    # The mask has 1 for real tokens and 0 for padding tokens. Only real\n    # tokens are attended to.\n    input_mask = [1 if mask_padding_with_zero else 0] * len(input_ids)\n\n    if do_padding:\n        # Zero-pad up to the sequence length.\n        padding_length = max_seq_length - len(input_ids)\n        if pad_on_left:\n            input_ids = ([pad_token_id] * padding_length) + input_ids\n            input_mask = ([0 if mask_padding_with_zero else 1] * padding_length) + input_mask\n            segment_ids = ([pad_token_segment_id] * padding_length) + segment_ids\n            label_ids = ([pad_token_label_id] * padding_length) + label_ids\n        else:\n            input_ids += [pad_token_id] * padding_length\n            input_mask += [0 if mask_padding_with_zero else 1] * padding_length\n            segment_ids += [pad_token_segment_id] * padding_length\n            label_ids += [pad_token_label_id] * padding_length\n\n        assert len(input_ids) == max_seq_length\n        assert len(input_mask) == max_seq_length\n        assert len(segment_ids) == max_seq_length\n        assert len(label_ids) == max_seq_length, f'failed for:\\n {args}'\n    else:\n        assert len(set(len(x) for x in [input_ids, input_mask, segment_ids, label_ids])) == 1\n    return input_ids, input_mask, segment_ids, label_ids\n\n\ndef main():\n    transformer = 'bert-base-uncased'\n    tokenizer: PreTrainedTokenizer = AutoTokenizer_.from_pretrained(transformer)\n    # _test_text_transform(tokenizer)\n    _test_sequence_transform(tokenizer)\n\n\ndef _test_text_transform(tokenizer):\n    transform = TransformerTextTokenizer(tokenizer, 'text')\n    sample = {'text': 'HanLP good'}\n    print(transform(sample))\n\n\ndef _test_sequence_transform(tokenizer):\n    transform = TransformerSequenceTokenizer(tokenizer, 'token')\n    sample = {'token': 'HanLP good'.split()}\n    print(transform(sample))\n\n\nif __name__ == '__main__':\n    main()\n"
  },
  {
    "path": "hanlp/transform/tsv_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-06-13 21:15\nimport functools\nfrom abc import ABC\nfrom typing import Tuple, Union, Optional, Iterable, List\n\nimport tensorflow as tf\n\nfrom hanlp_common.structure import SerializableDict\n\nfrom hanlp.common.transform_tf import Transform\nfrom hanlp.common.vocab_tf import VocabTF\nfrom hanlp.utils.io_util import generate_words_tags_from_tsv\nfrom hanlp.utils.tf_util import str_tensor_to_str\nfrom hanlp_common.util import merge_locals_kwargs\n\n\ndef dataset_from_tsv(tsv_file_path, word_vocab: VocabTF, char_vocab: VocabTF, tag_vocab: VocabTF, batch_size=32,\n                     shuffle=None, repeat=None, prefetch=1, lower=False, **kwargs):\n    generator = functools.partial(generate_words_tags_from_tsv, tsv_file_path, word_vocab, char_vocab, tag_vocab, lower)\n    return dataset_from_generator(generator, word_vocab, tag_vocab, batch_size, shuffle, repeat, prefetch,\n                                  **kwargs)\n\n\ndef dataset_from_generator(generator, word_vocab, tag_vocab, batch_size=32, shuffle=None, repeat=None, prefetch=1,\n                           **kwargs):\n    shapes = [None], [None]\n    types = tf.string, tf.string\n    defaults = word_vocab.pad_token, tag_vocab.pad_token if tag_vocab.pad_token else tag_vocab.first_token\n    dataset = tf.data.Dataset.from_generator(generator, output_shapes=shapes, output_types=types)\n    if shuffle:\n        if isinstance(shuffle, bool):\n            shuffle = 1024\n        dataset = dataset.shuffle(shuffle)\n    if repeat:\n        dataset = dataset.repeat(repeat)\n    dataset = dataset.padded_batch(batch_size, shapes, defaults).prefetch(prefetch)\n    return dataset\n\n\ndef vocab_from_tsv(tsv_file_path, lower=False, lock_word_vocab=False, lock_char_vocab=True, lock_tag_vocab=True) \\\n        -> Tuple[VocabTF, VocabTF, VocabTF]:\n    word_vocab = VocabTF()\n    char_vocab = VocabTF()\n    tag_vocab = VocabTF(unk_token=None)\n    with open(tsv_file_path, encoding='utf-8') as tsv_file:\n        for line in tsv_file:\n            cells = line.strip().split()\n            if cells:\n                word, tag = cells\n                if lower:\n                    word_vocab.add(word.lower())\n                else:\n                    word_vocab.add(word)\n                char_vocab.update(list(word))\n                tag_vocab.add(tag)\n    if lock_word_vocab:\n        word_vocab.lock()\n    if lock_char_vocab:\n        char_vocab.lock()\n    if lock_tag_vocab:\n        tag_vocab.lock()\n    return word_vocab, char_vocab, tag_vocab\n\n\nclass TsvTaggingFormat(Transform, ABC):\n    def file_to_inputs(self, filepath: str, gold=True):\n        assert gold, 'TsvTaggingFormat does not support reading non-gold files'\n        yield from generate_words_tags_from_tsv(filepath, gold=gold, lower=self.config.get('lower', False),\n                                                max_seq_length=self.max_seq_length)\n\n    @property\n    def max_seq_length(self):\n        return self.config.get('max_seq_length', None)\n\n\nclass TSVTaggingTransform(TsvTaggingFormat, Transform):\n    def __init__(self, config: SerializableDict = None, map_x=True, map_y=True, use_char=False, **kwargs) -> None:\n        super().__init__(**merge_locals_kwargs(locals(), kwargs))\n        self.word_vocab: Optional[VocabTF] = None\n        self.tag_vocab: Optional[VocabTF] = None\n        self.char_vocab: Optional[VocabTF] = None\n\n    def fit(self, trn_path: str, **kwargs) -> int:\n        self.word_vocab = VocabTF()\n        self.tag_vocab = VocabTF(pad_token=None, unk_token=None)\n        num_samples = 0\n        for words, tags in self.file_to_inputs(trn_path, True):\n            self.word_vocab.update(words)\n            self.tag_vocab.update(tags)\n            num_samples += 1\n        if self.char_vocab:\n            self.char_vocab = VocabTF()\n            for word in self.word_vocab.token_to_idx.keys():\n                if word in (self.word_vocab.pad_token, self.word_vocab.unk_token):\n                    continue\n                self.char_vocab.update(list(word))\n        return num_samples\n\n    def create_types_shapes_values(self) -> Tuple[Tuple, Tuple, Tuple]:\n        types = tf.string, tf.string\n        shapes = [None], [None]\n        values = self.word_vocab.pad_token, self.tag_vocab.first_token\n        return types, shapes, values\n\n    def inputs_to_samples(self, inputs, gold=False):\n        lower = self.config.get('lower', False)\n        if gold:\n            if lower:\n                for x, y in inputs:\n                    yield x.lower(), y\n            else:\n                yield from inputs\n        else:\n            for x in inputs:\n                yield x.lower() if lower else x, [self.padding_values[-1]] * len(x)\n\n    def x_to_idx(self, x) -> Union[tf.Tensor, Tuple]:\n        return self.word_vocab.lookup(x)\n\n    def y_to_idx(self, y) -> tf.Tensor:\n        return self.tag_vocab.lookup(y)\n\n    def X_to_inputs(self, X: Union[tf.Tensor, Tuple[tf.Tensor]]) -> Iterable:\n        for xs in X:\n            words = []\n            for x in xs:\n                words.append(str_tensor_to_str(x) if self.char_vocab else self.word_vocab.idx_to_token[int(x)])\n            yield words\n\n    def Y_to_outputs(self, Y: Union[tf.Tensor, Tuple[tf.Tensor]], gold=False,\n                     inputs=None, X=None, **kwargs) -> Iterable:\n        if not gold:\n            Y = tf.argmax(Y, axis=2)\n        for ys, xs in zip(Y, inputs):\n            tags = []\n            for y, x in zip(ys, xs):\n                tags.append(self.tag_vocab.idx_to_token[int(y)])\n            yield tags\n\n    def input_is_single_sample(self, input: Union[List[str], List[List[str]]]) -> bool:\n        return isinstance(input[0], str)\n\n    def input_truth_output_to_str(self, input: List[str], truth: List[str], output: List[str]):\n        text = ''\n        for word, gold_tag, pred_tag in zip(input, truth, output):\n            text += ' '.join([word, gold_tag, pred_tag]) + '\\n'\n\n        text += '\\n'\n        return text\n"
  },
  {
    "path": "hanlp/transform/txt_tf.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-10-24 15:07\nimport functools\nfrom abc import ABC\nfrom typing import Tuple, Union, List, Iterable\n\nimport tensorflow as tf\n\nfrom hanlp.common.transform_tf import Transform\nfrom hanlp.common.vocab_tf import VocabTF\nfrom hanlp.utils.io_util import get_resource\nfrom hanlp.utils.lang.zh.char_table import CharTable\nfrom hanlp.utils.span_util import bmes_of, bmes_to_words\nfrom hanlp.utils.string_util import split_long_sent\n\n\ndef generate_words_per_line(file_path):\n    with open(file_path, encoding='utf-8') as src:\n        for line in src:\n            cells = line.strip().split()\n            if not cells:\n                continue\n            yield cells\n\n\ndef words_to_bmes(words):\n    tags = []\n    for w in words:\n        if not w:\n            raise ValueError('{} contains None or zero-length word {}'.format(str(words), w))\n        if len(w) == 1:\n            tags.append('S')\n        else:\n            tags.extend(['B'] + ['M'] * (len(w) - 2) + ['E'])\n    return tags\n\n\ndef extract_ngram_features_and_tags(sentence, bigram_only=False, window_size=4, segmented=True):\n    \"\"\"\n    Feature extraction for windowed approaches\n    See Also https://github.com/chqiwang/convseg/\n    Parameters\n    ----------\n    sentence\n    bigram_only\n    window_size\n    segmented\n\n    Returns\n    -------\n\n    \"\"\"\n    chars, tags = bmes_of(sentence, segmented)\n    chars = CharTable.normalize_chars(chars)\n    ret = []\n    ret.append(chars)\n    # TODO: optimize ngram generation using https://www.tensorflow.org/api_docs/python/tf/strings/ngrams\n    ret.extend(extract_ngram_features(chars, bigram_only, window_size))\n    ret.append(tags)\n    return tuple(ret[:-1]), ret[-1]  # x, y\n\n\ndef extract_ngram_features(chars, bigram_only, window_size):\n    ret = []\n    if bigram_only:\n        chars = ['', ''] + chars + ['', '']\n        ret.append([a + b if a and b else '' for a, b in zip(chars[:-4], chars[1:])])\n        ret.append([a + b if a and b else '' for a, b in zip(chars[1:-3], chars[2:])])\n        ret.append([a + b if a and b else '' for a, b in zip(chars[2:-2], chars[3:])])\n        ret.append([a + b if a and b else '' for a, b in zip(chars[3:-1], chars[4:])])\n    elif window_size > 0:\n        chars = ['', '', ''] + chars + ['', '', '']\n        # single char\n        if window_size >= 1:\n            ret.append(chars[3:-3])\n        if window_size >= 2:\n            # bi chars\n            ret.append([a + b if a and b else '' for a, b in zip(chars[2:], chars[3:-3])])\n            ret.append([a + b if a and b else '' for a, b in zip(chars[3:-3], chars[4:])])\n        if window_size >= 3:\n            # tri chars\n            ret.append(\n                [a + b + c if a and b and c else '' for a, b, c in zip(chars[1:], chars[2:], chars[3:-3])])\n            ret.append(\n                [a + b + c if a and b and c else '' for a, b, c in zip(chars[2:], chars[3:-3], chars[4:])])\n            ret.append(\n                [a + b + c if a and b and c else '' for a, b, c in zip(chars[3:-3], chars[4:], chars[5:])])\n        if window_size >= 4:\n            # four chars\n            ret.append([a + b + c + d if a and b and c and d else '' for a, b, c, d in\n                        zip(chars[0:], chars[1:], chars[2:], chars[3:-3])])\n            ret.append([a + b + c + d if a and b and c and d else '' for a, b, c, d in\n                        zip(chars[1:], chars[2:], chars[3:-3], chars[4:])])\n            ret.append([a + b + c + d if a and b and c and d else '' for a, b, c, d in\n                        zip(chars[2:], chars[3:-3], chars[4:], chars[5:])])\n            ret.append([a + b + c + d if a and b and c and d else '' for a, b, c, d in\n                        zip(chars[3:-3], chars[4:], chars[5:], chars[6:])])\n    return ret\n\n\ndef generate_ngram_bmes(file_path, bigram_only=False, window_size=4, gold=True):\n    with open(file_path, encoding='utf-8') as src:\n        for line in src:\n            sentence = line.strip()\n            if not sentence:\n                continue\n            yield extract_ngram_features_and_tags(sentence, bigram_only, window_size, gold)\n\n\ndef vocab_from_txt(txt_file_path, bigram_only=False, window_size=4, **kwargs) -> Tuple[VocabTF, VocabTF, VocabTF]:\n    char_vocab, ngram_vocab, tag_vocab = VocabTF(), VocabTF(), VocabTF(pad_token=None, unk_token=None)\n    for X, Y in generate_ngram_bmes(txt_file_path, bigram_only, window_size, gold=True):\n        char_vocab.update(X[0])\n        for ngram in X[1:]:\n            ngram_vocab.update(filter(lambda x: x, ngram))\n        tag_vocab.update(Y)\n    return char_vocab, ngram_vocab, tag_vocab\n\n\ndef dataset_from_txt(txt_file_path: str, char_vocab: VocabTF, ngram_vocab: VocabTF, tag_vocab: VocabTF,\n                     bigram_only=False,\n                     window_size=4, segmented=True, batch_size=32, shuffle=None, repeat=None, prefetch=1):\n    generator = functools.partial(generate_ngram_bmes, txt_file_path, bigram_only, window_size, segmented)\n    return dataset_from_generator(generator, char_vocab, ngram_vocab, tag_vocab, bigram_only, window_size, batch_size,\n                                  shuffle, repeat, prefetch)\n\n\ndef dataset_from_generator(generator, char_vocab, ngram_vocab, tag_vocab, bigram_only=False, window_size=4,\n                           batch_size=32, shuffle=None, repeat=None, prefetch=1):\n    if bigram_only:\n        ngram_size = 4\n    else:\n        ngram_size = window_size * (window_size + 1) // 2\n    vec_dim = 2 + ngram_size\n    shapes = tuple([[None]] * (vec_dim - 1)), [None]\n    types = tuple([tf.string] * (vec_dim - 1)), tf.string\n    defaults = tuple([char_vocab.pad_token] + [\n        ngram_vocab.pad_token if ngram_vocab else char_vocab.pad_token] * ngram_size), (\n                   tag_vocab.pad_token if tag_vocab.pad_token else tag_vocab.first_token)\n    dataset = tf.data.Dataset.from_generator(generator, output_shapes=shapes, output_types=types)\n    if shuffle:\n        if isinstance(shuffle, bool):\n            shuffle = 1024\n        dataset = dataset.shuffle(shuffle)\n    if repeat:\n        dataset = dataset.repeat(repeat)\n    dataset = dataset.padded_batch(batch_size, shapes, defaults).prefetch(prefetch)\n    return dataset\n\n\nclass TxtFormat(Transform, ABC):\n    def file_to_inputs(self, filepath: str, gold=True):\n        filepath = get_resource(filepath)\n        with open(filepath, encoding='utf-8') as src:\n            for line in src:\n                sentence = line.strip()\n                if not sentence:\n                    continue\n                yield sentence\n\n\nclass TxtBMESFormat(TxtFormat, ABC):\n    def file_to_inputs(self, filepath: str, gold=True):\n        max_seq_length = self.config.get('max_seq_length', False)\n        if max_seq_length:\n            if 'transformer' in self.config:\n                max_seq_length -= 2  # allow for [CLS] and [SEP]\n            delimiter = set()\n            delimiter.update('。！？：；、，,;!?、,')\n        for text in super().file_to_inputs(filepath, gold):\n            chars, tags = bmes_of(text, gold)\n            if max_seq_length:\n                start = 0\n                for short_chars in split_long_sent(chars, delimiter, max_seq_length):\n                    end = start + len(short_chars)\n                    yield short_chars, tags[start:end]\n                    start = end\n            else:\n                yield chars, tags\n\n    def input_is_single_sample(self, input: Union[List[str], List[List[str]]]) -> bool:\n        return isinstance(input, str)\n\n    def inputs_to_samples(self, inputs, gold=False):\n        for chars, tags in (inputs if gold else zip(inputs, [None] * len(inputs))):\n            if not gold:\n                tags = [self.tag_vocab.safe_pad_token] * len(chars)\n            chars = CharTable.normalize_chars(chars)\n            yield chars, tags\n\n    def Y_to_outputs(self, Y: Union[tf.Tensor, Tuple[tf.Tensor]], gold=False, inputs=None, X=None,\n                     batch=None) -> Iterable:\n        yield from self.Y_to_tokens(self.tag_vocab, Y, gold, inputs)\n\n    def Y_to_tokens(self, tag_vocab, Y, gold, inputs):\n        if not gold:\n            Y = tf.argmax(Y, axis=2)\n        for text, ys in zip(inputs, Y):\n            tags = [tag_vocab.idx_to_token[int(y)] for y in ys[:len(text)]]\n            yield bmes_to_words(list(text), tags)\n"
  },
  {
    "path": "hanlp/utils/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-08-24 22:12\nfrom . import rules\n\n\ndef ls_resource_in_module(root) -> dict:\n    res = dict()\n    for k, v in root.__dict__.items():\n        if k.startswith('_') or v == root:\n            continue\n        if isinstance(v, str):\n            if v.startswith('http') and not v.endswith('/') and not v.endswith('#') and not v.startswith('_'):\n                res[k] = v\n        elif type(v).__name__ == 'module':\n            res.update(ls_resource_in_module(v))\n    if 'ALL' in root.__dict__ and isinstance(root.__dict__['ALL'], dict):\n        root.__dict__['ALL'].update(res)\n    return res\n"
  },
  {
    "path": "hanlp/utils/component_util.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-31 19:24\nimport os\nfrom hanlp_common.constant import HANLP_VERBOSE\nfrom hanlp_common.io import load_json, eprint, save_json\nfrom hanlp_common.reflection import object_from_classpath, str_to_type\nfrom hanlp import pretrained\nfrom hanlp import version\nfrom hanlp.common.component import Component\nfrom hanlp.utils.io_util import get_resource, get_latest_info_from_pypi, check_version_conflicts\nfrom hanlp_common.util import isdebugging\n\n\ndef load_from_meta_file(save_dir: str, meta_filename='meta.json', transform_only=False, verbose=HANLP_VERBOSE,\n                        **kwargs) -> Component:\n    \"\"\"\n    Load a component from a ``meta.json`` (legacy TensorFlow component) or a ``config.json`` file.\n\n    Args:\n        save_dir: The identifier.\n        meta_filename (str): The meta file of that saved component, which stores the classpath and version.\n        transform_only: Load and return only the transform.\n        **kwargs: Extra parameters passed to ``component.load()``.\n\n    Returns:\n\n        A component.\n    \"\"\"\n    identifier = save_dir\n    load_path = save_dir\n    save_dir = get_resource(save_dir)\n    if save_dir.endswith('.json'):\n        meta_filename = os.path.basename(save_dir)\n        save_dir = os.path.dirname(save_dir)\n    metapath = os.path.join(save_dir, meta_filename)\n    if not os.path.isfile(metapath):\n        tf_model = False\n        metapath = os.path.join(save_dir, 'config.json')\n    else:\n        tf_model = True\n    cls = None\n    if not os.path.isfile(metapath):\n        tips = ''\n        if save_dir.isupper():\n            from difflib import SequenceMatcher\n            similar_keys = sorted(pretrained.ALL.keys(),\n                                  key=lambda k: SequenceMatcher(None, k, identifier).ratio(),\n                                  reverse=True)[:5]\n            tips = f'Check its spelling based on the available keys:\\n' + \\\n                   f'{sorted(pretrained.ALL.keys())}\\n' + \\\n                   f'Tips: it might be one of {similar_keys}'\n        # These components are not intended to be loaded in this way, but I'm tired of explaining it again and again\n        if identifier in pretrained.word2vec.ALL.values():\n            save_dir = os.path.dirname(save_dir)\n            metapath = os.path.join(save_dir, 'config.json')\n            save_json({'classpath': 'hanlp.layers.embeddings.word2vec.Word2VecEmbeddingComponent',\n                       'embed': {'classpath': 'hanlp.layers.embeddings.word2vec.Word2VecEmbedding',\n                                 'embed': identifier, 'field': 'token', 'normalize': 'l2'},\n                       'hanlp_version': version.__version__}, metapath)\n        elif identifier in pretrained.fasttext.ALL.values():\n            save_dir = os.path.dirname(save_dir)\n            metapath = os.path.join(save_dir, 'config.json')\n            save_json({'classpath': 'hanlp.layers.embeddings.fast_text.FastTextEmbeddingComponent',\n                       'embed': {'classpath': 'hanlp.layers.embeddings.fast_text.FastTextEmbedding',\n                                 'filepath': identifier, 'src': 'token'},\n                       'hanlp_version': version.__version__}, metapath)\n        elif identifier in {pretrained.classifiers.LID_176_FASTTEXT_SMALL,\n                            pretrained.classifiers.LID_176_FASTTEXT_BASE}:\n            save_dir = os.path.dirname(save_dir)\n            metapath = os.path.join(save_dir, 'config.json')\n            save_json({'classpath': 'hanlp.components.classifiers.fasttext_classifier.FastTextClassifier',\n                       'model_path': identifier,\n                       'hanlp_version': version.__version__}, metapath)\n        else:\n            raise FileNotFoundError(f'The identifier {save_dir} resolves to a nonexistent meta file {metapath}. {tips}')\n    meta: dict = load_json(metapath)\n    cls = meta.get('classpath', cls)\n    if not cls:\n        cls = meta.get('class_path', None)  # For older version\n    if tf_model:\n        # tf models are trained with version < 2.1. To migrate them to 2.1, map their classpath to new locations\n        upgrade = {\n            'hanlp.components.tok_tf.TransformerTokenizerTF': 'hanlp.components.tokenizers.tok_tf.TransformerTokenizerTF',\n            'hanlp.components.pos.RNNPartOfSpeechTagger': 'hanlp.components.taggers.pos_tf.RNNPartOfSpeechTaggerTF',\n            'hanlp.components.pos_tf.RNNPartOfSpeechTaggerTF': 'hanlp.components.taggers.pos_tf.RNNPartOfSpeechTaggerTF',\n            'hanlp.components.pos_tf.CNNPartOfSpeechTaggerTF': 'hanlp.components.taggers.pos_tf.CNNPartOfSpeechTaggerTF',\n            'hanlp.components.ner_tf.TransformerNamedEntityRecognizerTF': 'hanlp.components.ner.ner_tf.TransformerNamedEntityRecognizerTF',\n            'hanlp.components.parsers.biaffine_parser.BiaffineDependencyParser': 'hanlp.components.parsers.biaffine_parser_tf.BiaffineDependencyParserTF',\n            'hanlp.components.parsers.biaffine_parser.BiaffineSemanticDependencyParser': 'hanlp.components.parsers.biaffine_parser_tf.BiaffineSemanticDependencyParserTF',\n            'hanlp.components.tok_tf.NgramConvTokenizerTF': 'hanlp.components.tokenizers.tok_tf.NgramConvTokenizerTF',\n            'hanlp.components.classifiers.transformer_classifier.TransformerClassifier': 'hanlp.components.classifiers.transformer_classifier_tf.TransformerClassifierTF',\n            'hanlp.components.taggers.transformers.transformer_tagger.TransformerTagger': 'hanlp.components.taggers.transformers.transformer_tagger_tf.TransformerTaggerTF',\n            'hanlp.components.tok.NgramConvTokenizer': 'hanlp.components.tokenizers.tok_tf.NgramConvTokenizerTF',\n        }\n        cls = upgrade.get(cls, cls)\n    assert cls, f'{meta_filename} doesn\\'t contain classpath field'\n    try:\n        obj: Component = object_from_classpath(cls)\n        if hasattr(obj, 'load'):\n            if transform_only:\n                # noinspection PyUnresolvedReferences\n                obj.load_transform(save_dir)\n            else:\n                if os.path.isfile(os.path.join(save_dir, 'config.json')):\n                    obj.load(save_dir, verbose=verbose, **kwargs)\n                else:\n                    obj.load(metapath, **kwargs)\n            obj.config['load_path'] = load_path\n        return obj\n    except ModuleNotFoundError as e:\n        if isdebugging():\n            raise e from None\n        else:\n            raise ModuleNotFoundError(\n                f'Some modules ({e.name} etc.) required by this model are missing. Please install the full version:'\n                '\\n\\n\\tpip install hanlp[full] -U') from None\n    except ValueError as e:\n        if e.args and isinstance(e.args[0], str) and 'Internet connection' in e.args[0]:\n            raise ConnectionError(\n                'Hugging Face 🤗 Transformers failed to download because your Internet connection is either off or bad.\\n'\n                'See https://hanlp.hankcs.com/docs/install.html#server-without-internet for solutions.') \\\n                from None\n        raise e from None\n    except Exception as e:\n        # Some users often install an incompatible tf and put the blame on HanLP. Teach them the basics.\n        try:\n            you_installed_wrong_versions, extras = check_version_conflicts(extras=('full',) if tf_model else None)\n        except Exception as check_e:\n            you_installed_wrong_versions, extras = None, None\n        if you_installed_wrong_versions:\n            raise version.NotCompatible(you_installed_wrong_versions + '\\nPlease reinstall HanLP in the proper way:' +\n                                        '\\n\\n\\tpip install --upgrade hanlp' + (\n                                            f'[{\",\".join(extras)}]' if extras else '')) from None\n        eprint(f'Failed to load {identifier}')\n        from pkg_resources import parse_version\n        model_version = meta.get(\"hanlp_version\", '2.0.0-alpha.0')\n        if model_version == '2.0.0':  # Quick fix: the first version used a wrong string\n            model_version = '2.0.0-alpha.0'\n        model_version = parse_version(model_version)\n        installed_version = parse_version(version.__version__)\n        try:\n            latest_version = get_latest_info_from_pypi()\n        except:\n            latest_version = None\n        if model_version > installed_version:\n            eprint(f'{identifier} was created with hanlp-{model_version}, '\n                   f'while you are running a lower version: {installed_version}. ')\n        if installed_version != latest_version:\n            eprint(\n                f'Please upgrade HanLP with:\\n'\n                f'\\n\\tpip install --upgrade hanlp\\n')\n        eprint(\n            'If the problem still persists, please submit an issue to https://github.com/hankcs/HanLP/issues\\n'\n            'When reporting an issue, make sure to paste the FULL ERROR LOG below.')\n\n        eprint(f'{\"ERROR LOG BEGINS\":=^80}')\n        import platform\n        eprint(f'OS: {platform.platform()}')\n        eprint(f'Python: {platform.python_version()}')\n        import torch\n        eprint(f'PyTorch: {torch.__version__}')\n        if tf_model:\n            try:\n                import tensorflow\n                tf_version = tensorflow.__version__\n                eprint(f'TensorFlow: {tf_version}')\n            except ModuleNotFoundError:\n                tf_version = 'not installed'\n                eprint(f'TensorFlow: {tf_version}')\n            except Exception as tf_e:\n                eprint(f'TensorFlow cannot be imported due to {tf_e.__class__.__name__}: {e}. '\n                       f'Note this is not a bug of HanLP, but rather a compatability issue caused by TensorFlow.')\n        eprint(f'HanLP: {version.__version__}')\n        import sys\n        sys.stderr.flush()\n        try:\n            if e.args and isinstance(e.args, tuple):\n                for i in range(len(e.args)):\n                    if isinstance(e.args[i], str):\n                        from hanlp_common.util import set_tuple_with\n                        e.args = set_tuple_with(e.args, e.args[i] + f'\\n{\"ERROR LOG ENDS\":=^80}', i)\n                        break\n        except:\n            pass\n        raise e from None\n\n\ndef load_from_meta(meta: dict) -> Component:\n    if 'load_path' in meta:\n        return load_from_meta_file(meta['load_path'])\n    cls = meta.get('class_path', None) or meta.get('classpath', None)\n    assert cls, f'{meta} doesn\\'t contain classpath field'\n    cls = str_to_type(cls)\n    return cls.from_config(meta)\n"
  },
  {
    "path": "hanlp/utils/file_read_backwards/__init__.py",
    "content": "# -*- coding: utf-8 -*-\n\nfrom .file_read_backwards import FileReadBackwards  # noqa: F401\n\n__author__ = \"\"\"Robin Robin\"\"\"\n__email__ = 'robinsquare42@gmail.com'\n__version__ = '2.0.0'\n"
  },
  {
    "path": "hanlp/utils/file_read_backwards/buffer_work_space.py",
    "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n\n\"\"\"BufferWorkSpace module.\"\"\"\n\nimport os\n\nnew_lines = [\"\\r\\n\", \"\\n\", \"\\r\"]\nnew_lines_bytes = [n.encode(\"ascii\") for n in new_lines]  # we only support encodings that's backward compat with ascii\n\n\nclass BufferWorkSpace:\n\n    \"\"\"It is a helper module for FileReadBackwards.\"\"\"\n\n    def __init__(self, fp, chunk_size):\n        \"\"\"Convention for the data.\n\n        When read_buffer is not None, it represents contents of the file from `read_position` onwards\n            that has not been processed/returned.\n        read_position represents the file pointer position that has been read into read_buffer\n            initialized to be just past the end of file.\n        \"\"\"\n        self.fp = fp\n        self.read_position = _get_file_size(self.fp)  # set the previously read position to the\n        self.read_buffer = None\n        self.chunk_size = chunk_size\n\n    def add_to_buffer(self, content, read_position):\n        \"\"\"Add additional bytes content as read from the read_position.\n\n        Args:\n          content(bytes): data to be added to buffer working BufferWorkSpac.\n          read_position(int): where in the file pointer the data was read from.\n\n        Returns:\n\n        \"\"\"\n        self.read_position = read_position\n        if self.read_buffer is None:\n            self.read_buffer = content\n        else:\n            self.read_buffer = content + self.read_buffer\n\n    def yieldable(self):\n        \"\"\" \"\"\"\n        if self.read_buffer is None:\n            return False\n\n        t = _remove_trailing_new_line(self.read_buffer)\n        n = _find_furthest_new_line(t)\n        if n >= 0:\n            return True\n\n        # we have read in entire file and have some unprocessed lines\n        if self.read_position == 0 and self.read_buffer is not None:\n            return True\n        return False\n\n    def return_line(self):\n        \"\"\"\n\n        Args:\n\n        Returns:\n          Precondition: self.yieldable() must be True\n\n        \"\"\"\n        assert(self.yieldable())\n\n        t = _remove_trailing_new_line(self.read_buffer)\n        i = _find_furthest_new_line(t)\n\n        if i >= 0:\n            l = i + 1\n            after_new_line = slice(l, None)\n            up_to_include_new_line = slice(0, l)\n            r = t[after_new_line]\n            self.read_buffer = t[up_to_include_new_line]\n        else:  # the case where we have read in entire file and at the \"last\" line\n            r = t\n            self.read_buffer = None\n        return r\n\n    def read_until_yieldable(self):\n        \"\"\"Read in additional chunks until it is yieldable.\"\"\"\n        while not self.yieldable():\n            read_content, read_position = _get_next_chunk(self.fp, self.read_position, self.chunk_size)\n            self.add_to_buffer(read_content, read_position)\n\n    def has_returned_every_line(self):\n        \"\"\" \"\"\"\n        if self.read_position == 0 and self.read_buffer is None:\n            return True\n        return False\n\n\ndef _get_file_size(fp):\n    return os.fstat(fp.fileno()).st_size\n\n\ndef _get_next_chunk(fp, previously_read_position, chunk_size):\n    \"\"\"Return next chunk of data that we would from the file pointer.\n\n    Args:\n      fp: file\n      previously_read_position: file pointer position that we have read from\n      chunk_size: desired read chunk_size\n\n    Returns:\n      (bytestring, int): data that has been read in, the file pointer position where the data has been read from\n\n    \"\"\"\n    seek_position, read_size = _get_what_to_read_next(fp, previously_read_position, chunk_size)\n    fp.seek(seek_position)\n    read_content = fp.read(read_size)\n    read_position = seek_position\n    return read_content, read_position\n\n\ndef _get_what_to_read_next(fp, previously_read_position, chunk_size):\n    \"\"\"Return information on which file pointer position to read from and how many bytes.\n\n    Args:\n      fp: \n      past_read_positon: int\n      chunk_size: int\n      previously_read_position: \n\n    Returns:\n      (int, int): The next seek position, how many bytes to read next\n\n    \"\"\"\n    seek_position = max(previously_read_position - chunk_size, 0)\n    read_size = chunk_size\n\n    # examples: say, our new_lines are potentially \"\\r\\n\", \"\\n\", \"\\r\"\n    # find a reading point where it is not \"\\n\", rewind further if necessary\n    # if we have \"\\r\\n\" and we read in \"\\n\",\n    # the next iteration would treat \"\\r\" as a different new line.\n    # Q: why don't I just check if it is b\"\\n\", but use a function ?\n    # A: so that we can potentially expand this into generic sets of separators, later on.\n    while seek_position > 0:\n        fp.seek(seek_position)\n        if _is_partially_read_new_line(fp.read(1)):\n            seek_position -= 1\n            read_size += 1  # as we rewind further, let's make sure we read more to compensate\n        else:\n            break\n\n    # take care of special case when we are back to the beginnin of the file\n    read_size = min(previously_read_position - seek_position, read_size)\n    return seek_position, read_size\n\n\ndef _remove_trailing_new_line(l):\n    \"\"\"Remove a single instance of new line at the end of l if it exists.\n\n    Args:\n      l: \n\n    Returns:\n      : bytestring\n\n    \"\"\"\n    # replace only 1 instance of newline\n    # match longest line first (hence the reverse=True), we want to match \"\\r\\n\" rather than \"\\n\" if we can\n    for n in sorted(new_lines_bytes, key=lambda x: len(x), reverse=True):\n        if l.endswith(n):\n            remove_new_line = slice(None, -len(n))\n            return l[remove_new_line]\n    return l\n\n\ndef _find_furthest_new_line(read_buffer):\n    \"\"\"Return -1 if read_buffer does not contain new line otherwise the position of the rightmost newline.\n\n    Args:\n      read_buffer: bytestring\n\n    Returns:\n      int: The right most position of new line character in read_buffer if found, else -1\n\n    \"\"\"\n    new_line_positions = [read_buffer.rfind(n) for n in new_lines_bytes]\n    return max(new_line_positions)\n\n\ndef _is_partially_read_new_line(b):\n    \"\"\"Return True when b is part of a new line separator found at index >= 1, False otherwise.\n\n    Args:\n      b: bytestring\n\n    Returns:\n      bool\n\n    \"\"\"\n    for n in new_lines_bytes:\n        if n.find(b) >= 1:\n            return True\n    return False\n"
  },
  {
    "path": "hanlp/utils/file_read_backwards/file_read_backwards.py",
    "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n\n\"\"\"FileReadBackwards module.\"\"\"\n\nimport io\nimport os\n\nfrom .buffer_work_space import BufferWorkSpace\n\nsupported_encodings = [\"utf-8\", \"ascii\", \"latin-1\"]  # any encodings that are backward compatible with ascii should work\n\n\nclass FileReadBackwards:\n\n    \"\"\"Class definition for `FileReadBackwards`.\n    \n    A `FileReadBackwards` will spawn a `FileReadBackwardsIterator` and keep an opened file handler.\n    \n    It can be used as a Context Manager. If done so, when exited, it will close its file handler.\n    \n    In any mode, `close()` can be called to close the file handler..\n\n    Args:\n\n    Returns:\n\n    \"\"\"\n\n    def __init__(self, path, encoding=\"utf-8\", chunk_size=io.DEFAULT_BUFFER_SIZE):\n        \"\"\"Constructor for FileReadBackwards.\n\n        Args:\n            path: Path to the file to be read\n            encoding (str): Encoding\n            chunk_size (int): How many bytes to read at a time\n        \"\"\"\n        if encoding.lower() not in supported_encodings:\n            error_message = \"{0} encoding was not supported/tested.\".format(encoding)\n            error_message += \"Supported encodings are '{0}'\".format(\",\".join(supported_encodings))\n            raise NotImplementedError(error_message)\n\n        self.path = path\n        self.encoding = encoding.lower()\n        self.chunk_size = chunk_size\n        self.iterator = FileReadBackwardsIterator(io.open(self.path, mode=\"rb\"), self.encoding, self.chunk_size)\n\n    def __iter__(self):\n        \"\"\"Return its iterator.\"\"\"\n        return self.iterator\n\n    def __enter__(self):\n        return self\n\n    def __exit__(self, exc_type, exc_val, exc_tb):\n        \"\"\"Closes all opened its file handler and propagates all exceptions on exit.\"\"\"\n        self.close()\n        return False\n\n    def close(self):\n        \"\"\"Closes all opened it s file handler.\"\"\"\n        self.iterator.close()\n\n    def readline(self):\n        \"\"\" \"\"\"\n\n        try:\n            r = next(self.iterator) + os.linesep\n            return r\n        except StopIteration:\n            return \"\"\n\n\nclass FileReadBackwardsIterator:\n    \"\"\"Iterator for `FileReadBackwards`.\n    \n    This will read backwards line by line a file. It holds an opened file handler.\n\n    Args:\n\n    Returns:\n\n    \"\"\"\n    def __init__(self, fp, encoding, chunk_size):\n        \"\"\"Constructor for FileReadBackwardsIterator\n\n        Args:\n            fp (File): A file that we wish to start reading backwards from\n            encoding (str): Encoding of the file\n            chunk_size (int): How many bytes to read at a time\n        \"\"\"\n        self.path = fp.name\n        self.encoding = encoding\n        self.chunk_size = chunk_size\n        self.__fp = fp\n        self.__buf = BufferWorkSpace(self.__fp, self.chunk_size)\n\n    def __iter__(self):\n        return self\n\n    def next(self):\n        \"\"\"Returns unicode string from the last line until the beginning of file.\n        \n        Gets exhausted if::\n        \n            * already reached the beginning of the file on previous iteration\n            * the file got closed\n        \n        When it gets exhausted, it closes the file handler.\n\n        Args:\n\n        Returns:\n\n        \"\"\"\n        # Using binary mode, because some encodings such as \"utf-8\" use variable number of\n        # bytes to encode different Unicode points.\n        # Without using binary mode, we would probably need to understand each encoding more\n        # and do the seek operations to find the proper boundary before issuing read\n        if self.closed:\n            raise StopIteration\n        if self.__buf.has_returned_every_line():\n            self.close()\n            raise StopIteration\n        self.__buf.read_until_yieldable()\n        r = self.__buf.return_line()\n        return r.decode(self.encoding)\n\n    __next__ = next\n\n    @property\n    def closed(self):\n        \"\"\"The status of the file handler.\n        \n        :return: True if the file handler is still opened. False otherwise.\n\n        Args:\n\n        Returns:\n\n        \"\"\"\n        return self.__fp.closed\n\n    def close(self):\n        \"\"\"Closes the file handler.\"\"\"\n        self.__fp.close()\n"
  },
  {
    "path": "hanlp/utils/init_util.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-27 13:25\nimport math\n\nimport torch\nfrom torch import nn\nimport functools\n\n\ndef embedding_uniform(tensor:torch.Tensor, seed=233):\n    gen = torch.Generator().manual_seed(seed)\n    with torch.no_grad():\n        fan_out = tensor.size(-1)\n        bound = math.sqrt(3.0 / fan_out)\n        return tensor.uniform_(-bound, bound, generator=gen)\n"
  },
  {
    "path": "hanlp/utils/io_util.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-08-26 15:02\nimport contextlib\nimport glob\nimport gzip\nimport json\nimport logging\nimport os\nimport platform\nimport random\nimport shlex\nimport shutil\nimport sys\nimport tarfile\nimport tempfile\nimport urllib\nimport zipfile\nfrom contextlib import contextmanager\nfrom pathlib import Path\nfrom subprocess import Popen, PIPE\nfrom typing import Tuple, Optional, Union, List\nfrom urllib.parse import urlparse\nfrom urllib.request import urlretrieve\n\nfrom hanlp_downloader import Downloader\nfrom hanlp_downloader.log import DownloadCallback\nfrom packaging.version import Version\n\nimport hanlp\nfrom hanlp_common.constant import HANLP_URL, HANLP_VERBOSE\nfrom hanlp.utils.log_util import logger, cprint, remove_color_tag\nfrom hanlp.utils.string_util import split_long_sentence_into\nfrom hanlp.utils.time_util import now_filename, CountdownTimer\nfrom hanlp.version import __version__\nfrom hanlp_common.io import eprint\n\n\ndef load_jsonl(path, verbose=False):\n    if verbose:\n        src = TimingFileIterator(path)\n    else:\n        src = open(path, encoding='utf-8')\n    for line in src:\n        yield json.loads(line)\n    if not verbose:\n        src.close()\n\n\ndef make_debug_corpus(path, delimiter=None, percentage=0.1, max_samples=100):\n    files = []\n    if os.path.isfile(path):\n        files.append(path)\n    elif os.path.isdir(path):\n        files += [os.path.join(path, f) for f in os.listdir(path) if\n                  os.path.isfile(os.path.join(path, f)) and '.debug' not in f and not f.startswith('.')]\n    else:\n        raise FileNotFoundError(path)\n    for filepath in files:\n        filename, file_extension = os.path.splitext(filepath)\n        if not delimiter:\n            if file_extension in {'.tsv', '.conll', '.conllx', '.conllu'}:\n                delimiter = '\\n\\n'\n            else:\n                delimiter = '\\n'\n        with open(filepath, encoding='utf-8') as src, open(filename + '.debug' + file_extension, 'w',\n                                                           encoding='utf-8') as out:\n            samples = src.read().strip().split(delimiter)\n            max_samples = min(max_samples, int(len(samples) * percentage))\n            out.write(delimiter.join(samples[:max_samples]))\n\n\ndef path_join(path, *paths):\n    return os.path.join(path, *paths)\n\n\ndef makedirs(path):\n    os.makedirs(path, exist_ok=True)\n    return path\n\n\ndef tempdir(name=None):\n    path = tempfile.gettempdir()\n    if name:\n        path = makedirs(path_join(path, name))\n    return path\n\n\ndef tempdir_human():\n    return tempdir(now_filename())\n\n\ndef temp_lock(path):\n    from filelock import FileLock\n    import hashlib\n    lock = FileLock(f\"{tempdir()}/.{hashlib.md5(path.encode('utf8')).hexdigest()}.lock\")\n    return lock\n\n\ndef hanlp_home_default():\n    \"\"\"Default data directory depending on the platform and environment variables\"\"\"\n    if windows():\n        return os.path.join(os.environ.get('APPDATA'), 'hanlp')\n    else:\n        return os.path.join(os.path.expanduser(\"~\"), '.hanlp')\n\n\ndef windows():\n    system = platform.system()\n    return system == 'Windows'\n\n\ndef hanlp_home():\n    \"\"\" Home directory for HanLP resources.\n\n    Returns:\n        Data directory in the filesystem for storage, for example when downloading models.\n\n    This home directory can be customized with the following shell command or equivalent environment variable on Windows\n    systems.\n\n    .. highlight:: bash\n    .. code-block:: bash\n\n        $ export HANLP_HOME=/data/hanlp\n\n    \"\"\"\n    return os.getenv('HANLP_HOME', hanlp_home_default())\n\n\ndef file_exist(filename) -> bool:\n    return os.path.isfile(filename)\n\n\ndef remove_file(filename):\n    if file_exist(filename):\n        os.remove(filename)\n\n\ndef parent_dir(path):\n    return os.path.normpath(os.path.join(path, os.pardir))\n\n\ndef download(url, save_path=None, save_dir=hanlp_home(), prefix=HANLP_URL, append_location=True, verbose=HANLP_VERBOSE):\n    if not save_path:\n        save_path = path_from_url(url, save_dir, prefix, append_location)\n    if os.path.isfile(save_path):\n        if verbose:\n            eprint('Using local {}, ignore {}'.format(save_path, url))\n        return save_path\n    else:\n        makedirs(parent_dir(save_path))\n        if verbose:\n            eprint('Downloading {} to {}'.format(url, save_path))\n        tmp_path = '{}.downloading'.format(save_path)\n        remove_file(tmp_path)\n        try:\n            downloader = Downloader(url, tmp_path, 4, headers={\n                'User-agent': f'HanLP/{__version__} ({platform.platform()})'})\n            if verbose:\n                downloader.subscribe(DownloadCallback(show_header=False))\n            downloader.start_sync()\n        except BaseException as e:\n            remove_file(tmp_path)\n            url = url.split('#')[0]\n            try:\n                installed_version, latest_version = check_outdated()\n            except:\n                installed_version, latest_version = None, None  # No Internet\n            if installed_version != latest_version:\n                # Always prompt user to upgrade whenever a new version is available\n                hints = f'[green]Please upgrade to the latest version ({latest_version}) with:[/green]' \\\n                        f'\\n\\n\\t[yellow]pip install -U hanlp[/yellow]\\n'\n            else:  # Otherwise, prompt user to re-try\n                hints = f'[green]Please re-try or download it to {save_path} by yourself '\n                if not windows():\n                    hints += f'with:[/green]\\n\\n\\t[yellow]wget {url} -O {save_path}[/yellow]\\n\\n'\n                else:\n                    hints += 'using some decent downloading tools.[/green]\\n'\n                if not url.startswith(HANLP_URL):\n                    hints += 'For third party data, unrestricted connectivity to the global network may be required.'\n                else:\n                    hints += 'See also https://hanlp.hankcs.com/docs/install.html#install-models for instructions.'\n            message = f'Download failed due to [red]{repr(e)}[/red].\\n' \\\n                      f'{hints}'\n            if verbose:\n                cprint(message)\n            if hasattr(e, 'msg'):\n                e.msg += '\\n' + remove_color_tag(message)\n            elif hasattr(e, 'args') and e.args and isinstance(e.args, tuple) and isinstance(e.args[0], str):\n                e.args = (e.args[0] + '\\n' + remove_color_tag(message),) + e.args[1:]\n            raise e from None\n        remove_file(save_path)\n        os.rename(tmp_path, save_path)\n    return save_path\n\n\ndef parse_url_path(url):\n    parsed: urllib.parse.ParseResult = urlparse(url)\n    path = parsed.path.strip('/')\n    return parsed.netloc, path\n\n\ndef uncompress(path, dest=None, remove=True, verbose=HANLP_VERBOSE):\n    \"\"\"Uncompress a file and clean up uncompressed files once an error is triggered.\n\n    Args:\n      path: The path to a compressed file\n      dest: The dest folder.\n      remove: Remove archive file after decompression.\n      verbose: ``True`` to print log message.\n\n    Returns:\n        Destination path.\n    \n    \"\"\"\n    # assert path.endswith('.zip')\n    prefix, ext = split_if_compressed(path)\n    folder_name = os.path.basename(prefix)\n    file_is_zip = ext == '.zip'\n    root_of_folder = None\n    if ext == '.gz':\n        try:\n            with gzip.open(path, 'rb') as f_in, open(prefix, 'wb') as f_out:\n                shutil.copyfileobj(f_in, f_out)\n        except Exception as e:\n            remove_file(prefix)\n            remove_file(path)\n            raise e\n    else:\n        try:\n            with zipfile.ZipFile(path, \"r\") if ext == '.zip' else tarfile.open(path, 'r:*') as archive:\n                if not dest:\n                    namelist = sorted(archive.namelist() if file_is_zip else archive.getnames())\n                    if namelist[0] == '.':\n                        namelist = namelist[1:]\n                        namelist = [p[len('./'):] if p.startswith('./') else p for p in namelist]\n                    if ext == '.tgz':\n                        roots = set(x.split('/')[0] for x in namelist)\n                        if len(roots) == 1:\n                            root_of_folder = next(iter(roots))\n                    else:\n                        # only one file, root_of_folder = ''\n                        root_of_folder = namelist[0].strip('/') if len(namelist) > 1 else ''\n                    if all(f.split('/')[0] == root_of_folder for f in namelist[1:]) or not root_of_folder:\n                        dest = os.path.dirname(path)  # only one folder, unzip to the same dir\n                    else:\n                        root_of_folder = None\n                        dest = prefix  # assume zip contains more than one file or folder\n                if verbose:\n                    eprint('Decompressing {} to {}'.format(path, dest))\n                archive.extractall(dest)\n                if root_of_folder:\n                    if root_of_folder != folder_name:\n                        # move root to match folder name\n                        os.rename(path_join(dest, root_of_folder), path_join(dest, folder_name))\n                    dest = path_join(dest, folder_name)\n                elif len(namelist) == 1:\n                    dest = path_join(dest, namelist[0])\n        except Exception as e:\n            remove_file(path)\n            if os.path.exists(prefix):\n                if os.path.isfile(prefix):\n                    os.remove(prefix)\n                elif os.path.isdir(prefix):\n                    shutil.rmtree(prefix)\n            raise e\n    if remove:\n        remove_file(path)\n    return dest\n\n\ndef split_if_compressed(path: str, compressed_ext=('.zip', '.tgz', '.gz', 'bz2', '.xz')) -> Tuple[str, Optional[str]]:\n    tar_gz = '.tar.gz'\n    if path.endswith(tar_gz):\n        root, ext = path[:-len(tar_gz)], tar_gz\n    else:\n        root, ext = os.path.splitext(path)\n    if ext in compressed_ext or ext == tar_gz:\n        return root, ext\n    return path, None\n\n\ndef get_resource(path: str, save_dir=hanlp_home(), extract=True, prefix=HANLP_URL, append_location=True,\n                 verbose=HANLP_VERBOSE):\n    \"\"\"Fetch real (local) path for a resource (model, corpus, whatever) to ``save_dir``.\n\n    Args:\n      path: A local path (which will returned as is) or a remote URL (which will be downloaded, decompressed then\n        returned).\n      save_dir: Where to store the resource (Default value = :meth:`hanlp.utils.io_util.hanlp_home`)\n      extract: Whether to unzip it if it's a zip file (Default value = True)\n      prefix: A prefix when matched with an URL (path), then that URL is considered to be official. For official\n        resources, they will not go to a folder called ``thirdparty`` under :const:`~hanlp_common.constants.HANLP_HOME`.\n      append_location: Whether to put unofficial files in a ``thirdparty`` folder.\n      verbose: Whether to print log messages.\n\n    Returns:\n      The real path to the resource.\n\n    \"\"\"\n    _path = path\n    path = hanlp.pretrained.ALL.get(path, path)\n    anchor: str = None\n    compressed = None\n    if os.path.isdir(path):\n        return path\n    elif os.path.isfile(path):\n        pass\n    elif path.startswith('http:') or path.startswith('https:'):\n        url = path\n        if '#' in url:\n            url, anchor = url.split('#', maxsplit=1)\n        realpath = path_from_url(path, save_dir, prefix, append_location)\n        realpath, compressed = split_if_compressed(realpath)\n        # check if resource is there\n        if anchor:\n            if anchor.startswith('/'):\n                # indicates the folder name has to be polished\n                anchor = anchor.lstrip('/')\n                parts = anchor.split('/')\n                renamed_realpath = str(Path(realpath).parent.joinpath(parts[0]))\n                if os.path.isfile(realpath + compressed):\n                    os.rename(realpath + compressed, renamed_realpath + compressed)\n                realpath = renamed_realpath\n                anchor = '/'.join(parts[1:])\n            child = path_join(realpath, anchor)\n            if os.path.exists(child):\n                return child\n        elif os.path.isdir(realpath) or (os.path.isfile(realpath) and (compressed and extract)):\n            return realpath\n        else:\n            if compressed:\n                pattern = realpath + '.*'\n                files = glob.glob(pattern)\n                files = list(filter(lambda x: not x.endswith('.downloading') and not x.endswith(compressed), files))\n                if files:\n                    if len(files) > 1:\n                        logger.debug(f'Found multiple files with {pattern}, will use the first one.')\n                    return files[0]\n        # realpath is where its path after exaction\n        if compressed:\n            realpath += compressed\n        with temp_lock(path):\n            if not os.path.isfile(realpath):\n                path = download(url=path, save_path=realpath, verbose=verbose)\n            else:\n                path = realpath\n    if extract and compressed:\n        with temp_lock(path):\n            if os.path.isfile(path):\n                path = uncompress(path, verbose=verbose)\n            else:  # other process must have already decompressed it and deleted it\n                return get_resource(_path, save_dir, extract, prefix, append_location, verbose)\n        if anchor:\n            path = path_join(path, anchor)\n\n    return path\n\n\ndef path_from_url(url, save_dir=hanlp_home(), prefix=HANLP_URL, append_location=True):\n    \"\"\"Map a URL to a local path.\n\n    Args:\n        url: Remote URL.\n        save_dir: The root folder to save this file.\n        prefix: The prefix of official website. Any URLs starting with this prefix will be considered official.\n        append_location: Whether to put unofficial files in a ``thirdparty`` folder.\n\n    Returns:\n        The real path that this URL is mapped to.\n    \"\"\"\n    if not save_dir:\n        save_dir = hanlp_home()\n    domain, relative_path = parse_url_path(url)\n    if append_location:\n        if not url.startswith(prefix):\n            save_dir = os.path.join(save_dir, 'thirdparty', domain)\n        else:\n            # remove the relative path in prefix\n            middle = prefix.split(domain)[-1].lstrip('/')\n            if relative_path.startswith(middle):\n                relative_path = relative_path[len(middle):]\n        realpath = os.path.join(save_dir, relative_path)\n    else:\n        realpath = os.path.join(save_dir, os.path.basename(relative_path))\n    return realpath\n\n\ndef human_bytes(file_size: int) -> str:\n    file_size /= 1024  # KB\n    if file_size > 1024:\n        file_size /= 1024  # MB\n        if file_size > 1024:\n            file_size /= 1024  # GB\n            return '%.1f GB' % file_size\n        return '%.1f MB' % file_size\n    return '%d KB' % file_size\n\n\ndef read_cells(filepath: str, delimiter='auto', strip=True, skip_header=False):\n    filepath = get_resource(filepath)\n    if delimiter == 'auto':\n        if filepath.endswith('.tsv'):\n            delimiter = '\\t'\n        elif filepath.endswith('.csv'):\n            delimiter = ','\n        else:\n            delimiter = None\n    with open(filepath, encoding='utf-8') as src:\n        if skip_header:\n            next(src)\n        for line in src:\n            line = line.strip()\n            if not line:\n                continue\n            cells = line.split(delimiter)\n            if strip:\n                cells = [c.strip() for c in cells]\n                yield cells\n\n\ndef replace_ext(filepath, ext) -> str:\n    \"\"\" Replace the extension of filepath to ext.\n\n    Args:\n        filepath: Filepath to be replaced.\n        ext: Extension to replace.\n\n    Returns:\n        A new path.\n    \"\"\"\n    file_prefix, _ = os.path.splitext(filepath)\n    return file_prefix + ext\n\n\ndef read_tsv_as_sents(tsv_file_path, ignore_prefix=None, delimiter=None):\n    sent = []\n    tsv_file_path = get_resource(tsv_file_path)\n    with open(tsv_file_path, encoding='utf-8') as tsv_file:\n        for line in tsv_file:\n            if ignore_prefix and line.startswith(ignore_prefix):\n                continue\n            line = line.strip()\n            cells = line.split(delimiter)\n            if line and cells:\n                sent.append(cells)\n            elif sent:\n                yield sent\n                sent = []\n    if sent:\n        yield sent\n\n\ndef generate_words_tags_from_tsv(tsv_file_path, lower=False, gold=True, max_seq_length=None, sent_delimiter=None,\n                                 char_level=False, hard_constraint=False):\n    for sent in read_tsv_as_sents(tsv_file_path):\n        words = [cells[0] for cells in sent]\n        if max_seq_length:\n            offset = 0\n            # try to split the sequence to make it fit into max_seq_length\n            for shorter_words in split_long_sentence_into(words, max_seq_length, sent_delimiter, char_level,\n                                                          hard_constraint):\n                if gold:\n                    shorter_tags = [cells[1] for cells in sent[offset:offset + len(shorter_words)]]\n                    offset += len(shorter_words)\n                else:\n                    shorter_tags = None\n                if lower:\n                    shorter_words = [word.lower() for word in shorter_words]\n                yield shorter_words, shorter_tags\n        else:\n            if gold:\n                try:\n                    tags = [cells[1] for cells in sent]\n                except:\n                    raise ValueError(f'Failed to load {tsv_file_path}: {sent}')\n            else:\n                tags = None\n            if lower:\n                words = [word.lower() for word in words]\n            yield words, tags\n\n\ndef split_file(filepath, train=0.8, dev=0.1, test=0.1, names=None, shuffle=False):\n    num_samples = 0\n    if filepath.endswith('.tsv'):\n        for sent in read_tsv_as_sents(filepath):\n            num_samples += 1\n    else:\n        with open(filepath, encoding='utf-8') as src:\n            for sample in src:\n                num_samples += 1\n    splits = {'train': train, 'dev': dev, 'test': test}\n    splits = dict((k, v) for k, v in splits.items() if v)\n    splits = dict((k, v / sum(splits.values())) for k, v in splits.items())\n    accumulated = 0\n    r = []\n    for k, v in splits.items():\n        r.append(accumulated)\n        accumulated += v\n        r.append(accumulated)\n        splits[k] = accumulated\n    if names is None:\n        names = {}\n    name, ext = os.path.splitext(filepath)\n    filenames = [names.get(split, name + '.' + split + ext) for split in splits.keys()]\n    outs = [open(f, 'w', encoding='utf-8') for f in filenames]\n    if shuffle:\n        shuffle = list(range(num_samples))\n        random.shuffle(shuffle)\n    if filepath.endswith('.tsv'):\n        src = read_tsv_as_sents(filepath)\n    else:\n        src = open(filepath, encoding='utf-8')\n    for idx, sample in enumerate(src):\n        if shuffle:\n            idx = shuffle[idx]\n        ratio = idx / num_samples\n        for sid, out in enumerate(outs):\n            if r[2 * sid] <= ratio < r[2 * sid + 1]:\n                if isinstance(sample, list):\n                    sample = '\\n'.join('\\t'.join(x) for x in sample) + '\\n\\n'\n                out.write(sample)\n                break\n    if not filepath.endswith('.tsv'):\n        src.close()\n    for out in outs:\n        out.close()\n    return filenames\n\n\ndef fileno(file_or_fd):\n    try:\n        fd = getattr(file_or_fd, 'fileno', lambda: file_or_fd)()\n    except:\n        return None\n    if not isinstance(fd, int):\n        raise ValueError(\"Expected a file (`.fileno()`) or a file descriptor\")\n    return fd\n\n\n@contextmanager\ndef stdout_redirected(to=os.devnull, stdout=None):\n    \"\"\"Redirect stdout to else where.\n    Copied from https://stackoverflow.com/questions/4675728/redirect-stdout-to-a-file-in-python/22434262#22434262\n\n    Args:\n      to:  Target device.\n      stdout:  Source device.\n\n    \"\"\"\n    if windows():  # This doesn't play well with windows\n        yield None\n        return\n    if stdout is None:\n        stdout = sys.stdout\n    stdout_fd = fileno(stdout)\n    if not stdout_fd:\n        yield None\n        return\n        # copy stdout_fd before it is overwritten\n    # NOTE: `copied` is inheritable on Windows when duplicating a standard stream\n    with os.fdopen(os.dup(stdout_fd), 'wb') as copied:\n        stdout.flush()  # flush library buffers that dup2 knows nothing about\n        try:\n            os.dup2(fileno(to), stdout_fd)  # $ exec >&to\n        except ValueError:  # filename\n            with open(to, 'wb') as to_file:\n                os.dup2(to_file.fileno(), stdout_fd)  # $ exec > to\n        try:\n            yield stdout  # allow code to be run with the redirected stdout\n        finally:\n            # restore stdout to its previous value\n            # NOTE: dup2 makes stdout_fd inheritable unconditionally\n            try:\n                stdout.flush()\n                os.dup2(copied.fileno(), stdout_fd)  # $ exec >&copied\n            except:\n                # This is the best we can do\n                pass\n\n\ndef get_exitcode_stdout_stderr(cmd):\n    \"\"\"Execute the external command and get its exitcode, stdout and stderr.\n    See https://stackoverflow.com/a/21000308/3730690\n\n    Args:\n      cmd: Command.\n\n    Returns:\n        Exit code, stdout, stderr.\n    \"\"\"\n    args = shlex.split(cmd)\n    proc = Popen(args, stdout=PIPE, stderr=PIPE)\n    out, err = proc.communicate()\n    exitcode = proc.returncode\n    return exitcode, out.decode('utf-8'), err.decode('utf-8')\n\n\ndef run_cmd(cmd: str) -> str:\n    exitcode, out, err = get_exitcode_stdout_stderr(cmd)\n    if exitcode:\n        raise RuntimeError(err + '\\nThe command is:\\n' + cmd)\n    return out\n\n\n@contextlib.contextmanager\ndef pushd(new_dir):\n    previous_dir = os.getcwd()\n    os.chdir(new_dir)\n    try:\n        yield\n    finally:\n        os.chdir(previous_dir)\n\n\ndef basename_no_ext(path):\n    basename = os.path.basename(path)\n    no_ext, ext = os.path.splitext(basename)\n    return no_ext\n\n\ndef file_cache(path: str, purge=False):\n    cache_name = path + '.cache'\n    cache_time = os.path.getmtime(cache_name) if os.path.isfile(cache_name) and not purge else 0\n    file_time = os.path.getmtime(path)\n    cache_valid = cache_time > file_time\n    return cache_name, cache_valid\n\n\ndef merge_files(files: List[str], dst: str):\n    with open(dst, 'wb') as write:\n        for f in files:\n            with open(f, 'rb') as read:\n                shutil.copyfileobj(read, write)\n\n\nclass TimingFileIterator(CountdownTimer):\n\n    def __init__(self, filepath) -> None:\n        super().__init__(os.path.getsize(filepath))\n        self.filepath = filepath\n\n    def __iter__(self):\n        if not os.path.isfile(self.filepath):\n            raise FileNotFoundError(self.filepath)\n        fp = open(self.filepath, encoding='utf-8', errors='ignore')\n        line = fp.readline()\n        while line:\n            yield line\n            self.current = fp.tell()\n            line = fp.readline()\n        fp.close()\n\n    def log(self, info=None, ratio_percentage=True, ratio=True, step=0, interval=0.5, erase=True,\n            logger: Union[logging.Logger, bool] = None, newline=False, ratio_width=None):\n        assert step == 0\n        super().log(info, ratio_percentage, ratio, step, interval, erase, logger, newline, ratio_width)\n\n    @property\n    def ratio(self) -> str:\n        return f'{human_bytes(self.current)}/{human_bytes(self.total)}'\n\n    @property\n    def ratio_width(self) -> int:\n        return len(f'{human_bytes(self.total)}') * 2 + 1\n\n    def close(self):\n        pass\n\n\ndef check_outdated(package='hanlp', version=__version__, repository_url='https://pypi.python.org/pypi/%s/json'):\n    \"\"\"Given the name of a package on PyPI and a version (both strings), checks\n    if the given version is the latest version of the package available.\n    Returns a 2-tuple (installed_version, latest_version)\n    `repository_url` is a `%` style format string\n    to use a different repository PyPI repository URL,\n    e.g. test.pypi.org or a private repository.\n    The string is formatted with the package name.\n    Adopted from https://github.com/alexmojaki/outdated/blob/master/outdated/__init__.py\n\n    Args:\n        package: Package name.\n        version: Installed version string.\n        repository_url: URL on pypi.\n\n    Returns:\n        Parsed installed version and latest version.\n    \"\"\"\n    installed_version = Version(version)\n    latest_version = get_latest_info_from_pypi(package, repository_url)\n    return installed_version, latest_version\n\n\ndef get_latest_info_from_pypi(package='hanlp', repository_url='https://pypi.python.org/pypi/%s/json'):\n    url = repository_url % package\n    response = urllib.request.urlopen(url).read()\n    return Version(json.loads(response)['info']['version'])\n\n\ndef check_version_conflicts(extras=None):\n    from pkg_resources import get_distribution, Requirement, WorkingSet, VersionConflict, DistributionNotFound\n    pkg = get_distribution('hanlp')\n    if not extras:\n        extras = pkg.extras\n    if isinstance(extras, list):\n        extras = tuple(extras)\n    requirements: List[Requirement] = pkg.requires(extras=extras)\n    error = None\n    try:\n        WorkingSet().resolve(\n            requirements, extras=extras\n        )\n    except VersionConflict as e:\n        error = e.with_context('hanlp').report()\n    except DistributionNotFound as e:\n        error = str(e)\n    return error, extras\n"
  },
  {
    "path": "hanlp/utils/lang/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-01-09 18:46\n\n__doc__ = '''\nThis package holds misc utils for specific languages.\n'''\n"
  },
  {
    "path": "hanlp/utils/lang/en/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-12-28 19:28\n"
  },
  {
    "path": "hanlp/utils/lang/en/english_tokenizer.py",
    "content": "#!/usr/bin/env python\n\"\"\"\nRegex-based word tokenizers.\n\nNote that small/full/half-width character variants are *not* covered.\nIf a text were to contains such characters, normalize it first.\nA modified version of https://github.com/fnl/segtok\n\n- dropped dependency on regex\n- dropped web_tokenize\n- supported concat word\n\n\"\"\"\n\n__author__ = 'Florian Leitner <florian.leitner@gmail.com>'\nfrom re import compile, UNICODE, VERBOSE\n\nSENTENCE_TERMINALS = '.!?\\u203C\\u203D\\u2047\\u2048\\u2049\\u3002' \\\n                     '\\uFE52\\uFE57\\uFF01\\uFF0E\\uFF1F\\uFF61'\n\"The list of valid Unicode sentence terminal characters.\"\n\n# Note that Unicode the category Pd is NOT a good set for valid word-breaking hyphens,\n# because it contains many dashes that should not be considered part of a word.\nHYPHENS = '\\u00AD\\u058A\\u05BE\\u0F0C\\u1400\\u1806\\u2010-\\u2012\\u2e17\\u30A0-'\n\"Any valid word-breaking hyphen, including ASCII hyphen minus.\"\n\nAPOSTROPHES = '\\'\\u00B4\\u02B9\\u02BC\\u2019\\u2032'\n\"\"\"All apostrophe-like marks, including the ASCII \"single quote\".\"\"\"\n\nAPOSTROPHE = r\"[\\u00B4\\u02B9\\u02BC\\u2019\\u2032]\"\n\"\"\"Any apostrophe-like marks, including \"prime\" but not the ASCII \"single quote\".\"\"\"\n\nLINEBREAK = r'(?:\\r\\n|\\n|\\r|\\u2028)'\n\"\"\"Any valid linebreak sequence (Windows, Unix, Mac, or U+2028).\"\"\"\n\nLETTER = r'[^\\W\\d_]'\n\"\"\"Any Unicode letter character that can form part of a word: Ll, Lm, Lt, Lu.\"\"\"\n\nNUMBER = r'\\d'\n\"\"\"Any Unicode number character: Nd or Nl.\"\"\"\n\nPOWER = r'\\u207B?[\\u00B9\\u00B2\\u00B3]'\n\"\"\"Superscript 1, 2, and 3, optionally prefixed with a minus sign.\"\"\"\n\nSUBDIGIT = r'[\\u2080-\\u2089]'\n\"\"\"Subscript digits.\"\"\"\n\nALNUM = LETTER[:-1] + NUMBER + ']'\n\"\"\"Any alphanumeric Unicode character: letter or number.\"\"\"\n\nHYPHEN = r'[%s]' % HYPHENS\n\nSPACE = r'\\s'\n\"\"\"Any unicode space character plus the (horizontal) tab.\"\"\"\n\nAPO_MATCHER = compile(APOSTROPHE, UNICODE)\n\"\"\"Matcher for any apostrophe.\"\"\"\n\nHYPHENATED_LINEBREAK = compile(\n    r'({alnum}{hyphen}){space}*?{linebreak}{space}*?({alnum})'.format(\n        alnum=ALNUM, hyphen=HYPHEN, linebreak=LINEBREAK, space=SPACE\n    ), UNICODE\n)\n\"\"\"\nThe pattern matches any alphanumeric Unicode character, followed by a hyphen,\na single line-break surrounded by optional (non-breaking) spaces,\nand terminates with a alphanumeric character on this next line.\nThe opening char and hyphen as well as the terminating char are captured in two groups.\n\"\"\"\n\nIS_POSSESSIVE = compile(r\"{alnum}+(?:{hyphen}{alnum}+)*(?:{apo}[sS]|[sS]{apo})$\".format(\n    alnum=ALNUM, hyphen=HYPHEN, apo=\"['\" + APOSTROPHE[1:]\n), UNICODE\n)\n\"\"\"A pattern that matches English words with a possessive s terminal form.\"\"\"\n\nIS_CONTRACTION = compile(r\"{alnum}+(?:{hyphen}{alnum}+)*{apo}(?:d|ll|m|re|s|t|ve)$\".format(\n    alnum=ALNUM, hyphen=HYPHEN, apo=\"['\" + APOSTROPHE[1:]\n), UNICODE\n)\n\"\"\"A pattern that matches tokens with valid English contractions ``'(d|ll|m|re|s|t|ve)``.\"\"\"\n\nMAP_CONCAT_WORD = {'aint': [2, 4], 'arent': [3, 5], 'cant': [2, 4], 'cannot': [3, 6], 'coulda': [5, 6],\n                   'couldnt': [5, 7], 'didnt': [3, 5], 'doncha': [2, 3, 6], 'dont': [2, 4],\n                   'doesnt': [4, 6], 'dunno': [2, 3, 5], 'finna': [3, 5], 'gimme': [3, 5], 'gonna': [3, 5],\n                   'gotta': [3, 5], 'hadnt': [3, 5], 'hasnt': [3, 5], 'havent': [4, 6], 'isnt': [2, 4],\n                   'itd': [2, 3], 'itll': [2, 4], 'lemme': [3, 5], 'lets': [3, 4], 'mightnt': [5, 7],\n                   'mustnt': [4, 6], 'shant': [3, 5], 'shoulda': [6, 7], 'shouldnt': [6, 8],\n                   'thatd': [4, 5], 'thatll': [4, 6], 'thats': [4, 5], 'theyd': [4, 5], 'theyre': [4, 6],\n                   'theyve': [4, 6], 'wanna': [3, 5], 'wasnt': [3, 5], 'weve': [2, 4], 'werent': [4, 6],\n                   'whadya': [3, 4, 6], 'whatcha': [4, 7], 'whatre': [4, 6], 'whats': [4, 5],\n                   'whatve': [4, 6], 'whatz': [4, 5], 'whod': [3, 4], 'wholl': [3, 5], 'woncha': [2, 3, 6],\n                   'wont': [2, 4], 'woulda': [5, 6], 'wouldnt': [5, 7], 'youd': [3, 4], 'youll': [3, 5],\n                   'youve': [3, 5], \"'tis\": [2, 4], \"'twas\": [2, 5], \"d'ye\": [2, 4], \"don'cha\": [2, 4, 7],\n                   \"i'mma\": [1, 3, 5], \"i'mmm\": [1, 5], \"more'n\": [4, 6], '’tis': [2, 4], '’twas': [2, 5],\n                   'd’ye': [2, 4], 'don’cha': [2, 4, 7], 'i’mma': [1, 3, 5], 'i’mmm': [1, 5],\n                   'more’n': [4, 6]}\n\nRE_APOSTROPHE = compile(r'(?i)[a-z](n[\\'\\u2019]t|[\\'\\u2019](ll|nt|re|ve|[dmstz]))(\\W|$)')\n\n\ndef split_possessive_markers(tokens):\n    \"\"\"\n    A function to split possessive markers at the end of alphanumeric (and hyphenated) tokens.\n\n    Takes the output of any of the tagger functions and produces and updated list.\n    To use it, simply wrap the tagger function, for example::\n\n    >>> my_sentence = \"This is Fred's latest book.\"\n    >>> split_possessive_markers(tokenize_english(my_sentence))\n    ['This', 'is', 'Fred', \"'s\", 'latest', 'book', '.']\n\n    :param tokens: a list of tokens\n    :returns: an updated list if a split was made or the original list otherwise\n    \"\"\"\n    idx = -1\n\n    for token in list(tokens):\n        idx += 1\n\n        if IS_POSSESSIVE.match(token) is not None:\n            if token[-1].lower() == 's' and token[-2] in APOSTROPHES:\n                tokens.insert(idx, token[:-2])\n                idx += 1\n                tokens[idx] = token[-2:]\n            elif token[-2].lower() == 's' and token[-1] in APOSTROPHES:\n                tokens.insert(idx, token[:-1])\n                idx += 1\n                tokens[idx] = token[-1:]\n\n    return tokens\n\n\ndef split_contractions(tokens):\n    \"\"\"\n    A function to split apostrophe contractions at the end of alphanumeric (and hyphenated) tokens.\n\n    Takes the output of any of the tagger functions and produces and updated list.\n\n    :param tokens: a list of tokens\n    :returns: an updated list if a split was made or the original list otherwise\n    \"\"\"\n    idx = -1\n\n    for token in list(tokens):\n        idx += 1\n\n        if IS_CONTRACTION.match(token) is not None:\n            length = len(token)\n\n            if length > 1:\n                for pos in range(length - 1, -1, -1):\n                    if token[pos] in APOSTROPHES:\n                        if 2 < length and pos + 2 == length and token[-1] == 't' and token[pos - 1] == 'n':\n                            pos -= 1\n\n                        tokens.insert(idx, token[:pos])\n                        idx += 1\n                        tokens[idx] = token[pos:]\n\n    return tokens\n\n\ndef _matches(regex):\n    \"\"\"Regular expression compiling function decorator.\"\"\"\n\n    def match_decorator(fn):\n        automaton = compile(regex, UNICODE | VERBOSE)\n        fn.split = automaton.split\n        fn.match = automaton.match\n        return fn\n\n    return match_decorator\n\n\n@_matches(r'\\s+')\ndef space_tokenizer(sentence):\n    \"\"\"\n    For a given input `sentence`, return a list of its tokens.\n\n    Split on Unicode spaces ``\\\\s+`` (i.e., any kind of **Unicode** space character).\n    The separating space characters are not included in the resulting token list.\n    \"\"\"\n    return [token for token in space_tokenizer.split(sentence) if token]\n\n\n@_matches(r'(%s+)' % ALNUM)\ndef symbol_tokenizer(sentence):\n    \"\"\"\n    The symbol tagger extends the :func:`space_tokenizer` by separating alphanumerics.\n\n    Separates alphanumeric Unicode character sequences in already space-split tokens.\n    \"\"\"\n    return [token for span in space_tokenizer(sentence) for\n            token in symbol_tokenizer.split(span) if token]\n\n\n@_matches(r\"\"\"((?:\n    # Dots, except ellipsis\n    {alnum} \\. (?!\\.\\.)\n    | # Comma, surrounded by digits (e.g., chemicals) or letters\n    {alnum} , (?={alnum})\n    | # Colon, surrounded by digits (e.g., time, references)\n    {number} : (?={number})\n    | # Hyphen, surrounded by digits (e.g., DNA endings: \"5'-ACGT-3'\") or letters\n    {alnum} {apo}? {hyphen} (?={alnum})  # incl. optional apostrophe for DNA segments\n    | # Apostophes, non-consecutive\n    {apo} (?!{apo})\n    | # ASCII single quote, surrounded by digits or letters (no dangling allowed)\n    {alnum} ' (?={alnum})\n    | # ASCII single quote after an s and at the token's end\n    s ' $\n    | # Terminal dimensions (superscript minus, 1, 2, and 3) attached to physical units\n    #  size-prefix                 unit-acronym    dimension\n    \\b [yzafpn\\u00B5mcdhkMGTPEZY]? {letter}{{1,3}} {power} $\n    | # Atom counts (subscript numbers) and ionization states (optional superscript\n    #   2 or 3 followed by a + or -) are attached to valid fragments of a chemical formula\n    \\b (?:[A-Z][a-z]?|[\\)\\]])+ {subdigit}+ (?:[\\u00B2\\u00B3]?[\\u207A\\u207B])?\n    | # Any (Unicode) letter, digit, or the underscore\n    {alnum}\n    )+)\"\"\".format(alnum=ALNUM, apo=APOSTROPHE, power=POWER, subdigit=SUBDIGIT,\n                  hyphen=HYPHEN, letter=LETTER, number=NUMBER))\ndef tokenize_english(sentence):\n    \"\"\"\n    A modified version of the segtok tagger: https://github.com/fnl/segtok\n    This tagger extends the alphanumeric :func:`symbol_tokenizer` by splitting fewer cases:\n\n    1. Dots appearing after a letter are maintained as part of the word, except for the last word\n       in a sentence if that dot is the sentence terminal. Therefore, abbreviation marks (words\n       containing or ending in a ``.``, like \"i.e.\") remain intact and URL or ID segments remain\n       complete (\"www.ex-ample.com\", \"EC1.2.3.4.5\", etc.). The only dots that never are attached\n       are triple dots (``...``; ellipsis).\n    2. Commas surrounded by alphanumeric characters are maintained in the word, too, e.g. ``a,b``.\n       Colons surrounded by digits are maintained, e.g., 'at 12:30pm' or 'Isaiah 12:3'.\n       Commas, semi-colons, and colons dangling at the end of a token are always spliced off.\n    3. Any two alphanumeric letters that are separated by a single hyphen are joined together;\n       Those \"inner\" hyphens may optionally be followed by a linebreak surrounded by spaces;\n       The spaces will be removed, however. For example, ``Hel- \\\\r\\\\n \\t lo`` contains a (Windows)\n       linebreak and will be returned as ``Hel-lo``.\n    4. Apostrophes are always allowed in words as long as they are not repeated; The single quote\n       ASCII letter ``'`` is only allowed as a terminal apostrophe after the letter ``s``,\n       otherwise it must be surrounded by letters. To support DNA and chemicals, a apostrophe\n       (prime) may be located before the hyphen, as in the single token \"5'-ACGT-3'\" (if any\n       non-ASCII hyphens are used instead of the shown single quote).\n    5. Superscript 1, 2, and 3, optionally prefixed with a superscript minus, are attached to a\n       word if it is no longer than 3 letters (optionally 4 if the first letter is a power prefix\n       in the range from yocto, y (10^-24) to yotta, Y (10^+24)).\n    6. Subscript digits are attached if prefixed with letters that look like a chemical formula.\n    \"\"\"\n    if not sentence:\n        return []\n    flat = not isinstance(sentence, list)\n    if flat:\n        sents = [sentence]\n    else:\n        sents = sentence\n    results = []\n    for sentence in sents:\n        pruned = HYPHENATED_LINEBREAK.sub(r'\\1\\2', sentence)\n        tokens = [token for span in space_tokenizer(pruned) for\n                  token in tokenize_english.split(span) if token]\n\n        # splice the sentence terminal off the last word/token if it has any at its borders\n        # only look for the sentence terminal in the last three tokens\n        for idx, word in enumerate(reversed(tokens[-3:]), 1):\n            if (tokenize_english.match(word) and not APO_MATCHER.match(word)) or \\\n                    any(t in word for t in SENTENCE_TERMINALS):\n                last = len(word) - 1\n\n                if 0 == last or u'...' == word:\n                    # any case of \"...\" or any single char (last == 0)\n                    pass  # leave the token as it is\n                elif any(word.rfind(t) == last for t in SENTENCE_TERMINALS):\n                    # \"stuff.\"\n                    tokens[-idx] = word[:-1]\n                    tokens.insert(len(tokens) - idx + 1, word[-1])\n                elif any(word.find(t) == 0 for t in SENTENCE_TERMINALS):\n                    # \".stuff\"\n                    tokens[-idx] = word[0]\n                    tokens.insert(len(tokens) - idx + 1, word[1:])\n\n                break\n\n        # keep splicing off any dangling commas and (semi-) colons\n        dirty = True\n        while dirty:\n            dirty = False\n\n            for idx, word in enumerate(reversed(tokens), 1):\n                while len(word) > 1 and word[-1] in u',;:':\n                    char = word[-1]  # the dangling comma/colon\n                    word = word[:-1]\n                    tokens[-idx] = word\n                    tokens.insert(len(tokens) - idx + 1, char)\n                    idx += 1\n                    dirty = True\n                if dirty:\n                    break  # restart check to avoid index errors\n\n        # split concat words\n        chunks = []\n        for token in tokens:\n            t = MAP_CONCAT_WORD.get(token.lower(), None)\n            if t:\n                i = 0\n                for j in t:\n                    chunks.append(token[i:j])\n                    i = j\n            else:\n                chunks.append(token)\n        tokens = chunks\n        # split APOSTROPHE\n        chunks = []\n        for token in tokens:\n            m = RE_APOSTROPHE.search(token)\n            if m:\n                chunks.append(token[:m.start(1)])\n                chunks.append(token[m.start(1):m.end(1)])\n                if m.end(1) < len(token):\n                    chunks.append(token[m.end(1):])\n            else:\n                chunks.append(token)\n        tokens = chunks\n        results.append(tokens)\n    return results[0] if flat else results"
  },
  {
    "path": "hanlp/utils/lang/ja/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-05-13 13:24\n"
  },
  {
    "path": "hanlp/utils/lang/ja/bert_tok.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-05-13 13:24\nfrom typing import Union, Optional\n\nfrom transformers import BertTokenizerFast, TensorType, BatchEncoding, BertJapaneseTokenizer as _BertJapaneseTokenizer\nfrom transformers.file_utils import PaddingStrategy\nfrom transformers.tokenization_utils_base import TextInput, PreTokenizedInput, EncodedInput, TruncationStrategy\n\n\nclass BertJapaneseTokenizer(_BertJapaneseTokenizer):\n    # We may need to customize character level tokenization to handle English words and URLs\n    pass\n\n\nclass BertJapaneseTokenizerFast(BertTokenizerFast):\n    def encode_plus(\n            self,\n            text: Union[TextInput, PreTokenizedInput, EncodedInput],\n            text_pair: Optional[Union[TextInput, PreTokenizedInput, EncodedInput]] = None,\n            add_special_tokens: bool = True,\n            padding: Union[bool, str, PaddingStrategy] = False,\n            truncation: Union[bool, str, TruncationStrategy] = False,\n            max_length: Optional[int] = None,\n            stride: int = 0,\n            is_split_into_words: bool = False,\n            pad_to_multiple_of: Optional[int] = None,\n            return_tensors: Optional[Union[str, TensorType]] = None,\n            return_token_type_ids: Optional[bool] = None,\n            return_attention_mask: Optional[bool] = None,\n            return_overflowing_tokens: bool = False,\n            return_special_tokens_mask: bool = False,\n            return_offsets_mapping: bool = False,\n            return_length: bool = False,\n            verbose: bool = True,\n            **kwargs\n    ) -> BatchEncoding:\n        \"\"\"\n        Tokenize and prepare for the model a sequence or a pair of sequences.\n\n        .. warning::\n            This method is deprecated, ``__call__`` should be used instead.\n\n        Args:\n            text (:obj:`str`, :obj:`List[str]` or :obj:`List[int]` (the latter only for not-fast tokenizers)):\n                The first sequence to be encoded. This can be a string, a list of strings (tokenized string using the\n                ``tokenize`` method) or a list of integers (tokenized string ids using the ``convert_tokens_to_ids``\n                method).\n            text_pair (:obj:`str`, :obj:`List[str]` or :obj:`List[int]`, `optional`):\n                Optional second sequence to be encoded. This can be a string, a list of strings (tokenized string using\n                the ``tokenize`` method) or a list of integers (tokenized string ids using the\n                ``convert_tokens_to_ids`` method).\n        \"\"\"\n        text = list(text)\n        is_split_into_words = True\n        encoding = BertJapaneseTokenizer.encode_plus(self,\n                                                     text,\n                                                     text_pair,\n                                                     add_special_tokens,\n                                                     padding,\n                                                     truncation,\n                                                     max_length,\n                                                     stride,\n                                                     is_split_into_words,\n                                                     pad_to_multiple_of,\n                                                     return_tensors,\n                                                     return_token_type_ids,\n                                                     return_attention_mask,\n                                                     return_overflowing_tokens,\n                                                     return_special_tokens_mask,\n                                                     return_offsets_mapping,\n                                                     return_length,\n                                                     verbose,\n                                                     **kwargs\n                                                     )\n        offsets = encoding.encodings[0].offsets\n        fixed_offsets = [(b + i, e + i) for i, (b, e) in enumerate(offsets)]\n        # TODO: This doesn't work with rust tokenizers\n        encoding.encodings[0].offsets.clear()\n        encoding.encodings[0].offsets.extend(fixed_offsets)\n        return encoding\n"
  },
  {
    "path": "hanlp/utils/lang/zh/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-01-09 18:47"
  },
  {
    "path": "hanlp/utils/lang/zh/char_table.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-01-09 19:07\nfrom typing import List\n\nfrom hanlp.utils.io_util import get_resource\nfrom hanlp_common.io import load_json\n\nHANLP_CHAR_TABLE_TXT = 'https://file.hankcs.com/corpus/char_table.zip#CharTable.txt'\nHANLP_CHAR_TABLE_JSON = 'https://file.hankcs.com/corpus/char_table.json.zip'\n\n\nclass CharTable:\n    convert = {}\n\n    @staticmethod\n    def convert_char(c):\n        if not CharTable.convert:\n            CharTable._init()\n        return CharTable.convert.get(c, c)\n\n    @staticmethod\n    def normalize_text(text: str) -> str:\n        return ''.join(CharTable.convert_char(c) for c in text)\n\n    @staticmethod\n    def normalize_chars(chars: List[str]) -> List[str]:\n        return [CharTable.convert_char(c) for c in chars]\n\n    @staticmethod\n    def _init():\n        CharTable.convert = CharTable.load()\n\n    @staticmethod\n    def load():\n        mapper = {}\n        with open(get_resource(HANLP_CHAR_TABLE_TXT), encoding='utf-8') as src:\n            for line in src:\n                cells = line.rstrip('\\n')\n                if len(cells) != 3:\n                    continue\n                a, _, b = cells\n                mapper[a] = b\n        return mapper\n\n\nclass JsonCharTable(CharTable):\n\n    @staticmethod\n    def load():\n        return load_json(get_resource(HANLP_CHAR_TABLE_JSON))\n\n\n"
  },
  {
    "path": "hanlp/utils/lang/zh/localization.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-05 02:09\n\ntask = {\n    'dep': '依存句法树',\n    'token': '单词',\n    'pos': '词性',\n    'ner': '命名实体',\n    'srl': '语义角色'\n}\n\npos = {\n    'VA': '表语形容词', 'VC': '系动词', 'VE': '动词有无', 'VV': '其他动词', 'NR': '专有名词', 'NT': '时间名词', 'NN': '其他名词',\n    'LC': '方位词', 'PN': '代词', 'DT': '限定词', 'CD': '概数词', 'OD': '序数词', 'M': '量词', 'AD': '副词', 'P': '介词',\n    'CC': '并列连接词', 'CS': '从属连词', 'DEC': '补语成分“的”', 'DEG': '属格“的”', 'DER': '表结果的“得”', 'DEV': '表方式的“地”',\n    'AS': '动态助词', 'SP': '句末助词', 'ETC': '表示省略', 'MSP': '其他小品词', 'IJ': '句首感叹词', 'ON': '象声词',\n    'LB': '长句式表被动', 'SB': '短句式表被动', 'BA': '把字句', 'JJ': '其他名词修饰语', 'FW': '外来语', 'PU': '标点符号',\n    'NOI': '噪声', 'URL': '网址'\n}\n\nner = {\n    'NT': '机构团体', 'NS': '地名', 'NR': '人名'\n}\n\ndep = {\n    'nn': '复合名词修饰', 'punct': '标点符号', 'nsubj': '名词性主语', 'conj': '连接性状语', 'dobj': '直接宾语', 'advmod': '名词性状语',\n    'prep': '介词性修饰语', 'nummod': '数词修饰语', 'amod': '形容词修饰语', 'pobj': '介词性宾语', 'rcmod': '相关关系', 'cpm': '补语',\n    'assm': '关联标记', 'assmod': '关联修饰', 'cc': '并列关系', 'elf': '类别修饰', 'ccomp': '从句补充', 'det': '限定语', 'lobj': '时间介词',\n    'range': '数量词间接宾语', 'asp': '时态标记', 'tmod': '时间修饰语', 'plmod': '介词性地点修饰', 'attr': '属性', 'mmod': '情态动词',\n    'loc': '位置补语', 'top': '主题', 'pccomp': '介词补语', 'etc': '省略关系', 'lccomp': '位置补语', 'ordmod': '量词修饰',\n    'xsubj': '控制主语', 'neg': '否定修饰', 'rcomp': '结果补语', 'comod': '并列联合动词', 'vmod': '动词修饰', 'prtmod': '小品词',\n    'ba': '把字关系', 'dvpm': '地字修饰', 'dvpmod': '地字动词短语', 'prnmod': '插入词修饰', 'cop': '系动词', 'pass': '被动标记',\n    'nsubjpass': '被动名词主语', 'clf': '类别修饰', 'dep': '依赖关系', 'root': '核心关系'\n}\n"
  },
  {
    "path": "hanlp/utils/log_util.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-08-24 22:12\nimport datetime\nimport io\nimport logging\nimport os\nimport sys\nfrom logging import LogRecord\n\nimport termcolor\n\nfrom hanlp_common.constant import IPYTHON\n\n\nclass ColoredFormatter(logging.Formatter):\n    def __init__(self, fmt=None, datefmt=None, style='%', enable=True):\n        super().__init__(fmt, datefmt, style)\n        self.enable = enable\n\n    def formatMessage(self, record: LogRecord) -> str:\n        message = super().formatMessage(record)\n        if self.enable:\n            return color_format(message)\n        else:\n            return remove_color_tag(message)\n\n\ndef init_logger(name=None, root_dir=None, level=logging.INFO, mode='w',\n                fmt=\"%(asctime)s %(levelname)s %(message)s\",\n                datefmt='%Y-%m-%d %H:%M:%S') -> logging.Logger:\n    if not name:\n        name = datetime.datetime.now().strftime(\"%Y-%m-%d_%H.%M.%S\")\n    rootLogger = logging.getLogger(os.path.join(root_dir, name) if root_dir else name)\n    rootLogger.propagate = False\n\n    consoleHandler = logging.StreamHandler(sys.stdout)  # stderr will be rendered as red which is bad\n    consoleHandler.setFormatter(ColoredFormatter(fmt, datefmt=datefmt))\n    attached_to_std = False\n    for handler in rootLogger.handlers:\n        if isinstance(handler, logging.StreamHandler):\n            if handler.stream == sys.stderr or handler.stream == sys.stdout:\n                attached_to_std = True\n                break\n    if not attached_to_std:\n        rootLogger.addHandler(consoleHandler)\n    rootLogger.setLevel(level)\n    consoleHandler.setLevel(level)\n\n    if root_dir:\n        os.makedirs(root_dir, exist_ok=True)\n        log_path = \"{0}/{1}.log\".format(root_dir, name)\n        fileHandler = logging.FileHandler(log_path, mode=mode)\n        fileHandler.setFormatter(ColoredFormatter(fmt, datefmt=datefmt, enable=False))\n        rootLogger.addHandler(fileHandler)\n        fileHandler.setLevel(level)\n\n    return rootLogger\n\n\nlogger = init_logger(name='hanlp', level=os.environ.get('HANLP_LOG_LEVEL', 'INFO'))\n\n\ndef enable_debug(debug=True):\n    logger.setLevel(logging.DEBUG if debug else logging.ERROR)\n\n\nclass ErasablePrinter(object):\n    def __init__(self, out=sys.stderr):\n        self._last_print_width = 0\n        self.out = out\n\n    def erase(self):\n        if self._last_print_width:\n            if IPYTHON:\n                self.out.write(\"\\r\")\n                self.out.write(\" \" * self._last_print_width)\n            else:\n                self.out.write(\"\\b\" * self._last_print_width)\n                self.out.write(\" \" * self._last_print_width)\n                self.out.write(\"\\b\" * self._last_print_width)\n            self.out.write(\"\\r\")  # \\r is essential when multi-lines were printed\n            self._last_print_width = 0\n\n    def print(self, msg: str, color=True):\n        self.erase()\n        if color:\n            if IPYTHON:\n                msg, _len = color_format_len(msg)\n                _len = len(msg)\n            else:\n                msg, _len = color_format_len(msg)\n            self._last_print_width = _len\n        else:\n            self._last_print_width = len(msg)\n        self.out.write(msg)\n        self.out.flush()\n\n\n_printer = ErasablePrinter()\n\n\ndef flash(line: str, color=True):\n    _printer.print(line, color)\n\n\ndef color_format(msg: str):\n    for tag in termcolor.COLORS, termcolor.HIGHLIGHTS, termcolor.ATTRIBUTES:\n        for c, v in tag.items():\n            start, end = f'[{c}]', f'[/{c}]'\n            msg = msg.replace(start, '\\033[%dm' % v).replace(end, termcolor.RESET)\n    return msg\n\n\ndef remove_color_tag(msg: str):\n    for tag in termcolor.COLORS, termcolor.HIGHLIGHTS, termcolor.ATTRIBUTES:\n        for c, v in tag.items():\n            start, end = f'[{c}]', f'[/{c}]'\n            msg = msg.replace(start, '').replace(end, '')\n    return msg\n\n\ndef color_format_len(msg: str):\n    _len = len(msg)\n    for tag in termcolor.COLORS, termcolor.HIGHLIGHTS, termcolor.ATTRIBUTES:\n        for c, v in tag.items():\n            start, end = f'[{c}]', f'[/{c}]'\n            msg, delta = _replace_color_offset(msg, start, '\\033[%dm' % v)\n            _len -= delta\n            msg, delta = _replace_color_offset(msg, end, termcolor.RESET)\n            _len -= delta\n    return msg, _len\n\n\ndef _replace_color_offset(msg: str, color: str, ctrl: str):\n    chunks = msg.split(color)\n    delta = (len(chunks) - 1) * len(color)\n    return ctrl.join(chunks), delta\n\n\ndef cprint(*args, file=None, **kwargs):\n    out = io.StringIO()\n    print(*args, file=out, **kwargs)\n    text = out.getvalue()\n    out.close()\n    c_text = color_format(text)\n    print(c_text, end='', file=file)\n\n\ndef main():\n    # cprint('[blink][yellow]...[/yellow][/blink]')\n    # show_colors_and_formats()\n    show_colors()\n    # print('previous', end='')\n    # for i in range(10):\n    #     flash(f'[red]{i}[/red]')\n\n\ndef show_colors_and_formats():\n    msg = ''\n    for c in termcolor.COLORS.keys():\n        for h in termcolor.HIGHLIGHTS.keys():\n            for a in termcolor.ATTRIBUTES.keys():\n                msg += f'[{c}][{h}][{a}] {c}+{h}+{a} [/{a}][/{h}][/{c}]'\n    logger.info(msg)\n\n\ndef show_colors():\n    msg = ''\n    for c in termcolor.COLORS.keys():\n        cprint(f'[{c}]\"{c}\",[/{c}]')\n\n\n# Generates tables for Doxygen flavored Markdown.  See the Doxygen\n# documentation for details:\n#   http://www.doxygen.nl/manual/markdown.html#md_tables\n\n# Translation dictionaries for table alignment\n\n\nif __name__ == '__main__':\n    main()\n"
  },
  {
    "path": "hanlp/utils/rules.py",
    "content": "import re\n\n_SEPARATOR = r'@'\n_RE_SENTENCE = re.compile(r'(\\S.+?[.!?])(?=\\s+|$)|(\\S.+?)(?=[\\n]|$)', re.UNICODE)\n_AB_SENIOR = re.compile(r'([A-Z][a-z]{1,2}\\.)\\s(\\w)', re.UNICODE)\n_AB_ACRONYM = re.compile(r'(\\.[a-zA-Z]\\.)\\s(\\w)', re.UNICODE)\n_UNDO_AB_SENIOR = re.compile(r'([A-Z][a-z]{1,2}\\.)' + _SEPARATOR + r'(\\w)', re.UNICODE)\n_UNDO_AB_ACRONYM = re.compile(r'(\\.[a-zA-Z]\\.)' + _SEPARATOR + r'(\\w)', re.UNICODE)\n\n\ndef _replace_with_separator(text, separator, regexs):\n    replacement = r\"\\1\" + separator + r\"\\2\"\n    result = text\n    for regex in regexs:\n        result = regex.sub(replacement, result)\n    return result\n\n\ndef split_sentence(text, best=True):\n    text = re.sub(r'([。！？?])([^”’])', r\"\\1\\n\\2\", text)\n    text = re.sub(r'(\\.{6})([^”’])', r\"\\1\\n\\2\", text)\n    text = re.sub(r'(…{2})([^”’])', r\"\\1\\n\\2\", text)\n    text = re.sub(r'([。！？?][”’])([^，。！？?])', r'\\1\\n\\2', text)\n    for chunk in text.split(\"\\n\"):\n        chunk = chunk.strip()\n        if not chunk:\n            continue\n        if not best:\n            yield chunk\n            continue\n        processed = _replace_with_separator(chunk, _SEPARATOR, [_AB_SENIOR, _AB_ACRONYM])\n        sents = list(_RE_SENTENCE.finditer(processed))\n        if not sents:\n            yield chunk\n            continue\n        for sentence in sents:\n            sentence = _replace_with_separator(sentence.group(), r\" \", [_UNDO_AB_SENIOR, _UNDO_AB_ACRONYM])\n            yield sentence\n\n\n"
  },
  {
    "path": "hanlp/utils/span_util.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-06-12 20:34\nimport warnings\nfrom typing import Dict, List, Tuple, Callable, Set, Optional\n\n\ndef generate_words_per_line(file_path):\n    with open(file_path, encoding='utf-8') as src:\n        for line in src:\n            cells = line.strip().split()\n            if not cells:\n                continue\n            yield cells\n\n\ndef words_to_bmes(words):\n    tags = []\n    for w in words:\n        if not w:\n            raise ValueError('{} contains None or zero-length word {}'.format(str(words), w))\n        if len(w) == 1:\n            tags.append('S')\n        else:\n            tags.extend(['B'] + ['M'] * (len(w) - 2) + ['E'])\n    return tags\n\n\ndef words_to_bi(words):\n    tags = []\n    for w in words:\n        if not w:\n            raise ValueError('{} contains None or zero-length word {}'.format(str(words), w))\n        tags.extend(['B'] + ['I'] * (len(w) - 1))\n    return tags\n\n\ndef bmes_to_words(chars, tags):\n    result = []\n    if len(chars) == 0:\n        return result\n    word = chars[0]\n\n    for c, t in zip(chars[1:], tags[1:]):\n        if t == 'B' or t == 'S':\n            result.append(word)\n            word = ''\n        word += c\n    if len(word) != 0:\n        result.append(word)\n\n    return result\n\n\ndef bmes_to_spans(tags):\n    result = []\n    offset = 0\n    pre_offset = 0\n    for t in tags[1:]:\n        offset += 1\n        if t == 'B' or t == 'S':\n            result.append((pre_offset, offset))\n            pre_offset = offset\n    if offset != len(tags):\n        result.append((pre_offset, len(tags)))\n\n    return result\n\n\ndef bmes_of(sentence, segmented):\n    if segmented:\n        chars = []\n        tags = []\n        words = sentence.split()\n        for w in words:\n            chars.extend(list(w))\n            if len(w) == 1:\n                tags.append('S')\n            else:\n                tags.extend(['B'] + ['M'] * (len(w) - 2) + ['E'])\n    else:\n        chars = list(sentence)\n        tags = ['S'] * len(chars)\n    return chars, tags\n\n\ndef iobes_to_bilou(src, dst):\n    with open(src) as src, open(dst, 'w') as out:\n        for line in src:\n            line = line.strip()\n            if not line:\n                out.write('\\n')\n                continue\n            word, tag = line.split('\\t')\n            if tag.startswith('E-'):\n                tag = 'L-' + tag[2:]\n            elif tag.startswith('S-'):\n                tag = 'U-' + tag[2:]\n            out.write(f'{word}\\t{tag}\\n')\n\n\ndef allowed_transitions(constraint_type: str, labels: Dict[int, str]) -> List[Tuple[int, int]]:\n    \"\"\"\n    Given labels and a constraint type, returns the allowed transitions. It will\n    additionally include transitions for the start and end states, which are used\n    by the conditional random field.\n\n    # Parameters\n\n    constraint_type : `str`, required\n        Indicates which constraint to apply. Current choices are\n        \"BIO\", \"IOB1\", \"BIOUL\", and \"BMES\".\n    labels : `Dict[int, str]`, required\n        A mapping {label_id -> label}. Most commonly this would be the value from\n        Vocabulary.get_index_to_token_vocabulary()\n\n    # Returns\n\n    `List[Tuple[int, int]]`\n        The allowed transitions (from_label_id, to_label_id).\n    \"\"\"\n    num_labels = len(labels)\n    start_tag = num_labels\n    end_tag = num_labels + 1\n    labels_with_boundaries = list(labels.items()) + [(start_tag, \"START\"), (end_tag, \"END\")]\n\n    allowed = []\n    for from_label_index, from_label in labels_with_boundaries:\n        if from_label in (\"START\", \"END\"):\n            from_tag = from_label\n            from_entity = \"\"\n        else:\n            from_tag = from_label[0]\n            from_entity = from_label[1:]\n        for to_label_index, to_label in labels_with_boundaries:\n            if to_label in (\"START\", \"END\"):\n                to_tag = to_label\n                to_entity = \"\"\n            else:\n                to_tag = to_label[0]\n                to_entity = to_label[1:]\n            if is_transition_allowed(constraint_type, from_tag, from_entity, to_tag, to_entity):\n                allowed.append((from_label_index, to_label_index))\n    return allowed\n\n\ndef is_transition_allowed(\n    constraint_type: str, from_tag: str, from_entity: str, to_tag: str, to_entity: str\n):\n    \"\"\"\n    Given a constraint type and strings `from_tag` and `to_tag` that\n    represent the origin and destination of the transition, return whether\n    the transition is allowed under the given constraint type.\n\n    # Parameters\n\n    constraint_type : `str`, required\n        Indicates which constraint to apply. Current choices are\n        \"BIO\", \"IOB1\", \"BIOUL\", and \"BMES\".\n    from_tag : `str`, required\n        The tag that the transition originates from. For example, if the\n        label is `I-PER`, the `from_tag` is `I`.\n    from_entity : `str`, required\n        The entity corresponding to the `from_tag`. For example, if the\n        label is `I-PER`, the `from_entity` is `PER`.\n    to_tag : `str`, required\n        The tag that the transition leads to. For example, if the\n        label is `I-PER`, the `to_tag` is `I`.\n    to_entity : `str`, required\n        The entity corresponding to the `to_tag`. For example, if the\n        label is `I-PER`, the `to_entity` is `PER`.\n\n    # Returns\n\n    `bool`\n        Whether the transition is allowed under the given `constraint_type`.\n    \"\"\"\n\n    if to_tag == \"START\" or from_tag == \"END\":\n        # Cannot transition into START or from END\n        return False\n\n    if constraint_type == \"BIOUL\":\n        if from_tag == \"START\":\n            return to_tag in (\"O\", \"B\", \"U\")\n        if to_tag == \"END\":\n            return from_tag in (\"O\", \"L\", \"U\")\n        return any(\n            [\n                # O can transition to O, B-* or U-*\n                # L-x can transition to O, B-*, or U-*\n                # U-x can transition to O, B-*, or U-*\n                from_tag in (\"O\", \"L\", \"U\") and to_tag in (\"O\", \"B\", \"U\"),\n                # B-x can only transition to I-x or L-x\n                # I-x can only transition to I-x or L-x\n                from_tag in (\"B\", \"I\") and to_tag in (\"I\", \"L\") and from_entity == to_entity,\n            ]\n        )\n    elif constraint_type == \"BIO\":\n        if from_tag == \"START\":\n            return to_tag in (\"O\", \"B\")\n        if to_tag == \"END\":\n            return from_tag in (\"O\", \"B\", \"I\")\n        return any(\n            [\n                # Can always transition to O or B-x\n                to_tag in (\"O\", \"B\"),\n                # Can only transition to I-x from B-x or I-x\n                to_tag == \"I\" and from_tag in (\"B\", \"I\") and from_entity == to_entity,\n            ]\n        )\n    elif constraint_type == \"IOB1\":\n        if from_tag == \"START\":\n            return to_tag in (\"O\", \"I\")\n        if to_tag == \"END\":\n            return from_tag in (\"O\", \"B\", \"I\")\n        return any(\n            [\n                # Can always transition to O or I-x\n                to_tag in (\"O\", \"I\"),\n                # Can only transition to B-x from B-x or I-x, where\n                # x is the same tag.\n                to_tag == \"B\" and from_tag in (\"B\", \"I\") and from_entity == to_entity,\n            ]\n        )\n    elif constraint_type == \"BMES\":\n        if from_tag == \"START\":\n            return to_tag in (\"B\", \"S\")\n        if to_tag == \"END\":\n            return from_tag in (\"E\", \"S\")\n        return any(\n            [\n                # Can only transition to B or S from E or S.\n                to_tag in (\"B\", \"S\") and from_tag in (\"E\", \"S\"),\n                # Can only transition to M-x from B-x, where\n                # x is the same tag.\n                to_tag == \"M\" and from_tag in (\"B\", \"M\") and from_entity == to_entity,\n                # Can only transition to E-x from B-x or M-x, where\n                # x is the same tag.\n                to_tag == \"E\" and from_tag in (\"B\", \"M\") and from_entity == to_entity,\n            ]\n        )\n    else:\n        raise ValueError(f\"Unknown constraint type: {constraint_type}\")\n\n\nTypedSpan = Tuple[int, Tuple[int, int]]\nTypedStringSpan = Tuple[str, Tuple[int, int]]\n\n\nclass InvalidTagSequence(Exception):\n    def __init__(self, tag_sequence=None):\n        super().__init__()\n        self.tag_sequence = tag_sequence\n\n    def __str__(self):\n        return \" \".join(self.tag_sequence)\n\n\nT = str\n\n\ndef enumerate_spans(\n        sentence: List[T],\n        offset: int = 0,\n        max_span_width: int = None,\n        min_span_width: int = 1,\n        filter_function: Callable[[List[T]], bool] = None,\n) -> List[Tuple[int, int]]:\n    \"\"\"\n    Given a sentence, return all token spans within the sentence. Spans are `inclusive`.\n    Additionally, you can provide a maximum and minimum span width, which will be used\n    to exclude spans outside of this range.\n\n    Finally, you can provide a function mapping `List[T] -> bool`, which will\n    be applied to every span to decide whether that span should be included. This\n    allows filtering by length, regex matches, pos tags or any Spacy `Token`\n    attributes, for example.\n\n    # Parameters\n\n    sentence : `List[T]`, required.\n        The sentence to generate spans for. The type is generic, as this function\n        can be used with strings, or Spacy `Tokens` or other sequences.\n    offset : `int`, optional (default = `0`)\n        A numeric offset to add to all span start and end indices. This is helpful\n        if the sentence is part of a larger structure, such as a document, which\n        the indices need to respect.\n    max_span_width : `int`, optional (default = `None`)\n        The maximum length of spans which should be included. Defaults to len(sentence).\n    min_span_width : `int`, optional (default = `1`)\n        The minimum length of spans which should be included. Defaults to 1.\n    filter_function : `Callable[[List[T]], bool]`, optional (default = `None`)\n        A function mapping sequences of the passed type T to a boolean value.\n        If `True`, the span is included in the returned spans from the\n        sentence, otherwise it is excluded..\n    \"\"\"\n    max_span_width = max_span_width or len(sentence)\n    filter_function = filter_function or (lambda x: True)\n    spans: List[Tuple[int, int]] = []\n\n    for start_index in range(len(sentence)):\n        last_end_index = min(start_index + max_span_width, len(sentence))\n        first_end_index = min(start_index + min_span_width - 1, len(sentence))\n        for end_index in range(first_end_index, last_end_index):\n            start = offset + start_index\n            end = offset + end_index\n            # add 1 to end index because span indices are inclusive.\n            if filter_function(sentence[slice(start_index, end_index + 1)]):\n                spans.append((start, end))\n    return spans\n\n\ndef bio_tags_to_spans(\n        tag_sequence: List[str], classes_to_ignore: List[str] = None\n) -> List[TypedStringSpan]:\n    \"\"\"\n    Given a sequence corresponding to BIO tags, extracts spans.\n    Spans are inclusive and can be of zero length, representing a single word span.\n    Ill-formed spans are also included (i.e those which do not start with a \"B-LABEL\"),\n    as otherwise it is possible to get a perfect precision score whilst still predicting\n    ill-formed spans in addition to the correct spans. This function works properly when\n    the spans are unlabeled (i.e., your labels are simply \"B\", \"I\", and \"O\").\n\n    # Parameters\n\n    tag_sequence : `List[str]`, required.\n        The integer class labels for a sequence.\n    classes_to_ignore : `List[str]`, optional (default = `None`).\n        A list of string class labels `excluding` the bio tag\n        which should be ignored when extracting spans.\n\n    # Returns\n\n    spans : `List[TypedStringSpan]`\n        The typed, extracted spans from the sequence, in the format (label, (span_start, span_end)).\n        Note that the label `does not` contain any BIO tag prefixes.\n    \"\"\"\n    classes_to_ignore = classes_to_ignore or []\n    spans: Set[Tuple[str, Tuple[int, int]]] = set()\n    span_start = 0\n    span_end = 0\n    active_conll_tag = None\n    for index, string_tag in enumerate(tag_sequence):\n        # Actual BIO tag.\n        bio_tag = string_tag[0]\n        if bio_tag not in [\"B\", \"I\", \"O\"]:\n            raise InvalidTagSequence(tag_sequence)\n        conll_tag = string_tag[2:]\n        if bio_tag == \"O\" or conll_tag in classes_to_ignore:\n            # The span has ended.\n            if active_conll_tag is not None:\n                spans.add((active_conll_tag, (span_start, span_end)))\n            active_conll_tag = None\n            # We don't care about tags we are\n            # told to ignore, so we do nothing.\n            continue\n        elif bio_tag == \"B\":\n            # We are entering a new span; reset indices\n            # and active tag to new span.\n            if active_conll_tag is not None:\n                spans.add((active_conll_tag, (span_start, span_end)))\n            active_conll_tag = conll_tag\n            span_start = index\n            span_end = index\n        elif bio_tag == \"I\" and conll_tag == active_conll_tag:\n            # We're inside a span.\n            span_end += 1\n        else:\n            # This is the case the bio label is an \"I\", but either:\n            # 1) the span hasn't started - i.e. an ill formed span.\n            # 2) The span is an I tag for a different conll annotation.\n            # We'll process the previous span if it exists, but also\n            # include this span. This is important, because otherwise,\n            # a model may get a perfect F1 score whilst still including\n            # false positive ill-formed spans.\n            if active_conll_tag is not None:\n                spans.add((active_conll_tag, (span_start, span_end)))\n            active_conll_tag = conll_tag\n            span_start = index\n            span_end = index\n    # Last token might have been a part of a valid span.\n    if active_conll_tag is not None:\n        spans.add((active_conll_tag, (span_start, span_end)))\n    return list(spans)\n\n\ndef iob1_tags_to_spans(\n        tag_sequence: List[str], classes_to_ignore: List[str] = None\n) -> List[TypedStringSpan]:\n    \"\"\"\n    Given a sequence corresponding to IOB1 tags, extracts spans.\n    Spans are inclusive and can be of zero length, representing a single word span.\n    Ill-formed spans are also included (i.e., those where \"B-LABEL\" is not preceded\n    by \"I-LABEL\" or \"B-LABEL\").\n\n    # Parameters\n\n    tag_sequence : `List[str]`, required.\n        The integer class labels for a sequence.\n    classes_to_ignore : `List[str]`, optional (default = `None`).\n        A list of string class labels `excluding` the bio tag\n        which should be ignored when extracting spans.\n\n    # Returns\n\n    spans : `List[TypedStringSpan]`\n        The typed, extracted spans from the sequence, in the format (label, (span_start, span_end)).\n        Note that the label `does not` contain any BIO tag prefixes.\n    \"\"\"\n    classes_to_ignore = classes_to_ignore or []\n    spans: Set[Tuple[str, Tuple[int, int]]] = set()\n    span_start = 0\n    span_end = 0\n    active_conll_tag = None\n    prev_bio_tag = None\n    prev_conll_tag = None\n    for index, string_tag in enumerate(tag_sequence):\n        curr_bio_tag = string_tag[0]\n        curr_conll_tag = string_tag[2:]\n\n        if curr_bio_tag not in [\"B\", \"I\", \"O\"]:\n            raise InvalidTagSequence(tag_sequence)\n        if curr_bio_tag == \"O\" or curr_conll_tag in classes_to_ignore:\n            # The span has ended.\n            if active_conll_tag is not None:\n                spans.add((active_conll_tag, (span_start, span_end)))\n            active_conll_tag = None\n        elif _iob1_start_of_chunk(prev_bio_tag, prev_conll_tag, curr_bio_tag, curr_conll_tag):\n            # We are entering a new span; reset indices\n            # and active tag to new span.\n            if active_conll_tag is not None:\n                spans.add((active_conll_tag, (span_start, span_end)))\n            active_conll_tag = curr_conll_tag\n            span_start = index\n            span_end = index\n        else:\n            # bio_tag == \"I\" and curr_conll_tag == active_conll_tag\n            # We're continuing a span.\n            span_end += 1\n\n        prev_bio_tag = string_tag[0]\n        prev_conll_tag = string_tag[2:]\n    # Last token might have been a part of a valid span.\n    if active_conll_tag is not None:\n        spans.add((active_conll_tag, (span_start, span_end)))\n    return list(spans)\n\n\ndef _iob1_start_of_chunk(\n        prev_bio_tag: Optional[str],\n        prev_conll_tag: Optional[str],\n        curr_bio_tag: str,\n        curr_conll_tag: str,\n) -> bool:\n    if curr_bio_tag == \"B\":\n        return True\n    if curr_bio_tag == \"I\" and prev_bio_tag == \"O\":\n        return True\n    if curr_bio_tag != \"O\" and prev_conll_tag != curr_conll_tag:\n        return True\n    return False\n\n\ndef bioul_tags_to_spans(\n        tag_sequence: List[str], classes_to_ignore: List[str] = None\n) -> List[TypedStringSpan]:\n    \"\"\"\n    Given a sequence corresponding to BIOUL tags, extracts spans.\n    Spans are inclusive and can be of zero length, representing a single word span.\n    Ill-formed spans are not allowed and will raise `InvalidTagSequence`.\n    This function works properly when the spans are unlabeled (i.e., your labels are\n    simply \"B\", \"I\", \"O\", \"U\", and \"L\").\n\n    # Parameters\n\n    tag_sequence : `List[str]`, required.\n        The tag sequence encoded in BIOUL, e.g. [\"B-PER\", \"L-PER\", \"O\"].\n    classes_to_ignore : `List[str]`, optional (default = `None`).\n        A list of string class labels `excluding` the bio tag\n        which should be ignored when extracting spans.\n\n    # Returns\n\n    spans : `List[TypedStringSpan]`\n        The typed, extracted spans from the sequence, in the format (label, (span_start, span_end)).\n    \"\"\"\n    spans = []\n    classes_to_ignore = classes_to_ignore or []\n    index = 0\n    while index < len(tag_sequence):\n        label = tag_sequence[index]\n        if label[0] == \"U\":\n            spans.append((label.partition(\"-\")[2], (index, index)))\n        elif label[0] == \"B\":\n            start = index\n            while label[0] != \"L\":\n                index += 1\n                if index >= len(tag_sequence):\n                    raise InvalidTagSequence(tag_sequence)\n                label = tag_sequence[index]\n                if not (label[0] == \"I\" or label[0] == \"L\"):\n                    raise InvalidTagSequence(tag_sequence)\n            spans.append((label.partition(\"-\")[2], (start, index)))\n        else:\n            if label != \"O\":\n                raise InvalidTagSequence(tag_sequence)\n        index += 1\n    return [span for span in spans if span[0] not in classes_to_ignore]\n\n\ndef iobes_tags_to_spans(\n        tag_sequence: List[str], classes_to_ignore: List[str] = None\n) -> List[TypedStringSpan]:\n    \"\"\"\n    Given a sequence corresponding to BIOUL tags, extracts spans.\n    Spans are inclusive and can be of zero length, representing a single word span.\n    Ill-formed spans are not allowed and will raise `InvalidTagSequence`.\n    This function works properly when the spans are unlabeled (i.e., your labels are\n    simply \"B\", \"I\", \"O\", \"U\", and \"L\").\n\n    # Parameters\n\n    tag_sequence : `List[str]`, required.\n        The tag sequence encoded in BIOUL, e.g. [\"B-PER\", \"L-PER\", \"O\"].\n    classes_to_ignore : `List[str]`, optional (default = `None`).\n        A list of string class labels `excluding` the bio tag\n        which should be ignored when extracting spans.\n\n    # Returns\n\n    spans : `List[TypedStringSpan]`\n        The typed, extracted spans from the sequence, in the format (label, (span_start, span_end)).\n    \"\"\"\n    spans = []\n    classes_to_ignore = classes_to_ignore or []\n    index = 0\n    while index < len(tag_sequence):\n        label = tag_sequence[index]\n        if label[0] == \"S\":\n            spans.append((label.partition(\"-\")[2], (index, index)))\n        elif label[0] == \"B\":\n            start = index\n            while label[0] != \"E\":\n                index += 1\n                if index >= len(tag_sequence):\n                    raise InvalidTagSequence(tag_sequence)\n                label = tag_sequence[index]\n                if not (label[0] == \"I\" or label[0] == \"E\"):\n                    raise InvalidTagSequence(tag_sequence)\n            spans.append((label.partition(\"-\")[2], (start, index)))\n        else:\n            if label != \"O\":\n                raise InvalidTagSequence(tag_sequence)\n        index += 1\n    return [span for span in spans if span[0] not in classes_to_ignore]\n\n\ndef iob1_to_bioul(tag_sequence: List[str]) -> List[str]:\n    warnings.warn(\n        \"iob1_to_bioul has been replaced with 'to_bioul' to allow more encoding options.\",\n        FutureWarning,\n    )\n    return to_bioul(tag_sequence)\n\n\ndef to_bioul(tag_sequence: List[str], encoding: str = \"IOB1\") -> List[str]:\n    \"\"\"\n    Given a tag sequence encoded with IOB1 labels, recode to BIOUL.\n\n    In the IOB1 scheme, I is a token inside a span, O is a token outside\n    a span and B is the beginning of span immediately following another\n    span of the same type.\n\n    In the BIO scheme, I is a token inside a span, O is a token outside\n    a span and B is the beginning of a span.\n\n    # Parameters\n\n    tag_sequence : `List[str]`, required.\n        The tag sequence encoded in IOB1, e.g. [\"I-PER\", \"I-PER\", \"O\"].\n    encoding : `str`, optional, (default = `\"IOB1\"`).\n        The encoding type to convert from. Must be either \"IOB1\" or \"BIO\".\n\n    # Returns\n\n    bioul_sequence : `List[str]`\n        The tag sequence encoded in IOB1, e.g. [\"B-PER\", \"L-PER\", \"O\"].\n    \"\"\"\n    if encoding not in {\"IOB1\", \"BIO\"}:\n        raise ValueError(f\"Invalid encoding {encoding} passed to 'to_bioul'.\")\n\n    def replace_label(full_label, new_label):\n        # example: full_label = 'I-PER', new_label = 'U', returns 'U-PER'\n        parts = list(full_label.partition(\"-\"))\n        parts[0] = new_label\n        return \"\".join(parts)\n\n    def pop_replace_append(in_stack, out_stack, new_label):\n        # pop the last element from in_stack, replace the label, append\n        # to out_stack\n        tag = in_stack.pop()\n        new_tag = replace_label(tag, new_label)\n        out_stack.append(new_tag)\n\n    def process_stack(stack, out_stack):\n        # process a stack of labels, add them to out_stack\n        if len(stack) == 1:\n            # just a U token\n            pop_replace_append(stack, out_stack, \"U\")\n        else:\n            # need to code as BIL\n            recoded_stack = []\n            pop_replace_append(stack, recoded_stack, \"L\")\n            while len(stack) >= 2:\n                pop_replace_append(stack, recoded_stack, \"I\")\n            pop_replace_append(stack, recoded_stack, \"B\")\n            recoded_stack.reverse()\n            out_stack.extend(recoded_stack)\n\n    # Process the tag_sequence one tag at a time, adding spans to a stack,\n    # then recode them.\n    bioul_sequence = []\n    stack: List[str] = []\n\n    for label in tag_sequence:\n        # need to make a dict like\n        # token = {'token': 'Matt', \"labels\": {'conll2003': \"B-PER\"}\n        #                   'gold': 'I-PER'}\n        # where 'gold' is the raw value from the CoNLL data set\n\n        if label == \"O\" and len(stack) == 0:\n            bioul_sequence.append(label)\n        elif label == \"O\" and len(stack) > 0:\n            # need to process the entries on the stack plus this one\n            process_stack(stack, bioul_sequence)\n            bioul_sequence.append(label)\n        elif label[0] == \"I\":\n            # check if the previous type is the same as this one\n            # if it is then append to stack\n            # otherwise this start a new entity if the type\n            # is different\n            if len(stack) == 0:\n                if encoding == \"BIO\":\n                    raise InvalidTagSequence(tag_sequence)\n                stack.append(label)\n            else:\n                # check if the previous type is the same as this one\n                this_type = label.partition(\"-\")[2]\n                prev_type = stack[-1].partition(\"-\")[2]\n                if this_type == prev_type:\n                    stack.append(label)\n                else:\n                    if encoding == \"BIO\":\n                        raise InvalidTagSequence(tag_sequence)\n                    # a new entity\n                    process_stack(stack, bioul_sequence)\n                    stack.append(label)\n        elif label[0] == \"B\":\n            if len(stack) > 0:\n                process_stack(stack, bioul_sequence)\n            stack.append(label)\n        else:\n            raise InvalidTagSequence(tag_sequence)\n\n    # process the stack\n    if len(stack) > 0:\n        process_stack(stack, bioul_sequence)\n\n    return bioul_sequence\n\n\ndef bmes_tags_to_spans(\n        tag_sequence: List[str], classes_to_ignore: List[str] = None\n) -> List[TypedStringSpan]:\n    \"\"\"\n    Given a sequence corresponding to BMES tags, extracts spans.\n    Spans are inclusive and can be of zero length, representing a single word span.\n    Ill-formed spans are also included (i.e those which do not start with a \"B-LABEL\"),\n    as otherwise it is possible to get a perfect precision score whilst still predicting\n    ill-formed spans in addition to the correct spans.\n    This function works properly when the spans are unlabeled (i.e., your labels are\n    simply \"B\", \"M\", \"E\" and \"S\").\n\n    # Parameters\n\n    tag_sequence : `List[str]`, required.\n        The integer class labels for a sequence.\n    classes_to_ignore : `List[str]`, optional (default = `None`).\n        A list of string class labels `excluding` the bio tag\n        which should be ignored when extracting spans.\n\n    # Returns\n\n    spans : `List[TypedStringSpan]`\n        The typed, extracted spans from the sequence, in the format (label, (span_start, span_end)).\n        Note that the label `does not` contain any BIO tag prefixes.\n    \"\"\"\n\n    def extract_bmes_tag_label(text):\n        bmes_tag = text[0]\n        label = text[2:]\n        return bmes_tag, label\n\n    spans: List[Tuple[str, List[int]]] = []\n    prev_bmes_tag: Optional[str] = None\n    for index, tag in enumerate(tag_sequence):\n        bmes_tag, label = extract_bmes_tag_label(tag)\n        if bmes_tag in (\"B\", \"S\"):\n            # Regardless of tag, we start a new span when reaching B & S.\n            spans.append((label, [index, index]))\n        elif bmes_tag in (\"M\", \"E\") and prev_bmes_tag in (\"B\", \"M\") and spans[-1][0] == label:\n            # Only expand the span if\n            # 1. Valid transition: B/M -> M/E.\n            # 2. Matched label.\n            spans[-1][1][1] = index\n        else:\n            # Best effort split for invalid span.\n            spans.append((label, [index, index]))\n        # update previous BMES tag.\n        prev_bmes_tag = bmes_tag\n\n    classes_to_ignore = classes_to_ignore or []\n    return [\n        # to tuple.\n        (span[0], (span[1][0], span[1][1]))\n        for span in spans\n        if span[0] not in classes_to_ignore\n    ]"
  },
  {
    "path": "hanlp/utils/string_util.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-08-25 00:19\nimport unicodedata\nfrom typing import List, Dict, Tuple\n\n\ndef format_scores(results: Dict[str, float]) -> str:\n    return ' - '.join(f'{k}: {v:.4f}' for (k, v) in results.items())\n\n\ndef ispunct(token):\n    return all(unicodedata.category(char).startswith('P')\n               for char in token)\n\n\ndef split_long_sentence_into(tokens: List[str], max_seq_length, sent_delimiter=None, char_level=False,\n                             hard_constraint=False):\n    punct_offset = [i for i, x in enumerate(tokens) if\n                    ((sent_delimiter and x in sent_delimiter) or (not sent_delimiter and ispunct(x)))]\n    if not punct_offset:\n        # treat every token as punct\n        punct_offset = [i for i in range(len(tokens))]\n    punct_offset += [len(tokens)]\n    token_to_char_offset = []\n    if char_level:\n        offset = 0\n        for token in tokens:\n            token_to_char_offset.append(offset)\n            offset += len(token)\n        token_to_char_offset.append(offset)\n\n    start = 0\n    for i, offset in enumerate(punct_offset[:-1]):\n        end = punct_offset[i + 1]\n        length_at_next_punct = _len(start, end, token_to_char_offset, char_level)\n        if length_at_next_punct >= max_seq_length:\n            if hard_constraint:\n                yield from _gen_short_sent(tokens, start, offset, max_seq_length, token_to_char_offset, char_level)\n            else:\n                yield tokens[start: offset + 1]\n            start = offset + 1\n    offset = punct_offset[-1]\n    if start < offset:\n        offset -= 1\n        length_at_next_punct = _len(start, offset, token_to_char_offset, char_level)\n        if length_at_next_punct >= max_seq_length and hard_constraint:\n            yield from _gen_short_sent(tokens, start, offset, max_seq_length, token_to_char_offset, char_level)\n        else:\n            yield tokens[start:]\n\n\ndef _gen_short_sent(tokens, start, offset, max_seq_length, token_to_char_offset, char_level):\n    while start <= offset:\n        for j in range(offset + 1, start, -1):\n            if _len(start, j, token_to_char_offset, char_level) <= max_seq_length or j == start + 1:\n                yield tokens[start: j]\n                start = j\n                break\n\n\ndef _len(start, end, token_to_char_offset, char_level):\n    if char_level:\n        length_at_next_punct = token_to_char_offset[end] - token_to_char_offset[start]\n    else:\n        length_at_next_punct = end - start\n    return length_at_next_punct\n\n\ndef guess_delimiter(tokens):\n    if all(ord(c) < 128 for c in ''.join(tokens)):\n        delimiter_in_entity = ' '\n    else:\n        delimiter_in_entity = ''\n    return delimiter_in_entity\n\n\ndef split_long_sent(sent, delimiters, max_seq_length):\n    parts = []\n    offset = 0\n    for idx, char in enumerate(sent):\n        if char in delimiters:\n            parts.append(sent[offset:idx + 1])\n            offset = idx + 1\n    if not parts:\n        yield sent\n        return\n    short = []\n    for idx, part in enumerate(parts):\n        short += part\n        if idx == len(parts) - 1:\n            yield short\n        else:\n            if len(short) + len(parts[idx + 1]) > max_seq_length:\n                yield short\n                short = []\n\n\ndef possible_tokenization(text: str) -> List[Tuple[str]]:\n    \"\"\"Enumerate all possible tokenizations of a text.\n\n    Args:\n        text: A text.\n\n    Returns: All possible tokenizations.\n\n    \"\"\"\n    states = [((), ())]\n    for c in text:\n        new_states = []\n        for t, b in states:\n            # to split\n            new_states.append((t + (''.join(b + (c,)),), ()))\n            # not to split\n            new_states.append((t, b + (c,)))\n        states = new_states\n    return [t for t, b in states if not b]\n"
  },
  {
    "path": "hanlp/utils/tf_util.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-08-27 01:27\nimport json\nimport logging\nimport os\nimport random\nfrom typing import List\n\nimport numpy as np\n\nfrom hanlp_common.constant import PAD\n\n\ndef set_gpu(idx=0):\n    \"\"\"Restrict TensorFlow to only use the GPU of idx\n\n    Args:\n      idx:  (Default value = 0)\n\n    Returns:\n\n    \n    \"\"\"\n    gpus = get_visible_gpus()\n    if gpus:\n        try:\n            tf.config.experimental.set_visible_devices(gpus[idx], 'GPU')\n            logical_devices = tf.config.experimental.list_logical_devices('GPU')\n            assert len(logical_devices) == 1\n        except RuntimeError as e:\n            # Virtual devices must be set before GPUs have been initialized\n            # print(e)\n            raise e\n\n\ndef get_visible_gpus():\n    gpus = tf.config.experimental.list_physical_devices('GPU')\n    return gpus\n\n\ndef set_gpu_memory_growth(growth=True):\n    gpus = get_visible_gpus()\n    if gpus:\n        try:\n            # Currently, memory growth needs to be the same across GPUs\n            for gpu in gpus:\n                tf.config.experimental.set_memory_growth(gpu, growth)\n        except RuntimeError as e:\n            # Memory growth must be set before GPUs have been initialized\n            # print(e)\n            raise e\n\n\ndef nice_gpu():\n    \"\"\"Use GPU nicely.\"\"\"\n    set_gpu_memory_growth()\n    set_gpu()\n\n\ndef shut_up_python_logging():\n    logging.getLogger('tensorflow').setLevel(logging.ERROR)\n    import absl.logging\n    logging.root.removeHandler(absl.logging._absl_handler)\n    absl.logging._warn_preinit_stderr = False\n\n\ndef set_tf_loglevel(level=logging.ERROR):\n    if level >= logging.FATAL:\n        os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'\n        os.environ['TF_CPP_MIN_VLOG_LEVEL'] = '3'\n    if level >= logging.ERROR:\n        os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'\n        os.environ['TF_CPP_MIN_VLOG_LEVEL'] = '2'\n    if level >= logging.WARNING:\n        os.environ['TF_CPP_MIN_LOG_LEVEL'] = '1'\n        os.environ['TF_CPP_MIN_VLOG_LEVEL'] = '1'\n    else:\n        os.environ['TF_CPP_MIN_LOG_LEVEL'] = '0'\n        os.environ['TF_CPP_MIN_VLOG_LEVEL'] = '0'\n    shut_up_python_logging()\n    logging.getLogger('tensorflow').setLevel(level)\n\n\nset_tf_loglevel()\n\nshut_up_python_logging()\nimport tensorflow as tf\n\nnice_gpu()\n\n\ndef size_of_dataset(dataset: tf.data.Dataset) -> int:\n    count = 0\n    for element in dataset.unbatch().batch(1):\n        count += 1\n    return count\n\n\ndef summary_of_model(model: tf.keras.Model):\n    \"\"\"https://stackoverflow.com/a/53668338/3730690\n\n    Args:\n      model: tf.keras.Model: \n\n    Returns:\n\n    \n    \"\"\"\n    if not model.built:\n        return 'model structure unknown until calling fit() with some data'\n    line_list = []\n    model.summary(print_fn=lambda x: line_list.append(x))\n    summary = \"\\n\".join(line_list)\n    return summary\n\n\ndef register_custom_cls(custom_cls, name=None):\n    if not name:\n        name = custom_cls.__name__\n    tf.keras.utils.get_custom_objects()[name] = custom_cls\n\n\ndef set_seed_tf(seed=233):\n    tf.random.set_seed(seed)\n    np.random.seed(seed)\n    random.seed(seed)\n\n\ndef nice():\n    nice_gpu()\n    set_seed_tf()\n\n\ndef hanlp_register(arg):\n    \"\"\"Registers a class with the Keras serialization framework.\n\n    Args:\n      arg: \n\n    Returns:\n\n    \"\"\"\n    class_name = arg.__name__\n    registered_name = 'HanLP' + '>' + class_name\n\n    # if tf_inspect.isclass(arg) and not hasattr(arg, 'get_config'):\n    #     raise ValueError(\n    #         'Cannot register a class that does not have a get_config() method.')\n\n    tf.keras.utils.get_custom_objects()[registered_name] = arg\n\n    return arg\n\n\ndef tensor_is_eager(tensor: tf.Tensor):\n    return hasattr(tensor, 'numpy')\n\n\ndef copy_mask(src: tf.Tensor, dst: tf.Tensor):\n    mask = getattr(src, '_keras_mask', None)\n    if mask is not None:\n        dst._keras_mask = mask\n    return mask\n\n\ndef get_callback_by_class(callbacks: List[tf.keras.callbacks.Callback], cls) -> tf.keras.callbacks.Callback:\n    for callback in callbacks:\n        if isinstance(callback, cls):\n            return callback\n\n\ndef tf_bernoulli(shape, p, dtype=None):\n    return tf.keras.backend.random_binomial(shape, p, dtype)\n\n\ndef str_tensor_to_str(str_tensor: tf.Tensor) -> str:\n    return str_tensor.numpy().decode('utf-8')\n\n\ndef str_tensor_2d_to_list(str_tensor: tf.Tensor, pad=PAD) -> List[List[str]]:\n    l = []\n    for i in str_tensor:\n        sent = []\n        for j in i:\n            j = str_tensor_to_str(j)\n            if j == pad:\n                break\n            sent.append(j)\n        l.append(sent)\n    return l\n\n\ndef str_tensor_to_list(pred):\n    return [tag.predict('utf-8') for tag in pred]\n\n\ndef format_metrics(metrics: List[tf.keras.metrics.Metric]):\n    return ' - '.join(f'{m.name}: {m.result():.4f}' for m in metrics)\n\n\nclass NumpyEncoder(json.JSONEncoder):\n    def default(self, obj):\n        \"\"\"Special json encoder for numpy types\n        See https://interviewbubble.com/typeerror-object-of-type-float32-is-not-json-serializable/\n\n        Args:\n            obj: Object to be json encoded.\n\n        Returns:\n            Json string.\n        \"\"\"\n        if isinstance(obj, (np.int_, np.intc, np.intp, np.int8,\n                            np.int16, np.int32, np.int64, np.uint8,\n                            np.uint16, np.uint32, np.uint64)):\n            return int(obj)\n        elif isinstance(obj, (np.float_, np.float16, np.float32,\n                              np.float64)):\n            return float(obj)\n        elif isinstance(obj, (np.ndarray,)):  #### This is the fix\n            return obj.tolist()\n        return json.JSONEncoder.default(self, obj)"
  },
  {
    "path": "hanlp/utils/time_util.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-08-27 00:01\nimport datetime\nimport logging\nimport sys\nimport time\nfrom typing import Union\n\nfrom hanlp.utils.log_util import ErasablePrinter, color_format, color_format_len\n\n\ndef human_time_delta(days, hours, minutes, seconds, delimiter=' ') -> str:\n    units = locals().copy()\n    units.pop('delimiter')\n    non_zero = False\n    result = []\n    for key, val in sorted(units.items()):\n        append = False\n        if non_zero:\n            append = True\n        elif val:\n            non_zero = True\n            append = True\n        if append:\n            result.append('{} {}'.format(val, key[0]))\n    if not non_zero:\n        return '0 s'\n    return delimiter.join(result)\n\n\ndef seconds_to_time_delta(seconds):\n    seconds = round(seconds)\n    days = seconds // 86400\n    hours = seconds // 3600 % 24\n    minutes = seconds // 60 % 60\n    seconds = seconds % 60\n    return days, hours, minutes, seconds\n\n\ndef report_time_delta(seconds, human=True):\n    days, hours, minutes, seconds = seconds_to_time_delta(seconds)\n    if human:\n        return human_time_delta(days, hours, minutes, seconds)\n    return days, hours, minutes, seconds\n\n\nclass HumanTimeDelta(object):\n\n    def __init__(self, delta_seconds) -> None:\n        super().__init__()\n        self.delta_seconds = delta_seconds\n\n    def report(self, human=True):\n        return report_time_delta(self.delta_seconds, human)\n\n    def __str__(self) -> str:\n        return self.report(human=True)\n\n    def __truediv__(self, scalar):\n        return HumanTimeDelta(self.delta_seconds / scalar)\n\n\nclass CountdownTimer(ErasablePrinter):\n\n    def __init__(self, total: int, out=sys.stdout) -> None:\n        super().__init__(out=out)\n        self.total = total\n        self.current = 0\n        self.start = time.time()\n        self.finished_in = None\n        self.last_log_time = 0\n\n    def update(self, n=1):\n        self.current += n\n        self.current = min(self.total, self.current)\n        if self.current == self.total:\n            self.finished_in = time.time() - self.start\n\n    @property\n    def ratio(self) -> str:\n        return f'{self.current}/{self.total}'\n\n    @property\n    def ratio_percentage(self) -> str:\n        return f'{self.current / self.total:.2%}'\n\n    @property\n    def eta(self) -> float:\n        elapsed = self.elapsed\n        if self.finished_in:\n            eta = 0\n        else:\n            eta = elapsed / max(self.current, 0.1) * (self.total - self.current)\n\n        return eta\n\n    @property\n    def elapsed(self) -> float:\n        if self.finished_in:\n            elapsed = self.finished_in\n        else:\n            elapsed = time.time() - self.start\n        return elapsed\n\n    @property\n    def elapsed_human(self) -> str:\n        return human_time_delta(*seconds_to_time_delta(self.elapsed))\n\n    @property\n    def elapsed_average(self) -> float:\n        return self.elapsed / self.current\n\n    @property\n    def elapsed_average_human(self) -> str:\n        return human_time_delta(*seconds_to_time_delta(self.elapsed_average))\n\n    @property\n    def eta_human(self) -> str:\n        return human_time_delta(*seconds_to_time_delta(self.eta))\n\n    @property\n    def total_time(self) -> float:\n        elapsed = self.elapsed\n        if self.finished_in:\n            t = self.finished_in\n        else:\n            t = elapsed / max(self.current, 1) * self.total\n\n        return t\n\n    @property\n    def total_time_human(self) -> str:\n        return human_time_delta(*seconds_to_time_delta(self.total_time))\n\n    def stop(self, total=None):\n        if not self.finished_in or total:\n            self.finished_in = time.time() - self.start\n            if not total:\n                self.total = self.current\n            else:\n                self.current = total\n                self.total = total\n\n    @property\n    def et_eta(self):\n        _ = self.elapsed\n        if self.finished_in:\n            return self.elapsed\n        else:\n            return self.eta\n\n    @property\n    def et_eta_human(self):\n        text = human_time_delta(*seconds_to_time_delta(self.et_eta))\n        if self.finished_in:\n            return f'ET: {text}'\n        else:\n            return f'ETA: {text}'\n\n    @property\n    def finished(self):\n        return self.total == self.current\n\n    def log(self, info=None, ratio_percentage=True, ratio=True, step=1, interval=0.5, erase=True,\n            logger: Union[logging.Logger, bool] = None, newline=False, ratio_width=None):\n        self.update(step)\n        now = time.time()\n        if now - self.last_log_time > interval or self.finished:\n            cells = []\n            if ratio_percentage:\n                cells.append(self.ratio_percentage)\n            if ratio:\n                ratio = self.ratio\n                if not ratio_width:\n                    ratio_width = self.ratio_width\n                ratio = ratio.rjust(ratio_width)\n                cells.append(ratio)\n            cells += [info, self.et_eta_human]\n            cells = [x for x in cells if x]\n            msg = f'{\" \".join(cells)}'\n            self.last_log_time = now\n            self.print(msg, newline, erase, logger)\n\n    @property\n    def ratio_width(self) -> int:\n        return len(f'{self.total}') * 2 + 1\n\n    def print(self, msg, newline=False, erase=True, logger=None):\n        self.erase()\n        msg_len = 0 if newline else len(msg)\n        if self.finished and logger:\n            sys.stdout.flush()\n            if isinstance(logger, logging.Logger):\n                logger.info(msg)\n        else:\n            msg, msg_len = color_format_len(msg)\n            sys.stdout.write(msg)\n            if newline:\n                sys.stdout.write('\\n')\n                msg_len = 0\n        self._last_print_width = msg_len\n        if self.finished and not logger:\n            if erase:\n                self.erase()\n            else:\n                sys.stdout.write(\"\\n\")\n                self._last_print_width = 0\n        sys.stdout.flush()\n\n\nclass Timer(object):\n    def __init__(self) -> None:\n        self.last = time.time()\n\n    def start(self):\n        self.last = time.time()\n\n    def stop(self) -> HumanTimeDelta:\n        now = time.time()\n        seconds = now - self.last\n        self.last = now\n        return HumanTimeDelta(seconds)\n\n\ndef now_human(year='y'):\n    now = datetime.datetime.now()\n    return now.strftime(f\"%{year}-%m-%d %H:%M:%S\")\n\n\ndef now_datetime():\n    return now_human('Y')\n\n\ndef now_filename(fmt=\"%y%m%d_%H%M%S\"):\n    \"\"\"Generate filename using current datetime, in 20180102_030405 format\n\n    Args:\n      fmt:  (Default value = \"%y%m%d_%H%M%S\")\n\n    Returns:\n\n    \n    \"\"\"\n    now = datetime.datetime.now()\n    return now.strftime(fmt)\n"
  },
  {
    "path": "hanlp/utils/torch_util.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-05-09 15:52\nimport os\nimport random\nimport time\nfrom typing import List, Union, Dict, Tuple\n\nimport numpy as np\nimport torch\nfrom pynvml import nvmlDeviceGetHandleByIndex, nvmlDeviceGetMemoryInfo, nvmlInit, nvmlShutdown, nvmlDeviceGetCount\nfrom torch import nn\nfrom torch.nn.utils.rnn import pad_sequence\n\nfrom hanlp.utils.io_util import get_resource, replace_ext, TimingFileIterator\nfrom hanlp.utils.log_util import logger, flash\nfrom hanlp_common.constant import HANLP_VERBOSE\nfrom hanlp_common.io import load_pickle, save_pickle\n\n\ndef gpus_available() -> Dict[int, float]:\n    if not torch.cuda.is_available():\n        return dict()\n    try:\n        nvmlInit()\n        gpus = {}\n        visible_devices = os.environ.get('CUDA_VISIBLE_DEVICES', None)\n        if visible_devices is None:\n            visible_devices = list(range(nvmlDeviceGetCount()))\n        else:\n            visible_devices = {int(x.strip()) for x in visible_devices.split(',')}\n        for i, real_id in enumerate(visible_devices):\n            h = nvmlDeviceGetHandleByIndex(real_id)\n            info = nvmlDeviceGetMemoryInfo(h)\n            total = info.total\n            free = info.free\n            ratio = free / total\n            gpus[i] = ratio\n            # print(f'total    : {info.total}')\n            # print(f'free     : {info.free}')\n            # print(f'used     : {info.used}')\n            # t = torch.cuda.get_device_properties(0).total_memory\n            # c = torch.cuda.memory_cached(0)\n            # a = torch.cuda.memory_allocated(0)\n            # print(t, c, a)\n        nvmlShutdown()\n        return dict(sorted(gpus.items(), key=lambda x: x[1], reverse=True))\n    except Exception as e:\n        logger.debug(f'Failed to get gpu info due to {e}')\n        return dict((i, 1.0) for i in range(torch.cuda.device_count()))\n\n\ndef cuda_devices(query=None) -> List[int]:\n    \"\"\"Decide which GPUs to use\n\n    Args:\n      query:  (Default value = None)\n\n    Returns:\n\n    \n    \"\"\"\n    if isinstance(query, list):\n        if len(query) == 0:\n            return [-1]\n        return query\n    if query is None:\n        query = gpus_available()\n        if not query:\n            return []\n        size, idx = max((v, k) for k, v in query.items())\n        # When multiple GPUs have the same size, randomly pick one to avoid conflicting\n        gpus_with_same_size = [k for k, v in query.items() if v == size]\n        query = random.choice(gpus_with_same_size)\n    if isinstance(query, float):\n        gpus = gpus_available()\n        if not query:\n            return []\n        query = [k for k, v in gpus.items() if v > query]\n    elif isinstance(query, int):\n        query = [query]\n    return query\n\n\ndef pad_lists(sequences: List[List], dtype=torch.long, padding_value=0):\n    return pad_sequence([torch.tensor(x, dtype=dtype) for x in sequences], True, padding_value)\n\n\ndef set_seed(seed=233, dont_care_speed=False):\n    \"\"\"Copied from https://github.com/huggingface/transformers/blob/7b75aa9fa55bee577e2c7403301ed31103125a35/src/transformers/trainer.py#L76\n\n    Args:\n      seed:  (Default value = 233)\n      dont_care_speed: True may have a negative single-run performance impact, but ensures deterministic\n\n    Returns:\n\n    \n    \"\"\"\n    random.seed(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    # ^^ safe to call this function even if cuda is not available\n    torch.cuda.manual_seed_all(seed)\n    if dont_care_speed:\n        torch.backends.cudnn.deterministic = True\n        torch.backends.cudnn.benchmark = False\n\n\ndef batched_index_select(input, index, dim=1):\n    \"\"\"\n\n    Args:\n      input: B x * x ... x *\n      index: B x M\n      dim:  (Default value = 1)\n\n    Returns:\n\n    \n    \"\"\"\n    views = [input.shape[0]] + [1 if i != dim else -1 for i in range(1, len(input.shape))]\n    expanse = list(input.shape)\n    expanse[0] = -1\n    expanse[dim] = -1\n    index = index.view(views).expand(expanse)\n    return torch.gather(input, dim, index)\n\n\ndef truncated_normal_(tensor, mean=0, std=1):\n    size = tensor.shape\n    tmp = tensor.new_empty(size + (4,)).normal_()\n    valid = (tmp < 2) & (tmp > -2)\n    ind = valid.max(-1, keepdim=True)[1]\n    tensor.data.copy_(tmp.gather(-1, ind).squeeze(-1))\n    tensor.data.mul_(std).add_(mean)\n    return tensor\n\n\ndef dtype_of(e: Union[int, bool, float]):\n    if isinstance(e, bool):\n        return torch.bool\n    if isinstance(e, int):\n        return torch.long\n    if isinstance(e, float):\n        return torch.float\n    raise ValueError(f'Unsupported type of {repr(e)}')\n\n\ndef mean_model(model: torch.nn.Module):\n    return float(torch.mean(torch.stack([torch.sum(p) for p in model.parameters() if p.requires_grad])))\n\n\ndef main():\n    start = time.time()\n    print(gpus_available())\n    print(time.time() - start)\n    # print(gpus_available())\n    # print(cuda_devices())\n    # print(cuda_devices(0.1))\n\n\nif __name__ == '__main__':\n    main()\n\n\ndef clip_grad_norm(model: nn.Module, grad_norm, transformer: nn.Module = None, transformer_grad_norm=None):\n    if transformer_grad_norm is None:\n        if grad_norm is not None:\n            nn.utils.clip_grad_norm_(filter(lambda p: p.requires_grad, model.parameters()), grad_norm)\n    else:\n        is_transformer = []\n        non_transformer = []\n        transformer = set(transformer.parameters())\n        for p in model.parameters():\n            if not p.requires_grad:\n                continue\n            if p in transformer:\n                is_transformer.append(p)\n            else:\n                non_transformer.append(p)\n        nn.utils.clip_grad_norm_(non_transformer, grad_norm)\n        nn.utils.clip_grad_norm_(is_transformer, transformer_grad_norm)\n\n\ndef load_word2vec(path, delimiter=' ', cache=True) -> Tuple[Dict[str, np.ndarray], int]:\n    realpath = get_resource(path)\n    binpath = replace_ext(realpath, '.pkl')\n    if cache:\n        try:\n            flash('Loading word2vec from cache [blink][yellow]...[/yellow][/blink]')\n            word2vec, dim = load_pickle(binpath)\n            flash('')\n            return word2vec, dim\n        except IOError:\n            pass\n\n    dim = None\n    word2vec = dict()\n    f = TimingFileIterator(realpath)\n    for idx, line in enumerate(f):\n        f.log('Loading word2vec from text file [blink][yellow]...[/yellow][/blink]')\n        line = line.rstrip().split(delimiter)\n        if len(line) > 2:\n            if dim is None:\n                dim = len(line)\n            else:\n                if len(line) != dim:\n                    logger.warning('{}#{} length mismatches with {}'.format(path, idx + 1, dim))\n                    continue\n            word, vec = line[0], line[1:]\n            word2vec[word] = np.array(vec, dtype=np.float32)\n    dim -= 1\n    if cache:\n        flash('Caching word2vec [blink][yellow]...[/yellow][/blink]')\n        save_pickle((word2vec, dim), binpath)\n        flash('')\n    return word2vec, dim\n\n\ndef load_word2vec_as_vocab_tensor(path, delimiter=' ', cache=True) -> Tuple[Dict[str, int], torch.Tensor]:\n    realpath = get_resource(path)\n    vocab_path = replace_ext(realpath, '.vocab')\n    matrix_path = replace_ext(realpath, '.pt')\n    if cache:\n        try:\n            if HANLP_VERBOSE:\n                flash('Loading vocab and matrix from cache [blink][yellow]...[/yellow][/blink]')\n            vocab = load_pickle(vocab_path)\n            matrix = torch.load(matrix_path, map_location='cpu')\n            if HANLP_VERBOSE:\n                flash('')\n            return vocab, matrix\n        except IOError:\n            pass\n\n    word2vec, dim = load_word2vec(path, delimiter, cache)\n    vocab = dict((k, i) for i, k in enumerate(word2vec.keys()))\n    matrix = torch.Tensor(np.stack(list(word2vec.values())))\n    if cache:\n        flash('Caching vocab and matrix [blink][yellow]...[/yellow][/blink]')\n        save_pickle(vocab, vocab_path)\n        torch.save(matrix, matrix_path)\n        flash('')\n    return vocab, matrix\n\n\ndef save_word2vec(word2vec: dict, filepath, delimiter=' '):\n    with open(filepath, 'w', encoding='utf-8') as out:\n        for w, v in word2vec.items():\n            out.write(f'{w}{delimiter}')\n            out.write(f'{delimiter.join(str(x) for x in v)}\\n')\n\n\ndef lengths_to_mask(seq_len, max_len=None):\n    r\"\"\"\n    .. code-block::\n\n        >>> seq_len = torch.arange(2, 16)\n        >>> mask = lengths_to_mask(seq_len)\n        >>> print(mask.size())\n        torch.Size([14, 15])\n        >>> seq_len = np.arange(2, 16)\n        >>> mask = lengths_to_mask(seq_len)\n        >>> print(mask.shape)\n        (14, 15)\n        >>> seq_len = torch.arange(2, 16)\n        >>> mask = lengths_to_mask(seq_len, max_len=100)\n        >>>print(mask.size())\n        torch.Size([14, 100])\n\n    :param torch.LongTensor seq_len: (B,)\n    :param int max_len: max sequence length。\n    :return:  torch.Tensor  (B, max_len)\n    \"\"\"\n    assert seq_len.dim() == 1, f\"seq_len can only have one dimension, got {seq_len.dim() == 1}.\"\n    batch_size = seq_len.size(0)\n    max_len = int(max_len) if max_len else seq_len.max().long()\n    broad_cast_seq_len = torch.arange(max_len).expand(batch_size, -1).to(seq_len)\n    mask = broad_cast_seq_len.lt(seq_len.unsqueeze(1))\n\n    return mask\n\n\ndef activation_from_name(name: str):\n    return getattr(torch.nn, name)\n\n\ndef filter_state_dict_safely(model_state: dict, load_state: dict):\n    safe_state = dict()\n    for k, v in load_state.items():\n        model_v = model_state.get(k, None)\n        if model_v is not None and model_v.shape == v.shape:\n            safe_state[k] = v\n    return safe_state\n"
  },
  {
    "path": "hanlp/version.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 19:26\n\n__version__ = '2.1.0-beta.64'\n\"\"\"HanLP version\"\"\"\n\n\nclass NotCompatible(Exception):\n    pass\n"
  },
  {
    "path": "plugins/README.md",
    "content": "# Plugins for HanLP\n\nThis directory contains modules shared across several individual packages or non core APIs.\nIf you plan to submit any plugins, please put it here too.\n\nFor developers, run the following set-up.\n\n```bash\npip install -e hanlp_trie\npip install -e hanlp_common\npip install -e hanlp_restful\n```"
  },
  {
    "path": "plugins/hanlp_common/README.md",
    "content": "# Common utilities and structures for HanLP\n\n[中文](https://github.com/hankcs/HanLP/tree/doc-zh) | [1.x](https://github.com/hankcs/HanLP/tree/1.x) | [forum](https://bbs.hankcs.com/) | [docker](https://github.com/WalterInSH/hanlp-jupyter-docker)\n\nThe multilingual NLP library for researchers and companies, built on PyTorch and TensorFlow 2.x, for advancing state-of-the-art deep learning techniques in both academia and industry. HanLP was designed from day one to be efficient, user friendly and extendable. It comes with pretrained models for various human languages including English, Chinese and many others. Currently, HanLP 2.0 is in alpha stage with more killer features on the roadmap. Discussions are welcomed on our [forum](https://bbs.hankcs.com/), while bug reports and feature requests are reserved for GitHub issues. For Java users, please checkout the [1.x](https://github.com/hankcs/HanLP/tree/1.x) branch.\n\n\n## Installation\n\n```bash\npip install hanlp\n```\n\n## License\n\nHanLP is licensed under **Apache License 2.0**. You can use HanLP in your commercial products for free. We would appreciate it if you add a link to HanLP on your website.\n\n"
  },
  {
    "path": "plugins/hanlp_common/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-16 22:20\n"
  },
  {
    "path": "plugins/hanlp_common/hanlp_common/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-16 22:21\n"
  },
  {
    "path": "plugins/hanlp_common/hanlp_common/amr.py",
    "content": "# MIT License\n#\n# Copyright (c) 2019 Sheng Zhang\n#\n# Permission is hereby granted, free of charge, to any person obtaining a copy\n# of this software and associated documentation files (the \"Software\"), to deal\n# in the Software without restriction, including without limitation the rights\n# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n# copies of the Software, and to permit persons to whom the Software is\n# furnished to do so, subject to the following conditions:\n#\n# The above copyright notice and this permission notice shall be included in all\n# copies or substantial portions of the Software.\n#\n# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n# SOFTWARE.\n\nimport json\nimport logging\nimport re\nimport traceback\nfrom collections import Counter, defaultdict\n\nfrom hanlp_common.io import eprint\n\ntry:\n    import networkx as nx\n    import penman\n    from penman import Triple\nexcept ModuleNotFoundError:\n    traceback.print_exc()\n    eprint('AMR support requires the full version which can be installed via:\\n'\n           'pip install hanlp_common[full]')\n    exit(1)\n\nDEFAULT_PADDING_TOKEN = \"@@PADDING@@\"\nDEFAULT_OOV_TOKEN = \"@@UNKNOWN@@\"\nlogger = logging.getLogger('amr')\n\n# Disable inverting ':mod' relation.\npenman.AMRCodec._inversions.pop('domain')\npenman.AMRCodec._deinversions.pop('mod')\n\namr_codec = penman.AMRCodec(indent=6)\n\nWORDSENSE_RE = re.compile(r'-\\d\\d$')\nQUOTED_RE = re.compile(r'^\".*\"$')\n\n\ndef is_abstract_token(token):\n    return re.search(r'^([A-Z]+_)+\\d+$', token) or re.search(r'^\\d0*$', token)\n\n\ndef is_english_punct(c):\n    return re.search(r'^[,.?!:;\"\\'-(){}\\[\\]]$', c)\n\n\ndef find_similar_token(token, tokens):\n    token = re.sub(r'-\\d\\d$', '', token)  # .lower())\n    for i, t in enumerate(tokens):\n        if token == t:\n            return tokens[i]\n        # t = t.lower()\n        # if (token == t or\n        #     (t.startswith(token) and len(token) > 3) or\n        #     token + 'd' == t or\n        #     token + 'ed' == t or\n        #     re.sub('ly$', 'le', t) == token or\n        #     re.sub('tive$', 'te', t) == token or\n        #     re.sub('tion$', 'te', t) == token or\n        #     re.sub('ied$', 'y', t) == token or\n        #     re.sub('ly$', '', t) == token\n        # ):\n        #     return tokens[i]\n    return None\n\n\nclass AMR:\n\n    def __init__(self,\n                 id=None,\n                 sentence=None,\n                 graph=None,\n                 tokens=None,\n                 lemmas=None,\n                 pos_tags=None,\n                 ner_tags=None,\n                 abstract_map=None,\n                 misc=None):\n        self.id = id\n        self.sentence = sentence\n        self.graph = graph\n        self.tokens = tokens\n        self.lemmas = lemmas\n        self.pos_tags = pos_tags\n        self.ner_tags = ner_tags\n        self.abstract_map = abstract_map\n        self.misc = misc\n\n    def is_named_entity(self, index):\n        return self.ner_tags[index] not in ('0', 'O')\n\n    def get_named_entity_span(self, index):\n        if self.ner_tags is None or not self.is_named_entity(index):\n            return []\n        span = [index]\n        tag = self.ner_tags[index]\n        prev = index - 1\n        while prev > 0 and self.ner_tags[prev] == tag:\n            span.append(prev)\n            prev -= 1\n        next = index + 1\n        while next < len(self.ner_tags) and self.ner_tags[next] == tag:\n            span.append(next)\n            next += 1\n        return span\n\n    def find_span_indexes(self, span):\n        for i, token in enumerate(self.tokens):\n            if token == span[0]:\n                _span = self.tokens[i: i + len(span)]\n                if len(_span) == len(span) and all(x == y for x, y in zip(span, _span)):\n                    return list(range(i, i + len(span)))\n        return None\n\n    def replace_span(self, indexes, new, pos=None, ner=None):\n        self.tokens = self.tokens[:indexes[0]] + new + self.tokens[indexes[-1] + 1:]\n        self.lemmas = self.lemmas[:indexes[0]] + new + self.lemmas[indexes[-1] + 1:]\n        if pos is None:\n            pos = [self.pos_tags[indexes[0]]]\n        self.pos_tags = self.pos_tags[:indexes[0]] + pos + self.pos_tags[indexes[-1] + 1:]\n        if ner is None:\n            ner = [self.ner_tags[indexes[0]]]\n        self.ner_tags = self.ner_tags[:indexes[0]] + ner + self.ner_tags[indexes[-1] + 1:]\n\n    def remove_span(self, indexes):\n        self.replace_span(indexes, [], [], [])\n\n    def __repr__(self):\n        fields = []\n        for k, v in dict(\n                id=self.id,\n                snt=self.sentence,\n                tokens=self.tokens,\n                lemmas=self.lemmas,\n                pos_tags=self.pos_tags,\n                ner_tags=self.ner_tags,\n                abstract_map=self.abstract_map,\n                misc=self.misc,\n                graph=self.graph\n        ).items():\n            if v is None:\n                continue\n            if k == 'misc':\n                fields += v\n            elif k == 'graph':\n                fields.append(str(v))\n            else:\n                if not isinstance(v, str):\n                    v = json.dumps(v)\n                fields.append('# ::{} {}'.format(k, v))\n        return '\\n'.join(fields)\n\n    def get_src_tokens(self):\n        return self.lemmas if self.lemmas else self.sentence.split()\n\n\nclass AMRNode:\n    attribute_priority = [\n        'instance', 'quant', 'mode', 'value', 'name', 'li', 'mod', 'frequency',\n        'month', 'day', 'year', 'time', 'unit', 'decade', 'poss'\n    ]\n\n    def __init__(self, identifier, attributes=None, copy_of=None):\n        self.identifier = identifier\n        if attributes is None:\n            self.attributes = []\n        else:\n            self.attributes = attributes\n            # self._sort_attributes()\n        self._num_copies = 0\n        self.copy_of = copy_of\n\n    def _sort_attributes(self):\n        def get_attr_priority(attr):\n            if attr in self.attribute_priority:\n                return self.attribute_priority.index(attr), attr\n            if not re.search(r'^(ARG|op|snt)', attr):\n                return len(self.attribute_priority), attr\n            else:\n                return len(self.attribute_priority) + 1, attr\n\n        self.attributes.sort(key=lambda x: get_attr_priority(x[0]))\n\n    def __hash__(self):\n        return hash(self.identifier)\n\n    def __eq__(self, other):\n        if not isinstance(other, AMRNode):\n            return False\n        return self.identifier == other.identifier\n\n    def __repr__(self):\n        ret = str(self.identifier)\n        for k, v in self.attributes:\n            if k == 'instance':\n                ret += ' / ' + v\n                break\n        return ret\n\n    def __str__(self):\n        ret = repr(self)\n        for key, value in self.attributes:\n            if key == 'instance':\n                continue\n            ret += '\\n\\t:{} {}'.format(key, value)\n        return ret\n\n    @property\n    def instance(self):\n        for key, value in self.attributes:\n            if key == 'instance':\n                return value\n        else:\n            return None\n\n    @property\n    def ops(self):\n        ops = []\n        for key, value in self.attributes:\n            if re.search(r'op\\d+', key):\n                ops.append((int(key[2:]), value))\n        if len(ops):\n            ops.sort(key=lambda x: x[0])\n        return [v for k, v in ops]\n\n    def copy(self):\n        attributes = None\n        if self.attributes is not None:\n            attributes = self.attributes[:]\n        self._num_copies += 1\n        copy = AMRNode(self.identifier + '_copy_{}'.format(self._num_copies), attributes, self)\n        return copy\n\n    def remove_attribute(self, attr, value):\n        self.attributes.remove((attr, value))\n\n    def add_attribute(self, attr, value):\n        self.attributes.append((attr, value))\n\n    def replace_attribute(self, attr, old, new):\n        index = self.attributes.index((attr, old))\n        self.attributes[index] = (attr, new)\n\n    def get_frame_attributes(self):\n        for k, v in self.attributes:\n            if isinstance(v, str) and re.search(r'-\\d\\d$', v):\n                yield k, v\n\n    def get_senseless_attributes(self):\n        for k, v in self.attributes:\n            if isinstance(v, str) and not re.search(r'-\\d\\d$', v):\n                yield k, v\n\n\nclass AMRGraph(penman.Graph):\n    edge_label_priority = (\n        'mod name time location degree poss domain quant manner unit purpose topic condition part-of compared-to '\n        'duration source ord beneficiary concession direction frequency consist-of example medium location-of '\n        'manner-of quant-of time-of instrument prep-in destination accompanier prep-with extent instrument-of age '\n        'path concession-of subevent-of prep-as prep-to prep-against prep-on prep-for degree-of prep-under part '\n        'condition-of prep-without topic-of season duration-of poss-of prep-from prep-at range purpose-of source-of '\n        'subevent example-of value path-of scale conj-as-if prep-into prep-by prep-on-behalf-of medium-of prep-among '\n        'calendar beneficiary-of prep-along-with extent-of age-of frequency-of dayperiod accompanier-of '\n        'destination-of prep-amid prep-toward prep-in-addition-to ord-of name-of weekday direction-of prep-out-of '\n        'timezone subset-of'.split())\n\n    def __init__(self, penman_graph):\n        super(AMRGraph, self).__init__()\n        self._triples = penman_graph._triples\n        self._top = penman_graph._top\n        self._build_extras()\n        self._src_tokens = []\n\n    def __str__(self):\n        self._triples = penman.alphanum_order(self._triples)\n        return amr_codec.encode(self)\n\n    def _build_extras(self):\n        G = nx.DiGraph()\n\n        self.variable_to_node = {}\n        for v in self.variables():\n            if type(v) is not str:\n                continue\n            attributes = [(t.relation, t.target) for t in self.attributes(source=v)]\n            node = AMRNode(v, attributes)\n            G.add_node(node)\n            self.variable_to_node[v] = node\n\n        edge_set = set()\n        for edge in self.edges():\n            if type(edge.source) is not str:\n                continue\n            source = self.variable_to_node[edge.source]\n            target = self.variable_to_node[edge.target]\n            relation = edge.relation\n\n            if relation == 'instance':\n                continue\n\n            if source == target:\n                continue\n\n            if edge.inverted:\n                source, target, relation = target, source, amr_codec.invert_relation(edge.relation)\n\n            if (source, target) in edge_set:\n                target = target.copy()\n\n            edge_set.add((source, target))\n            G.add_edge(source, target, label=relation)\n\n        self._G = G\n\n    def attributes(self, source=None, relation=None, target=None):\n        # Refine attributes because there's a bug in penman.attributes()\n        # See https://github.com/goodmami/penman/issues/29\n        attrmatch = lambda a: (\n                (source is None or source == a.source) and\n                (relation is None or relation == a.relation) and\n                (target is None or target == a.target)\n        )\n        variables = self.variables()\n        attrs = [t for t in self.triples() if t.target not in variables or t.relation == 'instance']\n        return list(filter(attrmatch, attrs))\n\n    def _update_penman_graph(self, triples):\n        self._triples = triples\n        if self._top not in self.variables():\n            self._top = None\n\n    def is_name_node(self, node):\n        edges = list(self._G.in_edges(node))\n        return any(self._G[source][target].get('label', None) == 'name' for source, target in edges)\n\n    def get_name_node_type(self, node):\n        edges = list(self._G.in_edges(node))\n        for source, target in edges:\n            if self._G[source][target].get('label', None) == 'name':\n                return source.instance\n        raise KeyError\n\n    def get_name_node_wiki(self, node):\n        edges = list(self._G.in_edges(node))\n        for source, target in edges:\n            if self._G[source][target].get('label', None) == 'name':\n                for attr, value in source.attributes:\n                    if attr == 'wiki':\n                        if value != '-':\n                            value = value[1:-1]  # remove quotes\n                        return value\n        return None\n\n    def set_name_node_wiki(self, node, wiki):\n        edges = list(self._G.in_edges(node))\n        parent = None\n        for source, target in edges:\n            if self._G[source][target].get('label', None) == 'name':\n                parent = source\n                break\n        if parent:\n            if wiki != '-':\n                wiki = '\"{}\"'.format(wiki)\n            self.add_node_attribute(parent, 'wiki', wiki)\n\n    def is_date_node(self, node):\n        return node.instance == 'date-entity'\n\n    def add_edge(self, source, target, label):\n        self._G.add_edge(source, target, label=label)\n        t = penman.Triple(source=source.identifier, relation=label, target=target.identifier)\n        triples = self._triples + [t]\n        triples = penman.alphanum_order(triples)\n        self._update_penman_graph(triples)\n\n    def remove_edge(self, x, y):\n        if isinstance(x, AMRNode) and isinstance(y, AMRNode):\n            self._G.remove_edge(x, y)\n        if isinstance(x, AMRNode):\n            x = x.identifier\n        if isinstance(y, AMRNode):\n            y = y.identifier\n        triples = [t for t in self._triples if not (t.source == x and t.target == y)]\n        self._update_penman_graph(triples)\n\n    def update_edge_label(self, x, y, old, new):\n        self._G[x][y]['label'] = new\n        triples = []\n        for t in self._triples:\n            if t.source == x.identifier and t.target == y.identifier and t.relation == old:\n                t = Triple(x.identifier, new, y.identifier)\n            triples.append(t)\n        self._update_penman_graph(triples)\n\n    def add_node(self, instance):\n        identifier = instance[0]\n        assert identifier.isalpha()\n        if identifier in self.variables():\n            i = 2\n            while identifier + str(i) in self.variables():\n                i += 1\n            identifier += str(i)\n        triples = self._triples + [Triple(identifier, 'instance', instance)]\n        self._triples = penman.alphanum_order(triples)\n\n        node = AMRNode(identifier, [('instance', instance)])\n        self._G.add_node(node)\n        return node\n\n    def remove_node(self, node):\n        self._G.remove_node(node)\n        triples = [t for t in self._triples if t.source != node.identifier]\n        self._update_penman_graph(triples)\n\n    def replace_node_attribute(self, node, attr, old, new):\n        node.replace_attribute(attr, old, new)\n        triples = []\n        found = False\n        for t in self._triples:\n            if t.source == node.identifier and t.relation == attr and t.target == old:\n                found = True\n                t = penman.Triple(source=node.identifier, relation=attr, target=new)\n            triples.append(t)\n        if not found:\n            raise KeyError\n        self._triples = penman.alphanum_order(triples)\n\n    def remove_node_attribute(self, node, attr, value):\n        node.remove_attribute(attr, value)\n        triples = [t for t in self._triples if\n                   not (t.source == node.identifier and t.relation == attr and t.target == value)]\n        self._update_penman_graph(triples)\n\n    def add_node_attribute(self, node, attr, value):\n        node.add_attribute(attr, value)\n        t = penman.Triple(source=node.identifier, relation=attr, target=value)\n        self._triples = penman.alphanum_order(self._triples + [t])\n\n    def remove_node_ops(self, node):\n        ops = []\n        for attr, value in node.attributes:\n            if re.search(r'^op\\d+$', attr):\n                ops.append((attr, value))\n        for attr, value in ops:\n            self.remove_node_attribute(node, attr, value)\n\n    def remove_subtree(self, root):\n        children = []\n        removed_nodes = set()\n        for _, child in list(self._G.edges(root)):\n            self.remove_edge(root, child)\n            children.append(child)\n        for child in children:\n            if len(list(self._G.in_edges(child))) == 0:\n                removed_nodes.update(self.remove_subtree(child))\n        if len(list(self._G.in_edges(root))) == 0:\n            self.remove_node(root)\n            removed_nodes.add(root)\n        return removed_nodes\n\n    def get_subtree(self, root, max_depth):\n        if max_depth == 0:\n            return []\n        nodes = [root]\n        children = [child for _, child in self._G.edges(root)]\n        nodes += children\n        for child in children:\n            if len(list(self._G.in_edges(child))) == 1:\n                nodes = nodes + self.get_subtree(child, max_depth - 1)\n        return nodes\n\n    def get_nodes(self):\n        return self._G.nodes\n\n    def get_edges(self):\n        return self._G.edges\n\n    def set_src_tokens(self, sentence):\n        if type(sentence) is not list:\n            sentence = sentence.split(\" \")\n        self._src_tokens = sentence\n\n    def get_src_tokens(self):\n        return self._src_tokens\n\n    def get_list_node(self, replace_copy=True):\n        visited = defaultdict(int)\n        node_list = []\n\n        def dfs(node, relation, parent):\n\n            node_list.append((\n                node if node.copy_of is None or not replace_copy else node.copy_of,\n                relation,\n                parent if parent.copy_of is None or not replace_copy else parent.copy_of))\n\n            if len(self._G[node]) > 0 and visited[node] == 0:\n                visited[node] = 1\n                for child_node, child_relation in self.sort_edges(self._G[node].items()):\n                    dfs(child_node, child_relation[\"label\"], node)\n\n        dfs(\n            self.variable_to_node[self._top],\n            'root',\n            self.variable_to_node[self._top]\n        )\n\n        return node_list\n\n    def sort_edges(self, edges):\n        return edges\n\n    def get_tgt_tokens(self):\n        node_list = self.get_list_node()\n\n        tgt_token = []\n        visited = defaultdict(int)\n\n        for node, relation, parent_node in node_list:\n            instance = [attr[1] for attr in node.attributes if attr[0] == \"instance\"]\n            assert len(instance) == 1\n            tgt_token.append(str(instance[0]))\n\n            if len(node.attributes) > 1 and visited[node] == 0:\n                for attr in node.attributes:\n                    if attr[0] != \"instance\":\n                        tgt_token.append(str(attr[1]))\n\n            visited[node] = 1\n\n        return tgt_token\n\n    def get_list_data(self, amr, bos=None, eos=None, bert_tokenizer=None, max_tgt_length=None):\n        node_list = self.get_list_node()\n\n        tgt_tokens = []\n        head_tags = []\n        head_indices = []\n\n        node_to_idx = defaultdict(list)\n        visited = defaultdict(int)\n\n        def update_info(node, relation, parent, token):\n            head_indices.append(1 + node_to_idx[parent][-1])\n            head_tags.append(relation)\n            tgt_tokens.append(str(token))\n\n        for node, relation, parent_node in node_list:\n\n            node_to_idx[node].append(len(tgt_tokens))\n\n            instance = [attr[1] for attr in node.attributes if attr[0] == \"instance\"]\n            assert len(instance) == 1\n            instance = instance[0]\n\n            update_info(node, relation, parent_node, instance)\n\n            if len(node.attributes) > 1 and visited[node] == 0:\n                for attr in node.attributes:\n                    if attr[0] != \"instance\":\n                        update_info(node, attr[0], node, attr[1])\n\n            visited[node] = 1\n\n        def trim_very_long_tgt_tokens(tgt_tokens, head_tags, head_indices, node_to_idx):\n            tgt_tokens = tgt_tokens[:max_tgt_length]\n            head_tags = head_tags[:max_tgt_length]\n            head_indices = head_indices[:max_tgt_length]\n            for node, indices in node_to_idx.items():\n                invalid_indices = [index for index in indices if index >= max_tgt_length]\n                for index in invalid_indices:\n                    indices.remove(index)\n            return tgt_tokens, head_tags, head_indices, node_to_idx\n\n        if max_tgt_length is not None:\n            tgt_tokens, head_tags, head_indices, node_to_idx = trim_very_long_tgt_tokens(\n                tgt_tokens, head_tags, head_indices, node_to_idx)\n\n        copy_offset = 0\n        if bos:\n            tgt_tokens = [bos] + tgt_tokens\n            copy_offset += 1\n        if eos:\n            tgt_tokens = tgt_tokens + [eos]\n\n        head_indices[node_to_idx[self.variable_to_node[self.top]][0]] = 0\n\n        # Target side Coreference\n        tgt_copy_indices = [i for i in range(len(tgt_tokens))]\n\n        for node, indices in node_to_idx.items():\n            if len(indices) > 1:\n                copy_idx = indices[0] + copy_offset\n                for token_idx in indices[1:]:\n                    tgt_copy_indices[token_idx + copy_offset] = copy_idx\n\n        tgt_copy_map = [(token_idx, copy_idx) for token_idx, copy_idx in enumerate(tgt_copy_indices)]\n\n        for i, copy_index in enumerate(tgt_copy_indices):\n            # Set the coreferred target to 0 if no coref is available.\n            if i == copy_index:\n                tgt_copy_indices[i] = 0\n\n        tgt_token_counter = Counter(tgt_tokens)\n        tgt_copy_mask = [0] * len(tgt_tokens)\n        for i, token in enumerate(tgt_tokens):\n            if tgt_token_counter[token] > 1:\n                tgt_copy_mask[i] = 1\n\n        def add_source_side_tags_to_target_side(_src_tokens, _src_tags):\n            assert len(_src_tags) == len(_src_tokens)\n            tag_counter = defaultdict(lambda: defaultdict(int))\n            for src_token, src_tag in zip(_src_tokens, _src_tags):\n                tag_counter[src_token][src_tag] += 1\n\n            tag_lut = {DEFAULT_OOV_TOKEN: DEFAULT_OOV_TOKEN,\n                       DEFAULT_PADDING_TOKEN: DEFAULT_OOV_TOKEN}\n            for src_token in set(_src_tokens):\n                tag = max(tag_counter[src_token].keys(), key=lambda x: tag_counter[src_token][x])\n                tag_lut[src_token] = tag\n\n            tgt_tags = []\n            for tgt_token in tgt_tokens:\n                sim_token = find_similar_token(tgt_token, _src_tokens)\n                if sim_token is not None:\n                    index = _src_tokens.index(sim_token)\n                    tag = _src_tags[index]\n                else:\n                    tag = DEFAULT_OOV_TOKEN\n                tgt_tags.append(tag)\n\n            return tgt_tags, tag_lut\n\n        # Source Copy\n        src_tokens = self.get_src_tokens()\n        src_token_ids = None\n        src_token_subword_index = None\n        src_pos_tags = amr.pos_tags\n        src_copy_vocab = SourceCopyVocabulary(src_tokens)\n        src_copy_indices = src_copy_vocab.index_sequence(tgt_tokens)\n        src_copy_map = src_copy_vocab.get_copy_map(src_tokens)\n        tgt_pos_tags, pos_tag_lut = add_source_side_tags_to_target_side(src_tokens, src_pos_tags)\n\n        if bert_tokenizer is not None:\n            src_token_ids, src_token_subword_index = bert_tokenizer.tokenize(src_tokens, True)\n\n        src_must_copy_tags = [1 if is_abstract_token(t) else 0 for t in src_tokens]\n        src_copy_invalid_ids = set(src_copy_vocab.index_sequence(\n            [t for t in src_tokens if is_english_punct(t)]))\n\n        return {\n            \"tgt_tokens\": tgt_tokens,\n            \"tgt_pos_tags\": tgt_pos_tags,\n            \"tgt_copy_indices\": tgt_copy_indices,\n            \"tgt_copy_map\": tgt_copy_map,\n            \"tgt_copy_mask\": tgt_copy_mask,\n            \"src_tokens\": src_tokens,\n            \"src_token_ids\": src_token_ids,\n            \"src_token_subword_index\": src_token_subword_index,\n            \"src_must_copy_tags\": src_must_copy_tags,\n            \"src_pos_tags\": src_pos_tags,\n            \"src_copy_vocab\": src_copy_vocab,\n            \"src_copy_indices\": src_copy_indices,\n            \"src_copy_map\": src_copy_map,\n            \"pos_tag_lut\": pos_tag_lut,\n            \"head_tags\": head_tags,\n            \"head_indices\": head_indices,\n            \"src_copy_invalid_ids\": src_copy_invalid_ids\n        }\n\n    @classmethod\n    def decode(cls, raw_graph_string):\n        _graph = amr_codec.decode(raw_graph_string)\n        return cls(_graph)\n\n    @classmethod\n    def from_lists(cls, all_list):\n        head_tags = all_list['head_tags']\n        head_indices = all_list['head_indices']\n        tgt_tokens = all_list['tokens']\n\n        tgt_copy_indices = all_list['coref']\n        variables = []\n        variables_count = defaultdict(int)\n        for i, token in enumerate(tgt_tokens):\n            if tgt_copy_indices[i] != i:\n                variables.append(variables[tgt_copy_indices[i]])\n            else:\n                if token[0] in variables_count:\n                    variables.append(token[0] + str(variables_count[token[0]]))\n                else:\n                    variables.append(token[0])\n\n                variables_count[token[0]] += 1\n\n        Triples = []\n        for variable, token in zip(variables, tgt_tokens):\n            Triples.append(Triple(variable, \"instance\", token))\n            Triples.append(\n                Triple(\n                    head_indices[variable],\n                    head_tags[variable],\n                    variable\n                )\n            )\n\n    @classmethod\n    def from_prediction(cls, prediction):\n\n        def is_attribute_value(value):\n            return re.search(r'(^\".*\"$|^[^a-zA-Z]+$)', value) is not None\n\n        def is_attribute_edge(label):\n            return label in ('instance', 'mode', 'li', 'value', 'month', 'year', 'day', 'decade', 'ARG6')\n\n        def normalize_number(text):\n            if re.search(r'^\\d+,\\d+$', text):\n                text = text.replace(',', '')\n            return text\n\n        def abstract_node(value):\n            return re.search(r'^([A-Z]+|DATE_ATTRS|SCORE_ENTITY|ORDINAL_ENTITY)_\\d+$', value)\n\n        def abstract_attribute(value):\n            return re.search(r'^_QUANTITY_\\d+$', value)\n\n        def correct_multiroot(heads):\n            for i in range(1, len(heads)):\n                if heads[i] == 0:\n                    heads[i] = 1\n            return heads\n\n        nodes = [normalize_number(n) for n in prediction['nodes']]\n        heads = correct_multiroot(prediction['heads'])\n        corefs = [int(x) for x in prediction['corefs']]\n        head_labels = prediction['head_labels']\n\n        triples = []\n        top = None\n        # Build the variable map from variable to instance.\n        variable_map = {}\n        for coref_index in corefs:\n            node = nodes[coref_index - 1]\n            head_label = head_labels[coref_index - 1]\n            if (re.search(r'[/:\\\\()]', node) or is_attribute_value(node) or\n                    is_attribute_edge(head_label) or abstract_attribute(node)):\n                continue\n            variable_map['vv{}'.format(coref_index)] = node\n        for head_index in heads:\n            if head_index == 0:\n                continue\n            node = nodes[head_index - 1]\n            coref_index = corefs[head_index - 1]\n            variable_map['vv{}'.format(coref_index)] = node\n        # Build edge triples and other attribute triples.\n        for i, head_index in enumerate(heads):\n            if head_index == 0:\n                top_variable = 'vv{}'.format(corefs[i])\n                if top_variable not in variable_map:\n                    variable_map[top_variable] = nodes[i]\n                top = top_variable\n                continue\n            head_variable = 'vv{}'.format(corefs[head_index - 1])\n            modifier = nodes[i]\n            modifier_variable = 'vv{}'.format(corefs[i])\n            label = head_labels[i]\n            assert head_variable in variable_map\n            if modifier_variable in variable_map:\n                triples.append((head_variable, label, modifier_variable))\n            else:\n                # Add quotes if there's a backslash.\n                if re.search(r'[/:\\\\()]', modifier) and not re.search(r'^\".*\"$', modifier):\n                    modifier = '\"{}\"'.format(modifier)\n                triples.append((head_variable, label, modifier))\n\n        for var, node in variable_map.items():\n            if re.search(r'^\".*\"$', node):\n                node = node[1:-1]\n            if re.search(r'[/:\\\\()]', node):\n                parts = re.split(r'[/:\\\\()]', node)\n                for part in parts[::-1]:\n                    if len(part):\n                        node = part\n                        break\n                else:\n                    node = re.sub(r'[/:\\\\()]', '_', node)\n            triples.append((var, 'instance', node))\n\n        if len(triples) == 0:\n            triples.append(('vv1', 'instance', 'string-entity'))\n            top = 'vv1'\n        triples.sort(key=lambda x: int(x[0].replace('vv', '')))\n        graph = penman.Graph()\n        graph._top = top\n        graph._triples = [penman.Triple(*t) for t in triples]\n        graph = cls(graph)\n        try:\n            GraphRepair.do(graph, nodes)\n            amr_codec.encode(graph)\n        except Exception as e:\n            graph._top = top\n            graph._triples = [penman.Triple(*t) for t in triples]\n            graph = cls(graph)\n        return graph\n\n\nclass SourceCopyVocabulary:\n    def __init__(self, sentence, pad_token=DEFAULT_PADDING_TOKEN, unk_token=DEFAULT_OOV_TOKEN):\n        if type(sentence) is not list:\n            sentence = sentence.split(\" \")\n\n        self.src_tokens = sentence\n        self.pad_token = pad_token\n        self.unk_token = unk_token\n\n        self.token_to_idx = {self.pad_token: 0, self.unk_token: 1}\n        self.idx_to_token = {0: self.pad_token, 1: self.unk_token}\n\n        self.vocab_size = 2\n\n        for token in sentence:\n            if token not in self.token_to_idx:\n                self.token_to_idx[token] = self.vocab_size\n                self.idx_to_token[self.vocab_size] = token\n                self.vocab_size += 1\n\n    def get_token_from_idx(self, idx):\n        return self.idx_to_token[idx]\n\n    def get_token_idx(self, token):\n        return self.token_to_idx.get(token, self.token_to_idx[self.unk_token])\n\n    def index_sequence(self, list_tokens):\n        return [self.get_token_idx(token) for token in list_tokens]\n\n    def get_copy_map(self, list_tokens):\n        src_indices = [self.get_token_idx(self.unk_token)] + self.index_sequence(list_tokens)\n        return [\n            (src_idx, src_token_idx) for src_idx, src_token_idx in enumerate(src_indices)\n        ]\n\n    def get_special_tok_list(self):\n        return [self.pad_token, self.unk_token]\n\n    def __repr__(self):\n        return json.dumps(self.idx_to_token)\n\n\ndef is_similar(instances1, instances2):\n    if len(instances1) < len(instances2):\n        small = instances1\n        large = instances2\n    else:\n        small = instances2\n        large = instances1\n    coverage1 = sum(1 for x in small if x in large) / len(small)\n    coverage2 = sum(1 for x in large if x in small) / len(large)\n    return coverage1 > .8 and coverage2 > .8\n\n\nclass GraphRepair:\n\n    def __init__(self, graph, nodes):\n        self.graph = graph\n        self.nodes = nodes\n        self.repaired_items = set()\n\n    @staticmethod\n    def do(graph, nodes):\n        gr = GraphRepair(graph, nodes)\n        gr.remove_redundant_edges()\n        gr.remove_unknown_nodes()\n\n    def remove_unknown_nodes(self):\n        graph = self.graph\n        nodes = [node for node in graph.get_nodes()]\n        for node in nodes:\n            for attr, value in node.attributes:\n                if value == '@@UNKNOWN@@' and attr != 'instance':\n                    graph.remove_node_attribute(node, attr, value)\n            if node.instance == '@@UNKNOWN@@':\n                if len(list(graph._G.edges(node))) == 0:\n                    for source, target in list(graph._G.in_edges(node)):\n                        graph.remove_edge(source, target)\n                    graph.remove_node(node)\n                    self.repaired_items.add('remove-unknown-node')\n\n    def remove_redundant_edges(self):\n        \"\"\"\n        Edge labels such as ARGx, ARGx-of, and 'opx' should only appear at most once\n        in each node's outgoing edges.\n        \"\"\"\n        graph = self.graph\n        nodes = [node for node in graph.get_nodes()]\n        removed_nodes = set()\n        for node in nodes:\n            if node in removed_nodes:\n                continue\n            edges = list(graph._G.edges(node))\n            edge_counter = defaultdict(list)\n            for source, target in edges:\n                label = graph._G[source][target]['label']\n                # `name`, `ARGx`, and `ARGx-of` should only appear once.\n                if label == 'name':  # or label.startswith('ARG'):\n                    edge_counter[label].append(target)\n                # the target of `opx' should only appear once.\n                elif label.startswith('op') or label.startswith('snt'):\n                    edge_counter[str(target.instance)].append(target)\n                else:\n                    edge_counter[label + str(target.instance)].append(target)\n            for label, children in edge_counter.items():\n                if len(children) == 1:\n                    continue\n                if label == 'name':\n                    # remove redundant edges.\n                    for target in children[1:]:\n                        if len(list(graph._G.in_edges(target))) == 1 and len(list(graph._G.edges(target))) == 0:\n                            graph.remove_edge(node, target)\n                            graph.remove_node(target)\n                            removed_nodes.add(target)\n                            self.repaired_items.add('remove-redundant-edge')\n                    continue\n                visited_children = set()\n                groups = []\n                for i, target in enumerate(children):\n                    if target in visited_children:\n                        continue\n                    subtree_instances1 = [n.instance for n in graph.get_subtree(target, 5)]\n                    group = [(target, subtree_instances1)]\n                    visited_children.add(target)\n                    for _t in children[i + 1:]:\n                        if _t in visited_children or target.instance != _t.instance:\n                            continue\n                        subtree_instances2 = [n.instance for n in graph.get_subtree(_t, 5)]\n                        if is_similar(subtree_instances1, subtree_instances2):\n                            group.append((_t, subtree_instances2))\n                            visited_children.add(_t)\n                    groups.append(group)\n                for group in groups:\n                    if len(group) == 1:\n                        continue\n                    kept_target, _ = max(group, key=lambda x: len(x[1]))\n                    for target, _ in group:\n                        if target == kept_target:\n                            continue\n                        graph.remove_edge(node, target)\n                        removed_nodes.update(graph.remove_subtree(target))\n"
  },
  {
    "path": "plugins/hanlp_common/hanlp_common/configurable.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-16 22:24\nfrom hanlp_common.reflection import str_to_type, classpath_of\n\n\nclass Configurable(object):\n    @staticmethod\n    def from_config(config: dict, **kwargs):\n        \"\"\"Build an object from config.\n\n        Args:\n          config: A ``dict`` holding parameters for its constructor. It has to contain a `classpath` key,\n                    which has a classpath str as its value. ``classpath`` will determine the type of object\n                    being deserialized.\n          kwargs: Arguments not used.\n\n        Returns: A deserialized object.\n\n        \"\"\"\n        cls = config.get('classpath', None)\n        assert cls, f'{config} doesn\\'t contain classpath field'\n        cls = str_to_type(cls)\n        deserialized_config = dict(config)\n        for k, v in config.items():\n            if isinstance(v, dict) and 'classpath' in v:\n                deserialized_config[k] = Configurable.from_config(v)\n        if cls.from_config == Configurable.from_config:\n            deserialized_config.pop('classpath')\n            return cls(**deserialized_config)\n        else:\n            return cls.from_config(deserialized_config)\n\n\nclass AutoConfigurable(Configurable):\n    @property\n    def config(self) -> dict:\n        \"\"\"\n        The config of this object, which are public properties. If any properties needs to be excluded from this config,\n        simply declare it with prefix ``_``.\n        \"\"\"\n        return dict([('classpath', classpath_of(self))] +\n                    [(k, v.config if hasattr(v, 'config') else v)\n                     for k, v in self.__dict__.items() if\n                     not k.startswith('_')])\n\n    def __repr__(self) -> str:\n        return repr(self.config)\n"
  },
  {
    "path": "plugins/hanlp_common/hanlp_common/conll.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-19 20:50\nfrom typing import Union, List\n\nfrom hanlp_common.structure import SerializableDict\nfrom hanlp_common.visualization import pretty_tree_horizontal, make_table, markdown_table\n\n\nclass CoNLLWord(SerializableDict):\n    def __init__(self, id, form, lemma=None, cpos=None, pos=None, feats=None, head=None, deprel=None, phead=None,\n                 pdeprel=None):\n        \"\"\"CoNLL (:cite:`buchholz-marsi-2006-conll`) format template, see http://anthology.aclweb.org/W/W06/W06-2920.pdf\n\n        Args:\n            id (int):\n                Token counter, starting at 1 for each new sentence.\n            form (str):\n                Word form or punctuation symbol.\n            lemma (str):\n                Lemma or stem (depending on the particular treebank) of word form, or an underscore if not available.\n            cpos (str):\n                Coarse-grained part-of-speech tag, where the tagset depends on the treebank.\n            pos (str):\n                Fine-grained part-of-speech tag, where the tagset depends on the treebank.\n            feats (str):\n                Unordered set of syntactic and/or morphological features (depending on the particular treebank),\n                or an underscore if not available.\n            head (Union[int, List[int]]):\n                Head of the current token, which is either a value of ID,\n                or zero (’0’) if the token links to the virtual root node of the sentence.\n            deprel (Union[str, List[str]]):\n                Dependency relation to the HEAD.\n            phead (int):\n                Projective head of current token, which is either a value of ID or zero (’0’),\n                or an underscore if not available.\n            pdeprel (str):\n                Dependency relation to the PHEAD, or an underscore if not available.\n        \"\"\"\n        self.id = sanitize_conll_int_value(id)\n        self.form = form\n        self.cpos = cpos\n        self.pos = pos\n        self.head = sanitize_conll_int_value(head)\n        self.deprel = deprel\n        self.lemma = lemma\n        self.feats = feats\n        self.phead = phead\n        self.pdeprel = pdeprel\n\n    def __str__(self):\n        if isinstance(self.head, list):\n            return '\\n'.join('\\t'.join(['_' if v is None else v for v in values]) for values in [\n                [str(self.id), self.form, self.lemma, self.cpos, self.pos, self.feats,\n                 None if head is None else str(head), deprel, self.phead, self.pdeprel] for head, deprel in\n                zip(self.head, self.deprel)\n            ])\n        values = [str(self.id), self.form, self.lemma, self.cpos, self.pos, self.feats,\n                  None if self.head is None else str(self.head), self.deprel, self.phead, self.pdeprel]\n        return '\\t'.join(['_' if v is None else v for v in values])\n\n    @property\n    def nonempty_fields(self):\n        \"\"\"\n        Get the values of nonempty fields as a list.\n        \"\"\"\n        return list(f for f in\n                    [self.form, self.lemma, self.cpos, self.pos, self.feats, self.head, self.deprel, self.phead,\n                     self.pdeprel] if f)\n\n    def get_pos(self):\n        \"\"\"\n        Get the precisest pos for this word.\n\n        Returns: ``self.pos`` or ``self.cpos``.\n\n        \"\"\"\n        return self.pos or self.cpos\n\n\nclass CoNLLUWord(SerializableDict):\n    def __init__(self, id: Union[int, str], form, lemma=None, upos=None, xpos=None, feats=None, head=None, deprel=None,\n                 deps=None,\n                 misc=None):\n        \"\"\"CoNLL-U format template, see https://universaldependencies.org/format.html\n\n        Args:\n\n            id (Union[int, str]):\n                Token counter, starting at 1 for each new sentence.\n            form (Union[str, None]):\n                Word form or punctuation symbol.\n            lemma (str):\n                Lemma or stem (depending on the particular treebank) of word form, or an underscore if not available.\n            upos (str):\n                Universal part-of-speech tag.\n            xpos (str):\n                Language-specific part-of-speech tag; underscore if not available.\n            feats (str):\n                List of morphological features from the universal feature inventory or from a defined language-specific extension; underscore if not available.\n            head (int):\n                Head of the current token, which is either a value of ID,\n                or zero (’0’) if the token links to the virtual root node of the sentence.\n            deprel (str):\n                Dependency relation to the HEAD.\n            deps (Union[List[Tuple[int, str], str]):\n                Projective head of current token, which is either a value of ID or zero (’0’),\n                or an underscore if not available.\n            misc (str):\n                Dependency relation to the PHEAD, or an underscore if not available.\n        \"\"\"\n        self.id = sanitize_conll_int_value(id)\n        self.form = form\n        self.upos = upos\n        self.xpos = xpos\n        if isinstance(head, list):\n            assert deps is None, 'When head is a list, deps has to be None'\n            assert isinstance(deprel, list), 'When head is a list, deprel has to be a list'\n            assert len(deprel) == len(head), 'When head is a list, deprel has to match its length'\n            deps = list(zip(head, deprel))\n            head = None\n            deprel = None\n        self.head = sanitize_conll_int_value(head)\n        self.deprel = deprel\n        self.lemma = lemma\n        self.feats = feats\n        if deps == '_':\n            deps = None\n        if isinstance(deps, str):\n            self.deps = []\n            for pair in deps.split('|'):\n                h, r = pair.split(':')\n                h = int(h)\n                self.deps.append((h, r))\n        else:\n            self.deps = deps\n        self.misc = misc\n\n    def __str__(self):\n        deps = self.deps\n        if not deps:\n            deps = None\n        else:\n            deps = '|'.join(f'{h}:{r}' for h, r in deps)\n        values = [str(self.id), self.form, self.lemma, self.upos, self.xpos, self.feats,\n                  str(self.head) if self.head is not None else None, self.deprel, deps, self.misc]\n        return '\\t'.join(['_' if v is None else v for v in values])\n\n    @property\n    def nonempty_fields(self):\n        \"\"\"\n        Get the values of nonempty fields as a list.\n        \"\"\"\n        return list(f for f in\n                    [self.form, self.lemma, self.upos, self.xpos, self.feats, self.head, self.deprel, self.deps,\n                     self.misc] if f)\n\n    def get_pos(self):\n        \"\"\"\n        Get the precisest pos for this word.\n\n        Returns: ``self.xpos`` or ``self.upos``\n\n        \"\"\"\n        return self.xpos or self.upos\n\n\nclass CoNLLSentence(list):\n    def __init__(self, words=None):\n        \"\"\"\n        A list of :class:`~hanlp_common.conll.CoNLLWord` or :class:`~hanlp_common.conll.CoNLLUWord`. It is a sub-class\n        of :class:`list` and its words can be accessed in the same way as accessing list elements.\n\n        Args:\n            words (list[Union[CoNLLWord, CoNLLUWord]]): A list of words.\n        \"\"\"\n        super().__init__()\n        if words:\n            self.extend(words)\n\n    def __str__(self):\n        return '\\n'.join([word.__str__() for word in self])\n\n    @staticmethod\n    def from_str(conll: str, conllu=False):\n        \"\"\"Build a CoNLLSentence from CoNLL-X format str\n\n        Args:\n          conll (str): CoNLL-X or CoNLL-U format string\n          conllu:  ``True`` to build :class:`~hanlp_common.conll.CoNLLUWord` for each token.\n\n        Returns:\n            A :class:`~hanlp_common.conll.CoNLLSentence`.\n        \"\"\"\n        words: List[CoNLLWord] = []\n        prev_id = None\n        for line in conll.strip().split('\\n'):\n            if line.startswith('#'):\n                continue\n            cells = line.split('\\t')\n            cells = [None if c == '_' else c for c in cells]\n            if '-' in cells[0]:\n                continue\n            cells[0] = int(cells[0])\n            cells[6] = int(cells[6])\n            if cells[0] != prev_id:\n                words.append(CoNLLUWord(*cells) if conllu else CoNLLWord(*cells))\n            else:\n                if isinstance(words[-1].head, list):\n                    words[-1].head.append(cells[6])\n                    words[-1].deprel.append(cells[7])\n                else:\n                    words[-1].head = [words[-1].head] + [cells[6]]\n                    words[-1].deprel = [words[-1].deprel] + [cells[7]]\n            prev_id = cells[0]\n        if conllu:\n            for word in words:  # type: CoNLLUWord\n                if isinstance(word.head, list):\n                    assert not word.deps\n                    word.deps = list(zip(word.head, word.deprel))\n                    word.head = None\n                    word.deprel = None\n        return CoNLLSentence(words)\n\n    @staticmethod\n    def from_file(path: str, conllu=False):\n        \"\"\"Build a CoNLLSentence from ``.conllx`` or ``.conllu`` file\n\n        Args:\n          path: Path to the file.\n          conllu:  ``True`` to build :class:`~hanlp_common.conll.CoNLLUWord` for each token.\n\n        Returns:\n            A :class:`~hanlp_common.conll.CoNLLSentence`.\n        \"\"\"\n        with open(path) as src:\n            return [CoNLLSentence.from_str(x, conllu) for x in src.read().split('\\n\\n') if x.strip()]\n\n    @staticmethod\n    def from_dict(d: dict, conllu=False):\n        \"\"\"Build a CoNLLSentence from a dict.\n\n        Args:\n            d: A dict storing a list for each field, where each index corresponds to a token.\n            conllu: ``True`` to build :class:`~hanlp_common.conll.CoNLLUWord` for each token.\n\n        Returns:\n            A :class:`~hanlp_common.conll.CoNLLSentence`.\n        \"\"\"\n        if conllu:\n            headings = ['ID', 'FORM', 'LEMMA', 'UPOS', 'XPOS', 'FEATS', 'HEAD', 'DEPREL', 'DEPS', 'MISC']\n        else:\n            headings = ['ID', 'FORM', 'LEMMA', 'CPOS', 'POS', 'FEATS', 'HEAD', 'DEPREL', 'PHEAD', 'PDEPREL']\n        words: List[Union[CoNLLWord, CoNLLUWord]] = []\n        for cells in zip(*list(d[f] for f in headings)):\n            words.append(CoNLLUWord(*cells) if conllu else CoNLLWord(*cells))\n        return CoNLLSentence(words)\n\n    def to_markdown(self, headings: Union[str, List[str]] = 'auto') -> str:\n        r\"\"\"Convert into markdown string.\n\n        Args:\n            headings: ``auto`` to automatically detect the word type. When passed a list of string, they are treated as\n                        headings for each field.\n\n        Returns:\n            A markdown representation of this sentence.\n        \"\"\"\n        cells = [str(word).split('\\t') for word in self]\n        if headings == 'auto':\n            if isinstance(self[0], CoNLLWord):\n                headings = ['ID', 'FORM', 'LEMMA', 'CPOS', 'POS', 'FEATS', 'HEAD', 'DEPREL', 'PHEAD', 'PDEPREL']\n            else:  # conllu\n                headings = ['ID', 'FORM', 'LEMMA', 'UPOS', 'XPOS', 'FEATS', 'HEAD', 'DEPREL', 'DEPS', 'MISC']\n                for each in cells:\n                    # if '|' in each[8]:\n                    # each[8] = f'`{each[8]}`'\n                    each[8] = each[8].replace('|', '⎮')\n        alignment = [('^', '>'), ('^', '<'), ('^', '<'), ('^', '<'), ('^', '<'), ('^', '<'), ('^', '>'), ('^', '<'),\n                     ('^', '<'), ('^', '<')]\n        text = markdown_table(headings, cells, alignment=alignment)\n        return text\n\n    def to_tree(self, extras: List[str] = None) -> str:\n        \"\"\"Convert into a pretty tree string which can be printed to show the tree structure.\n\n        Args:\n            extras: Extra table to be aligned to this tree.\n\n        Returns:\n            A pretty tree string along with extra table if passed any.\n        \"\"\"\n        arrows = []\n        for word in self:  # type: Union[CoNLLWord, CoNLLUWord]\n            if word.head:\n                arrows.append({'from': word.head - 1, 'to': word.id - 1})\n        tree = pretty_tree_horizontal(arrows)\n        rows = [['Dep Tree', 'Token', 'Relation']]\n        has_lem = all(x.lemma for x in self)\n        has_pos = all(x.get_pos() for x in self)\n        if has_lem:\n            rows[0].append('Lemma')\n        if has_pos:\n            rows[0].append('PoS')\n        if extras:\n            rows[0].extend(extras[0])\n        for i, (word, arc) in enumerate(zip(self, tree)):\n            cell_per_word = [arc]\n            cell_per_word.append(word.form)\n            cell_per_word.append(word.deprel)\n            if has_lem:\n                cell_per_word.append(word.lemma)\n            if has_pos:\n                cell_per_word.append(word.get_pos())\n            if extras:\n                cell_per_word.extend(extras[i + 1])\n            rows.append(cell_per_word)\n        return make_table(rows, insert_header=True)\n\n    @property\n    def projective(self):\n        \"\"\"\n        ``True`` if this tree is projective.\n        \"\"\"\n        return isprojective([x.head for x in self])\n\n\nclass CoNLLSentenceList(list):\n\n    def __str__(self) -> str:\n        return '\\n\\n'.join(str(x) for x in self)\n\n\ndef sanitize_conll_int_value(value: Union[str, int]):\n    if value is None or isinstance(value, int):\n        return value\n    if value == '_':\n        return None\n    if isinstance(value, str):\n        return int(value)\n    return value\n\n\ndef isprojective(sequence):\n    r\"\"\"\n    Checks if a dependency tree is projective.\n    This also works for partial annotation.\n\n    Besides the obvious crossing arcs, the examples below illustrate two non-projective cases\n    which are hard to detect in the scenario of partial annotation.\n\n    Args:\n        sequence (list[int]):\n            A list of head indices.\n\n    Returns:\n        ``True`` if the tree is projective, ``False`` otherwise.\n\n    Examples:\n        >>> isprojective([2, -1, 1])  # -1 denotes un-annotated cases\n        False\n        >>> isprojective([3, -1, 2])\n        False\n    \"\"\"\n\n    pairs = [(h, d) for d, h in enumerate(sequence, 1) if h >= 0]\n    for i, (hi, di) in enumerate(pairs):\n        for hj, dj in pairs[i + 1:]:\n            (li, ri), (lj, rj) = sorted([hi, di]), sorted([hj, dj])\n            if li <= hj <= ri and hi == dj:\n                return False\n            if lj <= hi <= rj and hj == di:\n                return False\n            if (li < lj < ri or li < rj < ri) and (li - lj) * (ri - rj) > 0:\n                return False\n    return True\n"
  },
  {
    "path": "plugins/hanlp_common/hanlp_common/constant.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-06-13 22:41\nimport os\n\nPAD = '<pad>'\n'''Padding token.'''\nUNK = '<unk>'\n'''Unknown token.'''\nCLS = '[CLS]'\nBOS = '<bos>'\nEOS = '<eos>'\nROOT = BOS\nIDX = '_idx_'\n'''Key for index.'''\nHANLP_URL = os.getenv('HANLP_URL', 'https://file.hankcs.com/hanlp/')\n'''Resource URL.'''\nHANLP_VERBOSE = os.environ.get('HANLP_VERBOSE', '1').lower() in ('1', 'true', 'yes')\n'''Enable verbose or not.'''\nNULL = '<null>'\nPRED = 'PRED'\n\nIPYTHON = os.environ.get('HANLP_IPYTHON', '1').lower() in ('1', 'true', 'yes')  # Allow the user to disable IPYTHON\nif IPYTHON:\n    try:\n        # noinspection PyUnresolvedReferences,PyStatementEffect\n        get_ipython\n    except NameError:\n        IPYTHON = False\n"
  },
  {
    "path": "plugins/hanlp_common/hanlp_common/document.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-31 04:16\nimport json\nimport re\nimport warnings\nfrom typing import List, Union\n\nfrom phrasetree.tree import Tree\n\nfrom hanlp_common.conll import CoNLLUWord, CoNLLSentence, CoNLLSentenceList\nfrom hanlp_common.constant import PRED, IPYTHON\nfrom hanlp_common.util import collapse_json, prefix_match\nfrom hanlp_common.visualization import tree_to_list, list_to_tree, render_labeled_span, make_table\n\n\nclass Document(dict):\n    def __init__(self, *args, **kwargs) -> None:\n        r\"\"\"A dict structure holding parsed annotations. A document is a subclass of ``dict`` and it supports every\n        interface of ``dict``\\. Additionally, it supports interfaces to deal with various linguistic structures. Its\n        ``str`` and ``dict`` representations are made to be compatible with JSON serialization.\n\n        Args:\n            *args: An iterator of key-value pairs.\n            **kwargs: Arguments from ``**`` operator.\n\n        Examples::\n\n            # Create a document\n            doc = Document(\n                tok=[[\"晓美焰\", \"来到\", \"北京\", \"立方庭\", \"参观\", \"自然\", \"语义\", \"科技\", \"公司\"]],\n                pos=[[\"NR\", \"VV\", \"NR\", \"NR\", \"VV\", \"NN\", \"NN\", \"NN\", \"NN\"]],\n                ner=[[[\"晓美焰\", \"PERSON\", 0, 1], [\"北京立方庭\", \"LOCATION\", 2, 4],\n                      [\"自然语义科技公司\", \"ORGANIZATION\", 5, 9]]],\n                dep=[[[2, \"nsubj\"], [0, \"root\"], [4, \"name\"], [2, \"dobj\"], [2, \"conj\"],\n                      [9, \"compound\"], [9, \"compound\"], [9, \"compound\"], [5, \"dobj\"]]]\n            )\n\n            # print(doc) or str(doc) to get its JSON representation\n            print(doc)\n\n            # Access an annotation by its task name\n            print(doc['tok'])\n\n            # Get number of sentences\n            print(f'It has {doc.count_sentences()} sentence(s)')\n\n            # Access the n-th sentence\n            print(doc.squeeze(0)['tok'])\n\n            # Pretty print it right in your console or notebook\n            doc.pretty_print()\n\n            # To save the pretty prints in a str\n            pretty_text: str = '\\n\\n'.join(doc.to_pretty())\n\n        \"\"\"\n        super().__init__(*args, **kwargs)\n        for k, v in list(self.items()):\n            if not v:\n                continue\n            if k == 'con':\n                if isinstance(v, Tree) or isinstance(v[0], Tree):\n                    continue\n                flat = isinstance(v[0], str)\n                if flat:\n                    v = [v]\n                ls = []\n                for each in v:\n                    if not isinstance(each, Tree):\n                        ls.append(list_to_tree(each))\n                if flat:\n                    ls = ls[0]\n                self[k] = ls\n            elif k == 'amr':\n                from hanlp_common.amr import AMRGraph\n                import penman\n                if isinstance(v, AMRGraph) or isinstance(v[0], AMRGraph):\n                    continue\n                flat = isinstance(v[0][0], str)\n                if flat:\n                    v = [v]\n                graphs = [AMRGraph(penman.Graph(triples)) for triples in v]\n                if flat:\n                    graphs = graphs[0]\n                self[k] = graphs\n\n    def to_json(self, ensure_ascii=False, indent=2) -> str:\n        \"\"\"Convert to json string.\n\n        Args:\n            ensure_ascii: ``False`` to allow for non-ascii text.\n            indent: Indent per nested structure.\n\n        Returns:\n            A text representation in ``str``.\n\n        \"\"\"\n        d = self.to_dict()\n        text = json.dumps(d, ensure_ascii=ensure_ascii, indent=indent, default=lambda o: repr(o))\n        text = collapse_json(text, 4)\n        return text\n\n    def to_dict(self):\n        \"\"\"Convert to a json compatible dict.\n\n        Returns:\n            A dict representation.\n        \"\"\"\n        d = dict(self)\n        for k, v in self.items():\n            if v == [] or v is None:\n                continue\n            if k == 'con':\n                if not isinstance(v, Tree) and not isinstance(v[0], Tree):\n                    continue\n                flat = isinstance(v, Tree)\n                if flat:\n                    v = [v]\n                ls = []\n                for each in v:\n                    if isinstance(each, Tree):\n                        ls.append(tree_to_list(each))\n                if flat:\n                    ls = ls[0]\n                d[k] = ls\n        return d\n\n    def __str__(self) -> str:\n        return self.to_json()\n\n    def to_conll(self, tok='tok', lem='lem', pos='pos', fea='fea', dep='dep', sdp='sdp') -> Union[\n        CoNLLSentence, List[CoNLLSentence]]:\n        \"\"\"\n        Convert to :class:`~hanlp_common.conll.CoNLLSentence`.\n\n        Args:\n            tok (str): Field name for tok.\n            lem (str): Field name for lem.\n            pos (str): Field name for upos.\n            fea (str): Field name for feats.\n            dep (str): Field name for dependency parsing.\n            sdp (str): Field name for semantic dependency parsing.\n\n        Returns:\n            A :class:`~hanlp_common.conll.CoNLLSentence` representation.\n\n        \"\"\"\n        tok = prefix_match(tok, self)\n        lem = prefix_match(lem, self)\n        pos = prefix_match(pos, self)\n        fea = prefix_match(fea, self)\n        dep = prefix_match(dep, self)\n        sdp = prefix_match(sdp, self)\n        results = CoNLLSentenceList()\n        if not tok or not self[tok]:\n            return results\n        self = self._to_doc_without_spans(tok)\n        flat = isinstance(self[tok][0], str)\n        if flat:\n            d = Document((k, [v]) for k, v in self.items())\n        else:\n            d = self\n        for sample in [dict(zip(d, t)) for t in zip(*d.values())]:\n            def get(_k, _i):\n                _v = sample.get(_k, None)\n                if not _v:\n                    return None\n                return _v[_i]\n\n            sent = CoNLLSentence()\n\n            for i, _tok in enumerate(sample[tok]):\n                _dep = get(dep, i)\n                if not _dep:\n                    _dep = (None, None)\n                sent.append(\n                    CoNLLUWord(i + 1, form=_tok, lemma=get(lem, i), upos=get(pos, i), feats=get(fea, i), head=_dep[0],\n                               deprel=_dep[1],\n                               deps=None if not get(sdp, i) else '|'.join(f'{x[0]}:{x[1]}' for x in get(sdp, i))))\n            results.append(sent)\n        if flat:\n            return results[0]\n        return results\n\n    def to_pretty(self, tok='tok', lem='lem', pos='pos', dep='dep', sdp='sdp', ner='ner', srl='srl', con='con',\n                  show_header=True, html=False) -> Union[str, List[str]]:\n        \"\"\"\n        Convert to a pretty text representation which can be printed to visualize linguistic structures.\n\n        Args:\n            tok: Token key.\n            lem: Lemma key.\n            pos: Part-of-speech key.\n            dep: Dependency parse tree key.\n            sdp: Semantic dependency tree/graph key. SDP visualization has not been implemented yet.\n            ner: Named entity key.\n            srl: Semantic role labeling key.\n            con: Constituency parsing key.\n            show_header: ``True`` to include a header which indicates each field with its name.\n            html: ``True`` to output HTML format so that non-ASCII characters can align correctly.\n\n        Returns:\n            A pretty string.\n\n        \"\"\"\n        results = []\n        tok = prefix_match(tok, self)\n        pos = prefix_match(pos, self)\n        ner = prefix_match(ner, self)\n        conlls = self.to_conll(tok=tok, lem=lem, pos=pos, dep=dep, sdp=sdp)\n        flat = isinstance(conlls, CoNLLSentence)\n        if flat:\n            conlls: List[CoNLLSentence] = [conlls]\n\n        def condense(block_, extras_=None):\n            text_ = make_table(block_, insert_header=False)\n            text_ = [x.split('\\t', 1) for x in text_.split('\\n')]\n            text_ = [[x[0], x[1].replace('\\t', '')] for x in text_]\n            if extras_:\n                for r, s in zip(extras_, text_):\n                    r.extend(s)\n            return text_\n\n        for i, conll in enumerate(conlls):\n            conll: CoNLLSentence = conll\n            tokens = [x.form for x in conll]\n            length = len(conll)\n            extras = [[] for j in range(length + 1)]\n            if ner in self:\n                ner_samples = self[ner]\n                if flat:\n                    ner_samples = [ner_samples]\n                ner_per_sample = ner_samples[i]\n                # For nested NER, use the longest span\n                start_offsets = [None for i in range(length)]\n                for ent, label, b, e in ner_per_sample:\n                    if not start_offsets[b] or e > start_offsets[b][-1]:\n                        start_offsets[b] = (ent, label, b, e)\n                ner_per_sample = [y for y in start_offsets if y]\n                header = ['Token', 'NER', 'Type']\n                block = [[] for _ in range(length + 1)]\n                _ner = []\n                _type = []\n                offset = 0\n                for ent, label, b, e in ner_per_sample:\n                    render_labeled_span(b, e, _ner, _type, label, offset)\n                    offset = e\n                if offset != length:\n                    _ner.extend([''] * (length - offset))\n                    _type.extend([''] * (length - offset))\n                if any(_type):\n                    block[0].extend(header)\n                    for j, (_s, _t) in enumerate(zip(_ner, _type)):\n                        block[j + 1].extend((tokens[j], _s, _t))\n                    text = condense(block, extras)\n\n            if srl in self:\n                srl_samples = self[srl]\n                if flat:\n                    srl_samples = [srl_samples]\n                srl_per_sample = srl_samples[i]\n                for k, pas in enumerate(srl_per_sample):\n                    if not pas:\n                        continue\n                    block = [[] for _ in range(length + 1)]\n                    header = ['Token', 'SRL', f'PA{k + 1}']\n                    _srl = []\n                    _type = []\n                    offset = 0\n                    p_index = None\n                    for _, label, b, e in pas:\n                        render_labeled_span(b, e, _srl, _type, label, offset)\n                        offset = e\n                        if label == PRED:\n                            p_index = b\n                    if len(_srl) != length:\n                        _srl.extend([''] * (length - offset))\n                        _type.extend([''] * (length - offset))\n                    if p_index is not None:\n                        _srl[p_index] = '╟──►'\n                        # _type[j] = 'V'\n                        if len(block) != len(_srl) + 1:\n                            # warnings.warn(f'Unable to visualize overlapped spans: {pas}')\n                            continue\n                        block[0].extend(header)\n                        while len(_srl) < length:\n                            _srl.append('')\n                        while len(_type) < length:\n                            _type.append('')\n                        for j, (_s, _t) in enumerate(zip(_srl, _type)):\n                            block[j + 1].extend((tokens[j], _s, _t))\n                    text = condense(block, extras)\n            if con in self:\n                con_samples: Tree = self[con]\n                if flat:\n                    con_samples: List[Tree] = [con_samples]\n                tree = con_samples[i]\n                block = [[] for _ in range(length + 1)]\n                block[0].extend(('Token', 'PoS'))\n                for j, t in enumerate(tree.pos()):\n                    block[j + 1].extend(t)\n\n                for height in range(2, tree.height() + (0 if len(tree) == 1 else 1)):\n                    offset = 0\n                    spans = []\n                    labels = []\n                    for k, subtree in enumerate(tree.subtrees(lambda x: x.height() == height)):\n                        subtree: Tree = subtree\n                        b, e = offset, offset + len(subtree.leaves())\n                        if height >= 3:\n                            b, e = subtree[0].center, subtree[-1].center + 1\n                        subtree.center = b + (e - b) // 2\n                        render_labeled_span(b, e, spans, labels, subtree.label(), offset, unidirectional=True)\n                        offset = e\n                    if len(spans) != length:\n                        spans.extend([''] * (length - len(spans)))\n                    if len(labels) != length:\n                        labels.extend([''] * (length - len(labels)))\n                    if height < 3:\n                        continue\n                    block[0].extend(['', f'{height}'])\n                    for j, (_s, _t) in enumerate(zip(spans, labels)):\n                        block[j + 1].extend((_s, _t))\n                    # check short arrows and increase their length\n                    for j, arrow in enumerate(spans):\n                        if not arrow:\n                            # -1 current tag ; -2 arrow to current tag ; -3 = prev tag ; -4 = arrow to prev tag\n                            if block[j + 1][-3] or block[j + 1][-4] == '───►':\n                                if height > 3:\n                                    if block[j + 1][-3]:\n                                        block[j + 1][-1] = block[j + 1][-3]\n                                        block[j + 1][-2] = '───►'\n                                    else:\n                                        block[j + 1][-1] = '────'\n                                        block[j + 1][-2] = '────'\n                                    block[j + 1][-3] = '────'\n                                    if block[j + 1][-4] == '───►':\n                                        block[j + 1][-4] = '────'\n                                else:\n                                    block[j + 1][-1] = '────'\n                                if block[j + 1][-1] == '────':\n                                    block[j + 1][-2] = '────'\n                                if not block[j + 1][-4]:\n                                    block[j + 1][-4] = '────'\n                # If the root label is shorter than the level number, extend it to the same length\n                level_len = len(block[0][-1])\n                for row in block[1:]:\n                    if row[-1] and len(row[-1]) < level_len:\n                        row[-1] = row[-1] + ' ' * (level_len - len(row[-1]))\n\n                text = condense(block)\n                # Cosmetic issues\n                for row in text[1:]:\n                    while '  ─' in row[1]:\n                        row[1] = row[1].replace('  ─', ' ──')\n                    row[1] = row[1].replace('─ ─', '───')\n                    row[1] = re.sub(r'([►─])([\\w-]*)(\\s+)([│├])', lambda\n                        m: f'{m.group(1)}{m.group(2)}{\"─\" * len(m.group(3))}{\"┤\" if m.group(4) == \"│\" else \"┼\"}',\n                                    row[1])\n                    row[1] = re.sub(r'►(─+)►', r'─\\1►', row[1])\n                for r, s in zip(extras, text):\n                    r.extend(s)\n            # warnings.warn('Unable to visualize non-projective trees.')\n            if dep in self and conll.projective:\n                text = conll.to_tree(extras)\n                if not show_header:\n                    text = text.split('\\n')\n                    text = '\\n'.join(text[2:])\n                results.append(text)\n            elif any(extras):\n                results.append(make_table(extras, insert_header=True))\n            else:\n                results.append(' '.join(['/'.join(str(f) for f in x.nonempty_fields) for x in conll]))\n        if html:\n            def to_html(pretty_text: str) -> str:\n                lines = [x for x in pretty_text.split('\\n') if x]\n                cells = []\n                for line in lines:\n                    cells.append(line.split('\\t'))\n\n                num_cols = len(cells[0])\n                cols = []\n\n                for i in range(num_cols):\n                    cols.append([])\n                    for row in cells:\n                        cols[-1].append(row[i])\n\n                html = '<div style=\"display: table; padding-bottom: 1rem;\">'\n                for i, each in enumerate(cols):\n                    html += '<pre style=\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,' \\\n                            'Liberation Mono,Courier New,monospace; white-space: nowrap; line-height: 128%; padding: 0;\">'\n                    if i != len(cols) - 1:\n                        each = [x + ' ' for x in each]\n                    html += '<br>'.join([x.replace(' ', '&nbsp;') for x in each])\n                    html += '</pre>'\n                html += '</div>'\n                return html\n\n            results = [to_html(x) for x in results]\n        if flat:\n            return results[0]\n        return results\n\n    def pretty_print(self, tok='tok', lem='lem', pos='pos', dep='dep', sdp='sdp', ner='ner', srl='srl', con='con',\n                     show_header=True, html=IPYTHON):\n        \"\"\"\n        Print a pretty text representation which visualizes linguistic structures.\n\n        Args:\n            tok: Token key.\n            lem: Lemma key.\n            pos: Part-of-speech key.\n            dep: Dependency parse tree key.\n            sdp: Semantic dependency tree/graph key. SDP visualization has not been implemented yet.\n            ner: Named entity key.\n            srl: Semantic role labeling key.\n            con: Constituency parsing key.\n            show_header: ``True`` to print a header which indicates each field with its name.\n            html: ``True`` to output HTML format so that non-ASCII characters can align correctly.\n\n        \"\"\"\n        results = self.to_pretty(tok, lem, pos, dep, sdp, ner, srl, con, show_header, html=html)\n        if isinstance(results, str):\n            results = [results]\n        if html and IPYTHON:\n            from IPython.core.display import display, HTML\n            display(HTML('<br>'.join(results)))\n        else:\n            sent_new_line = '\\n\\n' if any('\\n' in x for x in results) else '\\n'\n            print(sent_new_line.join(results))\n\n    def translate(self, lang, tok='tok', pos='pos', dep='dep', sdp='sdp', ner='ner', srl='srl'):\n        \"\"\"\n        Translate tags for each annotation. This is an inplace operation.\n\n        .. Attention:: Note that the translated document might not print well in terminal due to non-ASCII characters.\n\n        Args:\n            lang: Target language to be translated to.\n            tok: Token key.\n            pos: Part-of-speech key.\n            dep: Dependency parse tree key.\n            sdp: Semantic dependency tree/graph key. SDP visualization has not been implemented yet.\n            ner: Named entity key.\n            srl: Semantic role labeling key.\n\n        Returns:\n            The translated document.\n\n        \"\"\"\n        if lang == 'zh':\n            from hanlp.utils.lang.zh import localization\n        else:\n            raise NotImplementedError(f'No translation for {lang}. '\n                                      f'Please contribute to our translation at https://github.com/hankcs/HanLP')\n        flat = isinstance(self[tok][0], str)\n        for task, name in zip(['pos', 'ner', 'dep', 'sdp', 'srl'], [pos, ner, dep, sdp, srl]):\n            annotations = self.get(name, None)\n            if not annotations:\n                continue\n            if flat:\n                annotations = [annotations]\n            translate: dict = getattr(localization, name, None)\n            if not translate:\n                continue\n            for anno_per_sent in annotations:\n                for i, v in enumerate(anno_per_sent):\n                    if task == 'ner' or task == 'dep':\n                        v[1] = translate.get(v[1], v[1])\n                    else:\n                        anno_per_sent[i] = translate.get(v, v)\n        return self\n\n    def squeeze(self, i=0):\n        r\"\"\"\n        Squeeze the dimension of each field into one. It's intended to convert a nested document like ``[[sent_i]]``\n        to ``[sent_i]``. When there are multiple sentences, only the ``i-th`` one will be returned. Note this is not an\n        inplace operation.\n\n        Args:\n            i: Keep the element at ``index`` for all ``list``\\s.\n\n        Returns:\n            A squeezed document with only one sentence.\n\n        \"\"\"\n        sq = Document()\n        for k, v in self.items():\n            sq[k] = v[i] if isinstance(v, list) else v\n        return sq\n\n    def _to_doc_without_spans(self, tok: str):\n        \"\"\"\n        Remove the spans attached to tokens and return a new document.\n\n        Args:\n            tok: The key to tokens.\n\n        Returns:\n            A new document or itself.\n\n        \"\"\"\n        tokens: Union[List[str], List[List[str]], List[str, int, int],\n                      List[List[str, int, int]]] = self[tok]\n        if isinstance(tokens[0], str):\n            return self\n        elif isinstance(tokens[0][-1], int):\n            tokens = [x[0] for x in tokens]\n        elif isinstance(tokens[0][-1], str):\n            return self\n        else:\n            tokens = [[t[0] for t in x] for x in tokens]\n        d = Document(**self)\n        d[tok] = tokens\n        return d\n\n    def get_by_prefix(self, prefix: str):\n        \"\"\"\n        Get value by the prefix of a key.\n\n        Args:\n            prefix: The prefix of a key. If multiple keys are matched, only the first one will be used.\n\n        Returns:\n            The value assigned with the matched key.\n        \"\"\"\n        key = prefix_match(prefix, self)\n        if not key:\n            return None\n        return self[key]\n\n    def count_sentences(self) -> int:\n        \"\"\"\n        Count number of sentences in this document.\n\n        Returns:\n            Number of sentences.\n        \"\"\"\n        tok = self.get_by_prefix('tok')\n        if isinstance(tok[0], str):\n            return 1\n        return len(tok)\n"
  },
  {
    "path": "plugins/hanlp_common/hanlp_common/io.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-16 22:38\nimport json\nimport os\nimport pickle\nimport sys\nfrom typing import Union\n\n\ndef save_pickle(item, path):\n    with open(path, 'wb') as f:\n        pickle.dump(item, f)\n\n\ndef load_pickle(path):\n    with open(path, 'rb') as f:\n        return pickle.load(f)\n\n\ndef save_json(item: Union[dict, list, str, int, float], path: str, ensure_ascii=False, cls=None,\n              default=lambda o: repr(o), indent=2):\n    dirname = os.path.dirname(path)\n    if dirname:\n        os.makedirs(dirname, exist_ok=True)\n    with open(path, 'w', encoding='utf-8') as out:\n        json.dump(item, out, ensure_ascii=ensure_ascii, indent=indent, cls=cls, default=default)\n\n\ndef load_json(path):\n    with open(path, encoding='utf-8') as src:\n        return json.load(src)\n\n\ndef filename_is_json(filename):\n    filename, file_extension = os.path.splitext(filename)\n    return file_extension in ['.json', '.jsonl']\n\n\ndef eprint(*args, **kwargs):\n    print(*args, file=sys.stderr, **kwargs)\n"
  },
  {
    "path": "plugins/hanlp_common/hanlp_common/reflection.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 16:41\nimport importlib\nimport inspect\n\n\ndef classpath_of(obj) -> str:\n    \"\"\"get the full class path of object\n\n    Args:\n      obj: return:\n\n    Returns:\n\n    \"\"\"\n    if inspect.isfunction(obj):\n        return module_path_of(obj)\n    return \"{0}.{1}\".format(obj.__class__.__module__, obj.__class__.__name__)\n\n\ndef module_path_of(func) -> str:\n    return inspect.getmodule(func).__name__ + '.' + func.__name__\n\n\ndef object_from_classpath(classpath, **kwargs):\n    classpath = str_to_type(classpath)\n    if inspect.isfunction(classpath):\n        return classpath\n    return classpath(**kwargs)\n\n\ndef str_to_type(classpath):\n    \"\"\"convert class path in str format to a type\n\n    Args:\n      classpath: class path\n\n    Returns:\n      type\n\n    \"\"\"\n    module_name, class_name = classpath.rsplit(\".\", 1)\n    cls = getattr(importlib.import_module(module_name), class_name)\n    return cls\n\n\ndef type_to_str(type_object) -> str:\n    \"\"\"convert a type object to class path in str format\n\n    Args:\n      type_object: type\n\n    Returns:\n      class path\n\n    \"\"\"\n    cls_name = str(type_object)\n    assert cls_name.startswith(\"<class '\"), 'illegal input'\n    cls_name = cls_name[len(\"<class '\"):]\n    assert cls_name.endswith(\"'>\"), 'illegal input'\n    cls_name = cls_name[:-len(\"'>\")]\n    return cls_name\n"
  },
  {
    "path": "plugins/hanlp_common/hanlp_common/structure.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-19 20:56\nimport json\nfrom collections import OrderedDict\n\nfrom hanlp_common.io import filename_is_json, save_pickle, load_pickle, save_json, load_json\n\n\nclass Serializable(object):\n    \"\"\"A super class for save/load operations.\"\"\"\n\n    def save(self, path, fmt=None):\n        if not fmt:\n            if filename_is_json(path):\n                self.save_json(path)\n            else:\n                self.save_pickle(path)\n        elif fmt in ['json', 'jsonl']:\n            self.save_json(path)\n        else:\n            self.save_pickle(path)\n\n    def load(self, path, fmt=None):\n        if not fmt:\n            if filename_is_json(path):\n                self.load_json(path)\n            else:\n                self.load_pickle(path)\n        elif fmt in ['json', 'jsonl']:\n            self.load_json(path)\n        else:\n            self.load_pickle(path)\n\n    def save_pickle(self, path):\n        \"\"\"Save to path\n\n        Args:\n          path:\n\n        Returns:\n\n\n        \"\"\"\n        save_pickle(self, path)\n\n    def load_pickle(self, path):\n        \"\"\"Load from path\n\n        Args:\n          path(str): file path\n\n        Returns:\n\n\n        \"\"\"\n        item = load_pickle(path)\n        return self.copy_from(item)\n\n    def save_json(self, path):\n        save_json(self.to_dict(), path)\n\n    def load_json(self, path):\n        item = load_json(path)\n        return self.copy_from(item)\n\n    # @abstractmethod\n    def copy_from(self, item):\n        self.__dict__ = item.__dict__\n        # raise NotImplementedError('%s.%s()' % (self.__class__.__name__, inspect.stack()[0][3]))\n\n    def to_json(self, ensure_ascii=False, indent=2, sort=False) -> str:\n        d = self.to_dict()\n        if sort:\n            d = OrderedDict(sorted(d.items()))\n        return json.dumps(d, ensure_ascii=ensure_ascii, indent=indent, default=lambda o: repr(o))\n\n    def to_dict(self) -> dict:\n        return self.__dict__\n\n\nclass SerializableDict(Serializable, dict):\n\n    def save_json(self, path):\n        save_json(self, path)\n\n    def copy_from(self, item):\n        if isinstance(item, dict):\n            self.clear()\n            self.update(item)\n\n    def __getattr__(self, key):\n        if key.startswith('__'):\n            return dict.__getattr__(key)\n        return self.__getitem__(key)\n\n    def __setattr__(self, key, value):\n        return self.__setitem__(key, value)\n\n    def to_dict(self) -> dict:\n        return self"
  },
  {
    "path": "plugins/hanlp_common/hanlp_common/util.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-10-27 19:09\nimport math\nfrom typing import Union, Any, List, Optional, Tuple, Iterable, Dict\nimport inspect\nfrom itertools import chain, combinations\n\n\ndef powerset(iterable, descending=False):\n    \"\"\"\n    powerset([1,2,3]) --> () (1,) (2,) (3,) (1,2) (1,3) (2,3) (1,2,3)\n\n    Args:\n        iterable:\n\n    Returns:\n\n    \"\"\"\n    s = list(iterable)\n    sizes = range(len(s), -1, -1) if descending else range(len(s) + 1)\n    return chain.from_iterable(combinations(s, r) for r in sizes)\n\n\ndef isdebugging():\n    \"\"\"See Also https://stackoverflow.com/questions/333995/how-to-detect-that-python-code-is-being-executed-through-the-debugger\"\"\"\n    for frame in inspect.stack():\n        if frame[1].endswith(\"pydevd.py\"):\n            return True\n    return False\n\n\ndef list_is_list_of_lists(sent: Union[Any, List[Any]]) -> Optional[bool]:\n    if not sent:\n        return None\n    return isinstance(sent[0], list)\n\n\ndef set_tuple_with(t: Tuple, v, at=0) -> Tuple:\n    t = list(t)\n    t[at] = v\n    return tuple(t)\n\n\ndef consume_keys_from_dict(keys: Iterable, d: dict) -> dict:\n    consumed = {}\n    for k in keys:\n        if k in d:\n            consumed[k] = d.pop(k)\n    return consumed\n\n\ndef merge_dict(d: dict, overwrite=False, inplace=False, **kwargs):\n    \"\"\"Merging the provided dict with other kvs\n\n    Args:\n      d: \n      kwargs: \n      d: dict: \n      overwrite:  (Default value = False)\n      inplace:  (Default value = False)\n      **kwargs: \n\n    Returns:\n\n    \n    \"\"\"\n    nd = dict([(k, v) for k, v in d.items()] + [(k, v) for k, v in kwargs.items() if overwrite or k not in d])\n    if inplace:\n        d.update(nd)\n        return d\n    return nd\n\n\ndef merge_locals_kwargs(locals: dict, kwargs: dict = None, excludes=('self', 'kwargs', '__class__')):\n    if not kwargs:\n        kwargs = dict()\n    return merge_dict(dict((k, v) for k, v in list(locals.items())\n                           if k not in excludes), **kwargs)\n\n\ndef infer_space_after(sent: List[str]):\n    last_token = None\n    quote_count: int = 0\n    # infer whitespace after field\n    whitespace_after = [True] * len(sent)\n    for token in range(len(sent)):\n        if sent[token] == '\"':\n            quote_count += 1\n            if quote_count % 2 != 0:\n                whitespace_after[token] = False\n            elif last_token is not None:\n                whitespace_after[last_token] = False\n\n        if last_token is not None:\n\n            if sent[token] in [\".\", \":\", \",\", \";\", \")\", \"n't\", \"!\", \"?\"]:\n                whitespace_after[last_token] = False\n\n            if sent[token].startswith(\"'\"):\n                whitespace_after[last_token] = False\n\n        if sent[token] in [\"(\"]:\n            whitespace_after[token] = False\n\n        last_token = token\n    return whitespace_after\n\n\ndef collapse_json(text, indent=12):\n    \"\"\"Compacts a string of json data by collapsing whitespace after the\n    specified indent level\n    \n    NOTE: will not produce correct results when indent level is not a multiple\n    of the json indent level\n\n    Args:\n      text: \n      indent:  (Default value = 12)\n\n    Returns:\n\n    \"\"\"\n    initial = \" \" * indent\n    out = []  # final json output\n    sublevel = []  # accumulation list for sublevel entries\n    pending = None  # holder for consecutive entries at exact indent level\n    for line in text.splitlines():\n        if line.startswith(initial):\n            if line[indent] == \" \":\n                # found a line indented further than the indent level, so add\n                # it to the sublevel list\n                if pending:\n                    # the first item in the sublevel will be the pending item\n                    # that was the previous line in the json\n                    sublevel.append(pending)\n                    pending = None\n                item = line.strip()\n                sublevel.append(item)\n                if item.endswith(\",\"):\n                    sublevel.append(\" \")\n            elif sublevel:\n                # found a line at the exact indent level *and* we have sublevel\n                # items. This means the sublevel items have come to an end\n                sublevel.append(line.strip())\n                out.append(\"\".join(sublevel))\n                sublevel = []\n            else:\n                # found a line at the exact indent level but no items indented\n                # further, so possibly start a new sub-level\n                if pending:\n                    # if there is already a pending item, it means that\n                    # consecutive entries in the json had the exact same\n                    # indentation and that last pending item was not the start\n                    # of a new sublevel.\n                    out.append(pending)\n                pending = line.rstrip()\n        else:\n            if pending:\n                # it's possible that an item will be pending but not added to\n                # the output yet, so make sure it's not forgotten.\n                out.append(pending)\n                pending = None\n            if sublevel:\n                out.append(\"\".join(sublevel))\n            out.append(line)\n    return \"\\n\".join(out)\n\n\nclass DummyContext(object):\n    def __enter__(self):\n        pass\n\n    def __exit__(self, exc_type, exc_val, exc_tb):\n        pass\n\n\ndef merge_list_of_dict(samples: List[Dict]) -> dict:\n    batch = {}\n    for each in samples:\n        for k, v in each.items():\n            vs = batch.get(k, None)\n            if vs is None:\n                vs = []\n                batch[k] = vs\n            vs.append(v)\n    return batch\n\n\ndef split_dict(batch: Dict[str, Any]) -> List[Dict[str, Any]]:\n    samples = []\n    batch = dict((k, v) for k, v in batch.items() if isinstance(v, list))\n    num_samples = len(max(batch.values(), key=len))\n    for i in range(num_samples):\n        samples.append(dict((k, v[i]) for k, v in batch.items()))\n    return samples\n\n\ndef reorder(samples: List, order: List[int]) -> List:\n    return [samples[i] for i in sorted(range(len(order)), key=lambda k: order[k])]\n\n\ndef k_fold(k, total, i):\n    trn = math.ceil(i / k * total)\n    tst = math.ceil((i + 1) / k * total)\n    return list(range(0, trn)) + list(range(tst, total)), list(range(trn, tst))\n\n\ndef dfs(graph, start):\n    seen = set()\n    path = []\n    q = [start]\n    while q:\n        v = q.pop()\n        if v not in seen:\n            seen.add(v)\n            path.append(v)\n            q.extend(graph[v])\n\n    return path\n\n\ndef topological_sort(graph, start):\n    seen = set()\n    stack = []\n    order = []\n    q = [start]\n    while q:\n        v = q.pop()\n        if v not in seen:\n            seen.add(v)\n            q.extend(graph[v])\n\n            while stack and v not in graph[stack[-1]]:\n                order.append(stack.pop())\n            stack.append(v)\n\n    return stack + order[::-1]\n\n\ndef prefix_match(target, sources: Iterable[str]):\n    if target is None:\n        return None\n    if target in sources:\n        return target\n    for each in sources:\n        if each.startswith(target):\n            return each\n"
  },
  {
    "path": "plugins/hanlp_common/hanlp_common/visualization.py",
    "content": "# -*- coding:utf-8 -*-\n# Modified from https://github.com/tylerneylon/explacy\nimport io\nfrom collections import defaultdict\nfrom pprint import pprint\n\nfrom phrasetree.tree import Tree\n\n\ndef make_table(rows, insert_header=False):\n    col_widths = [max(len(s) for s in col) for col in zip(*rows[1:])]\n    rows[0] = [x[:l] for x, l in zip(rows[0], col_widths)]\n    fmt = '\\t'.join('%%-%ds' % width for width in col_widths)\n    if insert_header:\n        rows.insert(1, ['─' * width for width in col_widths])\n    return '\\n'.join(fmt % tuple(row) for row in rows)\n\n\ndef _start_end(arrow):\n    start, end = arrow['from'], arrow['to']\n    mn = min(start, end)\n    mx = max(start, end)\n    return start, end, mn, mx\n\n\ndef pretty_tree_horizontal(arrows, _do_print_debug_info=False):\n    \"\"\"Print the dependency tree horizontally\n\n    Args:\n      arrows: \n      _do_print_debug_info:  (Default value = False)\n\n    Returns:\n\n    \"\"\"\n    # Set the base height; these may increase to allow room for arrowheads after this.\n    arrows_with_deps = defaultdict(set)\n    for i, arrow in enumerate(arrows):\n        arrow['underset'] = set()\n        if _do_print_debug_info:\n            print('Arrow %d: \"%s\" -> \"%s\"' % (i, arrow['from'], arrow['to']))\n        num_deps = 0\n        start, end, mn, mx = _start_end(arrow)\n        for j, other in enumerate(arrows):\n            if arrow is other:\n                continue\n            o_start, o_end, o_mn, o_mx = _start_end(other)\n            if ((start == o_start and mn <= o_end <= mx) or\n                    (start != o_start and mn <= o_start <= mx)):\n                num_deps += 1\n                if _do_print_debug_info:\n                    print('%d is over %d' % (i, j))\n                arrow['underset'].add(j)\n        arrow['num_deps_left'] = arrow['num_deps'] = num_deps\n        arrows_with_deps[num_deps].add(i)\n\n    if _do_print_debug_info:\n        print('')\n        print('arrows:')\n        pprint(arrows)\n\n        print('')\n        print('arrows_with_deps:')\n        pprint(arrows_with_deps)\n\n    # Render the arrows in characters. Some heights will be raised to make room for arrowheads.\n    sent_len = (max([max(arrow['from'], arrow['to']) for arrow in arrows]) if arrows else 0) + 1\n    lines = [[] for i in range(sent_len)]\n    num_arrows_left = len(arrows)\n    while num_arrows_left > 0:\n\n        assert len(arrows_with_deps[0])\n\n        arrow_index = arrows_with_deps[0].pop()\n        arrow = arrows[arrow_index]\n        src, dst, mn, mx = _start_end(arrow)\n\n        # Check the height needed.\n        height = 3\n        if arrow['underset']:\n            height = max(arrows[i]['height'] for i in arrow['underset']) + 1\n        height = max(height, 3, len(lines[dst]) + 3)\n        arrow['height'] = height\n\n        if _do_print_debug_info:\n            print('')\n            print('Rendering arrow %d: \"%s\" -> \"%s\"' % (arrow_index,\n                                                        arrow['from'],\n                                                        arrow['to']))\n            print('  height = %d' % height)\n\n        goes_up = src > dst\n\n        # Draw the outgoing src line.\n        if lines[src] and len(lines[src]) < height:\n            lines[src][-1].add('w')\n        while len(lines[src]) < height - 1:\n            lines[src].append(set(['e', 'w']))\n        if len(lines[src]) < height:\n            lines[src].append({'e'})\n        lines[src][height - 1].add('n' if goes_up else 's')\n\n        # Draw the incoming dst line.\n        lines[dst].append(u'►')\n        while len(lines[dst]) < height:\n            lines[dst].append(set(['e', 'w']))\n        lines[dst][-1] = set(['e', 's']) if goes_up else set(['e', 'n'])\n\n        # Draw the adjoining vertical line.\n        for i in range(mn + 1, mx):\n            while len(lines[i]) < height - 1:\n                lines[i].append(' ')\n            lines[i].append(set(['n', 's']))\n\n        # Update arrows_with_deps.\n        for arr_i, arr in enumerate(arrows):\n            if arrow_index in arr['underset']:\n                arrows_with_deps[arr['num_deps_left']].remove(arr_i)\n                arr['num_deps_left'] -= 1\n                arrows_with_deps[arr['num_deps_left']].add(arr_i)\n\n        num_arrows_left -= 1\n\n    return render_arrows(lines)\n\n\ndef render_arrows(lines):\n    arr_chars = {'ew': u'─',\n                 'ns': u'│',\n                 'en': u'└',\n                 'es': u'┌',\n                 'enw': u'┴',\n                 'ensw': u'┼',\n                 'ens': u'├',\n                 'esw': u'┬'}\n    # Convert the character lists into strings.\n    max_len = max(len(line) for line in lines)\n    for i in range(len(lines)):\n        lines[i] = [arr_chars[''.join(sorted(ch))] if type(ch) is set else ch for ch in lines[i]]\n        lines[i] = ''.join(reversed(lines[i]))\n        lines[i] = ' ' * (max_len - len(lines[i])) + lines[i]\n    return lines\n\n\ndef render_span(begin, end, unidirectional=False):\n    if end - begin == 1:\n        return ['───►']\n    elif end - begin == 2:\n        return [\n            '──┐',\n            '──┴►',\n        ] if unidirectional else [\n            '◄─┐',\n            '◄─┴►',\n        ]\n\n    rows = []\n    for i in range(begin, end):\n        if i == (end - begin) // 2 + begin:\n            rows.append('  ├►')\n        elif i == begin:\n            rows.append('──┐' if unidirectional else '◄─┐')\n        elif i == end - 1:\n            rows.append('──┘' if unidirectional else '◄─┘')\n        else:\n            rows.append('  │')\n    return rows\n\n\ndef tree_to_list(T):\n    return [T.label(), [tree_to_list(t) if isinstance(t, Tree) else t for t in T]]\n\n\ndef list_to_tree(L):\n    if isinstance(L, str):\n        return L\n    return Tree(L[0], [list_to_tree(child) for child in L[1]])\n\n\ndef render_labeled_span(b, e, spans, labels, label, offset, unidirectional=False):\n    spans.extend([''] * (b - offset))\n    spans.extend(render_span(b, e, unidirectional))\n    center = b + (e - b) // 2\n    labels.extend([''] * (center - offset))\n    labels.append(label)\n    labels.extend([''] * (e - center - 1))\n\n\ndef main():\n    # arrows = [{'from': 1, 'to': 0}, {'from': 2, 'to': 1}, {'from': 2, 'to': 4}, {'from': 2, 'to': 5},\n    #           {'from': 4, 'to': 3}]\n    # lines = pretty_tree_horizontal(arrows)\n    # print('\\n'.join(lines))\n    # print('\\n'.join([\n    #     '◄─┐',\n    #     '  │',\n    #     '  ├►',\n    #     '  │',\n    #     '◄─┘',\n    # ]))\n    print('\\n'.join(render_span(7, 12)))\n\n\nif __name__ == '__main__':\n    main()\nleft_rule = {'<': ':', '^': ':', '>': '-'}\nright_rule = {'<': '-', '^': ':', '>': ':'}\n\n\ndef evalute_field(record, field_spec):\n    \"\"\"Evalute a field of a record using the type of the field_spec as a guide.\n\n    Args:\n      record:\n      field_spec:\n\n    Returns:\n\n    \"\"\"\n    if type(field_spec) is int:\n        return str(record[field_spec])\n    elif type(field_spec) is str:\n        return str(getattr(record, field_spec))\n    else:\n        return str(field_spec(record))\n\n\ndef markdown_table(headings, records, fields=None, alignment=None, file=None):\n    \"\"\"Generate a Doxygen-flavor Markdown table from records.\n    See https://stackoverflow.com/questions/13394140/generate-markdown-tables\n\n    file -- Any object with a 'write' method that takes a single string\n        parameter.\n    records -- Iterable.  Rows will be generated from this.\n    fields -- List of fields for each row.  Each entry may be an integer,\n        string or a function.  If the entry is an integer, it is assumed to be\n        an index of each record.  If the entry is a string, it is assumed to be\n        a field of each record.  If the entry is a function, it is called with\n        the record and its return value is taken as the value of the field.\n    headings -- List of column headings.\n    alignment - List of pairs alignment characters.  The first of the pair\n        specifies the alignment of the header, (Doxygen won't respect this, but\n        it might look good, the second specifies the alignment of the cells in\n        the column.\n\n        Possible alignment characters are:\n            '<' = Left align\n            '>' = Right align (default for cells)\n            '^' = Center (default for column headings)\n\n    Args:\n      headings:\n      records:\n      fields:  (Default value = None)\n      alignment:  (Default value = None)\n      file:  (Default value = None)\n\n    Returns:\n\n    \"\"\"\n    if not file:\n        file = io.StringIO()\n    num_columns = len(headings)\n    if not fields:\n        fields = list(range(num_columns))\n    assert len(headings) == num_columns\n\n    # Compute the table cell data\n    columns = [[] for i in range(num_columns)]\n    for record in records:\n        for i, field in enumerate(fields):\n            columns[i].append(evalute_field(record, field))\n\n    # Fill out any missing alignment characters.\n    extended_align = alignment if alignment is not None else [('^', '<')]\n    if len(extended_align) > num_columns:\n        extended_align = extended_align[0:num_columns]\n    elif len(extended_align) < num_columns:\n        extended_align += [('^', '>') for i in range(num_columns - len(extended_align))]\n\n    heading_align, cell_align = [x for x in zip(*extended_align)]\n\n    field_widths = [len(max(column, key=len)) if len(column) > 0 else 0\n                    for column in columns]\n    heading_widths = [max(len(head), 2) for head in headings]\n    column_widths = [max(x) for x in zip(field_widths, heading_widths)]\n\n    _ = ' | '.join(['{:' + a + str(w) + '}'\n                    for a, w in zip(heading_align, column_widths)])\n    heading_template = '| ' + _ + ' |'\n    _ = ' | '.join(['{:' + a + str(w) + '}'\n                    for a, w in zip(cell_align, column_widths)])\n    row_template = '| ' + _ + ' |'\n\n    _ = ' | '.join([left_rule[a] + '-' * (w - 2) + right_rule[a]\n                    for a, w in zip(cell_align, column_widths)])\n    ruling = '| ' + _ + ' |'\n\n    file.write(heading_template.format(*headings).rstrip() + '\\n')\n    file.write(ruling.rstrip() + '\\n')\n    for row in zip(*columns):\n        file.write(row_template.format(*row).rstrip() + '\\n')\n    if isinstance(file, io.StringIO):\n        text = file.getvalue()\n        file.close()\n        return text\n"
  },
  {
    "path": "plugins/hanlp_common/setup.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 19:26\nfrom os.path import abspath, join, dirname\nfrom setuptools import find_packages, setup\n\nthis_dir = abspath(dirname(__file__))\nwith open(join(this_dir, 'README.md'), encoding='utf-8') as file:\n    long_description = file.read()\n\nsetup(\n    name='hanlp_common',\n    version='0.0.22',\n    description='HanLP: Han Language Processing',\n    long_description=long_description,\n    long_description_content_type=\"text/markdown\",\n    url='https://github.com/hankcs/HanLP',\n    author='hankcs',\n    author_email='hankcshe@gmail.com',\n    license='Apache License 2.0',\n    classifiers=[\n        'Intended Audience :: Science/Research',\n        'Intended Audience :: Developers',\n        \"Development Status :: 3 - Alpha\",\n        'Operating System :: OS Independent',\n        \"License :: OSI Approved :: Apache Software License\",\n        'Programming Language :: Python :: 3 :: Only',\n        'Topic :: Scientific/Engineering :: Artificial Intelligence',\n        \"Topic :: Text Processing :: Linguistic\"\n    ],\n    keywords='corpus,machine-learning,NLU,NLP',\n    packages=find_packages(exclude=['docs', 'tests*']),\n    include_package_data=True,\n    install_requires=[\n        'phrasetree>=0.0.9',\n    ],\n    extras_require={\n        # These AMR dependencies might not be necessary for most people.\n        'full': [\n            'networkx',\n            'penman==0.6.2',\n        ],\n    },\n    python_requires='>=3.6',\n)\n"
  },
  {
    "path": "plugins/hanlp_demo/README.md",
    "content": "# Demos and examples for HanLP\n\nThis package is intended for demonstration purpose and won't be released to pypi. **Training requires a fair understanding of Linux and Python which might not be the case for everybody.**\n\nYou need a Linux/macOS system with Internet on because some corpora and bash scripts will be downloaded during training. Training on Windows might work if you are an expert but we believe it's very rare.\n\nYour `python` command needs to be Python2 while `python3` needs to be Python3.\n\nYou need to install this package and run it from the **root** folder of HanLP.\n\n```bash\npip install -e plugins/hanlp_demo\npython3 plugins/hanlp_demo/hanlp_demo/zh/train/open_small.py\n```\n\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-11-29 17:48\n\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/block_windows.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-07-28 21:38\nfrom hanlp.utils.io_util import windows\n\nassert not windows(), 'Windows is not supported for this script. Please run it on Linux systems.'\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/en/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-01-01 17:55"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/en/demo_amr.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2022-01-25 19:09\nimport hanlp\n\namr_parser = hanlp.load(hanlp.pretrained.amr.AMR3_SEQ2SEQ_BART_LARGE)\namr = amr_parser('The boy wants the girl to believe him.')\nprint(amr)\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/en/demo_dep.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-01-01 17:55\nimport hanlp\n\nsyntactic_parser = hanlp.load(hanlp.pretrained.dep.PTB_BIAFFINE_DEP_EN)\nsent = [('Is', 'VBZ'),\n        ('this', 'DT'),\n        ('the', 'DT'),\n        ('future', 'NN'),\n        ('of', 'IN'),\n        ('chamber', 'NN'),\n        ('music', 'NN'),\n        ('?', '.')]\ntree = syntactic_parser(sent)\nprint(tree)\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/en/demo_lm.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-02-11 09:14\nimport hanlp\n\nlm = hanlp.load(hanlp.pretrained.rnnlm.FLAIR_LM_FW_WMT11_EN_TF)\nprint(''.join(lm.generate_text(list('hello'))))\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/en/demo_ner.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-01-03 22:50\nimport hanlp\n\nrecognizer = hanlp.load(hanlp.pretrained.ner.CONLL03_NER_BERT_BASE_CASED_EN)\nprint(recognizer([\"President\", \"Obama\", \"is\", \"speaking\", \"at\", \"the\", \"White\", \"House\", \".\"]))\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/en/demo_pipeline.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-01-04 21:05\nimport hanlp\nfrom hanlp.utils.lang.en.english_tokenizer import tokenize_english\n\ntokenizer = tokenize_english\ntagger = hanlp.load(hanlp.pretrained.pos.PTB_POS_RNN_FASTTEXT_EN)\nsyntactic_parser = hanlp.load(hanlp.pretrained.dep.PTB_BIAFFINE_DEP_EN)\nsemantic_parser = hanlp.load(hanlp.pretrained.sdp.SEMEVAL15_PAS_BIAFFINE_EN)\n\npipeline = hanlp.pipeline() \\\n    .append(hanlp.utils.rules.split_sentence, output_key='sentences') \\\n    .append(tokenizer, output_key='tokens') \\\n    .append(tagger, output_key='part_of_speech_tags') \\\n    .append(syntactic_parser, input_key=('tokens', 'part_of_speech_tags'), output_key='syntactic_dependencies',\n            conll=False) \\\n    .append(semantic_parser, input_key=('tokens', 'part_of_speech_tags'), output_key='semantic_dependencies',\n            conll=False)\nprint(pipeline)\n\ntext = '''Jobs and Wozniak co-founded Apple in 1976 to sell Wozniak's Apple I personal computer.\nTogether the duo gained fame and wealth a year later with the Apple II.\n'''\n\ndoc = pipeline(text)\nprint(doc)\n\n# You can save the config to disk for deploying or sharing.\npipeline.save('en.json')\n# Then load it smoothly.\ndeployed = hanlp.load('en.json')\nprint(deployed)\nprint(deployed(text))\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/en/demo_pos.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-01-03 22:16\nimport hanlp\n\ntagger = hanlp.load(hanlp.pretrained.pos.PTB_POS_RNN_FASTTEXT_EN)\nprint(tagger([['I', 'banked', '2', 'dollars', 'in', 'a', 'bank', '.'],\n              ['Is', 'this', 'the', 'future', 'of', 'chamber', 'music', '?']]))\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/en/demo_sdp.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-01-03 15:26\nimport hanlp\nfrom hanlp_common.conll import CoNLLSentence\n\n# semeval15 offers three independent annotations over the Penn Treebank (PTB)\nsemantic_parser = hanlp.load(hanlp.pretrained.sdp.SEMEVAL15_PAS_BIAFFINE_EN)\n# semantic_parser = hanlp.load(hanlp.pretrained.sdp.SEMEVAL15_DM_BIAFFINE_EN)\n# semantic_parser = hanlp.load(hanlp.pretrained.sdp.SEMEVAL15_PSD_BIAFFINE_EN)\nsent = [('Is', 'VBZ'),\n        ('this', 'DT'),\n        ('the', 'DT'),\n        ('future', 'NN'),\n        ('of', 'IN'),\n        ('chamber', 'NN'),\n        ('music', 'NN'),\n        ('?', '.')]\ntree = semantic_parser(sent)  # type:CoNLLSentence\nprint(tree)\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/en/demo_sentiment_analysis.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-01-01 03:52\nimport hanlp\n\nclassifier = hanlp.load('SST2_ALBERT_BASE_EN')\nprint(classifier.predict('I feel lucky'))\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/en/demo_tok.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-01-02 19:41\nfrom hanlp.utils.lang.en.english_tokenizer import tokenize_english\n\ntext = \"\"\"\\\nDon't go gentle into that good night.\n\"\"\"\nprint(tokenize_english(text))\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/en/train_sst2_albert_base.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-11-10 17:41\nimport os\n\nfrom hanlp.components.classifiers.transformer_classifier_tf import TransformerClassifierTF\n\nfrom tests import cdroot\n\nfrom hanlp.datasets.glu.glue import STANFORD_SENTIMENT_TREEBANK_2_DEV, STANFORD_SENTIMENT_TREEBANK_2_TRAIN, \\\n    STANFORD_SENTIMENT_TREEBANK_2_TEST\n\ncdroot()\nsave_dir = os.path.join('data', 'model', 'sst', 'sst2_albert_base')\nclassifier = TransformerClassifierTF()\nclassifier.fit(STANFORD_SENTIMENT_TREEBANK_2_TRAIN, STANFORD_SENTIMENT_TREEBANK_2_DEV, save_dir,\n               transformer='albert-base-v2')\nclassifier.load(save_dir)\nprint(classifier('it\\' s a charming and often affecting journey'))\nclassifier.evaluate(STANFORD_SENTIMENT_TREEBANK_2_TEST, save_dir=save_dir)\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/ja/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-05-17 22:30\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/ja/demo_mtl.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-05-17 22:30\nimport hanlp\nfrom hanlp_common.document import Document\n\nHanLP = hanlp.load(hanlp.pretrained.mtl.NPCMJ_UD_KYOTO_TOK_POS_CON_BERT_BASE_CHAR_JA)\ndoc: Document = HanLP([\n    '2021年、HanLPv2.1は次世代の最先端多言語NLP技術を本番環境に導入します。',\n    '奈須きのこは1973年11月28日に千葉県円空山で生まれ、ゲーム制作会社「ノーツ」の設立者だ。',\n])\nprint(doc)\ndoc.pretty_print()\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/mul/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-31 22:25\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/mul/demo_lid.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2022-09-28 16:49\nimport hanlp\n\nlid = hanlp.load(hanlp.pretrained.classifiers.LID_176_FASTTEXT_BASE)\n\nprint(lid('In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques to production environments.'))\nlang, prob = lid('2021年、HanLPv2.1は次世代の最先端多言語NLP技術を本番環境に導入します。', prob=True)\nprint(f'{lang} language identified with probability {prob:.3%}')\nprint(lid('2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', topk=2))\n\n# For a combination of languages, predict top-k languages with probabilities:\ntext = '''\n2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。\nIn 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques to production environments.\n'''\n\nprint(lid(text, topk=3, prob=True))\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/mul/demo_lid_restful.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2022-09-28 16:49\nfrom hanlp_restful import HanLPClient\n\nHanLP = HanLPClient('https://hanlp.hankcs.com/api', auth=None, language='mul')\n\nprint(HanLP.language_identification([\n    'In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques to production environment.',\n    '2021年、HanLPv2.1は次世代の最先端多言語NLP技術を本番環境に導入します。',\n    '2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。',\n]))\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/mul/demo_mtl.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-31 13:51\nimport hanlp\nfrom hanlp_common.document import Document\n\nHanLP = hanlp.load(hanlp.pretrained.mtl.UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_XLMR_BASE)\ndoc: Document = HanLP([\n    'In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques to production environment.',\n    '2021年、HanLPv2.1は次世代の最先端多言語NLP技術を本番環境に導入します。',\n    '2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。',\n])\nprint(doc)\ndoc.pretty_print()\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/mul/train/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2023-02-21 19:40\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/mul/train/mul_base.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-03 14:24\nfrom hanlp.common.dataset import SortingSamplerBuilder\nfrom hanlp.common.transform import NormalizeToken\nfrom hanlp.components.mtl.multi_task_learning import MultiTaskLearning\nfrom hanlp.components.mtl.tasks.tok.tag_tok import TaggingTokenization\nfrom hanlp.components.mtl.tasks.ud import UniversalDependenciesParsing\nfrom hanlp.datasets.parsing.ptb import PTB_TOKEN_MAPPING\nfrom hanlp.datasets.parsing.ud.ud210m import UD_210_MULTILINGUAL_TRAIN, UD_210_MULTILINGUAL_DEV, \\\n    UD_210_MULTILINGUAL_TEST\nfrom hanlp.layers.embeddings.contextual_word_embedding import ContextualWordEmbedding\nfrom hanlp.utils.log_util import cprint\nfrom tests import cdroot\n\n\ndef main():\n    cdroot()\n    transformer = \"nreimers/mMiniLMv2-L12-H384-distilled-from-XLMR-Large\"\n    tasks = {\n        'tok': TaggingTokenization(\n            'data/mtl/mul/tok/train.tsv',\n            'data/mtl/mul/tok/dev.tsv',\n            'data/mtl/mul/tok/test.tsv',\n            SortingSamplerBuilder(batch_size=128, batch_max_tokens=12800),\n            hard_constraint=True,\n            tagging_scheme='BMES',\n            delimiter='\\t',\n            max_seq_len=256,\n            char_level=True,\n            lr=1e-3,\n        ),\n        'ud': UniversalDependenciesParsing(\n            UD_210_MULTILINGUAL_TRAIN,\n            UD_210_MULTILINGUAL_DEV,\n            UD_210_MULTILINGUAL_TEST,\n            SortingSamplerBuilder(batch_size=128, batch_max_tokens=12800),\n            lr=1e-3,\n            dependencies='tok',\n            max_seq_len=256,\n        ),\n    }\n    mtl = MultiTaskLearning()\n    save_dir = 'data/model/mtl/ud_ontonotes_tok_pos_lem_fea_ner_srl_dep_sdp_con_mMiniLMv2L12'\n    cprint(f'Model will be saved in [cyan]{save_dir}[/cyan]')\n    mtl.fit(\n        ContextualWordEmbedding(\n            'token',\n            transformer,\n            average_subwords=True,\n            max_sequence_length=512,\n            word_dropout=.2,\n        ),\n        tasks,\n        save_dir,\n        30,\n        lr=1e-3,\n        encoder_lr=5e-5,\n        grad_norm=1,\n        gradient_accumulation=8,\n        eval_trn=False,\n        transform=NormalizeToken(PTB_TOKEN_MAPPING, 'token'),\n        tau=0.5,\n        cache='data/cache/ud/mtl',\n    )\n    cprint(f'Model saved in [cyan]{save_dir}[/cyan]')\n    mtl.load(save_dir)\n    mtl['tok'].dict_force = {\"'s\", \"n't\", \"'ll\", \"'m\", \"'d\", \"'ve\", \"'re\"}\n    mtl['ud'].config.tree = True\n    mtl.save_config(save_dir)\n    for k, v in mtl.tasks.items():\n        v.trn = tasks[k].trn\n        v.dev = tasks[k].dev\n        v.tst = tasks[k].tst\n    mtl.evaluate(save_dir)\n    doc = mtl(['In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques to production environments.',\n               '2021年、HanLPv2.1は次世代の最先端多言語NLP技術を本番環境に導入します。',\n               '2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。'])\n    doc.pretty_print()\n\n\nif __name__ == '__main__':\n    main()\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/sent_split.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-31 14:23\nimport hanlp\n\nsplit_sent = hanlp.load(hanlp.pretrained.eos.UD_CTB_EOS_MUL)\noutput = split_sent('3.14 is pi. “你好！！！”——他说。劇場版「Fate/stay night [HF]」最終章公開カウントダウン！')\nprint('\\n'.join(output))\n# See also https://hanlp.hankcs.com/docs/api/hanlp/components/eos.html\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-31 13:51\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/abstractive_summarization_restful.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/absum_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fabsum_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp_restful -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 创建客户端\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"4M7ka0K5OMWU\",\n    \"outputId\": \"d74f0749-0587-454a-d7c9-7418d45ce534\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from hanlp_restful import HanLPClient\\n\",\n    \"HanLP = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh') # auth不填则匿名，zh中文，mul多语种\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"BMW528wGNulM\"\n   },\n   \"source\": [\n    \"#### 申请秘钥\\n\",\n    \"由于服务器算力有限，匿名用户每分钟限2次调用。如果你需要更多调用次数，[建议申请免费公益API秘钥auth](https://bbs.hanlp.com/t/hanlp2-1-restful-api/53)。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 生成式自动摘要\\n\",\n    \"生成式自动摘要（Abstractive Summarization）任务的目标是为文章生成一段简短的概括性摘要。 生成的摘要有可能出现原文中不存在的新短语或新句子，并且整体流畅性较高。\\n\",\n    \"### 中文\\n\",\n    \"生成式自动摘要任务的输入为一段文本：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"936d439a-e1ff-4308-d2aa-775955558594\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"'长江证券：看好大金属品种中的铜铝钢'\"\n      ]\n     },\n     \"execution_count\": 2,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"HanLP.abstractive_summarization('''\\n\",\n    \"每经AI快讯，2月4日，长江证券研究所金属行业首席分析师王鹤涛表示，2023年海外经济衰退，美债现处于历史高位，\\n\",\n    \"黄金的趋势是值得关注的；在国内需求修复的过程中，看好大金属品种中的铜铝钢。\\n\",\n    \"此外，在细分的小品种里，建议关注两条主线，一是新能源，比如锂、钴、镍、稀土，二是专精特新主线。（央视财经）\\n\",\n    \"''')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"jj1Jk-2sPHYx\"\n   },\n   \"source\": [\n    \"返回值为一段摘要。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### 英文\\n\",\n    \"按照HanLP一贯的多语种设计，任何语言都支持。由于服务器GPU资源限制，目前英文接口暂未上线。如果你有相应需求，欢迎前往论坛发起请愿。\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"absum_restful.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/amr_restful.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/amr_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Famr_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp_restful -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 创建客户端\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"4M7ka0K5OMWU\",\n    \"outputId\": \"d74f0749-0587-454a-d7c9-7418d45ce534\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from hanlp_restful import HanLPClient\\n\",\n    \"HanLP = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh') # auth不填则匿名，zh中文，mul多语种\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"BMW528wGNulM\"\n   },\n   \"source\": [\n    \"#### 申请秘钥\\n\",\n    \"由于服务器算力有限，匿名用户每分钟限2次调用。如果你需要更多调用次数，[建议申请免费公益API秘钥auth](https://bbs.hanlp.com/t/hanlp2-1-restful-api/53)。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 抽象意义表示\\n\",\n    \"### 中文\\n\",\n    \"抽象意义表示任务的输入为一段文本或已分词完毕的句子：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"936d439a-e1ff-4308-d2aa-775955558594\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"1\"\n      ]\n     },\n     \"execution_count\": 2,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"graphs = HanLP.abstract_meaning_representation('男孩希望女孩相信他。')\\n\",\n    \"len(graphs)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"jj1Jk-2sPHYx\"\n   },\n   \"source\": [\n    \"返回值为每个句子相应的AMR图的Meaning Representation格式：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'id': '0',\\n\",\n       \" 'input': '男孩 希望 女孩 相信 他 。',\\n\",\n       \" 'nodes': [{'id': 0,\\n\",\n       \"   'label': '男孩',\\n\",\n       \"   'anchors': [{'from': 0, 'to': 2}, {'from': 12, 'to': 13}]},\\n\",\n       \"  {'id': 1, 'label': '希望-01', 'anchors': [{'from': 3, 'to': 5}]},\\n\",\n       \"  {'id': 2, 'label': '女孩', 'anchors': [{'from': 6, 'to': 8}]},\\n\",\n       \"  {'id': 3, 'label': '相信-01', 'anchors': [{'from': 9, 'to': 11}]}],\\n\",\n       \" 'edges': [{'source': 1, 'target': 3, 'label': 'arg1'},\\n\",\n       \"  {'source': 1, 'target': 0, 'label': 'arg0'},\\n\",\n       \"  {'source': 3, 'target': 2, 'label': 'arg0'},\\n\",\n       \"  {'source': 3, 'target': 0, 'label': 'arg1'}],\\n\",\n       \" 'tops': [1],\\n\",\n       \" 'framework': 'amr'}\"\n      ]\n     },\n     \"execution_count\": 3,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"graph = graphs[0]\\n\",\n    \"graph\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"注意上面“男孩”有2个anchor，分别对应“男孩”和“他”。也就是说，MR格式其实包含了指代消解的结果。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### 可视化\\n\",\n    \"指定`visualization='svg'`即可得到矢量图可视化。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"image/svg+xml\": [\n       \"<svg xmlns=\\\"http://www.w3.org/2000/svg\\\" xmlns:xlink=\\\"http://www.w3.org/1999/xlink\\\" width=\\\"171pt\\\" height=\\\"298pt\\\" viewBox=\\\"0.00 0.00 170.78 297.55\\\">\\n\",\n       \"<g id=\\\"graph0\\\" class=\\\"graph\\\" transform=\\\"scale(1 1) rotate(0) translate(4 293.55)\\\">\\n\",\n       \"<title>0</title>\\n\",\n       \"<polygon fill=\\\"white\\\" stroke=\\\"transparent\\\" points=\\\"-4,4 -4,-293.55 166.78,-293.55 166.78,4 -4,4\\\"/>\\n\",\n       \"<!-- top -->\\n\",\n       \"<!-- 1 -->\\n\",\n       \"<g id=\\\"node2\\\" class=\\\"node\\\">\\n\",\n       \"<title>1</title>\\n\",\n       \"<ellipse fill=\\\"none\\\" stroke=\\\"black\\\" cx=\\\"59.53\\\" cy=\\\"-197.46\\\" rx=\\\"45.01\\\" ry=\\\"19.18\\\"/>\\n\",\n       \"<text text-anchor=\\\"start\\\" x=\\\"37.53\\\" y=\\\"-193.26\\\" font-family=\\\"Times,serif\\\" font-size=\\\"14.00\\\">希望-01</text>\\n\",\n       \"</g>\\n\",\n       \"<!-- top&#45;&gt;1 -->\\n\",\n       \"<g id=\\\"edge1\\\" class=\\\"edge\\\">\\n\",\n       \"<title>top-&gt;1</title>\\n\",\n       \"<path fill=\\\"none\\\" stroke=\\\"black\\\" d=\\\"M59.53,-253.47C59.53,-245.51 59.53,-235.82 59.53,-226.81\\\"/>\\n\",\n       \"<polygon fill=\\\"black\\\" stroke=\\\"black\\\" points=\\\"63.03,-226.68 59.53,-216.68 56.03,-226.68 63.03,-226.68\\\"/>\\n\",\n       \"</g>\\n\",\n       \"<!-- 0 -->\\n\",\n       \"<g id=\\\"node3\\\" class=\\\"node\\\">\\n\",\n       \"<title>0</title>\\n\",\n       \"<ellipse fill=\\\"none\\\" stroke=\\\"black\\\" cx=\\\"32.53\\\" cy=\\\"-19.09\\\" rx=\\\"32.55\\\" ry=\\\"19.18\\\"/>\\n\",\n       \"<text text-anchor=\\\"start\\\" x=\\\"19.53\\\" y=\\\"-14.89\\\" font-family=\\\"Times,serif\\\" font-size=\\\"14.00\\\">男孩</text>\\n\",\n       \"</g>\\n\",\n       \"<!-- 1&#45;&gt;0 -->\\n\",\n       \"<g id=\\\"edge2\\\" class=\\\"edge\\\">\\n\",\n       \"<title>1-&gt;0</title>\\n\",\n       \"<path fill=\\\"none\\\" stroke=\\\"black\\\" d=\\\"M52.8,-178.56C47.93,-164.78 41.68,-145.14 38.53,-127.37 33.85,-100.97 32.53,-70.4 32.28,-48.67\\\"/>\\n\",\n       \"<polygon fill=\\\"black\\\" stroke=\\\"black\\\" points=\\\"35.78,-48.45 32.23,-38.47 28.78,-48.48 35.78,-48.45\\\"/>\\n\",\n       \"<text text-anchor=\\\"middle\\\" x=\\\"51.03\\\" y=\\\"-104.58\\\" font-family=\\\"Times,serif\\\" font-size=\\\"14.00\\\">arg0</text>\\n\",\n       \"</g>\\n\",\n       \"<!-- 3 -->\\n\",\n       \"<g id=\\\"node4\\\" class=\\\"node\\\">\\n\",\n       \"<title>3</title>\\n\",\n       \"<ellipse fill=\\\"none\\\" stroke=\\\"black\\\" cx=\\\"117.53\\\" cy=\\\"-108.28\\\" rx=\\\"45.01\\\" ry=\\\"19.18\\\"/>\\n\",\n       \"<text text-anchor=\\\"start\\\" x=\\\"95.53\\\" y=\\\"-104.08\\\" font-family=\\\"Times,serif\\\" font-size=\\\"14.00\\\">相信-01</text>\\n\",\n       \"</g>\\n\",\n       \"<!-- 1&#45;&gt;3 -->\\n\",\n       \"<g id=\\\"edge3\\\" class=\\\"edge\\\">\\n\",\n       \"<title>1-&gt;3</title>\\n\",\n       \"<path fill=\\\"none\\\" stroke=\\\"black\\\" d=\\\"M71.26,-178.82C79.52,-166.41 90.71,-149.59 100.01,-135.6\\\"/>\\n\",\n       \"<polygon fill=\\\"black\\\" stroke=\\\"black\\\" points=\\\"103.18,-137.16 105.81,-126.89 97.35,-133.28 103.18,-137.16\\\"/>\\n\",\n       \"<text text-anchor=\\\"middle\\\" x=\\\"104.03\\\" y=\\\"-149.17\\\" font-family=\\\"Times,serif\\\" font-size=\\\"14.00\\\">arg1</text>\\n\",\n       \"</g>\\n\",\n       \"<!-- 3&#45;&gt;0 -->\\n\",\n       \"<g id=\\\"edge4\\\" class=\\\"edge\\\">\\n\",\n       \"<title>3-&gt;0</title>\\n\",\n       \"<path fill=\\\"none\\\" stroke=\\\"black\\\" d=\\\"M101.14,-90.47C88.07,-77.07 69.64,-58.15 55.16,-43.31\\\"/>\\n\",\n       \"<polygon fill=\\\"black\\\" stroke=\\\"black\\\" points=\\\"57.49,-40.69 48.01,-35.97 52.48,-45.57 57.49,-40.69\\\"/>\\n\",\n       \"<text text-anchor=\\\"middle\\\" x=\\\"92.03\\\" y=\\\"-59.98\\\" font-family=\\\"Times,serif\\\" font-size=\\\"14.00\\\">arg1</text>\\n\",\n       \"</g>\\n\",\n       \"<!-- 2 -->\\n\",\n       \"<g id=\\\"node5\\\" class=\\\"node\\\">\\n\",\n       \"<title>2</title>\\n\",\n       \"<ellipse fill=\\\"none\\\" stroke=\\\"black\\\" cx=\\\"117.53\\\" cy=\\\"-19.09\\\" rx=\\\"32.55\\\" ry=\\\"19.18\\\"/>\\n\",\n       \"<text text-anchor=\\\"start\\\" x=\\\"104.53\\\" y=\\\"-14.89\\\" font-family=\\\"Times,serif\\\" font-size=\\\"14.00\\\">女孩</text>\\n\",\n       \"</g>\\n\",\n       \"<!-- 3&#45;&gt;2 -->\\n\",\n       \"<g id=\\\"edge5\\\" class=\\\"edge\\\">\\n\",\n       \"<title>3-&gt;2</title>\\n\",\n       \"<path fill=\\\"none\\\" stroke=\\\"black\\\" d=\\\"M117.53,-88.79C117.53,-77.03 117.53,-61.59 117.53,-48.32\\\"/>\\n\",\n       \"<polygon fill=\\\"black\\\" stroke=\\\"black\\\" points=\\\"121.03,-48.26 117.53,-38.26 114.03,-48.26 121.03,-48.26\\\"/>\\n\",\n       \"<text text-anchor=\\\"middle\\\" x=\\\"130.03\\\" y=\\\"-59.98\\\" font-family=\\\"Times,serif\\\" font-size=\\\"14.00\\\">arg0</text>\\n\",\n       \"</g>\\n\",\n       \"</g>\\n\",\n       \"</svg>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.SVG object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"from IPython.display import SVG, display\\n\",\n    \"\\n\",\n    \"def show_svg(g):\\n\",\n    \"    display(SVG(data=g['svg']))\\n\",\n    \"    \\n\",\n    \"graph = HanLP.abstract_meaning_representation('男孩希望女孩相信他。', visualization='svg')[0]\\n\",\n    \"show_svg(graph)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### 多语种支持\\n\",\n    \"除了中文外，支持的语言列表：\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"#### 英文\\n\",\n    \"目前，HanLP服务器还支持英文AMR：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"image/svg+xml\": [\n       \"<svg xmlns=\\\"http://www.w3.org/2000/svg\\\" xmlns:xlink=\\\"http://www.w3.org/1999/xlink\\\" width=\\\"192pt\\\" height=\\\"298pt\\\" viewBox=\\\"0.00 0.00 191.82 297.55\\\">\\n\",\n       \"<g id=\\\"graph0\\\" class=\\\"graph\\\" transform=\\\"scale(1 1) rotate(0) translate(4 293.55)\\\">\\n\",\n       \"<title>0</title>\\n\",\n       \"<polygon fill=\\\"white\\\" stroke=\\\"transparent\\\" points=\\\"-4,4 -4,-293.55 187.82,-293.55 187.82,4 -4,4\\\"/>\\n\",\n       \"<!-- top -->\\n\",\n       \"<!-- 1 -->\\n\",\n       \"<g id=\\\"node2\\\" class=\\\"node\\\">\\n\",\n       \"<title>1</title>\\n\",\n       \"<ellipse fill=\\\"none\\\" stroke=\\\"black\\\" cx=\\\"46.67\\\" cy=\\\"-197.46\\\" rx=\\\"46.84\\\" ry=\\\"19.18\\\"/>\\n\",\n       \"<text text-anchor=\\\"start\\\" x=\\\"24.17\\\" y=\\\"-193.26\\\" font-family=\\\"Times,serif\\\" font-size=\\\"14.00\\\">want-01</text>\\n\",\n       \"</g>\\n\",\n       \"<!-- top&#45;&gt;1 -->\\n\",\n       \"<g id=\\\"edge1\\\" class=\\\"edge\\\">\\n\",\n       \"<title>top-&gt;1</title>\\n\",\n       \"<path fill=\\\"none\\\" stroke=\\\"black\\\" d=\\\"M46.67,-253.47C46.67,-245.51 46.67,-235.82 46.67,-226.81\\\"/>\\n\",\n       \"<polygon fill=\\\"black\\\" stroke=\\\"black\\\" points=\\\"50.17,-226.68 46.67,-216.68 43.17,-226.68 50.17,-226.68\\\"/>\\n\",\n       \"</g>\\n\",\n       \"<!-- 0 -->\\n\",\n       \"<g id=\\\"node3\\\" class=\\\"node\\\">\\n\",\n       \"<title>0</title>\\n\",\n       \"<ellipse fill=\\\"none\\\" stroke=\\\"black\\\" cx=\\\"41.67\\\" cy=\\\"-19.09\\\" rx=\\\"29.9\\\" ry=\\\"19.18\\\"/>\\n\",\n       \"<text text-anchor=\\\"start\\\" x=\\\"31.17\\\" y=\\\"-14.89\\\" font-family=\\\"Times,serif\\\" font-size=\\\"14.00\\\">boy</text>\\n\",\n       \"</g>\\n\",\n       \"<!-- 1&#45;&gt;0 -->\\n\",\n       \"<g id=\\\"edge2\\\" class=\\\"edge\\\">\\n\",\n       \"<title>1-&gt;0</title>\\n\",\n       \"<path fill=\\\"none\\\" stroke=\\\"black\\\" d=\\\"M44.32,-178.31C42.65,-164.4 40.55,-144.71 39.67,-127.37 38.32,-100.63 39.11,-70.1 40.07,-48.47\\\"/>\\n\",\n       \"<polygon fill=\\\"black\\\" stroke=\\\"black\\\" points=\\\"43.58,-48.48 40.56,-38.33 36.58,-48.15 43.58,-48.48\\\"/>\\n\",\n       \"<text text-anchor=\\\"middle\\\" x=\\\"52.17\\\" y=\\\"-104.58\\\" font-family=\\\"Times,serif\\\" font-size=\\\"14.00\\\">arg0</text>\\n\",\n       \"</g>\\n\",\n       \"<!-- 3 -->\\n\",\n       \"<g id=\\\"node4\\\" class=\\\"node\\\">\\n\",\n       \"<title>3</title>\\n\",\n       \"<ellipse fill=\\\"none\\\" stroke=\\\"black\\\" cx=\\\"128.67\\\" cy=\\\"-108.28\\\" rx=\\\"55.31\\\" ry=\\\"19.18\\\"/>\\n\",\n       \"<text text-anchor=\\\"start\\\" x=\\\"99.67\\\" y=\\\"-104.08\\\" font-family=\\\"Times,serif\\\" font-size=\\\"14.00\\\">believe-01</text>\\n\",\n       \"</g>\\n\",\n       \"<!-- 1&#45;&gt;3 -->\\n\",\n       \"<g id=\\\"edge3\\\" class=\\\"edge\\\">\\n\",\n       \"<title>1-&gt;3</title>\\n\",\n       \"<path fill=\\\"none\\\" stroke=\\\"black\\\" d=\\\"M62.87,-179.23C74.92,-166.42 91.55,-148.74 105.08,-134.36\\\"/>\\n\",\n       \"<polygon fill=\\\"black\\\" stroke=\\\"black\\\" points=\\\"107.92,-136.45 112.22,-126.76 102.82,-131.65 107.92,-136.45\\\"/>\\n\",\n       \"<text text-anchor=\\\"middle\\\" x=\\\"105.17\\\" y=\\\"-149.17\\\" font-family=\\\"Times,serif\\\" font-size=\\\"14.00\\\">arg1</text>\\n\",\n       \"</g>\\n\",\n       \"<!-- 3&#45;&gt;0 -->\\n\",\n       \"<g id=\\\"edge4\\\" class=\\\"edge\\\">\\n\",\n       \"<title>3-&gt;0</title>\\n\",\n       \"<path fill=\\\"none\\\" stroke=\\\"black\\\" d=\\\"M111.48,-90.05C97.99,-76.53 79.08,-57.58 64.35,-42.82\\\"/>\\n\",\n       \"<polygon fill=\\\"black\\\" stroke=\\\"black\\\" points=\\\"66.61,-40.13 57.07,-35.53 61.66,-45.08 66.61,-40.13\\\"/>\\n\",\n       \"<text text-anchor=\\\"middle\\\" x=\\\"103.17\\\" y=\\\"-59.98\\\" font-family=\\\"Times,serif\\\" font-size=\\\"14.00\\\">arg1</text>\\n\",\n       \"</g>\\n\",\n       \"<!-- 2 -->\\n\",\n       \"<g id=\\\"node5\\\" class=\\\"node\\\">\\n\",\n       \"<title>2</title>\\n\",\n       \"<ellipse fill=\\\"none\\\" stroke=\\\"black\\\" cx=\\\"128.67\\\" cy=\\\"-19.09\\\" rx=\\\"28.07\\\" ry=\\\"19.18\\\"/>\\n\",\n       \"<text text-anchor=\\\"start\\\" x=\\\"119.17\\\" y=\\\"-14.89\\\" font-family=\\\"Times,serif\\\" font-size=\\\"14.00\\\">girl</text>\\n\",\n       \"</g>\\n\",\n       \"<!-- 3&#45;&gt;2 -->\\n\",\n       \"<g id=\\\"edge5\\\" class=\\\"edge\\\">\\n\",\n       \"<title>3-&gt;2</title>\\n\",\n       \"<path fill=\\\"none\\\" stroke=\\\"black\\\" d=\\\"M128.67,-88.79C128.67,-77.03 128.67,-61.59 128.67,-48.32\\\"/>\\n\",\n       \"<polygon fill=\\\"black\\\" stroke=\\\"black\\\" points=\\\"132.17,-48.26 128.67,-38.26 125.17,-48.26 132.17,-48.26\\\"/>\\n\",\n       \"<text text-anchor=\\\"middle\\\" x=\\\"141.17\\\" y=\\\"-59.98\\\" font-family=\\\"Times,serif\\\" font-size=\\\"14.00\\\">arg0</text>\\n\",\n       \"</g>\\n\",\n       \"</g>\\n\",\n       \"</svg>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.SVG object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"graph = HanLP.abstract_meaning_representation('The boy wants the girl to believe him.',\\n\",\n    \"                                      language='en', visualization='svg')[0]\\n\",\n    \"show_svg(graph)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"用户可以通过指定`language`参数来实现英文抽象意义表示的分析：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'id': '0',\\n\",\n       \" 'input': 'The boy wants the girl to believe him .',\\n\",\n       \" 'nodes': [{'id': 0, 'label': 'boy'},\\n\",\n       \"  {'id': 1, 'label': 'wants-01'},\\n\",\n       \"  {'id': 2, 'label': 'girl'},\\n\",\n       \"  {'id': 3, 'label': 'believe-01'}],\\n\",\n       \" 'edges': [{'source': 3, 'target': 0, 'label': 'arg1'},\\n\",\n       \"  {'source': 1, 'target': 3, 'label': 'arg1'},\\n\",\n       \"  {'source': 3, 'target': 2, 'label': 'arg0'},\\n\",\n       \"  {'source': 1, 'target': 0, 'label': 'arg0'}],\\n\",\n       \" 'tops': [1],\\n\",\n       \" 'framework': 'amr'}\"\n      ]\n     },\n     \"execution_count\": 6,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"HanLP.abstract_meaning_representation(tokens=[['The', 'boy', 'wants', 'the', 'girl', 'to', 'believe', 'him', '.']], \\n\",\n    \"                                            language='en')[0]\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"amr_stl.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/amr_stl.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/amr_stl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Famr_stl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp[amr] -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 加载模型\\n\",\n    \"HanLP的工作流程是先加载模型，模型的标示符存储在`hanlp.pretrained`这个包中，按照NLP任务归类。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"4M7ka0K5OMWU\",\n    \"outputId\": \"d74f0749-0587-454a-d7c9-7418d45ce534\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'AMR3_SEQ2SEQ_BART_LARGE': 'https://file.hankcs.com/hanlp/amr/amr3_seq2seq_bart_large_83.30_20220125_114450.zip',\\n\",\n       \" 'MRP2020_AMR_ENG_ZHO_XLM_BASE': 'http://download.hanlp.com/amr/extra/amr-eng-zho-xlm-roberta-base_20220412_223756.zip',\\n\",\n       \" 'MRP2020_AMR_ZHO_MENGZI_BASE': 'http://download.hanlp.com/amr/extra/amr-zho-mengzi-base_20220415_101941.zip'}\"\n      ]\n     },\n     \"execution_count\": 1,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"import hanlp\\n\",\n    \"hanlp.pretrained.amr.ALL # 语种见名称最后一个字段或相应语料库\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"BMW528wGNulM\"\n   },\n   \"source\": [\n    \"调用`hanlp.load`进行加载，模型会自动下载到本地缓存。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"0tmKBu7sNAXX\",\n    \"outputId\": \"df2de87b-27f5-4c72-8eb2-25ceefdd8270\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"amr = hanlp.load('MRP2020_AMR_ENG_ZHO_XLM_BASE')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 抽象意义表示\\n\",\n    \"抽象意义表示任务的输入为一个或多个句子，`MRP2020_AMR_ENG_ZHO_XLM_BASE`要求提供分词完毕的句子：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"936d439a-e1ff-4308-d2aa-775955558594\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"graph = amr([\\\"男孩\\\", \\\"希望\\\", \\\"女孩\\\", \\\"相信\\\", \\\"他\\\", \\\"。\\\"])\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"jj1Jk-2sPHYx\"\n   },\n   \"source\": [\n    \"返回对象为[penman.Graph](https://penman.readthedocs.io/en/latest/api/penman.graph.html)类型：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"<AMRGraph object (top=x2) at 12603529872>\"\n      ]\n     },\n     \"execution_count\": 4,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"graph\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"打印时为友好格式：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"(x2 / 希望-01\\n\",\n      \"    :arg1 (x4 / 相信-01\\n\",\n      \"              :arg0 (x3 / 女孩)\\n\",\n      \"              :arg1 x1)\\n\",\n      \"    :arg0 (x1 / 男孩))\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(graph)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"该AMR的可视化结果为：\\n\",\n    \"\\n\",\n    \"![amr-zh](https://hanlp.hankcs.com/backend/v2/amr_svg?tokens=%E7%94%B7%E5%AD%A9%20%E5%B8%8C%E6%9C%9B%20%E5%A5%B3%E5%AD%A9%20%E7%9B%B8%E4%BF%A1%20%E4%BB%96%20%E3%80%82&language=zh&scale=1)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"`MRP2020_AMR_ENG_ZHO_XLM_BASE`其实是一个Meaning Representation Parsing模型，支持输出Meaning Representation（MR）格式，该格式比AMR的表达力更强：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'id': '0',\\n\",\n       \" 'input': '男孩 希望 女孩 相信 他 。',\\n\",\n       \" 'nodes': [{'id': 0,\\n\",\n       \"   'label': '男孩',\\n\",\n       \"   'anchors': [{'from': 0, 'to': 2}, {'from': 12, 'to': 13}]},\\n\",\n       \"  {'id': 1, 'label': '希望-01', 'anchors': [{'from': 3, 'to': 5}]},\\n\",\n       \"  {'id': 2, 'label': '女孩', 'anchors': [{'from': 6, 'to': 8}]},\\n\",\n       \"  {'id': 3, 'label': '相信-01', 'anchors': [{'from': 9, 'to': 11}]}],\\n\",\n       \" 'edges': [{'source': 1, 'target': 3, 'label': 'arg1'},\\n\",\n       \"  {'source': 1, 'target': 0, 'label': 'arg0'},\\n\",\n       \"  {'source': 3, 'target': 2, 'label': 'arg0'},\\n\",\n       \"  {'source': 3, 'target': 0, 'label': 'arg1'}],\\n\",\n       \" 'tops': [1],\\n\",\n       \" 'framework': 'amr'}\"\n      ]\n     },\n     \"execution_count\": 6,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"amr([\\\"男孩\\\", \\\"希望\\\", \\\"女孩\\\", \\\"相信\\\", \\\"他\\\", \\\"。\\\"], output_amr=False)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"注意上面“男孩”有2个anchor，分别对应“男孩”和“他”。也就是说，MR格式其实包含了指代消解的结果。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## 多语种支持\\n\",\n    \"`MRP2020_AMR_ENG_ZHO_XLM_BASE`同时还是一个Cross-Lingual模型，支持的语言列表：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"[['amr', 'eng'], ['amr', 'zho']]\"\n      ]\n     },\n     \"execution_count\": 7,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"amr.config.frameworks\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"用户可以通过指定language参数来实现英文抽象意义表示的分析：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 8,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"(w1 / wants-01\\n\",\n      \"    :arg1 (b2 / believe-01\\n\",\n      \"              :arg0 (g1 / girl)\\n\",\n      \"              :arg1 b1)\\n\",\n      \"    :arg0 (b1 / boy))\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(amr(['The', 'boy', 'wants', 'the', 'girl', 'to', 'believe', 'him', '.'], language='eng'))\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"为了达到最佳效果，建议同时提供每个词的词干：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 9,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"(w1 / want-01\\n\",\n      \"    :arg1 (b2 / believe-01\\n\",\n      \"              :arg0 (g1 / girl)\\n\",\n      \"              :arg1 b1)\\n\",\n      \"    :arg0 (b1 / boy))\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(amr([('The', 'the'), ('boy', 'boy'), ('wants', 'want'), ('the', 'the'), ('girl', 'girl'), ('to', 'to'),\\n\",\n    \"              ('believe', 'believe'), ('him', 'he'), ('.', '.')], language='eng'))\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"该AMR的可视化结果为：\\n\",\n    \"\\n\",\n    \"![amr-en](https://hanlp.hankcs.com/backend/v2/amr_svg?tokens=The%20boy%20wants%20the%20girl%20to%20believe%20him%20.&language=en&scale=1)\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"amr_stl.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/classification_restful.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/classification_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fclassification_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"nf9TgeCTC0OT\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"jaW4eu6kC0OU\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp_restful -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"_xI_bLAaC0OU\"\n   },\n   \"source\": [\n    \"## 创建客户端\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"IYwV-UkNNzFp\",\n    \"outputId\": \"54065443-9b0a-444c-f6c0-c701bc86400b\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from hanlp_restful import HanLPClient\\n\",\n    \"HanLP = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh') # auth不填则匿名，zh中文，mul多语种\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\",\n    \"pycharm\": {\n     \"name\": \"#%% md\\n\"\n    }\n   },\n   \"source\": [\n    \"#### 申请秘钥\\n\",\n    \"由于服务器算力有限，匿名用户每分钟限2次调用。如果你需要更多调用次数，[建议申请免费公益API秘钥auth](https://bbs.hanlp.com/t/hanlp2-1-restful-api/53)。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 文本分类\\n\",\n    \"文本分类任务的输入为文档以及分类模型，以新闻领域的`news_zh`为例：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {\n    \"id\": \"BqEmDMGGOtk3\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"'科技'\"\n      ]\n     },\n     \"execution_count\": 4,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"HanLP.text_classification('2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', model='news_zh')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"SwaPn1hjC0OW\"\n   },\n   \"source\": [\n    \"返回值为文档最可能的类目。HanLP支持返回类目对应的概率（置信度）：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"egpWwHKxC0OX\",\n    \"outputId\": \"f7c77687-dd75-4fa2-dbd2-be6bda8a3fff\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"['科技', 0.999642014503479]\"\n      ]\n     },\n     \"execution_count\": 5,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"HanLP.text_classification('2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', model='news_zh', prob=True)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"kq_j5TLFC0OX\"\n   },\n   \"source\": [\n    \"HanLP也支持返回概率最高的`topk`个类目：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"isJhzYyIC0OX\",\n    \"outputId\": \"683c8489-dffc-426e-f95b-e91dfb373260\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"['科技', '家居']\"\n      ]\n     },\n     \"execution_count\": 7,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"HanLP.text_classification('2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', model='news_zh', topk=2)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"该功能对于混合了多个主题的文档而言特别实用：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 10,\n   \"metadata\": {\n    \"scrolled\": true\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'时尚': 0.6342714428901672,\\n\",\n       \" '家居': 0.359315425157547,\\n\",\n       \" '科技': 0.0013340614968910813,\\n\",\n       \" '体育': 0.001275017624720931,\\n\",\n       \" '房产': 0.0010209722677245736,\\n\",\n       \" '娱乐': 0.0006360886618494987,\\n\",\n       \" '财经': 0.0005668793455697596,\\n\",\n       \" '游戏': 0.00037119409535080194,\\n\",\n       \" '教育': 0.00029694309341721237,\\n\",\n       \" '股票': 0.0002858955995179713,\\n\",\n       \" '星座': 0.0002288677787873894,\\n\",\n       \" '彩票': 0.00022682634880766273,\\n\",\n       \" '时政': 0.0001005345256999135,\\n\",\n       \" '社会': 6.985480285948142e-05}\"\n      ]\n     },\n     \"execution_count\": 10,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"text = '''\\n\",\n    \"改了好几次，感觉终于可以确定了。\\n\",\n    \"这次的真丝是做了古董感的米金色染色，法蕾也做了同样的颜色。\\n\",\n    \"真丝软糯的手感和温柔的光泽感，在即将结束的冬天，显得格外的美好。\\n\",\n    \"'''\\n\",\n    \"\\n\",\n    \"HanLP.text_classification(text, model='news_zh', topk=True, prob=True)\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"classification_restful.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/con_mtl.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/con_mtl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fcon_mtl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 加载模型\\n\",\n    \"HanLP的工作流程是先加载模型，模型的标示符存储在`hanlp.pretrained`这个包中，按照NLP任务归类。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"id\": \"0tmKBu7sNAXX\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'OPEN_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/mtl/open_tok_pos_ner_srl_dep_sdp_con_electra_small_20201223_035557.zip',\\n\",\n       \" 'OPEN_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH': 'https://file.hankcs.com/hanlp/mtl/open_tok_pos_ner_srl_dep_sdp_con_electra_base_20201223_201906.zip',\\n\",\n       \" 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_small_20210111_124159.zip',\\n\",\n       \" 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_base_20210111_124519.zip',\\n\",\n       \" 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ERNIE_GRAM_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_ernie_gram_base_aug_20210904_145403.zip',\\n\",\n       \" 'UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_MT5_SMALL': 'https://file.hankcs.com/hanlp/mtl/ud_ontonotes_tok_pos_lem_fea_ner_srl_dep_sdp_con_mt5_small_20210228_123458.zip',\\n\",\n       \" 'UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_XLMR_BASE': 'https://file.hankcs.com/hanlp/mtl/ud_ontonotes_tok_pos_lem_fea_ner_srl_dep_sdp_con_xlm_base_20210602_211620.zip',\\n\",\n       \" 'NPCMJ_UD_KYOTO_TOK_POS_CON_BERT_BASE_CHAR_JA': 'https://file.hankcs.com/hanlp/mtl/npcmj_ud_kyoto_tok_pos_ner_dep_con_srl_bert_base_char_ja_20210914_133742.zip'}\"\n      ]\n     },\n     \"execution_count\": 1,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"import hanlp\\n\",\n    \"hanlp.pretrained.mtl.ALL # MTL多任务，具体任务见模型名称，语种见名称最后一个字段或相应语料库\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"EmZDmLn9aGxG\"\n   },\n   \"source\": [\n    \"调用`hanlp.load`进行加载，模型会自动下载到本地缓存。自然语言处理分为许多任务，分词只是最初级的一个。与其每个任务单独创建一个模型，不如利用HanLP的联合模型一次性完成多个任务：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"HanLP = hanlp.load(hanlp.pretrained.mtl.CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 短语句法分析\\n\",\n    \"任务越少，速度越快。如指定仅执行短语句法分析：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"2a0d392f-b99a-4a18-fc7f-754e2abe2e34\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"doc = HanLP(['2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', '阿婆主来到北京立方庭参观自然语义科技公司。'], tasks='con')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"返回值为一个[Document](https://hanlp.hankcs.com/docs/api/common/document.html):\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"{\\n\",\n      \"  \\\"tok/fine\\\": [\\n\",\n      \"    [\\\"2021年\\\", \\\"HanLPv2.1\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次\\\", \\\"世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多\\\", \\\"语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"],\\n\",\n      \"    [\\\"阿婆主\\\", \\\"来到\\\", \\\"北京\\\", \\\"立方庭\\\", \\\"参观\\\", \\\"自然\\\", \\\"语义\\\", \\\"科技\\\", \\\"公司\\\", \\\"。\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"con\\\": [\\n\",\n      \"    [\\\"TOP\\\", [[\\\"IP\\\", [[\\\"NP\\\", [[\\\"_\\\", [\\\"2021年\\\"]]]], [\\\"NP\\\", [[\\\"_\\\", [\\\"HanLPv2.1\\\"]]]], [\\\"VP\\\", [[\\\"PP\\\", [[\\\"_\\\", [\\\"为\\\"]], [\\\"NP\\\", [[\\\"_\\\", [\\\"生产\\\"]], [\\\"_\\\", [\\\"环境\\\"]]]]]], [\\\"VP\\\", [[\\\"_\\\", [\\\"带来\\\"]], [\\\"NP\\\", [[\\\"ADJP\\\", [[\\\"NP\\\", [[\\\"ADJP\\\", [[\\\"_\\\", [\\\"次\\\"]]]], [\\\"NP\\\", [[\\\"_\\\", [\\\"世代\\\"]]]]]], [\\\"ADVP\\\", [[\\\"_\\\", [\\\"最\\\"]]]], [\\\"VP\\\", [[\\\"_\\\", [\\\"先进\\\"]]]]]], [\\\"_\\\", [\\\"的\\\"]], [\\\"NP\\\", [[\\\"QP\\\", [[\\\"_\\\", [\\\"多\\\"]]]], [\\\"NP\\\", [[\\\"_\\\", [\\\"语种\\\"]]]]]], [\\\"NP\\\", [[\\\"_\\\", [\\\"NLP\\\"]], [\\\"_\\\", [\\\"技术\\\"]]]]]]]]]], [\\\"_\\\", [\\\"。\\\"]]]]]],\\n\",\n      \"    [\\\"TOP\\\", [[\\\"IP\\\", [[\\\"NP\\\", [[\\\"_\\\", [\\\"阿婆主\\\"]]]], [\\\"VP\\\", [[\\\"VP\\\", [[\\\"_\\\", [\\\"来到\\\"]], [\\\"NP\\\", [[\\\"_\\\", [\\\"北京\\\"]], [\\\"_\\\", [\\\"立方庭\\\"]]]]]], [\\\"VP\\\", [[\\\"_\\\", [\\\"参观\\\"]], [\\\"NP\\\", [[\\\"_\\\", [\\\"自然\\\"]], [\\\"_\\\", [\\\"语义\\\"]], [\\\"_\\\", [\\\"科技\\\"]], [\\\"_\\\", [\\\"公司\\\"]]]]]]]], [\\\"_\\\", [\\\"。\\\"]]]]]]\\n\",\n      \"  ]\\n\",\n      \"}\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(doc)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"`doc['con']`为Tree类型，是list的子类。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"wxctCigrTKu-\"\n   },\n   \"source\": [\n    \"可视化短语句法树：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"Zo08uquCTFSk\",\n    \"outputId\": \"c6077f2d-7084-4f4b-a3bc-9aa9951704ea\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Token&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>为&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>生产&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>环境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>带来&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先进&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>的&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>语种&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技术&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">P&nbsp;&nbsp;&nbsp;&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;9&nbsp;<br>───────────────────────────────────────────────────────<br>_───────────────────────────────────────────►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;<br>_───────────────────────────────────────────►NP────┤&nbsp;&nbsp;&nbsp;<br>_──────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├────────────────────────►PP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──┴►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──────────────────────────────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───►ADJP──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP────┤&nbsp;&nbsp;&nbsp;<br>_───►NP&nbsp;───┴►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───────────►ADVP──┼►ADJP──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►IP<br>_───────────►VP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──────────────────────────┤&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───►QP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───►NP&nbsp;───┴────────►NP────┤&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──┴────────────────►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──────────────────────────────────────────────────┘&nbsp;&nbsp;&nbsp;</pre></div><br><div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;<br>───&nbsp;<br>阿婆主&nbsp;<br>来到&nbsp;&nbsp;<br>北京&nbsp;&nbsp;<br>立方庭&nbsp;<br>参观&nbsp;&nbsp;<br>自然&nbsp;&nbsp;<br>语义&nbsp;&nbsp;<br>科技&nbsp;&nbsp;<br>公司&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">P&nbsp;&nbsp;&nbsp;&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6&nbsp;<br>───────────────────────────────<br>_───────────────────►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;<br>_──────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──┴►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP────┤&nbsp;&nbsp;&nbsp;<br>_──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►IP<br>_&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_&nbsp;&nbsp;├►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──────────────────────────┘&nbsp;&nbsp;&nbsp;</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"doc.pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"将第一个短语树转换为bracketed格式：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"(TOP\\n\",\n      \"  (IP\\n\",\n      \"    (NP (_ 2021年))\\n\",\n      \"    (NP (_ HanLPv2.1))\\n\",\n      \"    (VP\\n\",\n      \"      (PP (_ 为) (NP (_ 生产) (_ 环境)))\\n\",\n      \"      (VP\\n\",\n      \"        (_ 带来)\\n\",\n      \"        (NP\\n\",\n      \"          (ADJP\\n\",\n      \"            (NP (ADJP (_ 次)) (NP (_ 世代)))\\n\",\n      \"            (ADVP (_ 最))\\n\",\n      \"            (VP (_ 先进)))\\n\",\n      \"          (_ 的)\\n\",\n      \"          (NP (QP (_ 多)) (NP (_ 语种)))\\n\",\n      \"          (NP (_ NLP) (_ 技术)))))\\n\",\n      \"    (_ 。)))\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(doc['con'][0])\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"将第一个短语树转换为list格式：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"['TOP',\\n\",\n       \" [['IP',\\n\",\n       \"   [['NP', [['_', ['2021年']]]],\\n\",\n       \"    ['NP', [['_', ['HanLPv2.1']]]],\\n\",\n       \"    ['VP',\\n\",\n       \"     [['PP', [['_', ['为']], ['NP', [['_', ['生产']], ['_', ['环境']]]]]],\\n\",\n       \"      ['VP',\\n\",\n       \"       [['_', ['带来']],\\n\",\n       \"        ['NP',\\n\",\n       \"         [['ADJP',\\n\",\n       \"           [['NP', [['ADJP', [['_', ['次']]]], ['NP', [['_', ['世代']]]]]],\\n\",\n       \"            ['ADVP', [['_', ['最']]]],\\n\",\n       \"            ['VP', [['_', ['先进']]]]]],\\n\",\n       \"          ['_', ['的']],\\n\",\n       \"          ['NP', [['QP', [['_', ['多']]]], ['NP', [['_', ['语种']]]]]],\\n\",\n       \"          ['NP', [['_', ['NLP']], ['_', ['技术']]]]]]]]]],\\n\",\n       \"    ['_', ['。']]]]]]\"\n      ]\n     },\n     \"execution_count\": 7,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"doc['con'][0].to_list()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"XOsWkOqQfzlr\"\n   },\n   \"source\": [\n    \"为已分词的句子执行短语句法分析：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 8,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"bLZSTbv_f3OA\",\n    \"outputId\": \"111c0be9-bac6-4eee-d5bd-a972ffc34844\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Token&nbsp;<br>─────&nbsp;<br>hanlp&nbsp;<br>为&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>生产&nbsp;&nbsp;&nbsp;&nbsp;<br>环境&nbsp;&nbsp;&nbsp;&nbsp;<br>带来&nbsp;&nbsp;&nbsp;&nbsp;<br>次世代&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先进&nbsp;&nbsp;&nbsp;&nbsp;<br>的&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多语种&nbsp;&nbsp;&nbsp;<br>nlp&nbsp;&nbsp;&nbsp;<br>技术&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">P&nbsp;&nbsp;&nbsp;&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;9&nbsp;<br>───────────────────────────────────────────────────────<br>_───────────────────────────────────────────►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;<br>_──────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├────────────────────────►PP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──┴►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──────────────────────────────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP────┤&nbsp;&nbsp;&nbsp;<br>_───►ADVP──┼►VP&nbsp;────►IP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►IP<br>_───►VP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──────────────────────────┤&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───────────────────►NP────┼►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───────────────────►NP────┤&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───────────────────►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──────────────────────────────────────────────────┘&nbsp;&nbsp;&nbsp;</pre></div><br><div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;<br>───&nbsp;<br>我&nbsp;&nbsp;&nbsp;<br>的&nbsp;&nbsp;&nbsp;<br>希望&nbsp;&nbsp;<br>是&nbsp;&nbsp;&nbsp;<br>希望&nbsp;&nbsp;<br>张晚霞&nbsp;<br>的&nbsp;&nbsp;&nbsp;<br>背影&nbsp;&nbsp;<br>被&nbsp;&nbsp;&nbsp;<br>晚霞&nbsp;&nbsp;<br>映红&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">P&nbsp;&nbsp;&nbsp;&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;9&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;10&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;11<br>───────────────────────────────────────────────────────────────────────<br>_───►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>_──────────┴►DNP&nbsp;──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>_───────────►NP&nbsp;───┴────────────────────────────────────────►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;<br>_──────────────────────────────────────────────────────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──────────────────────────────────────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP────┤&nbsp;&nbsp;&nbsp;<br>_──────────┴►DNP&nbsp;──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP&nbsp;────►IP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───────────►NP&nbsp;───┴────────►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►IP<br>_──────────────────────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►IP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───►VP&nbsp;───┴►IP&nbsp;────►CP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──────────────────────────────────────────────────────────────────┘&nbsp;&nbsp;&nbsp;</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"HanLP([\\n\",\n    \"    [\\\"HanLP\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"],\\n\",\n    \"    [\\\"我\\\", \\\"的\\\", \\\"希望\\\", \\\"是\\\", \\\"希望\\\", \\\"张晚霞\\\", \\\"的\\\", \\\"背影\\\", \\\"被\\\", \\\"晚霞\\\", \\\"映红\\\", \\\"。\\\"]\\n\",\n    \"  ], tasks='con', skip_tasks='tok*').pretty_print()\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"con_mtl.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/con_restful.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/con_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fcon_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp_restful -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 创建客户端\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"id\": \"0tmKBu7sNAXX\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from hanlp_restful import HanLPClient\\n\",\n    \"HanLP = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh') # auth不填则匿名，zh中文，mul多语种\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"EmZDmLn9aGxG\"\n   },\n   \"source\": [\n    \"#### 申请秘钥\\n\",\n    \"由于服务器算力有限，匿名用户每分钟限2次调用。如果你需要更多调用次数，[建议申请免费公益API秘钥auth](https://bbs.hanlp.com/t/hanlp2-1-restful-api/53)。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 短语句法分析\\n\",\n    \"任务越少，速度越快。如指定仅执行短语句法分析：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"2a0d392f-b99a-4a18-fc7f-754e2abe2e34\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"doc = HanLP('2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', tasks='con')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"返回值为一个[Document](https://hanlp.hankcs.com/docs/api/common/document.html):\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"{\\n\",\n      \"  \\\"tok/fine\\\": [\\n\",\n      \"    [\\\"2021年\\\", \\\"HanLPv2.1\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次\\\", \\\"世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多\\\", \\\"语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"con\\\": [\\n\",\n      \"    [\\\"TOP\\\", [[\\\"IP\\\", [[\\\"NP\\\", [[\\\"_\\\", [\\\"2021年\\\"]]]], [\\\"NP\\\", [[\\\"_\\\", [\\\"HanLPv2.1\\\"]]]], [\\\"VP\\\", [[\\\"PP\\\", [[\\\"_\\\", [\\\"为\\\"]], [\\\"NP\\\", [[\\\"_\\\", [\\\"生产\\\"]], [\\\"_\\\", [\\\"环境\\\"]]]]]], [\\\"VP\\\", [[\\\"_\\\", [\\\"带来\\\"]], [\\\"NP\\\", [[\\\"IP\\\", [[\\\"VP\\\", [[\\\"NP\\\", [[\\\"QP\\\", [[\\\"CLP\\\", [[\\\"_\\\", [\\\"次\\\"]]]]]], [\\\"NP\\\", [[\\\"_\\\", [\\\"世代\\\"]]]]]], [\\\"ADVP\\\", [[\\\"_\\\", [\\\"最\\\"]]]], [\\\"VP\\\", [[\\\"_\\\", [\\\"先进\\\"]]]]]]]], [\\\"_\\\", [\\\"的\\\"]], [\\\"NP\\\", [[\\\"QP\\\", [[\\\"_\\\", [\\\"多\\\"]]]], [\\\"NP\\\", [[\\\"_\\\", [\\\"语种\\\"]]]]]], [\\\"NP\\\", [[\\\"_\\\", [\\\"NLP\\\"]], [\\\"_\\\", [\\\"技术\\\"]]]]]]]]]], [\\\"_\\\", [\\\"。\\\"]]]]]]\\n\",\n      \"  ]\\n\",\n      \"}\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(doc)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"`doc['con']`为Tree类型，是list的子类。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"wxctCigrTKu-\"\n   },\n   \"source\": [\n    \"可视化短语句法树：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"Zo08uquCTFSk\",\n    \"outputId\": \"c6077f2d-7084-4f4b-a3bc-9aa9951704ea\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Token&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>为&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>生产&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>环境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>带来&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先进&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>的&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>语种&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技术&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">P&nbsp;&nbsp;&nbsp;&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;9&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;10&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;11<br>───────────────────────────────────────────────────────────────────────<br>_───────────────────────────────────────────────────────────►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;<br>_───────────────────────────────────────────────────────────►NP────┤&nbsp;&nbsp;&nbsp;<br>_──────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├────────────────────────────────────────►PP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──┴►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──────────────────────────────────────────────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───►CLP&nbsp;───►QP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP────┤&nbsp;&nbsp;&nbsp;<br>_───────────►NP&nbsp;───┴►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───────────────────►ADVP──┼►VP&nbsp;────►IP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►IP<br>_───────────────────►VP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──────────────────────────────────────────┤&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───►QP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───►NP&nbsp;───┴────────────────────────►NP────┤&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──┴────────────────────────────────►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──────────────────────────────────────────────────────────────────┘&nbsp;&nbsp;&nbsp;</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"doc.pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"转换为bracketed格式：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"(TOP\\n\",\n      \"  (IP\\n\",\n      \"    (NP (_ 2021年))\\n\",\n      \"    (NP (_ HanLPv2.1))\\n\",\n      \"    (VP\\n\",\n      \"      (PP (_ 为) (NP (_ 生产) (_ 环境)))\\n\",\n      \"      (VP\\n\",\n      \"        (_ 带来)\\n\",\n      \"        (NP\\n\",\n      \"          (IP\\n\",\n      \"            (VP\\n\",\n      \"              (NP (QP (CLP (_ 次))) (NP (_ 世代)))\\n\",\n      \"              (ADVP (_ 最))\\n\",\n      \"              (VP (_ 先进))))\\n\",\n      \"          (_ 的)\\n\",\n      \"          (NP (QP (_ 多)) (NP (_ 语种)))\\n\",\n      \"          (NP (_ NLP) (_ 技术)))))\\n\",\n      \"    (_ 。)))\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(doc['con'][0])\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"XOsWkOqQfzlr\"\n   },\n   \"source\": [\n    \"为已分词的句子执行短语句法分析：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"bLZSTbv_f3OA\",\n    \"outputId\": \"111c0be9-bac6-4eee-d5bd-a972ffc34844\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Token&nbsp;<br>─────&nbsp;<br>hanlp&nbsp;<br>为&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>生产&nbsp;&nbsp;&nbsp;&nbsp;<br>环境&nbsp;&nbsp;&nbsp;&nbsp;<br>带来&nbsp;&nbsp;&nbsp;&nbsp;<br>次世代&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先进&nbsp;&nbsp;&nbsp;&nbsp;<br>的&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多语种&nbsp;&nbsp;&nbsp;<br>nlp&nbsp;&nbsp;&nbsp;<br>技术&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">P&nbsp;&nbsp;&nbsp;&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;9&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;10&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;11&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;12<br>───────────────────────────────────────────────────────────────────────────────<br>_───────────────────────────────────────────────────────────────────►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;<br>_──────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├────────────────────────────────────────────────►PP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──┴►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──────────────────────────────────────────────────────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───────────►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP────┤&nbsp;&nbsp;&nbsp;<br>_───►ADVP──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP&nbsp;────►IP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►IP<br>_───►VP&nbsp;───┴►VP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►CP&nbsp;────►CP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──────────────────────────────────┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──────────────────────────────────────────────────┼►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───►NP&nbsp;───┴────────────────────────────────►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──────────────────────────────────────────────────────────────────────────┘&nbsp;&nbsp;&nbsp;</pre></div><br><div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;<br>───&nbsp;<br>我&nbsp;&nbsp;&nbsp;<br>的&nbsp;&nbsp;&nbsp;<br>希望&nbsp;&nbsp;<br>是&nbsp;&nbsp;&nbsp;<br>希望&nbsp;&nbsp;<br>张晚霞&nbsp;<br>的&nbsp;&nbsp;&nbsp;<br>背影&nbsp;&nbsp;<br>被&nbsp;&nbsp;&nbsp;<br>晚霞&nbsp;&nbsp;<br>映红&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">P&nbsp;&nbsp;&nbsp;&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;9&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;10&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;11<br>───────────────────────────────────────────────────────────────────────<br>_───►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>_──────────┴►DNP&nbsp;──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>_───────────►NP&nbsp;───┴────────────────────────────────────────►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;<br>_──────────────────────────────────────────────────────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──────────────────────────────────────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP────┤&nbsp;&nbsp;&nbsp;<br>_──────────┴►DNP&nbsp;──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP&nbsp;────►IP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───────────►NP&nbsp;───┴────────►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►IP<br>_──────────────────────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►IP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_───►VP&nbsp;───┴►IP&nbsp;────►CP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>_──────────────────────────────────────────────────────────────────┘&nbsp;&nbsp;&nbsp;</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"HanLP(tokens=[\\n\",\n    \"    [\\\"HanLP\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"],\\n\",\n    \"    [\\\"我\\\", \\\"的\\\", \\\"希望\\\", \\\"是\\\", \\\"希望\\\", \\\"张晚霞\\\", \\\"的\\\", \\\"背影\\\", \\\"被\\\", \\\"晚霞\\\", \\\"映红\\\", \\\"。\\\"]\\n\",\n    \"  ], tasks='con').pretty_print()\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"con_restful.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/con_stl.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/con_stl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fcon_stl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 加载模型\\n\",\n    \"HanLP的工作流程是先加载模型，模型的标示符存储在`hanlp.pretrained`这个包中，按照NLP任务归类。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"id\": \"0tmKBu7sNAXX\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'CTB9_CON_ELECTRA_SMALL': 'https://file.hankcs.com/hanlp/constituency/ctb9_con_electra_small_20220215_230116.zip',\\n\",\n       \" 'CTB9_CON_FULL_TAG_ELECTRA_SMALL': 'https://file.hankcs.com/hanlp/constituency/ctb9_full_tag_con_electra_small_20220118_103119.zip'}\"\n      ]\n     },\n     \"execution_count\": 1,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"import hanlp\\n\",\n    \"hanlp.pretrained.constituency.ALL # 语种见名称最后一个字段或相应语料库\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"EmZDmLn9aGxG\"\n   },\n   \"source\": [\n    \"调用`hanlp.load`进行加载，模型会自动下载到本地缓存。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"con = hanlp.load('CTB9_CON_FULL_TAG_ELECTRA_SMALL')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 短语句法分析\\n\",\n    \"输入为已分词的一个或多个句子：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"2a0d392f-b99a-4a18-fc7f-754e2abe2e34\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"trees = con([[\\\"2021年\\\", \\\"HanLPv2.1\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次\\\", \\\"世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多\\\", \\\"语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"], [\\\"阿婆主\\\", \\\"来到\\\", \\\"北京\\\", \\\"立方庭\\\", \\\"参观\\\", \\\"自然\\\", \\\"语义\\\", \\\"科技\\\", \\\"公司\\\", \\\"。\\\"]], tasks='con')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"返回值为一个`Tree`的数组:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"[['TOP', [['IP', [['NP-TMP', [['_', ['2021年']]]], ['NP-PN-SBJ', [['_', ['HanLPv2.1']]]], ['VP', [['PP-BNF', [['_', ['为']], ['NP', [['_', ['生产']], ['_', ['环境']]]]]], ['VP', [['_', ['带来']], ['NP-OBJ', [['CP', [['CP', [['IP', [['VP', [['NP', [['DP', [['_', ['次']]]], ['NP', [['_', ['世代']]]]]], ['ADVP', [['_', ['最']]]], ['VP', [['_', ['先进']]]]]]]], ['_', ['的']]]]]], ['NP', [['QP', [['_', ['多']]]], ['NP', [['_', ['语种']]]]]], ['NP', [['_', ['NLP']], ['_', ['技术']]]]]]]]]], ['_', ['。']]]]]], ['TOP', [['IP', [['NP-SBJ', [['_', ['阿婆主']]]], ['VP', [['VP', [['_', ['来到']], ['NP-OBJ', [['_', ['北京']], ['NP-PN', [['_', ['立方庭']]]]]]]], ['VP', [['_', ['参观']], ['NP-OBJ', [['_', ['自然']], ['_', ['语义']], ['_', ['科技']], ['_', ['公司']]]]]]]], ['_', ['。']]]]]]]\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(trees)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"转换为bracketed格式：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"(TOP\\n\",\n      \"  (IP\\n\",\n      \"    (NP-TMP (_ 2021年))\\n\",\n      \"    (NP-PN-SBJ (_ HanLPv2.1))\\n\",\n      \"    (VP\\n\",\n      \"      (PP-BNF (_ 为) (NP (_ 生产) (_ 环境)))\\n\",\n      \"      (VP\\n\",\n      \"        (_ 带来)\\n\",\n      \"        (NP-OBJ\\n\",\n      \"          (CP\\n\",\n      \"            (CP\\n\",\n      \"              (IP\\n\",\n      \"                (VP\\n\",\n      \"                  (NP (DP (_ 次)) (NP (_ 世代)))\\n\",\n      \"                  (ADVP (_ 最))\\n\",\n      \"                  (VP (_ 先进))))\\n\",\n      \"              (_ 的)))\\n\",\n      \"          (NP (QP (_ 多)) (NP (_ 语种)))\\n\",\n      \"          (NP (_ NLP) (_ 技术)))))\\n\",\n      \"    (_ 。)))\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(trees[0])\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## 组装流水线\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"短语成分树的第一层non-terminal一般是词性标签，所以经常与词性标注一起使用。为此，先加载一个词性标注器：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"pos = hanlp.load(hanlp.pretrained.pos.CTB9_POS_ELECTRA_SMALL)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"然后创建一个函数将词性标签和句法树组装起来:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"from hanlp_common.document import Document\\n\",\n    \"def merge_pos_into_con(doc:Document):\\n\",\n    \"    flat = isinstance(doc['pos'][0], str)\\n\",\n    \"    if flat:\\n\",\n    \"        doc = Document((k, [v]) for k, v in doc.items())\\n\",\n    \"    for tree, tags in zip(doc['con'], doc['pos']):\\n\",\n    \"        offset = 0\\n\",\n    \"        for subtree in tree.subtrees(lambda t: t.height() == 2):\\n\",\n    \"            tag = subtree.label()\\n\",\n    \"            if tag == '_':\\n\",\n    \"                subtree.set_label(tags[offset])\\n\",\n    \"            offset += 1\\n\",\n    \"    if flat:\\n\",\n    \"        doc = doc.squeeze()\\n\",\n    \"    return doc\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"之后就可以用一个流水线将三者组装起来了：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 8,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"nlp = hanlp.pipeline() \\\\\\n\",\n    \"    .append(pos, input_key='tok', output_key='pos') \\\\\\n\",\n    \"    .append(con, input_key='tok', output_key='con') \\\\\\n\",\n    \"    .append(merge_pos_into_con, input_key='*')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"该流水线的结构如下：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 9,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"[tok->TransformerTagger->pos, tok->CRFConstituencyParser->con, None->merge_pos_into_con->None]\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(nlp)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"传入一个已分词的句子试试：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 10,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"{\\n\",\n      \"  \\\"tok\\\": [\\n\",\n      \"    \\\"2021年\\\",\\n\",\n      \"    \\\"HanLPv2.1\\\",\\n\",\n      \"    \\\"带来\\\",\\n\",\n      \"    \\\"最\\\",\\n\",\n      \"    \\\"先进\\\",\\n\",\n      \"    \\\"的\\\",\\n\",\n      \"    \\\"多\\\",\\n\",\n      \"    \\\"语种\\\",\\n\",\n      \"    \\\"NLP\\\",\\n\",\n      \"    \\\"技术\\\",\\n\",\n      \"    \\\"。\\\"\\n\",\n      \"  ],\\n\",\n      \"  \\\"pos\\\": [\\n\",\n      \"    \\\"NT\\\",\\n\",\n      \"    \\\"NR\\\",\\n\",\n      \"    \\\"VV\\\",\\n\",\n      \"    \\\"AD\\\",\\n\",\n      \"    \\\"VA\\\",\\n\",\n      \"    \\\"DEC\\\",\\n\",\n      \"    \\\"CD\\\",\\n\",\n      \"    \\\"NN\\\",\\n\",\n      \"    \\\"NR\\\",\\n\",\n      \"    \\\"NN\\\",\\n\",\n      \"    \\\"PU\\\"\\n\",\n      \"  ],\\n\",\n      \"  \\\"con\\\": [\\n\",\n      \"    \\\"TOP\\\",\\n\",\n      \"    [[\\\"IP\\\", [[\\\"NP-TMP\\\", [[\\\"NT\\\", [\\\"2021年\\\"]]]], [\\\"NP-PN-SBJ\\\", [[\\\"NR\\\", [\\\"HanLPv2.1\\\"]]]], [\\\"VP\\\", [[\\\"VV\\\", [\\\"带来\\\"]], [\\\"NP-OBJ\\\", [[\\\"CP\\\", [[\\\"CP\\\", [[\\\"IP\\\", [[\\\"VP\\\", [[\\\"ADVP\\\", [[\\\"AD\\\", [\\\"最\\\"]]]], [\\\"VP\\\", [[\\\"VA\\\", [\\\"先进\\\"]]]]]]]], [\\\"DEC\\\", [\\\"的\\\"]]]]]], [\\\"NP\\\", [[\\\"QP\\\", [[\\\"CD\\\", [\\\"多\\\"]]]], [\\\"NP\\\", [[\\\"NN\\\", [\\\"语种\\\"]]]]]], [\\\"NP\\\", [[\\\"NR\\\", [\\\"NLP\\\"]], [\\\"NN\\\", [\\\"技术\\\"]]]]]]]], [\\\"PU\\\", [\\\"。\\\"]]]]]\\n\",\n      \"  ]\\n\",\n      \"}\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"doc = nlp(tok=[\\\"2021年\\\", \\\"HanLPv2.1\\\", \\\"带来\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多\\\", \\\"语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"])\\n\",\n    \"print(doc)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"流水线的输出也是一个Document，所以支持可视化：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 11,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Token&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>带来&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先进&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>的&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>语种&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技术&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">PoS&nbsp;&nbsp;&nbsp;&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;9&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;10<br>────────────────────────────────────────────────────────────────────────<br>NT&nbsp;─────────────────────────────────────────────────────►NP-TMP&nbsp;────┐&nbsp;&nbsp;&nbsp;<br>NR&nbsp;─────────────────────────────────────────────────────►NP-PN-SBJ──┤&nbsp;&nbsp;&nbsp;<br>VV&nbsp;────────────────────────────────────────────────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>AD&nbsp;───►ADVP──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>VA&nbsp;───►VP&nbsp;───┴►VP&nbsp;────►IP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>DEC──────────────────────────┴►CP&nbsp;────►CP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP─────────┼►IP<br>CD&nbsp;───►QP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NN&nbsp;───►NP&nbsp;───┴────────────────────────►NP────┼►NP-OBJ──┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NR&nbsp;──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NN&nbsp;──┴────────────────────────────────►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>PU&nbsp;─────────────────────────────────────────────────────────────────┘&nbsp;&nbsp;&nbsp;</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"doc.pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"如果要分析原始文本的话，分词是第一步，所以先加载一个分词器：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 12,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"tok = hanlp.load(hanlp.pretrained.tok.COARSE_ELECTRA_SMALL_ZH)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"然后将分词器插入到流水线的第一级：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 13,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"[None->TransformerTaggingTokenizer->tok,\\n\",\n       \" tok->TransformerTagger->pos,\\n\",\n       \" tok->CRFConstituencyParser->con,\\n\",\n       \" None->merge_pos_into_con->None]\"\n      ]\n     },\n     \"execution_count\": 13,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"nlp.insert(0, tok, output_key='tok')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"然后就可以直接分析原始文本了：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 14,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"(TOP\\n\",\n      \"  (IP\\n\",\n      \"    (NT 2021)\\n\",\n      \"    (M 年)\\n\",\n      \"    (NP-PN-SBJ (NR HanLPv2.1))\\n\",\n      \"    (VP\\n\",\n      \"      (VV 带来)\\n\",\n      \"      (NP-OBJ\\n\",\n      \"        (CP (CP (IP (VP (ADVP (AD 最)) (VP (VA 先进)))) (DEC 的)))\\n\",\n      \"        (NP (QP (CD 多)) (NP (NN 语种)))\\n\",\n      \"        (NP (NR NLP) (NN 技术))))\\n\",\n      \"    (PU 。)))\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(nlp('2021年HanLPv2.1带来最先进的多语种NLP技术。')['con'])\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"你明白吗？HanLP是为聪明人设计的，只要你足够聪明，你就可以优雅地实现各种功能。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## 操作短语树的技巧\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"短语结构树的类型为`phrasetree.tree.Tree`，提供了许多接口，此处列举其中一些常用的接口。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 15,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"(TOP\\n\",\n      \"  (IP\\n\",\n      \"    (NP-TMP (NT 2021年))\\n\",\n      \"    (NP-PN-SBJ (NR HanLPv2.1))\\n\",\n      \"    (VP\\n\",\n      \"      (VV 带来)\\n\",\n      \"      (NP-OBJ\\n\",\n      \"        (CP (CP (IP (VP (ADVP (AD 最)) (VP (VA 先进)))) (DEC 的)))\\n\",\n      \"        (NP (QP (CD 多)) (NP (NN 语种)))\\n\",\n      \"        (NP (NR NLP) (NN 技术))))\\n\",\n      \"    (PU 。)))\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"tree = doc['con'] # tree数组的话则需要doc['con'][0]\\n\",\n    \"print(tree)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### 按高度枚举子树\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 16,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"子树：(VP (ADVP (AD 最)) (VP (VA 先进)))\\t标签：VP\\t短语：['最', '先进']\\n\",\n      \"子树：(NP (QP (CD 多)) (NP (NN 语种)))\\t标签：NP\\t短语：['多', '语种']\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"for subtree in tree.subtrees(lambda t: t.height() == 4):\\n\",\n    \"    print(f'子树：{subtree}\\\\t标签：{subtree.label()}\\\\t短语：{subtree.leaves()}')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### 按标签枚举子树\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 17,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"(NP (QP (CD 多)) (NP (NN 语种)))\\n\",\n      \"(NP (NN 语种))\\n\",\n      \"(NP (NR NLP) (NN 技术))\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"for subtree in tree.subtrees(lambda t: t.label() == 'NP'):\\n\",\n    \"    print(subtree)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### 遍历子节点\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 18,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"父节点(NP (NR NLP) (NN 技术))的子节点有：\\n\",\n      \"(NR NLP)\\n\",\n      \"(NN 技术)\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(f'父节点{subtree}的子节点有：')\\n\",\n    \"for child in subtree:\\n\",\n    \"    print(child)\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"con_stl.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/cor_restful.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/cor_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fcor_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp_restful -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 创建客户端\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"id\": \"0tmKBu7sNAXX\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from hanlp_restful import HanLPClient\\n\",\n    \"HanLP = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh') # auth不填则匿名，zh中文，mul多语种\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"EmZDmLn9aGxG\"\n   },\n   \"source\": [\n    \"#### 申请秘钥\\n\",\n    \"由于服务器算力有限，匿名用户每分钟限2次调用。如果你需要更多调用次数，[建议申请免费公益API秘钥auth](https://bbs.hanlp.com/t/hanlp2-1-restful-api/53)。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 指代消解\\n\",\n    \"任务越少，速度越快。如指定仅执行指代消解：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"2a0d392f-b99a-4a18-fc7f-754e2abe2e34\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"ret = HanLP.coreference_resolution('我姐送我她的猫。我很喜欢它。')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"返回值为一个包含分词结果与簇的dict:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"True\"\n      ]\n     },\n     \"execution_count\": 5,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"ret == {'clusters': [\\n\",\n    \"              [['我', 0, 1], ['我', 3, 4], ['我', 8, 9]], # 指代说话人\\n\",\n    \"              [['我姐', 0, 2], ['她', 4, 5]],             # 指代说话人的姐姐\\n\",\n    \"              [['她的猫', 4, 7], ['它', 11, 12]]],        # 指代说话人的姐姐的猫\\n\",\n    \"        'tokens': ['我', '姐', '送', '我', '她', '的', '猫', '。', '我', '很', '喜欢', '它', '。']}\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"wxctCigrTKu-\"\n   },\n   \"source\": [\n    \"对应如下结构：\\n\",\n    \"![cor](https://file.hankcs.com/img/coref_demo_small.png)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"XOsWkOqQfzlr\"\n   },\n   \"source\": [\n    \"为已分词的句子执行指代消解：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"bLZSTbv_f3OA\",\n    \"outputId\": \"111c0be9-bac6-4eee-d5bd-a972ffc34844\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"clusters = HanLP.coreference_resolution(tokens=[['我', '姐', '送', '我', '她', '的', '猫', '。'],\\n\",\n    \"                                                ['我', '很', '喜欢', '它', '。']])\\n\",\n    \"             \"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"返回值为簇的list：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 8,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"True\"\n      ]\n     },\n     \"execution_count\": 8,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"clusters == [\\n\",\n    \"              [['我', 0, 1], ['我', 3, 4], ['我', 8, 9]], # 指代说话人\\n\",\n    \"              [['我姐', 0, 2], ['她', 4, 5]],             # 指代说话人的姐姐\\n\",\n    \"              [['她的猫', 4, 7], ['它', 11, 12]]]         # 指代说话人的姐姐的猫\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"cor_restful.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/demo_amr.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2022-04-12 22:19\nimport hanlp\n\nparser = hanlp.load(hanlp.pretrained.amr.MRP2020_AMR_ENG_ZHO_XLM_BASE)\n\n# For Chinese:\nprint(parser([\"男孩\", \"希望\", \"女孩\", \"相信\", \"他\", \"。\"]))\nprint(parser([\"男孩\", \"希望\", \"女孩\", \"相信\", \"他\", \"。\"], output_amr=False))\n\n# For English:\nprint(parser(['The', 'boy', 'wants', 'the', 'girl', 'to', 'believe', 'him', '.'], language='eng'))\n# It's suggested to also feed the lemma for stabler performance.\nprint(parser([('The', 'the'), ('boy', 'boy'), ('wants', 'want'), ('the', 'the'), ('girl', 'girl'), ('to', 'to'),\n              ('believe', 'believe'), ('him', 'he'), ('.', '.')], language='eng'))\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/demo_custom_dict.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-15 22:26\nimport hanlp\nfrom hanlp.components.mtl.multi_task_learning import MultiTaskLearning\nfrom hanlp.components.mtl.tasks.tok.tag_tok import TaggingTokenization\n\n# 加载多任务模型\nHanLP: MultiTaskLearning = hanlp.load(hanlp.pretrained.mtl.CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH)\n# 获取分词任务（以tok开头的任务都是分词任务，以细分标准为例）\ntok: TaggingTokenization = HanLP['tok/fine']\n\ntok.dict_force = tok.dict_combine = None\nprint(f'不挂词典:\\n{HanLP(\"商品和服务项目\")[\"tok/fine\"]}')\n\ntok.dict_force = {'和服', '服务项目'}\nprint(f'强制模式:\\n{HanLP(\"商品和服务项目\")[\"tok/fine\"]}')  # 慎用，详见《自然语言处理入门》第二章\n\ntok.dict_force = {'和服务': ['和', '服务']}\nprint(f'强制校正:\\n{HanLP(\"正向匹配商品和服务、任何和服务必按上述切分\")[\"tok/fine\"]}')\n\ntok.dict_force = None\ntok.dict_combine = {'和服', '服务项目'}\nprint(f'合并模式:\\n{HanLP(\"商品和服务项目\")[\"tok/fine\"]}')\n\n# 需要算法基础才能理解，初学者可参考 http://nlp.hankcs.com/book.php\n# See also https://hanlp.hankcs.com/docs/api/hanlp/components/tokenizers/transformer.html\n\n# 含有空格、制表符等（Transformer tokenizer去掉的字符）的词语需要用tuple的形式提供\ntok.dict_combine = {('iPad', 'Pro'), '2个空格'}\nprint(f'空格匹配：\\n{HanLP(\"如何评价iPad Pro ？iPad  Pro有2个空格\", tasks=\"tok/fine\")[\"tok/fine\"]}')\n# 聪明的用户请继续阅读：tuple词典中的字符串其实等价于该字符串的所有可能的切分方式\nprint(f'词典内容：\\n{dict(tok.dict_combine.config[\"dictionary\"]).keys()}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/demo_custom_dict_stl.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-15 22:26\nimport hanlp\nfrom hanlp.components.tokenizers.transformer import TransformerTaggingTokenizer\n\n# 加载一个旧版本单任务模型演示分词错误（最新版已经修复）：\ntok: TransformerTaggingTokenizer = hanlp.load('https://file.hankcs.com/hanlp/tok/coarse_electra_small_20220220_013548.zip')\n\ntok.dict_force = tok.dict_combine = None\nprint(f'不挂词典:\\n{tok(\"首相和川普通电话\")}')\n\ntok.dict_force = {'川普'}\nprint(f'强制模式:\\n{tok([\"首相和川普通电话\", \"银川普通人与川普通电话讲四川普通话\"])}')  # 慎用，详见《自然语言处理入门》第二章\n\ntok.dict_force = {'川普通电话': ['川普', '通', '电话']}\nprint(f'强制校正:\\n{tok([\"首相和川普通电话\", \"银川普通人与川普通电话讲四川普通话\"])}')\n\ntok.dict_force = None\ntok.dict_combine = {'美国总统'}\nprint(f'合并模式:\\n{tok(\"首相和川普通电话，川普是美国总统。\")}')\n\n# 需要算法基础才能理解，初学者可参考 http://nlp.hankcs.com/book.php\n# See also https://hanlp.hankcs.com/docs/api/hanlp/components/tokenizers/transformer.html\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/demo_del_tasks.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-02-03 13:28\nimport hanlp\nfrom hanlp.components.mtl.multi_task_learning import MultiTaskLearning\nfrom hanlp_common.document import Document\n\nHanLP: MultiTaskLearning = hanlp.load(hanlp.pretrained.mtl.OPEN_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH)\ntasks = list(HanLP.tasks.keys())\nprint(tasks)  # Pick what you need from what we have\nfor task in tasks:\n    if task not in ('tok', 'pos'):\n        del HanLP[task]\n# You can save it as a new component\n# HanLP.save('path/to/new/component')\n# HanLP.load('path/to/new/component')\nprint(HanLP.tasks.keys())\ndoc: Document = HanLP(['2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', 'up主来到北京立方庭参观自然语义科技公司。'])\nprint(doc)\ndoc.pretty_print()\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/demo_document.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2022-10-26 23:40\nfrom hanlp_common.document import Document\n\n# Create a document or get a document from HanLP.parse\ndoc = Document(\n    tok=[[\"晓美焰\", \"来到\", \"北京\", \"立方庭\", \"参观\", \"自然\", \"语义\", \"科技\", \"公司\"]],\n    pos=[[\"NR\", \"VV\", \"NR\", \"NR\", \"VV\", \"NN\", \"NN\", \"NN\", \"NN\"]],\n    ner=[[[\"晓美焰\", \"PERSON\", 0, 1], [\"北京立方庭\", \"LOCATION\", 2, 4],\n          [\"自然语义科技公司\", \"ORGANIZATION\", 5, 9]]],\n    dep=[[[2, \"nsubj\"], [0, \"root\"], [4, \"name\"], [2, \"dobj\"], [2, \"conj\"],\n          [9, \"compound\"], [9, \"compound\"], [9, \"compound\"], [5, \"dobj\"]]]\n)\n\n# print(doc) or str(doc) to get its JSON representation\nprint(doc)\n\n# Access an annotation by its task name\nprint(doc['tok'])\n\n# Get number of sentences\nprint(f'It has {doc.count_sentences()} sentence(s)')\n\n# Access the n-th sentence\nprint(doc.squeeze(0)['tok'])\n\n# Pretty print it right in your console or notebook\ndoc.pretty_print()\n\n# To save the pretty prints in a str\npretty_text: str = '\\n\\n'.join(doc.to_pretty())\n\n# Create a document from a dict\ndoc = Document({\n    \"tok/fine\": [\n        [\"晓美焰\", \"来到\", \"北京\", \"立方庭\", \"参观\", \"自然\", \"语义\", \"科技\", \"公司\", \"。\"]\n    ],\n    \"tok/coarse\": [\n        [\"晓美焰\", \"来到\", \"北京立方庭\", \"参观\", \"自然语义科技公司\", \"。\"]\n    ],\n    \"pos/ctb\": [\n        [\"NR\", \"VV\", \"NR\", \"NR\", \"VV\", \"NN\", \"NN\", \"NN\", \"NN\", \"PU\"]\n    ],\n    \"pos/pku\": [\n        [\"nr\", \"v\", \"ns\", \"nz\", \"v\", \"n\", \"n\", \"n\", \"n\", \"w\"]\n    ],\n    \"ner/msra\": [\n        [[\"晓美焰\", \"PERSON\", 0, 1], [\"北京立方庭\", \"LOCATION\", 2, 4], [\"自然语义科技公司\", \"ORGANIZATION\", 5, 9]]\n    ],\n    \"ner/ontonotes\": [\n        [[\"晓美焰\", \"PERSON\", 0, 1], [\"北京\", \"GPE\", 2, 3], [\"立方庭\", \"FAC\", 3, 4], [\"自然语义科技公司\", \"ORG\", 5, 9]]\n    ],\n    \"srl\": [\n        [[[\"晓美焰\", \"ARG0\", 0, 1], [\"来到\", \"PRED\", 1, 2], [\"北京立方庭\", \"ARG1\", 2, 4]],\n         [[\"晓美焰\", \"ARG0\", 0, 1], [\"参观\", \"PRED\", 4, 5], [\"自然语义科技公司\", \"ARG1\", 5, 9]]]\n    ],\n    \"dep\": [\n        [[2, \"nsubj\"], [0, \"root\"], [4, \"name\"], [2, \"dobj\"], [2, \"conj\"], [9, \"compound\"], [9, \"compound\"],\n         [9, \"compound\"], [5, \"dobj\"], [2, \"punct\"]]\n    ]\n})\n# Pretty print using a different NER annotation\ndoc.pretty_print(ner='ner/ontonotes')\n# Get the first annotation for NER\nprint(doc.get_by_prefix('ner'))\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/demo_mlm.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2022-01-29 21:11\nfrom hanlp.components.lm.mlm import MaskedLanguageModel\n\nmlm = MaskedLanguageModel()\nmlm.load('bert-base-chinese')\nprint(mlm('生活的真谛是[MASK]。'))\n\n# Batching is always faster\nprint(mlm(['生活的真谛是[MASK]。', '巴黎是[MASK][MASK]的首都。']))\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/demo_mtl.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-31 13:51\nimport hanlp\nfrom hanlp_common.document import Document\n\n# CLOSE是自然语义标注的闭源语料库，BASE是中号模型，ZH中文\nHanLP = hanlp.load(hanlp.pretrained.mtl.CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH)\n# 默认执行全部任务\ndoc: Document = HanLP(['2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', '阿婆主来到北京立方庭参观自然语义科技公司。'])\n# 返回类型Document是dict的子类，打印出来兼容JSON\nprint(doc)\n# 即时可视化，防止换行请最大化窗口，推荐在Jupyter Notebook里调用\ndoc.pretty_print()\n# 指定可视化OntoNotes标准的NER\n# doc.pretty_print(ner='ner/ontonotes', pos='pku')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/demo_ner_dict.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-04-29 11:06\nimport hanlp\nfrom hanlp.components.mtl.tasks.ner.tag_ner import TaggingNamedEntityRecognition\nfrom hanlp.utils.io_util import get_resource\n\nHanLP = hanlp.load(hanlp.pretrained.mtl.CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ERNIE_GRAM_ZH)\nner: TaggingNamedEntityRecognition = HanLP['ner/msra']\nner.dict_whitelist = {'午饭后': 'TIME'}\ndoc = HanLP('2021年测试高血压是138，时间是午饭后2点45，低血压是44', tasks='ner/msra')\ndoc.pretty_print()\nprint(doc['ner/msra'])\n\nner.dict_tags = {('名字', '叫', '金华'): ('O', 'O', 'S-PERSON')}\nHanLP('他在浙江金华出生，他的名字叫金华。', tasks='ner/msra').pretty_print()\n\n# HanLP.save(get_resource(hanlp.pretrained.mtl.CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ERNIE_GRAM_ZH))\n\n# 需要算法基础才能理解，初学者可参考 http://nlp.hankcs.com/book.php\n# See https://hanlp.hankcs.com/docs/api/hanlp/components/mtl/tasks/ner/tag_ner.html\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/demo_parse_constituency.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2022-01-18 11:09\nfrom hanlp_common.document import Document\nimport hanlp\n\ncon = hanlp.load(hanlp.pretrained.constituency.CTB9_CON_FULL_TAG_ELECTRA_SMALL)\n# To speed up, parse multiple sentences at once, and use a GPU.\nprint(con([\"2021年\", \"HanLPv2.1\", \"带来\", \"最\", \"先进\", \"的\", \"多\", \"语种\", \"NLP\", \"技术\", \"。\"]))\n\n\n# The rest of this tutorial is written for clever users.\n# The first level of non-terminals are PoS tags. So usually a PoS model is piped.\ndef merge_pos_into_con(doc: Document):\n    flat = isinstance(doc['pos'][0], str)\n    if flat:\n        doc = Document((k, [v]) for k, v in doc.items())\n    for tree, tags in zip(doc['con'], doc['pos']):\n        offset = 0\n        for subtree in tree.subtrees(lambda t: t.height() == 2):\n            tag = subtree.label()\n            if tag == '_':\n                subtree.set_label(tags[offset])\n            offset += 1\n    if flat:\n        doc = doc.squeeze()\n    return doc\n\n\npos = hanlp.load(hanlp.pretrained.pos.CTB9_POS_ELECTRA_SMALL)\nnlp = hanlp.pipeline() \\\n    .append(pos, input_key='tok', output_key='pos') \\\n    .append(con, input_key='tok', output_key='con') \\\n    .append(merge_pos_into_con, input_key='*')\nprint(f'The pipeline looks like this: {nlp}')\ndoc = nlp(tok=[\"2021年\", \"HanLPv2.1\", \"带来\", \"最\", \"先进\", \"的\", \"多\", \"语种\", \"NLP\", \"技术\", \"。\"])\nprint(doc)\ndoc.pretty_print()\n\n# If you need to parse raw text, simply add a tokenizer into this pipeline.\ntok = hanlp.load(hanlp.pretrained.tok.COARSE_ELECTRA_SMALL_ZH)\nnlp.insert(0, tok, output_key='tok')\nprint(f'The pipeline looks like this: {nlp}')\ndoc = nlp('2021年HanLPv2.1带来最先进的多语种NLP技术。')\nprint(doc)\ndoc.pretty_print()\n\n# ATTENTION: Pipelines are usually slower than MTL but they are more flexible.\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/demo_pipeline.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-12-28 20:47\nimport hanlp\n\n# Pipeline allows blending multiple callable functions no matter they are a rule, a TensorFlow component or a PyTorch\n# one. However, it's slower than the MTL framework.\n# pos = hanlp.load(hanlp.pretrained.pos.CTB9_POS_ALBERT_BASE)  # In case both tf and torch are used, load tf first.\n\nHanLP = hanlp.pipeline() \\\n    .append(hanlp.utils.rules.split_sentence, output_key='sentences') \\\n    .append(hanlp.load('CTB9_TOK_ELECTRA_SMALL'), output_key='tok') \\\n    .append(hanlp.load('CTB9_POS_ELECTRA_SMALL'), output_key='pos') \\\n    .append(hanlp.load('MSRA_NER_ELECTRA_SMALL_ZH'), output_key='ner', input_key='tok') \\\n    .append(hanlp.load('CTB9_DEP_ELECTRA_SMALL', conll=False), output_key='dep', input_key='tok') \\\n    .append(hanlp.load('CTB9_CON_ELECTRA_SMALL'), output_key='con', input_key='tok')\n\ndoc = HanLP('2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。阿婆主来到北京立方庭参观自然语义科技公司。')\nprint(doc)\ndoc.pretty_print()\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/demo_pos_dict.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-15 22:26\nimport hanlp\nfrom hanlp.components.mtl.multi_task_learning import MultiTaskLearning\nfrom hanlp.components.mtl.tasks.pos import TransformerTagging\nfrom hanlp.components.mtl.tasks.tok.tag_tok import TaggingTokenization\nfrom tests import cdroot\n\ncdroot()\nHanLP: MultiTaskLearning = hanlp.load(hanlp.pretrained.mtl.CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH)\n\n# Demonstrates custom dict in part-of-speech tagging\npos: TransformerTagging = HanLP['pos/ctb']\n\nprint(f'自定义单个词性:')\npos.dict_tags = {'HanLP': 'state-of-the-art-tool'}\nHanLP(\"HanLP为生产环境带来次世代最先进的多语种NLP技术。\", tasks='pos/ctb').pretty_print()\n\nprint(f'根据上下文自定义词性:')\npos.dict_tags = {('的', '希望'): ('补语成分', '名词'), '希望': '动词'}\nHanLP(\"我的希望是希望张晚霞的背影被晚霞映红。\", tasks='pos/ctb').pretty_print()\n\n# 需要算法基础才能理解，初学者可参考 http://nlp.hankcs.com/book.php\n# See also https://hanlp.hankcs.com/docs/api/hanlp/components/taggers/transformer_tagger.html\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/demo_sts.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-05-24 13:15\nimport hanlp\n\nsim = hanlp.load(hanlp.pretrained.sts.STS_ELECTRA_BASE_ZH)\nprint(sim([\n    ['看图猜一电影名', '看图猜电影'],\n    ['无线路由器怎么无线上网', '无线上网卡和无线路由器怎么用'],\n    ['北京到上海的动车票', '上海到北京的动车票'],\n]))\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/demo_word2vec.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-12-12 18:33\nimport hanlp\nimport torch\n\nword2vec = hanlp.load(hanlp.pretrained.word2vec.CONVSEG_W2V_NEWS_TENSITE_WORD_PKU)\nvec = word2vec('先进')\nprint(vec)\n\nprint(torch.nn.functional.cosine_similarity(word2vec('先进'), word2vec('优秀'), dim=0))\nprint(torch.nn.functional.cosine_similarity(word2vec('先进'), word2vec('水果'), dim=0))\n\nprint('获取语义最相似的词语：')\nprint(word2vec.most_similar('上海'))\n# print(word2vec.most_similar(['上海', '寒冷'])) # batching更快\n\nprint('非常寒冷是OOV所以无法获取：')\nprint(word2vec.most_similar('非常寒冷'))\nprint('但是在doc2vec模式下OOV也可以进行相似度计算：')\nprint(word2vec.most_similar('非常寒冷', doc2vec=True))\nprint('甚至可以处理短文本：')\nprint(word2vec.most_similar('国家图书馆推出2022年春节主题活动', doc2vec=True))\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/dep_mtl.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/dep_mtl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fdep_mtl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 加载模型\\n\",\n    \"HanLP的工作流程是先加载模型，模型的标示符存储在`hanlp.pretrained`这个包中，按照NLP任务归类。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"id\": \"0tmKBu7sNAXX\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'OPEN_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/mtl/open_tok_pos_ner_srl_dep_sdp_con_electra_small_20201223_035557.zip',\\n\",\n       \" 'OPEN_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH': 'https://file.hankcs.com/hanlp/mtl/open_tok_pos_ner_srl_dep_sdp_con_electra_base_20201223_201906.zip',\\n\",\n       \" 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_small_20210111_124159.zip',\\n\",\n       \" 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_base_20210111_124519.zip',\\n\",\n       \" 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ERNIE_GRAM_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_ernie_gram_base_aug_20210904_145403.zip',\\n\",\n       \" 'UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_MT5_SMALL': 'https://file.hankcs.com/hanlp/mtl/ud_ontonotes_tok_pos_lem_fea_ner_srl_dep_sdp_con_mt5_small_20210228_123458.zip',\\n\",\n       \" 'UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_XLMR_BASE': 'https://file.hankcs.com/hanlp/mtl/ud_ontonotes_tok_pos_lem_fea_ner_srl_dep_sdp_con_xlm_base_20210602_211620.zip',\\n\",\n       \" 'NPCMJ_UD_KYOTO_TOK_POS_CON_BERT_BASE_CHAR_JA': 'https://file.hankcs.com/hanlp/mtl/npcmj_ud_kyoto_tok_pos_ner_dep_con_srl_bert_base_char_ja_20210914_133742.zip'}\"\n      ]\n     },\n     \"execution_count\": 2,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"import hanlp\\n\",\n    \"hanlp.pretrained.mtl.ALL # MTL多任务，具体任务见模型名称，语种见名称最后一个字段或相应语料库\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"EmZDmLn9aGxG\"\n   },\n   \"source\": [\n    \"调用`hanlp.load`进行加载，模型会自动下载到本地缓存。自然语言处理分为许多任务，分词只是最初级的一个。与其每个任务单独创建一个模型，不如利用HanLP的联合模型一次性完成多个任务：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"HanLP = hanlp.load(hanlp.pretrained.mtl.CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 依存句法分析\\n\",\n    \"任务越少，速度越快。如指定仅执行依存句法分析：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"2a0d392f-b99a-4a18-fc7f-754e2abe2e34\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"doc = HanLP(['2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', '阿婆主来到北京立方庭参观自然语义科技公司。'], tasks='dep')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"返回值为一个[Document](https://hanlp.hankcs.com/docs/api/common/document.html):\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"{\\n\",\n      \"  \\\"tok/fine\\\": [\\n\",\n      \"    [\\\"2021年\\\", \\\"HanLPv2.1\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次\\\", \\\"世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多\\\", \\\"语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"],\\n\",\n      \"    [\\\"阿婆主\\\", \\\"来到\\\", \\\"北京\\\", \\\"立方庭\\\", \\\"参观\\\", \\\"自然\\\", \\\"语义\\\", \\\"科技\\\", \\\"公司\\\", \\\"。\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"dep\\\": [\\n\",\n      \"    [[6, \\\"tmod\\\"], [6, \\\"nsubj\\\"], [6, \\\"prep\\\"], [5, \\\"nn\\\"], [3, \\\"pobj\\\"], [0, \\\"root\\\"], [8, \\\"amod\\\"], [15, \\\"nn\\\"], [10, \\\"advmod\\\"], [15, \\\"rcmod\\\"], [10, \\\"assm\\\"], [13, \\\"nummod\\\"], [15, \\\"nn\\\"], [15, \\\"nn\\\"], [6, \\\"dobj\\\"], [6, \\\"punct\\\"]],\\n\",\n      \"    [[2, \\\"nsubj\\\"], [0, \\\"root\\\"], [4, \\\"nn\\\"], [2, \\\"dobj\\\"], [2, \\\"conj\\\"], [9, \\\"nn\\\"], [9, \\\"nn\\\"], [9, \\\"nn\\\"], [5, \\\"dobj\\\"], [2, \\\"punct\\\"]]\\n\",\n      \"  ]\\n\",\n      \"}\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(doc)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"`doc['dep']`为句子们的依存句法树列表，第`i`个二元组表示第`i`个单词的`[中心词的下标, 与中心词的依存关系]`。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"wxctCigrTKu-\"\n   },\n   \"source\": [\n    \"可视化依存句法树：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"Zo08uquCTFSk\",\n    \"outputId\": \"c6077f2d-7084-4f4b-a3bc-9aa9951704ea\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Dep Tree    \\tToken    \\tRelati\\n\",\n      \"────────────\\t─────────\\t──────\\n\",\n      \" ┌─────────►\\t2021年    \\ttmod  \\n\",\n      \" │┌────────►\\tHanLPv2.1\\tnsubj \\n\",\n      \" ││┌─►┌─────\\t为        \\tprep  \\n\",\n      \" │││  │  ┌─►\\t生产       \\tnn    \\n\",\n      \" │││  └─►└──\\t环境       \\tpobj  \\n\",\n      \"┌┼┴┴────────\\t带来       \\troot  \\n\",\n      \"││       ┌─►\\t次        \\tamod  \\n\",\n      \"││  ┌───►└──\\t世代       \\tnn    \\n\",\n      \"││  │    ┌─►\\t最        \\tadvmod\\n\",\n      \"││  │┌──►├──\\t先进       \\trcmod \\n\",\n      \"││  ││   └─►\\t的        \\tassm  \\n\",\n      \"││  ││   ┌─►\\t多        \\tnummod\\n\",\n      \"││  ││┌─►└──\\t语种       \\tnn    \\n\",\n      \"││  │││  ┌─►\\tNLP      \\tnn    \\n\",\n      \"│└─►└┴┴──┴──\\t技术       \\tdobj  \\n\",\n      \"└──────────►\\t。        \\tpunct \\n\",\n      \"\\n\",\n      \"Dep Tree    \\tTok\\tRelat\\n\",\n      \"────────────\\t───\\t─────\\n\",\n      \"         ┌─►\\t阿婆主\\tnsubj\\n\",\n      \"┌┬────┬──┴──\\t来到 \\troot \\n\",\n      \"││    │  ┌─►\\t北京 \\tnn   \\n\",\n      \"││    └─►└──\\t立方庭\\tdobj \\n\",\n      \"│└─►┌───────\\t参观 \\tconj \\n\",\n      \"│   │  ┌───►\\t自然 \\tnn   \\n\",\n      \"│   │  │┌──►\\t语义 \\tnn   \\n\",\n      \"│   │  ││┌─►\\t科技 \\tnn   \\n\",\n      \"│   └─►└┴┴──\\t公司 \\tdobj \\n\",\n      \"└──────────►\\t。  \\tpunct\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"doc.pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"转换为CoNLL格式：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"1\\t2021年\\t_\\t_\\t_\\t_\\t6\\ttmod\\t_\\t_\\n\",\n      \"2\\tHanLPv2.1\\t_\\t_\\t_\\t_\\t6\\tnsubj\\t_\\t_\\n\",\n      \"3\\t为\\t_\\t_\\t_\\t_\\t6\\tprep\\t_\\t_\\n\",\n      \"4\\t生产\\t_\\t_\\t_\\t_\\t5\\tnn\\t_\\t_\\n\",\n      \"5\\t环境\\t_\\t_\\t_\\t_\\t3\\tpobj\\t_\\t_\\n\",\n      \"6\\t带来\\t_\\t_\\t_\\t_\\t0\\troot\\t_\\t_\\n\",\n      \"7\\t次\\t_\\t_\\t_\\t_\\t8\\tamod\\t_\\t_\\n\",\n      \"8\\t世代\\t_\\t_\\t_\\t_\\t15\\tnn\\t_\\t_\\n\",\n      \"9\\t最\\t_\\t_\\t_\\t_\\t10\\tadvmod\\t_\\t_\\n\",\n      \"10\\t先进\\t_\\t_\\t_\\t_\\t15\\trcmod\\t_\\t_\\n\",\n      \"11\\t的\\t_\\t_\\t_\\t_\\t10\\tassm\\t_\\t_\\n\",\n      \"12\\t多\\t_\\t_\\t_\\t_\\t13\\tnummod\\t_\\t_\\n\",\n      \"13\\t语种\\t_\\t_\\t_\\t_\\t15\\tnn\\t_\\t_\\n\",\n      \"14\\tNLP\\t_\\t_\\t_\\t_\\t15\\tnn\\t_\\t_\\n\",\n      \"15\\t技术\\t_\\t_\\t_\\t_\\t6\\tdobj\\t_\\t_\\n\",\n      \"16\\t。\\t_\\t_\\t_\\t_\\t6\\tpunct\\t_\\t_\\n\",\n      \"\\n\",\n      \"1\\t阿婆主\\t_\\t_\\t_\\t_\\t2\\tnsubj\\t_\\t_\\n\",\n      \"2\\t来到\\t_\\t_\\t_\\t_\\t0\\troot\\t_\\t_\\n\",\n      \"3\\t北京\\t_\\t_\\t_\\t_\\t4\\tnn\\t_\\t_\\n\",\n      \"4\\t立方庭\\t_\\t_\\t_\\t_\\t2\\tdobj\\t_\\t_\\n\",\n      \"5\\t参观\\t_\\t_\\t_\\t_\\t2\\tconj\\t_\\t_\\n\",\n      \"6\\t自然\\t_\\t_\\t_\\t_\\t9\\tnn\\t_\\t_\\n\",\n      \"7\\t语义\\t_\\t_\\t_\\t_\\t9\\tnn\\t_\\t_\\n\",\n      \"8\\t科技\\t_\\t_\\t_\\t_\\t9\\tnn\\t_\\t_\\n\",\n      \"9\\t公司\\t_\\t_\\t_\\t_\\t5\\tdobj\\t_\\t_\\n\",\n      \"10\\t。\\t_\\t_\\t_\\t_\\t2\\tpunct\\t_\\t_\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(doc.to_conll())\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"XOsWkOqQfzlr\"\n   },\n   \"source\": [\n    \"为已分词的句子执行依存句法分析：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 9,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"bLZSTbv_f3OA\",\n    \"outputId\": \"111c0be9-bac6-4eee-d5bd-a972ffc34844\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Dep Tree   \\tToken\\tRelati\\n\",\n      \"───────────\\t─────\\t──────\\n\",\n      \" ┌────────►\\tHanLP\\tnsubj \\n\",\n      \" │┌─►┌─────\\t为    \\tprep  \\n\",\n      \" ││  │  ┌─►\\t生产   \\tnn    \\n\",\n      \" ││  └─►└──\\t环境   \\tpobj  \\n\",\n      \"┌┼┴────────\\t带来   \\troot  \\n\",\n      \"││  ┌─────►\\t次世代  \\tnn    \\n\",\n      \"││  │   ┌─►\\t最    \\tadvmod\\n\",\n      \"││  │┌─►├──\\t先进   \\trcmod \\n\",\n      \"││  ││  └─►\\t的    \\tassm  \\n\",\n      \"││  ││ ┌──►\\t多语种  \\tnn    \\n\",\n      \"││  ││ │┌─►\\tNLP  \\tnn    \\n\",\n      \"│└─►└┴─┴┴──\\t技术   \\tdobj  \\n\",\n      \"└─────────►\\t。    \\tpunct \\n\",\n      \"\\n\",\n      \"Dep Tree        \\tTok\\tRelation \\n\",\n      \"────────────────\\t───\\t─────────\\n\",\n      \"          ┌─►┌──\\t我  \\tassmod   \\n\",\n      \"          │  └─►\\t的  \\tassm     \\n\",\n      \"       ┌─►└─────\\t希望 \\ttop      \\n\",\n      \"┌┬─────┴────────\\t是  \\troot     \\n\",\n      \"│└─►┌───────────\\t希望 \\tccomp    \\n\",\n      \"│   │     ┌─►┌──\\t张晚霞\\tassmod   \\n\",\n      \"│   │     │  └─►\\t的  \\tassm     \\n\",\n      \"│   │  ┌─►└─────\\t背影 \\tnsubjpass\\n\",\n      \"│   └─►└──┬─────\\t被  \\tccomp    \\n\",\n      \"│         │  ┌─►\\t晚霞 \\tnsubj    \\n\",\n      \"│         └─►└──\\t映红 \\tdep      \\n\",\n      \"└──────────────►\\t。  \\tpunct    \\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"HanLP([\\n\",\n    \"    [\\\"HanLP\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"],\\n\",\n    \"    [\\\"我\\\", \\\"的\\\", \\\"希望\\\", \\\"是\\\", \\\"希望\\\", \\\"张晚霞\\\", \\\"的\\\", \\\"背影\\\", \\\"被\\\", \\\"晚霞\\\", \\\"映红\\\", \\\"。\\\"]\\n\",\n    \"  ], tasks='dep', skip_tasks='tok*').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"#### 注意\\n\",\n    \"Native API的输入单位限定为句子，需使用[多语种分句模型](https://github.com/hankcs/HanLP/blob/master/plugins/hanlp_demo/hanlp_demo/sent_split.py)或[基于规则的分句函数](https://github.com/hankcs/HanLP/blob/master/hanlp/utils/rules.py#L19)先行分句。RESTful同时支持全文、句子、已分词的句子。除此之外，RESTful和native两种API的语义设计完全一致，用户可以无缝互换。\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"dep_mtl.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/dep_restful.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/dep_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fdep_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp_restful -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 创建客户端\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"id\": \"0tmKBu7sNAXX\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from hanlp_restful import HanLPClient\\n\",\n    \"HanLP = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh') # auth不填则匿名，zh中文，mul多语种\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"EmZDmLn9aGxG\"\n   },\n   \"source\": [\n    \"#### 申请秘钥\\n\",\n    \"由于服务器算力有限，匿名用户每分钟限2次调用。如果你需要更多调用次数，[建议申请免费公益API秘钥auth](https://bbs.hanlp.com/t/hanlp2-1-restful-api/53)。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 依存句法分析\\n\",\n    \"任务越少，速度越快。如指定仅执行依存句法分析：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"2a0d392f-b99a-4a18-fc7f-754e2abe2e34\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"doc = HanLP('2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', tasks='dep')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"返回值为一个[Document](https://hanlp.hankcs.com/docs/api/common/document.html):\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"{\\n\",\n      \"  \\\"tok/fine\\\": [\\n\",\n      \"    [\\\"2021年\\\", \\\"HanLPv2.1\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次\\\", \\\"世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多\\\", \\\"语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"dep\\\": [\\n\",\n      \"    [[6, \\\"tmod\\\"], [6, \\\"nsubj\\\"], [6, \\\"prep\\\"], [5, \\\"nn\\\"], [3, \\\"pobj\\\"], [0, \\\"root\\\"], [8, \\\"clf\\\"], [10, \\\"dep\\\"], [10, \\\"advmod\\\"], [15, \\\"rcmod\\\"], [10, \\\"cpm\\\"], [13, \\\"nummod\\\"], [15, \\\"nn\\\"], [15, \\\"nn\\\"], [6, \\\"dobj\\\"], [6, \\\"punct\\\"]]\\n\",\n      \"  ]\\n\",\n      \"}\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(doc)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"`doc['dep']`为句子们的依存句法树列表，第`i`个二元组表示第`i`个单词的`[中心词的下标, 与中心词的依存关系]`。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"wxctCigrTKu-\"\n   },\n   \"source\": [\n    \"可视化依存句法树：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"Zo08uquCTFSk\",\n    \"outputId\": \"c6077f2d-7084-4f4b-a3bc-9aa9951704ea\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Dep Tree     \\tToken    \\tRelati\\n\",\n      \"─────────────\\t─────────\\t──────\\n\",\n      \"  ┌─────────►\\t2021年    \\ttmod  \\n\",\n      \"  │┌────────►\\tHanLPv2.1\\tnsubj \\n\",\n      \"  ││┌─►┌─────\\t为        \\tprep  \\n\",\n      \"  │││  │  ┌─►\\t生产       \\tnn    \\n\",\n      \"  │││  └─►└──\\t环境       \\tpobj  \\n\",\n      \"┌┬┴┴┴────────\\t带来       \\troot  \\n\",\n      \"││        ┌─►\\t次        \\tclf   \\n\",\n      \"││     ┌─►└──\\t世代       \\tdep   \\n\",\n      \"││     │  ┌─►\\t最        \\tadvmod\\n\",\n      \"││  ┌─►└──┼──\\t先进       \\trcmod \\n\",\n      \"││  │     └─►\\t的        \\tcpm   \\n\",\n      \"││  │     ┌─►\\t多        \\tnummod\\n\",\n      \"││  │  ┌─►└──\\t语种       \\tnn    \\n\",\n      \"││  │  │  ┌─►\\tNLP      \\tnn    \\n\",\n      \"│└─►└──┴──┴──\\t技术       \\tdobj  \\n\",\n      \"└───────────►\\t。        \\tpunct \\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"doc.pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"转换为CoNLL格式：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"1\\t2021年\\t_\\t_\\t_\\t_\\t6\\ttmod\\t_\\t_\\n\",\n      \"2\\tHanLPv2.1\\t_\\t_\\t_\\t_\\t6\\tnsubj\\t_\\t_\\n\",\n      \"3\\t为\\t_\\t_\\t_\\t_\\t6\\tprep\\t_\\t_\\n\",\n      \"4\\t生产\\t_\\t_\\t_\\t_\\t5\\tnn\\t_\\t_\\n\",\n      \"5\\t环境\\t_\\t_\\t_\\t_\\t3\\tpobj\\t_\\t_\\n\",\n      \"6\\t带来\\t_\\t_\\t_\\t_\\t0\\troot\\t_\\t_\\n\",\n      \"7\\t次\\t_\\t_\\t_\\t_\\t8\\tclf\\t_\\t_\\n\",\n      \"8\\t世代\\t_\\t_\\t_\\t_\\t10\\tdep\\t_\\t_\\n\",\n      \"9\\t最\\t_\\t_\\t_\\t_\\t10\\tadvmod\\t_\\t_\\n\",\n      \"10\\t先进\\t_\\t_\\t_\\t_\\t15\\trcmod\\t_\\t_\\n\",\n      \"11\\t的\\t_\\t_\\t_\\t_\\t10\\tcpm\\t_\\t_\\n\",\n      \"12\\t多\\t_\\t_\\t_\\t_\\t13\\tnummod\\t_\\t_\\n\",\n      \"13\\t语种\\t_\\t_\\t_\\t_\\t15\\tnn\\t_\\t_\\n\",\n      \"14\\tNLP\\t_\\t_\\t_\\t_\\t15\\tnn\\t_\\t_\\n\",\n      \"15\\t技术\\t_\\t_\\t_\\t_\\t6\\tdobj\\t_\\t_\\n\",\n      \"16\\t。\\t_\\t_\\t_\\t_\\t6\\tpunct\\t_\\t_\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(doc.to_conll())\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"XOsWkOqQfzlr\"\n   },\n   \"source\": [\n    \"为已分词的句子执行依存句法分析：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"bLZSTbv_f3OA\",\n    \"outputId\": \"111c0be9-bac6-4eee-d5bd-a972ffc34844\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Dep Tree   \\tToken\\tRelati\\n\",\n      \"───────────\\t─────\\t──────\\n\",\n      \" ┌────────►\\tHanLP\\tnsubj \\n\",\n      \" │┌─►┌─────\\t为    \\tprep  \\n\",\n      \" ││  │  ┌─►\\t生产   \\tnn    \\n\",\n      \" ││  └─►└──\\t环境   \\tpobj  \\n\",\n      \"┌┼┴────────\\t带来   \\troot  \\n\",\n      \"││     ┌──►\\t次世代  \\tdep   \\n\",\n      \"││     │┌─►\\t最    \\tadvmod\\n\",\n      \"││  ┌─►└┼──\\t先进   \\trcmod \\n\",\n      \"││  │   └─►\\t的    \\tcpm   \\n\",\n      \"││  │  ┌──►\\t多语种  \\tnn    \\n\",\n      \"││  │  │┌─►\\tNLP  \\tnn    \\n\",\n      \"│└─►└──┴┴──\\t技术   \\tdobj  \\n\",\n      \"└─────────►\\t。    \\tpunct \\n\",\n      \"\\n\",\n      \"Dep Tree        \\tTok\\tRelation \\n\",\n      \"────────────────\\t───\\t─────────\\n\",\n      \"          ┌─►┌──\\t我  \\tassmod   \\n\",\n      \"          │  └─►\\t的  \\tassm     \\n\",\n      \"       ┌─►└─────\\t希望 \\ttop      \\n\",\n      \"┌┬─────┴────────\\t是  \\troot     \\n\",\n      \"│└─►┌───────────\\t希望 \\tccomp    \\n\",\n      \"│   │     ┌─►┌──\\t张晚霞\\tassmod   \\n\",\n      \"│   │     │  └─►\\t的  \\tassm     \\n\",\n      \"│   │  ┌─►└─────\\t背影 \\tnsubjpass\\n\",\n      \"│   └─►└──┬─────\\t被  \\tccomp    \\n\",\n      \"│         │  ┌─►\\t晚霞 \\tnsubj    \\n\",\n      \"│         └─►└──\\t映红 \\tdep      \\n\",\n      \"└──────────────►\\t。  \\tpunct    \\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"HanLP(tokens=[\\n\",\n    \"    [\\\"HanLP\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"],\\n\",\n    \"    [\\\"我\\\", \\\"的\\\", \\\"希望\\\", \\\"是\\\", \\\"希望\\\", \\\"张晚霞\\\", \\\"的\\\", \\\"背影\\\", \\\"被\\\", \\\"晚霞\\\", \\\"映红\\\", \\\"。\\\"]\\n\",\n    \"  ], tasks='dep').pretty_print()\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"dep_restful.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/dep_stl.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/dep_stl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fdep_stl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 加载模型\\n\",\n    \"HanLP的工作流程是先加载模型，模型的标示符存储在`hanlp.pretrained`这个包中，按照NLP任务归类。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"4M7ka0K5OMWU\",\n    \"outputId\": \"69cdad22-d94d-41fb-9591-1c29515a3da9\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'CTB5_BIAFFINE_DEP_ZH': 'https://file.hankcs.com/hanlp/dep/biaffine_ctb5_20191229_025833.zip',\\n\",\n       \" 'CTB7_BIAFFINE_DEP_ZH': 'https://file.hankcs.com/hanlp/dep/biaffine_ctb7_20200109_022431.zip',\\n\",\n       \" 'CTB9_DEP_ELECTRA_SMALL': 'https://file.hankcs.com/hanlp/dep/ctb9_dep_electra_small_20220216_100306.zip',\\n\",\n       \" 'PMT1_DEP_ELECTRA_SMALL': 'https://file.hankcs.com/hanlp/dep/pmt_dep_electra_small_20220218_134518.zip',\\n\",\n       \" 'CTB9_UDC_ELECTRA_SMALL': 'https://file.hankcs.com/hanlp/dep/udc_dep_electra_small_20220218_095452.zip',\\n\",\n       \" 'PTB_BIAFFINE_DEP_EN': 'https://file.hankcs.com/hanlp/dep/ptb_dep_biaffine_20200101_174624.zip'}\"\n      ]\n     },\n     \"execution_count\": 1,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"import hanlp\\n\",\n    \"hanlp.pretrained.dep.ALL # 语种见名称最后一个字段或相应语料库\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"BMW528wGNulM\"\n   },\n   \"source\": [\n    \"调用`hanlp.load`进行加载，模型会自动下载到本地缓存：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"id\": \"0tmKBu7sNAXX\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"dep = hanlp.load(hanlp.pretrained.dep.CTB9_DEP_ELECTRA_SMALL)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 依存句法分析\\n\",\n    \"依存句法分析任务的输入为已分词的一个或多个句子：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {\n    \"id\": \"BqEmDMGGOtk3\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"tree = dep([\\\"2021年\\\", \\\"HanLPv2.1\\\", \\\"带来\\\", \\\"次\\\", \\\"世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多\\\", \\\"语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"])\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"jj1Jk-2sPHYx\"\n   },\n   \"source\": [\n    \"返回对象为[CoNLLSentence](https://hanlp.hankcs.com/docs/api/common/conll.html#hanlp_common.conll.CoNLLSentence)类型：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"U_PGm06m6K20\",\n    \"outputId\": \"a25c6452-5032-42b3-d501-99158380c487\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"[{'id': 1,\\n\",\n       \"  'form': '2021年',\\n\",\n       \"  'cpos': None,\\n\",\n       \"  'pos': None,\\n\",\n       \"  'head': 3,\\n\",\n       \"  'deprel': 'tmod',\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'phead': None,\\n\",\n       \"  'pdeprel': None},\\n\",\n       \" {'id': 2,\\n\",\n       \"  'form': 'HanLPv2.1',\\n\",\n       \"  'cpos': None,\\n\",\n       \"  'pos': None,\\n\",\n       \"  'head': 3,\\n\",\n       \"  'deprel': 'nsubj',\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'phead': None,\\n\",\n       \"  'pdeprel': None},\\n\",\n       \" {'id': 3,\\n\",\n       \"  'form': '带来',\\n\",\n       \"  'cpos': None,\\n\",\n       \"  'pos': None,\\n\",\n       \"  'head': 0,\\n\",\n       \"  'deprel': 'root',\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'phead': None,\\n\",\n       \"  'pdeprel': None},\\n\",\n       \" {'id': 4,\\n\",\n       \"  'form': '次',\\n\",\n       \"  'cpos': None,\\n\",\n       \"  'pos': None,\\n\",\n       \"  'head': 5,\\n\",\n       \"  'deprel': 'det',\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'phead': None,\\n\",\n       \"  'pdeprel': None},\\n\",\n       \" {'id': 5,\\n\",\n       \"  'form': '世代',\\n\",\n       \"  'cpos': None,\\n\",\n       \"  'pos': None,\\n\",\n       \"  'head': 7,\\n\",\n       \"  'deprel': 'dep',\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'phead': None,\\n\",\n       \"  'pdeprel': None},\\n\",\n       \" {'id': 6,\\n\",\n       \"  'form': '最',\\n\",\n       \"  'cpos': None,\\n\",\n       \"  'pos': None,\\n\",\n       \"  'head': 7,\\n\",\n       \"  'deprel': 'advmod',\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'phead': None,\\n\",\n       \"  'pdeprel': None},\\n\",\n       \" {'id': 7,\\n\",\n       \"  'form': '先进',\\n\",\n       \"  'cpos': None,\\n\",\n       \"  'pos': None,\\n\",\n       \"  'head': 12,\\n\",\n       \"  'deprel': 'rcmod',\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'phead': None,\\n\",\n       \"  'pdeprel': None},\\n\",\n       \" {'id': 8,\\n\",\n       \"  'form': '的',\\n\",\n       \"  'cpos': None,\\n\",\n       \"  'pos': None,\\n\",\n       \"  'head': 7,\\n\",\n       \"  'deprel': 'cpm',\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'phead': None,\\n\",\n       \"  'pdeprel': None},\\n\",\n       \" {'id': 9,\\n\",\n       \"  'form': '多',\\n\",\n       \"  'cpos': None,\\n\",\n       \"  'pos': None,\\n\",\n       \"  'head': 10,\\n\",\n       \"  'deprel': 'nummod',\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'phead': None,\\n\",\n       \"  'pdeprel': None},\\n\",\n       \" {'id': 10,\\n\",\n       \"  'form': '语种',\\n\",\n       \"  'cpos': None,\\n\",\n       \"  'pos': None,\\n\",\n       \"  'head': 12,\\n\",\n       \"  'deprel': 'nn',\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'phead': None,\\n\",\n       \"  'pdeprel': None},\\n\",\n       \" {'id': 11,\\n\",\n       \"  'form': 'NLP',\\n\",\n       \"  'cpos': None,\\n\",\n       \"  'pos': None,\\n\",\n       \"  'head': 12,\\n\",\n       \"  'deprel': 'nn',\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'phead': None,\\n\",\n       \"  'pdeprel': None},\\n\",\n       \" {'id': 12,\\n\",\n       \"  'form': '技术',\\n\",\n       \"  'cpos': None,\\n\",\n       \"  'pos': None,\\n\",\n       \"  'head': 3,\\n\",\n       \"  'deprel': 'dobj',\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'phead': None,\\n\",\n       \"  'pdeprel': None},\\n\",\n       \" {'id': 13,\\n\",\n       \"  'form': '。',\\n\",\n       \"  'cpos': None,\\n\",\n       \"  'pos': None,\\n\",\n       \"  'head': 3,\\n\",\n       \"  'deprel': 'punct',\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'phead': None,\\n\",\n       \"  'pdeprel': None}]\"\n      ]\n     },\n     \"execution_count\": 4,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"tree\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"Gn_RQa_Z6K20\"\n   },\n   \"source\": [\n    \"打印时为CoNLL格式：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"26P1LGzv6K20\",\n    \"outputId\": \"c78ffdb0-3cd7-492d-f55e-0d50120faffb\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"1\\t2021年\\t_\\t_\\t_\\t_\\t3\\ttmod\\t_\\t_\\n\",\n      \"2\\tHanLPv2.1\\t_\\t_\\t_\\t_\\t3\\tnsubj\\t_\\t_\\n\",\n      \"3\\t带来\\t_\\t_\\t_\\t_\\t0\\troot\\t_\\t_\\n\",\n      \"4\\t次\\t_\\t_\\t_\\t_\\t5\\tdet\\t_\\t_\\n\",\n      \"5\\t世代\\t_\\t_\\t_\\t_\\t7\\tdep\\t_\\t_\\n\",\n      \"6\\t最\\t_\\t_\\t_\\t_\\t7\\tadvmod\\t_\\t_\\n\",\n      \"7\\t先进\\t_\\t_\\t_\\t_\\t12\\trcmod\\t_\\t_\\n\",\n      \"8\\t的\\t_\\t_\\t_\\t_\\t7\\tcpm\\t_\\t_\\n\",\n      \"9\\t多\\t_\\t_\\t_\\t_\\t10\\tnummod\\t_\\t_\\n\",\n      \"10\\t语种\\t_\\t_\\t_\\t_\\t12\\tnn\\t_\\t_\\n\",\n      \"11\\tNLP\\t_\\t_\\t_\\t_\\t12\\tnn\\t_\\t_\\n\",\n      \"12\\t技术\\t_\\t_\\t_\\t_\\t3\\tdobj\\t_\\t_\\n\",\n      \"13\\t。\\t_\\t_\\t_\\t_\\t3\\tpunct\\t_\\t_\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(tree)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"如果不需要CoNLL格式的话，也许`conll=False`时的输出更加简洁：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"[(3, 'tmod'),\\n\",\n       \" (3, 'nsubj'),\\n\",\n       \" (0, 'root'),\\n\",\n       \" (5, 'det'),\\n\",\n       \" (7, 'dep'),\\n\",\n       \" (7, 'advmod'),\\n\",\n       \" (12, 'rcmod'),\\n\",\n       \" (7, 'cpm'),\\n\",\n       \" (10, 'nummod'),\\n\",\n       \" (12, 'nn'),\\n\",\n       \" (12, 'nn'),\\n\",\n       \" (3, 'dobj'),\\n\",\n       \" (3, 'punct')]\"\n      ]\n     },\n     \"execution_count\": 6,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"dep([\\\"2021年\\\", \\\"HanLPv2.1\\\", \\\"带来\\\", \\\"次\\\", \\\"世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多\\\", \\\"语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"], conll=False)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### 可视化\\n\",\n    \"你可以构造一个`Document`实现漂亮的可视化：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Dep&nbsp;Tree&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────────&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌──►&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│┌─►&nbsp;<br>┌┬───────┴┴──&nbsp;<br>││&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>││&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌─►└──&nbsp;<br>││&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;┌─►&nbsp;<br>││&nbsp;&nbsp;┌─►└──┼──&nbsp;<br>││&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;└─►&nbsp;<br>││&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>││&nbsp;&nbsp;│&nbsp;&nbsp;┌─►└──&nbsp;<br>││&nbsp;&nbsp;│&nbsp;&nbsp;│&nbsp;&nbsp;┌─►&nbsp;<br>│└─►└──┴──┴──&nbsp;<br>└───────────►&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Token&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>带来&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先进&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>的&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>语种&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技术&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Relati<br>──────<br>tmod&nbsp;&nbsp;<br>nsubj&nbsp;<br>root&nbsp;&nbsp;<br>det&nbsp;&nbsp;&nbsp;<br>dep&nbsp;&nbsp;&nbsp;<br>advmod<br>rcmod&nbsp;<br>cpm&nbsp;&nbsp;&nbsp;<br>nummod<br>nn&nbsp;&nbsp;&nbsp;&nbsp;<br>nn&nbsp;&nbsp;&nbsp;&nbsp;<br>dobj&nbsp;&nbsp;<br>punct&nbsp;</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"from hanlp_common.document import Document\\n\",\n    \"doc = Document(\\n\",\n    \"    tok=[\\\"2021年\\\", \\\"HanLPv2.1\\\", \\\"带来\\\", \\\"次\\\", \\\"世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多\\\", \\\"语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"],\\n\",\n    \"    dep=[(3, 'tmod'), (3, 'nsubj'), (0, 'root'), (5, 'det'), (7, 'dep'), (7, 'advmod'), (12, 'rcmod'), (7, 'cpm'), (10, 'nummod'), (12, 'nn'), (12, 'nn'), (3, 'dobj'), (3, 'punct')]\\n\",\n    \")\\n\",\n    \"doc.pretty_print()\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"dep_stl.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/extractive_summarization_restful.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/extractive_summarization_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fextractive_summarization_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp_restful -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 创建客户端\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"4M7ka0K5OMWU\",\n    \"outputId\": \"d74f0749-0587-454a-d7c9-7418d45ce534\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from hanlp_restful import HanLPClient\\n\",\n    \"HanLP = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh') # auth不填则匿名，zh中文，mul多语种\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"BMW528wGNulM\"\n   },\n   \"source\": [\n    \"#### 申请秘钥\\n\",\n    \"由于服务器算力有限，匿名用户每分钟限2次调用。如果你需要更多调用次数，[建议申请免费公益API秘钥auth](https://bbs.hanlp.com/t/hanlp2-1-restful-api/53)。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 抽取式自动摘要\\n\",\n    \"抽取式自动摘要的目标是从文章中筛选出一些作为摘要的中心句子：既要紧扣要点，又要避免赘语。\\n\",\n    \"### 中文\\n\",\n    \"抽取式自动摘要任务的输入为一段文本和所需的摘要句子数量的最大值`topk`：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"936d439a-e1ff-4308-d2aa-775955558594\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'据DigiTimes报道，在上海疫情趋缓，防疫管控开始放松后，苹果供应商广达正在逐步恢复其中国工厂的MacBook产品生产。': 0.9999685883522034,\\n\",\n       \" '仍有许多苹果笔记本用户在等待3月和4月订购的MacBook Pro机型到货，由于苹果的供应问题，他们的发货时间被大大推迟了。': 0.5798477530479431,\\n\",\n       \" '尽管MacBook Pro的生产逐渐恢复，但供应问题预计依然影响2022年第三季度的产品销售。': 0.5435440540313721}\"\n      ]\n     },\n     \"execution_count\": 2,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"text = '''\\n\",\n    \"据DigiTimes报道，在上海疫情趋缓，防疫管控开始放松后，苹果供应商广达正在逐步恢复其中国工厂的MacBook产品生产。\\n\",\n    \"据供应链消息人士称，生产厂的订单拉动情况正在慢慢转强，这会提高MacBook Pro机型的供应量，并缩短苹果客户在过去几周所经历的延长交货时间。\\n\",\n    \"仍有许多苹果笔记本用户在等待3月和4月订购的MacBook Pro机型到货，由于苹果的供应问题，他们的发货时间被大大推迟了。\\n\",\n    \"据分析师郭明錤表示，广达是高端MacBook Pro的唯一供应商，自防疫封控依赖，MacBook Pro大部分型号交货时间增加了三到五周，\\n\",\n    \"一些高端定制型号的MacBook Pro配置要到6月底到7月初才能交货。\\n\",\n    \"尽管MacBook Pro的生产逐渐恢复，但供应问题预计依然影响2022年第三季度的产品销售。\\n\",\n    \"苹果上周表示，防疫措施和元部件短缺将继续使其难以生产足够的产品来满足消费者的强劲需求，这最终将影响苹果6月份的收入。\\n\",\n    \"'''\\n\",\n    \"HanLP.extractive_summarization(text, topk=3)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"jj1Jk-2sPHYx\"\n   },\n   \"source\": [\n    \"返回值为最多`topk`个摘要句子以及相应的权重，权重取值区间为$[0, 1]$。由于Trigram Blocking技巧，实际返回的摘要句数量可能小于`topk`。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"#### 可视化\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"\\n\",\n       \"<span style=\\\"background-color:rgba(255, 255, 0, 0.9999685883522034);\\\">据DigiTimes报道，在上海疫情趋缓，防疫管控开始放松后，苹果供应商广达正在逐步恢复其中国工厂的MacBook产品生产。</span>\\n\",\n       \"据供应链消息人士称，生产厂的订单拉动情况正在慢慢转强，这会提高MacBook Pro机型的供应量，并缩短苹果客户在过去几周所经历的延长交货时间。\\n\",\n       \"<span style=\\\"background-color:rgba(255, 255, 0, 0.5798477530479431);\\\">仍有许多苹果笔记本用户在等待3月和4月订购的MacBook Pro机型到货，由于苹果的供应问题，他们的发货时间被大大推迟了。</span>\\n\",\n       \"据分析师郭明錤表示，广达是高端MacBook Pro的唯一供应商，自防疫封控依赖，MacBook Pro大部分型号交货时间增加了三到五周，\\n\",\n       \"一些高端定制型号的MacBook Pro配置要到6月底到7月初才能交货。\\n\",\n       \"<span style=\\\"background-color:rgba(255, 255, 0, 0.5435440540313721);\\\">尽管MacBook Pro的生产逐渐恢复，但供应问题预计依然影响2022年第三季度的产品销售。</span>\\n\",\n       \"<span style=\\\"background-color:rgba(255, 255, 0, 0.17781692743301392);\\\">苹果上周表示，防疫措施和元部件短缺将继续使其难以生产足够的产品来满足消费者的强劲需求，这最终将影响苹果6月份的收入。</span>\\n\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"def highlight(text, scores):\\n\",\n    \"    for k, v in scores.items():\\n\",\n    \"        text = text.replace(k, f'<span style=\\\"background-color:rgba(255, 255, 0, {v});\\\">{k}</span>')\\n\",\n    \"    from IPython.display import display, HTML\\n\",\n    \"    display(HTML(text))\\n\",\n    \"\\n\",\n    \"scores = HanLP.extractive_summarization(text, topk=100)\\n\",\n    \"highlight(text, scores)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"#### 繁体中文\\n\",\n    \"HanLP的抽取式自动摘要接口支持繁体中文：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'華爾街日報周二（3日）報導，根據知情人透露，日前已宣布將以440億美元買下推特（Twitter）並下市的馬斯克，曾經跟一些潛在投資人說，他可以在短短幾年後，再將這家社群媒體公司重新上市。': 0.9999818205833435,\\n\",\n       \" '消息來源說，特斯拉創辦人兼執行長馬斯克表示，他計劃在買下推特後最短三年內，就展開推特的首次公開發行股票。': 0.503434419631958,\\n\",\n       \" '根據之前華爾街日報的報導，馬斯克為購買推特籌現金時，與私募股權公司等投資人討論出資事宜，Apollo Global Management有興趣參與。': 0.2688594460487366}\"\n      ]\n     },\n     \"execution_count\": 4,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"text = '''\\n\",\n    \"華爾街日報周二（3日）報導，根據知情人透露，日前已宣布將以440億美元買下推特（Twitter）並下市的馬斯克，曾經跟一些潛在投資人說，他可以在短短幾年後，再將這家社群媒體公司重新上市。\\n\",\n    \"消息來源說，特斯拉創辦人兼執行長馬斯克表示，他計劃在買下推特後最短三年內，就展開推特的首次公開發行股票。\\n\",\n    \"馬斯克買推特的交易案預期在今年稍後走完程序，包括獲得股東同意以及監管機關核准等步驟。\\n\",\n    \"根據之前華爾街日報的報導，馬斯克為購買推特籌現金時，與私募股權公司等投資人討論出資事宜，Apollo Global Management有興趣參與。\\n\",\n    \"私募股權公司通常都先買下公司將之私有化，把公司移出眾人注目的焦點之外以後，整頓公司，接著再把公司上市，時間常是五年左右。\\n\",\n    \"華爾街日報指出，馬斯克暗示他對推特有類似的規劃的話，有助說服潛在投資人，他會很快行動，改善推特的營運和獲利。\\n\",\n    \"'''\\n\",\n    \"scores = HanLP.extractive_summarization(text)\\n\",\n    \"scores\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"\\n\",\n       \"<span style=\\\"background-color:rgba(255, 255, 0, 0.9999818205833435);\\\">華爾街日報周二（3日）報導，根據知情人透露，日前已宣布將以440億美元買下推特（Twitter）並下市的馬斯克，曾經跟一些潛在投資人說，他可以在短短幾年後，再將這家社群媒體公司重新上市。</span>\\n\",\n       \"<span style=\\\"background-color:rgba(255, 255, 0, 0.503434419631958);\\\">消息來源說，特斯拉創辦人兼執行長馬斯克表示，他計劃在買下推特後最短三年內，就展開推特的首次公開發行股票。</span>\\n\",\n       \"馬斯克買推特的交易案預期在今年稍後走完程序，包括獲得股東同意以及監管機關核准等步驟。\\n\",\n       \"<span style=\\\"background-color:rgba(255, 255, 0, 0.2688594460487366);\\\">根據之前華爾街日報的報導，馬斯克為購買推特籌現金時，與私募股權公司等投資人討論出資事宜，Apollo Global Management有興趣參與。</span>\\n\",\n       \"私募股權公司通常都先買下公司將之私有化，把公司移出眾人注目的焦點之外以後，整頓公司，接著再把公司上市，時間常是五年左右。\\n\",\n       \"華爾街日報指出，馬斯克暗示他對推特有類似的規劃的話，有助說服潛在投資人，他會很快行動，改善推特的營運和獲利。\\n\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"highlight(text, scores)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### 英文\\n\",\n    \"按照HanLP一贯的多语种设计，任何语言都支持。由于服务器GPU资源限制，目前英文接口暂未上线。如果你有相应需求，欢迎前往论坛发起请愿。\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"extractive_summarization_restful.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/gec_restful.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/gec_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fgec_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp_restful -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 创建客户端\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"id\": \"0tmKBu7sNAXX\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from hanlp_restful import HanLPClient\\n\",\n    \"HanLP = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh') # auth不填则匿名，zh中文，mul多语种\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"EmZDmLn9aGxG\"\n   },\n   \"source\": [\n    \"#### 申请秘钥\\n\",\n    \"由于服务器算力有限，匿名用户每分钟限2次调用。如果你需要更多调用次数，[建议申请免费公益API秘钥auth](https://bbs.hanlp.com/t/hanlp2-1-restful-api/53)。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 语法纠错\\n\",\n    \"输入短文本，执行语法纠错：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"2a0d392f-b99a-4a18-fc7f-754e2abe2e34\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"['每个青年都应当有远大的抱负。', '有的同学对语言很有兴趣。', '我市本地居民约占全市人口的70%。']\"\n      ]\n     },\n     \"execution_count\": 2,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"HanLP.grammatical_error_correction(['每个青年都应当有远大的报复。', '有的同学对语言很兴趣。', '我市本地居民约占全市人口的70%多。'])\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"返回值是每段短文本的修改结果列表。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## 测试版\\n\",\n    \"当前版本为测试版，暂时仅支持拼写、标点和简单的语法错误，HanLP的线上模型和语料库仍然在迭代发展中。欢迎广大用户将测试版的问题反馈到[论坛](https://bbs.hankcs.com/c/text-generation/gec/30)，我们将在下一个版本中，将HanLP的文本纠错能力提升到高考语文水平。\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"gec_restful.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/keyphrase_restful.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/keyphrase_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fkeyphrase_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp_restful -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 创建客户端\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"4M7ka0K5OMWU\",\n    \"outputId\": \"d74f0749-0587-454a-d7c9-7418d45ce534\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from hanlp_restful import HanLPClient\\n\",\n    \"HanLP = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh') # auth不填则匿名，zh中文，mul多语种\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"BMW528wGNulM\"\n   },\n   \"source\": [\n    \"#### 申请秘钥\\n\",\n    \"由于服务器算力有限，匿名用户每分钟限2次调用。如果你需要更多调用次数，[建议申请免费公益API秘钥auth](https://bbs.hanlp.com/t/hanlp2-1-restful-api/53)。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 关键词提取\\n\",\n    \"关键词（短语）提取的目标是文本中最具有代表性的关键词以及短语。\\n\",\n    \"### 中文\\n\",\n    \"关键词提取任务的输入为一段文本和所需的关键词数量`topk`：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"936d439a-e1ff-4308-d2aa-775955558594\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'自然语言处理': 0.800000011920929,\\n\",\n       \" 'HanLP的全部性能': 0.5256577134132385,\\n\",\n       \" '一门博大精深的学科': 0.42154020071029663}\"\n      ]\n     },\n     \"execution_count\": 2,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"HanLP.keyphrase_extraction('自然语言处理是一门博大精深的学科，掌握理论才能发挥出HanLP的全部性能。 '\\n\",\n    \"                           '《自然语言处理入门》是一本配套HanLP的NLP入门书，助你零起点上手自然语言处理。', topk=3)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"jj1Jk-2sPHYx\"\n   },\n   \"source\": [\n    \"返回值为`topk`个关键词以及相应的权重，权重取值区间为$[0, 1]$。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"关键词提取并不仅限于短文本，长文章也一样支持：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'新冠病毒核酸阳性感染': 0.888239324092865,\\n\",\n       \" '确诊病例': 0.8868124485015869,\\n\",\n       \" '本土无症状感染者': 0.8557102680206299,\\n\",\n       \" '属地社区（村屯）': 0.8164600133895874,\\n\",\n       \" '疫情防控工作': 0.7749382853507996,\\n\",\n       \" '我市疫情防控要求': 0.7502512335777283,\\n\",\n       \" '症状': 0.669366180896759,\\n\",\n       \" '我市疫情形势': 0.6673010587692261,\\n\",\n       \" '感染': 0.6663177013397217,\\n\",\n       \" '本土确诊病例': 0.6464788317680359}\"\n      ]\n     },\n     \"execution_count\": 3,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"doc = '''\\n\",\n    \"4月15日0-24时，长春市新增本土确诊病例157例（含57例无症状感染者转为确诊病例），新增本土无症状感染者407例。\\n\",\n    \"以上人员均为隔离管控期间筛查新冠病毒核酸阳性感染者。\\n\",\n    \"当前我市疫情形势严峻，为做好全市疫情防控工作，尽快恢复正常社会秩序和经济社会发展，长春市新冠肺炎疫情防控工作领导小组办公室提醒广大市民，\\n\",\n    \"请严格遵守我市疫情防控要求，配合各部门落实好防控措施，进一步提高防范意识，坚持规范戴口罩、勤洗手、常通风、保持社交距离、不聚餐、不聚集，\\n\",\n    \"减少疾病感染风险。一旦出现发热、干咳、乏力、咽痛、嗅味觉减退或丧失等不适症状，应及时向属地社区（村屯）或疾控机构报告。\\n\",\n    \"'''\\n\",\n    \"HanLP.keyphrase_extraction(doc)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"#### 可视化\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"\\n\",\n       \"4月15日0-24时，长春市新增本土<span style=\\\"background-color:rgba(255, 255, 0, 0.8868124485015869);\\\">确诊病例</span>157例（含57例无<span style=\\\"background-color:rgba(255, 255, 0, 0.669366180896759);\\\">症状</span><span style=\\\"background-color:rgba(255, 255, 0, 0.6663177013397217);\\\">感染</span>者转为<span style=\\\"background-color:rgba(255, 255, 0, 0.8868124485015869);\\\">确诊病例</span>），新增<span style=\\\"background-color:rgba(255, 255, 0, 0.8557102680206299);\\\">本土无<span style=\\\"background-color:rgba(255, 255, 0, 0.669366180896759);\\\">症状</span><span style=\\\"background-color:rgba(255, 255, 0, 0.6663177013397217);\\\">感染</span>者</span>407例。\\n\",\n       \"以上人员均为隔离管控期间筛查<span style=\\\"background-color:rgba(255, 255, 0, 0.888239324092865);\\\">新冠病毒核酸阳性<span style=\\\"background-color:rgba(255, 255, 0, 0.6663177013397217);\\\">感染</span></span>者。\\n\",\n       \"当前<span style=\\\"background-color:rgba(255, 255, 0, 0.6673010587692261);\\\">我市疫情形势</span>严峻，为做好全市<span style=\\\"background-color:rgba(255, 255, 0, 0.7749382853507996);\\\">疫情防控工作</span>，尽快恢复正常社会秩序和经济社会发展，长春市新冠肺炎<span style=\\\"background-color:rgba(255, 255, 0, 0.7749382853507996);\\\">疫情防控工作</span>领导小组办公室提醒广大市民，\\n\",\n       \"请严格遵守<span style=\\\"background-color:rgba(255, 255, 0, 0.7502512335777283);\\\">我市疫情防控要求</span>，配合各部门落实好防控措施，进一步提高防范意识，坚持规范戴口罩、勤洗手、常通风、保持社交距离、不聚餐、不聚集，\\n\",\n       \"减少疾病<span style=\\\"background-color:rgba(255, 255, 0, 0.6663177013397217);\\\">感染</span>风险。一旦出现发热、干咳、乏力、咽痛、嗅味觉减退或丧失等不适<span style=\\\"background-color:rgba(255, 255, 0, 0.669366180896759);\\\">症状</span>，应及时向<span style=\\\"background-color:rgba(255, 255, 0, 0.8164600133895874);\\\">属地社区（村屯）</span>或疾控机构报告。\\n\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"def highlight(text, scores):\\n\",\n    \"    for k, v in scores.items():\\n\",\n    \"        text = text.replace(k, f'<span style=\\\"background-color:rgba(255, 255, 0, {v});\\\">{k}</span>')\\n\",\n    \"    from IPython.display import display, HTML\\n\",\n    \"    display(HTML(text))\\n\",\n    \"\\n\",\n    \"scores = HanLP.keyphrase_extraction(doc)\\n\",\n    \"highlight(doc, scores)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### 英文\\n\",\n    \"按照HanLP一贯的多语种设计，任何语言都支持。由于服务器GPU资源限制，目前英文接口暂未上线。如果你有相应需求，欢迎前往论坛发起请愿。\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"keyphrase_restful.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/lid_restful.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/lid_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Flid_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"nf9TgeCTC0OT\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"jaW4eu6kC0OU\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp_restful -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"_xI_bLAaC0OU\"\n   },\n   \"source\": [\n    \"## 创建客户端\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"IYwV-UkNNzFp\",\n    \"outputId\": \"54065443-9b0a-444c-f6c0-c701bc86400b\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from hanlp_restful import HanLPClient\\n\",\n    \"HanLP = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh') # auth不填则匿名，zh中文，mul多语种\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\",\n    \"pycharm\": {\n     \"name\": \"#%% md\\n\"\n    }\n   },\n   \"source\": [\n    \"#### 申请秘钥\\n\",\n    \"由于服务器算力有限，匿名用户每分钟限2次调用。如果你需要更多调用次数，[建议申请免费公益API秘钥auth](https://bbs.hanlp.com/t/hanlp2-1-restful-api/53)。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 语种识别\\n\",\n    \"语种识别任务的输入为一个或多个文档：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"id\": \"BqEmDMGGOtk3\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"'en'\"\n      ]\n     },\n     \"execution_count\": 2,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"HanLP.language_identification('In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques to production environments.')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"SwaPn1hjC0OW\"\n   },\n   \"source\": [\n    \"返回对象为[ISO 639-1编码](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes)。HanLP支持返回语种对应的概率（置信度）：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"egpWwHKxC0OX\",\n    \"outputId\": \"f7c77687-dd75-4fa2-dbd2-be6bda8a3fff\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"['ja', 0.9976244568824768]\"\n      ]\n     },\n     \"execution_count\": 3,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"HanLP.language_identification('2021年、HanLPv2.1は次世代の最先端多言語NLP技術を本番環境に導入します。', prob=True)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"kq_j5TLFC0OX\"\n   },\n   \"source\": [\n    \"HanLP也支持返回概率最高的`topk`个语种：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"isJhzYyIC0OX\",\n    \"outputId\": \"683c8489-dffc-426e-f95b-e91dfb373260\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"['zh', 'ja']\"\n      ]\n     },\n     \"execution_count\": 4,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"HanLP.language_identification('2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', topk=2)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"该功能对于混合了多个语种的文档而言特别实用：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'zh': 0.3952908217906952,\\n\",\n       \" 'en': 0.37189167737960815,\\n\",\n       \" 'ja': 0.056213412433862686}\"\n      ]\n     },\n     \"execution_count\": 5,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"text = '''\\n\",\n    \"2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。\\n\",\n    \"In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques to production environments.\\n\",\n    \"'''\\n\",\n    \"\\n\",\n    \"HanLP.language_identification(text, topk=3, prob=True)\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"lid_restful.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/lid_stl.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/lid_stl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Flid_stl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"nf9TgeCTC0OT\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"jaW4eu6kC0OU\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp[fasttext] -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"_xI_bLAaC0OU\"\n   },\n   \"source\": [\n    \"## 加载模型\\n\",\n    \"HanLP的工作流程是先加载模型，模型的标示符存储在`hanlp.pretrained`这个包中，按照NLP任务归类。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"IYwV-UkNNzFp\",\n    \"outputId\": \"54065443-9b0a-444c-f6c0-c701bc86400b\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'CHNSENTICORP_BERT_BASE_ZH': 'https://file.hankcs.com/hanlp/classification/chnsenticorp_bert_base_20211228_163210.zip',\\n\",\n       \" 'SST2_ALBERT_BASE_EN': 'https://file.hankcs.com/hanlp/classification/sst2_albert_base_20211228_164917.zip',\\n\",\n       \" 'LID_176_FASTTEXT_BASE': 'https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.bin',\\n\",\n       \" 'LID_176_FASTTEXT_SMALL': 'https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.ftz'}\"\n      ]\n     },\n     \"execution_count\": 1,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"import hanlp\\n\",\n    \"hanlp.pretrained.classifiers.ALL # 任务见第一个字段\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\",\n    \"pycharm\": {\n     \"name\": \"#%% md\\n\"\n    }\n   },\n   \"source\": [\n    \"调用`hanlp.load`进行加载，模型会自动下载到本地缓存。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stderr\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Warning : `load_model` does not return WordVectorModel or SupervisedModel any more, but a `FastText` object which is very similar.\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"lid = hanlp.load('LID_176_FASTTEXT_BASE')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 语种识别\\n\",\n    \"语种识别任务的输入为一个或多个文档：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {\n    \"id\": \"BqEmDMGGOtk3\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"'en'\"\n      ]\n     },\n     \"execution_count\": 3,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"lid('In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques to production environments.')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"SwaPn1hjC0OW\"\n   },\n   \"source\": [\n    \"返回对象为[ISO 639-1编码](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes)。HanLP支持返回语种对应的概率（置信度）：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"egpWwHKxC0OX\",\n    \"outputId\": \"f7c77687-dd75-4fa2-dbd2-be6bda8a3fff\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"('ja', 0.9976244568824768)\"\n      ]\n     },\n     \"execution_count\": 4,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"lid('2021年、HanLPv2.1は次世代の最先端多言語NLP技術を本番環境に導入します。', prob=True)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"kq_j5TLFC0OX\"\n   },\n   \"source\": [\n    \"HanLP也支持返回概率最高的`topk`个语种：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"isJhzYyIC0OX\",\n    \"outputId\": \"683c8489-dffc-426e-f95b-e91dfb373260\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"['zh', 'ja']\"\n      ]\n     },\n     \"execution_count\": 5,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"lid('2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', topk=2)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"该功能对于混合了多个语种的文档而言特别实用：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'zh': 0.3952908217906952,\\n\",\n       \" 'en': 0.37189167737960815,\\n\",\n       \" 'ja': 0.056213412433862686}\"\n      ]\n     },\n     \"execution_count\": 6,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"text = '''\\n\",\n    \"2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。\\n\",\n    \"In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques to production environments.\\n\",\n    \"'''\\n\",\n    \"\\n\",\n    \"lid(text, topk=3, prob=True)\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"lid_stl.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/ner_mtl.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/ner_mtl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fner_mtl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\",\n    \"pycharm\": {\n     \"name\": \"#%% md\\n\"\n    }\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"0tmKBu7sNAXX\",\n    \"pycharm\": {\n     \"name\": \"#%% md\\n\"\n    }\n   },\n   \"source\": [\n    \"## 加载模型\\n\",\n    \"HanLP的工作流程是先加载模型，模型的标示符存储在`hanlp.pretrained`这个包中，按照NLP任务归类。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"EmZDmLn9aGxG\",\n    \"outputId\": \"38469cbe-d56c-4648-b103-b67e6d22aeff\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'OPEN_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/mtl/open_tok_pos_ner_srl_dep_sdp_con_electra_small_20201223_035557.zip',\\n\",\n       \" 'OPEN_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH': 'https://file.hankcs.com/hanlp/mtl/open_tok_pos_ner_srl_dep_sdp_con_electra_base_20201223_201906.zip',\\n\",\n       \" 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_small_20210111_124159.zip',\\n\",\n       \" 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_base_20210111_124519.zip',\\n\",\n       \" 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ERNIE_GRAM_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_ernie_gram_base_aug_20210904_145403.zip',\\n\",\n       \" 'UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_MT5_SMALL': 'https://file.hankcs.com/hanlp/mtl/ud_ontonotes_tok_pos_lem_fea_ner_srl_dep_sdp_con_mt5_small_20210228_123458.zip',\\n\",\n       \" 'UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_XLMR_BASE': 'https://file.hankcs.com/hanlp/mtl/ud_ontonotes_tok_pos_lem_fea_ner_srl_dep_sdp_con_xlm_base_20210602_211620.zip',\\n\",\n       \" 'NPCMJ_UD_KYOTO_TOK_POS_CON_BERT_BASE_CHAR_JA': 'https://file.hankcs.com/hanlp/mtl/npcmj_ud_kyoto_tok_pos_ner_dep_con_srl_bert_base_char_ja_20210914_133742.zip'}\"\n      ]\n     },\n     \"execution_count\": 3,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"import hanlp\\n\",\n    \"hanlp.pretrained.mtl.ALL # MTL多任务，具体任务见模型名称，语种见名称最后一个字段或相应语料库\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"w0lm87NUsMwW\"\n   },\n   \"source\": [\n    \"调用`hanlp.load`进行加载，模型会自动下载到本地缓存。自然语言处理分为许多任务，分词只是最初级的一个。与其每个任务单独创建一个模型，不如利用HanLP的联合模型一次性完成多个任务：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {\n    \"id\": \"6Evnxsa0sMwW\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"HanLP = hanlp.load(hanlp.pretrained.mtl.CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"bPUHdNJ-sMwW\"\n   },\n   \"source\": [\n    \"## 命名实体识别\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"wxctCigrTKu-\"\n   },\n   \"source\": [\n    \"同时执行所有标准的命名实体识别：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"Zo08uquCTFSk\",\n    \"outputId\": \"21be671b-ead0-43c9-cc3a-32c305d8be29\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"{\\n\",\n      \"  \\\"tok/fine\\\": [\\n\",\n      \"    [\\\"2021年\\\", \\\"HanLPv2.1\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次\\\", \\\"世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多\\\", \\\"语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"],\\n\",\n      \"    [\\\"阿婆主\\\", \\\"来到\\\", \\\"北京\\\", \\\"立方庭\\\", \\\"参观\\\", \\\"自然\\\", \\\"语义\\\", \\\"科技\\\", \\\"公司\\\", \\\"。\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"ner/msra\\\": [\\n\",\n      \"    [[\\\"2021年\\\", \\\"DATE\\\", 0, 1], [\\\"HanLPv2.1\\\", \\\"WWW\\\", 1, 2]],\\n\",\n      \"    [[\\\"北京\\\", \\\"LOCATION\\\", 2, 3], [\\\"立方庭\\\", \\\"LOCATION\\\", 3, 4], [\\\"自然语义科技公司\\\", \\\"ORGANIZATION\\\", 5, 9]]\\n\",\n      \"  ],\\n\",\n      \"  \\\"ner/pku\\\": [\\n\",\n      \"    [],\\n\",\n      \"    [[\\\"北京立方庭\\\", \\\"ns\\\", 2, 4], [\\\"自然语义科技公司\\\", \\\"nt\\\", 5, 9]]\\n\",\n      \"  ],\\n\",\n      \"  \\\"ner/ontonotes\\\": [\\n\",\n      \"    [[\\\"2021年\\\", \\\"DATE\\\", 0, 1], [\\\"HanLPv2.1\\\", \\\"ORG\\\", 1, 2]],\\n\",\n      \"    [[\\\"北京立方庭\\\", \\\"FAC\\\", 2, 4], [\\\"自然语义科技公司\\\", \\\"ORG\\\", 5, 9]]\\n\",\n      \"  ]\\n\",\n      \"}\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(HanLP(['2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', '阿婆主来到北京立方庭参观自然语义科技公司。'], tasks='ner*'))\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"每个四元组表示`[命名实体, 类型标签, 起始下标, 终止下标]`，下标指的是命名实体在单词数组中的下标，单词数组默认为第一个以`tok`开头的数组。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"cqEWnj_7p2Lf\"\n   },\n   \"source\": [\n    \"任务越少，速度越快。如指定仅执行命名实体识别，默认MSRA标准：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 572\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"33790ca9-7013-456f-c1cb-e5ddce90a457\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Token    \\tNER Type        \\n\",\n      \"─────────\\t────────────────\\n\",\n      \"2021年    \\t───►DATE        \\n\",\n      \"HanLPv2.1\\t───►WWW         \\n\",\n      \"为        \\t                \\n\",\n      \"生产       \\t                \\n\",\n      \"环境       \\t                \\n\",\n      \"带来       \\t                \\n\",\n      \"次世代      \\t───►DATE        \\n\",\n      \"最        \\t                \\n\",\n      \"先进       \\t                \\n\",\n      \"的        \\t                \\n\",\n      \"多        \\t                \\n\",\n      \"语种       \\t                \\n\",\n      \"NLP      \\t                \\n\",\n      \"技术       \\t                \\n\",\n      \"。        \\t                \\n\",\n      \"阿婆主      \\t                \\n\",\n      \"来到       \\t                \\n\",\n      \"北京       \\t◄─┐             \\n\",\n      \"立方庭      \\t◄─┴►ORGANIZATION\\n\",\n      \"参观       \\t                \\n\",\n      \"自然       \\t◄─┐             \\n\",\n      \"语义       \\t  │             \\n\",\n      \"科技       \\t  ├►ORGANIZATION\\n\",\n      \"公司       \\t◄─┘             \\n\",\n      \"。        \\t                \\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"HanLP('2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。阿婆主来到北京立方庭参观自然语义科技公司。', tasks='ner').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"jj1Jk-2sPHYx\"\n   },\n   \"source\": [\n    \"执行OntoNotes命名实体识别：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 572\n    },\n    \"id\": \"1goEC7znPNkI\",\n    \"outputId\": \"2a97331c-a5fb-4d3c-ccf2-ce2186616c57\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Token    \\tNER Type\\n\",\n      \"─────────\\t────────\\n\",\n      \"2021年    \\t───►DATE\\n\",\n      \"HanLPv2.1\\t───►ORG \\n\",\n      \"为        \\t        \\n\",\n      \"生产       \\t        \\n\",\n      \"环境       \\t        \\n\",\n      \"带来       \\t        \\n\",\n      \"次世代      \\t        \\n\",\n      \"最        \\t        \\n\",\n      \"先进       \\t        \\n\",\n      \"的        \\t        \\n\",\n      \"多        \\t        \\n\",\n      \"语种       \\t        \\n\",\n      \"NLP      \\t        \\n\",\n      \"技术       \\t        \\n\",\n      \"。        \\t        \\n\",\n      \"阿婆主      \\t        \\n\",\n      \"来到       \\t        \\n\",\n      \"北京       \\t◄─┐     \\n\",\n      \"立方庭      \\t◄─┴►ORG \\n\",\n      \"参观       \\t        \\n\",\n      \"自然       \\t◄─┐     \\n\",\n      \"语义       \\t  │     \\n\",\n      \"科技       \\t  ├►ORG \\n\",\n      \"公司       \\t◄─┘     \\n\",\n      \"。        \\t        \\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"HanLP('2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。阿婆主来到北京立方庭参观自然语义科技公司。', tasks='ner/ontonotes').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"#### 注意\\n\",\n    \"Native API的输入单位限定为句子，需使用[多语种分句模型](https://github.com/hankcs/HanLP/blob/master/plugins/hanlp_demo/hanlp_demo/sent_split.py)或[基于规则的分句函数](https://github.com/hankcs/HanLP/blob/master/hanlp/utils/rules.py#L19)先行分句。RESTful同时支持全文、句子、已分词的句子。除此之外，RESTful和native两种API的语义设计完全一致，用户可以无缝互换。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"P7CNTDBRsiYa\"\n   },\n   \"source\": [\n    \"## 自定义词典\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"ZXtRTXlBsmtw\"\n   },\n   \"source\": [\n    \"自定义词典是NER任务的成员变量，要操作自定义词典，先获取一个NER任务。以MSRA为例：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 8,\n   \"metadata\": {\n    \"id\": \"QgY22h0AszsA\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"ner = HanLP['ner/msra']\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"_6fPzuyps98H\"\n   },\n   \"source\": [\n    \"### 白名单词典\\n\",\n    \"白名单词典中的词语会尽量被输出。当然，HanLP以统计为主，词典的优先级很低。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 9,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 321\n    },\n    \"id\": \"plNDyWhws5qg\",\n    \"outputId\": \"7120d400-022c-42e9-fca9-febe3745d2c9\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Token\\tNER Type   \\n\",\n      \"─────\\t───────────\\n\",\n      \"2021年\\t───►DATE   \\n\",\n      \"测试   \\t           \\n\",\n      \"高血压  \\t           \\n\",\n      \"是    \\t           \\n\",\n      \"138  \\t───►INTEGER\\n\",\n      \"，    \\t           \\n\",\n      \"时间   \\t           \\n\",\n      \"是    \\t           \\n\",\n      \"午饭   \\t◄─┐        \\n\",\n      \"后    \\t◄─┴►TIME   \\n\",\n      \"2点45 \\t───►TIME   \\n\",\n      \"，    \\t           \\n\",\n      \"低血压  \\t           \\n\",\n      \"是    \\t           \\n\",\n      \"44   \\t───►INTEGER\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"ner.dict_whitelist = {'午饭后': 'TIME'}\\n\",\n    \"doc = HanLP('2021年测试高血压是138，时间是午饭后2点45，低血压是44', tasks='ner/msra')\\n\",\n    \"doc.pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"aR_8TICmtw_E\"\n   },\n   \"source\": [\n    \"### 强制词典\\n\",\n    \"如果你读过[《自然语言处理入门》](http://nlp.hankcs.com/book.php)，你就会理解BMESO标注集，于是你可以直接干预统计模型预测的标签，拿到最高优先级的权限。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 10,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 268\n    },\n    \"id\": \"sWPljj3stsEA\",\n    \"outputId\": \"99c4c281-a5b6-46bb-dffd-c1722fee7aee\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"To\\tNER Type    \\n\",\n      \"──\\t────────────\\n\",\n      \"他 \\t            \\n\",\n      \"在 \\t            \\n\",\n      \"浙江\\t───►LOCATION\\n\",\n      \"金华\\t───►LOCATION\\n\",\n      \"出生\\t            \\n\",\n      \"， \\t            \\n\",\n      \"他 \\t            \\n\",\n      \"的 \\t            \\n\",\n      \"名字\\t            \\n\",\n      \"叫 \\t            \\n\",\n      \"金华\\t───►PERSON  \\n\",\n      \"。 \\t            \\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"ner.dict_tags = {('名字', '叫', '金华'): ('O', 'O', 'S-PERSON')}\\n\",\n    \"HanLP('他在浙江金华出生，他的名字叫金华。', tasks='ner/msra').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"fkTC0GFxtinZ\"\n   },\n   \"source\": [\n    \"### 黑名单词典\\n\",\n    \"黑名单中的词语绝对不会被当做命名实体。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 11,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 268\n    },\n    \"id\": \"bIJpgdGauLJK\",\n    \"outputId\": \"e74ec7ba-00fd-4958-d772-a1d1c40d1033\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"To\\tNER Type    \\n\",\n      \"──\\t────────────\\n\",\n      \"他 \\t            \\n\",\n      \"在 \\t            \\n\",\n      \"浙江\\t───►LOCATION\\n\",\n      \"金华\\t            \\n\",\n      \"出生\\t            \\n\",\n      \"， \\t            \\n\",\n      \"他 \\t            \\n\",\n      \"的 \\t            \\n\",\n      \"名字\\t            \\n\",\n      \"叫 \\t            \\n\",\n      \"金华\\t            \\n\",\n      \"。 \\t            \\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"ner.dict_blacklist = {'金华'}\\n\",\n    \"HanLP('他在浙江金华出生，他的名字叫金华。', tasks='ner/msra').pretty_print()\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"ner_mtl.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/ner_restful.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/ner_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fner_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"pip install hanlp_restful -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 创建客户端\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"id\": \"0tmKBu7sNAXX\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from hanlp_restful import HanLPClient\\n\",\n    \"HanLP = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh') # auth不填则匿名，zh中文，mul多语种\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"EmZDmLn9aGxG\"\n   },\n   \"source\": [\n    \"#### 申请秘钥\\n\",\n    \"由于服务器算力有限，匿名用户每分钟限2次调用。如果你需要更多调用次数，[建议申请免费公益API秘钥auth](https://bbs.hanlp.com/t/hanlp2-1-restful-api/53)。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 命名实体识别\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"wxctCigrTKu-\"\n   },\n   \"source\": [\n    \"同时执行所有标准的命名实体识别：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"Zo08uquCTFSk\",\n    \"outputId\": \"21be671b-ead0-43c9-cc3a-32c305d8be29\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"{\\n\",\n      \"  \\\"tok/fine\\\": [\\n\",\n      \"    [\\\"2021年\\\", \\\"HanLPv2.1\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次\\\", \\\"世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多\\\", \\\"语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"],\\n\",\n      \"    [\\\"阿婆主\\\", \\\"来到\\\", \\\"北京\\\", \\\"立方庭\\\", \\\"参观\\\", \\\"自然\\\", \\\"语义\\\", \\\"科技\\\", \\\"公司\\\", \\\"。\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"ner/msra\\\": [\\n\",\n      \"    [[\\\"2021年\\\", \\\"DATE\\\", 0, 1], [\\\"HanLPv2.1\\\", \\\"ORGANIZATION\\\", 1, 2]],\\n\",\n      \"    [[\\\"北京立方庭\\\", \\\"LOCATION\\\", 2, 4], [\\\"自然语义科技公司\\\", \\\"ORGANIZATION\\\", 5, 9]]\\n\",\n      \"  ],\\n\",\n      \"  \\\"ner/pku\\\": [\\n\",\n      \"    [],\\n\",\n      \"    [[\\\"北京\\\", \\\"ns\\\", 2, 3], [\\\"立方庭\\\", \\\"ns\\\", 3, 4], [\\\"自然语义科技公司\\\", \\\"nt\\\", 5, 9]]\\n\",\n      \"  ],\\n\",\n      \"  \\\"ner/ontonotes\\\": [\\n\",\n      \"    [[\\\"2021年\\\", \\\"DATE\\\", 0, 1], [\\\"次世代\\\", \\\"DATE\\\", 6, 8]],\\n\",\n      \"    [[\\\"北京\\\", \\\"FAC\\\", 2, 3], [\\\"立方庭\\\", \\\"LOC\\\", 3, 4], [\\\"自然语义科技公司\\\", \\\"ORG\\\", 5, 9]]\\n\",\n      \"  ]\\n\",\n      \"}\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(HanLP('2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。阿婆主来到北京立方庭参观自然语义科技公司。', tasks='ner*'))\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"每个四元组表示`[命名实体, 类型标签, 起始下标, 终止下标]`，下标指的是命名实体在单词数组中的下标，单词数组默认为第一个以`tok`开头的数组。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"cqEWnj_7p2Lf\"\n   },\n   \"source\": [\n    \"任务越少，速度越快。如指定仅执行命名实体识别，默认MSRA标准：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 572\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"33790ca9-7013-456f-c1cb-e5ddce90a457\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Token    \\tNER Type        \\n\",\n      \"─────────\\t────────────────\\n\",\n      \"2021年    \\t───►DATE        \\n\",\n      \"HanLPv2.1\\t───►ORGANIZATION\\n\",\n      \"为        \\t                \\n\",\n      \"生产       \\t                \\n\",\n      \"环境       \\t                \\n\",\n      \"带来       \\t                \\n\",\n      \"次        \\t                \\n\",\n      \"世代       \\t                \\n\",\n      \"最        \\t                \\n\",\n      \"先进       \\t                \\n\",\n      \"的        \\t                \\n\",\n      \"多        \\t                \\n\",\n      \"语种       \\t                \\n\",\n      \"NLP      \\t                \\n\",\n      \"技术       \\t                \\n\",\n      \"。        \\t                \\n\",\n      \"\\n\",\n      \"Tok\\tNER Type        \\n\",\n      \"───\\t────────────────\\n\",\n      \"阿婆主\\t                \\n\",\n      \"来到 \\t                \\n\",\n      \"北京 \\t◄─┐             \\n\",\n      \"立方庭\\t◄─┴►LOCATION    \\n\",\n      \"参观 \\t                \\n\",\n      \"自然 \\t◄─┐             \\n\",\n      \"语义 \\t  │             \\n\",\n      \"科技 \\t  ├►ORGANIZATION\\n\",\n      \"公司 \\t◄─┘             \\n\",\n      \"。  \\t                \\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"HanLP('2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。阿婆主来到北京立方庭参观自然语义科技公司。', tasks='ner').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"jj1Jk-2sPHYx\"\n   },\n   \"source\": [\n    \"执行OntoNotes命名实体识别：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 572\n    },\n    \"id\": \"1goEC7znPNkI\",\n    \"outputId\": \"2a97331c-a5fb-4d3c-ccf2-ce2186616c57\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Token    \\tNER Type\\n\",\n      \"─────────\\t────────\\n\",\n      \"2021年    \\t───►DATE\\n\",\n      \"HanLPv2.1\\t        \\n\",\n      \"为        \\t        \\n\",\n      \"生产       \\t        \\n\",\n      \"环境       \\t        \\n\",\n      \"带来       \\t        \\n\",\n      \"次        \\t◄─┐     \\n\",\n      \"世代       \\t◄─┴►DATE\\n\",\n      \"最        \\t        \\n\",\n      \"先进       \\t        \\n\",\n      \"的        \\t        \\n\",\n      \"多        \\t        \\n\",\n      \"语种       \\t        \\n\",\n      \"NLP      \\t        \\n\",\n      \"技术       \\t        \\n\",\n      \"。        \\t        \\n\",\n      \"\\n\",\n      \"Tok\\tNER Typ\\n\",\n      \"───\\t───────\\n\",\n      \"阿婆主\\t       \\n\",\n      \"来到 \\t       \\n\",\n      \"北京 \\t───►FAC\\n\",\n      \"立方庭\\t───►LOC\\n\",\n      \"参观 \\t       \\n\",\n      \"自然 \\t◄─┐    \\n\",\n      \"语义 \\t  │    \\n\",\n      \"科技 \\t  ├►ORG\\n\",\n      \"公司 \\t◄─┘    \\n\",\n      \"。  \\t       \\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"HanLP('2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。阿婆主来到北京立方庭参观自然语义科技公司。', tasks='ner/ontonotes').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"XOsWkOqQfzlr\"\n   },\n   \"source\": [\n    \"为已分词的句子执行命名实体识别：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 161\n    },\n    \"id\": \"bLZSTbv_f3OA\",\n    \"outputId\": \"6a0e1e76-f581-4fd1-8a78-ef97d9429e87\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Token   \\tNER Type        \\n\",\n      \"────────\\t────────────────\\n\",\n      \"阿婆主     \\t                \\n\",\n      \"来到      \\t                \\n\",\n      \"北京立方庭   \\t───►LOCATION    \\n\",\n      \"参观      \\t                \\n\",\n      \"自然语义科技公司\\t───►ORGANIZATION\\n\",\n      \"。       \\t                \\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"HanLP(tokens=[[\\\"阿婆主\\\", \\\"来到\\\", \\\"北京立方庭\\\", \\\"参观\\\", \\\"自然语义科技公司\\\", \\\"。\\\"]], tasks='ner').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"ner_restful.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/ner_stl.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/ner_stl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fner_stl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\",\n    \"pycharm\": {\n     \"name\": \"#%% md\\n\"\n    }\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"0tmKBu7sNAXX\",\n    \"pycharm\": {\n     \"name\": \"#%% md\\n\"\n    }\n   },\n   \"source\": [\n    \"## 加载模型\\n\",\n    \"HanLP的工作流程是先加载模型，模型的标示符存储在`hanlp.pretrained`这个包中，按照NLP任务归类。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"EmZDmLn9aGxG\",\n    \"outputId\": \"0d55f7a1-3a4c-4170-e60f-da7473208e3f\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'MSRA_NER_BERT_BASE_ZH': 'https://file.hankcs.com/hanlp/ner/ner_bert_base_msra_20211227_114712.zip',\\n\",\n       \" 'MSRA_NER_ALBERT_BASE_ZH': 'https://file.hankcs.com/hanlp/ner/msra_ner_albert_base_20211228_173323.zip',\\n\",\n       \" 'MSRA_NER_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/ner/msra_ner_electra_small_20210807_154832.zip',\\n\",\n       \" 'CONLL03_NER_BERT_BASE_CASED_EN': 'https://file.hankcs.com/hanlp/ner/ner_conll03_bert_base_cased_en_20211227_121443.zip'}\"\n      ]\n     },\n     \"execution_count\": 1,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"import hanlp\\n\",\n    \"hanlp.pretrained.ner.ALL # 语种见名称最后一个字段或相应语料库\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"VDT-qmLyvDST\"\n   },\n   \"source\": [\n    \"调用`hanlp.load`进行加载，模型会自动下载到本地缓存。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {\n    \"id\": \"Tzu5Qi-xvDST\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"ner = hanlp.load(hanlp.pretrained.ner.MSRA_NER_ELECTRA_SMALL_ZH)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 命名实体识别\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"wxctCigrTKu-\"\n   },\n   \"source\": [\n    \"命名实体识别任务的输入为已分词的句子：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"Zo08uquCTFSk\",\n    \"outputId\": \"864da076-7113-4685-e27a-1856e69bdd2a\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"[[('2021年', 'DATE', 0, 1)], [('北京', 'LOCATION', 2, 3), ('立方庭', 'LOCATION', 3, 4), ('自然语义科技公司', 'ORGANIZATION', 5, 9)]]\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(ner([[\\\"2021年\\\", \\\"HanLPv2.1\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次\\\", \\\"世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多\\\", \\\"语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"], [\\\"阿婆主\\\", \\\"来到\\\", \\\"北京\\\", \\\"立方庭\\\", \\\"参观\\\", \\\"自然\\\", \\\"语义\\\", \\\"科技\\\", \\\"公司\\\", \\\"。\\\"]], tasks='ner*'))\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"每个四元组表示`[命名实体, 类型标签, 起始下标, 终止下标]`，下标指的是命名实体在单词数组中的下标。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## 自定义词典\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"自定义词典是NER任务的成员变量：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"None\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(ner.dict_whitelist)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### 白名单词典\\n\",\n    \"白名单词典中的词语会尽量被输出。当然，HanLP以统计为主，词典的优先级很低。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 8,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"[('2021年', 'DATE', 0, 1),\\n\",\n       \" ('138', 'INTEGER', 4, 5),\\n\",\n       \" ('午饭后', 'TIME', 8, 10),\\n\",\n       \" ('2点45', 'TIME', 10, 11),\\n\",\n       \" ('44', 'INTEGER', 14, 15)]\"\n      ]\n     },\n     \"execution_count\": 8,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"ner.dict_whitelist = {'午饭后': 'TIME'}\\n\",\n    \"ner(['2021年', '测试', '高血压', '是', '138', '，', '时间', '是', '午饭', '后', '2点45', '，', '低血压', '是', '44'])\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### 强制词典\\n\",\n    \"如果你读过[《自然语言处理入门》](http://nlp.hankcs.com/book.php)，你就会理解BMESO标注集，于是你可以直接干预统计模型预测的标签，拿到最高优先级的权限。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 9,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"[('浙江', 'LOCATION', 2, 3), ('金华', 'LOCATION', 3, 4), ('金华', 'PERSON', 10, 11)]\"\n      ]\n     },\n     \"execution_count\": 9,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"ner.dict_tags = {('名字', '叫', '金华'): ('O', 'O', 'S-PERSON')}\\n\",\n    \"ner(['他', '在', '浙江', '金华', '出生', '，', '他', '的', '名字', '叫', '金华', '。'])\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### 黑名单词典\\n\",\n    \"黑名单中的词语绝对不会被当做命名实体。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 10,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"[('浙江', 'LOCATION', 2, 3)]\"\n      ]\n     },\n     \"execution_count\": 10,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"ner.dict_blacklist = {'金华'}\\n\",\n    \"ner(['他', '在', '浙江', '金华', '出生', '，', '他', '的', '名字', '叫', '金华', '。'])\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"ner_stl.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/pos_mtl.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"colab_type\": \"text\",\n    \"id\": \"view-in-github\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/pos_mtl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fpos_mtl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 加载模型\\n\",\n    \"HanLP的工作流程是先加载模型，模型的标示符存储在`hanlp.pretrained`这个包中，按照NLP任务归类。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"4M7ka0K5OMWU\",\n    \"outputId\": \"50ad002e-4363-46cd-8f5d-b6d6aad3e957\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'OPEN_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/mtl/open_tok_pos_ner_srl_dep_sdp_con_electra_small_20201223_035557.zip',\\n\",\n       \" 'OPEN_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH': 'https://file.hankcs.com/hanlp/mtl/open_tok_pos_ner_srl_dep_sdp_con_electra_base_20201223_201906.zip',\\n\",\n       \" 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_small_20210111_124159.zip',\\n\",\n       \" 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_base_20210111_124519.zip',\\n\",\n       \" 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ERNIE_GRAM_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_ernie_gram_base_aug_20210904_145403.zip',\\n\",\n       \" 'UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_MT5_SMALL': 'https://file.hankcs.com/hanlp/mtl/ud_ontonotes_tok_pos_lem_fea_ner_srl_dep_sdp_con_mt5_small_20210228_123458.zip',\\n\",\n       \" 'UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_XLMR_BASE': 'https://file.hankcs.com/hanlp/mtl/ud_ontonotes_tok_pos_lem_fea_ner_srl_dep_sdp_con_xlm_base_20210602_211620.zip',\\n\",\n       \" 'NPCMJ_UD_KYOTO_TOK_POS_CON_BERT_BASE_CHAR_JA': 'https://file.hankcs.com/hanlp/mtl/npcmj_ud_kyoto_tok_pos_ner_dep_con_srl_bert_base_char_ja_20210914_133742.zip'}\"\n      ]\n     },\n     \"execution_count\": 1,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"import hanlp\\n\",\n    \"hanlp.pretrained.mtl.ALL # MTL多任务，具体任务见模型名称，语种见名称最后一个字段或相应语料库\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"BMW528wGNulM\"\n   },\n   \"source\": [\n    \"调用`hanlp.load`进行加载，模型会自动下载到本地缓存。自然语言处理分为许多任务，分词只是最初级的一个。与其每个任务单独创建一个模型，不如利用HanLP的联合模型一次性完成多个任务：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"id\": \"0tmKBu7sNAXX\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"HanLP = hanlp.load(hanlp.pretrained.mtl.CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 词性标注\\n\",\n    \"任务越少，速度越快。如指定仅执行词性标注，默认CTB标准：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"5ad7fd22-651a-4403-d897-a9492eb15854\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">HanLP/NR&nbsp;为/P&nbsp;生产/NN&nbsp;环境/NN&nbsp;带来/VV&nbsp;次/JJ&nbsp;世代/NN&nbsp;最/AD&nbsp;先进/JJ&nbsp;的/DEG&nbsp;多语种/NN&nbsp;NLP/NR&nbsp;技术/NN&nbsp;。/PU</pre></div><br><div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">我/PN&nbsp;的/DEG&nbsp;希望/NN&nbsp;是/VC&nbsp;希望/VV&nbsp;张晚霞/NR&nbsp;的/DEG&nbsp;背影/NN&nbsp;被/LB&nbsp;晚霞/NN&nbsp;映红/VV&nbsp;。/PU</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"HanLP(['HanLP为生产环境带来次世代最先进的多语种NLP技术。', '我的希望是希望张晚霞的背影被晚霞映红。'], tasks='pos').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"jj1Jk-2sPHYx\"\n   },\n   \"source\": [\n    \"注意上面两个“希望”的词性各不相同，一个是名词另一个是动词。\\n\",\n    \"执行PKU词性标注：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"1goEC7znPNkI\",\n    \"outputId\": \"586afd5d-db0d-41bd-f7de-411f37062a8c\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">HanLP/nx&nbsp;为/p&nbsp;生产/vn&nbsp;环境/n&nbsp;带来/v&nbsp;次/b&nbsp;世代/n&nbsp;最/d&nbsp;先进/a&nbsp;的/u&nbsp;多语种/n&nbsp;NLP/nx&nbsp;技术/n&nbsp;。/w</pre></div><br><div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">我/r&nbsp;的/u&nbsp;希望/n&nbsp;是/v&nbsp;希望/v&nbsp;张晚霞/nr&nbsp;的/u&nbsp;背影/n&nbsp;被/p&nbsp;晚霞/n&nbsp;映红/v&nbsp;。/w</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"HanLP(['HanLP为生产环境带来次世代最先进的多语种NLP技术。', '我的希望是希望张晚霞的背影被晚霞映红。'], tasks='pos/pku').pretty_print()\\n\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"wxctCigrTKu-\"\n   },\n   \"source\": [\n    \"同时执行所有标准的词性标注：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"Zo08uquCTFSk\",\n    \"outputId\": \"d2b3eb65-06e6-47a6-d954-04cae27d6c51\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"{\\n\",\n      \"  \\\"tok/fine\\\": [\\n\",\n      \"    [\\\"HanLP\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次\\\", \\\"世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"],\\n\",\n      \"    [\\\"我\\\", \\\"的\\\", \\\"希望\\\", \\\"是\\\", \\\"希望\\\", \\\"张晚霞\\\", \\\"的\\\", \\\"背影\\\", \\\"被\\\", \\\"晚霞\\\", \\\"映红\\\", \\\"。\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"pos/ctb\\\": [\\n\",\n      \"    [\\\"NR\\\", \\\"P\\\", \\\"NN\\\", \\\"NN\\\", \\\"VV\\\", \\\"JJ\\\", \\\"NN\\\", \\\"AD\\\", \\\"JJ\\\", \\\"DEG\\\", \\\"NN\\\", \\\"NR\\\", \\\"NN\\\", \\\"PU\\\"],\\n\",\n      \"    [\\\"PN\\\", \\\"DEG\\\", \\\"NN\\\", \\\"VC\\\", \\\"VV\\\", \\\"NR\\\", \\\"DEG\\\", \\\"NN\\\", \\\"LB\\\", \\\"NN\\\", \\\"VV\\\", \\\"PU\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"pos/pku\\\": [\\n\",\n      \"    [\\\"nx\\\", \\\"p\\\", \\\"vn\\\", \\\"n\\\", \\\"v\\\", \\\"b\\\", \\\"n\\\", \\\"d\\\", \\\"a\\\", \\\"u\\\", \\\"n\\\", \\\"nx\\\", \\\"n\\\", \\\"w\\\"],\\n\",\n      \"    [\\\"r\\\", \\\"u\\\", \\\"n\\\", \\\"v\\\", \\\"v\\\", \\\"nr\\\", \\\"u\\\", \\\"n\\\", \\\"p\\\", \\\"n\\\", \\\"v\\\", \\\"w\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"pos/863\\\": [\\n\",\n      \"    [\\\"w\\\", \\\"p\\\", \\\"v\\\", \\\"n\\\", \\\"v\\\", \\\"a\\\", \\\"nt\\\", \\\"d\\\", \\\"a\\\", \\\"u\\\", \\\"n\\\", \\\"ws\\\", \\\"n\\\", \\\"w\\\"],\\n\",\n      \"    [\\\"r\\\", \\\"u\\\", \\\"n\\\", \\\"vl\\\", \\\"v\\\", \\\"nh\\\", \\\"u\\\", \\\"n\\\", \\\"p\\\", \\\"n\\\", \\\"v\\\", \\\"w\\\"]\\n\",\n      \"  ]\\n\",\n      \"}\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(HanLP(['HanLP为生产环境带来次世代最先进的多语种NLP技术。', '我的希望是希望张晚霞的背影被晚霞映红。'], tasks='pos*'))\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"以`pos`开头的字段为词性，以`tok`开头的第一个数组为单词，两者按下标一一对应。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"#### 注意\\n\",\n    \"Native API的输入单位限定为句子，需使用[多语种分句模型](https://github.com/hankcs/HanLP/blob/master/plugins/hanlp_demo/hanlp_demo/sent_split.py)或[基于规则的分句函数](https://github.com/hankcs/HanLP/blob/master/hanlp/utils/rules.py#L19)先行分句。RESTful同时支持全文、句子、已分词的句子。除此之外，RESTful和native两种API的语义设计完全一致，用户可以无缝互换。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"suUL042zPpLj\"\n   },\n   \"source\": [\n    \"## 自定义词典\\n\",\n    \"自定义词典为词性标注任务的成员变量，要操作自定义词典，先获取一个词性标注任务，以CTB标准为例：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"AzYShIssP6kq\",\n    \"outputId\": \"640cefa5-1d6d-464b-81d2-83c66e2081f2\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"<hanlp.components.mtl.tasks.pos.TransformerTagging at 0x160950910>\"\n      ]\n     },\n     \"execution_count\": 6,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"pos = HanLP['pos/ctb']\\n\",\n    \"pos\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"1q4MUpgVQNlu\"\n   },\n   \"source\": [\n    \"自定义单个词性：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 35\n    },\n    \"id\": \"2zZkH9tRQOoi\",\n    \"outputId\": \"ed0bb8fe-2e68-4c58-e11e-ff6a0cc69ae4\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">HanLP/state-of-the-art-tool&nbsp;为/P&nbsp;生产/NN&nbsp;环境/NN&nbsp;带来/VV&nbsp;次/JJ&nbsp;世代/NN&nbsp;最/AD&nbsp;先进/JJ&nbsp;的/DEG&nbsp;多语种/NN&nbsp;NLP/NR&nbsp;技术/NN&nbsp;。/PU</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"pos.dict_tags = {'HanLP': 'state-of-the-art-tool'}\\n\",\n    \"HanLP(\\\"HanLP为生产环境带来次世代最先进的多语种NLP技术。\\\", tasks='pos/ctb').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"F-9gAeIVQUFG\"\n   },\n   \"source\": [\n    \"根据上下文自定义词性：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 8,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 35\n    },\n    \"id\": \"F8M8cyBrQduw\",\n    \"outputId\": \"16ef7f82-50ff-478f-c3ea-8e768b0cea31\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">我/PN&nbsp;的/补语成分&nbsp;希望/名词&nbsp;是/VC&nbsp;希望/动词&nbsp;张晚霞/NR&nbsp;的/DEG&nbsp;背影/NN&nbsp;被/LB&nbsp;晚霞/NN&nbsp;映红/VV&nbsp;。/PU</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"pos.dict_tags = {('的', '希望'): ('补语成分', '名词'), '希望': '动词'}\\n\",\n    \"HanLP(\\\"我的希望是希望张晚霞的背影被晚霞映红。\\\", tasks='pos/ctb').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"需要算法基础才能理解，初学者可参考[《自然语言处理入门》](http://nlp.hankcs.com/book.php)。\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"include_colab_link\": true,\n   \"name\": \"pos_mtl.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/pos_restful.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/pos_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fpos_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"pip install hanlp_restful -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 创建客户端\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"id\": \"0tmKBu7sNAXX\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from hanlp_restful import HanLPClient\\n\",\n    \"HanLP = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh') # auth不填则匿名，zh中文，mul多语种\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"EmZDmLn9aGxG\"\n   },\n   \"source\": [\n    \"#### 申请秘钥\\n\",\n    \"由于服务器算力有限，匿名用户每分钟限2次调用。如果你需要更多调用次数，[建议申请免费公益API秘钥auth](https://bbs.hanlp.com/t/hanlp2-1-restful-api/53)。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 词性标注\\n\",\n    \"任务越少，速度越快。如指定仅执行词性标注，默认CTB标准：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"2a0d392f-b99a-4a18-fc7f-754e2abe2e34\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; padding-bottom: 1rem;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap; line-height: 128%; padding: 0;\\\">HanLP/NR&nbsp;为/P&nbsp;生产/NN&nbsp;环境/NN&nbsp;带来/VV&nbsp;次世代/NN&nbsp;最/AD&nbsp;先进/JJ&nbsp;的/DEG&nbsp;多/CD&nbsp;语种/NN&nbsp;NLP/NN&nbsp;技术/NN&nbsp;。/PU</pre></div><br><div style=\\\"display: table; padding-bottom: 1rem;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap; line-height: 128%; padding: 0;\\\">我/PN&nbsp;的/DEG&nbsp;希望/NN&nbsp;是/VC&nbsp;希望/VV&nbsp;张晚霞/NR&nbsp;的/DEG&nbsp;背影/NN&nbsp;被/LB&nbsp;晚霞/NN&nbsp;映红/VV&nbsp;。/PU</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"HanLP('HanLP为生产环境带来次世代最先进的多语种NLP技术。我的希望是希望张晚霞的背影被晚霞映红。', tasks='pos').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"jj1Jk-2sPHYx\"\n   },\n   \"source\": [\n    \"注意上面两个“希望”的词性各不相同，一个是名词另一个是动词。\\n\",\n    \"\\n\",\n    \"### 执行PKU词性标注\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"1goEC7znPNkI\",\n    \"outputId\": \"7a3fde55-7577-49eb-92c8-48146aaa89d3\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; padding-bottom: 1rem;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap; line-height: 128%; padding: 0;\\\">HanLP/nx&nbsp;为/p&nbsp;生产/vn&nbsp;环境/n&nbsp;带来/v&nbsp;次世代/n&nbsp;最/d&nbsp;先进/a&nbsp;的/u&nbsp;多/a&nbsp;语种/n&nbsp;NLP/nx&nbsp;技术/n&nbsp;。/w</pre></div><br><div style=\\\"display: table; padding-bottom: 1rem;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap; line-height: 128%; padding: 0;\\\">我/r&nbsp;的/u&nbsp;希望/n&nbsp;是/v&nbsp;希望/v&nbsp;张晚霞/nr&nbsp;的/u&nbsp;背影/n&nbsp;被/p&nbsp;晚霞/n&nbsp;映红/v&nbsp;。/w</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"HanLP('HanLP为生产环境带来次世代最先进的多语种NLP技术。我的希望是希望张晚霞的背影被晚霞映红。', tasks='pos/pku').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### 执行粗颗粒度分词和PKU词性标注\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; padding-bottom: 1rem;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap; line-height: 128%; padding: 0;\\\">阿婆主/n&nbsp;来到/v&nbsp;北京立方庭/ns&nbsp;参观/v&nbsp;自然语义科技公司/n&nbsp;。/w</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"HanLP('阿婆主来到北京立方庭参观自然语义科技公司。', tasks=['tok/coarse', 'pos/pku'], skip_tasks='tok/fine').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"举一反三，你可以指定其他pos标注集（ctb、863等）。用户有多聪明，HanLP就有多强大。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"wxctCigrTKu-\"\n   },\n   \"source\": [\n    \"### 同时执行所有标准的词性标注\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"Zo08uquCTFSk\",\n    \"outputId\": \"c6077f2d-7084-4f4b-a3bc-9aa9951704ea\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"{\\n\",\n      \"  \\\"tok/fine\\\": [\\n\",\n      \"    [\\\"HanLP\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多\\\", \\\"语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"],\\n\",\n      \"    [\\\"我\\\", \\\"的\\\", \\\"希望\\\", \\\"是\\\", \\\"希望\\\", \\\"张晚霞\\\", \\\"的\\\", \\\"背影\\\", \\\"被\\\", \\\"晚霞\\\", \\\"映红\\\", \\\"。\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"pos/ctb\\\": [\\n\",\n      \"    [\\\"NR\\\", \\\"P\\\", \\\"NN\\\", \\\"NN\\\", \\\"VV\\\", \\\"NN\\\", \\\"AD\\\", \\\"JJ\\\", \\\"DEG\\\", \\\"CD\\\", \\\"NN\\\", \\\"NN\\\", \\\"NN\\\", \\\"PU\\\"],\\n\",\n      \"    [\\\"PN\\\", \\\"DEG\\\", \\\"NN\\\", \\\"VC\\\", \\\"VV\\\", \\\"NR\\\", \\\"DEG\\\", \\\"NN\\\", \\\"LB\\\", \\\"NN\\\", \\\"VV\\\", \\\"PU\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"pos/pku\\\": [\\n\",\n      \"    [\\\"nx\\\", \\\"p\\\", \\\"vn\\\", \\\"n\\\", \\\"v\\\", \\\"n\\\", \\\"d\\\", \\\"a\\\", \\\"u\\\", \\\"a\\\", \\\"n\\\", \\\"nx\\\", \\\"n\\\", \\\"w\\\"],\\n\",\n      \"    [\\\"r\\\", \\\"u\\\", \\\"n\\\", \\\"v\\\", \\\"v\\\", \\\"nr\\\", \\\"u\\\", \\\"n\\\", \\\"p\\\", \\\"n\\\", \\\"v\\\", \\\"w\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"pos/863\\\": [\\n\",\n      \"    [\\\"w\\\", \\\"p\\\", \\\"v\\\", \\\"n\\\", \\\"v\\\", \\\"n\\\", \\\"d\\\", \\\"a\\\", \\\"u\\\", \\\"a\\\", \\\"n\\\", \\\"w\\\", \\\"n\\\", \\\"w\\\"],\\n\",\n      \"    [\\\"r\\\", \\\"u\\\", \\\"v\\\", \\\"vl\\\", \\\"v\\\", \\\"nh\\\", \\\"u\\\", \\\"n\\\", \\\"p\\\", \\\"n\\\", \\\"v\\\", \\\"w\\\"]\\n\",\n      \"  ]\\n\",\n      \"}\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(HanLP('HanLP为生产环境带来次世代最先进的多语种NLP技术。我的希望是希望张晚霞的背影被晚霞映红。', tasks='pos*'))\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"以`pos`开头的字段为词性，以`tok`开头的第一个数组为单词，两者按下标一一对应。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"XOsWkOqQfzlr\"\n   },\n   \"source\": [\n    \"### 为已分词的句子执行词性标注\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"bLZSTbv_f3OA\",\n    \"outputId\": \"111c0be9-bac6-4eee-d5bd-a972ffc34844\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; padding-bottom: 1rem;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap; line-height: 128%; padding: 0;\\\">HanLP/NR&nbsp;为/P&nbsp;生产环境/NN&nbsp;带来/VV&nbsp;次世代/NN&nbsp;最/AD&nbsp;先进/JJ&nbsp;的/DEG&nbsp;多语种/NN&nbsp;NLP/NN&nbsp;技术/NN&nbsp;。/PU</pre></div><br><div style=\\\"display: table; padding-bottom: 1rem;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap; line-height: 128%; padding: 0;\\\">我/PN&nbsp;的/DEG&nbsp;希望/NN&nbsp;是/VC&nbsp;希望/VV&nbsp;张晚霞/NR&nbsp;的/DEG&nbsp;背影/NN&nbsp;被/LB&nbsp;晚霞/NN&nbsp;映红/VV&nbsp;。/PU</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"HanLP(tokens=[\\n\",\n    \"    [\\\"HanLP\\\", \\\"为\\\", \\\"生产环境\\\", \\\"带来\\\", \\\"次世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"],\\n\",\n    \"    [\\\"我\\\", \\\"的\\\", \\\"希望\\\", \\\"是\\\", \\\"希望\\\", \\\"张晚霞\\\", \\\"的\\\", \\\"背影\\\", \\\"被\\\", \\\"晚霞\\\", \\\"映红\\\", \\\"。\\\"]\\n\",\n    \"  ], tasks='pos').pretty_print()\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"pos_restful.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.7.13\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/pos_stl.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/pos_stl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fpos_stl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 加载模型\\n\",\n    \"HanLP的工作流程是先加载模型，模型的标示符存储在`hanlp.pretrained`这个包中，按照NLP任务归类。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"4M7ka0K5OMWU\",\n    \"outputId\": \"d74f0749-0587-454a-d7c9-7418d45ce534\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'CTB5_POS_RNN': 'https://file.hankcs.com/hanlp/pos/ctb5_pos_rnn_20200113_235925.zip',\\n\",\n       \" 'CTB5_POS_RNN_FASTTEXT_ZH': 'https://file.hankcs.com/hanlp/pos/ctb5_pos_rnn_fasttext_20191230_202639.zip',\\n\",\n       \" 'CTB9_POS_ALBERT_BASE': 'https://file.hankcs.com/hanlp/pos/ctb9_albert_base_20211228_163935.zip',\\n\",\n       \" 'CTB9_POS_ELECTRA_SMALL_TF': 'https://file.hankcs.com/hanlp/pos/pos_ctb_electra_small_20211227_121341.zip',\\n\",\n       \" 'CTB9_POS_ELECTRA_SMALL': 'https://file.hankcs.com/hanlp/pos/pos_ctb_electra_small_20220215_111944.zip',\\n\",\n       \" 'CTB9_POS_RADICAL_ELECTRA_SMALL': 'https://file.hankcs.com/hanlp/pos/pos_ctb_radical_electra_small_20220215_111932.zip',\\n\",\n       \" 'C863_POS_ELECTRA_SMALL': 'https://file.hankcs.com/hanlp/pos/pos_863_electra_small_20220217_101958.zip',\\n\",\n       \" 'PKU_POS_ELECTRA_SMALL': 'https://file.hankcs.com/hanlp/pos/pos_pku_electra_small_20220217_142436.zip',\\n\",\n       \" 'PKU98_POS_ELECTRA_SMALL': 'https://file.hankcs.com/hanlp/pos/pos_pku_electra_small_20210808_125158.zip',\\n\",\n       \" 'PTB_POS_RNN_FASTTEXT_EN': 'https://file.hankcs.com/hanlp/pos/ptb_pos_rnn_fasttext_20200103_145337.zip'}\"\n      ]\n     },\n     \"execution_count\": 1,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"import hanlp\\n\",\n    \"hanlp.pretrained.pos.ALL # 语种见名称最后一个字段或相应语料库\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"BMW528wGNulM\"\n   },\n   \"source\": [\n    \"调用`hanlp.load`进行加载，模型会自动下载到本地缓存：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"0tmKBu7sNAXX\",\n    \"outputId\": \"df2de87b-27f5-4c72-8eb2-25ceefdd8270\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stderr\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Downloading https://file.hankcs.com/hanlp/pos/ctb9_pos_electra_small_20220118_164341.zip to /root/.hanlp/pos/ctb9_pos_electra_small_20220118_164341.zip\\n\",\n      \"100%  43.6 MiB  21.2 MiB/s ETA:  0 s [=========================================]\\n\",\n      \"Decompressing /root/.hanlp/pos/ctb9_pos_electra_small_20220118_164341.zip to /root/.hanlp/pos\\n\",\n      \"Downloading https://file.hankcs.com/hanlp/transformers/electra_zh_small_20210706_125427.zip to /root/.hanlp/transformers/electra_zh_small_20210706_125427.zip\\n\",\n      \"100%  41.2 KiB  41.2 KiB/s ETA:  0 s [=========================================]\\n\",\n      \"Decompressing /root/.hanlp/transformers/electra_zh_small_20210706_125427.zip to /root/.hanlp/transformers\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"pos = hanlp.load(hanlp.pretrained.pos.CTB9_POS_ELECTRA_SMALL)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 词性标注\\n\",\n    \"词性标注任务的输入为已分词的一个或多个句子：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"936d439a-e1ff-4308-d2aa-775955558594\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"['PN', 'DEG', 'NN', 'VC', 'VV', 'NR', 'DEG', 'NN', 'LB', 'NR', 'VV', 'PU']\"\n      ]\n     },\n     \"execution_count\": 7,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"pos([\\\"我\\\", \\\"的\\\", \\\"希望\\\", \\\"是\\\", \\\"希望\\\", \\\"张晚霞\\\", \\\"的\\\", \\\"背影\\\", \\\"被\\\", \\\"晚霞\\\", \\\"映红\\\", \\\"。\\\"])\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"jj1Jk-2sPHYx\"\n   },\n   \"source\": [\n    \"注意上面两个“希望”的词性各不相同，一个是名词另一个是动词。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"suUL042zPpLj\"\n   },\n   \"source\": [\n    \"## 自定义词典\\n\",\n    \"自定义词典为词性标注任务的成员变量，以CTB标准为例：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 8,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"AzYShIssP6kq\",\n    \"outputId\": \"99b2607b-b618-4876-bbea-9f8c24859a85\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"None\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(pos.dict_tags)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"1q4MUpgVQNlu\"\n   },\n   \"source\": [\n    \"自定义单个词性：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 10,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"2zZkH9tRQOoi\",\n    \"outputId\": \"4f92a907-10c3-4798-e7b9-914b8f577b2c\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"['state-of-the-art-tool',\\n\",\n       \" 'P',\\n\",\n       \" 'NN',\\n\",\n       \" 'NN',\\n\",\n       \" 'VV',\\n\",\n       \" 'JJ',\\n\",\n       \" 'NN',\\n\",\n       \" 'AD',\\n\",\n       \" 'VA',\\n\",\n       \" 'DEC',\\n\",\n       \" 'NN',\\n\",\n       \" 'NN',\\n\",\n       \" 'NN',\\n\",\n       \" 'PU']\"\n      ]\n     },\n     \"execution_count\": 10,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"pos.dict_tags = {'HanLP': 'state-of-the-art-tool'}\\n\",\n    \"pos([\\\"HanLP\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次\\\", \\\"世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"])\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"F-9gAeIVQUFG\"\n   },\n   \"source\": [\n    \"根据上下文自定义词性：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 12,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"F8M8cyBrQduw\",\n    \"outputId\": \"24fa7ff0-305d-4d71-925e-f369b1c50e96\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"['PN', '补语成分', '名词', 'VC', '动词', 'NR', 'DEG', 'NN', 'LB', 'NR', 'VV', 'PU']\"\n      ]\n     },\n     \"execution_count\": 12,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"pos.dict_tags = {('的', '希望'): ('补语成分', '名词'), '希望': '动词'}\\n\",\n    \"pos([\\\"我\\\", \\\"的\\\", \\\"希望\\\", \\\"是\\\", \\\"希望\\\", \\\"张晚霞\\\", \\\"的\\\", \\\"背影\\\", \\\"被\\\", \\\"晚霞\\\", \\\"映红\\\", \\\"。\\\"])\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"需要算法基础才能理解，初学者可参考[《自然语言处理入门》](http://nlp.hankcs.com/book.php)。\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"pos_stl.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/sdp_mtl.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/sdp_mtl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fsdp_mtl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## 加载模型\\n\",\n    \"HanLP的工作流程是先加载模型，模型的标示符存储在`hanlp.pretrained`这个包中，按照NLP任务归类。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'OPEN_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/mtl/open_tok_pos_ner_srl_dep_sdp_con_electra_small_20201223_035557.zip',\\n\",\n       \" 'OPEN_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH': 'https://file.hankcs.com/hanlp/mtl/open_tok_pos_ner_srl_dep_sdp_con_electra_base_20201223_201906.zip',\\n\",\n       \" 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_small_20210111_124159.zip',\\n\",\n       \" 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_base_20210111_124519.zip',\\n\",\n       \" 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ERNIE_GRAM_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_ernie_gram_base_aug_20210904_145403.zip',\\n\",\n       \" 'UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_MT5_SMALL': 'https://file.hankcs.com/hanlp/mtl/ud_ontonotes_tok_pos_lem_fea_ner_srl_dep_sdp_con_mt5_small_20210228_123458.zip',\\n\",\n       \" 'UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_XLMR_BASE': 'https://file.hankcs.com/hanlp/mtl/ud_ontonotes_tok_pos_lem_fea_ner_srl_dep_sdp_con_xlm_base_20210602_211620.zip',\\n\",\n       \" 'NPCMJ_UD_KYOTO_TOK_POS_CON_BERT_BASE_CHAR_JA': 'https://file.hankcs.com/hanlp/mtl/npcmj_ud_kyoto_tok_pos_ner_dep_con_srl_bert_base_char_ja_20210914_133742.zip'}\"\n      ]\n     },\n     \"execution_count\": 1,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"import hanlp\\n\",\n    \"hanlp.pretrained.mtl.ALL # MTL多任务，具体任务见模型名称，语种见名称最后一个字段或相应语料库\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\",\n    \"pycharm\": {\n     \"name\": \"#%% md\\n\"\n    }\n   },\n   \"source\": [\n    \"调用`hanlp.load`进行加载，模型会自动下载到本地缓存。自然语言处理分为许多任务，分词只是最初级的一个。与其每个任务单独创建一个模型，不如利用HanLP的联合模型一次性完成多个任务：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"HanLP = hanlp.load(hanlp.pretrained.mtl.CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 语义依存分析\\n\",\n    \"任务越少，速度越快。如指定仅执行语义依存分析：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"2a0d392f-b99a-4a18-fc7f-754e2abe2e34\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"doc = HanLP('2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', tasks='sdp')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"返回值为一个[Document](https://hanlp.hankcs.com/docs/api/common/document.html):\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"{\\n\",\n      \"  \\\"tok/fine\\\": [\\n\",\n      \"    \\\"2021年\\\",\\n\",\n      \"    \\\"HanLPv2.1\\\",\\n\",\n      \"    \\\"为\\\",\\n\",\n      \"    \\\"生产\\\",\\n\",\n      \"    \\\"环境\\\",\\n\",\n      \"    \\\"带来\\\",\\n\",\n      \"    \\\"次\\\",\\n\",\n      \"    \\\"世代\\\",\\n\",\n      \"    \\\"最\\\",\\n\",\n      \"    \\\"先进\\\",\\n\",\n      \"    \\\"的\\\",\\n\",\n      \"    \\\"多\\\",\\n\",\n      \"    \\\"语种\\\",\\n\",\n      \"    \\\"NLP\\\",\\n\",\n      \"    \\\"技术\\\",\\n\",\n      \"    \\\"。\\\"\\n\",\n      \"  ],\\n\",\n      \"  \\\"sdp\\\": [\\n\",\n      \"    [[6, \\\"Time\\\"]],\\n\",\n      \"    [[6, \\\"Exp\\\"]],\\n\",\n      \"    [[5, \\\"mPrep\\\"]],\\n\",\n      \"    [[5, \\\"Desc\\\"]],\\n\",\n      \"    [[6, \\\"Datv\\\"]],\\n\",\n      \"    [[13, \\\"dDesc\\\"]],\\n\",\n      \"    [[0, \\\"Root\\\"], [8, \\\"Desc\\\"], [13, \\\"Desc\\\"]],\\n\",\n      \"    [[15, \\\"Time\\\"]],\\n\",\n      \"    [[10, \\\"mDegr\\\"]],\\n\",\n      \"    [[15, \\\"Desc\\\"]],\\n\",\n      \"    [[10, \\\"mAux\\\"]],\\n\",\n      \"    [[8, \\\"Quan\\\"], [13, \\\"Quan\\\"]],\\n\",\n      \"    [[15, \\\"Desc\\\"]],\\n\",\n      \"    [[15, \\\"Nmod\\\"]],\\n\",\n      \"    [[6, \\\"Pat\\\"]],\\n\",\n      \"    [[6, \\\"mPunc\\\"]]\\n\",\n      \"  ]\\n\",\n      \"}\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(doc)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"`doc['sdp']`字段代表语义依存图的数组格式，数组中第`i`个子数组代表第`i`个单词的语义依存关系，子数组中每个二元组的格式为`[中心词的下标, 与中心词的语义依存关系]`。每个单词的语义依存关系可能有零个、一个或多个（任意数量）。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"转换为[CoNLLSentence](https://hanlp.hankcs.com/docs/api/common/conll.html#hanlp_common.conll.CoNLLSentence)格式更容易观察：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"1\\t2021年\\t_\\t_\\t_\\t_\\t_\\t_\\t6:Time\\t_\\n\",\n      \"2\\tHanLPv2.1\\t_\\t_\\t_\\t_\\t_\\t_\\t6:Exp\\t_\\n\",\n      \"3\\t为\\t_\\t_\\t_\\t_\\t_\\t_\\t5:mPrep\\t_\\n\",\n      \"4\\t生产\\t_\\t_\\t_\\t_\\t_\\t_\\t5:Desc\\t_\\n\",\n      \"5\\t环境\\t_\\t_\\t_\\t_\\t_\\t_\\t6:Datv\\t_\\n\",\n      \"6\\t带来\\t_\\t_\\t_\\t_\\t_\\t_\\t13:dDesc\\t_\\n\",\n      \"7\\t次\\t_\\t_\\t_\\t_\\t_\\t_\\t0:Root|8:Desc|13:Desc\\t_\\n\",\n      \"8\\t世代\\t_\\t_\\t_\\t_\\t_\\t_\\t15:Time\\t_\\n\",\n      \"9\\t最\\t_\\t_\\t_\\t_\\t_\\t_\\t10:mDegr\\t_\\n\",\n      \"10\\t先进\\t_\\t_\\t_\\t_\\t_\\t_\\t15:Desc\\t_\\n\",\n      \"11\\t的\\t_\\t_\\t_\\t_\\t_\\t_\\t10:mAux\\t_\\n\",\n      \"12\\t多\\t_\\t_\\t_\\t_\\t_\\t_\\t8:Quan|13:Quan\\t_\\n\",\n      \"13\\t语种\\t_\\t_\\t_\\t_\\t_\\t_\\t15:Desc\\t_\\n\",\n      \"14\\tNLP\\t_\\t_\\t_\\t_\\t_\\t_\\t15:Nmod\\t_\\n\",\n      \"15\\t技术\\t_\\t_\\t_\\t_\\t_\\t_\\t6:Pat\\t_\\n\",\n      \"16\\t。\\t_\\t_\\t_\\t_\\t_\\t_\\t6:mPunc\\t_\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(doc.to_conll())\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"XOsWkOqQfzlr\"\n   },\n   \"source\": [\n    \"为已分词的句子执行语义依存分析：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"bLZSTbv_f3OA\",\n    \"outputId\": \"111c0be9-bac6-4eee-d5bd-a972ffc34844\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"1\\tHanLP\\t_\\t_\\t_\\t_\\t_\\t_\\t5:Exp\\t_\\n\",\n      \"2\\t为\\t_\\t_\\t_\\t_\\t_\\t_\\t4:mPrep\\t_\\n\",\n      \"3\\t生产\\t_\\t_\\t_\\t_\\t_\\t_\\t4:Desc\\t_\\n\",\n      \"4\\t环境\\t_\\t_\\t_\\t_\\t_\\t_\\t5:Datv\\t_\\n\",\n      \"5\\t带来\\t_\\t_\\t_\\t_\\t_\\t_\\t0:Root\\t_\\n\",\n      \"6\\t次世代\\t_\\t_\\t_\\t_\\t_\\t_\\t12:Time\\t_\\n\",\n      \"7\\t最\\t_\\t_\\t_\\t_\\t_\\t_\\t8:mDegr\\t_\\n\",\n      \"8\\t先进\\t_\\t_\\t_\\t_\\t_\\t_\\t12:Desc\\t_\\n\",\n      \"9\\t的\\t_\\t_\\t_\\t_\\t_\\t_\\t8:mAux\\t_\\n\",\n      \"10\\t多语种\\t_\\t_\\t_\\t_\\t_\\t_\\t12:Desc\\t_\\n\",\n      \"11\\tNLP\\t_\\t_\\t_\\t_\\t_\\t_\\t12:Nmod\\t_\\n\",\n      \"12\\t技术\\t_\\t_\\t_\\t_\\t_\\t_\\t5:Pat\\t_\\n\",\n      \"13\\t。\\t_\\t_\\t_\\t_\\t_\\t_\\t5:mPunc\\t_\\n\",\n      \"\\n\",\n      \"1\\t我\\t_\\t_\\t_\\t_\\t_\\t_\\t3:Poss\\t_\\n\",\n      \"2\\t的\\t_\\t_\\t_\\t_\\t_\\t_\\t1:mAux\\t_\\n\",\n      \"3\\t希望\\t_\\t_\\t_\\t_\\t_\\t_\\t4:Exp\\t_\\n\",\n      \"4\\t是\\t_\\t_\\t_\\t_\\t_\\t_\\t11:mMod\\t_\\n\",\n      \"5\\t希望\\t_\\t_\\t_\\t_\\t_\\t_\\t4:dClas\\t_\\n\",\n      \"6\\t张晚霞\\t_\\t_\\t_\\t_\\t_\\t_\\t8:Poss\\t_\\n\",\n      \"7\\t的\\t_\\t_\\t_\\t_\\t_\\t_\\t6:mAux\\t_\\n\",\n      \"8\\t背影\\t_\\t_\\t_\\t_\\t_\\t_\\t11:Pat\\t_\\n\",\n      \"9\\t被\\t_\\t_\\t_\\t_\\t_\\t_\\t10:mPrep\\t_\\n\",\n      \"10\\t晚霞\\t_\\t_\\t_\\t_\\t_\\t_\\t11:Exp\\t_\\n\",\n      \"11\\t映红\\t_\\t_\\t_\\t_\\t_\\t_\\t5:dCont\\t_\\n\",\n      \"12\\t。\\t_\\t_\\t_\\t_\\t_\\t_\\t4:mPunc\\t_\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(HanLP([\\n\",\n    \"    [\\\"HanLP\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"],\\n\",\n    \"    [\\\"我\\\", \\\"的\\\", \\\"希望\\\", \\\"是\\\", \\\"希望\\\", \\\"张晚霞\\\", \\\"的\\\", \\\"背影\\\", \\\"被\\\", \\\"晚霞\\\", \\\"映红\\\", \\\"。\\\"]\\n\",\n    \"  ], tasks='sdp', skip_tasks='tok*').to_conll())\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"#### 注意\\n\",\n    \"Native API的输入单位限定为句子，需使用[多语种分句模型](https://github.com/hankcs/HanLP/blob/master/plugins/hanlp_demo/hanlp_demo/sent_split.py)或[基于规则的分句函数](https://github.com/hankcs/HanLP/blob/master/hanlp/utils/rules.py#L19)先行分句。RESTful同时支持全文、句子、已分词的句子。除此之外，RESTful和native两种API的语义设计完全一致，用户可以无缝互换。\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"sdp_mtl.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/sdp_restful.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/sdp_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fsdp_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp_restful -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 创建客户端\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"id\": \"0tmKBu7sNAXX\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from hanlp_restful import HanLPClient\\n\",\n    \"HanLP = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh') # auth不填则匿名，zh中文，mul多语种\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"EmZDmLn9aGxG\"\n   },\n   \"source\": [\n    \"#### 申请秘钥\\n\",\n    \"由于服务器算力有限，匿名用户每分钟限2次调用。如果你需要更多调用次数，[建议申请免费公益API秘钥auth](https://bbs.hanlp.com/t/hanlp2-1-restful-api/53)。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 语义依存分析\\n\",\n    \"任务越少，速度越快。如指定仅执行语义依存分析：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"2a0d392f-b99a-4a18-fc7f-754e2abe2e34\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"doc = HanLP('2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', tasks='sdp')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"返回值为一个[Document](https://hanlp.hankcs.com/docs/api/common/document.html):\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"{\\n\",\n      \"  \\\"tok/fine\\\": [\\n\",\n      \"    [\\\"2021年\\\", \\\"HanLPv2.1\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次\\\", \\\"世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多\\\", \\\"语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"sdp\\\": [\\n\",\n      \"    [[[6, \\\"Time\\\"]], [[6, \\\"Agt\\\"]], [[5, \\\"mPrep\\\"]], [[5, \\\"Desc\\\"]], [[6, \\\"Datv\\\"]], [[0, \\\"Root\\\"]], [[8, \\\"Qp\\\"]], [[15, \\\"TDur\\\"]], [[10, \\\"mDegr\\\"]], [[15, \\\"Desc\\\"]], [[10, \\\"mAux\\\"]], [[13, \\\"Quan\\\"]], [[15, \\\"Desc\\\"]], [[15, \\\"Nmod\\\"]], [[6, \\\"Cont\\\"]], [[6, \\\"mPunc\\\"]]]\\n\",\n      \"  ]\\n\",\n      \"}\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(doc)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"`doc['sdp']`字段代表语义依存图的数组格式，数组中第`i`个子数组代表第`i`个单词的语义依存关系，子数组中每个二元组的格式为`[中心词的下标, 与中心词的语义依存关系]`。每个单词的语义依存关系可能有零个、一个或多个（任意数量）。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"转换为[CoNLLSentence](https://hanlp.hankcs.com/docs/api/common/conll.html#hanlp_common.conll.CoNLLSentence)格式更容易观察：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"1\\t2021年\\t_\\t_\\t_\\t_\\t_\\t_\\t6:Time\\t_\\n\",\n      \"2\\tHanLPv2.1\\t_\\t_\\t_\\t_\\t_\\t_\\t6:Agt\\t_\\n\",\n      \"3\\t为\\t_\\t_\\t_\\t_\\t_\\t_\\t5:mPrep\\t_\\n\",\n      \"4\\t生产\\t_\\t_\\t_\\t_\\t_\\t_\\t5:Desc\\t_\\n\",\n      \"5\\t环境\\t_\\t_\\t_\\t_\\t_\\t_\\t6:Datv\\t_\\n\",\n      \"6\\t带来\\t_\\t_\\t_\\t_\\t_\\t_\\t0:Root\\t_\\n\",\n      \"7\\t次\\t_\\t_\\t_\\t_\\t_\\t_\\t8:Qp\\t_\\n\",\n      \"8\\t世代\\t_\\t_\\t_\\t_\\t_\\t_\\t15:TDur\\t_\\n\",\n      \"9\\t最\\t_\\t_\\t_\\t_\\t_\\t_\\t10:mDegr\\t_\\n\",\n      \"10\\t先进\\t_\\t_\\t_\\t_\\t_\\t_\\t15:Desc\\t_\\n\",\n      \"11\\t的\\t_\\t_\\t_\\t_\\t_\\t_\\t10:mAux\\t_\\n\",\n      \"12\\t多\\t_\\t_\\t_\\t_\\t_\\t_\\t13:Quan\\t_\\n\",\n      \"13\\t语种\\t_\\t_\\t_\\t_\\t_\\t_\\t15:Desc\\t_\\n\",\n      \"14\\tNLP\\t_\\t_\\t_\\t_\\t_\\t_\\t15:Nmod\\t_\\n\",\n      \"15\\t技术\\t_\\t_\\t_\\t_\\t_\\t_\\t6:Cont\\t_\\n\",\n      \"16\\t。\\t_\\t_\\t_\\t_\\t_\\t_\\t6:mPunc\\t_\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(doc.to_conll())\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"XOsWkOqQfzlr\"\n   },\n   \"source\": [\n    \"为已分词的句子执行语义依存分析：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"bLZSTbv_f3OA\",\n    \"outputId\": \"111c0be9-bac6-4eee-d5bd-a972ffc34844\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"1\\tHanLP\\t_\\t_\\t_\\t_\\t_\\t_\\t5:Agt\\t_\\n\",\n      \"2\\t为\\t_\\t_\\t_\\t_\\t_\\t_\\t4:mPrep\\t_\\n\",\n      \"3\\t生产\\t_\\t_\\t_\\t_\\t_\\t_\\t4:Desc\\t_\\n\",\n      \"4\\t环境\\t_\\t_\\t_\\t_\\t_\\t_\\t5:Datv\\t_\\n\",\n      \"5\\t带来\\t_\\t_\\t_\\t_\\t_\\t_\\t0:Root\\t_\\n\",\n      \"6\\t次世代\\t_\\t_\\t_\\t_\\t_\\t_\\t12:Time\\t_\\n\",\n      \"7\\t最\\t_\\t_\\t_\\t_\\t_\\t_\\t8:mDegr\\t_\\n\",\n      \"8\\t先进\\t_\\t_\\t_\\t_\\t_\\t_\\t12:Desc\\t_\\n\",\n      \"9\\t的\\t_\\t_\\t_\\t_\\t_\\t_\\t8:mAux\\t_\\n\",\n      \"10\\t多语种\\t_\\t_\\t_\\t_\\t_\\t_\\t12:Desc\\t_\\n\",\n      \"11\\tNLP\\t_\\t_\\t_\\t_\\t_\\t_\\t12:Nmod\\t_\\n\",\n      \"12\\t技术\\t_\\t_\\t_\\t_\\t_\\t_\\t5:Cont\\t_\\n\",\n      \"13\\t。\\t_\\t_\\t_\\t_\\t_\\t_\\t5:mPunc\\t_\\n\",\n      \"\\n\",\n      \"1\\t我\\t_\\t_\\t_\\t_\\t_\\t_\\t3:Poss\\t_\\n\",\n      \"2\\t的\\t_\\t_\\t_\\t_\\t_\\t_\\t1:mAux\\t_\\n\",\n      \"3\\t希望\\t_\\t_\\t_\\t_\\t_\\t_\\t0:Root|4:Exp\\t_\\n\",\n      \"4\\t是\\t_\\t_\\t_\\t_\\t_\\t_\\t5:mMod\\t_\\n\",\n      \"5\\t希望\\t_\\t_\\t_\\t_\\t_\\t_\\t4:dClas\\t_\\n\",\n      \"6\\t张晚霞\\t_\\t_\\t_\\t_\\t_\\t_\\t8:Poss\\t_\\n\",\n      \"7\\t的\\t_\\t_\\t_\\t_\\t_\\t_\\t6:mAux\\t_\\n\",\n      \"8\\t背影\\t_\\t_\\t_\\t_\\t_\\t_\\t11:Pat\\t_\\n\",\n      \"9\\t被\\t_\\t_\\t_\\t_\\t_\\t_\\t10:mPrep\\t_\\n\",\n      \"10\\t晚霞\\t_\\t_\\t_\\t_\\t_\\t_\\t11:Exp\\t_\\n\",\n      \"11\\t映红\\t_\\t_\\t_\\t_\\t_\\t_\\t5:dCont\\t_\\n\",\n      \"12\\t。\\t_\\t_\\t_\\t_\\t_\\t_\\t5:mPunc\\t_\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(HanLP(tokens=[\\n\",\n    \"    [\\\"HanLP\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"],\\n\",\n    \"    [\\\"我\\\", \\\"的\\\", \\\"希望\\\", \\\"是\\\", \\\"希望\\\", \\\"张晚霞\\\", \\\"的\\\", \\\"背影\\\", \\\"被\\\", \\\"晚霞\\\", \\\"映红\\\", \\\"。\\\"]\\n\",\n    \"  ], tasks='sdp').to_conll())\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"sdp_restful.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/sdp_stl.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/sdp_stl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fsdp_stl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"nf9TgeCTC0OT\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"jaW4eu6kC0OU\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"_xI_bLAaC0OU\"\n   },\n   \"source\": [\n    \"## 加载模型\\n\",\n    \"HanLP的工作流程是先加载模型，模型的标示符存储在`hanlp.pretrained`这个包中，按照NLP任务归类。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"IYwV-UkNNzFp\",\n    \"outputId\": \"54065443-9b0a-444c-f6c0-c701bc86400b\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'SEMEVAL16_NEWS_BIAFFINE_ZH': 'https://file.hankcs.com/hanlp/sdp/semeval16-news-biaffine_20191231_235407.zip',\\n\",\n       \" 'SEMEVAL16_TEXT_BIAFFINE_ZH': 'https://file.hankcs.com/hanlp/sdp/semeval16-text-biaffine_20200101_002257.zip',\\n\",\n       \" 'SEMEVAL16_ALL_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/sdp/semeval16_sdp_electra_small_20220208_122026.zip',\\n\",\n       \" 'SEMEVAL15_PAS_BIAFFINE_EN': 'https://file.hankcs.com/hanlp/sdp/semeval15_biaffine_pas_20200103_152405.zip',\\n\",\n       \" 'SEMEVAL15_PSD_BIAFFINE_EN': 'https://file.hankcs.com/hanlp/sdp/semeval15_biaffine_psd_20200106_123009.zip',\\n\",\n       \" 'SEMEVAL15_DM_BIAFFINE_EN': 'https://file.hankcs.com/hanlp/sdp/semeval15_biaffine_dm_20200106_122808.zip'}\"\n      ]\n     },\n     \"execution_count\": 1,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"import hanlp\\n\",\n    \"hanlp.pretrained.sdp.ALL # 语种见名称最后一个字段或相应语料库\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\",\n    \"pycharm\": {\n     \"name\": \"#%% md\\n\"\n    }\n   },\n   \"source\": [\n    \"调用`hanlp.load`进行加载，模型会自动下载到本地缓存。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"sdp = hanlp.load('SEMEVAL16_ALL_ELECTRA_SMALL_ZH')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 语义依存分析\\n\",\n    \"语义依存分析的输入为已分词的一个或多个句子：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {\n    \"id\": \"BqEmDMGGOtk3\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"graph = sdp([\\\"2021年\\\", \\\"HanLPv2.1\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次\\\", \\\"世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多\\\", \\\"语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"])\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"SwaPn1hjC0OW\"\n   },\n   \"source\": [\n    \"返回对象为[CoNLLSentence](https://hanlp.hankcs.com/docs/api/common/conll.html#hanlp_common.conll.CoNLLSentence)类型：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"egpWwHKxC0OX\",\n    \"outputId\": \"f7c77687-dd75-4fa2-dbd2-be6bda8a3fff\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"[{'id': 1,\\n\",\n       \"  'form': '2021年',\\n\",\n       \"  'upos': None,\\n\",\n       \"  'xpos': None,\\n\",\n       \"  'head': None,\\n\",\n       \"  'deprel': None,\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'deps': [(6, 'Time')],\\n\",\n       \"  'misc': None},\\n\",\n       \" {'id': 2,\\n\",\n       \"  'form': 'HanLPv2.1',\\n\",\n       \"  'upos': None,\\n\",\n       \"  'xpos': None,\\n\",\n       \"  'head': None,\\n\",\n       \"  'deprel': None,\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'deps': [(6, 'Exp')],\\n\",\n       \"  'misc': None},\\n\",\n       \" {'id': 3,\\n\",\n       \"  'form': '为',\\n\",\n       \"  'upos': None,\\n\",\n       \"  'xpos': None,\\n\",\n       \"  'head': None,\\n\",\n       \"  'deprel': None,\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'deps': [(5, 'mPrep')],\\n\",\n       \"  'misc': None},\\n\",\n       \" {'id': 4,\\n\",\n       \"  'form': '生产',\\n\",\n       \"  'upos': None,\\n\",\n       \"  'xpos': None,\\n\",\n       \"  'head': None,\\n\",\n       \"  'deprel': None,\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'deps': [(5, 'Desc')],\\n\",\n       \"  'misc': None},\\n\",\n       \" {'id': 5,\\n\",\n       \"  'form': '环境',\\n\",\n       \"  'upos': None,\\n\",\n       \"  'xpos': None,\\n\",\n       \"  'head': None,\\n\",\n       \"  'deprel': None,\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'deps': [(6, 'Datv')],\\n\",\n       \"  'misc': None},\\n\",\n       \" {'id': 6,\\n\",\n       \"  'form': '带来',\\n\",\n       \"  'upos': None,\\n\",\n       \"  'xpos': None,\\n\",\n       \"  'head': None,\\n\",\n       \"  'deprel': None,\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'deps': [(2, 'eSucc')],\\n\",\n       \"  'misc': None},\\n\",\n       \" {'id': 7,\\n\",\n       \"  'form': '次',\\n\",\n       \"  'upos': None,\\n\",\n       \"  'xpos': None,\\n\",\n       \"  'head': None,\\n\",\n       \"  'deprel': None,\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'deps': [(8, 'Desc'), (13, 'Desc')],\\n\",\n       \"  'misc': None},\\n\",\n       \" {'id': 8,\\n\",\n       \"  'form': '世代',\\n\",\n       \"  'upos': None,\\n\",\n       \"  'xpos': None,\\n\",\n       \"  'head': None,\\n\",\n       \"  'deprel': None,\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'deps': [(0, 'Root'), (15, 'Time')],\\n\",\n       \"  'misc': None},\\n\",\n       \" {'id': 9,\\n\",\n       \"  'form': '最',\\n\",\n       \"  'upos': None,\\n\",\n       \"  'xpos': None,\\n\",\n       \"  'head': None,\\n\",\n       \"  'deprel': None,\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'deps': [(10, 'mDegr')],\\n\",\n       \"  'misc': None},\\n\",\n       \" {'id': 10,\\n\",\n       \"  'form': '先进',\\n\",\n       \"  'upos': None,\\n\",\n       \"  'xpos': None,\\n\",\n       \"  'head': None,\\n\",\n       \"  'deprel': None,\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'deps': [(15, 'Desc')],\\n\",\n       \"  'misc': None},\\n\",\n       \" {'id': 11,\\n\",\n       \"  'form': '的',\\n\",\n       \"  'upos': None,\\n\",\n       \"  'xpos': None,\\n\",\n       \"  'head': None,\\n\",\n       \"  'deprel': None,\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'deps': [(10, 'mAux')],\\n\",\n       \"  'misc': None},\\n\",\n       \" {'id': 12,\\n\",\n       \"  'form': '多',\\n\",\n       \"  'upos': None,\\n\",\n       \"  'xpos': None,\\n\",\n       \"  'head': None,\\n\",\n       \"  'deprel': None,\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'deps': [(10, 'mDegr'), (13, 'Quan')],\\n\",\n       \"  'misc': None},\\n\",\n       \" {'id': 13,\\n\",\n       \"  'form': '语种',\\n\",\n       \"  'upos': None,\\n\",\n       \"  'xpos': None,\\n\",\n       \"  'head': None,\\n\",\n       \"  'deprel': None,\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'deps': [(15, 'Desc')],\\n\",\n       \"  'misc': None},\\n\",\n       \" {'id': 14,\\n\",\n       \"  'form': 'NLP',\\n\",\n       \"  'upos': None,\\n\",\n       \"  'xpos': None,\\n\",\n       \"  'head': None,\\n\",\n       \"  'deprel': None,\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'deps': [(15, 'Desc')],\\n\",\n       \"  'misc': None},\\n\",\n       \" {'id': 15,\\n\",\n       \"  'form': '技术',\\n\",\n       \"  'upos': None,\\n\",\n       \"  'xpos': None,\\n\",\n       \"  'head': None,\\n\",\n       \"  'deprel': None,\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'deps': [(6, 'Pat')],\\n\",\n       \"  'misc': None},\\n\",\n       \" {'id': 16,\\n\",\n       \"  'form': '。',\\n\",\n       \"  'upos': None,\\n\",\n       \"  'xpos': None,\\n\",\n       \"  'head': None,\\n\",\n       \"  'deprel': None,\\n\",\n       \"  'lemma': None,\\n\",\n       \"  'feats': None,\\n\",\n       \"  'deps': [(6, 'mPunc')],\\n\",\n       \"  'misc': None}]\"\n      ]\n     },\n     \"execution_count\": 5,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"graph\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"kq_j5TLFC0OX\"\n   },\n   \"source\": [\n    \"打印为为CoNLL格式：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"isJhzYyIC0OX\",\n    \"outputId\": \"683c8489-dffc-426e-f95b-e91dfb373260\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"1\\t2021年\\t_\\t_\\t_\\t_\\t_\\t_\\t6:Time\\t_\\n\",\n      \"2\\tHanLPv2.1\\t_\\t_\\t_\\t_\\t_\\t_\\t6:Exp\\t_\\n\",\n      \"3\\t为\\t_\\t_\\t_\\t_\\t_\\t_\\t5:mPrep\\t_\\n\",\n      \"4\\t生产\\t_\\t_\\t_\\t_\\t_\\t_\\t5:Desc\\t_\\n\",\n      \"5\\t环境\\t_\\t_\\t_\\t_\\t_\\t_\\t6:Datv\\t_\\n\",\n      \"6\\t带来\\t_\\t_\\t_\\t_\\t_\\t_\\t2:eSucc\\t_\\n\",\n      \"7\\t次\\t_\\t_\\t_\\t_\\t_\\t_\\t8:Desc|13:Desc\\t_\\n\",\n      \"8\\t世代\\t_\\t_\\t_\\t_\\t_\\t_\\t0:Root|15:Time\\t_\\n\",\n      \"9\\t最\\t_\\t_\\t_\\t_\\t_\\t_\\t10:mDegr\\t_\\n\",\n      \"10\\t先进\\t_\\t_\\t_\\t_\\t_\\t_\\t15:Desc\\t_\\n\",\n      \"11\\t的\\t_\\t_\\t_\\t_\\t_\\t_\\t10:mAux\\t_\\n\",\n      \"12\\t多\\t_\\t_\\t_\\t_\\t_\\t_\\t10:mDegr|13:Quan\\t_\\n\",\n      \"13\\t语种\\t_\\t_\\t_\\t_\\t_\\t_\\t15:Desc\\t_\\n\",\n      \"14\\tNLP\\t_\\t_\\t_\\t_\\t_\\t_\\t15:Desc\\t_\\n\",\n      \"15\\t技术\\t_\\t_\\t_\\t_\\t_\\t_\\t6:Pat\\t_\\n\",\n      \"16\\t。\\t_\\t_\\t_\\t_\\t_\\t_\\t6:mPunc\\t_\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(graph)\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"sdp_stl.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/sentiment_restful.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/sentiment_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fsentiment_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"nf9TgeCTC0OT\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"jaW4eu6kC0OU\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp_restful -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"_xI_bLAaC0OU\"\n   },\n   \"source\": [\n    \"## 创建客户端\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"IYwV-UkNNzFp\",\n    \"outputId\": \"54065443-9b0a-444c-f6c0-c701bc86400b\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from hanlp_restful import HanLPClient\\n\",\n    \"HanLP = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh') # auth不填则匿名，zh中文，mul多语种\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\",\n    \"pycharm\": {\n     \"name\": \"#%% md\\n\"\n    }\n   },\n   \"source\": [\n    \"#### 申请秘钥\\n\",\n    \"由于服务器算力有限，匿名用户每分钟限2次调用。如果你需要更多调用次数，[建议申请免费公益API秘钥auth](https://bbs.hanlp.com/t/hanlp2-1-restful-api/53)。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 情感分析\\n\",\n    \"情感分析任务的输入为文档：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"id\": \"BqEmDMGGOtk3\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"0.8418035507202148\"\n      ]\n     },\n     \"execution_count\": 2,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"HanLP.sentiment_analysis('2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"SwaPn1hjC0OW\"\n   },\n   \"source\": [\n    \"返回值为文档的情感极性，表示为$[-1, +1]$之间的数值，数值的正负代表正负面情绪，数值的绝对值代表情感的强烈程度。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"egpWwHKxC0OX\",\n    \"outputId\": \"f7c77687-dd75-4fa2-dbd2-be6bda8a3fff\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"0.8327275514602661\"\n      ]\n     },\n     \"execution_count\": 3,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"HanLP.sentiment_analysis('看哭了。感人肺腑。')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"kq_j5TLFC0OX\"\n   },\n   \"source\": [\n    \"注意返回值的符号代表正负情感：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"isJhzYyIC0OX\",\n    \"outputId\": \"683c8489-dffc-426e-f95b-e91dfb373260\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"-0.8850911855697632\"\n      ]\n     },\n     \"execution_count\": 4,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"HanLP.sentiment_analysis('看哭了。难看哭了。')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"绝对值的大小代表情感的强烈程度：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"-0.9190718531608582\"\n      ]\n     },\n     \"execution_count\": 5,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"HanLP.sentiment_analysis('看哭了。难看哭了！！！')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"长文档一样支持：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {\n    \"scrolled\": true\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"0.9505730271339417\"\n      ]\n     },\n     \"execution_count\": 6,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"text = '''“这是一部男人必看的电影。”人人都这么说。但单纯从性别区分，就会让这电影变狭隘。\\n\",\n    \"《肖申克的救赎》突破了男人电影的局限，通篇几乎充满令人难以置信的温馨基调，而电影里最伟大的主题是“希望”。\\n\",\n    \"当我们无奈地遇到了如同肖申克一般囚禁了心灵自由的那种囹圄，我们是无奈的老布鲁克，灰心的瑞德，还是智慧的安迪？\\n\",\n    \"运用智慧，信任希望，并且勇敢面对恐惧心理，去打败它？\\n\",\n    \"经典的电影之所以经典，因为他们都在做同一件事——让你从不同的角度来欣赏希望的美好。'''\\n\",\n    \"HanLP.sentiment_analysis(text)\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"sentiment_restful.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/srl_mtl.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/srl_mtl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fsrl_mtl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 加载模型\\n\",\n    \"HanLP的工作流程是先加载模型，模型的标示符存储在`hanlp.pretrained`这个包中，按照NLP任务归类。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"id\": \"0tmKBu7sNAXX\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'OPEN_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/mtl/open_tok_pos_ner_srl_dep_sdp_con_electra_small_20201223_035557.zip',\\n\",\n       \" 'OPEN_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH': 'https://file.hankcs.com/hanlp/mtl/open_tok_pos_ner_srl_dep_sdp_con_electra_base_20201223_201906.zip',\\n\",\n       \" 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_small_20210111_124159.zip',\\n\",\n       \" 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_base_20210111_124519.zip',\\n\",\n       \" 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ERNIE_GRAM_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_ernie_gram_base_aug_20210904_145403.zip',\\n\",\n       \" 'UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_MT5_SMALL': 'https://file.hankcs.com/hanlp/mtl/ud_ontonotes_tok_pos_lem_fea_ner_srl_dep_sdp_con_mt5_small_20210228_123458.zip',\\n\",\n       \" 'UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_XLMR_BASE': 'https://file.hankcs.com/hanlp/mtl/ud_ontonotes_tok_pos_lem_fea_ner_srl_dep_sdp_con_xlm_base_20210602_211620.zip',\\n\",\n       \" 'NPCMJ_UD_KYOTO_TOK_POS_CON_BERT_BASE_CHAR_JA': 'https://file.hankcs.com/hanlp/mtl/npcmj_ud_kyoto_tok_pos_ner_dep_con_srl_bert_base_char_ja_20210914_133742.zip'}\"\n      ]\n     },\n     \"execution_count\": 1,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"import hanlp\\n\",\n    \"hanlp.pretrained.mtl.ALL # MTL多任务，具体任务见模型名称，语种见名称最后一个字段或相应语料库\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"EmZDmLn9aGxG\"\n   },\n   \"source\": [\n    \"调用`hanlp.load`进行加载，模型会自动下载到本地缓存。自然语言处理分为许多任务，分词只是最初级的一个。与其每个任务单独创建一个模型，不如利用HanLP的联合模型一次性完成多个任务：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"HanLP = hanlp.load(hanlp.pretrained.mtl.CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 语义角色分析\\n\",\n    \"任务越少，速度越快。如指定仅执行语义角色分析：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"2a0d392f-b99a-4a18-fc7f-754e2abe2e34\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"doc = HanLP('2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', tasks='srl')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"返回值为一个[Document](https://hanlp.hankcs.com/docs/api/common/document.html):\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"{\\n\",\n      \"  \\\"tok/fine\\\": [\\n\",\n      \"    \\\"2021年\\\",\\n\",\n      \"    \\\"HanLPv2.1\\\",\\n\",\n      \"    \\\"为\\\",\\n\",\n      \"    \\\"生产\\\",\\n\",\n      \"    \\\"环境\\\",\\n\",\n      \"    \\\"带来\\\",\\n\",\n      \"    \\\"次\\\",\\n\",\n      \"    \\\"世代\\\",\\n\",\n      \"    \\\"最\\\",\\n\",\n      \"    \\\"先进\\\",\\n\",\n      \"    \\\"的\\\",\\n\",\n      \"    \\\"多\\\",\\n\",\n      \"    \\\"语种\\\",\\n\",\n      \"    \\\"NLP\\\",\\n\",\n      \"    \\\"技术\\\",\\n\",\n      \"    \\\"。\\\"\\n\",\n      \"  ],\\n\",\n      \"  \\\"srl\\\": [\\n\",\n      \"    [[\\\"2021年\\\", \\\"ARGM-TMP\\\", 0, 1], [\\\"HanLPv2.1\\\", \\\"ARG0\\\", 1, 2], [\\\"为生产环境\\\", \\\"ARG2\\\", 2, 5], [\\\"带来\\\", \\\"PRED\\\", 5, 6], [\\\"次世代最先进的多语种NLP技术\\\", \\\"ARG1\\\", 6, 15]],\\n\",\n      \"    [[\\\"最\\\", \\\"ARGM-ADV\\\", 8, 9], [\\\"先进\\\", \\\"PRED\\\", 9, 10], [\\\"技术\\\", \\\"ARG0\\\", 14, 15]]\\n\",\n      \"  ]\\n\",\n      \"}\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(doc)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"`doc['srl']`字段为语义角色标注结果，每个四元组的格式为`[论元或谓词, 语义角色标签, 起始下标, 终止下标]`。其中，谓词的语义角色标签为`PRED`，起止下标对应以`tok`开头的第一个单词数组。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"wxctCigrTKu-\"\n   },\n   \"source\": [\n    \"可视化谓词论元结构：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"Zo08uquCTFSk\",\n    \"outputId\": \"c6077f2d-7084-4f4b-a3bc-9aa9951704ea\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Token    \\tSRL PA1     \\tToken    \\tSRL PA2     \\n\",\n      \"─────────\\t────────────\\t─────────\\t────────────\\n\",\n      \"2021年    \\t───►ARGM-TMP\\t2021年    \\t            \\n\",\n      \"HanLPv2.1\\t───►ARG0    \\tHanLPv2.1\\t            \\n\",\n      \"为        \\t◄─┐         \\t为        \\t            \\n\",\n      \"生产       \\t  ├►ARG2    \\t生产       \\t            \\n\",\n      \"环境       \\t◄─┘         \\t环境       \\t            \\n\",\n      \"带来       \\t╟──►PRED    \\t带来       \\t            \\n\",\n      \"次        \\t◄─┐         \\t次        \\t            \\n\",\n      \"世代       \\t  │         \\t世代       \\t            \\n\",\n      \"最        \\t  │         \\t最        \\t───►ARGM-ADV\\n\",\n      \"先进       \\t  │         \\t先进       \\t╟──►PRED    \\n\",\n      \"的        \\t  ├►ARG1    \\t的        \\t            \\n\",\n      \"多        \\t  │         \\t多        \\t            \\n\",\n      \"语种       \\t  │         \\t语种       \\t            \\n\",\n      \"NLP      \\t  │         \\tNLP      \\t            \\n\",\n      \"技术       \\t◄─┘         \\t技术       \\t───►ARG0    \\n\",\n      \"。        \\t            \\t。        \\t            \\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"doc.pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"遍历谓词论元结构：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"第1个谓词论元结构：\\n\",\n      \"2021年 = ARGM-TMP at [0, 1]\\n\",\n      \"HanLPv2.1 = ARG0 at [1, 2]\\n\",\n      \"为生产环境 = ARG2 at [2, 5]\\n\",\n      \"带来 = PRED at [5, 6]\\n\",\n      \"次世代最先进的多语种NLP技术 = ARG1 at [6, 15]\\n\",\n      \"第2个谓词论元结构：\\n\",\n      \"最 = ARGM-ADV at [8, 9]\\n\",\n      \"先进 = PRED at [9, 10]\\n\",\n      \"技术 = ARG0 at [14, 15]\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"for i, pas in enumerate(doc['srl']):\\n\",\n    \"    print(f'第{i+1}个谓词论元结构：')\\n\",\n    \"    for form, role, begin, end in pas:\\n\",\n    \"        print(f'{form} = {role} at [{begin}, {end}]')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"XOsWkOqQfzlr\"\n   },\n   \"source\": [\n    \"为已分词的句子执行语义角色分析：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"bLZSTbv_f3OA\",\n    \"outputId\": \"111c0be9-bac6-4eee-d5bd-a972ffc34844\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Token\\tSRL PA1 \\tToken\\tSRL PA2     \\n\",\n      \"─────\\t────────\\t─────\\t────────────\\n\",\n      \"HanLP\\t───►ARG0\\tHanLP\\t            \\n\",\n      \"为    \\t◄─┐     \\t为    \\t            \\n\",\n      \"生产   \\t  ├►ARG2\\t生产   \\t            \\n\",\n      \"环境   \\t◄─┘     \\t环境   \\t            \\n\",\n      \"带来   \\t╟──►PRED\\t带来   \\t            \\n\",\n      \"次世代  \\t◄─┐     \\t次世代  \\t            \\n\",\n      \"最    \\t  │     \\t最    \\t───►ARGM-ADV\\n\",\n      \"先进   \\t  │     \\t先进   \\t╟──►PRED    \\n\",\n      \"的    \\t  ├►ARG1\\t的    \\t            \\n\",\n      \"多语种  \\t  │     \\t多语种  \\t            \\n\",\n      \"NLP  \\t  │     \\tNLP  \\t            \\n\",\n      \"技术   \\t◄─┘     \\t技术   \\t───►ARG0    \\n\",\n      \"。    \\t        \\t。    \\t            \\n\",\n      \"\\n\",\n      \"Tok\\tSRL PA1 \\tTok\\tSRL PA2 \\tTok\\tSRL PA3 \\n\",\n      \"───\\t────────\\t───\\t────────\\t───\\t────────\\n\",\n      \"我  \\t◄─┐     \\t我  \\t        \\t我  \\t        \\n\",\n      \"的  \\t  ├►ARG0\\t的  \\t        \\t的  \\t        \\n\",\n      \"希望 \\t◄─┘     \\t希望 \\t        \\t希望 \\t        \\n\",\n      \"是  \\t╟──►PRED\\t是  \\t        \\t是  \\t        \\n\",\n      \"希望 \\t◄─┐     \\t希望 \\t╟──►PRED\\t希望 \\t        \\n\",\n      \"张晚霞\\t  │     \\t张晚霞\\t◄─┐     \\t张晚霞\\t        \\n\",\n      \"的  \\t  │     \\t的  \\t  │     \\t的  \\t        \\n\",\n      \"背影 \\t  ├►ARG1\\t背影 \\t  │     \\t背影 \\t        \\n\",\n      \"被  \\t  │     \\t被  \\t  ├►ARG1\\t被  \\t        \\n\",\n      \"晚霞 \\t  │     \\t晚霞 \\t  │     \\t晚霞 \\t───►ARG0\\n\",\n      \"映红 \\t◄─┘     \\t映红 \\t◄─┘     \\t映红 \\t╟──►PRED\\n\",\n      \"。  \\t        \\t。  \\t        \\t。  \\t        \\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"HanLP([\\n\",\n    \"    [\\\"HanLP\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"],\\n\",\n    \"    [\\\"我\\\", \\\"的\\\", \\\"希望\\\", \\\"是\\\", \\\"希望\\\", \\\"张晚霞\\\", \\\"的\\\", \\\"背影\\\", \\\"被\\\", \\\"晚霞\\\", \\\"映红\\\", \\\"。\\\"]\\n\",\n    \"  ], tasks='srl', skip_tasks='tok*').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"#### 注意\\n\",\n    \"Native API的输入单位限定为句子，需使用[多语种分句模型](https://github.com/hankcs/HanLP/blob/master/plugins/hanlp_demo/hanlp_demo/sent_split.py)或[基于规则的分句函数](https://github.com/hankcs/HanLP/blob/master/hanlp/utils/rules.py#L19)先行分句。RESTful同时支持全文、句子、已分词的句子。除此之外，RESTful和native两种API的语义设计完全一致，用户可以无缝互换。\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"srl_mtl.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/srl_restful.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/srl_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fsrl_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp_restful -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 创建客户端\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"id\": \"0tmKBu7sNAXX\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from hanlp_restful import HanLPClient\\n\",\n    \"HanLP = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh') # auth不填则匿名，zh中文，mul多语种\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"EmZDmLn9aGxG\"\n   },\n   \"source\": [\n    \"#### 申请秘钥\\n\",\n    \"由于服务器算力有限，匿名用户每分钟限2次调用。如果你需要更多调用次数，[建议申请免费公益API秘钥auth](https://bbs.hanlp.com/t/hanlp2-1-restful-api/53)。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 语义角色分析\\n\",\n    \"任务越少，速度越快。如指定仅执行语义角色分析：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"2a0d392f-b99a-4a18-fc7f-754e2abe2e34\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"doc = HanLP('2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', tasks='srl')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"返回值为一个[Document](https://hanlp.hankcs.com/docs/api/common/document.html):\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"{\\n\",\n      \"  \\\"tok/fine\\\": [\\n\",\n      \"    [\\\"2021年\\\", \\\"HanLPv2.1\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次\\\", \\\"世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多\\\", \\\"语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"srl\\\": [\\n\",\n      \"    [[[\\\"2021年\\\", \\\"ARGM-TMP\\\", 0, 1], [\\\"HanLPv2.1\\\", \\\"ARG0\\\", 1, 2], [\\\"为生产环境\\\", \\\"ARG2\\\", 2, 5], [\\\"带来\\\", \\\"PRED\\\", 5, 6], [\\\"次世代最先进的多语种NLP技术\\\", \\\"ARG1\\\", 6, 15]], [[\\\"次世代\\\", \\\"ARGM-TMP\\\", 6, 8], [\\\"最\\\", \\\"ARGM-ADV\\\", 8, 9], [\\\"先进\\\", \\\"PRED\\\", 9, 10], [\\\"NLP技术\\\", \\\"ARG0\\\", 13, 15]]]\\n\",\n      \"  ]\\n\",\n      \"}\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(doc)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"`doc['srl']`字段为语义角色标注结果，每个四元组的格式为`[论元或谓词, 语义角色标签, 起始下标, 终止下标]`。其中，谓词的语义角色标签为`PRED`，起止下标对应以`tok`开头的第一个单词数组。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"wxctCigrTKu-\"\n   },\n   \"source\": [\n    \"可视化谓词论元结构：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"Zo08uquCTFSk\",\n    \"outputId\": \"c6077f2d-7084-4f4b-a3bc-9aa9951704ea\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Token    \\tSRL PA1     \\tToken    \\tSRL PA2     \\n\",\n      \"─────────\\t────────────\\t─────────\\t────────────\\n\",\n      \"2021年    \\t───►ARGM-TMP\\t2021年    \\t            \\n\",\n      \"HanLPv2.1\\t───►ARG0    \\tHanLPv2.1\\t            \\n\",\n      \"为        \\t◄─┐         \\t为        \\t            \\n\",\n      \"生产       \\t  ├►ARG2    \\t生产       \\t            \\n\",\n      \"环境       \\t◄─┘         \\t环境       \\t            \\n\",\n      \"带来       \\t╟──►PRED    \\t带来       \\t            \\n\",\n      \"次        \\t◄─┐         \\t次        \\t◄─┐         \\n\",\n      \"世代       \\t  │         \\t世代       \\t◄─┴►ARGM-TMP\\n\",\n      \"最        \\t  │         \\t最        \\t───►ARGM-ADV\\n\",\n      \"先进       \\t  │         \\t先进       \\t╟──►PRED    \\n\",\n      \"的        \\t  ├►ARG1    \\t的        \\t            \\n\",\n      \"多        \\t  │         \\t多        \\t            \\n\",\n      \"语种       \\t  │         \\t语种       \\t            \\n\",\n      \"NLP      \\t  │         \\tNLP      \\t◄─┐         \\n\",\n      \"技术       \\t◄─┘         \\t技术       \\t◄─┴►ARG0    \\n\",\n      \"。        \\t            \\t。        \\t            \\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"doc.pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"遍历谓词论元结构：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 9,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"第1个谓词论元结构：\\n\",\n      \"2021年 = ARGM-TMP at [0, 1]\\n\",\n      \"HanLPv2.1 = ARG0 at [1, 2]\\n\",\n      \"为生产环境 = ARG2 at [2, 5]\\n\",\n      \"带来 = PRED at [5, 6]\\n\",\n      \"次世代最先进的多语种NLP技术 = ARG1 at [6, 15]\\n\",\n      \"第2个谓词论元结构：\\n\",\n      \"次世代 = ARGM-TMP at [6, 8]\\n\",\n      \"最 = ARGM-ADV at [8, 9]\\n\",\n      \"先进 = PRED at [9, 10]\\n\",\n      \"NLP技术 = ARG0 at [13, 15]\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"for i, pas in enumerate(doc['srl'][0]):\\n\",\n    \"    print(f'第{i+1}个谓词论元结构：')\\n\",\n    \"    for form, role, begin, end in pas:\\n\",\n    \"        print(f'{form} = {role} at [{begin}, {end}]')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"XOsWkOqQfzlr\"\n   },\n   \"source\": [\n    \"为已分词的句子执行语义角色分析：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 11,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"bLZSTbv_f3OA\",\n    \"outputId\": \"111c0be9-bac6-4eee-d5bd-a972ffc34844\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Token\\tSRL PA1 \\tToken\\tSRL PA2     \\n\",\n      \"─────\\t────────\\t─────\\t────────────\\n\",\n      \"HanLP\\t───►ARG0\\tHanLP\\t            \\n\",\n      \"为    \\t◄─┐     \\t为    \\t            \\n\",\n      \"生产   \\t  ├►ARG2\\t生产   \\t            \\n\",\n      \"环境   \\t◄─┘     \\t环境   \\t            \\n\",\n      \"带来   \\t╟──►PRED\\t带来   \\t            \\n\",\n      \"次世代  \\t◄─┐     \\t次世代  \\t───►ARGM-TMP\\n\",\n      \"最    \\t  │     \\t最    \\t───►ARGM-ADV\\n\",\n      \"先进   \\t  │     \\t先进   \\t╟──►PRED    \\n\",\n      \"的    \\t  ├►ARG1\\t的    \\t            \\n\",\n      \"多语种  \\t  │     \\t多语种  \\t            \\n\",\n      \"NLP  \\t  │     \\tNLP  \\t            \\n\",\n      \"技术   \\t◄─┘     \\t技术   \\t───►ARG0    \\n\",\n      \"。    \\t        \\t。    \\t            \\n\",\n      \"\\n\",\n      \"Tok\\tSRL PA1 \\tTok\\tSRL PA2 \\tTok\\tSRL PA3 \\n\",\n      \"───\\t────────\\t───\\t────────\\t───\\t────────\\n\",\n      \"我  \\t◄─┐     \\t我  \\t        \\t我  \\t        \\n\",\n      \"的  \\t  ├►ARG0\\t的  \\t        \\t的  \\t        \\n\",\n      \"希望 \\t◄─┘     \\t希望 \\t        \\t希望 \\t        \\n\",\n      \"是  \\t╟──►PRED\\t是  \\t        \\t是  \\t        \\n\",\n      \"希望 \\t◄─┐     \\t希望 \\t╟──►PRED\\t希望 \\t        \\n\",\n      \"张晚霞\\t  │     \\t张晚霞\\t◄─┐     \\t张晚霞\\t◄─┐     \\n\",\n      \"的  \\t  │     \\t的  \\t  │     \\t的  \\t  ├►ARG1\\n\",\n      \"背影 \\t  ├►ARG1\\t背影 \\t  │     \\t背影 \\t◄─┘     \\n\",\n      \"被  \\t  │     \\t被  \\t  ├►ARG1\\t被  \\t        \\n\",\n      \"晚霞 \\t  │     \\t晚霞 \\t  │     \\t晚霞 \\t───►ARG0\\n\",\n      \"映红 \\t◄─┘     \\t映红 \\t◄─┘     \\t映红 \\t╟──►PRED\\n\",\n      \"。  \\t        \\t。  \\t        \\t。  \\t        \\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"HanLP(tokens=[\\n\",\n    \"    [\\\"HanLP\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"],\\n\",\n    \"    [\\\"我\\\", \\\"的\\\", \\\"希望\\\", \\\"是\\\", \\\"希望\\\", \\\"张晚霞\\\", \\\"的\\\", \\\"背影\\\", \\\"被\\\", \\\"晚霞\\\", \\\"映红\\\", \\\"。\\\"]\\n\",\n    \"  ], tasks='srl', skip_tasks='tok*').pretty_print()\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"srl_restful.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/srl_stl.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/srl_mtl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fsrl_mtl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 加载模型\\n\",\n    \"HanLP的工作流程是先加载模型，模型的标示符存储在`hanlp.pretrained`这个包中，按照NLP任务归类。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"id\": \"0tmKBu7sNAXX\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'CPB3_SRL_ELECTRA_SMALL': 'https://file.hankcs.com/hanlp/srl/cpb3_electra_small_crf_has_transform_20220218_135910.zip'}\"\n      ]\n     },\n     \"execution_count\": 1,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"import hanlp\\n\",\n    \"hanlp.pretrained.srl.ALL # 语种见名称最后一个字段或相应语料库\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"EmZDmLn9aGxG\"\n   },\n   \"source\": [\n    \"调用`hanlp.load`进行加载，模型会自动下载到本地缓存：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"srl = hanlp.load('CPB3_SRL_ELECTRA_SMALL')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 语义角色分析\\n\",\n    \"为已分词的句子执行语义角色分析：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"2a0d392f-b99a-4a18-fc7f-754e2abe2e34\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"[[('2021年', 'ARGM-TMP', 0, 1),\\n\",\n       \"  ('HanLPv2.1', 'ARG0', 1, 2),\\n\",\n       \"  ('为生产环境', 'ARG2', 2, 5),\\n\",\n       \"  ('带来', 'PRED', 5, 6),\\n\",\n       \"  ('次世代最先进的多语种NLP技术', 'ARG1', 6, 15)],\\n\",\n       \" [('次世代', 'ARGM-TMP', 6, 8),\\n\",\n       \"  ('最', 'ARGM-ADV', 8, 9),\\n\",\n       \"  ('先进', 'PRED', 9, 10),\\n\",\n       \"  ('技术', 'ARG0', 14, 15)]]\"\n      ]\n     },\n     \"execution_count\": 4,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"srl(['2021年', 'HanLPv2.1', '为', '生产', '环境', '带来', '次', '世代', '最', '先进', '的', '多', '语种', 'NLP', '技术', '。'])\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"语义角色标注结果中每个四元组的格式为`[论元或谓词, 语义角色标签, 起始下标, 终止下标]`。其中，谓词的语义角色标签为`PRED`，起止下标对应单词数组。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"遍历谓词论元结构：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"第1个谓词论元结构：\\n\",\n      \"2021年 = ARGM-TMP at [0, 1]\\n\",\n      \"HanLPv2.1 = ARG0 at [1, 2]\\n\",\n      \"为生产环境 = ARG2 at [2, 5]\\n\",\n      \"带来 = PRED at [5, 6]\\n\",\n      \"次世代最先进的多语种NLP技术 = ARG1 at [6, 15]\\n\",\n      \"第2个谓词论元结构：\\n\",\n      \"次世代 = ARGM-TMP at [6, 8]\\n\",\n      \"最 = ARGM-ADV at [8, 9]\\n\",\n      \"先进 = PRED at [9, 10]\\n\",\n      \"技术 = ARG0 at [14, 15]\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"for i, pas in enumerate(srl(['2021年', 'HanLPv2.1', '为', '生产', '环境', '带来', '次', '世代', '最', '先进', '的', '多', '语种', 'NLP', '技术', '。'])):\\n\",\n    \"    print(f'第{i+1}个谓词论元结构：')\\n\",\n    \"    for form, role, begin, end in pas:\\n\",\n    \"        print(f'{form} = {role} at [{begin}, {end}]')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"#### 注意\\n\",\n    \"Native API的输入单位限定为句子，需使用[多语种分句模型](https://github.com/hankcs/HanLP/blob/master/plugins/hanlp_demo/hanlp_demo/sent_split.py)或[基于规则的分句函数](https://github.com/hankcs/HanLP/blob/master/hanlp/utils/rules.py#L19)先行分句。RESTful同时支持全文、句子、已分词的句子。除此之外，RESTful和native两种API的语义设计完全一致，用户可以无缝互换。\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"srl_mtl.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/sts_restful.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/sts_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fsts_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp_restful -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 创建客户端\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"id\": \"0tmKBu7sNAXX\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from hanlp_restful import HanLPClient\\n\",\n    \"HanLP = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh') # auth不填则匿名，zh中文，mul多语种\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"EmZDmLn9aGxG\"\n   },\n   \"source\": [\n    \"#### 申请秘钥\\n\",\n    \"由于服务器算力有限，匿名用户每分钟限2次调用。如果你需要更多调用次数，[建议申请免费公益API秘钥auth](https://bbs.hanlp.com/t/hanlp2-1-restful-api/53)。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 语义文本相似度\\n\",\n    \"输入两段短文本组成的二元组列表，执行语义文本相似度：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"2a0d392f-b99a-4a18-fc7f-754e2abe2e34\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"[0.9764469861984253, 0.0, 0.003458738327026367]\"\n      ]\n     },\n     \"execution_count\": 2,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"HanLP.semantic_textual_similarity([\\n\",\n    \"    ('看图猜一电影名', '看图猜电影'),\\n\",\n    \"    ('无线路由器怎么无线上网', '无线上网卡和无线路由器怎么用'),\\n\",\n    \"    ('北京到上海的动车票', '上海到北京的动车票'),\\n\",\n    \"])\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"sts_restful.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/sts_stl.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/sts_stl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Fsts_stl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 加载模型\\n\",\n    \"HanLP的工作流程是先加载模型，模型的标示符存储在`hanlp.pretrained`这个包中，按照NLP任务归类。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"id\": \"0tmKBu7sNAXX\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"import hanlp\\n\",\n    \"hanlp.pretrained.sts.ALL # 语种见名称最后一个字段或相应语料库\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"EmZDmLn9aGxG\"\n   },\n   \"source\": [\n    \"调用`hanlp.load`进行加载，模型会自动下载到本地缓存：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"sts = hanlp.load(hanlp.pretrained.sts.STS_ELECTRA_BASE_ZH)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 语义文本相似度\\n\",\n    \"输入两段短文本组成的二元组列表，执行语义文本相似度：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"2a0d392f-b99a-4a18-fc7f-754e2abe2e34\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"[0.9764469861984253, 0.0, 0.003458738327026367]\"\n      ]\n     },\n     \"execution_count\": 2,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"sts([\\n\",\n    \"    ('看图猜一电影名', '看图猜电影'),\\n\",\n    \"    ('无线路由器怎么无线上网', '无线上网卡和无线路由器怎么用'),\\n\",\n    \"    ('北京到上海的动车票', '上海到北京的动车票'),\\n\",\n    \"])\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"sts_stl.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-31 20:36\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/demo_classifier.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-01-01 03:52\nfrom hanlp.datasets.classification.sentiment import CHNSENTICORP_ERNIE_TEST\n\nimport hanlp\n\nclassifier = hanlp.load('CHNSENTICORP_BERT_BASE_ZH')\nprint(classifier.predict('前台客房服务态度非常好！早餐很丰富，房价很干净。再接再厉！'))\n\n# predict a whole file in batch mode\noutputs = classifier.predict(classifier.transform.file_to_inputs(CHNSENTICORP_ERNIE_TEST), gold=True)\nprint(outputs[:5])\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/demo_client.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-01-08 04:43\n# pip3 install tensorflow-serving-api-gpu\nimport grpc\nimport tensorflow as tf\nfrom tensorflow_core.python.framework import tensor_util\nfrom tensorflow_serving.apis import predict_pb2, prediction_service_pb2_grpc\nimport hanlp\nfrom hanlp.common.keras_component import KerasComponent\n\ntagger: KerasComponent = hanlp.load(hanlp.pretrained.pos.CTB5_POS_RNN, transform_only=True)\ntransform = tagger.transform\ndel tagger\n\ninputs = [['商品', '和', '服务'],\n          ['我', '的', '希望', '是', '希望', '和平']]\n\nsamples = next(iter(transform.inputs_to_dataset(inputs)))[0]\nprint(samples)\n\nchannel = grpc.insecure_channel('{host}:{port}'.format(host='localhost', port=8500))\nstub = prediction_service_pb2_grpc.PredictionServiceStub(channel)\nrequest = predict_pb2.PredictRequest()\nrequest.model_spec.name = 'ctb5_pos_rnn_20191229_015325'\nrequest.model_spec.signature_name = 'serving_default'\nrequest.inputs['embedding_input'].CopyFrom(\n    tf.make_tensor_proto(samples, dtype=tf.float32))\nresult = stub.Predict(request, 10.0)  # 10 secs timeout\nprint(result)\nprediction = tensor_util.MakeNdarray(result.outputs['dense'])\nprint(prediction)\n\nprint(list(transform.Y_to_outputs(prediction, inputs=inputs)))\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/demo_cws.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 21:25\nimport hanlp\n\ntokenizer = hanlp.load(hanlp.pretrained.tok.LARGE_ALBERT_BASE)\nprint(tokenizer('商品和服务'))\nprint(tokenizer(['萨哈夫说，伊拉克将同联合国销毁伊拉克大规模杀伤性武器特别委员会继续保持合作。',\n                 '上海华安工业（集团）公司董事长谭旭光和秘书张晚霞来到美国纽约现代艺术博物馆参观。',\n                 'HanLP支援臺灣正體、香港繁體，具有新詞辨識能力的中文斷詞系統']))\n\ntext = 'NLP统计模型没有加规则，聪明人知道自己加。英文、数字、自定义词典统统都是规则。'\nprint(tokenizer(text))\n\ndic = {'自定义词典': 'custom_dict', '聪明人': 'smart'}\n\n\ndef split_by_dic(text: str):\n    # We use regular expression for the sake of simplicity.\n    # However, you should use some trie trees for production\n    import re\n    p = re.compile('(' + '|'.join(dic.keys()) + ')')\n    sents, offset, words = [], 0, []\n    for m in p.finditer(text):\n        if offset < m.start():\n            sents.append(text[offset: m.start()])\n            words.append((m.group(), dic[m.group()]))\n            offset = m.end()\n    if offset < len(text):\n        sents.append(text[offset:])\n        words.append((None, None))\n    flat = []\n    for pred, (word, tag) in zip(tokenizer(sents), words):\n        flat.extend(pred)\n        if word:\n            flat.append((word, tag))\n    return flat\n\n\nprint(split_by_dic(text))\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/demo_cws_trie.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 21:25\nfrom hanlp_trie.trie import Trie\n\nimport hanlp\n\ntokenizer = hanlp.load('LARGE_ALBERT_BASE')\ntext = 'NLP统计模型没有加规则，聪明人知道自己加。英文、数字、自定义词典统统都是规则。'\nprint(tokenizer(text))\n\ntrie = Trie()\ntrie.update({'自定义词典': 'custom_dict', '聪明人': 'smart'})\n\n\ndef split_sents(text: str, trie: Trie):\n    words = trie.parse_longest(text)\n    sents = []\n    pre_start = 0\n    offsets = []\n    for start, end, value in words:\n        if pre_start != start:\n            sents.append(text[pre_start: start])\n            offsets.append(pre_start)\n        pre_start = end\n    if pre_start != len(text):\n        sents.append(text[pre_start:])\n        offsets.append(pre_start)\n    return sents, offsets, words\n\n\nprint(split_sents(text, trie))\n\n\ndef merge_parts(parts, offsets, words):\n    items = [(i, p) for (i, p) in zip(offsets, parts)]\n    items += [(start, [value]) for (start, end, value) in words]\n    return [each for x in sorted(items) for each in x[1]]\n\n\ntokenizer = hanlp.pipeline() \\\n    .append(split_sents, output_key=('parts', 'offsets', 'words'), trie=trie) \\\n    .append(tokenizer, input_key='parts', output_key='tokens') \\\n    .append(merge_parts, input_key=('tokens', 'offsets', 'words'), output_key='merged')\n\nprint(tokenizer(text))\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/demo_dep.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 21:25\nimport hanlp\n\nsyntactic_parser = hanlp.load(hanlp.pretrained.dep.CTB7_BIAFFINE_DEP_ZH)\nsent = [('蜡烛', 'NN'), ('两', 'CD'), ('头', 'NN'), ('烧', 'VV')]\ntree = syntactic_parser(sent)\nprint(tree)\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/demo_fasttext.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-12-12 18:33\nimport hanlp\nimport torch\n\n# fasttext is a `torch.nn.Module`. Unless you know how to code in\n# PyTorch, otherwise don't bother to use this.\nfasttext = hanlp.load(hanlp.pretrained.fasttext.FASTTEXT_WIKI_300_ZH)\n\nvec = fasttext('单词')\nprint(vec)\n\nprint(torch.nn.functional.cosine_similarity(fasttext('单词'), fasttext('词语'), dim=0))\nprint(torch.nn.functional.cosine_similarity(fasttext('单词'), fasttext('今天'), dim=0))\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/demo_multiprocess.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-02-15 11:30\nimport multiprocessing\nimport hanlp\n\ntokenizer = hanlp.load(hanlp.pretrained.tok.LARGE_ALBERT_BASE)\n\n\ndef worker(job):\n    print(job)\n    print(tokenizer(job))\n\n\nif __name__ == '__main__':\n    num_proc = 2\n    # Important! The python multiprocessing package defaults to just call fork when creating a child process.\n    # This cannot work when the child process calls async code (i.e TensorFlow is multithreaded).\n    # See https://github.com/tensorflow/tensorflow/issues/8220#issuecomment-302826884\n    # See https://sefiks.com/2019/03/20/tips-and-tricks-for-gpu-and-multiprocessing-in-tensorflow/\n    multiprocessing.set_start_method('spawn', force=True)  # only spawn works with TensorFlow\n    with multiprocessing.Pool(num_proc) as pool:\n        pool.map(worker, [f'给{i}号进程的任务' for i in range(num_proc)])\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/demo_ner.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-30 19:52\nimport hanlp\n\nrecognizer = hanlp.load(hanlp.pretrained.ner.MSRA_NER_BERT_BASE_ZH)\nprint(recognizer.predict([list('上海华安工业（集团）公司董事长谭旭光和秘书张晚霞来到美国纽约现代艺术博物馆参观。'),\n                          list('萨哈夫说，伊拉克将同联合国销毁伊拉克大规模杀伤性武器特别委员会继续保持合作。')]))\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/demo_pipeline.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-31 03:24\n\nimport hanlp\n\ntokenizer = hanlp.load('LARGE_ALBERT_BASE')\ntagger = hanlp.load('CTB9_POS_ALBERT_BASE')\nsyntactic_parser = hanlp.load('CTB7_BIAFFINE_DEP_ZH')\nsemantic_parser = hanlp.load('SEMEVAL16_TEXT_BIAFFINE_ZH')\n\npipeline = hanlp.pipeline() \\\n    .append(hanlp.utils.rules.split_sentence, output_key='sentences') \\\n    .append(tokenizer, output_key='tokens') \\\n    .append(tagger, output_key='part_of_speech_tags') \\\n    .append(syntactic_parser, input_key=('tokens', 'part_of_speech_tags'), output_key='syntactic_dependencies', conll=False) \\\n    .append(semantic_parser, input_key=('tokens', 'part_of_speech_tags'), output_key='semantic_dependencies', conll=False)\nprint(pipeline)\n\ntext = '''HanLP是一系列模型与算法组成的自然语言处理工具包，目标是普及自然语言处理在生产环境中的应用。\nHanLP具备功能完善、性能高效、架构清晰、语料时新、可自定义的特点。\n内部算法经过工业界和学术界考验，配套书籍《自然语言处理入门》已经出版。\n'''\n\ndoc = pipeline(text)\nprint(doc)\n# By default the doc is json serializable, it holds true if your pipes output json serializable object too.\n# print(json.dumps(doc, ensure_ascii=False, indent=2))\n\n# You can save the config to disk for deploying or sharing.\npipeline.save('zh.json')\n# Then load it smoothly.\ndeployed = hanlp.load('zh.json')\nprint(deployed)\nprint(deployed(text))\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/demo_pos.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 21:25\nimport hanlp\nfrom hanlp.pretrained.pos import CTB9_POS_ALBERT_BASE\n\ntagger = hanlp.load(CTB9_POS_ALBERT_BASE)\nprint(tagger.predict(['我', '的', '希望', '是', '希望', '世界', '和平']))\nprint(tagger.predict([['支持', '批处理', '地', '预测'], ['速度', '更', '快']]))\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/demo_sdp.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-31 23:55\nimport hanlp\n\nsemantic_parser = hanlp.load('SEMEVAL16_NEWS_BIAFFINE_ZH')\nsent = [('蜡烛', 'NN'), ('两', 'CD'), ('头', 'NN'), ('烧', 'VV')]\nprint(semantic_parser(sent))\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/demo_serving.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-01-06 20:23\nimport hanlp\nfrom hanlp.common.keras_component import KerasComponent\n\ntagger: KerasComponent = hanlp.load(hanlp.pretrained.pos.CTB5_POS_RNN)\nprint(tagger('商品 和 服务'.split()))\ntagger.serve()\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2021-12-26 23:25\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/cws/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-01-01 20:55"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/cws/train_ctb6_cws_albert.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 22:22\n\nfrom hanlp.components.tokenizers.tok_tf import TransformerTokenizerTF\nfrom hanlp.datasets.tokenization.ctb6 import CTB6_CWS_TRAIN, CTB6_CWS_DEV, CTB6_CWS_TEST\nfrom tests import cdroot\n\ncdroot()\ntokenizer = TransformerTokenizerTF()\nsave_dir = 'data/model/cws_bert_albert_ctb6'\ntokenizer.fit(CTB6_CWS_TRAIN, CTB6_CWS_DEV, save_dir,\n              transformer='/home/ubuntu/hankcs/laser/data/transformer/albert_base_tf2',\n              metrics='f1', learning_rate=5e-5, epochs=3)\ntokenizer.load(save_dir)\nprint(tokenizer.predict(['中央民族乐团离开北京前往维也纳', '商品和服务']))\ntokenizer.evaluate(CTB6_CWS_TEST, save_dir=save_dir)\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/cws/train_ctb6_cws_bert.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 22:22\n\nfrom hanlp.components.tokenizers.tok_tf import TransformerTokenizerTF\nfrom hanlp.datasets.tokenization.ctb6 import CTB6_CWS_TRAIN, CTB6_CWS_DEV, CTB6_CWS_TEST\nfrom tests import cdroot\n\ncdroot()\ntokenizer = TransformerTokenizerTF()\nsave_dir = 'data/model/cws_bert_base_ctb6'\ntokenizer.fit(CTB6_CWS_TRAIN, CTB6_CWS_DEV, save_dir, transformer='chinese_L-12_H-768_A-12',\n              metrics='f1')\ntokenizer.load(save_dir)\nprint(tokenizer.predict(['中央民族乐团离开北京前往维也纳', '商品和服务']))\ntokenizer.evaluate(CTB6_CWS_TEST, save_dir=save_dir)\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/cws/train_ctb6_cws_convseg.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 22:22\nimport tensorflow as tf\n\nfrom hanlp.components.tokenizers.tok_tf import NgramConvTokenizerTF\nfrom hanlp.datasets.tokenization.ctb6 import CTB6_CWS_TRAIN, CTB6_CWS_DEV, CTB6_CWS_TEST\nfrom hanlp.pretrained.word2vec import CONVSEG_W2V_NEWS_TENSITE_CHAR\nfrom tests import cdroot\n\ncdroot()\ntokenizer = NgramConvTokenizerTF()\nsave_dir = 'data/model/cws/ctb6_cws'\noptimizer = tf.keras.optimizers.Adam(learning_rate=0.001,\n                                     epsilon=1e-8, clipnorm=5)\ntokenizer.fit(CTB6_CWS_TRAIN,\n              CTB6_CWS_DEV,\n              save_dir,\n              word_embed={'class_name': 'HanLP>Word2VecEmbedding',\n                          'config': {\n                              'trainable': True,\n                              'filepath': CONVSEG_W2V_NEWS_TENSITE_CHAR,\n                              'expand_vocab': False,\n                              'lowercase': False,\n                          }},\n              optimizer=optimizer,\n              window_size=0,\n              weight_norm=True)\ntokenizer.evaluate(CTB6_CWS_TEST, save_dir=save_dir, output=False)\nprint(tokenizer.predict(['中央民族乐团离开北京前往维也纳', '商品和服务']))\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/cws/train_large_bert_cws.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-21 15:39\nfrom hanlp.components.tokenizers.tok_tf import TransformerTokenizerTF\nfrom hanlp.datasets.tokenization.ctb6 import CTB6_CWS_DEV, CTB6_CWS_TEST\nfrom tests import cdroot\n\ncdroot()\ntokenizer = TransformerTokenizerTF()\nsave_dir = 'data/model/cws_bert_base_100million'\ntokenizer.fit('data/cws/large/all.txt', CTB6_CWS_DEV, save_dir, transformer='bert-base-chinese',\n              metrics='accuracy', batch_size=32)\ntokenizer.load(save_dir, metrics='f1')\nprint(tokenizer.predict(['中央民族乐团离开北京前往维也纳', '商品和服务']))\ntokenizer.evaluate(CTB6_CWS_TEST, save_dir=save_dir)\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/cws/train_large_conv_cws.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-29 21:58\n\nimport tensorflow as tf\n\nfrom hanlp.components.tokenizers.tok_tf import NgramConvTokenizerTF\nfrom hanlp.datasets.cws.ctb import CTB6_CWS_TRAIN, CTB6_CWS_DEV, CTB6_CWS_TEST\nfrom hanlp.pretrained.word2vec import CONVSEG_W2V_NEWS_TENSITE_CHAR\nfrom tests import cdroot\n\ncdroot()\ntokenizer = NgramConvTokenizerTF()\nsave_dir = 'data/model/cws/ctb6_cws'\noptimizer = tf.keras.optimizers.Adam(learning_rate=0.001,\n                                     epsilon=1e-8, clipnorm=5)\ntokenizer.fit(CTB6_CWS_TRAIN,\n              CTB6_CWS_DEV,\n              save_dir,\n              word_embed={'class_name': 'HanLP>Word2VecEmbedding',\n                          'config': {\n                              'trainable': True,\n                              'filepath': CONVSEG_W2V_NEWS_TENSITE_CHAR,\n                              'expand_vocab': False,\n                              'lowercase': False,\n                          }},\n              optimizer=optimizer,\n              window_size=0,\n              weight_norm=True)\ntokenizer.evaluate(CTB6_CWS_TEST, save_dir=save_dir, output=False)\nprint(tokenizer.predict(['中央民族乐团离开北京前往维也纳', '商品和服务']))\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/cws/train_large_cws_albert.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 22:22\nfrom hanlp.components.tokenizers.tok_tf import TransformerTokenizerTF\nfrom hanlp.datasets.tokenization.ctb6 import CTB6_CWS_DEV, CTB6_CWS_TEST\nfrom tests import cdroot\n\ncdroot()\ntokenizer = TransformerTokenizerTF()\nsave_dir = 'data/model/large_corpus_cws_albert_base'\ntokenizer.fit('data/cws/large/all.txt',\n              CTB6_CWS_DEV, save_dir,\n              transformer='uer/albert-base-chinese-cluecorpussmall',\n              max_seq_length=128,\n              metrics='accuracy', learning_rate=5e-5, epochs=3)\ntokenizer.load(save_dir, metrics='f1')\nprint(tokenizer.predict(['中央民族乐团离开北京前往维也纳', '商品和服务']))\ntokenizer.evaluate(CTB6_CWS_TEST, save_dir=save_dir)\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/cws/train_large_cws_electra.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 22:22\nfrom hanlp.components.tokenizers.tok_tf import TransformerTokenizerTF\nfrom hanlp.datasets.tokenization.ctb6 import CTB6_CWS_DEV, CTB6_CWS_TEST\nfrom tests import cdroot\n\ncdroot()\ntokenizer = TransformerTokenizerTF()\nsave_dir = 'data/model/large_corpus_cws_electra_small'\ntokenizer.fit('data/cws/large/all.txt',\n              CTB6_CWS_DEV, save_dir,\n              transformer='hfl/chinese-electra-small-discriminator',\n              max_seq_length=128,\n              metrics='accuracy', learning_rate=5e-5, epochs=10)\ntokenizer.load(save_dir, metrics='f1')\nprint(tokenizer.predict(['中央民族乐团离开北京前往维也纳', '商品和服务']))\ntokenizer.evaluate(CTB6_CWS_TEST, save_dir=save_dir)\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/cws/train_large_rnn_cws.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-21 15:39\nimport tensorflow as tf\n\nfrom hanlp.components.tokenizers.tok_tf import RNNTokenizerTF\nfrom hanlp.datasets.cws.ctb import CTB6_CWS_TEST, CTB6_CWS_DEV\nfrom hanlp.pretrained.word2vec import RADICAL_CHAR_EMBEDDING_100\nfrom tests import cdroot\n\ncdroot()\n\ntokenizer = RNNTokenizerTF()\nsave_dir = 'data/model/cws/large_rnn_cws'\noptimizer = tf.keras.optimizers.Adam(learning_rate=0.001,\n                                     epsilon=1e-8, clipnorm=5)\ntokenizer.fit('data/cws/large/all.txt',\n              CTB6_CWS_DEV,\n              save_dir,\n              embeddings={'class_name': 'HanLP>Word2VecEmbedding',\n                          'config': {\n                              'trainable': True,\n                              'filepath': RADICAL_CHAR_EMBEDDING_100,\n                              'expand_vocab': False,\n                              'lowercase': False,\n                          }},\n              early_stopping_patience=5,\n              batch_size=64,\n              max_seq_len=64,\n              metrics='accuracy'\n              )\ntokenizer.load(save_dir, metrics='f1')\ntokenizer.evaluate(CTB6_CWS_TEST, save_dir=save_dir, output=False)\nprint(tokenizer.predict(['中央民族乐团离开北京前往维也纳', '商品和服务']))\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/cws/train_msr_cws_albert.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 22:22\n\nfrom hanlp.components.tokenizers.tok import TransformerTokenizer\nfrom hanlp.datasets.cws.ctb import CTB6_CWS_TEST\nfrom hanlp.datasets.tokenization.sighan2005.msr import SIGHAN2005_MSR_VALID, SIGHAN2005_MSR_TRAIN\nfrom tests import cdroot\n\ncdroot()\ntokenizer = TransformerTokenizer()\nsave_dir = 'data/model/msr_cws_albert_base'\ntokenizer.fit(SIGHAN2005_MSR_TRAIN, SIGHAN2005_MSR_VALID, save_dir,\n              transformer='albert_base_zh',\n              max_seq_length=150,\n              metrics='f1', learning_rate=5e-5, epochs=10)\ntokenizer.load(save_dir)\nprint(tokenizer.predict(['中央民族乐团离开北京前往维也纳', '商品和服务']))\ntokenizer.evaluate(CTB6_CWS_TEST, save_dir=save_dir)\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/cws/train_msr_cws_bert.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-21 15:39\nfrom hanlp.components.tokenizers.tok_tf import TransformerTokenizerTF\nfrom hanlp.datasets.tokenization.sighan2005.msr import SIGHAN2005_MSR_TRAIN, SIGHAN2005_MSR_DEV, SIGHAN2005_MSR_TEST\nfrom tests import cdroot\n\ncdroot()\ntokenizer = TransformerTokenizerTF()\nsave_dir = 'data/model/cws_bert_base_msra'\ntokenizer.fit(SIGHAN2005_MSR_TRAIN, SIGHAN2005_MSR_DEV, save_dir, transformer='bert-base-chinese',\n              metrics='f1')\n# tagger.load(save_dir)\nprint(tokenizer.predict(['中央民族乐团离开北京前往维也纳', '商品和服务']))\ntokenizer.evaluate(SIGHAN2005_MSR_TEST, save_dir=save_dir)\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/cws/train_msr_cws_ngram_conv.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-21 15:39\nimport tensorflow as tf\n\nfrom hanlp.components.tokenizers.tok_tf import NgramConvTokenizerTF\nfrom hanlp.datasets.tokenization.sighan2005.msr import SIGHAN2005_MSR_TRAIN, SIGHAN2005_MSR_DEV, SIGHAN2005_MSR_TEST\nfrom hanlp.pretrained.word2vec import CONVSEG_W2V_NEWS_TENSITE_CHAR\nfrom tests import cdroot\n\ncdroot()\ntokenizer = NgramConvTokenizerTF()\nsave_dir = 'data/model/cws/convseg-msr-nocrf-noembed'\ntokenizer.fit(SIGHAN2005_MSR_TRAIN,\n              SIGHAN2005_MSR_DEV,\n              save_dir,\n              word_embed={'class_name': 'HanLP>Word2VecEmbedding',\n                          'config': {\n                              'trainable': True,\n                              'filepath': CONVSEG_W2V_NEWS_TENSITE_CHAR,\n                              'expand_vocab': False,\n                              'lowercase': False,\n                          }},\n              optimizer=tf.keras.optimizers.Adam(learning_rate=0.001,\n                                                 epsilon=1e-8, clipnorm=5),\n              epochs=100,\n              window_size=0,\n              metrics='f1',\n              weight_norm=True)\nprint(tokenizer.predict(['中央民族乐团离开北京前往维也纳', '商品和服务']))\ntokenizer.evaluate(SIGHAN2005_MSR_TEST, save_dir=save_dir)\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/cws/train_msr_cws_ngram_conv_embed.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-21 15:39\nimport tensorflow as tf\n\nfrom hanlp.components.tokenizers.tok import NgramConvTokenizer\nfrom hanlp.datasets.tokenization.sighan2005.msr import SIGHAN2005_MSR_TRAIN, SIGHAN2005_MSR_VALID, SIGHAN2005_MSR_TEST\nfrom hanlp.pretrained.word2vec import CONVSEG_W2V_NEWS_TENSITE_CHAR, CONVSEG_W2V_NEWS_TENSITE_WORD_MSR\nfrom tests import cdroot\n\ncdroot()\ntokenizer = NgramConvTokenizer()\nsave_dir = 'data/model/cws/convseg-msr-nocrf-noembed'\ntokenizer.fit(SIGHAN2005_MSR_TRAIN,\n              SIGHAN2005_MSR_VALID,\n              save_dir,\n              word_embed={'class_name': 'HanLP>Word2VecEmbedding',\n                          'config': {\n                              'trainable': True,\n                              'filepath': CONVSEG_W2V_NEWS_TENSITE_CHAR,\n                              'expand_vocab': False,\n                              'lowercase': False,\n                          }},\n              ngram_embed={'class_name': 'HanLP>Word2VecEmbedding',\n                           'config': {\n                               'trainable': True,\n                               'filepath': CONVSEG_W2V_NEWS_TENSITE_WORD_MSR,\n                               'expand_vocab': True,\n                               'lowercase': False,\n                           }},\n              optimizer=tf.keras.optimizers.Adam(learning_rate=0.001,\n                                                 epsilon=1e-8, clipnorm=5),\n              epochs=3,\n              window_size=4,\n              metrics='f1',\n              weight_norm=True)\nprint(tokenizer.predict(['中央民族乐团离开北京前往维也纳', '商品和服务']))\ntokenizer.load(save_dir, metrics='f1')\ntokenizer.evaluate(SIGHAN2005_MSR_TEST, save_dir=save_dir)\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/cws/train_pku980106_conv_cws.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-21 15:39\nimport tensorflow as tf\n\nfrom hanlp.components.tokenizers.tok_tf import NgramConvTokenizerTF\nfrom hanlp.pretrained.word2vec import RADICAL_CHAR_EMBEDDING_100\nfrom tests import cdroot\n\ncdroot()\n\ntokenizer = NgramConvTokenizerTF()\nsave_dir = 'data/model/cws/pku98_6m_conv_ngram'\noptimizer = tf.keras.optimizers.Adam(learning_rate=0.001,\n                                     epsilon=1e-8, clipnorm=5)\ntokenizer.fit('data/cws/pku98/199801-06-seg.txt',\n              'data/cws/pku98/test_pku98_name_merged.txt',\n              save_dir,\n              word_embed={'class_name': 'HanLP>Word2VecEmbedding',\n                          'config': {\n                              'trainable': False,\n                              'filepath': RADICAL_CHAR_EMBEDDING_100,\n                              'expand_vocab': True,\n                              'lowercase': False,\n                          }},\n              optimizer=optimizer,\n              window_size=0,\n              weight_norm=True)\ntokenizer.evaluate('data/cws/pku98/test_pku98_name_merged.txt', save_dir=save_dir, output=False)\nprint(tokenizer.predict(['中央民族乐团离开北京前往维也纳', '商品和服务']))\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/cws/train_pku980106_rnn_cws.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-21 15:39\nimport tensorflow as tf\n\nfrom hanlp.components.tokenizers.tok_tf import RNNTokenizerTF\nfrom hanlp.pretrained.word2vec import RADICAL_CHAR_EMBEDDING_100\nfrom tests import cdroot\n\ncdroot()\n\ntokenizer = RNNTokenizerTF()\nsave_dir = 'data/model/cws/pku_6m_rnn_cws'\noptimizer = tf.keras.optimizers.Adam(learning_rate=0.001,\n                                     epsilon=1e-8, clipnorm=5)\ntokenizer.fit('data/cws/pku98/199801-06-seg.txt',\n              'data/cws/pku98/pku98_test.txt',\n              save_dir,\n              embeddings={'class_name': 'HanLP>Word2VecEmbedding',\n                          'config': {\n                              'trainable': False,\n                              'filepath': RADICAL_CHAR_EMBEDDING_100,\n                              'expand_vocab': True,\n                              'lowercase': False,\n                          }}\n              )\ntokenizer.evaluate('data/cws/pku98/pku98_test.txt', save_dir=save_dir, output=False)\nprint(tokenizer.predict(['中央民族乐团离开北京前往维也纳', '商品和服务']))\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/cws/train_pku_conv_cws.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-21 15:39\nfrom hanlp.datasets.tokenization.sighan2005 import SIGHAN2005_PKU_TRAIN, SIGHAN2005_PKU_DEV, SIGHAN2005_PKU_TEST\nfrom hanlp.pretrained.word2vec import CONVSEG_W2V_NEWS_TENSITE_CHAR\nfrom hanlp.utils.tf_util import nice\nfrom tests import cdroot\nimport tensorflow as tf\n\nnice()\ncdroot()\nfrom hanlp.components.tokenizers.tok_tf import NgramConvTokenizerTF\n\ntokenizer = NgramConvTokenizerTF()\nsave_dir = 'data/model/cws/sighan2005-pku-convseg'\noptimizer = tf.keras.optimizers.Adam(learning_rate=0.001,\n                                     epsilon=1e-8, clipnorm=5)\ntokenizer.fit(SIGHAN2005_PKU_TRAIN,\n              SIGHAN2005_PKU_DEV,\n              save_dir,\n              word_embed={'class_name': 'HanLP>Word2VecEmbedding',\n                          'config': {\n                              'trainable': True,\n                              'filepath': CONVSEG_W2V_NEWS_TENSITE_CHAR,\n                              'expand_vocab': False,\n                              'lowercase': False,\n                          }},\n              optimizer=optimizer,\n              window_size=0,\n              weight_norm=True)\ntokenizer.evaluate(SIGHAN2005_PKU_TEST, save_dir=save_dir, output=False)\n# print(tagger.tag(list('中央民族乐团离开北京前往维也纳')))\n# print(tagger.predict('中央民族乐团离开北京前往维也纳'))\nprint(tokenizer.predict(['中央民族乐团离开北京前往维也纳', '商品和服务']))\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/finetune_msra_ner_albert.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 23:15\nimport hanlp\nfrom hanlp.components.ner.ner_tf import TransformerNamedEntityRecognizerTF\nfrom hanlp.datasets.ner.msra import MSRA_NER_CHAR_LEVEL_TRAIN, MSRA_NER_CHAR_LEVEL_DEV, MSRA_NER_CHAR_LEVEL_TEST\nfrom tests import cdroot\n\ncdroot()\nrecognizer = TransformerNamedEntityRecognizerTF()\nsave_dir = 'data/model/ner/finetune_ner_albert_base_zh_msra'\nrecognizer.fit(MSRA_NER_CHAR_LEVEL_TRAIN, MSRA_NER_CHAR_LEVEL_DEV, save_dir, transformer='albert_base_zh',\n               finetune=hanlp.pretrained.ner.MSRA_NER_ALBERT_BASE_ZH)\nrecognizer.load(save_dir)\nprint(recognizer.predict(list('上海华安工业（集团）公司董事长谭旭光和秘书张晚霞来到美国纽约现代艺术博物馆参观。')))\nrecognizer.evaluate(MSRA_NER_CHAR_LEVEL_TEST, save_dir=save_dir)\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_chnsenticorp_bert.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-30 21:01\nfrom hanlp.components.classifiers.transformer_classifier_tf import TransformerClassifierTF, TransformerTextTransform\nfrom hanlp.datasets.classification.sentiment import CHNSENTICORP_ERNIE_TRAIN, CHNSENTICORP_ERNIE_TEST, \\\n    CHNSENTICORP_ERNIE_DEV\nfrom tests import cdroot\n\ncdroot()\nsave_dir = 'data/model/classification/chnsenticorp_bert_base'\nclassifier = TransformerClassifierTF(TransformerTextTransform(y_column=0))\nclassifier.fit(CHNSENTICORP_ERNIE_TRAIN, CHNSENTICORP_ERNIE_DEV, save_dir,\n               transformer='bert-base-chinese')\nclassifier.load(save_dir)\nprint(classifier.predict('前台客房服务态度非常好！早餐很丰富，房价很干净。再接再厉！'))\nclassifier.evaluate(CHNSENTICORP_ERNIE_TEST, save_dir=save_dir)\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_conll03_ner_bert.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-10-25 21:34\nfrom hanlp.components.ner.ner_tf import TransformerNamedEntityRecognizerTF\nfrom hanlp.datasets.ner.conll03 import CONLL03_EN_TRAIN, CONLL03_EN_DEV, CONLL03_EN_TEST\nfrom tests import cdroot\n\ncdroot()\ntagger = TransformerNamedEntityRecognizerTF()\nsave_dir = 'data/model/ner/ner_conll03_bert_base_cased_en'\ntagger.fit(CONLL03_EN_TRAIN, CONLL03_EN_DEV, save_dir, transformer='bert-base-cased',\n           metrics='accuracy')\ntagger.load(save_dir, metrics='f1')\nprint(tagger.predict('West Indian all-rounder Phil Simmons eats apple .'.split()))\ntagger.evaluate(CONLL03_EN_TEST, save_dir=save_dir, output=False, batch_size=32)\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_conll03_ner_flair.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-10-25 21:34\n\nimport tensorflow as tf\n\nfrom hanlp.components.ner.ner_tf import RNNNamedEntityRecognizerTF\nfrom hanlp.datasets.ner.conll03 import CONLL03_EN_TRAIN, CONLL03_EN_TEST\nfrom hanlp.pretrained.glove import GLOVE_6B_100D\nfrom hanlp.pretrained.rnnlm import FLAIR_LM_FW_WMT11_EN_TF, FLAIR_LM_BW_WMT11_EN_TF\nfrom tests import cdroot\n\ncdroot()\ntagger = RNNNamedEntityRecognizerTF()\nsave_dir = 'data/model/conll03-ner-rnn-flair'\ntagger.fit(CONLL03_EN_TRAIN, CONLL03_EN_TEST, save_dir, epochs=100,\n           optimizer=tf.keras.optimizers.Adam(learning_rate=0.1,\n                                              beta_1=0.9,\n                                              beta_2=0.999,\n                                              epsilon=1e-8),\n           loss='crf',\n           rnn_units=256,\n           embeddings=[\n               {'class_name': 'HanLP>Word2VecEmbedding',\n                'config': {\n                    'trainable': False,\n                    'embeddings_initializer': 'zero',\n                    'filepath': GLOVE_6B_100D,\n                    'expand_vocab': True,\n                    'lowercase': False\n                }},\n               {'class_name': 'HanLP>ContextualStringEmbedding',\n                'config': {\n                    'trainable': False,\n                    'forward_model_path': FLAIR_LM_FW_WMT11_EN_TF,\n                    'backward_model_path': FLAIR_LM_BW_WMT11_EN_TF\n                }}\n           ],\n           rnn_output_dropout=0.5,\n           rnn_input_dropout=0.5,\n           batch_size=32,\n           metrics='f1',\n           anneal_factor=0.5,\n           patience=2,\n           )\nprint(tagger.predict('West Indian all-rounder Phil Simmons eats apple .'.split()))\n# print(tagger.predict([['This', 'is', 'an', 'old', 'story'],\n#                       ['Not', 'this', 'year', '.']]))\n# [['DT', 'VBZ', 'DT', 'JJ', 'NN'], ['RB', 'DT', 'NN', '.']]\n# tagger.load(save_dir)\ntagger.evaluate(CONLL03_EN_TEST, save_dir=save_dir, output=False)\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_ctb5_dep.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 18:33\nfrom hanlp.components.parsers.biaffine_parser_tf import BiaffineDependencyParserTF\nfrom hanlp.datasets.parsing.ctb5 import CTB5_DEP_TRAIN, CTB5_DEP_DEV, CTB5_DEP_TEST\nfrom hanlp.pretrained.word2vec import CTB5_FASTTEXT_300_CN\nfrom tests import cdroot\n\ncdroot()\nsave_dir = 'data/model/dep/biaffine_ctb'\nparser = BiaffineDependencyParserTF()\nparser.fit(CTB5_DEP_TRAIN, CTB5_DEP_DEV, save_dir,\n           pretrained_embed={'class_name': 'HanLP>Word2VecEmbedding',\n                             'config': {\n                                 'trainable': False,\n                                 'embeddings_initializer': 'zero',\n                                 'filepath': CTB5_FASTTEXT_300_CN,\n                                 'expand_vocab': True,\n                                 'lowercase': True,\n                                 'normalize': True,\n                             }},\n           )\nparser.load(save_dir)\nsentence = [('中国', 'NR'), ('批准', 'VV'), ('设立', 'VV'), ('外商', 'NN'), ('投资', 'NN'), ('企业', 'NN'), ('逾', 'VV'),\n            ('三十万', 'CD'), ('家', 'M')]\nprint(parser.predict(sentence))\nparser.evaluate(CTB5_DEP_TEST, save_dir)\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_ctb5_pos_rnn.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 22:46\nfrom hanlp.components.taggers.pos_tf import RNNPartOfSpeechTaggerTF\nfrom hanlp.datasets.pos.ctb5 import CTB5_POS_TRAIN, CTB5_POS_DEV, CTB5_POS_TEST\nfrom hanlp.pretrained.fasttext import FASTTEXT_WIKI_300_ZH\nfrom tests import cdroot\n\ncdroot()\ntagger = RNNPartOfSpeechTaggerTF()\nsave_dir = 'data/model/pos/ctb5_pos_rnn_fasttext'\ntagger.fit(CTB5_POS_TRAIN, CTB5_POS_DEV, save_dir, embeddings={'class_name': 'HanLP>FastTextEmbedding',\n                                                                 'config': {'filepath': FASTTEXT_WIKI_300_ZH}}, )\ntagger.evaluate(CTB5_POS_TEST, save_dir=save_dir)\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_ctb7_dep.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 18:33\nfrom hanlp.components.parsers.biaffine_parser_tf import BiaffineDependencyParserTF\nfrom hanlp.datasets.parsing.ctb5 import CIP_W2V_100_CN\nfrom hanlp.datasets.parsing.ctb7 import CTB7_DEP_TRAIN, CTB7_DEP_DEV, CTB7_DEP_TEST\nfrom tests import cdroot\n\ncdroot()\nsave_dir = 'data/model/dep/biaffine_ctb7'\nparser = BiaffineDependencyParserTF()\nparser.fit(CTB7_DEP_TRAIN, CTB7_DEP_DEV, save_dir,\n           pretrained_embed={'class_name': 'HanLP>Word2VecEmbedding',\n                             'config': {\n                                 'trainable': False,\n                                 'embeddings_initializer': 'zero',\n                                 'filepath': CIP_W2V_100_CN,\n                                 'expand_vocab': True,\n                                 'lowercase': True,\n                                 'normalize': True,\n                             }},\n           )\nparser.load(save_dir)\nsentence = [('中国', 'NR'), ('批准', 'VV'), ('设立', 'VV'), ('外商', 'NN'), ('投资', 'NN'), ('企业', 'NN'), ('逾', 'VV'),\n            ('三十万', 'CD'), ('家', 'M')]\nprint(parser.predict(sentence))\nparser.evaluate(CTB7_DEP_TEST, save_dir)\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_ctb9_pos_albert.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 23:15\nfrom hanlp.components.taggers.transformers.transformer_tagger_tf import TransformerTaggerTF\nfrom tests import cdroot\n\ncdroot()\ntagger = TransformerTaggerTF()\nsave_dir = 'data/model/pos/ctb9_albert_base'\ntagger.fit('data/pos/ctb9/train.tsv',\n           'data/pos/ctb9/test.tsv',\n           save_dir,\n           transformer='uer/albert-base-chinese-cluecorpussmall',\n           max_seq_length=130,\n           warmup_steps_ratio=0.1,\n           epochs=20,\n           learning_rate=5e-5)\ntagger.load(save_dir)\nprint(tagger(['我', '的', '希望', '是', '希望', '和平']))\ntagger.evaluate('data/pos/ctb9/test.tsv', save_dir=save_dir)\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_ctb9_pos_electra.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 23:15\nfrom hanlp.components.taggers.transformers.transformer_tagger_tf import TransformerTaggerTF\nfrom tests import cdroot\n\ncdroot()\ntagger = TransformerTaggerTF()\nsave_dir = 'data/model/pos/ctb9_electra_small_zh_epoch_20'\ntagger.fit('data/pos/ctb9/train.tsv',\n           'data/pos/ctb9/test.tsv',\n           save_dir,\n           transformer='hfl/chinese-electra-small-discriminator',\n           max_seq_length=130,\n           warmup_steps_ratio=0.1,\n           epochs=20,\n           learning_rate=5e-5)\ntagger.load(save_dir)\nprint(tagger(['我', '的', '希望', '是', '希望', '和平']))\ntagger.evaluate('data/pos/ctb9/test.tsv', save_dir=save_dir)\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_msra_ner_albert.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 23:15\nfrom hanlp.components.ner.ner_tf import TransformerNamedEntityRecognizerTF\nfrom hanlp.datasets.ner.msra import MSRA_NER_CHAR_LEVEL_TRAIN, MSRA_NER_CHAR_LEVEL_DEV, MSRA_NER_CHAR_LEVEL_TEST\nfrom tests import cdroot\n\ncdroot()\nrecognizer = TransformerNamedEntityRecognizerTF()\nsave_dir = 'data/model/ner/msra_ner_albert_base'\nrecognizer.fit(MSRA_NER_CHAR_LEVEL_TRAIN, MSRA_NER_CHAR_LEVEL_DEV, save_dir,\n               transformer='uer/albert-base-chinese-cluecorpussmall',\n               learning_rate=5e-5,\n               metrics='accuracy')  # Use accuracy to speed up training\nrecognizer.load(save_dir, metrics='f1')\nprint(recognizer.predict(list('上海华安工业（集团）公司董事长谭旭光和秘书张晚霞来到美国纽约现代艺术博物馆参观。')))\nrecognizer.evaluate(MSRA_NER_CHAR_LEVEL_TEST, save_dir=save_dir)\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_msra_ner_bert.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 23:15\nfrom hanlp.components.ner.ner_tf import TransformerNamedEntityRecognizerTF\nfrom hanlp.datasets.ner.msra import MSRA_NER_CHAR_LEVEL_TRAIN, MSRA_NER_CHAR_LEVEL_DEV, MSRA_NER_CHAR_LEVEL_TEST\nfrom tests import cdroot\n\ncdroot()\nrecognizer = TransformerNamedEntityRecognizerTF()\nsave_dir = 'data/model/ner/ner_bert_base_msra_1'\nrecognizer.fit(MSRA_NER_CHAR_LEVEL_TRAIN, MSRA_NER_CHAR_LEVEL_DEV, save_dir, transformer='bert-base-chinese',\n               metrics='accuracy')  # accuracy is faster\nrecognizer.load(save_dir, metrics='f1')\nprint(recognizer.predict(list('上海华安工业（集团）公司董事长谭旭光和秘书张晚霞来到美国纽约现代艺术博物馆参观。')))\nrecognizer.evaluate(MSRA_NER_CHAR_LEVEL_TEST, save_dir=save_dir)\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_msra_ner_electra.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 23:15\nfrom hanlp.components.ner.ner_tf import TransformerNamedEntityRecognizerTF\nfrom hanlp.datasets.ner.msra import MSRA_NER_CHAR_LEVEL_TRAIN, MSRA_NER_CHAR_LEVEL_DEV, MSRA_NER_CHAR_LEVEL_TEST\nfrom tests import cdroot\n\ncdroot()\nrecognizer = TransformerNamedEntityRecognizerTF()\nsave_dir = 'data/model/ner/ner_electra_small_zh_msra_sparse_categorical_crossentropy'\nrecognizer.fit(MSRA_NER_CHAR_LEVEL_TRAIN, MSRA_NER_CHAR_LEVEL_DEV, save_dir,\n               transformer='hfl/chinese-electra-small-discriminator',\n               learning_rate=5e-5,\n               metrics='accuracy')  # Use accuracy to speed up training\nrecognizer.load(save_dir, metrics='f1')\nprint(recognizer.predict(list('上海华安工业（集团）公司董事长谭旭光和秘书张晚霞来到美国纽约现代艺术博物馆参观。')))\nrecognizer.evaluate(MSRA_NER_CHAR_LEVEL_TEST, save_dir=save_dir)\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_msra_ner_ngram_conv.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 23:15\nfrom hanlp.components.ner.ner_tf import NgramConvNamedEntityRecognizerTF\nfrom hanlp.datasets.ner.msra import MSRA_NER_CHAR_LEVEL_TRAIN, MSRA_NER_CHAR_LEVEL_DEV, MSRA_NER_CHAR_LEVEL_TEST\nfrom hanlp.pretrained.word2vec import CONVSEG_W2V_NEWS_TENSITE_CHAR, \\\n    CONVSEG_W2V_NEWS_TENSITE_WORD_MSR\nfrom tests import cdroot\n\ncdroot()\nrecognizer = NgramConvNamedEntityRecognizerTF()\nsave_dir = 'data/model/ner/msra_ner_ngram_conv'\nrecognizer.fit(MSRA_NER_CHAR_LEVEL_TRAIN, MSRA_NER_CHAR_LEVEL_DEV, save_dir,\n               word_embed={'class_name': 'HanLP>Word2VecEmbedding',\n                           'config': {\n                               'trainable': True,\n                               'filepath': CONVSEG_W2V_NEWS_TENSITE_CHAR,\n                               'expand_vocab': False,\n                               'lowercase': False,\n                           }},\n               ngram_embed={'class_name': 'HanLP>Word2VecEmbedding',\n                            'config': {\n                                'trainable': True,\n                                'filepath': CONVSEG_W2V_NEWS_TENSITE_WORD_MSR,\n                                'expand_vocab': True,\n                                'lowercase': False,\n                            }},\n               weight_norm=True)\nrecognizer.evaluate(MSRA_NER_CHAR_LEVEL_TEST, save_dir)\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_msra_ner_rnn.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 23:15\nfrom hanlp.components.ner.ner_tf import RNNNamedEntityRecognizerTF\nfrom hanlp.datasets.ner.msra import MSRA_NER_CHAR_LEVEL_TRAIN, MSRA_NER_CHAR_LEVEL_DEV, MSRA_NER_CHAR_LEVEL_TEST\nfrom hanlp.pretrained.word2vec import RADICAL_CHAR_EMBEDDING_100\nfrom tests import cdroot\n\ncdroot()\nrecognizer = RNNNamedEntityRecognizerTF()\nsave_dir = 'data/model/ner/msra_ner_rnn'\nrecognizer.fit(MSRA_NER_CHAR_LEVEL_TRAIN, MSRA_NER_CHAR_LEVEL_DEV, save_dir,\n               embeddings=RADICAL_CHAR_EMBEDDING_100,\n               embedding_trainable=True,\n               epochs=100)\nrecognizer.evaluate(MSRA_NER_CHAR_LEVEL_TEST, save_dir)\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_ptb_dep_biaffine_albert.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-03-07 23:48\nfrom hanlp.metrics.parsing import conllx_eval\n\nfrom hanlp.datasets.parsing.ptb import PTB_SD330_DEV, PTB_SD330_TRAIN, PTB_SD330_TEST, PTB_TOKEN_MAPPING\nfrom hanlp.components.parsers.biaffine_parser_tf import BiaffineTransformerDependencyParserTF\nfrom tests import cdroot\n\ncdroot()\nsave_dir = 'data/model/dep/ptb_albert3'\nparser = BiaffineTransformerDependencyParserTF()\nparser.fit(PTB_SD330_TRAIN, PTB_SD330_DEV, save_dir,\n           'albert-xxlarge-v2',\n           batch_size=256,\n           warmup_steps_ratio=.1,\n           token_mapping=PTB_TOKEN_MAPPING,\n           samples_per_batch=150,\n           transformer_dropout=.33,\n           learning_rate=2e-3,\n           learning_rate_transformer=1e-5,\n           # early_stopping_patience=10,\n           )\nparser.load(save_dir)\n# output = f'{save_dir}/test.predict.conll'\nparser.evaluate(PTB_SD330_TEST, save_dir, warm_up=False)\n# uas, las = conllx_eval.evaluate(PTB_SD330_TEST, output)\n# print(f'Official UAS: {uas:.4f} LAS: {las:.4f}')\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_ptb_dep_biaffine_bert.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-03-07 23:48\nfrom hanlp.metrics.parsing import conllx_eval\n\nfrom hanlp.datasets.parsing.ptb import PTB_SD330_DEV, PTB_SD330_TRAIN, PTB_SD330_TEST, PTB_TOKEN_MAPPING\nfrom hanlp.components.parsers.biaffine_parser_tf import BiaffineTransformerDependencyParserTF\nfrom tests import cdroot\n\ncdroot()\nsave_dir = 'data/model/dep/ptb_bert_1e-5'\nparser = BiaffineTransformerDependencyParserTF()\n# parser.fit(PTB_SD330_TRAIN, PTB_SD330_DEV, save_dir, 'bert-base-uncased',\n#            batch_size=3000,\n#            warmup_steps_ratio=.1,\n#            token_mapping=PTB_TOKEN_MAPPING,\n#            samples_per_batch=150,\n#            transformer_dropout=.33,\n#            learning_rate=2e-3,\n#            learning_rate_transformer=1e-5,\n#            # early_stopping_patience=10,\n#            )\nparser.load(save_dir, tree='tarjan')\n# output = f'{save_dir}/test.predict.conll'\nparser.evaluate(PTB_SD330_TEST, save_dir, warm_up=False)\n# uas, las = conllx_eval.evaluate(PTB_SD330_TEST, output)\n# print(f'Official UAS: {uas:.4f} LAS: {las:.4f}')\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_ptb_dep_biaffine_bert_96.6.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-03-07 23:48\n\nfrom hanlp.datasets.parsing.ptb import PTB_SD330_DEV, PTB_SD330_TRAIN, PTB_SD330_TEST, PTB_TOKEN_MAPPING\nfrom hanlp.components.parsers.biaffine_parser_tf import BiaffineTransformerDependencyParserTF\nfrom tests import cdroot\nfrom hanlp.metrics.parsing import conllx_eval\n\ncdroot()\nsave_dir = 'data/model/dep/ptb_bert_96.61'\nparser = BiaffineTransformerDependencyParserTF()\n# parser.fit(PTB_SD330_TRAIN, PTB_SD330_DEV, save_dir, 'bert-base-uncased',\n#            batch_size=3000,\n#            warmup_steps_ratio=.1,\n#            token_mapping=PTB_TOKEN_MAPPING,\n#            samples_per_batch=150,\n#            )\nparser.load(save_dir)\noutput = f'{save_dir}/test.predict.conll'\nparser.evaluate(PTB_SD330_TEST, save_dir, warm_up=False, output=output)\nuas, las = conllx_eval.evaluate(PTB_SD330_TEST, output)\nprint(f'Official UAS: {uas:.4f} LAS: {las:.4f}')\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_ptb_dep_biaffine_bert_positional.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-03-07 23:48\nfrom hanlp.metrics.parsing import conllx_eval\n\nfrom hanlp.datasets.parsing.ptb import PTB_SD330_DEV, PTB_SD330_TRAIN, PTB_SD330_TEST, PTB_TOKEN_MAPPING\nfrom hanlp.components.parsers.biaffine_parser_tf import BiaffineTransformerDependencyParserTF\nfrom tests import cdroot\n\ncdroot()\nsave_dir = 'data/model/dep/ptb_bert_positional_diff_lr'\nparser = BiaffineTransformerDependencyParserTF()\nparser.fit(PTB_SD330_TRAIN, PTB_SD330_DEV, save_dir, 'bert-base-uncased',\n           batch_size=3000,\n           warmup_steps_ratio=.1,\n           token_mapping=PTB_TOKEN_MAPPING,\n           samples_per_batch=150,\n           transformer_dropout=.33,\n           learning_rate=1e-4,\n           learning_rate_transformer=1e-5,\n           d_positional=128,\n           # early_stopping_patience=10,\n           )\n# parser.load(save_dir)\n# output = f'{save_dir}/test.predict.conll'\nparser.evaluate(PTB_SD330_TEST, save_dir, warm_up=False)\n# uas, las = conllx_eval.evaluate(PTB_SD330_TEST, output)\n# print(f'Official UAS: {uas:.4f} LAS: {las:.4f}')\n# print(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_albert.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-03-07 23:48\nfrom hanlp.metrics.parsing import conllx_eval\n\nfrom hanlp.datasets.parsing.ptb import PTB_SD330_DEV, PTB_SD330_TRAIN, PTB_SD330_TEST, PTB_TOKEN_MAPPING\nfrom hanlp.components.parsers.biaffine_parser_tf import BiaffineTransformerDependencyParserTF, \\\n    StructuralAttentionDependencyParserTF\nfrom hanlp.pretrained.glove import GLOVE_840B_300D\nfrom tests import cdroot\n\ncdroot()\nsave_dir = 'data/model/dep/ptb_sa_glove'\nparser = StructuralAttentionDependencyParserTF()\n# parser.fit(PTB_SD330_TRAIN, PTB_SD330_DEV, save_dir, 'bert-base-uncased',\n#            batch_size=3000,\n#            warmup_steps_ratio=.1,\n#            token_mapping=PTB_TOKEN_MAPPING,\n#            samples_per_batch=150,\n#            transformer_dropout=.33,\n#            masked_lm_dropout=.33,\n#            # learning_rate=2e-3,\n#            # learning_rate_transformer=1e-5,\n#            masked_lm_embed={'class_name': 'HanLP>Word2VecEmbedding',\n#                             'config': {\n#                                 'trainable': False,\n#                                 # 'embeddings_initializer': 'zero',\n#                                 'filepath': GLOVE_840B_300D,\n#                                 'expand_vocab': False,\n#                                 'lowercase': True,\n#                                 'cpu': False\n#                             }}\n#            # alpha=1,\n#            # early_stopping_patience=10,\n#            # num_decoder_layers=2,\n#            )\nparser.load(save_dir)\n# output = f'{save_dir}/test.predict.conll'\nparser.evaluate(PTB_SD330_TEST, save_dir, warm_up=False)\n# uas, las = conllx_eval.evaluate(PTB_SD330_TEST, output)\n# print(f'Official UAS: {uas:.4f} LAS: {las:.4f}')\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_albert_topk.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-03-07 23:48\nfrom hanlp.metrics.parsing import conllx_eval\n\nfrom hanlp.datasets.parsing.ptb import PTB_SD330_DEV, PTB_SD330_TRAIN, PTB_SD330_TEST, PTB_TOKEN_MAPPING\nfrom hanlp.components.parsers.biaffine_parser_tf import BiaffineTransformerDependencyParserTF, \\\n    StructuralAttentionDependencyParserTF\nfrom hanlp.pretrained.glove import GLOVE_840B_300D\nfrom tests import cdroot\n\ncdroot()\nsave_dir = 'data/model/dep/ptb_sa_topk'\nparser = StructuralAttentionDependencyParserTF()\nparser.fit(PTB_SD330_TRAIN, PTB_SD330_DEV, save_dir, 'bert-base-uncased',\n           batch_size=3000,\n           warmup_steps_ratio=.1,\n           token_mapping=PTB_TOKEN_MAPPING,\n           samples_per_batch=150,\n           transformer_dropout=.33,\n           masked_lm_dropout=.33,\n           learning_rate=2e-3,\n           learning_rate_transformer=1e-5,\n\n           # alpha=1,\n           # early_stopping_patience=10,\n           # num_decoder_layers=2,\n           )\nparser.load(save_dir)\n# output = f'{save_dir}/test.predict.conll'\nparser.evaluate(PTB_SD330_TEST, save_dir, warm_up=False)\n# uas, las = conllx_eval.evaluate(PTB_SD330_TEST, output)\n# print(f'Official UAS: {uas:.4f} LAS: {las:.4f}')\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_bert.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-03-07 23:48\nfrom hanlp.metrics.parsing import conllx_eval\n\nfrom hanlp.datasets.parsing.ptb import PTB_SD330_DEV, PTB_SD330_TRAIN, PTB_SD330_TEST, PTB_TOKEN_MAPPING\nfrom hanlp.components.parsers.biaffine_parser_tf import BiaffineTransformerDependencyParserTF, \\\n    StructuralAttentionDependencyParserTF\nfrom hanlp.pretrained.glove import GLOVE_840B_300D\nfrom tests import cdroot\n\ncdroot()\nsave_dir = 'data/model/dep/ptb_sa_glove'\nparser = StructuralAttentionDependencyParserTF()\n# parser.fit(PTB_SD330_TRAIN, PTB_SD330_DEV, save_dir, 'bert-base-uncased',\n#            batch_size=3000,\n#            warmup_steps_ratio=.1,\n#            token_mapping=PTB_TOKEN_MAPPING,\n#            samples_per_batch=150,\n#            transformer_dropout=.33,\n#            masked_lm_dropout=.33,\n#            # learning_rate=2e-3,\n#            # learning_rate_transformer=1e-5,\n#            masked_lm_embed={'class_name': 'HanLP>Word2VecEmbedding',\n#                             'config': {\n#                                 'trainable': False,\n#                                 # 'embeddings_initializer': 'zero',\n#                                 'filepath': GLOVE_840B_300D,\n#                                 'expand_vocab': False,\n#                                 'lowercase': True,\n#                                 'cpu': False\n#                             }}\n#            # alpha=1,\n#            # early_stopping_patience=10,\n#            # num_decoder_layers=2,\n#            )\nparser.load(save_dir)\n# output = f'{save_dir}/test.predict.conll'\nparser.evaluate(PTB_SD330_TEST, save_dir, warm_up=False)\n# uas, las = conllx_eval.evaluate(PTB_SD330_TEST, output)\n# print(f'Official UAS: {uas:.4f} LAS: {las:.4f}')\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_ptb_dep_sa_pos_bert.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-03-07 23:48\nfrom hanlp.metrics.parsing import conllx_eval\n\nfrom hanlp.datasets.parsing.ptb import PTB_SD330_DEV, PTB_SD330_TRAIN, PTB_SD330_TEST, PTB_TOKEN_MAPPING\nfrom hanlp.components.parsers.biaffine_parser_tf import BiaffineTransformerDependencyParserTF, \\\n    StructuralAttentionDependencyParserTF\nfrom hanlp.pretrained.glove import GLOVE_840B_300D\nfrom tests import cdroot\n\ncdroot()\nsave_dir = 'data/model/dep/ptb_sa_bert_joint_pos'\nparser = StructuralAttentionDependencyParserTF()\nparser.fit('data/ptb-dep/train.conllx', 'data/ptb-dep/dev.conllx', save_dir, 'bert-base-uncased',\n           batch_size=256,\n           warmup_steps_ratio=.1,\n           token_mapping=PTB_TOKEN_MAPPING,\n           samples_per_batch=150,\n           transformer_dropout=.33,\n           masked_lm_dropout=.33,\n           learning_rate=2e-3,\n           learning_rate_transformer=1e-5,\n           joint_pos=True\n           # alpha=1,\n           # early_stopping_patience=10,\n           # num_decoder_layers=2,\n           )\n# parser.load(save_dir)\n# output = f'{save_dir}/test.predict.conll'\nparser.evaluate('data/ptb-dep/test.conllx', save_dir, warm_up=False)\n# uas, las = conllx_eval.evaluate(PTB_SD330_TEST, output)\n# print(f'Official UAS: {uas:.4f} LAS: {las:.4f}')\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_ptb_pos_rnn_fasttext.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-10-25 21:34\n\nimport tensorflow as tf\n\nfrom hanlp.components.taggers.pos_tf import RNNPartOfSpeechTaggerTF\nfrom hanlp.pretrained.fasttext import FASTTEXT_CC_300_EN\nfrom tests import cdroot\n\ncdroot()\ntagger = RNNPartOfSpeechTaggerTF()\nsave_dir = 'data/model/pos/ptb_pos_rnn_fasttext'\noptimizer = tf.keras.optimizers.SGD(lr=0.015)\n# optimizer = 'adam'\ntagger.fit('data/ptb-pos/train.tsv',\n           'data/ptb-pos/dev.tsv',\n           batch_size=10,\n           save_dir=save_dir,\n           embeddings={'class_name': 'HanLP>FastTextEmbedding',\n                       'config': {'filepath': FASTTEXT_CC_300_EN}},\n           optimizer=optimizer,\n           lr_decay_per_epoch=0.05,\n           rnn_units=100,\n           rnn_input_dropout=0.5,\n           rnn_output_dropout=0.5,\n           epochs=100,\n           verbose=True)\ntagger.load(save_dir)\ntagger.evaluate('data/ptb-pos/test.tsv', save_dir=save_dir, output=False)\nprint(tagger.predict(['This' 'time', 'is', 'for', 'dinner']))\nprint(tagger.predict([['This', 'is', 'an', 'old', 'story'],\n                      ['Not', 'this', 'year', '.']]))\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_semeval15_dm.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-01-01 18:26\nfrom hanlp.components.parsers.biaffine_parser_tf import BiaffineSemanticDependencyParserTF\nfrom hanlp.pretrained.glove import GLOVE_6B_100D\nfrom tests import cdroot\n\ncdroot()\nsave_dir = 'data/model/sdp/semeval15_biaffine_dm'\nparser = BiaffineSemanticDependencyParserTF()\nparser.fit('data/semeval15/en.dm.train.conll', 'data/semeval15/en.dm.dev.conll', save_dir,\n           pretrained_embed={'class_name': 'HanLP>Word2VecEmbedding',\n                             'config': {\n                                 'trainable': False,\n                                 'embeddings_initializer': 'zero',\n                                 'filepath': GLOVE_6B_100D,\n                                 'expand_vocab': True,\n                                 'lowercase': True,\n                                 'normalize': True,\n                             }},\n           )\nparser.load(save_dir)  # disable variational dropout during evaluation so as to use CudaLSTM\nsentence = [('Is', 'VBZ'), ('this', 'DT'), ('the', 'DT'), ('future', 'NN'), ('of', 'IN'), ('chamber', 'NN'),\n            ('music', 'NN'), ('?', '.')]\nprint(parser.predict(sentence))\nparser.evaluate('data/semeval15/en.id.dm.auto.conllu', save_dir)\nparser.evaluate('data/semeval15/en.ood.dm.auto.conllu', save_dir)\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_semeval15_pas.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-01-01 18:26\nfrom hanlp.components.parsers.biaffine_parser_tf import BiaffineSemanticDependencyParserTF\nfrom hanlp.pretrained.glove import GLOVE_6B_100D\nfrom tests import cdroot\n\ncdroot()\nsave_dir = 'data/model/sdp/semeval15_biaffine_pas'\nparser = BiaffineSemanticDependencyParserTF()\nparser.fit('data/semeval15/en.pas.train.conll', 'data/semeval15/en.pas.dev.conll', save_dir,\n           pretrained_embed={'class_name': 'HanLP>Word2VecEmbedding',\n                             'config': {\n                                 'trainable': False,\n                                 'embeddings_initializer': 'zero',\n                                 'filepath': GLOVE_6B_100D,\n                                 'expand_vocab': True,\n                                 'lowercase': True,\n                                 'normalize': True,\n                             }},\n           )\nparser.load(save_dir)  # disable variational dropout during evaluation so as to use CudaLSTM\nsentence = [('Is', 'VBZ'), ('this', 'DT'), ('the', 'DT'), ('future', 'NN'), ('of', 'IN'), ('chamber', 'NN'),\n            ('music', 'NN'), ('?', '.')]\nprint(parser.predict(sentence))\nparser.evaluate('data/semeval15/en.id.pas.conll', save_dir)\nparser.evaluate('data/semeval15/en.ood.pas.conll', save_dir)\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_semeval15_psd.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-01-01 18:26\nfrom hanlp.components.parsers.biaffine_parser_tf import BiaffineSemanticDependencyParserTF\nfrom hanlp.pretrained.glove import GLOVE_6B_100D\nfrom tests import cdroot\n\ncdroot()\nsave_dir = 'data/model/sdp/semeval15_biaffine_psd'\nparser = BiaffineSemanticDependencyParserTF()\nparser.fit('data/semeval15/en.psd.train.conll', 'data/semeval15/en.psd.dev.conll', save_dir,\n           pretrained_embed={'class_name': 'HanLP>Word2VecEmbedding',\n                             'config': {\n                                 'trainable': False,\n                                 'embeddings_initializer': 'zero',\n                                 'filepath': GLOVE_6B_100D,\n                                 'expand_vocab': True,\n                                 'lowercase': True,\n                                 'normalize': True,\n                             }},\n           )\nparser.load(save_dir)  # disable variational dropout during evaluation so as to use CudaLSTM\nsentence = [('Is', 'VBZ'), ('this', 'DT'), ('the', 'DT'), ('future', 'NN'), ('of', 'IN'), ('chamber', 'NN'),\n            ('music', 'NN'), ('?', '.')]\nprint(parser.predict(sentence))\nparser.evaluate('data/semeval15/en.id.psd.conll', save_dir)\nparser.evaluate('data/semeval15/en.ood.psd.conll', save_dir)\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_semeval16_news.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-26 23:20\nfrom hanlp.datasets.parsing.semeval16 import SEMEVAL2016_NEWS_TRAIN, SEMEVAL2016_NEWS_DEV, SEMEVAL2016_NEWS_TEST\nfrom hanlp.pretrained.word2vec import SEMEVAL16_EMBEDDINGS_300_NEWS_CN\nfrom hanlp.utils.tf_util import nice\n\nnice()\nfrom hanlp.components.parsers.biaffine_parser_tf import BiaffineSemanticDependencyParserTF\nfrom tests import cdroot\n\ncdroot()\nsave_dir = 'data/model/sdp/semeval16-news'\nparser = BiaffineSemanticDependencyParserTF()\nparser.fit(SEMEVAL2016_NEWS_TRAIN, SEMEVAL2016_NEWS_DEV, save_dir,\n           pretrained_embed={'class_name': 'HanLP>Word2VecEmbedding',\n                             'config': {\n                                 'trainable': False,\n                                 'embeddings_initializer': 'zero',\n                                 'filepath': SEMEVAL16_EMBEDDINGS_300_NEWS_CN,\n                                 'expand_vocab': True,\n                                 'lowercase': True,\n                                 'normalize': True,\n                             }},\n           )\nparser.load(save_dir)\nsentence = [('中国', 'NR'), ('批准', 'VV'), ('设立', 'VV'), ('外商', 'NN'), ('投资', 'NN'), ('企业', 'NN'), ('逾', 'VV'),\n            ('三十万', 'CD'), ('家', 'M')]\nprint(parser.predict(sentence))\nparser.evaluate(SEMEVAL2016_NEWS_TEST, save_dir)\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tf/train/train_semeval16_text.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-26 23:20\nfrom hanlp.datasets.parsing.semeval16 import SEMEVAL2016_TEXT_TRAIN, SEMEVAL2016_TEXT_DEV, SEMEVAL2016_TEXT_TEST\nfrom hanlp.pretrained.word2vec import SEMEVAL16_EMBEDDINGS_300_TEXT_CN\nfrom hanlp.utils.tf_util import nice\n\nnice()\nfrom hanlp.components.parsers.biaffine_parser_tf import BiaffineSemanticDependencyParserTF\nfrom tests import cdroot\n\ncdroot()\nsave_dir = 'data/model/sdp/semeval16-text'\nparser = BiaffineSemanticDependencyParserTF()\nparser.fit(SEMEVAL2016_TEXT_TRAIN, SEMEVAL2016_TEXT_DEV, save_dir,\n           pretrained_embed={'class_name': 'HanLP>Word2VecEmbedding',\n                             'config': {\n                                 'trainable': False,\n                                 'embeddings_initializer': 'zero',\n                                 'filepath': SEMEVAL16_EMBEDDINGS_300_TEXT_CN,\n                                 'expand_vocab': True,\n                                 'lowercase': True,\n                                 'normalize': True,\n                             }},\n           )\nparser.load(save_dir)\nsentence = [('中国', 'NR'), ('批准', 'VV'), ('设立', 'VV'), ('外商', 'NN'), ('投资', 'NN'), ('企业', 'NN'), ('逾', 'VV'),\n            ('三十万', 'CD'), ('家', 'M')]\nprint(parser.predict(sentence))\nparser.evaluate(SEMEVAL2016_TEXT_TEST, save_dir)\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tok_mtl.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"colab_type\": \"text\",\n    \"id\": \"view-in-github\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/tok_mtl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Ftok_mtl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 加载模型\\n\",\n    \"HanLP的工作流程是先加载模型，模型的标示符存储在`hanlp.pretrained`这个包中，按照NLP任务归类。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"4M7ka0K5OMWU\",\n    \"outputId\": \"9a1dc26a-786a-4dce-c013-7ae5017a8805\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'OPEN_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/mtl/open_tok_pos_ner_srl_dep_sdp_con_electra_small_20201223_035557.zip',\\n\",\n       \" 'OPEN_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH': 'https://file.hankcs.com/hanlp/mtl/open_tok_pos_ner_srl_dep_sdp_con_electra_base_20201223_201906.zip',\\n\",\n       \" 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_small_20210111_124159.zip',\\n\",\n       \" 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_base_20210111_124519.zip',\\n\",\n       \" 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ERNIE_GRAM_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_ernie_gram_base_aug_20210904_145403.zip',\\n\",\n       \" 'UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_MT5_SMALL': 'https://file.hankcs.com/hanlp/mtl/ud_ontonotes_tok_pos_lem_fea_ner_srl_dep_sdp_con_mt5_small_20210228_123458.zip',\\n\",\n       \" 'UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_XLMR_BASE': 'https://file.hankcs.com/hanlp/mtl/ud_ontonotes_tok_pos_lem_fea_ner_srl_dep_sdp_con_xlm_base_20210602_211620.zip',\\n\",\n       \" 'NPCMJ_UD_KYOTO_TOK_POS_CON_BERT_BASE_CHAR_JA': 'https://file.hankcs.com/hanlp/mtl/npcmj_ud_kyoto_tok_pos_ner_dep_con_srl_bert_base_char_ja_20210914_133742.zip'}\"\n      ]\n     },\n     \"execution_count\": 1,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"import hanlp\\n\",\n    \"hanlp.pretrained.mtl.ALL # MTL多任务，具体任务见模型名称，语种见名称最后一个字段或相应语料库\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"BMW528wGNulM\"\n   },\n   \"source\": [\n    \"调用`hanlp.load`进行加载，模型会自动下载到本地缓存。自然语言处理分为许多任务，分词只是最初级的一个。与其每个任务单独创建一个模型，不如利用HanLP的联合模型一次性完成多个任务：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"0tmKBu7sNAXX\",\n    \"outputId\": \"e0187328-c6d2-47fe-cf84-c5b44703940b\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"HanLP = hanlp.load(hanlp.pretrained.mtl.CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 分词\\n\",\n    \"任务越少，速度越快。如指定仅执行分词，默认细粒度：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 35\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"387cbf30-4d70-44b1-d64b-b7a5c22ae31e\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"阿婆主 来到 北京 立方庭 参观 自然 语义 科技 公司 。\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"HanLP('阿婆主来到北京立方庭参观自然语义科技公司。', tasks='tok').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"jj1Jk-2sPHYx\"\n   },\n   \"source\": [\n    \"执行粗颗粒度分词：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 35\n    },\n    \"id\": \"1goEC7znPNkI\",\n    \"outputId\": \"ddf15a17-2f5d-4bc3-d145-908fb6176552\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"阿婆主 来到 北京立方庭 参观 自然语义科技公司 。\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"HanLP('阿婆主来到北京立方庭参观自然语义科技公司。', tasks='tok/coarse').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"wxctCigrTKu-\"\n   },\n   \"source\": [\n    \"同时执行细粒度和粗粒度分词：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"Zo08uquCTFSk\",\n    \"outputId\": \"bf24a01a-a09b-4b78-fdec-2bb705b4becb\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'tok/fine': ['阿婆主', '来到', '北京', '立方庭', '参观', '自然', '语义', '科技', '公司', '。'],\\n\",\n       \" 'tok/coarse': ['阿婆主', '来到', '北京立方庭', '参观', '自然语义科技公司', '。']}\"\n      ]\n     },\n     \"execution_count\": 5,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"HanLP('阿婆主来到北京立方庭参观自然语义科技公司。', tasks='tok*')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"`coarse`为粗分，`fine`为细分。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"#### 注意\\n\",\n    \"Native API的输入单位限定为句子，需使用[多语种分句模型](https://github.com/hankcs/HanLP/blob/master/plugins/hanlp_demo/hanlp_demo/sent_split.py)或[基于规则的分句函数](https://github.com/hankcs/HanLP/blob/master/hanlp/utils/rules.py#L19)先行分句。RESTful同时支持全文、句子、已分词的句子。除此之外，RESTful和native两种API的语义设计完全一致，用户可以无缝互换。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"suUL042zPpLj\"\n   },\n   \"source\": [\n    \"## 自定义词典\\n\",\n    \"自定义词典为分词任务的成员变量，要操作自定义词典，先获取分词任务，以细分标准为例：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"AzYShIssP6kq\",\n    \"outputId\": \"7f07897c-8a97-4193-855d-d9e296581d0c\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"<hanlp.components.mtl.tasks.tok.tag_tok.TaggingTokenization at 0x1527337f0>\"\n      ]\n     },\n     \"execution_count\": 6,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"tok = HanLP['tok/fine']\\n\",\n    \"tok\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"自定义词典为分词任务的成员变量：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"metadata\": {\n    \"id\": \"1q4MUpgVQNlu\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"(None, None)\"\n      ]\n     },\n     \"execution_count\": 7,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"tok.dict_combine, tok.dict_force\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"2zZkH9tRQOoi\",\n    \"outputId\": \"c231c35b-1a5f-4b54-e5c3-8680d2cc1515\",\n    \"pycharm\": {\n     \"name\": \"#%% md\\n\"\n    }\n   },\n   \"source\": [\n    \"HanLP支持合并和强制两种优先级的自定义词典，以满足不同场景的需求。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"F-9gAeIVQUFG\",\n    \"pycharm\": {\n     \"name\": \"#%% md\\n\"\n    }\n   },\n   \"source\": [\n    \"不挂词典：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 8,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"F8M8cyBrQduw\",\n    \"outputId\": \"c3bf7ec5-b1d4-4207-a979-2c85754c7cd7\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"商品 和 服务 项目\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"tok.dict_force = tok.dict_combine = None\\n\",\n    \"HanLP(\\\"商品和服务项目\\\", tasks='tok/fine').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"DDqQxqQaTayv\",\n    \"pycharm\": {\n     \"name\": \"#%% md\\n\"\n    }\n   },\n   \"source\": [\n    \"### 强制模式\\n\",\n    \"强制模式优先输出正向最长匹配到的自定义词条（慎用，详见[《自然语言处理入门》](http://nlp.hankcs.com/book.php)第二章）：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 9,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"bjnEqDaATdVr\",\n    \"outputId\": \"3a282acc-5716-45e4-e1e2-96eefb8ee342\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"商品 和服 务 项目\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"tok.dict_force = {'和服', '服务项目'}\\n\",\n    \"HanLP(\\\"商品和服务项目\\\", tasks='tok/fine').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"ldKAnVoSTgxb\",\n    \"pycharm\": {\n     \"name\": \"#%% md\\n\"\n    }\n   },\n   \"source\": [\n    \"与大众的朴素认知不同，词典优先级最高未必是好事，极有可能匹配到不该分出来的自定义词语，导致歧义。自定义词语越长，越不容易发生歧义。这启发我们将强制模式拓展为强制校正功能。\\n\",\n    \"\\n\",\n    \"强制校正原理相似，但会将匹配到的自定义词条替换为相应的分词结果:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 10,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"bwIu0f6wTgbF\",\n    \"outputId\": \"b941b079-5202-420a-e7f3-8f1617a2545c\",\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"商品 和 服务 项目\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"tok.dict_force = {'和服务': ['和', '服务']}\\n\",\n    \"HanLP(\\\"商品和服务项目\\\", tasks='tok/fine').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### 合并模式\\n\",\n    \"合并模式的优先级低于统计模型，即`dict_combine`会在统计模型的分词结果上执行最长匹配并合并匹配到的词条。一般情况下，推荐使用该模式。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 11,\n   \"metadata\": {\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"商品 和 服务项目\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"tok.dict_force = None\\n\",\n    \"tok.dict_combine = {'和服', '服务项目'}\\n\",\n    \"HanLP(\\\"商品和服务项目\\\", tasks='tok/fine').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"9aRzEeRvTlRr\"\n   },\n   \"source\": [\n    \"需要算法基础才能理解，初学者可参考[《自然语言处理入门》](http://nlp.hankcs.com/book.php)。\\n\",\n    \"#### 空格单词\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"含有空格、制表符等（Transformer tokenizer去掉的字符）的词语需要用`tuple`的形式提供：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 12,\n   \"metadata\": {\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"['如何', '评价', 'iPad Pro', '？', 'iPad  Pro', '有', '2个空格']\"\n      ]\n     },\n     \"execution_count\": 12,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"tok.dict_combine = {('iPad', 'Pro'), '2个空格'}\\n\",\n    \"HanLP(\\\"如何评价iPad Pro ？iPad  Pro有2个空格\\\", tasks='tok/fine')['tok/fine']\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"聪明的用户请继续阅读，`tuple`词典中的字符串其实等价于该字符串的所有可能的切分方式：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 13,\n   \"metadata\": {\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"dict_keys([('2', '个', '空格'), ('2', '个', '空', '格'), ('2', '个空', '格'), ('2', '个空格'), ('2个', '空', '格'), ('2个', '空格'), ('2个空格',), ('iPad', 'Pro'), ('2个空', '格')])\"\n      ]\n     },\n     \"execution_count\": 13,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"dict(tok.dict_combine.config[\\\"dictionary\\\"]).keys()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## 单词位置\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"HanLP支持输出每个单词在文本中的原始位置，以便用于搜索引擎等场景。在词法分析中，非语素字符（空格、换行、制表符等）会被剔除，此时需要额外的位置信息才能定位每个单词：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 14,\n   \"metadata\": {\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"[['2021 年', 0, 6], ['HanLPv2.1', 7, 16], ['为', 17, 18], ['生产', 18, 20], ['环境', 20, 22], ['带来', 22, 24], ['次', 24, 25], ['世代', 25, 27], ['最', 27, 28], ['先进', 28, 30], ['的', 30, 31], ['多', 31, 32], ['语种', 32, 34], ['NLP', 34, 37], ['技术', 37, 39], ['。', 39, 40]]\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"tok.config.output_spans = True\\n\",\n    \"sent = '2021 年\\\\nHanLPv2.1 为生产环境带来次世代最先进的多语种NLP技术。'\\n\",\n    \"word_offsets = HanLP(sent, tasks='tok/fine')['tok/fine']\\n\",\n    \"print(word_offsets)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"返回格式为三元组（单词，单词的起始下标，单词的终止下标），下标以字符级别计量。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 15,\n   \"metadata\": {\n    \"pycharm\": {\n     \"name\": \"#%%\\n\"\n    }\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"for word, begin, end in word_offsets:\\n\",\n    \"    assert word == sent[begin:end]\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"authorship_tag\": \"ABX9TyNRpO7rdchCK1UmB0nQmPrG\",\n   \"collapsed_sections\": [],\n   \"include_colab_link\": true,\n   \"name\": \"tok_mtl.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tok_restful.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/tok_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Ftok_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"pip install hanlp_restful -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 创建客户端\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"id\": \"0tmKBu7sNAXX\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from hanlp_restful import HanLPClient\\n\",\n    \"HanLP = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh') # auth不填则匿名，zh中文，mul多语种\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"EmZDmLn9aGxG\"\n   },\n   \"source\": [\n    \"#### 申请秘钥\\n\",\n    \"由于服务器算力有限，匿名用户每分钟限2次调用。如果你需要更多调用次数，[建议申请免费公益API秘钥auth](https://bbs.hanlp.com/t/hanlp2-1-restful-api/53)。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 分词\\n\",\n    \"HanLP线上模型训练自`9970`万字的大型综合语料库，覆盖新闻、社交媒体、金融、法律等多个领域，是已知范围内**全世界最大**的中文分词语料库。语料库规模决定实际效果，面向生产环境的语料库应当在千万字量级。自然语义的语言学专家一直在持续标注该语料库，与时俱进保持最先进的分词质量。\\n\",\n    \"在分词标准上，HanLP提供细粒度和粗粒度两种颗粒度，细粒度适合搜索引擎业务，粗粒度适合文本挖掘业务。\\n\",\n    \"### 细粒度分词\\n\",\n    \"默认细粒度：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"[['商品', '和', '服务', '。'],\\n\",\n       \" ['阿婆主', '来到', '北京', '立方庭', '参观', '自然', '语义', '科技', '公司', '。']]\"\n      ]\n     },\n     \"execution_count\": 2,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"HanLP.tokenize('商品和服务。阿婆主来到北京立方庭参观自然语义科技公司。')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"用户也可以直接将`HanLP`当作函数调用，并且打印漂亮的分词结果：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 35\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"6fbb3eac-df26-4a55-8ba9-975d6cede227\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">商品&nbsp;和&nbsp;服务&nbsp;。</pre></div><br><div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">阿婆主&nbsp;来到&nbsp;北京&nbsp;立方庭&nbsp;参观&nbsp;自然&nbsp;语义&nbsp;科技&nbsp;公司&nbsp;。</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"HanLP('商品和服务。阿婆主来到北京立方庭参观自然语义科技公司。', tasks='tok').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"返回类型为[Document](https://hanlp.hankcs.com/docs/api/common/document.html)，是`dict`的子类，拓展了很多操作各种语言学结构的方法。\\n\",\n    \"\\n\",\n    \"两个接口都会对文本进行分句，所以返回的结果一定是句子的列表。推荐在不超过服务器允许的最大长度的前提下，尽量传入整篇文章，以提高分词速度。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"jj1Jk-2sPHYx\"\n   },\n   \"source\": [\n    \"### 粗粒度分词\\n\",\n    \"执行粗颗粒度分词：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"[['商品', '和', '服务', '。'], ['阿婆主', '来到', '北京', '立方庭', '参观', '自然语义科技公司']]\"\n      ]\n     },\n     \"execution_count\": 4,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"HanLP.tokenize('商品和服务。阿婆主来到北京立方庭参观自然语义科技公司', coarse=True)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"或者直接当函数调用：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 35\n    },\n    \"id\": \"1goEC7znPNkI\",\n    \"outputId\": \"ddf15a17-2f5d-4bc3-d145-908fb6176552\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">阿婆主&nbsp;来到&nbsp;北京&nbsp;立方庭&nbsp;参观&nbsp;自然语义科技公司&nbsp;。</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"HanLP('阿婆主来到北京立方庭参观自然语义科技公司。', tasks='tok/coarse').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"wxctCigrTKu-\"\n   },\n   \"source\": [\n    \"### 同时执行细粒度和粗粒度分词\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"Zo08uquCTFSk\",\n    \"outputId\": \"bf24a01a-a09b-4b78-fdec-2bb705b4becb\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'tok/fine': [['阿婆主', '来到', '北京', '立方庭', '参观', '自然', '语义', '科技', '公司', '。']],\\n\",\n       \" 'tok/coarse': [['阿婆主', '来到', '北京', '立方庭', '参观', '自然语义科技公司', '。']]}\"\n      ]\n     },\n     \"execution_count\": 6,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"HanLP('阿婆主来到北京立方庭参观自然语义科技公司。', tasks='tok*')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"`fine`为细分，`coarse`为粗分。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### 多语种分词\\n\",\n    \"得益于语言无关的设计，HanLP支持包括简繁中英日俄法德在内的104种语言上的分词。这一切，只需指定`language='mul'`即可实现。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">In&nbsp;2021&nbsp;,&nbsp;HanLPv2.1&nbsp;delivers&nbsp;state-of-the-art&nbsp;multilingual&nbsp;NLP&nbsp;techniques&nbsp;to&nbsp;production&nbsp;environments&nbsp;.</pre></div><br><div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">2021&nbsp;年&nbsp;、&nbsp;HanLPv2.1&nbsp;は&nbsp;次&nbsp;世代&nbsp;の&nbsp;最&nbsp;先端&nbsp;多&nbsp;言語&nbsp;NLP&nbsp;技術&nbsp;を&nbsp;本番&nbsp;環境&nbsp;に&nbsp;導入&nbsp;します&nbsp;。</pre></div><br><div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">2021&nbsp;年&nbsp;HanLPv2.1&nbsp;为&nbsp;生产&nbsp;环境&nbsp;带来&nbsp;次世代&nbsp;最&nbsp;先进的&nbsp;多&nbsp;语种&nbsp;NLP&nbsp;技术&nbsp;。</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"HanLP(['In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques to production environments.',\\n\",\n    \"       '2021年、HanLPv2.1は次世代の最先端多言語NLP技術を本番環境に導入します。',\\n\",\n    \"       '2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。'], tasks='tok', language='mul').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"自然语言处理分为许多任务，分词只是最初级的一个。也许大家只听说过中文分词，但HanLP并不局限于分词。HanLP的使命是普及最前沿的自然语言处理技术到生产环境，所以在其他教程中你会见到许多更高级的NLP任务以及相应的API用法。\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"tok_restful.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tok_stl.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/tok_stl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Ftok_stl.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 加载模型\\n\",\n    \"HanLP的工作流程是先加载模型，模型的标示符存储在`hanlp.pretrained`这个包中，按照NLP任务归类。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"4M7ka0K5OMWU\",\n    \"outputId\": \"f931579a-f5a8-487a-a89e-33d5477584c3\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'SIGHAN2005_PKU_CONVSEG': 'https://file.hankcs.com/hanlp/tok/sighan2005-pku-convseg_20200110_153722.zip',\\n\",\n       \" 'SIGHAN2005_MSR_CONVSEG': 'https://file.hankcs.com/hanlp/tok/convseg-msr-nocrf-noembed_20200110_153524.zip',\\n\",\n       \" 'CTB6_CONVSEG': 'https://file.hankcs.com/hanlp/tok/ctb6_convseg_nowe_nocrf_20200110_004046.zip',\\n\",\n       \" 'PKU_NAME_MERGED_SIX_MONTHS_CONVSEG': 'https://file.hankcs.com/hanlp/tok/pku98_6m_conv_ngram_20200110_134736.zip',\\n\",\n       \" 'LARGE_ALBERT_BASE': 'https://file.hankcs.com/hanlp/tok/large_corpus_cws_albert_base_20211228_160926.zip',\\n\",\n       \" 'SIGHAN2005_PKU_BERT_BASE_ZH': 'https://file.hankcs.com/hanlp/tok/sighan2005_pku_bert_base_zh_20201231_141130.zip',\\n\",\n       \" 'COARSE_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/tok/coarse_electra_small_20220616_012050.zip',\\n\",\n       \" 'FINE_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/tok/fine_electra_small_20220615_231803.zip',\\n\",\n       \" 'CTB9_TOK_ELECTRA_SMALL': 'https://file.hankcs.com/hanlp/tok/ctb9_electra_small_20220215_205427.zip',\\n\",\n       \" 'CTB9_TOK_ELECTRA_BASE': 'http://download.hanlp.com/tok/extra/ctb9_tok_electra_base_20220426_111949.zip',\\n\",\n       \" 'CTB9_TOK_ELECTRA_BASE_CRF': 'http://download.hanlp.com/tok/extra/ctb9_tok_electra_base_crf_20220426_161255.zip',\\n\",\n       \" 'MSR_TOK_ELECTRA_BASE_CRF': 'http://download.hanlp.com/tok/extra/msra_crf_electra_base_20220507_113936.zip',\\n\",\n       \" 'UD_TOK_MMINILMV2L6': 'https://file.hankcs.com/hanlp/tok/ud_tok_mMiniLMv2L6_no_space_mul_20220619_091824.zip',\\n\",\n       \" 'UD_TOK_MMINILMV2L12': 'https://file.hankcs.com/hanlp/tok/ud_tok_mMiniLMv2L12_no_space_mul_20220619_091159.zip'}\"\n      ]\n     },\n     \"execution_count\": 1,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"import hanlp\\n\",\n    \"hanlp.pretrained.tok.ALL # 语种见名称最后一个字段或相应语料库\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"BMW528wGNulM\"\n   },\n   \"source\": [\n    \"调用`hanlp.load`进行加载，模型会自动下载到本地缓存。自然语言处理分为许多任务，分词只是最初级的一个。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"0tmKBu7sNAXX\",\n    \"outputId\": \"8977891f-9e64-4e39-8ce6-264a791541a3\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"<hanlp.components.tokenizers.transformer.TransformerTaggingTokenizer at 0x10420e5b0>\"\n      ]\n     },\n     \"execution_count\": 2,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"tok = hanlp.load(hanlp.pretrained.tok.COARSE_ELECTRA_SMALL_ZH)\\n\",\n    \"tok\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### 进阶知识\\n\",\n    \"你可以通过加载不同的模型实现各种颗粒度、各种分词标准、各种领域的中文分词。其中，coarse和fine模型训练自`9970`万字的大型综合语料库，覆盖新闻、社交媒体、金融、法律等多个领域，是已知范围内**全世界最大**的中文分词语料库。语料库规模决定实际效果，面向生产环境的语料库应当在千万字量级。欢迎用户在自己的语料上[训练或微调模型](https://github.com/hankcs/HanLP/tree/master/plugins/hanlp_demo/hanlp_demo/zh/train)以适应新领域。语料库标注标准决定最终的分词标准，模型的准确率决定多大程度上再现该分词标准。更多背景知识请参考[《自然语言处理入门》](http://nlp.hankcs.com/book.php)。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"KYH1oEKkctuy\"\n   },\n   \"source\": [\n    \"## 执行分词\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"uzex--zFcqKB\",\n    \"outputId\": \"a4db6808-1039-4803-84af-2687cce0fa7b\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"[['商品', '和', '服务', '。'], ['晓美焰', '来到', '北京立方庭', '参观', '自然语义科技公司']]\"\n      ]\n     },\n     \"execution_count\": 3,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"tok(['商品和服务。', '晓美焰来到北京立方庭参观自然语义科技公司'])\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### 细分标准\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"你可以通过加载`FINE_ELECTRA_SMALL_ZH`模型实现细粒度中文分词：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"tok_fine = hanlp.load(hanlp.pretrained.tok.FINE_ELECTRA_SMALL_ZH)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"无论哪个模型，分词器的接口是完全一致的：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"['晓美焰', '来到', '北京', '立方庭', '参观', '自然', '语义', '科技', '公司']\"\n      ]\n     },\n     \"execution_count\": 5,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"tok_fine('晓美焰来到北京立方庭参观自然语义科技公司')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### 无限长度\\n\",\n    \"众所周知，Transformer的输入有长度限制（通常是512）。幸运地是，HanLP的滑动窗口技巧完美地突破了该限制。只要你的内存（显存）足够，HanLP就可以处理无限长的句子。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"### 并行分词\\n\",\n    \"无论是CPU还是GPU，同时传入多个句子都将并行分词。也就是说，仅花费1个句子的时间可以处理多个句子。然而工作研究中的文本通常是一篇文档，而不是许多句子。此时可以利用HanLP提供的分句功能和流水线模式优雅应对，既能处理长文本又能并行化。只需创建一个流水线`pipeline`，第一级管道分句，第二级管道分词：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"[['量体裁衣', '，', 'HanLP', '提供', 'RESTful', '和', 'native', '两种', 'API', '。'],\\n\",\n       \" ['两者', '在', '语义', '上', '保持', '一致', '，', '在', '代码', '上', '坚持', '开源', '。']]\"\n      ]\n     },\n     \"execution_count\": 6,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"HanLP = hanlp.pipeline() \\\\\\n\",\n    \"    .append(hanlp.utils.rules.split_sentence) \\\\\\n\",\n    \"    .append(tok)\\n\",\n    \"HanLP('量体裁衣，HanLP提供RESTful和native两种API。两者在语义上保持一致，在代码上坚持开源。')\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"返回结果是每个句子的分词`list`，如果要将它们合并到一个`list`里该怎么办呢？聪明的用户可能已经想到了，再加一级`lambda`管道：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"['量体裁衣', '，', 'HanLP', '提供', 'RESTful', '和', 'native', '两种', 'API', '。', '两者', '在', '语义', '上', '保持', '一致', '，', '在', '代码', '上', '坚持', '开源', '。']\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"HanLP.append(lambda sents: sum(sents, []))\\n\",\n    \"print(HanLP('量体裁衣，HanLP提供RESTful和native两种API。两者在语义上保持一致，在代码上坚持开源。'))\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"suUL042zPpLj\"\n   },\n   \"source\": [\n    \"## 自定义词典\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"1q4MUpgVQNlu\"\n   },\n   \"source\": [\n    \"智者千虑，必有一失。模型偶尔也会犯错误，比如某个旧版本模型在不挂词典时会犯以下错误（最新版已经修复）：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 8,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"2zZkH9tRQOoi\",\n    \"outputId\": \"a74db6c6-0a71-411c-de78-60621a43eded\",\n    \"scrolled\": true\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"['首相', '和', '川', '普通', '电话']\"\n      ]\n     },\n     \"execution_count\": 8,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"tok = hanlp.load('https://file.hankcs.com/hanlp/tok/coarse_electra_small_20220220_013548.zip')\\n\",\n    \"tok.dict_force = tok.dict_combine = None\\n\",\n    \"tok(\\\"首相和川普通电话\\\")\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"上面分词任务两个成员变量`dict_force`和`dict_combine`为自定义词典：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 9,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 35\n    },\n    \"id\": \"AzYShIssP6kq\",\n    \"outputId\": \"ce3bb1aa-5042-47d7-8ac9-7ed0fd478c77\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"(None, None)\"\n      ]\n     },\n     \"execution_count\": 9,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"tok.dict_combine, tok.dict_force\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"HanLP支持合并和强制两种优先级的自定义词典，以满足不同场景的需求。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"F-9gAeIVQUFG\"\n   },\n   \"source\": [\n    \"### 强制模式\\n\",\n    \"强制模式`dict_force`优先输出正向最长匹配到的自定义词条，在这个案例中，用户的第一反应也许是将`川普`加入到`dict_force`中，强制分词器输出`川普`：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 10,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"F8M8cyBrQduw\",\n    \"outputId\": \"c156513c-d13c-47f1-bc3a-c73a8649ddb1\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"[['首相', '和', '川普', '通', '电话'],\\n\",\n       \" ['银', '川普', '通人', '与', '川普', '通', '电话', '讲', '四', '川普', '通话']]\"\n      ]\n     },\n     \"execution_count\": 10,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"tok.dict_force = {'川普'}\\n\",\n    \"tok([\\\"首相和川普通电话\\\", \\\"银川普通人与川普通电话讲四川普通话\\\"])\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"DDqQxqQaTayv\"\n   },\n   \"source\": [\n    \"然而与大众的朴素认知不同，词典优先级最高未必是好事。极有可能匹配到不该分出来的自定义词语，导致歧义。即便是将`普通人`或`普通话`加入到词典中也无济于事，因为在正向最长匹配第二个句子的过程中，会匹配到`川普`而不会匹配后两者。这也解释了为什么自定义词典中存在的词可能分不出来：当歧义发生时，两个词语发生交叉冲突，自然有所取舍，无法同时输出两者。那种同时输出句子或长单词中所有可能的单词，并且允许单词交叉的算法，并非分词，而是多模式字符串匹配。你需要基本的算法知识才能理解这一点，总之一般情况下应当慎用强制模式，详见[《自然语言处理入门》](http://nlp.hankcs.com/book.php)第二章。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"自定义词语越长，越不容易发生歧义。这启发我们将强制模式拓展为强制校正功能。强制校正原理相似，但会将匹配到的自定义词条替换为相应的分词结果:\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 11,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"bjnEqDaATdVr\",\n    \"outputId\": \"2e694aed-a71f-4a28-d981-0767d9e263e9\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"[['首相', '和', '川普', '通', '电话'],\\n\",\n       \" ['银川', '普通人', '与', '川普', '通', '电话', '讲', '四川', '普通话']]\"\n      ]\n     },\n     \"execution_count\": 11,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"tok.dict_force = {'川普通电话': ['川普', '通', '电话']}\\n\",\n    \"tok([\\\"首相和川普通电话\\\", \\\"银川普通人与川普通电话讲四川普通话\\\"])\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"强制校正是一种短平快的规则补丁，需要针对每种可能产生歧义的语境，截取一个片段执行校正。当你积累了很多歧义片段与相应的校正补丁后，其实就应该考虑微调模型。微调可以让模型增量式学习这些歧义语境，摆脱对补丁规则的依赖，同时举一反三应对新的语境。从错误中积累经验，用经验预测未来，这就是机器学习与人工智能的魅力。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"事实上，“川普通电话”这种例子不需要词典即可分对。只需提供给神经网络足够的上下文线索（这也是真实文本所具备的），告诉神经网络“川普是美国总统”：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 12,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"[['首相', '和', '川普', '通', '电话', '，', '川普', '是', '美国', '总统', '。'], ['银川', '普通人', '与', '川普', '通', '电话', '讲', '四川', '普通话', '，', '川普', '是', '美国', '总统', '。']]\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"tok.dict_force = tok.dict_combine = None\\n\",\n    \"print(tok([\\\"首相和川普通电话，川普是美国总统。\\\", \\\"银川普通人与川普通电话讲四川普通话，川普是美国总统。\\\"]))\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"9aRzEeRvTlRr\"\n   },\n   \"source\": [\n    \"在上面的例子中，虽然词典对“川普”没有施加任何影响，但是更丰富的上下文促进了神经网络对语境的理解，使其得出了正确的结果。深度学习中的神经网络似乎展示了些许智能，感兴趣的初学者可参考[《自然语言处理入门》](http://nlp.hankcs.com/book.php)。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"ldKAnVoSTgxb\"\n   },\n   \"source\": [\n    \"### 合并模式\\n\",\n    \"合并模式的优先级低于统计模型，即`dict_combine`会在统计模型的分词结果上执行最长匹配并合并匹配到的词条。一般情况下，推荐使用该模式。比如，将“美国总统”加入`dict_combine`后会合并`['美国', '总统']`，而不会合并`['美国', '总', '统筹部']`为`['美国总统', '筹部']`：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 13,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"bwIu0f6wTgbF\",\n    \"outputId\": \"22807b6a-3472-431b-d1e3-95f6b761c84c\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"[['首相', '和', '川普', '通', '电话', '，', '川普', '是', '美国总统', '。'], ['银川', '普通人', '与', '川普', '通', '电话', '讲', '四川', '普通话', '，', '川普', '是', '美国总统', '。'], ['美国', '总统筹部', '部长', '是', '谁', '？']]\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"tok.dict_force = None\\n\",\n    \"tok.dict_combine = {'美国总统'}\\n\",\n    \"print(tok([\\\"首相和川普通电话，川普是美国总统。\\\", \\\"银川普通人与川普通电话讲四川普通话，川普是美国总统。\\\", \\\"美国总统筹部部长是谁？\\\"]))\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"#### 空格单词\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"含有空格、制表符等（Transformer tokenizer去掉的字符）的词语需要用`tuple`的形式提供：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 14,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"['如何', '评价', 'iPad Pro', '？', 'iPad  Pro', '有', '2个空格']\"\n      ]\n     },\n     \"execution_count\": 14,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"tok.dict_combine = {('iPad', 'Pro'), '2个空格'}\\n\",\n    \"tok(\\\"如何评价iPad Pro ？iPad  Pro有2个空格\\\")\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"聪明的用户请继续阅读，`tuple`词典中的字符串其实等价于该字符串的所有可能的切分方式：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 15,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"dict_keys([('iPad', 'Pro'), ('2个空格',), ('2', '个', '空格'), ('2', '个', '空', '格'), ('2', '个空格'), ('2', '个空', '格'), ('2个', '空', '格'), ('2个', '空格'), ('2个空', '格')])\"\n      ]\n     },\n     \"execution_count\": 15,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"dict(tok.dict_combine.config[\\\"dictionary\\\"]).keys()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## 单词位置\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"HanLP支持输出每个单词在文本中的原始位置，以便用于搜索引擎等场景。在词法分析中，非语素字符（空格、换行、制表符等）会被剔除，此时需要额外的位置信息才能定位每个单词：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 16,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"[['2021', 0, 4], ['年', 5, 6], ['HanLPv2.1', 7, 16], ['为', 17, 18], ['生产', 18, 20], ['环境', 20, 22], ['带来', 22, 24], ['次', 24, 25], ['世代', 25, 27], ['最', 27, 28], ['先进', 28, 30], ['的', 30, 31], ['多', 31, 32], ['语种', 32, 34], ['NLP', 34, 37], ['技术', 37, 39], ['。', 39, 40]]\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"tok.config.output_spans = True\\n\",\n    \"sent = '2021 年\\\\nHanLPv2.1 为生产环境带来次世代最先进的多语种NLP技术。'\\n\",\n    \"word_offsets = tok(sent)\\n\",\n    \"print(word_offsets)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"返回格式为三元组（单词，单词的起始下标，单词的终止下标），下标以字符级别计量。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 17,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"for word, begin, end in word_offsets:\\n\",\n    \"    assert word == sent[begin:end]\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"## 多语种支持\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"得益于语言无关的设计，以及大规模多语种语料库，最近HanLP发布了支持[130种语言](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/tok.html#hanlp.pretrained.tok.UD_TOK_MMINILMV2L12)的单任务分词器。用法与中文分词器相同：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 18,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"mul = hanlp.load(hanlp.pretrained.tok.UD_TOK_MMINILMV2L6)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 19,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"[['In', '2021', ',', 'HanLPv2.1', 'delivers', 'state-of-the-art', 'multilingual', 'NLP', 'techniques', 'to', 'production', 'environments', '.'], ['2021年', '、', 'HanLPv2.1', 'は', '次世代', 'の', '最', '先端', '多', '言語', 'NLP', '技術', 'を', '本番', '環境', 'に', '導入', 'し', 'ます', '。'], ['2021年', 'HanLPv2.1', '为', '生产', '环境', '带来', '次', '世代', '最', '先进', '的', '多语种', 'NLP', '技术', '。'], ['奈須きのこ', 'は', '1973年', '11月', '28日', 'に', '千葉', '県', '円空山', 'で', '生まれ', '、', 'ゲーム', '制作', '会社', '「', 'ノーツ', '」', 'の', '設立', '者', 'だ', '。']]\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(mul([\\n\",\n    \"    'In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques to production environments.',\\n\",\n    \"    '2021年、HanLPv2.1は次世代の最先端多言語NLP技術を本番環境に導入します。',\\n\",\n    \"    '2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。',\\n\",\n    \"    '奈須きのこは1973年11月28日に千葉県円空山で生まれ、ゲーム制作会社「ノーツ」の設立者だ。'\\n\",\n    \"]))\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"目前，多语种分词器的效果并不如单语种好。欢迎在你自己的单语种语料上自行训练新模型，也欢迎开源你的语料和模型。\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"authorship_tag\": \"ABX9TyPxXzYAXgLUW5uKV7v0/2iP\",\n   \"collapsed_sections\": [],\n   \"name\": \"tok_stl.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/train/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-31 20:12\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/train/finetune_ner.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2023-10-18 18:49\nimport os\n\nimport hanlp\nfrom hanlp.components.ner.transformer_ner import TransformerNamedEntityRecognizer\nfrom tests import cdroot\n\ncdroot()\n\nyour_training_corpus = 'data/ner/finetune/word_to_iobes.tsv'\nyour_development_corpus = your_training_corpus  # Use a different one in reality\nsave_dir = 'data/ner/finetune/model'\n\nif not os.path.exists(your_training_corpus):\n    os.makedirs(os.path.dirname(your_training_corpus), exist_ok=True)\n    with open(your_training_corpus, 'w') as out:\n        out.write(\n'''训练\\tB-NLP\n语料\\tE-NLP\n为\\tO\nIOBES\\tO\n格式\\tO\n'''\n        )\n\nner = TransformerNamedEntityRecognizer()\nif not os.path.exists(save_dir):\n    print('Start fine-tuning ')\n    ner.fit(\n        trn_data=your_training_corpus,\n        dev_data=your_development_corpus,\n        save_dir=save_dir,\n        epochs=50,  # Since the corpus is small, overfit it\n        finetune=hanlp.pretrained.ner.MSRA_NER_ELECTRA_SMALL_ZH,\n        # You MUST set the same parameters with the fine-tuning model:\n        average_subwords=True,\n        transformer='hfl/chinese-electra-180g-small-discriminator',\n    )\nelse:\n    print('Load fine-tuned model')\n    ner = hanlp.load(save_dir)\n\nHanLP = hanlp.pipeline()\\\n    .append(hanlp.load(hanlp.pretrained.tok.FINE_ELECTRA_SMALL_ZH), output_key='tok')\\\n    .append(ner, output_key='ner')\nHanLP(['训练语料为IOBES格式', '晓美焰来到北京立方庭参观自然语义科技公司。']).pretty_print()\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/train/open_base.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-03 14:24\nfrom hanlp_demo import block_windows\nfrom hanlp.common.dataset import SortingSamplerBuilder\nfrom hanlp.common.transform import NormalizeCharacter\nfrom hanlp.components.mtl.multi_task_learning import MultiTaskLearning\nfrom hanlp.components.mtl.tasks.constituency import CRFConstituencyParsing\nfrom hanlp.components.mtl.tasks.dep import BiaffineDependencyParsing\nfrom hanlp.components.mtl.tasks.ner.tag_ner import TaggingNamedEntityRecognition\nfrom hanlp.components.mtl.tasks.pos import TransformerTagging\nfrom hanlp.components.mtl.tasks.sdp import BiaffineSemanticDependencyParsing\nfrom hanlp.components.mtl.tasks.srl.bio_srl import SpanBIOSemanticRoleLabeling\nfrom hanlp.components.mtl.tasks.tok.tag_tok import TaggingTokenization\nfrom hanlp.datasets.ner.msra import MSRA_NER_TOKEN_LEVEL_SHORT_IOBES_TRAIN, MSRA_NER_TOKEN_LEVEL_SHORT_IOBES_DEV, \\\n    MSRA_NER_TOKEN_LEVEL_SHORT_IOBES_TEST\nfrom hanlp.datasets.parsing.ctb8 import CTB8_POS_TRAIN, CTB8_POS_DEV, CTB8_POS_TEST, CTB8_SD330_TEST, CTB8_SD330_DEV, \\\n    CTB8_SD330_TRAIN, CTB8_CWS_TRAIN, CTB8_CWS_DEV, CTB8_CWS_TEST, CTB8_BRACKET_LINE_NOEC_TRAIN, \\\n    CTB8_BRACKET_LINE_NOEC_DEV, CTB8_BRACKET_LINE_NOEC_TEST\nfrom hanlp.datasets.parsing.semeval16 import SEMEVAL2016_TEXT_TRAIN_CONLLU, SEMEVAL2016_TEXT_TEST_CONLLU, \\\n    SEMEVAL2016_TEXT_DEV_CONLLU\nfrom hanlp.datasets.srl.ontonotes5.chinese import ONTONOTES5_CONLL12_CHINESE_TEST, ONTONOTES5_CONLL12_CHINESE_DEV, \\\n    ONTONOTES5_CONLL12_CHINESE_TRAIN\nfrom hanlp.layers.embeddings.contextual_word_embedding import ContextualWordEmbedding\nfrom hanlp.layers.transformers.relative_transformer import RelativeTransformerEncoder\nfrom hanlp.utils.lang.zh.char_table import HANLP_CHAR_TABLE_JSON\nfrom hanlp.utils.log_util import cprint\nfrom tests import cdroot\n\ncdroot()\ntasks = {\n    'tok': TaggingTokenization(\n        CTB8_CWS_TRAIN,\n        CTB8_CWS_DEV,\n        CTB8_CWS_TEST,\n        SortingSamplerBuilder(batch_size=32),\n        max_seq_len=510,\n        hard_constraint=True,\n        char_level=True,\n        tagging_scheme='BMES',\n        lr=1e-3,\n        transform=NormalizeCharacter(HANLP_CHAR_TABLE_JSON, 'token'),\n    ),\n    'pos': TransformerTagging(\n        CTB8_POS_TRAIN,\n        CTB8_POS_DEV,\n        CTB8_POS_TEST,\n        SortingSamplerBuilder(batch_size=32),\n        hard_constraint=True,\n        max_seq_len=510,\n        char_level=True,\n        dependencies='tok',\n        lr=1e-3,\n    ),\n    'ner': TaggingNamedEntityRecognition(\n        MSRA_NER_TOKEN_LEVEL_SHORT_IOBES_TRAIN,\n        MSRA_NER_TOKEN_LEVEL_SHORT_IOBES_DEV,\n        MSRA_NER_TOKEN_LEVEL_SHORT_IOBES_TEST,\n        SortingSamplerBuilder(batch_size=32),\n        lr=1e-3,\n        secondary_encoder=RelativeTransformerEncoder(768, k_as_x=True),\n        dependencies='tok',\n    ),\n    'srl': SpanBIOSemanticRoleLabeling(\n        ONTONOTES5_CONLL12_CHINESE_TRAIN,\n        ONTONOTES5_CONLL12_CHINESE_DEV,\n        ONTONOTES5_CONLL12_CHINESE_TEST,\n        SortingSamplerBuilder(batch_size=32, batch_max_tokens=2048),\n        lr=1e-3,\n        crf=True,\n        dependencies='tok',\n    ),\n    'dep': BiaffineDependencyParsing(\n        CTB8_SD330_TRAIN,\n        CTB8_SD330_DEV,\n        CTB8_SD330_TEST,\n        SortingSamplerBuilder(batch_size=32),\n        lr=1e-3,\n        tree=True,\n        punct=True,\n        dependencies='tok',\n    ),\n    'sdp': BiaffineSemanticDependencyParsing(\n        SEMEVAL2016_TEXT_TRAIN_CONLLU,\n        SEMEVAL2016_TEXT_DEV_CONLLU,\n        SEMEVAL2016_TEXT_TEST_CONLLU,\n        SortingSamplerBuilder(batch_size=32),\n        lr=1e-3,\n        apply_constraint=True,\n        punct=True,\n        dependencies='tok',\n    ),\n    'con': CRFConstituencyParsing(\n        CTB8_BRACKET_LINE_NOEC_TRAIN,\n        CTB8_BRACKET_LINE_NOEC_DEV,\n        CTB8_BRACKET_LINE_NOEC_TEST,\n        SortingSamplerBuilder(batch_size=32),\n        lr=1e-3,\n        dependencies='tok',\n    )\n}\nmtl = MultiTaskLearning()\nsave_dir = 'data/model/mtl/open_tok_pos_ner_srl_dep_sdp_con_electra_base'\nmtl.fit(\n    ContextualWordEmbedding('token',\n                            \"hfl/chinese-electra-180g-base-discriminator\",\n                            average_subwords=True,\n                            max_sequence_length=512,\n                            word_dropout=.1),\n    tasks,\n    save_dir,\n    30,\n    lr=1e-3,\n    encoder_lr=5e-5,\n    grad_norm=1,\n    gradient_accumulation=2,\n    eval_trn=False,\n)\ncprint(f'Model saved in [cyan]{save_dir}[/cyan]')\nmtl.evaluate(save_dir)\nmtl.load(save_dir)\nprint(mtl('华纳音乐旗下的新垣结衣在12月21日于日本武道馆举办歌手出道活动'))\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/train/open_small.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-12-03 14:24\nfrom hanlp_demo import block_windows\nfrom hanlp.common.dataset import SortingSamplerBuilder\nfrom hanlp.common.transform import NormalizeCharacter\nfrom hanlp.components.mtl.multi_task_learning import MultiTaskLearning\nfrom hanlp.components.mtl.tasks.constituency import CRFConstituencyParsing\nfrom hanlp.components.mtl.tasks.dep import BiaffineDependencyParsing\nfrom hanlp.components.mtl.tasks.ner.tag_ner import TaggingNamedEntityRecognition\nfrom hanlp.components.mtl.tasks.pos import TransformerTagging\nfrom hanlp.components.mtl.tasks.sdp import BiaffineSemanticDependencyParsing\nfrom hanlp.components.mtl.tasks.srl.bio_srl import SpanBIOSemanticRoleLabeling\nfrom hanlp.components.mtl.tasks.tok.tag_tok import TaggingTokenization\nfrom hanlp.datasets.ner.msra import MSRA_NER_TOKEN_LEVEL_SHORT_IOBES_TEST, MSRA_NER_TOKEN_LEVEL_SHORT_IOBES_DEV, \\\n    MSRA_NER_TOKEN_LEVEL_SHORT_IOBES_TRAIN\nfrom hanlp.datasets.parsing.ctb8 import CTB8_POS_TRAIN, CTB8_POS_DEV, CTB8_POS_TEST, CTB8_SD330_TEST, CTB8_SD330_DEV, \\\n    CTB8_SD330_TRAIN, CTB8_CWS_TRAIN, CTB8_CWS_DEV, CTB8_CWS_TEST, CTB8_BRACKET_LINE_NOEC_TEST, \\\n    CTB8_BRACKET_LINE_NOEC_DEV, CTB8_BRACKET_LINE_NOEC_TRAIN\nfrom hanlp.datasets.parsing.semeval16 import SEMEVAL2016_TEXT_TRAIN_CONLLU, SEMEVAL2016_TEXT_TEST_CONLLU, \\\n    SEMEVAL2016_TEXT_DEV_CONLLU\nfrom hanlp.datasets.srl.ontonotes5.chinese import ONTONOTES5_CONLL12_CHINESE_TEST, ONTONOTES5_CONLL12_CHINESE_DEV, \\\n    ONTONOTES5_CONLL12_CHINESE_TRAIN\nfrom hanlp.layers.embeddings.contextual_word_embedding import ContextualWordEmbedding\nfrom hanlp.layers.transformers.relative_transformer import RelativeTransformerEncoder\nfrom hanlp.utils.lang.zh.char_table import HANLP_CHAR_TABLE_JSON\nfrom hanlp.utils.log_util import cprint\nfrom tests import cdroot\n\ncdroot()\ntasks = {\n    'tok': TaggingTokenization(\n        CTB8_CWS_TRAIN,\n        CTB8_CWS_DEV,\n        CTB8_CWS_TEST,\n        SortingSamplerBuilder(batch_size=32),\n        max_seq_len=510,\n        hard_constraint=True,\n        char_level=True,\n        tagging_scheme='BMES',\n        lr=1e-3,\n        transform=NormalizeCharacter(HANLP_CHAR_TABLE_JSON, 'token'),\n    ),\n    'pos': TransformerTagging(\n        CTB8_POS_TRAIN,\n        CTB8_POS_DEV,\n        CTB8_POS_TEST,\n        SortingSamplerBuilder(batch_size=32),\n        hard_constraint=True,\n        max_seq_len=510,\n        char_level=True,\n        dependencies='tok',\n        lr=1e-3,\n    ),\n    'ner': TaggingNamedEntityRecognition(\n        MSRA_NER_TOKEN_LEVEL_SHORT_IOBES_TRAIN,\n        MSRA_NER_TOKEN_LEVEL_SHORT_IOBES_DEV,\n        MSRA_NER_TOKEN_LEVEL_SHORT_IOBES_TEST,\n        SortingSamplerBuilder(batch_size=32),\n        max_seq_len=510,\n        hard_constraint=True,\n        char_level=True,\n        lr=1e-3,\n        secondary_encoder=RelativeTransformerEncoder(256, k_as_x=True, feedforward_dim=128),\n        dependencies='tok',\n    ),\n    'srl': SpanBIOSemanticRoleLabeling(\n        ONTONOTES5_CONLL12_CHINESE_TRAIN,\n        ONTONOTES5_CONLL12_CHINESE_DEV,\n        ONTONOTES5_CONLL12_CHINESE_TEST,\n        SortingSamplerBuilder(batch_size=32, batch_max_tokens=1280),\n        lr=1e-3,\n        crf=True,\n        dependencies='tok',\n    ),\n    'dep': BiaffineDependencyParsing(\n        CTB8_SD330_TRAIN,\n        CTB8_SD330_DEV,\n        CTB8_SD330_TEST,\n        SortingSamplerBuilder(batch_size=32),\n        lr=1e-3,\n        tree=True,\n        proj=True,\n        punct=True,\n        dependencies='tok',\n    ),\n    'sdp': BiaffineSemanticDependencyParsing(\n        SEMEVAL2016_TEXT_TRAIN_CONLLU,\n        SEMEVAL2016_TEXT_DEV_CONLLU,\n        SEMEVAL2016_TEXT_TEST_CONLLU,\n        SortingSamplerBuilder(batch_size=32),\n        lr=1e-3,\n        apply_constraint=True,\n        punct=True,\n        dependencies='tok',\n    ),\n    'con': CRFConstituencyParsing(\n        CTB8_BRACKET_LINE_NOEC_TRAIN,\n        CTB8_BRACKET_LINE_NOEC_DEV,\n        CTB8_BRACKET_LINE_NOEC_TEST,\n        SortingSamplerBuilder(batch_size=32),\n        lr=1e-3,\n        dependencies='tok',\n    )\n}\nmtl = MultiTaskLearning()\nsave_dir = 'data/model/mtl/open_tok_pos_ner_srl_dep_sdp_con_electra_small'\ncprint(f'Model will be saved in [cyan]{save_dir}[/cyan]')\nmtl.fit(\n    ContextualWordEmbedding('token',\n                            \"hfl/chinese-electra-180g-small-discriminator\",\n                            average_subwords=True,\n                            max_sequence_length=512,\n                            word_dropout=.1),\n    tasks,\n    save_dir,\n    30,\n    lr=1e-3,\n    encoder_lr=5e-5,\n    grad_norm=1,\n    gradient_accumulation=1,\n    eval_trn=False,\n)\ncprint(f'Model saved in [cyan]{save_dir}[/cyan]')\nmtl.evaluate(save_dir)\nmtl.load(save_dir)\nmtl('华纳音乐旗下的新垣结衣在12月21日于日本武道馆举办歌手出道活动').pretty_print()\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/train_sota_bert_pku.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-08-11 02:47\nfrom hanlp.common.dataset import SortingSamplerBuilder\nfrom hanlp.components.tokenizers.transformer import TransformerTaggingTokenizer\nfrom hanlp.datasets.tokenization.sighan2005.pku import SIGHAN2005_PKU_TRAIN_ALL, SIGHAN2005_PKU_TEST\nfrom tests import cdroot\n\ncdroot()\ntokenizer = TransformerTaggingTokenizer()\nsave_dir = 'data/model/cws/sighan2005_pku_bert_base_96.7'\ntokenizer.fit(\n    SIGHAN2005_PKU_TRAIN_ALL,\n    SIGHAN2005_PKU_TEST,  # Conventionally, no devset is used. See Tian et al. (2020).\n    save_dir,\n    'bert-base-chinese',\n    max_seq_len=300,\n    char_level=True,\n    hard_constraint=True,\n    sampler_builder=SortingSamplerBuilder(batch_size=32),\n    epochs=3,\n    adam_epsilon=1e-6,\n    warmup_steps=0.1,\n    weight_decay=0.01,\n    word_dropout=0.1,\n    seed=1660853059,\n)\ntokenizer.evaluate(SIGHAN2005_PKU_TEST, save_dir)\nprint(f'Model saved in {save_dir}')\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tst_restful.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WfGpInivS0fG\"\n   },\n   \"source\": [\n    \"<h2 align=\\\"center\\\">点击下列图标在线运行HanLP</h2>\\n\",\n    \"<div align=\\\"center\\\">\\n\",\n    \"\\t<a href=\\\"https://colab.research.google.com/github/hankcs/HanLP/blob/doc-zh/plugins/hanlp_demo/hanlp_demo/zh/tst_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://colab.research.google.com/assets/colab-badge.svg\\\" alt=\\\"Open In Colab\\\"/></a>\\n\",\n    \"\\t<a href=\\\"https://mybinder.org/v2/gh/hankcs/HanLP/doc-zh?filepath=plugins%2Fhanlp_demo%2Fhanlp_demo%2Fzh%2Ftst_restful.ipynb\\\" target=\\\"_blank\\\"><img src=\\\"https://mybinder.org/badge_logo.svg\\\" alt=\\\"Open In Binder\\\"/></a>\\n\",\n    \"</div>\\n\",\n    \"\\n\",\n    \"## 安装\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"IYwV-UkNNzFp\"\n   },\n   \"source\": [\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {\n    \"id\": \"1Uf_u7ddMhUt\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"!pip install hanlp_restful -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pp-1KqEOOJ4t\"\n   },\n   \"source\": [\n    \"## 创建客户端\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"id\": \"0tmKBu7sNAXX\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from hanlp_restful import HanLPClient\\n\",\n    \"HanLP = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh') # auth不填则匿名，zh中文，mul多语种\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"EmZDmLn9aGxG\"\n   },\n   \"source\": [\n    \"#### 申请秘钥\\n\",\n    \"由于服务器算力有限，匿名用户每分钟限2次调用。如果你需要更多调用次数，[建议申请免费公益API秘钥auth](https://bbs.hanlp.com/t/hanlp2-1-restful-api/53)。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"elA_UyssOut_\"\n   },\n   \"source\": [\n    \"## 文本风格转换\\n\",\n    \"输入短文本以及目标风格，执行文本风格转换：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 70\n    },\n    \"id\": \"BqEmDMGGOtk3\",\n    \"outputId\": \"2a0d392f-b99a-4a18-fc7f-754e2abe2e34\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"['国家对中石油寄予巨大期望。', '要用创新推动高质量发展。']\"\n      ]\n     },\n     \"execution_count\": 2,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"HanLP.text_style_transfer(['国家对中石油抱有很大的期望.', '要用创新去推动高质量的发展。'],\\n\",\n    \"                          target_style='gov_doc')\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"tst_restful.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/hanlp_demo/zh/tutorial.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"BZPSH4VkK7J2\"\n   },\n   \"source\": [\n    \"欢迎来到HanLP在线交互环境，这是一个Jupyter记事本，可以输入任意Python代码并在线执行。请点击左上角【Run】来运行这篇NLP教程。\\n\",\n    \"\\n\",\n    \"\\n\",\n    \"\\n\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"XxPAiNwSK7J4\"\n   },\n   \"source\": [\n    \"## 安装\\n\",\n    \"量体裁衣，HanLP提供**RESTful**（云端）和**native**（本地）两种API，分别面向轻量级和海量级两种场景。无论何种API何种语言，HanLP接口在语义上保持一致，你可以**任选一种**API来运行本教程。\\n\",\n    \"\\n\",\n    \"### 轻量级RESTful API\\n\",\n    \"\\n\",\n    \"仅数KB，适合敏捷开发、移动APP等场景。简单易用，无需GPU配环境，**强烈推荐**，秒速安装：\\n\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"lgMa4kbfK7J5\",\n    \"outputId\": \"5bb662d8-1665-4bcc-c517-70d1c4bc4837\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Requirement already satisfied: hanlp_restful in /usr/local/lib/python3.7/dist-packages (0.0.7)\\n\",\n      \"Requirement already satisfied: hanlp-common in /usr/local/lib/python3.7/dist-packages (from hanlp_restful) (0.0.9)\\n\",\n      \"Requirement already satisfied: phrasetree in /usr/local/lib/python3.7/dist-packages (from hanlp-common->hanlp_restful) (0.0.8)\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"!pip install hanlp_restful\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"N4G6GbNmK7J6\"\n   },\n   \"source\": [\n    \"创建客户端，填入服务器地址：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {\n    \"id\": \"3XM9-3-oK7J6\"\n   },\n   \"outputs\": [],\n   \"source\": [\n    \"from hanlp_restful import HanLPClient\\n\",\n    \"HanLP = HanLPClient('https://www.hanlp.com/api', auth=None, language='zh') # auth不填则匿名，zh中文，mul多语种\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pbeFH9jmK7J7\"\n   },\n   \"source\": [\n    \"调用`parse`接口，传入一篇文章，得到HanLP精准的分析结果。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"mNJPvZ_3K7J7\",\n    \"outputId\": \"4048d0d6-2dad-4582-e327-f99338f8f72b\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"{\\n\",\n      \"  \\\"tok/fine\\\": [\\n\",\n      \"    [\\\"2021年\\\", \\\"HanLPv2.1\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次\\\", \\\"世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多\\\", \\\"语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"],\\n\",\n      \"    [\\\"阿婆主\\\", \\\"来到\\\", \\\"北京\\\", \\\"立方庭\\\", \\\"参观\\\", \\\"自然\\\", \\\"语义\\\", \\\"科技\\\", \\\"公司\\\", \\\"。\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"tok/coarse\\\": [\\n\",\n      \"    [\\\"2021年\\\", \\\"HanLPv2.1\\\", \\\"为\\\", \\\"生产环境\\\", \\\"带来\\\", \\\"次世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"],\\n\",\n      \"    [\\\"阿婆主\\\", \\\"来到\\\", \\\"北京立方庭\\\", \\\"参观\\\", \\\"自然语义科技公司\\\", \\\"。\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"pos/ctb\\\": [\\n\",\n      \"    [\\\"NT\\\", \\\"NR\\\", \\\"P\\\", \\\"NN\\\", \\\"NN\\\", \\\"VV\\\", \\\"JJ\\\", \\\"NN\\\", \\\"AD\\\", \\\"JJ\\\", \\\"DEG\\\", \\\"CD\\\", \\\"NN\\\", \\\"NR\\\", \\\"NN\\\", \\\"PU\\\"],\\n\",\n      \"    [\\\"NN\\\", \\\"VV\\\", \\\"NR\\\", \\\"NR\\\", \\\"VV\\\", \\\"NN\\\", \\\"NN\\\", \\\"NN\\\", \\\"NN\\\", \\\"PU\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"pos/pku\\\": [\\n\",\n      \"    [\\\"t\\\", \\\"nx\\\", \\\"p\\\", \\\"vn\\\", \\\"n\\\", \\\"v\\\", \\\"b\\\", \\\"n\\\", \\\"d\\\", \\\"a\\\", \\\"u\\\", \\\"a\\\", \\\"n\\\", \\\"nx\\\", \\\"n\\\", \\\"w\\\"],\\n\",\n      \"    [\\\"n\\\", \\\"v\\\", \\\"ns\\\", \\\"ns\\\", \\\"v\\\", \\\"n\\\", \\\"n\\\", \\\"n\\\", \\\"n\\\", \\\"w\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"pos/863\\\": [\\n\",\n      \"    [\\\"nt\\\", \\\"w\\\", \\\"p\\\", \\\"v\\\", \\\"n\\\", \\\"v\\\", \\\"a\\\", \\\"nt\\\", \\\"d\\\", \\\"a\\\", \\\"u\\\", \\\"a\\\", \\\"n\\\", \\\"ws\\\", \\\"n\\\", \\\"w\\\"],\\n\",\n      \"    [\\\"n\\\", \\\"v\\\", \\\"ns\\\", \\\"n\\\", \\\"v\\\", \\\"n\\\", \\\"n\\\", \\\"n\\\", \\\"n\\\", \\\"w\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"ner/msra\\\": [\\n\",\n      \"    [[\\\"2021年\\\", \\\"DATE\\\", 0, 1], [\\\"HanLPv2.1\\\", \\\"ORGANIZATION\\\", 1, 2]],\\n\",\n      \"    [[\\\"北京立方庭\\\", \\\"LOCATION\\\", 2, 4], [\\\"自然语义科技公司\\\", \\\"ORGANIZATION\\\", 5, 9]]\\n\",\n      \"  ],\\n\",\n      \"  \\\"ner/pku\\\": [\\n\",\n      \"    [],\\n\",\n      \"    [[\\\"北京立方庭\\\", \\\"ns\\\", 2, 4], [\\\"自然语义科技公司\\\", \\\"nt\\\", 5, 9]]\\n\",\n      \"  ],\\n\",\n      \"  \\\"ner/ontonotes\\\": [\\n\",\n      \"    [[\\\"2021年\\\", \\\"DATE\\\", 0, 1], [\\\"HanLPv2.1\\\", \\\"ORG\\\", 1, 2]],\\n\",\n      \"    [[\\\"北京立方庭\\\", \\\"FAC\\\", 2, 4], [\\\"自然语义科技公司\\\", \\\"ORG\\\", 5, 9]]\\n\",\n      \"  ],\\n\",\n      \"  \\\"srl\\\": [\\n\",\n      \"    [[[\\\"2021年\\\", \\\"ARGM-TMP\\\", 0, 1], [\\\"HanLPv2.1\\\", \\\"ARG0\\\", 1, 2], [\\\"为生产环境\\\", \\\"ARG2\\\", 2, 5], [\\\"带来\\\", \\\"PRED\\\", 5, 6], [\\\"次世代最先进的多语种NLP技术\\\", \\\"ARG1\\\", 6, 15]], [[\\\"最\\\", \\\"ARGM-ADV\\\", 8, 9], [\\\"先进\\\", \\\"PRED\\\", 9, 10], [\\\"技术\\\", \\\"ARG0\\\", 14, 15]]],\\n\",\n      \"    [[[\\\"阿婆主\\\", \\\"ARG0\\\", 0, 1], [\\\"来到\\\", \\\"PRED\\\", 1, 2], [\\\"北京立方庭\\\", \\\"ARG1\\\", 2, 4]], [[\\\"阿婆主\\\", \\\"ARG0\\\", 0, 1], [\\\"参观\\\", \\\"PRED\\\", 4, 5], [\\\"自然语义科技公司\\\", \\\"ARG1\\\", 5, 9]]]\\n\",\n      \"  ],\\n\",\n      \"  \\\"dep\\\": [\\n\",\n      \"    [[6, \\\"tmod\\\"], [6, \\\"nsubj\\\"], [6, \\\"prep\\\"], [5, \\\"nn\\\"], [3, \\\"pobj\\\"], [0, \\\"root\\\"], [8, \\\"amod\\\"], [15, \\\"nn\\\"], [10, \\\"advmod\\\"], [15, \\\"rcmod\\\"], [10, \\\"assm\\\"], [13, \\\"nummod\\\"], [15, \\\"nn\\\"], [15, \\\"nn\\\"], [6, \\\"dobj\\\"], [6, \\\"punct\\\"]],\\n\",\n      \"    [[2, \\\"nsubj\\\"], [0, \\\"root\\\"], [4, \\\"nn\\\"], [2, \\\"dobj\\\"], [2, \\\"conj\\\"], [9, \\\"nn\\\"], [9, \\\"nn\\\"], [9, \\\"nn\\\"], [5, \\\"dobj\\\"], [2, \\\"punct\\\"]]\\n\",\n      \"  ],\\n\",\n      \"  \\\"sdp\\\": [\\n\",\n      \"    [[[6, \\\"Time\\\"]], [[6, \\\"Exp\\\"]], [[5, \\\"mPrep\\\"]], [[5, \\\"Desc\\\"]], [[6, \\\"Datv\\\"]], [[13, \\\"dDesc\\\"]], [[0, \\\"Root\\\"], [8, \\\"Desc\\\"], [13, \\\"Desc\\\"]], [[15, \\\"Time\\\"]], [[10, \\\"mDegr\\\"]], [[15, \\\"Desc\\\"]], [[10, \\\"mAux\\\"]], [[8, \\\"Quan\\\"], [13, \\\"Quan\\\"]], [[15, \\\"Desc\\\"]], [[15, \\\"Nmod\\\"]], [[6, \\\"Pat\\\"]], [[6, \\\"mPunc\\\"]]],\\n\",\n      \"    [[[2, \\\"Agt\\\"], [5, \\\"Agt\\\"]], [[0, \\\"Root\\\"]], [[4, \\\"Loc\\\"]], [[2, \\\"Lfin\\\"]], [[2, \\\"ePurp\\\"]], [[8, \\\"Nmod\\\"]], [[9, \\\"Nmod\\\"]], [[9, \\\"Nmod\\\"]], [[5, \\\"Datv\\\"]], [[5, \\\"mPunc\\\"]]]\\n\",\n      \"  ],\\n\",\n      \"  \\\"con\\\": [\\n\",\n      \"    [\\\"TOP\\\", [[\\\"IP\\\", [[\\\"NP\\\", [[\\\"NT\\\", [\\\"2021年\\\"]]]], [\\\"NP\\\", [[\\\"NR\\\", [\\\"HanLPv2.1\\\"]]]], [\\\"VP\\\", [[\\\"PP\\\", [[\\\"P\\\", [\\\"为\\\"]], [\\\"NP\\\", [[\\\"NN\\\", [\\\"生产\\\"]], [\\\"NN\\\", [\\\"环境\\\"]]]]]], [\\\"VP\\\", [[\\\"VV\\\", [\\\"带来\\\"]], [\\\"NP\\\", [[\\\"ADJP\\\", [[\\\"NP\\\", [[\\\"ADJP\\\", [[\\\"JJ\\\", [\\\"次\\\"]]]], [\\\"NP\\\", [[\\\"NN\\\", [\\\"世代\\\"]]]]]], [\\\"ADVP\\\", [[\\\"AD\\\", [\\\"最\\\"]]]], [\\\"VP\\\", [[\\\"JJ\\\", [\\\"先进\\\"]]]]]], [\\\"DEG\\\", [\\\"的\\\"]], [\\\"NP\\\", [[\\\"QP\\\", [[\\\"CD\\\", [\\\"多\\\"]]]], [\\\"NP\\\", [[\\\"NN\\\", [\\\"语种\\\"]]]]]], [\\\"NP\\\", [[\\\"NR\\\", [\\\"NLP\\\"]], [\\\"NN\\\", [\\\"技术\\\"]]]]]]]]]], [\\\"PU\\\", [\\\"。\\\"]]]]]],\\n\",\n      \"    [\\\"TOP\\\", [[\\\"IP\\\", [[\\\"NP\\\", [[\\\"NN\\\", [\\\"阿婆主\\\"]]]], [\\\"VP\\\", [[\\\"VP\\\", [[\\\"VV\\\", [\\\"来到\\\"]], [\\\"NP\\\", [[\\\"NR\\\", [\\\"北京\\\"]], [\\\"NR\\\", [\\\"立方庭\\\"]]]]]], [\\\"VP\\\", [[\\\"VV\\\", [\\\"参观\\\"]], [\\\"NP\\\", [[\\\"NN\\\", [\\\"自然\\\"]], [\\\"NN\\\", [\\\"语义\\\"]], [\\\"NN\\\", [\\\"科技\\\"]], [\\\"NN\\\", [\\\"公司\\\"]]]]]]]], [\\\"PU\\\", [\\\"。\\\"]]]]]]\\n\",\n      \"  ]\\n\",\n      \"}\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"doc = HanLP.parse(\\\"2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。阿婆主来到北京立方庭参观自然语义科技公司。\\\")\\n\",\n    \"print(doc)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"w4E8Kn_nK7J8\"\n   },\n   \"source\": [\n    \"#### 可视化\\n\",\n    \"输出结果是一个可以`json`化的`dict`，键为[NLP任务名](https://hanlp.hankcs.com/docs/data_format.html#naming-convention)，值为分析结果。关于标注集含义，请参考[《语言学标注规范》](https://hanlp.hankcs.com/docs/annotations/index.html)及[《格式规范》](https://hanlp.hankcs.com/docs/data_format.html)。我们购买、标注或采用了世界上量级最大、种类最多的语料库用于联合多语种多任务学习，所以HanLP的标注集也是覆盖面最广的。通过`doc.pretty_print`，可以在等宽字体环境中得到可视化，你需要取消换行才能对齐可视化结果。我们已经发布HTML环境的可视化，在Jupyter Notebook中自动对齐中文。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 575\n    },\n    \"id\": \"GZ79la4LK7J8\",\n    \"outputId\": \"b9bd5dc0-52f9-4b42-93fd-7c4e49214ace\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Dep&nbsp;Tree&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────&nbsp;<br>&nbsp;┌─────────►&nbsp;<br>&nbsp;│┌────────►&nbsp;<br>&nbsp;││┌─►┌─────&nbsp;<br>&nbsp;│││&nbsp;&nbsp;│&nbsp;&nbsp;┌─►&nbsp;<br>&nbsp;│││&nbsp;&nbsp;└─►└──&nbsp;<br>┌┼┴┴────────&nbsp;<br>││&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>││&nbsp;&nbsp;┌───►└──&nbsp;<br>││&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>││&nbsp;&nbsp;│┌──►├──&nbsp;<br>││&nbsp;&nbsp;││&nbsp;&nbsp;&nbsp;└─►&nbsp;<br>││&nbsp;&nbsp;││&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>││&nbsp;&nbsp;││┌─►└──&nbsp;<br>││&nbsp;&nbsp;│││&nbsp;&nbsp;┌─►&nbsp;<br>│└─►└┴┴──┴──&nbsp;<br>└──────────►&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Token&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>为&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>生产&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>环境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>带来&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先进&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>的&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>语种&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技术&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Relati&nbsp;<br>──────&nbsp;<br>tmod&nbsp;&nbsp;&nbsp;<br>nsubj&nbsp;&nbsp;<br>prep&nbsp;&nbsp;&nbsp;<br>nn&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>pobj&nbsp;&nbsp;&nbsp;<br>root&nbsp;&nbsp;&nbsp;<br>amod&nbsp;&nbsp;&nbsp;<br>nn&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>advmod&nbsp;<br>rcmod&nbsp;&nbsp;<br>assm&nbsp;&nbsp;&nbsp;<br>nummod&nbsp;<br>nn&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>nn&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>dobj&nbsp;&nbsp;&nbsp;<br>punct&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">PoS&nbsp;<br>───&nbsp;<br>NT&nbsp;&nbsp;<br>NR&nbsp;&nbsp;<br>P&nbsp;&nbsp;&nbsp;<br>NN&nbsp;&nbsp;<br>NN&nbsp;&nbsp;<br>VV&nbsp;&nbsp;<br>JJ&nbsp;&nbsp;<br>NN&nbsp;&nbsp;<br>AD&nbsp;&nbsp;<br>JJ&nbsp;&nbsp;<br>DEG&nbsp;<br>CD&nbsp;&nbsp;<br>NN&nbsp;&nbsp;<br>NR&nbsp;&nbsp;<br>NN&nbsp;&nbsp;<br>PU&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>为&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>生产&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>环境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>带来&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先进&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>的&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>语种&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技术&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">NER&nbsp;Type&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────────&nbsp;<br>───►DATE&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>───►ORGANIZATION&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>为&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>生产&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>环境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>带来&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先进&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>的&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>语种&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技术&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">SRL&nbsp;PA1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────&nbsp;<br>───►ARGM-TMP&nbsp;<br>───►ARG0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;├►ARG2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>╟──►PRED&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;├►ARG1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>为&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>生产&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>环境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>带来&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先进&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>的&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>语种&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技术&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">SRL&nbsp;PA2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>───►ARGM-ADV&nbsp;<br>╟──►PRED&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>───►ARG0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>为&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>生产&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>环境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>带来&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先进&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>的&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>语种&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技术&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">PoS&nbsp;&nbsp;&nbsp;&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;9&nbsp;<br>─────────────────────────────────────────────────────────<br>NT&nbsp;───────────────────────────────────────────►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;<br>NR&nbsp;───────────────────────────────────────────►NP────┤&nbsp;&nbsp;&nbsp;<br>P&nbsp;───────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NN&nbsp;──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├────────────────────────►PP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NN&nbsp;──┴►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>VV&nbsp;──────────────────────────────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>JJ&nbsp;───►ADJP──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP────┤&nbsp;&nbsp;&nbsp;<br>NN&nbsp;───►NP&nbsp;───┴►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>AD&nbsp;───────────►ADVP──┼►ADJP──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►IP<br>JJ&nbsp;───────────►VP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>DEG──────────────────────────┤&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>CD&nbsp;───►QP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NN&nbsp;───►NP&nbsp;───┴────────►NP────┤&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NR&nbsp;──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NN&nbsp;──┴────────────────►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>PU&nbsp;──────────────────────────────────────────────────┘&nbsp;&nbsp;&nbsp;</pre></div><br><div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Dep&nbsp;Tree&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>┌┬────┬──┴──&nbsp;<br>││&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;┌─►&nbsp;<br>││&nbsp;&nbsp;&nbsp;&nbsp;└─►└──&nbsp;<br>│└─►┌───────&nbsp;<br>│&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;┌───►&nbsp;<br>│&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;│┌──►&nbsp;<br>│&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;││┌─►&nbsp;<br>│&nbsp;&nbsp;&nbsp;└─►└┴┴──&nbsp;<br>└──────────►&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;<br>───&nbsp;<br>阿婆主&nbsp;<br>来到&nbsp;&nbsp;<br>北京&nbsp;&nbsp;<br>立方庭&nbsp;<br>参观&nbsp;&nbsp;<br>自然&nbsp;&nbsp;<br>语义&nbsp;&nbsp;<br>科技&nbsp;&nbsp;<br>公司&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Relat&nbsp;<br>─────&nbsp;<br>nsubj&nbsp;<br>root&nbsp;&nbsp;<br>nn&nbsp;&nbsp;&nbsp;&nbsp;<br>dobj&nbsp;&nbsp;<br>conj&nbsp;&nbsp;<br>nn&nbsp;&nbsp;&nbsp;&nbsp;<br>nn&nbsp;&nbsp;&nbsp;&nbsp;<br>nn&nbsp;&nbsp;&nbsp;&nbsp;<br>dobj&nbsp;&nbsp;<br>punct&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Po&nbsp;<br>──&nbsp;<br>NN&nbsp;<br>VV&nbsp;<br>NR&nbsp;<br>NR&nbsp;<br>VV&nbsp;<br>NN&nbsp;<br>NN&nbsp;<br>NN&nbsp;<br>NN&nbsp;<br>PU&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;<br>───&nbsp;<br>阿婆主&nbsp;<br>来到&nbsp;&nbsp;<br>北京&nbsp;&nbsp;<br>立方庭&nbsp;<br>参观&nbsp;&nbsp;<br>自然&nbsp;&nbsp;<br>语义&nbsp;&nbsp;<br>科技&nbsp;&nbsp;<br>公司&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">NER&nbsp;Type&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────────&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┴►LOCATION&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;├►ORGANIZATION&nbsp;<br>◄─┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;<br>───&nbsp;<br>阿婆主&nbsp;<br>来到&nbsp;&nbsp;<br>北京&nbsp;&nbsp;<br>立方庭&nbsp;<br>参观&nbsp;&nbsp;<br>自然&nbsp;&nbsp;<br>语义&nbsp;&nbsp;<br>科技&nbsp;&nbsp;<br>公司&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">SRL&nbsp;PA1&nbsp;&nbsp;<br>────────&nbsp;<br>───►ARG0&nbsp;<br>╟──►PRED&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┴►ARG1&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;<br>───&nbsp;<br>阿婆主&nbsp;<br>来到&nbsp;&nbsp;<br>北京&nbsp;&nbsp;<br>立方庭&nbsp;<br>参观&nbsp;&nbsp;<br>自然&nbsp;&nbsp;<br>语义&nbsp;&nbsp;<br>科技&nbsp;&nbsp;<br>公司&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">SRL&nbsp;PA2&nbsp;&nbsp;<br>────────&nbsp;<br>───►ARG0&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>╟──►PRED&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;├►ARG1&nbsp;<br>◄─┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;<br>───&nbsp;<br>阿婆主&nbsp;<br>来到&nbsp;&nbsp;<br>北京&nbsp;&nbsp;<br>立方庭&nbsp;<br>参观&nbsp;&nbsp;<br>自然&nbsp;&nbsp;<br>语义&nbsp;&nbsp;<br>科技&nbsp;&nbsp;<br>公司&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Po&nbsp;&nbsp;&nbsp;&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6&nbsp;<br>────────────────────────────────<br>NN───────────────────►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;<br>VV──────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NR──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NR──┴►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>VV──────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP────┤&nbsp;&nbsp;&nbsp;<br>NN──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►IP<br>NN&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NN&nbsp;&nbsp;├►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NN──┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>PU──────────────────────────┘&nbsp;&nbsp;&nbsp;</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {\n      \"tags\": []\n     },\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"doc.pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WIKyCLQJK7J9\"\n   },\n   \"source\": [\n    \"#### 申请秘钥\\n\",\n    \"由于服务器算力有限，匿名用户每分钟限2次调用。如果你需要更多调用次数，[建议申请免费公益API秘钥auth](https://bbs.hanlp.com/t/hanlp2-1-restful-api/53)。\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"PcZAZopQK7J9\"\n   },\n   \"source\": [\n    \"### 海量级native API\\n\",\n    \"\\n\",\n    \"依赖PyTorch、TensorFlow等深度学习技术，适合**专业**NLP工程师、研究者以及本地海量数据场景。要求Python 3.6以上，支持Windows，推荐*nix。可以在CPU上运行，推荐GPU/TPU。\\n\",\n    \"\\n\",\n    \"无论是Windows、Linux还是macOS，HanLP的安装只需一句话搞定。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"bjRdHxl1K7J-\",\n    \"outputId\": \"659d7920-c857-4eb8-f45f-dba84366688a\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"Requirement already satisfied: hanlp in /usr/local/lib/python3.7/dist-packages (2.1.0a54)\\n\",\n      \"Requirement already satisfied: sentencepiece>=0.1.91torch>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from hanlp) (0.1.96)\\n\",\n      \"Requirement already satisfied: toposort==1.5 in /usr/local/lib/python3.7/dist-packages (from hanlp) (1.5)\\n\",\n      \"Requirement already satisfied: alnlp in /usr/local/lib/python3.7/dist-packages (from hanlp) (1.0.0rc27)\\n\",\n      \"Requirement already satisfied: hanlp-common>=0.0.9 in /usr/local/lib/python3.7/dist-packages (from hanlp) (0.0.9)\\n\",\n      \"Requirement already satisfied: hanlp-downloader in /usr/local/lib/python3.7/dist-packages (from hanlp) (0.0.23)\\n\",\n      \"Requirement already satisfied: hanlp-trie>=0.0.2 in /usr/local/lib/python3.7/dist-packages (from hanlp) (0.0.2)\\n\",\n      \"Requirement already satisfied: transformers>=4.1.1 in /usr/local/lib/python3.7/dist-packages (from hanlp) (4.9.1)\\n\",\n      \"Requirement already satisfied: termcolor in /usr/local/lib/python3.7/dist-packages (from hanlp) (1.1.0)\\n\",\n      \"Requirement already satisfied: pynvml in /usr/local/lib/python3.7/dist-packages (from hanlp) (11.0.0)\\n\",\n      \"Requirement already satisfied: phrasetree in /usr/local/lib/python3.7/dist-packages (from hanlp-common>=0.0.9->hanlp) (0.0.8)\\n\",\n      \"Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers>=4.1.1->hanlp) (3.0.12)\\n\",\n      \"Requirement already satisfied: sacremoses in /usr/local/lib/python3.7/dist-packages (from transformers>=4.1.1->hanlp) (0.0.45)\\n\",\n      \"Requirement already satisfied: tokenizers<0.11,>=0.10.1 in /usr/local/lib/python3.7/dist-packages (from transformers>=4.1.1->hanlp) (0.10.3)\\n\",\n      \"Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from transformers>=4.1.1->hanlp) (21.0)\\n\",\n      \"Requirement already satisfied: huggingface-hub==0.0.12 in /usr/local/lib/python3.7/dist-packages (from transformers>=4.1.1->hanlp) (0.0.12)\\n\",\n      \"Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.7/dist-packages (from transformers>=4.1.1->hanlp) (5.4.1)\\n\",\n      \"Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers>=4.1.1->hanlp) (2019.12.20)\\n\",\n      \"Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.7/dist-packages (from transformers>=4.1.1->hanlp) (4.41.1)\\n\",\n      \"Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from transformers>=4.1.1->hanlp) (2.23.0)\\n\",\n      \"Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from transformers>=4.1.1->hanlp) (4.6.1)\\n\",\n      \"Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.7/dist-packages (from transformers>=4.1.1->hanlp) (1.19.5)\\n\",\n      \"Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from huggingface-hub==0.0.12->transformers>=4.1.1->hanlp) (3.7.4.3)\\n\",\n      \"Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->transformers>=4.1.1->hanlp) (2.4.7)\\n\",\n      \"Requirement already satisfied: torch in /usr/local/lib/python3.7/dist-packages (from alnlp->hanlp) (1.9.0+cu102)\\n\",\n      \"Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata->transformers>=4.1.1->hanlp) (3.5.0)\\n\",\n      \"Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->transformers>=4.1.1->hanlp) (1.24.3)\\n\",\n      \"Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->transformers>=4.1.1->hanlp) (3.0.4)\\n\",\n      \"Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->transformers>=4.1.1->hanlp) (2.10)\\n\",\n      \"Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->transformers>=4.1.1->hanlp) (2021.5.30)\\n\",\n      \"Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers>=4.1.1->hanlp) (1.0.1)\\n\",\n      \"Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers>=4.1.1->hanlp) (7.1.2)\\n\",\n      \"Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers>=4.1.1->hanlp) (1.15.0)\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"!pip install hanlp -U\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"dHhIRwgqK7J-\"\n   },\n   \"source\": [\n    \"#### 加载模型\\n\",\n    \"HanLP的工作流程是先加载模型，模型的标示符存储在`hanlp.pretrained`这个包中，按照NLP任务归类。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"KHY6bsG_K7J-\",\n    \"outputId\": \"208c12b6-2702-4ee7-a03a-f053b7ad3479\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"{'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_base_20210111_124519.zip',\\n\",\n       \" 'CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/mtl/close_tok_pos_ner_srl_dep_sdp_con_electra_small_20210111_124159.zip',\\n\",\n       \" 'NPCMJ_UD_KYOTO_TOK_POS_CON_BERT_BASE_CHAR_JA': 'https://file.hankcs.com/hanlp/mtl/npcmj_ud_kyoto_tok_pos_ner_dep_con_srl_bert_base_char_ja_20210517_225654.zip',\\n\",\n       \" 'OPEN_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH': 'https://file.hankcs.com/hanlp/mtl/open_tok_pos_ner_srl_dep_sdp_con_electra_base_20201223_201906.zip',\\n\",\n       \" 'OPEN_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH': 'https://file.hankcs.com/hanlp/mtl/open_tok_pos_ner_srl_dep_sdp_con_electra_small_20201223_035557.zip',\\n\",\n       \" 'UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_MT5_SMALL': 'https://file.hankcs.com/hanlp/mtl/ud_ontonotes_tok_pos_lem_fea_ner_srl_dep_sdp_con_mt5_small_20210228_123458.zip',\\n\",\n       \" 'UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_XLMR_BASE': 'https://file.hankcs.com/hanlp/mtl/ud_ontonotes_tok_pos_lem_fea_ner_srl_dep_sdp_con_xlm_base_20210602_211620.zip'}\"\n      ]\n     },\n     \"execution_count\": 6,\n     \"metadata\": {\n      \"tags\": []\n     },\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"import hanlp\\n\",\n    \"hanlp.pretrained.mtl.ALL # MTL多任务，具体任务见模型名称，语种见名称最后一个字段或相应语料库\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"WDT3Hks0K7J_\"\n   },\n   \"source\": [\n    \"调用`hanlp.load`进行加载，模型会自动下载到本地缓存。自然语言处理分为许多任务，分词只是最初级的一个。与其每个任务单独创建一个模型，不如利用HanLP的联合模型一次性完成多个任务：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"4Cj8a73rK7J_\",\n    \"outputId\": \"a92ac736-6e61-4949-8d35-56c773faf950\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stderr\",\n     \"output_type\": \"stream\",\n     \"text\": []\n    }\n   ],\n   \"source\": [\n    \"HanLP = hanlp.load(hanlp.pretrained.mtl.CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_BASE_ZH)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pBqH_My8K7J_\"\n   },\n   \"source\": [\n    \"## 多任务批量分析\\n\",\n    \"客户端创建完毕，或者模型加载完毕后，就可以传入一个或多个句子进行分析了：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 8,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"B58npfkHK7J_\",\n    \"outputId\": \"69fed02d-39cb-4b4c-d2c8-d0edc25970ea\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"{\\n\",\n      \"  \\\"tok/fine\\\": [\\n\",\n      \"    [\\\"2021年\\\", \\\"HanLPv2.1\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次\\\", \\\"世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多\\\", \\\"语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"],\\n\",\n      \"    [\\\"阿婆主\\\", \\\"来到\\\", \\\"北京\\\", \\\"立方庭\\\", \\\"参观\\\", \\\"自然\\\", \\\"语义\\\", \\\"科技\\\", \\\"公司\\\", \\\"。\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"tok/coarse\\\": [\\n\",\n      \"    [\\\"2021年\\\", \\\"HanLPv2.1\\\", \\\"为\\\", \\\"生产\\\", \\\"环境\\\", \\\"带来\\\", \\\"次世代\\\", \\\"最\\\", \\\"先进\\\", \\\"的\\\", \\\"多语种\\\", \\\"NLP\\\", \\\"技术\\\", \\\"。\\\"],\\n\",\n      \"    [\\\"阿婆主\\\", \\\"来到\\\", \\\"北京立方庭\\\", \\\"参观\\\", \\\"自然语义科技公司\\\", \\\"。\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"pos/ctb\\\": [\\n\",\n      \"    [\\\"NT\\\", \\\"NR\\\", \\\"P\\\", \\\"NN\\\", \\\"NN\\\", \\\"VV\\\", \\\"JJ\\\", \\\"NN\\\", \\\"AD\\\", \\\"JJ\\\", \\\"DEG\\\", \\\"CD\\\", \\\"NN\\\", \\\"NR\\\", \\\"NN\\\", \\\"PU\\\"],\\n\",\n      \"    [\\\"NN\\\", \\\"VV\\\", \\\"NR\\\", \\\"NR\\\", \\\"VV\\\", \\\"NN\\\", \\\"NN\\\", \\\"NN\\\", \\\"NN\\\", \\\"PU\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"pos/pku\\\": [\\n\",\n      \"    [\\\"t\\\", \\\"nx\\\", \\\"p\\\", \\\"vn\\\", \\\"n\\\", \\\"v\\\", \\\"b\\\", \\\"n\\\", \\\"d\\\", \\\"a\\\", \\\"u\\\", \\\"a\\\", \\\"n\\\", \\\"nx\\\", \\\"n\\\", \\\"w\\\"],\\n\",\n      \"    [\\\"n\\\", \\\"v\\\", \\\"ns\\\", \\\"ns\\\", \\\"v\\\", \\\"n\\\", \\\"n\\\", \\\"n\\\", \\\"n\\\", \\\"w\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"pos/863\\\": [\\n\",\n      \"    [\\\"nt\\\", \\\"w\\\", \\\"p\\\", \\\"v\\\", \\\"n\\\", \\\"v\\\", \\\"a\\\", \\\"nt\\\", \\\"d\\\", \\\"a\\\", \\\"u\\\", \\\"a\\\", \\\"n\\\", \\\"ws\\\", \\\"n\\\", \\\"w\\\"],\\n\",\n      \"    [\\\"n\\\", \\\"v\\\", \\\"ns\\\", \\\"n\\\", \\\"v\\\", \\\"n\\\", \\\"n\\\", \\\"n\\\", \\\"n\\\", \\\"w\\\"]\\n\",\n      \"  ],\\n\",\n      \"  \\\"ner/msra\\\": [\\n\",\n      \"    [[\\\"2021年\\\", \\\"DATE\\\", 0, 1], [\\\"HanLPv2.1\\\", \\\"WWW\\\", 1, 2]],\\n\",\n      \"    [[\\\"北京\\\", \\\"LOCATION\\\", 2, 3], [\\\"立方庭\\\", \\\"LOCATION\\\", 3, 4], [\\\"自然语义科技公司\\\", \\\"ORGANIZATION\\\", 5, 9]]\\n\",\n      \"  ],\\n\",\n      \"  \\\"ner/pku\\\": [\\n\",\n      \"    [],\\n\",\n      \"    [[\\\"北京立方庭\\\", \\\"ns\\\", 2, 4], [\\\"自然语义科技公司\\\", \\\"nt\\\", 5, 9]]\\n\",\n      \"  ],\\n\",\n      \"  \\\"ner/ontonotes\\\": [\\n\",\n      \"    [[\\\"2021年\\\", \\\"DATE\\\", 0, 1], [\\\"HanLPv2.1\\\", \\\"ORG\\\", 1, 2]],\\n\",\n      \"    [[\\\"北京立方庭\\\", \\\"FAC\\\", 2, 4], [\\\"自然语义科技公司\\\", \\\"ORG\\\", 5, 9]]\\n\",\n      \"  ],\\n\",\n      \"  \\\"srl\\\": [\\n\",\n      \"    [[[\\\"2021年\\\", \\\"ARGM-TMP\\\", 0, 1], [\\\"HanLPv2.1\\\", \\\"ARG0\\\", 1, 2], [\\\"为生产环境\\\", \\\"ARG2\\\", 2, 5], [\\\"带来\\\", \\\"PRED\\\", 5, 6], [\\\"次世代最先进的多语种NLP技术\\\", \\\"ARG1\\\", 6, 15]], [[\\\"最\\\", \\\"ARGM-ADV\\\", 8, 9], [\\\"先进\\\", \\\"PRED\\\", 9, 10], [\\\"技术\\\", \\\"ARG0\\\", 14, 15]]],\\n\",\n      \"    [[[\\\"阿婆主\\\", \\\"ARG0\\\", 0, 1], [\\\"来到\\\", \\\"PRED\\\", 1, 2], [\\\"北京立方庭\\\", \\\"ARG1\\\", 2, 4]], [[\\\"阿婆主\\\", \\\"ARG0\\\", 0, 1], [\\\"参观\\\", \\\"PRED\\\", 4, 5], [\\\"自然语义科技公司\\\", \\\"ARG1\\\", 5, 9]]]\\n\",\n      \"  ],\\n\",\n      \"  \\\"dep\\\": [\\n\",\n      \"    [[6, \\\"tmod\\\"], [6, \\\"nsubj\\\"], [6, \\\"prep\\\"], [5, \\\"nn\\\"], [3, \\\"pobj\\\"], [0, \\\"root\\\"], [8, \\\"amod\\\"], [15, \\\"nn\\\"], [10, \\\"advmod\\\"], [15, \\\"rcmod\\\"], [10, \\\"assm\\\"], [13, \\\"nummod\\\"], [15, \\\"nn\\\"], [15, \\\"nn\\\"], [6, \\\"dobj\\\"], [6, \\\"punct\\\"]],\\n\",\n      \"    [[2, \\\"nsubj\\\"], [0, \\\"root\\\"], [4, \\\"nn\\\"], [2, \\\"dobj\\\"], [2, \\\"conj\\\"], [9, \\\"nn\\\"], [9, \\\"nn\\\"], [9, \\\"nn\\\"], [5, \\\"dobj\\\"], [2, \\\"punct\\\"]]\\n\",\n      \"  ],\\n\",\n      \"  \\\"sdp\\\": [\\n\",\n      \"    [[[6, \\\"Time\\\"]], [[6, \\\"Exp\\\"]], [[5, \\\"mPrep\\\"]], [[5, \\\"Desc\\\"]], [[6, \\\"Datv\\\"]], [[13, \\\"dDesc\\\"]], [[0, \\\"Root\\\"], [8, \\\"Desc\\\"], [13, \\\"Desc\\\"]], [[15, \\\"Time\\\"]], [[10, \\\"mDegr\\\"]], [[15, \\\"Desc\\\"]], [[10, \\\"mAux\\\"]], [[8, \\\"Quan\\\"], [13, \\\"Quan\\\"]], [[15, \\\"Desc\\\"]], [[15, \\\"Nmod\\\"]], [[6, \\\"Pat\\\"]], [[6, \\\"mPunc\\\"]]],\\n\",\n      \"    [[[2, \\\"Agt\\\"], [5, \\\"Agt\\\"]], [[0, \\\"Root\\\"]], [[4, \\\"Loc\\\"]], [[2, \\\"Lfin\\\"]], [[2, \\\"ePurp\\\"]], [[8, \\\"Nmod\\\"]], [[9, \\\"Nmod\\\"]], [[9, \\\"Nmod\\\"]], [[5, \\\"Datv\\\"]], [[5, \\\"mPunc\\\"]]]\\n\",\n      \"  ],\\n\",\n      \"  \\\"con\\\": [\\n\",\n      \"    [\\\"TOP\\\", [[\\\"IP\\\", [[\\\"NP\\\", [[\\\"NT\\\", [\\\"2021年\\\"]]]], [\\\"NP\\\", [[\\\"NR\\\", [\\\"HanLPv2.1\\\"]]]], [\\\"VP\\\", [[\\\"PP\\\", [[\\\"P\\\", [\\\"为\\\"]], [\\\"NP\\\", [[\\\"NN\\\", [\\\"生产\\\"]], [\\\"NN\\\", [\\\"环境\\\"]]]]]], [\\\"VP\\\", [[\\\"VV\\\", [\\\"带来\\\"]], [\\\"NP\\\", [[\\\"ADJP\\\", [[\\\"NP\\\", [[\\\"ADJP\\\", [[\\\"JJ\\\", [\\\"次\\\"]]]], [\\\"NP\\\", [[\\\"NN\\\", [\\\"世代\\\"]]]]]], [\\\"ADVP\\\", [[\\\"AD\\\", [\\\"最\\\"]]]], [\\\"VP\\\", [[\\\"JJ\\\", [\\\"先进\\\"]]]]]], [\\\"DEG\\\", [\\\"的\\\"]], [\\\"NP\\\", [[\\\"QP\\\", [[\\\"CD\\\", [\\\"多\\\"]]]], [\\\"NP\\\", [[\\\"NN\\\", [\\\"语种\\\"]]]]]], [\\\"NP\\\", [[\\\"NR\\\", [\\\"NLP\\\"]], [\\\"NN\\\", [\\\"技术\\\"]]]]]]]]]], [\\\"PU\\\", [\\\"。\\\"]]]]]],\\n\",\n      \"    [\\\"TOP\\\", [[\\\"IP\\\", [[\\\"NP\\\", [[\\\"NN\\\", [\\\"阿婆主\\\"]]]], [\\\"VP\\\", [[\\\"VP\\\", [[\\\"VV\\\", [\\\"来到\\\"]], [\\\"NP\\\", [[\\\"NR\\\", [\\\"北京\\\"]], [\\\"NR\\\", [\\\"立方庭\\\"]]]]]], [\\\"VP\\\", [[\\\"VV\\\", [\\\"参观\\\"]], [\\\"NP\\\", [[\\\"NN\\\", [\\\"自然\\\"]], [\\\"NN\\\", [\\\"语义\\\"]], [\\\"NN\\\", [\\\"科技\\\"]], [\\\"NN\\\", [\\\"公司\\\"]]]]]]]], [\\\"PU\\\", [\\\"。\\\"]]]]]]\\n\",\n      \"  ]\\n\",\n      \"}\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"doc = HanLP(['2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', '阿婆主来到北京立方庭参观自然语义科技公司。'])\\n\",\n    \"print(doc)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"tvuxfWPYK7J_\"\n   },\n   \"source\": [\n    \"## 可视化\\n\",\n    \"输出结果是一个可以`json`化的`dict`，键为[NLP任务名](https://hanlp.hankcs.com/docs/data_format.html#naming-convention)，值为分析结果。关于标注集含义，请参考[《语言学标注规范》](https://hanlp.hankcs.com/docs/annotations/index.html)及[《格式规范》](https://hanlp.hankcs.com/docs/data_format.html)。我们购买、标注或采用了世界上量级最大、种类最多的语料库用于联合多语种多任务学习，所以HanLP的标注集也是覆盖面最广的。通过`doc.pretty_print`，可以在等宽字体环境中得到可视化，你需要取消换行才能对齐可视化结果。我们已经发布HTML环境的可视化，在Jupyter Notebook中自动对齐中文。\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 9,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 575\n    },\n    \"id\": \"M8WxTdlAK7KA\",\n    \"outputId\": \"a027a302-74d8-48c9-b30d-45ebf8741c1e\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Dep&nbsp;Tree&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────&nbsp;<br>&nbsp;┌─────────►&nbsp;<br>&nbsp;│┌────────►&nbsp;<br>&nbsp;││┌─►┌─────&nbsp;<br>&nbsp;│││&nbsp;&nbsp;│&nbsp;&nbsp;┌─►&nbsp;<br>&nbsp;│││&nbsp;&nbsp;└─►└──&nbsp;<br>┌┼┴┴────────&nbsp;<br>││&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>││&nbsp;&nbsp;┌───►└──&nbsp;<br>││&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>││&nbsp;&nbsp;│┌──►├──&nbsp;<br>││&nbsp;&nbsp;││&nbsp;&nbsp;&nbsp;└─►&nbsp;<br>││&nbsp;&nbsp;││&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>││&nbsp;&nbsp;││┌─►└──&nbsp;<br>││&nbsp;&nbsp;│││&nbsp;&nbsp;┌─►&nbsp;<br>│└─►└┴┴──┴──&nbsp;<br>└──────────►&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Token&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>为&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>生产&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>环境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>带来&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先进&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>的&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>语种&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技术&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Relati&nbsp;<br>──────&nbsp;<br>tmod&nbsp;&nbsp;&nbsp;<br>nsubj&nbsp;&nbsp;<br>prep&nbsp;&nbsp;&nbsp;<br>nn&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>pobj&nbsp;&nbsp;&nbsp;<br>root&nbsp;&nbsp;&nbsp;<br>amod&nbsp;&nbsp;&nbsp;<br>nn&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>advmod&nbsp;<br>rcmod&nbsp;&nbsp;<br>assm&nbsp;&nbsp;&nbsp;<br>nummod&nbsp;<br>nn&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>nn&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>dobj&nbsp;&nbsp;&nbsp;<br>punct&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">PoS&nbsp;<br>───&nbsp;<br>NT&nbsp;&nbsp;<br>NR&nbsp;&nbsp;<br>P&nbsp;&nbsp;&nbsp;<br>NN&nbsp;&nbsp;<br>NN&nbsp;&nbsp;<br>VV&nbsp;&nbsp;<br>JJ&nbsp;&nbsp;<br>NN&nbsp;&nbsp;<br>AD&nbsp;&nbsp;<br>JJ&nbsp;&nbsp;<br>DEG&nbsp;<br>CD&nbsp;&nbsp;<br>NN&nbsp;&nbsp;<br>NR&nbsp;&nbsp;<br>NN&nbsp;&nbsp;<br>PU&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>为&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>生产&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>环境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>带来&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先进&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>的&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>语种&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技术&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">NER&nbsp;Type&nbsp;<br>────────&nbsp;<br>───►DATE&nbsp;<br>───►WWW&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>为&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>生产&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>环境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>带来&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先进&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>的&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>语种&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技术&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">SRL&nbsp;PA1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────&nbsp;<br>───►ARGM-TMP&nbsp;<br>───►ARG0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;├►ARG2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>╟──►PRED&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;├►ARG1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>为&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>生产&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>环境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>带来&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先进&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>的&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>语种&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技术&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">SRL&nbsp;PA2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>───►ARGM-ADV&nbsp;<br>╟──►PRED&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>───►ARG0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>为&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>生产&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>环境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>带来&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先进&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>的&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>语种&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技术&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">PoS&nbsp;&nbsp;&nbsp;&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;9&nbsp;<br>─────────────────────────────────────────────────────────<br>NT&nbsp;───────────────────────────────────────────►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;<br>NR&nbsp;───────────────────────────────────────────►NP────┤&nbsp;&nbsp;&nbsp;<br>P&nbsp;───────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NN&nbsp;──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├────────────────────────►PP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NN&nbsp;──┴►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>VV&nbsp;──────────────────────────────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>JJ&nbsp;───►ADJP──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP────┤&nbsp;&nbsp;&nbsp;<br>NN&nbsp;───►NP&nbsp;───┴►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>AD&nbsp;───────────►ADVP──┼►ADJP──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►IP<br>JJ&nbsp;───────────►VP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>DEG──────────────────────────┤&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>CD&nbsp;───►QP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NN&nbsp;───►NP&nbsp;───┴────────►NP────┤&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NR&nbsp;──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NN&nbsp;──┴────────────────►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>PU&nbsp;──────────────────────────────────────────────────┘&nbsp;&nbsp;&nbsp;</pre></div><br><div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Dep&nbsp;Tree&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>┌┬────┬──┴──&nbsp;<br>││&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;┌─►&nbsp;<br>││&nbsp;&nbsp;&nbsp;&nbsp;└─►└──&nbsp;<br>│└─►┌───────&nbsp;<br>│&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;┌───►&nbsp;<br>│&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;│┌──►&nbsp;<br>│&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;││┌─►&nbsp;<br>│&nbsp;&nbsp;&nbsp;└─►└┴┴──&nbsp;<br>└──────────►&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;<br>───&nbsp;<br>阿婆主&nbsp;<br>来到&nbsp;&nbsp;<br>北京&nbsp;&nbsp;<br>立方庭&nbsp;<br>参观&nbsp;&nbsp;<br>自然&nbsp;&nbsp;<br>语义&nbsp;&nbsp;<br>科技&nbsp;&nbsp;<br>公司&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Relat&nbsp;<br>─────&nbsp;<br>nsubj&nbsp;<br>root&nbsp;&nbsp;<br>nn&nbsp;&nbsp;&nbsp;&nbsp;<br>dobj&nbsp;&nbsp;<br>conj&nbsp;&nbsp;<br>nn&nbsp;&nbsp;&nbsp;&nbsp;<br>nn&nbsp;&nbsp;&nbsp;&nbsp;<br>nn&nbsp;&nbsp;&nbsp;&nbsp;<br>dobj&nbsp;&nbsp;<br>punct&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Po&nbsp;<br>──&nbsp;<br>NN&nbsp;<br>VV&nbsp;<br>NR&nbsp;<br>NR&nbsp;<br>VV&nbsp;<br>NN&nbsp;<br>NN&nbsp;<br>NN&nbsp;<br>NN&nbsp;<br>PU&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;<br>───&nbsp;<br>阿婆主&nbsp;<br>来到&nbsp;&nbsp;<br>北京&nbsp;&nbsp;<br>立方庭&nbsp;<br>参观&nbsp;&nbsp;<br>自然&nbsp;&nbsp;<br>语义&nbsp;&nbsp;<br>科技&nbsp;&nbsp;<br>公司&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">NER&nbsp;Type&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────────&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>───►LOCATION&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>───►LOCATION&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;├►ORGANIZATION&nbsp;<br>◄─┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;<br>───&nbsp;<br>阿婆主&nbsp;<br>来到&nbsp;&nbsp;<br>北京&nbsp;&nbsp;<br>立方庭&nbsp;<br>参观&nbsp;&nbsp;<br>自然&nbsp;&nbsp;<br>语义&nbsp;&nbsp;<br>科技&nbsp;&nbsp;<br>公司&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">SRL&nbsp;PA1&nbsp;&nbsp;<br>────────&nbsp;<br>───►ARG0&nbsp;<br>╟──►PRED&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┴►ARG1&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;<br>───&nbsp;<br>阿婆主&nbsp;<br>来到&nbsp;&nbsp;<br>北京&nbsp;&nbsp;<br>立方庭&nbsp;<br>参观&nbsp;&nbsp;<br>自然&nbsp;&nbsp;<br>语义&nbsp;&nbsp;<br>科技&nbsp;&nbsp;<br>公司&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">SRL&nbsp;PA2&nbsp;&nbsp;<br>────────&nbsp;<br>───►ARG0&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>╟──►PRED&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;├►ARG1&nbsp;<br>◄─┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;<br>───&nbsp;<br>阿婆主&nbsp;<br>来到&nbsp;&nbsp;<br>北京&nbsp;&nbsp;<br>立方庭&nbsp;<br>参观&nbsp;&nbsp;<br>自然&nbsp;&nbsp;<br>语义&nbsp;&nbsp;<br>科技&nbsp;&nbsp;<br>公司&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Po&nbsp;&nbsp;&nbsp;&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6&nbsp;<br>────────────────────────────────<br>NN───────────────────►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;<br>VV──────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NR──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NR──┴►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>VV──────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP────┤&nbsp;&nbsp;&nbsp;<br>NN──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►IP<br>NN&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NN&nbsp;&nbsp;├►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NN──┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>PU──────────────────────────┘&nbsp;&nbsp;&nbsp;</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {\n      \"tags\": []\n     },\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"doc.pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"_B2HDiZgK7KA\"\n   },\n   \"source\": [\n    \"## 指定任务\\n\",\n    \"简洁的接口也支持灵活的参数，任务越少，速度越快。如指定仅执行分词：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 10,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 35\n    },\n    \"id\": \"9Mnys4t2K7KA\",\n    \"outputId\": \"88d72a72-c095-4f6d-df0b-d881887087ce\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">阿婆主&nbsp;来到&nbsp;北京&nbsp;立方庭&nbsp;参观&nbsp;自然&nbsp;语义&nbsp;科技&nbsp;公司&nbsp;。</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {\n      \"tags\": []\n     },\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"HanLP('阿婆主来到北京立方庭参观自然语义科技公司。', tasks='tok').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"s5RkVkVkK7KA\"\n   },\n   \"source\": [\n    \"### 执行粗颗粒度分词\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 11,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 35\n    },\n    \"id\": \"5R_PwELlK7KA\",\n    \"outputId\": \"5ce2c037-eb44-481f-9de2-dc0d4122e7c4\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">阿婆主&nbsp;来到&nbsp;北京立方庭&nbsp;参观&nbsp;自然语义科技公司&nbsp;。</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {\n      \"tags\": []\n     },\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"HanLP('阿婆主来到北京立方庭参观自然语义科技公司。', tasks='tok/coarse').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"pTrajkHEK7KB\"\n   },\n   \"source\": [\n    \"### 执行分词和PKU词性标注\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 12,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 35\n    },\n    \"id\": \"kkkgVKFqK7KB\",\n    \"outputId\": \"e9f9879b-47ce-459a-e089-923de1c6436c\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">阿婆主/n&nbsp;来到/v&nbsp;北京/ns&nbsp;立方庭/ns&nbsp;参观/v&nbsp;自然/n&nbsp;语义/n&nbsp;科技/n&nbsp;公司/n&nbsp;。/w</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {\n      \"tags\": []\n     },\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"HanLP('阿婆主来到北京立方庭参观自然语义科技公司。', tasks='pos/pku').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"YLLTVY0RK7KB\"\n   },\n   \"source\": [\n    \"### 执行粗颗粒度分词和PKU词性标注\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 13,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 35\n    },\n    \"id\": \"5qSlqbcfK7KB\",\n    \"outputId\": \"66944459-bc22-4bd9-e4af-4d2aba9316f3\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">阿婆主/n&nbsp;来到/v&nbsp;北京立方庭/ns&nbsp;参观/v&nbsp;自然语义科技公司/n&nbsp;。/w</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {\n      \"tags\": []\n     },\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"HanLP('阿婆主来到北京立方庭参观自然语义科技公司。', tasks=['tok/coarse', 'pos/pku'], skip_tasks='tok/fine').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"3nNojvHiK7KB\"\n   },\n   \"source\": [\n    \"### 执行分词和MSRA标准NER\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 14,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 225\n    },\n    \"id\": \"tTVoEPiAK7KB\",\n    \"outputId\": \"b8dc8c24-3392-4712-d1b6-e2dc8b7710e8\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;<br>───&nbsp;<br>阿婆主&nbsp;<br>来到&nbsp;&nbsp;<br>北京&nbsp;&nbsp;<br>立方庭&nbsp;<br>参观&nbsp;&nbsp;<br>自然&nbsp;&nbsp;<br>语义&nbsp;&nbsp;<br>科技&nbsp;&nbsp;<br>公司&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">NER&nbsp;Type&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────────<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>───►LOCATION&nbsp;&nbsp;&nbsp;&nbsp;<br>───►LOCATION&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;├►ORGANIZATION<br>◄─┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {\n      \"tags\": []\n     },\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"HanLP('阿婆主来到北京立方庭参观自然语义科技公司。', tasks='ner/msra').pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"uG2wYTfmK7KB\"\n   },\n   \"source\": [\n    \"### 执行分词、词性标注和依存句法分析\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 15,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 225\n    },\n    \"id\": \"WXl6f7zyK7KC\",\n    \"outputId\": \"8671e0e4-d0c3-40f4-a4db-ba9aaec225ab\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Dep&nbsp;Tree&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>┌┬────┬──┴──&nbsp;<br>││&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;┌─►&nbsp;<br>││&nbsp;&nbsp;&nbsp;&nbsp;└─►└──&nbsp;<br>│└─►┌───────&nbsp;<br>│&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;┌───►&nbsp;<br>│&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;│┌──►&nbsp;<br>│&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;││┌─►&nbsp;<br>│&nbsp;&nbsp;&nbsp;└─►└┴┴──&nbsp;<br>└──────────►&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;<br>───&nbsp;<br>阿婆主&nbsp;<br>来到&nbsp;&nbsp;<br>北京&nbsp;&nbsp;<br>立方庭&nbsp;<br>参观&nbsp;&nbsp;<br>自然&nbsp;&nbsp;<br>语义&nbsp;&nbsp;<br>科技&nbsp;&nbsp;<br>公司&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Relat&nbsp;<br>─────&nbsp;<br>nsubj&nbsp;<br>root&nbsp;&nbsp;<br>nn&nbsp;&nbsp;&nbsp;&nbsp;<br>dobj&nbsp;&nbsp;<br>conj&nbsp;&nbsp;<br>nn&nbsp;&nbsp;&nbsp;&nbsp;<br>nn&nbsp;&nbsp;&nbsp;&nbsp;<br>nn&nbsp;&nbsp;&nbsp;&nbsp;<br>dobj&nbsp;&nbsp;<br>punct&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Po<br>──<br>NN<br>VV<br>NR<br>NR<br>VV<br>NN<br>NN<br>NN<br>NN<br>PU</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {\n      \"tags\": []\n     },\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"doc = HanLP('阿婆主来到北京立方庭参观自然语义科技公司。', tasks=['pos', 'dep'])\\n\",\n    \"doc.pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"ocxM3LsGK7KC\"\n   },\n   \"source\": [\n    \"转换为CoNLL格式：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 16,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"NtKmSB_0K7KC\",\n    \"outputId\": \"cc9245b3-32c2-4d35-88a8-a7d91127eca7\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"1\\t阿婆主\\t_\\tNN\\t_\\t_\\t2\\tnsubj\\t_\\t_\\n\",\n      \"2\\t来到\\t_\\tVV\\t_\\t_\\t0\\troot\\t_\\t_\\n\",\n      \"3\\t北京\\t_\\tNR\\t_\\t_\\t4\\tnn\\t_\\t_\\n\",\n      \"4\\t立方庭\\t_\\tNR\\t_\\t_\\t2\\tdobj\\t_\\t_\\n\",\n      \"5\\t参观\\t_\\tVV\\t_\\t_\\t2\\tconj\\t_\\t_\\n\",\n      \"6\\t自然\\t_\\tNN\\t_\\t_\\t9\\tnn\\t_\\t_\\n\",\n      \"7\\t语义\\t_\\tNN\\t_\\t_\\t9\\tnn\\t_\\t_\\n\",\n      \"8\\t科技\\t_\\tNN\\t_\\t_\\t9\\tnn\\t_\\t_\\n\",\n      \"9\\t公司\\t_\\tNN\\t_\\t_\\t5\\tdobj\\t_\\t_\\n\",\n      \"10\\t。\\t_\\tPU\\t_\\t_\\t2\\tpunct\\t_\\t_\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(doc.to_conll())\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"PNBo-kETK7KC\"\n   },\n   \"source\": [\n    \"### 执行分词、词性标注和短语成分分析\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 17,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 225\n    },\n    \"id\": \"Ja8dib6XK7KC\",\n    \"outputId\": \"a972f5bb-ae23-47a9-cd9f-6070a5b39f50\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;<br>───&nbsp;<br>阿婆主&nbsp;<br>来到&nbsp;&nbsp;<br>北京&nbsp;&nbsp;<br>立方庭&nbsp;<br>参观&nbsp;&nbsp;<br>自然&nbsp;&nbsp;<br>语义&nbsp;&nbsp;<br>科技&nbsp;&nbsp;<br>公司&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Po&nbsp;&nbsp;&nbsp;&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6&nbsp;<br>────────────────────────────────<br>NN───────────────────►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;<br>VV──────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NR──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NR──┴►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>VV──────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP────┤&nbsp;&nbsp;&nbsp;<br>NN──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►IP<br>NN&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NN&nbsp;&nbsp;├►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NN──┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>PU──────────────────────────┘&nbsp;&nbsp;&nbsp;</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {\n      \"tags\": []\n     },\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"doc = HanLP('阿婆主来到北京立方庭参观自然语义科技公司。', tasks=['pos', 'con'])\\n\",\n    \"doc.pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"Mg3DhvjhK7KC\"\n   },\n   \"source\": [\n    \"#### 将短语结构树以bracketed形式打印\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 18,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"kE8iBZNUK7KC\",\n    \"outputId\": \"79e2a72d-e473-41ca-c054-9595a4dd5971\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"(TOP\\n\",\n      \"  (IP\\n\",\n      \"    (NP (NN 阿婆主))\\n\",\n      \"    (VP\\n\",\n      \"      (VP (VV 来到) (NP (NR 北京) (NR 立方庭)))\\n\",\n      \"      (VP (VV 参观) (NP (NN 自然) (NN 语义) (NN 科技) (NN 公司))))\\n\",\n      \"    (PU 。)))\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"print(doc['con'])  # str(doc['con'])会将短语结构列表转换为括号形式\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"MfleaY_pK7KC\"\n   },\n   \"source\": [\n    \"关于标注集含义，请参考[《语言学标注规范》](https://hanlp.hankcs.com/docs/annotations/index.html)及[《格式规范》](https://hanlp.hankcs.com/docs/data_format.html)。我们购买、标注或采用了世界上量级最大、种类最多的语料库用于联合多语种多任务学习，所以HanLP的标注集也是覆盖面最广的。\\n\",\n    \"\\n\",\n    \"## 多语种支持\\n\",\n    \"总之，可以通过tasks参数灵活调用各种NLP任务。除了中文联合模型之外，你可以在文档中通过找到许多其他语种的模型，比如日语：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 19,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\"\n    },\n    \"id\": \"oJP8dvfvK7KD\",\n    \"outputId\": \"2262ccdb-7cf5-4859-8d6c-18300e54c22e\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stderr\",\n     \"output_type\": \"stream\",\n     \"text\": []\n    }\n   ],\n   \"source\": [\n    \"ja = hanlp.load(hanlp.pretrained.mtl.NPCMJ_UD_KYOTO_TOK_POS_CON_BERT_BASE_CHAR_JA)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 20,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 991\n    },\n    \"id\": \"3WPvCbH2K7KD\",\n    \"outputId\": \"46a9435d-ed5b-47ef-99c6-71d7ee0fc6e8\"\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Dep&nbsp;Tree&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>──────────────&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>┌─────────►├──&nbsp;<br>│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;└─►&nbsp;<br>│&nbsp;&nbsp;&nbsp;┌────────►&nbsp;<br>│&nbsp;&nbsp;&nbsp;│┌───────►&nbsp;<br>│&nbsp;&nbsp;&nbsp;││&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>│&nbsp;&nbsp;&nbsp;││┌───►├──&nbsp;<br>│&nbsp;&nbsp;&nbsp;│││&nbsp;&nbsp;&nbsp;&nbsp;└─►&nbsp;<br>│&nbsp;&nbsp;&nbsp;│││┌─────►&nbsp;<br>│&nbsp;&nbsp;&nbsp;││││┌────►&nbsp;<br>│&nbsp;&nbsp;&nbsp;│││││┌───►&nbsp;<br>│&nbsp;&nbsp;&nbsp;││││││┌──►&nbsp;<br>│&nbsp;&nbsp;&nbsp;│││││││┌─►&nbsp;<br>│┌─►└┴┴┴┴┴┴┼──&nbsp;<br>││&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;└─►&nbsp;<br>││&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>││&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌─►├──&nbsp;<br>││&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;└─►&nbsp;<br>└┴──────┴┬┬┬──&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;││└─►&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│└──►&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;└───►&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Token&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>、&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>は&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>の&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先端&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>言語&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技術&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>を&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>本番&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>環境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>に&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>導入&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>し&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>ます&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Relation&nbsp;<br>────────&nbsp;<br>nummod&nbsp;&nbsp;&nbsp;<br>obl&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>punct&nbsp;&nbsp;&nbsp;&nbsp;<br>compound&nbsp;<br>case&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>compound&nbsp;<br>nmod&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>case&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>compound&nbsp;<br>compound&nbsp;<br>compound&nbsp;<br>compound&nbsp;<br>compound&nbsp;<br>obj&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>case&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>compound&nbsp;<br>obl&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>case&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>root&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>aux&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>aux&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>punct&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">PoS&nbsp;<br>───&nbsp;<br>NUM&nbsp;<br>CL&nbsp;&nbsp;<br>PU&nbsp;&nbsp;<br>NPR&nbsp;<br>P&nbsp;&nbsp;&nbsp;<br>N&nbsp;&nbsp;&nbsp;<br>N&nbsp;&nbsp;&nbsp;<br>P&nbsp;&nbsp;&nbsp;<br>N&nbsp;&nbsp;&nbsp;<br>N&nbsp;&nbsp;&nbsp;<br>NUM&nbsp;<br>N&nbsp;&nbsp;&nbsp;<br>N&nbsp;&nbsp;&nbsp;<br>N&nbsp;&nbsp;&nbsp;<br>P&nbsp;&nbsp;&nbsp;<br>N&nbsp;&nbsp;&nbsp;<br>N&nbsp;&nbsp;&nbsp;<br>P&nbsp;&nbsp;&nbsp;<br>VB&nbsp;&nbsp;<br>VB0&nbsp;<br>AX&nbsp;&nbsp;<br>PU&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>、&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>は&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>の&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先端&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>言語&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技術&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>を&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>本番&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>環境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>に&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>導入&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>し&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>ます&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">NER&nbsp;Type&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┴►DATE&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>───►ARTIFACT&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>、&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>は&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>の&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先端&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>言語&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技術&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>を&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>本番&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>環境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>に&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>導入&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>し&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>ます&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">SRL&nbsp;PA1&nbsp;&nbsp;<br>────────&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>───►修飾&nbsp;&nbsp;&nbsp;<br>╟──►PRED&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>、&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>は&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>の&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先端&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>言語&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技術&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>を&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>本番&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>環境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>に&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>導入&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>し&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>ます&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">SRL&nbsp;PA3&nbsp;&nbsp;<br>────────&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┴►修飾&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>╟──►PRED&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>、&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>は&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>の&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先端&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>言語&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技術&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>を&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>本番&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>環境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>に&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>導入&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>し&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>ます&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">SRL&nbsp;PA4&nbsp;&nbsp;<br>────────&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;├►修飾&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┴►ノ&nbsp;&nbsp;&nbsp;&nbsp;<br>╟──►PRED&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>、&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>は&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>の&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先端&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>言語&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技術&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>を&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>本番&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>環境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>に&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>導入&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>し&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>ます&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">SRL&nbsp;PA5&nbsp;&nbsp;<br>────────&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>───►修飾&nbsp;&nbsp;&nbsp;<br>╟──►PRED&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>、&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>は&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>の&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先端&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>言語&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技術&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>を&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>本番&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>環境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>に&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>導入&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>し&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>ます&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">SRL&nbsp;PA6&nbsp;&nbsp;<br>────────&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;├►時間&nbsp;&nbsp;&nbsp;<br>◄─┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┴►ガ&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;├►ヲ&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;├►ニ&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>╟──►PRED&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>、&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>は&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>の&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先端&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>言語&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技術&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>を&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>本番&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>環境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>に&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>導入&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>し&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>ます&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">PoS&nbsp;&nbsp;&nbsp;&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;8&nbsp;<br>────────────────────────────────────────────────────<br>NUM──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>CL&nbsp;──┴►NUMCLP────────&nbsp;───────────────────►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;<br>PU&nbsp;────────&nbsp;─────────&nbsp;──────────────────────────┤&nbsp;&nbsp;&nbsp;<br>NPR───►NP&nbsp;─────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>P&nbsp;─────────&nbsp;───┴►────&nbsp;───────────────────►PP────┤&nbsp;&nbsp;&nbsp;<br>N&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>N&nbsp;───┴►NP&nbsp;─────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>P&nbsp;─────────&nbsp;───┴►PP&nbsp;────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>N&nbsp;─────────&nbsp;─────────&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>N&nbsp;────►NP&nbsp;──────►CONJP──┤&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NUM────────&nbsp;─────────&nbsp;&nbsp;&nbsp;├►NML&nbsp;──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>N&nbsp;─────────&nbsp;─────────&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►IP<br>N&nbsp;─────────&nbsp;─────────&nbsp;──┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>N&nbsp;─────────&nbsp;─────────&nbsp;──────────┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►PP────┤&nbsp;&nbsp;&nbsp;<br>P&nbsp;─────────&nbsp;─────────&nbsp;──────────────────┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>N&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>N&nbsp;───┴►NP&nbsp;─────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>P&nbsp;─────────&nbsp;───┴►────&nbsp;───────────────────►PP────┤&nbsp;&nbsp;&nbsp;<br>VB&nbsp;────────&nbsp;─────────&nbsp;──────────────────────────┤&nbsp;&nbsp;&nbsp;<br>VB0────────&nbsp;─────────&nbsp;──────────────────────────┤&nbsp;&nbsp;&nbsp;<br>AX&nbsp;────────&nbsp;─────────&nbsp;──────────────────────────┤&nbsp;&nbsp;&nbsp;<br>PU&nbsp;────────&nbsp;─────────&nbsp;──────────────────────────┘&nbsp;&nbsp;&nbsp;</pre></div><br><div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Dep&nbsp;Tree&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>──────────────&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>┌─────────►├──&nbsp;<br>│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;└─►&nbsp;<br>│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌─────►&nbsp;<br>│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│┌────►&nbsp;<br>│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;││┌───►&nbsp;<br>│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│││┌──►&nbsp;<br>│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;││││┌─►&nbsp;<br>│&nbsp;&nbsp;&nbsp;┌─►└┴┴┴┼──&nbsp;<br>│&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;└─►&nbsp;<br>│&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>│&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;┌─►└──&nbsp;<br>│&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;┌─►&nbsp;<br>│&nbsp;&nbsp;&nbsp;│┌─►└──┼──&nbsp;<br>│&nbsp;&nbsp;&nbsp;││&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;└─►&nbsp;<br>│┌─►└┴─────┬──&nbsp;<br>││&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;└─►&nbsp;<br>││&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌──►&nbsp;<br>││&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│┌─►&nbsp;<br>││&nbsp;&nbsp;&nbsp;┌─►┌┬┼┼──&nbsp;<br>││&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;│││└─►&nbsp;<br>││&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;││└──►&nbsp;<br>││&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;│└───►&nbsp;<br>││&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;└────►&nbsp;<br>││&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>└┴───┴────┬┼──&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│└─►&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;└──►&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Toke&nbsp;<br>────&nbsp;<br>奈須&nbsp;&nbsp;&nbsp;<br>きのこ&nbsp;&nbsp;<br>は&nbsp;&nbsp;&nbsp;&nbsp;<br>1973&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;<br>11&nbsp;&nbsp;&nbsp;<br>月&nbsp;&nbsp;&nbsp;&nbsp;<br>28&nbsp;&nbsp;&nbsp;<br>日&nbsp;&nbsp;&nbsp;&nbsp;<br>に&nbsp;&nbsp;&nbsp;&nbsp;<br>千葉&nbsp;&nbsp;&nbsp;<br>県&nbsp;&nbsp;&nbsp;&nbsp;<br>円空&nbsp;&nbsp;&nbsp;<br>山&nbsp;&nbsp;&nbsp;&nbsp;<br>で&nbsp;&nbsp;&nbsp;&nbsp;<br>生まれ&nbsp;&nbsp;<br>、&nbsp;&nbsp;&nbsp;&nbsp;<br>ゲーム&nbsp;&nbsp;<br>制作&nbsp;&nbsp;&nbsp;<br>会社&nbsp;&nbsp;&nbsp;<br>「&nbsp;&nbsp;&nbsp;&nbsp;<br>ノーツ&nbsp;&nbsp;<br>」&nbsp;&nbsp;&nbsp;&nbsp;<br>の&nbsp;&nbsp;&nbsp;&nbsp;<br>設立&nbsp;&nbsp;&nbsp;<br>者&nbsp;&nbsp;&nbsp;&nbsp;<br>だ&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Relation&nbsp;<br>────────&nbsp;<br>compound&nbsp;<br>nsubj&nbsp;&nbsp;&nbsp;&nbsp;<br>case&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>compound&nbsp;<br>compound&nbsp;<br>compound&nbsp;<br>compound&nbsp;<br>nummod&nbsp;&nbsp;&nbsp;<br>obl&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>case&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>compound&nbsp;<br>nmod&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>compound&nbsp;<br>obl&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>case&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>acl&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>punct&nbsp;&nbsp;&nbsp;&nbsp;<br>compound&nbsp;<br>compound&nbsp;<br>nmod&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>punct&nbsp;&nbsp;&nbsp;&nbsp;<br>compound&nbsp;<br>punct&nbsp;&nbsp;&nbsp;&nbsp;<br>case&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>compound&nbsp;<br>root&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>cop&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>punct&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">PoS&nbsp;<br>───&nbsp;<br>NPR&nbsp;<br>NPR&nbsp;<br>P&nbsp;&nbsp;&nbsp;<br>NUM&nbsp;<br>CL&nbsp;&nbsp;<br>NUM&nbsp;<br>CL&nbsp;&nbsp;<br>NUM&nbsp;<br>CL&nbsp;&nbsp;<br>P&nbsp;&nbsp;&nbsp;<br>NPR&nbsp;<br>NPR&nbsp;<br>NPR&nbsp;<br>NPR&nbsp;<br>P&nbsp;&nbsp;&nbsp;<br>VB&nbsp;&nbsp;<br>PU&nbsp;&nbsp;<br>N&nbsp;&nbsp;&nbsp;<br>N&nbsp;&nbsp;&nbsp;<br>N&nbsp;&nbsp;&nbsp;<br>PUL&nbsp;<br>NPR&nbsp;<br>PUR&nbsp;<br>P&nbsp;&nbsp;&nbsp;<br>N&nbsp;&nbsp;&nbsp;<br>N&nbsp;&nbsp;&nbsp;<br>AX&nbsp;&nbsp;<br>PU&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;<br>────&nbsp;<br>奈須&nbsp;&nbsp;&nbsp;<br>きのこ&nbsp;&nbsp;<br>は&nbsp;&nbsp;&nbsp;&nbsp;<br>1973&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;<br>11&nbsp;&nbsp;&nbsp;<br>月&nbsp;&nbsp;&nbsp;&nbsp;<br>28&nbsp;&nbsp;&nbsp;<br>日&nbsp;&nbsp;&nbsp;&nbsp;<br>に&nbsp;&nbsp;&nbsp;&nbsp;<br>千葉&nbsp;&nbsp;&nbsp;<br>県&nbsp;&nbsp;&nbsp;&nbsp;<br>円空&nbsp;&nbsp;&nbsp;<br>山&nbsp;&nbsp;&nbsp;&nbsp;<br>で&nbsp;&nbsp;&nbsp;&nbsp;<br>生まれ&nbsp;&nbsp;<br>、&nbsp;&nbsp;&nbsp;&nbsp;<br>ゲーム&nbsp;&nbsp;<br>制作&nbsp;&nbsp;&nbsp;<br>会社&nbsp;&nbsp;&nbsp;<br>「&nbsp;&nbsp;&nbsp;&nbsp;<br>ノーツ&nbsp;&nbsp;<br>」&nbsp;&nbsp;&nbsp;&nbsp;<br>の&nbsp;&nbsp;&nbsp;&nbsp;<br>設立&nbsp;&nbsp;&nbsp;<br>者&nbsp;&nbsp;&nbsp;&nbsp;<br>だ&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">NER&nbsp;Type&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────────&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┴►PERSON&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;├►DATE&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;├►LOCATION&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>───►ORGANIZATION&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;<br>────&nbsp;<br>奈須&nbsp;&nbsp;&nbsp;<br>きのこ&nbsp;&nbsp;<br>は&nbsp;&nbsp;&nbsp;&nbsp;<br>1973&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;<br>11&nbsp;&nbsp;&nbsp;<br>月&nbsp;&nbsp;&nbsp;&nbsp;<br>28&nbsp;&nbsp;&nbsp;<br>日&nbsp;&nbsp;&nbsp;&nbsp;<br>に&nbsp;&nbsp;&nbsp;&nbsp;<br>千葉&nbsp;&nbsp;&nbsp;<br>県&nbsp;&nbsp;&nbsp;&nbsp;<br>円空&nbsp;&nbsp;&nbsp;<br>山&nbsp;&nbsp;&nbsp;&nbsp;<br>で&nbsp;&nbsp;&nbsp;&nbsp;<br>生まれ&nbsp;&nbsp;<br>、&nbsp;&nbsp;&nbsp;&nbsp;<br>ゲーム&nbsp;&nbsp;<br>制作&nbsp;&nbsp;&nbsp;<br>会社&nbsp;&nbsp;&nbsp;<br>「&nbsp;&nbsp;&nbsp;&nbsp;<br>ノーツ&nbsp;&nbsp;<br>」&nbsp;&nbsp;&nbsp;&nbsp;<br>の&nbsp;&nbsp;&nbsp;&nbsp;<br>設立&nbsp;&nbsp;&nbsp;<br>者&nbsp;&nbsp;&nbsp;&nbsp;<br>だ&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">SRL&nbsp;PA1&nbsp;&nbsp;<br>────────&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┴►ノ？&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>╟──►PRED&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;<br>────&nbsp;<br>奈須&nbsp;&nbsp;&nbsp;<br>きのこ&nbsp;&nbsp;<br>は&nbsp;&nbsp;&nbsp;&nbsp;<br>1973&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;<br>11&nbsp;&nbsp;&nbsp;<br>月&nbsp;&nbsp;&nbsp;&nbsp;<br>28&nbsp;&nbsp;&nbsp;<br>日&nbsp;&nbsp;&nbsp;&nbsp;<br>に&nbsp;&nbsp;&nbsp;&nbsp;<br>千葉&nbsp;&nbsp;&nbsp;<br>県&nbsp;&nbsp;&nbsp;&nbsp;<br>円空&nbsp;&nbsp;&nbsp;<br>山&nbsp;&nbsp;&nbsp;&nbsp;<br>で&nbsp;&nbsp;&nbsp;&nbsp;<br>生まれ&nbsp;&nbsp;<br>、&nbsp;&nbsp;&nbsp;&nbsp;<br>ゲーム&nbsp;&nbsp;<br>制作&nbsp;&nbsp;&nbsp;<br>会社&nbsp;&nbsp;&nbsp;<br>「&nbsp;&nbsp;&nbsp;&nbsp;<br>ノーツ&nbsp;&nbsp;<br>」&nbsp;&nbsp;&nbsp;&nbsp;<br>の&nbsp;&nbsp;&nbsp;&nbsp;<br>設立&nbsp;&nbsp;&nbsp;<br>者&nbsp;&nbsp;&nbsp;&nbsp;<br>だ&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">SRL&nbsp;PA2&nbsp;&nbsp;<br>────────&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;├►ガ&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;├►時間&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;├►デ&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>╟──►PRED&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;<br>────&nbsp;<br>奈須&nbsp;&nbsp;&nbsp;<br>きのこ&nbsp;&nbsp;<br>は&nbsp;&nbsp;&nbsp;&nbsp;<br>1973&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;<br>11&nbsp;&nbsp;&nbsp;<br>月&nbsp;&nbsp;&nbsp;&nbsp;<br>28&nbsp;&nbsp;&nbsp;<br>日&nbsp;&nbsp;&nbsp;&nbsp;<br>に&nbsp;&nbsp;&nbsp;&nbsp;<br>千葉&nbsp;&nbsp;&nbsp;<br>県&nbsp;&nbsp;&nbsp;&nbsp;<br>円空&nbsp;&nbsp;&nbsp;<br>山&nbsp;&nbsp;&nbsp;&nbsp;<br>で&nbsp;&nbsp;&nbsp;&nbsp;<br>生まれ&nbsp;&nbsp;<br>、&nbsp;&nbsp;&nbsp;&nbsp;<br>ゲーム&nbsp;&nbsp;<br>制作&nbsp;&nbsp;&nbsp;<br>会社&nbsp;&nbsp;&nbsp;<br>「&nbsp;&nbsp;&nbsp;&nbsp;<br>ノーツ&nbsp;&nbsp;<br>」&nbsp;&nbsp;&nbsp;&nbsp;<br>の&nbsp;&nbsp;&nbsp;&nbsp;<br>設立&nbsp;&nbsp;&nbsp;<br>者&nbsp;&nbsp;&nbsp;&nbsp;<br>だ&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">SRL&nbsp;PA3&nbsp;&nbsp;<br>────────&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┴►ノ&nbsp;&nbsp;&nbsp;&nbsp;<br>╟──►PRED&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;<br>────&nbsp;<br>奈須&nbsp;&nbsp;&nbsp;<br>きのこ&nbsp;&nbsp;<br>は&nbsp;&nbsp;&nbsp;&nbsp;<br>1973&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;<br>11&nbsp;&nbsp;&nbsp;<br>月&nbsp;&nbsp;&nbsp;&nbsp;<br>28&nbsp;&nbsp;&nbsp;<br>日&nbsp;&nbsp;&nbsp;&nbsp;<br>に&nbsp;&nbsp;&nbsp;&nbsp;<br>千葉&nbsp;&nbsp;&nbsp;<br>県&nbsp;&nbsp;&nbsp;&nbsp;<br>円空&nbsp;&nbsp;&nbsp;<br>山&nbsp;&nbsp;&nbsp;&nbsp;<br>で&nbsp;&nbsp;&nbsp;&nbsp;<br>生まれ&nbsp;&nbsp;<br>、&nbsp;&nbsp;&nbsp;&nbsp;<br>ゲーム&nbsp;&nbsp;<br>制作&nbsp;&nbsp;&nbsp;<br>会社&nbsp;&nbsp;&nbsp;<br>「&nbsp;&nbsp;&nbsp;&nbsp;<br>ノーツ&nbsp;&nbsp;<br>」&nbsp;&nbsp;&nbsp;&nbsp;<br>の&nbsp;&nbsp;&nbsp;&nbsp;<br>設立&nbsp;&nbsp;&nbsp;<br>者&nbsp;&nbsp;&nbsp;&nbsp;<br>だ&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">SRL&nbsp;PA4&nbsp;&nbsp;<br>────────&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;├►ガ&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;├►ヲ&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>╟──►PRED&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;<br>────&nbsp;<br>奈須&nbsp;&nbsp;&nbsp;<br>きのこ&nbsp;&nbsp;<br>は&nbsp;&nbsp;&nbsp;&nbsp;<br>1973&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;<br>11&nbsp;&nbsp;&nbsp;<br>月&nbsp;&nbsp;&nbsp;&nbsp;<br>28&nbsp;&nbsp;&nbsp;<br>日&nbsp;&nbsp;&nbsp;&nbsp;<br>に&nbsp;&nbsp;&nbsp;&nbsp;<br>千葉&nbsp;&nbsp;&nbsp;<br>県&nbsp;&nbsp;&nbsp;&nbsp;<br>円空&nbsp;&nbsp;&nbsp;<br>山&nbsp;&nbsp;&nbsp;&nbsp;<br>で&nbsp;&nbsp;&nbsp;&nbsp;<br>生まれ&nbsp;&nbsp;<br>、&nbsp;&nbsp;&nbsp;&nbsp;<br>ゲーム&nbsp;&nbsp;<br>制作&nbsp;&nbsp;&nbsp;<br>会社&nbsp;&nbsp;&nbsp;<br>「&nbsp;&nbsp;&nbsp;&nbsp;<br>ノーツ&nbsp;&nbsp;<br>」&nbsp;&nbsp;&nbsp;&nbsp;<br>の&nbsp;&nbsp;&nbsp;&nbsp;<br>設立&nbsp;&nbsp;&nbsp;<br>者&nbsp;&nbsp;&nbsp;&nbsp;<br>だ&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">SRL&nbsp;PA5&nbsp;&nbsp;<br>────────&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;├►ガ&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>╟──►PRED&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;<br>────&nbsp;<br>奈須&nbsp;&nbsp;&nbsp;<br>きのこ&nbsp;&nbsp;<br>は&nbsp;&nbsp;&nbsp;&nbsp;<br>1973&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;<br>11&nbsp;&nbsp;&nbsp;<br>月&nbsp;&nbsp;&nbsp;&nbsp;<br>28&nbsp;&nbsp;&nbsp;<br>日&nbsp;&nbsp;&nbsp;&nbsp;<br>に&nbsp;&nbsp;&nbsp;&nbsp;<br>千葉&nbsp;&nbsp;&nbsp;<br>県&nbsp;&nbsp;&nbsp;&nbsp;<br>円空&nbsp;&nbsp;&nbsp;<br>山&nbsp;&nbsp;&nbsp;&nbsp;<br>で&nbsp;&nbsp;&nbsp;&nbsp;<br>生まれ&nbsp;&nbsp;<br>、&nbsp;&nbsp;&nbsp;&nbsp;<br>ゲーム&nbsp;&nbsp;<br>制作&nbsp;&nbsp;&nbsp;<br>会社&nbsp;&nbsp;&nbsp;<br>「&nbsp;&nbsp;&nbsp;&nbsp;<br>ノーツ&nbsp;&nbsp;<br>」&nbsp;&nbsp;&nbsp;&nbsp;<br>の&nbsp;&nbsp;&nbsp;&nbsp;<br>設立&nbsp;&nbsp;&nbsp;<br>者&nbsp;&nbsp;&nbsp;&nbsp;<br>だ&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">PoS&nbsp;&nbsp;&nbsp;&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;9&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;10&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;11<br>───────────────────────────────────────────────────────────────────────────<br>NPR──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NPR──┴►NP&nbsp;─────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>P&nbsp;─────────&nbsp;───┴────────────────────────────────────────────────►PP&nbsp;───┐&nbsp;&nbsp;&nbsp;<br>NUM──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>CL&nbsp;──┴►NUMCLP──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NUM──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>CL&nbsp;──┴►NUMCLP──┼►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NUM──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>CL&nbsp;──┴►NUMCLP──┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►PP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>P&nbsp;─────────&nbsp;───────────┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NPR──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NPR──┴►PP&nbsp;─────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NPR────────&nbsp;&nbsp;&nbsp;&nbsp;├►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├────────────────────────────────►IP────┤&nbsp;&nbsp;&nbsp;<br>NPR────────&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►PP────┤&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>P&nbsp;─────────&nbsp;───────────┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>VB&nbsp;────────&nbsp;───────────────────┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►IP<br>PU&nbsp;────────&nbsp;───────────────────────────────────────────────────────────┤&nbsp;&nbsp;&nbsp;<br>N&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>N&nbsp;───┴►NP&nbsp;──────►PRN&nbsp;──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>N&nbsp;─────────&nbsp;───────────┴►NP&nbsp;────►PRN&nbsp;──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>PUL────────&nbsp;───────────────────────────┤&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NPR────────&nbsp;───────────────────────────┼►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>PUR────────&nbsp;───────────────────────────┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►PP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>P&nbsp;─────────&nbsp;───────────────────────────────────┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►IP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>N&nbsp;─────────&nbsp;───────────────────────────────────────────┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►NP────┤&nbsp;&nbsp;&nbsp;<br>N&nbsp;─────────&nbsp;───────────────────────────────────────────────────┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>AX&nbsp;────────&nbsp;───────────────────────────────────────────────────────────┤&nbsp;&nbsp;&nbsp;<br>PU&nbsp;────────&nbsp;───────────────────────────────────────────────────────────┘&nbsp;&nbsp;&nbsp;</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {\n      \"tags\": []\n     },\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"ja(['2021年、HanLPv2.1は次世代の最先端多言語NLP技術を本番環境に導入します。',\\n\",\n    \"    '奈須きのこは1973年11月28日に千葉県円空山で生まれ、ゲーム制作会社「ノーツ」の設立者だ。',]).pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"NifrOGlNK7KD\"\n   },\n   \"source\": [\n    \"以及支持[130种语言](https://hanlp.hankcs.com/docs/api/hanlp/pretrained/mtl.html#hanlp.pretrained.mtl.UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_MMINILMV2L6)的多语种联合模型：\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 21,\n   \"metadata\": {\n    \"colab\": {\n     \"base_uri\": \"https://localhost:8080/\",\n     \"height\": 1000\n    },\n    \"id\": \"ae-4j5sbK7KD\",\n    \"outputId\": \"2777cc5d-c1c5-4091-b754-0c220dafea8a\"\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stderr\",\n     \"output_type\": \"stream\",\n     \"text\": []\n    },\n    {\n     \"data\": {\n      \"text/html\": [\n       \"<div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Dep&nbsp;Tree&nbsp;&nbsp;&nbsp;<br>──────────&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;┌─►├──&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;└─►&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;┌─►&nbsp;<br>┌┬┬─┴──┴──&nbsp;<br>│││&nbsp;&nbsp;┌───►&nbsp;<br>│││&nbsp;&nbsp;│┌──►&nbsp;<br>│││&nbsp;&nbsp;││┌─►&nbsp;<br>││└─►└┴┴──&nbsp;<br>││&nbsp;&nbsp;&nbsp;&nbsp;┌──►&nbsp;<br>││&nbsp;&nbsp;&nbsp;&nbsp;│┌─►&nbsp;<br>│└───►└┴──&nbsp;<br>└────────►&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Token&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────────&nbsp;<br>In&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>2021&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>,&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>delivers&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>state-of-the-art&nbsp;<br>multilingual&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>techniques&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>to&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>production&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>environments&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Relation&nbsp;<br>────────&nbsp;<br>case&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>obl&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>punct&nbsp;&nbsp;&nbsp;&nbsp;<br>nsubj&nbsp;&nbsp;&nbsp;&nbsp;<br>root&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>amod&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>amod&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>compound&nbsp;<br>obj&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>case&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>compound&nbsp;<br>obl&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>punct&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Lemma&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────────&nbsp;<br>in&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>2021&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>,&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HANlpv2.1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>deliver&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>state-of-the-art&nbsp;<br>multilingual&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>technique&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>to&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>production&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>environment&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">PoS&nbsp;&nbsp;&nbsp;<br>─────&nbsp;<br>ADP&nbsp;&nbsp;&nbsp;<br>NUM&nbsp;&nbsp;&nbsp;<br>PUNCT&nbsp;<br>PROPN&nbsp;<br>VERB&nbsp;&nbsp;<br>ADJ&nbsp;&nbsp;&nbsp;<br>ADJ&nbsp;&nbsp;&nbsp;<br>PROPN&nbsp;<br>NOUN&nbsp;&nbsp;<br>ADP&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;&nbsp;<br>NOUN&nbsp;&nbsp;<br>PUNCT&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────────&nbsp;<br>In&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>2021&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>,&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>delivers&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>state-of-the-art&nbsp;<br>multilingual&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>techniques&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>to&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>production&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>environments&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">NER&nbsp;Type&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>───────────────&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>───►DATE&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>───►WORK_OF_ART&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────────&nbsp;<br>In&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>2021&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>,&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>delivers&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>state-of-the-art&nbsp;<br>multilingual&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>techniques&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>to&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>production&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>environments&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">SRL&nbsp;PA1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┴►ARGM-TMP&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>───►ARG0&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>╟──►PRED&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;├►ARG2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────────&nbsp;<br>In&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>2021&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>,&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>delivers&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>state-of-the-art&nbsp;<br>multilingual&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>techniques&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>to&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>production&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>environments&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>.&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">PoS&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6<br>──────────────────────────────────<br>ADP&nbsp;───────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NUM&nbsp;────►NP&nbsp;───┴────────►PP&nbsp;───┐&nbsp;&nbsp;<br>PUNCT──────────────────────────┤&nbsp;&nbsp;<br>PROPN───────────────────►NP────┤&nbsp;&nbsp;<br>VERB&nbsp;──────────────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;<br>ADJ&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;<br>ADJ&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;<br>PROPN&nbsp;&nbsp;├────────►NP────┼►VP────┼►S<br>NOUN&nbsp;──┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;<br>ADP&nbsp;───────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;<br>NOUN&nbsp;──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►PP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;<br>NOUN&nbsp;──┴►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;<br>PUNCT──────────────────────────┘&nbsp;&nbsp;</pre></div><br><div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Dep&nbsp;Tree&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────────&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>┌────────►├──&nbsp;<br>│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;└─►&nbsp;<br>│┌───────►┌──&nbsp;<br>││&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;└─►&nbsp;<br>││&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>││&nbsp;&nbsp;&nbsp;┌───►├──&nbsp;<br>││&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;└─►&nbsp;<br>││&nbsp;&nbsp;&nbsp;│┌─────►&nbsp;<br>││&nbsp;&nbsp;&nbsp;││┌────►&nbsp;<br>││&nbsp;&nbsp;&nbsp;│││┌───►&nbsp;<br>││&nbsp;&nbsp;&nbsp;││││┌──►&nbsp;<br>││&nbsp;&nbsp;&nbsp;│││││┌─►&nbsp;<br>││┌─►└┴┴┴┴┼──&nbsp;<br>│││&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;└─►&nbsp;<br>│││&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>│││&nbsp;&nbsp;&nbsp;&nbsp;┌─►├──&nbsp;<br>│││&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;└─►&nbsp;<br>└┴┴────┴─┬┬──&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│└─►&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;└──►&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Token&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>、&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>は&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>の&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先端&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>言語&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技術&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>を&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>本番&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>環境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>に&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>導入&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>します&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Relation&nbsp;<br>────────&nbsp;<br>nummod&nbsp;&nbsp;&nbsp;<br>obl&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>punct&nbsp;&nbsp;&nbsp;&nbsp;<br>nsubj&nbsp;&nbsp;&nbsp;&nbsp;<br>case&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>compound&nbsp;<br>nmod&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>case&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>compound&nbsp;<br>compound&nbsp;<br>compound&nbsp;<br>compound&nbsp;<br>compound&nbsp;<br>obj&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>case&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>compound&nbsp;<br>obl&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>case&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>root&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>aux&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>punct&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Lemma&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>、&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HANLPV2.1&nbsp;<br>は&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>の&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先端&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>言語&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技術&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>を&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>本番&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>環境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>に&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>導入&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>します&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">PoS&nbsp;&nbsp;&nbsp;<br>─────&nbsp;<br>NUM&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;&nbsp;<br>PUNCT&nbsp;<br>NOUN&nbsp;&nbsp;<br>ADP&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;&nbsp;<br>NOUN&nbsp;&nbsp;<br>ADP&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;&nbsp;<br>NOUN&nbsp;&nbsp;<br>NOUN&nbsp;&nbsp;<br>NOUN&nbsp;&nbsp;<br>NOUN&nbsp;&nbsp;<br>NOUN&nbsp;&nbsp;<br>ADP&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;&nbsp;<br>NOUN&nbsp;&nbsp;<br>ADP&nbsp;&nbsp;&nbsp;<br>VERB&nbsp;&nbsp;<br>AUX&nbsp;&nbsp;&nbsp;<br>PUNCT&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>、&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>は&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>の&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先端&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>言語&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技術&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>を&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>本番&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>環境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>に&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>導入&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>します&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">NER&nbsp;Type&nbsp;<br>────────&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┴►DATE&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>、&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>は&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>の&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先端&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>言語&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技術&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>を&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>本番&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>環境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>に&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>導入&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>します&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">PoS&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;8&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;9&nbsp;<br>───────────────────────────────────────────────────────────<br>NUM&nbsp;───────────────────────────────────────────────────┐&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;──────────────────────────────────────────────────┤&nbsp;&nbsp;&nbsp;<br>PUNCT──────────────────────────────────────────────────┤&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;──────────────────────────────────────────────────┤&nbsp;&nbsp;&nbsp;<br>ADP&nbsp;───────────────────────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;──────────────────────────┤&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;──────────────────────────┤&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>ADP&nbsp;───────────────────────────┼►VP&nbsp;────►VP&nbsp;────►IP────┤&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;───►ADJP──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;───►ADJP──┴►ADJP──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;───────────►ADJP──┴►ADJP──┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►IP<br>NOUN&nbsp;──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;&nbsp;&nbsp;├►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;──┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>ADP&nbsp;───────────┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;──────────────────┼►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;──────────────────┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>ADP&nbsp;────────────────────►PP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>VERB&nbsp;──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├────────►NP────┤&nbsp;&nbsp;&nbsp;<br>AUX&nbsp;───┴────────────────────────►VP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>PUNCT──────────────────────────────────────────────────┘&nbsp;&nbsp;&nbsp;</pre></div><br><div style=\\\"display: table; line-height: 128%;\\\"><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Dep&nbsp;Tree&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>&nbsp;&nbsp;&nbsp;┌────►└──&nbsp;<br>&nbsp;&nbsp;&nbsp;│┌──────►&nbsp;<br>&nbsp;&nbsp;&nbsp;││&nbsp;&nbsp;&nbsp;┌──►&nbsp;<br>&nbsp;&nbsp;&nbsp;││&nbsp;&nbsp;&nbsp;│┌─►&nbsp;<br>&nbsp;&nbsp;&nbsp;││┌─►└┴──&nbsp;<br>┌┬─┴┴┴──────&nbsp;<br>││&nbsp;&nbsp;┌──────►&nbsp;<br>││&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>││&nbsp;&nbsp;│┌──►└──&nbsp;<br>││&nbsp;&nbsp;││&nbsp;&nbsp;&nbsp;┌─►&nbsp;<br>││&nbsp;&nbsp;││┌─►└──&nbsp;<br>││&nbsp;&nbsp;│││&nbsp;&nbsp;┌─►&nbsp;<br>│└─►└┴┴──┴──&nbsp;<br>└──────────►&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Token&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>为&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>生产&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>环境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>带来&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先进的&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>语种&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技术&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Relation&nbsp;&nbsp;<br>─────────&nbsp;<br>nummod&nbsp;&nbsp;&nbsp;&nbsp;<br>nmod:tmod&nbsp;<br>nsubj&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>case&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>nmod&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>obl&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>root&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>nmod&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>advmod&nbsp;&nbsp;&nbsp;&nbsp;<br>amod&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>nummod&nbsp;&nbsp;&nbsp;&nbsp;<br>nmod&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>nmod&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>obj&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>punct&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Lemma&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HANlpv2.1&nbsp;<br>为&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>生产&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>环境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>带来&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先进的&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>语种&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技术&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">PoS&nbsp;&nbsp;&nbsp;<br>─────&nbsp;<br>NUM&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;&nbsp;<br>X&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>ADP&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;&nbsp;<br>NOUN&nbsp;&nbsp;<br>VERB&nbsp;&nbsp;<br>NOUN&nbsp;&nbsp;<br>ADV&nbsp;&nbsp;&nbsp;<br>ADJ&nbsp;&nbsp;&nbsp;<br>NUM&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;&nbsp;<br>X&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;&nbsp;<br>PUNCT&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>为&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>生产&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>环境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>带来&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先进的&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>语种&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技术&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">NER&nbsp;Type&nbsp;&nbsp;&nbsp;<br>──────────&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┴►DATE&nbsp;&nbsp;&nbsp;<br>───►PERSON&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>为&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>生产&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>环境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>带来&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先进的&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>语种&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技术&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">SRL&nbsp;PA1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>────────────&nbsp;<br>◄─┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>◄─┴►ARGM-TMP&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>╟──►PRED&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">Tok&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>─────────&nbsp;<br>2021&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>年&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>HanLPv2.1&nbsp;<br>为&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>生产&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>环境&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>带来&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>次世代&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>最&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>先进的&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>多&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>语种&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NLP&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>技术&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>。&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</pre><pre style=\\\"display: table-cell; font-family: SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,monospace; white-space: nowrap;\\\">PoS&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;7&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;8&nbsp;<br>───────────────────────────────────────────────────<br>NUM&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;──┴────────────────────────────────►NP&nbsp;───┐&nbsp;&nbsp;&nbsp;<br>X&nbsp;──────────────────────────────────────►NP────┤&nbsp;&nbsp;&nbsp;<br>ADP&nbsp;───────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├────────────────►PP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;──┴►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>VERB&nbsp;──────────────────────────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP────┤&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;───────────►ADJP──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>ADV&nbsp;────►ADVP──┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►VP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►IP<br>ADJ&nbsp;────►ADJP──┴►ADJP──┤&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NUM&nbsp;────►QP&nbsp;───┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;├►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;───►NP&nbsp;───┴►NP────┤&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>X&nbsp;─────┐&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>NOUN&nbsp;──┴────────►NP&nbsp;───┘&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;│&nbsp;&nbsp;&nbsp;<br>PUNCT──────────────────────────────────────────┘&nbsp;&nbsp;&nbsp;</pre></div>\"\n      ],\n      \"text/plain\": [\n       \"<IPython.core.display.HTML object>\"\n      ]\n     },\n     \"metadata\": {\n      \"tags\": []\n     },\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"from hanlp.utils.torch_util import gpus_available\\n\",\n    \"if gpus_available(): # 建议在GPU上运行XLMR_BASE，否则运行mini模型\\n\",\n    \"    mul = hanlp.load(hanlp.pretrained.mtl.UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_XLMR_BASE)\\n\",\n    \"else:\\n\",\n    \"    if 'ja' in globals(): # Binder内存只有2G，释放已加载的模型\\n\",\n    \"        del ja\\n\",\n    \"    mul = hanlp.load(hanlp.pretrained.mtl.UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_MMINILMV2L6)\\n\",\n    \"mul(['In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques to production environments.',\\n\",\n    \"     '2021年、HanLPv2.1は次世代の最先端多言語NLP技術を本番環境に導入します。',\\n\",\n    \"     '2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。']).pretty_print()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {\n    \"id\": \"0QV_93CjK7KD\"\n   },\n   \"source\": [\n    \"你可以在下面输入你想执行的代码~\"\n   ]\n  }\n ],\n \"metadata\": {\n  \"accelerator\": \"GPU\",\n  \"colab\": {\n   \"collapsed_sections\": [],\n   \"name\": \"tutorial.ipynb\",\n   \"provenance\": []\n  },\n  \"kernelspec\": {\n   \"display_name\": \"Python 3 (ipykernel)\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.12\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n"
  },
  {
    "path": "plugins/hanlp_demo/setup.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 19:26\nfrom os.path import abspath, join, dirname\nfrom setuptools import find_packages, setup\n\nthis_dir = abspath(dirname(__file__))\nwith open(join(this_dir, 'README.md'), encoding='utf-8') as file:\n    long_description = file.read()\n\nsetup(\n    name='hanlp_demo',\n    version='0.0.1',\n    description='HanLP: Han Language Processing',\n    long_description=long_description,\n    long_description_content_type=\"text/markdown\",\n    url='https://github.com/hankcs/HanLP',\n    author='hankcs',\n    author_email='hankcshe@gmail.com',\n    license='Apache License 2.0',\n    classifiers=[\n        'Intended Audience :: Science/Research',\n        'Intended Audience :: Developers',\n        \"Development Status :: 3 - Alpha\",\n        'Operating System :: OS Independent',\n        \"License :: OSI Approved :: Apache Software License\",\n        'Programming Language :: Python :: 3 :: Only',\n        'Topic :: Scientific/Engineering :: Artificial Intelligence',\n        \"Topic :: Text Processing :: Linguistic\"\n    ],\n    keywords='corpus,machine-learning,NLU,NLP',\n    packages=find_packages(exclude=['docs', 'tests*']),\n    include_package_data=True,\n    install_requires=[\n        'hanlp_common'\n    ],\n    python_requires='>=3.6',\n)\n"
  },
  {
    "path": "plugins/hanlp_restful/README.md",
    "content": "# RESTFul API Client for HanLP\n\n[中文](https://github.com/hankcs/HanLP/tree/doc-zh) | [1.x](https://github.com/hankcs/HanLP/tree/1.x) | [forum](https://bbs.hankcs.com/) | [docker](https://github.com/WalterInSH/hanlp-jupyter-docker)\n\nThe multilingual NLP library for researchers and companies, built on PyTorch and TensorFlow 2.x, for advancing state-of-the-art deep learning techniques in both academia and industry. HanLP was designed from day one to be efficient, user friendly and extendable. It comes with pretrained models for various human languages including English, Chinese and many others. Currently, HanLP 2.0 is in alpha stage with more killer features on the roadmap. Discussions are welcomed on our [forum](https://bbs.hankcs.com/), while bug reports and feature requests are reserved for GitHub issues. For Java users, please checkout the [1.x](https://github.com/hankcs/HanLP/tree/1.x) branch.\n\n\n## Installation\n\n```bash\npip install hanlp-restful\n```\n\n## License\n\nHanLP is licensed under **Apache License 2.0**. You can use HanLP in your commercial products for free. We would appreciate it if you add a link to HanLP on your website.\n\n"
  },
  {
    "path": "plugins/hanlp_restful/hanlp_restful/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-11-29 17:48\nimport json\nfrom typing import Union, List, Optional, Dict, Any, Tuple\nfrom urllib.error import HTTPError\nfrom urllib.parse import urlencode\nfrom urllib.request import Request, urlopen\nfrom hanlp_common.document import Document\n\ntry:\n    # noinspection PyUnresolvedReferences\n    import requests\n\n\n    def _post(url, form: Dict[str, Any], headers: Dict[str, Any], timeout=60, verify=True) -> str:\n        response = requests.post(url, json=form, headers=headers, timeout=timeout, verify=verify)\n        if response.status_code != 200:\n            raise HTTPError(url, response.status_code, response.text, response.headers, None)\n        return response.text\nexcept ImportError:\n    import ssl\n\n\n    def _post(url, form: Dict[str, Any], headers: Dict[str, Any], timeout=60, verify=True) -> str:\n        request = Request(url, json.dumps(form).encode())\n        for k, v in headers.items():\n            request.add_header(k, v)\n        ctx = None\n        if not verify:\n            ctx = ssl.create_default_context()\n            ctx.check_hostname = False\n            ctx.verify_mode = ssl.CERT_NONE\n        return urlopen(request, timeout=timeout, context=ctx).read().decode()\n\n\nclass HanLPClient(object):\n\n    def __init__(self, url: str, auth: str = None, language=None, timeout=60, verify=True) -> None:\n        \"\"\"\n\n        Args:\n            url (str): An API endpoint to a service provider.\n            auth (str): An auth key licenced from a service provider.\n            language (str): The default language for each :func:`~hanlp_restful.HanLPClient.parse` call.\n                Contact the service provider for the list of languages supported.\n                Conventionally, ``zh`` is used for Chinese and ``mul`` for multilingual.\n                Leave ``None`` to use the default language on server.\n            timeout (int): Maximum waiting time in seconds for a request.\n            verify (bool): ``True`` to enable SSL cert verification. You can also pass ``verify`` the path to a CA_BUNDLE\n                file or directory with certificates of trusted CAs (``requests`` required).\n        \"\"\"\n        super().__init__()\n        self._language = language\n        self._timeout = timeout\n        self._url = url\n        if auth is None:\n            import os\n            auth = os.getenv('HANLP_AUTH', None)\n        self._auth = auth\n        self._verify = verify\n\n    def parse(self,\n              text: Union[str, List[str]] = None,\n              tokens: List[List[str]] = None,\n              tasks: Optional[Union[str, List[str]]] = None,\n              skip_tasks: Optional[Union[str, List[str]]] = None,\n              language: str = None,\n              ) -> Document:\n        \"\"\"\n        Parse a piece of text.\n\n        Args:\n            text: A document (str), or a list of sentences (List[str]).\n            tokens: A list of sentences where each sentence is a list of tokens.\n            tasks: The tasks to predict. Use ``tasks=[...]`` to run selected tasks only. Dependent tasks will be\n                automatically selected.\n            skip_tasks: The tasks to skip. Use ``skip_tasks='tok/fine'`` to enable coarse tokenization for all tasks.\n                Use ``tasks=['tok/coarse', ...]`` and ``skip_tasks='tok/fine'`` to enable coarse tokenization for\n                selected tasks.\n            language: The language of input text or tokens. ``None`` to use the default language on server.\n\n        Returns:\n            A :class:`~hanlp_common.document.Document`.\n\n        Examples::\n\n            # Use tasks=[...] to run selected tasks only\n            HanLP('晓美焰来到自然语义科技公司', tasks=['pos', 'ner'])\n\n            # Use skip_tasks='tok/fine' to enable coarse tokenization for all tasks\n            HanLP('晓美焰来到自然语义科技公司', skip_tasks='tok/fine')\n\n            # Use tasks=['tok/coarse', ...] and skip_tasks='tok/fine' to enable\n            # coarse tokenization for selected tasks\n            HanLP('晓美焰来到自然语义科技公司', tasks=['tok/coarse','pos'],skip_tasks='tok/fine')\n\n\n        Raises:\n            HTTPError: Any errors happening on the Internet side or the server side. Refer to the ``code`` and ``msg``\n                of the exception for more details. A list of common errors :\n\n        - ``400 Bad Request`` indicates that the server cannot process the request due to a client\n          fault (e.g., text too long, language unsupported).\n        - ``401 Unauthorized`` indicates that the request lacks **valid** ``auth`` credentials for the API.\n        - ``422 Unprocessable Entity`` indicates that the content type of the request entity is not in\n          proper json format.\n        - ``429 Too Many Requests`` indicates the user has sent too many requests in a given\n          amount of time (\"rate limiting\").\n\n        \"\"\"\n        assert text or tokens, 'At least one of text or tokens has to be specified.'\n        response = self._send_post_json(self._url + '/parse', {\n            'text': text,\n            'tokens': tokens,\n            'tasks': tasks,\n            'skip_tasks': skip_tasks,\n            'language': language or self._language\n        })\n        return Document(response)\n\n    def __call__(self,\n                 text: Union[str, List[str]] = None,\n                 tokens: List[List[str]] = None,\n                 tasks: Optional[Union[str, List[str]]] = None,\n                 skip_tasks: Optional[Union[str, List[str]]] = None,\n                 language: str = None,\n                 ) -> Document:\n        \"\"\"\n        A shortcut of :meth:`~hanlp_restful.HanLPClient.parse`.\n        \"\"\"\n        return self.parse(text, tokens, tasks, skip_tasks, language)\n\n    def about(self) -> Dict[str, Any]:\n        \"\"\"Get the information about server and your client.\n\n        Returns:\n            A dict containing your rate limit and server version etc.\n\n        \"\"\"\n        info = self._send_get_json(self._url + '/about', {})\n        return Document(info)\n\n    def _send_post(self, url, form: Dict[str, Any]):\n        request = Request(url, json.dumps(form).encode())\n        self._add_headers(request)\n        return self._fire_request(request)\n\n    def _fire_request(self, request):\n        return urlopen(request, timeout=self._timeout).read().decode()\n\n    def _send_post_json(self, url, form: Dict[str, Any]):\n        headers = dict()\n        if self._auth:\n            headers['Authorization'] = f'Basic {self._auth}'\n        return json.loads(_post(url, form, headers, self._timeout, verify=self._verify))\n\n    def _send_get(self, url, form: Dict[str, Any]):\n        request = Request(url + '?' + urlencode(form))\n        self._add_headers(request)\n        return self._fire_request(request)\n\n    def _add_headers(self, request):\n        if self._auth:\n            request.add_header('Authorization', f'Basic {self._auth}')\n\n    def _send_get_json(self, url, form: Dict[str, Any]):\n        return json.loads(self._send_get(url, form))\n\n    def text_style_transfer(self, text: Union[str, List[str]], target_style: str, language: str = None) \\\n            -> Union[str, List[str]]:\n        \"\"\" Text style transfer aims to change the style of the input text to the target style while preserving its\n        content.\n\n        Args:\n            text: Source text.\n            target_style: Target style.\n            language: The language of input text. ``None`` to use the default language.\n\n        Returns:\n            Text or a list of text of the target style.\n\n        Examples::\n\n            HanLP.text_style_transfer(['国家对中石油抱有很大的期望.', '要用创新去推动高质量的发展。'],\n                                      target_style='gov_doc')\n            # Output:\n            [\n                '国家对中石油寄予厚望。',\n                '要以创新驱动高质量发展。'\n            ]\n\n            HanLP.text_style_transfer('我看到了窗户外面有白色的云和绿色的森林',\n                                      target_style='modern_poetry')\n            # Output:\n            '我看见窗外的白云绿林'\n        \"\"\"\n        response = self._send_post_json(self._url + '/text_style_transfer',\n                                        {'text': text, 'target_style': target_style,\n                                         'language': language or self._language})\n        return response\n\n    def semantic_textual_similarity(self, text: Union[Tuple[str, str], List[Tuple[str, str]]], language: str = None) \\\n            -> Union[float, List[float]]:\n        \"\"\" Semantic textual similarity deals with determining how similar two pieces of texts are.\n\n        Args:\n            text: A pair or pairs of text.\n            language: The language of input text. ``None`` to use the default language.\n\n        Returns:\n            Similarities.\n\n        Examples::\n\n            HanLP.semantic_textual_similarity([\n                ('看图猜一电影名', '看图猜电影'),\n                ('无线路由器怎么无线上网', '无线上网卡和无线路由器怎么用'),\n                ('北京到上海的动车票', '上海到北京的动车票'),\n            ])\n            # Output:\n            [\n                0.9764469, # Similarity of ('看图猜一电影名', '看图猜电影')\n                0.0,       # Similarity of ('无线路由器怎么无线上网', '无线上网卡和无线路由器怎么用')\n                0.0034587  # Similarity of ('北京到上海的动车票', '上海到北京的动车票')\n            ]\n        \"\"\"\n        response = self._send_post_json(self._url + '/semantic_textual_similarity',\n                                        {'text': text, 'language': language or self._language})\n        return response\n\n    def coreference_resolution(self, text: Optional[str] = None, tokens: Optional[List[List[str]]] = None,\n                               speakers: Optional[List[str]] = None, language: Optional[str] = None) -> Union[\n        Dict[str, Union[List[str], List[List[Tuple[str, int, int]]]]], List[List[Tuple[str, int, int]]]]:\n        r\"\"\" Coreference resolution is the task of clustering mentions in text that refer to the same underlying\n        real world entities.\n\n        Args:\n            text: A piece of text, usually a document without tokenization.\n            tokens: A list of sentences where each sentence is a list of tokens.\n            speakers: A list of speakers where each speaker is a ``str`` representing the speaker's ID, e.g., ``Tom``.\n            language: The language of input text. ``None`` to use the default language.\n\n        Returns:\n            When ``text`` is specified, return the clusters and tokens. Otherwise just the clusters, In this case, you need to ``sum(tokens, [])`` in order to match the span indices with tokens\n\n        Examples::\n\n            HanLP.coreference_resolution('我姐送我她的猫。我很喜欢它。')\n            # Output:\n            {'clusters': [\n                          [['我', 0, 1], ['我', 3, 4], ['我', 8, 9]], # 指代说话人\n                          [['我姐', 0, 2], ['她', 4, 5]],             # 指代说话人的姐姐\n                          [['她的猫', 4, 7], ['它', 11, 12]]],        # 指代说话人的姐姐的猫\n             'tokens': ['我', '姐', '送', '我', '她', '的', '猫', '。',\n                        '我', '很', '喜欢', '它', '。']}\n\n            HanLP.coreference_resolution(\n            tokens=[['我', '姐', '送', '我', '她', '的', '猫', '。'],\n                    ['我', '很', '喜欢', '它', '。']])\n            # Output:\n                         [\n                          [['我', 0, 1], ['我', 3, 4], ['我', 8, 9]], # 指代说话人\n                          [['我姐', 0, 2], ['她', 4, 5]],             # 指代说话人的姐姐\n                          [['她的猫', 4, 7], ['它', 11, 12]]],        # 指代说话人的姐姐的猫\n\n        .. image:: https://file.hankcs.com/img/coref_demo_small.png\n            :alt: Coreference resolution visualization\n        \"\"\"\n        response = self._send_post_json(self._url + '/coreference_resolution',\n                                        {'text': text, 'tokens': tokens, 'speakers': speakers,\n                                         'language': language or self._language})\n        return response\n\n    def tokenize(self, text: Union[str, List[str]], coarse: Optional[bool] = None, language=None) -> List[List[str]]:\n        \"\"\" Split a document into sentences and tokenize them. Note that it is always faster to tokenize a whole\n        document than to tokenize each sentence one by one. So avoid calling this method sentence by sentence but put\n        sentences into a ``list`` and pass them to the ``text`` argument.\n\n        Args:\n            text: A document (``str``), or a list of sentences (``List[str]``).\n            coarse: Whether to perform coarse-grained or fine-grained tokenization.\n            language: The language of input text. ``None`` to use the default language.\n\n        Returns:\n            A list of tokenized sentences.\n\n        Examples::\n\n            # Avoid tokenizing sentence by sentence, it is expensive:\n            HanLP.tokenize('商品和服务。')\n            [['商品', '和', '服务', '。']]\n            HanLP.tokenize('阿婆主来到北京立方庭参观自然语义科技公司')\n            [['阿婆主', '来到', '北京', '立方庭', '参观', '自然', '语义', '科技', '公司']]\n\n            # Instead, the following codes are much faster:\n            HanLP.tokenize('商品和服务。阿婆主来到北京立方庭参观自然语义科技公司')\n            [['商品', '和', '服务', '。'],\n             ['阿婆主', '来到', '北京', '立方庭', '参观', '自然', '语义', '科技', '公司']]\n\n            # To tokenize with coarse-grained standard:\n            HanLP.tokenize('商品和服务。阿婆主来到北京立方庭参观自然语义科技公司', coarse=True)\n            [['商品', '和', '服务', '。'],\n             ['阿婆主', '来到', '北京', '立方庭', '参观', '自然语义科技公司']]\n\n            # To tokenize pre-segmented sentences:\n            HanLP.tokenize(['商品和服务。', '当下雨天地面积水分外严重'])\n            [['商品', '和', '服务', '。'],\n             ['当', '下雨天', '地面', '积水', '分', '外', '严重']]\n\n            # Multilingual tokenization by specifying language='mul':\n            HanLP.tokenize(\n                ['In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques\n                 'to production environment.',\n                 '2021年、HanLPv2.1は次世代の最先端多言語NLP技術を本番環境に導入します。',\n                 '2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。'], language='mul')\n            [['In', '2021', ',', 'HanLPv2.1', 'delivers', 'state-of-the-art', 'multilingual',\n              'NLP', 'techniques', 'to', 'production', 'environment', '.'],\n             ['2021', '年', '、', 'HanLPv2.1', 'は', '次', '世代', 'の', '最', '先端', '多',\n              '言語', 'NLP', '技術', 'を', '本番', '環境', 'に', '導入', 'します', '。'],\n             ['2021', '年', 'HanLPv2.1', '为', '生产', '环境', '带来', '次世代', '最', '先进的',\n              '多', '语种', 'NLP', '技术', '。']]\n        \"\"\"\n        language = language or self._language\n        if coarse and language and language != 'zh':\n            raise NotImplementedError(f'Coarse tokenization not supported for {language}. Please set language=\"zh\".')\n        doc = self.parse(text=text, tasks='tok/coarse' if coarse is True else 'tok', language=language)\n        return next(iter(doc.values()))\n\n    def abstract_meaning_representation(self,\n                                        text: Union[str, List[str]] = None,\n                                        tokens: List[List[str]] = None,\n                                        language: str = None,\n                                        visualization: str = None,\n                                        ) -> List[Dict]:\n        \"\"\"Abstract Meaning Representation (AMR) captures “who is doing what to whom” in a sentence. Each sentence is\n        represented as a rooted, directed, acyclic graph consisting of nodes (concepts) and edges (relations).\n\n        Args:\n            text: A document (str), or a list of sentences (List[str]).\n            tokens: A list of sentences where each sentence is a list of tokens.\n            language: The language of input text or tokens. ``None`` to use the default language on server.\n            visualization: Set to `dot` or `svg` to obtain coresspodning visualization.\n\n        Returns:\n            Graphs in meaning represenation format.\n\n        Examples::\n\n            HanLP.abstract_meaning_representation('男孩希望女孩相信他。')\n            HanLP.abstract_meaning_representation('The boy wants the girl to believe him.',\n                                                  language='en')\n\n        .. image:: https://hanlp.hankcs.com/backend/v2/amr_svg?tokens=%E7%94%B7%E5%AD%A9%20%E5%B8%8C%E6%9C%9B%20%E5%A5%B3%E5%AD%A9%20%E7%9B%B8%E4%BF%A1%20%E4%BB%96%20%E3%80%82&language=zh&scale=1\n            :alt: Abstract Meaning Representation\n\n        .. image:: https://hanlp.hankcs.com/backend/v2/amr_svg?tokens=The%20boy%20wants%20the%20girl%20to%20believe%20him%20.&language=en&scale=1\n            :alt: Abstract Meaning Representation\n\n        \"\"\"\n        assert text or tokens, 'At least one of text or tokens has to be specified.'\n        return self._send_post_json(self._url + '/abstract_meaning_representation', {\n            'text': text,\n            'tokens': tokens,\n            'language': language or self._language,\n            'visualization': visualization,\n        })\n\n    def keyphrase_extraction(\n            self,\n            text: str,\n            topk: int = 10,\n            language: str = None,\n    ) -> Dict[str, float]:\n        \"\"\" Keyphrase extraction aims to identify keywords or phrases reflecting the main topics of a document.\n\n        Args:\n            text: The text content of the document. Preferably the concatenation of the title and the content.\n            topk: The number of top-K ranked keywords or keyphrases.\n            language: The language of input text or tokens. ``None`` to use the default language on server.\n\n        Returns:\n            A dictionary containing each keyword or keyphrase and its ranking score :math:`s`, :math:`s \\in [0, 1]`.\n\n        Examples::\n\n            HanLP.keyphrase_extraction(\n                '自然语言处理是一门博大精深的学科，掌握理论才能发挥出HanLP的全部性能。 '\n                '《自然语言处理入门》是一本配套HanLP的NLP入门书，助你零起点上手自然语言处理。', topk=3)\n            # Output:\n            {'自然语言处理': 0.800000011920929,\n             'HanLP的全部性能': 0.5258446335792542,\n             '一门博大精深的学科': 0.421421080827713}\n        \"\"\"\n        assert text, 'Text has to be specified.'\n        return self._send_post_json(self._url + '/keyphrase_extraction', {\n            'text': text,\n            'language': language or self._language,\n            'topk': topk,\n        })\n\n    def extractive_summarization(\n            self,\n            text: str,\n            topk: int = 3,\n            language: str = None,\n    ) -> Dict[str, float]:\n        \"\"\" Single document summarization is the task of selecting a subset of the sentences which best\n        represents a summary of the document, with a balance of salience and redundancy.\n\n        Args:\n            text: The text content of the document.\n            topk: The maximum number of top-K ranked sentences. Note that due to Trigram Blocking tricks, the actual\n                number of returned sentences could be less than ``topk``.\n            language: The language of input text or tokens. ``None`` to use the default language on server.\n\n        Returns:\n            A dictionary containing each sentence and its ranking score :math:`s \\in [0, 1]`.\n\n        Examples::\n\n            HanLP.extractive_summarization('''\n            据DigiTimes报道，在上海疫情趋缓，防疫管控开始放松后，苹果供应商广达正在逐步恢复其中国工厂的MacBook产品生产。\n            据供应链消息人士称，生产厂的订单拉动情况正在慢慢转强，这会提高MacBook Pro机型的供应量，并缩短苹果客户在过去几周所经历的延长交货时间。\n            仍有许多苹果笔记本用户在等待3月和4月订购的MacBook Pro机型到货，由于苹果的供应问题，他们的发货时间被大大推迟了。\n            据分析师郭明錤表示，广达是高端MacBook Pro的唯一供应商，自防疫封控依赖，MacBook Pro大部分型号交货时间增加了三到五周，\n            一些高端定制型号的MacBook Pro配置要到6月底到7月初才能交货。\n            尽管MacBook Pro的生产逐渐恢复，但供应问题预计依然影响2022年第三季度的产品销售。\n            苹果上周表示，防疫措施和元部件短缺将继续使其难以生产足够的产品来满足消费者的强劲需求，这最终将影响苹果6月份的收入。\n            ''')\n            # Output:\n            {'据DigiTimes报道，在上海疫情趋缓，防疫管控开始放松后，苹果供应商广达正在逐步恢复其中国工厂的MacBook产品生产。': 0.9999,\n             '仍有许多苹果笔记本用户在等待3月和4月订购的MacBook Pro机型到货，由于苹果的供应问题，他们的发货时间被大大推迟了。': 0.5800,\n             '尽管MacBook Pro的生产逐渐恢复，但供应问题预计依然影响2022年第三季度的产品销售。': 0.5422}\n        \"\"\"\n        assert text, 'Text has to be non-empty.'\n        return self._send_post_json(self._url + '/extractive_summarization', {\n            'text': text,\n            'language': language or self._language,\n            'topk': topk,\n        })\n\n    def abstractive_summarization(\n            self,\n            text: str,\n            language: str = None,\n    ) -> str:\n        r\"\"\" Abstractive Summarization is the task of generating a short and concise summary that captures the\n        salient ideas of the source text. The generated summaries potentially contain new phrases and sentences that\n        may not appear in the source text.\n\n        Args:\n            text: The text content of the document.\n            language: The language of input text or tokens. ``None`` to use the default language on server.\n\n        Returns:\n            Summarization.\n\n        Examples::\n\n            HanLP.abstractive_summarization('''\n            每经AI快讯，2月4日，长江证券研究所金属行业首席分析师王鹤涛表示，2023年海外经济衰退，美债现处于历史高位，\n            黄金的趋势是值得关注的；在国内需求修复的过程中，看好大金属品种中的铜铝钢。\n            此外，在细分的小品种里，建议关注两条主线，一是新能源，比如锂、钴、镍、稀土，二是专精特新主线。（央视财经）\n            ''')\n            # Output:\n            '长江证券：看好大金属品种中的铜铝钢'\n        \"\"\"\n        assert text, 'Text has to be non-empty.'\n        return self._send_post_json(self._url + '/abstractive_summarization', {\n            'text': text,\n            'language': language or self._language,\n        })\n\n    def grammatical_error_correction(self, text: Union[str, List[str]], language: str = None) \\\n            -> Union[str, List[str]]:\n        \"\"\" Grammatical Error Correction (GEC) is the task of correcting different kinds of errors in text such as\n        spelling, punctuation, grammatical, and word choice errors.\n\n        Args:\n            text: Text potentially containing different kinds of errors such as spelling, punctuation,\n                grammatical, and word choice errors.\n            language: The language of input text. ``None`` to use the default language.\n\n        Returns:\n            Corrected text.\n\n        Examples::\n\n            HanLP.grammatical_error_correction(['每个青年都应当有远大的报复。',\n                                                '有的同学对语言很兴趣。'])\n            # Output:\n            [\n                '每个青年都应当有远大的抱负。',\n                '有的同学对语言很有兴趣。'\n            ]\n\n        \"\"\"\n        response = self._send_post_json(self._url + '/grammatical_error_correction',\n                                        {'text': text,\n                                         'language': language or self._language})\n        return response\n\n    def text_classification(self, text: Union[str, List[str]], model, topk=False, prob=False) -> Union[\n        str, Dict[str, float], List[Union[str, Dict[str, float]]]]:\n        \"\"\"\n        Text classification is the task of assigning a sentence or document an appropriate category.\n        The categories depend on the chosen dataset and can range from topics.\n\n        Args:\n            text: A document or a list of documents.\n            model: The model to use for prediction.\n            topk: ``True`` or ``int`` to return the top-k labels.\n            prob: Return also probabilities.\n\n        Returns:\n\n            Classification results.\n        \"\"\"\n        response = self._send_post_json(self._url + '/text_classification',\n                                        {'text': text, 'model': model, 'topk': topk, 'prob': prob})\n        return response\n\n    def sentiment_analysis(self, text: Union[str, List[str]], language=None) -> Union[float, List[float]]:\n        r\"\"\"\n        Sentiment analysis is the task of classifying the polarity of a given text. For instance,\n        a text-based tweet can be categorized into either \"positive\", \"negative\", or \"neutral\".\n\n        Args:\n            text: A document or a list of documents.\n            language (str): The default language for each :func:`~hanlp_restful.HanLPClient.parse` call.\n                Contact the service provider for the list of languages supported.\n                Conventionally, ``zh`` is used for Chinese and ``mul`` for multilingual.\n                Leave ``None`` to use the default language on server.\n\n        Returns:\n\n            Sentiment polarity as a numerical value which measures how positive the sentiment is.\n\n        Examples::\n\n            HanLP.language_identification('''“这是一部男人必看的电影。”人人都这么说。但单纯从性别区分，就会让这电影变狭隘。\n            《肖申克的救赎》突破了男人电影的局限，通篇几乎充满令人难以置信的温馨基调，而电影里最伟大的主题是“希望”。\n            当我们无奈地遇到了如同肖申克一般囚禁了心灵自由的那种囹圄，我们是无奈的老布鲁克，灰心的瑞德，还是智慧的安迪？\n            运用智慧，信任希望，并且勇敢面对恐惧心理，去打败它？\n            经典的电影之所以经典，因为他们都在做同一件事——让你从不同的角度来欣赏希望的美好。''')\n            0.9505730271339417\n        \"\"\"\n        response = self._send_post_json(self._url + '/sentiment_analysis',\n                                        {'text': text, 'language': language or self._language})\n        return response\n\n    def language_identification(self, text: Union[str, List[str]], topk=False, prob=False) -> Union[\n        str, Dict[str, float], List[Union[str, Dict[str, float]]]]:\n        \"\"\"\n        Identify the language of a given text.\n\n        Args:\n            text: A document or a list of documents.\n            topk: ``True`` or ``int`` to return the top-k languages.\n            prob: Return also probabilities.\n\n        Returns:\n\n            Identified language in `ISO 639-1 codes`_.\n\n        Examples::\n\n            HanLP.language_identification(\n            'In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques.')\n            'en'\n            lang, prob = HanLP.language_identification(\n            '2021年、HanLPv2.1は次世代の最先端多言語NLP技術を本番環境に導入します。', prob=True)\n            ('ja', 0.9976244568824768)\n            HanLP.language_identification(\n            '2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', topk=2)\n            ['zh', 'ja']\n            HanLP.language_identification(\n            '2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。', topk=3, prob=True)\n            {'zh': 0.3952908217906952, 'en': 0.37189167737960815, 'ja': 0.056213412433862686}\n\n        .. _ISO 639-1 codes:\n           https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes\n        \"\"\"\n        return self.text_classification(text, 'lid', topk, prob)\n"
  },
  {
    "path": "plugins/hanlp_restful/setup.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 19:26\nfrom os.path import abspath, join, dirname\nfrom setuptools import find_packages, setup\n\nthis_dir = abspath(dirname(__file__))\nwith open(join(this_dir, 'README.md'), encoding='utf-8') as file:\n    long_description = file.read()\n\nsetup(\n    name='hanlp_restful',\n    version='0.0.23',\n    description='HanLP: Han Language Processing',\n    long_description=long_description,\n    long_description_content_type=\"text/markdown\",\n    url='https://github.com/hankcs/HanLP',\n    author='hankcs',\n    author_email='hankcshe@gmail.com',\n    license='Apache License 2.0',\n    classifiers=[\n        'Intended Audience :: Science/Research',\n        'Intended Audience :: Developers',\n        \"Development Status :: 3 - Alpha\",\n        'Operating System :: OS Independent',\n        \"License :: OSI Approved :: Apache Software License\",\n        'Programming Language :: Python :: 3 :: Only',\n        'Topic :: Scientific/Engineering :: Artificial Intelligence',\n        \"Topic :: Text Processing :: Linguistic\"\n    ],\n    keywords='corpus,machine-learning,NLU,NLP',\n    packages=find_packages(exclude=['docs', 'tests*']),\n    include_package_data=True,\n    install_requires=[\n        'hanlp_common'\n    ],\n    python_requires='>=3.6',\n)\n"
  },
  {
    "path": "plugins/hanlp_restful/tests/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-11-29 18:05\n"
  },
  {
    "path": "plugins/hanlp_restful/tests/test_client.py",
    "content": "import unittest\n\nfrom hanlp_restful import HanLPClient\n\n\nclass TestClient(unittest.TestCase):\n\n    def setUp(self) -> None:\n        self.HanLP = HanLPClient('https://hanlp.hankcs.com/api', auth=None)  # Fill in your auth\n\n    def test_raw_text(self):\n        text = '2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。阿婆主来到北京立方庭参观自然语义科技公司。'\n        doc = self.HanLP.parse(text)\n\n    def test_sents(self):\n        text = ['2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。',\n                '阿婆主来到北京立方庭参观自然语义科技公司。']\n        doc = self.HanLP(text)\n\n    def test_tokens(self):\n        tokens = [\n            [\"2021年\", \"HanLPv2.1\", \"为\", \"生产\", \"环境\", \"带来\", \"次\", \"世代\", \"最\", \"先进\", \"的\", \"多语种\", \"NLP\", \"技术\", \"。\"],\n            [\"英\", \"首相\", \"与\", \"特朗普\", \"通\", \"电话\", \"讨论\", \"华为\", \"与\", \"苹果\", \"公司\", \"。\"]\n        ]\n        doc = self.HanLP(tokens=tokens, tasks=['ner*', 'srl', 'dep'])\n\n    def test_sents_mul(self):\n        text = ['In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques to production environment.',\n                '2021年、HanLPv2.1は次世代の最先端多言語NLP技術を本番環境に導入します。',\n                '2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。']\n        doc = self.HanLP.parse(text, language='mul')\n\n    def test_tokenize(self):\n        print(self.HanLP.tokenize('商品和服务。阿婆主来到北京立方庭参观自然语义科技公司'))\n        print(self.HanLP.tokenize('商品和服务。阿婆主来到北京立方庭参观自然语义科技公司', coarse=True))\n        print(self.HanLP.tokenize(['商品和服务。', '当下雨天地面积水分外严重']))\n        print(self.HanLP.tokenize(\n            ['In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques to production environment.',\n             '2021年、HanLPv2.1は次世代の最先端多言語NLP技術を本番環境に導入します。',\n             '2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。'], language='mul'))\n\n    def test_coreference_resolution(self):\n        print(self.HanLP.coreference_resolution('我姐送我她的猫。我很喜欢它。'))\n\n    def test_text_style_transfer(self):\n        print(self.HanLP.text_style_transfer('国家对中石油抱有很大的期望.', target_style='gov_doc'))\n        print(self.HanLP.text_style_transfer('打工人，打工魂，打工都是人上人', target_style='gov_doc'))\n        print(self.HanLP.text_style_transfer('我看到了窗户外面有白色的云和绿色的森林', target_style='modern_poetry'))\n\n    def test_abstract_meaning_representation(self):\n        print(self.HanLP.abstract_meaning_representation('男孩希望女孩相信他。'))\n        print(self.HanLP.abstract_meaning_representation('男孩希望女孩相信他。', visualization='dot'))\n        print(self.HanLP.abstract_meaning_representation('男孩希望女孩相信他。', visualization='svg'))\n        print(self.HanLP.abstract_meaning_representation(tokens=[['男孩', '希望', '女孩', '相信', '他', '。']]))\n        print(self.HanLP.abstract_meaning_representation('The boy wants the girl to believe him.', language='en'))\n\n    def test_keyphrase_extraction(self):\n        print(self.HanLP.keyphrase_extraction(\n            '自然语言处理是一门博大精深的学科，掌握理论才能发挥出HanLP的全部性能。 '\n            '《自然语言处理入门》是一本配套HanLP的NLP入门书，助你零起点上手自然语言处理。', topk=3))\n\n    def test_extractive_summarization(self):\n        text = '''\n        据DigiTimes报道，在上海疫情趋缓，防疫管控开始放松后，苹果供应商广达正在逐步恢复其中国工厂的MacBook产品生产。\n        据供应链消息人士称，生产厂的订单拉动情况正在慢慢转强，这会提高MacBook Pro机型的供应量，并缩短苹果客户在过去几周所经历的延长交货时间。\n        仍有许多苹果笔记本用户在等待3月和4月订购的MacBook Pro机型到货，由于苹果的供应问题，他们的发货时间被大大推迟了。\n        据分析师郭明錤表示，广达是高端MacBook Pro的唯一供应商，自防疫封控依赖，MacBook Pro大部分型号交货时间增加了三到五周，\n        一些高端定制型号的MacBook Pro配置要到6月底到7月初才能交货。\n        尽管MacBook Pro的生产逐渐恢复，但供应问题预计依然影响2022年第三季度的产品销售。\n        苹果上周表示，防疫措施和元部件短缺将继续使其难以生产足够的产品来满足消费者的强劲需求，这最终将影响苹果6月份的收入。\n            '''\n        print(self.HanLP.extractive_summarization(text))\n\n\nif __name__ == '__main__':\n    unittest.main()\n"
  },
  {
    "path": "plugins/hanlp_restful_golang/README.md",
    "content": "# gohanlp\n\nGolang RESTful Client for HanLP\n\nWe have moved to https://github.com/hankcs/gohanlp"
  },
  {
    "path": "plugins/hanlp_restful_java/pom.xml",
    "content": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<project xmlns=\"http://maven.apache.org/POM/4.0.0\"\n         xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\n         xsi:schemaLocation=\"http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd\">\n    <modelVersion>4.0.0</modelVersion>\n\n    <groupId>com.hankcs.hanlp.restful</groupId>\n    <artifactId>hanlp-restful</artifactId>\n    <version>0.0.15</version>\n\n    <name>HanLP RESTful Client in Java</name>\n    <url>https://github.com/hankcs/HanLP</url>\n    <description>\n        HanLP: Han Language Processing\n    </description>\n    <organization>\n        <name>hankcs</name>\n        <url>http://www.hankcs.com/</url>\n    </organization>\n    <licenses>\n        <license>\n            <name>Apache License Version 2.0</name>\n            <url>https://www.apache.org/licenses/LICENSE-2.0.html</url>\n        </license>\n    </licenses>\n    <inceptionYear>2020</inceptionYear>\n    <developers>\n        <developer>\n            <name>hankcs</name>\n            <email>cnhankmc@gmail.com</email>\n            <url>http://www.hankcs.com</url>\n        </developer>\n    </developers>\n    <scm>\n        <connection>scm:git@github.com:hankcs/HanLP.git</connection>\n        <developerConnection>scm:git@github.com:hankcs/HanLP.git</developerConnection>\n        <url>git@github.com:hankcs/HanLP.git</url>\n    </scm>\n\n    <build>\n        <plugins>\n            <plugin>\n                <groupId>org.apache.maven.plugins</groupId>\n                <artifactId>maven-compiler-plugin</artifactId>\n                <configuration>\n                    <source>8</source>\n                    <target>8</target>\n                </configuration>\n            </plugin>\n            <plugin>\n                <artifactId>maven-source-plugin</artifactId>\n                <version>2.4</version>\n                <executions>\n                    <execution>\n                        <id>attach-sources</id>\n                        <goals>\n                            <goal>jar</goal>\n                        </goals>\n                    </execution>\n                </executions>\n            </plugin>\n            <plugin>\n                <groupId>org.apache.maven.plugins</groupId>\n                <artifactId>maven-javadoc-plugin</artifactId>\n                <version>2.9.1</version>\n                <executions>\n                    <execution>\n                        <phase>package</phase>\n                        <goals>\n                            <goal>jar</goal>\n                        </goals>\n                    </execution>\n                </executions>\n            </plugin>\n            <plugin>\n                <groupId>org.apache.maven.plugins</groupId>\n                <artifactId>maven-gpg-plugin</artifactId>\n                <version>1.6</version>\n                <executions>\n                    <execution>\n                        <phase>verify</phase>\n                        <goals>\n                            <goal>sign</goal>\n                        </goals>\n                        <configuration>\n                            <!-- This is necessary for gpg to not try to use the pinentry programs -->\n                            <gpgArguments>\n                                <arg>--pinentry-mode</arg>\n                                <arg>loopback</arg>\n                            </gpgArguments>\n                        </configuration>\n                    </execution>\n                </executions>\n            </plugin>\n        </plugins>\n    </build>\n\n    <dependencies>\n        <dependency>\n            <groupId>com.fasterxml.jackson.core</groupId>\n            <artifactId>jackson-databind</artifactId>\n            <version>2.14.1</version>\n        </dependency>\n        <dependency>\n            <groupId>org.junit.jupiter</groupId>\n            <artifactId>junit-jupiter</artifactId>\n            <version>RELEASE</version>\n            <scope>test</scope>\n        </dependency>\n    </dependencies>\n\n    <distributionManagement>\n        <snapshotRepository>\n            <id>maven-repo</id>\n            <url>https://oss.sonatype.org/content/repositories/snapshots/</url>\n        </snapshotRepository>\n        <repository>\n            <id>maven-repo</id>\n            <url>https://oss.sonatype.org/service/local/staging/deploy/maven2/</url>\n        </repository>\n    </distributionManagement>\n</project>"
  },
  {
    "path": "plugins/hanlp_restful_java/src/main/java/com/hankcs/hanlp/restful/BaseInput.java",
    "content": "/*\n * <author>Han He</author>\n * <email>me@hankcs.com</email>\n * <create-date>2020-12-27 12:07 AM</create-date>\n *\n * <copyright file=\"Input.java\">\n * Copyright (c) 2020, Han He. All Rights Reserved, http://www.hankcs.com/\n * See LICENSE file in the project root for full license information.\n * </copyright>\n */\npackage com.hankcs.hanlp.restful;\n\n/**\n * @author hankcs\n */\npublic class BaseInput\n{\n    public String[] tasks;\n    public String[] skip_tasks;\n    public String language;\n\n    public BaseInput(String[] tasks, String[] skipTasks, String language)\n    {\n        this.tasks = tasks;\n        this.skip_tasks = skipTasks;\n        this.language = language;\n    }\n}\n"
  },
  {
    "path": "plugins/hanlp_restful_java/src/main/java/com/hankcs/hanlp/restful/CoreferenceResolutionOutput.java",
    "content": "/*\n * <author>Han He</author>\n * <email>me@hankcs.com</email>\n * <create-date>2021-10-16 4:43 PM</create-date>\n *\n * <copyright file=\"CoreferenceResolutionOutput.java\">\n * Copyright (c) 2021, Han He. All Rights Reserved, http://www.hankcs.com/\n * See LICENSE file in the project root for full license information.\n * </copyright>\n */\npackage com.hankcs.hanlp.restful;\n\nimport java.util.ArrayList;\nimport java.util.List;\nimport java.util.Set;\n\n/**\n * A data class for coreference resolution\n *\n * @author hankcs\n */\npublic class CoreferenceResolutionOutput\n{\n    public List<Set<Span>> clusters;\n    public ArrayList<String> tokens;\n\n    public CoreferenceResolutionOutput(List<Set<Span>> clusters, ArrayList<String> tokens)\n    {\n        this.clusters = clusters;\n        this.tokens = tokens;\n    }\n}\n"
  },
  {
    "path": "plugins/hanlp_restful_java/src/main/java/com/hankcs/hanlp/restful/DocumentInput.java",
    "content": "/*\n * <author>Han He</author>\n * <email>me@hankcs.com</email>\n * <create-date>2020-12-27 12:09 AM</create-date>\n *\n * <copyright file=\"TextInput.java\">\n * Copyright (c) 2020, Han He. All Rights Reserved, http://www.hankcs.com/\n * See LICENSE file in the project root for full license information.\n * </copyright>\n */\npackage com.hankcs.hanlp.restful;\n\n/**\n * @author hankcs\n */\npublic class DocumentInput extends BaseInput\n{\n    public String text;\n\n    public DocumentInput(String text, String[] tasks, String[] skipTasks, String language)\n    {\n        super(tasks, skipTasks, language);\n        this.text = text;\n    }\n}\n"
  },
  {
    "path": "plugins/hanlp_restful_java/src/main/java/com/hankcs/hanlp/restful/HanLPClient.java",
    "content": "/*\n * <author>Han He</author>\n * <email>me@hankcs.com</email>\n * <create-date>2020-12-26 11:54 PM</create-date>\n *\n * <copyright file=\"HanLPClient.java\">\n * Copyright (c) 2020, Han He. All Rights Reserved, http://www.hankcs.com/\n * See LICENSE file in the project root for full license information.\n * </copyright>\n */\npackage com.hankcs.hanlp.restful;\n\n\nimport com.fasterxml.jackson.databind.ObjectMapper;\nimport com.hankcs.hanlp.restful.mrp.MeaningRepresentation;\n\nimport java.io.BufferedReader;\nimport java.io.IOException;\nimport java.io.InputStreamReader;\nimport java.io.OutputStream;\nimport java.net.HttpURLConnection;\nimport java.net.URL;\nimport java.nio.charset.StandardCharsets;\nimport java.util.*;\n\n/**\n * A RESTful client implementing the data format specification of HanLP.\n *\n * @author hankcs\n * @see <a href=\"https://hanlp.hankcs.com/docs/data_format.html\">Data Format</a>\n */\npublic class HanLPClient\n{\n    private String url;\n    private String auth;\n    private String language;\n    private int timeout;\n    private ObjectMapper mapper;\n\n    /**\n     * @param url      An API endpoint to a service provider.\n     * @param auth     An auth key licenced by a service provider.\n     * @param language The language this client will be expecting. Contact the service provider for the list of\n     *                 languages supported. Conventionally, zh is used for Chinese and mul for multilingual.\n     *                 Leave null to use the default language on server.\n     * @param timeout  Maximum waiting time in seconds for a request.\n     */\n    public HanLPClient(String url, String auth, String language, int timeout)\n    {\n        if (auth == null)\n        {\n            auth = System.getenv().getOrDefault(\"HANLP_AUTH\", null);\n        }\n        this.url = url;\n        this.auth = auth;\n        this.language = language;\n        this.timeout = timeout * 1000;\n        this.mapper = new ObjectMapper();\n    }\n\n    /**\n     * @param url  An API endpoint to a service provider.\n     * @param auth An auth key licenced by a service provider.\n     */\n    public HanLPClient(String url, String auth)\n    {\n        this(url, auth, null, 5);\n    }\n\n    /**\n     * Parse a raw document.\n     *\n     * @param text      Document content which can have multiple sentences.\n     * @param tasks     Tasks to perform.\n     * @param skipTasks Tasks to skip.\n     * @return Parsed annotations.\n     * @throws IOException HTTP exception.\n     * @see <a href=\"https://hanlp.hankcs.com/docs/data_format.html\">Data Format</a>\n     */\n    public Map<String, List> parse(String text, String[] tasks, String[] skipTasks) throws IOException\n    {\n        //noinspection unchecked\n        return mapper.readValue(post(\"/parse\", new DocumentInput(text, tasks, skipTasks, language)), Map.class);\n    }\n\n    /**\n     * Parse a raw document.\n     *\n     * @param text Document content which can have multiple sentences.\n     * @return Parsed annotations.\n     * @throws IOException HTTP exception.\n     * @see <a href=\"https://hanlp.hankcs.com/docs/data_format.html\">Data Format</a>\n     */\n    public Map<String, List> parse(String text) throws IOException\n    {\n        return parse(text, null, null);\n    }\n\n    /**\n     * Parse an array of sentences.\n     *\n     * @param sentences Multiple sentences to parse.\n     * @param tasks     Tasks to perform.\n     * @param skipTasks Tasks to skip.\n     * @return Parsed annotations.\n     * @throws IOException HTTP exception.\n     * @see <a href=\"https://hanlp.hankcs.com/docs/data_format.html\">Data Format</a>\n     */\n    public Map<String, List> parse(String[] sentences, String[] tasks, String[] skipTasks) throws IOException\n    {\n        //noinspection unchecked\n        return mapper.readValue(post(\"/parse\", new SentenceInput(sentences, tasks, skipTasks, language)), Map.class);\n    }\n\n    /**\n     * Parse an array of sentences.\n     *\n     * @param sentences Multiple sentences to parse.\n     * @return Parsed annotations.\n     * @throws IOException HTTP exception.\n     * @see <a href=\"https://hanlp.hankcs.com/docs/data_format.html\">Data Format</a>\n     */\n    public Map<String, List> parse(String[] sentences) throws IOException\n    {\n        return parse(sentences, null, null);\n    }\n\n    /**\n     * Parse an array of pre-tokenized sentences.\n     *\n     * @param tokens    Multiple pre-tokenized sentences to parse.\n     * @param tasks     Tasks to perform.\n     * @param skipTasks Tasks to skip.\n     * @return Parsed annotations.\n     * @throws IOException HTTP exception.\n     * @see <a href=\"https://hanlp.hankcs.com/docs/data_format.html\">Data Format</a>\n     */\n    public Map<String, List> parse(String[][] tokens, String[] tasks, String[] skipTasks) throws IOException\n    {\n        //noinspection unchecked\n        return mapper.readValue(post(\"/parse\", new TokenInput(tokens, tasks, skipTasks, language)), Map.class);\n    }\n\n    /**\n     * Parse an array of pre-tokenized sentences.\n     *\n     * @param tokens Multiple pre-tokenized sentences to parse.\n     * @return Parsed annotations.\n     * @throws IOException HTTP exception.\n     * @see <a href=\"https://hanlp.hankcs.com/docs/data_format.html\">Data Format</a>\n     */\n    public Map<String, List> parse(String[][] tokens) throws IOException\n    {\n        return parse(tokens, null, null);\n    }\n\n    /**\n     * Split a document into sentences and tokenize them.\n     *\n     * @param text   A document.\n     * @param coarse Whether to perform coarse-grained or fine-grained tokenization.\n     * @return A list of tokenized sentences.\n     * @throws IOException HTTP exception.\n     */\n    public List<List<String>> tokenize(String text, Boolean coarse) throws IOException\n    {\n        String[] tasks;\n        if (coarse != null)\n        {\n            if (coarse)\n                tasks = new String[]{\"tok/coarse\"};\n            else\n                tasks = new String[]{\"tok/fine\"};\n        }\n        else\n            tasks = new String[]{\"tok\"};\n        Map<String, List> doc = parse(text, tasks, null);\n        //noinspection unchecked\n        return doc.values().iterator().next();\n    }\n\n    /**\n     * Split a document into sentences and tokenize them using fine-grained standard.\n     *\n     * @param text A document.\n     * @return A list of tokenized sentences.\n     * @throws IOException HTTP exception.\n     */\n    public List<List<String>> tokenize(String text) throws IOException\n    {\n        return tokenize(text, null);\n    }\n\n    /**\n     * Text style transfer aims to change the style of the input text to the target style while preserving its content.\n     *\n     * @param text        Source text.\n     * @param targetStyle Target style.\n     * @return Text of the target style.\n     */\n    public List<String> textStyleTransfer(List<String> text, String targetStyle) throws IOException\n    {\n        Map<String, Object> input = new HashMap<>();\n        input.put(\"text\", text);\n        input.put(\"target_style\", targetStyle);\n        input.put(\"language\", language);\n        //noinspection unchecked\n        return mapper.readValue(post(\"/text_style_transfer\", input), List.class);\n    }\n\n    /**\n     * Text style transfer aims to change the style of the input text to the target style while preserving its content.\n     *\n     * @param text        Source text.\n     * @param targetStyle Target style.\n     * @return Text of the target style.\n     */\n    public String textStyleTransfer(String text, String targetStyle) throws IOException\n    {\n        Map<String, Object> input = new HashMap<>();\n        input.put(\"text\", text);\n        input.put(\"target_style\", targetStyle);\n        input.put(\"language\", language);\n        return mapper.readValue(post(\"/text_style_transfer\", input), String.class);\n    }\n\n    /**\n     * Grammatical Error Correction (GEC) is the task of correcting different kinds of errors in text such as\n     * spelling, punctuation, grammatical, and word choice errors.\n     *\n     * @param text Text potentially containing different kinds of errors such as spelling, punctuation,\n     *             grammatical, and word choice errors.\n     * @return Corrected text.\n     */\n    public List<String> grammaticalErrorCorrection(List<String> text) throws IOException\n    {\n        Map<String, Object> input = new HashMap<>();\n        input.put(\"text\", text);\n        input.put(\"language\", language);\n        //noinspection unchecked\n        return mapper.readValue(post(\"/grammatical_error_correction\", input), List.class);\n    }\n\n    /**\n     * Grammatical Error Correction (GEC) is the task of correcting different kinds of errors in text such as\n     * spelling, punctuation, grammatical, and word choice errors.\n     *\n     * @param text Text potentially containing different kinds of errors such as spelling, punctuation,\n     *             grammatical, and word choice errors.\n     * @return Corrected text.\n     */\n    public String[] grammaticalErrorCorrection(String[] text) throws IOException\n    {\n        Map<String, Object> input = new HashMap<>();\n        input.put(\"text\", text);\n        input.put(\"language\", language);\n        //noinspection unchecked\n        return mapper.readValue(post(\"/grammatical_error_correction\", input), String[].class);\n    }\n\n    /**\n     * Grammatical Error Correction (GEC) is the task of correcting different kinds of errors in text such as\n     * spelling, punctuation, grammatical, and word choice errors.\n     *\n     * @param text Text potentially containing different kinds of errors such as spelling, punctuation,\n     *             grammatical, and word choice errors.\n     * @return Corrected text.\n     */\n    public String grammaticalErrorCorrection(String text) throws IOException\n    {\n        Map<String, Object> input = new HashMap<>();\n        input.put(\"text\", text);\n        input.put(\"language\", language);\n        return mapper.readValue(post(\"/grammatical_error_correction\", input), String.class);\n    }\n\n    /**\n     * Semantic textual similarity deals with determining how similar two pieces of texts are.\n     *\n     * @param textA The first text.\n     * @param textB The second text.\n     * @return Their similarity.\n     * @throws IOException HTTP errors.\n     */\n    public Double semanticTextualSimilarity(String textA, String textB) throws IOException\n    {\n        Map<String, Object> input = new HashMap<>();\n        input.put(\"text\", new String[]{textA, textB});\n        input.put(\"language\", language);\n        return mapper.readValue(post(\"/semantic_textual_similarity\", input), Double.class);\n    }\n\n    /**\n     * Semantic textual similarity deals with determining how similar two pieces of texts are.\n     *\n     * @param text The pairs of text.\n     * @return Their similarities.\n     * @throws IOException HTTP errors.\n     */\n    public List<Double> semanticTextualSimilarity(String[][] text) throws IOException\n    {\n        Map<String, Object> input = new HashMap<>();\n        input.put(\"text\", text);\n        input.put(\"language\", language);\n        //noinspection unchecked\n        return mapper.readValue(post(\"/semantic_textual_similarity\", input), List.class);\n    }\n\n    /**\n     * Coreference resolution is the task of clustering mentions in text that refer to the same underlying real world entities.\n     *\n     * @param text A piece of text, usually a document without tokenization.\n     * @return Coreference resolution clusters and tokens.\n     * @throws IOException HTTP errors.\n     */\n    public CoreferenceResolutionOutput coreferenceResolution(String text) throws IOException\n    {\n        Map<String, Object> input = new HashMap<>();\n        input.put(\"text\", text);\n        input.put(\"language\", language);\n        //noinspection unchecked\n        Map<String, List> response = mapper.readValue(post(\"/coreference_resolution\", input), Map.class);\n        //noinspection unchecked\n        List<List<List>> clusters = response.get(\"clusters\");\n        return new CoreferenceResolutionOutput(_convert_clusters(clusters), (ArrayList<String>) response.get(\"tokens\"));\n    }\n\n    /**\n     * Coreference resolution is the task of clustering mentions in text that refer to the same underlying real world entities.\n     *\n     * @param tokens   A list of sentences where each sentence is a list of tokens.\n     * @param speakers A list of speakers where each speaker is a String representing the speaker's ID, e.g., \"Tom\".\n     * @return Coreference resolution clusters.\n     * @throws IOException HTTP errors.\n     */\n    public List<Set<Span>> coreferenceResolution(String[][] tokens, String[] speakers) throws IOException\n    {\n        Map<String, Object> input = new HashMap<>();\n        input.put(\"tokens\", tokens);\n        input.put(\"speakers\", speakers);\n        input.put(\"language\", language);\n        //noinspection unchecked\n        List<List<List>> clusters = mapper.readValue(post(\"/coreference_resolution\", input), List.class);\n        return _convert_clusters(clusters);\n    }\n\n    /**\n     * Coreference resolution is the task of clustering mentions in text that refer to the same underlying real world entities.\n     *\n     * @param tokens A list of sentences where each sentence is a list of tokens.\n     * @return Coreference resolution clusters.\n     * @throws IOException HTTP errors.\n     */\n    public List<Set<Span>> coreferenceResolution(String[][] tokens) throws IOException\n    {\n        Map<String, Object> input = new HashMap<>();\n        input.put(\"tokens\", tokens);\n        input.put(\"language\", language);\n        //noinspection unchecked\n        List<List<List>> clusters = mapper.readValue(post(\"/coreference_resolution\", input), List.class);\n        return _convert_clusters(clusters);\n    }\n\n    private static List<Set<Span>> _convert_clusters(List<List<List>> clusters)\n    {\n        List<Set<Span>> results = new ArrayList<>(clusters.size());\n        for (List<List> cluster : clusters)\n        {\n            Set<Span> spans = new LinkedHashSet<>();\n            for (List span : cluster)\n            {\n                spans.add(new Span((String) span.get(0), (Integer) span.get(1), (Integer) span.get(2)));\n            }\n            results.add(spans);\n        }\n        return results;\n    }\n\n    /**\n     * Abstract Meaning Representation (AMR) captures “who is doing what to whom” in a sentence. Each sentence is\n     * represented as a rooted, directed, acyclic graph consisting of nodes (concepts) and edges (relations).\n     *\n     * @param text A piece of text, usually a document without tokenization.\n     * @return AMR graphs.\n     * @throws IOException HTTP errors.\n     */\n    public MeaningRepresentation[] abstractMeaningRepresentation(String text) throws IOException\n    {\n        Map<String, Object> input = new HashMap<>();\n        input.put(\"text\", text);\n        input.put(\"language\", language);\n        return mapper.readValue(post(\"/abstract_meaning_representation\", input), MeaningRepresentation[].class);\n    }\n\n    /**\n     * Abstract Meaning Representation (AMR) captures “who is doing what to whom” in a sentence. Each sentence is\n     * represented as a rooted, directed, acyclic graph consisting of nodes (concepts) and edges (relations).\n     *\n     * @param tokens A list of sentences where each sentence is a list of tokens.\n     * @return AMR graphs.\n     * @throws IOException HTTP errors.\n     */\n    public MeaningRepresentation[] abstractMeaningRepresentation(String[][] tokens) throws IOException\n    {\n        Map<String, Object> input = new HashMap<>();\n        input.put(\"tokens\", tokens);\n        input.put(\"language\", language);\n        return mapper.readValue(post(\"/abstract_meaning_representation\", input), MeaningRepresentation[].class);\n    }\n\n    /**\n     * Keyphrase extraction aims to identify keywords or phrases reflecting the main topics of a document.\n     *\n     * @param text The text content of the document. Preferably the concatenation of the title and the content.\n     * @param topk The number of top-K ranked keywords or keyphrases.\n     * @return A dictionary containing each keyphrase and its ranking score s between 0 and 1.\n     * @throws IOException HTTP errors.\n     */\n    public Map<String, Double> keyphraseExtraction(String text, int topk) throws IOException\n    {\n        Map<String, Object> input = new HashMap<>();\n        input.put(\"text\", text);\n        input.put(\"topk\", topk);\n        input.put(\"language\", language);\n        //noinspection unchecked\n        return mapper.readValue(post(\"/keyphrase_extraction\", input), LinkedHashMap.class);\n    }\n\n    /**\n     * Single document summarization is the task of selecting a subset of the sentences which best\n     * represents a summary of the document, with a balance of salience and redundancy.\n     *\n     * @param text The text content of the document.\n     * @return A dictionary containing each sentence and its ranking score s between 0 and 1.\n     * @throws IOException HTTP errors.\n     */\n    public Map<String, Double> extractiveSummarization(String text) throws IOException\n    {\n        return extractiveSummarization(text, 3);\n    }\n\n    /**\n     * Single document summarization is the task of selecting a subset of the sentences which best\n     * represents a summary of the document, with a balance of salience and redundancy.\n     *\n     * @param text The text content of the document.\n     * @param topk The maximum number of top-K ranked sentences. Note that due to Trigram Blocking tricks, the actual\n     *             number of returned sentences could be less than ``topk``.\n     * @return A dictionary containing each sentence and its ranking score s between 0 and 1.\n     * @throws IOException HTTP errors.\n     */\n    public Map<String, Double> extractiveSummarization(String text, int topk) throws IOException\n    {\n        Map<String, Object> input = new HashMap<>();\n        input.put(\"text\", text);\n        input.put(\"topk\", topk);\n        input.put(\"language\", language);\n        //noinspection unchecked\n        return mapper.readValue(post(\"/extractive_summarization\", input), LinkedHashMap.class);\n    }\n\n    /**\n     * Abstractive Summarization is the task of generating a short and concise summary that captures the\n     * salient ideas of the source text. The generated summaries potentially contain new phrases and sentences that\n     * may not appear in the source text.\n     *\n     * @param text The text content of the document.\n     * @return Summarization.\n     * @throws IOException HTTP errors.\n     */\n    public String abstractiveSummarization(String text) throws IOException\n    {\n        Map<String, Object> input = new HashMap<>();\n        input.put(\"text\", text);\n        input.put(\"language\", language);\n        //noinspection unchecked\n        return mapper.readValue(post(\"/abstractive_summarization\", input), String.class);\n    }\n\n    /**\n     * Text classification is the task of assigning a sentence or document an appropriate category.\n     * The categories depend on the chosen dataset and can range from topics.\n     *\n     * @param text  The text content of the document.\n     * @param model The model to use for prediction.\n     * @return Classification results.\n     * @throws IOException HTTP errors.\n     */\n    public String textClassification(String text, String model) throws IOException\n    {\n        return (String) textClassification(text, model, false, false);\n    }\n\n\n    /**\n     * Sentiment analysis is the task of classifying the polarity of a given text. For instance,\n     * a text-based tweet can be categorized into either \"positive\", \"negative\", or \"neutral\".\n     *\n     * @param text The text content of the document.\n     * @return Sentiment polarity as a numerical value which measures how positive the sentiment is.\n     * @throws IOException HTTP errors.\n     */\n    public Double sentimentAnalysis(String text) throws IOException\n    {\n        Map<String, Object> input = new HashMap<>();\n        input.put(\"text\", text);\n        input.put(\"language\", language);\n        //noinspection unchecked\n        return mapper.readValue(post(\"/sentiment_analysis\", input), Double.class);\n    }\n\n\n    /**\n     * Text classification is the task of assigning a sentence or document an appropriate category.\n     * The categories depend on the chosen dataset and can range from topics.\n     *\n     * @param text  A document or a list of documents.\n     * @param model The model to use for prediction.\n     * @param topk  `true` or `int` to return the top-k languages.\n     * @param prob  Return also probabilities.\n     * @return Classification results.\n     * @throws IOException HTTP errors.\n     */\n    public Object textClassification(Object text, String model, Object topk, boolean prob) throws IOException\n    {\n        Map<String, Object> input = new HashMap<>();\n        input.put(\"text\", text);\n        input.put(\"model\", model);\n        input.put(\"topk\", topk);\n        input.put(\"prob\", prob);\n        //noinspection unchecked\n        return mapper.readValue(post(\"/text_classification\", input), Object.class);\n    }\n\n    /**\n     * Recognize the language of a given text.\n     *\n     * @param text The text content of the document.\n     * @return Identified language in <a href=\"https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes\">ISO 639-1 codes</a>.\n     * @throws IOException HTTP errors.\n     */\n    public String languageIdentification(String text) throws IOException\n    {\n        return textClassification(text, \"lid\");\n    }\n\n    /**\n     * Recognize the language of a given text.\n     *\n     * @param text The text content of the document.\n     * @return Identified language in <a href=\"https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes\">ISO 639-1 codes</a>.\n     * @throws IOException HTTP errors.\n     */\n    public List<String> languageIdentification(String[] text) throws IOException\n    {\n        return (List<String>) textClassification(text, \"lid\", false, false);\n    }\n\n    /**\n     * Keyphrase extraction aims to identify keywords or phrases reflecting the main topics of a document.\n     *\n     * @param text The text content of the document. Preferably the concatenation of the title and the content.\n     * @return A dictionary containing 10 keyphrases and their ranking scores s between 0 and 1.\n     * @throws IOException HTTP errors.\n     */\n    public Map<String, Double> keyphraseExtraction(String text) throws IOException\n    {\n        return keyphraseExtraction(text, 10);\n    }\n\n    private String post(String api, Object input_) throws IOException\n    {\n        URL url = new URL(this.url + api);\n\n        HttpURLConnection con = (HttpURLConnection) url.openConnection();\n        con.setRequestMethod(\"POST\");\n        if (auth != null)\n            con.setRequestProperty(\"Authorization\", \"Basic \" + auth);\n        con.setRequestProperty(\"Content-Type\", \"application/json; utf-8\");\n        con.setRequestProperty(\"Accept\", \"application/json\");\n        con.setDoOutput(true);\n        con.setConnectTimeout(timeout);\n        con.setReadTimeout(timeout);\n\n        String jsonInputString = mapper.writeValueAsString(input_);\n\n        try (OutputStream os = con.getOutputStream())\n        {\n            byte[] input = jsonInputString.getBytes(StandardCharsets.UTF_8);\n            os.write(input, 0, input.length);\n        }\n\n        int code = con.getResponseCode();\n        if (code != 200)\n        {\n            StringBuilder response = new StringBuilder();\n            try (BufferedReader br = new BufferedReader(new InputStreamReader(con.getErrorStream(), StandardCharsets.UTF_8)))\n            {\n                String responseLine;\n                while ((responseLine = br.readLine()) != null)\n                {\n                    response.append(responseLine.trim());\n                }\n            }\n            String error = String.format(\"Request failed, status code = %d, error = %s\", code, con.getResponseMessage());\n            try\n            {\n                Map detail = mapper.readValue(response.toString(), Map.class);\n                error = (String) detail.get(\"detail\");\n            }\n            catch (Exception ignored)\n            {\n            }\n            throw new IOException(error);\n        }\n\n        StringBuilder response = new StringBuilder();\n        try (BufferedReader br = new BufferedReader(new InputStreamReader(con.getInputStream(), StandardCharsets.UTF_8)))\n        {\n            String responseLine;\n            while ((responseLine = br.readLine()) != null)\n            {\n                response.append(responseLine.trim());\n            }\n        }\n        return response.toString();\n    }\n\n}\n"
  },
  {
    "path": "plugins/hanlp_restful_java/src/main/java/com/hankcs/hanlp/restful/SentenceInput.java",
    "content": "/*\n * <author>Han He</author>\n * <email>me@hankcs.com</email>\n * <create-date>2020-12-27 12:09 AM</create-date>\n *\n * <copyright file=\"SentenceInput.java\">\n * Copyright (c) 2020, Han He. All Rights Reserved, http://www.hankcs.com/\n * See LICENSE file in the project root for full license information.\n * </copyright>\n */\npackage com.hankcs.hanlp.restful;\n\n/**\n * @author hankcs\n */\npublic class SentenceInput extends BaseInput\n{\n    public String[] text;\n\n    public SentenceInput(String[] text, String[] tasks, String[] skipTasks, String language)\n    {\n        super(tasks, skipTasks, language);\n        this.text = text;\n    }\n}\n"
  },
  {
    "path": "plugins/hanlp_restful_java/src/main/java/com/hankcs/hanlp/restful/Span.java",
    "content": "/*\n * <author>Han He</author>\n * <email>me@hankcs.com</email>\n * <create-date>2021-10-16 4:26 PM</create-date>\n *\n * <copyright file=\"Span.java\">\n * Copyright (c) 2021, Han He. All Rights Reserved, http://www.hankcs.com/\n * See LICENSE file in the project root for full license information.\n * </copyright>\n */\npackage com.hankcs.hanlp.restful;\n\nimport java.util.Objects;\n\n/**\n * A common data format to represent a span.\n *\n * @author hankcs\n */\npublic class Span\n{\n    /**\n     * The raw form of a span, which can be either a token, an entity or a mention etc.\n     */\n    public String form;\n    /**\n     * The inclusive beginning offset of a span.\n     */\n    public int begin;\n    /**\n     * The exclusive ending offset of a span.\n     */\n    public int end;\n\n    public Span(String form, int begin, int end)\n    {\n        this.form = form;\n        this.begin = begin;\n        this.end = end;\n    }\n\n    @Override\n    public boolean equals(Object o)\n    {\n        if (this == o) return true;\n        if (o == null || getClass() != o.getClass()) return false;\n        Span span = (Span) o;\n        return begin == span.begin &&\n                end == span.end &&\n                form.equals(span.form);\n    }\n\n    @Override\n    public int hashCode()\n    {\n        return Objects.hash(form, begin, end);\n    }\n\n    @Override\n    public String toString()\n    {\n        return String.format(\"[%d, %d) = %s\", begin, end, form);\n    }\n}\n"
  },
  {
    "path": "plugins/hanlp_restful_java/src/main/java/com/hankcs/hanlp/restful/TokenInput.java",
    "content": "/*\n * <author>Han He</author>\n * <email>me@hankcs.com</email>\n * <create-date>2020-12-27 12:09 AM</create-date>\n *\n * <copyright file=\"TokenInput.java\">\n * Copyright (c) 2020, Han He. All Rights Reserved, http://www.hankcs.com/\n * See LICENSE file in the project root for full license information.\n * </copyright>\n */\npackage com.hankcs.hanlp.restful;\n\n/**\n * @author hankcs\n */\npublic class TokenInput extends BaseInput\n{\n    public String[][] tokens;\n\n    public TokenInput(String[][] tokens, String[] tasks, String[] skipTasks, String language)\n    {\n        super(tasks, skipTasks, language);\n        this.tokens = tokens;\n    }\n}\n"
  },
  {
    "path": "plugins/hanlp_restful_java/src/main/java/com/hankcs/hanlp/restful/mrp/Anchor.java",
    "content": "/*\n * <author>Han He</author>\n * <email>me@hankcs.com</email>\n * <create-date>2022-04-13 8:58 AM</create-date>\n *\n * <copyright file=\"Anchor.java\">\n * Copyright (c) 2022, Han He. All Rights Reserved, http://www.hankcs.com/\n * See LICENSE file in the project root for full license information.\n * </copyright>\n */\npackage com.hankcs.hanlp.restful.mrp;\n\n/**\n * @author hankcs\n */\npublic class Anchor\n{\n    public String from;\n    public String to;\n}\n"
  },
  {
    "path": "plugins/hanlp_restful_java/src/main/java/com/hankcs/hanlp/restful/mrp/Edge.java",
    "content": "/*\n * <author>Han He</author>\n * <email>me@hankcs.com</email>\n * <create-date>2022-04-13 9:01 AM</create-date>\n *\n * <copyright file=\"Edge.java\">\n * Copyright (c) 2022, Han He. All Rights Reserved, http://www.hankcs.com/\n * See LICENSE file in the project root for full license information.\n * </copyright>\n */\npackage com.hankcs.hanlp.restful.mrp;\n\n/**\n * @author hankcs\n */\npublic class Edge\n{\n    public int source;\n    public int target;\n    public String label;\n}\n"
  },
  {
    "path": "plugins/hanlp_restful_java/src/main/java/com/hankcs/hanlp/restful/mrp/MeaningRepresentation.java",
    "content": "/*\n * <author>Han He</author>\n * <email>me@hankcs.com</email>\n * <create-date>2022-04-13 8:57 AM</create-date>\n *\n * <copyright file=\"MeaningRepresentation.java\">\n * Copyright (c) 2022, Han He. All Rights Reserved, http://www.hankcs.com/\n * See LICENSE file in the project root for full license information.\n * </copyright>\n */\npackage com.hankcs.hanlp.restful.mrp;\n\n/**\n * Graph-based meaning representation.\n *\n * @author hankcs\n */\npublic class MeaningRepresentation\n{\n    public String id;\n    public String input;\n    public Node[] nodes;\n    public Edge[] edges;\n    public String[] tops;\n    public String framework;\n}\n"
  },
  {
    "path": "plugins/hanlp_restful_java/src/main/java/com/hankcs/hanlp/restful/mrp/Node.java",
    "content": "/*\n * <author>Han He</author>\n * <email>me@hankcs.com</email>\n * <create-date>2022-04-13 8:57 AM</create-date>\n *\n * <copyright file=\"Node.java\">\n * Copyright (c) 2022, Han He. All Rights Reserved, http://www.hankcs.com/\n * See LICENSE file in the project root for full license information.\n * </copyright>\n */\npackage com.hankcs.hanlp.restful.mrp;\n\n/**\n * @author hankcs\n */\npublic class Node\n{\n    public int id;\n    public String label;\n    public String[] properties;\n    public String[] values;\n    public Anchor[] anchors;\n}\n"
  },
  {
    "path": "plugins/hanlp_restful_java/src/test/java/com/hankcs/hanlp/restful/HanLPClientTest.java",
    "content": "package com.hankcs.hanlp.restful;\n\nimport com.fasterxml.jackson.core.JsonProcessingException;\nimport com.fasterxml.jackson.databind.ObjectMapper;\nimport org.junit.jupiter.api.BeforeEach;\nimport org.junit.jupiter.api.Test;\n\nimport java.io.IOException;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\n\nclass HanLPClientTest\n{\n    HanLPClient client;\n\n    @BeforeEach\n    void setUp()\n    {\n        client = new HanLPClient(\"https://hanlp.hankcs.com/api\", null);\n    }\n\n    @org.junit.jupiter.api.Test\n    void parseText() throws IOException\n    {\n        Map<String, List> doc = client.parse(\"2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。英首相与特朗普通电话讨论华为与苹果公司。\");\n        prettyPrint(doc);\n    }\n\n    @org.junit.jupiter.api.Test\n    void parseSentences() throws IOException\n    {\n        Map<String, List> doc = client.parse(new String[]{\n                \"2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。\",\n                \"英首相与特朗普通电话讨论华为与苹果公司。\"\n        });\n        prettyPrint(doc);\n    }\n\n    @org.junit.jupiter.api.Test\n    void parseTokens() throws IOException\n    {\n        Map<String, List> doc = client.parse(new String[][]{\n                new String[]{\"2021年\", \"HanLPv2.1\", \"为\", \"生产\", \"环境\", \"带来\", \"次\", \"世代\", \"最\", \"先进\", \"的\", \"多语种\", \"NLP\", \"技术\", \"。\"},\n                new String[]{\"英\", \"首相\", \"与\", \"特朗普\", \"通\", \"电话\", \"讨论\", \"华为\", \"与\", \"苹果\", \"公司\", \"。\"},\n        });\n        prettyPrint(doc);\n    }\n\n    @Test\n    void parseCoarse() throws IOException\n    {\n        Map<String, List> doc = client.parse(\n                \"阿婆主来到北京立方庭参观自然语义科技公司。\",\n                new String[]{\"tok/coarse\", \"pos\", \"dep\"},\n                new String[]{\"tok/fine\"});\n        prettyPrint(doc);\n    }\n\n    @Test\n    void tokenize() throws IOException\n    {\n        List<List<String>> fine = client.tokenize(\"2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。阿婆主来到北京立方庭参观自然语义科技公司。\");\n        System.out.println(fine);\n        List<List<String>> coarse = client.tokenize(\"2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。阿婆主来到北京立方庭参观自然语义科技公司。\", true);\n        System.out.println(coarse);\n    }\n\n    @Test\n    void textStyleTransfer() throws IOException\n    {\n        String doc = client.textStyleTransfer(\"国家对中石油抱有很大的期望.\", \"gov_doc\");\n        prettyPrint(doc);\n    }\n\n    @Test\n    void semanticTextualSimilarity() throws IOException\n    {\n        Double similarity = client.semanticTextualSimilarity(\"看图猜一电影名\", \"看图猜电影\");\n        prettyPrint(similarity);\n        List<Double> similarities = client.semanticTextualSimilarity(new String[][]{\n                new String[]{\"看图猜一电影名\", \"看图猜电影\"},\n                new String[]{\"北京到上海的动车票\", \"上海到北京的动车票\"}\n        });\n        for (Double similarityPerPair : similarities)\n        {\n            prettyPrint(similarityPerPair);\n        }\n    }\n\n    @Test\n    void coreferenceResolutionText() throws IOException\n    {\n        CoreferenceResolutionOutput clusters = client.coreferenceResolution(\"我姐送我她的猫。我很喜欢它。\");\n        prettyPrint(clusters);\n    }\n\n    @Test\n    void coreferenceResolutionTokens() throws IOException\n    {\n        List<Set<Span>> clusters = client.coreferenceResolution(\n                new String[][]{\n                        new String[]{\"我\", \"姐\", \"送\", \"我\", \"她\", \"的\", \"猫\", \"。\"},\n                        new String[]{\"我\", \"很\", \"喜欢\", \"它\", \"。\"}});\n        prettyPrint(clusters);\n    }\n\n    @Test\n    void coreferenceResolutionTokensWithSpeakers() throws IOException\n    {\n        List<Set<Span>> clusters = client.coreferenceResolution(\n                new String[][]{\n                        new String[]{\"我\", \"姐\", \"送\", \"我\", \"她\", \"的\", \"猫\", \"。\"},\n                        new String[]{\"我\", \"很\", \"喜欢\", \"它\", \"。\"}},\n                new String[]{\"张三\", \"张三\"});\n        prettyPrint(clusters);\n    }\n\n    @Test\n    void keyphraseExtraction() throws IOException\n    {\n        prettyPrint(client.keyphraseExtraction(\n                \"自然语言处理是一门博大精深的学科，掌握理论才能发挥出HanLP的全部性能。\" +\n                        \"《自然语言处理入门》是一本配套HanLP的NLP入门书，助你零起点上手自然语言处理。\", 3));\n    }\n\n    @Test\n    void extractiveSummarization() throws IOException\n    {\n        prettyPrint(client.extractiveSummarization(\n                \"据DigiTimes报道，在上海疫情趋缓，防疫管控开始放松后，苹果供应商广达正在逐步恢复其中国工厂的MacBook产品生产。\\n\" +\n                        \"据供应链消息人士称，生产厂的订单拉动情况正在慢慢转强，这会提高MacBook Pro机型的供应量，并缩短苹果客户在过去几周所经历的延长交货时间。\\n\" +\n                        \"仍有许多苹果笔记本用户在等待3月和4月订购的MacBook Pro机型到货，由于苹果的供应问题，他们的发货时间被大大推迟了。\\n\" +\n                        \"据分析师郭明錤表示，广达是高端MacBook Pro的唯一供应商，自防疫封控依赖，MacBook Pro大部分型号交货时间增加了三到五周，\\n\" +\n                        \"一些高端定制型号的MacBook Pro配置要到6月底到7月初才能交货。\\n\" +\n                        \"尽管MacBook Pro的生产逐渐恢复，但供应问题预计依然影响2022年第三季度的产品销售。\\n\" +\n                        \"苹果上周表示，防疫措施和元部件短缺将继续使其难以生产足够的产品来满足消费者的强劲需求，这最终将影响苹果6月份的收入。\"));\n    }\n\n    @Test\n    void abstractiveSummarization() throws IOException\n    {\n        prettyPrint(client.abstractiveSummarization(\n                \"每经AI快讯，2月4日，长江证券研究所金属行业首席分析师王鹤涛表示，2023年海外经济衰退，美债现处于历史高位，\\n\" +\n                        \"黄金的趋势是值得关注的；在国内需求修复的过程中，看好大金属品种中的铜铝钢。\\n\" +\n                        \"此外，在细分的小品种里，建议关注两条主线，一是新能源，比如锂、钴、镍、稀土，二是专精特新主线。（央视财经）\"));\n    }\n\n    @Test\n    void abstractMeaningRepresentationText() throws IOException\n    {\n        prettyPrint(client.abstractMeaningRepresentation(\"男孩希望女孩相信他。阿婆主来到北京立方庭参观自然语义科技公司。\"));\n    }\n\n    @Test\n    void abstractMeaningRepresentationTokens() throws IOException\n    {\n        prettyPrint(client.abstractMeaningRepresentation(new String[][]{\n                new String[]{\"2021年\", \"HanLPv2.1\", \"为\", \"生产\", \"环境\", \"带来\", \"次\", \"世代\", \"最\", \"先进\", \"的\", \"多语种\", \"NLP\", \"技术\", \"。\"},\n                new String[]{\"英\", \"首相\", \"与\", \"特朗普\", \"通\", \"电话\", \"讨论\", \"华为\", \"与\", \"苹果\", \"公司\", \"。\"}}));\n    }\n\n    @Test\n    void grammaticalErrorCorrection() throws IOException\n    {\n        prettyPrint(client.grammaticalErrorCorrection(new String[]{\"每个青年都应当有远大的报复。\", \"有的同学对语言很兴趣。\"}));\n    }\n\n    @Test\n    void languageIdentification() throws IOException\n    {\n        prettyPrint(client.languageIdentification(new String[]{\n                \"In 2021, HanLPv2.1 delivers state-of-the-art multilingual NLP techniques to production environment.\",\n                \"2021年、HanLPv2.1は次世代の最先端多言語NLP技術を本番環境に導入します。\",\n                \"2021年 HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。\",\n        }));\n    }\n\n    @Test\n    void sentimentAnalysis() throws IOException\n    {\n        prettyPrint(client.sentimentAnalysis(\n                \"“这是一部男人必看的电影。”人人都这么说。但单纯从性别区分，就会让这电影变狭隘。《肖申克的救赎》突破了男人电影的局限，通篇几乎充满令人难以置信的温馨基调，而电影里最伟大的主题是“希望”。 当我们无奈地遇到了如同肖申克一般囚禁了心灵自由的那种囹圄，我们是无奈的老布鲁克，灰心的瑞德，还是智慧的安迪？运用智慧，信任希望，并且勇敢面对恐惧心理，去打败它？ 经典的电影之所以经典，因为他们都在做同一件事——让你从不同的角度来欣赏希望的美好。\"\n        ));\n    }\n\n    void prettyPrint(Object object) throws JsonProcessingException\n    {\n        ObjectMapper mapper = new ObjectMapper();\n        System.out.println(mapper.writerWithDefaultPrettyPrinter().writeValueAsString(object));\n    }\n}"
  },
  {
    "path": "plugins/hanlp_restful_java/src/test/java/com/hankcs/hanlp/restful/MeaningRepresentationTest.java",
    "content": "package com.hankcs.hanlp.restful;\n\nimport com.fasterxml.jackson.core.JsonProcessingException;\nimport com.fasterxml.jackson.databind.ObjectMapper;\nimport com.hankcs.hanlp.restful.mrp.MeaningRepresentation;\nimport org.junit.jupiter.api.BeforeEach;\nimport org.junit.jupiter.api.Test;\n\nimport java.io.IOException;\nimport java.util.List;\nimport java.util.Map;\nimport java.util.Set;\n\nclass MeaningRepresentationTest\n{\n\n\n    @Test\n    void parseText() throws IOException\n    {\n        String json = \"[{\\\"id\\\": \\\"0\\\", \\\"input\\\": \\\"北京 大学 计算 语言学 研究所 和 富士通 研究 开发 中心 有限公司 ， 得到 了 人民日报社 新闻 信息 中心 的 语料库 。\\\", \\\"nodes\\\": [{\\\"id\\\": 0, \\\"label\\\": \\\"name\\\", \\\"properties\\\": [\\\"op1\\\", \\\"op2\\\"], \\\"values\\\": [\\\"北京\\\", \\\"大学\\\"], \\\"anchors\\\": [{\\\"from\\\": 0, \\\"to\\\": 2}, {\\\"from\\\": 3, \\\"to\\\": 5}]}, {\\\"id\\\": 1, \\\"label\\\": \\\"university\\\", \\\"anchors\\\": []}, {\\\"id\\\": 2, \\\"label\\\": \\\"name\\\", \\\"properties\\\": [\\\"op1\\\", \\\"op2\\\", \\\"op4\\\"], \\\"values\\\": [\\\"计算\\\", \\\"语言学\\\", \\\"\\\"], \\\"anchors\\\": [{\\\"from\\\": 6, \\\"to\\\": 8}, {\\\"from\\\": 9, \\\"to\\\": 12}, {\\\"from\\\": 13, \\\"to\\\": 16}]}, {\\\"id\\\": 3, \\\"label\\\": \\\"research-institute\\\", \\\"anchors\\\": []}, {\\\"id\\\": 4, \\\"label\\\": \\\"and\\\", \\\"anchors\\\": []}, {\\\"id\\\": 5, \\\"label\\\": \\\"name\\\", \\\"properties\\\": [\\\"op1\\\", \\\"op2\\\", \\\"op3\\\", \\\"op4\\\", \\\"op5\\\"], \\\"values\\\": [\\\"富士通\\\", \\\"研究\\\", \\\"开发\\\", \\\"中心\\\", \\\"有限公司\\\"], \\\"anchors\\\": [{\\\"from\\\": 19, \\\"to\\\": 22}, {\\\"from\\\": 23, \\\"to\\\": 25}, {\\\"from\\\": 26, \\\"to\\\": 28}, {\\\"from\\\": 29, \\\"to\\\": 31}, {\\\"from\\\": 32, \\\"to\\\": 36}]}, {\\\"id\\\": 6, \\\"label\\\": \\\"company\\\", \\\"anchors\\\": []}, {\\\"id\\\": 7, \\\"label\\\": \\\"得到-01\\\", \\\"anchors\\\": [{\\\"from\\\": 39, \\\"to\\\": 41}]}, {\\\"id\\\": 8, \\\"label\\\": \\\"了\\\", \\\"anchors\\\": [{\\\"from\\\": 42, \\\"to\\\": 43}]}, {\\\"id\\\": 9, \\\"label\\\": \\\"name\\\", \\\"properties\\\": [\\\"op1\\\"], \\\"values\\\": [\\\"人民日报社\\\"], \\\"anchors\\\": [{\\\"from\\\": 44, \\\"to\\\": 49}]}, {\\\"id\\\": 10, \\\"label\\\": \\\"organization\\\", \\\"anchors\\\": []}, {\\\"id\\\": 11, \\\"label\\\": \\\"name\\\", \\\"properties\\\": [\\\"op1\\\", \\\"op2\\\", \\\"op3\\\"], \\\"values\\\": [\\\"新闻\\\", \\\"信息\\\", \\\"中心\\\"], \\\"anchors\\\": [{\\\"from\\\": 50, \\\"to\\\": 52}, {\\\"from\\\": 53, \\\"to\\\": 55}, {\\\"from\\\": 56, \\\"to\\\": 58}]}, {\\\"id\\\": 12, \\\"label\\\": \\\"organization\\\", \\\"anchors\\\": []}, {\\\"id\\\": 13, \\\"label\\\": \\\"语料库\\\", \\\"anchors\\\": [{\\\"from\\\": 61, \\\"to\\\": 64}]}], \\\"edges\\\": [{\\\"source\\\": 7, \\\"target\\\": 8, \\\"label\\\": \\\"aspect\\\"}, {\\\"source\\\": 7, \\\"target\\\": 4, \\\"label\\\": \\\"arg0\\\"}, {\\\"source\\\": 10, \\\"target\\\": 9, \\\"label\\\": \\\"name\\\"}, {\\\"source\\\": 4, \\\"target\\\": 6, \\\"label\\\": \\\"op2\\\"}, {\\\"source\\\": 7, \\\"target\\\": 13, \\\"label\\\": \\\"arg1\\\"}, {\\\"source\\\": 6, \\\"target\\\": 5, \\\"label\\\": \\\"name\\\"}, {\\\"source\\\": 12, \\\"target\\\": 11, \\\"label\\\": \\\"name\\\"}, {\\\"source\\\": 3, \\\"target\\\": 2, \\\"label\\\": \\\"name\\\"}, {\\\"source\\\": 1, \\\"target\\\": 0, \\\"label\\\": \\\"name\\\"}, {\\\"source\\\": 13, \\\"target\\\": 12, \\\"label\\\": \\\"poss\\\"}, {\\\"source\\\": 4, \\\"target\\\": 3, \\\"label\\\": \\\"op1\\\"}, {\\\"source\\\": 12, \\\"target\\\": 9, \\\"label\\\": \\\"name\\\"}, {\\\"source\\\": 1, \\\"target\\\": 3, \\\"label\\\": \\\"part\\\"}], \\\"tops\\\": [7], \\\"framework\\\": \\\"amr\\\"}]\";\n        ObjectMapper mapper = new ObjectMapper();\n        MeaningRepresentation[] graphs = mapper.readValue(json, MeaningRepresentation[].class);\n        prettyPrint(graphs);\n    }\n\n\n    void prettyPrint(Object object) throws JsonProcessingException\n    {\n        ObjectMapper mapper = new ObjectMapper();\n        System.out.println(mapper.writerWithDefaultPrettyPrinter().writeValueAsString(object));\n    }\n}"
  },
  {
    "path": "plugins/hanlp_trie/README.md",
    "content": "# Trie interface and implementation for HanLP\n\n[中文](https://github.com/hankcs/HanLP/tree/doc-zh) | [1.x](https://github.com/hankcs/HanLP/tree/1.x) | [forum](https://bbs.hankcs.com/) | [docker](https://github.com/WalterInSH/hanlp-jupyter-docker)\n\nThe multilingual NLP library for researchers and companies, built on PyTorch and TensorFlow 2.x, for advancing state-of-the-art deep learning techniques in both academia and industry. HanLP was designed from day one to be efficient, user friendly and extendable. It comes with pretrained models for various human languages including English, Chinese and many others. Currently, HanLP 2.0 is in alpha stage with more killer features on the roadmap. Discussions are welcomed on our [forum](https://bbs.hankcs.com/), while bug reports and feature requests are reserved for GitHub issues. For Java users, please checkout the [1.x](https://github.com/hankcs/HanLP/tree/1.x) branch.\n\n## Installation\n\n```bash\npip install hanlp\n```\n\n\n## License\n\nHanLP is licensed under **Apache License 2.0**. You can use HanLP in your commercial products for free. We would appreciate it if you add a link to HanLP on your website.\n\n"
  },
  {
    "path": "plugins/hanlp_trie/hanlp_trie/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-11-29 17:48\nfrom .trie import Trie\nfrom .dictionary import DictInterface, TrieDict\n"
  },
  {
    "path": "plugins/hanlp_trie/hanlp_trie/dictionary.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-11-29 17:53\nfrom abc import ABC, abstractmethod\nfrom typing import List, Tuple, Any, Dict, Union, Sequence, Iterable, Optional\n\nfrom hanlp_common.configurable import Configurable\nfrom hanlp_common.reflection import classpath_of\nfrom hanlp_trie.trie import Trie\n\n\nclass DictInterface(ABC):\n    @abstractmethod\n    def tokenize(self, text: Union[str, Sequence[str]]) -> List[Tuple[int, int, Any]]:\n        \"\"\"Implement this method to tokenize a piece of text into a list of non-intersect spans, each span is a tuple\n        of ``(begin_offset, end_offset, label)``, where label is some properties related to this span and downstream\n        tasks have the freedom to define what kind of labels they want.\n\n        Args:\n            text: The text to be tokenized.\n\n        Returns:\n              A list of tokens.\n\n        \"\"\"\n        pass\n\n    def split(self, text: Union[str, Sequence[str]]) -> List[Tuple[int, int, Any]]:\n        \"\"\"Like the :meth:`str.split`, this method splits a piece of text into chunks by taking the keys in this\n        dictionary as delimiters. It performs longest-prefix-matching on text and split it whenever a longest key is\n        matched. Unlike the :meth:`str.split`, it inserts matched keys into the results list right after where they are\n        found. So that the text can be restored by joining chunks in the results list.\n\n        Args:\n            text: A piece of text.\n\n        Returns:\n            A list of chunks, each chunk is a span of ``(begin_offset, end_offset, label)``, where label is some\n            properties related to this span and downstream tasks.\n        \"\"\"\n        offset = 0\n        spans = []\n        for begin, end, label in self.tokenize(text):\n            if begin > offset:\n                spans.append(text[offset:begin])\n            spans.append((begin, end, label))\n            offset = end\n        if offset < len(text):\n            spans.append(text[offset:])\n        return spans\n\n\nclass TrieDict(Trie, DictInterface, Configurable):\n    def __init__(self, dictionary: Optional[Union[Dict[Iterable[str], Any], Iterable[str]]] = None) -> None:\n        r\"\"\"\n        A dict-like structure for fast custom dictionary strategies in tokenization and tagging. It is built with\n        a dict of key-value pairs or a set of strings. When a set is passed in, it will be turned into a dict where each\n        key is assigned with a boolean value ``True``.\n\n        Args:\n            dictionary: A custom dictionary of string-value pairs.\n        \"\"\"\n        super().__init__(dictionary)\n\n    def tokenize(self, text: Union[str, Sequence[str]]) -> List[Tuple[int, int, Any]]:\n        return self.parse_longest(text)\n\n    def split_batch(self, data: List[str]) -> Tuple[List[str], List[int], List[List[Tuple[int, int, Any]]]]:\n        \"\"\" A handy method to perform longest-prefix-matching on a batch of sentences. It tokenize each sentence, record\n        the chunks being either a key in the dict or a span outside of the dict. The spans are then packed into a new\n        batch and returned along with the following information:\n\n            - which sentence a span belongs to\n            - the matched keys along with their spans and values.\n\n        This method bridges the gap between statistical models and rule-based gazetteers.\n        It's used in conjunction with :meth:`~hanlp_trie.dictionary.TrieDict.merge_batch`.\n\n        Args:\n            data: A batch of sentences.\n\n        Returns:\n            A tuple of the new batch, the belonging information and the keys.\n        \"\"\"\n        new_data, new_data_belongs, parts = [], [], []\n        for idx, sent in enumerate(data):\n            parts.append([])\n            found = self.tokenize(sent)\n            if found:\n                pre_start = 0\n                for start, end, info in found:\n                    if start > pre_start:\n                        new_data.append(sent[pre_start:start])\n                        new_data_belongs.append(idx)\n                    pre_start = end\n                    parts[idx].append((start, end, info))\n                if pre_start != len(sent):\n                    new_data.append(sent[pre_start:])\n                    new_data_belongs.append(idx)\n            else:\n                new_data.append(sent)\n                new_data_belongs.append(idx)\n        return new_data, new_data_belongs, parts\n\n    @staticmethod\n    def merge_batch(data, new_outputs, new_data_belongs, parts):\n        \"\"\" A helper method to merge the outputs of split batch back by concatenating the output per span with the key\n        used to split it. It's used in conjunction with :meth:`~hanlp_trie.dictionary.TrieDict.split_batch`.\n\n        Args:\n            data: Split batch.\n            new_outputs: Outputs of the split batch.\n            new_data_belongs: Belonging information.\n            parts: The keys.\n\n        Returns:\n            Merged outputs.\n        \"\"\"\n        outputs = []\n        segments = []\n        for idx in range(len(data)):\n            segments.append([])\n        for o, b in zip(new_outputs, new_data_belongs):\n            dst = segments[b]\n            dst.append(o)\n        for s, p, sent in zip(segments, parts, data):\n            s: list = s\n            if p:\n                dst = []\n                offset = 0\n                for start, end, info in p:\n                    while offset < start:\n                        head = s.pop(0)\n                        offset += sum(len(token) for token in head)\n                        dst += head\n                    if isinstance(info, list):\n                        dst += info\n                    elif isinstance(info, str):\n                        dst.append(info)\n                    else:\n                        dst.append(sent[start:end])\n                    offset = end\n                if s:\n                    assert len(s) == 1\n                    dst += s[0]\n                outputs.append(dst)\n            else:\n                outputs.append(s[0])\n        return outputs\n\n    @property\n    def config(self):\n        return {\n            'classpath': classpath_of(self),\n            'dictionary': dict(self.items())\n        }\n\n\nclass TupleTrieDict(TrieDict):\n    def __init__(self, dictionary: Optional[Union[Dict[Iterable[str], Any], Iterable[str]]] = None) -> None:\n        r\"\"\"\n        A dict-like structure for fast custom dictionary strategies in tokenization and tagging. It is built with\n        a dict of key-value pairs or a set of strings. When a set is passed in, it will be turned into a dict where each\n        key is assigned with a boolean value ``True``. In comparison to ``TrieDict``, ``TupleTrieDict`` additionally\n        supports serializing/deserializing tuple-as-keys dict.\n\n        Args:\n            dictionary: A custom dictionary of string-value pairs.\n        \"\"\"\n        if isinstance(dictionary, list) and dictionary and isinstance(dictionary[0], (list, tuple)):\n            _d = dict()\n            for k, v in dictionary:\n                _d[tuple(k)] = v\n            dictionary = _d\n        super().__init__(dictionary)\n\n    @property\n    def config(self):\n        return {\n            'classpath': classpath_of(self),\n            'dictionary': list(self.items(prefix=()))\n        }\n\n    def parse_longest(self, text: Sequence[str]) -> List[Tuple[int, int, Any]]:\n        \"\"\"Longest-prefix-matching which tries to match the longest keyword sequentially from the head of the text till\n        its tail. By definition, the matches won't overlap with each other.\n\n        Args:\n            text: A piece of text. In HanLP's design, it doesn't really matter whether this is a str or a list of str.\n                The trie will transit on either types properly, which means a list of str simply defines a list of\n                transition criteria while a str defines each criterion as a character.\n\n        Returns:\n            A tuple of ``(begin, end, value)``.\n\n        \"\"\"\n        found = []\n        i = 0\n        while i < len(text):\n            state = self.transit(text[i:i + 1])\n            if state:\n                to = i + 1\n                end = to\n                value = state._value\n                for to in range(i + 1, len(text)):\n                    state = state.transit(text[to:to + 1])\n                    if not state:\n                        break\n                    if state._value is not None:\n                        value = state._value\n                        end = to + 1\n                if value is not None:\n                    found.append((i, end, value))\n                    i = end - 1\n            i += 1\n        return found\n"
  },
  {
    "path": "plugins/hanlp_trie/hanlp_trie/trie.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-01-04 23:46\nfrom typing import Dict, Any, List, Tuple, Sequence, Union, Iterable, Optional\n\n\nclass Node(object):\n    def __init__(self, value=None) -> None:\n        \"\"\"A node in a trie tree.\n\n        Args:\n            value: The value associated with this node.\n        \"\"\"\n        self._children = {}\n        self._value = value\n\n    def _get_or_add_child(self, char):\n        child = self._children.get(char)\n        if child is None:\n            child = Node(None)\n            self._children[char] = child\n        return child\n\n    def transit(self, key):\n        \"\"\"Transit the state of a Deterministic Finite Automata (DFA) with key.\n\n        Args:\n            key: A sequence of criterion (tokens or characters) used to transit to a new state.\n\n        Returns:\n            A new state if the transition succeeded, otherwise ``None``.\n\n        \"\"\"\n        state = self\n        for char in key:\n            state = state._children.get(char)\n            if state is None:\n                break\n        return state\n\n    def _walk(self, prefix: Union[str, tuple], ordered=False):\n        for char, child in sorted(self._children.items()) if ordered else self._children.items():\n            prefix_new = prefix + (char if isinstance(prefix, str) else (char,))\n            if child._value:\n                yield prefix_new, child._value\n            yield from child._walk(prefix_new)\n\n\nclass Trie(Node):\n    def __init__(self, tokens: Optional[Union[Dict[str, Any], Iterable[str]]] = None) -> None:\n        \"\"\"A referential implementation of the trie (:cite:`10.1145/1457838.1457895`) structure. It stores a dict by\n        assigning each key/value pair a :class:`~hanlp_trie.trie.Node` in a trie tree. It provides get/set/del/items\n        methods just like a :class:`dict` does. Additionally, it also provides longest-prefix-matching and keywords\n        lookup against a piece of text, which are very helpful in rule-based Natural Language Processing.\n\n        Args:\n            tokens: A set of keys or a dict mapping.\n        \"\"\"\n        super().__init__()\n        self._size = 0\n        if tokens:\n            if isinstance(tokens, dict):\n                for k, v in tokens.items():\n                    self[k] = v\n            else:\n                for k in tokens:\n                    self[k] = True\n\n    def __contains__(self, key):\n        return self[key] is not None\n\n    def __getitem__(self, key):\n        state = self.transit(key)\n        if state is None:\n            return None\n        return state._value\n\n    def __setitem__(self, key, value):\n        state = self\n        for char in key[:-1]:\n            state = state._get_or_add_child(char)\n\n        leaf = state._get_or_add_child(key[-1])\n        if leaf._value is None:\n            self._size += 1\n        leaf._value = value\n\n    def __delitem__(self, key):\n        state = self.transit(key)\n        if state is not None:\n            state._value = None\n            self._size -= 1\n\n    def update(self, dic: Dict[str, Any]):\n        for k, v in dic.items():\n            self[k] = v\n        return self\n\n    def parse(self, text: Sequence[str]) -> List[Tuple[int, int, Any]]:\n        \"\"\"Keywords lookup which takes a piece of text as input, and lookup all occurrences of keywords in it. These\n        occurrences can overlap with each other.\n\n        Args:\n            text: A piece of text. In HanLP's design, it doesn't really matter whether this is a str or a list of str.\n                The trie will transit on either types properly, which means a list of str simply defines a list of\n                transition criteria while a str defines each criterion as a character.\n\n        Returns:\n            A tuple of ``(begin, end, value)``.\n        \"\"\"\n        found = []\n        for i in range(len(text)):\n            state = self\n            for j in range(i, len(text)):\n                state = state.transit(text[j])\n                if state:\n                    if state._value is not None:\n                        found.append((i, j + 1, state._value))\n                else:\n                    break\n        return found\n\n    def parse_longest(self, text: Sequence[str]) -> List[Tuple[int, int, Any]]:\n        \"\"\"Longest-prefix-matching which tries to match the longest keyword sequentially from the head of the text till\n        its tail. By definition, the matches won't overlap with each other.\n\n        Args:\n            text: A piece of text. In HanLP's design, it doesn't really matter whether this is a str or a list of str.\n                The trie will transit on either types properly, which means a list of str simply defines a list of\n                transition criteria while a str defines each criterion as a character.\n\n        Returns:\n            A tuple of ``(begin, end, value)``.\n\n        \"\"\"\n        found = []\n        i = 0\n        while i < len(text):\n            state = self.transit(text[i])\n            if state:\n                to = i + 1\n                end = to\n                value = state._value\n                for to in range(i + 1, len(text)):\n                    state = state.transit(text[to])\n                    if not state:\n                        break\n                    if state._value is not None:\n                        value = state._value\n                        end = to + 1\n                if value is not None:\n                    found.append((i, end, value))\n                    i = end - 1\n            i += 1\n        return found\n\n    def items(self, ordered=False, prefix=''):\n        yield from self._walk(prefix, ordered)\n\n    def __len__(self):\n        return self._size\n\n    def __bool__(self):\n        return bool(len(self))\n"
  },
  {
    "path": "plugins/hanlp_trie/setup.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 19:26\nfrom os.path import abspath, join, dirname\nfrom setuptools import find_packages, setup\n\nthis_dir = abspath(dirname(__file__))\nwith open(join(this_dir, 'README.md'), encoding='utf-8') as file:\n    long_description = file.read()\n\nsetup(\n    name='hanlp_trie',\n    version='0.0.5',\n    description='HanLP: Han Language Processing',\n    long_description=long_description,\n    long_description_content_type=\"text/markdown\",\n    url='https://github.com/hankcs/HanLP',\n    author='hankcs',\n    author_email='hankcshe@gmail.com',\n    license='Apache License 2.0',\n    classifiers=[\n        'Intended Audience :: Science/Research',\n        'Intended Audience :: Developers',\n        \"Development Status :: 3 - Alpha\",\n        'Operating System :: OS Independent',\n        \"License :: OSI Approved :: Apache Software License\",\n        'Programming Language :: Python :: 3 :: Only',\n        'Topic :: Scientific/Engineering :: Artificial Intelligence',\n        \"Topic :: Text Processing :: Linguistic\"\n    ],\n    keywords='corpus,machine-learning,NLU,NLP',\n    packages=find_packages(exclude=['docs', 'tests*']),\n    include_package_data=True,\n    install_requires=[\n        'hanlp_common'\n    ],\n    python_requires='>=3.6',\n)\n"
  },
  {
    "path": "plugins/hanlp_trie/tests/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2020-11-29 18:05\n"
  },
  {
    "path": "plugins/hanlp_trie/tests/test_trie.py",
    "content": "import unittest\n\nfrom hanlp_trie import Trie\n\n\nclass TestTrie(unittest.TestCase):\n    def build_small_trie(self):\n        return Trie({'商品': 'goods', '和': 'and', '和服': 'kimono', '服务': 'service', '务': 'business'})\n\n    def assert_results_valid(self, text, results, trie):\n        for begin, end, value in results:\n            self.assertEqual(value, trie[text[begin:end]])\n\n    def test_parse(self):\n        trie = self.build_small_trie()\n        text = '商品和服务'\n        parse_result = trie.parse(text)\n        self.assert_results_valid(text, parse_result, trie)\n        self.assertEqual([(0, 2, 'goods'),\n                          (2, 3, 'and'),\n                          (2, 4, 'kimono'),\n                          (3, 5, 'service'),\n                          (4, 5, 'business')],\n                         parse_result)\n\n    def test_parse_longest(self):\n        trie = self.build_small_trie()\n        text = '商品和服务'\n        parse_longest_result = trie.parse_longest(text)\n        self.assert_results_valid(text, parse_longest_result, trie)\n        self.assertEqual([(0, 2, 'goods'), (2, 4, 'kimono'), (4, 5, 'business')],\n                         parse_longest_result)\n\n    def test_items(self):\n        trie = self.build_small_trie()\n        items = list(trie.items())\n        self.assertEqual([('商品', 'goods'), ('和', 'and'), ('和服', 'kimono'), ('服务', 'service'), ('务', 'business')], items)\n\n    def test_len(self):\n        trie = self.build_small_trie()\n        self.assertEqual(len(trie), 5)\n        trie['和'] = '&'\n        self.assertEqual(len(trie), 5)\n        del trie['和']\n        self.assertEqual(len(trie), 4)\n        trie['和'] = '&'\n        self.assertEqual(len(trie), 5)\n\n\nif __name__ == '__main__':\n    unittest.main()\n"
  },
  {
    "path": "plugins/hanlp_trie/tests/test_trie_dict.py",
    "content": "import unittest\n\nfrom hanlp_trie import TrieDict\n\n\nclass TestTrieDict(unittest.TestCase):\n\n    def setUp(self) -> None:\n        super().setUp()\n        self.text = '第一个词语很重要，第二个词语也很重要'\n        self.trie_dict = TrieDict({'重要': 'important'})\n\n    def test_tokenize(self):\n        self.assertEqual([(6, 8, 'important'), (16, 18, 'important')], self.trie_dict.tokenize(self.text))\n\n    def test_split_batch(self):\n        data = [self.text]\n        new_data, new_data_belongs, parts = self.trie_dict.split_batch(data)\n        predictions = [list(x) for x in new_data]\n        self.assertSequenceEqual(\n            [['第', '一', '个', '词', '语', '很', 'important', '，', '第', '二', '个', '词', '语', '也', '很', 'important']],\n            self.trie_dict.merge_batch(data, predictions, new_data_belongs, parts))\n\n    def test_tokenize_2(self):\n        t = TrieDict({'次世代', '生产环境'})\n        self.assertSequenceEqual(t.tokenize('2021年HanLPv2.1为生产环境带来次世代最先进的多语种NLP技术。'),\n                                 [(15, 19, True), (21, 24, True)])\n\n    def test_empty_dict(self):\n        trie_dict = TrieDict()\n        self.assertFalse(bool(trie_dict))\n        trie_dict['one'] = 1\n        self.assertTrue(bool(trie_dict))\n        del trie_dict['one']\n        self.assertFalse(bool(trie_dict))\n\n\nif __name__ == '__main__':\n    unittest.main()\n"
  },
  {
    "path": "setup.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-12-28 19:26\nimport sys\nfrom os.path import abspath, join, dirname\nfrom setuptools import find_packages, setup\n\nthis_dir = abspath(dirname(__file__))\nwith open(join(this_dir, 'README.md'), encoding='utf-8') as file:\n    long_description = file.read()\nversion = {}\nwith open(join(this_dir, \"hanlp\", \"version.py\")) as fp:\n    exec(fp.read(), version)\n\nFASTTEXT = 'fasttext-wheel==0.9.2'\nsys_version_info = sys.version_info\n\nTOKENIZERS = []\nif (sys_version_info.major, sys_version_info.minor) == (3, 6) and sys.platform in {'darwin', 'win32'}:\n    TOKENIZERS = ['tokenizers==0.10.3']\n\nextras_require = {\n    'amr': [\n        'penman==1.2.1',\n        'networkx>=2.5.1',\n        'perin-parser>=0.0.12',\n    ],\n    'fasttext': [FASTTEXT],\n    'tf': [FASTTEXT, 'tensorflow>=2.6.0,<2.14']\n}\nextras_require['full'] = list(set(sum(extras_require.values(), [])))\n\nsetup(\n    name='hanlp',\n    version=version['__version__'],\n    description='HanLP: Han Language Processing',\n    long_description=long_description,\n    long_description_content_type=\"text/markdown\",\n    url='https://github.com/hankcs/HanLP',\n    author='hankcs',\n    author_email='hankcshe@gmail.com',\n    license='Apache License 2.0',\n    classifiers=[\n        'Intended Audience :: Science/Research',\n        'Intended Audience :: Developers',\n        \"Development Status :: 4 - Beta\",\n        'Operating System :: OS Independent',\n        \"License :: OSI Approved :: Apache Software License\",\n        'Programming Language :: Python :: 3.6',\n        'Programming Language :: Python :: 3.7',\n        'Programming Language :: Python :: 3.8',\n        'Programming Language :: Python :: 3.9',\n        'Programming Language :: Python :: 3.10',\n        'Topic :: Scientific/Engineering :: Artificial Intelligence',\n        \"Topic :: Text Processing :: Linguistic\"\n    ],\n    keywords='corpus,machine-learning,NLU,NLP',\n    packages=find_packages(exclude=['docs', 'tests*']),\n    include_package_data=True,\n    install_requires=[\n        'termcolor',\n        'pynvml',\n        'toposort==1.5',\n        'transformers>=4.1.1',\n        'sentencepiece>=0.1.91',  # Essential for tokenization_bert_japanese\n        'torch>=1.6.0',\n        'hanlp-common>=0.0.22',\n        'hanlp-trie>=0.0.4',\n        'hanlp-downloader',\n        *TOKENIZERS,\n    ],\n    extras_require=extras_require,\n    python_requires='>=3.6',\n)\n"
  },
  {
    "path": "tests/__init__.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2019-06-13 23:43\nimport os\n\nroot = os.path.abspath(os.path.join(os.path.dirname(__file__), os.pardir))\n\n\ndef cdroot():\n    \"\"\"\n    cd to project root, so models are saved in the root folder\n    \"\"\"\n    os.chdir(root)\n"
  },
  {
    "path": "tests/test_config_tracker.py",
    "content": "import unittest\n\nfrom hanlp.common.structure import ConfigTracker\n\n\nclass MyClass(ConfigTracker):\n    def __init__(self, i_need_this='yes') -> None:\n        super().__init__(locals())\n\n\nclass TestConfigTracker(unittest.TestCase):\n    def test_init(self):\n        obj = MyClass()\n        self.assertEqual(obj.config.get('i_need_this', None), 'yes')\n\n\nif __name__ == '__main__':\n    unittest.main()\n"
  },
  {
    "path": "tests/test_mtl.py",
    "content": "import hanlp\nimport unittest\nfrom multiprocessing.dummy import Pool\nfrom hanlp_common.document import Document\n\nmtl = hanlp.load(hanlp.pretrained.mtl.CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ELECTRA_SMALL_ZH, devices=-1)\n\n\ndef tokenize(mtl, text):\n    return mtl(text, tasks='tok/fine')['tok/fine']\n\n\nclass TestMultiTaskLearning(unittest.TestCase):\n    def test_mtl_single_sent(self):\n        doc: Document = mtl('商品和服务')\n        self.assertSequenceEqual(doc['tok/fine'], [\"商品\", \"和\", \"服务\"])\n\n    def test_mtl_multiple_sents(self):\n        doc: Document = mtl(['商品和服务', '研究生命'])\n        self.assertSequenceEqual(doc['tok/fine'], [\n            [\"商品\", \"和\", \"服务\"],\n            [\"研究\", \"生命\"]\n        ])\n\n    def test_mtl_empty_str(self):\n        mtl('')\n        mtl(' ')\n        mtl([''])\n        mtl([' '])\n        mtl(['', ' '])\n        mtl(['', ' ', 'good'])\n        mtl([[]], skip_tasks='tok*')\n\n    def test_skip_tok(self):\n        pre_tokenized_sents = [\n            [\"商品和服务\", '一个', '词'],\n            [\"研究\", \"生命\"]\n        ]\n        doc: Document = mtl(pre_tokenized_sents, skip_tasks='tok*')\n        self.assertSequenceEqual(doc['tok'], pre_tokenized_sents)\n\n    def test_sdp_as_the_first_task(self):\n        doc: Document = mtl(['人', '吃', '鱼'], tasks='sdp', skip_tasks='tok*')\n        self.assertDictEqual(\n            doc.to_dict(),\n            {\n                \"sdp\": [\n                    [(2, \"Agt\")],\n                    [(0, \"Root\")],\n                    [(2, \"Pat\")]\n                ],\n                \"tok\": [\n                    \"人\",\n                    \"吃\",\n                    \"鱼\"\n                ]\n            }\n        )\n\n    def test_threading(self):\n        num_proc = 8\n        with Pool(num_proc) as pool:\n            results = pool.starmap(tokenize, [(mtl, '商品和服务')] * num_proc)\n            self.assertSequenceEqual(results, [['商品', '和', '服务']] * num_proc)\n\n    def test_emoji(self):\n        self.assertSequenceEqual(mtl('( ͡° ͜ʖ ͡ °)你好', tasks='tok/fine')['tok/fine'],\n                                 [\"(\", \" ͡\", \"°\", \" ͜\", \"ʖ\", \" ͡ \", \"°\", \")\", \"你\", \"好\"])\n        mtl['tok/fine'].dict_combine = {'( ͡° ͜ʖ ͡ °)'}\n        self.assertSequenceEqual(mtl('( ͡° ͜ʖ ͡ °)你好', tasks='tok/fine')['tok/fine'],\n                                 [\"( ͡° ͜ʖ ͡ °)\", \"你\", \"好\"])\n\n    def test_unicode_removed_by_hf(self):\n        self.assertSequenceEqual(mtl('͡', tasks='tok/fine')['tok/fine'], ['͡'])\n\n    def test_space(self):\n        task = 'tok/fine'\n        doc: Document = mtl('商品 和服务', tasks=task)\n        self.assertSequenceEqual(doc[task], [\"商品\", \"和\", \"服务\"])\n        mtl[task].dict_combine = {('iPad', 'Pro'), '2个空格'}\n        self.assertSequenceEqual(mtl(\"如何评价iPad Pro ？iPad  Pro有2个空格\", tasks=task)[task],\n                                 ['如何', '评价', 'iPad Pro', '？', 'iPad  Pro', '有', '2个空格'])\n\n    def test_transform(self):\n        task = 'tok/fine'\n        mtl[task].dict_force = {'用户ID'}\n        self.assertSequenceEqual(mtl(\"我的用户ID跟你的用户id不同\", tasks=task)[task],\n                                 ['我', '的', '用户ID', '跟', '你', '的', '用户', 'id', '不同'])\n\n    def test_tok_offset(self):\n        task = 'tok/fine'\n        tok = mtl[task]\n        tok.config.output_spans = True\n        tok.dict_force = None\n        tok.dict_combine = None\n        sent = '我先去看医生'\n\n        for t, b, e in mtl(sent, tasks=task)[task]:\n            self.assertEqual(t, sent[b:e])\n\n        tok.dict_combine = {'先去'}\n        for t, b, e in mtl(sent, tasks=task)[task]:\n            self.assertEqual(t, sent[b:e])\n\n        tok.config.output_spans = False\n        tok.dict_force = None\n        tok.dict_combine = None\n\n\nif __name__ == '__main__':\n    unittest.main()\n"
  },
  {
    "path": "tests/test_pipeline.py",
    "content": "import unittest\nimport hanlp\n\n\nclass TestPipeLine(unittest.TestCase):\n    def test_copy(self):\n        pipe = hanlp.pipeline().append(hanlp.utils.rules.split_sentence)\n        copied_pipe = pipe.copy()\n        test_text = \"今天天气真好。我要去散步。\"\n        assert pipe is not copied_pipe\n        copied_pipe.append(lambda sent: \"\".join(sent))\n        assert pipe(test_text) != copied_pipe(test_text)\n\nif __name__ == '__main__':\n    unittest.main()\n"
  },
  {
    "path": "tests/test_rules.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2022-03-22 17:17\nimport unittest\n\nfrom hanlp.utils.rules import split_sentence\n\n\nclass TestRules(unittest.TestCase):\n    def test_eos(self):\n        self.assertListEqual(list(split_sentence('叶')), ['叶'])\n        self.assertListEqual(list(split_sentence('他说：“加油。”谢谢')), ['他说：“加油。”', '谢谢'])\n        self.assertListEqual(list(split_sentence('Go to hankcs.com. Yes.')), ['Go to hankcs.com.', 'Yes.'])\n\n\nif __name__ == '__main__':\n    unittest.main()\n"
  },
  {
    "path": "tests/test_string_util.py",
    "content": "# -*- coding:utf-8 -*-\n# Author: hankcs\n# Date: 2022-03-22 17:17\nimport unittest\n\nfrom hanlp.utils.string_util import possible_tokenization\n\n\nclass TestStringUtility(unittest.TestCase):\n    def test_enumerate_tokenization(self):\n        text = '商品和服务'\n        toks = possible_tokenization(text)\n        assert len(set(toks)) == 2 ** (len(text) - 1)\n        for each in toks:\n            assert ''.join(each) == text\n\n\nif __name__ == '__main__':\n    unittest.main()\n"
  }
]